Implementing Device To Cloud Messaging Using IoT Hub

|  Posted: April 11, 2017  |  Categories: Microsoft Azure

When sending messages from millions of devices, we need to have a robust solution, which allows us to easily connect our devices, and can handle large amounts of messages.

To make sure our devices are running as we expect them to, we will also want to be notified of our devices states. IoT Hub, built on technologies like Event Hubs and Service Bus, gives us these capabilities and more, while allowing us to scale up and down as needed. This is the second post in my IoT Hub series, and here we will see the possibilities we have to send messages to IoT Hub and process them. We will continue on the scenario we set in the first blog post of the series, and start receiving sensor readings about the temperature, RPM and alerts of the engines on our simulated ship.

IoT Hub Architecture

Prerequisites

In this blog post, we will be using other Azure technologies as well. When implementing the scenario described here, please make sure te prepare the following items.

  • Service Bus namespace including a queue used for routes
  • Azure Storage account including a blob container linked to your IoT Hub as described here. Make sure file notifications are enabled.

Device To Cloud

When looking at a product like IoT Hub, one of the most important features is the capability to easily send messages like telemetry data, sensor readings and such to it. IoT Hub implements device-to-cloud messaging using a streaming messaging pattern, allowing a high volume of events to be processed by multiple readers. Open the project for the simulated engine we created in the first blog post, and add a class representing our engine readings.

Add EngineReadings Class

 

This class will be sent to in the message to IoT Hub, and will contain the readings from our engine and the properties to identify for which engine the message was created.

public class EngineReading
{
    public string ShipName;
    public string SerialNumber;
    public DateTime CreatedDateTime;
    public int RPM;
    public int Temperature;
}

To make it easy to change the serial number and the name of the ship, we will store these in the settings of the application.

Create Settings

 

We will now create the method which will create some dummy engine reading (which in a real life scenario would of course be read from your actual sensors), and send this to IoT Hub.

private async static void SendEngineReading(object state)
{
	// Create dummy reading
	var engineReading = new EngineReading
	{
		SerialNumber = Settings.Default.EngineSerialNumber,
		ShipName = Settings.Default.Ship,
		CreatedDateTime = DateTime.Now,
		RPM = 500 + new Random().Next(-30, 30),
		Temperature = 450 + new Random().Next(-50, 70)
	};

	// Send message to IoT Hub
	await client.SendEventAsync(new Message(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(engineReading))));
}

And finally we will need to add a timer which triggers sending the message. For this we will add the following line to the Main method, just after the SetDesiredPropertyUpdateCallback call we created in the first blog post.

var timer = new Timer(SendEngineReading, null, 0, 5000);

Consumer Groups

The messages we will now send, will all be handled by IoT Hub, and sent to all its consumer groups. Each consumer group can be read by a single reader, or by a set of readers. In case a set of readers is used, each message will only be read by a single reader in the set. This means that in case we want to send a message to multiple readers, for example both to Stream Analytics and to a custom implementation using the Event Processor Host library, we will need to create multiple consumer groups in IoT Hub. Luckily this can be done easily using the portal. Open your IoT Hub, go to the Endpoints blade, open the Events endpoint and add consumer groups for all the readers (sets) you want.

Add Consumer Groups

 

Routes

We now have set up the messaging to IoT Hub, and can process our messages in a streaming way, but what if we want to have certain messages have priority over other messages? This is especially powerful when handling large amounts of events, which might take some time to be processed in a streaming way, when we have certain alerts which should be processed as quickly as possible. Fortunately IoT Hub knows the concept of routes, which means we can filter our messages on properties set on the message, and send these to an alternate endpoint, for which we can use Event Hubs, Service Bus Queues and Service Bus Topics. We will start by updating our code in SendEngineReading to set a property in case the current temperature of the engine is over the maximum temperature we specified.

private async static void SendEngineReading(object state)
{
    // Create dummy reading
    var engineReading = new EngineReading
    {
        SerialNumber = Settings.Default.EngineSerialNumber,
        ShipName = Settings.Default.Ship,
        CreatedDateTime = DateTime.Now,
        RPM = 500 + new Random().Next(-30, 30),
        Temperature = 450 + new Random().Next(-50, 70)
    };

    var messageToSend = new Message(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(engineReading)));

    if (engineReading.Temperature > maximumTemperature)
    {
        // Add alert property
        messageToSend.Properties.Add("alert", "HighTemperature");
    }

    // Send message to IoT Hub
    await client.SendEventAsync(messageToSend);
}

This will make sure a property is added in case of the engine’s temperature being too high, so lets add a route to IoT Hub to send these messages to a Service Bus Queue (as indicated in the prerequisites). We will first need to create a custom endpoint to our queue.

Add Custom Endpoint

 

Once the endpoint is created, we will add a route which uses the following filter to send specific messages to this new endpoint.

alert="HighTemperature"

Add Route For Alert

Our messages with the alert will now be routed to the Service Bus Queue, where we can pick it up for further processing.

Set Reported Properties

In the first blog post of this series we already worked with desired properties, which are used to communicate a desired configuration from IoT Hub to the device. In this post, we will look into its counterpart reported properties, which are used to send actual configurations from a device to IoT Hub. This is especially useful for devices which may not always have a connection to the internet, as we can use the desired properties to start a process as soon as the device comes online, and use the reported properties to be notified of changes in the progress.

In the first blog post in this series we created the EngineManagement application, which we will expand on in this post to initiate a firmware update. We will use the a reported property to report back the progress of of updating the firmware. For this we will first create the following method in the SimulatedEngine program, which will be used to update the reported properties back to IoT Hub.

private static async void UpdateReportedProperties(Guid configId, int version, string status)
{
    var reportedProperties = new TwinCollection
    {
        ["firmwareConfig"] = new
        {
            configId = configId,
            version = version,
            status = status
        }
    };
    await client.UpdateReportedPropertiesAsync(reportedProperties);
}

Next we will create a new method to retrieve the desired properties for the firmware, simulate updating the firmware, and reported properties to report the progress to the backend. To simulate a long running process we will add a delay, to show how we can use different statuses to let our backend know how much progress has been made.

private static async void UpdateFirmware()
{
    var deviceTwin = await client.GetTwinAsync();

    int firmwareVersion = deviceTwin.Properties.Desired["firmwareConfig"]["firmwareVersion"];
    string downloadLink = deviceTwin.Properties.Desired["firmwareConfig"]["downloadLink"];
    Guid configId = deviceTwin.Properties.Desired["firmwareConfig"]["configId"];

    // Make sure desired properties section exists
    if (!deviceTwin.Properties.Reported.Contains("firmwareConfig"))
    {
        UpdateReportedProperties(Guid.Empty, 0, "Initial");
        deviceTwin = await client.GetTwinAsync();
    }

    // Check if desired properties are updated
    if (deviceTwin.Properties.Reported["firmwareConfig"]["configId"] == configId)
    {
        return;
    }

    // Simulate firmware upgrade, reporting back the progress
    UpdateReportedProperties(configId, firmwareVersion, "Downloading");
    Thread.Sleep(10000);
    UpdateReportedProperties(configId, firmwareVersion, "Applying");
    Thread.Sleep(20000);
    UpdateReportedProperties(configId, firmwareVersion, "Success");
}

Since an update of the desired properties does not trigger the callback in case the device was offline when the update was done, we should do this check every time the application starts, so make dure to call this new method from the Main method. Since we will also want to call this method in case the callback does get triggered, which happens in case the desired properties get updated when the device is online, we will change the DesiredPropertyUpdated to the following.

private static Task DesiredPropertyUpdated(TwinCollection desiredProperties, object userContext)
{
    // Get maximum temperature
    maximumTemperature = desiredProperties["temperatureConfig"]["maximumTemperature"];
    UpdateFirmware();
    return null;
}

Next go to the EngineManagement project we created in the first blog post of this series, and update the form to the following.

Add Firmware Controls

For the event handler of the Apply button, we will implement the following method, which updates the desired properties, and tracks the progress.

private async void ButtonApplyFirmware_Click(object sender, EventArgs e)
{
    var deviceTwin = await registry.GetTwinAsync(comboBoxSerialNumber.Text);

    if (deviceTwin == null)
    {
        groupBoxFirmware.Enabled = false;
        labelProgress.Text = "Invalid device!";
    }

    var configId = Guid.NewGuid();

    // Create patch which updates the device twin
    var patch = new
    {
        properties = new
        {
            desired = new
            {
                // Set firmware
                firmwareConfig = new
                {
                    configId = configId,
                    firmwareVersion = numericUpDownVersion.Value,
                    downloadLink = textBoxDownloadLink.Text
                }
            }
        }
    };

    await registry.UpdateTwinAsync(comboBoxSerialNumber.Text, JsonConvert.SerializeObject(patch), deviceTwin.ETag);

    while (true)
    {
        if (deviceTwin.Properties.Reported.Contains("firmwareConfig")
            && deviceTwin.Properties.Reported["firmwareConfig"]["configId"] == configId)
        {
            var status = deviceTwin.Properties.Reported["firmwareConfig"]["status"];
            labelProgress.Text = $"Progress: {status}";

            if(status == "Success")
            {
                break;
            }
        }

        Thread.Sleep(1000);
    }
}

File Upload

In most communication from your devices we will be sending relativly small messages to IoT Hub, like telemetry data, readings, etc. However sometimes we might have a need to send larger files, like images or logfiles. For these requirements, IoT Hub provides file upload capabilities, which we will use in our scenario to upload a log file after having updated the firmware of the device. Start by adding the following method in the program for our simulated engine.

private static async void UploadLogFile()
{
    using (var sourceData = new FileStream(logFilePath, FileMode.Open))
    {
        await client.UploadToBlobAsync($"{DateTime.Now.ToString("yyyyMMddHHmmss")}.log", sourceData);
    }
}

 

Now update the UpdateFirmware method to do the actual logging, and call the UploadLogFile method once done to upload the log.

private static async void UpdateFirmware()
{
    var deviceTwin = await client.GetTwinAsync();

    int firmwareVersion = deviceTwin.Properties.Desired["firmwareConfig"]["firmwareVersion"];
    string downloadLink = deviceTwin.Properties.Desired["firmwareConfig"]["downloadLink"];
    Guid configId = deviceTwin.Properties.Desired["firmwareConfig"]["configId"];

    // Make sure desired properties section exists
    if (!deviceTwin.Properties.Reported.Contains("firmwareConfig"))
    {
        UpdateReportedProperties(Guid.Empty, 0, "Initial");
        deviceTwin = await client.GetTwinAsync();
    }

    // Check if desired properties are updated
    if (deviceTwin.Properties.Reported["firmwareConfig"]["configId"] == configId)
    {
        return;
    }

    // Simulate firmware upgrade, reporting back the progress
    File.WriteAllText(logFilePath, $"{DateTime.Now}: Download from {downloadLink} started{Environment.NewLine}");
    UpdateReportedProperties(configId, firmwareVersion, "Downloading");
    Thread.Sleep(10000);
    File.AppendAllText(logFilePath, $"{DateTime.Now}: Download from {downloadLink} finished{Environment.NewLine}");
    File.AppendAllText(logFilePath, $"{DateTime.Now}: Applying firmware update{Environment.NewLine}");
    UpdateReportedProperties(configId, firmwareVersion, "Applying");
    Thread.Sleep(20000);
    File.AppendAllText(logFilePath, $"{DateTime.Now}: Firmware update finished successfully");
    UpdateReportedProperties(configId, firmwareVersion, "Success");

    UploadLogFile();
}

Conclusion

In this post, we have created an application we have implemented communication from our devices to IoT Hub. We can now send device-to-cloud messages, which will be routed to IoT Hub’s internal Event Hub or to a Service Bus queue depending on the properties set on the message. In this post we also implemented reported properties, which we can use to asynchronously report our device’s properties back to our backend. And to top it all of, we can send large files to IoT Hub by using the file upload capabilities. The complete code of the first two blog posts in this series can be downloaded here.

IoT Hub Blog Series

In case you missed the other articles from this IoT Hub series, take a look here.

Blog 1: Device Administration Using Azure IoT Hub
Blog 2: Implementing Device To Cloud Messaging Using IoT Hub
Blog 3: Using IoT Hub for Cloud to Device Messaging

Author: Eldert Grootenboer

Eldert is a Microsoft Integration Architect and Azure MVP from the Netherlands, currently working at Motion10, mainly focused on IoT and BizTalk Server and Azure integration. He comes from a .NET background, and has been in the IT since 2006. He has been working with BizTalk since 2010 and since then has expanded into Azure and surrounding technologies as well. Eldert loves working in integration projects, as each project brings new challenges and there is always something new to learn. In his spare time Eldert likes to be active in the integration community and get his hands dirty on new technologies. He can be found on Twitter at @egrootenboer and has a blog at http://blog.eldert.net/.

One Platform Operations, Monitoring and Analytics Software
BizTalk360

microsoft biztalk

Learn more

Over 500 customers across 30+ countries depend on BizTalk360

ServiceBus360

Azure Composite Application

Learn more

Start manage and monitor your Azure Services in minutes

One Platform - Operations, Monitoring and Analytics Software
BizTalk360

microsoft biztalk

Learn more

Over 500 customers across 30+ countries depend on BizTalk360

One Platform - Operations, Monitoring and Analytics Software
ServiceBus360

Azure Composite Application

Learn more

Start manage and monitor your Azure Services in minutes

Back to Top