Firmware Update Archives

Tag Archive for: Firmware Update

Understanding Golioth OTA Firmware Updates

Over-the-Air (OTA) firmware updates are table stakes for Internet of Things (IoT) devices. Once a device is in the field, OTA means you can make changes by updating the firmware remotely.

The ability to update devices remotely is great, but that alone is the bare minimum of functionality. Options for targeting specific devices for each firmware update are crucial to building a fleet that scales.

For example, at Golioth we use a small subset of our deployed fleet to test releases before rolling them out to all devices. In some cases we have different hardware in the same fleet that each need different binaries. And when adding new features we conditionally roll out release candidates to certain team members for testing.

All of these features are built into Golioth and ready for you to use. Recently one of our customers opened a forum thread asking how firmware version, device blueprints, and device tags work together to determine the OTA each device receives. It’s a great question that we’ll dive into today!

What are OTA Firmware Updates?

Over-the-Air (OTA) firmware updates are a method of using a network connection to send a device a new firmware version that it then validates and runs.

At Golioth, we’ve used multiple types of network connections to accomplish this, including cellular, WiFi, Ethernet, and Thread. Our Golioth Firmware SDK demonstrates the feature, using MCUboot to store two copies of firmware (the currently running version, and the newly received update). Each image is cryptographically signed so it can be verified for authenticity and integrity before the device uses the new image.

Golioth applies firmware updates on the device side using the semantic version number. The Golioth servers make these update versions available based on their sequence number. Let’s unpack the differences.

Demystifying Semantic Versions and Sequence Numbers

Remember this rule of thumb: devices will always match the semantic version of a firmware release, while the Golioth servers will always advertise the most recent sequence number.

Devices match semantic version

Devices have no awareness of “newer” or “older” firmware releases. They merely check if the version of firmware currently running is an exact match for the semantic version being advertised by the server. It they do not match, the device will download the binary and update itself to the version available from the server. This might be a newer semantic version, or an older one, the only thing that matters is that the device sees a different version is available.

Device serial output indicates version 1.0.100 is currently running. The manifest received from Golioth shows main-1.0.99 as the latest version. The device begins downloading. We call this a roll back, the device is only aware that the versions don’t match and the server is the source of truth.

This is crucial in delivering the ability to “roll back” a firmware update. The Golioth web console has a rollout button that facilitates automatic roll back when you unselect the most recently uploaded firmware.

The server advertises the most recent sequence number

The Golioth server separates OTA into two distinct parts: the artifact and the release. The artifact is the binary itself which has a package name (the default name is main) and a semantic version number. The release is created using an existing artifact, adding a time-based sequence number (new releases have higher sequence numbers) and controlling whether or not the release is rolled out to devices.

Each of the artifacts is a unique binary with its own package name and semantic version. This fleet uses all the same hardware so no Blueprint has been assigned to these artifacts

The Golioth servers will check to ensure artifacts have unique name & version combos — you can only upload main - 1.2.3 once. The next artifact will need a different version number or package name. (The only way around this is to delete the existing artifact so you may reuse the package/version number.)

Assigning a blueprint to an Artifact makes it unique. The rules from the previous paragraph still apply, but artifacts with a blueprint will only be compared to other artifacts with the same blueprint.

Finally, the file hash uploaded with each artifact must be unique from all other artifacts. The Golioth web console will check the file, and issue an error if the same binary is uploaded more than once. This is a safety feature to help ensure that the wrong artifact isn’t uploaded by mistake.

Each release requires one artifact to be assigned. Notice that the main-1.0.1 artifact has been used multiple times, targeting devices with different tags assigned. The releases are made available to devices by enabling the Rollout toggle.

Golioth releases require an artifact, but the semantic version number will not be used to decide which release is advertised to devices. Instead, the release with the newest sequence number (meaning the most recently created release) will be advertised to devices.

There are many conditions to determining which release is advertised

It is important to understand that there are several ways to target devices with a release. The most obvious is the Rollout setting—a release will only be advertised if the rollout is enabled. Package names, device blueprints, and device tags are also used to determine which releases will be advertised to any given device in your fleet.

Applying Blueprints to Firmware Updates

If all devices in your fleet use the exact same hardware, they may all be able to run the same firmware. But in many cases, a fleet will have more than one hardware variant and need more than one compiled version of the same firmware release. For instance, if you have some devices that use an nRF9160 (cellular) and others that use an NXP i.MX RT1024 (Ethernet) you must run different firmware compiled specifically for those two distinct devices.

Golioth uses device blueprints to account for this issue. When you create devices on the Golioth cloud, you can choose one device blueprint to assign to the device. The same blueprint may be selected when uploading an artifact or creating a release.

When a device has a blueprint assigned, it will receive notification of releases with the newest sequence number and the matching blueprint.

Using Tags to Target OTA

Device tags can be assigned in a similar way to device blueprints but you may assign multiple tags (blueprints are limited to a single assignment per device/artifact/release).

Device tags must match exactly to receive a release notification. This means if you have a device with two tags and a release with only one tag, the device will not be notified. That said, if you roll out a release with zero tags selected, all devices in your fleet will be notified of the release no matter what tags are assigned to those devices.

This summary page for a single device in our fleet shows that this is a member of the “release-candidate” tag group. The device reports which version of firmware it is currently running, as shown in this view.

Multiple releases may be created using the same artifact. As an example, at Golioth we will roll out a release that targets the release candidate tag, so that devices that we have previously identified for testing new features will receive the update but the larger fleet will not. When testing is complete and the firmware artifact is determined to behave as we expected, a second release using the same artifact will be made without any tags, prompting the rest of the fleet to download and apply the OTA update.

How Does Golioth OTA Fit Your Needs?

Wow, this is a lot. Thanks for sticking with me through this post. I hope you will agree that there is quite a bit more to OTA than just providing an update file.

We’d love to hear your feedback. We’re especially interested to know your thoughts on how our system approaches tags. The last thing a fleet manager wants is a “surprise” update to a device they weren’t expecting. This is why we’ve gone to great lengths to implement granular control, and a multitude of options. But we’re always keen on using customer feedback to improve, so let us know on the Golioth Forum or hit us up at the DevRel email address.

Until next time, happy OTA updating!

January 11, 2024/by Mike Szczys

Cloud

How to use the Golioth REST API for OTA firmware updates

Golioth’s web console has a super easy and intuitive way to send Over-The-Air (OTA) updates to devices in the field. But for ultimate flexibility, you can choose to use the Golioth REST API for managing OTA firmware updates.

Every IoT device should have the ability to update firmware remotely. OTA is a crucial feature for fixing bugs and keeping fleet security up-to-date. But the device itself is only one half of the equation, you also need a management tool to get the new firmware to the device. Golioth is that management tool, and every one of our features are available using our REST API. Today we’ll take a look at how firmware update artifacts and releases work, using Python for the demonstration.

Overview of OTA from the Golioth Console

Before jumping into the REST API, let’s take a quick look at the Golioth Console OTA flow to help better understand the abstract concepts. There are two parts to rolling out an OTA update: artifacts and releases.

The artifacts screen shown here facilitates the upload of the firmware binary that will be served to the devices in your fleet. During the upload process you assign a package name (usually main) and version number (e.g. 1.0.1), with an optional setting for Blueprint (a way of grouping device types on Golioth)

Creating a release is the second part of creating an OTA update. You can select Blueprint and Device Tags to narrow down which devices are being targeted. The release is associated with an Artifact (uploaded in the previous step). Optional “release flags” are a way to associate some additional information about the release.

Releases include a rollout toggle button. This is a way to stage a release and devices will only be notified when the rollout toggle is turned on. If you ever need to go back to previous firmware version, this button serves as a one-click rollback!

Creating OTA Artifacts and Releases with the Golioth REST API

Now that we’ve reviewed Artifacts and Releases, let’s use the REST API to create some!

The Open API page from the Golioth Doc site

First off, the Open API section of our Docs is the best place to test out your REST API calls. Just click the Authorize button at the top and give it an API key from your Golioth Project. From there you can try out the commands live on that page.

Python API calls

import requests
import base64

For this demo I’ll use Python 3.10.6 to make the API calls. This is pretty easy to do using the requests and base64 packages.

Uploading an Artifact

def upload_artifact(api_key, proj_id, version, package, filename):
    artifact_upload_url = "https://api.golioth.io/v1/artifacts"
    headers = {'x-api-key': api_key}
    with open(filename, 'rb') as f:

        data = {
            "projectId": proj_id,
            "content": base64.standard_b64encode(f.read()).decode(),
            "version": version,
            #"blueprintId": "string",
            "package": package
            }

    jsonData = requests.post(artifact_upload_url, json=data, headers=headers).json()
    if 'data' in jsonData:
        if jsonData['data']['version'] == version:
            print("Artifact {}-{} created successfully for project {}".format(package, version, proj_id))
    elif 'code' in jsonData:
        print("Error code {}: {}".format(jsonData['code'], jsonData['message']))
    return jsonData

The artifact upload requires the API key and project name. The package name is usually main and semantic versioning is used for the version number. I have not assigned a Blueprint in this example but I did leave a commented line if you choose to do so.

>>> result = upload_artifact("V010sGuWtXXM1htPCHBjLGfrW6GlsKDt", "developer-training", "1.2.3", "main", "new_firmware.bin")
Artifact main-1.2.3 created successfully for project developer-training
>>> import json
>>> print(json.dumps(result, indent=4))
{
    "data": {
        "id": "63ceb040c345ce2e0256ac30",
        "version": "1.2.3",
        "package": "main",
        "createdAt": "2023-01-23T16:05:20.235Z",
        "updatedAt": "2023-01-23T16:05:20.235Z",
        "binaryInfo": {
            "digests": {
                "sha256": {
                    "digest": "0b101d7b0ad1330ec49471d6feb0debc02e022a99e839c7951a446c1539802e6",
                    "size": 32,
                    "type": "sha256"
                }
            },
            "headerSize": 0,
            "imageSize": 1219152,
            "tlvTotalSize": 0,
            "type": "default",
            "version": ""
        },
        "size": "1219152"
    }
}

A JSON packet is returned when the upload is successful. It includes all pertinent information for your newly created artifact. This includes the id of the artifact which is needed to create a release.

Creating a Release

def create_release(api_key, proj_id, artifact_id):
    release_url = "https://api.golioth.io/v1/projects/{}/releases".format(proj_id)
    headers = {'x-api-key': api_key}
    data = {
        #"deviceTagIds": [ tagId ],
        "artifactIds": [ artifact_id ],
        "rollout": True
        }

    jsonData = requests.post(release_url, json=data, headers=headers).json()
    if 'code' in jsonData:
        print("Error code {}: {}".format(jsonData['code'], jsonData['message']))
    return jsonData

The release is created using the API key, project name, and artifact ID. In this example I’ve chosen to rollout the release at the same time as it is created, so the devices will be notified immediately that there is a new OTA firmware version available.

>>> result = create_release("V010sGuWtXXM1htPCHBjLGfrW6GlsKDt", "developer-training", "63ceb31ac345ce2e0256ac31")
>>> import json
>>> print(json.dumps(result, indent=4))
{
    "data": {
        "id": "63ceb3b9c345ce2e0256ac32",
        "createdAt": "2023-01-23T16:20:09.563Z",
        "updatedAt": "2023-01-23T16:20:09.563Z",
        "releaseTags": [],
        "deviceTagIds": [],
        "suitManifest": {
            "authentication-wrapper": [
                {
                    "algorithm-id": "sha256",
                    "digest-bytes": "2a3efb45029dc5d23cf0adf5e260ba53c9a78266b9ee28bdbf4ef20b43a2d6c7"
                }
            ],
            "manifest": {
                "common": {
                    "common-sequence": [
                        {
                            "id": "set-component-index",
                            "value": 0
                        },
                        {
                            "arg": {
                                "class-id": "53f7395e-0825-5970-badb-cc7158e49eaa",
                                "image-digest": {
                                    "algorithm-id": "sha256",
                                    "digest-bytes": "0b101d7b0ad1330ec49471d6feb0debc02e022a99e839c7951a446c1539802e6"
                                },
                                "image-size": 1219152,
                                "vendor-id": "36323535-6565-3863-6430-346536653332"
                            },
                            "id": "override-parameters"
                        },
                        {
                            "id": "vendor-identifier",
                            "value": 15
                        },
                        {
                            "id": "class-identifier",
                            "value": 15
                        }
                    ],
                    "components": [
                        [
                            "[email protected]"
                        ]
                    ]
                },
                "install": [
                    {
                        "id": "set-component-index",
                        "value": 0
                    },
                    {
                        "arg": {
                            "uri": "/.u/c/[email protected]"
                        },
                        "id": "set-parameters"
                    },
                    {
                        "id": "fetch",
                        "value": 2
                    },
                    {
                        "id": "image-match",
                        "value": 15
                    }
                ],
                "manifest-sequence-number": 1674490809,
                "manifest-version": 1,
                "run": [
                    {
                        "id": "set-component-index",
                        "value": 0
                    },
                    {
                        "id": "run",
                        "value": 2
                    }
                ],
                "validate": [
                    {
                        "id": "set-component-index",
                        "value": 0
                    },
                    {
                        "id": "image-match",
                        "value": 15
                    }
                ]
            }
        },
        "artifactIds": [
            "63ceb31ac345ce2e0256ac31"
        ],
        "rollout": true,
        "sequenceNumber": "1674490809563381303"
    }
}

Upon success, all details of the release are returned by the REST API.

Get the Artifact ID (or Release ID)

def get_artifact_id(api_key, proj_id, version, package):
    artifact_url = "https://api.golioth.io/v1/projects/{}/artifacts".format(proj_id)
    headers = {'x-api-key': api_key}
    
    jsonData = requests.get(artifact_url, headers=headers).json()

    for i in jsonData['list']:
        a_ver = i['version']
        a_package = i['package']
        if version == a_ver and package == a_package:
            return i['id']
    return None

The artifact ID is returned when uploading a new binary. However, if you need to get it after the fact there is a REST API call for that as well.

>>> result = get_artifact_id("V010sGuWtXXM1htPCHBjLGfrW6GlsKDt", "developer-training", "1.2.3", "main")
>>> print(result)
63ceb31ac345ce2e0256ac31

Artifacts are identified by the version number and package name, but it’s the ID that is needed when creating a release using the REST API. The same approach is used to query the Release ID, just change the URL for the API call:

release_url = "https://api.golioth.io/v1/projects/{}/releases".format(proj_id)

Rollout, rollback

def release_set_rollout(api_key, proj_id, release_id, rollout_state):
    release_url = "https://api.golioth.io/v1/projects/{}/releases".format(proj_id)
    rollout_url = release_url + "/" + release_id
    headers = {'x-api-key': api_key}
    data = { "rollout": rollout_state }

    jsonData = requests.patch(rollout_url, json=data, headers=headers).json()

    return jsonData

Finally, you can rollout and rollback releases using the Release ID.

>>> result = release_set_rollout("V010sGuWtXXM1htPCHBjLGfrW6GlsKDt", "developer-training", "63ceb3b9c345ce2e0256ac32", False)
>>> print(result['data']['rollout'])
False

When rollout change is successful, you will receive the complete release manifest. Here I’ve printed just the state of the rollout.

IoT any way you need it

We’ve gone to great lengths to make it easy to build your IoT fleet. For most users this means a straightforward GUI experience on the web console. But every one of those features are available programmatically for those who need it. Today we showed OTA functionality, but you can just as easily create a custom UI for your fleet.

We’d love to hear about how are you using the REST API. Start a thread in the Golioth Forum to share your story!

January 24, 2023/by Mike Szczys

Device

How to do ESP32 OTA Updates using Golioth and ESP-IDF

The most sought-after Golioth feature is OTA, also known as Over-the-Air firmware updates. When you put an IoT device into the field it’s crucial that you be able to push firmware updates to it without human intervention. Golioth makes simplifies the process for your ESP-IDF projects.

Today we’re walking through the OTA process:

Build and flash the initial firmware to the device
Provision the device with credentials that will be persistent across firmware updates
Build a new revision of the firmware
Upload the firmware to Golioth and roll it out as a release
Observe the device detecting, downloading, and running the new firmware

Prerequisites:

Please ensure that you have installed a copy of the the ESP-IDF v4.4.2 to your computer. Today’s article will use an ESP32 but this will work with other variants like the ESP32s2, ESP32c3, etc.

Clone a copy of the Golioth Firmware SDK (which includes ESP-IDF support). To do, please follow the “Cloning this repo section” in the README.

Commands in this guide are based on a Linux operating system with the ESP-IDF and Golioth Firmware SDK installed in the home directory (~/). However, these are cross-platform tools and are easy to adapt to your system and your preferred install directories.

Build and flash the initial firmware

In a classic Chicken-or-Egg scenario, to perform a Golioth OTA update your device needs to be running firmware built for Golioth OTA. We can use the golioth_basics example which is ready to run without changes.

First, let’s make sure our ESP-IDF is set to the correct version and enabled for this session:

cd ~/esp-idf
git fetch
git checkout v4.4.2
git submodule update --init --recursive ./install.sh all
source export.sh

Now move to the ESP-IDF section of the Golioth Firmware SDK, specifically the golioth_basics example. We’ll build, flash, and run this code on the ESP32:

cd ~/golioth-firmware-sdk/examples/esp_idf/golioth_basics
idf.py build
idf.py flash
idf.py monitor

On some systems you will need to hold the boot button on the ESP32 in order to flash the code. I find to get the monitor command to work I need to first hold the boot button, then press reset when the screen says “waiting for download”. One last tip: CTRL-] is used to exit from the idf.py monitor screen.

Assign device credentials and connect

There are a number of ways to assign credentials to your device (including Bluetooth via your browser!) but perhaps the easiest is to type them into the shell. Head over to the Golioth Console and select your device’s credentials tab. (If you don’t have an account, sign up for the Dev Tier now, your first 50 devices are free.)

Use your the shell window to set the credentials. Here you can see the process, with the four commands for Golioth and WiFi credentials highlighted:

Type 'help' to get the list of commands.
Use UP/DOWN arrows to navigate through command history.
Press TAB when typing command name to auto-complete.
esp32> W (2212) golioth_example: WiFi and golioth credentials are not set
W (2212) golioth_example: Use the shell settings commands to set them, then restart
esp32> 
esp32> settings set golioth/psk-id 20221122201152-esp32@developer-training
Setting golioth/psk-id saved
esp32> settings set golioth/psk 5bfb64ad29dce4e3dd30ab10c5b95a6a
Setting golioth/psk saved
esp32> settings set wifi/ssid MyWifiAp
Setting wifi/ssid saved
esp32> settings set wifi/psk MyWifiPassword
Setting wifi/psk saved
esp32> reset

The final command resets the device so that it will use the new credentials. You should see this output:

I (4235) esp_netif_handlers: sta ip: 192.168.1.159, mask: 255.255.255.0, gw: 192.168.1.1
I (4235) example_wifi: WiFi Connected. Got IP:192.168.1.159
I (4235) example_wifi: Connected to AP SSID: MyWifiAp
I (4255) golioth_mbox: Mbox created, bufsize: 2184, num_items: 20, item_size: 104
I (4255) golioth_basics: Waiting for connection to Golioth...
W (4295) wifi:<ba-add>idx:0 (ifx:0, c6:ff:d4:a8:fa:10), tid:0, ssn:1, winSize:64
I (4395) golioth_coap_client: Start CoAP session with host: coaps://coap.golioth.io
I (4405) libcoap: Setting PSK key

I (4415) golioth_coap_client: Entering CoAP I/O loop
I (4805) golioth_basics: Golioth client connected
I (4805) golioth_basics: Hello, Golioth!
I (4815) golioth_coap_client: Golioth CoAP client connected
I (4815) golioth_fw_update: Current firmware version: 1.2.5
I (5735) golioth_fw_update: Waiting to receive OTA manifest
I (5835) golioth_basics: Synchronously got my_int = 42
I (5845) golioth_basics: Entering endless loop
I (5845) golioth_basics: Sending hello! 0
I (5935) golioth_fw_update: Received OTA manifest
I (5935) golioth_fw_update: Manifest does not contain different firmware version. Nothing to do.
I (5945) golioth_fw_update: Waiting to receive OTA manifest
I (6545) golioth_basics: Callback got my_int = 42
W (9805) golioth_coap_client: CoAP message retransmitted
W (10335) golioth_coap_client: 4.00 (req type: 3, path: .c/status), len 59
I (15845) golioth_basics: Sending hello! 1
I (21965) wifi:bcn_timout,ap_probe_send_start
I (25855) golioth_basics: Sending hello! 2
I (35005) wifi:bcn_timout,ap_probe_send_start
I (35855) golioth_basics: Sending hello! 3

First the ESP32 connects to WiFi, then Golioth. After checking (and not finding) a firmware update available, this example begins sending hello messages every few seconds.

Now let’s do an OTA firmware update

We connected to Golioth with the device, now let’s build and upload a new firmware package to test the OTA capabilities. We’ll use the same code, updating the Current Version number which the device uses to identify when an update is needed. We’ll also change the string used in the log messages so it’s easy to recognize that our new firmware is running.

Change the source code version and rebuild

The file we need to update is a common file used by multiple Golioth SDKs. Edit the ~/golioth-firmware-sdk/examples/common/golioth_basics.c file:

#define TAG "golioth_basics_new"

// Current firmware version
static const char* _current_version = "1.2.6";

You can see I’ve appended “_new” to the tag name and incremented the version number to 1.2.6. Now we’re ready to rebuild… but remember, don’t flash this to your ESP32. We’re going to upload it to Golioth and perform a remote firmware update!

cd ~/golioth-firmware-sdk/examples/esp_idf/golioth_basics
idf.py build

The newly built binary is located in the build folder.

Upload firmware to Golioth and roll out a release

After much preamble we’ve arrived at the important moment.

To set the scene, the ESP32 that’s running on your desk is a remote IoT device taking sensor readings in a brick-and-mortar retail establishment in Waldorf, Maryland. We’ll push an update to it using a three-step process:

Upload the binary as an “artifact”
Create a “release” using the artifact
Click the “rollout” button to make the release live

Go to the Golioth Console and select Firmware Update→Artifacts from the left sidebar. Click the “Create” button.

The only thing we’re going to change on this window is the “Artifact Version”. Type in the same version number you entered in the sourcecode for this firmware (probably 1.2.6 if you’re following along). Click the upload icon in the middle of the window and choose the “golioth_basics.bin” file from the build directory where you ran the idf.py build command. Finally, click the Upload Artifact button.

You have the option here to use a Blueprint. I’m not detailing that today for brevity, but it’s a good practice to use Blueprints to organize your production devices.

Now let’s create a release based on the artifact. Click Firmware Update→Release from the left sidebar and click the Create button.

All we’re going to do here is to chose the artifact we previously created in the Artifacts box and press Create Release.

You have some options here, most notably you can choose to start the rollout as soon as the release is created. I prefer to wait and roll it out as a separate confirmation step in case I made some mistake along the way.

Note that you have the option here of selecting device Blueprints and Tags to make this a more targeted release. This window is telling me the release will apply to 51 devices (!). That’s okay here because these are all test devices on a test project that we use for training.

Finally, let’s roll out the release to our devices:

The Rollout button is all that stand between you and automatic updates. Click it and you will (almost) immediately see your device begin to download the new binary.

Here’s an awesome feature to keep in mind. When you have more than one release, you can use this button to rollback to previous versions. This means if you realize you released an update that has a bug, you can just toggle this button and all of your devices will automatically download the next-newest release that has rollout selected.

Watch your ESP32 update

If you read the source code for the golioth_basics example you will notice that it calls golioth_fw_update_init(client, _current_version);. That means the device has registered with the Golioth servers to receive updates when new firmware is available. Look in the terminal output and you will see the result:

I (175827) golioth_basics: Sending hello! 17
I (185007) golioth_fw_update: Received OTA manifest
I (185007) golioth_fw_update: Current version = 1.2.5, Target version = 1.2.6
I (185017) golioth_fw_update: State = Downloading
I (185317) golioth_fw_update: Image size = 1211744
I (185327) golioth_fw_update: Getting block index 0 (1/1184)
I (185827) golioth_basics: Sending hello! 18
W (187867) golioth_coap_client: CoAP message retransmitted
I (187947) fw_update_esp_idf: Writing to partition subtype 17 at offset 0x1a0000
I (187947) fw_update_esp_idf: Erasing flash
I (191627) golioth_fw_update: Getting block index 1 (2/1184)
I (191837) golioth_fw_update: Getting block index 2 (3/1184)
I (192037) golioth_fw_update: Getting block index 3 (4/1184)
I (192187) golioth_fw_update: Getting block index 4 (5/1184)
I (192447) golioth_fw_update: Getting block index 5 (6/1184)
I (192597) golioth_fw_update: Getting block index 6 (7/1184)

... snip ...

I (279837) golioth_fw_update: Getting block index 1181 (1182/1184) 
I (280107) golioth_fw_update: Getting block index 1182 (1183/1184) 
I (280317) golioth_fw_update: Getting block index 1183 (1184/1184) 
I (280457) golioth_fw_update: Total bytes written: 1211760 
I (280467) esp_image: segment 0: paddr=001a0020 vaddr=3f400020 size=29df0h (171504) map 
I (280527) esp_image: segment 1: paddr=001c9e18 vaddr=3ffbdb60 size=05868h ( 22632) 
I (280537) esp_image: segment 2: paddr=001cf688 vaddr=40080000 size=00990h ( 2448) 
I (280547) esp_image: segment 3: paddr=001d0020 vaddr=400d0020 size=d9a64h (891492) map 
I (280847) esp_image: segment 4: paddr=002a9a8c vaddr=40080990 size=1e2a0h (123552) 
I (280887) esp_image: segment 5: paddr=002c7d34 vaddr=50000000 size=00010h ( 16) 
I (280887) golioth_fw_update: State = Downloaded 
I (281127) golioth_fw_update: State = Updating 
I (281327) fw_update_esp_idf: Setting boot partition 
I (281337) esp_image: segment 0: paddr=001a0020 vaddr=3f400020 size=29df0h (171504) map 
I (281397) esp_image: segment 1: paddr=001c9e18 vaddr=3ffbdb60 size=05868h ( 22632) 
I (281417) esp_image: segment 2: paddr=001cf688 vaddr=40080000 size=00990h ( 2448) 
I (281417) esp_image: segment 3: paddr=001d0020 vaddr=400d0020 size=d9a64h (891492) map 
I (281717) esp_image: segment 4: paddr=002a9a8c vaddr=40080990 size=1e2a0h (123552) 
I (281757) esp_image: segment 5: paddr=002c7d34 vaddr=50000000 size=00010h ( 16) 
I (281827) golioth_fw_update: Rebooting into new image in 5 seconds 
I (282827) golioth_fw_update: Rebooting into new image in 4 seconds 
I (283827) golioth_fw_update: Rebooting into new image in 3 seconds 
I (284827) golioth_fw_update: Rebooting into new image in 2 seconds 
I (285827) golioth_fw_update: Rebooting into new image in 1 seconds

... snip ...

I (4267) esp_netif_handlers: sta ip: 192.168.1.159, mask: 255.255.255.0, gw: 192.168.1.1
I (4267) example_wifi: WiFi Connected. Got IP:192.168.1.159
I (4277) example_wifi: Connected to AP SSID: TheNewPeachRepublic
I (4287) golioth_mbox: Mbox created, bufsize: 2184, num_items: 20, item_size: 104
I (4287) golioth_basics_new: Waiting for connection to Golioth...
W (4297) wifi:<ba-add>idx:0 (ifx:0, c6:ff:d4:a8:fa:10), tid:0, ssn:1, winSize:64
I (4307) golioth_coap_client: Start CoAP session with host: coaps://coap.golioth.io
I (4307) libcoap: Setting PSK key

I (4317) golioth_coap_client: Entering CoAP I/O loop
I (4637) golioth_basics_new: Golioth client connected
I (4647) golioth_coap_client: Golioth CoAP client connected
I (4657) golioth_basics_new: Hello, Golioth!
I (4657) golioth_fw_update: Current firmware version: 1.2.6
I (4657) golioth_fw_update: Waiting for golioth client to connect before cancelling rollback
I (4677) golioth_fw_update: Firmware updated successfully!
I (4727) golioth_fw_update: State = Idle
I (5937) golioth_basics_new: Synchronously got my_int = 42
I (5937) golioth_basics_new: Entering endless loop
I (5937) golioth_basics_new: Sending hello! 0

First, the device compares version numbers and then begins to download blocks of the new firmware. Once downloaded it will reboot, connect to Golioth, and verify that it is running the newest version. The log labels near the end of the output now show golioth_basics_new, confirming one of the changes we made to our source code.

On the Golioth Console, the summary view for this device confirms the currently running firmware version is 1.2.6!

With Golioth, OTA is built into the SDK

Golioth has done the heavy lifting so that you don’t need to. Our SDK uses just the single API call to register your devices for firmware updates. Use the fleet management tools on the Golioth Cloud to provision your devices in groups and by hardware variants. These make it possible to target your test devices for the first round of updates, or push new features just to the devices on the fourth floor of your Des Moines plant.

These robust tools are crucial for successful, long-lasting IoT deployments, and and they’re ready for you to start using right now. If you have any questions, we’d love to talk! Reach out to us on the Golioth Forum or get in touch with the DevRel team for demo.

November 29, 2022/by Mike Szczys

Product Feature, Video

Update Firmware (DFU) over cellular (OTA) on the nRF9160 with Golioth

Implementing remote firmware updates is one of the most critical steps towards building a resilient IoT deployment. In short: it allows you to change your device firmware in the field without being physically present with the device. This can be critical for security fixes, for feature upgrades, and for extending the lifetime of your IoT implementation. Today we’re going to be looking at Over-The-Air (OTA) firmware updates (DFU) for the Nordic Semiconductor nRF9160 cellular module (SIP). The associated video explanation and walkthrough (below) was published on our YouTube channel a couple months ago, but we’re reviewing it on our blog today.

Why are IoT firmware updates difficult?

There are many difficult components to OTA DFU, including on the system design level, at the device level, and on the hosting service. At a system level view, the hard part of a firmware update is knowing which device should be receiving the update. From the device perspective, it’s critical to be able to recover from any issues during the update and then verify the complete update was received; using cellular as the method of connectivity means there may be intermittent connectivity. With embedded devices, there are fewer ways to have a device intelligently recover, as may be the case with a full Linux system. On the cloud side, having a reliable hosting and delivery system for firmware packages is also important. Simply sticking files on a FTP server represents a security risk and puts an outsized burden on the logic of the device in the field.

Firmware updates are easier with Golioth

Golioth provides a data pipe to cellular devices and other devices in the Zephyr ecosystem as a one stop shop for all data connectivity. This includes state data handling (digital twin), data streams coming back to the cloud, and logging. Once that data pipe is established, we can also push firmware updates down to devices utilizing transport layers like CoAP.

We also benefit from the fact that the Zephyr project utilizes MCUBoot as the default bootloader. This open source bootloader (to which we are contributing members!) makes it easy to interact with from within an application. It’s as simple as pushing an updated firmware binary into memory and then calling an API to use that code upon reboot. Our recent blog posts around the ESP32 showcased how the newly-published port of MCUboot to the ESP32 platform enabled DFU on those parts. The Nordic Semiconductor nRF9160 has had this capability from the beginning because of its deep integration in the Zephyr ecosystem.

A key point shown in the video below is the signing of images using MCUboot. Having a signed firmware binary means there is a way to check that the entire package is there. From a security perspective, the signing allows the device to validate that the firmware image that has arrive is secure and unchanged in transit.

Managing firmware rollout

The Golioth Console enables our users to manage a wide variety of devices in their IoT fleet. A deployment with 10,000+ devices would be difficult to manage using scripts alone. Withouth the ability to visualize which devices have been updated or not means there is higher likelihood of costly mistakes.

To aid in partitioning device deployments, users can tag devices to create logical representations of different groupings. You might use a tag to identify some devices as early recipients of a test firmware image. Blueprints can be used to group devices that match a certain hardware profile. In this way, Golioth users can target key portions of their device fleet to ensure that only those specific devices are being updated at any given time.

Artifacts and Releases

Golioth implements two distinct features that will allow for more complex firmware packages. The first is an Artifact. this is simply a binary that is uploaded to the Golioth cloud. For your security, Golioth has no knowledge of what is inside that image, only how it has been signed. As such, this might be a firmware image, a JPEG that will be displayed on a smart screen. An artifact can also be a binary blob being delivered to a device modem, or even firmware that’s being pushed down the line to other microcontrollers in the system. Each of these artifacts can be versioned and assigned a blueprint, so that they can only be deployed onto the eligible systems.

The second feature is the idea of a Release. Users can take individual Artifacts and bundle them together as a Release. For example, a Release might have an image to display on the screen, the base application firmware for the device we’re targeting (the nRF9160), and a firmware image for an auxiliary processor on board (e.g., an nRF52). Once the user is ready to deploy, they create a Release with matching tags and blueprints and an assigned Release version. Devices in the field that match the criteria set by the user on the console (tags, blueprints) will then be notified that a new device firmware release is available for them. The device in the field will start the firmware update process, downloading chunks of data.

Once the image is verified as complete, the device will utilize the MCUBoot API on the device and restart using that new image. In the event the Golioth user wants to roll back changes, they simply slide a switch on the Golioth Console and the device is alerted to download/restart the previous version of firmware. In this way, Golioth users have the ultimate flexibility for delivering firmware devices to a laser focused set of devices in their fleet.

Watch the demo

See this all happening in real time, including compiling a new firmware image, uploading it to the Golioth Console and watching the OTA DFU happen on the nRF9160.

February 24, 2022/by Chris Gammell

Cloud, Product Feature, Video

Device Firmware Update (DFU) with Golioth

“I’m sorry boss, I am working as fast as I can here. I reprogrammed about 36 out of the total 50 units, but this is slow going. I only have one programming cable and I need to disassemble the deployed units so I can get to the header on the boards first.”

A bad firmware image on your deployed IoT devices can mean ruined weekends, upset customers, and lost time. Many businesses pursue a network based firmware update so that they can push new versions to their devices. In fact, this is a critical part of the firmware development process, often a very early one. Developing or implementing a bootloader allows engineers to ship new control software to their devices. A straw poll on Twitter showed that some engineers spend a significant amount of time putting this tooling in place.

Straw poll: in the first 3 months of a new electronics project, how much of that time is dedicated to getting the bootloader/firwmare update flow working?

— Chris Gammell (@Chris_Gammell) October 11, 2021

While the “barely any time” group seems large, it also includes those who aren’t doing a custom bootloader, nor a bootloader that is networked:

Straw poll, part 2, electric boogaloo:

How many of your projects' bootloaders work over a network? ("Over The Air" or "OTA") Can you send an updated firmware package without a cable being involved?

Original poll linked above, esp curious about people who responded to that

— Chris Gammell (@Chris_Gammell) October 12, 2021

In the past, networked firmware updates took a significant amount of planning and coordination between hardware, firmware, software, and web teams. Golioth has collapsed this down to a simple process.

Update all the devices in your fleet with the click of a button

Golioth Device Firmware Update (DFU) is possible because the Golioth SDK is built on top of the Zephyr Project. Part of that implementation includes MCUboot, an open source bootloader. Using open source software up and down the stack, Golioth enables quick, secure deployment of firmware packages to IoT devices throughout the world. The Golioth Console enables easy management of firmware releases, including multi-part binary bundles, enabling updates for devices as diverse as smart speakers, digital signage, machine learning enabled sensor systems, multiple processor embedded devices, and more.

In the video linked below, Lead Engineer Alvaro Viebrantz demonstrates with Chris Gammell how to update the firmware of an nRF52 based device over Ethernet. The video includes code snippets in Zephyr and walking through the build process using the command line tool West. Once the firmware image is built, Alvaro showcases how to push the image to the Golioth cloud, package it for delivery, and then deploying to Golioth enabled devices.

No more fussing with programming cables out in the field, Golioth allows engineers to update their devices with new features, requested fixes, and efficiency improvements. Try it out today!

About Golioth

Golioth is a cloud company dedicated to making it easier to prototype, deploy, and manage IoT systems. Learn more by joining the Golioth Beta program and reading through Golioth Documentation.

October 14, 2021/by Chris Gammell