Download Images to IoT Devices Using OTA

Golioth Over-the-Air (OTA) Updates in most common cases are used for single-image firmware upgrade purposes. In that scenario, a device is notified about a new release. Such notification includes a release manifest, which contains information about new firmware. The most important metadata that a device gets is firmware version, hash, and URL (used to download the firmware).

Firmware is the only artifact that is tied to OTA release in that scenario. But the Golioth OTA service may also include multiple artifacts. This allows you to implement multi-image upgrades, e.g. when there are many MCUs on a single device. Golioth OTA even supports artifacts that are not firmware, but large blobs of data of any kind. Examples include AI models, images and arbitrary binary blobs.

Device with a display

This article shows an example application running on a device with a display. Implemented functionality is simple: just displaying an arbitrary image. In the future we would like to add more capabilities, so firmware upgrade will be implemented. Additionally we would like to change the displayed image without upgrading the whole firmware.

Multi-component OTA

The Golioth SDK exposes high-level APIs to easily setup single-image firmware upgrade (golioth_fw_update_init()). This automatically creates a thread that observes newest firmware release, upgrades it when notified and reboots to run the new version.

In the case of multi-component releases we will handle the manifest in application code. Let’s first implement a callback that gets executed when a new release is available:

struct ota_observe_data
{
    struct golioth_ota_manifest manifest;
    struct k_sem manifest_received;
};

static void on_ota_manifest(struct golioth_client *client,
                            const struct golioth_response *response,
                            const char *path,
                            const uint8_t *payload,
                            size_t payload_size,
                            void *arg)
{
    struct ota_observe_data *data = arg;

    LOG_INF("Manifest received");

    if (response->status != GOLIOTH_OK)
    {
        return;
    }

    LOG_HEXDUMP_INF(payload, payload_size, "Received OTA manifest");

    enum golioth_ota_state state = golioth_ota_get_state();
    if (state == GOLIOTH_OTA_STATE_DOWNLOADING)
    {
        GLTH_LOGW(TAG, "Ignoring manifest while download in progress");
        return;
    }

    enum golioth_status status =
        golioth_ota_payload_as_manifest(payload, payload_size, &data->manifest);
    if (status != GOLIOTH_OK)
    {
        GLTH_LOGE(TAG, "Failed to parse manifest: %s", golioth_status_to_str(status));
        return;
    }

    if (data->manifest.num_components > 0) {
        k_sem_give(&data->manifest_received);
    }
}

The above code checks whether the release manifest was received correctly and OTA is not already in progress. Then the CBOR encoded manifest is decoded with golioth_ota_payload_as_manifest(). If the manifest is valid and it contains at least one component, the main application thread is notified by releasing a semaphore with k_sem_give(&data->manifest_received).

Now it is time to start manifest observation in main() and wait until a release manifest is received:

int main(void)
{
    struct ota_observe_data ota_observe_data = {};

    /* ... */

    golioth_ota_observe_manifest_async(client, on_ota_manifest, &ota_observe_data);

    k_sem_take(&ota_observe_data.manifest_received, K_FOREVER));

    /* ... */
}

At this point the application continues execution after the manifest is successfully received and parsed. The next step is handling of received components:

int main(void)
{
    /* ... */

    for (size_t i = 0; i < ota_observe_data.manifest.num_components; i++) {
        struct golioth_ota_component *component = &ota_observe_data.manifest.components[i];
        uint8_t hash_bin[32];

        hex2bin(component->hash, strlen(component->hash), hash_bin, sizeof(hash_bin));

        struct component_desc *desc = component_by_name(component->package);
        if (!desc) {
            LOG_WRN("Unknown '%s' artifact package", component->package);
            continue;
        }

        if (desc->version ?
            (component_version_cmp(desc, component->version) == 0) :
            (component_hash_cmp(desc, hash_bin) == 0)) {
            continue;
        }

    LOG_INF("Updating %s package", component->package);

        status = golioth_ota_download_component(client, component, desc->write_block, NULL);
        if (status == GOLIOTH_OK) {
            reboot = true;
        }
    }

    /* ... */
}

Information about each component is stored in the ota_observe_data.manifest.components[] array. Either version or hash is compared with the received component. When it differs, the new component is downloaded with golioth_ota_download_component() API.

Firmware and background components require different handling. This is achieved with component_descs[] array and helper functions:

struct component_desc
{
    const char *name;
    const char *version;
    uint8_t hash[32];
    ota_component_block_write_cb write_block;
};

static struct component_desc component_descs[] = {
    { .name = "background", .write_block = write_to_storage },
    { .name = "main", .write_block = write_fw, .version = _current_version },
};

static int component_hash_update(struct component_desc *desc, uint8_t hash[32])
{
    memcpy(desc->hash, hash, 32);

    return 0;
}

static int component_hash_cmp(struct component_desc *desc, const uint8_t hash[32])
{
    return memcmp(desc->hash, hash, 32);
}

static int component_version_cmp(struct component_desc *desc, const char *version)
{
    return strcmp(desc->version, version);
}

static struct component_desc *component_by_name(const char *name)
{
    for (size_t i = 0; i < ARRAY_SIZE(component_descs); i++) {
        struct component_desc *desc = &component_descs[i];

        if (strcmp(desc->name, name) == 0) {
            return desc;
        }
    }

    return NULL;
}

Downloaded firmware is written to flash directly, into second application slot:

static struct flash_img_context flash;

enum golioth_status write_fw(const struct golioth_ota_component *component,
                             uint32_t block_idx,
                             uint8_t *block_buffer,
                             size_t block_size,
                             bool is_last,
                             void *arg)
{
    const char *filename = component->package;
    int err;

    LOG_INF("Writing %s block idx %u", filename, (unsigned int) block_idx);

    if (block_idx == 0) {
        err = flash_img_prepare(&flash);
        if (err) {
            return GOLIOTH_ERR_FAIL;
        }
    }

    err = flash_img_buffered_write(&flash, block_buffer, block_size, is_last);
    if (err) {
        LOG_ERR("Failed to write to flash: %d", err);
        return GOLIOTH_ERR_FAIL;
    }

    if (is_last) {
        LOG_INF("Requesting upgrade");

        err = boot_request_upgrade(BOOT_UPGRADE_TEST);
        if (err) {
            LOG_ERR("Failed to request upgrade: %d", err);
            return GOLIOTH_ERR_FAIL;
        }
    }

    return GOLIOTH_OK;
}

The background image is written to file system using write_to_storage() callback:

enum golioth_status write_to_storage(const struct golioth_ota_component *component,
                                     uint32_t block_idx,
                                     uint8_t *block_buffer,
                                     size_t block_size,
                                     bool is_last,
                                     void *arg)
{
    const char *filename = component->package;
    struct fs_file_t fp = {};
    fs_mode_t flags = FS_O_CREATE | FS_O_WRITE;
    char path[32];
    int err;
    ssize_t ret;

    LOG_INF("Writing %s block idx %u", filename, (unsigned int) block_idx);

    if (block_idx == 0) {
        flags |= FS_O_TRUNC;
    }

    sprintf(path, "/storage/%s", filename);

    err = fs_open(&fp, path, flags);
    if (err) {
        LOG_ERR("Failed to open %s: %d", filename, err);

        return GOLIOTH_ERR_FAIL;
    }

    err = fs_seek(&fp, block_idx * CONFIG_GOLIOTH_BLOCKWISE_DOWNLOAD_BUFFER_SIZE, FS_SEEK_SET);
    if (err) {
        goto fp_close;
    }

    ret = fs_write(&fp, block_buffer, block_size);
    if (ret < 0) {
        err = ret;
        goto fp_close;
    }

fp_close:
    fs_close(&fp);

    if (err) {
        return GOLIOTH_ERR_FAIL;
    }

    return GOLIOTH_OK;
}

Displaying (updated) background

Firmware is updated automatically during next boot, so there is nothing more needed to start using it. Background image, on the other hand, needs to be loaded from file system in the application code:

static lv_img_dsc_t img_background;

static int background_show(void)
{
    char hash[32] = {};
    struct fs_dirent dirent;
    struct fs_file_t background_fp = {};
    lv_img_header_t *img_header;
    uint8_t *buffer;
    int err;
    ssize_t ret;

    err = fs_stat("/storage/background", &dirent);
    if (err) {
        if (err == -ENOENT) {
            LOG_WRN("No background image found on FS");
        } else {
            LOG_ERR("Failed to check/stat background image: %d", err);
        }

        return err;
    }

    LOG_INF("Background image file size: %zu", dirent.size);

    buffer = malloc(dirent.size);
    if (!buffer) {
        LOG_ERR("Failed to allocate memory");
        return -ENOMEM;
    }

    err = fs_open(&background_fp, "/storage/background", FS_O_READ);
    if (err) {
        LOG_WRN("Failed to load background: %d", err);
        goto buffer_free;
    }

    ret = fs_read(&background_fp, buffer, dirent.size);
    if (ret < 0) {
        LOG_ERR("Failed to read: %zd", ret);
        err = ret;
        goto background_close;
    }

    if (ret != dirent.size) {
        LOG_ERR("ret (%d) != dirent.size (%d)", (int) ret, (int) dirent.size);
        err = -EIO;
        goto background_close;
    }

    err = mbedtls_sha256(buffer, dirent.size, hash, 0);
    if (err) {
        LOG_ERR("Failed to get update sha256: %d", err);
        goto background_close;
    }

    LOG_HEXDUMP_INF(hash, sizeof(hash), "hash");

    component_hash_update(&component_descs[1], hash);

    img_header = (void *)buffer;
    img_background.header = *img_header;
    img_background.data_size = dirent.size - sizeof(*img_header);
    img_background.data = &buffer[sizeof(*img_header)];

    lv_obj_t * background = lv_img_create(lv_scr_act());
    lv_img_set_src(background, &img_background);
    lv_obj_align(background, LV_ALIGN_CENTER, 0, 0);

background_close:
    fs_close(&background_fp);

buffer_free:
    free(buffer);

    return err;
}

Note that besides loading the background image, there is also SHA256 calculation using mbedtls_sha256(). This is needed to compare with the SHA256 hash received from OTA service in order to decide whether the background image needs to be updated.

Testing with native_sim

Round display with a black bezel around a white image with the Golioth Echo mascot at the center. A USB cable is plugged into the device on the right side of the screen.
XIAO ESP32S3 with Seeed Studio XIAO Round Display

The example-download-photo application is compatible with XIAO ESP32S3 with Seeed Studio XIAO Round Display. However it is possible to test with Native Simulator as well. To achieve that, the following command can be used:

# Build the example
west build -p -b native_sim/native/64 $(west topdir)/example-download-photo

# Run the example
west build -t run

Native Simulator uses the SDL library to emulate a display. On the first run it is blank because no background image is available. Now it is time to upload a background image as an OTA artifact and create a release. An example background image is included in the repository in background/Echo-Pose-Stand.bin. After rolling out an OTA release, this image is downloaded automatically to /storage/background file on the device, which is indicated with the following logs:

[00:00:01.310,007] <inf> example_download_photo: Received OTA manifest
...
[00:00:01.310,007] <inf> example_download_photo: component 0: package=background version=1.0.5 uri=/.u/c/[email protected] hash=6b4d243a362c0c4f63c535b2d2f7b8dfe4bcfbca69e7b2f8009f917458794c5e size=35716
[00:00:01.310,007] <inf> example_download_photo: Updating background package
[00:00:01.560,008] <inf> example_download_photo: Writing background block idx 0
[00:00:01.700,009] <inf> example_download_photo: Writing background block idx 1
...
[00:00:06.320,042] <inf> example_download_photo: Writing background block idx 34

Starting Native Simulator again shows the following screen:

Golioth Over-the-Air (OTA) Updates in most common cases are used for single-image firmware upgrade purposes. In that scenario, a device is notified about a new release. Such notification includes a release manifest, which contains information about new firmware. The most important metadata that a device gets is firmware version, hash, and URL (used to download the firmware).

Firmware is the only artifact that is tied to OTA release in that scenario. But the Golioth OTA service may also include multiple artifacts. This allows you to implement multi-image upgrades, e.g. when there are many MCUs on a single device. Golioth OTA even supports artifacts that are not firmware, but large blobs of data of any kind. Examples include AI models, images and arbitrary binary blobs.

Device with a display

This article shows an example application running on a device with a display. Implemented functionality is simple: just displaying an arbitrary image. In the future we would like to add more capabilities, so firmware upgrade will be implemented. Additionally we would like to change the displayed image without upgrading the whole firmware.

Multi-component OTA

The Golioth SDK exposes high-level APIs to easily setup single-image firmware upgrade (golioth_fw_update_init()). This automatically creates a thread that observes newest firmware release, upgrades it when notified and reboots to run the new version.

In the case of multi-component releases we will handle the manifest in application code. Let’s first implement a callback that gets executed when a new release is available:

struct ota_observe_data
{
    struct golioth_ota_manifest manifest;
    struct k_sem manifest_received;
};

static void on_ota_manifest(struct golioth_client *client,
                            const struct golioth_response *response,
                            const char *path,
                            const uint8_t *payload,
                            size_t payload_size,
                            void *arg)
{
    struct ota_observe_data *data = arg;

    LOG_INF("Manifest received");

    if (response->status != GOLIOTH_OK)
    {
        return;
    }

    LOG_HEXDUMP_INF(payload, payload_size, "Received OTA manifest");

    enum golioth_ota_state state = golioth_ota_get_state();
    if (state == GOLIOTH_OTA_STATE_DOWNLOADING)
    {
        GLTH_LOGW(TAG, "Ignoring manifest while download in progress");
        return;
    }

    enum golioth_status status =
        golioth_ota_payload_as_manifest(payload, payload_size, &data->manifest);
    if (status != GOLIOTH_OK)
    {
        GLTH_LOGE(TAG, "Failed to parse manifest: %s", golioth_status_to_str(status));
        return;
    }

    if (data->manifest.num_components > 0) {
        k_sem_give(&data->manifest_received);
    }
}

The above code checks whether the release manifest was received correctly and OTA is not already in progress. Then the CBOR encoded manifest is decoded with golioth_ota_payload_as_manifest(). If the manifest is valid and it contains at least one component, the main application thread is notified by releasing a semaphore with k_sem_give(&data->manifest_received).

Now it is time to start manifest observation in main() and wait until a release manifest is received:

int main(void)
{
    struct ota_observe_data ota_observe_data = {};

    /* ... */

    golioth_ota_observe_manifest_async(client, on_ota_manifest, &ota_observe_data);

    k_sem_take(&ota_observe_data.manifest_received, K_FOREVER));

    /* ... */
}

At this point the application continues execution after the manifest is successfully received and parsed. The next step is handling of received components:

int main(void)
{
    /* ... */

    for (size_t i = 0; i < ota_observe_data.manifest.num_components; i++) {
        struct golioth_ota_component *component = &ota_observe_data.manifest.components[i];
        uint8_t hash_bin[32];

        hex2bin(component->hash, strlen(component->hash), hash_bin, sizeof(hash_bin));

        struct component_desc *desc = component_by_name(component->package);
        if (!desc) {
            LOG_WRN("Unknown '%s' artifact package", component->package);
            continue;
        }

        if (desc->version ?
            (component_version_cmp(desc, component->version) == 0) :
            (component_hash_cmp(desc, hash_bin) == 0)) {
            continue;
        }

    LOG_INF("Updating %s package", component->package);

        status = golioth_ota_download_component(client, component, desc->write_block, NULL);
        if (status == GOLIOTH_OK) {
            reboot = true;
        }
    }

    /* ... */
}

Information about each component is stored in the ota_observe_data.manifest.components[] array. Either version or hash is compared with the received component. When it differs, the new component is downloaded with golioth_ota_download_component() API.

Firmware and background components require different handling. This is achieved with component_descs[] array and helper functions:

struct component_desc
{
    const char *name;
    const char *version;
    uint8_t hash[32];
    ota_component_block_write_cb write_block;
};

static struct component_desc component_descs[] = {
    { .name = "background", .write_block = write_to_storage },
    { .name = "main", .write_block = write_fw, .version = _current_version },
};

static int component_hash_update(struct component_desc *desc, uint8_t hash[32])
{
    memcpy(desc->hash, hash, 32);

    return 0;
}

static int component_hash_cmp(struct component_desc *desc, const uint8_t hash[32])
{
    return memcmp(desc->hash, hash, 32);
}

static int component_version_cmp(struct component_desc *desc, const char *version)
{
    return strcmp(desc->version, version);
}

static struct component_desc *component_by_name(const char *name)
{
    for (size_t i = 0; i < ARRAY_SIZE(component_descs); i++) {
        struct component_desc *desc = &component_descs[i];

        if (strcmp(desc->name, name) == 0) {
            return desc;
        }
    }

    return NULL;
}

Downloaded firmware is written to flash directly, into second application slot:

static struct flash_img_context flash;

enum golioth_status write_fw(const struct golioth_ota_component *component,
                             uint32_t block_idx,
                             uint8_t *block_buffer,
                             size_t block_size,
                             bool is_last,
                             void *arg)
{
    const char *filename = component->package;
    int err;

    LOG_INF("Writing %s block idx %u", filename, (unsigned int) block_idx);

    if (block_idx == 0) {
        err = flash_img_prepare(&flash);
        if (err) {
            return GOLIOTH_ERR_FAIL;
        }
    }

    err = flash_img_buffered_write(&flash, block_buffer, block_size, is_last);
    if (err) {
        LOG_ERR("Failed to write to flash: %d", err);
        return GOLIOTH_ERR_FAIL;
    }

    if (is_last) {
        LOG_INF("Requesting upgrade");

        err = boot_request_upgrade(BOOT_UPGRADE_TEST);
        if (err) {
            LOG_ERR("Failed to request upgrade: %d", err);
            return GOLIOTH_ERR_FAIL;
        }
    }

    return GOLIOTH_OK;
}

The background image is written to file system using write_to_storage() callback:

enum golioth_status write_to_storage(const struct golioth_ota_component *component,
                                     uint32_t block_idx,
                                     uint8_t *block_buffer,
                                     size_t block_size,
                                     bool is_last,
                                     void *arg)
{
    const char *filename = component->package;
    struct fs_file_t fp = {};
    fs_mode_t flags = FS_O_CREATE | FS_O_WRITE;
    char path[32];
    int err;
    ssize_t ret;

    LOG_INF("Writing %s block idx %u", filename, (unsigned int) block_idx);

    if (block_idx == 0) {
        flags |= FS_O_TRUNC;
    }

    sprintf(path, "/storage/%s", filename);

    err = fs_open(&fp, path, flags);
    if (err) {
        LOG_ERR("Failed to open %s: %d", filename, err);

        return GOLIOTH_ERR_FAIL;
    }

    err = fs_seek(&fp, block_idx * CONFIG_GOLIOTH_BLOCKWISE_DOWNLOAD_BUFFER_SIZE, FS_SEEK_SET);
    if (err) {
        goto fp_close;
    }

    ret = fs_write(&fp, block_buffer, block_size);
    if (ret < 0) {
        err = ret;
        goto fp_close;
    }

fp_close:
    fs_close(&fp);

    if (err) {
        return GOLIOTH_ERR_FAIL;
    }

    return GOLIOTH_OK;
}

Displaying (updated) background

Firmware is updated automatically during next boot, so there is nothing more needed to start using it. Background image, on the other hand, needs to be loaded from file system in the application code:

static lv_img_dsc_t img_background;

static int background_show(void)
{
    char hash[32] = {};
    struct fs_dirent dirent;
    struct fs_file_t background_fp = {};
    lv_img_header_t *img_header;
    uint8_t *buffer;
    int err;
    ssize_t ret;

    err = fs_stat("/storage/background", &dirent);
    if (err) {
        if (err == -ENOENT) {
            LOG_WRN("No background image found on FS");
        } else {
            LOG_ERR("Failed to check/stat background image: %d", err);
        }

        return err;
    }

    LOG_INF("Background image file size: %zu", dirent.size);

    buffer = malloc(dirent.size);
    if (!buffer) {
        LOG_ERR("Failed to allocate memory");
        return -ENOMEM;
    }

    err = fs_open(&background_fp, "/storage/background", FS_O_READ);
    if (err) {
        LOG_WRN("Failed to load background: %d", err);
        goto buffer_free;
    }

    ret = fs_read(&background_fp, buffer, dirent.size);
    if (ret < 0) {
        LOG_ERR("Failed to read: %zd", ret);
        err = ret;
        goto background_close;
    }

    if (ret != dirent.size) {
        LOG_ERR("ret (%d) != dirent.size (%d)", (int) ret, (int) dirent.size);
        err = -EIO;
        goto background_close;
    }

    err = mbedtls_sha256(buffer, dirent.size, hash, 0);
    if (err) {
        LOG_ERR("Failed to get update sha256: %d", err);
        goto background_close;
    }

    LOG_HEXDUMP_INF(hash, sizeof(hash), "hash");

    component_hash_update(&component_descs[1], hash);

    img_header = (void *)buffer;
    img_background.header = *img_header;
    img_background.data_size = dirent.size - sizeof(*img_header);
    img_background.data = &buffer[sizeof(*img_header)];

    lv_obj_t * background = lv_img_create(lv_scr_act());
    lv_img_set_src(background, &img_background);
    lv_obj_align(background, LV_ALIGN_CENTER, 0, 0);

background_close:
    fs_close(&background_fp);

buffer_free:
    free(buffer);

    return err;
}

Note that besides loading the background image, there is also SHA256 calculation using mbedtls_sha256(). This is needed to compare with the SHA256 hash received from OTA service in order to decide whether the background image needs to be updated.

Testing with native_sim

Round display with a black bezel around a white image with the Golioth Echo mascot at the center. A USB cable is plugged into the device on the right side of the screen.
XIAO ESP32S3 with Seeed Studio XIAO Round Display

The example-download-photo application is compatible with XIAO ESP32S3 with Seeed Studio XIAO Round Display. However it is possible to test with Native Simulator as well. To achieve that, the following command can be used:

# Build the example
west build -p -b native_sim/native/64 $(west topdir)/example-download-photo

# Run the example
west build -t run

Native Simulator uses the SDL library to emulate a display. On the first run it is blank because no background image is available. Now it is time to upload a background image as an OTA artifact and create a release. An example background image is included in the repository in background/Echo-Pose-Stand.bin. After rolling out an OTA release, this image is downloaded automatically to /storage/background file on the device, which is indicated with the following logs:

[00:00:01.310,007] <inf> example_download_photo: Received OTA manifest
...
[00:00:01.310,007] <inf> example_download_photo: component 0: package=background version=1.0.5 uri=/.u/c/[email protected] hash=6b4d243a362c0c4f63c535b2d2f7b8dfe4bcfbca69e7b2f8009f917458794c5e size=35716
[00:00:01.310,007] <inf> example_download_photo: Updating background package
[00:00:01.560,008] <inf> example_download_photo: Writing background block idx 0
[00:00:01.700,009] <inf> example_download_photo: Writing background block idx 1
...
[00:00:06.320,042] <inf> example_download_photo: Writing background block idx 34

Starting Native Simulator again shows the following screen:

Marcin Niestrój
Marcin Niestrój
Marcin is a firmware developer on the Golioth SDK, which is based on the Zephyr SDK. He has worked in the embedded space for 10 years, 4 of those on Zephyr. Past upstream contributions have focused on the networking stack. He has an extensive background combining hardware, firmware, and the cloud.

More from author

Start the discussion at forum.golioth.io

Related posts

Advertismentspot_img

Latest posts

Using the ESP32-C3 as an AT modem on the Aludel Elixir

We're preparing to do some testing around power consumption of different services on the Aludel Elixir, our open source hardware with the Nordic nRF9160...

West Commands Every Zephyr User Should Know

Zephyr's west meta tool can perform a vast number of useful operations. Here's a collection of both command and uncomon commands that we find ourselves reaching for when working on Zephyr-based IoT projects.

How we use Allure Report to understand Continuous Integration Tests

Allure Report is an open source tool to better understand testing outcomes. Golioth runs over 500 Hardware in the Loop tests for each pull request. Here's how we use Allure Report to make sense of it all.

Want to stay up to date with the latest news?

We would love to hear from you! Please fill in your details and we will stay in touch. It's that simple!