Golioth just rolled out a new settings service that lets you control your growing fleet of IoT devices. You can specify settings for your entire fleet, and override those global settings by individual device or for multiple devices that share the same blueprint.

Every IoT project needs some type of settings feature, from adjusting log levels and configuring the delay between sensor readings, to adjusting how frequently a cellular connection is used in order to conserve power. With the new settings service, the work is already done for you. A single settings change on the Golioth web console is applied to all devices listening for changes!

As you grow from dozens of devices to hundreds (and beyond), the Golioth settings service makes sure you can change device settings and confirm that those changes were received.

Demonstrating the settings service

Golioth settings service

The settings service is ready for you use right now. We have code samples for the Golioth Zephyr SDK and the Golioth ESP-IDF SDK. Let’s take it for a spin using the Zephyr samples.

I’ve compiled and flashed the Golioth Settings sample for an ESP32 device. It observes a LOOP_DELAY_S endpoint and uses that value to decide how long to wait before sending another “Hello” log message.

Project-wide settings

On the Golioth web console, I use the Device Settings option on the left sidebar to create the key/value pair for this setting. This is available to all devices in the project with firmware that is set up to observe the LOOP_DELAY_S settings endpoint.

Golioth device settings dialog

When viewing device logs, we can see the setting is observed as soon as the device connects to Golioth. The result is that the Hello messages are now issued ten seconds apart.

[00:00:19.930,000] <inf> golioth_system: Client connected!
[00:00:20.340,000] <inf> golioth: Payload
                                  a2 67 76 65 72 73 69 6f  6e 1a 62 e9 99 c4 68 73 |.gversio n.b...hs
                                  65 74 74 69 6e 67 73 a1  6c 4c 4f 4f 50 5f 44 45 |ettings. lLOOP_DE
                                  4c 41 59 5f 53 fb 40 24  00 00 00 00 00 00       |[email protected]$ ......  
[00:00:20.341,000] <dbg> golioth_hello: on_setting: Received setting: key = LOOP_DELAY_S, type = 2
[00:00:20.341,000] <inf> golioth_hello: Set loop delay to 10 seconds
[00:00:20.390,000] <inf> golioth_hello: Sending hello! 2
[00:00:30.391,000] <inf> golioth_hello: Sending hello! 3
[00:00:40.393,000] <inf> golioth_hello: Sending hello! 4

Settings by device or by blueprint

Of course, you don’t always want to have the same settings for all devices. Consider debugging a single device. It doesn’t make much sense to turn up the logging level or frequency of sensor reads for all devices. So with Golioth it’s easy to change the setting on just a single device.

settings change for a single device on the Golioth console

In the device view of the Golioth web console there is a settings tab. Here you can see the key, the value, and the level of the value. I have already changed the device-specific value in this screen so the level is being reported as “Device”.

[00:07:30.466,000] <inf> golioth_hello: Sending hello! 45
[00:07:40.468,000] <inf> golioth_hello: Sending hello! 46
[00:07:43.728,000] <inf> golioth: Payload
                                  a2 67 76 65 72 73 69 6f  6e 1a 62 e9 9c 17 68 73 |.gversio n.b...hs
                                  65 74 74 69 6e 67 73 a1  6c 4c 4f 4f 50 5f 44 45 |ettings. lLOOP_DE
                                  4c 41 59 5f 53 fb 40 00  00 00 00 00 00 00       |[email protected] ......  
[00:07:43.729,000] <dbg> golioth_hello: on_setting: Received setting: key = LOOP_DELAY_S, type = 2
[00:07:43.729,000] <inf> golioth_hello: Set loop delay to 2 seconds
[00:07:50.469,000] <inf> golioth_hello: Sending hello! 47
[00:07:52.471,000] <inf> golioth_hello: Sending hello! 48

When I made the change. the device was immediately notified and you can see from the timestamps that it began logging at a two-second cadence as expected.

Golioth settings applied at the blueprint level

It is also possible to change settings for a group of devices that share a common blueprint. Here you will find this setting by selecting Blueprint from the left sidebar and choosing your desired blueprint.

Settings are applied based on specificity. The device-level is the most specific and will be applied first, followed by blueprint-level, and finally project-level. Blueprints may be created and applied at any time, so if you later realize you need a more specific group you can change the blueprint for those devices.

Implementation: The two parts that make up the settings service

Fundamentally, there are two parts that make our device settings system work: the Golioth cloud services running on our servers and your firmware that is running on the devices. The Golioth device SDKs allow you to register a callback function that receives settings values each time a change is made to the settings on the cloud. You choose how the device should react to these settings, like updating a delay value, enabling/disabling features, changing log output levels, really anything you want to do.

Don’t worry if you already have devices in the field. You can add the settings service or make changes to how your device handles those settings, then use the Golioth OTA firmware update system to push out the new behavior.

Take control of your fleet

Scale is the hope for most IoT companies, but it’s also where the pain of IoT happens. You need to know you can control your devices, securely communicate with them, and perform updates as necessary. Golioth has you covered in all of these areas. The new settings service ensures that your ability to change how your fleet is performing doesn’t become outpaced by your growth.

On Tuesday we announced the Golioth ESP-IDF SDK that delivers all of Golioth’s excellent features to ESP32 projects built on Espressif’s FreeRTOS-based ESP-IDF ecosystem. The APIs included in our SDK make it dead simple to set up an encrypted connection with Golioth and begin sending and receiving data, controlling the device remotely, sending your logging messages up to the cloud, and of course performing Over-the-Air (OTA) updates on remote devices.

Today we dive into the code as Nick Miller, Golioth’s lead firmware engineer, takes us on a guided tour.

What does Golioth ESP-IDF SDK deliver?

All of the best features of Golioth’s device management cloud are available in our ESP-IDF SDK. The set of APIs are quite clever and take all of the heavy lift out of your hands. This includes:

  • Set, get, and observe data endpoints on the cloud
  • Write log data back to the cloud
  • Handle Over-the-Air (OTA) firmware updates
  • API calls–in both synchronous and asynchronous options–to suit your needs

Setup: Install ESP-IDF and clone the Golioth repo

To get started you need have the ESP-IDF installed and clone the Golioth ESP-IDF SDK. Instructions are available in the readme of our git repository, and there is also a quickstart on our docs site.

Our new SDK is a component for FreeRTOS, the real-time operating system used by the ESP-IDF. It’s the same operating system and build tools you’re used to, with the Golioth SDK sitting on top so that your devices can interact with the Golioth servers.

Stepping through the Golioth-Basics example

The best way to test-drive is with the Golioth-Basics example that is included in the SDK. It demonstrates assigning Golioth credentials to your device, sending/receiving data, observing data, sending log messages, and performing over-the-air (OTA) firmware updates. The app_main.c file is thoroughly commented to explain each API call in detail.

The example begins by initializing non-volatile storage, configuring a serial shell, checking for credentials, and connecting to WiFi. At that point we can start using the Golioth APIs.

Creating the Golioth system client

// Now we are ready to connect to the Golioth cloud.
//
// To start, we need to create a client. The function golioth_client_create will
// dynamically create a client and return a handle to it.
//
// The client itself runs in a separate task, so once this function returns,
// there will be a new task running in the background.
//
// As soon as the task starts, it will try to connect to Golioth using the
// CoAP protocol over DTLS, with the PSK ID and PSK for authentication.
golioth_client_t client =
        golioth_client_create(nvs_read_golioth_psk_id(), nvs_read_golioth_psk());

Everything starts of by instantiating a client to handle the connection for you. This client will be passed to all of the API calls so that the SDK knows where to send them.

Sending log messages

// We can also log messages "synchronously", meaning the function will block
// until one of 3 things happen (whichever comes first):
//
// 1. We receive a response to the request from the server
// 2. The user-provided timeout expires
// 3. The default client task timeout expires (GOLIOTH_COAP_RESPONSE_TIMEOUT_S)
//
// In this case, we will block for up to 2 seconds waiting for the server response.
// We'll check the return code to know whether a timeout happened.
//
// Any function provided by this SDK ending in _sync will have the same meaning.
golioth_status_t status = golioth_log_warn_sync(client, "app_main", "Sync log", 5);

Here you can see a log being written to Golioth. Notice that the client created in the previous code block is used as the first parameter. This logging call is synchronous, and will wait to ensure the log was received by the Golioth servers. There is also an asynchronous version available that provides the option to run a callback function when the log is received by Golioth.

Setting up OTA firmware updates

// For OTA, we will spawn a background task that will listen for firmware
// updates from Golioth and automatically update firmware on the device using
// Espressif's OTA library.
//
// This is optional, but most real applications will probably want to use this.
golioth_fw_update_init(client, _current_version);

OTA firmware updates are handled for you by the SDK. The line of code shown here is all it takes to register for updates. The app will then observe the firmware version available on the server. It will automatically begin the update process whenever you roll out a new firmware release on the Golioth Cloud.

Sending and receiving data

// There are a number of different functions you can call to get and set values in
// LightDB state, based on the type of value (e.g. int, bool, float, string, JSON).
golioth_lightdb_set_int_async(client, "my_int", 42, NULL, NULL);
// To asynchronously get a value from LightDB, a callback function must be provided
golioth_lightdb_get_async(client, "my_int", on_get_my_int, NULL);

The bread and butter of the IoT industry is the ability to send and received data. This code demonstrates asynchronous set and get functions. Notice that the get API call registers on_get_my_int as a callback function that will be executed to handle the data that arrives back from the Golioth servers.

A get command runs just once to fetch the requested data from Golioth. Another extremely useful approach is to observe the data using the golioth_lightdb_observe_async(). It works the same way as an asynchronous get call, but it will execute your callback every time the data on the server changes.

Putting it all together

In the second half of the video, Nick takes us through the process running the demo. He starts with setting up the ESP-IDF environment and compiling to code, and continues all the way through to viewing the device data on the web console.

You’re going to love working with the Golioth ESP-IDF SDK. It’s designed to deal with all the complexity of securely connecting and controlling your IoT devices. The API calls are easy to understand, and they make it painless to add Golioth to existing and future ESP-IDF based projects. Give it a try today using our free Dev Tier.

We’d love to hear what you’re planning to build. You can connect with us on the Golioth Discord server, ask questions over on the Golioth Forums, and share your demos by tagging the Golioth account on Twitter.

One of the most useful services in the Golioth Zephyr SDK is the ability to observe data changes on the cloud. A device can register any LightDB endpoint and the Golioth servers will notify it whenever changes happen. If your device is a door lock, an example endpoint might be “lock status”, which you would want to know about a server-side state change immediately.

This is slightly more complex to set up than something like a LightDB ‘Set’ API call. ‘Observe’ requires a callback function to handle the asynchronous reply from the Golioth servers. Today we’ll walk through how to add Golioth LightDB Observe functionality to any Zephyr application by:

  1. Adding a callback that is called every time observed data changes
  2. Registering the callback with a data endpoint
  3. Using on_message to trigger the callback
  4. Ensuring that on_message and on_connect are both registered

These techniques are all found in our LightDB Observe sample code which acts as the roadmap for this article.

Prerequisites

Today’s post assumes that you already have a device running Zephyr and you have already tested out an app that uses the Golioth Zephyr SDK. If you’re not there yet, don’t worry. You can sign up for our free Dev Tier that includes up to 50 devices, and follow the Golioth Quickstart Guide.

Your Zephyr workspace should already have Golioth installed as a module and your app (probably in main.c) is already instantiating a Golioth system client. Basically, you should see a block like this one somewhere in your code:

#include <net/coap.h>
#include <net/golioth/system_client.h>
static struct golioth_client *client = GOLIOTH_SYSTEM_CLIENT_GET();

If you don’t, checkout out our How to add Golioth to an existing Zephyr project blog post to get up to speed before moving on.

1. Add a callback function for observed data changes

The goal of this whole exercise is to enable your device to perform a task whenever data changes at your desired endpoint. Remember: Golioth LightDB endpoints are configurable by you! Whatever data you’d like to monitor, you can customize it to your needs.

The first thing we’ll do is create a callback function that will perform the task.

static int on_update(const struct coap_packet *response,
             struct coap_reply *reply,
             const struct sockaddr *from)
{
    char str[64];
    uint16_t payload_len;
    const uint8_t *payload;

    payload = coap_packet_get_payload(response, &payload_len);
    if (!payload) {
        LOG_WRN("packet did not contain data");
        return -ENOMSG;
    }

    if (payload_len + 1 > ARRAY_SIZE(str)) {
        payload_len = ARRAY_SIZE(str) - 1;
    }

    memcpy(str, payload, payload_len);
    str[payload_len] = '\0';

    LOG_DBG("payload: %s", log_strdup(str));

    return 0;
}

 

The majority of this callback is used to verify that string data was received from Golioth. Ultimately, line 22 is what you are interested in. For this example, we’re printing a log message with the string payload stored in the str array.

If your endpoint contains more than just one value, it may be useful to parse the JSON object and store the values. Also keep in mind that this callback will execute on the golioth system client thread, which is a different thread than the “main” thread running your application. This means:

  • The callback function should return quickly (under 10 ms). If that’s not enough time, you can use a Zephyr Workqueue to schedule the work on another thread.
  • If access to global data is required, access to the data must be protected by a mutex to avoid data races between threads.

2. Add coap_reply and register the callback in on_connect

We need to create an array of structs to store messages that arrive from Golioth. This struct is then registered with an endpoint and the callback we created in step 1.

#include <net/coap.h>

First ensure that you have included the CoAP header from Zephyr which defines the coap_reply struct and has some handy helper functions.

static struct coap_reply coap_replies[1];

Here I’ve created a coap_replies array with just one member because my example observes one single endpoint. If you want to observe multiple endpoints, you will need multiple callbacks and you must have one coap_reply for each callback.

static void golioth_on_connect(struct golioth_client *client)
{
    struct coap_reply *observe_reply;
    int err;

    coap_replies_clear(coap_replies, ARRAY_SIZE(coap_replies));

    observe_reply = coap_reply_next_unused(coap_replies, ARRAY_SIZE(coap_replies));

    /*
     * Observe the data stored at `/counter` in LightDB.
     * When that data is updated, the `on_update` callback
     * will be called.
     * This will get the value when first called, even if
     * the value doesn't change.
     */
    err = golioth_lightdb_observe(client,
                      GOLIOTH_LIGHTDB_PATH("counter"),
                      COAP_CONTENT_FORMAT_TEXT_PLAIN,
                      observe_reply, on_update);

    if (err) {
        LOG_WRN("failed to observe lightdb path: %d", err);
    }
}

Now we register the observation using the golioth_lightdb_observe() API call. The parameters passed to this function include:

  1. The endpoint to observe (the GOLIOTH_LIGHTDB_PATH macro indicates this is a LightDB State endpoint)
  2. The format, either plain text or CBOR (see the LightDB LED sample which demonstrates using CBOR serialization)
  3. A coap_reply struct to store the message and metadata received from Golioth
  4. The callback function to execute when a message is received from this endpoint

Notice that the helper function coap_reply_next_unused() is called to get the next available struct. This is important if you are registering multiple callbacks and should be used prior to each golioth_lightdb_observe() API call to get a pointer to a struct that isn’t already associated with another callback.

3. Add the processing function to on_message

This step is small but important, and seems to be the one I frequently forget and then scratch my head when my callback isn’t working.

Whenever a message is received from Golioth, the Golioth system client executes a callback that we usually call on_message. For our observed callbacks to work, we need to tell on_message about our coap_replies array.

static void golioth_on_message(struct golioth_client *client,
                   struct coap_packet *rx)
{
    /*
     * In order for the observe callback to be called,
     * we need to call this function.
     */
    coap_response_received(rx, NULL, coap_replies,
                   ARRAY_SIZE(coap_replies));
}

 

By calling Zephyr’s coap_response_received(), the CoAP packet will be parsed and the appropriate callback will be selected from the coap_replies struct (if one exists).

4. Ensure on_message and on_connect are both registered

The final step is to make sure that we’ve registered callbacks for when the Golioth system client connects and receives a message.

client->on_connect = golioth_on_connect;
client->on_message = golioth_on_message;
golioth_system_client_start();

This should be done in main() before the loop begins. The golioth_client struct should have already been instantiated in your code, in this example it was called client. The code above associates our two callbacks and starts the client running.

Observed data in action

Now that we’ve tied it all together, let’s test it out. Here’s the terminal output of my Zephyr app:

[00:00:09.328,000] <dbg> golioth_lightdb: main: Start Light DB observe sample
[00:00:09.328,000] <inf> golioth_system: Starting connect
[00:00:09.537,000] <inf> golioth_system: Client connected!
[00:00:09.739,000] <dbg> golioth_lightdb: on_update: payload: null
[00:00:48.653,000] <dbg> golioth_lightdb: on_update: payload: 42

You can see that at boot time, the observed data will be reported, which is great for setting defaults when your device first connects to observed data. When the endpoint was registered it didn’t exist on Golioth so a payload of null was returned. About 49 seconds later a payload of 42 is received. That’s when I added the endpoint and value in the Golioth Console.

On the cloud, this is an integer, but the device receives payloads as strings. You’ll need to validate received data on the device side to ensure expected behavior in your callback functions (beyond simply printing out the payload as I’m doing here). Give it a try for yourself using our LightDB Observe sample code.

Observing LightDB data gives your devices the ability to react to any changes without the need to poll like you would if you were using the golioth_lightdb_get() function. In addition to being notified each time the data changes, you’ll also get the current state when the observation is first registered (ie: at power-up). Single endpoints, or entire JSON objects can be observed, making it possible to group different types of state data to suit any need.

If you still have questions, or want to talk about how LightDB Observe works under the hood, head over to the Golioth Forum or ping us on the Golioth Discord.

Golioth is a device management cloud that is easy to add to any Zephyr project. Just list the Golioth Zephyr SDK as a module in your manifest file and the west tool will do the rest. Well, almost. Today I’ll walk through how to add Golioth to an existing Zephyr project. As an example, I’ll be using our hello sample. We chose this because it already has networking, which makes explaining things a bit easier. Generally any board that has led0 defined in the device tree and you can enable networking should be a good fit. What we want to do here is showcase the elements that allow you to add Golioth, so let’s dive in!

0. Understanding the west manifest

A Zephyr workspace is made up of a number of different code repositories all stored in the same tree. Zephyr uses a west manifest file to manage all of these repositories, including information like the origin URL of the repo, the commit hash to use, and where each repository should be placed in your local tree.

Think of the manifest as a code repository shopping list that the west update command uses to fill up your local Zephyr tree. There may be more than one manifest file, but today we’ll just focus on the main manifest.

1. Add Golioth to the west manifest

Start by locating your manifest file and opening it with a code editor.

~/zephyrproject $ west manifest --path
/home/mike/zephyrproject/zephyr/west.yml

Under projects: add an entry for the Golioth Zephyr SDK (highlighted in the abbreviated manifest sample below).

manifest:
  defaults:
    remote: upstream
 
  remotes:
    - name: upstream
      url-base: https://github.com/zephyrproject-rtos
 
  #
  # Please add items below based on alphabetical order
  projects:
      # Golioth repository.
    - name: golioth
      path: modules/lib/golioth
      revision: v0.2.0
      url: https://github.com/golioth/golioth-zephyr-sdk.git
      import:
        west-external.yml
    - name: canopennode
      revision: 53d3415c14d60f8f4bfca54bfbc5d5a667d7e724
...

Note that I have called out the v0.2.0release tag. You can set this to any of our release tags, to main, or to a specific commit.

Now run an update to incorporate the manifest changes:

west update

2. Select libraries to add to the build

We use KConfig to build in the libraries that Golioth needs. These changes are made in the prj.conf file of your application.

The first set of symbols deals with Zephyr’s networking stack. Of course every Internet of Things thing needs a network connection so you likely already have these symbols selected.

# Generic networking options
CONFIG_NETWORKING=y
CONFIG_NET_IPV4=y
CONFIG_NET_IPV6=n

Golioth is secure by default so we want to select the mbedtls libraries to handle the encryption layer.

# TLS configuration
CONFIG_MBEDTLS_ENABLE_HEAP=y
CONFIG_MBEDTLS_HEAP_SIZE=10240
CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=2048

Now let’s turn on the Golioth libraries and give them a bit of memory they can dynamically allocate from.

# Golioth Zephyr SDK
CONFIG_GOLIOTH=y
CONFIG_GOLIOTH_SYSTEM_CLIENT=y
 
CONFIG_MINIMAL_LIBC_MALLOC_ARENA_SIZE=256

Every device needs its own credentials because all connections to Golioth require security. There are a few ways to do this, but perhaps the simplest is to add them to the prj.conf file.

# Golioth credentials
CONFIG_GOLIOTH_SYSTEM_CLIENT_PSK_ID="[email protected]"
CONFIG_GOLIOTH_SYSTEM_CLIENT_PSK="cab43a035d4fe4dca327edfff6aa7935"

And finally, I’m going to enable back-end logging. This is not required for connecting to Golioth, but sending the Zephyr logs to the cloud is a really handy feature for remote devices.

# Optional for sending Zephyr logs to the Golioth cloud
CONFIG_NET_LOG=y
CONFIG_LOG_BACKEND_GOLIOTH=y
CONFIG_LOG_PROCESS_THREAD_STACK_SIZE=2048

I separated each step above for the sake of explanation. But the combination of these into one block is the boiler-plate configuration that I use on all of my new projects. See this all as one file in the prj.conf from our hello sample.

3. Instantiate the Golioth system client and say hello

So far we added Golioth as a Zephyr module and enabled the libraries using KConfig. Now it’s time to use the Golioth APIs in the main.c file.

The first step is to include the header files and instantiate a golioth_client object. The client is used to manage the connection with the Golioth servers. Including the coap header file is not strictly required to establish a connection, but it is necessary for processing the responses from Golioth, so I always include it.

#include <net/coap.h>
#include <net/golioth/system_client.h>
static struct golioth_client *client = GOLIOTH_SYSTEM_CLIENT_GET();

I also add a callback function to handle messages coming back from the Golioth cloud. Technically this is optional, but if you want two-way communication with the cloud you need it!

/* Callback for messages received from the Golioth servers */
static void golioth_on_message(struct golioth_client *client,
                   struct coap_packet *rx)
{
    uint16_t payload_len;
    const uint8_t *payload;
    uint8_t type;
 
    type = coap_header_get_type(rx);
    payload = coap_packet_get_payload(rx, &payload_len);
 
    printk("%s\n", payload);
}

In the main function, I register my callback, start the Golioth client, and call the golioth_send_hello() API.

/* Register callback and start Golioth system client */
client->on_message = golioth_on_message;
golioth_system_client_start();
 
while (1) {
    /* Say hello to Golioth */
    int ret = golioth_send_hello(client);
    if (ret) {
        printk("Failed to send hello! %d\n", ret);
    }
    else printk("Hello sent!\n");
}

When successful, the golioth_send_hello() call will prompt for a message back from the server which includes the name of the sending device. This is printed out by the golioth_on_message() callback.

Hello sent!
Hello blinky1060
Hello sent!
Hello blinky1060
Hello sent!
Hello blinky1060

Extra Credit: Back-end Logging

Observant readers have noticed that I enabled back-end logging using KConfig symbols but I didn’t use it in the C code. Here’s the really awesome part: just use Zephyr logging as you normally would and it will automatically be sent to your Golioth console.

At the top of main.c, include the log header file and register a logging module.

#include <logging/log.h>
LOG_MODULE_REGISTER(any_name_you_want, LOG_LEVEL_DBG);

In our C functions, call logs as you normal would.

LOG_ERR("An error message goes here!");
LOG_WRN("Careful, this is a warning!");
LOG_INF("Did you now that this is an info log?");
LOG_DBG("Oh no, why isn't my app working? Maybe this debug log will help.");

Now your log messages will appear on the Golioth console!

Further Reading

This covers the basics of adding Golioth to an existing Zephyr project. You can see the steps all in one place by looking at the Golioth code samples. Here are the examples that you’ll find immediately useful:

  • Hello – basics of connecting to Golioth and sending log messages (what was covered in this post)
  • Lightdb Set – send data to Golioth
  • Lightdb Observe – device will be notified whenever an endpoint on the Golioth cloud is updated

To go even deeper, you can see how we use the Golioth Over-the-Air (OTA) firmware update feature, and how to use the Zephyr settings subsystem for persistent credential storage. And remember, the Dev Tier of Golioth includes the first 50 devices, so you can try all of this out for free.

Image from Todd Lappin on flickr

Thread (and its common implementation known as OpenThread) is a networking technology that is quickly gaining adoption due to the forthcoming Matter standard. Thread has been around for many years, but as it grows, it becomes more accessible using off-the-shelf firmware and hardware. So it’s accessible, but maybe not all that straightforward. This post will help with that.

As we have more customers talking to us about using Golioth as a management layer for both Matter solutions and for standalone industrial Thread networks, we thought we should expand our tools to better fit the needs of engineers designing new systems.

In this article and associated examples, we will show you how to set up a network at home using common components. We’ll build a custom device that is communicating back over that network utilizing features of Golioth that extend the networking layer: over-the-air firmware updates, time series data tracking, command/control structures, and instant logging.

Parts of this Thread Guide

In the past, we have written about the basics of getting a Thread demo up and running, and shown it working on video. Today we build upon that work and showcase a new set of resources so you can build your own Thread network with custom devices.

  • YouTube video – A walkthrough of the setup steps and troubleshooting steps with a newly provisioned Thread network.
  • A tutorial site – Follow along with a simplified set of directions for replicating what we have built. We think is the shortest path towards getting a working Thread network on your desk or bench.
  • Code repository – Start from working code on the node devices to see how you can customize code and have your sensor data streaming back to Golioth quickly.
  • This blog post

Getting Started

The majority of the step-by-step instructions are contained in the tutorial site for Golioth and OpenThread. If you’re interested in immediately replicating and then extending our setup, head over to the tutorial site to learn more.

Let’s take a higher level look at the OpenThread Border Router (OTBR), OpenThread nodes, and Golioth device management layer that make up this Thread network example.

OpenThread Border Router  (OTBR)

This is the key part in a Thread network, as it allows any arbitrary number of nodes that are communicating with one another (meshing) to then reach the outside internet. Each Thread device has an IPv6 address, which is great: That means a node that is meshing with 30 other nodes and connected to the internet (through an OTBR) is directly addressable from the internet. That’s an important piece. We might expect a higher power WiFi based device to have an IP address (assigned from a router), but probably not a low power sensor. The OTBR does a lot of the routing of information and translation of packets coming from node devices.

We build a DIY version of the OTBR because there aren’t many commercially available (yet). We use a Raspberry Pi and an nRF52840 Dongle to create a pipeline out to the wider internet.

OpenThread Nodes

In our first video/blog about Thread, we had nodes talking to the internet through an OTBR. However, the nodes only blinked and sent back logging messages and we didn’t give detailed instructions on how to build them. In the tutorial site and the video, we show how we can execute arbitrary code to do higher level functions, like data logging. We use the Laird Connectivity BT510, which is a sensor node built with the Nordic Semiconductor nRF52840. It also has a range of sensors built in and is contained in a waterproof case. We think it’s a great platform for building a small, reliable Thread network and we used it in our Red Demo that we showcased at the 2022 Zephyr Developer Summit and Embedded World.

The BT510 is a board already supported in Zephyr, which means we can very easily compile firmware for it and access all of the sensor drivers that are built into Zephyr, no custom out-of-tree code required. We use the OpenThread networking stack that is native to Zephyr, and the Golioth SDK, which allows each node to be pre-configured to talk to the Golioth servers. We then enable things like LightDB Stream to regularly send back sensor data from the device through the Thread network.

Golioth Device Management Layer

The Golioth Device Management Layer/Platform is already ready for you. If you don’t have an account, you can sign up on the Golioth Console, which will guide you through creating your first device on the platform; you can use the credentials for that digital version of a device to control your first Thread node.

Once your Thread device is connected, you’ll be able to see how often the device is connecting, view the latest data and logs being sent back from the device, and check which firmware versions are on each device. Any data sent to the Golioth platform from a Thread node can be aggregated into an external visualization platform, or wholesale exported to 3rd party services (AWS, Azure, GCP) using Output Streams.

What will you build?

We are sharing the know-how to build a Thread network. Following this guide enables all of your devices on the Thread network to communicate back to the wider internet. As a hardware engineer, I don’t really want to mess about with the network layer, I just want something that works. Instead, I’d rather focus on the end application and building end devices (nodes) that are useful to customers and users.

With Golioth, Zephyr, OpenThread, and some off-the-shelf hardware, you can get started quickly and you can start to connect custom devices with a powerful interface to the internet. What will you build? Please let us know on our Discord, Forum, or on Twitter.

The conference season is in full swing, and the Golioth team will be showing off device cloud hardware demos at Embedded World in Nuremberg.

With the past few years of virtual meetings, we’re relieved that we can finally connect with our friends and colleagues in person. We’d love to see you swing by the Golioth booth to say hi. You’ll find us as among the Zephyr Project exhibits at Booth #4-170 in Hall 4.

The show floor is going to be pretty hectic, so if you want to get together for an more in-depth conversation, we’re happy to schedule a meeting in advance. Email the DevRel team and let us know.

A preview of the Golioth hardware demos

We had so much fun putting together a set of awesome hardware demos to show what we’ve been working on since coming out of stealth mode a year ago and completing our beta phase back in November. If you’re interested in trying out anything you see here, our Dev Tier is free for your first 50 device!

OpenThread

The new hotness on the IoT block is OpenThread, an open-source implementation of Thread. This mesh networking protocol is officially known as 802.15.4 and it works extremely well with Golioth’s device management cloud—we have already published our demo code, and have a full guide and blog post ready to roll next week while we’re at the conference.

Our demo uses a Raspberry Pi as the “border router”, although any Linux device will work nicely. A Nordic Semiconductor nRF52840 dongle connects to the Pi, acting as the radio co-processor (RCP) so that the Pi can communicate with the other devices on the Thread network. We’re connecting three Laird BT510 nodes (also based on the nRF52840) to build out the network.

These devices all have IPv6 addresses and connect to the wider internet via the border router. Of course Golioth is secure by default, so all communications are encrypted. You get the best features of Golioth, from full-featured data management to over-the-air (OTA) firmware upgrades.

IoT Devices reacting to one another

As we showed at the recent Zephyr Developer Summit, we color-coded all of our demos for easy reference. The Green Demo shows off the interconnected features of Golioth. It’s based around the concept of using smart grow lights in a greenhouse to supplement natural sunlight.

A light intensity sensor reports back the actual readings. If it’s a cloudy day, lower intensity readings are detected and the grow lights automatically swing into action, modulating their power usage to dial in the target light intensity. This saves energy, while ensuring the greenhouse yield and harvest dates aren’t affected by the fluctuation of natural light.

The target intensity can be set remotely through the Golioth Console. But for fun, we also made a dashboard you can load on your phone to adjust the setting.

Output Streams for your cloud team’s favorite platform

Your device data is stored on the Golioth Cloud and easy to work with. But your cloud team may already have infrastructure in place on a platform like Microsoft Azure, Amazon AWS, or Google Cloud Platform. No sweat! We can seamlessly connect to those with our output streams feature.

The Blue demo at our Embedded World booth shows off the ability to select which weather sensors (temperature, humidity, pressure) are being recorded, and select between sending the data to any of the three cloud platforms. This is also a great option if your team ever decides to move to a different cloud. You can put multiple output streams in place in order to test out the new setup before moving your production servers.

Mapping data and visualizing everything

Golioth’s Orange Demo started as a way to show that commercially-available hardware can be used with our platform. The Nordic Thingy91 is a cellular device that includes GPS, a sensor suite, and a rechargeable battery. This demo uses it as an asset tracker, caching GPS readings and uploading them to Golioth every few minutes.

But data is only useful if you can do something with it. We used the Grafana platform to visualize the data on a map interface. In fact, we have built up Grafana dashboards for all of the demos. It’s easy to make the data wow your customers with this interface. You can plot historical data using our REST API, and real-time data appears as soon as it’s received thanks to the open source Grafana plugin we wrote earlier this year.

Embedded World through Golioth’s eyes

Alas, only a small part of the IoT community will make it to Embedded World this year. Don’t worry, we’ll be posting video walkthroughs of our demos on our YouTube channel shortly after the conference. But don’t forget to follow us on Twitter as well, we’ll make sure to share the coolest things we see at the conference. And the offer to set up a chat is still good, even if it’s not in real life—the Developer Relations hotline is open!

Have you tried to use the Golioth Web Console yet? The interface delivers access and control for your entire IoT fleet. This means sending and receiving data in real-time, checking on the state of each device (including current firmware revision), reading logs from the hardware, and more.

Did you know that you can create your own User Interface (UI) like this using any stack you want? That’s because we used REST and WebSocket APIs to build the Golioth Console, and those same APIs are available for you to build any application to fit for our needs.

In this blog, post I will show how simple it is to set up a custom UI project using the Golioth APIs.

To demonstrate, we built a web interface around the image of an Adafruit MagTag, a development board we use for our developer training program. It’s pretty cool to click a button on the the image above and see the device in your hand react. Let’s dive in as I cover the necessary steps to pull it all together.

Golioth Cloud

We begin by using the Golioth Web Console to setup a project and create API Keys. You can also follow the authentication docs where it has a guide for you to do the same but using Golioth Command Line Interface (CLI).

  1. Sign-up/Sign-in on the Golioth Web Console
    • If you created a brand new account, you will see a QuickStart Wizard that guides you through the process of creating a new project and adding a new device
  2. Choose your project using the project selector at the middle-top (I’m calling mine Project 1)
  3. Create an API Key to authenticate on the API’s:
    • Clicking on Create, select API Key and hit Create at the pop-up;
  4. Create a Device
    • Go to Devices at the sidebar and hit Create
    • I’m calling mine magtag, and generating credentials automatically

Firmware

LightDB State - MagTag

LightDB State – MagTag

My physical MagTag is running the latest version of the MagTag Sample using Golioth’s Arduino SDK. With that firmware, my device is set up to use LightDB State to read and set data. That means it will listen for changes on the desired endpoint and update it’s status both physically and on the cloud’s state endpoint to match.

This state pattern is related to the concept of Digital Twin, which you can read more on our desired state vs. actual state blog post.

Front-end

The front-end was bootstrapped using Vite’s react-ts template. I’m also using Mantine as a component library for this application.

To set up our connection to both the REST and WebSocket APIs, we first need to understand how it should be built. For that, you can easily go to Golioth REST API Docs and Golioth WebSocket API Docs and follow through those steps.

The gist of it is that we need a few pieces of information to authenticate. This includes the Golioth API URL, the project ID we are targeting, and an API key from that project. Since we’re going to target a specific device as well, I also added the Device ID.

With those fields, we’re able to connect and listen to state changes using the WebSocket API. But we also want to display the current state, and be able to update the desired state from here.

Displaying Current State

The data coming from the current state allows us to build a virtual replica of the MagTag on the UI. With some CSS magic we can display its text, leds and light.

{
  "accX": 0.32,
  "accY": 0.11,
  "accZ": -9.89,
  "leds": {
    "0": "#00a2ff",
    "1": "#00ffee",
    "2": "#00ff84",
    "3": "#04ff00"
  },
  "light": 34,
  "text": "Golioth"
}

The theme is also reacting to the light sensor readings from the device. The UI changes to dark mode if the light level is low.

Setting Desired State

To set the desired state, we’re going to send some POST requests using our REST API, using the same API Keys that we input on the form.

LEDs:

Using a modal and a color picker, we can select the colors and press Save.

URL:

https://api.golioth.io/v1/projects/project-1/devices/6283eedd71e8739f42672114/data/desired/leds

Payload:

{
    "3": "#ff0000"
}

Using the same modal and form, we can set all leds states, by clicking on top of the led position.

Text:

Using the same logic, we can also use a modal and a textarea to set the text, with preserved line breaks.

URL:

https://api.golioth.io/v1/projects/project-1/devices/6283eedd71e8739f42672114/data/desired

Payload:

{
    "text": "MagTag"
}

Buttons:

Buttons don’t have an actual state, so here I just added some boxes on top of their positions on the MagTag image.

This way, when we click on them, the application will change the desired buzz state to true.

URL:

https://api.golioth.io/v1/projects/project-

/devices/6283eedd71e8739f42672114/data/desired/buzz

Payload:

true

LightDB State Result:

"desired": {
  "buzz": true,
  "leds": {
    "0": "#ff0000",
    "1": "#00ffee",
    "2": "#00ff84",
    "3": "#1ad57a"
  },
  "text": "MagTag"
}

The device is also listening for changes on the desired state, so when the buzz state changes to true, the device will take action. In this case, the actual MagTag will emit a buzz, and then change the desired buzz state back to 0.

Just a taste of what you can do with a custom UI

The simple steps I’ve shown here are just the beginning of what you can accomplish with your own custom user interface. If you are managing fleets for a customer, you may want to give them a simpler interface that only includes relevant controls and data. Whether it’s a Digital Twin like the MagTag in my example, or more traditional web interface, knowing that Golioth will work for any web-control need you come across is yet another powerful tool to have at your disposal.

See it in action

The Golioth Zephyr SDK has a new name, a new recommended install method, and a new recommended install directory name.

If you installed our SDK prior to May 2022, now is a great time to make one change to your manifest file and pull the newest version. We’ll walk you through that in the next section, but first let’s discuss what changed, and why we’re excited about it!

Last week we changed the name of our SDK from zephyr-sdk to golioth-zephyr-sdkto make it clear this code is for using Golioth device management features with Zephyr. We also updated our recommended install directory names to golioth-zephyr-workspace and golioth-ncs-workspace.

This second change differentiates the “vanilla” version of Zephyr from the specialized “nRF Connect SDK” (NCS) version of Zephyr that Nordic Semiconductor maintains for chips like the nRF52 and the nRF9160. It also prepares the way for Golioth to expand our platform support beyond Zephyr, which is extremely exciting for us.

For new installs, our getting started guide for ESP32 or for nRF9160 have already been updated and you won’t notice the difference. For existing installs, read on for simple steps to keep your local copy in sync with this new development.

Existing Golioth Zephyr SDK installs: How to update

A small manual update needs to be made to any Golioth SDK that was installed prior to May of 2022. If you previously followed our getting started docs, you have a folder called ~/zephyrproject for the Zephyr version of our SDK, or ~/zephyr-nrf for the NCS (nRF Connect SDK) version of our SDK.

Begin in that directory:

1. Edit the .west/config file

If you have the Golioth Zephyr SDK installed, change the manifest section of ~/zephyrproject/.west/config to match the following:

[manifest]
path = modules/lib/golioth
file = west-zephyr.yml

If you have the NCS version of the Golioth Zephyr SDK installed, change the manifest section of ~/zephyr-nrf/.west/config to match the follow:

[manifest]
path = modules/lib/golioth
file = west-ncs.yml

2. Update your Golioth remote, then pull and update the SDK

cd modules/lib/golioth
git checkout main
git remote set-url origin https://github.com/golioth/golioth-zephyr-sdk.git
git pull
west update

That’s it, your SDK is now up to date!

What changed for new installs: west init option and directory names

The install instructions for the Golioth SDK are very similar to what they were before this change. The most obvious difference is that we’ve moved away from using west.yml as the manifest file and instead use west-zephyr.yml for vanilla Zephyr, or west-ncs.yml for the Nordic “flavor” of Zephyr. When calling west init, we use a flag to chose one of these manifest files:

#Installing the Golioth Zephyr SDK:
west init -m https://github.com/golioth/golioth-zephyr-sdk.git --mf west-zephyr.yml ~/golioth-zephyr-workspace
cd golioth-zephyr-workspace
west update
#Installing the Golioth NCS SDK:
west init -m https://github.com/golioth/golioth-zephyr-sdk.git --mf west-ncs.yml ~/golioth-ncs-workspace
cd golioth-ncs-workspace
west update

This makes the installation process, and the update process for both approaches the same which it wasn’t before.

The old install directory (zephyrproject and zephry-nrf) have been updated to golioth-zephyr-workspace and golioth-ncs-workspace. This change does two things to better describe the contents of these directories. First, they are much more specific about the intended purpose of the folder contents. Second, calling these directories a workspace helps with understanding that you will find multiple repositories inside of each directory (the Golioth SDK, the Zephyr Project RTOS, and the nRF Connect SDK will also be there for NCS installs).

Golioth is a Rolling Stone

We are constantly improving how Golioth empowers you to manage your IoT infrastructure. These changes to the Golioth Zephyr SDK deliver a better workflow, and prepare Golioth to add SDKs for additional platforms in the future. We are dedicated to updating our users about changes that might impact their setup and tooling. If you have any questions on these changes, or want help getting up to speed with our tools, we’d love to hear from you on the Golioth Discord server!

Golioth connected to GCP PubSub

Golioth Output Streams enable IoT deployments built on the Golioth platform to export events to traditional cloud provider platforms. We now have support for Google Pub/Sub as a way to export your data into the GCP ecosystem.

In our initial announcement about Output Streams, we had exports for AWS SQS, Azure Event Hubs, and WebHooks. The first two act as buffers for the massive amounts of data coming into cloud systems from all over the web. These managed services can help to smooth out bursts of different requests hitting a server, acting as a queuing system so servers and serverless elements have time to “catch up” with the influx of new data arriving, ready to be processed. As deployments grow, they can also introduce redundancy and caching for different regions of the globe, while still acting like a monolithic service to the functions receiving input or providing output to these buffers.

Another cloud?

So why do we need yet another cloud service to receive data? Well, because users prefer it! With AWS, Azure, and now Google Cloud, we cover roughly 64% of the cloud infrastructure market according to Statista in Q4 of 2021.

Cloud market share Q4 2021

Image courtesy of Statista

We think it’s a bit crazy that cloud teams determine how IoT device makers run the services that go onto small devices. That means that an IoT Platform like Golioth shouldn’t make decisions going in the other direction! Some Golioth clients who are development shops have 3 different clients that want their data to go to 3 different cloud providers. Dev shops need to be nimble and have a lot of output options to serve their clients’ needs. Even within organizations at large companies, data might be flying to one provider or another. Software teams have reasons they choose one cloud provider over another, often far above the decision process of any particular engineer. We’re here to tell you, we don’t care where you send data. Your data should be able to go where you want it to go.

As a reminder, data can also stay on Golioth! Say you’re a hardware engineer looking to try things out and stream some sensor data to the cloud for visualization. You can directly interact with your data on the Golioth Cloud, including controlling your devices via our REST API. You can also use the WebHooks feature to push data out to charting programs like Datacake, Ubidots, or Grafana. You don’t need to export it immediately to an external provider, it’s just that some people have that as a business need given what their cloud teams are using.

Can I use Pub/Sub Directly?

It might be tempting to think that since you’re using Google’s Pub/Sub service, you could have your device talking directly to the service. It’s there to listen to different events, right?

This is a good opportunity to point out how Golioth is optimized to talk to constrained IoT devices. You could take a board using a cellular part like the nRF9160 and have it connect to the internet. If it wanted to talk to Google Pub/Sub, it would need to be making HTTP/2 requests and have authentication on the level that is usually required for one server talking to another. The amount of certificates, data overhead, and bandwidth would be a terrible choice for the nRF9160.

Instead, you could have your nRF9160 running a Zephyr application with the Golioth SDK talking to Golioth servers over CoAP, with CBOR encoding on the messages, and sending and receiving secure messages using LightDB. This will save bandwidth (data costs) and power for your cellular device. What’s more, you get great features like fleet management and OTA firmware updates through that same secure connection to your device. As you scale your fleet up to thousands or millions of devices, you’re still talking back to a similar connection provided by Golioth, but you’re piping the resulting data out to your cloud provider. When you’re ready to switch to a different one? Flip a switch! We’ll have even more cloud export options soon.

Implementing remote firmware updates is one of the most critical steps towards building a resilient IoT deployment. In short: it allows you to change your device firmware in the field without being physically present with the device. This can be critical for security fixes, for feature upgrades, and for extending the lifetime of your IoT implementation. Today we’re going to be looking at Over-The-Air (OTA) firmware updates (DFU) for the Nordic Semiconductor nRF9160 cellular module (SIP). The associated video explanation and walkthrough (below) was published on our YouTube channel a couple months ago, but we’re reviewing it on our blog today.

Why are IoT firmware updates difficult?

There are many difficult components to OTA DFU, including on the system design level, at the device level, and on the hosting service. At a system level view, the hard part of a firmware update is knowing which device should be receiving the update. From the device perspective, it’s critical to be able to recover from any issues during the update and then verify the complete update was received; using cellular as the method of connectivity means there may be intermittent connectivity. With embedded devices, there are fewer ways to have a device intelligently recover, as may be the case with a full Linux system.  On the cloud side, having a reliable hosting and delivery system for firmware packages is also important. Simply sticking files on a FTP server represents a security risk and puts an outsized burden on the logic of the device in the field.

Firmware updates are easier with Golioth

Golioth provides a data pipe to cellular devices and other devices in the Zephyr ecosystem as a one stop shop for all data connectivity. This includes state data handling (digital twin), data streams coming back to the cloud, and logging.  Once that data pipe is established, we can also push firmware updates down to devices utilizing transport layers like CoAP.

We also benefit from the fact that the Zephyr project utilizes MCUBoot as the default bootloader. This open source bootloader (to which we are contributing members!) makes it easy to interact with from within an application. It’s as simple as pushing an updated firmware binary into memory and then calling an API to use that code upon reboot. Our recent blog posts around the ESP32 showcased how the newly-published port of MCUboot to the ESP32 platform enabled DFU on those parts. The Nordic Semiconductor nRF9160 has had this capability from the beginning because of its deep integration in the Zephyr ecosystem.

A key point shown in the video below is the signing of images using MCUboot.  Having a signed firmware binary means there is a way to check that the entire package is there. From a security perspective, the signing allows the device to validate that the firmware image that has arrive is secure and unchanged in transit.

Managing firmware rollout

The Golioth Console enables our users to manage a wide variety of devices in their IoT fleet. A deployment with 10,000+ devices would be difficult to manage using scripts alone. Withouth the ability to visualize which devices have been updated or not means there is higher likelihood of costly mistakes.

To aid in partitioning device deployments, users can tag devices to create logical representations of different groupings. You might use a tag to identify some devices as early recipients of a test firmware image. Blueprints can be used to group devices that match a certain hardware profile. In this way, Golioth users can target key portions of their device fleet to ensure that only those specific devices are being updated at any given time.

Artifacts and Releases

Golioth implements two distinct features that will allow for more complex firmware packages. The first is an Artifact. this is simply a binary that is uploaded to the Golioth cloud. For your security, Golioth has no knowledge of what is inside that image, only how it has been signed. As such, this might be a firmware image, a JPEG that will be displayed on a smart screen. An artifact can also be a binary blob being delivered to a device modem, or even firmware that’s being pushed down the line to other microcontrollers in the system. Each of these artifacts can be versioned and assigned a blueprint, so that they can only be deployed onto the eligible systems.

The second feature is the idea of a Release. Users can take individual Artifacts and bundle them together as a Release. For example, a Release might have an image to display on the screen, the base application firmware for the device we’re targeting (the nRF9160), and a firmware image for an auxiliary processor on board (e.g., an nRF52). Once the user is ready to deploy, they create a Release with matching tags and blueprints and an assigned Release version. Devices in the field that match the criteria set by the user on the console (tags, blueprints) will then be notified that a new device firmware release is available for them. The device in the field will start the firmware update process, downloading chunks of data.

Once the image is verified as complete, the device will utilize the MCUBoot API on the device and restart using that new image. In the event the Golioth user wants to roll back changes, they simply slide a switch on the Golioth Console and the device is alerted to download/restart the previous version of firmware. In this way, Golioth users have the ultimate flexibility for delivering firmware devices to a laser focused set of devices in their fleet.

Watch the demo

See this all happening in real time, including compiling a new firmware image, uploading it to the Golioth Console and watching the OTA DFU happen on the nRF9160.