Golioth is a device management cloud that is easy to add to any Zephyr project. Just list the Golioth Zephyr SDK as a module in your manifest file and the west tool will do the rest. Well, almost. Today I’ll walk through how to add Golioth to an existing Zephyr project. As an example, I’ll be using our hello sample. We chose this because it already has networking, which makes explaining things a bit easier. Generally any board that has led0 defined in the device tree and you can enable networking should be a good fit. What we want to do here is showcase the elements that allow you to add Golioth, so let’s dive in!

0. Understanding the west manifest

A Zephyr workspace is made up of a number of different code repositories all stored in the same tree. Zephyr uses a west manifest file to manage all of these repositories, including information like the origin URL of the repo, the commit hash to use, and where each repository should be placed in your local tree.

Think of the manifest as a code repository shopping list that the west update command uses to fill up your local Zephyr tree. There may be more than one manifest file, but today we’ll just focus on the main manifest.

1. Add Golioth to the west manifest

Start by locating your manifest file and opening it with a code editor.

~/zephyrproject $ west manifest --path
/home/mike/zephyrproject/zephyr/west.yml

Under projects: add an entry for the Golioth Zephyr SDK (highlighted in the abbreviated manifest sample below).

manifest:
  defaults:
    remote: upstream
 
  remotes:
    - name: upstream
      url-base: https://github.com/zephyrproject-rtos
 
  #
  # Please add items below based on alphabetical order
  projects:
      # Golioth repository.
    - name: golioth
      path: modules/lib/golioth
      revision: v0.2.0
      url: https://github.com/golioth/golioth-zephyr-sdk.git
      import:
        west-external.yml
    - name: canopennode
      revision: 53d3415c14d60f8f4bfca54bfbc5d5a667d7e724
...

Note that I have called out the v0.2.0release tag. You can set this to any of our release tags, to main, or to a specific commit.

Now run an update to incorporate the manifest changes:

west update

2. Select libraries to add to the build

We use KConfig to build in the libraries that Golioth needs. These changes are made in the prj.conf file of your application.

The first set of symbols deals with Zephyr’s networking stack. Of course every Internet of Things thing needs a network connection so you likely already have these symbols selected.

# Generic networking options
CONFIG_NETWORKING=y
CONFIG_NET_IPV4=y
CONFIG_NET_IPV6=n

Golioth is secure by default so we want to select the mbedtls libraries to handle the encryption layer.

# TLS configuration
CONFIG_MBEDTLS_ENABLE_HEAP=y
CONFIG_MBEDTLS_HEAP_SIZE=10240
CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=2048

Now let’s turn on the Golioth libraries and give them a bit of memory they can dynamically allocate from.

# Golioth Zephyr SDK
CONFIG_GOLIOTH=y
CONFIG_GOLIOTH_SYSTEM_CLIENT=y
 
CONFIG_MINIMAL_LIBC_MALLOC_ARENA_SIZE=256

Every device needs its own credentials because all connections to Golioth require security. There are a few ways to do this, but perhaps the simplest is to add them to the prj.conf file.

# Golioth credentials
CONFIG_GOLIOTH_SYSTEM_CLIENT_PSK_ID="220220711145215-blinky1060@golioth-settings-demo"
CONFIG_GOLIOTH_SYSTEM_CLIENT_PSK="cab43a035d4fe4dca327edfff6aa7935"

And finally, I’m going to enable back-end logging. This is not required for connecting to Golioth, but sending the Zephyr logs to the cloud is a really handy feature for remote devices.

# Optional for sending Zephyr logs to the Golioth cloud
CONFIG_NET_LOG=y
CONFIG_LOG_BACKEND_GOLIOTH=y
CONFIG_LOG_PROCESS_THREAD_STACK_SIZE=2048

I separated each step above for the sake of explanation. But the combination of these into one block is the boiler-plate configuration that I use on all of my new projects. See this all as one file in the prj.conf from our hello sample.

3. Instantiate the Golioth system client and say hello

So far we added Golioth as a Zephyr module and enabled the libraries using KConfig. Now it’s time to use the Golioth APIs in the main.c file.

The first step is to include the header files and instantiate a golioth_client object. The client is used to manage the connection with the Golioth servers. Including the coap header file is not strictly required to establish a connection, but it is necessary for processing the responses from Golioth, so I always include it.

#include <net/coap.h>
#include <net/golioth/system_client.h>
static struct golioth_client *client = GOLIOTH_SYSTEM_CLIENT_GET();

I also add a callback function to handle messages coming back from the Golioth cloud. Technically this is optional, but if you want two-way communication with the cloud you need it!

/* Callback for messages received from the Golioth servers */
static void golioth_on_message(struct golioth_client *client,
                   struct coap_packet *rx)
{
    uint16_t payload_len;
    const uint8_t *payload;
    uint8_t type;
 
    type = coap_header_get_type(rx);
    payload = coap_packet_get_payload(rx, &payload_len);
 
    printk("%s\n", payload);
}

In the main function, I register my callback, start the Golioth client, and call the golioth_send_hello() API.

/* Register callback and start Golioth system client */
client->on_message = golioth_on_message;
golioth_system_client_start();
 
while (1) {
    /* Say hello to Golioth */
    int ret = golioth_send_hello(client);
    if (ret) {
        printk("Failed to send hello! %d\n", ret);
    }
    else printk("Hello sent!\n");
}

When successful, the golioth_send_hello() call will prompt for a message back from the server which includes the name of the sending device. This is printed out by the golioth_on_message() callback.

Hello sent!
Hello blinky1060
Hello sent!
Hello blinky1060
Hello sent!
Hello blinky1060

Extra Credit: Back-end Logging

Observant readers have noticed that I enabled back-end logging using KConfig symbols but I didn’t use it in the C code. Here’s the really awesome part: just use Zephyr logging as you normally would and it will automatically be sent to your Golioth console.

At the top of main.c, include the log header file and register a logging module.

#include <logging/log.h>
LOG_MODULE_REGISTER(any_name_you_want, LOG_LEVEL_DBG);

In our C functions, call logs as you normal would.

LOG_ERR("An error message goes here!");
LOG_WRN("Careful, this is a warning!");
LOG_INF("Did you now that this is an info log?");
LOG_DBG("Oh no, why isn't my app working? Maybe this debug log will help.");

Now your log messages will appear on the Golioth console!

Further Reading

This covers the basics of adding Golioth to an existing Zephyr project. You can see the steps all in one place by looking at the Golioth code samples. Here are the examples that you’ll find immediately useful:

  • Hello – basics of connecting to Golioth and sending log messages (what was covered in this post)
  • Lightdb Set – send data to Golioth
  • Lightdb Observe – device will be notified whenever an endpoint on the Golioth cloud is updated

To go even deeper, you can see how we use the Golioth Over-the-Air (OTA) firmware update feature, and how to use the Zephyr settings subsystem for persistent credential storage. And remember, the Dev Tier of Golioth includes the first 50 devices, so you can try all of this out for free.

Image from Todd Lappin on flickr

This post was written about Thread in July 2022. We are leaving it here for historical purposes, but we suggest that you check out some of our more recent work with Thread on the Golioth Reference Design template and using commercially available Thread Border Routers.

Thread (and its common implementation known as OpenThread) is a networking technology that is quickly gaining adoption due to the forthcoming Matter standard. Thread has been around for many years, but as it grows, it becomes more accessible using off-the-shelf firmware and hardware. So it’s accessible, but maybe not all that straightforward. This post will help with that.

As we have more customers talking to us about using Golioth as a management layer for both Matter solutions and for standalone industrial Thread networks, we thought we should expand our tools to better fit the needs of engineers designing new systems.

In this article and associated examples, we will show you how to set up a network at home using common components. We’ll build a custom device that is communicating back over that network utilizing features of Golioth that extend the networking layer: over-the-air firmware updates, time series data tracking, command/control structures, and instant logging.

Parts of this Thread Guide

In the past, we have written about the basics of getting a Thread demo up and running, and shown it working on video. Today we build upon that work and showcase a new set of resources so you can build your own Thread network with custom devices.

  • YouTube video – A walkthrough of the setup steps and troubleshooting steps with a newly provisioned Thread network.
  • A tutorial site – Follow along with a simplified set of directions for replicating what we have built. We think is the shortest path towards getting a working Thread network on your desk or bench.
  • Code repository – Start from working code on the node devices to see how you can customize code and have your sensor data streaming back to Golioth quickly.
  • This blog post

Getting Started

The majority of the step-by-step instructions are contained in the tutorial site for Golioth and OpenThread. If you’re interested in immediately replicating and then extending our setup, head over to the tutorial site to learn more.

Let’s take a higher level look at the OpenThread Border Router (OTBR), OpenThread nodes, and Golioth device management layer that make up this Thread network example.

OpenThread Border Router  (OTBR)

This is the key part in a Thread network, as it allows any arbitrary number of nodes that are communicating with one another (meshing) to then reach the outside internet. Each Thread device has an IPv6 address, which is great: That means a node that is meshing with 30 other nodes and connected to the internet (through an OTBR) is directly addressable from the internet. That’s an important piece. We might expect a higher power WiFi based device to have an IP address (assigned from a router), but probably not a low power sensor. The OTBR does a lot of the routing of information and translation of packets coming from node devices.

We build a DIY version of the OTBR because there aren’t many commercially available (yet). We use a Raspberry Pi and an nRF52840 Dongle to create a pipeline out to the wider internet.

OpenThread Nodes

In our first video/blog about Thread, we had nodes talking to the internet through an OTBR. However, the nodes only blinked and sent back logging messages and we didn’t give detailed instructions on how to build them. In the tutorial site and the video, we show how we can execute arbitrary code to do higher level functions, like data logging. We use the Laird Connectivity BT510, which is a sensor node built with the Nordic Semiconductor nRF52840. It also has a range of sensors built in and is contained in a waterproof case. We think it’s a great platform for building a small, reliable Thread network and we used it in our Red Demo that we showcased at the 2022 Zephyr Developer Summit and Embedded World.

The BT510 is a board already supported in Zephyr, which means we can very easily compile firmware for it and access all of the sensor drivers that are built into Zephyr, no custom out-of-tree code required. We use the OpenThread networking stack that is native to Zephyr, and the Golioth SDK, which allows each node to be pre-configured to talk to the Golioth servers. We then enable things like LightDB Stream to regularly send back sensor data from the device through the Thread network.

Golioth Device Management Layer

The Golioth Device Management Layer/Platform is already ready for you. If you don’t have an account, you can sign up on the Golioth Console, which will guide you through creating your first device on the platform; you can use the credentials for that digital version of a device to control your first Thread node.

Once your Thread device is connected, you’ll be able to see how often the device is connecting, view the latest data and logs being sent back from the device, and check which firmware versions are on each device. Any data sent to the Golioth platform from a Thread node can be aggregated into an external visualization platform, or wholesale exported to 3rd party services (AWS, Azure, GCP) using Output Streams.

What will you build?

We are sharing the know-how to build a Thread network. Following this guide enables all of your devices on the Thread network to communicate back to the wider internet. As a hardware engineer, I don’t really want to mess about with the network layer, I just want something that works. Instead, I’d rather focus on the end application and building end devices (nodes) that are useful to customers and users.

With Golioth, Zephyr, OpenThread, and some off-the-shelf hardware, you can get started quickly and you can start to connect custom devices with a powerful interface to the internet. What will you build? Please let us know on our Discord, Forum, or on Twitter.

Have you tried to use the Golioth Web Console yet? The interface delivers access and control for your entire IoT fleet. This means sending and receiving data in real-time, checking on the state of each device (including current firmware revision), reading logs from the hardware, and more.

Did you know that you can create your own User Interface (UI) like this using any stack you want? That’s because we used REST and WebSocket APIs to build the Golioth Console, and those same APIs are available for you to build any application to fit for our needs.

In this blog, post I will show how simple it is to set up a custom UI project using the Golioth APIs.

To demonstrate, we built a web interface around the image of an Adafruit MagTag, a development board we use for our developer training program. It’s pretty cool to click a button on the the image above and see the device in your hand react. Let’s dive in as I cover the necessary steps to pull it all together.

Golioth Cloud

We begin by using the Golioth Web Console to setup a project and create API Keys. You can also follow the authentication docs where it has a guide for you to do the same but using Golioth Command Line Interface (CLI).

  1. Sign-up/Sign-in on the Golioth Web Console
    • If you created a brand new account, you will see a QuickStart Wizard that guides you through the process of creating a new project and adding a new device
  2. Choose your project using the project selector at the middle-top (I’m calling mine Project 1)
  3. Create an API Key to authenticate on the API’s:
    • Clicking on Create, select API Key and hit Create at the pop-up;
  4. Create a Device
    • Go to Devices at the sidebar and hit Create
    • I’m calling mine magtag, and generating credentials automatically

Firmware

LightDB State - MagTag

LightDB State – MagTag

My physical MagTag is running the latest version of the MagTag Sample using Golioth’s Arduino SDK. With that firmware, my device is set up to use LightDB State to read and set data. That means it will listen for changes on the desired endpoint and update it’s status both physically and on the cloud’s state endpoint to match.

Note: The arduino-sdk repository showcased in this post is deprecated. GoliothLabs has two experimental repositories that may work as replacements:

This state pattern is related to the concept of Digital Twin, which you can read more on our desired state vs. actual state blog post.

Front-end

The front-end was bootstrapped using Vite’s react-ts template. I’m also using Mantine as a component library for this application.

To set up our connection to both the REST and WebSocket APIs, we first need to understand how it should be built. For that, you can easily go to Golioth REST API Docs and Golioth WebSocket API Docs and follow through those steps.

The gist of it is that we need a few pieces of information to authenticate. This includes the Golioth API URL, the project ID we are targeting, and an API key from that project. Since we’re going to target a specific device as well, I also added the Device ID.

With those fields, we’re able to connect and listen to state changes using the WebSocket API. But we also want to display the current state, and be able to update the desired state from here.

Displaying Current State

The data coming from the current state allows us to build a virtual replica of the MagTag on the UI. With some CSS magic we can display its text, leds and light.

{
  "accX": 0.32,
  "accY": 0.11,
  "accZ": -9.89,
  "leds": {
    "0": "#00a2ff",
    "1": "#00ffee",
    "2": "#00ff84",
    "3": "#04ff00"
  },
  "light": 34,
  "text": "Golioth"
}

The theme is also reacting to the light sensor readings from the device. The UI changes to dark mode if the light level is low.

Setting Desired State

To set the desired state, we’re going to send some POST requests using our REST API, using the same API Keys that we input on the form.

LEDs:

Using a modal and a color picker, we can select the colors and press Save.

URL:

https://api.golioth.io/v1/projects/project-1/devices/6283eedd71e8739f42672114/data/desired/leds

Payload:

{
    "3": "#ff0000"
}

Using the same modal and form, we can set all leds states, by clicking on top of the led position.

Text:

Using the same logic, we can also use a modal and a textarea to set the text, with preserved line breaks.

URL:

https://api.golioth.io/v1/projects/project-1/devices/6283eedd71e8739f42672114/data/desired

Payload:

{
    "text": "MagTag"
}

Buttons:

Buttons don’t have an actual state, so here I just added some boxes on top of their positions on the MagTag image.

This way, when we click on them, the application will change the desired buzz state to true.

URL:

https://api.golioth.io/v1/projects/project-

/devices/6283eedd71e8739f42672114/data/desired/buzz

Payload:

true

LightDB State Result:

"desired": {
  "buzz": true,
  "leds": {
    "0": "#ff0000",
    "1": "#00ffee",
    "2": "#00ff84",
    "3": "#1ad57a"
  },
  "text": "MagTag"
}

The device is also listening for changes on the desired state, so when the buzz state changes to true, the device will take action. In this case, the actual MagTag will emit a buzz, and then change the desired buzz state back to 0.

Just a taste of what you can do with a custom UI

The simple steps I’ve shown here are just the beginning of what you can accomplish with your own custom user interface. If you are managing fleets for a customer, you may want to give them a simpler interface that only includes relevant controls and data. Whether it’s a Digital Twin like the MagTag in my example, or more traditional web interface, knowing that Golioth will work for any web-control need you come across is yet another powerful tool to have at your disposal.

See it in action

The Golioth Zephyr SDK has a new name, a new recommended install method, and a new recommended install directory name.

If you installed our SDK prior to May 2022, now is a great time to make one change to your manifest file and pull the newest version. We’ll walk you through that in the next section, but first let’s discuss what changed, and why we’re excited about it!

Last week we changed the name of our SDK from zephyr-sdk to golioth-zephyr-sdkto make it clear this code is for using Golioth device management features with Zephyr. We also updated our recommended install directory names to golioth-zephyr-workspace and golioth-ncs-workspace.

This second change differentiates the “vanilla” version of Zephyr from the specialized “nRF Connect SDK” (NCS) version of Zephyr that Nordic Semiconductor maintains for chips like the nRF52 and the nRF9160. It also prepares the way for Golioth to expand our platform support beyond Zephyr, which is extremely exciting for us.

For new installs, our getting started guide for ESP32 or for nRF9160 have already been updated and you won’t notice the difference. For existing installs, read on for simple steps to keep your local copy in sync with this new development.

Existing Golioth Zephyr SDK installs: How to update

A small manual update needs to be made to any Golioth SDK that was installed prior to May of 2022. If you previously followed our getting started docs, you have a folder called ~/zephyrproject for the Zephyr version of our SDK, or ~/zephyr-nrf for the NCS (nRF Connect SDK) version of our SDK.

Begin in that directory:

1. Edit the .west/config file

If you have the Golioth Zephyr SDK installed, change the manifest section of ~/zephyrproject/.west/config to match the following:

[manifest]
path = modules/lib/golioth
file = west-zephyr.yml

If you have the NCS version of the Golioth Zephyr SDK installed, change the manifest section of ~/zephyr-nrf/.west/config to match the follow:

[manifest]
path = modules/lib/golioth
file = west-ncs.yml

2. Update your Golioth remote, then pull and update the SDK

cd modules/lib/golioth
git checkout main
git remote set-url origin https://github.com/golioth/golioth-zephyr-sdk.git
git pull
west update

That’s it, your SDK is now up to date!

What changed for new installs: west init option and directory names

The install instructions for the Golioth SDK are very similar to what they were before this change. The most obvious difference is that we’ve moved away from using west.yml as the manifest file and instead use west-zephyr.yml for vanilla Zephyr, or west-ncs.yml for the Nordic “flavor” of Zephyr. When calling west init, we use a flag to chose one of these manifest files:

#Installing the Golioth Zephyr SDK:
west init -m https://github.com/golioth/golioth-zephyr-sdk.git --mf west-zephyr.yml ~/golioth-zephyr-workspace
cd golioth-zephyr-workspace
west update
#Installing the Golioth NCS SDK:
west init -m https://github.com/golioth/golioth-zephyr-sdk.git --mf west-ncs.yml ~/golioth-ncs-workspace
cd golioth-ncs-workspace
west update

This makes the installation process, and the update process for both approaches the same which it wasn’t before.

The old install directory (zephyrproject and zephry-nrf) have been updated to golioth-zephyr-workspace and golioth-ncs-workspace. This change does two things to better describe the contents of these directories. First, they are much more specific about the intended purpose of the folder contents. Second, calling these directories a workspace helps with understanding that you will find multiple repositories inside of each directory (the Golioth SDK, the Zephyr Project RTOS, and the nRF Connect SDK will also be there for NCS installs).

Golioth is a Rolling Stone

We are constantly improving how Golioth empowers you to manage your IoT infrastructure. These changes to the Golioth Zephyr SDK deliver a better workflow, and prepare Golioth to add SDKs for additional platforms in the future. We are dedicated to updating our users about changes that might impact their setup and tooling. If you have any questions on these changes, or want help getting up to speed with our tools, we’d love to hear from you on the Golioth Discord server!

Golioth connected to GCP PubSub

Golioth Output Streams enable IoT deployments built on the Golioth platform to export events to traditional cloud provider platforms. We now have support for Google Pub/Sub as a way to export your data into the GCP ecosystem.

In our initial announcement about Output Streams, we had exports for AWS SQS, Azure Event Hubs, and WebHooks. The first two act as buffers for the massive amounts of data coming into cloud systems from all over the web. These managed services can help to smooth out bursts of different requests hitting a server, acting as a queuing system so servers and serverless elements have time to “catch up” with the influx of new data arriving, ready to be processed. As deployments grow, they can also introduce redundancy and caching for different regions of the globe, while still acting like a monolithic service to the functions receiving input or providing output to these buffers.

Another cloud?

So why do we need yet another cloud service to receive data? Well, because users prefer it! With AWS, Azure, and now Google Cloud, we cover roughly 64% of the cloud infrastructure market according to Statista in Q4 of 2021.

Cloud market share Q4 2021

Image courtesy of Statista

We think it’s a bit crazy that cloud teams determine how IoT device makers run the services that go onto small devices. That means that an IoT Platform like Golioth shouldn’t make decisions going in the other direction! Some Golioth clients who are development shops have 3 different clients that want their data to go to 3 different cloud providers. Dev shops need to be nimble and have a lot of output options to serve their clients’ needs. Even within organizations at large companies, data might be flying to one provider or another. Software teams have reasons they choose one cloud provider over another, often far above the decision process of any particular engineer. We’re here to tell you, we don’t care where you send data. Your data should be able to go where you want it to go.

As a reminder, data can also stay on Golioth! Say you’re a hardware engineer looking to try things out and stream some sensor data to the cloud for visualization. You can directly interact with your data on the Golioth Cloud, including controlling your devices via our REST API. You can also use the WebHooks feature to push data out to charting programs like Datacake, Ubidots, or Grafana. You don’t need to export it immediately to an external provider, it’s just that some people have that as a business need given what their cloud teams are using.

Can I use Pub/Sub Directly?

It might be tempting to think that since you’re using Google’s Pub/Sub service, you could have your device talking directly to the service. It’s there to listen to different events, right?

This is a good opportunity to point out how Golioth is optimized to talk to constrained IoT devices. You could take a board using a cellular part like the nRF9160 and have it connect to the internet. If it wanted to talk to Google Pub/Sub, it would need to be making HTTP/2 requests and have authentication on the level that is usually required for one server talking to another. The amount of certificates, data overhead, and bandwidth would be a terrible choice for the nRF9160.

Instead, you could have your nRF9160 running a Zephyr application with the Golioth SDK talking to Golioth servers over CoAP, with CBOR encoding on the messages, and sending and receiving secure messages using LightDB. This will save bandwidth (data costs) and power for your cellular device. What’s more, you get great features like fleet management and OTA firmware updates through that same secure connection to your device. As you scale your fleet up to thousands or millions of devices, you’re still talking back to a similar connection provided by Golioth, but you’re piping the resulting data out to your cloud provider. When you’re ready to switch to a different one? Flip a switch! We’ll have even more cloud export options soon.

Implementing remote firmware updates is one of the most critical steps towards building a resilient IoT deployment. In short: it allows you to change your device firmware in the field without being physically present with the device. This can be critical for security fixes, for feature upgrades, and for extending the lifetime of your IoT implementation. Today we’re going to be looking at Over-The-Air (OTA) firmware updates (DFU) for the Nordic Semiconductor nRF9160 cellular module (SIP). The associated video explanation and walkthrough (below) was published on our YouTube channel a couple months ago, but we’re reviewing it on our blog today.

Why are IoT firmware updates difficult?

There are many difficult components to OTA DFU, including on the system design level, at the device level, and on the hosting service. At a system level view, the hard part of a firmware update is knowing which device should be receiving the update. From the device perspective, it’s critical to be able to recover from any issues during the update and then verify the complete update was received; using cellular as the method of connectivity means there may be intermittent connectivity. With embedded devices, there are fewer ways to have a device intelligently recover, as may be the case with a full Linux system.  On the cloud side, having a reliable hosting and delivery system for firmware packages is also important. Simply sticking files on a FTP server represents a security risk and puts an outsized burden on the logic of the device in the field.

Firmware updates are easier with Golioth

Golioth provides a data pipe to cellular devices and other devices in the Zephyr ecosystem as a one stop shop for all data connectivity. This includes state data handling (digital twin), data streams coming back to the cloud, and logging.  Once that data pipe is established, we can also push firmware updates down to devices utilizing transport layers like CoAP.

We also benefit from the fact that the Zephyr project utilizes MCUBoot as the default bootloader. This open source bootloader (to which we are contributing members!) makes it easy to interact with from within an application. It’s as simple as pushing an updated firmware binary into memory and then calling an API to use that code upon reboot. Our recent blog posts around the ESP32 showcased how the newly-published port of MCUboot to the ESP32 platform enabled DFU on those parts. The Nordic Semiconductor nRF9160 has had this capability from the beginning because of its deep integration in the Zephyr ecosystem.

A key point shown in the video below is the signing of images using MCUboot.  Having a signed firmware binary means there is a way to check that the entire package is there. From a security perspective, the signing allows the device to validate that the firmware image that has arrive is secure and unchanged in transit.

Managing firmware rollout

The Golioth Console enables our users to manage a wide variety of devices in their IoT fleet. A deployment with 10,000+ devices would be difficult to manage using scripts alone. Withouth the ability to visualize which devices have been updated or not means there is higher likelihood of costly mistakes.

To aid in partitioning device deployments, users can tag devices to create logical representations of different groupings. You might use a tag to identify some devices as early recipients of a test firmware image. Blueprints can be used to group devices that match a certain hardware profile. In this way, Golioth users can target key portions of their device fleet to ensure that only those specific devices are being updated at any given time.

Artifacts and Releases

Golioth implements two distinct features that will allow for more complex firmware packages. The first is an Artifact. this is simply a binary that is uploaded to the Golioth cloud. For your security, Golioth has no knowledge of what is inside that image, only how it has been signed. As such, this might be a firmware image, a JPEG that will be displayed on a smart screen. An artifact can also be a binary blob being delivered to a device modem, or even firmware that’s being pushed down the line to other microcontrollers in the system. Each of these artifacts can be versioned and assigned a blueprint, so that they can only be deployed onto the eligible systems.

The second feature is the idea of a Release. Users can take individual Artifacts and bundle them together as a Release. For example, a Release might have an image to display on the screen, the base application firmware for the device we’re targeting (the nRF9160), and a firmware image for an auxiliary processor on board (e.g., an nRF52). Once the user is ready to deploy, they create a Release with matching tags and blueprints and an assigned Release version. Devices in the field that match the criteria set by the user on the console (tags, blueprints) will then be notified that a new device firmware release is available for them. The device in the field will start the firmware update process, downloading chunks of data.

Once the image is verified as complete, the device will utilize the MCUBoot API on the device and restart using that new image. In the event the Golioth user wants to roll back changes, they simply slide a switch on the Golioth Console and the device is alerted to download/restart the previous version of firmware. In this way, Golioth users have the ultimate flexibility for delivering firmware devices to a laser focused set of devices in their fleet.

Watch the demo

See this all happening in real time, including compiling a new firmware image, uploading it to the Golioth Console and watching the OTA DFU happen on the nRF9160.

Golioth recently released Output Streams. This new feature is immediately available to all users on their Golioth Console account. It allows you to pipe every event happening on the Golioth backend out to the cloud of your choice. Golioth’s capabilities as a device infrastructure provider (fleet management, OTA firmware update, device state maintenance, and deep device logging) can now be decoupled from the data storage and processing portion of your IoT deployment. When integrating with your internal or external cloud team, piping large volumes of data into their system is now as easy as flipping a switch.

How Golioth Users Access Data

You have hundreds or thousands of devices sending data back to the Golioth cloud. How do you get at that data? There are now 3 different ways you can access your device data:

REST API

Our REST API gives you access to all of the data on the platform, but on a per-request basis. So if you know you have a smart switch #5633 is out in the world at job site #1324, you might want to know the current status of that switch. You could send a GET request to the LightDB State variable that determines whether the switch is on or off. Or you could send a POST request to actually change the state of that switch. If you’re using LightDB Stream instead, you could do a GET request for the latest piece of data.

The key point is that you have to specifically call for variables in the platform (polling). If someone was flipping that switch every 3 seconds, but you only check every minute, you’re not going to catch every time that it switches. What’s more, you won’t get all of the other rich data that the Golioth platform collects around events. It’s still useful to be able to read and write the data, but doesn’t capture everything.

WebSockets

Websockets are a feature we introduced a few months ago and then made a Grafana plugin for listening to them. These are more direct connections to subscribe to realtime data. So if you were interested in not just the current state of the switch described above, but wanted to see a chart of (albeit boring) data showing the switch’s state, you could capture that by using LightDB Stream and WebSockets. Each of those switch events would be a “high” or “low” on the chart. Using WebSockets, you could point a Grafana instance to watch that variable and it would immediately update as the new switch data is available on the Golioth Cloud. Instead of polling, as would be required when using the REST API, the real time chart will update automatically.

Output Streams

Output Streams are an even richer way of outputting data from the Golioth Cloud. This is useful if you wanted to know not only the time that the switch changed state, but also who changed the state: was it a user flipping a physical switch on the device? Was it an app attempting to change the state of the switch to enact change in the physical world? Was it a troubleshooting command from the Golioth Console or via a REST API call? This context-rich data can now pass out of the Golioth Cloud and into the platform of your choice. When you are reviewing the data in your AWS or Azure instance, you can see that a user attempted to turn on the switch from the app on their phone at 8:30 am yesterday, and then they flipped the physical switch at 8:31. Not only is this incredibly useful data for troubleshooting problems, context-rich data can enable more meaningful software interactions for your users.

Integrations are easier

As of this writing, Golioth Output Streams support the following three integrations:

The first two are cloud specific ingestion engines for events happening outside the Amazon and Microsoft ecosystems. If your cloud team is using AWS, the SQS integration (also shown in the video below) will allow them to receive events from Golioth; you can select which events to send to AWS during Output Streams setup. Queuing (the “Q” in “SQS”) is a major feature of this input to AWS. It captures a high volume of incoming data events and buffers them for processing as your system catches up. As your Golioth deployment grows to thousands of devices or more, this will be a critical component to ensure every datapoint is captured. We will write more in the future about DevOps and large deployments using Golioth. The story for Azure Event Hubs is similar, it buffers incoming data, with some differences in setup and implementation.

Webhooks are a more generic class of Output Stream, which allows even easier integration to 3rd Party Platforms. As explained above, Output Streams enable delivery of a more context-rich piece of data to outside platforms. What’s more, many of these platforms are pre-configured to accept Webhooks. So as a user, I can now target the external platform’s address and immediately start parsing data in outside platforms such as visualization tools. Datacake did an early preview of this, showing how incoming Webhooks can be ingested and processed by their system for charting device data. Webhooks allow fast prototyping with a variety of visualization platforms.

Watch the demo and try it yourself

We discussed the nature of Output Streams and showed a demonstration of sending events to an AWS SQS instance. Watch the video below for more info or sign up for a free Dev Tier account and try Golioth Output Streams today!

As a hardware engineer, I have been interested in finding a platform to help me manage my devices for a long time. Seriously, there’s recorded proof. Back in 2018, I was on a podcast talking about my desire to have a management platform for all things IoT. If you continue listening to me describe what I wanted, it’s clear I was looking for what Golioth now offers. At the time though, I was specifically looking for firmware update services (what Golioth calls “Device Firmware Update” or “DFU”) for my small groups of ESP32 devices.

Firmware update for remote devices continues to be our most popular feature, not least of all because we support so many hardware platforms. Any device that meets the 3 following criteria is capable of firmware update using Golioth:

  • Firmware built with Zephyr
  • Has a network connection recognized in the Zephyr networking layer (including external modems if there is a sufficient driver)
  • Implements an MCUboot bootloader on the hardware

Our APIs for DFU are flexible enough that there are other combinations that can work, but the above criteria are enables hundreds of hardware devices. For the ESP32, it was that last point that was not yet enabled. The Espressif Zephyr team has been hard at work enabling more and more features and as of a few weeks ago, MCUboot has been pushed upstream and works on ESP32 devices. What’s more, we have tested this and the ESP32 now has DFU capabilities on Golioth deployments.

A bootloader interlude

What is a bootloader, anyway? A bootloader is the small segment of helper code that handles loading your main application code onto a processor when it starts up. It normally lives at the very beginning of memory and is the first thing that runs when a processor boots. The added advantage is you can tell the bootloader to try out different code than it normally runs, as is the case with a firmware update. If you have loaded a program to a microcontroller over a USB cable (without a programmer) on platforms like an Arduino, you are almost certainly running a serial bootloader. Something on your computer tells the board, “I’m shipping you a new firmware image”. Once it’s loaded, the bootloader restarts and tries out that new image. In this way, the bootloader handles updates to the code that is running on a processor at any given time.

In more complex examples, say you have application code that is able to download binaries over a network interface like WiFi or cellular from an awesome API service like Golioth. Once the application code verifies the contents are all properly loaded into the “trial” section of memory, we can then tell the bootloader to try that new code out. This is the basic idea behind the DFU sample in the Golioth SDK.

Possible, but intricate

The good news is you can build and load MCUboot onto your ESP32. This means you have an open source bootloader that enables integrations like Golioth DFU. The less-good news is this is still a somewhat young implementation and that means you have to build the MCUboot image yourself. We expect this flow will get easier over time and we will continue to update as this workflow changes. For now, let’s look at the steps required to get the bootloader built and loaded from a Linux (Ubuntu) system to an ESP32 (other OSes may be possible, but not featured here). After that, we’ll build Golioth sample code which will be loaded onto the ESP32 by this newly built bootloader.

Build MCUboot

This section of the guide is based on directions on the MCUboot repo for Espressif. Those directions are being replicated here for freezing a known working version and to improve visibility to the community. Additional commands were added to make things more clear and to standardize the install process. For instance, below we will be filling in all of the required variables to match the esp32 target.

Install additional packages required for MCUboot

These are the repository files for MCUboot and also the Python dependencies

cd ~
git clone [email protected]:mcu-tools/mcuboot.git
cd ~/mcuboot
pip3 install --user -r scripts/requirements.txt

Update the submodules needed by the Espressif port. This may take a while.

git submodule update --init --recursive --checkout boot/espressif/hal/esp-idf

Next, get the Mbed TLS submodule required by MCUboot.

git submodule update --init --recursive ext/mbedtls

Now we need to install IDF dependencies and set environment variables via a script. This step may take some time. If you happen to be using a Python virtualenv on your command line, exit it by typing deactivate; this is because the script instantiates a virtualenv for you using shell commands.

cd ~/mcuboot/boot/espressif/hal/esp-idf
./install.sh
. ./export.sh
cd ../..

Building the bootloader

The Espressif port of the MCUboot bootloader is built using the toolchain and tools provided by ESP-IDF. Additional configuration related to MCUboot features and slot partitioning may be made using the bootloader.conf. As a reminder, we are specifically building for the esp32 target here.

First we’re going to compile and generate the ELF:

cmake -DCMAKE_TOOLCHAIN_FILE=tools/toolchain-esp32.cmake -DMCUBOOT_TARGET=esp32 -B build -GNinja
cmake --build build/

Next we will convert the ELF to the final bootloader binary image, ready to be flashed. In this case, we are targeting a device with a 4MB flash size. This may work on larger devices, but you should check your specific device to ensure you are targeting an ESP32 part with the proper flash region:

 
esptool.py --chip esp32 elf2image --flash_mode dio --flash_freq 40m --flash_size 4MB -o build/mcuboot_esp32.bin build/mcuboot_esp32.elf

Finally, we are going to flash this binary onto the ESP32, which should be connected over USB. As we said in the bootloader interlude above, the bootloader is a chunk of code that lives at the beginning of memory. We need to offset where the code is actually going to start, based upon a normal memory configuration. In the case of the ESP32, the offset is at 0x1000, so we’ll set the variableBOOTLOADER_FLASH_OFFSET = 0x1000.

 

esptool.py -p /dev/ttyUSB0 -b 115200 --before default_reset --after hard_reset --chip esp32 write_flash --flash_mode dio --flash_size 4MB --flash_freq 40m 0x1000 build/mcuboot_esp32.bin

We are using /dev/ttyUSB0 because this is for an Ubuntu based system. An OSX (Mac) machine will likely target UART at /dev/tty.SLAB_USBtoUART

Build your image

Now we have a bootloader on the ESP32, hooray!

From here, we want to build a Golioth example for DFU to showcase the capabilities of the platform. The ESP32’s new MCUboot bootloader won’t do anything on its own, we need to pass data to it through the MCUboot API (taken care of in the Golioth sample). In order to fetch that data from the Golioth servers, we need to have application code that checks whether firmware is up-to-date and whether it should be fetching a new version. This is built into the Golioth DFU sample code, as well. We will build an initial version of this application code locally on our machine and manually push it onto the ESP32 using USB.

Building your initial DFU sample image to load directly on the ESP32

The steps below assume your Zephyr directory is located at ~/zephyrproject. If you haven’t built and tried any other samples for the ESP32, it’s probably worthwhile going through the ESP32 quickstart guide, which will take you through all of the required toolchain install steps for Zephyr. If your toolchain is in place, we can move on to edit the DFU sample’s prj.conffile:

 
cd ~/zephyrproject/modules/lib/golioth
nano samples/dfu/prj.conf

We’ll add the following to this file, like we do on all ESP32 samples. The first set of variables enables WiFi access onto your local network, the second set are your credentials for your device on the Golioth Cloud.

CONFIG_ESP32_WIFI_SSID="YOUR_NETWORK_NAME"
CONFIG_ESP32_WIFI_PASSWORD="YOUR_NETWORK_PW"

CONFIG_GOLIOTH_SYSTEM_CLIENT_PSK_ID="DEVICE_CRED_ID"
CONFIG_GOLIOTH_SYSTEM_CLIENT_PSK="DEVICE_PSK"

Then we’ll build the DFU sample

west build -b esp32 samples/dfu -p

Now we have a binary that we need to sign so that MCUboot knows this is a secure version of firmware. The key used here will not be unique and you will see warnings that this key should not be used in production. We agree! Generating and using a different key will be showcased in a future post. But you still need to sign the image. Note that this first image will be signed without a firmware version.

west sign -t imgtool -- --key ../../../bootloader/mcuboot/root-rsa-2048.pem

Finally we’ll flash this image to the board over USB.

west flash --bin-file build/zephyr/zephyr.signed.bin --hex-file build/zephyr/zephyr.signed.hex

Building a new DFU sample image to load directly onto the Golioth cloud

OK! Now we have a Golioth DFU sample loaded onto our ESP32 using the custom MCUboot bootloader. When the device boots up, the bootloader will first load this binary into the ESP32 and it’ll start processing. The application-level code will be monitoring for changes on the Golioth servers and waiting to hear if there is an update to the firmware version. So that’s our last step: building a version to load on the Golioth Console.

This build will look very similar to the last set of steps, with a few crucial differences. First, we build the firmware (no changes are needed for the source code):

 
west build -b esp32 samples/dfu -p 

Then we sign the image. Note that we are now giving this new binary a version number. This helps make it easy to tell which version is going onto your fleet of devices.

west sign -t imgtool --no-hex -B new.bin -- --key ../../../bootloader/mcuboot/root-rsa-2048.pem --version 1.0.0

Finally we upload this to the Golioth Cloud. This can be done from the command line using goliothctl or you can upload an image directly as an “Artifact” using the GUI. We walked through using the GUI in the nRF91 OTA DFU video, and have included screenshots of the process below.

Artifact upload process using command line tools:

 
goliothctl login
goliothctl dfu artifact create ./new.bin --version 1.0.0

Artifact upload process using the Golioth Console GUI:

Drag “new.bin” onto the file upload section and set the Version number

We see that the Artifact list shows a hash based on the signing of the image, and the Golioth Cloud has recognized that the image was built for MCUboot. We did not include a Blueprint, but we also could have done that to only target devices that also had a specific Blueprint assigned.

Ready, set, firmware update

Finally, we’ll create a Release that includes this Artifact. The Release is capable of bundling multiple binaries together, but in our case we only have one. So we’ll include that into a new release that we create on the Golioth Console

I’m targeting the device that is tagged desk (because it’s, you know…on my desk). Only devices with matching tags will get the update. Also note that Rollout is not yet selected.

This is the Release page showing this release, which includes the main-1.0.0 Artifact uploaded in the previous step. In the above screenshot, I have clicked Rollout which means any devices matching desk will try to download and install this image. I open a serial terminal to my ESP32 with MCUboot bootloader installed using this command

 
sudo screen /dev/ttyUSB0 115200

And I see the upload process starting! After it downloads the various pieces of the new v1.0.0 image, MCUboot restarts the processor and loads the new image. At boot time, we see it checking the server to see if it has the latest release. The key message is Desired version (1.0.0) matches current firmware version!

More updates to come

It’d be a stretch to say that was a short or smooth process. With so many steps involved, it might take a couple of tries to get the bootloader working smoothly. If you have run the DFU process a few times, the flow for compiling and uploading to the server becomes like second nature. The MCUboot build process should only have to be done the first time. But we want to keep improving the process for our customers and our partners.

If you have trouble with this process, be sure to hop over to our Discord server for live help from the Golioth team or post on our Forum for help in longer form. Happy updating!

IoT devices have the potential to generate a LOT of data. That’s kind of the point, right? You have a fleet spread out among a geographic area, or factory, or warehouse, or wherever you’re trying to measure data and you pipe it all back to the internet to collate that data for use elsewhere.

How do we make sense of all that data?

Other posts on this blog have showcased how you might chart the data using Grafana, or pipe it to an app using WebSockets. But before you do that, you probably want to get a feel for what’s in your dataset, especially as you’re streaming more and more data back to the platform. Today we’re going to explore that data using the Golioth Query Builder.

A note about images in this post: We are using screenshots of a lot of text, so we recommend clicking on the images in order to get a larger/better view of the feature under discussion.

What is the Golioth Query Builder?

The Golioth Query Builder is built into the Golioth Console. Any time you are looking at data sent through the LightDB Stream database (and associated API), you will be looking at a Query. It’s accessed via the green button in the upper right corner of your LightDB Stream View:

Viewing datasets

So you have data piling up in your database and you want to try to make sense of it. In my case, I was streaming built-in sensor data from my Nordic Semiconductor Thingy91 over LightDB Stream to Golioth. Every minute the device sends the following:

  • Temperature
  • Pressure
  • Humidity
  • Gas
  • Accelerometer (as separate values for X, Y, and Z)

By default, your Query Builder has 3 fields pre-populated: time, deviceid, and *

This means that when you look at the output, there are columns that correspond to those fields. time is the timestamp of when the data was sent back (which is stored in LightDB Stream by default, as it is a time-series database). deviceid is the name of the device you’re looking at.

There is a secondary filter at the top of the page which can also narrow it to a particular piece of hardware. In the image above, you see this as “streamy91”, the name I gave to my device. This means that only the data from that one device will show on this page, though I can remove that device-specific filter and look at data across multiple devices in my deployment.

Finally, the * (star) operator is a directive to display all of the remaining data. This is formatted depending on how you created your LightDB Stream data from the firmware on your device. As a result, I see datapoints coming back that look like this:

Notice that the data is in its raw form, delivered in a hierarchical view (each data point is listed under sensor). This is how the Zephyr program on my nRF9160 is sending data back to the platform. It’s a useful way to organize and data, but makes it hard to view the data directly.

Cleaning up data

One of the most obvious values of the Query Builder is to clean up and modify how we view data coming back over LightDB Stream. Instead of having data as a JSON style hierarchy, I can break it out into side-by-side columns. I can also apply labels to make things a bit easier to read:

Already the table is much more readable. I left out accelerometer data as I wasn’t moving the device very much while this data was collected:

If I also want to see the raw data on this chart, I just add an additional field for * and it will add that hierarchical view as an additional column. In the above example we only have 7 data fields, but you can imagine with a complex product, you might be sending back tens or hundreds of fields that might be in that hierarchical view. Query builder already makes it easier to choose what data you want to display at any given time.

New views on the data

Now that we can better view the data and use visual scanning through the data, we see just how much of it is there. For a physical measurement like Temperature, we might not care about data every minute as it’s being sent back. In fact, we might want to average the values to smooth out small spikes in the data. We can easily add a modifier on any of the parameters:

We choose avg to indicate we want to average the temperature field, float to indicate that field is representing a number, and we set the “Time Bucket” field to be 5m (5 minutes), though we can bucket by other conventions such as 10s (10 seconds), 1h (1 hour), or 2d (2 days). The higher the value for the time bucket, the more datapoints will be averaged together.

You’ll note here that the time column now is in 5 minute increments, however there are multiple datapoints at each of those buckets. That’s because we only averaged the sensor.temp data and not all of the fields we’re trying to show. Let’s now average all of the data shown on the screen and also add a new field that calculates the count of the sensor.temp column. This will showcase how many samples are in each bucket:

And now we see that each of these values have been averaged:

Take your queries with you

This is just the tip of the iceberg when it comes to displaying and filtering your data. We can set up even more complex queries using the filter, especially when implementing boolean functions. And though we are only looking at the output from one device in the example above, we might want to average across tens of devices that have the same tag (see our post on the power of tagging), such as devices that are in a test group, or are in a particular geographic area. The important thing is that the query builder gives users insight into how their data is streaming back from devices in the field and how they might be able to use it for their end goals.

On that topic, the query builder lives up to its namesake: It is actually building the query for you. You can peek under the hood by switching from the “Builder” view to the “JSON” view:

The JSON view allows you to copy the JSON query that Golioth is using on the Console to manipulate the data on the LightDB Stream database.

You can take this query and use it to interact with the Golioth REST API from outside visualization tools, such as Grafana.

Parse your data, gain more insight

The Golioth Query Builder allows users to get a better understanding of what range of data is coming back to the cloud from their embedded devices. With a platform like Golioth, you can write code using our Zephyr SDK, send a firmware update using our DFU service, determine what data should be transported back to the cloud, and now viewing that data in a real time and more nuanced way. We are continually seeking to shorten the cycle from embedded programming to cloud data.

Like many of the Golioth features, the best way to learn is to try it yourself. The Golioth Dev Tier means you can build out a small deployment and see the power of easily sending data back through the Golioth cloud. Sign up for the Golioth Console today and start streaming data back using platforms like the nRF9160 or the ESP32. If you’d like to read more about how you can filter and slice your data to make it more useful, check out our Querying LightDB Stream documentation, to see even more ways to interact with data on the Golioth platform.

The new WebSockets feature on Golioth allows users to stream data out to external services, such as visualization platforms and web apps. This includes showing “real-time” data from sources on embedded devices, such as sensors. Let’s look at how these are different than previous offerings.

What is a WebSocket?

A WebSocket is a persistent connection that runs over TCP connections between two entities, normally a browser and a server. Thought about in the abstract, it is a datapipe that allows bidirectional information. Golioth is using these to push and pull data between the Golioth Cloud and another endpoint, such as a browser or visualization tool.

How is this different?

As a hardware engineer, I find myself falling into the familiar trap of, “it’s all just moving data around!”. But that, of course, is wrong. Once your data moves from your device up to the Golioth cloud, you normally want it to go somewhere or to do something. Readers familiar with our other service offerings will remember that we have a REST API. If you want to pull data using the REST API, you need to request that next chunk of data. An external application like an Android app would need to ask for an update from the API on a regular basis, perhaps every second. This pull action is akin to “polling” in a microcontroller. Hardware and firmware designers will know the downsides to this method, especially in terms of processing overhead. But it happens all over the web and there are lots of reasons to use the REST API. In fact, this is how we update the Golioth Console when users are viewing LightDB data:

With WebSockets, you are not polling; you are being pushed data as it appears on the platform. That is, “from the perspective of the web application showcasing the data”, it’s a push, instead of a pull. The WebSocket connection must remain open in order to receive this data, but if it is, any new data will flow to the web application on the other side. Combining a WebSocket with LightDB Stream (Golioth’s time-series database) allows you to potentially stream data from a device up to the Golioth cloud and then out to a charting service like Grafana. In this way, it’s a conduit for sensor data in “real-time” (overhead will cause some delay between the action and the charting of the event).

Let’s imagine environmental sensors connected to an nRF9160 development board like the CircuitDojo nRF9160 feather. In fact, there is a demo from Jared Wolff (creator of the nRF91 Feather), showcasing the environmental add-on board called the “Air Quality Wing”. That demo implements this using the REST API. Now instead of Grafana constantly pinging the REST API for the next data point, we’ll be able to update that demo to have data streaming directly to Grafana as a change happens. We’ll have demos showcasing this functionality in the near future.

Get started today

Our team released new documentation about WebSockets alongside the new functionality. We love to hear from users putting features to good use, so be sure to stop by Golioth Office Hours on our Discord server to show off what you have built.