A key mission at Golioth is to make it easier for hardware and firmware developers to connect devices to the internet. We do that in two ways:

  1. Providing easy-to-use APIs and SDKs for IoT devices to connect to Golioth Cloud endpoints.
  2. Training developers how to use the Device side code.

We have done many successful training sessions so far, showing individuals and companies how to connect their first devices. Along the way, we have learned that there is a large unfulfilled need in the market for training in the IoT space. So we’re doing it again! We’ll show people how to connect devices and get access to things like:

  • Secure Over-The-Air Updates to constrained devices
  • Command and Control over remote devices
  • Learn how to create and modify settings for remote devices
  • Understand how to implement data tracking from your device

We will be running our first training open to the public on December 14th, 2022. Read more below if you’d like to take part.

Training challenges

Once again we’ll be training developers from afar. We did this back in October for a select group of hardware engineers looking to learn more about Zephyr:

Click to learn more about our experience back in October of 2022

The upcoming training will be built upon the lessons we learned during that training, and our last in-person training at the Hackaday Superconference. In both cases, we used Kasm to provide fully remote development environments so that users don’t need to install anything on their local machine (there are directions on how to do that after the training is over). We think this is an important piece to ensure people can get started quickly.

How to use Zephyr

We currently offer 3 SDKs as part of our device support, including an ESP-IDF SDK, a Modus Toolbox SDK, and a Zephyr SDK (including the Nordic Connect SDK variant). These SDKs cover a wide range of embedded hardware from different vendors.

The training includes some segments that detail how to use Zephyr, a Real Time Operating System (RTOS) that covers a wide range of different hardware platforms. We use it on many of our internal hardware reference designs at Golioth, and it was the first platform we launched. Hardware and firmware engineers who are new to Real Time Operating Systems will continue their learning journey by understanding how the RTOS connects to sensors and low level GPIO and how to manipulate different elements of the subsystems. Once a trainee understands how to get the data off of an external component (like a sensor), the Golioth Zephyr SDK makes it a simple task to forward that data along to the Golioth Cloud.

Requirements and background

We have referred to “Hardware and Firmware Engineers” in this article, because we expect that intermediate to expert level engineers will get the most out of this training. If you are brand new to understanding C or if you have never tried programming embedded hardware before, this might be a frustrating experience. If you would like some pointers to starter content that might prepare you for the training, please ask on our Forum and we will try to get you a customized list of resources that will help prepare you for future versions of this training.

Logistics

  • We are not charging for this training
  • We will be capping the training at 30 people
  • All attendees will be on a first-come, first-served basis
  • Those who are accepted for this training will receive an email with more details
  • You will be expected to purchase your own hardware
    • Details will be sent with your acceptance to this training
    • Be sure you leave enough time for shipping from your local distributor
  • Signing up to take part and not attending will disqualify you from future training

Sign up here


If you have clicked “submit” and don’t see any changes, please scroll back up to the top

The Golioth Zephyr SDK is now 100% easier to use. The new v0.4.0 release was tagged on Wednesday and delivers all of the Golioth features with less coding work needed to use them in your application. Highlights include:

  • Asynchronous function/callback declaration is greatly simplified
    • User-defined data can now be passed to callbacks
  • Synchronous versions of each function were added
  • API is now CoAP agnostic (reply handling happens transparently)
  • User code no longer needs to register an on_message() callback
  • Verified with the latest Zephyr v3.2.0 and NCS v2.1.0

The release brings with it many other improvements that make your applications better, even without knowing about them. For the curious, check out the changelog for full details.

Update your code for the new API (or not)

The new API includes some breaking changes and to deal with this you have two main options:

  1. Update legacy code to use the new API
  2. Lock your legacy code to a previous Golioth version

1. Updating legacy code

The Golioth Docs have been updated for the new API, and reading through the Firmware section will give you a great handle on how everything works. The source of truth continues to be the Golioth Zephyr SDK reference (Doxygen).

Updating to the new API is not difficult. I’ve just finished working on that for a number of our hardware demos, including the magtag-demo repository we use for our Developer Training sessions. The structure of your program will remain largely the same, with Golioth function names and parameters being the most noticeable change.

Consider the following code that uses the new API to perform an asynchronous get operation for LightDB State data:

/* The callback function */
static int counter_handler(struct golioth_req_rsp *rsp)
{
    if (rsp->err) {
        LOG_ERR("Failed to receive counter value: %d", rsp->err);
        return rsp->err;
    }

    LOG_HEXDUMP_INF(rsp->data, rsp->len, "Counter (async)");

    return 0;
}

/* Register the LightDB Get callback from somewhere in your code */
static int my_function(void)
{
    int err;
    err = golioth_lightdb_get_cb(client, "counter",
                                 GOLIOTH_CONTENT_FORMAT_APP_JSON,
                                 counter_handler, NULL);
}

Previously, the application code would have needed to allocate a coap_reply, pass it as a parameter in the get function call, use the on_message callback to process the reply, then unpack the payload in the reply callback before acting on it. All of that busy work is gone now!

With the move to the v0.4.0 API, we don’t need to worry about anything other than:

  • Registering the callback function
  • Working with the data (or an error message) when we hear back from Golioth.

You can see the response struct makes the data itself, the length of the data, and the error message available in a very straightforward way.

A keen eye already noticed the NULL as the final parameter. This is a void * type that lets you pass your user-defined data to the callback. Any value that’s 4-bytes or less can be passed directly, or you can pass a pointer to a struct packed with information. Just be sure to be mindful of the memory allocation lifespan of what you pass.

All of the asynchronous API function calls follow this same pattern for callbacks and requests. The synchronous calls are equally simple to understand. I found the Golioth sample applications to be a great blueprint for updating the API calls in my application code. The changelog also mentions each API-altering commit which you may find useful for additional migration info.

The Golioth Forum is a great place to ask questions and share your tips and tricks when getting to know the new syntax.

2. Locking older projects to an earlier Golioth

While we recommend updating your applications, if you do have the option to continue using an older version of Golioth instead. For that, we recommend using a west manifest to lock your project to a specific version of Golioth.

Manifest files specify the repository URL and tag/hash/branch that should be checked out. That version is used when running west update, which then imports a version of Zephyr and all supporting modules specified in the Golioth SDK manifest to be sure they can build the project in peace and harmony.

By adding a manifest to your project that references the Golioth Zephyr SDK v0.3.1 (the latest stable version before the API change) you can ensure your application will build in the future without the need to change your code. Please see our forum thread on using a west manifest to set up a “standalone” repository for more information.

A friendlier interface improves your Zephyr experience

Version 0.4.0 of the Golioth Zephyr SDK greatly improves the ease-of-use when adding device management features to your IoT applications. Device credentials and a few easy-to-use APIs are all it takes to build in data handling, command and control, and Over-the-Air updates into your device firmware. With Golioth’s Dev Tier your first 50 devices are free so you can start today, and as always, get in touch with us if you have any questions along the way!

One of the best tools in the Zephyr ecosystem is the ability to include different code modules in the build configuration. This feature leverages CMake and Kconfig, two tools that are core to Zephyr. But these tools aren’t limited to officially approved code modules. Today we’ll cover some CMake and Kconfig tricks for including common code in your own Zephyr applications.

The problem: common code across different variations of your app

Earlier this month we posted about the developer training that Golioth offers. It centers around a hardware development board called the MagTag, for which we’ve put together a code repository with many different examples. This includes code to drive the ePaper display, update four ws2812 LEDs, service button presses, and parse JSON objects.

The problem is that we need to use a common set of code in most, but not all of the training samples.

Luckily we’ve seen this problem before. The samples in the Golioth Zephyr SDK use common code for networking and shell settings features. Each feature from that common code has its own KConfig symbol, like GOLIOTH_SAMPLE_WIFI_SETTINGS, that is selected to include it in the build. By defining these in the board-specific conf files, libraries are only built for the devices that actually need them. For instance, an Ethernet-connected device doesn’t need WiFi settings.

The example from the Golioth SDK is a good one to study, but the MagTag implementation is a bit simpler so let’s walk through that code.

Golioth’s Developer Training is a self-guided experience that you can explore at your own pace. If you are interested in setting up a training with Golioth staff for your organization, please contact our Developer Relations team, we’d love to chat!

Directory Structure

Directory structure for a Zephyr project with multiple appsFrom the directory structure of the training you can see that we have five different Zephyr programs in this repository, each with their own boards and src subdirectories.

Common code for each app is placed in the magtag-common directory. Header files for each library are located in the the magtag-common/include/magtag-common directory. That might seem a bit convoluted, but it makes for sensible include names like #include "magtag-common/buttons.h".

Creating Kconfig symbols

Common code is individually enabled with a Kconfig symbol. There are a few ways to approach this. One of the easiest is to assign one symbol to indicate your app uses the common code, and then one symbol for each specific library in the common folder. This is done with a Kconfig file inside of the magtag-common directory.

menuconfig MAGTAG_COMMON
    bool "Common helper code for Golioth MagTag Demo"
    help
      Build and link common code that is shared across MagTag samples.

if MAGTAG_COMMON

config MAGTAG_ACCELEROMETER
    bool "Handle accelerometer readings"
    help
      Get accel from DeviceTree and write readings to shared sensor struct

config MAGTAG_BUTTONS
    bool "Process button reads"
    help
      Configure buttons for interrupts with callbacks

config MAGTAG_EPAPER
    bool "2.9\" grayscale ePaper driver"
    help
      Hardware driver for ePaper, including text and partial writes

config MAGTAG_WS2812
    bool "ws2812 helper functions"
    help
      Intialize and update LED color and state

endif # MAGTAG_COMMON

From this code snippet you can see the library-specific symbols are only defined if the MAGTAG_COMMON symbol has been selected.

Using CMake to build in the libraries

Now that we’ve created the Kconfig symbols, a CMakeLists.txt file in the magtag-common directory is used to build in the code based.

zephyr_library_sources_ifdef(CONFIG_MAGTAG_ACCELEROMETER accelerometer/accel.c)
zephyr_library_sources_ifdef(CONFIG_MAGTAG_BUTTONS buttons/buttons.c)
zephyr_library_sources_ifdef(CONFIG_MAGTAG_EPAPER epaper/magtag_epaper.c)
zephyr_library_sources_ifdef(CONFIG_MAGTAG_EPAPER epaper/magtag_epaper_hal.c)
zephyr_library_sources_ifdef(CONFIG_MAGTAG_WS2812 ws2812/ws2812_control.c)

zephyr_include_directories(include)

The include directory is added by default, but the source files are only added if the associated symbol is defined.

Using the common code in a Zephyr application

So far, the common code is completely separate from the each of the apps we want to build. Let’s use the golioth-demo app as an example. To tell the app about the common code, the subdirectory needs to be added to the CMakeLists.txt file in the application subfolder.

cmake_minimum_required(VERSION 3.20.0)

list(APPEND OVERLAY_CONFIG "../credentials.conf")

find_package(Zephyr REQUIRED HINTS $ENV{ZEPHYR_BASE})
project(lightdb)

add_subdirectory_ifdef(CONFIG_MAGTAG_COMMON ../magtag-common magtag-common)

target_sources(app PRIVATE src/main.c)

This codes makes sure that the common directory is only added to the build when the MAGTAG_COMMON symbol is selected. This completes the plumbing work. In this example, the app selects the common code in the prj.conf file:

# MagTag Common Files
CONFIG_MAGTAG_COMMON=y
CONFIG_MAGTAG_EPAPER=y
CONFIG_MAGTAG_WS2812=y
CONFIG_MAGTAG_ACCELEROMETER=y
CONFIG_MAGTAG_BUTTONS=y

And of course, you must include the header files to access the library functions. Here’s an excerpt of the main.c from the golioth-demo app:

/* MagTag specific hardware includes */
#include "magtag-common/magtag_epaper.h"
#include "magtag-common/ws2812_control.h"
#include "magtag-common/accel.h"
#include "magtag-common/buttons.h"

Wrapping up

MagTag Kconfig options shown in menuconfig

Abstracting your common code makes it a lot easier to maintain. This approach also adds entries in menuconfig for each of the libraries. This is a user-friendly feature as it allows you to convey more details in the help file. For more complex uses (like the Golioth samples common libs) it also makes it easier to see the symbol dependency hierarchy.

This example shows the libraries as a part of the repository, however this will work just as well if you commit them to their own repo. From there, they can be included as a git submodule, or with a little creativity you can get your west manifest file to pull the repo whenever west update is run. This is the approach that we’ll be taking with future development. Doing so allows us to lock the project to a specific hash of the common code so that future changes don’t break application code.

Understanding hardware and firmware is the first step towards building any kind of “thing” that will be part of an “Internet of Things” (IoT) deployment. For new chips and firmware ecosystems, this often means going through training.

Golioth Developer Relations (the group I’m part of) is tasked with making sure our users understand how to get their hardware connected to the Golioth Cloud. We have been doing in-person (as safety protocols allow) and remote training for Golioth users.

Last week, we did our first training for the broader public, a group of hardware engineers interested in learning more about Zephyr and Golioth. I wanted to discuss some of the tools and methods we used to do a fully remote hardware training. If you’re interested in taking part in the training in the future, please let us know in the associated forum post or by email. There are more details on the available options at the end of this post.

How we do fully remote hardware training

The main challenge with virtual gatherings is that trainees are not sitting in the same room as the trainers. This is an easy problem to understand, but a harder one to solve.

Hardware

The first step was standardization on the hardware. To achieve that goal, we created a list of the bare minimum of parts and shipped them out to all attendees, even those who already had some of the hardware. We wanted to make sure that we had exactly the same hardware in hand. Our trainees were in the US and Europe, so we ordered through DigiKey, with DHL shipping at least a week in advance. The average cost was about $60 of materials in the US, and $85 to Europe, both numbers including shipping, VAT, tarriffs, etc.

For the hardware itself, we use the Adafruit MagTag. It contains the Espressif ESP32-S2 as the main processor on board. This time around we focused on Zephyr for this training, but the board also works great with the Golioth ESP-IDF SDK. Remember, Golioth has 3 SDKs currently (vanilla Zephyr, NCS, ESP-IDF), and we are always working to enable other hardware platforms.

Adafruit MagTag

Video meeting

You might think that after 3 years of a global pandemic, the world would have figured out video software. But the key difference between a simple video call and our remote training is that it’s not just 10 people sitting in a single video session. We want to be able to dynamically “move” between different conversations, especially as trainers.

To achieve this, we used Gather.town. It utilizes tiny avatars you can move around the screen with your keyboard like a video game. As your avatar gets closer to another avatar, your video and audio pops up to those in range. The rugs in the image below also allow “private” areas which work kind of like mini conference rooms. We found this allowed us to dynamically help smaller groups of people while also making announcements to the entire group. Trainees could also choose to work together on different components of the training material.

Our Gather.town Training Location for Hardware Developers

Tooling

The biggest challenge with any training is dealing with firmware tooling, specifically with Zephyr. We have been working on solving this issue for a while. The Zephyr installation continues to get simpler overall, thanks to the Zephyr dev team updating what gets installed for each user. But we still have trainees “walking through the door” (figurative in this case) with a wide range of computers where the tools will be installed. A range of different operating systems (Windows, Mac, Linux–with different distributions) means that there is no one standardized setup. What’s more, if someone has tried out Zephyr in the past, it’s possible there are interactions between the old installation and the new one, not to mention a variety of dependencies within each machine. If we could, we would ask everyone to show up with a brand new installation of Windows or Linux on a laptop…but that’s unlikely to happen.

We wrote in the past about using Kasm and Docker to have a standardized install. Last week’s training represents the first time we used it with real users. In short, it worked! We collaborated with the wonderful team at KasmWeb (makers of the Kasm tool) who gave us seats of their Cloud offering to use with our trainees.

Zero install embedded training with Zephyr using Kasm and Docker

The first step was created a dockerfile and a Docker image that contained everything required for the training. This way we were able to spin up a standardized, browser based UI environment for everyone taking the training. The entire Zephyr toolchain and Espressif support tools were pre-installed in that image, in addition to VScode, a terminal, a browser, and text editing tools.

Trainees open a browser window that looks like Ubuntu (it is, under the hood), and immediately start compiling Zephyr code. At past in-person training sessions, we had issues with bandwidth just downloading the tool packages. While we still needed bandwidth to communicate with the browser-based UI, all of the processing and storage is done on the cloud. Our plan is to use Kasm at our next in-person training.

We continue to evolve our strategy for making training a more seamless process and offering a range of different training to our customers.

Kasm instance of a Docker container with Zephyr / Espressif / Golioth tools installed

Training material

The final piece of the puzzle is the training material itself. Again, this is an evolving solution, because we continue to add more material, and because we track changes to the Zephyr SDK over time.

On Training.Golioth.io, we take people through getting signed up for Golioth, creating a first binary for the MagTag using their Golioth credentials, and then launch them into understanding different parts of Zephyr. We take developers all the way through working with the DeviceTree and understanding how to interact with an in-tree sensor in Zephyr. Our goal is to help hardware and firmware engineers understand how to use some of the most critical parts of an RTOS, and then use those features to communicate back to the Cloud, thereby making their device an “IoT” based device.

training.golioth.io

If you’re interested in trying the training without Golioth present, you can do the “asynchronous” version of our training. Training.Golioth.io is free to the public. You won’t have access to our Kasm server, but you will be able to install the Zephyr toolchain using supplemental directions. Please utilize the Golioth Forums for any questions you have about training.

Results

We scheduled a 2 hour training session, as we know that engineers’ time is valuable. In that time, 100% of the trainees were able to get their devices connected to the internet, pass data back and forth between Device and Cloud, and explore the features of Zephyr. As in any training, each person worked at a different pace and was able to get to different checkpoints. Each trainee could use their hardware and training site at home for later study.

An unfortunate side effect of having multiple tools, including training materials, browser based UI, and video software is a lot of tab/window switching to keep track of it all. In-person training will cut down on the video window, but it would be nice to further integrate the training to include all required materials in one place. We had a “browser within a browser” (Firefox installed within the Kasm container), but that consumed a lot of virtual memory and wasn’t an optimal solution.

The other major restriction is that the binaries our trainees compiled were not directly loaded onto the MagTag hardware. Because we worked inside a virtual (browser based) container, there was no direct connection to the hardware. We also talked about this in the past article about Kasm based development. Because virtualization tools are created to be independent from the hardware, it is difficult to then reverse course and say, “We want this container to talk to USB!”.

We are working with the Kasm team and looking at different ways to program binaries directly from inside the container, and would love to hear about other options people have seen. We solved this during the training by having trainees install the Espressif esptooldirectly on their machines. This still involved some amount of install, but it was limited to a small program. When a trainee completed a build, they could download the binary image from the container to their local machine and load it onto the MagTag using esptool. After that, the MagTag was able to talk back to Golioth.

Future training

We are excited to continue iterating on this training in the future and would love to hear from people that are interested in participating. Here are the options you can try:

Zephyr threads, workers, and message queues

Zephyr is a Real Time Operating System (RTOS). That means it’s built to let you do multiple things at the same time using a single (or limited number) of processing cores. To be fair, you’re not doing things at the same time; the RTOS shares processing time across all of the tasks, with a priority system for the more important ones.

The trick with an RTOS is to design your applications so that they utilize the real-time-ness and you don’t miss reacting to real-time events like receiving data from a network, taking sensor readings, or reacting to user input.

In this post, we’ll discuss the difference between Zephyr threads and work queues and show you why you might want to use one versus the other. We’ll also discuss how to use message queues to pass around data between running processes.

Zephyr Threads

A thread is a set of commands to be executed by the CPU. The while(1) loop that runs inside of the main function of your application is a thread. But Zephyr allows you to create multiple threads. Each thread operates independently, with Zephyr’s built-in scheduler deciding when to run each thread.

For example, the Golioth Blue Demo run animations on the LEDs to indicate what mode the device was in. Instead of trying to get the loop in the main thread to update the lights on a tight schedule, we use a thread to run the animations. The main thread of the program deals with flow of the things happening on the device, and it can suspend/resume the animation thread as necessary.

I like to think of threads as tasks that need keep running, things that don’t return like a discrete function would. Your thread should have a while(1) and can call other functions just like you would in your main loop. It is up to you to yield processor control back to the scheduler so that it may run other threads. This is pretty easy, just call any of the k_sleep() functions or k_yield().

/* Every thread needs its own stack in RAM */
#define ANIMATE_PING_STACK	1024

/* Define and initialize the new thread */
K_THREAD_DEFINE(animate_ping_tid, ANIMATE_PING_STACK,
            animate_ping_thread, NULL, NULL, NULL,
            K_LOWEST_APPLICATION_THREAD_PRIO, 0, 0);

/* All animations handled by this thread */
extern void animate_ping_thread(void *d0, void *d1, void *d2) {
    uint8_t spinner_idx = 0;
    while(1) {
        led_state &= ~LEDS_LOGO;
        led_state |= leds_logo_order[spinner_idx];
        refresh_leds(led_state);
        if (++spinner_idx > 7) spinner_idx = 0;
        /* Control is yielded back to the scheduler during */
        /* sleep between animation frames                  */
        k_sleep(K_MSEC(100));
    }
}

int main(void) {
    /* Start animation thread running */
    k_thread_start(animate_ping_tid);

    while(1) {
        /* normal program flow */
    }
}

The example shown here runs an animation when an IoT device is trying to connect to a network. When it begins running, the LEDs will update every 100 milliseconds. Because the loop sleeps in between frames, control is given back to the scheduler or other tasks. Elsewhere in the program, we call k_thread_suspend(animate_ping_tid) and k_thread_resume(animate_ping_tid) to stop and start the animation.

Just be careful that you don’t have multiple threads trying to access the same resource at the same time (protect those resources with a mutex).

Zephyr Work Queues

A Zephyr workqueue is a thread with extra features bolted on. It also requires less setup and management than threads.

The work queue has a built-in buffer to store pending work (functions you want to run). Each item you add will be executed in the order you submitted it, like a “to-do list” for your processor. This is a perfect place for something that you want to run once and return from, but don’t want to do it inside of an interrupt service routine.

The “work” you submit to a work queue is a function. So you’re telling the work queue “run this function, then run this other function, then run this other…” you get the idea. Just set it and forget it–the Zephyr scheduler will run take care of popping work out of the queue and running it until the queue is empty. Many of the sensor readings we do in our demos are handled through work queues.

The work queue expects a specific type of function usually referred to as a “work handler”. You don’t actually give the handler any information, you just tell your work queue you want to add your handler to the list of pending work.

/* Work handler function */
void button_action_work_handler(struct k_work *work) {
    do_something_because_a_button_was_pressed();
}

/* Register the work handler */
K_WORK_DEFINE(button_action_work, button_action_work_handler);

/* Add a work item to the system workqueue from a button interrupt function */
void button_pressed(const struct device *dev, struct gpio_callback *cb,
            uint32_t pins)
{
    k_work_submit(&button_action_work);
}

This code snippet demonstrates using a work queue when a button is pressed. You want to spend as little time as possible in the button interrupt function. All this handler does is queue up a button action function that will be run by the scheduler, sometime after the interrupt service routine ends.

The example above uses the system workqueue. You also have the option of declaring your own workqueues which will need its own stack (remember… it’s a thread with extra features). If your application is freezing up when using the system workqueue, check to see if you’ve run our of stack space and set CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE accordingly.

The challenge here is how to get data to the worker since we can’t pass any parameters. How do you know which button was pressed? One approach is to place your work item inside of a struct along with the data you want to pass (button number, etc.). Then in the handler you can use CONTAINER_OF() to access the data in the structure. This is beyond the scope of today’s post, so let’s talk about a bit simpler approach we often take: using a message queue.

Zephyr Message Queues

A Zephyr Message Queue is a collection of fixed-sized data that is safe to access from multiple threads. You decide what the data is (a variable, a struct, a pointer, etc.) and how many of those objects the queue can contain (limited by your available RAM). Zephyr handles all of the logic necessary for adding to and removing from the queue.

#define BUTTON_ACTION_ARRAY_SIZE	16
#define BUTTON_ACTION_LEN		4
K_MSGQ_DEFINE(button_action_msgq,
        BUTTON_ACTION_LEN,
        BUTTON_ACTION_ARRAY_SIZE,
        4);

void button_pressed(const struct device *dev, struct gpio_callback *cb,
            uint32_t pins)
{
    /* add the current button states to the message queue */
    k_msgq_put(&button_action_msgq, &pins, K_NO_WAIT);

    /* This is a good place to queue a worker to process the buttons */
}

/* Work handler to process actions from button presses */
void button_action_work_handler(struct k_work *work) {
    /* Process everything in the message queue */
    while (k_msgq_num_used_get(&button_action_msgq)) {
        uint32_t button_states;
        k_msgq_get(&button_action_msgq, &button_states, K_NO_WAIT);

        /* Run some function based on which button was pressed */
        if (pins & BUTTON_0) {
            do_something_because_a_button_was_pressed();
        }
        /* Give the schedule a chance to run other tasks */
        k_yield();
    }
}

Continuing with our button analogy, the example above uses a message queue to record the button states for processing after the interrupt service routine returns. I’ve highlighted the lines that are crucial to the message queue system. Notice that the work handler uses a while loop to check if the message queue is empty. This is a nice pattern for processing all of the queued message. I’ve used k_yield() between loops to give the scheduler a chance to run other tasks.

Message queues are a spectacular tool for IoT applications. Golioth has used Message Queues to cache GPS readings while the cell modem is turned off on our Orange Demo. Periodically, the modem will turn on, send all of the readings from the queue to the Golioth servers, then turn off the radio once again to save power.

Threads, Workers, and Queues

We often focus on the “ecosystem” aspect of Zephyr, because we like that so many silicon vendors contribute to the code base. But it’s important to remember that Zephyr is a Real Time Operating System and has a fully featured scheduler. We’ve only just scraped the surface of what you can do with it. Hopefully the above explanations helped you to begin thinking in these patterns, understanding how to divide up the application work, and how to pass data between different parts of your code.

It’s no secret that we like Zephyr Real Time Operating System (RTOS) around here. Nor is it a secret that we’re very interested in the Internet of Things (IoT). Golioth Founder and CEO Jonathan Beri gave a talk at the 2022 Zephyr Developer Summit (ZDS) about why those two things are a perfect match, and how Zephyr can help you create an IoT product faster.

A high level overview

Since this talk was on the first day of ZDS, it was focused on people less familiar with the Zephyr ecosystem. So what should a beginner know about Zephyr and its relationship to IoT devices?

Batteries are included

Zephyr has a couple of key features that makes it a “one stop shop” for building an IoT device. Some other RTOSes rely on the engineer to piece together libraries and implementations to get started, which either means more startup time or reliance on hardware vendors to make a ready-to-go solution for you. Instead, Zephyr provides individual elements already configured to work together:

  • Device drivers
  • An OS kernel (including scheduler)
  • Services
  • A build system

Tying together the ecosystem with these tools provides a consistent experience. This also leads to another very important aspect.

Chip vendor buy-in

Because Zephyr has a well defined system, the chip vendors develop code to make their parts work within the ecosystem. With other RTOSes, that script is flipped; the RTOS is made to work with abstraction layers within the vendor specific ecosystem, meaning there is less likelihood of interoperability with other vendor solutions.

In addition to the chipsets from the vendors, a wide range of boards are supported. Targeting specific boards makes it easier to get dev boards working when getting started, and remapping pins to specific functions on a board doesn’t require a configurator tool like those available in many vendor IDEs.

Talking to the Internet

If you choose an RTOS that isn’t a broader ecosystem, you need to either define your own network layers or lean on a vendor implementation to do it for you. In Zephyr, the network layer and the protocol definitions are done as a part of the community. That means that chip, module, and software vendors are all working towards a common implementation. Add in the fact that there are modems and abstraction layers all the way up the stack, and an engineer using Zephyr doesn’t need to think about all of the pieces it takes to connect to the internet. At Golioth, we are able to switch between various networking technologies easily when using Zephyr.

Protocol Variety

Golioth talks to the internet at a very high level, so network access “just works”. This comes into focus as a small embedded device can talk to the variety of internet protocols available. Most people understand HTTP, but also consider the more common IoT favorites like MQTT and CoAP. Golioth favors CoAP but also supports MQTT. This enables implementations in other parts of the network stack. A good example is devices running OpenThread, a Thread network protocol implementation. As we showed in our recent post about Thread, a small Zephyr-based device utilizes 6LoWPAN and talks over CoAP using UDP packets to talk back to the Golioth cloud.

Last but not least: Security

Zephyr enables a secure connection back to the internet through the network stack and with features like DTLS, the basis of a secure connection over UDP. Security is a deeper topic in Zephyr though, all the way down to secure bootloaders and working within things like TrustZone. Talking to secure elements requires drivers that understand how to communicate with particular chip features (internal or external to a microcontroller). At a very high level, Zephyr focuses on things like Software Bill of Materials (SBOM), so your security teams understand the various software you are pulling into the build.

Built for IoT

Zephyr not only ❤️ internet…it’s purpose built for enabling IoT devices. Paired with Golioth, a Zephyr device has a higher likelihood of getting to market and growing to a massive scale. We’d love to hear your thoughts about the video on our Forum, our Discord, or by email.

 

I feel like I begin every Zephyr article talking about how this RTOS’s greatest strength and greatest weakness is the extent of its hardware abstraction. Such is the case with the Zephyr Pin Control (pinctrl) system that became the default approach for pin configuration and multiplexing back in June. This development made great strides in standardizing how to work with pinmux configuration across many vendors, and unified the process of reassigning pin functions at runtime. These gains are at the expense of needing to understand the new syntax. Let’s dig in!

What is pinctrl?

The pinctrl (Pin Control) system is a standardized way of assigning peripheral functions to pins, a concept adopted from Linux. Pin Control lets us define which pins will be used for special functions (SPI/i2c/UART) and how those pins will be configured (pull up/down resistors, slew rate).

In the past we would have remapped an i2c pin inside the i2c node itself using a syntax like sda-pin = <33>;. With pinctrl we don’t necessarily touch the i2c node–it is instead given the address of a pinctrl nodelabel. The pinctrl-state nodes are where actual pins are specified, using a syntax that might look like &i2c1_sda_gpio33. (The exact format of pin addressing varies by vendor. We’ll discuss how to look up that info.)

So what do we gain from this change? One example is reconfiguring a chip for low-power before entering sleep mode. pinctrl automatically looks for two states: default and sleep. When changing from the default state to the sleep state, pinctrl will automatically reconfigure all of your power-sucking pins for low-power, and set them back up again upon wake.

GPIO mapping and usage remains separate for pinctrl. If you’re blinking LEDs or reading buttons, you don’t need to use pinctrl. When you think ‘pinctrl’, think ‘special-function peripherals’.

Example: use pinctrl to change i2c pins

A great exercise when learning pinctrl is to reassign pin locations on an existing board. I’ve been working with the ESP32s2 and there is a “saola” board definition in Zephyr. If we study the pinctrl file for that board we find that i2c1 normally maps GPIO3 and GPIO4:

i2c1_default: i2c1_default {
	group1 {
		pinmux = <I2C1_SDA_GPIO3>,
			 <I2C1_SCL_GPIO4>;
		bias-pull-up;
		drive-open-drain;
		output-high;
	};
};

This pinctrl node uses a nodelabel of i2c1_default which is associated with the i2c1 peripheral in the board’s .dts file:

&i2c1 {
    clock-frequency = <I2C_BITRATE_STANDARD>;
    pinctrl-0 = <&i2c1_default>;
    pinctrl-names = "default";
};

Let’s say we wanted to change these pins to GPIO33 and GPIO34. We could make our own pinctrl node with a different node label and change which nodelabel is used for i2c1. However, the easiest thing is to simply change the pin assignments in the default node, which allows us to inherit all the other settings from the board definition. This is done by adding an esp32s2_saola.overlay in the boards directory of our application. Remember, an “overlay” allows you to tell the build system that the local changes in your build directory are more important than the default settings in Zephyr, and to adopt the local changes.

&pinctrl {
    i2c1_default: i2c1_default {
        group1 {
            pinmux = <I2C1_SDA_GPIO33>,
                     <I2C1_SCL_GPIO34>;
        };
    };
};

Whenever remapping pins, it’s important to check that the new pin assignments are available. In our case, GPIO33 is available, but GPIO34 is mapped to the pinctrl default for SPI3. To avoid a conflict, that pin should be remapped, or if the SPI3 peripheral is not used the SPI3 node can be deactivated or deleted.

Read the bindings pages for vendor-specific pinctrl info

In the example above we used I2C1_SDA_GPIO33 to remap a pin. This syntax is unique to Espressif chips, and will be different for Nordic, Microchip, NXP, Infineon, or any other vendor chips. The same assignment for a Nordic chip would be something like NRF_PSEL(TWIM_SDA, 0, 30). We admit, this is annoying. But Zephyr is an ecosystem, in addition to being an RTOS. Each vendor does the hard work of making their specific chips work with the rest of the code, an this is the cost of them being in control.

Vendors have documented their pinctrl implementation in their device bindings pages. Head over to the Zephyr bindings index and look for the pinctrl entry for your vendor. Espressif’s entry points to a set of macro definitions that lay out every possible function for each pin. Nordic’s entry indicates that rather than define each pin, we can use a macro to call out the nRF pin function along with the port and pin assignments.

/* Nordic pinctrl */
i2c2_default: i2c2_default {
    group1 {
        psels = <NRF_PSEL(TWIM_SDA, 0, 30)>,
            <NRF_PSEL(TWIM_SCL, 0, 31)>;
    };
};

/* Espressif pinctrl */
i2c1_default: i2c1_default {
    group1 {
        pinmux = <I2C1_SDA_GPIO3>,
             <I2C1_SCL_GPIO4>;
        bias-pull-up;
        drive-open-drain;
        output-high;
    };
};

The syntax itself is different from one vendor to another. The snippets above demonstrate that, in addition to pin assignment differences, Nordic uses psels to set the pins while Espressif uses tmux. Always check the bindings index pages, and when in doubt, look to the board definition files as examples.

Dynamic control

A full guide on dynamic control is out of scope for this post. To touch on the topic, choosing to use dynamic pinctrl makes it possible to update pin function at runtime by moving the pin configuration from ROM to RAM. The pinctrl_update_states() function is then used to assign a new state, executing the necessary configuration changes in the process.

Check the pinctrl test app for an example of dynamic control.

The good and the bad of pinctrl

I generally think the move to pinctrl is a good one. Sure, there are syntax differences between vendors, but that was true before the change as well. Now you can expect to find all important pin assignments inside of the pinctrl state nodes, which is a bit easier to reference, and makes it easier to see not just the pin numbers, but the configuration (slew rate, pull up/down, etc.).

i2c1_default: i2c1_default {
    phandle = < 0x5 >;
    group1 {
        pinmux = < 0x301821 >, < 0x2f97e2 >;
        bias-pull-up;
        drive-open-drain;
        output-high;
    };
};

One change that I have not been as happy about is in troubleshooting Devicetree issues. After building a project, it’s often useful to inspect the build/zephyr/zephyr.dts file that is generated. I’ve included an example above, can you tell me if my pins have been properly assigned?

The more you use pinctrl, the more useful it becomes. I’m sure you’ve noticed that some nodes include groupN style groupings. This associates multiple pins with the same needs (ie: for SPI, these three pins are outputs, but this other pins is an input) which is a welcomed shorthand that saves us from assigning pin flags over and over again. Take some time to learn the new system and start taking advantage of its perks!

If there’s a driver built into Zephyr, controlling a part over Serial Peripheral Interface (SPI) is a snap. But there’s an ocean of parts out there and only so many built-in drivers in existence. Today I’m going to show you how to use generic SPI devices with Zephyr so you can try writing your own test code in userspace, to be used for the long term or in preparation for creating a driver.

SPI is simple, right?

In practice, communicating with SPI devices is simple…but only after you get the hardware peripheral configured for your chip. Zephyr is built for cross-compatibility, but that is sometimes its biggest usability flaw. The SPI bus and the devices themselves are abstracted so much that it’s hard to know where to begin if you need to do it all yourself.

I spent far too long searching the internet for a guide before I remembered that all Zephyr systems have tests. Don’t be me, check the test files before you head to Google.

It turns out that the Zephyr SPI driver tests tell us how to register a generic SPI device, and then how to talk to it.

Zephyr SPI Step-by-Step

For this experiment I have chosen perhaps the easiest SPI chip ever, the Microchip MCP3201 12-bit ADC. No need to send register settings to that device, you simply enable the chip-select pin and toggle the clock signal to start shifting out ADC readings. In short: all I need is to read 4-bytes. That’s it!

1. Enable SPI in KConfig

First things first, we need to tell Zephyr to build SPI support into the app. We also need the ability to control GPIO pins. Add these to your prf.conf file:

CONFIG_GPIO=y
CONFIG_SPI=y

Why enable a library unless you’re actually going to use it? Let’s add these line to main.c to ensure we can access all of the Zephyr SPI goodness:

#include <drivers/gpio.h>                                                                                                                                                     
#include <drivers/spi.h>

2. Add a SPI node in the overlay file

Next, we need to tell Zephyr where to find our SPI device. The idiomatic way of doing this is to use the DeviceTree. For this project I’m using a Sparkfun Thing Plus nrf9160 so I needed to remap some pins for this device.

&spi2 {
        status = "okay";
        cs-gpios = <&gpio0 18 GPIO_ACTIVE_LOW>;
        pinctrl-0 = <&spi2_default>;
        pinctrl-1 = <&spi2_sleep>;
        pinctrl-names = "default", "sleep";
        mcp3201: [email protected] {
                compatible = "vnd,spi-device";
                reg = <0>;
                spi-max-frequency = <1600000>;
                label = "mcp3201";
        };
};

&pinctrl {
        spi2_default: spi2_default {
                group1 {
                        psels = <NRF_PSEL(SPIM_SCK, 0, 19)>,
                                <NRF_PSEL(SPIM_MOSI, 0, 21)>,
                                <NRF_PSEL(SPIM_MISO, 0, 22)>;
                };
        };

        spi2_sleep: spi2_sleep {
                group1 {
                        psels = <NRF_PSEL(SPIM_SCK, 0, 19)>,
                                <NRF_PSEL(SPIM_MOSI, 0, 21)>,
                                <NRF_PSEL(SPIM_MISO, 0, 22)>;
                        low-power-enable;
                };
        };
};

Don’t be scared away, I had to include all of this in order to remap which pins were being used for SPI on this device. Your mileage will vary based on your chip, just make sure you choose the correct SPI node (spi2, in my case) and include the highlighted lines above as the added node.

Note that SPI requires a chip-select (CS) pin. Zephyr will handle this for you automatically. Study the example above. You will see one entry called cs-gpios. This is a comma-separated list inside of arrow-brackets. If you have multiple SPI devices, just add each CS pin inside those brackets. It’s a zero-indexed list, and the CS for the mcp3201 is the zeroth entry in that list.  I use an @0 in the path and reg = <0> to tell Zephyr which CS pin to use.

I’ve identified this device as compatible with vnd,spi-device. Technically I should be defining my own DeviceTree binding instead of leaning on this “vendor” definition that is meant for testing purposes only. To learn the recommended way, here’s a 2-hour Zephyr driver talk from ZDS 2022. Let’s not get bogged down in the voodoo of DeviceTree; shall we move along and leave the bindings creation process for a future post?

3. Set up the memory structure

The memory structure is a bit quirky for SPI reads. Generally speaking, you need to set up the memory structure to perfectly mirror how much data you want to read or write. I’m only going to be reading in this example.

I want to read four bytes so I need a four-byte array. That array needs to be a member of a spi_buf, and that spi_buf needs to be a member of a spi_buf_set:

uint8_t my_buffer[4] = {0};
struct spi_buf my_spi_buffer[1];
my_spi_buffer[0].buf = my_buffer;
my_spi_buffer[0].len = 4;
const struct spi_buf_set rx_buff = { my_spi_buffer, 1 };

Honestly this was very confusing for me to figure out. It might be this way for DMA-based SPI peripherals. Or maybe this schema enables you to kind of pre-parse data of different lengths. I haven’t found documentation on why this this structure is the way it is. If you have any insight, please let us know on the Golioth forum thread for this post.

Basically the SPI peripheral is going to shift bits until it runs out of room in your memory structure. So format this for exactly the length of data you need to read/write and Zephyr takes care of the rest.

4. Get the device from the DeviceTree

It’s time to get the SPI device from the DeviceTree. This is a great time to include the configuration for your SPI bus. I followed the Zephyr test file examples and first defined my config values:

#define SPI_OP  SPI_OP_MODE_MASTER |SPI_MODE_CPOL | SPI_MODE_CPHA | SPI_WORD_SET(8) | SPI_LINES_SINGLE

Once we get to the main function, it’s now a much simpler call to get your SPI device:

const struct spi_dt_spec mcp3201_dev =
                SPI_DT_SPEC_GET(DT_NODELABEL(mcp3201), SPI_OP, 0);

5. Issue read command

My standard advice for Zephyr applies here: always check the return codes. When working through this issue, the return codes and enabling debug-level logs in the SPI libraries helped me work out a problem with my memory structure.

ret = spi_read_dt(&mcp3201_dev, &rx_buff);
if (ret) { LOG_INF("spi_read status: %d", ret); }

What we have here is a SPI read operation which immediately prints out the return code if there were any errors. If not, the data you wanted to get from your device is now ready and waiting inside of the buffer that was passed in during the read command.

Further reading

It was quite a task for me to figure all of this out. It would have been much easier had I first looked to the test files (as I mentioned previously). You may want to write to a SPI device, or read and write at the same time. This same process can be used! You just need to also have a memory structure for the data you want to transmit, and you need to use the write, or the “transceive” versions of the SPI functions.

I suggest you start by reading through the SPI API reference. You should also make time to work through all of the functions in the spi_loopback test app which uses SPI in just about every way possible on Zephyr.

Beyond that, if you do have any questions, hit up the Zephyr discord or start a thread on the Golioth forums.

Founding Engineer of Golioth, Alvaro Viebrantz went to the Zephyr Developer Summit…and said the hardware isn’t the only part of an IoT system that matters. That’s bold! This was a room full of hardware and firmware designers!

In reality, this talk is about everything else you should consider when building a device using the Zephyr Real Time Operating System (RTOS). Alvaro goes through the many layers and considerations when architecting large scale Internet of Things (IoT) deployment, from the servers, to the apps, to how devices are communicating back to the cloud, and yes, all the way down to the code put onto the end device.

Top Down Design

Most IoT users first experience a device not from holding the hardware in their hand and seeing it do some particular action. Instead, users rely on things like Dashboards and Applications. The physical device is often hidden from the end user, at least inside an enclosure, and sometimes entirely (as in the case of something like a home monitor in your attic).

It’s important to start by mapping your data model, often before you even start your first schematic. What will your device do? How will it be accessed hierarchically? What other elements are part of the system or will be in the future?

Also important, you should follow the ownership of both the physical device and that device’s output data all the way up the chain back to the user. How will a user claim a device belongs to them? How will you organize the various levels of permission required? What kind of questions will your users seek to answer with the end device or with the entire fleet? (ie. “Do any vehicles in the fleet have low battery?”)

Who is in charge here?

After you have considered how you will configure your database and what the end user will see when they are interacting with data, you should move on to access levels and which parts of the system are accessible by different individuals. This may include access to the end application, but should also include access control to the configuration components of the system. You don’t want your end users to be able to access your device management console, even if they are the ultimate owner of the device. Instead, you will want to provision the devices so you know the unique identifiers of the end-device, and then assign that and the access level to the user.

Additional layers required for a realistic IoT deployment

As you continue to move “down the stack” from Applications all the way to the hardware, there is still a good chunk of Cloud in between. You’ll want to make sure you at least understand all of the elements you’ll likely want in your design, including:

  • Device management
  • Device state and configuration
  • Bidirectional command and control
  • Time-series data and events

Connecting to the network

We’re almost down at the device itself, but the connection to the network is another key consideration before we get there. Not just the method of connection (which ultimately relies on the hardware), but also the protocol and the pathways of each packet coming back. In Golioth’s case, we default to CoAP, which is a lower overhead, UDP based protocol. All of our device SDKs include support to make it easy to talk to CoAP endpoints on the Golioth servers. As Alvaro mentioned in his talk, and as our SDKs also feature, encoding can help to shrink messages even further, using things like protobuf, JSON, and CBOR.

Finally the hardware

By this point in the room, we were squarely back into the realm of expertise of Zephyr developers, though many now had a greater appreciation of all the other things required for IoT systems. Alvaro mentioned how some of the interaction of the connectivity on-device in Zephyr could affect your decision making further up the stack. This talk to a packed room showed developers how to plot out a successful IoT deployment beyond the limits of the hardware, and to think about IoT systems from the hardware, all the way up to the end user.

 

Controlling 10 devices is easy, controlling 10,000 is a different story. The trick is to plan for scale, which is what we specialize in here at Golioth.

A few weeks ago we announced the Golioth Device Settings Service that enables you to change settings for your entire fleet at the click of a button. Of course, you can drill down to groups of devices and single devices too. While the announcement post showed the Settings Service using our ESP-IDF SDK, today we’ll look at this setting service from the device point of view with Zephyr RTOS.

Device-side building blocks

The good news is that the Golioth Zephyr SDK has done all the hard work for you. We just need to do three things to start using the settings service in any Zephyr firmware project:

  1. Enable the settings service in KConfig
  2. Register a callback function when the device connects to Golioth
  3. Validate the data and do something useful when a new setting arrives

Code walk-through

Golioth’s settings sample code is a great starting point. It watches for a settings key named LOOP_DELAY_S to remotely adjust the delay between sending messages back to the server. Let’s walk through the important parts of that code for a better understanding of what is involved.

1. Enable settings in KConfig

CONFIG_GOLIOTH_SETTINGS=y

To turn the settings service on, the CONFIG_GOLIOTH_SETTINGS symbol needs to be selected. Add the code above to your prj.conf file.

2. Register a callback for settings updates

static void golioth_on_connect(struct golioth_client *client)
{
    if (IS_ENABLED(CONFIG_GOLIOTH_SETTINGS)) {
        int err = golioth_settings_register_callback(client, on_setting);

        if (err) {
            LOG_ERR("Failed to register settings callback: %d", err);
        }
    }
}

To do anything useful with the settings service, we need to be able to execute a function when new settings are received. Register the callback function each time the device connects to Golioth. Your app may already have a golioth_on_connect() function declared, look for the following code at the beginning of main and add it if it’s not already there.

	client->on_connect = golioth_on_connect;
	client->on_message = golioth_on_message;
	golioth_system_client_start();

Each time the Golioth Client detects a new connection, the golioth_on_connect() function will run, which in turn registers your on_setting() function to run.

3. Validate and process the received settings

enum golioth_settings_status on_setting(
        const char *key,
        const struct golioth_settings_value *value)
{
    LOG_DBG("Received setting: key = %s, type = %d", key, value->type);
    if (strcmp(key, "LOOP_DELAY_S") == 0) {
        /* This setting is expected to be numeric, return an error if it's not */
        if (value->type != GOLIOTH_SETTINGS_VALUE_TYPE_INT64) {
            return GOLIOTH_SETTINGS_VALUE_FORMAT_NOT_VALID;
        }

        /* This setting must be in range [1, 100], return an error if it's not */
        if (value->i64 < 1 || value->i64 > 100) {
            return GOLIOTH_SETTINGS_VALUE_OUTSIDE_RANGE;
        }

        /* Setting has passed all checks, so apply it to the loop delay */
        _loop_delay_s = (int32_t)value->i64;
        LOG_INF("Set loop delay to %d seconds", _loop_delay_s);

        return GOLIOTH_SETTINGS_SUCCESS;
    }

    /* If the setting is not recognized, we should return an error */
    return GOLIOTH_SETTINGS_KEY_NOT_RECOGNIZED;
}

The callback function will provide a key (the name of the settings) that is a string type, and a golioth_settings_value that includes the value, as well as an indicate of the value type (bool, float, string, or int64). The callback function should validate that the expected key was received, and test that the received value is the proper type and within your expected bounds before acting upon the data.

The final piece of the device settings puzzle is to return an enum value indicating the device status after receiving a new setting. This is used by the Golioth Cloud to indicate sync status. Our example demonstrates the GOLIOTH_SETTINGS_SUCCESS and GOLIOTH_SETTINGS_KEY_NOT_RECOGNIZED status values, but it’s worth checking out the doxygen reference for golioth_settings_status to see how different invalid key and value messages can be sent to Golioth for display in the web console.

Coordinating with the Golioth Cloud

The device settings key format is tightly specified by Golioth, allowing only “A-Z (upper case), 0-9 and underscore (_) characters”. We recommend that you add a key to the Golioth web console while writing your firmware to ensure the device knows what to expect.

Each device page includes a status indicator to show if the settings are in sync with the Golioth cloud.

Additional information is available when a device is out of sync. Here you can see that hovering over the information icon indicates the key is not valid. For this example I created a VERY_IMPORTANT_BOOL_7 but didn’t add support for that to the firmware. This is the result of our callback function returning GOLIOTH_SETTINGS_KEY_NOT_RECOGNIZED. In cases like this, Golioth’s Over-the-Air firmwmare update (OTA) service can help you to push a new firmware to your devices in the field that will recognize the newly created Settings variable.

IoT Fleet control on day one

The pain of IoT often comes when you need to scale. Golioth’s mission is to make sure you avoid that pain and are ready to grow starting from day one.

Our device settings service gives you the power to remotely change settings on your entire fleet, on a group of devices, and all the way down to a single device. You are able to verify your devices are in sync, and receive helpful information when they are not. Take Golioth for a test drive today, our dev tier is free for the first 50 devices.