Timon Skerutsch is a software and electronics engineer who enjoys the systems design aspect of the work the most, from the cloud all the way down to silicon. He is the founder of the product development consultancy Diodes Delight.

Last year, I made the decision to do a personal prototype project with Zephyr instead of my go-to prototyping choices of Arduino, MicroPython, and CircuitPython. The latter two I still use a lot for rapid prototyping, as scripting languages like Python are hard to beat.

For the forthcoming project, I already knew where I was heading. I needed a sensor for my garden irrigation system that tracks the water level in my water reservoir. I had two requirements:

  • See the water level live on-site
  • Have the data transmitted to the internet so that I can see how things are doing while I’m away

The system regularly had leaks and failures in the irrigation pipes, which caused the automated watering pump to not only waste water but also fail to water my plants. This is detrimental during a heat wave! That added an additional requirement of an alarm to alert me of abnormal situations. When I started the project, I reached for Zephyr first. Let’s talk about why.

Why Arduino, anyway?

Change is a constant; no more so, than the early design phases of a product, when requirements rapidly shift. Prototyping is a crucial aspect of any development process and we often employ different tools for this than we do for our production firmware.

In the firmware world, the Arduino framework has been an immense success. Not only for teaching firmware development, but also as a quick and easy way to try out concepts and develop prototype solutions.

Not without controversy though. To this day, many embedded engineers will utter a silent curse on Arduino whenever it is encountered on the way to production. The supposed prototype firmware has morphed into the production firmware and now that needs to be extended with very complex functionality.

When deadlines are near and the stress levels are high, it is very appealing to companies to just keep using the code base that already seems to do 90% of the job. A lot of products end up shipping firmware based on frameworks like Arduino and then try to deal with the “last 10% of the work”. If you have developed software for a while you probably know those famous last 10% can be 90% of the work, completely derailing budgets and timelines.

But what is the issue? At the end of the day, Arduino is just a very light HAL based on C and C++! You can do whatever you could do in a bare metal C project. The emphasis is on “light”, which is very beneficial when you have only very basic requirements. Maybe you can get by just fine with printf() debugging and hitting that compile button in your IDE.

The Arduino HAL’s success was in part due to its simplicity and fully integrated development flow but it offers very few solutions to the modern challenges firmware developers are facing. When you consider the complexities you are faced with in modern devices, Over-the-Air firmware updates, firmware update encryption, Continuous Integration, connection to a cloud service, tracking of device status and metrics, then things start to look different.

Zephyr is changing things up in the industry

The Zephyr Project Real Time Operating System (RTOS) has seen a lot of adoption over the past years, even by Arduino themselves. They became a member of the Zephyr Project in 2023 and now contribute to the code base.

Zephyr is a very “IoT aware” RTOS and offers a lot of robust solutions to many of the very complex topics I mentioned. It is also one of the main targets of the Golioth Firmware SDK for that reason.

When developers first start interacting with Zephyr they often tend to be a bit intimidated. Device Tree and KConfig may be familiar tools for Embedded Linux developers, but not for someone coming from bare metal C or FreeRTOS. (Editor’s note: this is why Golioth offers free training around Zephyr).

Zephyr-specific tooling like the west meta tool means there’s a lot to learn when you start diving into Zephyr. You might start to question if that work is worth it. Especially early on in a development process where you want to move quickly and prove your concepts. You might feel a huge system like Zephyr could slow you down.

Due to the steep learning curve, Zephyr does not really have a reputation for being a tool for prototyping. But I think Zephyr has very much a place in that phase of a project and it comes with a lot of benefits once you move beyond the prototype: You are already in an environment that won’t hold you back when it comes to solving the tough problems of modern production ready firmware.

Now that I am up the (arguably steep) learning curve associated with Zephyr, I think in many cases I can produce a working solution a lot quicker than with Arduino or even MicroPython.

Not just for production grade firmware

Since I was starting a prototype, I chose an ESP32 dev board I had laying around which came with a nice little OLED screen. For the sensor, I opted for an industrial liquid level sensor. They are essentially pressure sensors in a sturdy form factor that measure the pressure differential of the outside air and the pressure seen in the liquid container.

I needed something rugged and precise to track abnormal water usage so that was a perfect solution. I ended up getting a stable 0.2mm resolution for my water column, much more than I needed. The sensor is a simple 4-20mA current loop that you often see in industrial automation and I connected that to an external precision ADC.

My firmware needs included:

  • WiFi provisioning
  • Network and application protocol to get the data to a server
  • OTA to update the device remotely
  • A GUI to show water levels on the OLED
  • ADC reads to ingest the sensor data

I opted to use Golioth for networking and OTA. While not a typical service for a hobby project, they have a (recently updated) free tier for individuals and it made the whole thing really easy. It only takes a couple lines of code to integrate into any Zephyr project and makes transmitting data to a database as easy as Serial.print(). Having OTA available is a matter of a KConfig option. Most importantly I don’t need to manage an internet facing server application!

net_connect();

golioth_client_t client;
const golioth_client_config_t* client_config = golioth_sample_credentials_get();
client = golioth_client_create(client_config);
golioth_client_register_event_callback(client, on_client_event, NULL);

err = golioth_lightdb_set_int_sync(client, "water-level", water_level/1000, 2);
if (err) {
    LOG_WRN("Failed to transmit water-level: %d", err);
}

I could have directly implemented CoAP or MQTT and host my own server for the receiving side. Both protocols are natively supported by Zephyr, which means I have flexibility if I change my mind on the server side in the future.

OTA firmware updates is also a concept native to Zephyr and very important: no matter what platform I choose! The Arduino ESP32 core has an option for OTA but if you are looking at any other MCU you would have to implement that from scratch which is a whole project in itself.
In Zephyr this is all enabled by the fantastic MCUBoot bootloader.

Abstractions are your friend in a complex world

The platform agnostic nature of Zephyr is powerful. Say you have already written a lot of code and then notice that your chosen MCU does not actually fulfill your needs. You don’t need to start from scratch because all of these advanced APIs are fully abstracted in Zephyr. You can retarget your code to a different platform with minimal code changes. The primary work will be in recreating your devicetree for the new hardware.

Arduino also is known for abstraction, but when it comes to more complex features the Arduino HAL is not defining an interface. Generally, you tend to need to stick to a particular platform if you want to take advantage of the underlying hardware’s fancy features, that is if they are available at all (in Arduino).

Lock-in with a specific IC is a painful lesson we all learned during the chip shortages of 2021 and 2022. Device Tree overlays are a great tool to stay on-top of changing hardware and describe those changes in a clean way. That flexibility is not only important from a risk perspective. Staying flexible during the prototype stage (where requirements change rapidly) allows you to try out different sensors and peripherals.

Changing a sensor in Zephyr is a matter of changing a dozen lines of devicetree definitions without needing to touch a single line of C code. This is made easier when the sensor is “in tree”, but it not the only way to use a new sensor. Devicetree also becomes a powerful tool in the early days of hardware development where your product might go through many revisions and changes. People on your team might be working with different hardware revisions but require the latest bug fixes.

This can quickly become tough to manage in the firmware if you had pin or even IC changes. No need for a ton of #ifdef‘s; all you need is a set of devicetree overlays that describe your various board revisions, the C code can most often stay the same. This not only makes your life easier but also helps reduce mistakes and stale code.

If you are still trying out options you can also interactively work with sensors through the Zephyr Shell, which makes for a great workflow to quickly try out several sensor candidates without writing firmware.

During my project I was unsure whether to choose a different MCU, because the built-in ADC of the ESP32 is quite noisy. In the end, I kept the ESP32 and chose to use an external ADC. My code did not have to change because the ADC API abstracts that away. It was just a matter of defining what ADC my project should use in the devicetree. My code does not need to care if that is an external I2C device or a peripheral internal to my MCU.

/{
    zephyr,user {
        io-channels =
            <&adc_ext 0>;
    };
};

&i2c0 {
    status = "okay";
    adc_ext: mcp3421@68 {
        compatible = "microchip,mcp3421";
        reg = <0x68>;
        #io-channel-cells = <1>;
    };
};

&adc_ext {
    status = "okay";
    channel@0 {
        reg = <0>;
        zephyr,gain = "ADC_GAIN_1";
        zephyr,reference = "ADC_REF_INTERNAL";
        zephyr,acquisition-time = <ADC_ACQ_TIME_DEFAULT>;
        zephyr,resolution = <18>;
        zephyr,differential;
    };
};

That is the benefit of fully abstracted subsystems, your application’s assumptions can stay the same most of the time. Last minute system changes are less painful during firmware development.

Complex UI’s don’t have to be complex to build

For the GUI, I went with LVGL, a popular UI framework that has been integrated into Zephyr.
That was probably the most eye opening experience to me. Normally you would have to mess with display drivers that all work very differently depending on the plugged in display. Then I would need to write code to manually transfer the rendered framebuffer to that display, which  again, tends to work differently with each display.

In Zephyr all I have to do is to modify the devicetree for which display driver my OLED needs, the resolution, and the bus it is connected to.

&spi3 {
    status = "okay";
    st7789v_st7789v_ttgo_128x64: st7789v@0 {
        compatible = "sitronix,st7789v";
        spi-max-frequency = <20000000>;
        reg = <0>;
        cmd-data-gpios = <&gpio0 16 GPIO_ACTIVE_LOW>;
        reset-gpios = <&gpio0 23 GPIO_ACTIVE_LOW>;
        width = <135>;
        height = <240>;
        x-offset = <53>;
        y-offset = <40>;
        vcom = <0x19>;
        rgb-param = [CD 08 14];
    };
};

With that done you can write powerful UIs with just a couple lines of code, all the hard stuff is handled behind the scenes. I figured the UI part would be the majority of work for this project, but ended up being done in under an hour.

Often Arduino prototypes tend to have character displays or use the same old school bitmap font because fonts are hard and font systems even harder. In LVGL, you have an array of modern fonts available and it’s fairly easy to include your own font.

Arranging elements is also trivial in LVGL. No need to manually calculate a bunch of stuff like your text length. It has a lot of functions available for laying out complex arrangements.
You can build some really pretty smartphone level UIs with it. These run on very constrained hardware and it doesn’t cost you your sanity in the process!

const struct device *display_dev;
display_dev = DEVICE_DT_GET(DT_CHOSEN(zephyr_display));

lv_obj_t *level_label;
lv_obj_t *status_label;
static lv_style_t level_style;
lv_style_init(&level_style);
lv_style_set_text_font(&level_style, &lv_font_montserrat_48);

level_label = lv_label_create(lv_scr_act());
lv_obj_align(level_label, LV_ALIGN_CENTER, 0, -20);

status_label = lv_label_create(lv_scr_act());
lv_obj_align(status_label, LV_ALIGN_BOTTOM_MID, 0, -50);
lv_obj_add_style(level_label, &level_style, 0);
// display the current water level
lv_label_set_text_fmt(level_label, "%llu", water_level/1000U);
lv_task_handler();

Quick, once you know how to get around

Within a day I had firmware for my hardware and it even looked…pretty!
Since its creation, my device has dutifully reported the water level and withstood the winter season.

The application is ~180 lines of C code, including a lot of error handling. That is really not a lot of code, for so much complex functionality. This is only possible thanks to all of the available abstractions that make writing the actual application logic a breeze. In the background there are multiple threads running but my code doesn’t even need to be aware of that. It is all handled by the kernel and the well-written libraries.

While simple things like an ADC read can be very verbose in Zephyr and a bit more complicated than in Arduino land, the hard parts–the things we often perceive as the last 10%–are a whole lot easier! You don’t want to start empty handed when you are tasked to implement encrypted firmware updates, OTA, telemetry or integrating the cloud team’s new backend system.

The grass is not always greener, of course. Zephyr is still comparatively young in embedded terms and while there are a lot of devices already supported, it is hard to beat the vast amount of drivers available for Arduino. In some situations you might want to do a first board bring up with an existing Arduino driver before implementing a Zephyr driver. License permitting, the driver code could be the basis for a port to Zephyr.

One reason why I think it is so common for firmware to get “stuck” in Arduino-land is because there has not been a good (free) alternative if you wanted to write C (or C++) and not pigeon hole yourself early on into one specific obscure vendor toolchain. If there is nothing obvious to move to, it can make the decision even harder and it will be put off until it’s usually too late.

Concepts like devicetree and a complex build system like CMake can be daunting at first, but there are ripe benefits at the end of the learning curve. If you want to learn Zephyr, Golioth regularly offers free Zephyr training sessions or you can read some of the great blog posts that cover the more gnarly bits of Zephyr in digestible bite sized portions.

Chris Duf is a chip-to-cloud software architect, working on embedded systems and IoT applications. He is the author of the DTSh tool.

Devicetree and the Devicetree Source Format (DTS) are major parts of Zephyr RTOS. But understanding the DTS output of an entire system—not to mention troubleshooting devicetree errors—can prove to be quite a challenge. This is especially true for beginners.

In this article, I’ll first discuss why the content of these DTS files is interesting, then introduce the Devicetree Shell (DTSh), a simple command line tool for visualizing the devicetree in a DTS, with versatile export capabilities.

Let’s do a quick demo before diving into the details. At build-time, during the configuration phase, Zephyr generates a devicetree that will represent the system hardware during the actual build phase. Two files are generated:

  • build/zephyr/include/generated/devicetree_generated.h: the devicetree in C, that describes the hardware to the device driver model
  • build/zephyr/zephyr.dts: the same devicetree in Devicetree Source Format (DTS), generated for debugging

DTSh can be used to present a much more human-readable view of these results:

The Devicetree Shell

find --with-bus * --OR --on-bus * -T --format NYcd

You can try this yourself, with the following in mind:

  • DTSh should install and run fine on GNU Linux (including WSL) and macOS with Python 3.8 to 3.11
  • on Windows, the readline API, on which DTSh relies for auto-completion and command history, is no longer distributed with the Python Standard Library: as a consequence, the user experience will be significantly degraded on this platform

Devicetree and Zephyr’s Driver Model

Devicetree is first a specification for a standardized data structure that can describe a system hardware and its initial configuration. Beside the DTS format we’ve already mentioned, a binary blob format is also defined (Flattened Devicetree, DTB).

Devicetree is primarily designed as a complete interface between a boot program like a bootloader and a client program like an operating system. When using Devicetree, Linux applies this approach (see Linux and the Devicetree):

  • the DTS for the system hardware is compiled to DTB, and linked into the vmlinux image
  • the boot program eventually passes a pointer to the devicetree in DTB format to the operating system client
  • the Linux kernel then walks through the devicetree to populate its device model, allocating device structures dynamically for the enabled DT nodes

While the hardware that the Linux kernel targets can benefit from this design, Zephyr targets embedded systems, which are more constrained, especially regarding persistent memory:

  • the binary blob is then wasted .rodata (typically Flash memory)
  • the kernel API needed to walk through the devicetree is wasted .text (also typically Flash memory)
  • the run-time overhead is worthless, most of the time device structures will be statically allocated

Zephyr has therefore chosen a different approach:

  • the devicetree is not passed to the Zephyr client program (everything happens at build-time)
  • only the device structures for enabled DT nodes are statically allocated into the .rodata section
  • the API to access these devices structures is entirely macro based, and does not end up in the .text section

Compared to Linux, we could say that the devicetree is passed to Zephyr in C format at build-time.

And, as a Zephyr application developer:

  • You use macros like DT_PATH or DT_NODELABEL to get DT node identifiers (this relies on the generated devicetree_generated.h, which is included by zephyr/devicetree.h)
  • Others like DEVICE_DT_GET to get pointers to the corresponding device structures

All this macrobatics is well explained in a presentation Marti Bolivar gave at a Zephyr Developer Summit in 2022: a few things have changed but still really worth watching.

So, why should I be interested in the DTS files content ?

A one sentence answer (well, two) could be:

  • Because the generated DTS will always describe the whole system hardware from the board files (the .dts and .dtsi files, plus possible overlays), including disabled devices and buses
  • Since the DTS file is generated early during the configuration phase, chances are that it will exist even when the build eventually fails: here, checking the devicetree content will prove helpful in debugging common issues (you know, the DT_N_S_ and __device_dts_ord_ things)

Well, but then what application can I use to open this DTS file, other than a text editor?

DTSh is just that: a simple tool to navigate, visualize and search a devicetree described in a DTS file, along with the bindings that specify and document its content.

In the remainder of this article, we’ll install DTSh before we reconsider this question with a new tool on hand.

Install DTSh

For simplicity, the examples in this article will assume DTSh is installed alongside your Zephyr development environment (please refer to DTSh’s Getting Started Guide for details about installation methods): just enter pip install dtsh from the same prompt where you usually run west commands.

You should now be able to open DTS files in the Devicetree Shell, e.g.:

$ cd zephyr/samples/sensor/bme680
$ west build
$ dtsh
dtsh (0.2rc1): Shell-like interface with Devicetree
How to exit: q, or quit, or exit, or press Ctrl-D

/
> ls -l
 Name              Labels          Binding
 ───────────────────────────────────────────────────────
 chosen
 aliases
 soc
 pin-controller    pinctrl         nordic,nrf-pinctrl
 entropy_bt_hci    rng_hci         zephyr,bt-hci-entropy
 sw-pwm            sw_pwm          nordic,nrf-sw-pwm
 cpus
 leds                              gpio-leds
 pwmleds                           pwm-leds
 buttons                           gpio-keys
 connector         arduino_header  arduino-header-r3
 analog-connector  arduino_adc     arduino,uno-adc

In the above example:

  • no need to pass the DTS file path to DTSh: by default it will try to open the devicetree at build/zephyr/zephyr.dts, which the preceding west build command has just generated
  • no need to tell where to search for the YAML binding files: DTSh first tries to retrieve this information from a CMake cache at build/CMakeCache.txt, which should also exist

As you can see, it really presents itself like a shell: a prompt where you enter commands like cd, ls or find, but these operate here on device paths rather than on file system paths. And, as with a shell, you can redirect commands output to files.

Only a few principles are necessary to start using DTSh:

  • You can navigate the devicetree (change the current working branch) with the cd command like a hierarchical file system
  • Device paths can be absolute path names or relative paths from the current working branch, may start with DTS labels and support some globbing
  • By default, commands will output paths to represent nodes, just like their Unix homonyms represent files or directories (ls, find and tree); the --format FMT option will instead format the output to show the information you’re looking for (-l is a shortcut for a configurable default format)
  • Command output redirection also uses the POSIX syntax, e.g. tree -lR > devicetree.html (the file extension determines the format)
  • For a better default user experience, prefer a dark Terminal theme and run the shell full-screen or in a maximized window
  • Last but not least, don’t be afraid to press the TAB key twice: it will trigger contextual auto-completion from nearly everywhere

A word of advice: if a command line tool looks scary, don’t worry. The Devicetree Shell syntax is simple and consistent, and command line auto-completion and history will save you from both memorizing and typing most things.
The DTSh handbook provides detailed documentation and various examples.

Back to DTS files

Let’s go back to the use cases we put forward:

  • Quickly learn about a new hardware configuration
  • Debug mysterious Devicetree issues

Learn about Hardware Configurations

When starting a new project, or porting an existing application to another board, it’s nice to get some visual representation of what’s available to your program.

Let’s say you’re interested in the nRF52840 DK (nrf52840dk_nrf52840), which you haven’t yet used with Zephyr.

First, generate the corresponding devicetree, then open it in dtsh:

# Any Zephyr sample that supports the board would do just fine.
cd zephyr/samples/sensor/bme680

# Remember, the configuration phase is enough.
cmake -B build -DBOARD=nrf52840dk_nrf52840

# No need to pass arguments, dtsh will open build/zephyr/zephyr.dts
dtsh

Try the command tree --format NKYC: it will dump the whole devicetree as a detailed tree view, with all device labels and alias (Also Known As), buses and bindings, ending with something like in the screenshot below.

Devicetree as tree view

tree --format NKYC

You have probably noticed:

  • Some nodes appear dim: these are the disabled buses and devices
  • Most of the time, the content of the Binding column appears with a (possibly dashed) underline: these are hyperlinks to the corresponding binding files, hovering over the text will reveal their paths, and clicking while holding the Ctrl key will open them in with your default application for handling YAML files
  • The content of the Binding column may also be anchored to its parent: these are child-bindings specified in the same YAML file

But the most blindingly obvious was probably that the command literally dumped its output, making impossible to understand the tree view calmly before it disappears. This can be addressed in two ways:

  • Try tree --format NKYC --pager: this will pass the command output to the system pager, where you can take your time (press h to display the pagers’ help screen, q to exit the pager and return to the dtsh prompt)
  • You can also export the tree view to an HTML file, e.g. tree --format NKYC > nrf52840dk_nrf52840.html

Another similar use case could consist of finding some information without having to resort to the SoC’s datasheet. You may not have the datasheet readily available, nor do you want to have it littering your desk.

The size and base addresses of the different kind of available memory is typically something you can forget, especially if you work with a handful of boards.

/
> find -E --also-known-as (image|storage).* --format NLrd > memory.txt

/
> find --also-known-as ram -i --format NLrd >> memory.txt

You now have a simple and nice memory.txt file:

Name             Labels             Registers         Description
──────────────────────────────────────────────────────────────────────────────────────────────────────────────
partition@c000   slot0_partition    0xc000 (472 kB)   Each child node of the fixed-partitions node represents…
partition@82000  slot1_partition    0x82000 (472 kB)  Each child node of the fixed-partitions node represents…
partition@f8000  storage_partition  0xf8000 (32 kB)   Each child node of the fixed-partitions node represents…

Name             Labels  Registers            Description
──────────────────────────────────────────────────────────────────────────────
memory@20000000  sram0   0x20000000 (256 kB)  Generic on-chip SRAM description

Tip: find supports a handful of criteria to match devicetree nodes with. To get the full list, start a command line with find --, then press TAB twice. The auto-completion will enumerate all the options that have a long name (short option names begin with a single -): the criteria are those starting with --with unless another term really seems more natural, e.g. --also-known-as or --on-bus.

/
> find --[TAB][TAB]
-h --help                      print command help
--also-known-as PATTERN        match labels or aliases
--chosen-for PATTERN           match chosen nodes
--count                        print matches count
--enabled-only                 filter out disabled nodes or branches
--format FMT                   node output format
--NOT                          negate the criterion chain
--on-bus PATTERN               match bus of appearance
--OR                           match any criterion instead of all
--order-by KEY                 sort nodes or branches
--pager                        page command output
--with-alias PATTERN           match aliases
--with-binding PATTERN         match binding's compatible or headline
--with-binding-depth EXPR      match child-binding depth
--with-bus PATTERN             match supported bus protocols
--with-compatible PATTERN      match compatible strings
--with-description PATTERN     grep binding's description
--with-device-label PATTERN    match device label
--with-dts-ord EXPR            match dependency ordinal
--with-irq-number EXPR         match IRQ numbers
--with-irq-priority EXPR       match IRQ priorities
--with-label PATTERN           match node labels
--with-name PATTERN            match node name
--with-path PATTERN            match path name
--with-reg-addr EXPR           match register addresses
--with-reg-size EXPR           match register sizes
--with-status PATTERN          match status string
--with-unit-addr EXPR          match unit address
--with-unit-name PATTERN       match unit name
--with-vendor PATTERN          match vendor prefix or name

DTS labels often match names in the SoC’s documentation:

/
> ls &[TAB][TAB]
acl                  Nordic nRF family ACL (Access Control List)
adc                  Nordic Semiconductor nRF family SAADC node
arduino_adc          ADC channels exposed on Arduino Uno (R3) headers…
arduino_header       GPIO pins exposed on Arduino Uno (R3) headers…
arduino_i2c          Nordic nRF family TWI (TWI master)…
arduino_serial       Nordic nRF family UARTE (UART with EasyDMA)
arduino_spi          Nordic nRF family SPIM (SPI master with EasyDMA)
boot_partition       Each child node of the fixed-partitions node represents…
button0              GPIO KEYS child node
... stripped ...

/
> ls &button0 -ld
Name      Labels   Binding
───────────────────────────────────────
button_0  button0  GPIO KEYS child node

/
> ls &timer0 -ld
Name            Labels  Binding
────────────────────────────────────────
timer@40008000  timer0  nordic,nrf-timer

/
> ls &i2c0 -ld
Name          Labels             Binding
───────────────────────────────────────────────
i2c@40003000  i2c0, arduino_i2c  nordic,nrf-twi

Debug Devicetree Issues

We are now equipped to look again at these mysterious DT_N_S_ and __device_dts_ord_ things.

Let’s start by disabling the I2C bus in boards/nrf52840dk_nrf52840.overlay to intentionally introduce a build error:

&i2c0 {
  status = "disabled";
  bme680@76 {
    compatible = "bosch,bme680";
    reg = <0x76>;
  };
};

west build will fail with one of these mysterious errors:

$ west build

... stripped ...

 from zephyr/drivers/sensor/bme680/bme680.c:14: error: '__device_dts_ord_124'
 undeclared here (not in a function); did you mean '__device_dts_ord_14'?
 89 | #define DEVICE_NAME_GET(dev_id) _CONCAT(__device_, dev_id)
    |                                         ^~~~~~~~~

... stripped ...

FATAL ERROR: command exited with status 2: /usr/bin/cmake --build zephyr/samples/sensor/bme680/build

First, you need to know that __device_dts_ord_124 reads “the device with dependency ordinal 14”, where the dependency ordinal is a non-negative integer value such that the value for a node is less than the value for all nodes that depend on it.

Knowing how to read C tokens like DT_N_S_soc_S_i2c_40003000 is also useful:

  • DT_N reads “devicetree node” (DT node)
  • _S_ represents the / that separates node names
  • DT_N_S_soc_S_i2c_40003000 is then the C token for the node with path name /soc/i2c@40003000

And indeed, if your search devicetree_generated.h for DT_N_S_soc_S_i2c_40003000, you’ll eventually come across the lines below (scattered among more than ten thousand):

#define DT_N_S_soc_S_i2c_40003000_ORD 124
#define DT_N_S_soc_S_i2c_40003000_STATUS_disabled 1
#define DT_N_S_soc_S_i2c_40003000_S_bme680_76_ORD 125
#define DT_N_S_soc_S_i2c_40003000_S_bme680_76_REQUIRES_ORDS 124

Which should help you conclude that the I2C bus the BME680 sensor should connect to is actually disabled.

Compare this to what DTSh can tell you with a single command line:

/
> find --with-dts-ord 124 --format psTd
Mysterious build error

A not so mysterious issue.

Epilogue

You should now be able to:

  • Open DTS files that represent your hardware configurations (hint: cd myproject && cmake -B build -DBOARD=myboard && dtsh)
  • Navigate (cd) and visualize (ls, tree) the devicetree
  • Experiment with find --[TAB][TAB] to search for buses, bindings, interrupts, and more
  • Experiment with the --format --[TAB][TAB] option to see the information you’re interested in
  • Experiment with command output redirection to text, HTML and SVG (hint: find --with-bus * --OR --on-bus * -T --format NYcd > buses.svg)
This brief introduction to DTSh misses interesting topics, don’t hesitate to skim through the documentation to find:
  • More usage examples
  • How to configure DTSh behavior and appearance with preferences and theme files
  • Learn more about the command line interface, auto-completion and history
  • A summary of useful key bindings (hint: Up and Down move backward and forward through the command history, respectively)
  • A detailed description of the built-in commands, their options and parameters

And remember, auto-completion (pressing TAB twice) is your friend, and can even list the available commands:

/
> [TAB][TAB]
alias     list aliased nodes
cd        change the current working branch
chosen    list chosen nodes
find      search branches for nodes
ls        list branch contents
pwd       print path of current working branch
tree      list branch contents in tree-like format

RFC: DTSh: DTS file viewer with a shell-like command line interface is a proposal to upstream the Devicetree Shell as a new Zephyr extension to West, such that you would eventually be able to just run with:

$ west build
$ west dtsh

This is the appropriate place to comment on: the potential usefulness of such a tool, the acceptability of the command line approach for beginners, the features you found interesting and those you would like to see added (accessing arbitrary node properties is definitely a miss), or the technical reasons why you think it is a good idea, or not, to upstream DTSh as a new West command. This is not the appropriate place to report bugs, though. Please open issues on the DTSh project instead.

Nordic’s nRF9160 cellular modem includes a great peripheral called the Key Management Unit (KMU). This secure key storage mechanism lets you write keys to it which cannot be read back. However, they can still be used for DTLS authentication. In this video and blog post I’ll walk you through how to use the feature with the newest release of the Golioth Firmware SDK.

Overview of secure storage and TLS Tag authentication

With the v0.10.0 release of the Golioth Firmware SDK, credentials may be selected using the TLS secure tag. One example of hardware that embraces this is the Nordic nRF9160, which implements Zephyr’s TLS credential management API. Credentials (either x509 certificates or pre-shared keys) are stored on the device using a security tag. Pass that tag to the Golioth Firmware SDK and enable offloaded DTLS sockets in order to utilize those securely stored secrets.

Since these credentials are stored separately from firmware, they are persistent and you can store multiple different credentials. At runtime, pass the security tag as a parameter when creating the Golioth client and you’re all set.

How to store credentials on the nRF9160

Storing credentials on the nRF9160 is accomplished in two steps: first generate and prepare the credentials, then place them on the device using AT commands. (Don’t worry, there are helper tools to generate the AT commands for you.)

Generating Certificates

Golioth has long supported certificate authentication. You can follow the Certificate Authentication guide in our Docs. You will need to use the .pem version of the device certificate and key, and you’ll also need the root certificate from the CA that issued Golioth’s server certificate (we use LetsEncrypt.org).

  1. Generate the root certificate and device certificate by following the docs page.
  2. Download the CA certificate from Let’s Encrypt

Now upload the public root certificate you generated (golioth.crt.pem) to Golioth so it may be used to authenticate your device. From the left sidebar choose Project Settings, select the Certificates tab, and click Add CA Certificate:

Upload public root certificate to Golioth

Generating PSK Credentials

You should not use PSK credentials for production purposes, certificates are far more secure. However, for demonstration purposes, here’s what you need to do to prepare your PSK:

  1. Use the Golioth console to add a new device.
  2. Copy the PSK-ID and PSK from your newly generated device.
  3. Use the following command to format your PSK as a HEX string—the format required by the nRF9160 (note this is only for the PSK, not for the PSK-ID).
echo "your-psk-here" | tr -d '\n' | xxd -ps -c 200

Loading Credentials onto the nRF9160

  1. Build and flash the Nordic at_client sample onto your device.
  2. Use the Nordic nRF Connect for Desktop tools to write credentials to the device.
    1. Older versions of this tool will use the LTE Link Monitor, newer versions will use the Cellular Monitor.
    2. Choose the security tag you wish to use.
    3. Add your credentials to the interface and use the Update certificates button to write to the device. I found that I wasn’t able to write all three certificate artifacts at the same time and instead needed to enter them one at a time.

Nordic Cellular Monitor used to store device credentials on nRF9160

The credentials are stored using AT commands over a serial connection. You do not need to use the Nordic desktop tools if you don’t care to as the commands can be sent from any serial terminal connection.

Configure Golioth to locate credentials by tag

Now that our credentials are stored on the device, it is a trivial process to adapt the Golioth client to use them. In the video, I stored my certificates at security tag 24. I simply pass this to my application by adding the following Kconfig symbols to the prj.conf file:

CONFIG_GOLIOTH_AUTH_METHOD_CERT=y
CONFIG_GOLIOTH_SAMPLE_AUTH_TYPE_TAG=y
CONFIG_GOLIOTH_COAP_CLIENT_CREDENTIALS_TAG=24

(If you are testing PSK authentication instead of certificates, do not use the Kconfig symbol that sets CERT as the auth method. Without it, PSK will be selected by default.)

You must also remove one symbol from the boards/nrf9160dk_nrf9160_ns.conf file. This symbol tells the application to skip the offloaded DTLS sockets, however, that’s exactly the feature we want when using tag-based credential selection. Remove this line to re-enable socket offloading:

# Remove this line from nrf9160dk_nrf9160_ns.conf
CONFIG_NET_SOCKETS_TLS_PRIORITY=35

With that in place, compile the application and flash it to your nRF9160. It will now use the stored credentials to authenticate with Golioth.

Understanding the Golioth client config

The Golioth samples are configured to use a common library that configures the Golioth client on your behalf. However, the configuration process is very simple as I demonstrated it in the video.

Create a config struct that chooses TLS tags. Use the struct to pass the security number where the credentials are stored. Here’s what that would look like for my credential authentication example:

/* Config for credential tag authentication */
struct golioth_client_config client_config = {
    .credentials =
        {
            .auth_type = GOLIOTH_TLS_AUTH_TYPE_TAG,
            .tag = 24,
        },
};

client = golioth_client_create(&client_config);

Device Credentials with Golioth

The new APIs present in the Golioth Firmware SDK make it really easy to select credentials using tags. We’d love to hear how you plan to utilize this functionality. Show off your project on the Golioth Forum, or hit us up at the DevRel email!

Zephyr has a lot of tricks up its sleeve and most recently I used it to enable power regulators on a custom Golioth board. Perhaps the most interesting part of this is that it can be done entirely with the configuration code, without needing to dive in to any of the C files. And as the icing on the cake, Zephyr even includes interactive shell for working with regulators!

If you’ve been following our blog for a while, or you’ve checked out our growing library of reference designs, you’ll know that we’ve been using an internal hardware platform codenamed “Aludel” for rapid prototyping. Chris Gammell has been busy working on a new version of the Aludel main board with the ability to shut off power to the mikroBUS Click board sockets for low-power operation.

For example, a GPIO from the onboard nRF9160 SIP is connected to the EN pin on the +5V boost regulator. This grants Zephyr firmware the power (ha!) to enable or disable the regulator:

When the first prototype boards arrived from the manufacturer, we needed to enable these power regulators as part of the hardware bring-up process. I was pleasantly surprised to discover that this is actually possible to do entirely with Devicetree & Kconfig without writing any C code!

Zephyr’s regulator bindings

Out of the box, Zephyr provides regulator-gpio and regulator-fixed devicetree bindings for variable and fixed-voltage regulators respectively. These bindings are used to define GPIO-controlled power regulators that can be enabled automatically when the OS boots.

Here’s the devicetree node we’re using for the +5V boost regulator on the Aludel board:

reg_mikrobus_5v: reg-mikrobus-5v {
    compatible = "regulator-fixed";
    regulator-name = "reg-mikrobus-5v";
    enable-gpios = <&gpio0 4 GPIO_ACTIVE_HIGH>;
    regulator-boot-on;
};

The compatible property matches this node to Zephyr’s regulator-fixed device tree binding, and the regulator-name property is just a descriptive name for the regulator output.

Next, the enable-gpios property is a phandle-array devicetree property which defines the GPIO connected to the regulator’s EN pin:

  • &gpio0 is the phandle for the GPIO0 controller node that controls the pin
  • 4 is the pin number on the controller
  • the pin is configured with the GPIO_ACTIVE_HIGH flag because the regulator EN pin is an active-high input

The regulator-boot-on property tells the regulator driver that the OS should turn on the regulator at boot (but shouldn’t prevent it from being turned off later). We just needed to set the following Kconfig symbol to enable the regulator drivers:

CONFIG_REGULATOR=y

Regulator shell

When you’re bringing up a new board, it’s really helpful to have full control over the power regulators from a diagnostic shell. Zephyr provides a regulator shell that allows you to interact with regulators on your board via the shell interface!

Set the following Kconfig symbols to enable the regulator shell:

CONFIG_SHELL=y
CONFIG_REGULATOR=y
CONFIG_REGULATOR_SHELL=y

With the regulator shell enabled, you can enable/disable the regulator. You can also get/set parameters like voltage, current limit, and operating modes.

Regulator shell example

First, we need to get the device node for the regulator. You can list the available devices and their states using the device list shell command:

uart:~$ device list
devices:
- nrf91_socket (READY)
- clock@5000 (READY)
- gpio@842500 (READY)
- psa-rng (READY)
- uart@9000 (READY)
- uart@8000 (READY)
- flash-controller@39000 (READY)
- i2c@a000 (READY)
- reg-mikrobus-5v (READY)
- reg-mikrobus-3v3 (READY)

To disable the reg-mikrobus-5v regulator, we can run the following shell command:

uart:~$ regulator disable reg-mikrobus-5v

To enable the reg-mikrobus-5v regulator again, we can run a similar shell command:

uart:~$ regulator enable reg-mikrobus-5v

Time to hit that play button on some Warren G!

🎵 “Regulators, mount up!” 🎵

Learn more about the Zephyr shell

Controlling power regulators is just one of the neat things you can do with the Zephyr shell. At Golioth we think the Zephyr shell is fantastic and we’ve written extensively about it on our blog. We also use it in the Golioth SDK for things like setting device credentials.

If you’d like to learn more about Zephyr in general, join us for a free Zephyr training. Our next session is coming up in February. Sign up now!

In the last blog post about the PPK2,we explained operating modes for embedded devices and how current consumption is generally measured. With Nordic’s Power Profiler Kit II, we were successful in reducing the current draw of Golioth’s hello sample by a factor of 10 by simply turning the modem off. But, truth be told, 4mA is still a significant current draw for battery-operated devices in Standby mode. The aim for today is to lower the current consumption even more.

General Power Saving Recommendations

Let’s discuss some general recommendations for lowering current consumption. Every device/product has a set of constraints. From the sensor reading period to the number of sensors connected to the MCU, the current consumption for your particular device may vary based on which function you are performing and how much you are asking of your device.

Disable Unused peripherals

Some peripherals are enabled by default, depending on the devicetree of the board you are using. Peripherals that are not in use should be disabled in the overlay file. Even though it might not seem obvious at first,  some peripherals could potentially draw current just by being enabled. And the result could be milliamps of unnecessary current draw.

Peripherals are enabled or disabled in the device overlay file. For the hello sample, we won’t use any of the peripherals, and we’ll disable them all in the nrf9160dk_nrf9160_ns.overlay file.

You only need to disable the peripherals which are not utilized in your project. You can use the Device Power Management module to put peripherals into sleep mode in inactivity periods; more about that later on.

Disable Serial Logging

By default, logging is performed over the serial port associated with the UART(E) peripheral with Nordic SoCs. If a device is expected to work without human interaction over the serial port, then there is no need for logging over serial output and having the associated current consumption. By doing this, we can reduce the current consumption by ~1mA.

  • Disable serial output with: CONFIG_SERIAL=n
  • Disable serial logging with: CONFIG_LOG=n
  • Disable the UART console with: CONFIG_UART_CONSOLE=n

For the purposes of the hello sample, we won’t disable the logging subsystem because we want the log messages to be sent to Golioth’s Logging device service, but we have disabled the uart0 peripheral, which is used as a serial console by default.

Enabling Device Power Management

The idea behind the Device Power Management module is to allow device drivers to handle power management operations. For instance, it turns off clocks and peripherals, lowering the current consumption. The Device PM module provides an interface that the device drivers use to be informed about entering the suspended state or resuming from the suspended state. This enables the application developer to suspend peripherals when the CPU goes to sleep, depending on the application’s behavior.

For example, we can turn of an SPI peripheral while it is not in use to reduce the current draw. A device driver must have an implementation of the PM action callback used by the PM subsystem to suspend or resume devices.

Since we won’t use any of the nRF9160 SoC peripherals in the hello sample, we won’t see any benefit from using the Device PM. Nordic’s documentation explains how to utilize the PM module on the external flash case.

Results

As in the previous blog post, we are going to connect to the cellular tower, Golioth Cloud, and send 5 hello messages; afterward, we’ll stop the Golioth system client and call the lte_lc_offline API function, which sets the device to flight mode, disabling both transmit and receive RF circuits and deactivating LTE and GNSS services.

 

From the picture above, we achieved an average current of 2.69 µA when all peripherals are disabled, the modem is turned off, and the CPU is in IDLE. It’s up to the application developer to decide when to enter this minimal operating state, and deal with the consequences when coming back online after this deep sleep. Things like ConnectionID can help cellular devices save power on handshakes with the tower when re-connecting.

Conclusion

We showed how disabling unused peripherals can benefit our current consumption bottom line and save milliamps in the process by achieving ~3uA current draw. In the next blog post, we’ll talk about Power Optimizations specific to the nRF9160 SoC and how to use Power Saving Mode (PSM) with the modem and eDRX method instead of turning the modem off completely.

Ozone is a free graphical debugger for embedded firmware from SEGGER. It’s a powerful tool that can give you deep visibility into what’s happening in your embedded system. It’s especially useful when debugging nRF9160 Zephyr apps. Sorting out multiple threads and multi image builds can be tough, but this is the tool you want.

In our previous post Taking the next step: Debugging with SEGGER Ozone and SystemView on Zephyr, Chris Gammell wrote about how to set up a SEGGER Ozone project to debug a Zephyr app running on the i.MX RT1060 Evaluation Kit. It’s a great general introduction to debugging a Zephyr app in Ozone and profiling the RTOS runtime behavior SystemView.

When I was trying to set up a similar Ozone project for debugging the Nordic nRF9160 SIP, I ran into a few snags along the way. Today, I’ll share what I’ve learned!

In this article, I’ll walk through how to:

  • Configure a nRF9160 Zephyr app for thread-aware debugging
  • Create an Ozone project for nRF9160 with the New Project Wizard
  • Modify the Ozone project to support debugging nRF9160 multi-image builds

Hardware Configuration

In the examples that follow, I’ll be using the Nordic nRF9160 DK board. This development kit from Nordic has a SEGGER J-Link OB debugger built into the board, so an external J-Link debugger is not required to follow-along with the examples.

 

Thread-awareness support in Zephyr

In a typical Zephyr app built using the Golioth Zephyr SDK, there will be multiple threads. For instance, one for the app’s main loop, one for the Golioth system client, and others for the UART shell, logging subsystem, network management, etc.

SEGGER provides a Zephyr RTOS plugin for Ozone that can show the status of each thread, but it requires that the Zephyr firmware is built with support for thread-aware debugging. Zephyr provides a CONFIG_DEBUG_THREAD_INFO Kconfig symbol that instructs the kernel to maintain a list of all threads and enables thread names to be visible in Ozone.

While you could simply add CONFIG_DEBUG_THREAD_INFO=y to your app’s prj.conf file, you probably only want to enable this extra debug info when you are building for debugging purposes. Instead, we can create an additional debug.conf Kconfig file that will only get merged in when we pass the -DEXTRA_CONF_FILE=debug.conf argument to the build system.

Since this article is about using Ozone for thread-aware debugging, we’ll use the zephyr/samples/basic/threads/ app from the nRF Connect SDK Zephyr repo as our example app in this article.

If this is your first time building one of the Zephyr sample apps, make sure to complete the nRF Connect SDK Installation Guide first to make sure your dev environment is set up correctly.

How to Enable Thread-Awareness

First, create a zephyr/samples/basic/threads/debug.conf file and add the following lines:

CONFIG_DEBUG_THREAD_INFO=y

# CONFIG_DEBUG_THREAD_INFO needs the heap memory pool to
# be defined for this app
CONFIG_HEAP_MEM_POOL_SIZE=256

Next, build the firmware, specifying the debug.conf file to be merged into the build configuration:

cd <your ncs workspace>/
west build -p -b nrf9160dk_nrf9160_ns zephyr/samples/basic/threads/ -- -DEXTRA_CONF_FILE="debug.conf"

If the build completed successfully, you’ll see the build/zephyr/zephyr.elf file we need to start a debugging session in Ozone.

Create the Ozone project

Now that we’ve built the firmware, you can launch Ozone and use the New Project Wizard to create an Ozone project:

Choose the nRF9160_xxAA device:

Select the J-Link device you want to use:

Select the build/zephyr/zephyr.elf ELF file we generated in the previous section:

Leave these fields the default values for now (we’ll update them later on):

After clicking “Finish”, you’ll see the Ozone project window appear.

In the “Console” window, run the following command to load the Zephyr RTOS plugin:

Project.SetOSPlugin("ZephyrPlugin.js");

You should now see a new “Zephyr” window in the Ozone project (if not, click on “View” → “Zephyr” to show the window):

Finally, save the project file by clicking on “File” → “Save Project as…”:

Start the debug session

Now that we’ve configured the Ozone project, we can start the debug session.

Click on “Debug” → “Start Debug Session” → “Download & Reset Program”:

Surprise! When the firmware starts to run, you’ll see a pop-up window indicating that the target has stopped in a HardFault exception state!

At this point, you might be wondering what’s going on here…

We’ve followed the same basic steps as outlined in our previous article, so why isn’t this working for the nRF9160?

Here’s a hint: the answer has to do with multi-image builds.

The missing step: flashing the merged image

You may have noticed that the board argument we passed to west build (nrf9160dk_nrf9160_ns) ends in _ns. This suffix is an indicator that the firmware will be built with Trusted Firmware-M (TF-M). This is the reference implementation of ARM’s IoT Security Framework called Platform Security Architecture (PSA).

TFM uses the ARM TrustZone security features of the nRF9160’s Cortex-M33 MCU to partition the MCU into a Secure Processing Environment (SPE) and Non-Secure Processing Environment (NSPE).

Here’s how the boot process works in a nutshell:

  1. When the MCU boots up, it starts executing in the secure environment (SPE).
  2. The boot process can optionally start with a secure bootloader chain using NSIB and/or MCUboot.
  3. If used, the bootloader starts TF-M, which configures a part of the MCU memory and peripherals to be non-secure.
  4. TF-M starts your Zephyr application which runs in the non-secure environment (NSPE).

When we build for _ns build targets, the TF-M image is automatically built and linked with the Zephyr app. If you look in the build/zephyr/ output directory, you’ll see a file named merged.hex, which is a single merged file containing the MCUboot bootloader (optional), the TF-M secure image, and the non-secure Zephyr app.

west flash knows to flash the full merged image, but Ozone doesn’t do this by default!

We need to configure Ozone to load the full merged image and start execution in the secure environment.

Fixing the Ozone project file

We’ll make a couple changes directly in the Ozone project file, which can be opened within Ozone by clicking “File” → “Edit Project File”:

Flash the merged image

Navigate to the TargetDownload section of the Ozone project file and add the following to configure Ozone to flash the merged image (changing the path to match the merged image file in your project):

/*********************************************************************
*
* TargetDownload
*
* Function description
* Replaces the default program download routine. Optional.
*
**********************************************************************
*/
void TargetDownload(void)
{
  Exec.Download("$(ProjectDir)/build/zephyr/merged.hex");
}

Fix the Vector Table & PC addresses

Navigate to the _SetupTarget section of the Ozone project file and make the following changes:

  1. Set the vector table address to 0
  2. Read the entry point program counter address from the vector table
/*********************************************************************
*
*       _SetupTarget
*
* Function description
*   Setup the target.
*   Called by AfterTargetReset() and AfterTargetDownload().
*
*   Auto-generated function. May be overridden by Ozone.
*
**********************************************************************
*/
void _SetupTarget(void) {
  unsigned int SP;
  unsigned int PC;
  unsigned int VectorTableAddr;

  VectorTableAddr = 0;
  //
  // Set up initial stack pointer
  //
  SP = Target.ReadU32(VectorTableAddr);
  if (SP != 0xFFFFFFFF) {
    Target.SetReg("SP", SP);
  }
  //
  // Set up entry point PC
  //
  PC = Target.ReadU32(VectorTableAddr + 4);
  if (PC != 0xFFFFFFFF) {
    Target.SetReg("PC", PC);
  } else {
    Util.Error("Project script error: failed to set up entry point PC", 1);
  }
}

When you save the project file, you should get a modal pop-up asking if you want to reload the project.

Choose “Yes”:

Restart the debug session:

After the image has been flashed to the device, you should see the debugger halted at main:

Click “Debug” → “Continue”:

The firmware should now run without exceptions!

Summary

Hopefully this helped you get started debugging the nRF9160 with Ozone.

The nRF9160 is fully supported in Zephyr, and has the highest level of support in the Golioth IoT device management platform (Continuously Verified). With Golioth, you can connect and secure your devices, send sensor data to the web, update firmware over-the-air, and scale your fleet with an instant IoT cloud.

Try it today—with Golioth’s Dev tier your first 50 devices are free!

Embedded developers always maintain sets of helper code that get used across multiple projects. With Zephyr RTOS, you can easily turn your helper code into a portable Zephyr module.

Creating a Zephyr module means you can version control your code, making changes in one centralized place, while targeting a specific git commit, tag, or branch of that code in a project. You only need to add two or three additional files to qualify as a Zephyr module. And once the module is published you can make the code available to your Zephyr projects simply by adding it to your manifest file.

A Working Example

Ostentus faceplate installed on Aludel-mini reference design

You may remember reading about Ostentus, Golioth’s custom faceplate for conference demos. This I2C display can be added to any Zephyr project by leveraging some helper code that simplifies sending data from the device to the faceplate. As this code will change in the future, it isn’t maintainable to copy it between projects, so we turned it into a Zephyr module. Let’s use this as an example. Here are the steps:

  1. Create a new repository to store your helper code
  2. Add a CMake file and an optional Kconfig file
  3. Add a module.yml file that tells Zephyr where to find files in the module repo

That creates the module, and the final step is to add it to your project’s West manifest.

1. Create a new repository

For our example, we have just one C file and one header file. I created a new repository to store these files. Place the header file in a directory named include; you may add subdirectories if you want there be a category-like prefix when the header is included (e.g. #include <yoursubdirectory/yourfile.h>).

Here are the important parts of my tree for this module:

➜ tree
.
├── include
│   └── libostentus.h
└── libostentus.c

2. Add CMake and Kconfig files

Next we need to add a CMake file to include our source code in the build. You also have the option of including a Kconfig file. I’m going to do this optional step because it allows me to create a symbol that will turn on or off this library in the project build.

First, I create the Kconfig file in the root of the repo that adds a unique symbol (LIB_OSTENTUS) for this library:

config LIB_OSTENTUS
    bool "Enable the helper library for the Golioth Ostentus faceplate"
    default n
    depends on I2C
    help
      Helper functions for controlling the Golioth Ostentus faceplate.
      Features include controlling LEDs, adding slides and slide data,
      enabling slideshows, etc.

Next, I add the C file and the header file to the build using a CMakeLists.txt file in the root directory of the repository. Note that this uses the Kconfig symbol created in the last step to decide whether or not to build with these files.

if (CONFIG_LIB_OSTENTUS)
  zephyr_include_directories(include)
  zephyr_library_sources(libostentus.c)
endif()

3. Add module.yml

The glue that holds this together is the module.yml file, which must be placed in a zephyr subdirectory of the repository. Here I tell it where to look for the CMake and Kconfig files, although there are other options that can be added to this file.

build:
  cmake: .
  kconfig: Kconfig

I find the relative paths of this file to be a bit confusing. But the gist of it is that this file will be located at zephyr/module.yml and the paths in the file are based on the parent of that zephyr subdirectory.

Congratulations, we now have a Zephyr module made up of the following files:

➜ tree
.
├── CMakeLists.txt
├── include
│   └── libostentus.h
├── Kconfig
├── libostentus.c
└── zephyr
    └── module.yml

Using the Module in a Zephyr Project

Using the newly created module follows the familiar Zephyr pattern of adding it to the projects section of your West manifest file. (You can find your manifest file by running west manifest --path.)

manifest:
  projects:
    - name: libostentus
      path: modules/lib/libostentus
      revision: v1.0.0
      url: https://github.com/golioth/libostentus

self:
  path: app

Calling west update will now checkout the helper code and place it in modules/lib/libostentus. Remember that I added a Kconfig symbol in my example, so I need to add that to the project prj.conf file or a <board>.conf file to include this code in the build.

CONFIG_LIB_OSTENTUS=y

Using the module in your C files is a simple matter of adding the #include and then calling the functions.

#include <libostentus.h>

int main(void)
{
    clear_memory();
    show_splash();

    k_sleep(K_FOREVER);
}

Summary

Golioth supports a wide variety of hardware and use cases, and part of our strategy to make that scalable is to write and maintain modular code. The Golioth Zephyr SDK is a Zephyr module, which makes it easy to include in your project and easy to lock to a known version until you’re ready to upgrade to a newer version. We’ve used this same Zephyr module approach for helper code when building our numerous reference designs. It works well and I encourage you to adopt modular practices for your own work.

This process of creating a Zephyr module is a nice improvement over copying and pasting code between projects because it implements reliable revision control. It is also an intermediary step between implementing a project-specific driver, and creating a modular Zephyr driver. The Ostentus faceplate should eventually be converted to a full Zephyr driver, and enabled directly from Devicetree. But that’s a post for another day!

In the meantime, if you find yourself in need of a device management service with robust OTA, data handling, fleet settings, RPC, and more, give us a try. With Golioth’s Dev tier your first 50 devices are free!

Troubleshooting IoT Cellular

Cellular connected IoT can be intimidating, especially when the cellular connection doesn’t work, or only works intermittently. Today we will explore Nordic’s LTE Link Monitor and Cellular Monitor applications to show how you can troubleshoot cellular connection using the nRF9160 DK as the development board.

Programming the modem firmware

The modem allows applications to send and receive data on the cellular network, and the application talks to the modem via the AT commands. Every nRF9160 SiP has modem firmware (separate from your application code) that is provided as a pre-compiled binary file, signed and encrypted by Nordic Semiconductor.

It’s always beneficial to update the modem firmware to the latest version. In fact, this is the first thing you should try. Make sure you have the nRF Connect for Desktop tool installed and follow these steps to do so:

  1.  Download the latest modem firmware (at the moment of writing, it’s v1.3.5),
  2. Open the Programmer Application from nRF Connect for Desktop
  3. Connect the nRF9160 DK to your computer and select it in the Programmer Application
  4. Click the Add file button, add the mfw_nrf9160_1.3.5.zip file
  5. Flash modem firmware with the Write button

LTE Link Monitor

If updating the firmware didn’t work, it’s time to look at the status of the cellular connection. LTE Link Monitor is part of the nRF Connect for Desktop tool. I works alongside a modem client application that monitors the modem status and activity using AT commands.

For this demo, we’ll use Nordic’s at_client sample for the nRF9160 DK, which can be found at nrf/samples/nrf9160/at_client (Nordic changed folder names over the summer so this may be in samples/cellular/at_client for you).

The AT Client sample demonstrates the asynchronous serial communication taking place over UART to the nRF9160 modem, enabling you to use an external computer to send AT commands to the LTE-M/NB-IoT modem. This facilitates the reading of responses or analyzing of events related to the nRF9160 modem.

Note that not all commands are supported in all modem firmware versions; that is why the first step was to update the modem firmware to the latest version.

Switch to NB-IoT standard with AT commands

In our recent blog post, we showed how to switch the preferred cellular connection standard from LTE-M to NB-IoT with configuration files. Now, we are going to do the same thing with AT commands and later show how the two are connected with Zephyr’s Kconfig Configuration System.

To switch from the default LTE-M connection standard to NB-IoT, do the following:

  • Flash the at_client example built for the nRF9160 DK,
  • open the LTE Link Monitor Application and connect to the development board,
  • check the modem system mode with AT%XSYSTEMMODE? command.

XSYSTEMMODE AT command is used for enabling system modes, and the command response syntax is:
%XSYSTEMMODE: <LTE_M_support>,<NB_IoT_support>,<GNSS_support>,<LTE_preference>.

The response we got was: 1,0,1,0, which means our device is set for LTE-M and GNSS support.
Now, let’s change it to NB-IoT.

  • Send AT%XSYSTEMMODE=0,1,0,0,
  • Send AT+CFUN=1 to set the device to full functionality
  • Send AT+CFUN?

After sending the AT+CFUN? command, the nRF9160 DK has connected to the cellular network T-Hrvatski Telekom using the NB-IoT standard and has obtained an IP address 178.160.19.177.


With the at_client sample, you can test and manually send all AT commands to the modem.

Cellular Monitor

Cellular Monitor is another nRF Connect for Desktop application used for capturing and analyzing modem traces to evaluate communication and view network parameters. Let’s use it with Golioth’s hello sample, which can be found at https://github.com/golioth/golioth-zephyr-sdk/tree/main/samples/hellogolioth/samples/hello.

But before we start, we need to add CONFIG_NRF_MODEM_LIB_TRACE=y to prj.conf file in the sample directory, PSK and PSK-ID as stated in the README file, build the sample, and flash the nRF9160 DK.

After that, open the Cellular Monitor Application, click the Start button in the upper left corner to capture a trace and reset the nRF9160 DK. When capturing a trace, the data is saved to the .mtrace binary file so you can view previously collected traces in the Cellular Monitor Application.

The trace data is categorized into the following 6 dashboard panels:

  • LTE Network
  • Device
  • Power Saving Mode
  • SIM
  • Connectivity Statistics
  • PDN (Packet Data Network)

The Packet Event Viewer visualizes communication at the AT command, Radio Resource Control (RRC), Non-access Stratum (NAS), and Internet Protocol (IP) levels.

For this example, I have enabled the CONFIG_LTE_NETWORK_MODE_NBIOT symbol in the configuration file, and we can see in the Packet Event Viewer the XSYSTEMMODE AT command is sent to the modem automatically during the configuration.

Conclusion

In this demo, we have shown how to start troubleshooting Cellular Network problems with LTE Link Monitor, how to use the Cellular Monitor Application, and how the modem can be configured with Zephyr’s Kconfig Configuration System. You should consider using these tools to observe your modem configuration when it is correctly functioning. This provides much of the intuition you will need when you are called upon to troubleshoot a misbehaving cellular modem!

Want to see what happens when your device is sending and receiving data over the network? Give Golioth a try, our Zephyr SDK has a number of ready-to-use samples and your first 50 devices are free on our Dev Tier.

We recently open-sourced the Golioth Reference Design Template that we have been using internally as the starting point for our growing library of reference designs. Out of the box, the template provides an end-to-end working firmware example showcasing all of Golioth’s key features. You can read more about what’s included in the Reference Design Template in the announcement post Open Sourcing Golioth Reference Designs and Template.

The Reference Design Template firmware currently supports two boards: the Nordic nRF9160-DK and the Golioth Aludel Mini. The Aludel Mini is an internal prototyping platform designed by Golioth for rapidly building and testing proof-of-concept ideas using widely available off-the-shelf modules. The Aludel Mini integrates a nice ePaper display (Ostentus) into the enclosure, and we have been using it to display custom sensor readings specific to each reference design—for example, on the DC Power Monitor it displays current & voltage, and on the Air Quality Monitor it displays particulate matter, CO₂, and weather data.

As I was working on some of the Golioth reference designs, I found myself occasionally checking the Golioth Console to confirm the firmware version was correct (for example after an OTA update) or to check the remaining battery level. The Golioth Console makes it easy to monitor values like this once the device has connected to the network, but it would be helpful to display this info immediately on the ePaper display when the device boots up (and without having to connect to the device’s UART console).

Luckily, this turned out to be a really easy addition!

Displaying the firmware version

The Golioth Reference Design Template is built with support for the MCUboot bootloader. The firmware version is defined and later made available to the Zephyr application firmware via the KConfig symbol CONFIG_MCUBOOT_IMAGE_VERSION.

When the app boots up, the firmware logs the current firmware version:

LOG_INF("Firmware version: %s", CONFIG_MCUBOOT_IMAGE_VERSION);

Now, it also displays a “Firmware” slide that shows the currently running firmware version on the Ostentus ePaper display:

Displaying the battery voltage

The Aludel Mini board uses a SparkFun nRF9160 Thing Plus module that has the ability to run off a Li-Poly battery. The battery voltage can be sampled using the ADC on the nRF9160 using the onboard voltage-divider circuit:

We can enable support for this voltage-divider circuit in Zephyr by adding a voltage-divider-compatible node to the board’s device tree definition. This specifies the ADC channel to use, the values of the resistors in the voltage divider circuit, and which GPIO is used to enable power to the divider:

/ {
  vbatt {
    compatible = "voltage-divider";
    io-channels = <&adc 7>;
    output-ohms = <100000>;
    full-ohms = <(100000 + 100000)>;
    power-gpios = <&gpio0 25 0>;
  };
};

The Zephyr tree provides an example of how to read this voltage divider circuit in zephyr/samples/boards/nrf/battery. I was able to use this sample as a starting point for quickly adding some simple battery monitoring to the Reference Design Template.

Now, whenever the template firmware is built with our custom  CONFIG_ALUDEL_BATTERY_MONITOR Kconfig symbol enabled, the firmware displays two slides: one that shows the current battery voltage and another that displays the estimated remaining battery level.

Golioth Reference Designs

Firmware version and battery status display slides are now included in all the Golioth reference designs we’ve published. You can check out the full set of reference designs on the Project page at https://projects.golioth.io. And as always, we’d love to hear about what you’re building. Show off your projects and ask questions over on the Golioth Forum.

If you take one thing away from this [talk], it should be this: Manifest files are a great way to manage revision control in your Zephyr applications.

Mike Szczys is my teammate in the Developer Relations group and the primary firmware engineer creating Golioth Reference Designs. We build on top of the Golioth Zephyr SDK to create showcase examples of how you can use Golioth firmware and cloud capabilities to create real-world applications quickly.

To keep everything straight, we rely heavily on Zephyr manifest files to ensure we are always building the correct code each time. We include things like the Zephyr codebase, the Golioth Zephyr SDK codebase, the Nordic Connect SDK codebase, and any custom code we write, all of which might be at different points in their lifecycles. It’s not a stretch to say that it’d be nearly impossible to manage all the code we use on a daily basis without manifest files. Mike used the knowledge he has built to give the above talk at the Zephyr Developer Summit 2023, which was part of the larger Embedded Open Source Summit in Prague in June of this year.

What is a manifest?

Zephyr utilizes a manifest file, which is part of the west meta-tool. These are a tough topic at first because they have multiple levels of inheritance possible. If you’re using a vendor-provided manifest file (or using one of the vendor SDK tools that hides the complexity), you likely will have an easy time following along their path. But once you want to start customizing your own project, you need to dig in and learn how they work.

A visualization of the west meta tool, showing how confusing things can get (image from Zephyr West page)

Mike points out that at their core, manifest files manage hierarchy and provide a revision-controlled record of what should be included in a project. You can directly call out which version of a piece of code you want to include. As the subsystems you utilize in your project continue to upgrade over time, you can choose to lock to the older version. More importantly, you can make a concerted effort to change the version and then test the upgrade has not broken your build or introduced unknown behaviors in your program.

Structuring your Zephyr application

While we normally avoid being overly prescriptive on this blog, we have a few opinions about how to find success with Zephyr applications. Some of this comes from seeing Zephyr projects go wrong in the past. Some examples of this are:

  • Cloning the Zephyr tree and putting it into your project repository
  • Placing your code among Zephyr samples
  • Having a standalone application repository (good) that references the Zephyr tree somewhere else on your machine (bad, because you don’t have a guarantee that the tree isn’t being used by some other Zephyr repo)

The Right Way™ (at least how we see it!) is to put a manifest file in your application and then specifically call out the versions of all dependencies. Asgeir Stavik Hustad guest-wrote about this method on the Golioth blog last year and we have taken those ideas and extended them even further (and changed a few things). The downside is that you will have many independent copies of the Zephyr tree on your hard drive…but hard drive space is cheap and making mistakes in a repo is very expensive. 

Examples and more!

I could continue to summarize the talk here, but it’s best to go and watch Mike’s examples as part of the talk. If you’re not a video person, feel free to scroll through the slides, embedded below. We love talking about Zephyr and would love to hear about your challenges and successes using manifests over on our forum.

https://www.youtube.com/watch?v=PVhu5rg_SGY