Yesterday I was upgrading a Golioth Reference Design to the newest version of the Golioth Firmware SDK and I encountered a network error I had never seen before. I was able to track it down fairly quickly using the debugging tools built into Zephyr. This process is quite handy, so today I’ll walk through how to debug a network error in Zephyr using GDB to help others hone their embedded debugging skills.

Encountering an Error

There are two errors shown below. The first is expected: the cell modem is not yet connected to the network so sending data will fail. But soon after the connection is established there is a second error highlighted below.

*** Booting nRF Connect SDK v2.5.2 ***
[00:00:00.465,942] <inf> fs_nvs: 2 Sectors of 4096 bytes
[00:00:00.465,942] <inf> fs_nvs: alloc wra: 0, fb8
[00:00:00.465,972] <inf> fs_nvs: data wra: 0, 68
[00:00:00.466,308] <dbg> golioth_powermonitor: main: Start Power Monitor Reference Design
[00:00:00.466,339] <inf> golioth_powermonitor: Firmware version: 1.2.0
[00:00:00.472,991] <inf> golioth_powermonitor: Modem firmware version: mfw_nrf9160_1.3.1
[00:00:00.474,456] <inf> golioth_powermonitor: Connecting to LTE, this may take some time...
[00:00:00.531,127] <inf> app_sensors: Device: ina260@40, 4.980000 V, 0.335000 A, 1.659999 W
[00:00:00.531,219] <inf> app_sensors: Device: ina260@41, 5.117499 V, 0.000000 A, 0.000000 W
[00:00:00.531,250] <dbg> app_sensors: app_sensors_read_and_stream: Ontime:      (ch0): 1        (ch1): 0
[00:00:00.531,463] <err> app_sensors: Failed to send sensor data to Golioth: 5
[00:00:02.397,918] <inf> lte_monitor: Network: Searching
[00:00:03.772,033] <inf> lte_monitor: Network: Registered (roaming)
[00:00:03.772,521] <inf> golioth_mbox: Mbox created, bufsize: 1232, num_items: 10, item_size: 112
[00:00:03.773,223] <inf> golioth_fw_update: Current firmware version: main - 1.2.0
[00:00:06.010,528] <inf> golioth_coap_client_zephyr: Golioth CoAP client connected
[00:00:06.010,803] <inf> golioth_powermonitor: Golioth client connected
[00:00:06.010,833] <inf> golioth_coap_client_zephyr: Entering CoAP I/O loop
[00:00:06.421,752] <dbg> app_state: async_handler: State successfully set
[00:00:06.443,542] <err> net_coap: 16 is > sizeof(coap_option->value)(12)!
[00:00:06.443,572] <dbg> app_state: app_state_desired_handler: desired
                                    66 61 6c 73 65                                   |false
[00:00:06.533,081] <inf> app_sensors: Device: ina260@40, 4.982499 V, 0.331250 A, 1.649999 W
[00:00:06.533,203] <inf> app_sensors: Device: ina260@41, 5.115000 V, 0.001250 A, 0.000000 W
[00:00:06.533,233] <dbg> app_sensors: app_sensors_read_and_stream: Ontime:      (ch0): 6003     (ch1): 1
[00:00:06.536,865] <inf> app_settings: Set loop delay to 10 seconds
[00:00:06.538,330] <inf> app_sensors: Device: ina260@40, 4.980000 V, 0.335000 A, 1.679999 W
[00:00:06.538,482] <inf> app_sensors: Device: ina260@41, 5.113749 V, 0.001250 A, 0.000000 W
[00:00:06.538,513] <dbg> app_sensors: app_sensors_read_and_stream: Ontime:      (ch0): 6008     (ch1): 6
[00:00:06.942,932] <dbg> app_settings: on_loop_delay_setting: Received LOOP_DELAY_S already matches local value.
[00:00:06.946,136] <dbg> app_settings: on_loop_delay_setting: Received LOOP_DELAY_S already matches local value.
[00:00:06.949,096] <dbg> app_settings: on_loop_delay_setting: Received LOOP_DELAY_S already matches local value.
[00:00:06.992,187] <inf> golioth_rpc: RPC observation established
[00:00:06.993,041] <inf> golioth_fw_update: Waiting to receive OTA manifest
[00:00:07.365,142] <dbg> app_sensors: get_cumulative_handler: Decoded: ch0: 1579017, ch1: 790405
[00:00:07.365,722] <dbg> app_state: async_handler: State successfully set
[00:00:07.367,553] <dbg> app_sensors: get_cumulative_handler: Decoded: ch0: 1579017, ch1: 790405
[00:00:07.368,133] <dbg> app_state: async_handler: State successfully set
[00:00:07.765,747] <inf> golioth_fw_update: Received OTA manifest
[00:00:07.765,777] <inf> golioth_fw_update: Manifest does not contain different firmware version. Nothing to do.
[00:00:07.765,808] <inf> golioth_fw_update: Waiting to receive OTA manifest

Hmmm, I wonder where this error came from?

<err> net_coap: 16 is > sizeof(coap_option->value)(12)!

I’ve never seen an error of this type before. Zephyr logging puts a logging tag at the beginning of each message and net_coap isn’t one that comes to mind. I started troubleshooting by “grepping”, or searching all files in a directory tree, for this message.

➜ rg net_coap
1903:void net_coap_init(void)

Seven different files were returned by rg (that’s ripgrep, which is just a different flavor of grep), but the first one is obviously what we want. You can see the exact net_coap name registered as the logging module tag.

Looking inside that file, I searched for the error message. I only searched for the > sizeof part of the error message since the rest is likely being added to the log using string substitution.

if (option) {
     * Make sure the option data will fit into the value field of
     * coap_option.
     * NOTE: To expand the size of the value field set:
    if (len > sizeof(option->value)) {
        NET_ERR("%u is > sizeof(coap_option->value)(%zu)!",
            len, sizeof(option->value));
        return -EINVAL;

Now we’re getting somewhere. The Zephyr contributor who worked on this code was even kind enough to leave comments on how to fix the error. However, I want to know what is causing the issue in the first place since I’m unfamiliar with this failure. Let’s use the debugger!

Using GDB to Debug Zephyr

Many boards that work with Zephyr have debugging support built right into the ecosystem. From the same directory where the build was run, I can run west attach to start GBD.

In GDB, first type mon reset to prepare the device to start from the beginning of the program. I know from the excerpt above that the error message is printed out from line 590 in the coap.c file. We can use the command b coap.c:590 to set a breakpoint, and start running using c for continue.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/mike/golioth-compile/reference-design-dc-power-monitor/app/build/zephyr/zephyr.elf...
Remote debugging using :2331
arch_cpu_idle () at /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/zephyr/arch/arm/core/aarch32/cpu_idle.S:143
143             cpsie   i
(gdb) mon reset
Resetting target
(gdb) b coap.c:590
Breakpoint 1 at 0x2139e: file /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/zephyr/subsys/net/lib/coap/coap.c, line 590.
(gdb) c

Breakpoint 1, parse_option (data=0x20013bd1 <rx_buffer> "hE\212\361\206)\035渠.>a\002R.d\adesired\r\003reset_cumulative\377false", offset=<optimized out>, pos=pos@entry=0x2001a54c <golioth_thread_stacks+5708>, max_len=<optimized out>, opt_delta=opt_delta@entry=0x2001a54e <golioth_thread_stacks+5710>, opt_len=opt_len@entry=0x2001a54a <golioth_thread_stacks+5706>, option=option@entry=0x2001a578 <golioth_thread_stacks+5752>) at /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/zephyr/subsys/net/lib/coap/coap.c:590
590                             NET_ERR("%u is > sizeof(coap_option->value)(%zu)!",

Great, we stopped where the error message is printed. At this point I want to know what my program was doing leading up to this moment. For this we can view the backtrace by typing bt.

(gdb) bt
#0  parse_option (data=0x20013bd1 <rx_buffer> "hE\212\361\206)\035渠.>a\002R.d\adesired\r\003reset_cumulative\377false", offset=<optimized out>,
    pos=pos@entry=0x2001a54c <golioth_thread_stacks+5708>, max_len=<optimized out>, opt_delta=opt_delta@entry=0x2001a54e <golioth_thread_stacks+5710>,
    opt_len=opt_len@entry=0x2001a54a <golioth_thread_stacks+5706>, option=option@entry=0x2001a578 <golioth_thread_stacks+5752>)
    at /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/zephyr/subsys/net/lib/coap/coap.c:590
#1  0x0004f8bc in coap_find_options (cpkt=cpkt@entry=0x20020950, code=code@entry=23, options=options@entry=0x2001a578 <golioth_thread_stacks+5752>, veclen=veclen@entry=1)
    at /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/zephyr/subsys/net/lib/coap/coap.c:907
#2  0x0004fa30 in coap_get_option_int (cpkt=cpkt@entry=0x20020950, code=code@entry=23)
    at /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/zephyr/subsys/net/lib/coap/coap.c:1282
#3  0x00031308 in golioth_coap_req_reply_handler (req=req@entry=0x20021558, response=response@entry=0x20020950)
    at /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/modules/lib/golioth-firmware-sdk/src/zephyr_coap_req.c:180
#4  0x00055e38 in golioth_coap_req_process_rx (client=client@entry=0x20020518, rx=rx@entry=0x20020950)
    at /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/modules/lib/golioth-firmware-sdk/src/zephyr_coap_req.c:362
#5  0x000326be in golioth_process_rx_data (len=<optimized out>, data=<optimized out>, client=0x20020518)
    at /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/modules/lib/golioth-firmware-sdk/src/coap_client_zephyr.c:866
#6  golioth_process_rx (client=0x20020518) at /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/modules/lib/golioth-firmware-sdk/src/coap_client_zephyr.c:949
#7  golioth_coap_client_thread (arg=0x20020518) at /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/modules/lib/golioth-firmware-sdk/src/coap_client_zephyr.c:1092
#8  0x0004d8a8 in z_thread_entry (entry=0x566ad <golioth_thread_main>, p1=<optimized out>, p2=<optimized out>, p3=<optimized out>)
    at /home/mike/golioth-compile/reference-design-dc-power-monitor/deps/zephyr/lib/os/thread_entry.c:48
#9  0xaaaaaaaa in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

The backtrack places the most recent function call at the top in position #0. Looking down the list I can see that starting on line #3 the Golioth SDK is calling a Zephyr CoAP function. Walking back through those function calls I established that we received a CoAP packet and are trying to decode the options stored in that packet.

I don’t really need to know how all of that packet handling is done… what is more important to me is to see the packet itself to help illuminate why there’s an option in it that is too big for the configured space. Luckily, GDB lets us look at what’s stored in memory.

Using GDB to Inspect Data in Memory

If we look at the coap.c sourcecode, we find the breakpoint we set is inside the of the parse_option function.

static int parse_option(uint8_t *data, uint16_t offset, uint16_t *pos,
            uint16_t max_len, uint16_t *opt_delta, uint16_t *opt_len,
            struct coap_option *option)

This has a data array as a parameter that likely has our coap packet in it. We can print this out to see the data. It’s as simple as p data, with data being the name of the variable.

(gdb) p data
$1 = (uint8_t *) 0x20013bd1 <rx_buffer> "hE\212\361\206)\035渠.>a\002R.d\adesired\r\003reset_cumulative\377false"

(Note: yes, That 渠 is what GDB actually outputs. Binary data sometimes has weird consequences, especially when there are unicode characters for symbols that match)

We’re getting somewhere, but this is not all that useful since it was printed as ASCII values instead of showing the actual hexadecimal data. Let’s print that out.

(gdb) p/x data@max_len
value has been optimized out

The p/x data@max_len command tells GDB to print hexidecimal data from an array called data and to use the max_len variable to determine how many bytes to print. But it looks like we’re stymied by the optimization of the program.

The max_len of the data array has already been optimized out and is unavailable to us. The next thing to do is to print out an arbitrary number of bytes by guessing at the length of the data array. Since we were already able to print it I’m guess it’s about 64 bytes and then using the ASCII values of the final parts of that string to figure out where the data actually ends:

(gdb) x/64xb data
0x20013bd1 <rx_buffer>: 0x68    0x45    0x8a    0xf1    0x86    0x29    0x1d    0xe6
0x20013bd9 <rx_buffer+8>:       0xb8    0xa0    0x2e    0x3e    0x61    0x02    0x52    0x2e
0x20013be1 <rx_buffer+16>:      0x64    0x07    0x64    0x65    0x73    0x69    0x72    0x65
0x20013be9 <rx_buffer+24>:      0x64    0x0d    0x03    0x72    0x65    0x73    0x65    0x74
0x20013bf1 <rx_buffer+32>:      0x5f    0x63    0x75    0x6d    0x75    0x6c    0x61    0x74
0x20013bf9 <rx_buffer+40>:      0x69    0x76    0x65    0xff    0x66    0x61    0x6c    0x73
0x20013c01 <rx_buffer+48>:      0x65    0x00    0x00    0x00    0x00    0x00    0x00    0x00
0x20013c09 <rx_buffer+56>:      0x00    0x00    0x00    0x00    0x00    0x00    0x00    0x00

The x/64xb data command prints out exactly what we’re after. In GDB the x command prints out memory contents (I always remember this as “examine”). The slash (/) adds the additional commands to print 64 hexidecimal (x) bytes (b) starting from the pointer address named data.

Decoding the CoAP Packet

After just a bit of cleanup, I have the data I’m after but it’s certainly not human readable. I like to use a site called Koap Online CoAP Decoder to take care of this for me:

When we go back to the original error message, the option that is too long is 16 characters. From the decoding above we see that the third option is a path called reset_cumulative that is 16 characters long. This is too long for the 12 character buffer we have configured in the Zephyr CoAP library!

I did this to myself! The application I’m working on is observing a Golioth LightDB State path and I chose a long name:

I followed the advice from the code comments in the Zephyr file and that fixed things right up.

# Adjust coap setting for a long (16-char) LightDB State sub-path

Make the Debugger Your Go-To

The worst part about using a debugger is usually setting things up. But in many cases, that work has already been done for you in the Zephyr ecosystem. Try out these skills the next time an unfamiliar error pops up in your embedded development work!

Manufacturing is marathon, not a sprint. Zephyr RTOS includes numerous features to help you at every step along the way, from initial prototype, to maintaining your hardware fleet in the field. Golioth’s Developer Releations lead, Chris Gammell, spoke on this topic at the 2024 Embedded Open Source Summit.

Chris’ approach boils down to breaking manufacturing into five distinct phases:

Golioth - going to production with hardware

  1. Early prototype
  2. Custom hardware
  3. First device in production
  4. Scaling production
  5. Maintaining a scaled fleet

The challenges of each phase exist whether or not you’re using Zephyr. But this RTOS has good tools you should utilize to smooth out many wrinkles. Let’s walk through each phase to see what is involved. The full set of talk slides is available at the bottom of this post.

Early Prototyping on Dev Boards

Chris always starts his prototyping out with commercially available development boards when possible. This means the hardware is in a known working state. Even if you haven’t finalized all of your hardware choices, Zephyr offers great portability so you can relatively easily change to a different part without the need to scrap your early work.

Zephyr also offers a number of tools for early tinkering. The menuconfig system is excellent to explore the configuration options available for the peripherals you have chosen. And the Zephyr shell is fantastic when validating new parts. For instance, the sensor and i2c shells facilitate live interaction with your sensors before getting down to the business of writing C code. Read about Golioth community member Timon switching over to Zephyr for prototyping.

Custom HW, First Pilot

As you move into your first pilot, this will likely be the first time you stand up custom hardware to ensure the system design works. Take time here to validate all of the parts in the design. Now is when you should be looking to see you have the feature coverage necessary to meet your needs. Confirm that the parts you have on the board are all needed, and ditch the ones that aren’t.

This is also a great time to begin planning for how you will test and provision each device. What kind of test points do you need? Ensure you’ve correctly routed the programming header and test placement for quick work during manufacturing.

Zephyr’s debugging features come into play during here. Consider the best setup for Zephyr’s logging system, whether that’s just turning it on and off, or changing up backends like the Golioth logging backend that sends logs to the cloud. Give thread-aware debugging suites like Ozone and Systemview a try before you need them. You’ll get a ton of insight to how your system is performing before a showstopper forces you to!

First Devices in Production

Pick a number, maybe that’s 100, of devices to join your first manufacturing run. This will be the first glimpse you have into some of the issues that will surface when you scale your production.

At this point, Chris likes to reach for the Zephyr board definitions and makes use of the support for board revisions. When peculiar behavior happens, the ease of compiling the same code for two different board revisions will help you discover if it’s something that’s always been there, or just arrived at the party.

commands to build firmware for different revisions of a board

Now is the time to set up your hardware-in-the-loop testing. You need to move fast and manual testing is the opposite of that. It’s not too late to adapt hardware for automated testing and you’ll thank yourself later. Once you have a programming and serial interface to the boards, Zephyr will swoop in with Twister and pytest that can be run on every PR and merge to catch problems early and run cycle tests far more frequently than you would otherwise.

Finally, don’t forget to plan for how you will perform firmware updates. Sure, you can plug USB cables into the 100 units you have in front of you, but that’s going to get really old when you do two patch releases in the same week. Don’t wait until you start to scale, set up your OTA updates now so you can begin testing automatic updates. With Golioth, you can do OTA from day one!

Scaling Production

This is it, time to turn the process up to 11 and start churning out boards. Smart decisions now will have a huge impact on your bottom line, so firm up those decisions on whether or not you need the top chip version in the family or can take it down a notch or two.

Consider how your choices affect cost after production. For instance, power budget is often a very large consideration. Zephyr includes a Power Management that should be used for battery-operated devices.

Network bandwidth is more directly related to monetary cost; optimizing your data usage leads to lower cellular and data usage bills. Even small savings scale! Consider configuring log levels to disable debug and info messages during normal operation. The Golioth settings service or an RPC can be used to remotely configure this. The same is true for what data is being streamed back to the servers and how frequently. We also recommend implementing a reboot RPC as a simple version of the “have you tried turning it off and back on again?” adage.

Chris touches on the topic of developing test stands for use during manufacturing. These interface with your hardware, and may work in conjunction with custom Zephyr shell commands to control the device during tests.

Maintaining a Scaled Fleet

You haven’t really crossed the finish line until your deployed devices reach their usable lifetime in the field. This means maintenance as myriad different operating conditions are sure to turn up unknown behavior.

If you followed Chris’ guidance in previous steps, your OTA update system is already in place and can be utilized to push out updates to address problems. Be sure to take advantage of simple things like Zephyr’s watchdog subsystem for automatic reboot when all else fails. But ultimately you want to fix the problems in place, so leveraging core dumps, and perhaps pushing fixes outside of full updates using the LLEXT feature in Zephyr is worth a look.


Give Chris’ talk a shot. There’s a ton of useful information there, whether this is your first rodeo or you’ve been rolling boards off of the production line since Chris was still in diapers. Manufacturing is defined by change. Embrace that concept and you’ll never be left behind.

Slides are below, the video is embedded at the top of the post.

How fast can a new Zephyr user go from zero to successful code compilation?

Golioth continues to invest in Zephyr RTOS capabilities: we have a super strong device SDK that works right on top of the Zephyr SDK, our Reference Designs are built on top of Zephyr, and we continue to train engineers every month on how to get booted quickly with Zephyr. When a fresh face walks through the door of our virtual training, we want to ensure a good experience for training.

In this talk at the Embedded Open Source Summit 2024, Firmware Engineer Mike Szczys describes how we utilize Dev Containers and GitHub Codespaces to provide a pre-installed Zephyr toolchain so that users have a seamless getting-started experience.

An evolution of training

Golioth has held a number of virtual training events using Kasm, something we have written about in the past (and gave a talk about at last year’s EOSS). We liked this method because it provided a pre-installed toolchain in a virtual environment. But it also included a “virtual desktop” environment and was built on top of generic “always on” cloud compute. Any time we did the training we would spin up new EC2 instances to serve users.

The problem is that the utilization was often low and we didn’t keep these resources available. Moreover, the users couldn’t take the work with them: when the session was over, their work was destroyed.

GitHub Codespaces fixes this up front: it’s a configuration script and setup that can deploy a full development environment onto a user’s own GitHub account. If someone wants to continue their work later in the day/week/month, they boot the Codespace back up and pick up where they left off. It’s also possible to utilize the setup to extend to their own projects, or implement interesting remote testing. And you can take it with you, because it’s build on the open standard of development containers.

Development Containers (or just: Dev Containers)

Utilizing Dev Containers only takes a couple of lines of configuration. However, understanding the pieces can be tricky, especially if you’re new to containers generally. Let’s peek at the main .devcontainer file:

  "image": "golioth/golioth-zephyr-base:0.16.3-SDK-v0",
  "workspaceMount": "source=${localWorkspaceFolder},target=/zephyr-training/app,type=bind",
  "workspaceFolder": "/zephyr-training",
  "onCreateCommand": "bash -i /zephyr-training/app/.devcontainer/",
  "remoteEnv": { "LC_ALL": "C" },
  "customizations": {
    "vscode": {
      "settings": {
        "cmake.configureOnOpen": false,
        "cmake.showOptionsMovedNotification": false,
        "C_Cpp.default.compilerPath": "/opt/toolchains/zephyr-sdk-0.16.3/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc",
        "C_Cpp.default.compileCommands": "/zephyr-training/app/build/compile_commands.json",
        "git.autofetch": false
      "extensions": [

The top 3 lines of the .devcontainer file we use for training includes 3 critical elements:

  1.  A base level image with Zephyr pre-installed on top of a minimal Linux environment
  2. Where to mount the local workspace folder inside this container
  3. The name of the workspace folder and where to drop the user when they boot a container

A critical element having the toolchain pre-installed on a Debian image. The nice part is that you can utilize others’ containers, including ours! If you want to customize containers for your own builds, you might need to learn some new items.

Other customizations allow you to set up how you’d like VS Code to look and which extensions you’d like to have available on boot.

Extending with

You can add other capabilities once the box has been initiated and is building. This includes pulling in repository information and setting up .bashrc elements that you might find useful from the command line.


west init -l app
west update
west zephyr-export
pip install -r deps/zephyr/scripts/requirements.txt
echo "alias ll='ls -lah'" >> $HOME/.bashrc
west completion bash > $HOME/west-completion.bash
echo 'source $HOME/west-completion.bash' >> $HOME/.bashrc
history -c

It’s key that the west environment was set up in the initial container, because this now utilizes those tools just like you would when setting up a local Zephyr environment on your computer using our docs.

Binary Downloads As The Main Output

If there’s one argument against this method, it’s that the output of the build is not directly loaded to your development board. We’re working on some capabilities to extend that (see the video for more details), but right now you need to download the binary (zephyr.hex) and load it to your board using a programming tool like nRF Connect For Desktop. That makes this method less optimal for rapid fire debugging. However, it’s possible to debug your program using GDB or something like Segger Ozone. It’s not the same “click to debug” experience that many hardware and firmware engineers are used to from vendor IDEs. We think the ability to get started in 60 seconds is worth the tradeoff.


View Mike’s slides below and watch the video for an in-depth look at how to create your own Codespaces installation and start with Zephyr today.

Golioth recently added the RAK5010 as a board that works “out of the box” with all of the Golioth Firmware SDK samples. This dev board offers a Nordic nRF52840 connected to a Quectel BG95 cellular modem–which is based on Qualcomm technology. This is quite a useful combination for a cell-connected sensor platform, GNSS/GPS asset tracker, Bluetooth gateway, and any number of other applications.

Honestly, a colossal number of boards work with Golioth thanks to Zephyr’s cross-vendor support. It’s just a matter of ensuring the board configuration is in place. Let’s dive into that today, to take a look at how we configured the RAK5010 for Zephyr and how to connect a programmer to flash the firmware.

Connecting a programmer to the RAK5010

The RAK5010 is programmed using JTAG over the ARM Single Wire Debug (SWD). There are a few things to remember with this configuration:

  • GPIO signals are 1.8V, so you must use a programmer that supports that logic level
  • The reset pin is not broken out to the headers, so software reset must be used

The SEGGER J-Link programmer is perfect for this, and already has runner support built into Zephyr.

J-link programmer connected to RAK5010 using a DIY IDC cable adapter

When I first started working with this board, used jumper cables to directly connect the pins on the RAK5010 to the pins inside the connector socket of the J-Link. The RAK5010 user guide has a connection diagram for this:

Image source: RAKwireless

I use my J-Link for a lot of different boards, so this quickly became annoying to hook back up each time I returned to it. Instead I ordered an adapter board and soldered my own little dongle that interfaces with a standard 1.27mm IDC cable
Front and back of a simple adapter that converts SWD 0.1" pins to a 1.27mm 10-pin IDC socket

I have also tested using a 1.8V JTAG programmer with this board. It will work, but I found for larger Zephyr programs I had to load the binary manually in the GDB console. Your mileage may vary.

Zephyr configuration for the RAK5010

Support for the RAKwireless RAK5010 is built into Zephyr. However, I found a few usability issues. These have already been accounted for in the board files for all of the Golioth’s Zephyr sample applications. Let’s walk through the details.

Use USB as a serial connection

If you want to use the USB port for serial output, the USB-CDC driver needs to be turned on and configured. This involves mapping the console in devicetree and enabling it in Kconfig:

/ {
  chosen {
    zephyr,console = &cdc_acm_uart0;
    zephyr,shell-uart = &cdc_acm_uart0;


&zephyr_udc0 {
  cdc_acm_uart0: cdc_acm_uart0 {
    compatible = "zephyr,cdc-acm-uart";


Use the Zephyr modem driver

The Quectel BG95 works well with Zephyr’s new(ish) modem driver. The board is supported by the in-tree cellular-modem sample and I cribbed most of the configuration from there.


# Networking

# Modem driver

The one quirk that I found, compared to other Zephyr networking, is that this board needed an explicit call to bring up the network interface. That’s now built-in to the Golioth common library, if you’re in need of help, check out the commit where it was added.

Build and flash for the RAK5010

Building and flashing is nearly the same for this board as any other. The one caveat is that if you don’t account for a lack of the reset pin, you’ll need to manually press the reset button (or power cycle) on the board for the binary to begin running. The –softreset flag shown below solves this for me. (This is the case for the default nrfjprog runner, I also tested the pyocd runner and found reset worked as expected).

west build -b rak5010_nrf52840
west flash --softreset

Other considerations

I had the chance to dig into the modem driver in a project that uses the RAK5010. It’s pretty incredible, and it looks like the chat script system will enable us to add any AT-command based modem to Zephyr with relatively little pain.

The chat script steps for the BG95 are in-tree. There is one step that always throws a timeout warning when first connecting. In my testing, this didn’t affect behavior (beyond delaying the connection for the few seconds for the timeout to occur).

As I write this post, v3.6.0 is the most recent Zephyr release. However, in a few weeks v3.7.0 will be released and this will upend board definitions. A new paradigm for defining boards has been adopted and it’s probably worth your time to get familiar with the new hardware model.

Jerónimo Agulló is an open-source projects enthusiast and a Zephyr RTOS contributor. He has worked on different IoT projects across different industries, from research centers such as the University of Sevilla, to large companies like Ferrovial Construction. Check out his work on his GitHub profile.

Location is a basic part of the most interesting IoT systems. It’s not just about loss prevention; geolocation opens the door to several applications and enhances the management of devices. In many cases, this involves Global Navigation Satellite System (GNSS).

Position is so important in IoT that the latest Zephyr release (v3.6.0) has added a new API for GNSS based on the NMEA0183 standard. Today I’m going to explain the basics of GNSS, what NMEA means, and dive into this awesome new Zephyr feature.

The basics of the NMEA standard

NMEA is the acronym for the National Marine Electronics Association, an organization created before GPS was invented. Its aim is creating better communications with manufacturers. I can assure you that they achieved their goal with GNSS.

Their most extended and well-known norm is the NMEA0183 protocol, which has become the GNSS standard for almost all manufacturers. It facilitates module integration and new applications development.

NMEA data is transmitted from a source such as a GPS module (known as a “Talker”) to equipment, such as our running Zephyr device (known as a “Listener”). One important aspect is that a single talker can communicate to many listeners.

From the electronic perspective, NMEA0183 originally used RS-422, but now uses several interfaces such as UART, USB, RS-232, WIFI, Bluetooth, and more. We can consider NMEA protocol as a common message structure standard. NMEA0183 protocol makes software developer life easier due to the standardization between GNSS devices.

NMEA message structure

Each NMEA sentence contains only ASCII characters, starting with the dollar symbol “$” and ending with the <CR> (Carriage return) and <LF> (Line feed) characters. The content of the message is a tuple, separated by commas. This starts with an NMEA identifier, followed by data, and ending by a checksum preceded by an asterisk “*”.

For a better understanding, let’s examine a popular NMEA sentence from a Quectel module. “GGA” sentence contains the Global Positioning System fix data, time, position, and fix related data for a GNSS receiver.

  • $GPGGA is the sentence identifier which can be split into “GP” which means the GNNS type, in this case GPS, and “GGA” which is the sentence identifier.
  • 102744.00 is the time of fix (hhmmss).
  • 6155.393269,N is the latitude (ddmm.mmm format, N for North).
  • 00848.433734,E is the longitude (dddmm.mmm format, E for East).
  • 1 is the fix quality indicator.
  • 03 is the number of satellites being tracked.
  • 1.6 is the horizontal dilution of precision.
  • 821.5,M is the altitude in meters above mean sea level.
  • 52.0,M is the height in meters of geoid separation.
  • *70 is the checksum (Note: this is not the correct checksum for this payload, we moved the location manually)


All talker devices don’t rely on GPS or on different constellations simultaneously. The most common constellations are GLONASS (GL), BEIDOU(DB or GB) and GALILEO (GA).

Besides GGA, other popular NMEA sentences are:

  • RMC: Recommended Minimum Navigation Information
    • This contains information similar to GGA such as the latitude and longitude and also the speed over ground (in knots), the date and the magnetic variation (in degrees).
  • GSV: Satellites in View
    • This NMEA message prints information such as the total number of satellites in view for each constellation and for each satellite its number (PRN), elevation in degrees, azimuth in degrees and Signal to Noise Ratio (SNR) in dB. Each GSV sentence contains the information of more than one satellite.

The previous NMEA sentences are named as “talker sentences”. In addition to that, there are “Proprietary sentences” and “query sentences”. On the one hand, the proprietary sentences start with “$P” and allow manufacturers to define custom NMEA sentences format for custom functions such as power management. On the other hand, the query sentences are the means for listener to request a particular sentence from a talker


If you’ve reached this point, it means that you already know and like Zephyr RTOS. If you’re not familiar with Zephyr, I encourage you to familiarize yourself with it by using this getting started guide.

Zephyr RTOS is not just a simple scheduler for simple applications. Zephyr is a whole operating system with drivers, services and common APIs which facilitates the new developments and sensors and boards exchange. I would go beyond and say that Zephyr is even a complete ecosystem with integration in awesome clouds such as Golioth.

A great example of this affirmation is the new Zephyr GNSS support in version v3.6.0. The GNSS API is built upon the modem subsystem, which provides the necessary modules to communicate with modems. The GNSS subsystem covers everything from sending and receiving commands to and from the modem, to parsing, creating and processing NMEA0183 messages.

The source code can be found under the path zephyr/drivers/gnss, which is divided into specific GNSS module drivers with custom features such as power management, some files with utils and a generic NMEA0183 driver. Covering in detail each of those utils would require a whole new post. However, I will provide a brief overview of each of them:

  • gnss_dump
    • A set of utilities to get, convert and print GNSS information into readable and useful data.
  • gnss_nmea0183 and gnss_parse
    • NMEA0183 utilities such as checksum calculation, parsing a ddmm.mmmm formatted coordinates to nano degrees or parsing the content of GGA, RMC and GSV NMEA messages.
  • gnss_nmea0183_match
    • This code is based on “modem_chat” match handlers, a Zephyr utility to process messages from modems. The callbacks to parse GGA, RMC and GSV messages are defined here.

Zephyr has created a generic NMEA driver which can be used for any NMEA talker. As any other driver, it is instantiated by the DEVICE_DT_INST_DEFINE macro. It includes the following code to call the callbacks defined in the gnss_nmea0183_match.c for corresponding NMEA identifier:

MODEM_CHAT_MATCH_WILDCARD("$??GGA,", ",*", gnss_nmea0183_match_gga_callback),
MODEM_CHAT_MATCH_WILDCARD("$??RMC,", ",*", gnss_nmea0183_match_rmc_callback),
MODEM_CHAT_MATCH_WILDCARD("$??GSV,", ",*", gnss_nmea0183_match_gsv_callback),

GNSS example in Zephyr

Now you know about NMEA0183 and the Zephyr GNSS API, so get your hands dirty and write some code! The full code can be found in my github repository:

KConfig Changes

In this example, I will show you how to configure a microcontroller in Zephyr to use a GNSS module which supports NMEA0183. For this aim, I will use an ESP32S3-Mini-1 development board along with an Air530z GNSS module. We’re going to modify our prj.conf file

  1. Add the CONFIG_GNSS=y kconfig variable to add the GNSS support.
  2. Add the variables CONFIG_GNSS_SATELLITES=y to display satellites’ information
  3. Add CONFIG_GNSS_DUMP_TO_LOG=y to print GNSS information like the fixation status, coordinates and time.

Devicetree Changes

Next, we’ll configure the device tree correctly for our GNSS module. In this case, the Air530z GNSS module uses an UART interface. By default, ESP32S3 has the UART0 configured in pins 43 and 44. However, this UART is configured at 115200 bps and used for both programming the board and by Zephyr to print messages into the serial terminal. This is not a good idea.

What is the solution? The use of another UART, for example, the UART1. However, it is not defined in esp32s3_devkitm.dts. So let ‘s do it!

According to GPIO, the UART1 uses the pins 17 and 18 (you can use others too). I have chosen these default pins and defined them in the device tree overlay using the pin control subsystem. The macro of the pins can be found in the file zephyr/include/zephyr/dt-bidings/pinctrl/esp32s3-pinctrl.h you can find your board pins in the corresponding header file of your board.


&pinctrl {

  uart1_pins: uart1_pins {
    group1 {
      pinmux = <UART1_TX_GPIO17>;

    group2 {
      pinmux = <UART1_RX_GPIO18>;


Then, in the UART node, I have added our above pin definition and a “gnss-nmea-generic” compatible device. I have created an alias to this GNSS device to facilitate the access from the main code.

&uart1 {

  status = "okay";
  current-speed = <9600>;
  pinctrl-0 = <&uart1_pins>;
  pinctrl-names = "default";
  gnssdev: gnss-nmea-generic {
    compatible = "gnss-nmea-generic";


Finally, the main code uses the GNSS_DATA_CALLBACK_DEFINE macro to define a callback for getting the information about satellites in view and another for GNSS information such as time, coordinates and tracking satellites. The following screenshot depicts the application output:


We managed to get the GNSS data!

Final thoughts

We have covered in this post what NMEA means and its importance in GNSS as well as the main NMEA messages, how the new Zephyr GNSS API relies on NMEA, and a sample of how to localize our device using Zephyr. Now, it is your turn to send the GNSS data to Golioth cloud!

In this post, we’ll walk through a couple methods for defining abstract hardware interfaces in Zephyr using the devicetree.

You’ve probably seen abstract hardware interfaces implemented on popular development boards. Many boards provide a standardized connector interface like mikroBUS, Feather, MicroMod, Arduino UNO, etc. A defining characteristic of these interfaces is the ability to plug in a range of peripheral boards that do different things, but still all use the same signals.

For example, a MikroE Weather Click board that implements the mikroBUS standard can be plugged into a LPCXpresso55S28 dev board or a STM32 M4 Clicker dev board without requiring any hardware changes to the Weather Click boards. The signals are always in the same place on the ClickBus form factor (even if each Click board doesn’t use all of them).

Using abstract interfaces can enable writing more portable devicetree definitions that can be reused across multiple boards.

Same app, multiple boards

We’ve been updating our collection of Golioth reference designs to support our Follow-Along Hardware boards. These are designs you can build yourself using off-the-shelf development boards.

Zephyr’s Devicetree support enables us to decouple the hardware definition for each board from the application source code. By using devicetree interfaces to the underlying hardware, we can target completely different boards (or different revisions of the same board) without any changes to the application code.

For example, the images below show two supported boards that run the same Air Quality Monitor reference design firmware:

Golioth Aludel Mini board

nRF9160 DK FAH board

Why use abstract hardware interfaces?

Look closely at the images above. You’ll notice that both platforms are hosting the exact same MikroE Click sensor modules.

These Click modules adhere to the mikroBUS socket specification, ensuring that any Click board can be plugged into any standard mikroBUS socket:

The Golioth Aludel Mini board in the left photo above includes two built-in MikroBUS sockets. The nRF9160 DK board in the right photo above uses an Arduino UNO Click shield to convert the onboard Arduino socket into two MikroBUS sockets.

The Weather Click board we’re using in the Air Quality Monitor reference design uses the I2C interface on the mikroBUS socket to read weather data from the onboard BME280 sensor.

On our custom Golioth Aludel Mini board, the mikroBUS I2C interface is connected to the nRF9160’s I2C1 peripheral. We could describe this hardware in the application’s aludel_mini_v1_sparkfun9160_ns.overlay file like this:

&i2c1 {
	status = "okay";

	bme280: bme280@76 {
		compatible = "bosch,bme280";
		reg = <0x76>;

However, on the off-the-shelf nRF9160 DK Follow-Along Hardware board, the mikroBUS I2C interface is connected to the nRF9160’s I2C2 peripheral (through the Arduino UNO Click shield). Similarly, we could describe this hardware in the application’s nrf9160dk_nrf9160_ns.overlay file like this:

&i2c2 {
	status = "okay";

	bme280: bme280@76 {
		compatible = "bosch,bme280";
		reg = <0x76>;

There are a couple downsides to this approach:

  1. It’s not very DRY. We are repeating the same bme280 node definition in each board overlay file in the application. If we swapped out the Weather Click board for a board with a different sensor, we’d have to update each board definition individually.
  2. It doesn’t reflect the actual hardware interface. The definitions above tightly couple the sensor hardware definitions to specific MCU peripherals on each board. As a result, we need a slightly different sensor configuration for each board.

Node labels as abstract interfaces

One simple way to address the downsides listed above is to define an abstract hardware interface using node labels.

Defining abstract hardware interfaces in boards

We can add node labels in a board’s devicetree definition for the mikroBUS UART, I2C, and SPI communication interfaces. Here’s the mikroBUS node labels for the Golioth Aludel Mini board from the aludel_mini_v1_sparkfun9160_common.dtsi file:

mikrobus_serial: &uart2 {};
mikrobus_i2c: &i2c1 {};
mikrobus_spi: &spi2 {};

Because the mikroBUS abstract interface is implemented directly by this board, these node labels map directly to specific peripherals on the nRF9160 MCU.

On the other hand, the nRF9160 DK doesn’t have any built-in mikroBUS sockets, so the nrf9160dk/nrf9160 board definition does not include these mikrobus_* node labels. However, if we enable the Arduino UNO Click shield (-DSHIELD="arduino_uno_click") when building for the nrf9160dk/nrf9160 board, the built-in arduino_uno_click.overlay will include the following mikrobus_* node labels:

mikrobus_spi: &arduino_spi {};
mikrobus_serial: &arduino_serial {};
mikrobus_i2c: &arduino_i2c {};

Since the mikroBUS abstract interface is implemented by the Arduino UNO Click shield, these node labels simply “pass through” to the arduino_* abstract interface implemented in the nrf9160dk_nrf9160_common.dtsi file.

(Currently, there is a bug with the mikrobus_i2c interface when using the arduino_uno_click shield with the nrf9160dk/nrf9160 board. See this PR for a fix.)

Using abstract hardware interfaces in overlays

Since all the boards supported by our application implement the mikrobus_* abstract interface, we can define the devicetree nodes for the Click boards in a single place using these node labels. For example, we could create a single application-level overlay app.overlay for the Weather Click board:

&mikrobus_i2c {
	status = "okay";

	bme280: bme280@76 {
		compatible = "bosch,bme280";
		reg = <0x76>;

Using abstract hardware interfaces in Shields

We can even take this one step further and turn this app.overlay into a mikroe_weather_click Zephyr Shield that we can use across multiple apps.

First, we need to add a boards/shields/mikroe_weather_click/Kconfig.shield with the following config:

	def_bool $(shields_list_contains,mikroe_weather_click)

Next, we add a boards/shields/mikroe_weather_click/mikroe_weather_click.overlay with the same devicetree code from the app.overlay above:

&mikrobus_i2c {
	status = "okay";

	bme280_mikroe_weather_click: bme280@76 {
		compatible = "bosch,bme280";
		reg = <0x76>;

Now we can simply include this shield overlay in any app by specifying the argument -DSHIELD="mikroe_weather_click" in the build command.

For example, to build the samples/sensor/bme280 app for the nRF9160 DK board + Arduino UNO Click shield + Weather Click shield:

west build -p -b nrf9160dk/nrf9160/ns samples/sensor/bme280 -- -DSHIELD="arduino_uno_click mikroe_weather_click"

(This PR adds the mikroe_weather_click shield to upstream Zephyr.)

GPIO Nexus Nodes

The mikroBUS interface described earlier defines a set of GPIO used for various purposes by the Click modules: SPI chip select (CS), enable (EN), reset (RST), interrupt (INT), etc. This is essentially an abstract interface for a group of GPIO pins. Any board that implements mikroBUS must map these pins on the socket to SoC-specific GPIO on the board.

So far, we’ve described how to define and use abstract interfaces via node labels. However, there is no single node in the devicetree that represents a GPIO pin. You can’t use a single phandle to represent one.

Instead, the Devicetree specification provides Nexus Nodes for creating these types of mappings. Specifically, GPIO nexus nodes provide a mapping between an abstract GPIO interface and the actual GPIO pins on a SoC.

For example, we can take a look at the mikroBUS GPIO nexus node defined for the lpcxpresso55s36 board:

mikrobus_header: mikrobus-connector {
	compatible = "mikro-bus";
	#gpio-cells = <2>;
	gpio-map-mask = <0xffffffff 0xffffffc0>;
	gpio-map-pass-thru = <0 0x3f>;
	gpio-map =	<0 0 &gpio1 9 0>,	/* AN  */
			/* Not a GPIO*/		/* RST */
			<2 0 &gpio0 20 0>,	/* CS   */
			<3 0 &gpio1 2 0>,	/* SCK  */
			<4 0 &gpio1 3 0>,	/* MISO */
			<5 0 &gpio0 26 0>,	/* MOSI */
						/* +3.3V */
						/* GND */
			<6 0 &gpio1 8 0>,	/* PWM  */
			<7 0 &gpio0 17 0>,	/* INT  */
			<8 0 &gpio1 24 0>,	/* RX   */
			<9 0 &gpio1 25 0>,	/* TX   */
			<10 0 &gpio1 30 0>,	/* SCL  */
			<11 0 &gpio1 21 0>;	/* SDA  */
						/* +5V */
						/* GND */

The main thing to highlight here is that the gpio-map is a mapping, where each row maps an abstract GPIO “specifier” (e.g. <&mikrobus_header 2 0>) to a real GPIO “specifier” on the LPC55S36 MCU (e.g. <&gpio0 20 0>). See the Zephyr documentation for GPIO nexus nodes and the devicetree spec for detailed descriptions of the other properties.

We can now refer to <&mikrobus_header 2 0> instead of <&gpio0 20 0> when referring to GPIOs in devices that use the mikroBUS interface. This is good because it means that we can define an app.overlay or a <shield>.overlay that is not tied to a specific SoC pin on a board.

Devicetree Aliases & Chosen Nodes

The methods we’ve shown above are helpful for defining abstract interfaces between hardware devices within devicetree. In this final section, we’ll show how to expose abstract hardware interfaces to your application via aliases & chosen nodes.

These provide a way to refer to abstract nodes without having any knowledge of the specific hardware implementation. In general, the idea is to use abstract node labels (mikrobus_i2c) to decouple devicetree definitions from board-specific devices, and use aliases/chosen nodes to decouple the application code from board-specific devices.

For example, it’s possible to build the zephyr/samples/basic/blinky app for many different boards without changing any of the application code. The only requirement is that each board’s devicetree definition must provide the board-specific implementation for the led0 alias—the abstract interface used by the application code:

leds {
	compatible = "gpio-leds";
	led0: led_0 {
		gpios = <&gpio0 2 0>;

aliases {
	led0 = &led0;

You can access this led0 alias in your application code like this:

#define LED0_NODE DT_ALIAS(led0)

static const struct gpio_dt_spec led = GPIO_DT_SPEC_GET(LED0_NODE, gpios);


Devicetree stuff is never clear on the first reading, but is super logical in the long run. Hopefully this was helpful in clarifying how to define and use abstract hardware interfaces in devicetree. If you have any follow up questions or comments, feel free to reach out in our forum.

Interested in learning more about Zephyr? Then, we’d love to have you join us for a free Zephyr developer training. Our next session is coming up in May. Sign up now!

Golioth works with Qualcomm

We’re excited to announce the latest update to the Golioth Firmware SDK, release 0.12.0, which now includes support for Zephyr’s newly introduced Modem Subsystem. This enhancement significantly increases the flexibility of our platform, enabling support for a broader array of cellular modem technologies, starting with Qualcomm. 0.12.0 adds support for the Quectel BG95, joining the Nordic Semiconductor nRF9160 (our go-to modem around here!) as a first class cellular modem. We also introduced additional ways to securely store credentials.

Zephyr’s New Modem Subsystem

Introduced in Zephyr 3.5.0, the Modem Subsystem is a unified interface for modem drivers. This addition simplifies the integration of cellular modems (and others) into Zephyr-based projects, greatly expanding the range of devices and technologies that developers can utilize effectively. For a detailed overview of the modem subsystem, check out this summary from Zephyr’s Developer Advocate, Benjamin Cabé.

Integration in Golioth Firmware SDK

With the integration of this modem subsystem in the Golioth Firmware SDK, Golioth users can now more flexibly incorporate a wider array of modem technologies into their IoT projects. There are a lot of great modems and module vendors in the market and providing choice is at the heart of what we do at Golioth.

First Supported Modem and Board

The first modem we are supporting with this updated SDK is the BG95 from Quectel, based on Qualcomm technology. The BG95 is paired with the nRF52840 on the RAK5010 development board from RAKwireless. This combination highlights the flexibility of Qualcomm’s technology integrated into Quectel’s hardware, offering developers robust tools for deploying cellular IoT solutions efficiently.

Why Qualcomm?

We chose to support Qualcomm modems because our community asked for it! Developers have different design needs and want maximum flexibility. They need more options that accommodate diverse business needs. Qualcomm chipsets offer the latest in connectivity protocols and radio technology at competitive prices. Qualcomm provides silicon and support for a wide ecosystem of module vendors, such as Quectel, U-Blox, Telit, and more. Golioth customers have used Qualcomm modems in their products in the past, but needed to do much of the integration engineering themselves. Zephyr’s Modem Subsystem makes it easier to develop applications that integrate Qualcomm modems. Connecting this wider range of modems to Golioth is more hands-off for the user, reducing complexity. Developers can focus more on innovation and less on technical hurdles.

Also in This Release

In addition to new modem support, this release introduces a another feature: DTLS socket offloading for Zephyr. This includes an example for the long-supported Nordic Semiconductor nRF9160.

DTLS socket offloading leverages a modem’s secure credential store, which allows for the use of secure, offloaded sockets. This means there is not any private key material in RAM. This can be a significant advantage as it helps reduce RAM usage, code size, CPU load, and power consumption. Actual benefits will vary depending on the application and how the code is utilized.

This new feature enhances device security and efficiency, contributing further to the versatility and robustness of the Golioth Firmware SDK. Mike previously wrote how to store credentials on the nRF9160 using TLS tags.

Getting Started

To get started with the latest SDK:

  1. Update to the newest release, 0.12.0, from the Golioth Firmware SDK repository.
  2. Explore the documentation and examples provided for integrating the RAK5010 board or try DTLS socket offloading with the nRF9160.
  3. Visit our community forums or support channels if you need help or want to discuss your projects.

Focused on Your Success

At Golioth, we’re committed to providing you with the tools and flexibility needed to succeed in the fast-evolving world of IoT. By adding support for new modems and enhancing the ways you can manage credentials, we aim to streamline your development process and empower your innovative projects. Whether you’re integrating the latest modem technology or implementing secure credential management, Golioth is here to support every step of your journey towards building smarter, more connected solutions.

In a previous post on Debugging nRF9160 Zephyr apps with Ozone, I showed how to use the New Project Wizard to create a new Ozone project for debugging a Zephyr app.

However, the project files generated by the New Project Wizard contain hard-coded paths. This makes them unsuitable for sharing with colleagues or checking them into a git repository.

In this post, I’ll demonstrate how to create a portable Ozone project file that you can check into your Zephyr application’s git repository.

Build the example app

To make this example concrete, we’ll be debugging the zephyr/samples/basic/blinky/ app from the nRF Connect SDK Zephyr repo. The firmware image will be built for the nrf9160dk_nrf9160_ns target running on the Nordic nRF9160 DK board. However, the concepts in the examples below should apply to any Zephyr app or board.

I’ve manually added an additional debug KConfig fragment file that we’ll be merging in to this app during the build process. This debug.conf file enables thread-aware debugging in Ozone:


# CONFIG_DEBUG_THREAD_INFO needs the heap memory pool to
# be defined for this app

To build the firmware for the example blinky app:

cd <your ncs workspace>/
west build -p -b nrf9160dk_nrf9160_ns zephyr/samples/basic/blinky/ -- -DEXTRA_CONF_FILE="debug.conf"

If the build completed successfully, you’ll see the build/zephyr/zephyr.elf & build/zephyr/merged.hex files we need to start a debugging session in Ozone.

Create a portable Ozone project file

Next, we’ll create a portable Ozone project file.

I’ll show the entire file first so you can see what it looks like all put together, and then I’ll walk through what each line in the file does in more detail.

I named my project file nRF9160.jdebug and located it in the root of my NCS workspace, but you can name this file anything you want (the file extension should be .jdebug):

void OnProjectLoad(void)
	Project.SetTIFSpeed("4 MHz");

void TargetDownload(void)

Ozone scripts

Project files in Ozone are scripts written in a simplified C-style language. Ozone defines a set of event handler functions that are called when specific debug events happen in the debugger.

The OnProjectLoad() event handler is executed when the project file is opened, so this is where we’ll do most of the setup for our project.

Project action functions

The nRF9160 DK board has an onboard SEGGER J-Link OB debugger which is connected to the nRF9160 SIP via SWD.

Ozone defines a set of project actions that can be used to configure the debugger settings:

  • Project.SetDevice("nRF9160_xxAA"); configures the debugger for the nRF9160 SIP
  • Project.SetTargetIF("SWD"); configures the debugger to use the Serial Wire Debug (SWD) protocol
  • Project.SetTIFSpeed("4 MHz"); sets the SWD signaling speed to 4 MHz

Next, we need to provide the debugger with a link to the System View Description (SVD) files for the ARM Cortex-M33F applications MCU in the nRF9160 SIP. Ozone comes bundled with a bunch of SVD files for different ARM cores, so we can provide a portable path to the file’s location by using the $(InstallDir) directory macro:

  • Project.AddSvdFile("$(InstallDir)/Config/CPU/Cortex-M33F.svd");

Since we’re debugging a Zephyr app, we can load the Zephyr RTOS awareness plugin automatically:

  • Project.SetOSPlugin("ZephyrPlugin.js");

File action functions

In addition, we need to provide Ozone with the path to the ELF file for the Zephyr app we’re trying to debug. We can use the $(ProjectDir) directory macro to provide a portable path to the ELF file relative to the directory containing the project file:

    • File.Open("$(ProjectDir)/build/zephyr/zephyr.elf");

Process replacement functions

In this example, we have a multi-image build for the nRF9160. We need to instruct the debugger to download the full merged image (not just the ELF). Ozone provides a set of process replacement functions that replace the default implementations of built-in debug operations. We can use the TargetDownload(); function to replace the default program download routine with one that loads the merged.hex file instead. Note that this also uses the $(ProjectDir) directory macro to provide a portable portable path to the merged image:

  • Exec.Download("$(ProjectDir)/build/zephyr/merged.hex");

Start the debug session

Now, click “File” → “Open…” and open the nRF9160.jdebug project file in Ozone.

Click on “Debug” → “Start Debug Session” → “Download & Reset Program” to start the debug session:

If you have multiple SEGGER devices plugged into your USB port, you’ll get a pop-up window asking which device should be used to flash the target:

If you don’t see the “Zephyr” window in the Ozone project, click on “View” → “Zephyr” to show the window:

That’s it!

You can relocate this file into any directory, including the git repository for your Zephyr app. Just make sure to update the relative paths in the file. Anyone who checks out the repo can quickly start a debug session without having to configure their own Ozone project.


Golioth can help you connect and secure your IoT devices, send sensor data to the web, update firmware over-the-air, and scale your fleet with an instant IoT cloud.

Try it today! You can make a small connected fleet and start sending data to and from the cloud immediately. Device management is free for individual users!

We are returning to Embedded World in 2024 (EW24) and will be showing off the things we’ve been working on at Golioth since last year. We’re also living a little closer to the edge and bringing some experiments and unreleased items to showcase on the floor. Demo gods, be kind!

We’ll be at the Zephyr booth, helping to showcase one of our favorite Real Time Operating Systems and Ecosystems (though not the only one we support!). Golioth generally only targets one part in our standard hardware setup, but we continuously verify a bunch of hardware on our Firmware SDK. This means that you can be certain that every new feature added to the Golioth Cloud and SDK is fully tested throughout our supported hardware. The great thing about Zephyr is that the underlying hardware works very similarly to all of our CVBs, because of the work from the silicon vendors that participate in the open source project. We’re excited because 2024 saw even more silicon vendors joining the fray!

Golioth Demos

Being part of a large project like Zephyr, we share time at the booth Kiosks. We’ll be at the Zephyr Booth (4-170) at the following times:

  • Tuesday, April 9th from 1500 to 1800
  • Wednesday, April 10th from 1200 to 1500
  • Thursday, April 11th from 0900 to 1130

All times CEST, which is GMT + 1

Hardware we’ll have on hand

  • Modbus Vibration monitor – This is our newest Reference Design that captures vibration and temperature data from a sensor normally used in industrial environments for large motors. The hardware in the Reference Design talks over RS-485 to the remote sensor and queries all of the available registers and publishes them to the Golioth cloud for processing and viewing.
  • Air Quality Monitor – It’s always interesting to see the trend lines of air quality metrics throughout the day. There are a lot of people that filter in and through the Nuremberg convention center, so we can track things like CO2 concentration just from the elevated number of bodies. The particulate counter is unlikely to go up a lot, but if there’s something like a cotton candy booth nearby, it could potentially have particulates flying through the air. I’ll be sure to go and investigate and clean up that particular air quality problem if the need arises.
  • Aludel Elixir – Powering our newer reference designs is the Aludel Elixir. This is an evolution of our previous designs (the Aludel Mini) that stitched together outside development boards that we loved like the Circuit Dojo nRF9160 Feather and the Sparkfun ThingPlus nRF9160. We pulled many of those components onto a bespoke PCB, and also added other components that can enhance any reference design. It all comes together in a custom milled case that enables maximum flexibility.

Aludel Elixir Rev B

Are you interested in your very own Aludel? Shoot me a note to discuss at Embedded World or fill out this form

Partner Demos:

AL2LOG at the AL2TECH booth (3A-335)

A render of the AL2LOG Unit

Our design partner AL2TECH based out of Italy is bringing an exciting new logging project based on the nRF9160 and an STM32 coprocessor. This is meant to target industrial applications that require a flexible way to capture data in a harsh environment.

The AL2LOG is a compact industrial logger with different sensor and actuator interfaces. This is based upon AL2TECH’s broad experience serving their clients in the energy meter/remote sensor monitoring industries.

The AL2LOG PCB (rev 1.0)

Some high level features include Cellular and GPS connectivity (nRF9160), as well as external module comms using USB-C, RS-485, and CAN. There is an Ultra Low Power acquisition and logging sub system (STM32) with a true 14 bit ADC. Industrial inputs including Digital Input, Dry Contact, Pulse Counter, 0-10V Voltage Analog Input, and 4-20mA Current Loop Input. There are outputs like Open Drain Output, and programmable supplies for devices downrange of the logger. The device can also switch external 220V via a relay. For powering the device you can use a 12-24V rail, or depend on the large D-Size High Power LTC Cells.

The device communicates back to Golioth using Zephyr/NCS and can be remotely updated using Golioth’s OTA service. Data routed from the field back to the cloud is accessible via the Golioth REST API.

BT Mesh demo from partner Ambient Sensors at the Zephyr booth (4-170)

BLE Mesh Demo from Ambient Sensors communicating through Golioth, on display at Embedded World

This demo shows Golioth controlling an LTE/BLE bridge (Thingy91) that communicates to a BLE Mesh composed of a series of lights that are running the Light Lightness Controller (the LC Server) model. Eight instances of the BBC micro:bit represent lights in this demo. The boards run BLE Mesh firmware on Nordic Semiconductor’s nRF52833, which could easily be controlling an LED driver instead of the demo LEDs.

The BLE SIG standard LC server code is running on each microbit, and is communicating with the Thingy91’s nRF52840, which communicates on the mesh as an LC client, Generic OnOff Client, and Light Lightness client. Using Golioth, the  demo remotely sends a command to turn the LC server on, and lights will then slowly dim over time (as is the functionality of the LC server). Normally lights dim after several hours, but for the purpose of the demo, we set that value to several seconds. Delay values may also be set via Golioth, showing how these settings can be tuned remotely if desired. Additionally, we have created commands to allow the Golioth user to turn on the lights, while turning off the LC Server (to prevent it from dimming the lights), and to set the lightness value to any desired value.

Golioth connectivity demo at the NXP FRDM Lab

Find Golioth at the FRDM Lab at EW24

NXP is another vendor that has great support in Zephyr. We’ll be showcasing Golioth’s solution on NXP boards at their FRDM Lab in the Messepark classroom (the main open area when you walk into the conference). You can also see all of NXPs new solutions at their main booth in the presentation hall, booth 4A-222.

Even more Golioth!

What’s that? You haven’t seen enough of us yet? You’d love to hear more on-site at Embedded World? Well let’s telegraph every place we’ll be! In addition to our time showcasing Golioth at the Zephyr booth, we’ll also be giving a couple of talks at the Zephyr stage (next to the Zephyr booth):

  • April 9th at 1400: “Building end-to-end Zephyr demos on IoT hardware”
  • April 10th at 1530: “Multi MCU OTA updates with Golioth and Simple Management Protocol”
  • April 11th at 1200: “Local toolchains, no more! Using GitHub Codespaces for training people on Zephyr throughout the world”

In the evenings (if we’re still standing), we plan to attend these open events and socialize with others in the industry:

  • IoT Stars – This is a short speaker event, as well as a networking session. We’ll have some of our portable demos with us.
  • PCBarts meetup – This is a more social event hosted by a local consulting and manufacturing company. Last year this was a wonderful group of engineers showing off their projects–personal and professional–after a long day at the tradeshow.

We really hope to see you at the conference! If you’d like to talk about Golioth and how we can work together, please fill out this form!

Timon Skerutsch is a software and electronics engineer who enjoys the systems design aspect of the work the most, from the cloud all the way down to silicon. He is the founder of the product development consultancy Diodes Delight.

Last year, I made the decision to do a personal prototype project with Zephyr instead of my go-to prototyping choices of Arduino, MicroPython, and CircuitPython. The latter two I still use a lot for rapid prototyping, as scripting languages like Python are hard to beat.

For the forthcoming project, I already knew where I was heading. I needed a sensor for my garden irrigation system that tracks the water level in my water reservoir. I had two requirements:

  • See the water level live on-site
  • Have the data transmitted to the internet so that I can see how things are doing while I’m away

The system regularly had leaks and failures in the irrigation pipes, which caused the automated watering pump to not only waste water but also fail to water my plants. This is detrimental during a heat wave! That added an additional requirement of an alarm to alert me of abnormal situations. When I started the project, I reached for Zephyr first. Let’s talk about why.

Why Arduino, anyway?

Change is a constant; no more so, than the early design phases of a product, when requirements rapidly shift. Prototyping is a crucial aspect of any development process and we often employ different tools for this than we do for our production firmware.

In the firmware world, the Arduino framework has been an immense success. Not only for teaching firmware development, but also as a quick and easy way to try out concepts and develop prototype solutions.

Not without controversy though. To this day, many embedded engineers will utter a silent curse on Arduino whenever it is encountered on the way to production. The supposed prototype firmware has morphed into the production firmware and now that needs to be extended with very complex functionality.

When deadlines are near and the stress levels are high, it is very appealing to companies to just keep using the code base that already seems to do 90% of the job. A lot of products end up shipping firmware based on frameworks like Arduino and then try to deal with the “last 10% of the work”. If you have developed software for a while you probably know those famous last 10% can be 90% of the work, completely derailing budgets and timelines.

But what is the issue? At the end of the day, Arduino is just a very light HAL based on C and C++! You can do whatever you could do in a bare metal C project. The emphasis is on “light”, which is very beneficial when you have only very basic requirements. Maybe you can get by just fine with printf() debugging and hitting that compile button in your IDE.

The Arduino HAL’s success was in part due to its simplicity and fully integrated development flow but it offers very few solutions to the modern challenges firmware developers are facing. When you consider the complexities you are faced with in modern devices, Over-the-Air firmware updates, firmware update encryption, Continuous Integration, connection to a cloud service, tracking of device status and metrics, then things start to look different.

Zephyr is changing things up in the industry

The Zephyr Project Real Time Operating System (RTOS) has seen a lot of adoption over the past years, even by Arduino themselves. They became a member of the Zephyr Project in 2023 and now contribute to the code base.

Zephyr is a very “IoT aware” RTOS and offers a lot of robust solutions to many of the very complex topics I mentioned. It is also one of the main targets of the Golioth Firmware SDK for that reason.

When developers first start interacting with Zephyr they often tend to be a bit intimidated. Device Tree and KConfig may be familiar tools for Embedded Linux developers, but not for someone coming from bare metal C or FreeRTOS. (Editor’s note: this is why Golioth offers free training around Zephyr).

Zephyr-specific tooling like the west meta tool means there’s a lot to learn when you start diving into Zephyr. You might start to question if that work is worth it. Especially early on in a development process where you want to move quickly and prove your concepts. You might feel a huge system like Zephyr could slow you down.

Due to the steep learning curve, Zephyr does not really have a reputation for being a tool for prototyping. But I think Zephyr has very much a place in that phase of a project and it comes with a lot of benefits once you move beyond the prototype: You are already in an environment that won’t hold you back when it comes to solving the tough problems of modern production ready firmware.

Now that I am up the (arguably steep) learning curve associated with Zephyr, I think in many cases I can produce a working solution a lot quicker than with Arduino or even MicroPython.

Not just for production grade firmware

Since I was starting a prototype, I chose an ESP32 dev board I had laying around which came with a nice little OLED screen. For the sensor, I opted for an industrial liquid level sensor. They are essentially pressure sensors in a sturdy form factor that measure the pressure differential of the outside air and the pressure seen in the liquid container.

I needed something rugged and precise to track abnormal water usage so that was a perfect solution. I ended up getting a stable 0.2mm resolution for my water column, much more than I needed. The sensor is a simple 4-20mA current loop that you often see in industrial automation and I connected that to an external precision ADC.

My firmware needs included:

  • WiFi provisioning
  • Network and application protocol to get the data to a server
  • OTA to update the device remotely
  • A GUI to show water levels on the OLED
  • ADC reads to ingest the sensor data

I opted to use Golioth for networking and OTA. While not a typical service for a hobby project, they have a (recently updated) free tier for individuals and it made the whole thing really easy. It only takes a couple lines of code to integrate into any Zephyr project and makes transmitting data to a database as easy as Serial.print(). Having OTA available is a matter of a KConfig option. Most importantly I don’t need to manage an internet facing server application!


golioth_client_t client;
const golioth_client_config_t* client_config = golioth_sample_credentials_get();
client = golioth_client_create(client_config);
golioth_client_register_event_callback(client, on_client_event, NULL);

err = golioth_lightdb_set_int_sync(client, "water-level", water_level/1000, 2);
if (err) {
    LOG_WRN("Failed to transmit water-level: %d", err);

I could have directly implemented CoAP or MQTT and host my own server for the receiving side. Both protocols are natively supported by Zephyr, which means I have flexibility if I change my mind on the server side in the future.

OTA firmware updates is also a concept native to Zephyr and very important: no matter what platform I choose! The Arduino ESP32 core has an option for OTA but if you are looking at any other MCU you would have to implement that from scratch which is a whole project in itself.
In Zephyr this is all enabled by the fantastic MCUBoot bootloader.

Abstractions are your friend in a complex world

The platform agnostic nature of Zephyr is powerful. Say you have already written a lot of code and then notice that your chosen MCU does not actually fulfill your needs. You don’t need to start from scratch because all of these advanced APIs are fully abstracted in Zephyr. You can retarget your code to a different platform with minimal code changes. The primary work will be in recreating your devicetree for the new hardware.

Arduino also is known for abstraction, but when it comes to more complex features the Arduino HAL is not defining an interface. Generally, you tend to need to stick to a particular platform if you want to take advantage of the underlying hardware’s fancy features, that is if they are available at all (in Arduino).

Lock-in with a specific IC is a painful lesson we all learned during the chip shortages of 2021 and 2022. Device Tree overlays are a great tool to stay on-top of changing hardware and describe those changes in a clean way. That flexibility is not only important from a risk perspective. Staying flexible during the prototype stage (where requirements change rapidly) allows you to try out different sensors and peripherals.

Changing a sensor in Zephyr is a matter of changing a dozen lines of devicetree definitions without needing to touch a single line of C code. This is made easier when the sensor is “in tree”, but it not the only way to use a new sensor. Devicetree also becomes a powerful tool in the early days of hardware development where your product might go through many revisions and changes. People on your team might be working with different hardware revisions but require the latest bug fixes.

This can quickly become tough to manage in the firmware if you had pin or even IC changes. No need for a ton of #ifdef‘s; all you need is a set of devicetree overlays that describe your various board revisions, the C code can most often stay the same. This not only makes your life easier but also helps reduce mistakes and stale code.

If you are still trying out options you can also interactively work with sensors through the Zephyr Shell, which makes for a great workflow to quickly try out several sensor candidates without writing firmware.

During my project I was unsure whether to choose a different MCU, because the built-in ADC of the ESP32 is quite noisy. In the end, I kept the ESP32 and chose to use an external ADC. My code did not have to change because the ADC API abstracts that away. It was just a matter of defining what ADC my project should use in the devicetree. My code does not need to care if that is an external I2C device or a peripheral internal to my MCU.

    zephyr,user {
        io-channels =
            <&adc_ext 0>;

&i2c0 {
    status = "okay";
    adc_ext: mcp3421@68 {
        compatible = "microchip,mcp3421";
        reg = <0x68>;
        #io-channel-cells = <1>;

&adc_ext {
    status = "okay";
    channel@0 {
        reg = <0>;
        zephyr,gain = "ADC_GAIN_1";
        zephyr,reference = "ADC_REF_INTERNAL";
        zephyr,acquisition-time = <ADC_ACQ_TIME_DEFAULT>;
        zephyr,resolution = <18>;

That is the benefit of fully abstracted subsystems, your application’s assumptions can stay the same most of the time. Last minute system changes are less painful during firmware development.

Complex UI’s don’t have to be complex to build

For the GUI, I went with LVGL, a popular UI framework that has been integrated into Zephyr.
That was probably the most eye opening experience to me. Normally you would have to mess with display drivers that all work very differently depending on the plugged in display. Then I would need to write code to manually transfer the rendered framebuffer to that display, which  again, tends to work differently with each display.

In Zephyr all I have to do is to modify the devicetree for which display driver my OLED needs, the resolution, and the bus it is connected to.

&spi3 {
    status = "okay";
    st7789v_st7789v_ttgo_128x64: st7789v@0 {
        compatible = "sitronix,st7789v";
        spi-max-frequency = <20000000>;
        reg = <0>;
        cmd-data-gpios = <&gpio0 16 GPIO_ACTIVE_LOW>;
        reset-gpios = <&gpio0 23 GPIO_ACTIVE_LOW>;
        width = <135>;
        height = <240>;
        x-offset = <53>;
        y-offset = <40>;
        vcom = <0x19>;
        rgb-param = [CD 08 14];

With that done you can write powerful UIs with just a couple lines of code, all the hard stuff is handled behind the scenes. I figured the UI part would be the majority of work for this project, but ended up being done in under an hour.

Often Arduino prototypes tend to have character displays or use the same old school bitmap font because fonts are hard and font systems even harder. In LVGL, you have an array of modern fonts available and it’s fairly easy to include your own font.

Arranging elements is also trivial in LVGL. No need to manually calculate a bunch of stuff like your text length. It has a lot of functions available for laying out complex arrangements.
You can build some really pretty smartphone level UIs with it. These run on very constrained hardware and it doesn’t cost you your sanity in the process!

const struct device *display_dev;
display_dev = DEVICE_DT_GET(DT_CHOSEN(zephyr_display));

lv_obj_t *level_label;
lv_obj_t *status_label;
static lv_style_t level_style;
lv_style_set_text_font(&level_style, &lv_font_montserrat_48);

level_label = lv_label_create(lv_scr_act());
lv_obj_align(level_label, LV_ALIGN_CENTER, 0, -20);

status_label = lv_label_create(lv_scr_act());
lv_obj_align(status_label, LV_ALIGN_BOTTOM_MID, 0, -50);
lv_obj_add_style(level_label, &level_style, 0);
// display the current water level
lv_label_set_text_fmt(level_label, "%llu", water_level/1000U);

Quick, once you know how to get around

Within a day I had firmware for my hardware and it even looked…pretty!
Since its creation, my device has dutifully reported the water level and withstood the winter season.

The application is ~180 lines of C code, including a lot of error handling. That is really not a lot of code, for so much complex functionality. This is only possible thanks to all of the available abstractions that make writing the actual application logic a breeze. In the background there are multiple threads running but my code doesn’t even need to be aware of that. It is all handled by the kernel and the well-written libraries.

While simple things like an ADC read can be very verbose in Zephyr and a bit more complicated than in Arduino land, the hard parts–the things we often perceive as the last 10%–are a whole lot easier! You don’t want to start empty handed when you are tasked to implement encrypted firmware updates, OTA, telemetry or integrating the cloud team’s new backend system.

The grass is not always greener, of course. Zephyr is still comparatively young in embedded terms and while there are a lot of devices already supported, it is hard to beat the vast amount of drivers available for Arduino. In some situations you might want to do a first board bring up with an existing Arduino driver before implementing a Zephyr driver. License permitting, the driver code could be the basis for a port to Zephyr.

One reason why I think it is so common for firmware to get “stuck” in Arduino-land is because there has not been a good (free) alternative if you wanted to write C (or C++) and not pigeon hole yourself early on into one specific obscure vendor toolchain. If there is nothing obvious to move to, it can make the decision even harder and it will be put off until it’s usually too late.

Concepts like devicetree and a complex build system like CMake can be daunting at first, but there are ripe benefits at the end of the learning curve. If you want to learn Zephyr, Golioth regularly offers free Zephyr training sessions or you can read some of the great blog posts that cover the more gnarly bits of Zephyr in digestible bite sized portions.