Troubleshooting high complexity systems like Zephyr requires more thorough tools. Menuconfig allows users to see the layers of their system and adjust settings without requiring a complete system recompilation.
The troubleshoot loop
Modify, compile, test.
Modify, compile, test.
Modify, compile, test.
Modify, compile, test.
How do we break out of this loop of trying to change different settings in a program, recompiling the entire thing, and then waiting for a build to finish? Sure, there are some tools to modify things if you’re step debugging, such as changing parameters in memory. But you can’t go and allocate new memory after compiling normally. So what happens when you need to change things? You find the #define in the code, change the parameter, and recompile. What a slow process!
Moving up the complexity stack
We move up the “complexity stack” from a bare-metal device to running a Real Time Operating System (RTOS) in order to get access to higher level functions. Not only does this allow us to abstract things like network interfaces and target different types of hardware, but it also allows us to add layers of software that would be untenable when running bare-metal firmware. The downside, of course, is that it’s more complex.
When you’re trying to figure out what is going wrong in a complex system like Zephyr, it can mean chasing problems through many layers of functions and threads. It’s hard to keep track of where things are and what is “in charge” when it comes time to change things.
Enter Menuconfig
Menuconfig is a tool borrowed from Linux development that works in a similar way: a high complexity system that needs some level of organization. Obviously, in full Linux systems, the complexity often will be even higher than in an RTOS. In the video below, Marcin shows how he uses Menuconfig to turn features on and off during debugging, including with the Golioth “hello” example. As recommended in the video, new Zephyr users can also utilize Menuconfig to explore the system and which characteristics are enabled and available.
https://blog.golioth.io/wp-content/uploads/2021/11/Zephyr-Menuconfig.png10801920Chris Gammellhttps://blog.golioth.io/wp-content/uploads/2023/09/Golioth_Logo_CoralWhite_RGB-1-320x127.pngChris Gammell2021-11-09 13:01:212024-01-26 11:03:28Troubleshoot Zephyr Like A Pro With Menuconfig
Let’s pretend you’re in the middle of a global chip shortage.
Surprise! There’s no need to pretend, as we are all currently in the middle of a global chip shortage. Right now it’s very difficult to source certain components.
“Why don’t hardware makers just switch out the components when they can’t source them during the chip shortage? In fact, why don’t people switch chips on a regular basis?”
As a generalization, switching costs for embedded devices are very high. If we were able to magically solve all of the switching costs for the hardware, you’d still need to deal with the switching costs of firmware and cloud platforms. This often is even more dire than than the hardware switching costs. It’s significant at both an individual level (rewriting firmware to target different architectures and board setups) and at an institutional level (maintaining different platforms and interoperability).
Operating systems and Real Time Operating Systems (RTOS) help by abstracting away a lot of the individual hardware details. When a new device is added to an RTOS, it needs to fit within the constraints of the system. If your board has an i2c sensor on it, you need to ensure your supporting firmware for that board or chipset capable of working with the elements of the RTOS. Then you can take advantage of the drivers already written for other boards/chipsets on the platform. Assuming you are willing to work within that system, you can start to supercharge your development. It’s possible to switch out components quickly and confidently, helping to alleviate the woes of the chip shortage currently underway.
Making the switch
Let’s say you have a board using an ESP32 module. Due to sourcing problems, you can no longer source a particular LED on your board, but you don’t want to change your PCB. Instead, you ask a technician wire in an extra LED to a spare pin you have on your production board that has a larger landing area. You need to build a firmware image to drive a different pin on the microcontroller than you previously were using. Now “LED2” (as it’s called in your program) is not driving pin 18, but is instead driving pin 22. With Zephyr overlays, the switch will take 5 minutes. As Marcin shows in the video below, the device tree overlay is where we map the signals internal to the firmware to the physical pins being used.
Now let’s say you cannot source the ESP32 at all, for some reason. You could create a new overlay file for a different target that works with Zephyr, assign the pins to target the functions you need on a new PCB containing a different chip, and then target that device. The time consuming aspect would be checking all of the functions are performing the same as your previous platform. But once you have decided on a new platform, assigning pins and functions to your new device would occur through overlay files.
How to use Zephyr Overlays
In the video below, we walk through the location and function of overlays in Zephyr. Marcin explains that customization of firmware images for particular hardware targets can be as simple as a different flag on the command line. In this particular example, we are showing how to change the pins for the ESP32 demo. Previously the ESP32 overlay was shown as part of our our LightDB Sample code (docs), which targeted an ESP32-DevKitC in that video.
The Zephyr Project is a popular and open source Real Time Operating System (RTOS) that enables complex features and easy connectivity to embedded devices. The project is focused on vendor participation, long-term support, and in-depth security development life cycles for products.
About Golioth
Golioth helps users to speed up development and increase the chances that pilots will be put into production with a commercial IoT development platform built for scale. We offer standardized interfaces for connecting embedded devices to the cloud and build out software ecosystems that allow your projects to get to market faster. Golioth uses Zephyr as part of the Golioth SDK to bootstrap application examples and show how to utilize the range of networking features Golioth enables via APIs.
