At Golioth, we talk about 3 things that make for likely hardware/firmware compatibility with our Cloud:

  • Running Zephyr RTOS
  • Have sufficient overhead to run the Golioth code in conjunction with other Zephyr code (about 2K extra code space)
  • A network interface in Zephyr

(this is not the only way to connect, just a good formula for getting started)

It’s that last point that disqualifies a bunch of boards in Zephyr. Maybe you love the STM32 feature set, but your board doesn’t have a modem to get to the internet. What then?

The great thing about Zephyr is that network interfaces are often abstracted to the point that you can add one after the fact to your board, say with a wire harness to a different PCB. If you’re at the design phase, you could also add the ESP32 as a co-processor to add connectivity. We have shown this in the past with Ethernet and with WiFi, and we’re working on a sample that adds a non-native cellular modem.

This article will show how to add WiFi to your Zephyr project in a cheap and efficient manner, using a $5 ESP32 board put into ESP-AT mode. Your project instantly has network connectivity (and a few other tricks too!).

AT commands? Like on my brick phone?

We’ll talk about the hardware in a bit, but the software part of this hinges on communication between processors using the ESP-AT command set.

AT Commands?? Like from the 80s?

Actually, exactly like that. And not just from your brick phone, the Hayes Command Set was created in 1981 for a 300 baud modem. It has survived 40 years later due to the easy connection over a serial interface (UART), which makes boards-to-board or chip-to-chip connectivity well understood and almost universally available. In fact, many of the cellular modems on the market if not using AT command sets directly (it has an ETSI standard), at least have an “AT mode” for setting up communications with cellular towers and troubleshooting.

The benefit is that the ESP32 acting as a secondary processor means a wide range of parts can talk over the UART interface. Though we’re talking about Zephyr in this post, a previous example showed a Cortex-m0+ running our Arduino SDK in conjunction with the ESP32 modem. On the Zephyr side of things, you can view the wide range of boards that are supported on our hardware catalog, including boards as powerful as the Altera Max 10 FPGA board and as small as the Seeeduino XIAO.

Set up the modem

The ESP32 AT command firmware is just a binary. If you find the proper module and chipset, you should be able to download it directly onto your board. The board the ESP32 module is mounted on doesn’t really matter, as long as you have access to the pins and can tell which pin on the PCB routes back to which pin on the module.

In this example, we are working with the ESP32-WROOM-32. This is one of the most common modules on the market today. You can find which module you have by looking at the laser etching on the metal can on the module itself.

I downloaded the latest binaries (V2.2.0.0 as of this writing) from the Espressif site. I will also show the command below using that version number, though you should use the newest version that is available. There is also a page that lists the different type of binaries and the associated pin numbers you’ll need to connect to when testing below.

esptool.py write_flash --verify 0x0 ~/Downloads/ESP32-WROOM-32-V2.2.0.0/factory/factory_WROOM-32.bin

Testing the modem

Once you have successfully programmed the modem, you’ll want to test it. This will involve manually typing in AT commands to a serial interface / terminal. While that might seem like an inefficient way to work with a modem, it’s a good skill set to have if you need to troubleshoot your setup at a later time.

You will need a USB to serial converter, or some other way to communicate with a UART. These are available on Amazon for $5 or less. You do not need any fancy features on this device.

If you’re using the ESP32-WROOM32 like me, you’ll have a setup like above. Hook up your USB to serial converter TX pin to pin 16 (ESP32 RX) and the converter’s RX pin to pin 17 (ESP32 TX). Note that there are pins labeled TX and RX on the dev kit, but those are the console output for the processor. The easy way to test is if you hit the Reset button (labeled “EN” on this board), you will see all of the boot sequence scrolling across the screen if hooked into TX/RX. If you are connected to the proper output (16/17), you will only see a ready prompt when the board is booted. Reminder to check the pin numbers if you’re using a different module than above.

In terms of the program to connect you to the USB to serial and communicate with the ESP32, a small warning about line endings. After initially using screen on Linux, I found that the line endings were not compatible with the ESP-AT family. I could see the ready prompt, but I could not enter any data. After some digging I found that you need to be able to send a Carriage Return / CR (\r) and a Line Feed / LF (\n). I followed this advice and downloaded and installed picocom and used the following command on the command line to launch a more interactive terminal: picocom /dev/ttyUSB0 --baud 115200 --omap crcrlf

This enabled me to try out various commands in the ESP-AT Command Set. Two in particular stood out to me as interesting, even though they are not implemented below:

  • AT+SLEEPWKCFG – Allows you to set the “light sleep” command for the modem and tell the modem which pin will be used for waking the modem.
  • AT+BLEGATTSSETATTR – This sets the GATT profile for the modem in Bluetooth LE mode. The command is actually just one of many commands…I didn’t realize that it was also possible to use the modem as a Bluetooth LE gateway as well!

Use the modem with samples

One hardware combination that is well supported in Golioth samples is the nRF52840 and the ESP32. Our “Hello” sample shows how you can configure the device and compile firmware for the nRF52840 while still taking advantage of the ESP-AT modem connected to it.

If you don’t have the nRF52840DK (Developer Kit), there are a range of other boards that will work. When you start actually running the demo, it will be very similar to our getting started using the ESP32 (natively), or the nRF9160. Our goal is to make a seamless experience once you have a network connection. We always love discussing projects in our forum, our Discord, and on Twitter.

Embedded software can be difficult to debug and the embedded developer’s tool chest is typically more limited than what is accessible when working with larger platforms. One of the most common debugging tools is the UART peripheral over which log output can be printed to report information regarding the state of a program’s execution and the contents of important items in device memory. This insight aids in determining the root cause of bugs and also verifying the success of a program that has run as intended. One of the lesser-known methods of obtaining such console log output is via Real Time Transfer (RTT).

Real Time Transfer is a Segger technology that is available when using a J-Link hardware debug probe. It is a high-performance bidirectional communication protocol for communication between a host CPU and target MCUs in the Cortex-M and RX processor families. And it has few notable advantages beyond what a UART can offer.

New to Golioth? Sign up for our newsletter to keep learning more about IoT development or create your free Golioth account to start building now.

RTT offers better timing; reduces dedicated pins

Debugging using print statements and the UART console may interrupt time-sensitive applications, but using RTT for console output preserves MCU real-time performance. And one of the biggest advantages of RTT is that UART pins are not needed.

RTT uses the SWD pins on the standard 10 pin J-Link header that you’ve likely already accounted for in order to program the chip. When designing a custom platform that will be used for development prior to manufacturing, UART pins do not need to be exposed in the design.

Enabling RTT in Zephyr

Both hardware and software configuration for RTT in Zephyr is trivial. A J-Link, an RTT-compatible debug target, and access to SWD programming pins round out the hardware requirements. For this example I’m using Zephyr OS, which has abstracted use of the RTT protocol. It is enabled with a few configuration options and greatly eases accessibility of the technology. I tested this example using the Nordic Semiconductor Thingy:91 as the target.

If you’re looking for a simple app to test this out on, try the Golioth Getting Started Guide and use the basic Hello application. The Kconfig directives shared below will simply move the console log output found in the Hello application from the standard UART pin routing and use the RTT over the SWD pins instead.

To enable RTT in a Zephyr application, add the following configurations to the application prj.conf file:

CONFIG_UART_CONSOLE=n
CONFIG_RTT_CONSOLE=y
CONFIG_USE_SEGGER_RTT=y

The first config will disable passing of console output to the standard UART path. The second configuration will enable the RTT console path. The third configuration enables use of the Segger RTT communication protocol.

Viewing RTT output on a computer

Viewing the output requires a special viewing tool provided in the free J-Link software package. Navigate to the folder on your computer that contains the SEGGER J-Link software tools. Find JLinkRTTViewer, and execute it. A window similar to the following will appear.

RTT setup screen

The default configurations are acceptable other than the Target Device. The tested example target is the Thingy:91 and the corresponding Target Device is the nrf9160_xxAA. Before clicking OK, build and flash the target with the configurations specified above.

The Viewer will not connect unless the associated software has been enabled to facilitate the connection. This is an important distinction between this method versus UART. The viewer will actively attempt a connection with the target and will fail to connect if the target has not been configured to use RTT previously. It will not monitor for an active connection to occur after programming. The Viewer will need to be restarted after flashing if it fails to connect.

RTT viewer window

The lower console window reports RTT Viewer activity and will indicate when it has successfully connected to the target. The Hello app output will be viewable in the “All terminals” tab, along with a terminal-specific tab (in this case Terminal 0) which include colored syntax highlighting.

Simple and Easy

There aren’t any secrets to using RTT, other than knowing that it is an option. But having the ability to see console output in cases where your target doesn’t have extra pins available, or when all of the UART periphersals are already in use, is a trick you’ll want to keep in mind. Real Time Transfer has a few other tricks up its sleeve, but those are a story for another post.

This guest post is contributed by Vojislav Milivojević, an Embedded software lead at IRNAS.

As embedded software engineers we usually have some associated hardware sitting next to our computer that is used for developing and testing our code. In most cases, this piece of hardware is connected to our computer with a so-called “programmer”, an additional tool that allows us to access processors and controllers for which we are developing code. Here we will explore the relationship between devices we are developing and a computer, but it won’t be a standard one, it will be a long-distance relationship.

I lead the firmware development at IRNAS, where we push the limits of efficient solution development on IoT devices, but since I live in a different country than the rest of my team, there are usually a lot of packages with PCBs going back and forth. While that is not a big problem, there are times when some pieces just cannot get to me quickly enough to meet customer demands. There have also been times where a specific LTE network is not available in my region. Overcoming this issue is usually done with remote desktop solutions that are not so efficient, or with some special equipment that in a nutshell is again a computer with some additional hardware. Since I needed such a solution, and none of the existing ones were able to give me a nice out-of-the-box experience, I decided to design and document a process that works for me and the complete IRNAS engineering team.

Using Segger tools

There are many solutions, commercial and open-source, that provide embedded development tools such as programmers, IDEs, logging features, etc. One of these is solutions providers is Segger, and their hardware sometimes comes as part of development boards which is really nice. At IRNAS we tend to favor using Segger J-Link tools as our ‘go-to’ solution for target flashing and debugging while building connected products. Besides that, they have a range of very useful features for embedded developers, and one of these tools is Segger Tunnel mode. This is a remote server that allows the users to connect to a J-Link programmer (and thus its connected target device) over the internet. This means a device located anywhere in the world can be debugged or brought up.

Mixing with Zephyr (west tool)

Since most of the projects that I am working on are using Zephyr RTOS, this means that the west tool is used for flashing, debugging, and many other things. West is a meta-tool that abstracts software for all different programmers and gives us the ability to easily flash for multiple targets while not needing to remember long command lines. West does support Segger J-Link for specific targets and it can be selected as one of the offered runners. The good thing about west is that it will let us pass commands to the selected runner which gives us the ability to fully utilize all the functions of the selected J-Link software.

Set up the hardware and software

In December of 2020 there was great news from Segger that the complete J-Link software is now available to run for Linux on ARM architecture, which meant that Raspberry Pi is now supported as a host machine! The idea was to connect a J-Link programmer to Raspberry Pi, add in some software, and we have ourselves a remote programming jig.

Components needed for this demo:

    • Raspberry Pi
    • J-Link programmer
    • Board with the target MCU

For the purposes of this demo, we will be using the Nordic Semiconductor nRF9160DK development kit since it already contains both a J-Link and the target MCU hardware. The board connects via USB to the Raspberry Pi which connects to power and Ethernet (WiFi is also an option).

nRF9160DK connected to Raspberry Pi

Now J-Link software needs to be installed on Raspberry Pi so it can work as a remote J-Link Server. In the Raspberry Pi user home directory, download and un-tar the Segger utilities for the Raspberry Pi (choose the Linux ARM 32-bit TGZ archive). Then configure the udev rules as per the README.txt file in the JLink_Linux_Vxxx_arm directory.

$ wget --post-data 'accept_license_agreement=accepted&non_emb_ctr=confirmed&submit=Download+software' https://www.segger.com/downloads/jlink/JLink_Linux_arm.tgz
$ tar xvf JLink_Linux_arm.tgz
$ cd JLink_Linux_V646g_arm/
$ cat README.txt
$ sudo cp 99-jlink.rules /etc/udev/rules.d/
$ sudo reboot

Next, it is time to start the remote server. On a GUI-based system, this can be done with a small application from Segger, but the good thing is that the CLI tool is also provided. I recommend checking all available options for this tool by starting it and then typing ? at the prompt.

pi@raspberrypi:~ $ JLinkRemoteServer
SEGGER J-Link Remote Server V7.22b
Compiled Jun 17 2021 17:32:35

'q' to quit '?' for help

Connected to J-Link with S/N 960012010

Waiting for client connections...
?Command line options:
? - Prints the list of available command line options
-Port - Specifies listening port of J-Link Remote Server
-UseTunnel - Specifies if tunneled connection shall be used
-SelectEmuBySN - Specifies to connect to a J-Link with a specific S/N
-TunnelServer - Specify a tunnel server to connect to (default: jlink.segger.com:19020)
-TunnelBySN - Specifies to identify at tunnel server via J-Link S/N
-TunnelByName - Specifies to identify at tunnel server via custom name
-TunnelPW - Specifies to protect the connection with a password
-TunnelEncrypt - Specifies to encrypt any transferred data of a tunneled connection
-TunnelPort - Specifies to connect to a tunnel server listening on a specific port
-select - <USB/IP>[=<SN/Hostname>] Specify how to connect to J-Link

Before entering the command we need to think of a name for our tunnel and a password. For me, this will be tunnel name: remote_nrf91 and password: 19frn. Then start the remote server with the command:

JLinkRemoteServer -UseTunnel -TunnelByName remote_nrf91 -TunnelPW 19frn

Demo time

To test this remote flashing we will build a demo application on our host computer. nRF Connect SDK (NCS) that is based on ZephyrRTOS contains some sample applications and we will use shell_module, which enables us to use shell commands over UART with nRF9160. The selected application is located in the ncs/zephyr/samples/subsys/shell/shell_module folder of NCS. To build it for nRF9160DK we will use the command:

west build -b nrf9160dk_nrf9160_ns -p

After that let’s flash the board that is connected to our remote Raspberry Pi. The default runner for flashing the nRF9160DK is nrfjprog, but instead of that, we want to use the J-Link supported runner. Since the west tool does not natively support remote flashing, parameters will be sent directly to the J-Link software using the --tool-opt flag.

west flash -r jlink --tool-opt="ip tunnel:remote_nrf91:19frn"

This will flash our target MCU that is connected to J-Link and Raspberry Pi. To validate the result, open the serial terminal on Raspberry Pi and see if shell commands are working.

minicom -D /dev/ttyACM1 -b 115200

Summary

While Segger provides very interesting tools for embedded developers, there is still some work that needs to be done so they are properly integrated into our development workflow. Remote flashing is just one part of all capabilities, so this can be a starting point for a great remote development setup!

What if you could open a text document on a device, write code, click save, and everything magically starts working? This is the promise of high level programming languages like CircuitPython. Golioth Labs now has an SDK to utilize the language’s fast prototyping capabilities. In addition to Golioth’s cloud functions, it’s super easy to pass data from a networked device up to the Golioth cloud. Click save to stream IoT device data to the cloud!

What is CircuitPython?

Adafruit created CircuitPython (CP), which started as a hardware specific fork of MicroPython (MP). Both CP and MP are based upon the ideas of the Python3 programming language, such as using an interpreter and basing language syntax on whitespace separators. The challenge is that the interpreter must live on a much smaller target than most computers running Python; fitting all of that into 128K of flash and 32K of RAM is a challenge! The hardware specific ports of CP encompass many of the Adafruit boards, but are really targeted around the chipset on those boards. Targets like the Microchip SAMD21 and SAMD51, the Raspberry Pi RP2040, the ESP32-S2, and more. The project continues to grow, both from Adafruit’s ongoing contributions and from a strong community contributing to the project.

Developing for prototyping

One of the strongest features of a language like CP is the ability to quickly iterate code. The “click save and your code starts running” is in contrast to the traditional method of compiling code on an external device (ie. your laptop), downloading it through a debugger (ie. a J-Link), debugging the code, and then watching the output. Because the CP interpreter is on the device itself, it processes the code as the device starts. While this means there is less room for user code, it is a much faster turn around once you click save.

During Golioth “Office Hours” on our community Discord (happening every Wednesday at 1 pm EST / 10 am PST), we had users asking for a faster way to prototype. Golioth developed an SDK in the “Golioth Labs” section of GitHub to interface with the Golioth API(s). This allows users with CircuitPython based programs to connect their IoT devices to the cloud and pass data back to Golioth. Much like the Arduino SDK experiment repo by Golioth Labs, we are interested in trying to extend the functionality of the Golioth cloud to many different platforms. If you have a hardware platform you would like supported, please post about it on our community forum!

Transport? I just want to get there!

As Alvaro points out in the video below, the transport layer is not something the user needs to care about with the Golioth CircuitPython SDK; any time you are working with a Golioth hosted SDK, you will be working at a higher level than the transport layer of communication, such as CoAP or MQTT. Many IoT platforms stop at the transport layer, notably MQTT examples. By moving up to working with APIs to LightDB State or LightDB Stream, you get additional functionality on the device side and you can better maintain your data on the Golioth console. Users always have the option to work at a lower level and peek under the hood at how the communication is happening. For people wanting to get started quickly, it’s important to have a high level way to start streaming data to the cloud.

Getting started with CircuitPython and Golioth

In the demo below, Alvaro showcases a sample setup using the following

First follow the Getting Started Guide to program your chosen hardware with the .UF2 file that represents the underlying CP tools, such as the interpreter and the code required to make your specific microcontroller into a CircuitPython based device.

The next step is hooking up the MicroMod device to the ESP32 modem using jumper wires. Read your board documentation to find an accessible UART port (TX/RX). Once completed, this enables the (Bluetooth based) nRF52840 to communicate over a UART connection to the modem and gain a connection to the internet using the ESP32’s WiFi interface. Credentials for the WiFi modem to connect to a local hotspot are sent from the nRF52840 to the ESP32 using AT Commands early in the CircuitPython program.

The example code on the Golioth CircuitPython SDK primarily lives in code.py. The device first connects using credentials for the WiFi network and PSK information for validating onto the Golioth cloud contained within secrets.py. After the UART is configured, the main processor (nRF52840 in the example below) starts communicating with the modem. In Alvaro’s example, he is sending the internal temperature of the processor using LightDB stream. He is able to send an update down to the board using LightDB State to remotely turn on an LED on the board, which is listening for changes on the /led path, similar to other examples that use LightDB State.

Watch the demo

It all comes together in the video below. Learn about how Golioth and CircuitPython pair to make a great combination for prototyping your next sensor project.

Troubleshooting high complexity systems like Zephyr requires more thorough tools. Menuconfig allows users to see the layers of their system and adjust settings without requiring a complete system recompilation.

The troubleshoot loop

Modify, compile, test.

Modify, compile, test.

Modify, compile, test.

Modify, compile, test.

How do we break out of this loop of trying to change different settings in a program, recompiling the entire thing, and then waiting for a build to finish? Sure, there are some tools to modify things if you’re step debugging, such as changing parameters in memory. But you can’t go and allocate new memory after compiling normally. So what happens when you need to change things? You find the #define in the code, change the parameter, and recompile. What a slow process!

Moving up the complexity stack

We move up the “complexity stack” from a bare-metal device to running a Real Time Operating System (RTOS) in order to get access to higher level functions. Not only does this allow us to abstract things like network interfaces and target different types of hardware, but it also allows us to add layers of software that would be untenable when running bare-metal firmware. The downside, of course, is that it’s more complex.

When you’re trying to figure out what is going wrong in a complex system like Zephyr, it can mean chasing problems through many layers of functions and threads. It’s hard to keep track of where things are and what is “in charge” when it comes time to change things.

Enter Menuconfig

Menuconfig is a tool borrowed from Linux development that works in a similar way: a high complexity system that needs some level of organization. Obviously, in full Linux systems, the complexity often will be even higher than in an RTOS. In the video below, Marcin shows how he uses Menuconfig to turn features on and off during debugging, including with the Golioth “hello” example. As recommended in the video, new Zephyr users can also utilize Menuconfig to explore the system and which characteristics are enabled and available.

 

 

One of the first challenges any embedded software developer faces is installing and configuring their development environment and toolchain. Toolchain version, silicon vendor libraries, Windows versus Linux, debug configuration, IDE settings, and environment variables are just a few components of the modern embedded developers workspace. The result of all this complexity is a fragile, hard to reproduce workspace for software often used in critical systems. We consider this developer experience equivalent to torture, and believe it is trapping value from reaching the market.

We know there is a better way. If the development environment can be entirely packaged and abstracted away from the developer they will be able to more quickly begin application development.  A remotely managed toolchain also facilitates more efficient teamwork. It eliminates the cryptic mantra: “Works On My Machine” accompanied by an obligatory shrug.

The Established Way

The traditional approach to eliminating toolchain headaches has typically been through the use of Integrated Development Environments (IDEs).  However, these packages are generally locked to a particular silicon vendor or compiler, may have paywalls to expose premium features, and can be constrained in feature availability. Our ‘gold standard’ for years has been the following:

  • Ubuntu Virtual Machine
  • Eclipse
  • GNU MCU Eclipse Tools
  • USB Passthrough from VM for debugging boards

Modern Web Options

Technologies like, VS Code, containerization, Microsoft’s Debug Protocol, and Language Server protocol have come together to enable a transformational developer experience. Most of the current approaches to bringing these technologies together in the market are built on top of some variation of VS Code. Each solution is vying to take advantage of VS Code’s capacity to run in the browser as seamlessly as it runs on a local machine.

One option is Github Codespaces.  Which option requires the user to be on a paid plan, is not focused on embedded development, and uses a closed source server that is closed source. Another option is Keil Studio.  Keil Studio is optimized for embedded development with ARM based microcontrollers. Pricing and roadmap are not yet established. It provides no terminal access and offers a limited number of embedded targets to work with.

Why We Chose Gitpod

We chose Gitpod in part for our mutually valued stance on open source, sustained active community engagement, and obsession with developer experience. Of note is Gitpod’s full-feature free tier. They provides 50 hours of running workspace per month; No payment details required. As a result, the psychological barrier of getting up to get one’s credit card is avoided. Fifty hours is enough time to introduce oneself to the Zephyr and Golioth ecosystems. Finally, Gitpod being open source enables us and our developers to optimize their workspaces to their needs.

 

Our Current Gitpod Workflow

Setting up the application begins with cloning the Example Application from Zephyr GitHub. Next, the Golioth SDK is added as a dependency.  Changes are then added to the .gitpod.yml and .gitpod.Dockerfile. After running ‘west update’ with the configurations in place, the hello application is copied from the Golioth SDK directory to the project root directory in place of or at the same level as the ‘app’ folder. Here is a link to the end result.

Also, some background info on Gitpod.

Our target embedded cloud developer experience would be one in which the developer instantiates the cloud development environment and has zero local tool dependencies.  They can then plug their debug hardware into any machine from anywhere with internet access and develop.  Our current implementation requires three local tools to facilitate debugging functionality with the current state of the Gitpod software and VS Code. Gitpod provides a Gitpod Local Companion which allows localhost access to any TCP port in a remote workspace. The second piece of software required locally is SSH.  SSH is necessary to establish an ssh tunnel between the Gitpod instance and local machine. The final software that is run on the local machine is JLinkGDBServerCL.

 

 

State of the Art

The technology to facilitate cloud-based development has arrived and it will enable remarkable gains in productivity and developer experience.  Unfortunately, we still have local dependencies and in the current state things are not optimized for use over the internet. Step debugging was accomplished, but some work remains for embedded cloud development to compete with a local development environment. With this in mind, a future blog post will show we actually can be effective developers when using this solution with a virtualized QEMU target.

To do this proper we’d serve an MS DAP compatible debug server such as Probe.rs to the developers browser, and hook it up to the target board using webUSB. A challenge that exists, is the lack of open source microcontroller debuggers written in JavaScript or WebAssembly.  The translation from C code to WebAssembly is not straightforward and can be error-prone. However, valid translators of Rust to WebAssembly do exist, and Probe.rs is an open-source debugger written in Rust.  We also need to convince Microsoft to push this PR forward.

Stay tuned for a future post about how to build, run, and debug Golioth examples with QEMU in less than 10 clicks.