,

Remote Zephyr development using Segger tunnel and a Raspberry Pi

Remote Zephyr DevelopmentPublic domain https://publicdomainvectors.org/en/free-clipart/Internet-e-mail-sign/40098.html https://publicdomainvectors.org/en/free-clipart/Microchip/40844.html https://publicdomainvectors.org/en/free-clipart/Arrow-shapes-vector/24597.html
This guest post is contributed by Vojislav Milivojević, an Embedded software lead at IRNAS.

As embedded software engineers we usually have some associated hardware sitting next to our computer that is used for developing and testing our code. In most cases, this piece of hardware is connected to our computer with a so-called “programmer”, an additional tool that allows us to access processors and controllers for which we are developing code. Here we will explore the relationship between devices we are developing and a computer, but it won’t be a standard one, it will be a long-distance relationship.

I lead the firmware development at IRNAS, where we push the limits of efficient solution development on IoT devices, but since I live in a different country than the rest of my team, there are usually a lot of packages with PCBs going back and forth. While that is not a big problem, there are times when some pieces just cannot get to me quickly enough to meet customer demands. There have also been times where a specific LTE network is not available in my region. Overcoming this issue is usually done with remote desktop solutions that are not so efficient, or with some special equipment that in a nutshell is again a computer with some additional hardware. Since I needed such a solution, and none of the existing ones were able to give me a nice out-of-the-box experience, I decided to design and document a process that works for me and the complete IRNAS engineering team.

Using Segger tools

There are many solutions, commercial and open-source, that provide embedded development tools such as programmers, IDEs, logging features, etc. One of these is solutions providers is Segger, and their hardware sometimes comes as part of development boards which is really nice. At IRNAS we tend to favor using Segger J-Link tools as our ‘go-to’ solution for target flashing and debugging while building connected products. Besides that, they have a range of very useful features for embedded developers, and one of these tools is Segger Tunnel mode. This is a remote server that allows the users to connect to a J-Link programmer (and thus its connected target device) over the internet. This means a device located anywhere in the world can be debugged or brought up.

Mixing with Zephyr (west tool)

Since most of the projects that I am working on are using Zephyr RTOS, this means that the west tool is used for flashing, debugging, and many other things. West is a meta-tool that abstracts software for all different programmers and gives us the ability to easily flash for multiple targets while not needing to remember long command lines. West does support Segger J-Link for specific targets and it can be selected as one of the offered runners. The good thing about west is that it will let us pass commands to the selected runner which gives us the ability to fully utilize all the functions of the selected J-Link software.

Set up the hardware and software

In December of 2020 there was great news from Segger that the complete J-Link software is now available to run for Linux on ARM architecture, which meant that Raspberry Pi is now supported as a host machine! The idea was to connect a J-Link programmer to Raspberry Pi, add in some software, and we have ourselves a remote programming jig.

Components needed for this demo:

    • Raspberry Pi
    • J-Link programmer
    • Board with the target MCU

For the purposes of this demo, we will be using the Nordic Semiconductor nRF9160DK development kit since it already contains both a J-Link and the target MCU hardware. The board connects via USB to the Raspberry Pi which connects to power and Ethernet (WiFi is also an option).

nRF9160DK connected to Raspberry Pi

Now J-Link software needs to be installed on Raspberry Pi so it can work as a remote J-Link Server. In the Raspberry Pi user home directory, download and un-tar the Segger utilities for the Raspberry Pi (choose the Linux ARM 32-bit TGZ archive). Then configure the udev rules as per the README.txt file in the JLink_Linux_Vxxx_arm directory.

$ wget --post-data 'accept_license_agreement=accepted&non_emb_ctr=confirmed&submit=Download+software' https://www.segger.com/downloads/jlink/JLink_Linux_arm.tgz
$ tar xvf JLink_Linux_arm.tgz
$ cd JLink_Linux_V646g_arm/
$ cat README.txt
$ sudo cp 99-jlink.rules /etc/udev/rules.d/
$ sudo reboot

Next, it is time to start the remote server. On a GUI-based system, this can be done with a small application from Segger, but the good thing is that the CLI tool is also provided. I recommend checking all available options for this tool by starting it and then typing ? at the prompt.

pi@raspberrypi:~ $ JLinkRemoteServer
SEGGER J-Link Remote Server V7.22b
Compiled Jun 17 2021 17:32:35

'q' to quit '?' for help

Connected to J-Link with S/N 960012010

Waiting for client connections...
?Command line options:
? - Prints the list of available command line options
-Port - Specifies listening port of J-Link Remote Server
-UseTunnel - Specifies if tunneled connection shall be used
-SelectEmuBySN - Specifies to connect to a J-Link with a specific S/N
-TunnelServer - Specify a tunnel server to connect to (default: jlink.segger.com:19020)
-TunnelBySN - Specifies to identify at tunnel server via J-Link S/N
-TunnelByName - Specifies to identify at tunnel server via custom name
-TunnelPW - Specifies to protect the connection with a password
-TunnelEncrypt - Specifies to encrypt any transferred data of a tunneled connection
-TunnelPort - Specifies to connect to a tunnel server listening on a specific port
-select - <USB/IP>[=<SN/Hostname>] Specify how to connect to J-Link

Before entering the command we need to think of a name for our tunnel and a password. For me, this will be tunnel name: remote_nrf91 and password: 19frn. Then start the remote server with the command:

JLinkRemoteServer -UseTunnel -TunnelByName remote_nrf91 -TunnelPW 19frn

Demo time

To test this remote flashing we will build a demo application on our host computer. nRF Connect SDK (NCS) that is based on ZephyrRTOS contains some sample applications and we will use shell_module, which enables us to use shell commands over UART with nRF9160. The selected application is located in the ncs/zephyr/samples/subsys/shell/shell_module folder of NCS. To build it for nRF9160DK we will use the command:

west build -b nrf9160dk_nrf9160_ns -p

After that let’s flash the board that is connected to our remote Raspberry Pi. The default runner for flashing the nRF9160DK is nrfjprog, but instead of that, we want to use the J-Link supported runner. Since the west tool does not natively support remote flashing, parameters will be sent directly to the J-Link software using the --tool-opt flag.

west flash -r jlink --tool-opt="ip tunnel:remote_nrf91:19frn"

This will flash our target MCU that is connected to J-Link and Raspberry Pi. To validate the result, open the serial terminal on Raspberry Pi and see if shell commands are working.

minicom -D /dev/ttyACM1 -b 115200

Summary

While Segger provides very interesting tools for embedded developers, there is still some work that needs to be done so they are properly integrated into our development workflow. Remote flashing is just one part of all capabilities, so this can be a starting point for a great remote development setup!

/by
,

Click “save” to stream IoT device data to the cloud — Golioth introduces CircuitPython SDK

What if you could open a text document on a device, write code, click save, and everything magically starts working? This is the promise of high level programming languages like CircuitPython. Golioth Labs now has an SDK to utilize the language’s fast prototyping capabilities. In addition to Golioth’s cloud functions, it’s super easy to pass data from a networked device up to the Golioth cloud. Click save to stream IoT device data to the cloud!

What is CircuitPython?

Adafruit created CircuitPython (CP), which started as a hardware specific fork of MicroPython (MP). Both CP and MP are based upon the ideas of the Python3 programming language, such as using an interpreter and basing language syntax on whitespace separators. The challenge is that the interpreter must live on a much smaller target than most computers running Python; fitting all of that into 128K of flash and 32K of RAM is a challenge! The hardware specific ports of CP encompass many of the Adafruit boards, but are really targeted around the chipset on those boards. Targets like the Microchip SAMD21 and SAMD51, the Raspberry Pi RP2040, the ESP32-S2, and more. The project continues to grow, both from Adafruit’s ongoing contributions and from a strong community contributing to the project.

Developing for prototyping

One of the strongest features of a language like CP is the ability to quickly iterate code. The “click save and your code starts running” is in contrast to the traditional method of compiling code on an external device (ie. your laptop), downloading it through a debugger (ie. a J-Link), debugging the code, and then watching the output. Because the CP interpreter is on the device itself, it processes the code as the device starts. While this means there is less room for user code, it is a much faster turn around once you click save.

During Golioth “Office Hours” on our community Discord (happening every Wednesday at 1 pm EST / 10 am PST), we had users asking for a faster way to prototype. Golioth developed an SDK in the “Golioth Labs” section of GitHub to interface with the Golioth API(s). This allows users with CircuitPython based programs to connect their IoT devices to the cloud and pass data back to Golioth. Much like the Arduino SDK experiment repo by Golioth Labs, we are interested in trying to extend the functionality of the Golioth cloud to many different platforms. If you have a hardware platform you would like supported, please post about it on our community forum!

Transport? I just want to get there!

As Alvaro points out in the video below, the transport layer is not something the user needs to care about with the Golioth CircuitPython SDK; any time you are working with a Golioth hosted SDK, you will be working at a higher level than the transport layer of communication, such as CoAP or MQTT. Many IoT platforms stop at the transport layer, notably MQTT examples. By moving up to working with APIs to LightDB State or LightDB Stream, you get additional functionality on the device side and you can better maintain your data on the Golioth console. Users always have the option to work at a lower level and peek under the hood at how the communication is happening. For people wanting to get started quickly, it’s important to have a high level way to start streaming data to the cloud.

Getting started with CircuitPython and Golioth

In the demo below, Alvaro showcases a sample setup using the following

First follow the Getting Started Guide to program your chosen hardware with the .UF2 file that represents the underlying CP tools, such as the interpreter and the code required to make your specific microcontroller into a CircuitPython based device.

The next step is hooking up the MicroMod device to the ESP32 modem using jumper wires. Read your board documentation to find an accessible UART port (TX/RX). Once completed, this enables the (Bluetooth based) nRF52840 to communicate over a UART connection to the modem and gain a connection to the internet using the ESP32’s WiFi interface. Credentials for the WiFi modem to connect to a local hotspot are sent from the nRF52840 to the ESP32 using AT Commands early in the CircuitPython program.

The example code on the Golioth CircuitPython SDK primarily lives in code.py. The device first connects using credentials for the WiFi network and PSK information for validating onto the Golioth cloud contained within secrets.py. After the UART is configured, the main processor (nRF52840 in the example below) starts communicating with the modem. In Alvaro’s example, he is sending the internal temperature of the processor using LightDB stream. He is able to send an update down to the board using LightDB State to remotely turn on an LED on the board, which is listening for changes on the /led path, similar to other examples that use LightDB State.

Watch the demo

It all comes together in the video below. Learn about how Golioth and CircuitPython pair to make a great combination for prototyping your next sensor project.

/by
, ,

Troubleshoot Zephyr Like A Pro With Menuconfig

Troubleshooting high complexity systems like Zephyr requires more thorough tools. Menuconfig allows users to see the layers of their system and adjust settings without requiring a complete system recompilation.

The troubleshoot loop

Modify, compile, test.

Modify, compile, test.

Modify, compile, test.

Modify, compile, test.

How do we break out of this loop of trying to change different settings in a program, recompiling the entire thing, and then waiting for a build to finish? Sure, there are some tools to modify things if you’re step debugging, such as changing parameters in memory. But you can’t go and allocate new memory after compiling normally. So what happens when you need to change things? You find the #define in the code, change the parameter, and recompile. What a slow process!

Moving up the complexity stack

We move up the “complexity stack” from a bare-metal device to running a Real Time Operating System (RTOS) in order to get access to higher level functions. Not only does this allow us to abstract things like network interfaces and target different types of hardware, but it also allows us to add layers of software that would be untenable when running bare-metal firmware. The downside, of course, is that it’s more complex.

When you’re trying to figure out what is going wrong in a complex system like Zephyr, it can mean chasing problems through many layers of functions and threads. It’s hard to keep track of where things are and what is “in charge” when it comes time to change things.

Enter Menuconfig

Menuconfig is a tool borrowed from Linux development that works in a similar way: a high complexity system that needs some level of organization. Obviously, in full Linux systems, the complexity often will be even higher than in an RTOS. In the video below, Marcin shows how he uses Menuconfig to turn features on and off during debugging, including with the Golioth “hello” example. As recommended in the video, new Zephyr users can also utilize Menuconfig to explore the system and which characteristics are enabled and available.

 

 

/0 Comments/by

We tried to debug a microcontroller from our browser

One of the first challenges any embedded software developer faces is installing and configuring their development environment and toolchain. Toolchain version, silicon vendor libraries, Windows versus Linux, debug configuration, IDE settings, and environment variables are just a few components of the modern embedded developers workspace. The result of all this complexity is a fragile, hard to reproduce workspace for software often used in critical systems. We consider this developer experience equivalent to torture, and believe it is trapping value from reaching the market.

We know there is a better way. If the development environment can be entirely packaged and abstracted away from the developer they will be able to more quickly begin application development.  A remotely managed toolchain also facilitates more efficient teamwork. It eliminates the cryptic mantra: “Works On My Machine” accompanied by an obligatory shrug.

The Established Way

The traditional approach to eliminating toolchain headaches has typically been through the use of Integrated Development Environments (IDEs).  However, these packages are generally locked to a particular silicon vendor or compiler, may have paywalls to expose premium features, and can be constrained in feature availability. Our ‘gold standard’ for years has been the following:

  • Ubuntu Virtual Machine
  • Eclipse
  • GNU MCU Eclipse Tools
  • USB Passthrough from VM for debugging boards

Modern Web Options

Technologies like, VS Code, containerization, Microsoft’s Debug Protocol, and Language Server protocol have come together to enable a transformational developer experience. Most of the current approaches to bringing these technologies together in the market are built on top of some variation of VS Code. Each solution is vying to take advantage of VS Code’s capacity to run in the browser as seamlessly as it runs on a local machine.

One option is Github Codespaces.  Which option requires the user to be on a paid plan, is not focused on embedded development, and uses a closed source server that is closed source. Another option is Keil Studio.  Keil Studio is optimized for embedded development with ARM based microcontrollers. Pricing and roadmap are not yet established. It provides no terminal access and offers a limited number of embedded targets to work with.

Why We Chose Gitpod

We chose Gitpod in part for our mutually valued stance on open source, sustained active community engagement, and obsession with developer experience. Of note is Gitpod’s full-feature free tier. They provides 50 hours of running workspace per month; No payment details required. As a result, the psychological barrier of getting up to get one’s credit card is avoided. Fifty hours is enough time to introduce oneself to the Zephyr and Golioth ecosystems. Finally, Gitpod being open source enables us and our developers to optimize their workspaces to their needs.

 

Our Current Gitpod Workflow

Setting up the application begins with cloning the Example Application from Zephyr GitHub. Next, the Golioth SDK is added as a dependency.  Changes are then added to the .gitpod.yml and .gitpod.Dockerfile. After running ‘west update’ with the configurations in place, the hello application is copied from the Golioth SDK directory to the project root directory in place of or at the same level as the ‘app’ folder. Here is a link to the end result.

Also, some background info on Gitpod.

Our target embedded cloud developer experience would be one in which the developer instantiates the cloud development environment and has zero local tool dependencies.  They can then plug their debug hardware into any machine from anywhere with internet access and develop.  Our current implementation requires three local tools to facilitate debugging functionality with the current state of the Gitpod software and VS Code. Gitpod provides a Gitpod Local Companion which allows localhost access to any TCP port in a remote workspace. The second piece of software required locally is SSH.  SSH is necessary to establish an ssh tunnel between the Gitpod instance and local machine. The final software that is run on the local machine is JLinkGDBServerCL.

 

 

State of the Art

The technology to facilitate cloud-based development has arrived and it will enable remarkable gains in productivity and developer experience.  Unfortunately, we still have local dependencies and in the current state things are not optimized for use over the internet. Step debugging was accomplished, but some work remains for embedded cloud development to compete with a local development environment. With this in mind, a future blog post will show we actually can be effective developers when using this solution with a virtualized QEMU target.

To do this proper we’d serve an MS DAP compatible debug server such as Probe.rs to the developers browser, and hook it up to the target board using webUSB. A challenge that exists, is the lack of open source microcontroller debuggers written in JavaScript or WebAssembly.  The translation from C code to WebAssembly is not straightforward and can be error-prone. However, valid translators of Rust to WebAssembly do exist, and Probe.rs is an open-source debugger written in Rust.  We also need to convince Microsoft to push this PR forward.

Stay tuned for a future post about how to build, run, and debug Golioth examples with QEMU in less than 10 clicks.

/0 Comments/by