A Remote Shell for Embedded IoT Devices

Have you ever wanted to directly interact with your IoT devices over SSH? Today we’ll talk about something similar, available to embedded devices on Golioth (with limitations).

This is a hack day project we did at Golioth to utilize Remote Procedure Calls (RPCs) in the Golioth Firmware SDK to send and receive data from an embedded cellular device. These RPCs directly interact with the Zephyr Shell. We then created a web frontend that can utilize the Golioth REST API to send commands down and see what is coming back on the shell. In the process, we can command the boards to do interesting things! See the video below for some examples.

How does it work?

The Golioth Firmware SDK supports multiple ecosystems and targets, but today I’m just going to be talking about our Zephyr port. It’s what I use most often for my own development and it’s what is enabled in this demo.

As we often write about on this blog, we love the Zephyr Shell. It’s a flexible tool that can be extended for a variety of capabilities depending on which KConfig symbols are selected at build time. And there are actually already different backends you can target (MQTT, Websocket, TELNET). The default is UART, which is the one I find myself using most often. But when you’re limited on available UARTs, RTT will work in a pinch and it’s easy to compile in that alternate KConfig symbol (CONFIG_SHELL_BACKEND_RTT).

In our case, we needed to create a custom backend and enable it with a custom KConfig (CONFIG_GOLIOTH_REMOTE_SHELL). In simple terms, we’re accepting input from the RPC, calling the shell, passing the resulting data to a buffer, and then passing that buffer to the Golioth RPC service. You send commands in, and receive output from something that looks like a UART…but might be thousands of miles away.

Custom shell commands extend this capability even further. I can add in gating logic and helper text to interact with pieces of my program over the shell. I showed an example of that in the video above, accepting inputs to the ostentus command, which is our library that talks to our custom ePaper front panel that we put onto Golioth Reference Designs. Now I can call that same shell command and push arbitrary text onto the display. It doesn’t have much interesting in the way of data coming back to the Remote Shell web interface, but it does highlight the ability to interact directly with a device or a user via that interface.

The Golioth Platform and Firmware SDK are the star players here. As mentioned above, there are existing shell backends like a WebSocket, but without a platform already talking to a fleet of devices, it remains a 1:1 tool. I’d need to somehow be able to reach out to a device that’s talking WebSockets and then coordinate that with which device I’m trying to find. The device management capabilities of Golioth are paramount for a widely available service.

The Frontend

The Golioth Remote Shell demo app is deployed to Netlify. You need three things to make it go:

• An API key for your Golioth account
• A project name you want to access within your account
• A device running the Remote Shell sample, which is built on RPCs and has a custom Shell backend

Once you are connected, you’ll see a dialog at the bottom showing the last connection time from the device to the cloud (using Golioth device metadata) and then you can type in a command like help and see the output from the shell returned to your browser-based interface:

Now you can interact with whichever shell commands are available on your device. I find the i2c, sensor, settings, and net shells to be particularly interesting. But check out this post about a range of Zephyr shells we like for IoT development.

Limitations of the Remote Shell

All of the testing I did with this was with an embedded cellular device. I normally target the nRF9160, but also I think it makes it more interesting as a demo. As a result, there are some delays. Even in a best case scenario, say the NXP RW612 connected over Ethernet, there are just more delays than going down the wire from a computer over a USB to UART port. So this won’t feel exactly like a shell. But well, it’s a remote shell!

RPCs also have some limitations. One is that they have a sensible timeout: 10 seconds, though it’s configurable both on the REST API and as a result, in the frontend we created. The more complex one to solve is that we don’t expect responses to be larger than a CoAP packet size, 1024 characters. Some shell commands have very verbose output, for instance help like shown above. Note that I said I enjoy the sensor shell…but it’s not actually being listed! That inspired me to create a simpler (less verbose) listing of all available commands called list:

Another somewhat obvious part of this remote shell…is that it’s not actually SSH! If I have access to SSH I can basically run an entire computing platform, up to the limits of that platform (ie. I need to be mindful of the onboard resources). The Zephyr shell was never intended to do the same amount of interaction, it’s really meant for debug and direct user input. If I want to pull in new capabilities of the shell, there isn’t a marketplace of standard applications like you can with Linux distros (for example: apt get). The variety in hardware basically requires that functionality be built for each bespoke platform using it.

However, it is possible to get new functionality! You can compile new functionality, send it out over Golioth’s OTA service, and get new functionality available over the remote shell. The main limitation would be things like the amount of available flash and RAM on your device, as running many many shells would be resource intensive. Pair the ability to use OTA update for each device with other Golioth functions like Settings, Stream / Pipelines, and State…and you have a true multitool for just about any problem you might want to solve with embedded.

Uses for a remote shell

Just to be clear, this is not needed for IoT operations. None of our customers implement something like this because it’s not particularly efficient, either for data transmission or for battery life. But I have always wanted a remote shell because it’s the way that I’m used to interacting with higher powered devices like a Raspberry Pi (over SSH). We also used to have this buried in our Firmware SDK, but it was available on so few devices that it wasn’t worth promoting. I always loved the idea of it though because of the flexibility. How else might you use a remote shell?

• Test – Run a self-test from the remote shell, even when a device is inside a temperature chamber during testing or fresh off the manufacturing line
• Troubleshoot – Scan for newly connected i2c devices and manually send commands using read/write capabilities
• Provision – Scan and connect to a new Wi-Fi network from a secondary connection like on the Elixir (has an nRF9160 and an ESP32-C3)
• Extend – Pull down linkable, loadable extensions (LLEXT subsystem) and dynamically load up code to work with different sensors that have been plugged in

The Golioth Remote Shell is not SSH, but if you really wanted to replicate some of the experiences, you could! We’d love to hear what you might use it for and how you’d like to interact with an embedded IoT device using the remote shell on our forum.