Exploring Golioth on Linux with Qualcomm

Recently, we teamed up with Qualcomm to showcase Golioth’s AI features. This demo stands out because we used the Qualcomm RB3 Gen 2 Development Kit running Linux. Staying true to our demo culture, we wanted to share how we pulled it off, what we learned about using Golioth with Linux, and where we might take this in the future.

Let’s dive in!

Wait, Golioth supports Linux??? 🐧

If you’ve been following us for a while, you probably know about our support for microcontrollers—from Zephyr tips to our cross-platform Firmware SDK. But you won’t find much mention of connecting Linux devices in our docs or blogs because we don’t officially support managing Linux-based devices. I say officially because we’ve actually had a Linux port for quite a while. It started as part of our CI testing infrastructure, helping us speed up tests on the Firmware SDK so that we can test more frequently than what you can do with physical devices.

Over the years we’ve received many requests to support Linux-based applications with a few different configurations. Sometimes a company was developing a product that had both an MCU and Linux Gateway (like a Thread Border Router) and wanted to manage the entire fleet with Golioth. Other customers were building a complex system that had both a MPU and MCU in the same device. And of course, many more are building a Linux edge-type device.

Since the scope of the Linux port was initially narrow, it was never designed to be a full “SDK”. Whenever a customer would ask if they could use the port for their embedded Linux device, we usually steered them away and pointed them to folks like Foundries.io or Balena.

Working with Foundries.io & AI Hub

We recently caught up with our friends at Foundries.io, who joined Qualcomm a few months ago, to see what they’ve been up to (we’ve collaborated in the past.) They were excited about some of Golioth’s new model management capabilities and connected us with the team from AI Hub (formally Tetra AI.) We discussed doing a joint demo together and Qualcomm wanted to highlight this new Linux-based device. Our mission is to be the universal connector for IoT so we were up for the challenge. After some brainstorming we got our hands on their latest devkit and got to work.

Getting the Firmware SDK up and running on Linux

Building the a Linux application that uses the Golioth Firmware SDK is as straightforward as building any other C program with CMake and requires minimal dependencies (see an example here). However, getting that application onto an embedded Linux device, and managing its lifecycle requires additional infrastructure.

Foundries.io is a perfect complement in this scenario, with existing support for the RB3 Gen2 devkit, and a simple container-based GitOps workflow for managing applications running on their Linux microPlatform (LmP) distro. Flashing the device with the LmP image, building an OCI image with a basic Golioth “Hello, World” application, and remotely deploying the application to the device only took minutes.

Leveraging the Hardware

The value of any embedded device is tied to how well the software is able to leverage the hardware, and the RB3 Gen 2 is an embarrassment of riches compared to the microcontrollers we usually interface with at Golioth. Based on the QCS6490 SoC, it includes a Kryo 670 CPU with 8 application processing cores, an Adreno 643L GPU, and a Hexagon DSP for accelerating AI workloads. Additionally, the RB3 Gen 2 Development Kit boasts a low and high resolution camera, as well as audio peripherals and an array of sensors.

AI Hub offers pre-tuned AI models optimized for Qualcomm hardware like the QCS6490, many of which leverage its robust image processing capabilities. Furthermore, Qualcomm provides the Intelligent Multimedia SDK (IM SDK), which includes a suite of GStreamer plugins that make it straightforward to access both peripherals and acceleration hardware. Combining these together with Golioth means that we can add connectivity to the equation, making it possible to stream data to the cloud, control processing remotely, and manage artifacts used in the processing pipeline.

Streaming Inference Results

We selected the YoloNAS model from AI Hub to perform object detection on the RB3 Gen2. The application constructed a GStreamer pipeline that pulled video from the high resolution camera, passed it to the model for inference, then composed the object detection results with the original video data in order to render a bounding box around objects before passing the final video to the display.

We also injected Golioth into the GStreamer pipeline, such that messages could be streamed to the cloud to notify when certain classes of objects were detected. As with all data streamed to Golioth, these messages could subsequently be routed to any other destination via Golioth Pipelines.

Remotely Controlling Image Processing

Outside of the GStreamer pipelines, we setup Golioth Remote Procedure Call (RPC) handlers that allowed for image processing and inference to be paused and resumed remotely. This functionality could be further extended to stream the current frame to an object storage destination via Golioth when processing is paused, all without requiring any physical intervention with the device.

Managing and Deploying AI Models

While Foundries.io handles application updates, being able to manage other types of artifacts used by applications, such as the AI models and labels, enables efficient updates without needing to rebuild and deploy. Integrating Golioth OTA into the application meant that the application was notified immediately when a new model was available, and was able to download and integrate it into the processing pipeline quickly.

Lessons and future explorations

We had set out to create Golioth application that would be useful on Linux and it was a success. We’ve proven to ourselves that Golioth’s services are useful for other IoT device types, especially embedded Linux, and that the Firmware SDK can work effectively in this context. Taking the code we developed we’ve already identified how we might evolve it into more of an agent or daemon and how we might better integrate with update mechanisms, especially on Yocto and Buildroot based distributions.

We’ll continue to explore the Linux for IoT space and see if and when it makes sense for us to do more here. Of course, you can count on us to continue to do more and more with MCUs. But we’re curious to hear from the community what they think – should Golioth invest in supporting Linux officially? What features or use cases would you like to see? Please share your thoughts on our forums!