A key mission at Golioth is to make it easier for hardware and firmware developers to connect devices to the internet. We do that in two ways:

  1. Providing easy-to-use APIs and SDKs for IoT devices to connect to Golioth Cloud endpoints.
  2. Training developers how to use the Device side code.

We have done many successful training sessions so far, showing individuals and companies how to connect their first devices. Along the way, we have learned that there is a large unfulfilled need in the market for training in the IoT space. So we’re doing it again! We’ll show people how to connect devices and get access to things like:

  • Secure Over-The-Air Updates to constrained devices
  • Command and Control over remote devices
  • Learn how to create and modify settings for remote devices
  • Understand how to implement data tracking from your device

We will be running our first training open to the public on December 14th, 2022. Read more below if you’d like to take part.

Training challenges

Once again we’ll be training developers from afar. We did this back in October for a select group of hardware engineers looking to learn more about Zephyr:

Click to learn more about our experience back in October of 2022

The upcoming training will be built upon the lessons we learned during that training, and our last in-person training at the Hackaday Superconference. In both cases, we used Kasm to provide fully remote development environments so that users don’t need to install anything on their local machine (there are directions on how to do that after the training is over). We think this is an important piece to ensure people can get started quickly.

How to use Zephyr

We currently offer 3 SDKs as part of our device support, including an ESP-IDF SDK, a Modus Toolbox SDK, and a Zephyr SDK (including the Nordic Connect SDK variant). These SDKs cover a wide range of embedded hardware from different vendors.

The training includes some segments that detail how to use Zephyr, a Real Time Operating System (RTOS) that covers a wide range of different hardware platforms. We use it on many of our internal hardware reference designs at Golioth, and it was the first platform we launched. Hardware and firmware engineers who are new to Real Time Operating Systems will continue their learning journey by understanding how the RTOS connects to sensors and low level GPIO and how to manipulate different elements of the subsystems. Once a trainee understands how to get the data off of an external component (like a sensor), the Golioth Zephyr SDK makes it a simple task to forward that data along to the Golioth Cloud.

Requirements and background

We have referred to “Hardware and Firmware Engineers” in this article, because we expect that intermediate to expert level engineers will get the most out of this training. If you are brand new to understanding C or if you have never tried programming embedded hardware before, this might be a frustrating experience. If you would like some pointers to starter content that might prepare you for the training, please ask on our Forum and we will try to get you a customized list of resources that will help prepare you for future versions of this training.

Logistics

  • We are not charging for this training
  • We will be capping the training at 30 people
  • All attendees will be on a first-come, first-served basis
  • Those who are accepted for this training will receive an email with more details
  • You will be expected to purchase your own hardware
    • Details will be sent with your acceptance to this training
    • Be sure you leave enough time for shipping from your local distributor
  • Signing up to take part and not attending will disqualify you from future training

Sign up here


If you have clicked “submit” and don’t see any changes, please scroll back up to the top

How the Golioth Developer Training paved the way

This is a guest post from Shrouk El-Attar, discussing her journey from the hardware space into the firmware space and how Golioth training has helped her understand building out IoT systems using Zephyr.

The Journey Started with Hardware

To me, hardware just makes sense! You have specific requirements, find a part that can fulfill them, read its datasheet, and then execute. Voila! Successful design achieved.

The worst thing about the process? I don’t know…maybe the Googling? Getting a calculation out by a factor of ten? The tedious BOM stock checking process? These things can be long and tiring, but still, the concepts make sense. Designing a PCB is easy to understand; you follow simple rules and let the copper tracks do the rest. Even things in the “RF Voodoo” territory make sense with well-developed RF design tools these days.

Not too long ago, I finally took the leap into the world of consulting as a self-proclaimed “Hardware Queen”. With that, I saw the terrifying rise in demand for the 2-in-1 engineering consultants: a hardware engineer who is also a firmware engineer.

The Firmware Roadblock

Firmware feels to me like hardware’s anarchist sister. I know her a little; we have some history. It might surprise you to know that she hasn’t always been this way with me.

Once upon a time, there was Arduino (I know, I know). I probably picked up my first Arduino in 2012. I’d heard about them long before then, but as an asylum seeker, I couldn’t afford to own one. That is until the UK officially recognised me as a refugee, and I could finally access higher education and buy my own shiny Arduino.

The first time I got to Blinky on my Arduino, it was like something clicked. I understood it. I understood every single bit of code I was writing on it. “I have a gift,” I thought. Perhaps learning Visual Basic (yup) back in 2007 laid down those basics for me to get to Blinky on Arduino and be a total boss at it.

From then on, I felt invincible. Whatever I could think of, I was able to make it. Self-watering plant? Check. NFC easter egg hunt? Check. Twitter-powered bubble machine? Check. Arduino felt like hardware and software meeting in perfect harmony in a world where anything I could think of was possible.

Shrouk’s Arduino Connected Plant: Light data is sent to cloud platform (left), phone notifications are sent when light data is critical (middle), plant “tweets” when light data reaches below a certain level (right).

My initial success was a sign I was going to be a firmware engineer. But soon after, my firmware dreams were shattered.

Stuck in the Firmware Valley

My first full-time engineering role was at Intel back in 2015, where I specialized in the then-brand-new field called IoT. “Can you imagine that by 2020 there will be 50 billion connected devices?” I thought to myself. I was so excited that I would be one of the people developing those devices. It was the future.

One of the first things I couldn’t wait to get my hands on was the all-shiny Intel Realsense 3D Camera. But when I downloaded my SDK, I didn’t know where to start. After days of struggling on my own, I could get some things working with the help of my senior colleagues. But none of it clicked. I had no idea what I was doing. I was not gifted. Frankly, I sucked at firmware.

I started specializing in hardware. As the years passed, some of my all-time favorite chips became the nRF5 MCUs by Nordic Semiconductor. What a hardware engineer’s dream. Can’t get to a specific pin during routing? Don’t worry; choose literally any other pin you can reach more easily, and re-route there! The nRF5 had an excellent reputation amongst my firmware colleagues too.

In 2019, I decided to give firmware another go on my favorite chip. I wasn’t too far down the toolchain setup, and I’d already started regretting it. What the heck were Make files?! It was an excruciating process. But–no exaggeration–one week later, I’d managed to set up the toolchain. Great, let’s get to Blinky! Save, build, flash. Error.

I spent hours, then finally figure out the error. Save, build, flash—another error. Repeat a zillion times. It turns out I had never set up the toolchain correctly in the first place. Months later, I finally found a training that would work, or so I thought. This turned out to be my biggest disappointment. Nothing in the training was up to date. Nothing worked the way it was supposed to. This was indeed the final time I’d try to touch firmware again. I was stuck. I was done.

The Golioth Way

Back in present day, as I was realizing the demand for the 2-in-1 hardware and firmware engineering consultant, a well timed post from Chris Gammell appeared on my feed. The post was about the Golioth Developer Training, targeted explicitly at hardware engineers. The 2-in-1 engineer could be me. Was I going to try firmware again? Absolutely.

The training took place entirely remotely with hardware engineers from all over the world. This is going to be it, for sure! But, from the very beginning, I was already struggling. “Here we go again”, I thought. I shouldn’t have thought I could do it. I struggled with minor details at first, like appending -d instead of -D to a build command, which apparently gives wildly different results. I copied and pasted the correct information but in the wrong fields. And each exercise took me way longer than the “estimated time” suggested for each section.

The training happened in a large group, but we did the exercises independently. Chris and Mike were checking in on each of us throughout, so they were able to help me fix anything I missed very quickly on a one-to-one basis. As the training went on, I started to notice that I needed their help less and less. What is this I feel? Some cautious optimism, perhaps?

The training method was active. I wasn’t sitting at my desk blindly following instructions. No, I was pushed to figure out the correct way to solve a problem at every stage. It was presented in bite-size information with many collapsed sections. The training also included links referencing concepts, e.g. Zephyr Pin Control, throughout, should I want to learn more. For the most part, I didn’t click those because I wanted to stay on track. My head was already full of too much firmware to learn. But I enjoyed that they were there so I could refer back to them in the future.

Then the happiest of accidents happened. I picked up a later stage of the training on my own, then realized that I had to redo the earlier parts to get to where I wanted to pick up from. Not going to lie; I was slightly annoyed. But while re-doing them, I saw that it wasn’t taking me very long at all. This time, I was well within the time estimates suggested for each exercise, if not much quicker. Was this working for me? Did I finally find a method to learn commercially viable firmware that works?

A light bulb went off when I realized that it took me over a week to get to Blinky on my previous firmware training, but within a couple of hours, I was already speaking to my Wi-Fi device through the Golioth Console. Not only that, but it was all through the Zephyr RTOS with Golioth. The same Zephyr RTOS I struggled endlessly with on the nRF5 hardware that I love. It was a miracle to think about being enabled for firmware on my favorite hardware platform. Coupled with the fact that Golioth is entirely scalable, i.e. the idea that I can start from an idea to production on the same platform, made Golioth a dream.

Golioth also helped me develop a community, some of whom I became close Discord friends with. We all need a Discord buddy to complain to about daily life as an electronics engineering consultant (Hi Seth!). And a decade after I picked up my first Arduino, Golioth Developer Training helped me finally break through that commercially viable firmware ceiling.

Next, I’ll work on reference designs and put the skills I learnt from the training to the test. Watch this space.

Editor note: If you’re interested in taking part as an individual or as a company, sign up for future training here.

Understanding hardware and firmware is the first step towards building any kind of “thing” that will be part of an “Internet of Things” (IoT) deployment. For new chips and firmware ecosystems, this often means going through training.

Golioth Developer Relations (the group I’m part of) is tasked with making sure our users understand how to get their hardware connected to the Golioth Cloud. We have been doing in-person (as safety protocols allow) and remote training for Golioth users.

Last week, we did our first training for the broader public, a group of hardware engineers interested in learning more about Zephyr and Golioth. I wanted to discuss some of the tools and methods we used to do a fully remote hardware training. If you’re interested in taking part in the training in the future, please let us know in the associated forum post or by email. There are more details on the available options at the end of this post.

How we do fully remote hardware training

The main challenge with virtual gatherings is that trainees are not sitting in the same room as the trainers. This is an easy problem to understand, but a harder one to solve.

Hardware

The first step was standardization on the hardware. To achieve that goal, we created a list of the bare minimum of parts and shipped them out to all attendees, even those who already had some of the hardware. We wanted to make sure that we had exactly the same hardware in hand. Our trainees were in the US and Europe, so we ordered through DigiKey, with DHL shipping at least a week in advance. The average cost was about $60 of materials in the US, and $85 to Europe, both numbers including shipping, VAT, tarriffs, etc.

For the hardware itself, we use the Adafruit MagTag. It contains the Espressif ESP32-S2 as the main processor on board. This time around we focused on Zephyr for this training, but the board also works great with the Golioth ESP-IDF SDK. Remember, Golioth has 3 SDKs currently (vanilla Zephyr, NCS, ESP-IDF), and we are always working to enable other hardware platforms.

Adafruit MagTag

Video meeting

You might think that after 3 years of a global pandemic, the world would have figured out video software. But the key difference between a simple video call and our remote training is that it’s not just 10 people sitting in a single video session. We want to be able to dynamically “move” between different conversations, especially as trainers.

To achieve this, we used Gather.town. It utilizes tiny avatars you can move around the screen with your keyboard like a video game. As your avatar gets closer to another avatar, your video and audio pops up to those in range. The rugs in the image below also allow “private” areas which work kind of like mini conference rooms. We found this allowed us to dynamically help smaller groups of people while also making announcements to the entire group. Trainees could also choose to work together on different components of the training material.

Our Gather.town Training Location for Hardware Developers

Tooling

The biggest challenge with any training is dealing with firmware tooling, specifically with Zephyr. We have been working on solving this issue for a while. The Zephyr installation continues to get simpler overall, thanks to the Zephyr dev team updating what gets installed for each user. But we still have trainees “walking through the door” (figurative in this case) with a wide range of computers where the tools will be installed. A range of different operating systems (Windows, Mac, Linux–with different distributions) means that there is no one standardized setup. What’s more, if someone has tried out Zephyr in the past, it’s possible there are interactions between the old installation and the new one, not to mention a variety of dependencies within each machine. If we could, we would ask everyone to show up with a brand new installation of Windows or Linux on a laptop…but that’s unlikely to happen.

We wrote in the past about using Kasm and Docker to have a standardized install. Last week’s training represents the first time we used it with real users. In short, it worked! We collaborated with the wonderful team at KasmWeb (makers of the Kasm tool) who gave us seats of their Cloud offering to use with our trainees.

Zero install embedded training with Zephyr using Kasm and Docker

The first step was created a dockerfile and a Docker image that contained everything required for the training. This way we were able to spin up a standardized, browser based UI environment for everyone taking the training. The entire Zephyr toolchain and Espressif support tools were pre-installed in that image, in addition to VScode, a terminal, a browser, and text editing tools.

Trainees open a browser window that looks like Ubuntu (it is, under the hood), and immediately start compiling Zephyr code. At past in-person training sessions, we had issues with bandwidth just downloading the tool packages. While we still needed bandwidth to communicate with the browser-based UI, all of the processing and storage is done on the cloud. Our plan is to use Kasm at our next in-person training.

We continue to evolve our strategy for making training a more seamless process and offering a range of different training to our customers.

Kasm instance of a Docker container with Zephyr / Espressif / Golioth tools installed

Training material

The final piece of the puzzle is the training material itself. Again, this is an evolving solution, because we continue to add more material, and because we track changes to the Zephyr SDK over time.

On Training.Golioth.io, we take people through getting signed up for Golioth, creating a first binary for the MagTag using their Golioth credentials, and then launch them into understanding different parts of Zephyr. We take developers all the way through working with the DeviceTree and understanding how to interact with an in-tree sensor in Zephyr. Our goal is to help hardware and firmware engineers understand how to use some of the most critical parts of an RTOS, and then use those features to communicate back to the Cloud, thereby making their device an “IoT” based device.

training.golioth.io

If you’re interested in trying the training without Golioth present, you can do the “asynchronous” version of our training. Training.Golioth.io is free to the public. You won’t have access to our Kasm server, but you will be able to install the Zephyr toolchain using supplemental directions. Please utilize the Golioth Forums for any questions you have about training.

Results

We scheduled a 2 hour training session, as we know that engineers’ time is valuable. In that time, 100% of the trainees were able to get their devices connected to the internet, pass data back and forth between Device and Cloud, and explore the features of Zephyr. As in any training, each person worked at a different pace and was able to get to different checkpoints. Each trainee could use their hardware and training site at home for later study.

An unfortunate side effect of having multiple tools, including training materials, browser based UI, and video software is a lot of tab/window switching to keep track of it all. In-person training will cut down on the video window, but it would be nice to further integrate the training to include all required materials in one place. We had a “browser within a browser” (Firefox installed within the Kasm container), but that consumed a lot of virtual memory and wasn’t an optimal solution.

The other major restriction is that the binaries our trainees compiled were not directly loaded onto the MagTag hardware. Because we worked inside a virtual (browser based) container, there was no direct connection to the hardware. We also talked about this in the past article about Kasm based development. Because virtualization tools are created to be independent from the hardware, it is difficult to then reverse course and say, “We want this container to talk to USB!”.

We are working with the Kasm team and looking at different ways to program binaries directly from inside the container, and would love to hear about other options people have seen. We solved this during the training by having trainees install the Espressif esptooldirectly on their machines. This still involved some amount of install, but it was limited to a small program. When a trainee completed a build, they could download the binary image from the container to their local machine and load it onto the MagTag using esptool. After that, the MagTag was able to talk back to Golioth.

Future training

We are excited to continue iterating on this training in the future and would love to hear from people that are interested in participating. Here are the options you can try:

It’s no secret that we like Zephyr Real Time Operating System (RTOS) around here. Nor is it a secret that we’re very interested in the Internet of Things (IoT). Golioth Founder and CEO Jonathan Beri gave a talk at the 2022 Zephyr Developer Summit (ZDS) about why those two things are a perfect match, and how Zephyr can help you create an IoT product faster.

A high level overview

Since this talk was on the first day of ZDS, it was focused on people less familiar with the Zephyr ecosystem. So what should a beginner know about Zephyr and its relationship to IoT devices?

Batteries are included

Zephyr has a couple of key features that makes it a “one stop shop” for building an IoT device. Some other RTOSes rely on the engineer to piece together libraries and implementations to get started, which either means more startup time or reliance on hardware vendors to make a ready-to-go solution for you. Instead, Zephyr provides individual elements already configured to work together:

  • Device drivers
  • An OS kernel (including scheduler)
  • Services
  • A build system

Tying together the ecosystem with these tools provides a consistent experience. This also leads to another very important aspect.

Chip vendor buy-in

Because Zephyr has a well defined system, the chip vendors develop code to make their parts work within the ecosystem. With other RTOSes, that script is flipped; the RTOS is made to work with abstraction layers within the vendor specific ecosystem, meaning there is less likelihood of interoperability with other vendor solutions.

In addition to the chipsets from the vendors, a wide range of boards are supported. Targeting specific boards makes it easier to get dev boards working when getting started, and remapping pins to specific functions on a board doesn’t require a configurator tool like those available in many vendor IDEs.

Talking to the Internet

If you choose an RTOS that isn’t a broader ecosystem, you need to either define your own network layers or lean on a vendor implementation to do it for you. In Zephyr, the network layer and the protocol definitions are done as a part of the community. That means that chip, module, and software vendors are all working towards a common implementation. Add in the fact that there are modems and abstraction layers all the way up the stack, and an engineer using Zephyr doesn’t need to think about all of the pieces it takes to connect to the internet. At Golioth, we are able to switch between various networking technologies easily when using Zephyr.

Protocol Variety

Golioth talks to the internet at a very high level, so network access “just works”. This comes into focus as a small embedded device can talk to the variety of internet protocols available. Most people understand HTTP, but also consider the more common IoT favorites like MQTT and CoAP. Golioth favors CoAP but also supports MQTT. This enables implementations in other parts of the network stack. A good example is devices running OpenThread, a Thread network protocol implementation. As we showed in our recent post about Thread, a small Zephyr-based device utilizes 6LoWPAN and talks over CoAP using UDP packets to talk back to the Golioth cloud.

Last but not least: Security

Zephyr enables a secure connection back to the internet through the network stack and with features like DTLS, the basis of a secure connection over UDP. Security is a deeper topic in Zephyr though, all the way down to secure bootloaders and working within things like TrustZone. Talking to secure elements requires drivers that understand how to communicate with particular chip features (internal or external to a microcontroller). At a very high level, Zephyr focuses on things like Software Bill of Materials (SBOM), so your security teams understand the various software you are pulling into the build.

Built for IoT

Zephyr not only ❤️ internet…it’s purpose built for enabling IoT devices. Paired with Golioth, a Zephyr device has a higher likelihood of getting to market and growing to a massive scale. We’d love to hear your thoughts about the video on our Forum, our Discord, or by email.

 

Google Summer of Code: Zephyr + Arduino

The Arduino ecosystem is ever-growing, and thanks to the excellent work of Dhruva Gole it’s coming to Zephyr as well.

For the 2022 Google Summer of Code with Golioth, Dhruva took on the challenge integrating the Arduino core with Zephyr RTOS as the base. Now the program is nearing completion and we’ve been excitedly building and running Arduino sketches that can directly call Zephyr libraries and subsystems. And of course, the cross-platform nature of Zephyr means there are suddenly many more boards that can run that Arduino code.

What is GSoC

Google Summer of Code is a program that focuses on bringing new contributions to Open Source. The program matches up participants (often students but not necessarily) with mentors to work on a specific goal previously accepted by the GSoC program.

Dhruva Gole finished his Bachelor of Technology in Electrical Engineering earlier this year. He has been working throughout the summer to build a Zephyr module that includes the Arduino core. This is the second GSoC program for Dhruva, who in 2021 worked on bringing Bela support to the BeagleBone AI platform. We’re also very happy to share the news that Dhruva began a job with Texas Instruments last month, congratulations!

It has been wonderful working with Dhruva, who has made a sizable contribution to the open source ecosystems of Zephyr and Arduino. At the same, as mentors we have been able to share a glimpse of how the Golioth engineering team operates. Finding our way through some hairy issues with the I2C system, mapping the Zephry Arduino header paradigm in DeviceTree, and getting the build system to work without extra command line arguments provided a great opportunity to work collectively.

What has been accomplished

Zephyr is a Real-Time Operating System (RTOS) that focuses on abstracting hardware functions so that the same code may be run on innumerable hardware platforms. This is an excellent underpinning for the Arduino code.

When successfully combined, Arduino sketches can be run on hardware that does not have an Arduino port. And at the same time, Zephyr features like multi-threading and the network stack become available to those Arduino sketches.

The milestones reached in the project thus far include:

  • Arduino Digital GPIO functions (e.g.: pinmode, digitalRead, digitalWrite)
  • Arduino time functions (e.g.: delay, millis)
  • I2C support via the Wire library API (works with external Arduino libraries)
  • A DeviceTree framework for mapping Arduino header pins to any Zephyr boards
  • Serial.println support in progress

How to use it

The project is organized as a Zephyr Module called Arduino-Zephyr-API. Any board that has Zephyr support can be defined in the variants directory of this module. This is accomplished by adding a DeviceTree overlay file (to map the pins of the board to the pin numbers from the Arduino ecosystem) and a pinmap header file which wraps Zephyr pin functions with the familiar Arduino API calls.

Using west, the Zephyr meta tool, Arduino sketches can be compiled and flashed to your target board. Beyond core functions, the addition of an RTOS opens up the ability to use multiple threads in Arduino. Dhruva has includes an excellent threading demonstration by blinking multiple LEDs asynchronously.

Rudimentary instructions for adding the module to your Zephyr installation are available in the project README. Documentation for adding boards is currently in progress and will be complete by the end of the GSoC program in September.

Challenges for future contributors

The heavy lifting of integrating the Arduino core with Zephyr has already been completed and is working well. But to use this, you must already have a Zephyr workspace installed (and know how to use it). Sure, Golioth has a quickstart that helps install Zephyr, but the ability to use this project from the Arduino IDE is desirable for existing Arduino users.

Unfortunately, there isn’t time left to work this level of integration. But contributions to the program are welcome and we hope that future contributors will take on this task.

There is also low-hanging fruit when it comes to implementing the most-used Arduino functions. For instance, random numbers, bits and bytes, and math functions should all be trivial to get working. More ambitious contributors may consider mapping the Zephyr ADC system to analogRead() and analogWrite().

TL;DR: we’ve enabled people to compile Zephyr programs from a computer with no toolchain installed, almost instantly.

Part of our charter at Golioth is to help people prototype and scale IoT devices faster. That’s why we offer an open source SDK built on top of Zephyr. We think this represents a “fast forward” or “cheat code” for quickly standing up an IoT device prototype. On the cloud side, our servers represent hundreds of hours of customization and testing; you can instantly connect and get access to resources that allow hardware and firmware developers to scale to thousands or millions of devices. But sometimes it can be scary to get started in a new ecosystem or Real Time Operating System (RTOS) like Zephyr, even if it will speed things up later. As such, we do public and private training for companies and individuals.

As part of the resources we offer, we maintain a Training site that walks people through how to get started using Zephyr, normally targeting remote training. You can follow along right now; you’ll need to purchase an Adafruit MagTag board and sign up for a free Dev Tier account, but everything else is covered on the training site. At the end of the training, you should understand how to interact with hardware in Zephyr and send data to and from the Golioth cloud over WiFi. It’s a short jump from there to re-target other hardware, including your custom designs.

The tripping points for the training often revolve around the installation process. This is multi-pronged:

  • The size of a Zephyr install is relatively large, even when you are only targeting a specific platform. Having multiple people in a room, even with good WiFi or network connectivity, means that the shared bandwidth will be a limiting factor. More trainees means slower downloads.
  • Everyone comes to training with a computer in a different state. They might have tried to install Zephyr tools in the past, or they might have a particularly rare Linux distro, or many other possible variations. It would be best if everyone showed up with a fresh OS install…but that is very unrealistic.
  • There are different expectations around how installations should go. Many embedded engineers are “Windows first” and expect a complete IDE for any new platform. Some silicon vendors help to support this in Zephyr, such as Nordic Semiconductor. But Zephyr was originally targeting Linux-based machines, and we have found the smoothest flow for installing tools for all of the platforms that Zephyr can target means you are Linux-first.

In this article, we’re going to talk about our attempt to normalize setups and have pre-installed tools using Kasm and Docker. These are not the only tools in this space; we have previously written about GitPod and are investigating GitHub Codespaces, but this is a look at one of the latest experiments we’re running at Golioth.

Kasm thin client

The concept of a browser based client or a “thin client” is nothing new. They were all the rage back in the day of time share servers (really those were “dumb terminals”) and then again in the 90s as computing was more ubiquitous throughout the office (with a centralized set of servers). The difference is that now things are much more graphical and running completely inside the browser.

Kasm was started in 2017 and includes an open source project run by Kasm Technologies. The company behind Kasm has a per seat licensing model or they will run the servers directly for you (once you’re past 5 trial seats). They specialize in visualizations around containers. Once you log into a Kasm server, you are able to launch a range of containers, normally a desktop view or a single app that will load up in your browser. You can try this for yourself on the Kasm demo page.

The server that we’re running on is a pre-configured image that I pulled from the Digital Ocean marketplace. I was able to install all of the required software on a provisioned server running in some unknown datacenter. All I did was log in the first time to get my credentials for a user and an admin, and the rest of my interaction was on the web interface that the Kasm server presents to me as an admin.

Docker

As a hardware engineer, Docker is one of those things I heard about for a long time and never really “got it”. I’m still not sure I do. But following the tutorial for customizing a Kasm container, I started to understand a bit more. In that set of tutorials I started from a base Operating System image (Ubuntu Focal) that allowed visualization through the browser. Then I was able to start customizing, adding things like custom files on the desktop, custom icons to launch programs I installed, or adding background images pulled in from the web. It was in this customization section that I could add all of the commands from the Golioth Docs for installing Zephyr tools.

My layman explanation of Docker would be “Creating a virtual computer where I can automatically install a bunch of software using shell scripts. Once I have built that virtual computer, I am able to use it over and over again, including different instances of that virtual computer (for this Kasm scenario)”. The analogy would be if I bought a bunch of laptops, had an install CD (remember those?) with all of the required software on them, and then I mailed the freshly installed laptop to everyone who is taking our training. Sound crazy? That’s one of the best solutions we have seen, where a trainer will bring a pelican case with 24 laptops freshly imaged to on-site training. Their training works flawlessly every time!

I don’t have much else to mention about Docker aside from the idea that it’s possible to script a bunch of install commands that match the install instructions we have on our Zephyr getting started guide. In fact, I used those very directions to build the container shown in the video above. So all I’m doing in this case is automating the install process, doing it once, and then deploying the container (with all of the software and dependencies installed) over and over again for different users.

Challenges

We don’t think this is the ultimate solution for our training, so much as an experiment that showcases what we can do with containerized solutions. There are some remaining challenges, and we would love to have some help from our community.

Loading firmware onto the device

Currently our plan (as shown in the video) is to have our users/trainees pull the final built binary to their local computer to run it on the device like the MagTag. This echoes the way the mbed online compiler worked.

If there is a bootloader and a USB to serial connection, it’s possible to directly load onto the embedded device. In the case of some Espressif boards, this would be something like having ESPtool.py installed locally on your machine. There are an increasing amount of tools that make this process easier, such as an ESP tool that allows you to load firmware using WebUSB. Certain specialized bootloaders like the one that comes default on the MagTag loads UF2 files. When the MagTag is plugged in over USB and a sequence of buttons are hit, the device shows up as a mass storage drive. You drop a UF2 formatted binary–which is just an alternative form of compiled format–onto the drive and the device reboots and starts running the code.

If it’s a board without a bootloader, the user would need to have a debugger and local tools to communicate with that debugger, such as a JLink device and JFlash software. This means they would still need some OS specific loader tools to get the binary into the embedded device. The user would not be able to take advantage of the built-in tools in west that allow direct loading onto the device.

You steppin’?

If you would like to do debugging instead of “printf/printk” debugging, you simply need to download a different file from the container. If you download the zephyr.elf file instead of the zephyr.bin file, you can load it into a 3rd party debugger like Segger Ozone (made by the same company as the JLink). We have done some experiments with this in the past, including also analyzing where the device is spending its time using SystemView. This would once again require installing local programs that could talk over the USB port to something like a JLink.

Experimental port forwarding and WebUSB

Some GDB debuggers/servers will host the control of the debugger over a port on the machine’s localhost. We have some experiments we’re trying where we forward this port to the container so we could directly run a debugger from a software debugger inside the container.

We have also heard some whispers of a WebUSB implementation that can tunnel to the container. So we could plug in a board on our host machine (ie. my laptop) and connect to it over WebUSB, and then forward all information along to the container machine (ie. the browser based desktop running on the Kasm server).

We would love to hear about other projects that are trying this.

Shared resources

The final challenge we are dealing with is the fact that we’re basically “renting” a computer to do exactly what we could be doing with the host machine sitting right in front of us. Most developers have access to very powerful machines and we are instead using the resources on a remote machine (the Kasm server). The cost of standardization is the cost of renting server time for each person in the workshop. It might be worth it, but it is a constraint and a challenge.

Containers are another tool

Anyone reading this with a web background is likely thinking, “Yeah, containers, cool, 2010 called and wants their headline back”. But we are excited about it because these tools are finally making their way into the historically sluggish embedded industry. While our use case of containers is mostly around zero-install-time training, others are using containers to automate their testing and implementing best software engineering practices for the range of devices they have on their desk or in the field.

We’d love to hear how you think we can improve our training and make it easier for you to learn more about Golioth, Zephyr, and building code instantly. Check out our forums, our Discord, ping us on Twitter, or send us an email at [email protected]

Hello from ZDS!

This week the Golioth team is at Zephyr Developer Summit. Previously we announced that we’ll be there and shared the talks we are presenting. We will post those shortly after the conference takes place. In the meantime, let’s recap how we got here in the first place and share a little bit more about what we’re showcasing.

Why Zephyr?

In short, because it helps our users. We are members of the Technical Steering Committee (TSC) and have been almost since the inception of the company. We built our first Device SDK on top of Zephyr because of the broad hardware support and high level integration of Golioth features into the Real Time Operating System (RTOS).

The assertion that “Zephyr helps our users” might be extra frustrating to beginners: Zephyr—and RTOSes more broadly—represents a tricky new set of skills that might be foreign to firmware or hardware engineers. For beginners coming from the hobby space, it can be an extra rude introduction into the world of command line compilation and large ecosystem. However, connecting to the internet is a difficult task, especially for custom hardware: we think that Zephyr represents a great first step towards managing those devices over time. We are committed to pushing for more user-friendly code and methods from the Zephyr foundation, and we will continue to publish best practices on our blog and our YouTube channel to help people get connected.

Showcase

One thing we’re excited about is showcasing how Golioth works to members of the community. We have been developing different “color coded” demos to make them a bit more memorable for folks that stop by our booth. Each of these demos feature a hardware (device) component and a dashboard component, in order to visualize the data that is on the Golioth Cloud.

This is the first time we have showcased the “Aludel”, which is our internal platform for prototyping ideas and switching out different development boards and plug-in sensors. We will post more about this in the future, including our talk on the subject.

Red Demo

The Red Demo is our showcase of devices running OpenThread on Zephyr; this is part of our larger interest in Thread, which we see as a very interesting way to connect a large range of sensors to the internet securely. We have been excited to show how we can use low power devices like the Nordic nRF52840 to communicate directly with the Golioth Cloud.

The devices we are using in this case are off-the-shelf multi-sensor nodes from Laird called the BT510. This hardware has additional sensors on the board which we integrated with LightDB Stream to send time-series data back to Golioth. This was fast work, thanks to Laird’s Zephyr support, it was as simple as calling out the board when we compiled the demo firmware.

We then capture the data from these on the Red Demo Dashboard, showing both historical and live data for the sensors.

 

Green Demo

The Green Demo showcases LightDB State, our real-time database that can be used to control a wide range of devices in a deployment. On the device side, it uses the Aludel platform to measure a light sensor, as would happen in a greenhouse. There is also a secondary Zephyr-based device inside a lamp, representing a grow light that might be inside a grow house. The lamp is set up to “listen” to commands from another node, in this case the Aludel.

LightDB State is used to control elements like “update rate” to control regulate flow of information. It also lets us monitor critical device variables on an ongoing basis and set up logic on the web to take actions as a result. Command and control variables can be set from multiple places, including a custom mobile app, the Golioth Console, a visualization platform, a web page, or (as is the case here) even from another device!

Our Green Demo Dashboard (below) again showcases live and historical information, as well as the current status of the connected lamp.

As an added bonus, we control some of the logic on the back end from a Node-RED instance, including control logic. That takes the light intensity sensor output and calculates how bright the lamp should be. Because this is written in Node-RED, we can include an additional input from a mobile app to control the “target intensity”. In this way, people at the booth can adjust the lamp output if the exhibition space is brighter or darker. Plus…it looks cool!

Blue Demo

The Blue Demo helps to showcase how data migrates into and out of Golioth. Using Output Streams, you can export all cloud events to 3rd party providers like AWS, Azure, and Google Cloud. Buttons on the Blue faceplate switch the output being sent back to the cloud. The sensor readings being exported to all 3 clouds can be turned on or off by changing which variables are exported from the device.

On the device side, we capture a sensor using our Aludel platform. The sensor is a BME280 (in-tree sensor in Zephyr), going through a feather form-factor dev board, talking to the network over a WizNet W5500 Ethernet external chip to the network. The Blue Demo Dashboard showcases the live data, and of course the data is being exported simultaneously to the 3 cloud platforms in real-time.

Orange Demo

Golioth is a “middleware” built on top of Zephyr RTOS, which means you can use it to implement new features on top of already-existing hardware. This demo uses the Nordic Semiconductor Thingy91 with custom firmware to send GPS data back over the cellular network to Golioth using LightDB Stream. This demo also has Golioth Logging and Device Firmware Update, which are easy to add to any project as an additional service for troubleshooting or in-field updates.

On the dashboard side, we wanted different ways to showcase this data, including “latest update”. Having access to the raw data is useful for anyone wanting to try asset tracking applications. We’re excited to be able to showcase this data as it dynamically flows into the Golioth Console and back out to the Grafana dashboard.

Future showcases

We’re excited to be showcasing our demos at the Zephyr Developer Summit, but these are moving targets! We will continue to update and pull in new feature for future events. We will be at Embedded World in two weeks (June 20-24th) and will have many of the same demos there.

The Zephyr Developer Summit (ZDS) is coming up June 7th-9th 2022 in Mountain View California at the Computer History Museum. Golioth will be there and we’re very excited to interact with fellow users, developers, and stakeholders in the open source real-time operating system (RTOS) known as Zephyr!

We love Zephyr

People reading this blog will not be surprised to know that we love Zephyr. We write about it quite often and it is the basis of our Zephyr-based SDK. As a result, many of our samples and demos are built using Zephyr. We often talk about Zephyr being an indicator that a hardware device will work with Golioth; all you need is a network connection, a board running Zephyr, and little bit of storage overhead to hold the Golioth code. It’s the hardware interoperability of Zephyr that allows Golioth users to target a wide range of platforms, including microcontrollers from Espressif, Infineon, Intel, Microchip, NXP, Nordic Semiconductor…and more being added every day!

Our plans at ZDS

We’re excited to be returning to ZDS. Last year we officially announced Golioth to the world at ZDS, and talked about how our platform works within the Zephyr ecosystem. We hope to have another year of connection, this time in person and online. Let’s look at how we’ll be participating.

Sponsoring/Showcase

We are helping to sponsor ZDS this year. We believe in the mission of the project and the conference and wanted to be part of it. We will also be showcasing Golioth at a vendor table at the conference. If you would like to see Golioth in action, you can stop by at any time to ask questions and see demos. You can, of course, also try out Golioth at any time using our Dev Tier plan, which gives anyone up to 50 free devices on the platform.

Giving Talks

We will be presenting a range of talks at ZDS:

  • What chip shortage? How we use Zephyr for truly modular hardware
    • Chris and Mike from Developer Relations will highlight the Aludel, an internal hardware platform we’ve built as a customizable solution that can switch out hardware pieces without major redesign. This modular hardware showcases a path for hardware and firmware teams to unify their codebase using Zephyr while targeting a wide range of hardware. Being able to swap out a sensor, microcontroller, or radio but keep the main board, or go from outdoor air monitoring to indoor monitoring is really powerful. Zephyr makes it much easier to create alternate builds and manage firmware pipelines to hardware variants.
  • Connecting Zephyr Logging to the Cloud over Constrained Channels
    • Our resident Zephyr expert Marcin will cover an approach to preparing Zephyr logging messages for transmission through a constrained networking layer, such as a cellular connection. This includes CBOR compression on all logging messages, including special handling around binary payloads. There is also an interface to a CoAP library to take advantage of smaller payloads and standardized format to a cloud backend. Additional tooling is included for selectable acknowledgement of messages, to handle high priority and high traffic scenarios.
  • Zephyr <3 Internet: How Zephyr speeds implementation for new IoT devices
    • I (Jonathan, CEO) will make a case to people outside of the Zephyr ecosystem on why they should adopt the platform and contrast the difficulties to other RTOS solutions. These networking concepts are so baked-in that it fundamentally changes the cost for anyone buying into the ecosystem. From vendors adding modems to developers building apps, the underlying framework saves time and engineering complexity.
  • End-to-end IoT development with Zephyr
    • Founding engineer Alvaro will cover the options for getting a Zephyr app connected (WiFi, Ethernet, Cellular), selecting the right data encoding (JSON/CBOR), securing the data transfer (DTLS/TLS), and choosing a protocol (HTTP/MQTT/COAP). But that’s not the end of the story, the cloud needs to manage devices allowed to connect, consume the data being received, open up options for using that data, and be aware of the continued state of the hardware. And once you have the data you need to build a user-facing application on top of it.

Giving a workshop

Hands-on demos are a critical part of understanding a new system. This is true of both Zephyr and of Golioth. We wanted to showcase how Golioth works to Zephyr users, while also helping people get a real piece of hardware talking to the cloud. We’re giving a workshop called “Hands-on with Zephyr-based IoT Hardware – Data Goes in, Data Comes Out, Data Goes Up“.  This is a hands-on developer training showing how to get a finished piece of hardware utilizing the various features that Zephyr has to offer. The main thrust of the training is getting up and running with the Zephyr toolchain, implementing examples on a piece of hardware (provided), and interacting with cloud services. The user will learn about various abstraction layers around things like CoAP and CBOR, and experience a real world example of a smart device talking back to the Golioth Cloud. This will also expose the user to web-side technologies and how they can export data to external commercial services like AWS, Azure, and GCP.

Meeting with users and partners

We love our community and are always looking to meet new people within it. Interested in setting up a time to discuss something? Email [email protected]

Should you attend?

If you’re a someone already developing for Zephyr and pushing code upstream, this is the best opportunity to meet with others from the community and continue to build your skills. We think this is a perfect event for you!

If you’re new to Zephyr, the content can seem a bit intimidating…but fear not! The first half day of the conference (June 7th starting at 1 pm) is the “Intro to Zephyr” day, and this is a great introduction to the platform and how you can build your skills using Zephyr. There are also reduced cost tickets for students, if you’re still learning. We think if you’re looking to build a product with Zephyr in the future, or already are building with Zephyr, it’s a worthwhile experience to be there.

See you there!

We’re excited to meet more people and hear the other great talks that will be happening at the 2022 Zephyr Developer Summit. While we definitely plan to share the talks after the fact, and you can also participate in the virtual conference, we still hope to see you there!

This guest post is contributed by Asgeir Stavik Hustad, a Golioth community member who is active on the Golioth Discord. Reach him on Twitter at @AsgeirSH.

This tutorial was inspired by and a response to the tutorial about how to build your Zephyr application in a standalone folder. I have done exactly that before, but I also wanted to include all my dependencies in that separate folder.

Background and motivation

I need to maintain different firmware with different Zephyr versions and trees. For example, I maintain the following directories:

  • Nordic’s Zephyr-variant (NCS) for the nRF9160
  • Base Zephyr for Atmel-MCUs
  • Base Zephyr, but locked to a particular version (ie. “2.7.0”)

We also have several custom boards. These are currently maintained in each project, but could be moved to a separate dependency if we want to use the same board overlay files in multiple projects.

Instead of trying to swap a single Zephyr-installation between all of these, I did some research into using west and its manifest file to automatically set up my project folders to include all dependencies. I also wanted to ensure our build server didn’t require any manual work to build different projects. The Zephyr docs present this topic in depth, and are recommended reading if you want to set this up.

Let’s look at how we can set up a project to fit a wider range of needs.

Project structure

Most of my projects are kept in my “Dev” folder, so for this example we’ll be using ~/Dev/app_zephyr as the root directory of the project.

I put my application source in application, which is further split into at least boards and src (you can add any folder you like here). You’ll note this is the same structure as any of the Zephyr or Golioth samples you see; in fact, you can copy a sample as the starting point (such as <Zephyr SDK Install location>/zephyr/samples/basic/blinky). The other folders include deps for dependencies and build for the build output folder.

Inside the root folder, add .west/config. This is a plain text file describing to west where it should look for the manifest file and where Zephyr should be placed.

[manifest]
path = application
[zephyr]
base = deps/zephyr

Drawbacks

  • The initial clone and west update of a project set up like this takes some time.
  • This method uses quite a bit of disk space because each project carries around the Zephyr dependencies, as opposed to having your application live within the Zephyr SDK.
  • Ensuring you get updates to all your projects means you need to update the projects in your manifest file to a new revision manually (not really a drawback in my eyes – I want control!)

Let’s go through the manifest-file itself step by step. It’s found in application/west.yml:

manifest:
  version: 0.7

  defaults:
    remote: zephyrproject
  remotes:
    - name: zephyrproject
      url-base: https://github.com/zephyrproject-rtos
    - name: mcutools
      url-base: https://github.com/mcu-tools
  projects:
    - name: zephyr
      repo-path: zephyr
      revision: v2.7.0
      import:
        path-prefix: deps
      path: zephyr
    - name: mcuboot
      remote: mcutools
      repo-path: mcuboot
      revision: v1.7.2
      path: deps/mcuboot

  self:
    path: application
  • The manifest version being set to 0.7 simply means west (the meta tool) must be at version 0.7 or higher to parse this correctly.
  • Default attributes for the project are not required, but in this case lists the main remote.
  • Remotes lists where west should look for project repos.
  • Projects lists the full range of repositories we’ll pull in as dependencies. This includes the revision, so we have control over upgrades when they are available. I want to prevent breaking changes from entering my project without my knowledge.
  • The self: path: application is used to further define where in the project tree the west.yml file is compared to the root of the project.

I feel that the projects key is the true turning point of this manifest. By adding to this we can make Zephyr pull any git repository we want, and put it in our dependencies-folder. We specify the project name, a remote (if different from the specified default), a repo-path on the remote (defaults to name), a revision (defaults to master) and a local path (with a slight footnote for this one).

There is a special key here as well that makes this work, import will make west import the projects from the manifest file of that project as well. This means that when running west update on this manifest, west will first clone all projects in this manifest, then run west update on the manifest file in the specified project, and clone all projects from that, with the specified path-prefix for all those. I’ve used this for the Zephyr-include, but not for the mcutools.

Build

In practice, this means that my project structure for the manifest file above after running west update will look like this:

- app_zephyr/
    - .west/
        - config
    - application/
        - boards/
            - arm/
                - ah_1202a/
        - src/
        - CMakeLists.txt
        - prj.conf
        - west.yml
    - deps/
        - mcuboot/
        - modules/
        - tools/
        - zephyr/
    - .gitignore

Your custom board *.dts-files can include all the root overlays from the Zephyr-dependency or any other projects. (I’ve also set this up so VS Code can do IntelliSense of these DTS files, that’s just a matter of setting the correct includePaths.)

From here, you can run west build and have it use your custom board-files, source and everything. In my case:

cd application
west build -p -b ah_1202a -d ../build

Revision Control

One of the benefits of a method like this is the reduced amount of files going into revision control. You don’t need to index all of the Zephyr directory files in your project repo. This is a bad idea anyway, given the size of the project and the almost certain guarantee they will be out of date the next time you pull your project. Locking the Zephyr version in west.yml will ensure that your project is always pulling from the expected version of an SDK or Zephyr repo. Adding a .gitignore file as shown below to your main repository will reduce your total footprint and only capture the unique elements of your project–your application code.

deps/
build/
.vscode/

Build it your way

The first step to building an optimal workflow for your company or personal development process is understanding how your build system works. The above method is far from the only way of doing things, but helps to give more precise control over what is tracked and what is pulled in from external sources.

This guest post is contributed by Ben Mawbey, a community member who is active on the Golioth Discord and frequently takes part in Office Hours.

Data wants to be visualized. The impact of showing a customer a slick plot of the information their devices have been collecting is massive compared to pointing at a few hundred lines of text from a log file or database query.

I was looking for some sort of dashboard or charting application for a demo for a sensor system we’ve built. I figured this will appease the clients’ need for pretty pictures over boring reports. I found exactly what I needed, and it only took me about 30 minutes to get it up and running.

Grafana graphing data

This looks great, even if you have no idea what the graphs mean! Let’s dig into how to get from numbers in a database to pretty pictures in a dashboard.

Pieces of the Puzzle: Golioth WebSockets, Node-RED, InfluxDB, and Grafana

Golioth is brilliant at getting your device data from the real world up to the cloud, climbing the IoT bean-stalk some would say. While abstracting some of the trickiest IoT problems, Golioth can present your time-series data as a convenient cloud resource using LightDB Stream service. By leveraging the built-in WebSockets support and some open-source tools we can rapidly store, manipulate and display this data!

Grafana is the open-source blockbuster for this application and can be easily set up to graph sensor data directly from Golioth LightDB Stream using the REST API. This has two major drawbacks:

  • Grafana must periodically poll for new data
  • While LightDB Stream does provide convenient data retention, I prefer to use my own data storage

Enter InfluxDB, the time-series database powerhouse and an ideal companion to Grafana when talking about IoT real-time applications. This pairing is so popular that the InfluxDB data source integration is baked right into Grafana! By utilizing InfluxDB to store our sensor data, we can perform more complex queries much faster.

The question remaining is how to shuffle data from Golioth to InfluxDB? There are many potential solutions to this hurdle, but my favorite is Node-RED which defines itself as a low code programming tool for wiring up event-driven systems.

Node-RED editor window

Node-RED uses graphical flows to connect data sources and destinations, bending protocols and translating data formats in innumerable ways

Node-RED has exploded in popularity and provides all sorts of integrations to connect your systems together. It provides simple blocks to perform actions and a slick graphical interface to wire up your data flows. Conceptually Node-RED acts as our rule engine to process and direct data.

Dashboards: Grafana and InfluxDB

System diagram

Grafana is immensely powerful at providing example custom views, data transformations, and alerting. That said, it is only as good as the quality of data you can provide. Having a tightly coupled InfluxDB instance with carefully curated data via Node-RED, allows you to quickly configure complex data queries on large datasets with low latency.

Before we can play with Grafana the first step is InfluxDB setup. After you’ve installed InfluxDB, create a new database on your InfluxDB instance:

> CREATE DATABASE golioth

Configure the InfluxDB Data Source in your Grafana instance by clicking the gear icon on the left sidebar, choosing and adding a data source, then searching for the InfluxDB plugin. Here’s how I’ve set up my data source:

Grafana InfluxDB source configuration

Assuming you have some data in your DB, we can quickly create a new time-series dashboard panel in Grafana and query the dataset using this integration:

Grafana panel configuration

This simple query shows how we structured the data in the DB, allowing us to select from a particular measurement, specifying a specific device identity tag and aggregating data points with specific time buckets. Adjusting the time range instantly updates the graph from our local InfluxDB instance.

Now how do we get the data from Golioth into InfluxDB?

Node-RED

Several networking integrations are provided with Node-RED. Presently the most relevant to Golioth would be HTTP nodes for REST API requests and the WebSockets node which is the easiest to configure.

You can see your sensor data collecting in the Golioth LightDB Stream by using the Golioth Console web interface:

Golioth Console showing LightDB stream data

We can use Node-RED flows to connect to Golioth via WebSockets and store the resulting data in our local DB:

NodeRED editor window

The nodes in this flow were set up as follows, taking care to give them appropriate names making it obvious to see their function. All of the nodes I’m using should come as part of every Node-RED installation except for the InfluxDB nodes. But don’t worry, these are trivial to install. On Linux is looks something like this:

cd ~/.node-red
npm install node-red-contrib-influxdb
sudo systemctl restart nodered.service

WebSockets Node:Node-RED websockets

First set up the credentials to your Golioth project using your generated API key and connect to the WebSockets LightDB Stream endpoint.

Debug Node:Node-RED debug

Drop a few of these nodes along the way and click the small green button to turn the debug log on. This is super handy to check data coming through and make sense of it.

JSON Node:Node-RED json

The LightDB Stream endpoint provides us with a JSON object representation containing our sensor data as well as meta information such as the data timestamp and device identity. This node allows us to parse this JSON into a javascript object so that we can work with it more easily in subsequent nodes.

Change Node:Node-RED change node

This node clearly shows the power of Node-RED as we can craft any sort of data manipulations or transformation.

We could do without this node and jump straight to InfluxDB, however, should any malformed data arrive we risk polluting the DB with bad data. By selectively transforming the incoming data and mapping it into a new object we can not only filter only good packets and arrange the measurement names but also add tags to build a solid data representation in the DB making our queries far more powerful.

InfluxDB Out Node:

Node-RED InfluxDB input nodeNode-RED InfluxDB server settings window

Finally, we can configure the data connection to our InfluxDB instance, set appropriately for your server configuration and database created earlier.

Assuming your flow is set up correctly, you should be able to see data collecting in your database. We know it works, but as I mentioned before, this visual is not going to impress our customers.

InfluxDB data

Revisiting the Grafana panel we previously created, you can see InfluxDB data is now being plotted!

Grafana graphing data

Corner Cases

One of the downsides of WebSockets is their ethereal nature, should there be some temporary connectivity issue, any data packets would be lost from the point of view of your InfluxDB database. A solution around this could be to set up another flow that executes periodically to sync with the LightDB Stream using the REST API. Node-RED could then be configured to check this data and add any missing values into the InfluxDB instance and prevent consistency issues.

Another concern with open-source self-hosted systems is security. It can be challenging to secure your server and services should they be public-facing. If you are handling sensitive data then it would be best to consult with an expert in this field. Fortunately, all of the tools discussed have subscription-based cloud services available that sort all of this out in the background.

Conclusion

Being able to set up a simple demo like this in less than 30 minutes demonstrates the power and flexibility of these modern open-source solutions. Coupled with the reliability and maintenance advantages of the Docker system, it’s a breeze to test locally on your desktop or Raspberry Pi and then deploy to production a moment later on your cloud server of choice. The rules engine and ease of wiring up blocks provided by Node-RED opens up a massive pool of possibilities, from countless other integrations to building intelligent processes. One such idea I would like to explore is integrating a device provisioning process into the flow such that we can link a device to a dataset or location during deployment or maintenance.