,

Thread network device management using Golioth

The following post was written about Thread in April 2022. We are leaving it here for historical purposes, but we suggest that you check out some of our more recent work with Thread on the Golioth Reference Design template and using commercially available Thread Border Routers.

Golioth is a flexible device management solution that allows you to manage devices that have IP addresses. This includes devices connecting over Ethernet, cellular, and Wi-Fi. Today we’re showcasing a demo of a Thread-based device. Utilizing the Golioth Console and API endpoints, it’s possible to manage a wide array of devices provisioned on the internet through a Thread Border Router. This enables even more low-power devices to reliably push data back to the internet for processing on the cloud.

In this post and the associated video, we show a demo of how developers can try out using Golioth to manage Thread based devices.

What is Thread?

Thread is a network protocol for low power IoT devices. It uses 6LoWPAN for mesh communications, in our case on 2.4GHz. In the examples above, we showcase Thread using OpenThread on an nRF52840 from Nordic Semiconductor. Multiple hardware vendors have Thread-based solutions, using Zephyr or other firmware solutions. Vit Prajzler shows how he is using the OpenThread documentation to build different types of devices that form a Thread mesh network, including a Radio Co-Processor (RCP) and a Full Thread Device (FTD).

Set up a Border Router

A Border Router is an element on a Thread Mesh network that allows the entire network to communicate with the broader internet. Each individual device (Node) has an IPv6 address, which allows Golioth to manage specific devices on a mesh.

While it’s possible to buy a few off-the-shelf Border Routers, we showcase building one using commonly available components. Vit is using a Linux computer (a Raspberry Pi), along with an nRF52840 dongle configured as an RCP to route network traffic from the mesh to the rest of the internet. In the case of his Thread network, Vit needs to include a translation layer (called NAT64) to talk from his IPv4-based router to the IPv6-based Thread network. In the future, when Vit gets an IPv6 address assigned to his home network from his ISP, no conversion will be required.

OpenThread software on a Linux device like a Raspberry Pi also has a web interface for managing connected devices. This is a useful way to visualize your network and the location of the Border Router within that network.

Using the Golioth interface in the Zephyr SDK

Once your Border Router has been set up, you can use the Golioth Zephyr SDK to compile an image for your device using OpenThread. In the video above, Vit is compiling for an nRF52840 (dongle). Once this device is loaded with Zephyr-based firmware and added to the mesh network–in this example, connecting directly to the Border Router–the device acts like any other IP connected device. It has an IPv6 address and can ping the address of the Border Router. It can also connect to various Golioth endpoints.

Vit demonstrates button presses and boot commands being logged in Zephyr, which then display over the UART. However, Zephyr logs are also tied to the Golioth logging end point (from the Thread network connection), so they can be viewed on the Golioth web console. As Vit presses a button on the Zephyr based device, the log message almost instantaneously shows up on the cloud.

Extending the examples

Towards the end of the video, Chris and Vit discuss how (Open)Thread-based devices built with Zephyr work very similarly to any other IP-based devices connecting to Golioth. This means that the other features that Golioth provides–such as control using LightDB State, or easy time series data tracking using LightDB Stream–is already in there. And yes, that means Firmware Updates as well. We’ll be showcasing these more in future videos.

We’re excited to bring Golioth and internet connectivity to even more devices, including the kind of teeny tiny low power ones that Thread enables. Please stop by the Golioth Discord or the Golioth Forums to ask questions and discuss the next device you’re working on!

Interested in Thread in a commercial application? Contact us at

 

/by
,

Golioth and Datacake enable device data visualization with a few clicks

Datacake makes beautiful dashboards for your IoT data. Golioth delivers the tools you need to manage your IoT hardware fleets. What if the two platforms worked well together? They do!

We’ve partnered with Datacake for an integration that easily connects your device data to create visually stunning presentations.

Demo

You’ll have no problem packing all kinds of display data into one computer screen with this setup. But one added bonus when working with Datacake is that you can generate publicly viewable links that include a dashboard optimized for mobile viewing. Think of sending your prospective customers a link to a sleek dashboard with live data from the proof-of-concept hardware you just bootstrapped.

Our demo today uses the temperature, humidity, and pressure data from a BME280 sensor, but the pattern is easily adapted to any visualization you need.

Setting up a Datacake Connection

Golioth and Datacake have integrations in place to communicate between the two platforms, so there are only a few steps to complete setup:

  1. Find your Golioth device ID
  2. Add a device on Datacake
  3. Use the Datacake endpoint to set up a Golioth output stream
  4. Select the specific data to visualize on your Datacake dashboard

For this demo, the weather-sensor sample code was used to stream temperature, humidity, and pressure. This will send real data from a BME280 sensor, or send simulated data if a sensor is not available.

Step 1: Golioth device ID

The prerequisite for this step is to have a device set up on Golioth to stream data. We have a free Dev Tier, so if you haven’t done so yet, register your account and set up a device.

Golioth Device ID

In the Golioth Console, select Devices from the left sidebar and click on your device in the resulting list. In the device view, as shown above, copy the Id, which we’ll need when adding a device in Datacake.

Step 2: Add a device in Datacake

The prerequisite for this step is to have a Datacake account. Your first two devices are free, so to give this guide a try, sign up for Datacake.

Add a device in Datacake

Once you have logged in, choose Devices from the left sidebar menu, then click the blue Add Device button in the upper right. As you can see above, you first click the API button at the top, select the New Product from template tile, and select Golioth.io from the list. Scroll down and click the blue Next button.

Datacake add device id and name

The resulting window asks us to enter a Serial Number, this the device ID we copied from the Golioth console in the previous step. (NOTE: double check that there are no spaces before or after your device ID.) Give this device a Name that makes sense to you. After clicking the Next button you will be asked to choose a plan for this device. There is a free plan option.

Datacake endpoint URL

You will be returned to the Device page. Click on the name of your newly created device and choose the Configuration tab. Scroll about half way down the page until you see the HTTP Endpoint URL. We need to give this to Golioth to authorize the output stream.

Step 3: Configure the Golioth Output Stream for Datacake

Back in the Golioth Console, select Output Streams from left sidebar menu, then select Datacake from the resulting list.

Golioth output stream configuration for Datacake

The Name field can be anything of your choosing. Paste the endpoint URL from Datacake into the HTTP Endpoint URL box as shown above, and click save.

Golioth Output Stream test

It’s a good idea to test the Webhook connection to Datacake that we just created to make sure it is working. After clicking save on the Webhook, choose the Custom Test tab in the window that appeared, and click Send Test Event.

Datacake show logs

Back in the configuration page of your Datacake device you will see a Show logs button in the right side of the HTTP Payload Decoder section. When you click it, a dialog opens. Wait a few seconds and you will see our {"data":{"hello":"world"},… test message appear. Success!

Step 4: Choose the data to show on your dashboard

Datacake sets up a default dashboard for Golioth data to display temperature, humidity, and pressure data. It expects data packets to stream in that look like the code below.

{
  "data": {
    "environment": {
      "humidity": 37.325195,
      "press": 98.508472,
      "temp": 23.46
    }
  },
  "device_id": "622fb52aa50e6c807d269a31",
  "project_id": "datacake-demo",
  "timestamp": {
    "nanos": 206247513,
    "seconds": 1647879865
  }
}

You can customize the Datacake interface to use the data of your choosing. In your device’s configuration tab, scroll down to the HTTP Payload Decoder section.

Important here is the var data assignment that you see highlighted in the screenshot. This should map to the object in your JSON data that holds the key/value pairs you wish to access. Look a few lines lower (44, 49, and 54 in the Datacake example) and you will see the values being mapped to field names for graphing on Datacake.

There is a Golioth specific guide in the Datacake documentation that goes into detail about these field values, and how to further test the decoder set up.

If you have been following along so far, and have a device actively sending data, then all you need to do is to click the Save button for the decoder settings to take effect and the the graphs will begin displaying live data. Click the Dashboard tab at the top of your device page and the Temperature, Humidity, and Pressure gauges will appear. In this view it’s easy to add more widgets and play around with how the data is presented. This is also where you can generate a public link to your dashboard.

Better visuals in less time

Datacake dashboard

Your customers want to know what’s going on with their IoT fleets. Being able to build out professional quality visualizations quickly is both impressive, and a way to free yourself up to work on other rollout issues. That’s why we continue to add to the Output Stream options here on Golioth and we’re particularly excited about this partnership with Datacake. You should give it a try!

/by
,

Click “save” to stream IoT device data to the cloud — Golioth introduces CircuitPython SDK

What if you could open a text document on a device, write code, click save, and everything magically starts working? This is the promise of high level programming languages like CircuitPython. Golioth Labs now has an SDK to utilize the language’s fast prototyping capabilities. In addition to Golioth’s cloud functions, it’s super easy to pass data from a networked device up to the Golioth cloud. Click save to stream IoT device data to the cloud!

What is CircuitPython?

Adafruit created CircuitPython (CP), which started as a hardware specific fork of MicroPython (MP). Both CP and MP are based upon the ideas of the Python3 programming language, such as using an interpreter and basing language syntax on whitespace separators. The challenge is that the interpreter must live on a much smaller target than most computers running Python; fitting all of that into 128K of flash and 32K of RAM is a challenge! The hardware specific ports of CP encompass many of the Adafruit boards, but are really targeted around the chipset on those boards. Targets like the Microchip SAMD21 and SAMD51, the Raspberry Pi RP2040, the ESP32-S2, and more. The project continues to grow, both from Adafruit’s ongoing contributions and from a strong community contributing to the project.

Developing for prototyping

One of the strongest features of a language like CP is the ability to quickly iterate code. The “click save and your code starts running” is in contrast to the traditional method of compiling code on an external device (ie. your laptop), downloading it through a debugger (ie. a J-Link), debugging the code, and then watching the output. Because the CP interpreter is on the device itself, it processes the code as the device starts. While this means there is less room for user code, it is a much faster turn around once you click save.

During Golioth “Office Hours” on our community Discord (happening every Wednesday at 1 pm EST / 10 am PST), we had users asking for a faster way to prototype. Golioth developed an SDK in the “Golioth Labs” section of GitHub to interface with the Golioth API(s). This allows users with CircuitPython based programs to connect their IoT devices to the cloud and pass data back to Golioth. Much like the Arduino SDK experiment repo by Golioth Labs, we are interested in trying to extend the functionality of the Golioth cloud to many different platforms. If you have a hardware platform you would like supported, please post about it on our community forum!

Transport? I just want to get there!

As Alvaro points out in the video below, the transport layer is not something the user needs to care about with the Golioth CircuitPython SDK; any time you are working with a Golioth hosted SDK, you will be working at a higher level than the transport layer of communication, such as CoAP or MQTT. Many IoT platforms stop at the transport layer, notably MQTT examples. By moving up to working with APIs to LightDB State or LightDB Stream, you get additional functionality on the device side and you can better maintain your data on the Golioth console. Users always have the option to work at a lower level and peek under the hood at how the communication is happening. For people wanting to get started quickly, it’s important to have a high level way to start streaming data to the cloud.

Getting started with CircuitPython and Golioth

In the demo below, Alvaro showcases a sample setup using the following

First follow the Getting Started Guide to program your chosen hardware with the .UF2 file that represents the underlying CP tools, such as the interpreter and the code required to make your specific microcontroller into a CircuitPython based device.

The next step is hooking up the MicroMod device to the ESP32 modem using jumper wires. Read your board documentation to find an accessible UART port (TX/RX). Once completed, this enables the (Bluetooth based) nRF52840 to communicate over a UART connection to the modem and gain a connection to the internet using the ESP32’s WiFi interface. Credentials for the WiFi modem to connect to a local hotspot are sent from the nRF52840 to the ESP32 using AT Commands early in the CircuitPython program.

The example code on the Golioth CircuitPython SDK primarily lives in code.py. The device first connects using credentials for the WiFi network and PSK information for validating onto the Golioth cloud contained within secrets.py. After the UART is configured, the main processor (nRF52840 in the example below) starts communicating with the modem. In Alvaro’s example, he is sending the internal temperature of the processor using LightDB stream. He is able to send an update down to the board using LightDB State to remotely turn on an LED on the board, which is listening for changes on the /led path, similar to other examples that use LightDB State.

Watch the demo

It all comes together in the video below. Learn about how Golioth and CircuitPython pair to make a great combination for prototyping your next sensor project.

/by
, ,

Building IoT Dashboards with Golioth, Grafana, and Node-RED

This guest post is contributed by Ben Mawbey, a community member who is active on the Golioth Discord and frequently takes part in Office Hours.

Data wants to be visualized. The impact of showing a customer a slick plot of the information their devices have been collecting is massive compared to pointing at a few hundred lines of text from a log file or database query.

I was looking for some sort of dashboard or charting application for a demo for a sensor system we’ve built. I figured this will appease the clients’ need for pretty pictures over boring reports. I found exactly what I needed, and it only took me about 30 minutes to get it up and running.

Grafana graphing data

This looks great, even if you have no idea what the graphs mean! Let’s dig into how to get from numbers in a database to pretty pictures in a dashboard.

Pieces of the Puzzle: Golioth WebSockets, Node-RED, InfluxDB, and Grafana

Golioth is brilliant at getting your device data from the real world up to the cloud, climbing the IoT bean-stalk some would say. While abstracting some of the trickiest IoT problems, Golioth can present your time-series data as a convenient cloud resource using LightDB Stream service. By leveraging the built-in WebSockets support and some open-source tools we can rapidly store, manipulate and display this data!

Grafana is the open-source blockbuster for this application and can be easily set up to graph sensor data directly from Golioth LightDB Stream using the REST API. This has two major drawbacks:

  • Grafana must periodically poll for new data
  • While LightDB Stream does provide convenient data retention, I prefer to use my own data storage

Enter InfluxDB, the time-series database powerhouse and an ideal companion to Grafana when talking about IoT real-time applications. This pairing is so popular that the InfluxDB data source integration is baked right into Grafana! By utilizing InfluxDB to store our sensor data, we can perform more complex queries much faster.

The question remaining is how to shuffle data from Golioth to InfluxDB? There are many potential solutions to this hurdle, but my favorite is Node-RED which defines itself as a low code programming tool for wiring up event-driven systems.

Node-RED editor window

Node-RED uses graphical flows to connect data sources and destinations, bending protocols and translating data formats in innumerable ways

Node-RED has exploded in popularity and provides all sorts of integrations to connect your systems together. It provides simple blocks to perform actions and a slick graphical interface to wire up your data flows. Conceptually Node-RED acts as our rule engine to process and direct data.

Dashboards: Grafana and InfluxDB

System diagram

Grafana is immensely powerful at providing example custom views, data transformations, and alerting. That said, it is only as good as the quality of data you can provide. Having a tightly coupled InfluxDB instance with carefully curated data via Node-RED, allows you to quickly configure complex data queries on large datasets with low latency.

Before we can play with Grafana the first step is InfluxDB setup. After you’ve installed InfluxDB, create a new database on your InfluxDB instance:

> CREATE DATABASE golioth

Configure the InfluxDB Data Source in your Grafana instance by clicking the gear icon on the left sidebar, choosing and adding a data source, then searching for the InfluxDB plugin. Here’s how I’ve set up my data source:

Grafana InfluxDB source configuration

Assuming you have some data in your DB, we can quickly create a new time-series dashboard panel in Grafana and query the dataset using this integration:

Grafana panel configuration

This simple query shows how we structured the data in the DB, allowing us to select from a particular measurement, specifying a specific device identity tag and aggregating data points with specific time buckets. Adjusting the time range instantly updates the graph from our local InfluxDB instance.

Now how do we get the data from Golioth into InfluxDB?

Node-RED

Several networking integrations are provided with Node-RED. Presently the most relevant to Golioth would be HTTP nodes for REST API requests and the WebSockets node which is the easiest to configure.

You can see your sensor data collecting in the Golioth LightDB Stream by using the Golioth Console web interface:

Golioth Console showing LightDB stream data

We can use Node-RED flows to connect to Golioth via WebSockets and store the resulting data in our local DB:

NodeRED editor window

The nodes in this flow were set up as follows, taking care to give them appropriate names making it obvious to see their function. All of the nodes I’m using should come as part of every Node-RED installation except for the InfluxDB nodes. But don’t worry, these are trivial to install. On Linux is looks something like this:

cd ~/.node-red
npm install node-red-contrib-influxdb
sudo systemctl restart nodered.service

WebSockets Node:Node-RED websockets

First set up the credentials to your Golioth project using your generated API key and connect to the WebSockets LightDB Stream endpoint.

Debug Node:Node-RED debug

Drop a few of these nodes along the way and click the small green button to turn the debug log on. This is super handy to check data coming through and make sense of it.

JSON Node:Node-RED json

The LightDB Stream endpoint provides us with a JSON object representation containing our sensor data as well as meta information such as the data timestamp and device identity. This node allows us to parse this JSON into a javascript object so that we can work with it more easily in subsequent nodes.

Change Node:Node-RED change node

This node clearly shows the power of Node-RED as we can craft any sort of data manipulations or transformation.

We could do without this node and jump straight to InfluxDB, however, should any malformed data arrive we risk polluting the DB with bad data. By selectively transforming the incoming data and mapping it into a new object we can not only filter only good packets and arrange the measurement names but also add tags to build a solid data representation in the DB making our queries far more powerful.

InfluxDB Out Node:

Node-RED InfluxDB input nodeNode-RED InfluxDB server settings window

Finally, we can configure the data connection to our InfluxDB instance, set appropriately for your server configuration and database created earlier.

Assuming your flow is set up correctly, you should be able to see data collecting in your database. We know it works, but as I mentioned before, this visual is not going to impress our customers.

InfluxDB data

Revisiting the Grafana panel we previously created, you can see InfluxDB data is now being plotted!

Grafana graphing data

Corner Cases

One of the downsides of WebSockets is their ethereal nature, should there be some temporary connectivity issue, any data packets would be lost from the point of view of your InfluxDB database. A solution around this could be to set up another flow that executes periodically to sync with the LightDB Stream using the REST API. Node-RED could then be configured to check this data and add any missing values into the InfluxDB instance and prevent consistency issues.

Another concern with open-source self-hosted systems is security. It can be challenging to secure your server and services should they be public-facing. If you are handling sensitive data then it would be best to consult with an expert in this field. Fortunately, all of the tools discussed have subscription-based cloud services available that sort all of this out in the background.

Conclusion

Being able to set up a simple demo like this in less than 30 minutes demonstrates the power and flexibility of these modern open-source solutions. Coupled with the reliability and maintenance advantages of the Docker system, it’s a breeze to test locally on your desktop or Raspberry Pi and then deploy to production a moment later on your cloud server of choice. The rules engine and ease of wiring up blocks provided by Node-RED opens up a massive pool of possibilities, from countless other integrations to building intelligent processes. One such idea I would like to explore is integrating a device provisioning process into the flow such that we can link a device to a dataset or location during deployment or maintenance.

/by

Embedded Rust with Golioth and the Nordic nRF9160

Team members Lachlan and Chris discuss an implementation of a Rust code sample with Golioth and the Nordic Semiconductor nRF9160. There is a demo of the code working in the video attached below.

What is Rust?

Rust is a high-level, general-purpose programming language, syntactically similar to C and C++. It is fast and memory-efficient and is specifically designed for performance and safety. It has no runtime or garbage collector, while taking advantage of the higher-level concepts and safety guarantees. Rust can run on embedded devices, and can be easily integrated with other languages, and is ideal for performance-critical services. Rust is still in early days of development, starting in 2010 and only coming to embedded devices a few years ago. The developer community around Rust is passionate and growing. While Rust and C/C++ share similarities, Rust is relatively faster than C/C++ in some cases. Rust also has the benefit of a package management system, which makes implementing different libraries easier than C (which existed long prior to the internet). Easy package management encourages code reuse by allowing libraries to be readily integrated into applications, not to mention building the overall community around Rust.

Why we’re trying it

Golioth was interested in trying out Rust as part of our Golioth Labs segment on GitHub, where the code is currently hosted. It represents one of many trials we have and will take on in order to showcase the Golioth platform. We are committed to showing ways to access Golioth services and APIs outside of our normal recommended path Golioth SDK based on Zephyr RTOS. Not only does this represent the reality for many users, it also showcases how Golioth services are accessible from many different programming paradigms, in addition to a wide range of hardware. Another good example from Golioth Labs is our Arduino SDK discussed in our last article.

We chose to try Rust on the nRF9160 because the hardware represents a key product on the platform. This Nordic part has great support from Golioth and Nordic Semiconductor, and is listed as “Verified + Quickstart” on the Golioth Hardware Catalog. While the quickstart is using Zephyr to get users up and started, it is a well-supported part internally at Golioth. The nRF9160 is also primed for Rust integration: Nordic Semiconductor provides a library for the NRF devices that exposes the API which communicates with the hardware. There is also a Rust binding for it which makes it quite easy to implement the rust code and enables you to connect to Golioth servers and send and receive data. In addition, Rust has some code abstraction features and doesn’t require a debugger.

Video explanation and demonstration

In the demo below, Lachlan shows how he is able to use the library and bindings to connect over CoAP to the Golioth servers and replicate many of the functions that exist in Golioth samples.

/by

Trying out Golioth with PlatformIO and Arduino Core

Note: The arduino-sdk repository showcased in this post is deprecated. GoliothLabs has two experimental repositories that may work as replacements:

In this post and associated video, we’re talking about interfacing hardware to Golioth using popular tools like PlatformIO and Arduino Core (API). Alvaro shows how to use an ESP32 board from Adafruit with the Arduino Core on PlatformIO to talk to the Golioth MQTT endpoint. Golioth has a range of tools and resources for hardware and firmware engineers to get their devices talking to the cloud.

Why PlatformIO?

PlatformIO is a plugin system built on top of VS Code that enables embedded developers to get up and running quickly with different Real Time Operating Systems (RTOS), chipsets, and boards. There is a library manager to help manage dependencies within projects, which removes much of the friction at the beginning of development projects. Currently they do not support Zephyr for ESP32 (a combination previously shown on this blog/channel), but they do support things like the ESP-IDF (based on FreeRTOS) and the Arduino core. Once you pull in the project files from the Golioth Arduino SDK page, PlatformIO will try to download all of the dependencies required to build firmware for a particular board.

What is Arduino Core?

The Arduino Core refers to the APIs that are available on different hardware devices in order to conform to the Arduino ecosystem. In the example below, it refers to Espressif’s Arduino Core port, which means that the ESP32 functions will align with what the Arduino IDE might be asking of that hardware. If you are using platforms outside of the Arduino IDE, you can still interact with many of the underlying functions that were written by hardware and firmware developers to make boards work with Arduino software.

PlatformIO also pulls in other libraries that are useful for processing data coming out of the hardware, such as the Arduino JSON library. So while it’s not the “teal and white IDE” that many people expect, it’s utilizing much of the same work that enabled people to work from within the IDE.

Golioth’s MQTT endpoint

MQTT is now available in preview on Golioth, but should have a similar user experience as other protocols on the platform. Since the early days of Golioth, we have offered device services over CoAP. We chose CoAP as a first offering because it is meant for low-power devices in potentially lossy environments, like cellular networks. MQTT is a popular transport protocol that many engineers use to initially connect their devices to the internet, only to need additional tooling on other platforms. From Golioth’s standpoint, the user doesn’t need to care about which one they’re using. Instead, we provide device services like LightDB, which allows the user to push and pull configuration data and state-based sensor readings from a device. LightDB stream allows for time-series data that needs to be aggregated and displayed on something like a chart. Golioth has other features like firmware updates and logging, that also operate over these protocols.

Showcase video

In the video below, Alvaro shows how the new MQTT endpoint and services from Golioth enable a similar experience that is shown using CoAP in other demos on the platform. What is different now is that we can utilize code from other platforms like Arduino Core, which already had MQTT libraries built-in. Alvaro showcases how he was able to quickly build an Arduino SDK for the Golioth Labs GitHub.

/by