Over the holidays I tried out a fun experiment that combines Arduino and PlatformIO with Golioth. The work was prompted by a customer with an existing Arduino code base. Their long-term plan is to port the code over to ESP-IDF, but this serves as an intermediary step while they work out their own library transition plans. It works for them and today I’ll cover how you can give it a try for yourself. But first, some disclaimers.

Golioth Labs means Experimental

We use the Golioth Labs organization on GitHub for our experimental projects. That means you should not build for production devices using any of those repositories. This experiment targets the ESP32 family of chips. We have full support for ESP-IDF in the Golioth Firmware SDK, which is what we recommend using.

That being said, what if you have an existing Arduino project and want to test out some Golioth features? This is a “what am I getting myself into” step, and depends on your code being able to run on an ESP32 since the experiment builds Arduino as a component of ESP-IDF. The “cherry on top” wraps the project up into PlatformIO, an IDE built around VScode that has become a popular alternative to the native Arduino IDE.

Test Driving Golioth + PlatformIO + Arduino

As a prerequisite you need to install VScode and the PlatformIO extension for it.

With those tools in place, let’s walk through how to clone the Golioth repository and set up submodules, then finish up by building and running the demo program.

Clone the Repository and Submodules

By default, PlatformIO stores new projects in the ~/Documents/PlatformIO/Projects folder. You may clone this project anywhere you like, but that’s as good of a place as any.

Navigate to your preferred folder and clone the GoliothLabs repo and submodules:

# Clone the repository
git clone https://github.com/goliothlabs/golioth_platformio_arduino.git
# Enter the newly created folder
cd golioth_platformio_arduino
# Initialize the submodules (this will take more than 10 minutes)
git submodule update --init --recursive
# Clone a nested repository (yes, this is odd)
git clone https://github.com/hathach/tinyusb.git third_party/esp32-arduino-lib-builder/components/arduino_tinyusb/tinyusb

That’s it for setup. The last step in the commands listed above is an odd one. According to Espressif’s Arduino as an ESP-IDF component documentation, the tinyusb library needs to be nested inside of the esp32-arduino-lib-builder library’s components folder.

Build and Run the Golioth Sample

  1. Start by importing the project into PlatformIO.In VScode, click the alien icon on the left sidebar, choose PIO Home→Open from the sidebar that appears, and select Open Project from the PlatformIO home screen.PlatformIO open project
  2. Next, set up your credentials.Make a copy of the credentials.h_example file and rename it credentials.h. In this file, place the credentials for your WiFi access point, and the PSK-ID/PSK for the device you created using the Golioth Console.Note: With our Dev tier, your first 50 devices are free. Use the Credentials tab when viewing a device to access the PSK-ID/PSK.credentials.h file
  3. Now build the project.Click on the alien icon on the left sidebar and choose Build from the Project Tasks menu. A terminal window will open and the build process will begin.PlatformIO build process
  4. Finally, flash the device and observe the output.From the Project Tasks menu, click on Upload and Monitor. The project will build, upload to the ESP32, and then open a terminal window to show the serial output from the ESP32.

You will see an output similar to the following:

I (942) wifi:new:<6,0>, old:<1,0>, ap:<255,255>, sta:<6,0>, prof:1
I (944) wifi:state: init -> auth (b0)
I (952) wifi:state: auth -> assoc (0)
I (957) wifi:state: assoc -> run (10)
I (970) wifi:connected with golioth-staff, aid = 1, channel 6, BW20, bssid = c2:ff:d4:a8:fa:10
I (970) wifi:security: WPA2-PSK, phy: bgn, rssi: -39
I (977) wifi:pm start, type: 1

I (1041) wifi:AP's beacon interval = 102400 us, DTIM period = 2
.W (1556) wifi:<ba-add>idx:0 (ifx:0, c2:ff:d4:a8:fa:10), tid:0, ssn:1, winSize:64
␛[0;32mI (1727) esp_netif_handlers: sta ip: 192.168.1.157, mask: 255.255.255.0, gw: 192.168.1.1␛[0m
.IP address: 
192.168.1.157
␛[0;32mI (1936) golioth_mbox: Mbox created, bufsize: 2184, num_items: 20, item_size: 104␛[0m
␛[0;32mI (1937) golioth_fw_update: Current firmware version: 1.0.0␛[0m
Hello, Golioth! #0
␛[0;32mI (1953) golioth_example: Updating counter to: 0␛[0m
␛[0;32mI (1987) golioth_coap_client: Start CoAP session with host: coaps://coap.golioth.io␛[0m
␛[0;32mI (1988) golioth_coap_client: Session PSK-ID: platformio-test@golioth-arduino-demo␛[0m
␛[0;32mI (1998) libcoap: Setting PSK key
␛[0m
␛[0;32mI (2005) golioth_coap_client: Entering CoAP I/O loop␛[0m
Hello, Golioth! #1
␛[0;32mI (6955) golioth_example: Updating counter to: 1␛[0m
␛[0;33mW (9968) golioth_coap_client: CoAP message retransmitted␛[0m
␛[0;32mI (11492) golioth_coap_client: Golioth CoAP client connected␛[0m
Hello, Golioth! #2
␛[0;32mI (11957) golioth_example: Updating counter to: 2␛[0m
␛[0;32mI (13544) golioth_fw_update: Waiting to receive OTA manifest␛[0m
Hello, Golioth! #3
␛[0;32mI (16959) golioth_example: Updating counter to: 3␛[0m
Hello, Golioth! #4
␛[0;32mI (21961) golioth_example: Updating counter to: 4␛[0m
Hello, Golioth! #5
␛[0;32mI (26963) golioth_example: Updating counter to: 5␛[0m
^CHello, Golioth! #6
␛[0;32mI (31965) golioth_example: Updating counter to: 6␛[0m
Hello, Golioth! #7
␛[0;32mI (36967) golioth_example: Updating counter to: 7␛[0m

In this example, we are sending Arduino Serial.print() commands and Golioth logs that are displayed on the terminal and sent to the server. They show an upcounting timer that is stored on the Golioth LightDB state service. Over-the-Air (OTA) firmware update is also available.

While this shows a very small subset of Golioth features, everything shown in our golioth_basics example code can be used with this project.

What to do Next

As I mentioned earlier, you should not build production hardware around this repo. This is merely a way to test drive Golioth with your existing Arduino code base. The longer term goal should be to transition your ESP32-based hardware over to the ESP-IDF flavor of the Golioth Firmware SDK. If you’re using non-ESP32 hardware, you should target the Golioth Zephyr SDK.

While Arduino and PlatformIO are great platforms, directly targeting the ESP-IDF framework results in a more stable and predictable build. The Golioth SDK is built as a component of ESP-IDF for rock-solid performance. Building for ESP-IDF is definitely the end goal if you are migrating an existing fleet, and the best place to start if you are bootstrapping new designs.

Do you have questions about how to best integrate Golioth for managing your IoT devices? We’d love to hear from you! Post a message on the Golioth Forum, or reach out to our Developer Relations folks to set up a meeting.

The human body is surprisingly adept at sensing temperature. And so when I sat up in bed my body immediately informed me something was not right with my furnace. Yes, it’s winter in Wisconsin, but it shouldn’t feel that cold inside the house. The furnace was fine, but the thermostat was not, and so begins the story of how Golioth ran my furnace over Christmas.

Twas the Week Before Christmas and Cold in the House

HVAC control panel

EIM that connects to the furnace/AC/HRV. The red/green wires exiting the bottom of the photo go to the Golioth Greenhouse Controller

I arose from my slumber and went to check the thermostat to find it displaying a message that it could not connect to the Equipment Interface Module (EIM). We have a furnace, air conditioner, and a heat recovery ventilator (HRV is a fancy name for a pair of fans that exchange stale inside air with fresh outside air). Being a geek, I wanted one way to control them all. That meant installing the Honeywell EIM to switch those subsystems. The thermostat then connects wirelessly to the EIM to provide automated control. But the chilly morning we’re discussing here, the thermostat wasn’t connecting at all!

After basic troubleshooting I used a jumper wire to short the white “run the furnace now” wire to the 24-volt red wire. The furnace kicked on and started warming up the house. There was nothing wrong with the furnace, but I no longer had an automatic way to control it.

But wait…industrial control is one place where Golioth shines! Could I use Golioth as my furnace controller until the built-in system was repaired?

Connecting the Golioth Greenhouse Controller to My House

Golioth Greenhouse Controller connected to the EIM

The Golioth Greenhouse Controller was placed outside of the utility closet to sense temperature readings from the room.

Just a week before this happened I wrote a blog post about the Golioth IoT Greenhouse Controller. It includes relays that can be switched automatically based on a temperature sensor. This is the definition of a thermostat. And these Golioth reference designs are built to be easily adaptable to similar applications. The hardware connection was dead simple: just run the red and white wires to the temperature-controlled relay on the greenhouse controller.

The logic in the firmware needed a quick tweak as greenhouses regulate temperature by taking action when it gets too hot (the opposite of heating a house). This required the “expert” firmware engineering step of changing a greater-than sign to a less-than sign. Luckily all of our reference designs have Over-the-Air (OTA) firmware update built-in so I was able to make this change while the device was in my basement, still connected to the furnace.

The temperature threshold is set via Golioth’s web console, so at that point I had actually upgraded my home system to include control from anywhere in the world! I also got a dashboard to keep my eye on temperature, pressure, and humidity.

Eating Your Own Dog Food (and Enjoying the Flavor)

So, why didn’t I just go out and buy another thermostat? The first reason is the ongoing chip shortage (now into it’s third year). The EIM part that was broken is about $80 normally, but there’s a 31 week lead time on it right now. The second reason is that the only wires running from the furnace to the thermostat are for supplying power (+24 V and Ground). Not only would a replacement thermostat be temporary, I would need to run new wires to give it furnace control or leave it in the basement.

For me this was a great philosophy experiment. We spend a lot of time planning and developing reference designs. Now I know a bit more about the user experience, and I like it! The hardware form-factor was easy to work with, and the approaches we take to control and feedback worked well for my application.

Temperature graph with many on/off cycles

Temperature control without hysteresis

That doesn’t mean it was a perfect fit. Our basic design doesn’t have hysteresis built in, which means that the controller was trying to cycle more often than normal as there was no deadband to allow the house to warm (or cool) a bit past the target temperature. I implemented a simple cycle time threshold that prevented switching between on and off more than once in a 15 minutes period. It’s an application-specific topic that could be added to an implementation guide for this reference design.

Temperature and humidity graphs with smooth rise and smooth response due to cycle-time control.

Temperature response over the same time period was much smoother with simple cycle-time control

I called several HVAC shops in town and the earliest estimates for a new part was the end of March. Online I was able to find a Honeywell “starter” kit included the replacement EIM with just a week before delivery. During this time the Golioth-controlled furnace kept the house warm, but also getting very very humid without the HRV to help regulate. Good news everyone, the Greenhouse demo has a humidity sensor and a second relay. You guessed, it. I added control for the ventilation as well!

I Built An IoT Thermostat in 20 Minutes

Now, I was already familiar with this demo unit, so take this with a grain of salt. But the purpose of the Golioth reference designs is to give businesses a running start when developing a proof of concept. I was able to substitute this one in place of my thermostat with just about 20 minutes of effort. It kept my house comfortable in the harshest conditions of December’s arctic blast, and it let me check temperature and adjust settings while I was working outside of the home during that time. This is the Internet of Things, and it’s never been easier.

Adding Golioth’s device management features to an existing ESP-IDF project is a snap. The Golioth Firmware SDK has excellent support for Espressif parts. Simply add Golioth as a component in your build, enable it in the CMake and Kconfig files, and start using the API functions.

Today I’m going to walk through how to do this using an ESP32 development board, however, any of the ESP32 family of chips (ESP32s2, ESP32c3, etc.) will work. Advanced users will also want to review the Golioth Firmware SDK Integration Guide which is the bases for this post.

Walkthrough

Since this article is about adding Golioth to an existing ESP-IDF application, I assume you already have the ESP-IDF installed. If you do not, you may consider following our ESP-IDF quickstart guide before continuing.

For illustration purposes, I’ve copied Espressif’s /esp-idf/examples/wifi/getting_started/station example code to a folder called esp_hello_golioth and will add the Golioth device management features to the program.

cp -r ~/esp-idf/examples/wifi/getting_started/station esp_hello_golioth

If you are following along, you will want to set your WiFi credentials either by using menuconfig or by changing the default values in main/Kconfig.projbuild.

1. Install Golioth as a component

Add the Golioth Firmware SDK as a submodule of your project. (Note that this means you need to have your project under version control. Hint: git init)

cd ~/esp_hello_golioth
git submodule add https://github.com/golioth/golioth-firmware-sdk.git third_party/golioth-firmware-sdk
git submodule update --init --recursive

2. Add the Golioth SDK to your CMake configuration

We need to tell CMake to include the Golioth SDK in the build. This is done by editing the project’s top-level CMakeLists.txt file and adding the following line before the project() directive in that file:

list(APPEND EXTRA_COMPONENT_DIRS third_party/golioth-firmware-sdk/port/esp_idf/components)

We also need to add the Golioth SDK as a dependency. This is done by editing the deps list in the CMake file in the main directory of the project. For this ESP-IDF example code, there are no existing deps so I added the following to the top of main/CMakeLists.txt:

set (deps
     "golioth_sdk"
)

3. Add Kconfig settings

Golioth needs MbedTLS so it must be enabled via the Kconfig system. I created an sdkconfig.defaults file in the top-level folder and added the following:

CONFIG_MBEDTLS_SSL_PROTO_DTLS=y
CONFIG_MBEDTLS_PSK_MODES=y
CONFIG_MBEDTLS_KEY_EXCHANGE_PSK=y
CONFIG_GOLIOTH_AUTO_LOG_TO_CLOUD=1
CONFIG_GOLIOTH_COAP_REQUEST_QUEUE_MAX_ITEMS=20

The final two symbols were added because I plan to send logging messages to the Golioth servers.

4. Use Golioth API calls in your C code

Now that we’ve unlocked Golioth we can start making API calls. Gain access to the functions by including the Golioth header file:

#include "golioth.h"

For this demo I created a counter to keep track of a forever loop. After a five-second pause, the counter value is sent to Golioth as a Log message, then sent as a LightDB State value. The highlighted code beginning at line 14 is what I added.

void app_main(void)
{
    //Initialize NVS
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    ESP_LOGI(TAG, "ESP_WIFI_MODE_STA");
    wifi_init_sta();

    //Everything above this line is unchanged from the ESP-IDF example code

    const char* psk_id = "your-golioth@psk-id";
    const char* psk = "your-golioth-psk";

    golioth_client_config_t config = {
            .credentials = {
                    .auth_type = GOLIOTH_TLS_AUTH_TYPE_PSK,
                    .psk = {
                            .psk_id = psk_id,
                            .psk_id_len = strlen(psk_id),
                            .psk = psk,
                            .psk_len = strlen(psk),
                    }}};
    golioth_client_t client = golioth_client_create(&config);

    uint16_t counter = 0;
    while(1) {
        GLTH_LOGI(TAG, "Hello, Golioth! #%d", counter);
        golioth_lightdb_set_int_async(client, "counter", counter, NULL, NULL);
        ++counter;
        vTaskDelay(5000 / portTICK_PERIOD_MS);
    }
}

For simplicity I hardcoded the Golioth credentials (PSK-ID/PSK) which is not a best practice as hardcoded credentials will not survive a firmware upgrade. Both shell and Bluetooth provisioning are demonstrated in the golioth_basics example code which stores these credentials in non-volatile flash.

See the results on the Golioth Console

The serial output of this app shows the Golioth client connecting and Log messages being sent.

I (937) wifi:AP's beacon interval = 102400 us, DTIM period = 2
I (1647) esp_netif_handlers: sta ip: 192.168.1.159, mask: 255.255.255.0, gw: 192.168.1.1
I (1647) wifi station: got ip:192.168.1.159
I (1647) wifi station: connected to ap SSID:TheNewPeachRepublic password:123456789101112internetplease
I (1657) golioth_mbox: Mbox created, bufsize: 2184, num_items: 20, item_size: 104
I (1667) wifi station: Hello, Golioth! #0
W (1677) wifi:<ba-add>idx:0 (ifx:0, c6:ff:d4:a8:fa:10), tid:0, ssn:1, winSize:64
I (1677) golioth_coap_client: Start CoAP session with host: coaps://coap.golioth.io
I (1697) libcoap: Setting PSK key

I (1697) golioth_coap_client: Entering CoAP I/O loop
I (1917) golioth_coap_client: Golioth CoAP client connected
I (6707) wifi station: Hello, Golioth! #1
I (11707) wifi station: Hello, Golioth! #2
I (16707) wifi station: Hello, Golioth! #3
I (21707) wifi station: Hello, Golioth! #4
I (26707) wifi station: Hello, Golioth! #5
I (31707) wifi station: Hello, Golioth! #6
I (36707) wifi station: Hello, Golioth! #7
I (41707) wifi station: Hello, Golioth! #8
I (46707) wifi station: Hello, Golioth! #9

Both Log messages and State data can be accessed from the tabs along the top of the device view in the Golioth Console. Navigate there by selecting Devices from the left sidebar menu and then choosing your device from the resulting list.

The log view shows each message as it comes in, and from the timestamps we can see that the five-second timer is working. For long-running operations, there are time-window selection tools as well as filters for log levels, devices, and modules.

The LightDB State view shows persistent data so you will always see the most recently reported value at the “counter” endpoint. Make sure the refresh dialog in the upper right is set to Auto-Refresh: Real-time to ensue you see the updates as they arrive from the device.

Give Golioth a try!

We built Golioth to make IoT design easier for firmware developers. We’d love it if you took our platform for a test-drive! With our Dev Tier, your first 50 devices are always free. Grab an ESP32 and you’ll have data management, command and control, and OTA firmware update in your toolbelt in no time.

Even people who have been living under a rock for the past couple of years know that there’s a global chip shortage. The correct response from the engineering community should be changes to our design philosophy and that’s the topic of the talk that Chris Gammell and I gave at the 2022 Zephyr Developer Summit back in June. With the right hardware and firmware approach, it’s possible to design truly modular hardware to respond to supply chain woes.

Standardization enabled the industrial revolution, and the electronics field is a direct descendant of those principles. You can rest easy in knowing that myriad parts exist to match your chosen operating voltage and communication scheme. But generally speaking it’s still quite painful to change microcontrollers and other key-components mid-way through product design.

Today’s hardware landscape has uprooted the old ways of doing things. You can’t wait out a 52-week lead time, so plan for the worst and hope for the best. Our talk covers the design philosophies we’ve adopted to make hardware “swapability” possible, while using Zephyr RTOS to manage the firmware side of things.

Meet the Golioth Aludel

Golioth exists to make IoT development straightforward and scalable–that’s not possible if you’re locked into specific hardware, especially these days. So we designed and built a modular platform to showcase this flexibility to our customers.

Golioth Aludel prototyping platform internal view

Called Aludel, Chris Gammell centered the design around a readily available industrial enclosure. The core concept is a PCB base that accepts any microcontroller board that uses the Feather standard. This way, every pin-location and function is standardized. Alongside the Feather you’ll find two headers that use the Click boards standard (PDF), a footprint for expansions boards facilitating i2c, SPI, UART, and more. The base includes Qwiic and Stemma headers for additional sensor connectivity, as well as terminal blocks, room for a battery, and provisions for external power.

The key customization element for Aludel is the faceplate. It’s a circuit board that can deliver a beautiful custom design to match any customer request. But on the inside, each faceplate has the same interface using a flat cable connection to the base board. Current versions of the faceplate feature a a screen, an EEPROM to identify the board, and a port expander that drives buttons, LEDs, and whatever else you need for the hardware UI.

The beauty of this is that electrically, it all just works. Need an nRF52, nRF9160, ESP32, STM32, or RP2040? They all already exist in Feather form factor. Need to change the type of temperature sensor you’re using? Want to add RS485, CANbus, MODBUS, 4-20 ma communication, or some other connectivity protocol? The hardware is ready for it–at most you might need to spin your own adapter board.

Now skeptics may look at the size of this with one abnormally high-arched eyebrow. But remember, this is a proof-of-concept platform. You can set it up for your prototyping, then take the block elements and boil them down into your final design. Prematurely optimizing for space means you will have many many more board runs as the parts change.

When you do a production run and your chip is no longer available, the same concepts will quickly allow you to test replacements and respin your production PCB to accommodate the changes.

Zephyr Alleviates your Firmware Headaches

“But wait!” you cry, “won’t someone think of the firmware?”. Indeed, someone has thought of the firmware. The Linux Foundation shepherds an amazing Real-Time Operating System called Zephyr RTOS that focuses on interoperability across a vast array of processor architectures and families. Even better, it handles the networking stack and has a standardized device model that makes it possible to switch peripherals (ie: sensors) without your C code even noticing.

We use Zephyr to maintain one firmware repository for the Aludel hardware that can be compiled for STM32F40, nRF52840, nRF9160, and ESP32 using your choice of Ethernet, WiFi, or Cellular connections. How is that even possible?

&i2c0 {
	bme280@76 {
		compatible = "bosch,bme280";
		reg = <0x76>;
		label = "BME280_I2C";
	};
};

&spi1 {
	compatible = "nordic,nrf-spi";
	status = "okay";
	cs-gpios = <&gpio0 3 GPIO_ACTIVE_LOW>,
	           <&gpio0 27 GPIO_ACTIVE_LOW>,
	           <&gpio1 8 GPIO_ACTIVE_LOW>;
	test_spi_w5500: w5500@0 {
		compatible = "wiznet,w5500";
		label = "w5500";
		reg = <0x0>;
		spi-max-frequency = <10000000>;
		int-gpios = <&gpio0 2 GPIO_ACTIVE_LOW>;
		reset-gpios = <&gpio0 30 GPIO_ACTIVE_LOW>;
	};
};

/ {
    aliases {
        aludeli2c = &i2c0;
        pca9539int = &interrupt_pin0;
    };
    gpio_keys {
        compatible = "gpio-keys";
        interrupt_pin0: int_pin_0 {
			gpios = < &gpio0 26 GPIO_ACTIVE_LOW >;
			label = "Port Expander Interrupt Pin";
		};
	};
};

Zephyr uses the DeviceTree standard to map how all of the hardware is connected. A vast amount of work has already been done for you by the manufacturers who maintain drivers for their chips and supply .dts (DeviceTree Syntax) files that specify pin functions and mappings. When writing firmware for your projects you supply DeviceTree overlay files that indicate sensors, buttons, LEDs, and anything else connected to your design. The C code looks up each device to receive all relevant information like device address, GPIO port/pin assignment and function.

const struct device *weather_dev = DEVICE_DT_GET_ANY(bosch_bme280);
sensor_sample_fetch(weather_dev);
sensor_channel_get(weather_dev, SENSOR_CHAN_AMBIENT_TEMP, &temp);
sensor_channel_get(weather_dev, SENSOR_CHAN_PRESS, &press);
sensor_channel_get(weather_dev, SENSOR_CHAN_HUMIDITY, &humidity);

The hardware abstraction doesn’t stop there. Zephyr also specifies abstraction for sensors. For instance, an IMU will have an X, Y, and Z output — Zephyr makes accessing these the same even if you have to move to an IMU from a different manufacturer due to parts shortages. You update the overlay files for the new build, and use a configuration file to turn on any specific libraries for the change, but the code you’re maintaining can continue on as if nothing happened.

Of course there are caveats which we cover in the talk. There is no one-size-fits-all in embedded engineering so you will eventually need to define your own .dts files for custom boards, and add peripherals that are unsupported in Zephyr. This is not a deal breaker, there is a template example for adding boards (and I highly recommend watching Jared Wolff’s video on the subject). And since Zephyr is open source, your customizations can be contributed upstream so others may also benefit.

Hardware will never again be the same

Electronics manufacturing will eventually emerge from the current shortages. But there are no signs of this happening soon, and it’s been ongoing for two years. We shouldn’t be trying to return to “normal”, but planning a better future. That’s what Golioth is doing for the Internet of Things. Choosing an IoT management platform shouldn’t require you to commit to one type of hardware. Your fleets should evolve, and our talk outlines how they can evolve. Golioth helps to manage your diverse and growing hardware fleet. Golioth is designed to make it easy to keep track of and control all of that hardware, whether that’s 100 devices or 100,000. Take Golioth for a test drive today.

How to use Ubidots data visualization

Golioth and Ubidots have teamed up to make it easy to visualize your Internet of Things device data. Ubidots makes it easy to white-label visualizations to produce a branded version of your service for end customers; you can also create end-users, in order to allow access to custom dashboards.

Our partnership has produced an integration that makes it a snap to connect your device data to an Ubidots dashboard so that it looks great for you and for your clients. Today I’ll walk you through the process of setting up a connection.

Ubidots data visualization dashboard

What you need to get started

To get started you just need a few basic things:

  • An account on Golioth (sign up)
  • An account on Ubidots (sign up)
  • A device that is sending data to Golioth

Creating your accounts is pretty easy; Golioth has a free dev tier, and Ubidots has a free Educational/Personal use option or a free trial period for their Business users. Everyone who has followed our Getting Started guide will already has a Golioth account.

If you don’t already have a device set up to send data, you can use the Golioth LightDB Stream example. It will simulate temperature data and send it back to the Golioth Cloud, a perfect dataset for your first Ubidots dashboard!

Setting up a connection between Golioth and Ubidots

To connect Golioth and Ubidots we need to take the follow steps:

  1. Configure an Ubidots plugin
  2. Configure a Golioth output stream
  3. Set up an Ubidots dashboard

1. Configure the plugin in Ubidots

Ubidots has a plugin specifically for Golioth. But before configuring the plugin we want to make sure we have a token to use for this project. From the Ubidots dashboard, click your profile picture in the upper right, choose API Credentials and click More under the tokens menu that appears. This will open the API Credentials page.

How to create an ubidots token

Use the plus sign to add a new token, then use the pencil icon to edit the name of the token.

Next, choose DevicesPlugins from the top menu. Click the plus to add a plugin, and choose Golioth. Click the arrow icon to get to the plugin configuration menu.

Choose the token you just created from the dropdown menu. In the LightDB Stream field, use the name of the endpoint where your device is sending data.

If you have a nested JSON, like {"env":{"temp":20.5}}, then you would set this value to “env” and the Ubidots plugin would pick up all of the key/value pairs inside of that JSON object.

If you are following along with the Golioth Stream sample, leave the LightDB Stream field blank, since we publish to the root-level directly. You can see in the code example below, we are setting the temperature value to the root-level “temp” key.

err = golioth_lightdb_set(client,
			GOLIOTH_LIGHTDB_STREAM_PATH(&amp;amp;quot;temp&amp;amp;quot;),
			COAP_CONTENT_FORMAT_TEXT_PLAIN,
			str_temperature,
			strlen(str_temperature));

Click the right arrow, give your plugin a name and description, and click the check mark icon to save. You will be redirected to a list of your plugins.

[screenshot]

Click on the one you just created and then select Decoder from the left sidebar.

Generally speaking, this page is already configured to work. We need to copy the Ubidots HTTPs Endpoint URL and use it to create a Golioth Output Stream in the next step.

If you want to learn more about the data format so that you can customize the decoder on this page, look at the DeviceStreamMessage in our Event Types documentation. For today’s example, the defaults will work.

2. Configure a Golioth output stream

Choose Ubidots as Output Stream

In the Golioth Console, select Output Streams from the left sidebar, click on Create an Output Stream, and choose Ubidots from the list.

Give your new Output Stream a name, and choose DEVICE_STREAM_TYPE as the event filter. The HTTPS Endpoint URL comes from the Ubidots dashboard. So does the auth token, which you can copy from the Ubidots dashboard by clicking your avatar in the upper right and choosing API Credentials(this reveals a panel that lets you copy tokens to the clipboard). Click save to finish setting up your Output Stream.

3. Set up an Ubidots dashboard

We’re ready to start visualizing! At the top of the Ubidots dashboard, select DataDashboards. You can create a new dashboard using the icon (sometimes called a “hamburger menu”) in the upper left.

On your new dashboard, click Add new Widget and choose a Gauge from the list.

Ubidots dashboard gauge settings

In the data section, click Add Variable, choose your device, and select the data you want to show. If your device is not listed, the endpoint may be configured wrong, or the device has not yet sent any data. There is a Logs section in the Ubidots plugins settings page that will be helpful in troubleshooting the issue.

The rest of the settings in this dialog decide how your data will be displayed. Set the name of the widget, and choose the range that suits you best before clicking the check mark icon to save the changes.

Temperature gauge

Now you have a gauge that shows temperature readings from your device. Of course there are many other visualization methods available to match your needs, so spend some time playing around with your new Ubidots dashboard!

Node-RED is a graphical programming tool that makes it easy to interact with web-based events. It’s called a “Low-code” solution because you can do a lot without writing much code. I love it because it’s the right balance of handling the hard stuff for me. Initial ease of use, while still making it possible to drop in custom code when I need it. I’ve been using that to control my IoT devices on the Golioth Cloud, and today I’m sharing how I do it.

Note: the concept of a node will be highlighted with bold text throughout this article to help clarify we’re discussing one of the blobs shown on screen in Node-RED

Why use Node-RED with Golioth

Earlier this year Ben Mawbey showed us how he uses Node-RED to manipulate data as it arrives on the Golioth cloud. This approach listens for realtime data using WebSockets. But this has a couple of limitations:

  • We will only see the data when it changes
  • We aren’t able to send or update data using WebSockets.

On that second point, the Golioth REST API lets us send/update data and query stored data.

Node-RED is a good choice here, since it’s relatively easy to set up a connection to WebSockets and the REST API. It’s powerful enough to do any data manipulation we want. It does need to run on a server, but on the upside that means it is always running and can be accessible to multiple end users via any web browser, no need to install an app. Let’s dive in!

Prerequisites

This example assumes you have already taken care of the following:

  • a Node-RED server
    • The Get Started page offers several solutions like running it locally on a Raspberry Pi, or on a cloud server
  • Golioth account, and a device added on the Golioth Console
    • As always, our getting started guide will walk you through all of these details
    • You can follow the demo below without having a hardware device, we just need to create a device on the Golioth Console to send data back and forth to the cloud

Connect Node-RED to WebSockets

The secret sauce in connecting a WebSocket is the URL which includes the Golioth API key. Go to the Golioth Console and choose API Keys from the left sidebar to create a new key:

The exact formatting of the WebSockets URL is detailed on our reference page, but I’ll show you how to do it here. Notice the three pieces of important data in the diagram below: project id, device id, and finally the API key.

I have chosen to connect to the dataendpoint to monitor LightDB state data. (If you want to monitor LightDB stream you can change data to stream.)

wss://api.golioth.io/v1/ws/projects/node-red-demo/devices/62695497404e12cd12628117/data?x-api-key=ynhJQQjLautKYLxQESuqB8VJLWMEVpH7

Setting up the Node-RED flow begins by adding a WebSocket-in node and connecting it to a debug node. I have also connected a JSON node for convenient data access later.

The WebSocket-in node is configured as follows:

The WebSocket type is “Connect to” which allows us to add a new URL. Click the pencil icon or the right side of that field and in the new window paste the URL. Because we are passing the API key as part of the URL, we do not need to add a TLS configuration.

Data will only appear on this WebSocket-in node when it first arrives on the Golioth Cloud. So generate an update by going to the Golioth Console and adding the key/value pairs shown above. In the Node-RED shown in the flow setup step, notice the debug output includes the raw payload as well as the JSON object.

Connect Node-RED to REST API

Setting up the REST API connection requires a bit of code, but uses the same project, device, and API key info from the last step. Two inject nodes (that send a 1 or a 0), a function node, an https request node, and a debug node complete this flow:

The inject node and http request node need a quick settings adjustment:

Set the payload of the inject node to pass 0 or 1 as a number (not a string). In the http request node, change the method to “set by msg.method”. The magic will all happen in the function node:

Code for this function is listed below. The variables at the top are used to set up the API URL, so you will need to enter your project id, device id, API key, and the LightDB state key that makes up the endpoint (led0 in my example).

var proj_id = "node-red-demo"
var dev_id = "62695497404e12cd12628117";
var api_key = "ynhJQQjLautKYLxQESuqB8VJLWMEVpH7";
var endpoint_name = "led0";

var dev_url = "https://api.golioth.io/v1/projects/" + proj_id + "/devices/";
var endpoint = "/data/" + endpoint_name;

var data = msg.payload;

var msg = {
	"method" : "PUT",
	"url" : dev_url + dev_id + endpoint,
	"headers" : {
		"Content-Type": "application/json",
		"x-api-key": api_key
	},
	"payload" : JSON.stringify(data)
};

return msg;

This code intercepts the incoming data (which will be a 0 or 1 from the inject node), formats it as a PUT command, and sends it to the http request node which will submit it to Golioth. In this demo, clicking one of the inject nodes updates the led0 value in LightDB state.

Head over to the Golioth Console and look at the LightDB state data for your device to see the changes. In practice, you can implement the desired state versus actual state principles discussed in my recent article and use this flow to update an LED on an actual piece of hardware.

Further Exercises: Dashboard/Web App

I’ve covered a lot of ground in this article and unfortunately have run out of column inches. But before signing off, I want to mention the potential for turning Node-RED flows into web apps.

The Node-RED dashboard node adds a UI which can be loaded on any browser. It looks spectacular with almost no work from us. Here you can see I’ve set up a display that shows the state of the LED, displays the latest value of the counter, and adds two buttons to turn the LED on or off.

If you want to test this out, here is an export of this flow that you can import into your Node-RED.

A word of caution: your flow contains an API key for accessing device information on Golioth. This must be kept secure. Please make time to review how to secure Node-RED and ensure that you authenticate users if you decide to build and share a web interface.

How to connect to Golioth with the ESP-IDF
PLEASE NOTE: This post was published in April of 2022. In July 2022, we released the Golioth ESP-IDF SDK, which is our recommended path for developing with the ESP-IDF. The post below is a great look “under the hood” to see what it took to connect to Golioth CoAP endpoints before our ESP-IDF SDK existed.

Golioth has great support for ESP32 devices. While our official SDK is built on the Zephyr RTOS, and we understand that there are some use cases where you might need to use the ESP-IDF instead.

The good news is that Golioth uses the CoAP protocol. So anything that can authenticate with Golioth and communicate over CoAP can be use with our platform. The ESP-IDF includes some CoAP examples, and allows pre-shared keys to be used for authentication, which means it will work with Golioth.

In this article, I’ll show you how to use VS Code to develop an embedded application that will connect to the Golioth Platform using the ESP-IDF instead of Zephyr. Let’s dive in.

Requirements

To follow this tutorial, you first need to have

  • A Golioth account
  • A Golioth project with at least one device
  • PSK-ID and PSK of your device
  • Visual Studio Code (VS Code) pre-installed

If you followed the Golioth Getting Started guide, you have already satisfied the first three requirements and will just need to install VS Code.

Install the ESP-IDF in VS Code

The ESP-IDF can be installed from VS Code. I suggest following this quick tutorial to install the ESP-IDF extension for VS Code, even if you already have manually installed it. We’ll only use the Graphical Interface provided by the extension in this tutorial.

Create a coap-client project from the ESP-IDF examples

Open the VS Code command pallet (ctrl+shift+p) and search for “ESP-IDF: New Project“

In the New Project window, enter:

  • project name: coap-golioth-hello
  • project location: <path-to-your-project>/
  • ESP-IDF board: custom board
  • ESP-IDF Target: Esp32 module
  • Serial Port: choose the serial port that your device is connected (in my case, /dev/ttyUSB0)

Click on “Choose Template” button, then click in the selection box and select “ESP-IDF” option

Next, a new windows will be open with ESP-IDF samples.

Scroll down and go to Protocol section and select coap_client sample option. Then click on “Create project using template coap_client”.

A status will appear at the bottom right corner showing the progress.

When finished, a new window will appear on the bottom right asking if you want to open the recently created project in a new window.  Click Yes.

After you create the Project and open it in VS Code, you’ll see the project structured as shown above.

Set up the Golioth platform as the CoAP destination

You will not need to change the the sample code contained in the main folder to make this tutorial work. The only change needed is to create a configuration file and set some variables.

To create the complete config file, we’ll use the menuconfig tool that the ESP-IDF extension makes available for us. Click the gear button in the VS Code bottom tool bar:

A new SDK Configuration Editor window will open:

We will set some configs in this windows.

First, at the left menu bar, click the Example CoAP Client Configuration:

Next, set the following values:

  • Target Uri to coaps://coap.golioth.net/hello
  • Preshared Key (PSK) to the PSK of your device which you’ll find in the Golioth Console
  • PSK Client Identity to the PSK-ID of your device which you’ll find in the Golioth Console
  • WiFi SSID and WiFi Password to the name and password of your WiFi network

There’s one last setting needed before saving your configuration. In the left menu bar, go to CoAP configuration and set the CoAP Encryption method to Pre-Shared Keys.

Finally, click save.

When your configuration is saved, the ESP-IDF extension will make some changes in your structure. It creates an sdkconfig.old file that contains the previous config before saving the new one, and also a new sdkconfig file that contains all the new configuration variables.

At this point, our tutorial is ready to be built and flashed to the device. If you check the Golioth Console before you first run the example, you’ll notice the device is reported as offline. If we did everything right, that’s about to change!

Run the CoAP example on ESP32

To build the firmware, click the cylinder button at the bottom tool bar and wait for the build process finish:

After the build finishes, ensure your device is connected to the USB port, click the flash button and, again, wait for the process to finish:

At this point, you have the firmware running on your device. But how can we see if it’s running properly?  Just click the monitor button, and you’ll start to see all the logs coming from your device through the serial port:

Above, you see some logs generated when the device is booting which includes some interesting information about the device.

Below, you can see logs related to the Golioth communication itself. As we set our Target Uri to coap.golioth.io/hello, this endpoint sends back a hello message every time the device connects to it:

So, from the device perspective, you can see that it managed to connect to the Golioth, right?

But let’s see from the Golioth Console perspective. Go to the Device section in the console and you will see the device is now reported as online:

Conclusion

At this point we can see that connecting a device to the Golioth using the Espressif ESP-IDF and the CoAP protocol is not hard. This post serves as a “Hello World!” to get the connection up and running. Now that you’ve done that, you can easily transfer data between your ESP32 and Golioth.

Espressif has included a lot of examples with their VS Code extension that will help you better understand how to use it and are worth further study. In my next tutorial I’ll continue using the CoAP protocol but this time we’ll see how to send a LightDB Stream package, which includes time-series data perfect for visualizing the information being collected by your devices.

Golioth has premier support for certain hardware platforms, including the Espressif ESP32 platform. In this post and associated videos, we will show you how to get started using Golioth and Zephyr on the ESP32 platform.

Re-Introducing Support for the ESP32

Long time Golioth watchers will note that we once had older versions of the videos below on our YouTube channel. Everybody is doing reboot movies these days, right?

We recorded new videos to showcase the updated onboarding process for developers to get started, including directions for getting started using Golioth. Previous videos showed a command line interface as the main way to interface to the Golioth cloud. Then and now, users could use goliothctl to query devices and pull data from the data streams on the Cloud such as LightDB State and LightDB Stream. Since then, the Golioth Console is the recommended way for users to get started. The graphical interface of the Golioth Console makes it even easier to add and provision new devices to a user’s Dev Tier account. A couple of clicks and you have a set of credentials that you can then add to your actual device (in this case, an ESP32).

Now that your device is provisioned, our Getting Started with ESP32 video shows an updated installation of Zephyr RTOS and how it can connect to the Golioth Cloud to send your first “hello” message. This is a walkthrough of our written directions on the topic, available on our docs site. There is a secondary video that shows the full walkthrough of installing the Zephyr toolchain for the ESP32.

Why we like the ESP32 and Zephyr

So why the ESP32? As a hardware designer, I don’t think it shines in any particular area, except for one. It is almost never the lowest power WiFi solution, nor is it the most powerful processor. It’s definitely a low cost processor, but high volume applications often have access to a range of parts that can meet price targets. It’s not the tooling, as it even uses a less common Xtensa core instead of an Arm core. What I like most about it is: you can buy it. In the current chip shortage, we have seen the ESP32 “on the shelves” throughout the time we have been developing.

On the software side, we like that the Espressif team has also enabled Zephyr support. They are still working on adding new features, including MCUboot support, which enables Over-The-Air updates for ESP32 based projects with Golioth. Zephyr is not their primary software support, as Espressif develops ESP-IDF, which is a custom FreeRTOS implementation for ESP32-based parts. While we don’t support ESP-IDF directly in our SDK, we have content coming out soon showing how users can directly connect to the Golioth cloud using ESP-IDF over CoAP. Overall, we like that the Espressif team and the surrounding community is devoted to extending capabilities of their parts to many different types of software and Real Time Operating systems, and giving it back to the community. We try to match this ethos.

Should you try the ESP32?

The ESP32 represents an affordable, available platform for you to try out Golioth on a real device. If you follow our docs or follow along with the videos below, you can expect to have a fully functional IoT device by the end of the tutorials. From there, you can go from a single prototype to scaling up a fleet of thousands of devices without needing a “cloud team” to manage your various services on the cloud.

Videos

The line between the hardware and software worlds continues to blur. One advantage is the range of software tools that are coming into the hardware and firmware space, such as “Continuous Integration, Continuous Deployment” (CI/CD). This ensures each commit of code to a repository is immediate run against a slate of tests, compiled using a standardized compiler, and deployed to eligible devices for testing. What if your IoT firmware deployments happened automatically just by typing git push?

In this post and the associated video, Lead Engineer Alvaro Viebrantz talks about a sample project that compiles and delivers firmware to eligible devices automatically using GitHub Actions. Follow along in our repository on the Golioth Labs page.

Note: The arduino-sdk repository showcased in this post is deprecated. GoliothLabs has two experimental repositories that may work as replacements:

Compiling in the cloud?

Most embedded engineers hear “cloud compilation” and start shaking their head no. Having an IDE or a local toolchain is the expectation for fast iteration and direct debugging of code using tried and true methods. But being “local only” also runs into dependencies that might be on your machine versus your co-worker’s machine. What’s worse, you might be lulled into a false sense of security and never venture past having a single machine that is capable of compiling critical device firmware. (Don’t believe me? Ask a long time firmware engineer about the steps they had to go through to keep that old computer running to compile code for the legacy project.)

Moving your final (production) builds to the cloud is beneficial because it removes localized dependencies. Distributed teams are increasingly common. Now when an engineer in Brazil, the US, Poland–or any other place your teammates might be–commit code to a repo, it receives the same treatment. If you have GitHub actions (or equivalent hooks) set up on your repository, it kicks off a compilation on a container on the cloud. The output firmware is then uploaded to Golioth. As we see in the video, it’s still possible to build on a local machine, but the standardization is very helpful as you move towards scaling your product and making production grade firmware.

Walking through the demo

Let’s recap the steps taking place in the video and discuss what is relevant about each step.

  • Build firmware locally
    • This showcases that while the toolchain is available on the cloud, it’s not only on the cloud. Developers can still create a local image for testing.
  • Load initial firmware to the device
    • Since the firmware is responsible for reacting to new firmware on the Golioth Cloud, we need to load an initial image to start checking whether an update is available. As Alvaro mentions, the firmware recognizes that the same version loaded on the device initially is also the latest on the cloud, so no action is taken.
  • Creating device credentials using the hardware ID
    • Golioth allows you to set a range of different IDs to help identify devices in the field. This is especially important as your fleet grows. As part of the provisioning process, Alvaro shows how the code on the firmware image can also extract a unique identifier that is programmed into the silicon at the factory, so it is possible to always verify which device is being identified, all the way down to the silicon.
  • Generate device credentials on Golioth
  • Set WiFi/Golioth credentials over the Console
    • Setting device credentials allows the Golioth Cloud to validate a device is able to talk to the network. Alvaro demonstrates using a serial connection with the device to add these credentials, a recent improvement that will enhance device programming in the factory or as users provision a new device.
  • Device connects and begins to send logs and LightDB State data
  • Make a small change and commit to the repo. Push a git tag.
    • Pushing the change to the repository and adding a tag is what alerts the GitHub actions logic to start compiling a new firmware image.
  • The firmware is built in the cloud as part of GitHub actions.
    • For this build, Alvaro is using our Arduino SDK along with PlatformIO. This is in a container that the GitHub action boots up and uses to standardize the firmware build.
  • The artifact is pushed to the Golioth cloud
    • As part of the setup process, the user will need to generate an API key to place on GitHub in order to allow GitHub to push the new firmware issue as an Artifact on Golioth.
  • Create a release using the Console and roll out the firmware to the eligible device.
    • This is a manual process in the video, but a user could also utilize the REST API in order to create a Release and set the release to be eligible for rollout to a set of devices that are tagged on Golioth (different than GitHub tags). Remember: Any action possible on the Console can be scripted using the REST API.

Extending the demo

One theme that is obvious throughout this video is the focus on moving your product to production. Programming devices as they come off the line is no small task and something we will continue to make content about here at Golioth. And future videos will describe methods for running real-world tests on hardware, something we are interested in given our support of hundreds of boards.

In the current demo, Alvaro shows loading credentials over serial. Past examples have also shown Bluetooth credentialing using MCUmgr. We’d love to hear more about how you’re creating your devices and how you want to program things as you move towards production. Please join us on our Discord or on our Forums to discuss more.

Looking for direct help getting your devices into production? Reach out at

Golioth is a flexible device management solution that allows you to manage devices that have IP addresses. This includes devices connecting over Ethernet, cellular, and Wi-Fi. Today we’re showcasing a demo of a Thread-based device. Utilizing the Golioth Console and API endpoints, it’s possible to manage a wide array of devices provisioned on the internet through a Thread Border Router. This enables even more low-power devices to reliably push data back to the internet for processing on the cloud.

In this post and the associated video, we show a demo of how developers can try out using Golioth to manage Thread based devices.

What is Thread?

Thread is a network protocol for low power IoT devices. It uses 6LoWPAN for mesh communications, in our case on 2.4GHz. In the examples above, we showcase Thread using OpenThread on an nRF52840 from Nordic Semiconductor. Multiple hardware vendors have Thread-based solutions, using Zephyr or other firmware solutions. Vit Prajzler shows how he is using the OpenThread documentation to build different types of devices that form a Thread mesh network, including a Radio Co-Processor (RCP) and a Full Thread Device (FTD).

Set up a Border Router

A Border Router is an element on a Thread Mesh network that allows the entire network to communicate with the broader internet. Each individual device (Node) has an IPv6 address, which allows Golioth to manage specific devices on a mesh.

While it’s possible to buy a few off-the-shelf Border Routers, we showcase building one using commonly available components. Vit is using a Linux computer (a Raspberry Pi), along with an nRF52840 dongle configured as an RCP to route network traffic from the mesh to the rest of the internet. In the case of his Thread network, Vit needs to include a translation layer (called NAT64) to talk from his IPv4-based router to the IPv6-based Thread network. In the future, when Vit gets an IPv6 address assigned to his home network from his ISP, no conversion will be required.

OpenThread software on a Linux device like a Raspberry Pi also has a web interface for managing connected devices. This is a useful way to visualize your network and the location of the Border Router within that network.

Using the Golioth interface in the Zephyr SDK

Once your Border Router has been set up, you can use the Golioth Zephyr SDK to compile an image for your device using OpenThread. In the video above, Vit is compiling for an nRF52840 (dongle). Once this device is loaded with Zephyr-based firmware and added to the mesh network–in this example, connecting directly to the Border Router–the device acts like any other IP connected device. It has an IPv6 address and can ping the address of the Border Router. It can also connect to various Golioth endpoints.

Vit demonstrates button presses and boot commands being logged in Zephyr, which then display over the UART. However, Zephyr logs are also tied to the Golioth logging end point (from the Thread network connection), so they can be viewed on the Golioth web console. As Vit presses a button on the Zephyr based device, the log message almost instantaneously shows up on the cloud.

Extending the examples

Towards the end of the video, Chris and Vit discuss how (Open)Thread-based devices built with Zephyr work very similarly to any other IP-based devices connecting to Golioth. This means that the other features that Golioth provides–such as control using LightDB State, or easy time series data tracking using LightDB Stream–is already in there. And yes, that means Firmware Updates as well. We’ll be showcasing these more in future videos.

We’re excited to bring Golioth and internet connectivity to even more devices, including the kind of teeny tiny low power ones that Thread enables. Please stop by the Golioth Discord or the Golioth Forums to ask questions and discuss the next device you’re working on!

Interested in Thread in a commercial application? Contact us at