Pretty much everyone that uses Golioth needs to send data from their IoT devices to the cloud. Most often this is sensor readings. But it doesn’t matter to us what the data is or where you need it to go. Golioth makes it easy to collect data from your IoT fleet and route that data different places based on a path supplied by the device along with configuration you establish in the cloud (pipelines).
For some of our customers, this data is quite large. Things like batch data consisting of many sensor readings along with timestamps, or just large payloads like images, audio, or video. Golioth handles large payloads in a way that is sensitive to constrained devices’ limitations. Those devices need to limit how much memory is used during the transmission. Today we’ll walk through how to use Golioth’s blockwise stream to send data from a device to the cloud.
Today’s post is for your first adventures with blockwise stream and we’ll only be using a synchronous API. However, an upcoming post will cover the more advances asynchronous API that is also available.
Streams and Pipelines, Oh My
When we refer to Stream we’re talking about a cloud service that receives data from devices in an IoT fleet. Sensor readings are a prime example: these are data packets sampled at different points in time that get pushed to the cloud.
Golioth uses a concept called Pipelines to transform and route this data. You must have a pipeline set up on the Golioth cloud that indicates what you want to happen with your data. For today, we’re going to use a Pipeline that listens on all paths for JSON-formatted data and routes it to Golioth’s own LightDB stream destination. But you can just as easily send CBOR or raw binary data and route it to GCP, Azure, AWS, your own server using webhooks, or a number of other available destinations.
To follow along with this guide, Make sure you have the Legacy LightDB Stream JSON pipeline example enabled on your Golioth project. If you don’t you can click this link to automatically add it.
Firmware Concepts for Blockwise Stream
Microcontrollers have RAM limitations that make it difficult to hold the entire data set in memory in order to stream it over your network connection. This is where the “blockwise” part comes in.
Golioth uses the CoAP protocol for securely transferring data to and from the cloud. The protocol has a payload limit of 1024 bytes per packet. The Golioth Firmware SDK implements an API that accepts 1024-byte blocks of data from your application, sending each to the cloud until the entire payload has been transferred. Your application needs to supply the data to the SDK using the following concepts:
- block index: the number of the current block in the overall transfer, beginning with 0.
- block size: the maximum size of a block (configurable with a default of 1024).
- data offset: the offset from the beginning of your data set from which to read the next block of memory. Your callback needs to calculate this by multiply the block index by the block size.
- buffer length: the length of the data (in bytes) supplied by your callback. This will be the same as the block size, except when sending the last block which may be smaller.
- last block: a boolean value your callback sets to indicate all data has been sent
Let’s apply these concepts in a practical firmware example.
Blockwise Stream Firmware Example
We want an example that’s easy to set up and easy to see when the data has arrived on the cloud. For this reason we’ll use some hardcoded JSON data. It’s human-readable, and we’ll send it to Golioth’s internal LightDB destination because you can view received data directly on your project page.
/* Sample text source: Mind Volume LIX, Issue 236 https://academic.oup.com/mind/article-abstract/LIX/236/433/986238 */
static const char json_data[] = "{\"my-quote\":\"1. The Imitation Game\\nI propose to consider the question, ‘Can machines think?’ This should begin with definitions of the meaning of the terms ‘machine’ and ‘think’. The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous. If the meaning of the words ‘machine’ and ‘think’ are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, ‘Can machines think?’ is to be sought in a statistical survey such as a Gallup poll. But this is absurd. Instead of attempting such a definition I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words.\\nThe new form of the problem can be described in terms of a game which we call the ‘imitation game’. It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart from the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either ‘X is A and Y is B’ or ‘X is B and Y is A’. The interrogator is allowed to put questions to A and B thus\"}";
static size_t json_data_len = strlen(json_data);
I’m using some simple example data that consists of a key-value pair in JSON format. The important thing is that the JSON itself is valid (otherwise it will be dropped by the server) and that the length of the data is correct.
Blockwise Async Callback and User Arguments
We will call a synchronous API but the firmware SDK uses an asynchronous callback to get each block of data from your application. This means that the API call will block your application program flow until all data has been sent.
This makes it really easy to keep the data in scope, but we need to make both the data pointer and the data length available in the async block read callback. You can do this with global variables but instead let’s pass a pointer as the callback argument. When the callback runs, it can cast the arg pointer to the correct type and access the data within.
struct block_stream_container {
uint8_t *data;
size_t data_len;
};
enum golioth_status stream_read_block(uint32_t block_idx,
uint8_t *block_buffer,
size_t *block_size,
bool *is_last,
void *arg)
{
if (NULL == arg)
{
LOG_ERR("Expected block_stream_container pointer as arg but got NULL");
return GOLIOTH_ERR_NULL;
}
struct block_stream_container *container = arg;
uint32_t offset = block_idx * *block_size;
/* Test to see if this is the final block of data we need to send */
if (container->data_len <= offset + *block_size)
{
/* Overwrite the block_size that was passed in with number of bytes in last block */
*block_size = container->data_len - offset;
/* Indicate to the Golioth Firmware SDK that this is the final block */
*is_last = true;
}
/* Copy our data into the supplied buffer */
memcpy(block_buffer, container->data + offset, *block_size);
LOG_INF("Sending block %u of %zu bytes.%s",
block_idx,
*block_size,
*is_last ? " Final block." : "");
return GOLIOTH_OK;
}
After testing to ensure the arg is not NULL it is cast to the block_stream_container we defined as part of the application. Now we can begin calculating the information we need to access the data.
The Golioth Firmware API supplied the parameters we need to target the next block of data. First, we can use block_idx and block_size to calculate the offset from the start of the data. This offset is where we begin reading after the start of the data set we are trying to upload.
Next, we need to know if there is enough data remaining to send a full block. Since the SDK has told us it wants block_size bytes of data, we first check to see if there’s more than that amount remaining (if so, this is not the last block). When we get to the end of the transfer, we will have a smaller block size (or exactly the block size and no more). In this case we need to set the is_last flag, and set the block_size to the actual amount of data for this final block.
With our calculations finished, we need to copy the data into the block_buffer supplied by the SDK. Return GOLIOTH_OK to show the data has been copied and is ready to upload. If there is a problem and you want to cancel the upload, return any other error code.
Initiating a Blockwise Stream
We’ve already done all the hard work in our callback. Initiating the transfer is rather simple.
static void send_blockwise_stream(void)
{
struct block_stream_container container;
container.data = (uint8_t *)json_data;
container.data_len = json_data_len;
enum golioth_status status = golioth_stream_set_blockwise_sync(client,
"my-test-path",
GOLIOTH_CONTENT_TYPE_JSON,
stream_read_block,
&container);
if (GOLIOTH_OK != status)
{
LOG_ERR("Failed to send blockwise stream: %d", status);
return;
}
LOG_INF("Blockwise data successfully sent.");
}
First we create the container struct we’re using to pass information to the callback. This just gets a pointer to the data and a total length. The API call then supplies the path to which we are sending the data; this may be anything you want but remember that the Golioth pipelines you have set up in the web console will key on this path. We also set the type to JSON, pass the callback function, and the address of our container struct which will be available in the callback.
This function will block until the send completes or there is an error (ensuring that our container struct remains in scope). Check the status code returned by this function to see if the operation was a success. Once complete, you may let the data go out of scope, the cloud already has it!
Putting it all together
You can run the example code for yourself. Start with the stream sample from the Golioth Firmware SDK, replacing the main.c file with the one below. Build, flash, and provision the code following the README in the SDK.
/*
* Copyright (c) 2025 Golioth, Inc.
*
* SPDX-License-Identifier: Apache-2.0
*/
#include <zephyr/logging/log.h>
LOG_MODULE_REGISTER(golioth_stream, LOG_LEVEL_DBG);
#include <golioth/client.h>
#include <golioth/stream.h>
#include <samples/common/sample_credentials.h>
#include <samples/common/net_connect.h>
#include <stdlib.h>
#include <string.h>
#include <zephyr/kernel.h>
static const char json_data[] = "{\"my-quote\":\"1. The Imitation Game\\nI propose to consider the question, ‘Can machines think?’ This should begin with definitions of the meaning of the terms ‘machine’ and ‘think’. The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous. If the meaning of the words ‘machine’ and ‘think’ are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, ‘Can machines think?’ is to be sought in a statistical survey such as a Gallup poll. But this is absurd. Instead of attempting such a definition I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words.\\nThe new form of the problem can be described in terms of a game which we call the ‘imitation game’. It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart from the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either ‘X is A and Y is B’ or ‘X is B and Y is A’. The interrogator is allowed to put questions to A and B thus\"}";
static size_t json_data_len = strlen(json_data);
struct golioth_client *client;
static K_SEM_DEFINE(connected, 0, 1);
static void on_client_event(struct golioth_client *client,
enum golioth_client_event event,
void *arg)
{
bool is_connected = (event == GOLIOTH_CLIENT_EVENT_CONNECTED);
if (is_connected)
{
k_sem_give(&connected);
}
LOG_INF("Golioth client %s", is_connected ? "connected" : "disconnected");
}
struct block_stream_container {
uint8_t *data;
size_t data_len;
};
enum golioth_status stream_read_block(uint32_t block_idx,
uint8_t *block_buffer,
size_t *block_size,
bool *is_last,
void *arg)
{
if (NULL == arg)
{
LOG_ERR("Expected block_stream_container pointer as arg but got NULL");
return GOLIOTH_ERR_NULL;
}
struct block_stream_container *container = arg;
uint32_t offset = block_idx * *block_size;
/* Test to see if this is the final block of data we need to send */
if (container->data_len <= offset + *block_size)
{
/* Overwrite the block_size that was passed in with number of bytes in last block */
*block_size = container->data_len - offset;
/* Indicate to the Golioth Firmware SDK that this is the final block */
*is_last = true;
}
/* Copy our data into the supplied buffer */
memcpy(block_buffer, container->data + offset, *block_size);
LOG_INF("Sending block %u of %zu bytes.%s",
block_idx,
*block_size,
*is_last ? " Final block." : "");
return GOLIOTH_OK;
}
static void send_blockwise_stream(void)
{
struct block_stream_container container;
container.data = (uint8_t *)json_data;
container.data_len = json_data_len;
enum golioth_status status = golioth_stream_set_blockwise_sync(client,
"my-test-path",
GOLIOTH_CONTENT_TYPE_JSON,
stream_read_block,
&container);
if (GOLIOTH_OK != status)
{
LOG_ERR("Failed to send blockwise stream: %d", status);
return;
}
LOG_INF("Blockwise data successfully sent.");
}
int main(void)
{
LOG_DBG("Start Golioth blockwise stream sample");
net_connect();
/* Note: In production, credentials should be saved in secure storage. For
* simplicity, we provide a utility that stores credentials as plaintext
* settings.
*/
const struct golioth_client_config *client_config = golioth_sample_credentials_get();
client = golioth_client_create(client_config);
golioth_client_register_event_callback(client, on_client_event, NULL);
k_sem_take(&connected, K_FOREVER);
/* Give network connection logs a few seconds to clear */
k_msleep(2000);
send_blockwise_stream();
return 0;
}
When monitoring the device output via a serial connection, you will see the log messages we added for each of the blocks:
*** Booting Zephyr OS build v4.1.0 *** *** Golioth Firmware SDK v0.20.0 *** [00:45:26.985,074] <inf> golioth_settings_autoload: Initializing settings subsystem [00:45:26.986,658] <inf> fs_nvs: 8 Sectors of 4096 bytes [00:45:26.986,675] <inf> fs_nvs: alloc wra: 0, fa0 [00:45:26.986,679] <inf> fs_nvs: data wra: 0, b6 [00:45:26.986,715] <inf> golioth_settings_autoload: Loading settings [00:45:26.986,912] <dbg> golioth_stream: main: Start Golioth blockwise stream sample [00:45:26.986,929] <inf> golioth_samples: Starting DHCP to obtain IP address [00:45:26.987,001] <inf> golioth_samples: Waiting to obtain IP address [00:45:35.004,339] <inf> net_dhcpv4: Received: 192.168.1.170 [00:45:35.004,816] <inf> golioth_mbox: Mbox created, bufsize: 1320, num_items: 10, item_size: 120 [00:45:35.269,926] <inf> golioth_coap_client_zephyr: Golioth CoAP client connected [00:45:35.270,055] <inf> golioth_stream: Golioth client connected [00:45:35.270,191] <inf> golioth_coap_client_zephyr: Entering CoAP I/O loop [00:45:37.270,335] <inf> golioth_stream: Sending block 0 of 1024 bytes. [00:45:37.322,012] <inf> golioth_stream: Sending block 1 of 349 bytes. Final block. [00:45:37.407,104] <inf> golioth_stream: Blockwise data successfully sent.
Once your device connects to Golioth, you can view the received stream data in the LightDB Stream tab of the device page.
Going Deeper with Blockwise Stream
Blockwise stream is a tool you should be using because it unlocks the ability to cache data locally into larger batches for upload at a convenient time. For instance, devices operating on battery power may opt to only upload data above a certain charge, and ambulatory devices may only have a network connection occasionally. Golioth’s data routing can separate readings using the batch destination and use the extract-timestamp transform to parse timestamps embedded in the data for accurate accounting of each reading.
While the example above is adequate for understanding how these block uploads work, in most cases you will not want to block program flow while uploading data. In an upcoming post we’ll cover using the asynchronous API, dynamic memory handling, and reading data from storage when using the blockwise stream feature.
No comments yet! Start the discussion at forum.golioth.io