CBOR Serialization in the Golioth Zephyr SDK: 20% Data Savings!

Golioth provides a lot of different services for your IoT fleet. Under the hood they all boil down to one thing: transferring some type of data to or from a constrained device. Whether it’s your microcontroller-based sensors sending back readings, or the new settings from the cloud being pushed to a device via Remote Procedure Call (RPC), it’s all data transfer. That data should be as efficient as possible, which is why Golioth uses CBOR for serialization. This saves bandwidth and radio-on time (ie: battery life). So you’d be wise to use CBOR in your application code. Let’s dive into an example!

What is CBOR?

CBOR is the Concise Binary Object Representation. It uses the JSON data model, but packs the data more tightly. The result is not human readable, but that’s the point. IoT networks are basically robots talking to other robots, we want to tailor our data packets with that in mind. Golioth makes it easier by doing the work under the hood (in our SDK) so you don’t have to think about it…you just reap the data savings.

Sending Data with CBOR

Let’s stream some data to Golioth using the LightDB Stream sample. Our data for this will include the following:

int32_t neg_reading = −2147483647
double temperature = 23.45
double pressure = 100.133019
double humidity = 32.204101
double accX = 0.480525
double accY = 0.156906
double accZ = -9.100571
char text[] = "Golioth"

If we were sending this as JSON, it would look pretty nice:

{
    "neg_reading": -2147483647,
    "weather": {
        "temperature": 23.45,
        "pressure": 100.133019,
        "humidity": 32.204101
    },
    "accelerometer": {
        "accX": 0.480525,
        "accY": 0.156906,
        "accZ": -9.100571
    },
    "text": "Golioth"
}

But we can reduce the data footprint with CBOR encoding by using the QCBOR library. This library is already installed in your workspace because the Golioth SDK uses it for all of the behind-the-scenes communications. For instance, when sending Remote Procedure Calls (RPCs), Device Settings information, and the firmware manifest for our Over-the-Air (OTA) firmware update service. Our resident Zephyr expert Marcin gave a talk at ZDS 2022 where he described in depth how we use CBOR as part of our logging service as well. Anywhere it makes sense for us to save you data, we do that for you.

CBOR Encoding Example Code

#include <qcbor/qcbor.h>

UsefulBuf_MAKE_STACK_UB(Buffer, 300);
QCBOREncodeContext EncodeCtx;
QCBOREncode_Init(&EncodeCtx, Buffer);

QCBOREncode_OpenMap(&EncodeCtx);
QCBOREncode_AddInt64ToMap(&EncodeCtx, "neg_reading", -2147483647);
QCBOREncode_OpenMapInMap(&EncodeCtx, "weather");
QCBOREncode_AddDoubleToMap(&EncodeCtx, "temperature", 23.45);
QCBOREncode_AddDoubleToMap(&EncodeCtx, "pressure", 100.133019);
QCBOREncode_AddDoubleToMap(&EncodeCtx, "humidity", 32.204101);
QCBOREncode_CloseMap(&EncodeCtx);
QCBOREncode_OpenMapInMap(&EncodeCtx, "accelerometer");
QCBOREncode_AddDoubleToMap(&EncodeCtx, "accX", 0.480525);
QCBOREncode_AddDoubleToMap(&EncodeCtx, "accY", 0.156906);
QCBOREncode_AddDoubleToMap(&EncodeCtx, "accZ", -9.100571);
QCBOREncode_CloseMap(&EncodeCtx);
QCBOREncode_AddSZStringToMap(&EncodeCtx, "text", "Golioth");
QCBOREncode_CloseMap(&EncodeCtx);

UsefulBufC EncodedCBOR;
QCBORError uErr;
uErr = QCBOREncode_Finish(&EncodeCtx, &EncodedCBOR);

if (uErr != QCBOR_SUCCESS) {
    LOG_ERR("Failed to encode CBOR: %d", uErr);
} else {
    LOG_DBG("CBOR data size is: %d", EncodedCBOR.len);
    LOG_HEXDUMP_DBG(EncodedCBOR.ptr, EncodedCBOR.len, "CBOR Object");

    err = golioth_stream_push_cb(client, "cbor_test",
                     GOLIOTH_CONTENT_FORMAT_APP_CBOR,
                     EncodedCBOR.ptr, EncodedCBOR.len,
                     async_error_handler, NULL);
    if (err) {
        LOG_WRN("Failed to push to LightDB Stream: %d", err);
        return;
    }
}

This code was put together by following the example.c file in the QCBOR library. It begins by allocating a 300 byte stack the library will use during encoding.

A map is then opened on line 5 using QCBOREncode_OpenMap(). This is similar to an opening-curly-bracket in JSON (think of maps like JSON objects). We can then add various data types to the map.

When it comes time to encode the "weather" data, we need another map nested inside the original. We call QCBOREncode_OpenMapInMap() on line 7, assigning a key to the object ("weather"), before adding temperature/pressure/humidity data with the  QCBOREncode_AddDoubleToMap() function.

Note that we close nested maps when done adding data using the QCBOREncode_CloseMap() function. It is also called at the very end to close the outermost map.

All of our data is held as a map until the encode step that actually creates the CBOR package. On line 20 we set up a UsefulBuf—a struct from the library that simply holds a pointer and a length. On line 22, QCBOREncode_Finish() is called to finish the encoding process. Our UsefulBuf is then used to log the packet and send it to Golioth.

When it comes to the Golioth API, the only difference between using JSON and using CBOR is the last four characters on line 31: GOLIOTH_CONTENT_FORMAT_APP_CBOR

When Golioth receives this data, the CBOR is unpacked and displayed in JSON format so that it is human-readable:

Comparing CBOR serialization to JSON

If you remove all of the whitespace from our initial JSON string, you end up with 186 bytes, but when encoded using CBOR that is reduced to 154 bytes.

That savings of 32 bytes is a reduction of 17.2%. Extrapolate that to every reading, from every device, over the life of your fleet. This makes a significant impact on cellular data charges and the power budget used when sending data.

Trying it yourself, and further Reading

Give CBOR a try in your own projects. With Golioth’s Dev Tier, your first 50 devices are free just for registering an account.

You can drop the C code shared above into the Golioth LightDB Stream sample to replicate what we’ve shared in this post. If you are looking for examples of CBOR decoding, it’s used in the Golioth RPC sample as well as the LightDB LED sample.

To learn more about the QCBOR library, start with the README in the repo. From there dig into the header files in the inc folder for the Doxygen comments. As always, we’d love to know what you’re building and we’re happy to answer questions. Post your thoughts on the Golioth Forum.