,

How to Add Golioth LightDB Observe to any Zephyr application

One of the most useful services in the Golioth Zephyr SDK is the ability to observe data changes on the cloud. A device can register any LightDB endpoint and the Golioth servers will notify it whenever changes happen. If your device is a door lock, an example endpoint might be “lock status”, which you would want to know about a server-side state change immediately.

This is slightly more complex to set up than something like a LightDB ‘Set’ API call. ‘Observe’ requires a callback function to handle the asynchronous reply from the Golioth servers. Today we’ll walk through how to add Golioth LightDB Observe functionality to any Zephyr application by:

  1. Adding a callback that is called every time observed data changes
  2. Registering the callback with a data endpoint
  3. Using on_message to trigger the callback
  4. Ensuring that on_message and on_connect are both registered

These techniques are all found in our LightDB Observe sample code which acts as the roadmap for this article.

Prerequisites

Today’s post assumes that you already have a device running Zephyr and you have already tested out an app that uses the Golioth Zephyr SDK. If you’re not there yet, don’t worry. You can sign up for our free Dev Tier that includes up to 50 devices, and follow the Golioth Quickstart Guide.

Your Zephyr workspace should already have Golioth installed as a module and your app (probably in main.c) is already instantiating a Golioth system client. Basically, you should see a block like this one somewhere in your code:

#include <net/coap.h>
#include <net/golioth/system_client.h>
static struct golioth_client *client = GOLIOTH_SYSTEM_CLIENT_GET();

If you don’t, checkout out our How to add Golioth to an existing Zephyr project blog post to get up to speed before moving on.

1. Add a callback function for observed data changes

The goal of this whole exercise is to enable your device to perform a task whenever data changes at your desired endpoint. Remember: Golioth LightDB endpoints are configurable by you! Whatever data you’d like to monitor, you can customize it to your needs.

The first thing we’ll do is create a callback function that will perform the task.

static int on_update(const struct coap_packet *response,
             struct coap_reply *reply,
             const struct sockaddr *from)
{
    char str[64];
    uint16_t payload_len;
    const uint8_t *payload;

    payload = coap_packet_get_payload(response, &payload_len);
    if (!payload) {
        LOG_WRN("packet did not contain data");
        return -ENOMSG;
    }

    if (payload_len + 1 > ARRAY_SIZE(str)) {
        payload_len = ARRAY_SIZE(str) - 1;
    }

    memcpy(str, payload, payload_len);
    str[payload_len] = '\0';

    LOG_DBG("payload: %s", log_strdup(str));

    return 0;
}

 

The majority of this callback is used to verify that string data was received from Golioth. Ultimately, line 22 is what you are interested in. For this example, we’re printing a log message with the string payload stored in the str array.

If your endpoint contains more than just one value, it may be useful to parse the JSON object and store the values. Also keep in mind that this callback will execute on the golioth system client thread, which is a different thread than the “main” thread running your application. This means:

  • The callback function should return quickly (under 10 ms). If that’s not enough time, you can use a Zephyr Workqueue to schedule the work on another thread.
  • If access to global data is required, access to the data must be protected by a mutex to avoid data races between threads.

2. Add coap_reply and register the callback in on_connect

We need to create an array of structs to store messages that arrive from Golioth. This struct is then registered with an endpoint and the callback we created in step 1.

#include <net/coap.h>

First ensure that you have included the CoAP header from Zephyr which defines the coap_reply struct and has some handy helper functions.

static struct coap_reply coap_replies[1];

Here I’ve created a coap_replies array with just one member because my example observes one single endpoint. If you want to observe multiple endpoints, you will need multiple callbacks and you must have one coap_reply for each callback.

static void golioth_on_connect(struct golioth_client *client)
{
    struct coap_reply *observe_reply;
    int err;

    coap_replies_clear(coap_replies, ARRAY_SIZE(coap_replies));

    observe_reply = coap_reply_next_unused(coap_replies, ARRAY_SIZE(coap_replies));

    /*
     * Observe the data stored at `/counter` in LightDB.
     * When that data is updated, the `on_update` callback
     * will be called.
     * This will get the value when first called, even if
     * the value doesn't change.
     */
    err = golioth_lightdb_observe(client,
                      GOLIOTH_LIGHTDB_PATH("counter"),
                      COAP_CONTENT_FORMAT_TEXT_PLAIN,
                      observe_reply, on_update);

    if (err) {
        LOG_WRN("failed to observe lightdb path: %d", err);
    }
}

Now we register the observation using the golioth_lightdb_observe() API call. The parameters passed to this function include:

  1. The endpoint to observe (the GOLIOTH_LIGHTDB_PATH macro indicates this is a LightDB State endpoint)
  2. The format, either plain text or CBOR (see the LightDB LED sample which demonstrates using CBOR serialization)
  3. A coap_reply struct to store the message and metadata received from Golioth
  4. The callback function to execute when a message is received from this endpoint

Notice that the helper function coap_reply_next_unused() is called to get the next available struct. This is important if you are registering multiple callbacks and should be used prior to each golioth_lightdb_observe() API call to get a pointer to a struct that isn’t already associated with another callback.

3. Add the processing function to on_message

This step is small but important, and seems to be the one I frequently forget and then scratch my head when my callback isn’t working.

Whenever a message is received from Golioth, the Golioth system client executes a callback that we usually call on_message. For our observed callbacks to work, we need to tell on_message about our coap_replies array.

static void golioth_on_message(struct golioth_client *client,
                   struct coap_packet *rx)
{
    /*
     * In order for the observe callback to be called,
     * we need to call this function.
     */
    coap_response_received(rx, NULL, coap_replies,
                   ARRAY_SIZE(coap_replies));
}

 

By calling Zephyr’s coap_response_received(), the CoAP packet will be parsed and the appropriate callback will be selected from the coap_replies struct (if one exists).

4. Ensure on_message and on_connect are both registered

The final step is to make sure that we’ve registered callbacks for when the Golioth system client connects and receives a message.

client->on_connect = golioth_on_connect;
client->on_message = golioth_on_message;
golioth_system_client_start();

This should be done in main() before the loop begins. The golioth_client struct should have already been instantiated in your code, in this example it was called client. The code above associates our two callbacks and starts the client running.

Observed data in action

Now that we’ve tied it all together, let’s test it out. Here’s the terminal output of my Zephyr app:

[00:00:09.328,000] <dbg> golioth_lightdb: main: Start Light DB observe sample
[00:00:09.328,000] <inf> golioth_system: Starting connect
[00:00:09.537,000] <inf> golioth_system: Client connected!
[00:00:09.739,000] <dbg> golioth_lightdb: on_update: payload: null
[00:00:48.653,000] <dbg> golioth_lightdb: on_update: payload: 42

You can see that at boot time, the observed data will be reported, which is great for setting defaults when your device first connects to observed data. When the endpoint was registered it didn’t exist on Golioth so a payload of null was returned. About 49 seconds later a payload of 42 is received. That’s when I added the endpoint and value in the Golioth Console.

On the cloud, this is an integer, but the device receives payloads as strings. You’ll need to validate received data on the device side to ensure expected behavior in your callback functions (beyond simply printing out the payload as I’m doing here). Give it a try for yourself using our LightDB Observe sample code.

Observing LightDB data gives your devices the ability to react to any changes without the need to poll like you would if you were using the golioth_lightdb_get() function. In addition to being notified each time the data changes, you’ll also get the current state when the observation is first registered (ie: at power-up). Single endpoints, or entire JSON objects can be observed, making it possible to group different types of state data to suit any need.

If you still have questions, or want to talk about how LightDB Observe works under the hood, head over to the Golioth Forum or ping us on the Golioth Discord.