How to parse JSON data in Zephyr

Storing and retrieving data from the cloud is the foundational concept of the Internet of Things. When two machines talk to one another they need to settle on a data format. Here at Golioth that means your microcontroller of choice is going to be sending and receiving JSON.

Wait a minute. JSON and microcontrollers? Take a breath, dry those sweaty palms, and keep reading! When it comes to working with the C language, parsing strings for punctuation delimiters doesn’t sound like much fun. That’s why it’s really nice that Zephyr has a built-in JSON library which does most of the work for you. The only thing you really need to do is to make a struct that tells Zephyr how the incoming data will be organized. At first blush this seems tedious, but what you get out of it is validation of both the key and the received value type.

The nice thing is that once you get the hang of it, parsing data and accessing those values becomes very easy. So today I’ll walk you through the process.

Why do we need to parse JSON?

It’s very easy to get data to and from the Golioth Cloud. One of the first examples you should try out is the lightdb/get sample that simple asks for an endpoint called /counter. The string you’ll receive is a key-value pair that looks like this:

{"counter":18}

For data this simple it would be trivial to iterate the string, look for a semicolon, and then test the following characters to make sure they are numbers. But that’s clunky, and breaks down when you start adding more key-value pairs and nested values. The JSON module will make sure the value you receive matches the key you were expecting, and that the variable type (string, int, or bool) is validated. You’ll want to use this library for sending complex data back to the cloud too. It includes an encoder that will build the JSON packet for you.

Configuring the JSON library

Enabling the library in your project is very simple. First, turn on the library in your prj.conf file:

CONFIG_JSON_LIBRARY=y

Next you include the header file in main.c of your app:

#include <data/json.h>

Setting up the struct

This part is a bit hard to wrap your head around because the helper code looks like a foreign language compared the readability of a JSON object. Here is the overview of what we need to accomplish in this section:

1. Build a set of all key-value pairs. This includes the name of the key and the data type of the value.
2. Package up all of the key-value pairs into a struct (which might itself include other structs) to match the way the expected JSON package will be structured.
3. Tell the library about our struct, which it will use as a map to encode or parse the JSON string.
4. Give the library a pointer to store the data. For encoding, this is a string pointer, for decoding this is a struct pointer.

Here’s a simple packet that might be received by a temperature controller listening to the Golioth Cloud for user settings:

{
  "unit": "c",
  "value": 37
}

In our c code we begin by describing all of the key-value pairs we expect to find in the JSON packet. Notice that we’re declaring variables. The type of the variable must match the expected data type of the JSON value. The name of the variable must match the expected name of the JSON key. Note that this isn’t something inherent about C; it’s how the library matches up the incoming data and rejects things that don’t fit that mold.

struct temperature {
  const char *unit;
  int value;
};

Next we declare a struct that will match the expected structure of the JSON packet. We invoke the JSON_OBJ_DESCR_PRIM() macro for each key-value pair. Here you can see we feed it temperature, the name of the field name from the struct, and a token that indicates the data type.

static const struct json_obj_descr temperature_descr[] = {
  JSON_OBJ_DESCR_PRIM(struct temperature, unit, JSON_TOK_STRING),
  JSON_OBJ_DESCR_PRIM(struct temperature, value, JSON_TOK_NUMBER),
};

Parsing JSON

Now we have a struct that contains all of our expected keys, and a descriptor struct that maps out the expected structure of the JSON packet. We’re ready to test it out!

Here is a concise bit of sample code to test out our setup. Note that I’m using Zephyr’s logging system to display values.

/* decode a single object */
uint8_t json_msg[] = "{\"unit\":\"c\",\"value\":30}";
struct temperature temp_results;

ret = json_obj_parse(json_msg, sizeof(json_msg),
				temperature_descr,
				ARRAY_SIZE(temperature_descr),
				&temp_results);

if (ret < 0)
{
	LOG_ERR("JSON Parse Error: %d", ret);
}
else
{
	LOG_INF("json_obj_parse return code: %d", ret);
	LOG_INF("Unit: %s", temp_results.unit);
	LOG_INF("Value: %d", temp_results.value);
}

On line 5 of the example we call json_obj_parse(), passing it our JSON string, length of that string, the descriptor we previously set up, the length of that descriptor (number of elements), and a copy of the struct where the results will be stored. The rest of the code tests whether an error code is being returned and prints out the values if the parsing was successful.

This works if the data is just right. However, checking for a negative error code isn’t enough. For instance, if valuekey of the JSON packet is received as a string ("30") instead of an int (30), the return code will not be negative, but the program will crash at runtime. This is a feature, not a bug, and it lets us parse JSON even when it’s not quite right. But we need to do more to test that the data is valid. Let’s do that, as well as looking at an example of parsing nested data.

Nested JSON

Our heater control example probably needs more than a temperature setting, it needs an on/off setting. Here’s what that JSON might look like:

{
  "heater_on": true,
  "heater_temp": {
    "unit": "c",
    "value": 30
  }
}

We build the struct in much the same way, except there are two steps here. First we set up the struct (and the descriptor) for the inner “heater_temp” object, then set up another struct and descriptor and maps the “heater_on” key/value and the “heater_temp” object:

struct temperature {
	const char *unit;
	int value;
};

struct heater_ctl {
	bool heater_on;
	struct temperature heater_temp;
};

static const struct json_obj_descr temperature_descr[] = {
	JSON_OBJ_DESCR_PRIM(struct temperature, unit, JSON_TOK_STRING),
	JSON_OBJ_DESCR_PRIM(struct temperature, value, JSON_TOK_NUMBER),
};

static const struct json_obj_descr heater_unit_descr[] = {
	JSON_OBJ_DESCR_PRIM(struct heater_ctl, heater_on, JSON_TOK_TRUE),
	JSON_OBJ_DESCR_OBJECT(struct heater_ctl, heater_temp, temperature_descr),
};

Note that we’re using the JSON_OBJ_DESCR_OBJECT()that maps the temperature descriptor for the nested data. Now we can parse our data:

uint8_t str[] = "{\"heater_on\":true,\"heater_temp\":{\"unit\":\"c\",\"value\":30}}";
struct heater_ctl heater_settings;
int expected_return_code = (1 << ARRAY_SIZE(heater_unit_descr)) - 1;
int ret = json_obj_parse(str, sizeof(str),
				heater_unit_descr,
				ARRAY_SIZE(heater_unit_descr),
				&heater_settings);

if (ret < 0)
{
	LOG_ERR("JSON Parse Error: %d", ret);
}
else if (ret != expected_return_code)
{
	LOG_ERR("Not all values decoded; Expected return code %d but got %d", expected_return_code, ret);
}
else
{
	LOG_INF("json_obj_parse return code: %d", ret);
	LOG_INF("calculated return code: %d", expected_return_code);
	if (heater_settings.heater_on)
	{
		LOG_INF("Heater On: True");
	}
	else
	{
		LOG_INF("Heater On: False");
	}
	LOG_INF("Unit: %s", heater_settings.heater_temp.unit);
	LOG_INF("Value: %d", heater_settings.heater_temp.value);
}

You must use JSON parse return codes to validate your data!

The json_obj_parse() function is going to return a positive-value that correlates to which tokens of the JSON object were successfully found and validated. Each token is represented by one bit in the return code.

The nested JSON example above presents a gotcha. We expect the parser to report back on the three tokens that are important to us (heater_on, unit, and value; in that order). What it actually does is report back on the tokens found in the top-level struct. So in this case a return code indicates that the parser found heater_on and heater_temp, the key for the nested data. It might have also found unit and value, or they may not have been present. We just don’t know for sure.

The solution when decoding json is to pretend the key to the nested struct doesn’t exist. Instead, we can just declare our important values:

struct temperature {
	bool heater_on;
	const char *unit;
	int value;
};

static const struct json_obj_descr temperature_descr[] = {
	JSON_OBJ_DESCR_PRIM(struct temperature, heater_on, JSON_TOK_TRUE),
	JSON_OBJ_DESCR_PRIM(struct temperature, unit, JSON_TOK_STRING),
	JSON_OBJ_DESCR_PRIM(struct temperature, value, JSON_TOK_NUMBER),
};

We will receive a return code indicating whether the keys in the descriptor were successfully decoded–the parser will set the corresponding bit when each value is validated. So we want to see bits 0, 1, and 2 set in the return code (0b111). If heater_on is not validated, we would received 0b110. Here is an illustration of the gotcha (top return code) and the fix (bottom return code).

It’s really important to check these bits before using the value. If we fail to do so, we’ll be using uninitialized values (bad data) or reading from unallocated memory (runtime crash).

So why did I even show you how to build structs for nested JSON? You need it when encoding data. The json_obj_encode() function will take the nested descriptor and encode a JSON string exactly as we expect to see it. It doesn’t matter as much when you’re in control of the data scheme used on the cloud side, but if you need to match an existing standard, this makes the encoding a snap.

Things to keep in mind, and further reading

When working with this library, remember that the variables you declare in structs must match the keys in the JSON string. The values must also match up. The JSON library currently supports three data types: string, int, and bool. In our example, you would probably want to use a float for Celsius temperature settings, but it’s not possible to parse that data type with this library.

If you’re ambitious, adding this support would be a great way to contribute to the Zephyr open source project! But we can work around the problem. When accessing lightDB state values on Golioth, it’s possible to directly request the value just by using the specific endpoint–in our example: .d/heat_control/heater_temp/value.

The Zephyr JSON library includes other very helpful features like array-handling. I haven’t been able to find additional documentation on these features beyond the the JSON API reference, but the automated tests are a great place to see all functions/macros at play. There is a ton of utility built into this and it’s worth getting to know the library by building a few examples. Once you get the hang of it, this a very accessible way to make sure you can use incoming JSON data and know that you have dependable values.

See it in action

Check out our recent video reviewing some of the JSON basics described above!