Using Custom Work Queues for Sensor Readings in Zephyr

Real-time operating systems (RTOS) are incredible at taking a single processor and wisely handling a number of concurrent operations. However, they aren’t headache-free. When it comes to controlling access to hardware peripherals, the Zephyr RTOS offers a number of subsystems that make it easier to handle concurrency issues. Today I’ll discuss using a custom work queue to ensure the sensor readings can be requested at any time, from any thread, while avoiding hard faults.

Sensor Readings and Concurrency

The generic “sensor reading” term is a bit sloppy. What we’re actually talking about here is a bus with one or more devices connected to it. We want to make sure that there aren’t multiple threads trying to access the bus at the same time.

For the purposes of this example, we’re going to set up a custom work queue to take sensor readings for a sensor connected via i2c. With this approach we can create a different instance of work for each device on the bus and the queue will ensure that they are read in the order each request is received. The queue will be the only thing accessing this bus.

Zephyr has a built-in work queue enabled by default called the system work queue. This runs on the main system thread. Since we’re using blocking sensor reads when calling the fetch command, it’s a good idea to separate these calls from the main thread so we’ll stand up our own custom work queue.

A Work Queue to Take Sensor Readings

When renewing your license you wait in line at the Department of Motor Vehicles (DMV). The work queue functions in the same way. However, what we’re doing under the hood is creating a new thread whose sole purpose is to service a work item until the queue is empty. We’re creating the DMV itself (the work queue) and the people in line (the work).

When creating a thread, we need a stack to handle memory and we need to assign a priority to the thread so the scheduler knows how important our work is. In order to use the queue, we need to have work items that tell the queue which functions to run when processing a queue item.

const struct device *const temp_sensor = DEVICE_DT_GET(DT_ALIAS(ambient_temp0));

#define SENSOR_WORKQ_STACK_SIZE 512
#define SENSOR_WORKQ_PRIO 5
struct k_work_q sensor_workq;
struct k_work sensor_work;

K_THREAD_STACK_DEFINE(sensor_workq_stack, SENSOR_WORKQ_STACK_SIZE);

static void sensor_work_handler(struct k_work *work) {
    int err = 0;

    err = sensor_sample_fetch(temp_sensor);
    if (0 != err) {
        LOG_ERR("Failed to fetch sensor reading: %d", err);
        return;
    }


    struct sensor_value temperature;
    err = sensor_channel_get(temp_sensor, SENSOR_CHAN_AMBIENT_TEMP, &temperature);
    if (0 != err) {
        LOG_ERR("Failed to get sensor value: %d", err);
        return;
    }

    LOG_INF("Temperature reading: %d.%06d", temperature.val1, abs(temperature.val2));
}

void sensor_init(void) {
    k_work_queue_init(&sensor_workq);
    k_work_queue_start(&sensor_workq,
                       sensor_workq_stack,
                       K_THREAD_STACK_SIZEOF(sensor_workq_stack),
                       SENSOR_WORKQ_PRIO,
                       NULL);
    k_work_init(&sensor_work, sensor_work_handler);
}

The sample details the following:

• Values for the work queue stack size and priority
• A struct for the work queue itself, and a struct for the work we will submit to it
• A macro to allocate the stack memory
• The work handler function that will be called when the work item is processed
• An initialization function to init/start the queue and init the work

This is the meat and potatoes of the operation, and most of our job is already done. The sensor_work_handler() uses a familiar pattern to fetch the sensor value from the hardware, and then read the value out of the sensor_value struct where it was stored.

To use the new work queue, we run the initialization function and then we can begin submitting work.

sensor_init();

while (true)
{
    k_work_submit_to_queue(&sensor_workq, &sensor_work);
    k_msleep(5000);
}

If you’re already familiar with the system work queue you may notice that instead of k_work_submit() we’re calling k_work_submit_to_queue(). This allows us to specify which work queue the work should be sent to.

Note that we declared our work item globally. If it’s already in the queue and we try to submit it again, nothing will happen. If you need to submit the same work multiple times you need to initialize multiple instances, each with their own pointers. There are also specialized approaches for rescheduling and delaying work items. For a guided tour of those topics, check out Chris Gammell’s post on Send-when-idle using Zephyr and Golioth Pouch.

When we build and run the code we can see that it outputs temperature readings on a regular cadence.

[01:05:31.971,657] <inf> hello_zephyr: Temperature reading: 24.750000
[01:05:36.971,771] <inf> hello_zephyr: Temperature reading: 24.812500
[01:05:41.971,855] <inf> hello_zephyr: Temperature reading: 24.875000
[01:05:46.971,960] <inf> hello_zephyr: Temperature reading: 24.937500
[01:05:51.972,050] <inf> hello_zephyr: Temperature reading: 24.875000
[01:05:56.972,149] <inf> hello_zephyr: Temperature reading: 24.937500

What about memory usage? In the shell we can view the memory usage for our work queue by using the kernel thread stacks command. Keep reading to see the output of that command.

Adding a Stack Name for Better Logs and Debugging

0x10045870                                  (real size  512):   unused   96     usage  416 /  512 (81 %)
0x1005a0d8 coap_client                      (real size 6144):   unused 4312     usage 1832 / 6144 (29 %)
0x100468c0 net_socket_service               (real size 1200):   unused  728     usage  472 / 1200 (39 %)
0x10046b58 rx_q[0]                          (real size 1792):   unused 1040     usage  752 / 1792 (41 %)
0x10046a68 net_mgmt                         (real size  800):   unused  552     usage  248 /  800 (31 %)
0x10045e40 shell_uart                       (real size 2048):   unused  888     usage 1160 / 2048 (56 %)
0x100470c8 sysworkq                         (real size 1024):   unused  496     usage  528 / 1024 (51 %)
0x10046d48 ENET_RX                          (real size 1600):   unused 1200     usage  400 / 1600 (25 %)
0x10045948 logging                          (real size 1536):   unused  664     usage  872 / 1536 (56 %)
0x10046e80 idle                             (real size  320):   unused  256     usage   64 /  320 (20 %)
0x10046f38 main                             (real size 4096):   unused 3264     usage  832 / 4096 (20 %)
0x100539a8 IRQ 00                           (real size 2048):   unused 1832     usage  216 / 2048 (10 %)

There are a couple of things that I pay attention to based on the kernel thread stacks output. First, our thread doesn’t name its stack so we have to deduce that the top line without a name is the memory usage for our sensor work queue. The second is that 81% of that stack is used, which makes me feel just a bit claustrophobic.

In this example we only have one instance of work that is being submitted to the queue, so our stack usage is not going to grow beyond what is shown after the work has run at least once. If there were other types of work, or if you have work handlers that use a variable amount of stack space, it’s important to do some testing to ensure you have enough head room to avoid an overflow.

void sensor_init(void) {
    k_work_queue_init(&sensor_workq);

    struct k_work_queue_config sensor_workq_stack_config = {.name = "sensor_workq"};
    k_work_queue_start(&sensor_workq,
                       sensor_workq_stack,
                       K_THREAD_STACK_SIZEOF(sensor_workq_stack),
                       SENSOR_WORKQ_PRIO,
                       &sensor_workq_stack_config);
    k_work_init(&sensor_work, sensor_work_handler);
}

Naming the stack is as simple as supplying a k_work_queue_config struct with the name member set. Once recompiled you can view the stacks with our brand new name showing proudly.

0x10045870 sensor_workq                     (real size  512):   unused   96     usage  416 /  512 (81 %)
0x1005a0d8 coap_client                      (real size 6144):   unused 4312     usage 1832 / 6144 (29 %)
0x100468c0 net_socket_service               (real size 1200):   unused  728     usage  472 / 1200 (39 %)
0x10046b58 rx_q[0]                          (real size 1792):   unused 1040     usage  752 / 1792 (41 %)
0x10046a68 net_mgmt                         (real size  800):   unused  552     usage  248 /  800 (31 %)
0x10045e40 shell_uart                       (real size 2048):   unused  888     usage 1160 / 2048 (56 %)
0x100470c8 sysworkq                         (real size 1024):   unused  496     usage  528 / 1024 (51 %)
0x10046d48 ENET_RX                          (real size 1600):   unused 1200     usage  400 / 1600 (25 %)
0x10045948 logging                          (real size 1536):   unused  664     usage  872 / 1536 (56 %)
0x10046e80 idle                             (real size  320):   unused  256     usage   64 /  320 (20 %)
0x10046f38 main                             (real size 4096):   unused 3264     usage  832 / 4096 (20 %)
0x100539a8 IRQ 00                           (real size 2048):   unused 1832     usage  216 / 2048 (10 %)

Setting this name is not just a handy way to understand the shell output for stack sizes. This will also be used for thread-aware debugging and shown when there is a hard-fault.

[00:04:12.155,700] <err> os: ***** USAGE FAULT *****
[00:04:12.155,717] <err> os:   Stack overflow (context area not valid)
[00:04:12.155,744] <err> os: r0/a1:  0x18026768  r1/a2:  0x61000000  r2/a3:  0xaaaaaaaa
[00:04:12.155,750] <err> os: r3/a4:  0x00000002 r12/ip:  0xaaaaaaaa r14/lr:  0xaaaaaaaa
[00:04:12.155,754] <err> os:  xpsr:  0xaaaaaa00
[00:04:12.155,781] <err> os: Faulting instruction address (r15/pc): 0xaaaaaaaa
[00:04:12.155,812] <err> os: >>> ZEPHYR FATAL ERROR 2: Stack overflow on CPU 0
[00:04:12.155,840] <err> os: Current thread: 0x10045870 (sensor_workq)
[00:04:12.290,057] <err> os: Halting system

It’s very easy to see from the ouptut above that the sensor_workq overflowed. If we hadn’t added the name, unknown would have been shown at the end of the Current thread log output.

Go Beyond Logging

It’s easy to envision a system that:

1. Takes readings on a regular cadence
2. Takes additional readings based on stimulus like motion or a threshold-based trigger
3. Takes additional readings on demand (from a button or network-based request)

The work queue is essential for coordinating between different threads (or hardware interrupts) from which these triggers arise. However, simply logging the data is not very useful.

There are two very handy approaches to recording those readings that are a natural follow to today’s demonstration. The first is using Zephyr message queues to cache sensor readings and sending them to the cloud as batch data. The second is using Zephyr’s ZBUS system to publish each reading to other processes listening/observing in the system. Stay tuned for more feature highlights from Zephyr and Golioth applications that put them to work.