Since the earliest days of the platform, Golioth has supported ingesting device logs and querying them via the console and management API. Combined with the Golioth Firmware SDK’s native integrations with RTOS logging subsystems, this functionality has made it easy to get started with basic device observability. Today we are expanding this capability by introducing support for routing device logs to any destination using Golioth Pipelines, relaxing the requirements on log format and enabling the use of your favorite SaaS and open source logging platforms.

The Old Way

Traditionally, the Golioth Firmware SDK has delivered device logs emitted by customer firmware to the Golioth Logs service. Developers do not need to directly interact with the service API, but rather use C macros, such as GLTH_LOGI, to emit messages at their desired log level. When using a supported RTOS, such as Zephyr, developers can also leverage the SDK’s logging subsystem integration to deliver log messages from libraries and drivers to the Golioth platform.

Behind the scenes, the Golioth Firmware SDK structures and encodes the log messages in a format that is compatible with the Golioth Logs service. It then buffers and sends the logs to the /logs endpoint of the Golioth device API. After the logs are delivered to Golioth, customers can query messages at the device and project level, filtering by metadata and time buckets.

The New Way

Since introducing Golioth Pipelines in May 2024, we have seen a wide range of customer product use cases be unlocked by the flexibility of the Golioth platform. The ability to easily send data in any format to any destination means that customers can easily integrate with existing cloud systems, drive down bandwidth costs using compact data encoding schemes, and preserve battery life by batching data payloads and delivering as bulk uploads. While many customers enjoy using Golioth’s internal data destinations, such as LightDB Stream for timeseries data and LightDB State for device status, we have always focused on solving the difficult problem of getting data to the cloud, then letting you use the best tool for the job.

In order to bring this same level of freedom to device logs, the latest version of the Golioth Firmware SDK updates the log processing functionality to route messages to Golioth Pipelines. We have introduced a new Golioth Logs data destination, which handles incoming logs on Pipelines. To replicate the previous functionality, customers can create the following Pipeline in their Golioth project to convert the CBOR encoded log messages to JSON then deliver them to the internal logs service. These logs are then viewable on the console or via the management API.

filter:
  path: "/logs"
  content_type: application/cbor
steps:
  - name: step-0
    transformer:
      type: cbor-to-json
      version: v1
    destination:
      type: logs
      version: v1

The path used for logs delivery can be changed using CONFIG_GOLIOTH_LOG_PIPELINES_PATH and customers who wish to upgrade to v0.22.0 of the Golioth Firmware SDK while preserving the previous behavior can set CONFIG_GOLIOTH_LOG_LEGACY=y.

With device logs flowing through Pipelines, messages can be transformed and routed to any of the existing data destinations, such as InfluxDB, and we’ll be introducing new destinations for more external logging services, as well as standardized vendor-neutral protocols, such as OpenTelemetry’s OTLP, in the coming weeks.

Getting Started

To get started, add a Pipeline to your existing Golioth project, or create a new project on Golioth (a logs Pipeline will be automatically created for you), then check out any of the example applications in the Golioth Firmware SDK. If you are interested in support for a specific logging service, let us know by creating a post on the Golioth Forum.