A new JSON Patch Pipelines transfomer is now generally available for Golioth users. It allows for modifying JSON payloads, or payloads that have been transformed into JSON in a previous pipeline step, in order to fit a specific structure or schema. To see this transformer in action, check out the documentation example or last week’s Friday Afternoon Stream.

Shaping Data for Different Destinations

Golioth frequently sits at the intersection of firmware and cloud platforms. One of our goals when launching Pipelines was to enable those to worlds to seamlessly interact, avoiding the need for one side to compromise to suit the other. For example, Pipelines can allow for devices to send data in a compact binary format, such as CBOR, then have it translated to a text-based representation, such as JSON, for delivery to a cloud service.

The json-patch transformer enhances this capability by not only changing the format, but also the structure and content. Furthermore, it allows for the structure required by the end destination to change over time, without requiring firmware updates. In the following example, fields are re-arranged to meet the requirements of the custom webhook data destination. Critically, if this destination changed, or a new one was added in the future, the pipeline could be updated, and the device could continue sending the same payloads.

filter:
  path: "*"
  content_type: application/cbor
steps:
  - name: change-format
    transformer:
      type: cbor-to-json
      version: v1
  - name: transform-and-deliver
    transformer:
      type: json-patch
      version: v1
      parameters:
        patch: |
          [
            {"op": "add", "path": "/environment", "value": {}},
            {"op": "add", "path": "/location", "value": {}},
            {"op": "move", "from": "/temp", "path": "/environment/temp"},
            {"op": "move", "from": "/lat", "path": "/location/lat"},
            {"op": "move", "from": "/long", "path": "/location/long"}
          ]
    destination:
      type: webhook
      version: v1
      parameters:
        url: https://my-backend.example.com/data
        headers:
          x-api-key: $BACKEND_API_KEY

Click here to use this pipeline in your project.

Conditional Data Manipulation

In some cases it may be desirable to conditionally patch a JSON object payload based on the contents of the payload, or the metadata associated with the device that sent it. Combining the json-patch transformer with other transformers demonstrates the power of Pipelines. The test operation in a JSON Patch document conditionally applies a patch if the criteria is met.

For example, in the following pipeline, the key-value pair demo: true is injected into the payload if the device ID matches 649998262fecb43eb2d39859. The device ID is made available when applying the patch using the inject-metadata transformer. The metadata is subsequently stripped from the payload to ensure extraneous information is not delivered to the final destination.

filter:
  path: "*"
  content_type: application/json
steps:
  - name: get-metadata
    transformer:
      type: inject-metadata
      version: v1
  - name: conditional-patch
    transformer:
      type: json-patch
      version: v1
      parameters:
        patch: |
          [
            {"op": "test", "path": "/device_id", "value": "649998262fecb43eb2d39859"},
            {"op": "add", "path": "/demo", "value": true}
          ]
  - name: remove-metadata
    transformer:
      type: json-patch
      version: v1
      parameters:
        patch: |
          [
            {"op": "remove", "path": "/device_id"},
            {"op": "remove", "path": "/project_id"},
            {"op": "remove", "path": "/timestamp"}
          ]
  - name: send-lightdb
    destination:
      type: lightdb-stream
      version: v1

Click here to use this pipeline in your project.

What’s Next

The json-patch transformer is the first of many new transformers and data destinations we’ll be adding over the next few weeks. If you have a use-case in mind, feel free to reach out to us on the forum!

Prior to joining Golioth a little over a year ago, I remember being struck by the number of services and features offered by the platform. OTA updates, logging, time-series data storage, a real-time database, settings — the list goes on. Coming from a cloud services background, I felt certain that Golioth was unique amongst its peers in trying to offer every IoT service a product could possibly need, but as I continued to research competitors I was surprised to see that this was the norm, not the exception.

Four tightly overlapping circles.

It seemed that every platform was racing to cover every service that an internet connected device could ever need. This has resulted in a confusing market for consumers, full of solutions with heavy overlap in service offerings. “Why does every platform feel the need to solve every problem?” I wondered to myself. In hindsight, the answer is obvious: connectivity.

In the cloud, where we reside in the luxurious confines of our precision cooled data centers, we don’t spend too much time sweating one more TCP connection. We’ll happily add a managed service to our architecture to avoid having to build something from scratch or, even more importantly, keep it running. In the world of embedded devices, we are not so fortunate.

A single device may be expected to run on battery for years, and network access may be limited, especially for devices in transit. For example, we’ve gone to great lengths to dramatically reduce how often some devices must perform DTLS handshakes. When deploying thousands, or millions of devices, the margins matter. It makes sense why a connected device platform would try to give you the kitchen sink on your one precious network connection. Still, I am skeptical of the ability of IoT-focused platforms to offer every cloud service you could ever need, and I feel pretty comfortable saying that considering we are building one ourselves! In other words, I am concerned that companies and individuals building connected device products are being prevented from using the right tool for the job.

A Platform for Building Platforms

Last week, we announced a massive new feature of the platform: Golioth Pipelines. This isn’t just another new service. Rather, Pipelines makes it significantly easier to use services outside of Golioth. The easiest way to understand this change is by looking at the architecture before and after.

Diagram showing devices talking to Golioth services.

Previously, devices established a secure connection to Golioth, then talked to each of our many services directly. Every service includes opinions about how data should be formatted, how requests should be made, and how responses should be returned. If data needed to go somewhere other than Golioth, devices would either need to establish an additional connection (expensive) or use LightDB Stream’s Output Streams. The benefit of this structure, especially when combined with the Golioth Firmware SDK, was that developers incurred very little cognitive overhead when deciding how to format and deliver their data.

Diagram showing devices talking to Golioth services, and LightDB Stream talking to external services.

However, there are a few drawbacks with this approach. First, LightDB Stream only accepts semi-structured data in the form of JSON or CBOR. These data formats are flexible and easy to use when prototyping, but are less efficient than alternative options. This is especially true if binary data is being transmitted and devices are forced to represent it in a text-based format such as base64.

The second drawback is that LightDB Stream is persistent. Any stream data flowing through Golioth to an external destination was stored in LightDB Stream in addition to being passed along to its final location. This not only incurs costs for users, but also results in data being resident on both Golioth and the external system to which it is being forwarded. In environments with stringent compliance requirements, this can be a significant blocker.

Diagram showing devices talking to Golioth services, but LightDB Stream is moved to be aligned with the external services and replaced by Pipelines.

Pipelines moves LightDB Stream from the point of ingestion to the point of persistence, making it available as an optional data destination alongside those outside of the Golioth platform. There are still advantages to using LightDB Stream, especially when prototyping, due to its native integration into the platform. Data can be viewed in the Golioth console directly, and there is no need to create external accounts or credentials. We’ve created a new category for these type of offerings: application services. Users may leverage them to build out their product, but the Golioth platform enables substituting alternatives as well.

For users that previously enjoyed having data live in LightDB Stream and an external destination, they can continue leveraging that pattern by creating a pipeline that routes data to multiple locations, ensuring that data is delivered in the proper format to each by leveraging transformers.

 

filter:
  path: /temp
  content_type: application/cbor
steps:
  - name: step-0
    destination:
      type: gcp-pubsub
      version: v1
      parameters:
        topic: projects/my-project/topics/my-topic
        service_account: $GCP_SERVICE_ACCOUNT
  - name: step-1
    transformer:
      type: cbor-to-json
      version: v1
  - name: step-2
    transformer:
      type: inject-path
      version: v1
    destination:
      type: lightdb-stream
      version: v1
  - name: step-3
    destination:
      type: influxdb
      version: v1
      parameters:
        url: https://us-east-1-1.aws.cloud2.influxdata.com
        token: $INFLUXDB_TOKEN
        bucket: device_data
        measurement: sensor_data

But Pipelines is more than just addressing the issues of LightDB Stream. Pipelines represents a paradigm shift in the architecture of the Golioth platform, which will enable additional features in the coming months. Rather than being constrained by an opinionated Golioth API, users can define their own APIs on Golioth by creating multiple pipelines with different path and content_type filters. This technical change reflects our renewed focus on Golioth’s core value proposition: device connectivity.

Rather than being constrained by an opinionated Golioth API, users can define their own APIs on Golioth by creating multiple pipelines with different path and content_type filters. This technical change reflects our renewed focus on Golioth’s core value proposition: device connectivity.

Behind the Scenes

While Pipelines represents a large change in user-facing functionality, the smooth rollout was enabled by a silent launch months before. Internally, all device traffic on Golioth was directed through a new component in our cloud infrastructure, the Constrained Device Gateway (cdg), a protocol-agnostic proxy we developed that is responsible for managing connection state and brokering bidirectional communication between devices and services. Hints of this new component have been present since the introduction of the Golioth Simulator, which talks to device services via HTTP rather than CoAP.

Diagram showing devices talking to Golioth services via the Constrained Device Gateway (cdg).

With cdg in place, we were able to easily begin routing a subset of traffic to the infrastructure that powers Pipelines. This enabled the internal use of Pipelines in production weeks prior to general availability, and has allowed for continued use of Output Streams for projects that were previously leveraging the functionality. These projects now have the opportunity to be automatically migrated from Output Streams to Pipelines when they are ready.

It also means that the pace at which we are able to safely introduce new services and features is dramatically increased. Previously, authoring a new service required deep understanding of how to handle CoAP request / response patterns and observations. Furthermore, when a new protocol was introduced, every service would need to be updated with support. These communication patterns are now generalized. Services can easily leverage the internal cdg SDK to both respond to requests from devices and dispatch notifications to them, all using their preferred protocol.

While this functionality is only available for internal services today, it opens the door for bringing some of the user-defined capabilities of Pipelines, which are unidirectional, to bidirectional services. Doing so would further enable Golioth users to define their own platform on top of a single secure connection between devices and the cloud, eliminating the “all or nothing” nature of the decision whether to use an IoT platform or build one in-house.

The Road Ahead

Complexity comes with optionality, and as we continue expanding the realm of what is possible on Golioth, we will also be continuing to invest in observability. In the coming weeks we’ll be introducing new features that give additional visibility into what data is flowing through Pipelines and where it is being delivered. We’ll also be offering greater insight into how often frequently individual devices are connecting to Golioth, and how much much data they are sending.

Realizing the vision of making Golioth the universal connector for IoT means giving more choice to our users, and we are excited for how this new architecture enables that goal. If you’re building a connected device product, check out Golioth today!

Today, we are thrilled to announce the launch of Pipelines, a powerful new set of features that redefines how you manage and route your IoT data on Golioth. Pipelines is the successor to our now-deprecated Output Streams and represents a significant upgrade in our functionality, scalability and user control.

Two years ago, we introduced Output Streams to seamlessly connect IoT data to various cloud services like AWS SQS, Azure Event Hubs, and GCP PubSub. This enabled Golioth users to efficiently stream sensor data for real-time processing, analytics, and storage, integrating easily with their existing cloud infrastructure.

Since then, we’ve gathered extensive feedback, and now we’re excited to introduce a more versatile solution for data routing: Pipelines. Previously, all stream data had to flow into LightDB Stream and conform to its JSON formatting requirements, which was restrictive for those with regulatory and data residency needs. With Pipelines, you can direct your data to LightDB Stream, your own database, or any other destination, in any format you choose.

Pipelines also introduces filtering and transformation features that simplify or even eliminate backend services through low-code configuration. Configurations are stored and managed as simple YAML files, making them easily deployable across multiple projects without manual recreation. This approach allows you to version your data routing configurations alongside your application code.

Pipelines Example Screenshot

Internally, Pipelines architecture is designed to support our future growth, enabling us to scale our data routing capabilities to billions of devices. This robust foundation allows us to quickly iterate and add new features rapidly, ensuring that our users always have access to the most powerful and flexible data management tools available.

All Golioth users can start taking advantage of Pipelines today. Projects that were previously only using LightDB Stream and did not have any Output Streams configured have been automatically migrated to Pipelines. Users in those projects will see two pipelines present, which together replicate the previous behavior of streaming to LightDB Stream. These pipelines can be modified or deleted, and new pipelines can be added to support additional data routing use-cases.

Projects with Output Streams configured will continue using the legacy system, but can be seamlessly migrated to Pipelines with no interruptions to data streaming. To do so, users in those projects must opt-in to migration.

New projects created on Golioth will have a minimal default pipeline created that transforms CBOR data to JSON and delivers it to LightDB Stream. This pipeline is compatible with Golioth firmware examples and training, but may be modified or removed by a user if alternative routing behavior is desired.

Pipelines are especially advantageous for users with specific data compliance requirements and those transmitting sensitive information, such as medical device products. Removing the requirement of routing data through LightDB Stream, where it is persisted on the Golioth platform, provides two main benefits:

  1. Regulatory Compliance: Users can route data to their own compliant storage solutions, making it suitable for many sensitive applications that require data not be persisted on other third-party platforms.
  2. Cost Savings: For users who do not need data persistence, routing data directly to other destinations can avoid the costs associated with streaming data to LightDB Stream. This flexibility allows for more efficient and cost-effective data management.

Getting Started with Pipelines

Alongside the launch of Pipelines, we have also released a new version of the Golioth Firmware SDK, v0.13.0 , which introduces new functionality to support streaming arbitrary binary data to destinations that support it. Previously, only CBOR and JSON data could be streamed to Golioth, as everything flowed through LightDB Stream, which only accepts JSON data. Now, rather than streaming data to LightDB Stream, data is sent to the Golioth platform and routed to its ultimate destination via the pipelines configured in a project. Devices using previous versions of the Golioth Firmware SDK will continue working as expected.

Pipelines can be configured in the Golioth Console using YAML, which defines filters and steps within your pipeline. Here’s an example:

filter:
  path: "*"
  content_type: application/cbor
steps:
  - name: step-0
    destination:
      type: gcp-pubsub
      version: v1
      parameters:
        topic: projects/my-project/topics/my-topic
        service_account: $GCP_SERVICE_ACCOUNT
  - name: step-1
    transformer:
      type: cbor-to-json
      version: v1
  - name: step-2
    transformer:
      type: inject-path
      version: v1
    destination:
      type: lightdb-stream
      version: v1
  - name: step-3
    destination:
      type: influxdb
      version: v1
      parameters:
        url: https://us-east-1-1.aws.cloud2.influxdata.com
        token: $INFLUXDB_TOKEN
        bucket: device_data
        measurement: sensor_data

This pipeline accepts CBOR data, delivers it to GCP PubSub, before transforming it to JSON and delivering it to both LightDB Stream (with the path injected) and InfluxDB. This is accomplished via three core components of Pipelines.

Filters

Filters route all or a subset of data to a pipeline. Currently, data may be filtered based on path and content_type. If either is not supplied, data with any value for the attribute will be matched. In this example, CBOR data sent on any path will be matched to the pipeline.

filter:
  path: "*"
  content_type: application/cbor

Transformers

Transformers modify the structure of a data message payload as it passes through a pipeline. A single transformer may be specified per step, but multiple steps can be chained to perform a sequence of transformations. This transformer will convert data from CBOR to JSON, then pass it along to the next step.

- name: step-1
  transformer:
    type: cbor-to-json

Destinations

Destinations define where the transformed data should be sent. Each step in a pipeline can have its own destination, allowing for complex routing configurations. When a step includes a transformer and a destination, the transformed data is only delivered to the destination in that step. This destination sends JSON data to LightDB Stream after nesting the object using the message path. The next step receives the data as it was prior to the path injection.

- name: step-2
    transformer:
      type: inject-path
    destination:
      type: lightdb-stream
      version: v1

The full list of Output Stream destinations is now available as Pipelines destinations (with more to come):

  • Azure Event Hub
  • AWS SQS
  • Datacake
  • GCP PubSub
  • InfluxDB
  • MongoDB Time Series
  • Ubidots
  • Webhooks

For detailed documentation visit, visit our Pipelines Documentation.

Updated Pricing Model

We’re keeping the same usage-based pricing as Output Streams but also introducing volume discounts. We want to emphasize transparent pricing optimized for MCUs and are revising the pricing structure for Pipelines to accommodate a wider range of data usage patterns, ensuring affordability and predictability in billing for both low and high bandwidth use cases while ensuring customers with large fleets of devices can enjoy discounts that come with scale.

Data routed to External Pipelines Destination
Data Volume (per Month) Per MB Price ($)
0 – 1 GB $0.40
1 – 10 GB $0.34
10 – 50 GB $0.28
50 – 150 GB $0.22
150 – 300 GB $0.16
300 – 600 GB $0.10
600 – 1 TB $0.04
1 TB+ $0.01
Data routed to LightDB Stream
Data Volume (per Month) Per MB Price ($)
0-1 TB + $0.001

The first 3MB of usage for Pipelines is free, allowing users who are prototyping to do so without needing to provide a credit card. This includes usage routing data to LightDB Stream through Pipelines.

For full details, visit Golioth Pricing.


Pipelines marks a significant step forward in Golioth’s IoT data routing capability, offering a new level of flexibility and control. We’re excited to see how you’ll use Pipelines to enhance your IoT projects. For more details and to get started, visit our Pipelines Documentation.

With our new infrastructure, we can rapidly add new destinations and transformations, so please let us know any you might use. Additionally, we’d love to hear about any logic you’re currently performing in backend services that we can help you streamline or even delete. If you have any questions or need assistance, don’t hesitate to reach out. Contact us at [email protected] or post in our community forum. We’re here to help!

Migration from Output Streams to Pipelines

As previously mentioned, projects currently using Output Streams will continue to leverage the legacy infrastructure until users opt-in to migration to Pipelines. We encourage users to try out Pipelines in a new project and opt-in existing projects when ready. Output Streams does not currently have an end of life date but we will be announcing one soon.

Stay tuned for more updates and happy streaming!

Golioth works with Qualcomm

We’re excited to announce the latest update to the Golioth Firmware SDK, release 0.12.0, which now includes support for Zephyr’s newly introduced Modem Subsystem. This enhancement significantly increases the flexibility of our platform, enabling support for a broader array of cellular modem technologies, starting with Qualcomm. 0.12.0 adds support for the Quectel BG95, joining the Nordic Semiconductor nRF9160 (our go-to modem around here!) as a first class cellular modem. We also introduced additional ways to securely store credentials.

Zephyr’s New Modem Subsystem

Introduced in Zephyr 3.5.0, the Modem Subsystem is a unified interface for modem drivers. This addition simplifies the integration of cellular modems (and others) into Zephyr-based projects, greatly expanding the range of devices and technologies that developers can utilize effectively. For a detailed overview of the modem subsystem, check out this summary from Zephyr’s Developer Advocate, Benjamin Cabé.

Integration in Golioth Firmware SDK

With the integration of this modem subsystem in the Golioth Firmware SDK, Golioth users can now more flexibly incorporate a wider array of modem technologies into their IoT projects. There are a lot of great modems and module vendors in the market and providing choice is at the heart of what we do at Golioth.

First Supported Modem and Board

The first modem we are supporting with this updated SDK is the BG95 from Quectel, based on Qualcomm technology. The BG95 is paired with the nRF52840 on the RAK5010 development board from RAKwireless. This combination highlights the flexibility of Qualcomm’s technology integrated into Quectel’s hardware, offering developers robust tools for deploying cellular IoT solutions efficiently.

Why Qualcomm?

We chose to support Qualcomm modems because our community asked for it! Developers have different design needs and want maximum flexibility. They need more options that accommodate diverse business needs. Qualcomm chipsets offer the latest in connectivity protocols and radio technology at competitive prices. Qualcomm provides silicon and support for a wide ecosystem of module vendors, such as Quectel, U-Blox, Telit, and more. Golioth customers have used Qualcomm modems in their products in the past, but needed to do much of the integration engineering themselves. Zephyr’s Modem Subsystem makes it easier to develop applications that integrate Qualcomm modems. Connecting this wider range of modems to Golioth is more hands-off for the user, reducing complexity. Developers can focus more on innovation and less on technical hurdles.

Also in This Release

In addition to new modem support, this release introduces a another feature: DTLS socket offloading for Zephyr. This includes an example for the long-supported Nordic Semiconductor nRF9160.

DTLS socket offloading leverages a modem’s secure credential store, which allows for the use of secure, offloaded sockets. This means there is not any private key material in RAM. This can be a significant advantage as it helps reduce RAM usage, code size, CPU load, and power consumption. Actual benefits will vary depending on the application and how the code is utilized.

This new feature enhances device security and efficiency, contributing further to the versatility and robustness of the Golioth Firmware SDK. Mike previously wrote how to store credentials on the nRF9160 using TLS tags.

Getting Started

To get started with the latest SDK:

  1. Update to the newest release, 0.12.0, from the Golioth Firmware SDK repository.
  2. Explore the documentation and examples provided for integrating the RAK5010 board or try DTLS socket offloading with the nRF9160.
  3. Visit our community forums or support channels if you need help or want to discuss your projects.

Focused on Your Success

At Golioth, we’re committed to providing you with the tools and flexibility needed to succeed in the fast-evolving world of IoT. By adding support for new modems and enhancing the ways you can manage credentials, we aim to streamline your development process and empower your innovative projects. Whether you’re integrating the latest modem technology or implementing secure credential management, Golioth is here to support every step of your journey towards building smarter, more connected solutions.

If you ask any seasoned hardware engineer, they will tell you there are only two types of people:

  1. Those who have accidentally swapped TX and RX
  2. And those who will
TX RX Fail

There’s even a fail badge of honor!

Despite our best efforts, mistakes can creep into hardware designs. Design specifications can change over time. Taking a hardware design from concept to production is a journey that nearly always involves iterating through multiple revisions of the PCB assembly.

In this post, we’ll walk through some of the tools that Zephyr & Golioth provide for managing multiple board revisions.

Let’s dive in and look at how you can support Rev ARev B, all the way to Rev X in your Zephyr firmware without losing your sanity!

Aludel Elixir Board

Here at Golioth, Chris Gammell has been designing a new rapid prototyping board called the “Aludel Elixir“. We use this board internally for developing and testing our growing collection of reference designs, and we’re using it for live demos at Embedded World 2024.

Aludel Elixir Rev B

The image above shows the 2nd hardware revision of the board (Rev B), which fixes some of the hardware issues we found when testing the 1st revision (Rev A).

Supporting the Rev B hardware requires changes to the Zephyr firmware that runs on the internal MCU in the nRF9160 SIP. However, since we have Rev A and Rev B hardware “in the wild”, we want to support building firmware for all current and future board revisions in Golioth projects—like our Reference Design Template.

Multiple board revisions in Zephyr

Fortunately, Zephyr has support for multiple board revisions as a standard part of the build system.

Note: Shortly after Zephyr 3.6.0 was released, a new hardware model was introduced to Zephyr. This new model overhauls the way both SoCs and boards are named and defined and is not backwards compatible. This blog post assumes the old hardware model used in Zephyr 3.6.0 or earlier.

Building for multiple board revisions in Zephyr

Before jumping into the implementation details, it’s helpful to see how an end-user would build for a specific board revision.

We can build a Golioth Zephyr app for a specific revision of the Aludel Elixir board by simply appending a @<revision> specifier to the board name:

# Build firmware for Rev A
west build -b aludel_elixir_ns@A

# Build firmware for Rev B
west build -b aludel_elixir_ns@B

# Build firmware for the "default" revision (which is currently Rev B)
west build -b aludel_elixir_ns

Adding multiple board revisions in Zephyr

The Zephyr Board Porting Guide has a detailed section on how to add support for multiple board revisions.

When we build for a board with a revision specifier—e.g. aludel_elixir_ns@B—the build system looks for a revision.cmake file in the board directory:

boards/arm/aludel_elixir
├── ...
└── revision.cmake

Here’s the revision.cmake file for the aludel_elixir board:

board_check_revision(
  FORMAT LETTER
  EXACT
  DEFAULT_REVISION B
  VALID_REVISIONS A B
)
  • FORMAT LETTER tells the build system that the revision format is “Letter revision matching” (A, B, C, etc)
  • EXACT requires that the revision is an exact match
  • DEFAULT_REVISION sets the revision to be used when no revision is specified (e.g. west build -b aludel_elixir_ns)
  • VALID_REVISIONS defines the set of valid revisions that can be specified

Kconfig settings for specific revisions

It’s possible to specify Kconfig symbols that are specific to a particular board revision by adding optional <board>_<revision>.conf files in the board directory. These will be merged into the board’s default Kconfig configuration.

For example, the Elixir Rev A board was accidentally built with the NB-IoT only variant of the nRF9160, which requires some Kconfig settings that only apply to the Rev A board revision.

boards/arm/aludel_elixir 
├── ...
├── aludel_elixir_A.conf
└── aludel_elixir_ns_A.conf

Devicetree overlays for specific revisions

It’s also possible to describe hardware changes in devicetree that are specific to a particular board revision by adding optional <board>_<revision>.overlay files in the board directory. These will be added to the common <board>.dts devicetree file.

For example, the Elixir Rev A board connects the spi2 peripheral to the mikroBUS socket headers, while the Rev B board connects the spi3 peripheral instead. We added devicetree overlay files for each board revision that specify the correct SPI peripheral to use:

boards/arm/aludel_elixir
├── ...
├── aludel_elixir_A.overlay
├── aludel_elixir_ns_A.overlay
├── aludel_elixir_B.overlay
└── aludel_elixir_ns_B.overlay

Distributing board definitions as a Zephyr Module

The golioth-zephyr-boards repo stores the Zephyr board definitions for the Aludel Elixir board revisions, allowing us to use them across multiple Zephyr projects as a Zephyr Module.

For example, here’s how it’s included in our Reference Design Template app via the west.yml manifest:

- name: golioth-zephyr-boards
  path: deps/modules/lib/golioth-boards
  revision: v1.1.1
  url: https://github.com/golioth/golioth-zephyr-boards

Note that it’s also possible to add application-specific Kconfig and devicetree overlay files for each board revision:

<app>/boards/
├── ...
├── aludel_elixir_ns_A.conf
├── aludel_elixir_ns_A.overlay
├── aludel_elixir_ns_B.conf
└── aludel_elixir_ns_B.overlay

If you leave off the @<revision> specifier, these will be applied to all revisions of the board:

<app>/boards/
├── ...
├── aludel_elixir_ns.conf
└── aludel_elixir_ns.overlay

Golioth Blueprints

In a real-world IoT deployment, it’s likely that a fleet of devices will have multiple hardware revisions deployed simultaneously. Golioth provides support for managing different hardware revisions through a concept called “Blueprints“.

Blueprints are a flexible way to segment devices based on variations in hardware characteristics, such as the board revision. For example, when creating a new over-the-air (OTA) firmware release in the Golioth console, a blueprint can be specified to limit the scope of the release to only Rev A hardware devices.

If you’d like a step-by-step introduction to deploying a fleet of IoT devices with Zephyr and Golioth, we’d love to have you join us for a free Zephyr developer training. Our next session is just two weeks away. Sign up now!

New Pricing with Golioth. Device Management should be free.

Today, we’re excited to roll out a major update to Golioth’s pricing. We are transitioning from per-device per-month fees to a usage-based model. This change allows individual developers to use Golioth’s device management features for free, with no limit on the number of devices connected, and significantly broadens accessibility for teams using our platform across a diverse array of IoT use cases. By doing so, we reinforce our belief that critical device management features, like Over-the-Air (OTA) updates, are a basic right for IoT developers and a necessity for creating secure products.

Introducing Our New Pricing Structure

Golioth’s new pricing model is designed to meet the varied requirements of our users, from individual developers to large enterprises:

  • Individual Developer Plan: Free
  • Teams Plan: $299/mo (user management and support)
  • Enterprise Plan: starting at $2,799/mo (features like SSO and private deployments)

This new model provides 1GB of OTA downloads and 200MB of log messages per month for free, sufficient for managing a moderately sized fleet in production, with no fees for device connections across all plans. For those who need more, additional usage is $0.35/MB for OTA downloads and $0.20/MB for logging. We continue to offer data ingestion and routing, with data streamed through Golioth at $1/GB to LightDB Stream and $0.40/MB for data streamed out to third-party services. Our new pricing model is designed to support developers at every stage of their journey, ensuring that Golioth remains the most accessible and developer-friendly IoT platform on the market.

Aligning Pricing with Device Behavior

Our updated pricing model is designed to accommodate the diverse behavior profiles of IoT devices. This is particularly advantageous for “sleepy devices” — those that wake up periodically to send sensor readings before going back to sleep. These types of devices, such as soil sensors, often operate with minimal data transmission, making them highly cost-sensitive under per-device fee structures. By eliminating these fees and focusing on actual usage, Golioth enables projects of every scale and complexity to leverage our advanced features without facing prohibitive costs.

Moreover, for projects with high bandwidth requirements, we offer volume discounts, ensuring that even large-scale deployments can be managed cost-effectively. This approach not only supports a broader range of IoT applications but also ensures that pricing scales sensibly with your project’s needs.

Why This Change Matters

Making device management free for individual developers emphasizes our commitment to the broader IoT community. OTA updates are crucial for the security, efficiency, and longevity of IoT products. We believe that every developer should have the capability to manage and update their devices remotely, without financial barriers.

This pricing update forces Golioth and our competitors to focus on innovation and creating unique value. We are moving away from the per-device fee model, recognizing it as outdated for supporting the wide range of IoT device use cases and behavior profiles. Instead, our focus is on creating a flexible developer experience, ensuring Golioth is compatible across numerous MCU platforms and can connect to any cloud destination, serving as the universal connector for IoT. This approach advances the IoT developer community and sets a new standard for what enterprises should expect from platform providers.

Looking Forward

We’re excited to see how these changes will enable developers to bring their visions to life with greater ease and less overhead. Our team is committed to continuously enhancing our platform and pricing to meet the evolving needs of the IoT industry.

For a detailed overview of our new pricing and to find the plan that’s right for you, visit our pricing page at Golioth Pricing.

We believe these changes will not only benefit our users but, also encourage a more innovative and secure IoT ecosystem. Thank you for your continued support, and we look forward to seeing what you create with Golioth.

As always, never hesitate to reach out if you have any questions, comments or feedback: [email protected]

View the official press release here. 

Black text on yellow background that reads "Understanding Your Golioth Usage".

Today we are launching a new feature in the Golioth console: usage visualization. All Golioth users can now access real-time usage data, broken down by project and service, on the settings page for organizations in which they are an admin.

Screenshot of Golioth console with usage metrics shown for an organization with three projects.

This feature follows a series of console updates, including the launch of the Golioth Simulator, and a restructuring of project and organization navigation. With each new release, Golioth is becoming more conducive to managing large fleets with greater visibility, while also enabling more seamless collaboration between users in an organization.

Usage data is currently displayed for the following services:

By default, the current month’s usage is displayed, but the selector in the top right can be used to change the range. Additional usage metrics, as well as more advanced filtering and visualization functionality will be introduced over the next few weeks.

Sign in to the Golioth console to view your usage data, and reach out on the forum with any feature requests or feedback!

Back in September, we released beta support for Zephyr RTOS in the Golioth Firmware SDK, as an additional option alongside our existing standalone Zephyr SDK. Today, we’re excited to remove the beta tag and announce full support for Zephyr in our cross-platform Firmware SDK. 

 

Over the last several months we’ve been improving the efficiency of the Firmware SDK and it is now on par with the standalone Zephyr SDK. We’ve expanded our automated test coverage, taken a fine-tooth comb through our APIs and documentation, and listened to and incorporated your feedback (thanks!). All of this means that the version of the SDK we’re releasing today is our best yet – and all of our customers will get these improvements at the same time, whether using Zephyr, ESP-IDF, or any other platform.

 

That’s why we chose to incorporate Zephyr support into our Firmware SDK and deprecate our standalone Zephyr SDK. We’re big fans of Zephyr, but our goal is to support the IoT community at large as a universal IoT connector, and that includes folks who choose to use other platforms. In order to provide the highest quality support to all of our users while moving as fast as possible to deliver new features and value, it made the most sense to invest in a single unified Firmware SDK.

 

Of course, our commitment to Zephyr itself hasn’t changed. We continue to be financial sponsors of the project, sit on both the Technical and Marketing Steering Committees, and provide free Zephyr training. And you’ll be able to find us at the Zephyr Developer Summit in Seattle in April.

Ready to get started?

You’ll find the same features and quality you’ve come to expect from Golioth in our Firmware SDK, but we’ve taken this opportunity to rework some of our APIs. Migrating from the Zephyr SDK is easy, and we’ve prepared a migration guide to help you along the way. For those beginning a new project, our getting started documentation will get you up and running in no time.

What does this mean for the Zephyr SDK?

As of today, our standalone Zephyr SDK is deprecated and not recommended for new designs. We may make periodic bug fixes and security updates to the SDK, but we won’t be adding new features or actively developing the standalone Zephyr SDK. This will result in accelerated delivery of features and support in our cross-platform Firmware SDK, as our team is now singularly focused.

 

We will reach formal end-of-life for the standalone Zephyr SDK on July 31, 2024, at which point we will no longer make any updates to the SDK. If you have any questions, please reach out to your Golioth account rep or contact us at [email protected].



At Golioth we are on a quest to improve the security, reliability, and efficiency of IoT products. Today we are launching the Golioth Simulator as part of an increasing focus on addressing the organizational challenges that come with building those products. The Simulator enables cloud engineering teams to generate data that mimics what devices deployed in the field will eventually send to Golioth, prior to the devices being ready. This eliminates bottlenecks caused by long hardware and firmware development cycles, and allows for the services that interact with device data to be developed in parallel.

Screenshot of Golioth Simulator with many instances of Golioth logo in background.

The Development Cycle Mismatch

Building connected products is challenging for a number of reasons, but one of the most prominent is the range of engineering work that is required. This is illustrated by comparing to the development of products that are either only software, or non-networked hardware. In the former, engineering work ranges from distributed systems and backend development to frontend and graphic design. In the latter, the range may be from electrical engineering and PCB design to firmware development.

IoT products essentially take those two spectra and join them.

Three boxes, one describing hardware products as a spectrum between hardware and firmware, one describing software products as a spectrum between backend and frontend, and one combining them to describe IoT products.

While this clearly increases the breadth of skills that are necessary for a team to possess, it also introduces a more subtle challenge: wildly different iteration cycles and development timelines. In the worst case scenario, the longest development cycle, which is typically hardware, will dictate the velocity of an entire project. A common fallacy is to compare the relative effort of software development to hardware development and conclude that the former can be delayed until the latter is in its final stages. In reality, any aspect of the engineering spectrum can derail a product.

However, even if a team properly prioritizes all aspects of developing the product, it is easy to fall into the trap of creating both implicit and explicit dependencies between components of the system. Mitigating these organizational hazards can be a force multiplier.

Innovation at the Interface

We frequently describe Golioth as the interface between firmware and cloud engineers. Sitting between these two groups comes with a responsibility to present a familiar set of tools to both parties, but also an opportunity to introduce new functionality that closes the gap between them. While the Golioth platform presents separate APIs for devices and servers, each designed to be familiar to the respective consumer, the commonality between them is data.

Ultimately, connected products are about producing data in one location, and consuming it, or some transformation of it in another location. Typically, the majority of data for our customers today is being produced by devices, and consumed by servers. All of this data flows through Golioth, meaning that customers have a consistent interface for both sending and receiving data. But what about the shape of the data itself?

Abstract diagram of a device and a server being connected by a common understanding of the shape of data.

In the end, it will depend on the behavior of devices, but software iteration that depends on managing physical devices, or even firmware running in an emulated environment, to produce data is sacrificing efficiency. In the ideal case, firmware and cloud teams use the shape of data as a contract between systems, and software engineers are able to easily produce data that adheres to that contract. If that contract changes in the future, the tooling should make testing and updating the software that interacts with it seamless. The Golioth Simulator is a first step in that direction.

How it Works & Getting Started

As of today, the Simulator is available in the Golioth console for all users. Navigating to any device in your fleet should result in the Simulator launcher appearing in the bottom right corner of your screen. Clicking on it will open a floating window.

Navigate to the Credentials tab of the chosen device, and copy your preferred PSK ID and PSK into the Simulator. Use the default JSON payload or enter your own, then click Send. If you navigate to the LightDB Stream tab, you should see a new message with your payload.

To view new messages as you send them, be sure to enable Auto-Refresh. Because the Simulator is actually interacting with Golioth device APIs, messages will also flow through to any Output Streams you may have configured in your project.

Next Steps & Feedback

The initial functionality of the Golioth Simulator is intentionally basic as we are interested in seeing how users leverage the new interface to increase their pace of development. If you have feedback about how you would like to see the Simulator evolve to support more use cases, please reach out to us on the forum.

Around these parts, we refer to Golioth as “the Universal Connector”. Our firmware SDK works seamlessly with your firmware project, our cloud APIs are directly mapped to device APIs within the SDK, and that singular connection from device to cloud allows your data to flow unhampered to the rest of the internet. As a result, we are always looking for the most useful services to route your device data to, while also offering you device management and Over The Air updates.

Memfault is a device diagnostics and telemetry platform that we have been friends with for a long time. If you want deep insight into what’s happening with your device in the field, Memfault is a great way to visualize what’s going right and wrong. We know them from their integrations into the Zephyr ecosystem, as well as their great writing on their Interrupt blog. Last month, we co-announced our partnership with them, working with NXP to integrate using the Zephyr Real-Time Operating System.

Golioth enables connections over CoAP to field devices and routes the data out to services like Memfault

Today, Memfault published a joint piece of content with our Cloud Lead, Dan Mangum, that explains more about how data flows through Golioth to the Memfault platform.

See it in action

If you’re anything like me, a live demo really solidifies how a thing works. NXP’s RT10xx family provides a great platform for showcasing Zephyr applications that can route data up to the cloud. We have been partners with NXP since last year and continue to find their software capabilities and seriously powerful processors a great addition to our Continuously Verified Boards, as well as excellent targets for all kinds of applications.

If you like demos, you’re in luck! We’re doing a joint webinar with Memfault and NXP on November 16th at 9 am PST / 12 pm EST / 6 pm CET. Register now to watch it live, or sign up and get access to the on-demand recording if you can’t make it.

Join our live webinar with NXP and Memfault on Thursday Nov 16th

Use this new integration on your project

The capabilities shown in the blog post linked above and during the webinar this Thursday include some functionality that is still in early preview. If you’re interested in having access to Golioth data routing that instantly enables the Memfault, sign up on our page here. You’ll be among the first to know when this capability is available for your projects. In the meantime, you can attend the webinar, try out other projects on Golioth, and discuss your latest IoT experiments on our forum.