Prior to joining Golioth a little over a year ago, I remember being struck by the number of services and features offered by the platform. OTA updates, logging, time-series data storage, a real-time database, settings — the list goes on. Coming from a cloud services background, I felt certain that Golioth was unique amongst its peers in trying to offer every IoT service a product could possibly need, but as I continued to research competitors I was surprised to see that this was the norm, not the exception.

It seemed that every platform was racing to cover every service that an internet connected device could ever need. This has resulted in a confusing market for consumers, full of solutions with heavy overlap in service offerings. “Why does every platform feel the need to solve every problem?” I wondered to myself. In hindsight, the answer is obvious: connectivity.

In the cloud, where we reside in the luxurious confines of our precision cooled data centers, we don’t spend too much time sweating one more TCP connection. We’ll happily add a managed service to our architecture to avoid having to build something from scratch or, even more importantly, keep it running. In the world of embedded devices, we are not so fortunate.

A single device may be expected to run on battery for years, and network access may be limited, especially for devices in transit. For example, we’ve gone to great lengths to dramatically reduce how often some devices must perform DTLS handshakes. When deploying thousands, or millions of devices, the margins matter. It makes sense why a connected device platform would try to give you the kitchen sink on your one precious network connection. Still, I am skeptical of the ability of IoT-focused platforms to offer every cloud service you could ever need, and I feel pretty comfortable saying that considering we are building one ourselves! In other words, I am concerned that companies and individuals building connected device products are being prevented from using the right tool for the job.

A Platform for Building Platforms

Last week, we announced a massive new feature of the platform: Golioth Pipelines. This isn’t just another new service. Rather, Pipelines makes it significantly easier to use services outside of Golioth. The easiest way to understand this change is by looking at the architecture before and after.

Previously, devices established a secure connection to Golioth, then talked to each of our many services directly. Every service includes opinions about how data should be formatted, how requests should be made, and how responses should be returned. If data needed to go somewhere other than Golioth, devices would either need to establish an additional connection (expensive) or use LightDB Stream’s Output Streams. The benefit of this structure, especially when combined with the Golioth Firmware SDK, was that developers incurred very little cognitive overhead when deciding how to format and deliver their data.

However, there are a few drawbacks with this approach. First, LightDB Stream only accepts semi-structured data in the form of JSON or CBOR. These data formats are flexible and easy to use when prototyping, but are less efficient than alternative options. This is especially true if binary data is being transmitted and devices are forced to represent it in a text-based format such as base64.

The second drawback is that LightDB Stream is persistent. Any stream data flowing through Golioth to an external destination was stored in LightDB Stream in addition to being passed along to its final location. This not only incurs costs for users, but also results in data being resident on both Golioth and the external system to which it is being forwarded. In environments with stringent compliance requirements, this can be a significant blocker.

Pipelines moves LightDB Stream from the point of ingestion to the point of persistence, making it available as an optional data destination alongside those outside of the Golioth platform. There are still advantages to using LightDB Stream, especially when prototyping, due to its native integration into the platform. Data can be viewed in the Golioth console directly, and there is no need to create external accounts or credentials. We’ve created a new category for these type of offerings: application services. Users may leverage them to build out their product, but the Golioth platform enables substituting alternatives as well.

For users that previously enjoyed having data live in LightDB Stream and an external destination, they can continue leveraging that pattern by creating a pipeline that routes data to multiple locations, ensuring that data is delivered in the proper format to each by leveraging transformers.

 

filter:
  path: /temp
  content_type: application/cbor
steps:
  - name: step-0
    destination:
      type: gcp-pubsub
      version: v1
      parameters:
        topic: projects/my-project/topics/my-topic
        service_account: $GCP_SERVICE_ACCOUNT
  - name: step-1
    transformer:
      type: cbor-to-json
      version: v1
  - name: step-2
    transformer:
      type: inject-path
      version: v1
    destination:
      type: lightdb-stream
      version: v1
  - name: step-3
    destination:
      type: influxdb
      version: v1
      parameters:
        url: https://us-east-1-1.aws.cloud2.influxdata.com
        token: $INFLUXDB_TOKEN
        bucket: device_data
        measurement: sensor_data

But Pipelines is more than just addressing the issues of LightDB Stream. Pipelines represents a paradigm shift in the architecture of the Golioth platform, which will enable additional features in the coming months. Rather than being constrained by an opinionated Golioth API, users can define their own APIs on Golioth by creating multiple pipelines with different path and content_type filters. This technical change reflects our renewed focus on Golioth’s core value proposition: device connectivity.

Rather than being constrained by an opinionated Golioth API, users can define their own APIs on Golioth by creating multiple pipelines with different path and content_type filters. This technical change reflects our renewed focus on Golioth’s core value proposition: device connectivity.

Behind the Scenes

While Pipelines represents a large change in user-facing functionality, the smooth rollout was enabled by a silent launch months before. Internally, all device traffic on Golioth was directed through a new component in our cloud infrastructure, the Constrained Device Gateway (cdg), a protocol-agnostic proxy we developed that is responsible for managing connection state and brokering bidirectional communication between devices and services. Hints of this new component have been present since the introduction of the Golioth Simulator, which talks to device services via HTTP rather than CoAP.

With cdg in place, we were able to easily begin routing a subset of traffic to the infrastructure that powers Pipelines. This enabled the internal use of Pipelines in production weeks prior to general availability, and has allowed for continued use of Output Streams for projects that were previously leveraging the functionality. These projects now have the opportunity to be automatically migrated from Output Streams to Pipelines when they are ready.

It also means that the pace at which we are able to safely introduce new services and features is dramatically increased. Previously, authoring a new service required deep understanding of how to handle CoAP request / response patterns and observations. Furthermore, when a new protocol was introduced, every service would need to be updated with support. These communication patterns are now generalized. Services can easily leverage the internal cdg SDK to both respond to requests from devices and dispatch notifications to them, all using their preferred protocol.

While this functionality is only available for internal services today, it opens the door for bringing some of the user-defined capabilities of Pipelines, which are unidirectional, to bidirectional services. Doing so would further enable Golioth users to define their own platform on top of a single secure connection between devices and the cloud, eliminating the “all or nothing” nature of the decision whether to use an IoT platform or build one in-house.

The Road Ahead

Complexity comes with optionality, and as we continue expanding the realm of what is possible on Golioth, we will also be continuing to invest in observability. In the coming weeks we’ll be introducing new features that give additional visibility into what data is flowing through Pipelines and where it is being delivered. We’ll also be offering greater insight into how often frequently individual devices are connecting to Golioth, and how much much data they are sending.

Realizing the vision of making Golioth the universal connector for IoT means giving more choice to our users, and we are excited for how this new architecture enables that goal. If you’re building a connected device product, check out Golioth today!

I love old* industrial standards like Modbus. It was created in 1979 and widely adopted by machine makers since then. It has survived and continues to be used because old industrial machinery just doesn’t seem to die. If the equipment doesn’t die and the equipment runs an industrial standard, by the transitive property, that standard isn’t going anywhere. In 2024 (a short 45 years after the inception of Modbus), there continues to be equipment out in the field that uses the standard. Some of that equipment is brand new, too!

Technically, Modbus is a communications protocol (OSI layer 7). But by default it doesn’t connect to the broader internet, it connects to other machines. For “Modbus RTU”, this involves hooking machines together using RS-485, another standard, but one for the data link layer (OSI layer 2).  It is hardened against interference for noisy industrial environments. As a result, it’s reliable. My experience in the industrial sector has taught me that more than anything else, reliability is key. It’s the same reason that 4-20 mA sensors are used, and the bulkiness of the electronics is normally not an issue. If a circuit requires a massive protection diode? Well, make the enclosure bigger! It needs to be reliable.

Normally you’d have a Modbus sensor talking back to an industrial controller like a PLC. This was the localized control that allows fast action on the factory floor (“the temperature sensor is too high, turn down the heating element”, etc). As the times modernized, so did the connectivity options on the PLCs. But in many cases, it’s overkill. If you just want to push the data from a sensor to the internet, it’s a lot of extra equipment. Not to mention, that bulky-yet-reliable equipment? It often comes with a hefty price tag.

A few weeks ago, we released a reference design that skips a lot of these steps. In fact, all that’s required is a piece of hardware that has an RS-485 interface and an internet connection. In our case, it’s the nRF9160 plus some breakout boards, talking back to the Golioth Cloud. We showcased the Modbus Vibration Monitor Reference Design using a motor monitoring sensor, but now we’re able to swap out that sensor for any other Modbus (RTU) sensor. That’s the power of an industrial standard. Now that we have a connected, embedded device talking Modbus, the rest is just firmware updates. Oh yeah, Golioth does that part too!

Modbus is far from the only standard out there. It benefits from wide ranging, long time adoption due to the fact that it’s openly published, royalty free, and easy to implement. But there are other sensors and end devices running things like EtherCAT, CAN, PROFIBUS, HART, BACnet, Foundation Fieldbus, DeviceNet, and more. Once you start to think of these protocols and standards as a common language among end devices, the power of a common platform like Golioth hardware, firmware, and Cloud software really starts to emerge. Let us know if you see a need for any of these other standards in your projects!

* somewhere an engineer is saying, “you think that’s old, check out this other industrial standard!”

A proper Ode

Oh, Modbus, protocol of precision, thou art,
In the realm of data, thou playest a crucial part.
With thy structure elegant, and format clear,
Thou bringest connectivity far and near.

From the industrial floors to automation’s lair,
Thy presence is felt, ubiquitous and fair.
In the language of ones and zeros, thou speak,
Binding machines together, strong and sleek.

With thy bit twiddling dance, thou dost choreograph,
In the symphony of automation, thou art the staff.
Thy messages traverse through wires unseen,
Connecting devices, a diligent machine.

In the world of SCADA, thou never are flawed,
Transmitting signals at 9600 baud.
Reliable, robust, thy virtues abound,
In the realm of control, thy praises resound.

Oh, Modbus, thou art a beacon bright,
Guiding engineers through the darkest night.
With thy registers and coils, thou dost empower,
Enabling systems to function by the hour.

Though newer protocols may rise and fall,
Thy legacy, Modbus, shall stand tall.
For in the annals of technology’s lore,
Thy name shall echo forevermore.

So here’s to thee, Modbus, noble and true,
In the world of automation, we honor you.
For without thy steadfast hand to guide,
The wheels of progress would surely slide.