How Golioth uses CloudEvents

Golioth helps you get your data to and from your constrained IoT device in the field. We make it really easy to ship data up to the Cloud where you can do more interesting things than merely interacting with the device out the field.

But what happens when the data gets to the Cloud?

Once your data is in an environment with many more resources, you probably want it to pass between different services. For the average Device creator, we make this easy with our Output Streams feature. Output Streams immediately export data to 3rd party Cloud providers like AWS, Azure, and GCP. When we built that service (and our other “Cloud-to-Cloud” integrations), we wanted to build on top of an open standard that would reduce the friction of getting your data from the Golioth cloud to your own systems. We decided to reuse a specification we adopted on our internal infrastructure called CloudEvents.

What are CloudEvents?

First off, we need to ask “what are events”? It sounds like a generic term but for cloud developers it means something quite specific. “Event Data” vs “Message Data” is a pattern in Cloud-to-Cloud communication.

CloudEvents are a standardized way of formatting event data so that it’s easier for one Cloud provider to understand incoming data from another Cloud provider. The standard includes things like the Type of event, the Source of the event, and the formatting of the event. If you’re trying to figure out where an event came from (ie: change the value of LED from ‘true’ to ‘false’), you’d need to know if that update request came from an app, the Golioth Console, a 3rd party source via the REST API, or some other previously validated entity talking to our Cloud.

The History of CloudEvents

People have been doing event-driven design (architecture) for a long time. But there was no standard, which resulted in a lot of reinventing of “templates” for events. When you encounter an HTTP event of JSON, the payload is not the important part for processing that event. Without a format for producers and consumers to agree on, it means wasted time and lack of interoperability. Cloud engineers and members of the Cloud Native Computing Foundation in the Linux Foundation started CloudEvents to develop a standard.

An Event Driven Architecture

Cloud software has gotten extremely complex. Services stream all sorts of a data, often times between internal and external and external teams. Event-driven designs have been increasingly popular among modern cloud solutions to manage this growing complexity. Think of a large system with lots of micro-services and serverless functions talking to each other. Getting agreement up front on the message design would slow down innovation and be rather difficult. Events allow for the the “componentization” of the these API contracts. It decouples them. And as long as the agreed upon format is written down somewhere, both the servers (producers) and clients (consumers) can move faster and the process of designing very complex systems becomes easier.

Golioth is a modern cloud native software too, and our architecture internally has an event-driven design. We use CloudEvents internally. For example, when a device sends a LightDB Stream value over CoAP, it is processed by our CoAP API Gateway. The gateway then converts that message into a CloudEvent and publishes to a message broker. The broker and that entire part of the cloud ensures that CloudEvent data makes it to our time-series database securely and reliably. That piece in the middle is performant, scalable, and secure thanks to our design that leans on modern event broker software.

If we didn’t use the event pattern, we’d have to develop a bespoke system to handle CoAP messages in the middle of our stack, which would likely be far more difficult to build and more fragile to maintain.

Why should you care?

If you’re a device maker, do you need to care about CloudEvents? Generally, no. Golioth makes it really easy to connect to the Cloud and handles everything you’ll need to do in order to interact with the rest of the Internet.

But you’ll benefit from our use of CloudEvents because of how easy it is to connect to Golioth in a standard way. When you are working with your Cloud team (or a customer’s Cloud team, if you do design for hire), it will be really easy to interact with their Cloud. Even bespoke Cloud infrastructure will be able to easily connect to the Golioth Cloud. Thanks to the standardization, there are a range of libraries to make it easier to interact with CloudEvent data to match how your Cloud team is operating.

The clearest way to see that in action is to look at our WebHooks. These are primed to hook into and leverage the CloudEvents ecosystem. For example, you can use any of the dozen or so SDKs in your favorite language (we’re fans of Go!) or a bunch of integrations to process the data on your cloud. If you’re using one of our native output streams with AWS, GCP, or Azure, you don’t have to worry about developing the integration. Under the covers? It’s using CloudEvents too.

What are the future of CloudEvents?

CloudEvents have a specification that addressed the most common use cases, like HTTP + JSON or Kafka + Protobuf. But it’s also an extensible spec, so they added IoT protocols like MQTT. Anyone can define their own event format and publish it.

At Golioth, we continue to add functionality to our Output Streams capability, and we’re able to integrate new ones with lower overhead because of CloudEvent standardization. If you’re interested in extending Golioth capability or need help passing data between our Cloud and another Cloud, please reach out on our Forum, our Discord, or by email.