Golioth Output Streams enable IoT deployments built on the Golioth platform to export events to traditional cloud provider platforms. We now have support for Google Pub/Sub as a way to export your data into the GCP ecosystem.
In our initial announcement about Output Streams, we had exports for AWS SQS, Azure Event Hubs, and WebHooks. The first two act as buffers for the massive amounts of data coming into cloud systems from all over the web. These managed services can help to smooth out bursts of different requests hitting a server, acting as a queuing system so servers and serverless elements have time to “catch up” with the influx of new data arriving, ready to be processed. As deployments grow, they can also introduce redundancy and caching for different regions of the globe, while still acting like a monolithic service to the functions receiving input or providing output to these buffers.
Another cloud?
So why do we need yet another cloud service to receive data? Well, because users prefer it! With AWS, Azure, and now Google Cloud, we cover roughly 64% of the cloud infrastructure market according to Statista in Q4 of 2021.
We think it’s a bit crazy that cloud teams determine how IoT device makers run the services that go onto small devices. That means that an IoT Platform like Golioth shouldn’t make decisions going in the other direction! Some Golioth clients who are development shops have 3 different clients that want their data to go to 3 different cloud providers. Dev shops need to be nimble and have a lot of output options to serve their clients’ needs. Even within organizations at large companies, data might be flying to one provider or another. Software teams have reasons they choose one cloud provider over another, often far above the decision process of any particular engineer. We’re here to tell you, we don’t care where you send data. Your data should be able to go where you want it to go.
As a reminder, data can also stay on Golioth! Say you’re a hardware engineer looking to try things out and stream some sensor data to the cloud for visualization. You can directly interact with your data on the Golioth Cloud, including controlling your devices via our REST API. You can also use the WebHooks feature to push data out to charting programs like Datacake, Ubidots, or Grafana. You don’t need to export it immediately to an external provider, it’s just that some people have that as a business need given what their cloud teams are using.
Can I use Pub/Sub Directly?
It might be tempting to think that since you’re using Google’s Pub/Sub service, you could have your device talking directly to the service. It’s there to listen to different events, right?
This is a good opportunity to point out how Golioth is optimized to talk to constrained IoT devices. You could take a board using a cellular part like the nRF9160 and have it connect to the internet. If it wanted to talk to Google Pub/Sub, it would need to be making HTTP/2 requests and have authentication on the level that is usually required for one server talking to another. The amount of certificates, data overhead, and bandwidth would be a terrible choice for the nRF9160.
Instead, you could have your nRF9160 running a Zephyr application with the Golioth SDK talking to Golioth servers over CoAP, with CBOR encoding on the messages, and sending and receiving secure messages using LightDB. This will save bandwidth (data costs) and power for your cellular device. What’s more, you get great features like fleet management and OTA firmware updates through that same secure connection to your device. As you scale your fleet up to thousands or millions of devices, you’re still talking back to a similar connection provided by Golioth, but you’re piping the resulting data out to your cloud provider. When you’re ready to switch to a different one? Flip a switch! We’ll have even more cloud export options soon.
