Tag Archive for: visualizing data

How to use Ubidots data visualization

Golioth and Ubidots have teamed up to make it easy to visualize your Internet of Things device data. Ubidots makes it easy to white-label visualizations to produce a branded version of your service for end customers; you can also create end-users, in order to allow access to custom dashboards.

Our partnership has produced an integration that makes it a snap to connect your device data to an Ubidots dashboard so that it looks great for you and for your clients. Today I’ll walk you through the process of setting up a connection.

Ubidots data visualization dashboard

What you need to get started

To get started you just need a few basic things:

  • An account on Golioth (sign up)
  • An account on Ubidots (sign up)
  • A device that is sending data to Golioth

Creating your accounts is pretty easy; Golioth has a free dev tier, and Ubidots has a free Educational/Personal use option or a free trial period for their Business users. Everyone who has followed our Getting Started guide will already has a Golioth account.

If you don’t already have a device set up to send data, you can use the Golioth LightDB Stream example. It will simulate temperature data and send it back to the Golioth Cloud, a perfect dataset for your first Ubidots dashboard!

Setting up a connection between Golioth and Ubidots

To connect Golioth and Ubidots we need to take the follow steps:

  1. Configure an Ubidots plugin
  2. Configure a Golioth output stream
  3. Set up an Ubidots dashboard

1. Configure the plugin in Ubidots

Ubidots has a plugin specifically for Golioth. But before configuring the plugin we want to make sure we have a token to use for this project. From the Ubidots dashboard, click your profile picture in the upper right, choose API Credentials and click More under the tokens menu that appears. This will open the API Credentials page.

How to create an ubidots token

Use the plus sign to add a new token, then use the pencil icon to edit the name of the token.

Next, choose DevicesPlugins from the top menu. Click the plus to add a plugin, and choose Golioth. Click the arrow icon to get to the plugin configuration menu.

Choose the token you just created from the dropdown menu. In the LightDB Stream field, use the name of the endpoint where your device is sending data.

If you have a nested JSON, like {"env":{"temp":20.5}}, then you would set this value to “env” and the Ubidots plugin would pick up all of the key/value pairs inside of that JSON object.

If you are following along with the Golioth Stream sample, leave the LightDB Stream field blank, since we publish to the root-level directly. You can see in the code example below, we are setting the temperature value to the root-level “temp” key.

err = golioth_lightdb_set(client,
			GOLIOTH_LIGHTDB_STREAM_PATH("temp"),
			COAP_CONTENT_FORMAT_TEXT_PLAIN,
			str_temperature,
			strlen(str_temperature));

Click the right arrow, give your plugin a name and description, and click the check mark icon to save. You will be redirected to a list of your plugins.

[screenshot]

Click on the one you just created and then select Decoder from the left sidebar.

Generally speaking, this page is already configured to work. We need to copy the Ubidots HTTPs Endpoint URL and use it to create a Golioth Output Stream in the next step.

If you want to learn more about the data format so that you can customize the decoder on this page, look at the DeviceStreamMessage in our Event Types documentation. For today’s example, the defaults will work.

2. Configure a Golioth output stream

Choose Ubidots as Output Stream

In the Golioth Console, select Output Streams from the left sidebar, click on Create an Output Stream, and choose Ubidots from the list.

Give your new Output Stream a name, and choose DEVICE_STREAM_TYPE as the event filter. The HTTPS Endpoint URL comes from the Ubidots dashboard. So does the auth token, which you can copy from the Ubidots dashboard by clicking your avatar in the upper right and choosing API Credentials(this reveals a panel that lets you copy tokens to the clipboard). Click save to finish setting up your Output Stream.

3. Set up an Ubidots dashboard

We’re ready to start visualizing! At the top of the Ubidots dashboard, select DataDashboards. You can create a new dashboard using the icon (sometimes called a “hamburger menu”) in the upper left.

On your new dashboard, click Add new Widget and choose a Gauge from the list.

Ubidots dashboard gauge settings

In the data section, click Add Variable, choose your device, and select the data you want to show. If your device is not listed, the endpoint may be configured wrong, or the device has not yet sent any data. There is a Logs section in the Ubidots plugins settings page that will be helpful in troubleshooting the issue.

The rest of the settings in this dialog decide how your data will be displayed. Set the name of the widget, and choose the range that suits you best before clicking the check mark icon to save the changes.

Temperature gauge

Now you have a gauge that shows temperature readings from your device. Of course there are many other visualization methods available to match your needs, so spend some time playing around with your new Ubidots dashboard!

This guest post is contributed by Ben Mawbey, a community member who is active on the Golioth Discord and frequently takes part in Office Hours.

Data wants to be visualized. The impact of showing a customer a slick plot of the information their devices have been collecting is massive compared to pointing at a few hundred lines of text from a log file or database query.

I was looking for some sort of dashboard or charting application for a demo for a sensor system we’ve built. I figured this will appease the clients’ need for pretty pictures over boring reports. I found exactly what I needed, and it only took me about 30 minutes to get it up and running.

Grafana graphing data

This looks great, even if you have no idea what the graphs mean! Let’s dig into how to get from numbers in a database to pretty pictures in a dashboard.

Pieces of the Puzzle: Golioth WebSockets, Node-RED, InfluxDB, and Grafana

Golioth is brilliant at getting your device data from the real world up to the cloud, climbing the IoT bean-stalk some would say. While abstracting some of the trickiest IoT problems, Golioth can present your time-series data as a convenient cloud resource using LightDB Stream service. By leveraging the built-in WebSockets support and some open-source tools we can rapidly store, manipulate and display this data!

Grafana is the open-source blockbuster for this application and can be easily set up to graph sensor data directly from Golioth LightDB Stream using the REST API. This has two major drawbacks:

  • Grafana must periodically poll for new data
  • While LightDB Stream does provide convenient data retention, I prefer to use my own data storage

Enter InfluxDB, the time-series database powerhouse and an ideal companion to Grafana when talking about IoT real-time applications. This pairing is so popular that the InfluxDB data source integration is baked right into Grafana! By utilizing InfluxDB to store our sensor data, we can perform more complex queries much faster.

The question remaining is how to shuffle data from Golioth to InfluxDB? There are many potential solutions to this hurdle, but my favorite is Node-RED which defines itself as a low code programming tool for wiring up event-driven systems.

Node-RED editor window

Node-RED uses graphical flows to connect data sources and destinations, bending protocols and translating data formats in innumerable ways

Node-RED has exploded in popularity and provides all sorts of integrations to connect your systems together. It provides simple blocks to perform actions and a slick graphical interface to wire up your data flows. Conceptually Node-RED acts as our rule engine to process and direct data.

Dashboards: Grafana and InfluxDB

System diagram

Grafana is immensely powerful at providing example custom views, data transformations, and alerting. That said, it is only as good as the quality of data you can provide. Having a tightly coupled InfluxDB instance with carefully curated data via Node-RED, allows you to quickly configure complex data queries on large datasets with low latency.

Before we can play with Grafana the first step is InfluxDB setup. After you’ve installed InfluxDB, create a new database on your InfluxDB instance:

> CREATE DATABASE golioth

Configure the InfluxDB Data Source in your Grafana instance by clicking the gear icon on the left sidebar, choosing and adding a data source, then searching for the InfluxDB plugin. Here’s how I’ve set up my data source:

Grafana InfluxDB source configuration

Assuming you have some data in your DB, we can quickly create a new time-series dashboard panel in Grafana and query the dataset using this integration:

Grafana panel configuration

This simple query shows how we structured the data in the DB, allowing us to select from a particular measurement, specifying a specific device identity tag and aggregating data points with specific time buckets. Adjusting the time range instantly updates the graph from our local InfluxDB instance.

Now how do we get the data from Golioth into InfluxDB?

Node-RED

Several networking integrations are provided with Node-RED. Presently the most relevant to Golioth would be HTTP nodes for REST API requests and the WebSockets node which is the easiest to configure.

You can see your sensor data collecting in the Golioth LightDB Stream by using the Golioth Console web interface:

Golioth Console showing LightDB stream data

We can use Node-RED flows to connect to Golioth via WebSockets and store the resulting data in our local DB:

NodeRED editor window

The nodes in this flow were set up as follows, taking care to give them appropriate names making it obvious to see their function. All of the nodes I’m using should come as part of every Node-RED installation except for the InfluxDB nodes. But don’t worry, these are trivial to install. On Linux is looks something like this:

cd ~/.node-red
npm install node-red-contrib-influxdb
sudo systemctl restart nodered.service

WebSockets Node:Node-RED websockets

First set up the credentials to your Golioth project using your generated API key and connect to the WebSockets LightDB Stream endpoint.

Debug Node:Node-RED debug

Drop a few of these nodes along the way and click the small green button to turn the debug log on. This is super handy to check data coming through and make sense of it.

JSON Node:Node-RED json

The LightDB Stream endpoint provides us with a JSON object representation containing our sensor data as well as meta information such as the data timestamp and device identity. This node allows us to parse this JSON into a javascript object so that we can work with it more easily in subsequent nodes.

Change Node:Node-RED change node

This node clearly shows the power of Node-RED as we can craft any sort of data manipulations or transformation.

We could do without this node and jump straight to InfluxDB, however, should any malformed data arrive we risk polluting the DB with bad data. By selectively transforming the incoming data and mapping it into a new object we can not only filter only good packets and arrange the measurement names but also add tags to build a solid data representation in the DB making our queries far more powerful.

InfluxDB Out Node:

Node-RED InfluxDB input nodeNode-RED InfluxDB server settings window

Finally, we can configure the data connection to our InfluxDB instance, set appropriately for your server configuration and database created earlier.

Assuming your flow is set up correctly, you should be able to see data collecting in your database. We know it works, but as I mentioned before, this visual is not going to impress our customers.

InfluxDB data

Revisiting the Grafana panel we previously created, you can see InfluxDB data is now being plotted!

Grafana graphing data

Corner Cases

One of the downsides of WebSockets is their ethereal nature, should there be some temporary connectivity issue, any data packets would be lost from the point of view of your InfluxDB database. A solution around this could be to set up another flow that executes periodically to sync with the LightDB Stream using the REST API. Node-RED could then be configured to check this data and add any missing values into the InfluxDB instance and prevent consistency issues.

Another concern with open-source self-hosted systems is security. It can be challenging to secure your server and services should they be public-facing. If you are handling sensitive data then it would be best to consult with an expert in this field. Fortunately, all of the tools discussed have subscription-based cloud services available that sort all of this out in the background.

Conclusion

Being able to set up a simple demo like this in less than 30 minutes demonstrates the power and flexibility of these modern open-source solutions. Coupled with the reliability and maintenance advantages of the Docker system, it’s a breeze to test locally on your desktop or Raspberry Pi and then deploy to production a moment later on your cloud server of choice. The rules engine and ease of wiring up blocks provided by Node-RED opens up a massive pool of possibilities, from countless other integrations to building intelligent processes. One such idea I would like to explore is integrating a device provisioning process into the flow such that we can link a device to a dataset or location during deployment or maintenance.