New Pipelines Transformer: Embed in JSON

A new Pipelines transformer for embedding data in JSON is now generally available for Golioth users. The embed-in-json transformer enables data streamed from devices to be embedded as a string value for a key in a JSON object, where the object can then be further augmented before being delivered to a destination that expects a JSON payload.

How It Works

The embed-in-json transformer accepts data of any content type, escapes it if necessary, then embeds it in JSON object with a key specified in the transformer parameters. For example, in the pipeline shown below, the data payload will be embedded as a UTF-8 string value for the key text.

filter:
  path: "*"
steps:
  - name: embed
    transformer:
      type: embed-in-json
      parameters:
        key: text
  - name: send-webhook
    destination:
      type: webhook
      version: v1
      parameters:
        url: $MY_WEBHOOK

Click here to use this pipeline in your Golioth project!

Therefore, if a device sent a payload containing hello, world, the POST request delivered to the webhook would be:

{"text": "hello, world"}

However, it is common to combine the embed-in-json transformer with other transformers. For example, if devices are sending binary data, it may be useful to encode that data as text before embedding it in the JSON object. One way to accomplish this is by utilizing the recently announced base64 transformer.

filter:
  path: "*"
steps:
  - name: encode
    transformer:
      type: base64
  - name: embed
    transformer:
      type: embed-in-json
      parameters:
        key: text
  - name: send-webhook
    destination:
      type: webhook
      version: v1
      parameters:
        url: $SLACK_WEBHOOK

Click here to use this pipeline in your Golioth project!

This specific pipeline is one we use in many of our internal Golioth projects when we want to temporarily inspect unknown binary data being sent from devices. Consider the following binary data (displayed as hex encoded for readability).

A1 64 74 65 6D 70 18 20

When presented to the pipeline, the data will first be Base64 encoded, yielding the following result.

oWR0ZW1wGCA=

Then, the Base64 encoded data will be embedded in a JSON object.

{"text": "oWR0ZW1wGCA="}

This payload is finally delivered to the Slack webhook, which results in the following message being delivered to a channel in our workspace.

Slack message showing base64 encoded data.

From there, we are able to inspect the data sent by the device, in this case determining that it is CBOR data with the following content.

A1             # map(1)
   64          # text(4)
      74656D70 # "temp"
   18 20       # unsigned(32)

Sometimes we even like to provide a little extra information to our messages. For example, it would be helpful if the message in the Slack channel also told us which device sent the payload. This can be accomplished by incorporating the inject-metadata transformer, then using the json-patch transformer to craft a payload that adheres to Slack’s rich message layout formatting.

filter:
  path: "*"
steps:
  - name: embed
    transformer:
      type: embed-in-json
      parameters:
        key: text
  - name: metadata
    transformer:
      type: inject-metadata
  - name: patch
    transformer:
      type: json-patch
      parameters:
        patch: |
          [
            {
              "op": "add",
              "path": "/blocks",
              "value": [
                {
                  "type": "rich_text",
                  "elements": [
                    {
                      "type": "rich_text_section",
                      "elements": [
                        {
                          "type": "text",
                          "text": "Device ID: ",
                          "style": {
                            "bold": true
                          }
                        },
                        {
                          "type": "text",
                          "text": "REPLACE"
                        }
                      ]
                    },
                    {
                      "type": "rich_text_section",
                      "elements": [
                        {
                          "type": "text",
                          "text": "Message: ",
                          "style": {
                            "bold": true
                          }
                        },
                        {
                          "type": "text",
                          "text": "REPLACE"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "op": "move",
              "from": "/device_id",
              "path": "/blocks/0/elements/0/elements/1/text"
            },
           {
              "op": "move",
              "from": "/data/text",
              "path": "/blocks/0/elements/1/elements/1/text"
            },
            {
              "op": "remove",
              "path": "/data"
            },
            {
              "op": "remove",
              "path": "/timestamp"
            },
            {
              "op": "remove",
              "path": "/device_id"
            },
            {
              "op": "remove",
              "path": "/project_id"
            }
          ]
  - name: send-webhook
    destination:
      type: webhook
      version: v1
      parameters:
        url: $SLACK_WEBHOOK

Click here to use this pipeline in your Golioth project!

The same payload through this pipeline now produces the following formatted message.

Slack message showing device ID and data message.

For more information on the embed-in-json transformer, go to the documentation.

What’s Next

Because of the broad set of services with APIs that accept JSON requests, the ability to embed data payloads using the embed-in-json transformer enables targeting many more destinations. We’ll be sharing more examples, and we look forward to hear more about how users are leveraging Golioth Pipelines on the forum.

Dan Mangum
Dan Mangum
Dan is an experienced engineering leader, having built products and teams at both large companies and small startups. He has a history of leadership in open source communities, and has worked across many layers of the technical stack, giving him unique insight into the constraints faced by Golioth’s customers and the requirements of a platform that enables their success.

Post Comments

No comments yet! Start the discussion at forum.golioth.io

More from this author

Related posts

spot_img

Latest posts

Detecting Text From Images Over A Cellular Link

This post and video demonstrate taking a photo with a low power camera and cellular-based microcontroller and transmitting to the cloud for processing. Rick Jen from the Microsoft Azure team shows how to accept those images and work with the Azure AI OCR service to extract useful text and store it in a database.

Managing OTA Updates for Multiple IoT Hardware Variations

Get the right OTA update to the correct device every time! Goilioth's firmware update system lets you compile with a custom package name for each different hardware variant, and target the rollout to these devices using Cohorts.

Custom Board, Custom Zephyr Devicetree (AirTag Session 3)

Sign up for the AirTag Clone webinar series to watch Golioth build out board definition files for the Orleon board, a Bluetooth sensor platform/playground. The upcoming session on May 23rd will walk through setting up all required build files.

Want to stay up to date with the latest news?

Subscribe to our newsletter and get updates every 2 weeks. Follow the latest blogs and industry trends.