New Pipelines Transformer: Embed in JSON

A new Pipelines transformer for embedding data in JSON is now generally available for Golioth users. The embed-in-json transformer enables data streamed from devices to be embedded as a string value for a key in a JSON object, where the object can then be further augmented before being delivered to a destination that expects a JSON payload.

How It Works

The embed-in-json transformer accepts data of any content type, escapes it if necessary, then embeds it in JSON object with a key specified in the transformer parameters. For example, in the pipeline shown below, the data payload will be embedded as a UTF-8 string value for the key text.

filter:
  path: "*"
steps:
  - name: embed
    transformer:
      type: embed-in-json
      parameters:
        key: text
  - name: send-webhook
    destination:
      type: webhook
      version: v1
      parameters:
        url: $MY_WEBHOOK

Click here to use this pipeline in your Golioth project!

Therefore, if a device sent a payload containing hello, world, the POST request delivered to the webhook would be:

{"text": "hello, world"}

However, it is common to combine the embed-in-json transformer with other transformers. For example, if devices are sending binary data, it may be useful to encode that data as text before embedding it in the JSON object. One way to accomplish this is by utilizing the recently announced base64 transformer.

filter:
  path: "*"
steps:
  - name: encode
    transformer:
      type: base64
  - name: embed
    transformer:
      type: embed-in-json
      parameters:
        key: text
  - name: send-webhook
    destination:
      type: webhook
      version: v1
      parameters:
        url: $SLACK_WEBHOOK

Click here to use this pipeline in your Golioth project!

This specific pipeline is one we use in many of our internal Golioth projects when we want to temporarily inspect unknown binary data being sent from devices. Consider the following binary data (displayed as hex encoded for readability).

A1 64 74 65 6D 70 18 20

When presented to the pipeline, the data will first be Base64 encoded, yielding the following result.

oWR0ZW1wGCA=

Then, the Base64 encoded data will be embedded in a JSON object.

{"text": "oWR0ZW1wGCA="}

This payload is finally delivered to the Slack webhook, which results in the following message being delivered to a channel in our workspace.

From there, we are able to inspect the data sent by the device, in this case determining that it is CBOR data with the following content.

A1             # map(1)
   64          # text(4)
      74656D70 # "temp"
   18 20       # unsigned(32)

Sometimes we even like to provide a little extra information to our messages. For example, it would be helpful if the message in the Slack channel also told us which device sent the payload. This can be accomplished by incorporating the inject-metadata transformer, then using the json-patch transformer to craft a payload that adheres to Slack’s rich message layout formatting.

filter:
  path: "*"
steps:
  - name: embed
    transformer:
      type: embed-in-json
      parameters:
        key: text
  - name: metadata
    transformer:
      type: inject-metadata
  - name: patch
    transformer:
      type: json-patch
      parameters:
        patch: |
          [
            {
              "op": "add",
              "path": "/blocks",
              "value": [
                {
                  "type": "rich_text",
                  "elements": [
                    {
                      "type": "rich_text_section",
                      "elements": [
                        {
                          "type": "text",
                          "text": "Device ID: ",
                          "style": {
                            "bold": true
                          }
                        },
                        {
                          "type": "text",
                          "text": "REPLACE"
                        }
                      ]
                    },
                    {
                      "type": "rich_text_section",
                      "elements": [
                        {
                          "type": "text",
                          "text": "Message: ",
                          "style": {
                            "bold": true
                          }
                        },
                        {
                          "type": "text",
                          "text": "REPLACE"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "op": "move",
              "from": "/device_id",
              "path": "/blocks/0/elements/0/elements/1/text"
            },
           {
              "op": "move",
              "from": "/data/text",
              "path": "/blocks/0/elements/1/elements/1/text"
            },
            {
              "op": "remove",
              "path": "/data"
            },
            {
              "op": "remove",
              "path": "/timestamp"
            },
            {
              "op": "remove",
              "path": "/device_id"
            },
            {
              "op": "remove",
              "path": "/project_id"
            }
          ]
  - name: send-webhook
    destination:
      type: webhook
      version: v1
      parameters:
        url: $SLACK_WEBHOOK

Click here to use this pipeline in your Golioth project!

The same payload through this pipeline now produces the following formatted message.

For more information on the embed-in-json transformer, go to the documentation.

What’s Next

Because of the broad set of services with APIs that accept JSON requests, the ability to embed data payloads using the embed-in-json transformer enables targeting many more destinations. We’ll be sharing more examples, and we look forward to hear more about how users are leveraging Golioth Pipelines on the forum.