New Pipelines Transformer: JSON Patch

A new JSON Patch Pipelines transformer is now generally available for Golioth users. It allows for modifying JSON payloads, or payloads that have been transformed into JSON in a previous pipeline step, in order to fit a specific structure or schema. To see this transformer in action, check out the documentation example or last week’s Friday Afternoon Stream.

Shaping Data for Different Destinations

Golioth frequently sits at the intersection of firmware and cloud platforms. One of our goals when launching Pipelines was to enable those to worlds to seamlessly interact, avoiding the need for one side to compromise to suit the other. For example, Pipelines can allow for devices to send data in a compact binary format, such as CBOR, then have it translated to a text-based representation, such as JSON, for delivery to a cloud service.

The json-patch transformer enhances this capability by not only changing the format, but also the structure and content. Furthermore, it allows for the structure required by the end destination to change over time, without requiring firmware updates. In the following example, fields are re-arranged to meet the requirements of the custom webhook data destination. Critically, if this destination changed, or a new one was added in the future, the pipeline could be updated, and the device could continue sending the same payloads.

filter:
  path: "*"
  content_type: application/cbor
steps:
  - name: change-format
    transformer:
      type: cbor-to-json
      version: v1
  - name: transform-and-deliver
    transformer:
      type: json-patch
      version: v1
      parameters:
        patch: |
          [
            {"op": "add", "path": "/environment", "value": {}},
            {"op": "add", "path": "/location", "value": {}},
            {"op": "move", "from": "/temp", "path": "/environment/temp"},
            {"op": "move", "from": "/lat", "path": "/location/lat"},
            {"op": "move", "from": "/long", "path": "/location/long"}
          ]
    destination:
      type: webhook
      version: v1
      parameters:
        url: https://my-backend.example.com/data
        headers:
          x-api-key: $BACKEND_API_KEY

Click here to use this pipeline in your project.

Conditional Data Manipulation

In some cases it may be desirable to conditionally patch a JSON object payload based on the contents of the payload, or the metadata associated with the device that sent it. Combining the json-patch transformer with other transformers demonstrates the power of Pipelines. The test operation in a JSON Patch document conditionally applies a patch if the criteria is met.

For example, in the following pipeline, the key-value pair demo: true is injected into the payload if the device ID matches 649998262fecb43eb2d39859. The device ID is made available when applying the patch using the inject-metadata transformer. The metadata is subsequently stripped from the payload to ensure extraneous information is not delivered to the final destination.

filter:
  path: "*"
  content_type: application/json
steps:
  - name: get-metadata
    transformer:
      type: inject-metadata
      version: v1
  - name: conditional-patch
    transformer:
      type: json-patch
      version: v1
      parameters:
        patch: |
          [
            {"op": "test", "path": "/device_id", "value": "649998262fecb43eb2d39859"},
            {"op": "add", "path": "/demo", "value": true}
          ]
  - name: remove-metadata
    transformer:
      type: json-patch
      version: v1
      parameters:
        patch: |
          [
            {"op": "remove", "path": "/device_id"},
            {"op": "remove", "path": "/project_id"},
            {"op": "remove", "path": "/timestamp"}
          ]
  - name: send-lightdb
    destination:
      type: lightdb-stream
      version: v1

Click here to use this pipeline in your project.

See it in action

What’s Next

The json-patch transformer is the first of many new transformers and data destinations we’ll be adding over the next few weeks. If you have a use-case in mind, feel free to reach out to us on the forum!