In-depth guide to Payload Parser

In-depth guide to Payload Parser

The Payload Parser is a data transformation tool in TagoIO that processes incoming device data before it's stored in your device's bucket. It acts as an intermediary layer that can decode, normalize, and modify data from IoT devices, ensuring it's in the correct format for storage and analysis.

When IoT devices send data to TagoIO, they often transmit it in formats that aren't immediately usable—such as hexadecimal strings, raw bytes, or proprietary encoding schemes. A Payload Parser is a JavaScript function that transforms this raw data into TagoIO's standardized JSON format, making it accessible for dashboards, analytics, and other platform features.


Supported Operations
  • Data Decoding: Convert hexadecimal payloads and byte arrays into readable values
  • Format Normalization: Transform any data structure into TagoIO's standard JSON format
  • Value Conversion: Perform calculations, unit conversions, and data type transformations
  • Protocol Support: Works with any communication protocol (LoRaWAN, Sigfox, MQTT, HTTP)
  • Data Filtering: Remove unwanted variables before storage
  • Variable Addition: Create new variables based on calculations or static values
  • Device Context Access: Read device tags and configuration parameters during parsing
Technical Constraints
  • No Historical Data Access: Cannot retrieve previously stored data from the device's bucket
  • Resource Isolation: Cannot access other TagoIO resources (dashboards, actions, other devices)
  • Execution Time Limit: Code must complete within 2 seconds
  • No External Requests: Cannot make HTTP requests or connect to external services
  • Synchronous Only: No support for asynchronous JavaScript operations

How Payload Parsers Work

Info
Device → Network Server → Payload Parser → Bucket Storage
         (if applicable)    (transformation)

When data arrives at your device:
  1. The network server (LoRaWAN, Sigfox, etc.) normalizes the data if applicable
  2. Your Payload Parser receives the normalized data
  3. The parser transforms the data according to your code
  4. Only the parser's output is stored in the bucket

Execution Context

The Payload Parser runs as an evaluated function with two parameters:
  1. // Your code runs inside this context:
  2. function payloadParser(payload, device) {
  3.   // payload: Array of data sent to the device
  4.   // device: Object containing device metadata
  5.   
  6.   // Your transformation code here
  7.   
  8.   payload = [] // Replaces the Payload to return the transformed data
  9. }


Getting Started

Enabling Payload Parser

  1. Navigate to your device in the TagoIO platform
  2. Select the Payload Parser tab
  3. Toggle the feature to Enable
  4. Choose a template from the snippet dropdown menu:
    1. Parse example for LoRaWAN: For LoRaWAN devices
    2. Parse example for Sigfox: For Sigfox devices
    3. Convert Raw JSON to TagoIO JSON: For HTTP devices sending JSON data
    4. Parse Example for MQTT Hexadecimal: If you're sending a hexadecimal data usi
    5. For MQTT devices, review this guide to build a custom parser

Code Structure

  1. // Available packages (pre-imported)
  2. // - dayjs: Lightweight date library
  3. // - timeUtils: TagoIO proprietary library to handle timezones

  5. // Function parameters
  6. // payload: Array of incoming data objects
  7. // device: Object with device metadata
  8. //   - device.tags: Array of device tags [{key, value}]
  9. //   - device.params: Array of config parameters [{key, value, sent}]

  11. // Example: Filter unwanted variables
  12. payload = payload.filter(item => item.variable !== 'unwanted_var');

    // Payload is automatically returned to the system with the new data packet

Working with Binary Data

Hexadecimal to Buffer Conversion

Most IoT sensors send data as hexadecimal strings. Convert them to buffers for easier manipulation:
  1. const payload_raw = payload.find(x => x.variable === 'payload');
  2. const buffer = Buffer.from(payload_raw.value, 'hex');

  4. // Example hex: "08FB383F"
  5. // Buffer array: [0x08, 0xFB, 0x38, 0x3F]
  6. // Decimal values: [8, 251, 56, 63]

Buffer Reading Methods

There are some methods available that help us when reading bytes from a buffer:

MethodDescriptionBytes ReadEndianness
readInt8(offset)Signed 8-bit integer1N/A
readUInt8(offset)Unsigned 8-bit integer1N/A
readInt16BE(offset)Signed 16-bit integer2Big Endian
readInt16LE(offset)Signed 16-bit integer2Little Endian
readUInt16BE(offset)Unsigned 16-bit integer2Big Endian
readUInt16LE(offset)Unsigned 16-bit integer2Little Endian
readInt32BE(offset)Signed 32-bit integer4Big Endian
readInt32LE(offset)Signed 32-bit integer4Little Endian
readUInt32BE(offset)Unsigned 32-bit integer4Big Endian
readUInt32LE(offset)Unsigned 32-bit integer4Little Endian


Bitwise Operations

Extract specific bits using masks:

  1. // Extract lower 4 bits
  2. const lowerNibble = buffer[0] & 0x0F;

  4. // Extract upper 4 bits
  5. const upperNibble = (buffer[0] & 0xF0) >> 4;

  7. // Check if specific bit is set
  8. const isBitSet = (buffer[0] & 0x80) !== 0;


Debugging Tools

An important step when running your payload parser script is to be able to test and debug it for any issue. In your device’s page, you should also see the Emulator and the Live Inspector section.

Live Inspector

The Live Inspector provides real-time visibility into data flow:
  • Incoming data: Raw payloads received by the device
  • Parser execution: Console output and errors
  • Stored data: Final data written to the bucket
  • MQTT events: Connection status and publishes

By using a console.log inside your payload parser, you can print content of variables and arithmetic results when the payload parser runs for any data that being stored to your device. This is very helpful when trying to understand why you’re receiving errors or unwanted results.

  1. console.log('Raw payload:', payload);
  2. console.log('Device tags:', device.tags);
  3. console.log('Parsed temperature:', temperature);

Live Inspector Example

Emulator

The Emulator allows you to test your parser without sending actual device data:

  1. Copy a raw payload from Live Inspector or craft test data
  2. Paste into the Emulator input field
  3. Execute to see parser results
Example emulator inputs:
  1. // LoRaWAN format
  2. [
  3.   { "variable": "payload", "value": "001100" },
  4.   { "variable": "port", "value": 100 }
  5. ]
  6.  
  7. // MQTT format
  8. [
  9.   {
  10.     "variable": "payload",
  11.     "value": "0109611395",
  12.     "metadata": { "mqtt_topic": "data" }
  13.   }
  14. ]

MQTT Emulator Example



Complete Example

Here's a practical parser for a temperature/humidity sensor:

  1. // Find the payload in incoming data
  2. const payload_raw = payload.find(x => 
  3.   x.variable === 'payload_raw' || 
  4.   x.variable === 'payload' || 
  5.   x.variable === 'data'
  6. );

  8. if (payload_raw) {
  9.   try {
  10.     // Convert hex to buffer
  11.     const buffer = Buffer.from(payload_raw.value, 'hex');
  12.     
  13.     // Decode according to device protocol:
  14.     // Byte 0: Protocol version
  15.     // Bytes 1-2: Temperature (int16, big-endian, scaled by 100)
  16.     // Bytes 3-4: Humidity (uint16, big-endian, scaled by 100)
  17.     
  18.     const data = [
  19.       { 
  20.         variable: 'protocol_version', 
  21.         value: buffer.readInt8(0) 
  22.       },
  23.       { 
  24.         variable: 'temperature', 
  25.         value: buffer.readInt16BE(1) / 100,
  26.         unit: '°C'
  27.       },
  28.       { 
  29.         variable: 'humidity', 
  30.         value: buffer.readUInt16BE(3) / 100,
  31.         unit: '%'
  32.       }
  33.     ];
  34.     
  35.     // Preserve metadata from original payload
  36.     payload = payload.concat(
  37.       data.map(x => ({
  38.         ...x,
  39.         group: String(payload_raw.serie || payload_raw.group),
  40.         time: String(payload_raw.time)
  41.       }))
  42.     );
  43.     
  44.   } catch (e) {
  45.     console.error('Parser error:', e);
  46.     payload = [{ variable: 'parse_error', value: e.message }];
  47.   }
  48. }


Scaling with Connectors

What are Connectors?

Connectors are reusable Payload Parser templates that can be applied to multiple devices. Instead of copying parser code to each device, you create a connector once and reference it from any compatible device.

Creating a Connector

  1. Navigate to Integrations in your account menu
  2. Click Create Connector
  3. Select supported networks (LoRaWAN, Sigfox, MQTT, etc.)
  4. Paste your tested Payload Parser code
  5. Add documentation and metadata (optional)
  6. Save the connector

Benefits

  • Centralized Updates: Modify parser logic in one place
  • Consistency: Ensure all devices use the same parsing logic
  • Sharing: Make connectors public for community use
  • Documentation: Include setup instructions and examples

More information regarding the connectors can be found at Connector Overview - TagoIO



Additional Resources