Chunk Management

Chunk Management

The Chunk Management feature is a powerful tool that allows you to easily manage the data of your Immutable devices, either to create backups or delete information.

By accessing the Device module and selecting a Device Data Optimized (Immutable) device type from your list, you will be able to find the Chunk Management section where there's a graph that shows your data chunks based on the period you chose when creating your device.

Export chunks to TagoIO Files

By selecting one or multiple chunks, you will be able to export them to your TagoIO Files in CSV format to create a backup of your data. In this process, the chunks are added to a queue and soon become available on the file address. This process can take several minutes, depending on the amount of data being exported.

File Address Template

During the exportation process, you will be able to customize the file address before exporting your data to TagoIO Files, and as a result, each chunk file will have its own file address. In the Advanced Settings, you can use key variables to change the path name that will be created inside your TagoIO Files. The variables you can use to customize your file address are listed below:

  • $DEVICE$ - the device ID

  • $CHUNK$ - the chunk ID 

  • $FROM$ - the chunk start date

  •  $TO$ - the chunk end date


For example, the following file address template:
  1. device/$DEVICE$/$FROM$_$TO$
Would result in a file address such as:
  1. /device/6297aa7691e70a00654f9816/2022-04-01_2022-06-30.csv
Date variables such as $FROM$ and $TO$ will be in the ISO 8601 format (YYYY-MM-DD)
When exporting a chunk to a location where a file already exists, all the data will be replaced with the new one.

Include Header

Activating the Include Header option will enable headers on CSV-generated files. Headers will describe the variable's data in each column, just like a column's name, but you can disable them in the Advanced Settings to keep your files filled only with rows of data.

How exporting is billed

Notice that exporting data consumes the Data Output service. The number of transactions billed is rounded to the next multiple of 10,000. For example:
  1. Exporting 700 registers will consume 10,000 data output transactions.
  2. Exporting 12,000 registers will consume 20,000 data output transactions.
  3. Exporting 32,000 registers will consume 40,000 data output transactions.
  4. Exporting 103,000 registers will consume 110,000 data output transactions.
Notes
The billing scheme mentioned above does not apply when fetching data using our API. For example, when using a GET request, the amount of data consumed will be equal to the amount of data fetched.

Delete chunks

You can delete chunks to optimize your data storage. This is an irreversible process, and once done there is no going back, so make sure to be certain before removing anything. You can check the chunk range information by hovering your mouse over the graph before deleting the data.
You can keep track of all interactions made on this feature through the Audit Log.

Automation

You can also access the chunk management functions using the SDK, Analysis, and Actions to automate processes to either backup or delete your data.

    • Related Articles

    • Device data management

      The device page provides tools for visualizing, filtering, editing, importing and exporting variables, facilitating the management of data received by your device. You can access these tools by navigating to the Device module, selecting a device, and ...

    • Team Management - Sharing your profile

      You can grant access to other TagoIO accounts to any of your Profiles by using the Team feature. By adding others to your team, they will have access to all the applications that run on that profile.  The invitation can only be sent to emails that ...

    • Data Export

      At TagoIO, you have multiple options for exporting data. You can export device data directly from the Admin panel on your device's page, through the API, or directly from Widgets in either the Admin panel or the TagoRun portal. Exporting data will ...

    • Rate Limits (Hard limits)

      When executing requests to TagoIO, you will have a limit on the number of requests that can be made during a certain time period. These limits are applied in short intervals every minute, allowing us to provide a reliable and scalable API that our ...

    • Data Retention Feature

      The Data Retention feature automatically removes old data from Immutable devices after the period you define. The goal is to help customers to avoid unnecessary costs with data that don’t need to be kept for a long period of time. The Data Retention ...