How to work with the new buckets and update applications for more efficient data storage

How to work with the new buckets and update applications for more efficient data storage

In this article we plan to explain the main changes between the Legacy Buckets and the new Immutable/Mutable buckets recently released by TagoIO.


The main goal is to help you perform the migration as smoothly as possible, answering any question you may have.



What are the new buckets?

You can see a detailed description of the new buckets on the Buckets documentation page.

Currently, there are two choices you can make: Immutable and Mutable


The Immutable bucket is a type of database optimized for massive amounts of data from IoT devices, you can now go beyond 1 trillion data points per application, delivering a very fast response time. As the name suggests, its structure doesn't allow users to remove or edit any data in the bucket. Removing data must be done through Data Retention only.


The Mutable bucket is mainly for devices that you will be using for database and application information. It can also be used for data provided from analyses that you plan to delete or edit frequently. It's a very fast bucket as well, with a storage capacity limited to 50,000 registers per bucket. Although not recommended, you can use it for IoT sensors too, but it doesn't have Data Retention and you will need to run Analysis to remove data from the bucket.


That said, the decision about the type of bucket you need to select relies more on your application, and how you will access the information you have in those two different types of buckets.




What are the changes to the API?

A couple of noticeable changes were also applied in the new buckets:


1. Data object keys

You can see a detailed description of data key at the Sending data documentation

  1. The key serie was renamed to group. This change doesn't affect its usability.

  2. The key origin was renamed to device. This change doesn't affect its usability.

The Legacy bucket received support for group and device key as well.

You shouldn't notice any impact by changing dashboards and analyses to the new parameters while keeping legacy buckets in the profile.


2. Device Creation

You can see a detailed description about device creation at the Device Create SDK method.

A new parameter key type was added while creating devices. It accepts the value of "legacy", "immutable" or "mutable". The Data retention had been updated as well, which now you must enter the new keys chunk_period and chunk_retention. Go to the documentation to get informed about Data Retention system.


Example:

  1. account.devices.create({

  2.   name: "Device #01",

  3.   network: "5bbd0d144051a50034cd19fb",

  4.   connector: "5f5a8f3351d4db99c40dece5",

  5.   type: "immutable",

  6. chunk_period: "quarter", chunk_retention: 2,

  7. })

The API route should include it as well. Notice that the mutable bucket doesn't allow for data retention parameters.


3. Addition of editData method (Mutable)

You can see a detailed description about editing data at the Device EditData SDK method.

This method is now allowed for Mutable buckets in order to allow you to edit specific data by its ID.


  1. const device = new Device({ token: "Device-Token" });

  2. device.editData({

  3.   id: "6aad0d144051a50034cd18b0",

  4.   value: 20

  5. })

4. Changes to deleteData method

You can a see detailed description about deleting data at the Device DeleteData SDK method.


The method for deleting data remains the same, except that now the difference between start_date and end_date can't go higher than 1 month. If the mentioned parameters aren't provided, it will delete the last 1 month of data by default.



5. Deleting Device automatically deletes the bucket

You can a see detailed description about deleting device at the Device Delete SDK method.


Deleting a device now automatically deletes the bucket, as having multiple devices by bucket is now deprecated.




FAQ

We created this section to answer frequently asked questions by our community.


Do I need to update dashboards, such as formulas, when I am using “serie” ?

As the old "serie" is now deprecated, it is highly recommended that you update dashboards and analyses to use the new keys.

That said, doing it right now is not necessary. The dashboards will continue to support old parameters and automatically get new ones if one is not available.


Do I need to update my analysis which is using the series and origin ?

While the old buckets will continue to respond with the “serie” and “origin” to the analysis, the new ones will not. Updating the analysis scripts is essential in this case.


What should I do if I need to edit the information in my Immutable device?

Basically, you can't edit anything or delete individual registers.

One option is to make use of the device Configuration Parameters, as it can keep the last data on any information you desire, and can be presented on the Device List widget. We are still investigating if we can deliver this information to other widgets as well.

Another option is to have two devices, an Immutable and a Mutable, for the same device. You can achieve that in different ways, and blueprint dashboards can work very well for two devices by using the Blueprint Device filters.