Group Actions
Engagement
  • Author Author: Jan Cumps
  • Date Created: Date Created
  • Views 2658 views
  • Likes 5 likes
  • Comments 1 comment
Related
Recommended

AWS IoT: convert a character array into JSON in a MQTT Rule

Jan Cumps

This is a little side note, related to the project Arduino Day Workshop: NanoDrone II: AI and Computer Vision with LoRa (Win a PSoC6 and a Pair of MKR 1300 Boards!).

In this project, 4 attributes are sent to Amazon Web Services, via MQTT.

We get the data as a fixed length char array. And at some point it needs to get converted into a JSON message.

 

 

fixed length record:

image

The red marks are not in the record. They indicate where decimal and field separators are.

 

JSON format:

{

  "longitude": 0.12345,

  "latitude": -0.12345,

  "msgtime": "12:23:44",

  "rating": 73,

  "timestamp": 1617734941770

}

 

 

The data flows from the Nanodrone to the base station over LoRa. Sender and receiver are an Arduino.

Then from the base station Arduino to a PSoC6 over UART, and from there over WiFi and internet to the AWS MQTT service in the cloud.

Somewhere, this conversion has to happen. We have 4 choices:

  • directly generate the information in JSON format on the drone's Arduino, and be done with it.
  • raw data record from the drone, and translate in the base station Arduino
  • raw until the PSoC6, and translate before sending it to AWS MQTT
  • keep the raw array until it arrives at AWS, and translate it into JSON there.

 

Each option will work. The choice will have impact on:

  • message size: fixed length array is smaller than in its JSON form.
  • format cost: the block that does the translation will do extra calculations (for the battery powered blocks: sleep less) and memory footprint increases.
  • bill: calculations on AWS have a cost (but this example fits in the free tier).

 

For this exercise, we choose to do the translation on the AWS side, in its IoT Rule engine.

We base our choice on:

  • keep the data size shorter for the wireless protocols,
  • save memory space in the embedded devices,
  • save battery life in those same devices by skipping conversion and deep-sleep instead.
  • make the big AWS system that has a fixed power supply, do the work.

 

Use an AWS Rule to convert to JSON

 

When you create fields in a Rule, they automatically become JSON attributes.

What we want to do here, is parse each of the fields in the record into an field in the rule.

The rule will kick in when a message arrives on the MQTT topic nanodrone.

And it will deliver the output to a destination. For this exercise, it's reposted on the MQTT service topic nanodrone_json.

 

Define the rule:

 

In AWS IoT, select the Act option in the sidebar, then Rules. Create a new one.

image

 

Create the rule code to split the char array in fields.

 

It's SQL:

 

SELECT 
cast(concat(substring(decode(encode(*, 'base64'),'base64'),0, 2),   ".", 
  substring(decode(encode(*, 'base64'),'base64'),3, 8)) as Decimal) as longitude
,  
cast(concat(substring(decode(encode(*, 'base64'),'base64'),8, 12), 
  ".", 
  substring(decode(encode(*, 'base64'),'base64'),12, 17)) as Decimal) as latitude
,  
concat(substring(decode(encode(*, 'base64'),'base64'),17, 19),
  ":",
  substring(decode(encode(*, 'base64'),'base64'),19, 21), 
  ":",
  substring(decode(encode(*, 'base64'),'base64'),21, 23) ) as msgtime 
,  
  cast( substring(decode(encode(*, 'base64'),'base64'),23, 26) as Int) as rating 
,
  timestamp() as timestamp  
FROM 'nanodrone'

 

the decode/encode pairs are needed because the char array is binary data for the rule, and this back-and-forth turns it into a String.

You will find the future JSON elements back as the field names in the SQL (e.g.: longitude).

 

Deliver the output to an AWS service:

 

You have many options here. For the Nanodrone, a good destination could be a DynamoDB table, analytical service, ...

For this exercise, I send the output back to the MQTT service, as new topic  nanodrone_json.

That makes it easy to see the results. And you can change that later.

 

image

Expanded:

image

 

When you accept everything, AWS will create a new role, that allows you to publish to the MQTT service.

If you have not done this before, accept the one that AWS will generate. It's a good closed one that only allows Publish on topic nanodrone_json.

 

Test:

 

You don't need the Nanodrone for this. Everything can be tested from within the AWS IoT console, via the MQTT tester:

First, subscribe to the two topics nanondrone and nanodrone_json.

 

image

 

 

Then, in the same test application, select the Publish tab.

Enter nanodrone as topic name, +0012345-00012345122344073 in the payload field.

 

image

 

Press the Publish button. Now check the two topics you subscribed to.

First nanodrone. It shows the exact message that was published:

 

image

 

Then nanodrone_json. It shows the JSON representation. Spot the extra timestamp that I sneaked into the Rule.

image

 

That's it. The goal was not to find the best and cleanest option. It's a self-training that turned into a blog post.

We may go for one of the other choices for the final drone design. If you would do this different, leave a comment.