I'm revisiting the Harting MICA Complete IIoT Starter Kit. A recent integration container is available. A service that can subscribe to messages, modify them and forward to another service. If you're working in e-commerce, EDI or integration projects, you'll be familiar with subscribe - map - deliver applications. This micro-service offers that on an IOT scale. Tiny footprint, specialised in small messages and low-key protocols.
The module allow you to listen to messages, translate them if you are interested, and pass them on. Key technologies:
|
Why Use an Integration Engine - and What Do They Do?
In e-commerce and message exchange facilities, it's a common approach to get messages, translate them, and pass them on to applications.
For external messages, this type of infrastructure (in particular the receiving part) is often sitting in a company's demilitarised zone. External partners can send orders, requests, shipping confirmations, ...
For internal use, there are in-network integration engines that route and adapt messages.
They can accept messages. Investigate who needs them. Then translate those messages in the right format for the receiving party.
Popular applications are Tibco, SAP PO/XI, Seeburger, Sterling Integrator, BizTalk
This paradigm is useful for IIOT setups too. It's good to have an infrastructure "at the edge" that can listen to messages, check where they are wanted, and transform them so that the destination can accept them.
The Harting Cloud Integration container does that. Light weight, so that it offers the important functions and keeps the footprint low. What you'd expect from an IIOT level service;
The Container's Services
Staying true to the embedded light-weight world, this container offers the 3 essential services needed, and uses light-weight solutions to implement them:
- listen to incoming messages: they support a few low-level protocols like http(s), tcp and udp, and: MQTT(s)
- map messages: JSON with JavaScript
- forward messages: identical protocols as the listen service.
I like this approach. Don't get side-tracked by the many choices that are out there. Focus and be able to run on a humble Linux powered ARM SBC.
When you use this together with a node-RED container, you have a low-key, flexible backend for IIOT messages. A routing/mapping engine and a workflow engine.
You could implement the Integration Engine services in node-RED flows too, by creating MQTT IN nodes, JavaScript Function nodes and MQTT out nodes.
But I prefer these declaration based routers.
Business scenarios where many diverse messages come in and have to be translated and routed, have been well-served by this approach.
Good to see that a light service is available to route IIOT flows.
How does the Harting Mica Container Work?
It's simple. Let's get into it.
There are 3 things to maintain:
- connections. Let's say an MQTT(s), UDP, HTTP(s), .... service.
- payload manipulation scripts. input and output are JSON. Language is JavaScript.
- routings that combine connections and scripts.
In this exercise, I'll use this simulation scenario:
- One MQTT service acts as the end point for all messages in the scennario: Harting's own MQTT service running on another MICA container.
- I'm setting up the Cloud Integrator to listen to any message arriving at the NewMessages, and post it to the ModifiedMessages topic.
- In between, I modify the payload.
In the real world, this could be us waiting for sensor messages on one service, and if it was indicating an under-temperature, adding dedicated markers before sending it to the messaging system of the alerting system.
- we're listening to any message on an MQTT service's New Messages entry point.
This is how the message looks like (sent by a node-RED flow in my example):
- when a message appears, we alter it:
replace "new message" with "modified message".
change the status from "new" to "modified"
add the mapping date as an attribute (viewed in the Cloud Integration runtime here):
- forward it to another MQTT topic for Modified Messages (I have a node-RED flow listening to those and dumping the content to a debug screen);
I'm using one MQTT service (could be two different ones) and two topics:
- NewMessages
- ModifiedMessages.
I'll be using 2 node-RED flows to send messages and to view routed results.
Connection
(used for sending and receiving here but can be 2 different ones)
The host is the MQTT service running on the same device, in another Linux container.
Routing
Then I set up the sender and receiver of the Route:
As explained before, both are using the same MQTT service in my example. But the left part is a listener, the right side a poster.
You can already see an entry that listens to /NewMessages topic and forwards to /ModifiedMessages.
The "/" part of the topic is the Topic Prefix in the Subscriber attributes.
Mapping
The in between box is grayed out. That's where we define the JavaScript "enrich_map.js" that will map the received message into a form that the destination wants:
We alter the "name" attribute, replace the "status" attribute and add a "date" attribute before handing over to the destination queue.
As shown earlier, this is the example payload we'll get on /NewMessages:
{
"name": "test payload: new message",
"status": "new"
}
This is what we'll forward to /ModifiedMessages:
{
"name":"test payload: modified message",
"status":"modified",
"date":"Sat Dec 05 2020"
}
Test Bed in node-RED
I use two node-RED flows to work with these topics:
One flow (the top one below) simulates JASON payload to the NewMessages topic. This could be a sensor sending data in the real world.
The other flow (bottom one) subscribes to the ModifiedMessages topic. It dumps the message payload to a debug window for this blog post. But in reality it could trigger an alert.
Monitoring Option 1: an MQTT Subscriber app:
I've subscribed to both topics in MQTT.fx, so that I can see the new message being posted, and the mapped message being posted:
Monitoring Option 2: Harting Mica Observer log
The container has a logging mechanism where you can see what happens at input, during mapping and at output:
It allows for fine granularity. You can flag each of the items individually for observation, then enable the observer to see activity.
In the image above, I enabled it for the listener, the mapper and the poster.
What Do You Get?
Time for a summary. What does the container offer:
- The subscribe -translate - push engine is a great asset in an IIOT scenario.
- You can reuse service connections in several scenarios.
- There's an Observe function that allows you to see what's happening at receiving, mapping and sending end of the routing funnel.
- Standards: MQTT, UDP, JSON.
Impovements?
Yes. There is room for improvement without increasing the footprint.
- currently, there's now way to reuse a JavaScript across routes. You can only create and save scripts. Not associate a script with different routes.
- the secure implementation is meh. I have a secure MQTT running, but can't get it to work with the service's MQTTS implementation. I'd have to know how its java stack is implemented. The documentation is dim.
- a decision engine at the inbound side may be useful. In particular with reusable JavaScript code. Then again: that's a design choice: reusability vs light-weight-ability.
- allow to suppress messages if the content is of no interest for the receiver? By returning null or an empty ("not interested") JSON object in the JavaScript?
Top Comments