The Cooker Connector - Blog #6: Back-End Data Architecture (Pi Chef Challenge)

I have been writing a lot about the hardware for my prototype. This post is a shift to the software side, and I will be describing the architecture of my back-end system at a high level. If you don’t already have a basic understanding of my Cooker Connector project, take a few minutes to catch up on my project plan.

Firebase

Firebase is a platform by Google that allows you to create a back end for your application with their cloud, and it’s designed to help you do this incredibly quickly. Firebase is a collection of different types of services that you can pick and choose from depending on your needs, and it’s mostly free of charge until you need to scale. I’m going to use the Realtime Database, Cloud Functions, and Cloud Messages for my project. You can read more about all Firebase has to offer at firebase.google.com.

The Realtime Database

There are thousands of cloud data storage services out there, but the Firebase Realtime Database is one of my favorites because of its unique and outstanding capabilities. It is a database, so yes, it stores and organizes your data, but it stands out because it also can be used like an instant communication channel between two or more devices. This is where it gets its “realtime” name. A device can watch a particular location in the database, and whenever data is written to that location, that device gets notified with the new data instantly. If the device goes offline, it transparently writes to a local database, and when the connection is restored, the two sides are synchronized. All of this is neatly bundled up into an SDK that does all of the hard work for you, so you can focus on building the parts of your app that are unique.

Once you set up a project on the Firebase developer console, you download a custom configuration file and an SDK for your language of choice, which you embed in your application. Once initialized, the SDK connects with the Firebase servers, and your application makes requests to the SDK to read and write data. Internally, the SDK opens a fast socket connection between your application and their servers, and whenever data is updated in the database, the changes are pushed immediately over this socket. The entire database is not necessarily replicated in your app, however. Only the portions of the database that you are interested in are transferred, and only when your application is listening for changes.

My Cooker Connector is a perfect use case for the Realtime Database because I need to log temperature data from one device and read it on another, but I also want these updates to be instant. In my system, the Raspberry Pi will write temperature data to the database, and my phone will subscribe as a listener to this data. Whenever a temperature data point is written to the database, it will immediately get pushed to the app running on my phone. I can also query the database to let me graph several hours of data points on my phone, and as new data points are generated, my graph will update in real time.

JSON Data Structure

The Realtime Database does not work like a traditional SQL database. It uses a tree structure that is essentially one big JSON object. This may seem unusual if you are used to modeling your data into tables, columns, and rows. Instead, every value has a key, and keys and values can be grouped together and nested under each other to form relationships. Every object must have a key that is unique among its siblings. Instead of arrays, Firebase prefers objects with unique keys. It takes a little while to wrap your head around at first if you’re not used to this model.

The Realtime Database has an unusual data model, but its design is powerful because it works across a distributed system of readers and writers, and it can even work offline temporarily and still keep everything in sync. An important concept that allows this to work is unique keys. Thousands of devices can insert objects to the same location all at once, without synchronization, because they all have the ability to generate unique keys based on random numbers and a clock.

Cooker Connector Data Architecture

I'm going to organize my temperature data in a way that matches how I want it to be displayed in the mobile app. I want to have a separate timeline of events for every time I use the system, so I'm going to have a concept of Recording Sessions. All the data points from a single Recording Session will be nested inside a session object. When a data point is generated by the Sensor Hub, it will know which session it belongs to, so it will know where to insert the data.

Figure 1. Block diagram of back-end data architecture.

Session Configuration

The Sensor Hub configuration will be stored in the database. Each Hub will have its own ID and will listen for changes to its configuration. When the user wants to start a new recording session, they will use the mobile app to select which inputs of which Sensor Hubs to use, the sample intervals, and how to interpret the ADC values of each reading, as well as the parameters for any alerts. All of these parameters will go into the configurations for each Sensor Hub, and the Sensor Hubs will generate data points accordingly. Each data point will be written to the database as soon as it is generated.

The mobile app, on the other hand, will not receive every data point every time one is generated. If the app had to be running and using the network for the duration of a 12-hour cook, it would quickly run out of battery. Instead, the app will only listen for updates while it is open. When it gets launched, it will reconnect and start listening for updates, and only then will it receive data points it doesn’t already know about. However, the user needs to be alerted as soon as the temperature goes out of the desired range, so this alert must be generated even if the app is not running. This is where Firebase Cloud Functions and push notifications come in to play.

Firebase Cloud Functions

Another incredibly useful service in the Firebase suite is Cloud Functions. These are pieces of code that run in the cloud, but you don't have to set up or maintain any servers. You literally write a Node.js function, upload it, and then Firebase runs it whenever it is triggered. For my temperature alerts, I'm going to configure my function to be triggered by a particular type of database write. When a Sensor Hub gets a reading that should trigger an alert, it will change the status of the alert object in the database, and this will trigger my function. When it sees an alert change from inactive to active, it will send a push notification to the mobile app. The conditions for the alert, such as which sensor input should stay below a certain temperature, will be configurable by the mobile app and will be stored in the database as well.

Push Notifications

If you have a smartphone, and you have pretty much any app installed, you likely see breaking news alerts, advertisements, and other notifications that pop up on your phone while you’re not actively using it. This message format is called a push notification because the message is “pushed” to your device from the server, and your device displays it whenever the server asks it to. In contrast, when you open your news app and ask it to refresh, you are choosing to “pull” information from the server to your device on demand. My application will pull historical temperature data from the Realtime Database on demand, but my Cloud Function will also push alerts down to it, so that the user is notified of potential problems as soon as possible. If I didn’t use a push message for the alert, the chef wouldn’t know about the alert until they decided to open the app, at which point it may be too late.

A key concept of push notifications is that it is possible for a message to be delivered to an application while that application is not running. This is a special thing that mobile apps can do. Normally, to receive a real-time message, an application would need to be running and have an open socket connection waiting to receive messages. This is inefficient when you consider that there may be hundreds of apps on a single phone that want to receive messages. If they were running all at the same time, this would consume a lot of memory, power, and network resources on an already constrained device. Instead, all of these message channels can be funneled into a single channel, which is controlled by an app with special permissions, and not by any individual app. This special broker is the only process that is allowed to have a long-running open socket connection, and it handles all incoming push messages for all apps. Only after a message comes for your application does it get launched by the broker. This message broker is Google Play Services, which is installed on every device that has the Google Play (app) store. Apple, Microsoft, Amazon, and other app stores have their own versions of this push message service as well.

Firebase Cloud Messages

My application will be running on an Android phone, so I will be using Firebase Cloud Messages (FCM) to push my alerts. When my Cloud Function decides it is time to send a push message, it will make a request to FCM with the content of the message, and FCM will take care of delivering the message to the app installed on my phone. However, the specific installation on my phone needs to be linked somehow to the alert so that FCM knows which app on which device I’m sending it to. This link is created when the app is first installed on the phone. One of the first things it will do on its first launch will be to register with FCM and obtain a unique identifier. This is a token consisting of many alphanumeric characters that is for all intents and purposes random, but it serves as the address to which messages can be sent. Once this token is issued to an installation of my app, my app will save it in the Realtime Database, where it can be looked up by my Cloud Function. The Cloud Function can then use it to send push messages. Once the app receives a message, it can alert the user.

Now that I have designed my back end at a high level, I can get to working on it. I’m sure that I will need to modify or add some pieces here or there, but this should be a good starting point. I hope you found this interesting, and thanks for keeping up with my progress!