Introduction
In this blog I explain how I created a solution, which displays real-time train arrivals data for a station platform (i.e. in one direction) on a 20x4 character OLED display using a Rasberry Pi PICO-W board. The motivation behind this project was two fold.
Firstly, I wanted to create a real world application using the skills I picked up when developing my LCD parallel PIO driver for the Raspberry Pi Pico, as explained here: /products/raspberry-pi/b/blog/posts/lcd-parallel-driver-using-pico-pio
And secondly, this project came about as a solution to a problem where I was spending a bit too much time each day checking my local railway network’s website (i.e. Irish Rail) for the next train arrival to Greystones versus those terminating in Bray as my teenage son kept phoning to ask to be collected from Bray claiming that he had too long a wait. So I thought about making my life easier and started exploring options, as Irish Rail, like many other public transport entities around the world, provides a real time API to extract the relevant data... and I just happened to have a brand new Raspberry Pico-W board waiting to be used.
Real-Time (Transit/Public Transport) Data Portals
It is now quite common for regional transit / public transport authorities to provide developers (via free registration) with an Application Programming Interface (API), giving access to live data feeds associated with the relevant transit / public transport network. These portals typically provide up-to-the minute (or every 30 second) updates of train or bus position and also provide estimated arrival times to the next stop, together with scheduled arrival and departure times.
Here are some links I came across showing what transit organisations currently provide to developers:
https://tfl.gov.uk/info-for/open-data-users/
https://digitransit.fi/en/developers/apis/4-realtime-api/
https://www.mbta.com/developers/gtfs-realtime
https://gitlab.com/LACMTA/gtfs_rail
https://www.translink.ca/about-us/doing-business-with-translink/app-developer-resources/rtti
https://www.nationaltransport.ie/attention-developers-upgrade-to-gtfs-realtime-api/
https://opendata.transport.nsw.gov.au/node/328/exploreapi
As you can see, most of these API’s have now adopted the preferred General Transit Feed Specification (GTFS). To learn more about this specification, there is a central documentation platform for the General Transit Feed Specification (http://gtfs.org/) which is maintained by https://mobilitydata.org/.
This specification has also been adopted by Google (https://developers.google.com/transit) for its own transit api and for overlaying data onto Google maps etc.
As you can see from the Google documentation portal, there are two format specifications. One is for schedules or static data, which forms the base specification and then there’s an extension which covers the real-time transit data updates.
Almost all of these API’s provide output in either XML or JSON data formats following a RESTful or SOAP GET or POST request. Most provide the output in human readable text format, as shown in this example below, but some are adopting the more efficient Protobuf binary message format (also developed by Google).
However, when reviewing the API’s system architecture and data formats, it became pretty clear that the intended end-point for using these data structures was never meant to be at the humble microcontroller as all require polling to obtain updates and the returned data can be quite resource intensive due to limited filtering options.
So I had to think of a simple means of extracting what I needed using some other means, which provided more computing resources.
Enter cloud-based Serverless Functions as a Service
Serverless Functions as a Service is something newish. There’s a quite a few options to choose from, such as from the usual suspects i.e. AWS Lambda / Azure / GCP, through to a couple of new entrants out there, such as the new Cloudflare Workers.
They all essentially provide you with similar event driven functionality, hosted in the cloud, which allows you to develop your application using familiar software, such as NodeJS for example.
As I had used Google Apps Script before on this Element14 project, I decided to use this option again as I found that Google Apps Script provided me with all the relevant functions to parse XML and extract the data I needed. I also had the option of storing data into a spreadsheet and presenting extracted data directly on a presentation slide, which is rather cool (although not needed for my intended solution).
Using Google Apps Script
For those who have not used Google Apps Script before, I suggest reading the learning guide and overview found on the Google webpage. According to the overview guide, Google Apps Script is a rapid application (Javascript-based) development platform that makes it fast and easy to create business applications that integrate with Google Workspace.
Similar to my CO2 monitoring application, I would use the GET (or POST) function to request the data I needed.
However, this time I decided to call the Google Apps Script function directly from the PICO-W microcontroller rather than use an intermediary messaging service, as I had done with my CO2 monitoring application, as I wanted to see how well it would perform. It also gave me the opportunity to test out some of the Arduino WiFi libraries built for the PICO-W board.
The part that was new to me was using the URL Fetch Service (https://developers.google.com/apps-script/reference/url-fetch) to retrieve and parse the XML data from the real-time train network API.
The next vital ingredient for my system was to get my Google App Script to return a JSON payload as a response to a GET request. This is handled by something called the Google Content Service. Thus to get my application to return JSON, I simply had to use:
return ContentService.createTextOutput(JSON.stringify(result)) .setMimeType(ContentService.MimeType.JSON);
Fortunately, this all proved quite straightforward and there was not much coding required.
function doGet(e) {
// Only need this if you want to store data in a sheet
var ss = SpreadsheetApp.openByUrl('https://docs.google.com/spreadsheets/d/--- Enter Unique Reference ----');
const url = '----- Enter Transit API URL Here ----------';
const xmlresponse = UrlFetchApp.fetch(url);
const XMLParsedDoc = XmlService.parse(xmlresponse.getContentText());
const XMLroot = XMLParsedDoc.getRootElement();
// Namespace is optional.
const IRNamespace = XmlService.getNamespace('--- taken from API URL ---');
// Then next is very much dependent on XML structure in terms of children and child etc.
const StationEntries = XMLroot.getChildren('-- specific name reference from XML node ---', IRNamespace);
var Destination = StationEntries[i].getChild('-- specific name reference from XML node ---', IRNamespace);
// --- add specific logic here to extract relevant data
// --- create your return object in JSON format
const Returnobj = {T1: [Responsedata[0][0], Responsedata[0][1], Responsedata[0][2], Responsedata[0][3]],
T2: [Responsedata[1][0], Responsedata[1][1], Responsedata[1][2], Responsedata[1][3]],
T3: [Responsedata[2][0], Responsedata[2][1], Responsedata[2][2], Responsedata[2][3]],
T4: [Responsedata[3][0], Responsedata[3][1], Responsedata[3][2], Responsedata[3][3]],
QT: Querytime.getValue(),
BA: TrainArrivalBray};
// Return object
return ContentService.createTextOutput(JSON.stringify(Returnobj)).setMimeType(ContentService.MimeType.JSON);
}
The final step in the process is then to deploy the application as a web app. This gives you a unique URL to use, which you simply copy into the Arduino code.
My Arduino Firmware for the PICO-W
To develop WiFi enabled PICO-W firmware using the Arduino IDE you currently need to use the Arduino Pico Board Manager setup by Earle F. Philhower, III. Details on how to include are found on the relevant GitHub page: https://github.com/earlephilhower/arduino-pico
This board package includes WiFi specific libraries such as HTTPClient, which also includes a number of familiar WiFi examples.
I based my solution off the BasicHttpsClient-Hard example.
The key point about this example is that it sets up the Pico-W board as an insecure client using the client.setInsecure() command. According to the example’s comment, it is a setting whereby it is not safe against MITM (man-in-the-middle) attacks. In other words it does not validate any data received/sent etc.
Then I needed to use another function to handle redirects (https.setFollowRedirects(HTTPC_STRICT_FOLLOW_REDIRECTS)) as this is how Google Apps Scripts returns data.
As mentioned in the introduction, the LED display part was all taken care of by my Pico PIO LCD library - as found here: /products/raspberry-pi/b/blog/posts/lcd-parallel-driver-using-pico-pio.
/**
Displays Train Arrival Times on a 20x4 LCD
obtained from Google Apps Script API
using the BasicHTTPSClient.ino example as
the template, obtained from the Board Manager and Examples found here:
https://github.com/earlephilhower/arduino-pico/blob/master/libraries/HTTPClient
Created on: 24.09.2022
Author C. Gerrish
MIT License
*/
#include <WiFi.h>
#include <HTTPClient.h>
#include "myWiFiCreds.h"
#include <ArduinoJson.h>
// include the Pico LCD PIO code library:
#include <LiquidCrystal_PicoPIO.h>
// define the pin numbers:
static const int RS_PIN = 14;
static const int EN_PIN = 15;
static const int BASE_PIN = 16;
static const int ROWORDER = 5;
static const char text1[] = ("My Train Arrivals ");
static const char text2[] = ("display for Bray ");
static const char text3[] = ("Station... using a ");
static const char text4[] = ("Raspberry PI PICO-W ");
static const char *ssid = STASSID;
static const char *pass = STAPSK;
uint8_t Row0 = 0;
uint8_t Row1 = 1;
uint8_t Row2 = 2;
uint8_t Row3 = 3;
bool firstTrainData = true;
WiFiMulti WiFiMulti;
// This class will provide the PIO memory offset for the capsensing PIO (only need once)
LCDPicoPIO PicoPIO(pio0, LCD_4BITMODE);
LiquidCrystal_PicoPIO lcd(PicoPIO, RS_PIN, EN_PIN, BASE_PIN);
String T1str = "";
String T2str = "";
String T3str = "";
String T4str = "";
String QTstr = "";
String BAstr = "";
String T1loc = "";
String T2loc = "";
String T3loc = "";
String T4loc = "";
void setup() {
pinMode(ROWORDER, INPUT_PULLUP);
T1str.reserve(32);
T2str.reserve(32);
T3str.reserve(32);
T4str.reserve(32);
T1loc.reserve(32);
T2loc.reserve(32);
T3loc.reserve(32);
T4loc.reserve(32);
QTstr.reserve(12);
BAstr.reserve(4);
Serial.begin(115200);
Serial.println();
Serial.println();
delay(200);
// set up the LCD's number of columns and rows:
lcd.begin(20, 4);
// Check row order as sometimes gets reversed
Serial.println("\r\nChecking LCD Row Order");
delay(700);
for (uint8_t i = 0; i < 5; i++) {
lcd.home();
lcd.clear();
if (digitalRead(ROWORDER)) {
Row0 = 1;
Row1 = 0;
Row2 = 3;
Row3 = 2;
}
else {
Row0 = 0;
Row1 = 1;
Row2 = 2;
Row3 = 3;
}
lcd.setCursor(0, Row0);
lcd.print("Testing...");
delay(600);
}
lcd.home();
lcd.clear();
// Print welcome message to the LCD.
lcd.setCursor(0, Row0);
lcd.print(text1);
lcd.setCursor(0, Row1);
lcd.print(text2);
lcd.setCursor(0, Row2);
lcd.print(text3);
lcd.setCursor(0, Row3);
lcd.print(text4);
for (uint8_t t = 4; t > 0; t--) {
Serial.printf("[SETUP] WAIT %d...\r\n", t);
Serial.flush();
delay(1000);
}
WiFi.mode(WIFI_STA);
WiFiMulti.addAP(ssid, pass);
}
void loop() {
// wait for WiFi connection
if ((WiFiMulti.run() == WL_CONNECTED)) {
HTTPClient https;
https.setInsecure(); // Note this is unsafe against MITM attacks
https.setFollowRedirects(HTTPC_STRICT_FOLLOW_REDIRECTS);
uint32_t t_request = millis();
Serial.println(F("\r\n[HTTPS] begin..."));
// configure target server and url
if (https.begin(GOOGLEAPPURL)) { // HTTP
// start connection and send HTTP header and body
int httpCode = https.GET();
// httpCode will be negative on error
if (httpCode > 0) {
lcd.home();
lcd.clear();
// HTTP header has been send and Server response header has been handled
Serial.printf("[HTTPS] GET... code: %d", httpCode);
Serial.printf(" (response time: %d msec)\r\n", (millis() - t_request));
// file found at server
if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
const String& payload = https.getString();
Serial.println(payload);
// Allocate the JSON document
DynamicJsonDocument doc(640);
// Deserialize the JSON document
DeserializationError error = deserializeJson(doc, payload.c_str());
https.end();
// Test if parsing succeeds.
if (error) {
Serial.print(F("deserializeJson() failed: "));
Serial.println(error.f_str());
lcd.home();
lcd.clear();
lcd.setCursor(0, Row0);
lcd.print("Retrieval Error... ");
delay(60000);
}
else {
QTstr = String(doc["QT"]);
BAstr = String(doc["BA"]);
Serial.print(F("\r\nLatest BRAY Southbound Train Arrivals as at: "));
Serial.println(QTstr);
// Fetch values and format/shorten text to print on LCD
String arrTime = String(doc["T1"][1]);
if (arrTime.length() == 4) arrTime = " "+arrTime;
if (String(doc["T1"][0]).indexOf("Wexford") >=0) T1str = "1."+String(doc["T1"][0])+" "+arrTime;
else if (String(doc["T1"][0]).indexOf("Bray Train") >=0) T1str = "1."+String(doc["T1"][0])+" "+arrTime;
else T1str = "1."+String(doc["T1"][0])+" "+arrTime;
T1str.replace(" 0min", "Arrvd");
T1str.replace("Greystones", "G'st");
T1str.replace("Wexford", "WXFRD");
T1str.replace("Rosslare Europort Train", "Rosl Port");
arrTime = String(doc["T2"][1]);
if (arrTime.length() == 4) arrTime = " "+arrTime;
if (String(doc["T2"][0]).indexOf("Wexford") >=0) T2str = "2."+String(doc["T2"][0])+" "+arrTime;
else if (String(doc["T2"][0]).indexOf("Bray Train") >=0) T2str = "2."+String(doc["T2"][0])+" "+arrTime;
else T2str = "2."+String(doc["T2"][0])+" "+arrTime;
T2str.replace("Greystones", "G'st");
T2str.replace("Wexford", "WXFRD");
T2str.replace("Rosslare Europort Train", "Rosl Port");
arrTime = String(doc["T3"][1]);
if (arrTime.length() == 4) arrTime = " "+arrTime;
if (String(doc["T3"][0]).indexOf("Wexford") >=0) T3str = "3."+String(doc["T3"][0])+" "+arrTime;
else if (String(doc["T3"][0]).indexOf("Bray Train") >=0) T3str = "3."+String(doc["T3"][0])+" "+arrTime;
else T3str = "3."+String(doc["T3"][0])+" "+arrTime;
T3str.replace("Greystones", "G'st");
T3str.replace("Wexford", "WXFRD");
T3str.replace("Rosslare Europort Train", "Rosl Port");
arrTime = String(doc["T4"][1]);
if (arrTime.length() == 4) arrTime = " "+arrTime;
if (String(doc["T4"][0]).indexOf("Wexford") >=0) T4str = "4."+String(doc["T4"][0])+" "+arrTime;
else if (String(doc["T4"][0]).indexOf("Bray Train") >=0) T4str = "4."+String(doc["T4"][0])+" "+arrTime;
else T4str = "4."+String(doc["T4"][0])+" "+arrTime;
T4str.replace("Greystones", "G'st");
T4str.replace("Wexford", "WXFRD");
T4str.replace("Rosslare Europort Train", "Rosl Port");
T1loc = "1. "+String(doc["T1"][3]);
T1loc.replace("Arrived", "at");
T1loc.replace("Departed", "left");
T2loc = "2. "+String(doc["T2"][3]);
T2loc.replace("Arrived", "at");
T2loc.replace("Departed", "left");
T3loc = "3. "+String(doc["T3"][3]);
T3loc.replace("Arrived", "at");
T3loc.replace("Departed", "left");
T4loc = "4. "+String(doc["T4"][3]);
T4loc.replace("Arrived", "at");
T4loc.replace("Departed", "left");
Serial.println(T1str+" "+String(doc["T1"][2])+" "+String(doc["T1"][3]));
Serial.println(T2str+" "+String(doc["T2"][2])+" "+String(doc["T2"][3]));
Serial.println(T3str+" "+String(doc["T3"][2])+" "+String(doc["T3"][3]));
Serial.println(T4str+" "+String(doc["T4"][2])+" "+String(doc["T4"][3]));
for (uint8_t x = 0; x < 3; x++) {
lcd.home();
lcd.clear();
lcd.setCursor(0, Row0);
lcd.print(T1str);
lcd.setCursor(0, Row1);
lcd.print(T2str);
lcd.setCursor(0, Row2);
lcd.print(T3str);
lcd.setCursor(0, Row3);
lcd.print(T4str);
delay(14000);
// turn on automatic scrolling
lcd.home();
lcd.clear();
lcd.setCursor(0, Row0);
lcd.print(T1loc);
lcd.setCursor(0, Row1);
lcd.print(T2loc);
lcd.setCursor(0, Row2);
lcd.print(T3loc);
lcd.setCursor(0, Row3);
lcd.print(T4loc);
delay(4000);
}
if (firstTrainData && QTstr.length()>5) {
QTstr.trim();
int QTsecs = QTstr.substring(QTstr.length()-2).toInt();
// Found throuh experimentation that a GET request made a couple of seconds after the minute helps
// Syncing the GET request based on first response
if (BAstr.toInt() > 2) {
if (QTsecs > 15) {
QTsecs = 72 - QTsecs;
delay(QTsecs*1000);
Serial.print(F("\r\nAdditional Seconds Delayed (sync): ")); Serial.println(QTsecs);
}
firstTrainData = false;
}
}
}
}
} else {
Serial.printf("[HTTPS] GET... failed, error: %s\n", https.errorToString(httpCode).c_str());
https.end();
delay(30000);
}
} else {
Serial.println(F("[HTTPS} Unable to connect"));
delay(120000);
}
}
}
Demonstration
Here is a video showing the LCD output from my PICO-W project.
And here is a short video showing how data can be captured directly into Google sheets:
