Introduction & Requirements
In previous instalments we've covered all the details for the physical hardware for The Grid. Once it's all in the ground, however, a lot of software also needs to exist to control the show. We needed to write high level software that generates patterns to be displayed and implements a computer vision system for interactivity, and low level software to interface with the electronics and drive the switches. We also needed several debug and utility systems to assist us in creating the rest of the software. We chose Python for the bulk of our software as we're both familiar with it from our day jobs, and we felt we could quickly get this wide range of software up and running, from sophisticated computer vision code through to interfacing with hardware.
With this in mind we laid out a plan for our core software.
If you'd like to follow along at home, all the code we've written is on GitHub at https://github.com/cillian64/thegrid/tree/master/control/thegrid.
Core Control Software
`control.py` implements the core of the software, which is responsible for shuttling data between the "Patterns" (these decide which lights to turn on and when) and the "Sinks" (which act on that data, for example displaying a virtual Grid). It additionally exposes a control and information API over HTTP. The core contains the software entry point -- this is where execution really starts. When initialised, it starts the HTTP API and the tracking subsystem, loads any default sinks that have been configured in the settings file, and moves into the main loop.
In the main loop we check for any commands coming in from the HTTP API (`api.py`) and deal with them appropriately, for example by loading a new Pattern module or unloading a Sink. The currently loaded Pattern is executed to get the new display frame (i.e., which lights should be on or off) and this is sent to all active Sinks for outputting (You can have multiple Sinks loaded at the same time, for example driving the real grid alongside a debug display). The main loop then goes to sleep until it's time to display the next frame.
The HTTP API is designed to allow other scripts or remote controls (for example an Android app for selecting patterns) to control the system running The Grid without needing physical access to the control PC. In the end we implemented three functions in the API: you could use it to load a new Pattern (replacing the previous pattern) or load or unload a Sink. We chose HTTP because it's easy to work with from any number of other clients, and very simple to get running in side Python using the fantastic Flask library.
Patterns
A lot of the software work in The Grid went into creating good-looking patterns (all found in the `patterns/` directory). To create these we inherit from the Pattern class and define our own `update()` function which will return a NumPy 7x7 boolean array of poles to light up. The class can otherwise do anything it likes -- for example contain local state, interact with other libraries, or display a pattern based on the bitcoin blockchain!.
To make it easy to add new patterns we also created a registration system so that each new Pattern class simply uses the Python decorator `register_pattern` which allows a name and configuration object to be specified. The decorator then stores a reference to the class so that the core control code can find and load it, with a specific configuration (e.g. a number of players, or a particular data file to load).
Static Patterns
We then got started making actual patterns. The obvious ones to start with were simple looping geometric patterns, such as scrolling bars or expanding diamonds. Instead of trying to write code to generate the appropriate frames, we devised a simple text based format where we could draw each frame in a text file. Each pole is either indicated by a `.` (off) or a `*` (on), and each frame is followed by a delay time in milliseconds. For example:
...*...
..*.*..
.*...*.
*.....*
.*...*.
..*.*..
...*...
100
We then wrote a Pattern class that could load these text files and send the right frame through to the control code. We called these 'static patterns', stored in the static_patterns directory. By using the `register_pattern` decorator multiple times with a different filename configuration, it was easy to add all the different files to the list of available patterns.
We quickly found swapping between all these patterns quite tiresome, so created the Playlist pattern, which loads up a number of other Patterns and loop through them. This was quite satisfying as it demonstrated how modular our pattern system was! In just a few minutes we could make a Pattern out of existing Patterns and it appeared just like any other Pattern to the control code.
Music Patterns
We wanted to try patterns synchronised to music. We created a desktop app (also in Python) which allowed users to draw frames against a music timeline, with functions like snapping to the beat and repeating sections. This app could save out the pattern data, and we wrote a MusicPattern class that could load these files and play back the appropriate MP3 file at the same time as displaying the patterns.
Unfortunately creating these patterns still proved very labour intensive so we didn't have any good ones ready in time for EMF 2014. Hopefully next time!
We ended up with a few other patterns driven by Python code, including an automatic game of Snake, a lightning inspired grid filling pattern, and a simple Pong game.
Interactive patterns
Finally we also wanted computer vision based interactive patterns. This requires a working computer vision system!
The basic concept of this system was to have a camera watching the whole Grid and attempt to detect where people were in its field of view, and therefore locate them in Grid coordinates. Since The Grid is a very difficult environment to work in (dark, with rapidly changing bright light sources, and fast-moving people) we knew we were in for a challenge. We made sure our camera could see into the infra-red spectrum (lots of cheap CCTV cameras can do this!), which allowed us to illuminate The Grid with IR floodlights that are invisible to the human eye. We designed the poles so that as few LEDs as possible would be pointed directly towards the camera, which helped prevent blinding it. We even tried using an IR filter on the camera, but in the end it caused more problems than it solved -- we couldn't mount it properly on the camera, and got quite a lot of internal reflections between the camera glass and lens.
Once the camera is capturing images that are as good as we can make them, we have to detect people in those images. The first hope was to use a trained machine learning approach, which would recognise people-shaped outlines in the image. However not only does this require a lot of computational resources, but it turned out to be quite hard to recognise people from a high elevation in the dark. Instead we went with a second approach, where we filtered the images to detect motion, and then tagged areas of motion that were large enough and moving fast enough to be likely a person rather than Grid poles swaying in the breeze or random noise. By tracking these moving areas over time, and only accepting areas that were consistently large enough, we could eliminate many more false positives. Then we find the centroid of each moving area, which tells us where (in the camera's field of view) the person is. Combining this with the position of the four edges of The Grid, we can use a projection to find the person's location on the flat plane of The Grid itself. The code to do this is in cv.py.
This display shows a green plane at roughly chest-height, which is where we predict most people's centroids to be. Detected movement areas, after being grouped together, are filtered based on size, aspect ratio and location to remove the false positives (the red and yellow markers). After a point has been tracked for more than a few frames inside The Grid area we declare it a human centroid (the green box) and continue to track its movements through the grid.
This finally means we could tell patterns where on The Grid we thought people were. From here it's fairly easy to create a Pattern that lit up poles near people. We wanted to create a maze pattern where users could explore a dynamically generated maze by running through The Grid, but again unfortunately we ran out of time.
Sinks
The Sinks (in the `sinks/` directory) are responsible for using patterns' output data to do something useful. We ended up with a number of sinks - some would display the pattern on the computer screen (console.py and tkinter.py), useful for debugging and testing, and others wrote data to hardware in various ways (ftdi.py and leonardo.py).
Prototyping
The first Sink we created just printed a representation of the current grid status to the console, which was very simple but reasonably effective. A later upgrade added cursor control codes so we didn't spam the console quite so much. To improve matters we wrote a version with a graphical interface using Tkinter.
We made one hardware sink while testing and prototyping, which used the common FTDI chip in its bit banging mode to directly drive the shift registers on the electronics. This was pretty slow using libftdi in Python so we ending up abandoning this route.
Arduino
In the end, to drive the actual Grid hardware, we installed an Arduino Leonardo which communicated with the control software over serial, receiving 7-byte frames which could be directly written out to the shift registers. This was fast and simple to connect up. The Arduino sink uses PySerial for its serial communication.
Future & Conclusions
So for our debut, we had a working control system with its HTTP API, a collection of Patterns, and the hardware and screen Sinks. During EMF itself, friends and strangers wrote us a few new patterns, such as the lightning and snake patterns, as well as a few other pretty displays. We also put together most of the computer vision while sitting in a cold field at night! For next time, the main software improvement would definitely be getting the interactivity a bit more developed and stable. The systems we created for adding new Patterns worked very well, allowing people to create new patterns in just an hour or two during EMF.
Tune in next time for our final instalment, with details of the actual set-up and running at EMF and a retrospective.