Acknowledgements

The following is a mini-project completed by five 3rd year undergraduate students at University College London (UCL) taking a module on the Internet of Things. Credit goes to the following students who kindly agreed to have their project posted on the Element 14 website:

• Andy Boscor
• Galen Han
• Octav Rusuleanu
• Claudia Stanescu
• Natcha Sujaritworakun

Overview

The key features of the system developed incorporating the Grid Eye kit include the following:

• Live seat availability tracking.
• Analytics of library usage.
• Intelligent light and heat system controls based on seat occupancy.

A web app of this system can be accessed at the link below:

http://studyhunt.galenhan.com/

A video describing its operation can be viewed on YouTube:

https://www.youtube.com/watch?v=gtB67Mpynnc

The Arduino code repository employing the Grid Eye kit is available on Github:

https://github.com/ironbila/Grideye_arduino

Additional repositories include the following:

Backend repository:

https://github.com/Mastergalen/StudyHunt-Backend

Frontend repository:

https://github.com/Mastergalen/StudyHunt-Frontend

Project Description

StudyHunt is a library seat availability live monitoring system that allows students to swiftly find an empty seat to study with a simple glance at our web app.

Everybody knows that during exam periods the libraries are packed and finding a space to study is a great challenge. Thus, the need for investment in new technology that can help students see at a glance available study spaces in all parts of the libraries is becoming more and more necessary. This way, students can choose where to work, without having to walk around and waste precious time looking for free study spaces. Furthermore, knowing how people are distributed around the libraries, will help provide study spaces that are more efficiently used in terms of energy usage. Using the occupancy information as well as library environment data gathered from Wi-Fi enabled temperature and light sensors, will allow automatic tuning of the HVAC and lighting systems to keep all library rooms optimally comfortable at minimum expense, achieving sustainability.

HVAC and lighting systems form a total of approximately 40% of the EU’s annual energy consumption and, thus, it is important to make them more efficient. Not all areas of the libraries are created equally. Some parts of the library can get very busy at particular times of the day and less busy, or even unoccupied during others – which means that energy is wasted when the HVAC and lighting systems are turned on uniformly around a building that does not have uniform occupancy and usage. By conditioning spaces based on occupancy, energy efficient libraries can be achieved, while maintaining library environment effectiveness.

Currently, UCL staff manually count the number of students in libraries every hour. This solution, although is cheap to implement, costs a lot in terms of human labour and is quite inefficient as the seats are rarely updated. As such, if students check at the wrong time (before receiving the hourly update), they might arrive at the library and not be able to find seats. Furthermore, this does not provide a seat map which means that students have to roam around and find free spots. The need for our solution is therefore clear.

Privacy and Climate Change Research

Due to the recent news regarding government surveillance [1], studies have been made that show 80% of Americans feel insecure about their online communication [2]. This in itself is a worrying statistic that requires action to be taken regarding data privacy. The European Commission and Swiss Administration together with the US department of commerce have created a framework called the Privacy Shield [3], in which companies and providers can comply with a data protection act meant to keep user information safe. This requires companies having strict policies and informing users that their data is being sent to the Cloud or that it is being processed.

Our idea was to create a library seat finder that would also provide energy efficiency. As such, we needed a way of detecting which seats were occupied. The initial thought we had was to use a camera (similar to the Raspberry Pi Camera Module [4]) and a microphone to be able to get almost accurate readings of who was in a room. This, however, required the data to be sent to the Cloud to a server for processing, as the low computing of a microcontroller would make local processing infeasible for human detection. Consequently, this raised many privacy concerns. Users would have been filmed and recorded and would have been required to give their consent to this. As such, every time a user would enter the library, they would have to sign a form consenting to them being filmed and recorded. This would have obviously deterred users from the library and would have become tedious in case a user would stop agreeing at any point to their data being shared with a Cloud server. An idea to solve this would have been Digital Catapult’s Consent Receipts [5], which could have quickly given users an overview of what they were sharing and what was being done with it. This, however, would have still been tedious in case a user wanted to opt out of sharing data.

Due to the reasons presented in the previous paragraph, we have concluded that using a camera and microphone is not a feasible solution, as we would be gathering too many personal details about users such as their habits, their conversations, how much time they would spend using the library services, etc. To remove any issues, we moved onto using a heat map with the Panasonic Grid-Eye [6]. This allows detecting humans only based on their heat signature. As such, we are not gathering any information on their habits, conversations and there is no way of individually matching a user’s name to their library usage. Furthermore, all processing is done locally on the device and is not being sent over to the Cloud. What we are sending over, is simply the seats that are occupied.

There are growing concerns regarding global warming, pollution and various other threats to our planet that are not being taken seriously by the population. A study by Spence et al. [7] shows that only people directly related to climate change issues would actually change their behaviour and be more efficient in their usage of resources. As such, there is a definite need to automate energy efficiency and we decided that for every area within the library that is unused, we would turn down the lighting and heating system to prevent unneeded usage.

Competitor Analysis

The only company that is offering a similar solution, a live space finder which locates available workspaces in buildings, is Cad-Capture Data Services Limited. Their solution is called OccupEye [1], which has already been employed by three universities: University of Edinburgh, University of Kent and Birkbeck College. A fully-deployed OccupEye installation consists of sensors linked to a small number of receivers, the system controlled and monitored via data logging software. The sensors are wireless, battery-powered and based on passive data collection technology (PIR), triggered by both motion and heat. One single unit, having the size 11cm x 6.5cm x 2.7cm, can be used to monitor individual desk utilisation. However, our solution employs state-of-the-art technology, one single unit of size 10.15cm x 5.4cm x 5cm mounted on the ceiling being able of monitoring the occupancy of four spaces.

Their solution offers a live space finder, however, the User Experience is not consistent throughout the app. The desktop and tablet version offer a map of the seats to help users find their way to a space. The mobile version, however, does not seem to have a floor map, and as such creates an inconsistent experience to users (Fig. 1). Consequently, we decided to make our solution fully responsive and keep the same experience throughout all the devices and implement the map on the mobile version as well.

With the increasing trend of using Data Analytics to provide a better User Experience, we realised that OccupEye's solution does not offer any insights for users. It does not offer any notifications of busy times of certain spaces, nor does it offer any way for users to see when spaces get busy. Consequently, we decided to add these features to our solution.

Due to the previously mentioned research paper regarding Climate Change, there is an increasing need to automate energy efficiency in shared spaces. OccupEye however, does not make any smart decisions regarding heating and lighting. This is a lost opportunity, that our system chose to take advantage of.

Figure 1. OccupEye cross-platform User Experience

Privacy and Climate Change References

1. The Guardian, NSA files, https://www.theguardian.com/us-news/the-nsa-files

2.Unger, N., Dechand, S., Bonneau, J., Fahl, S., Perl, H., Goldberg, I. and Smith, M., 2015, May. SoK: secure messaging. In Security and Privacy (SP), 2015 IEEE Symposium on (pp. 232-249). IEEE.

3. Privacy Shield, https://www.privacyshield.gov/welcome

4. Raspberry Pi Camera Module, https://www.raspberrypi.org/products/camera-module-v2/

5. Consent Receipts, Digital Catapult, https://www.digitalcatapultcentre.org.uk/project/pd-receipt/

6. Panasonic GridEye, https://eu.industrial.panasonic.com/grideye-evalkit

7. Spence, A., Poortinga, W., Butler, C. and Pidgeon, N.F., 2011. Perceptions of climate change and willingness to save energy related to flood experience. Nature climate change, 1(1), pp.46-49.

Competitor References

1. OccupEye, http://www.occupeye.com/how-it-works-2/

Fig. 1. Occupeye, http://www.occupeye.com/live-space-finder/

Criteria

Our criteria for designing the system were:

• Track seat occupancy with high accuracy and low latency
• Allow students to easily find a free seat with their smartphone
• Scalable sensor design, theoretically allowing all 3,000 study spaces at UCL to be tracked
• Intelligently control HVAC and lighting to save energy
• Respect privacy of student's visiting the library
• Allow monitoring of library usage for UCL management
• Provide a seamless cross-platform experience with smart suggestions

Introduction

When faced with the challenge of determining the presence of people in a specific area there are several technologies available
that must be analysed in order to develop an efficient solution. Most of these technologies are based on infrared (IR)
detectors (which can be separated into two categories: photon detectors and thermal detectors), and image recognition
from camera.

Photon detectors are, however, not a popular choice of IR system, due to their cooling requirements which make them bulky,
expensive and inconvenient to use. Thermal detectors on the other hand, have no necessity of cooling and exhibit high-temperature
stability and low cost, making them a widely used IR system. Thermal detectors can be further divided into two categories:
thermopile and pyroelectric, i.e. passive infrared motion sensor. Although these two types have dominated the low-end
single-element infrared sensor market, only the former is suitable for array sensors. By using single-element thermopile
sensors and pyroelectric sensors, it is only possible to detect moving people, but not the position and presence of motionless
people and objects.

Proposed solutions

To meet the design criteria, we considered various solutions. The advantages and disadvantages for tracking
seat occupancy are also weighted.

Pressure sensors for each seat

A Piezoelectric sensor is installed in each chair to detect if the seat is taken. Moreover, the normally wasted kinetic energy
from when someone sits on or leaves the chair can be converted into a usable electrical energy to charge a battery or
directly power the sensor system. This allows the system to be self-sufficient.

We did however, have concerns regarding the comfort of adding such a sensor to the seat cushion. We then further started
dismissing this solution, as sensors needed to be installed on every chair and would have required a Bluetooth connection
to a central hub with WiFi that would then forward the sensor data to a cloud server. This implied that moving a chair
too far away from the hub would break the system, and if the central hub failed, so would the whole system. The solution
to this would have been to have mesh nodes, but that would have been a far too complex solution.

In addition, chairs tend to migrate around the building in a random fashion. If a chair is moved from the table, a seat map
would no longer be accurate and the entire map would need to be updated with the new location of the seat.

Ambient noise

There was the possibility of adding a microphone to our set-up to measure noise levels. Although this idea might have helped
for group study spaces where noise is allowed, a library is normally silent. Furthermore, due to the Data Privacy concerns,
previously mentioned in our Research Analysis, our solution would have required written consent from each library
user to be recorded and for the audio data to be analysed.

Image recognition

Cameras may be used to detect humans sitting in the library. As mentioned in our Research Analysis, we initially explored
this solution, but we quickly dropped it due to the difficulty of using image processing to accurately detect the exact
number of library users. Achieving a high degree of accuracy would be difficult, unless many cameras are installed around
the library. The processing power needed to perform the image recognition would have required our solution to send the
images to the cloud to be analysed, requiring a considerable amount of processing power and increasing server costs.
This would bring even more privacy issues that would have required written consent from the users. It would have also
been, in our opinion, too obtrusive due to the public getting more worried regarding data privacy.

GridEye above seats

An infrared array sensor extends the capabilities of single element thermopile sensors and pyroelectric detectors, by being
able to detect multiple people and identify their positions. The first ever example of such an infrared sensor array
and the core device of the system for estimating library occupancy is the Grid-EYE, a 64-pixel IR camera in an all-in-one
compact SMD package. The 64 thermopile elements are in an 8x8 grid format that detect actual temperature as well as temperature
gradients within a 60° viewing area, without any contact and providing thermal images as a result. Thus, the Grid-EYE
sensor defeats the traditional limitations of passive IR technology, enabling various advanced solutions. Furthermore,
it is the preferred technology choice in people detection applications, as there are no privacy issues as with traditional
video cameras, which do not preserve the privacy of the subjects. It is also a cost-effective and compact solution.

The GridEye sensor can be used to detect the presence of humans. When mounted on a ceiling, it can cover an area of around
2.7 m², making it suitable to detect whether 4 seats around a table are occupied or not. During initial testing
with the GridEye sensor, we found that it was able to accurately detect temperature differences over a small area. We
were confident that we would be able to write an algorithm that would be able to process the GridEye
sensor information into a 2x2 matrix of occupied/vacant boolean values for seats.

GridEye sensor at doorways

Rather than mounting a GridEye sensor on the ceiling for every four seats, we considered installing a GridEye simply at every
doorway. The GridEye would then be able to detect whether a person is leaving or entering the floor, such that it could
the keep an internal counter of how many seats are occupied on the floor. This dramatically reduces the cost, as a floor
with 100 seats and 2 exits would only require 2 grideye units, rather than 25 units, each covering 4 seats.

However, we were skeptical that this method would have a high enough accuracy, for example when a group of people exit and
enter through the doorway simultaneously, resulting in the counter becoming untrue and needing to be reset.

Furthermore, we would lose the granularity of knowing where exactly students are sitting on the floor. Heating and lighting
control would have to control the entire floor, rather than specific areas within the library.

Smart heating and lighting system

When faced with the idea of automating energy efficiency, we realised that we can use the seat occupancy to make smart decisions about unoccupied areas. As such, we set the target temperature low, to avoid turning on the heating system, and the lights turned off. If the student chooses to sit in an unoccupied area, where the lights are initially dimmed and the heating system set at a low temperature, then the environment gradually becomes optimal: the lights will be automatically switched on if there is not enough natural light and the heating system adjusted to a comfortable temperature.

Final solution

Ultimately, we decided to place GridEye sensors on the ceiling for every 4 seats, even though it is more expensive than placing them only at doorways, as it’s able to meet all our design criteria. Students would be able to know which seats exactly are available via our web app, due to the relatively high sensor density. The high granularity of seat tracking also enables us to implement the smart heating and lighting control system, improving the level of energy effiency. Other solutions like OccupEye only offer seat tracking, even though saving energy is a growing concern to the environment that can be helped through automation.

Furthermore, our solution takes great advantage of design principles to provide a good User Experience with an almost identical cross-platform experience and with smart suggestions regarding occupancy.

System Design Diagram

Below is a diagram of our final system.

1. GridEye Sensor and Light Sensor attached to Arduino board measure environment
2. Arduino processes sensor data locally
3. Processed data is sent via WiFi to cloud server via a RESTful API
4. Backend hosted on cloud server updates database with new sensor readings, e.g. temperature and seat occupancy
5. Changes in library environment trigger backend to send action commands to heating and lighting control systems
6. Students accessing frontend have the client request up-to-date live data via the RESTful API

We developed a prototype sensor unit that would have a GridEye, temperature and light sensor on it, along with WiFi capability.

GridEye Algorithm

The manual for the GridEye states that the optimum range for operation is up to 2 m. The ceiling of the science library is 3 m.

The human detection algorithm is comprised of two main parts. The setup and the main loop. Both components revolve around the GridEye sensor performing and returning successive readings of the temperature in the designated area. Before going into more details about the algorithm, we present a succinct explanation of how the Grideye sensor operates:

The Grideye sensor is a 64 pixel IR camera. This IR camera has 64 thermopile elements in an 8×8 grid format that detect absolute surface temperature without any contact. As such, in terms of code implementation, at every reading, the GridEye sensor returns a 64 floating point number array containing 64 temperature values in double format.

Since the measured area is an 8x8 grid corresponding to a 2.7 m² square area, we decided that the most optimal way in which we can use the GridEye is to position it on the ceiling such that it monitors 4 seats (arranged symmetrical) at the same time. In other words, we split the 64 doubles array in 4 x 16 doubles arrays, each corresponding to a single seat. Even though the precision of the sensor drops considerably by doing so, it is still big enough to perform a reliable human detection, not to mention that we increase our monitoring capacity four-fold.

The following visual presents an easy-to-understand but comprehensive reading result:

Going back to the detection algorithm, we previously mentioned that it contains 2 parts: the setup and the main loop. We will now explain in-depth how both of these parts function.

The setup has the role of performing a temperature calibration of the measured area as well as setting 4 thresholds (one for each seat) which will be used in the main loop. During the setup, the Grideye sensor will perform 128 readings, each reading being followed by an "average temperature storing operation". The average temperature storing operation is defined as follows: after receiving the 64 doubles array from the GridEye sensor, we split it into four quadrants as previously mentioned. Then, we compute the average temperature for each quadrant (using a simple mean average for each of the four 16 doubles arrays) and store it in another array (one for each quadrant) at index n. We then increment and carry on with the next Grideye reading. By doing so 128 times we will have four arrays, one for each quadrant,  which contain 128 average temperature values. For each quadrant, we set the threshold to be the maximum value stored in their respective array. This marks the end of the setup stage. (Note: the setup must be performed with no person located in the monitored area!)

Following the setup, the main loop comprises of successive GridEye readings which are yet again segmented in four quadrants. For each quadrant we perform the following operations:

• We compute the average temperature and store it in a variable "av_t".
• We compare av_t against the threshold of the quadrant. If av_t is bigger, a counter gets incremented. If av_t is smaller, the same counter gets decremented. There is one counter for each quadrant for a total of 4 counter for each GridEye device.
• Once the counter reaches a certain maximum cap, we can conclude that a person has occupied the seat in that quadrant.

To detect that a person has left a seat, we perform the exact same operations but reversely. Should the av_t variable be smaller than the threshold, we decrease the counter up until the minimum cap is reached, at which point we can conclude that the person has indeed left the seat.

This algorithm relies on the fact that a person will raise the average temperature in a quadrant above the maximum recorded one. Testing has shown that this is indeed the case. However, the increase in average temperature is most times minimal, ranging from +1 degree Celsius to +2 degrees Celsius above the threshold. This testing was performed in a room with ambient temperature below 25 degrees Celsius. During hot summers, the readings might change and lead to false positives. Luckily, more modern buildings will have a HVAC system, keeping the room temperature around 20-23°C.

In terms of privacy concerns or issues, there as none since no personal information about individuals is stored or used by our system. All of the operations involve only temperature readings which pose no threat to the privacy of the user.

We tested our system rigorously to ensure that it was meeting all of our requirements.

Testing Methodology

Testing Environment

We tested our prototype in an environment that tried to simulate the conditions of a library. We carried out these tests in the Engineering Hub.

A single sensor unit is capable of tracking 4 seats around a table. This is due to the sensing area of the GridEye sensor. Only a table with 4 seats needed to be setup in order to test a single sensor unit in isolation.

Furthermore, we created a web dashboard that simulates a virtual thermostat and light that can be controlled by the system, as we did not have enough resources to create real-life prototype boards for lighting and heating systems as well. We can then easily debug the temperature that is set by the system, as well as the lighting control system. At the same time, we had the web app open, to see if the web app itself is functioning correctly, such as displaying the correct number of free seats.

Seating Configuration Testing

We tried testing various different scenarios, to see if the sensor could detect all possible seat occupancy combinations:

• All seats are occupied
• 1 seat is occupied
• 2 seats directly next to each other are occupied
• 2 seats diagonal to each other are occupied
• 3 seats are occupied
• none of the seats are occupied.
• White paper spread across the table, white paper being more reflective than the table
• Desktop located on table emitting waste heat
• Allow students to easily find a free seat with their smartphone
• Scalable sensor design, theoretically allowing all 3,000 study spaces at UCL to be tracked
• Intelligently control HVAC and lighting to save energy
• Respect privacy of student's visiting the library
• Allow monitoring of library usage for UCL management
• Provide a seamless cross-platform experience with smart suggestions
• Our tests were only carried out with a virtual HVAC and lighting system, that simulates the light's ON/OFF state and the temperature set by the system. In a real-life system, adding internet capability to the HVAC system and lighting system could be of significant cost, due to the old age of the buildings at UCL.

Future Improvements

• A booking system can be implemented so that the users could reserve their seats via the app, to avoid many students running to an empty seat, only to discover that it has already been taken.
• An automatic empty seat finder can also be implemented so the users do not have to check the availability of each library one by one. In addition to that, the finder can suggest to the users the specific seat in the study space (based on user location) that will help maximise the energy efficiency of the building.

Impact of the Solution

• StudyHunt provides UCL with highly accurate analytics and reports, indicating when the library space is utilised most frequently. Additionally, it allows identification of the most popular areas utilised by students and thus enables the management team to make strategic estate decisions (e.g. library layout modification and expansion).
• UCL can achieve greener building energy usage from the smart lighting and heating system integrated with our StudyHunt solution.
• Users would find it valuable and useful to be able to look at a screen when they walk into the library and know immediately where available study space is available, enhancing the learning experience.