Group Actions
Engagement
  • Author Author: javagoza
  • Date Created: Date Created
  • Views 2321 views
  • Likes 13 likes
  • Comments 10 comments
Related
Recommended

Sensor Fusion for Firefighters. Introductory blog.

javagoza
Sensor Fusion for Firefighters. Introductory blog.

This is the introductory blog of my participation in the Experimenting with Sensor Fusion Challenge. First of all, I would like to thank the sponsors: Element14 and AMD Xilinx, for choosing my application to participate in this challenge. In this blog I will introduce myself and share my motivations to participate in this Design Challenge and the experiments I want to carry out.

Table of Contents

The Complete "Sensor Fusion for Firefighters" Blog Series

About me

I am a Spanish self-taught programmer. I studied Industrial Electrical Engineering specialized in Robotics, Electronics and Automatic Control in the 80's, but I have never worked as an electrical engineer and developed my career focused on high level computer programming. I currently work as a back-end software developer for payment solutions in the payment card industry, specialized in PCI and EMV compliance.

Me and the Spartan-7, motivation with the Sensor Fusion Design Challenge

The challenge of a Sensor Fusion design using FPGAs is something I'm curious about. I have suffered the performance problem of having multiple sensors connected to a classic microcontroller something that you can only solve by looking for a faster microcontroller or with more microcontrollers.

Since June 2022 I have written 21 project blogs using the Spartan-7 FPGA on the element14 community website. It has been a learning from zero knowledge about FPGAs. I still have a lot to learn, and I hope to complete part of my training with this challenge.

I love the format of elemet14's sponsored experimenting with a component challenge and have participated in a few of them before:

I want to experience how an FPGA like the Spartan-7 can free the microcontroller from such costly tasks as image processing using hardware accelerators.

Choosing a project to experiment

I have had several ideas to focus my experiments on solving a problem. Some of the ideas I've worked on:

  • Water leak detector in domestic pipes by detection of changes in the image of mechanical meters and classification of pipe vibration patterns.
  • Automatic meter reading of mechanical meters through OCR when detection of changes in the image of mechanical meters.
  • Unattended cooking monitor. Detection PIR + image change detection + thermal array.
  • Autonomous driving following lines using 9-degrees IMU + video input + ToF sensor.

Working on these ideas, I tried to experiment with the design of a convolutional neural network (CNN) in hardware with the FPGA Spartan-7 of the SP701 development board, although I managed to get Vivado to synthesize the circuit, the alerts about the junction temperature that was going to be reached and the use of 98% of all the DSPs have made me give up those projects.

I finally decided on another idea: A portable alert monitor for firefighters.

Why? Because being a sensor fusion project, this will combine up to 11 different sensors and will allow me to explore both the capture of images, their processing and the visualization of the processed images.

Alert monitor for firefighters

image

My intention is to experiment with the different functionalities of the AMD Xilinx SP701 development board and the AMD Xilinx Vivado development environment with the aim of gradually building a firefighter alert monitor that uses hardware accelerators with the Spartan7 FPGA.

By using a 9-degree inertial module, the system will be able to detect the orientation and position of the firefighter's head and create and display real-time thermal imaging and visual panoramas of the environment along with information from other sensors. The information will be displayed on a portable HDMI display.

The system combines a thermal array sensor, real video image, a magnetometer compass, and the IMU to detect movement and orientation of the head and a ToF sensor to estimate the distance to nearby objects.

Features

Head Controlled Panoramic Thermal Imaging Scanner

The monitor will display:

  • Real-time video output: Real-time image captured by the camera and processed using hardware accelerators.
  • Image overlaid with the values of:
    • the closest distance detected at the moment by the time-of-flight sensor.
    • the last temperature value detected by the temperature sensor in ºC
    • the last % relative humidity value detected by the humidity sensor in % rel humidity
    • the last VOC value detected by the gas sensor in ppb/ppm
  • Image superimposed with 360º thermal image panorama recorded while the firefighter turns 360º on himself/herself and corrected with the values provided by the IMU.
  • or alternatively a 360º panorama built from the images obtained as the firefighter performs a 360º turn on himself/herself
  • a graphic representation of linear compass controlled by the values obtained from the magnetometer in real time.
  • labels, frames and transparent backgrounds for a better visualization of the different informative areas of the alert monitor.

The panoramas rotate to always show the front image in the center with the help of the magnetometer used as a compass.

Alternatively, a 360º panorama built from the images obtained as the firefighter performs a 360º turn on himself/herself

Head Controlled Panoramic Thermal Imaging Scanner

Sensor Fusion

The alert monitor fuses or combines information from the following sensors:

  • Color camera
  • Thermal array sensor
  • 3D Gyroscope sensor
  • 3D Accelerometer sensor
  • Barometer sensor.
  • 3D Magnetometer sensor
  • Time Of Flight sensor
  • Temperature sensor
  • Humidity sensor
  • Air quality sensor
  • Slide switches

See hardware section

Layered construction of the alert monitor image

The image that the firefighter will see on the HDMI display will be built by mixing several layers that will take advantage of hardware acceleration.

Head Controlled Panoramic Thermal Imaging Scanner Layers

Conceptual diagram of the construction of the image

Using a Microblaze soft-processor, in a bare-metal configuration, to orchestrate communication with the sensors, the sensor data will be pre-processed and stored in memory.

In parallel, a hardware video core will take care of image processing and mixing of the different layers and will generate the synchronized image for the hdmi monitor.

image

The hardware

Pmod HYGRO: Temperature sensor, Humidity sensor Digilent Pmod HYGRO
Pomd ToF, Time Of Flight sensor Digilent Pmod ToF
Pmod CMPS2, Magnetometer Digilent Pmod CMPS2
Pmod NAV, 9-AXIS IMU PLUS BAROMETER Digilent Pmod NAV
SP701 EVALUATION BOARD, SPARTAN-7 FPGA AMDP XILINX SP701
Digilent PCAM 5C Digilent PCAM 5C
CCS811 Air Quality Sensor Digilent Pmod AQS
Melexis MLX90640 IR Array 110 Degree FOV, MLX90640

Planning for the experiments

I want to build the Alert Monitor in different phases through small projects that allow me to learn little by little. This is the initial plan although the order may vary depending on the problems that appear.

  • Reviewing MIPI design example, Demosaic (Debayer), Color Correctio and Gamma LUT
  • Experiment with the CSI-2 Subsystem IP with PCAM camera module
  • Experiment with the DSI Subsystem IP with HDMI monitors
  • Experiment with Frame Buffer IP, Test Pattern Generator IP and Video Scaler
  • Experiment with Microblaze control of CSI and DSI
  • Experiment with Video Mixer IP, VIdeo Timing Controller, RGB to DVI Video Encoder
  • Creating image overlays from AXI Streams
  • Building a virtual seven digits display overlay to display meter readings
  • Building a frame overlay with semi-transparent regions for the meters display
  • Reading and displaying values from the Diligent pmod Hygro
  • Reading and displaying values from the Diligent pmod NAV
  • Reading and displaying values from the Diligent pmod ToF
  • Reading and displaying values from the CCS811 Air Quality Sensor
  • Building a thermal image overlay with the Melexis MLX90640 IR Array sensor.
  • Creating a compass linear face overlay.
  • Barrel shifting the compass face overlay with the orientation from the magnetometer of the pmodNav IMU
  • Creating a thermal image panorama scan using he Melexis MLX90640 IR Array sensor and the IMU
  • Creating a visual image panorama scan using he Pcam sensor and the IMU
  • Barrel shifting the panoramas with the orientation from the magnetometer of the pmodNav IMU.
  • Putting all together: Alert monitor for firefighters

Related projects and blogs I've done

I hope that the project and the next blogs will be interesting for you. The fun begins!

The Complete "Sensor Fusion for Firefighters" Blog Series