element14 Community
element14 Community
    Register Log In
  • Site
  • Search
  • Log In Register
  • Community Hub
    Community Hub
    • What's New on element14
    • Feedback and Support
    • Benefits of Membership
    • Personal Blogs
    • Members Area
    • Achievement Levels
  • Learn
    Learn
    • Ask an Expert
    • eBooks
    • element14 presents
    • Learning Center
    • Tech Spotlight
    • STEM Academy
    • Webinars, Training and Events
    • Learning Groups
  • Technologies
    Technologies
    • 3D Printing
    • FPGA
    • Industrial Automation
    • Internet of Things
    • Power & Energy
    • Sensors
    • Technology Groups
  • Challenges & Projects
    Challenges & Projects
    • Design Challenges
    • element14 presents Projects
    • Project14
    • Arduino Projects
    • Raspberry Pi Projects
    • Project Groups
  • Products
    Products
    • Arduino
    • Avnet Boards Community
    • Dev Tools
    • Manufacturers
    • Multicomp Pro
    • Product Groups
    • Raspberry Pi
    • RoadTests & Reviews
  • Store
    Store
    • Visit Your Store
    • Choose another store...
      • Europe
      •  Austria (German)
      •  Belgium (Dutch, French)
      •  Bulgaria (Bulgarian)
      •  Czech Republic (Czech)
      •  Denmark (Danish)
      •  Estonia (Estonian)
      •  Finland (Finnish)
      •  France (French)
      •  Germany (German)
      •  Hungary (Hungarian)
      •  Ireland
      •  Israel
      •  Italy (Italian)
      •  Latvia (Latvian)
      •  
      •  Lithuania (Lithuanian)
      •  Netherlands (Dutch)
      •  Norway (Norwegian)
      •  Poland (Polish)
      •  Portugal (Portuguese)
      •  Romania (Romanian)
      •  Russia (Russian)
      •  Slovakia (Slovak)
      •  Slovenia (Slovenian)
      •  Spain (Spanish)
      •  Sweden (Swedish)
      •  Switzerland(German, French)
      •  Turkey (Turkish)
      •  United Kingdom
      • Asia Pacific
      •  Australia
      •  China
      •  Hong Kong
      •  India
      •  Korea (Korean)
      •  Malaysia
      •  New Zealand
      •  Philippines
      •  Singapore
      •  Taiwan
      •  Thailand (Thai)
      • Americas
      •  Brazil (Portuguese)
      •  Canada
      •  Mexico (Spanish)
      •  United States
      Can't find the country/region you're looking for? Visit our export site or find a local distributor.
  • Translate
  • Profile
  • Settings
Experimenting with Sensor Fusion
  • Challenges & Projects
  • Design Challenges
  • Experimenting with Sensor Fusion
  • More
  • Cancel
Experimenting with Sensor Fusion
Blog Sensor Fusion for Firefighters. Introductory blog.
  • Blog
  • Forum
  • Documents
  • Files
  • Members
  • Mentions
  • Sub-Groups
  • Tags
  • More
  • Cancel
  • New
Join Experimenting with Sensor Fusion to participate - click to join for free!
  • Share
  • More
  • Cancel
Group Actions
  • Group RSS
  • More
  • Cancel
Engagement
  • Author Author: javagoza
  • Date Created: 26 Oct 2022 4:12 PM Date Created
  • Views 2527 views
  • Likes 13 likes
  • Comments 10 comments
  • experimenting with sensor fusion
  • amd-xilinx
  • sp701
  • fpga_projects
  • xilinx
  • fpga
  • vivado
  • amd
  • spartan-7
  • sensor fusion
Related
Recommended

Sensor Fusion for Firefighters. Introductory blog.

javagoza
javagoza
26 Oct 2022
Sensor Fusion for Firefighters. Introductory blog.

This is the introductory blog of my participation in the Experimenting with Sensor Fusion Challenge. First of all, I would like to thank the sponsors: Element14 and AMD Xilinx, for choosing my application to participate in this challenge. In this blog I will introduce myself and share my motivations to participate in this Design Challenge and the experiments I want to carry out.


Table of Contents

  • About me
  • Me and the Spartan-7, motivation with the Sensor Fusion Design Challenge
  • Choosing a project to experiment
  • Alert monitor for firefighters
  • Features
    • Sensor Fusion
    • Layered construction of the alert monitor image
    • Conceptual diagram of the construction of the image
  • The hardware
  • Planning for the experiments
  • Related projects and blogs I've done
  • The Complete "Sensor Fusion for Firefighters" Blog Series

The Complete "Sensor Fusion for Firefighters" Blog Series

  • Sensor Fusion for Firefighters. Introductory blog
  • Sensor Fusion for Firefighters. Getting Started with the AMD Xilinx SP701
  • Sensor Fusion for Firefighters. AMD Xilinx SP701 - MIPI Video Pipe Camera to HDMI Display
  • Sensor Fusion for Firefighters. Displaying heads-up video on the live feed
  • Sensor Fusion for Firefighters. Environmental monitor heads-up display on Xilinx Spartan-7 SP701 development board
  • Sensor Fusion for Firefighters. Compass and Environmental HUD monitor with the Spartan-7
  • Sensor Fusion for Firefighters. Thermal Vision, Compass and Environmental HUD monitor with the Spartan-7
  • Sensor Fusion for Firefighters. Summary Blog

About me

I am a Spanish self-taught programmer. I studied Industrial Electrical Engineering specialized in Robotics, Electronics and Automatic Control in the 80's, but I have never worked as an electrical engineer and developed my career focused on high level computer programming. I currently work as a back-end software developer for payment solutions in the payment card industry, specialized in PCI and EMV compliance.


Me and the Spartan-7, motivation with the Sensor Fusion Design Challenge

The challenge of a Sensor Fusion design using FPGAs is something I'm curious about. I have suffered the performance problem of having multiple sensors connected to a classic microcontroller something that you can only solve by looking for a faster microcontroller or with more microcontrollers.

Since June 2022 I have written 21 project blogs using the Spartan-7 FPGA on the element14 community website. It has been a learning from zero knowledge about FPGAs. I still have a lot to learn, and I hope to complete part of my training with this challenge.

I love the format of elemet14's sponsored experimenting with a component challenge and have participated in a few of them before:

  • Design For A Cause 2021,
  • experimenting with Magnetic Components,
  • and the 7 Ways to Leave Your Spartan-6 FPGA program

I want to experience how an FPGA like the Spartan-7 can free the microcontroller from such costly tasks as image processing using hardware accelerators.


Choosing a project to experiment

I have had several ideas to focus my experiments on solving a problem. Some of the ideas I've worked on:

  • Water leak detector in domestic pipes by detection of changes in the image of mechanical meters and classification of pipe vibration patterns.
  • Automatic meter reading of mechanical meters through OCR when detection of changes in the image of mechanical meters.
  • Unattended cooking monitor. Detection PIR + image change detection + thermal array.
  • Autonomous driving following lines using 9-degrees IMU + video input + ToF sensor.

Working on these ideas, I tried to experiment with the design of a convolutional neural network (CNN) in hardware with the FPGA Spartan-7 of the SP701 development board, although I managed to get Vivado to synthesize the circuit, the alerts about the junction temperature that was going to be reached and the use of 98% of all the DSPs have made me give up those projects.

I finally decided on another idea: A portable alert monitor for firefighters.

Why? Because being a sensor fusion project, this will combine up to 11 different sensors and will allow me to explore both the capture of images, their processing and the visualization of the processed images.


Alert monitor for firefighters

image

My intention is to experiment with the different functionalities of the AMD Xilinx SP701 development board and the AMD Xilinx Vivado development environment with the aim of gradually building a firefighter alert monitor that uses hardware accelerators with the Spartan7 FPGA.

By using a 9-degree inertial module, the system will be able to detect the orientation and position of the firefighter's head and create and display real-time thermal imaging and visual panoramas of the environment along with information from other sensors. The information will be displayed on a portable HDMI display.

The system combines a thermal array sensor, real video image, a magnetometer compass, and the IMU to detect movement and orientation of the head and a ToF sensor to estimate the distance to nearby objects.


Features

Head Controlled Panoramic Thermal Imaging Scanner

The monitor will display:

  • Real-time video output: Real-time image captured by the camera and processed using hardware accelerators.
  • Image overlaid with the values of:
    • the closest distance detected at the moment by the time-of-flight sensor.
    • the last temperature value detected by the temperature sensor in ºC
    • the last % relative humidity value detected by the humidity sensor in % rel humidity
    • the last VOC value detected by the gas sensor in ppb/ppm
  • Image superimposed with 360º thermal image panorama recorded while the firefighter turns 360º on himself/herself and corrected with the values provided by the IMU.
  • or alternatively a 360º panorama built from the images obtained as the firefighter performs a 360º turn on himself/herself
  • a graphic representation of linear compass controlled by the values obtained from the magnetometer in real time.
  • labels, frames and transparent backgrounds for a better visualization of the different informative areas of the alert monitor.

The panoramas rotate to always show the front image in the center with the help of the magnetometer used as a compass.

Alternatively, a 360º panorama built from the images obtained as the firefighter performs a 360º turn on himself/herself

Head Controlled Panoramic Thermal Imaging Scanner

Sensor Fusion

The alert monitor fuses or combines information from the following sensors:

  • Color camera
  • Thermal array sensor
  • 3D Gyroscope sensor
  • 3D Accelerometer sensor
  • Barometer sensor.
  • 3D Magnetometer sensor
  • Time Of Flight sensor
  • Temperature sensor
  • Humidity sensor
  • Air quality sensor
  • Slide switches

See hardware section


Layered construction of the alert monitor image

The image that the firefighter will see on the HDMI display will be built by mixing several layers that will take advantage of hardware acceleration.

Head Controlled Panoramic Thermal Imaging Scanner Layers


Conceptual diagram of the construction of the image

Using a Microblaze soft-processor, in a bare-metal configuration, to orchestrate communication with the sensors, the sensor data will be pre-processed and stored in memory.

In parallel, a hardware video core will take care of image processing and mixing of the different layers and will generate the synchronized image for the hdmi monitor.

image


The hardware

Pmod HYGRO: Temperature sensor, Humidity sensor Digilent Pmod HYGRO
Pomd ToF, Time Of Flight sensor Digilent Pmod ToF
Pmod CMPS2, Magnetometer Digilent Pmod CMPS2
Pmod NAV, 9-AXIS IMU PLUS BAROMETER Digilent Pmod NAV
SP701 EVALUATION BOARD, SPARTAN-7 FPGA AMDP XILINX SP701
Digilent PCAM 5C Digilent PCAM 5C
CCS811 Air Quality Sensor Digilent Pmod AQS
Melexis MLX90640 IR Array 110 Degree FOV, MLX90640

Planning for the experiments

I want to build the Alert Monitor in different phases through small projects that allow me to learn little by little. This is the initial plan although the order may vary depending on the problems that appear.

  • Reviewing MIPI design example, Demosaic (Debayer), Color Correctio and Gamma LUT
  • Experiment with the CSI-2 Subsystem IP with PCAM camera module
  • Experiment with the DSI Subsystem IP with HDMI monitors
  • Experiment with Frame Buffer IP, Test Pattern Generator IP and Video Scaler
  • Experiment with Microblaze control of CSI and DSI
  • Experiment with Video Mixer IP, VIdeo Timing Controller, RGB to DVI Video Encoder
  • Creating image overlays from AXI Streams
  • Building a virtual seven digits display overlay to display meter readings
  • Building a frame overlay with semi-transparent regions for the meters display
  • Reading and displaying values from the Diligent pmod Hygro
  • Reading and displaying values from the Diligent pmod NAV
  • Reading and displaying values from the Diligent pmod ToF
  • Reading and displaying values from the CCS811 Air Quality Sensor
  • Building a thermal image overlay with the Melexis MLX90640 IR Array sensor.
  • Creating a compass linear face overlay.
  • Barrel shifting the compass face overlay with the orientation from the magnetometer of the pmodNav IMU
  • Creating a thermal image panorama scan using he Melexis MLX90640 IR Array sensor and the IMU
  • Creating a visual image panorama scan using he Pcam sensor and the IMU
  • Barrel shifting the panoramas with the orientation from the magnetometer of the pmodNav IMU.
  • Putting all together: Alert monitor for firefighters

Related projects and blogs I've done

  • Melexis MLX90640 IR Array: Arty S7 50 VGA Thermal Imaging Camera ,
  • MIPI Camera: Kria KV260 - 01- Starting with the Smart Camera Firmware
  • Pmod HYGRO: Arty S7 50 Rapid Prototyping - Environmental Monitor
  • CCS811 Air Quality Sensor: Environmental Monitor on Arduino Nano 33 IoT, Environmental Monitor Revisited
  • Pmod ToF ISL29501: Arty S7 50 ArtyBot ToF Sensor for Obstacle Avoidance
  • TFT LCD Virtual Clock Display Library
  • Pmod CMPS2: Rube Goldberd Device The Quantenkompassinator

I hope that the project and the next blogs will be interesting for you. The fun begins!

The Complete "Sensor Fusion for Firefighters" Blog Series

  • Sensor Fusion for Firefighters. Introductory blog
  • Sensor Fusion for Firefighters. Getting Started with the AMD Xilinx SP701
  • Sensor Fusion for Firefighters. AMD Xilinx SP701 - MIPI Video Pipe Camera to HDMI Display
  • Sensor Fusion for Firefighters. Displaying heads-up video on the live feed
  • Sensor Fusion for Firefighters. Environmental monitor heads-up display on Xilinx Spartan-7 SP701 development board
  • Sensor Fusion for Firefighters. Compass and Environmental HUD monitor with the Spartan-7
  • Sensor Fusion for Firefighters. Thermal Vision, Compass and Environmental HUD monitor with the Spartan-7
  • Sensor Fusion for Firefighters. Summary Blog
  • Sign in to reply
  • DAB
    DAB over 2 years ago in reply to javagoza

    I always try to stretch ideas.

    Having worked with maps, navigation and other technologies, it is easy for me to see where additional capability can be added to fun projects.

    • Cancel
    • Vote Up 0 Vote Down
    • Sign in to reply
    • More
    • Cancel
  • javagoza
    javagoza over 2 years ago in reply to DAB

    DAB, thanks. You're a naughty little devil, I don't know if it's said that way with affection.

    That map was in my first drawings inspired by the video games of my teenage children with whom I have to share a table for the projects.

    It would be a dream but right now I don't have enough knowledge to tackle that problem. But you just piqued my curiosity and if things go well with the rest of the project it would be fun to try.

    I envision a start button to have the first reference in a predetermined place on the map and a selection of places for the end of the route.

    • Cancel
    • Vote Up 0 Vote Down
    • Sign in to reply
    • More
    • Cancel
  • Sudeep AJ
    Sudeep AJ over 2 years ago

    Brilliant idea! I am impressed by how you broadened the domain by including more than just Pmod and Pcam sensors in the project.

    • Cancel
    • Vote Up 0 Vote Down
    • Sign in to reply
    • More
    • Cancel
  • DAB
    DAB over 2 years ago

    Very interesting project.

    Have you considered uploading a building plan into the software?

    That way you could put the firefighter's current position over the plan to show them where they are in relation to where they might want to go or find the closest way out.

    • Cancel
    • Vote Up 0 Vote Down
    • Sign in to reply
    • More
    • Cancel
  • javagoza
    javagoza over 2 years ago in reply to scottiebabe

    Fire engine Thanks Fire engine! 

    • Cancel
    • Vote Up 0 Vote Down
    • Sign in to reply
    • More
    • Cancel
>
element14 Community

element14 is the first online community specifically for engineers. Connect with your peers and get expert answers to your questions.

  • Members
  • Learn
  • Technologies
  • Challenges & Projects
  • Products
  • Store
  • About Us
  • Feedback & Support
  • FAQs
  • Terms of Use
  • Privacy Policy
  • Legal and Copyright Notices
  • Sitemap
  • Cookies

An Avnet Company © 2025 Premier Farnell Limited. All Rights Reserved.

Premier Farnell Ltd, registered in England and Wales (no 00876412), registered office: Farnell House, Forge Lane, Leeds LS12 2NE.

ICP 备案号 10220084.

Follow element14

  • X
  • Facebook
  • linkedin
  • YouTube