Today there is only a small update: Last Sunday I posted a video with temperature data overlay but it had a little problem because the camera data was a little bit delayed.
Now I fixed this.
I used threads and created an extra thread which only reads the camera input. I used the standard thread library of C++ 11 (std::thread). To compile the program you have to enable C++ 11 with the compiler option "-std=c++11" and add -pthread to the libraries in the Makefile.
The usage of threads is quite simple because in this example you don't have to synchronize anything. The main thread starts a second thread which reads the camera input in an endless loop and saves the data to an OpenCV mat. The main thread reads this mat, copies it and does image processing with the copied data. So main thread and camera thread don't write on the same data and since there is no common write access it does not have to be synchronize. The worst issue with this approach could be that main thread is reading the data of the mat while the camera thread is writing new image data to the map. This could lead to glitches in the image data. IMHO this is negligible.
So here is the whole source code of my program. I also included the I2C improvements of my last post.
#include <stdio.h>
#include <stdlib.h>
#include <linux/i2c-dev.h>
#include <fcntl.h>
#include <unistd.h>
#include <opencv2/opencv.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/contrib/contrib.hpp>
#include <thread>
// global variables for exchange between threads
cv::VideoCapture cap; // create camera input
cv::Mat cameraImage; // create opencv mat for camera
void cameraThread(void) // function for the camera thread
{
while(1) // loop forever
{
cap >> cameraImage; // copy camera input to opencv mat
}
}
int main(void)
{
int file;
int addr=0x68; // adress of AMG88xx
int x,y; // variables to got through the array
signed short int internal_temp;
signed short int pixel_temp[64]; // array for pixel temperatures
int end=1; // variable to end program
printf("Pi Chef Stove Assistant Demo with AMG88xx\n"); // print start message
if((file=open("/dev/i2c-1",O_RDWR))<0) // open i2c-bus
{
perror("cannot open i2c-1");
exit(1);
}
if(ioctl(file, I2C_SLAVE, addr)<0) // open slave
{
perror("cannot open slave address");
exit(1);
}
internal_temp=(signed short)(i2c_smbus_read_word_data(file, 0x0e)<<4); // read internal temperature from sensor
internal_temp=internal_temp/16; // recalculate correct value
printf("Internal Temp: %f C (0x%04X = %i)\n",(float)internal_temp*0.0625,internal_temp,internal_temp); // print internal temperature
cv::Mat cameraImageGray; // create opencv mat for grayscale camera image
cv::Mat cameraImageBig(320,320,CV_8UC3); // create opencv mat for camera with final resolution
cap.open(0); // open camera
cap.set(CV_CAP_PROP_FRAME_WIDTH, 320); // change camera width to 320 - we do not need more
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 240); // change camera height to 240
cv::VideoWriter outputVideo; // create video output
outputVideo.open("output.avi", CV_FOURCC('M','J','P','G'), 15, cv::Size(320,320), true); // set video output to 15 fps and MJPG
if (!outputVideo.isOpened()) // check if generation of video output was successful
{
perror("Could not open the output video for write\n");
exit(1);
}
cv::Mat outSmall(8,8,CV_8UC1); // create opencv mat for sensor data
cv::Mat outSmallnorm(8,8,CV_8UC1); // create opencv mat for normalized data
cv::Mat outColor; // create opencv mat for color output
cv::Mat combined; // create opencv mat for combined output
cap >> cameraImage; // copy camera input to opencv mat to get data to startup
std::thread tcam(cameraThread); // start extra thread to get camera input
while(end==1) // check end variable
{
cv::TickMeter t;
t.start(); // start timer
x=i2c_smbus_read_i2c_block_data(file,0x80,32,(__u8*)pixel_temp)// read first 32 byte / 16 temperature pixels from sensor
x=i2c_smbus_read_i2c_block_data(file,0xa0,32,(__u8*)pixel_temp+32); // read next 32 byte / 16 temperature pixels from sensor
x=i2c_smbus_read_i2c_block_data(file,0xc0,32,(__u8*)pixel_temp+64); // read next 32 byte / 16 temperature pixels from sensor
x=i2c_smbus_read_i2c_block_data(file,0xe0,32,(__u8*)pixel_temp+96); // read last 32 byte / 16 temperature pixels from sensor
for(x=0;x<64;x++)
{
pixel_temp[x]=(signed short)(pixel_temp[x]<<4)/16; // set pixel_temp to original value
}
for(x=0;x<8;x++)
{
for(y=0;y<8;y++)
{
outSmall.at(7-x,7-y)=pixel_temp[x*8+y]; // save data to opencv mat and rotate it
}
}
cv::normalize(outSmall,outSmallnorm,255,0,cv::NORM_MINMAX); // normalize Mat to values between 0 and 255
cv::resize(outSmallnorm,outSmallnorm,cv::Size(320,320)); // resize Mat to 320 x 320 pixel
cv::applyColorMap(outSmallnorm,outColor,cv::COLORMAP_JET); // generate colored output with colormap
cv::cvtColor(cameraImage,cameraImageGray,CV_RGB2GRAY); // convert camera image to grayscale
cv::cvtColor(cameraImageGray,cameraImageGray,CV_GRAY2RGB); // make cameraImage 3 channels again
cameraImageGray.copyTo(cameraImageBig(cv::Rect(0,40,320,240)));// copy camera ingae to mat with same resolution as temperature mat
cv::addWeighted(cameraImageBig,0.5,outColor,0.5,0.0,combined); // combine camera mat and temperature mat into one single image
cv::imshow("combined",combined); // display mat on screen
outputVideo << combined; // add frame to video
t.stop(); // stop timer
printf("Time: %f ms\n", (double)t.getTimeMilli() / t.getCounter()); // print result of timer
char key = cv::waitKey(1); // check keys for input
if(key=='e') end=0; // end if e was pressed
}
printf("ended regularly!\n"); // print end message
close(file);
return 0;
}
This program also measures the execution time of the main thread. The execution time of the main thread is about 60 ms. The refresh rate of the raspberry pi camera is 30 fps, so about 33 ms for each frame, which is way faster than the main thread. This explains why in a single thread application the camera buffer fills up and adds a delay to the camera input. With an extra thread the camera buffer is emptied as soon as a new image arrives and the main thread always gets the latest image and no delay is added.
The processor load on a Raspberry Pi 3 is about 60% on the first core, 25% on the second core and none on core 3 and 4.
Here is a remake of the video of last Sunday:
Top Comments