Stove Assistant - Bernhard - Pi Chef #4.9 - Eliminating Camera Delay

17 Mar 2018

Today there is only a small update: Last Sunday I posted a video with temperature data overlay but it had a little problem because the camera data was a little bit delayed.

Now I fixed this.

I used threads and created an extra thread which only reads the camera input. I used the standard thread library of C++ 11 (std::thread). To compile the program you have to enable C++ 11 with the compiler option "-std=c++11" and add -pthread to the libraries in the Makefile.

The usage of threads is quite simple because in this example you don't have to synchronize anything. The main thread starts a second thread which reads the camera input in an endless loop and saves the data to an OpenCV mat. The main thread reads this mat, copies it and does image processing with the copied data. So main thread and camera thread don't write on the same data and since there is no common write access it does not have to be synchronize. The worst issue with this approach could be that main thread is reading the data of the mat while the camera thread is writing new image data to the map. This could lead to glitches in the image data. IMHO this is negligible.

So here is the whole source code of my program. I also included the I2C improvements of my last post.

#include <stdio.h>
#include <stdlib.h>
#include <linux/i2c-dev.h>
#include <fcntl.h>
#include <unistd.h>
#include <opencv2/opencv.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/contrib/contrib.hpp>
#include <thread>


// global variables for exchange between threads
cv::VideoCapture cap;    // create camera input
cv::Mat cameraImage;  // create opencv mat for camera

void cameraThread(void)    // function for the camera thread
{
    while(1)    // loop forever
    {
        cap >> cameraImage;    // copy camera input to opencv mat
    }
}

int main(void)
{
    int file;
    int addr=0x68;        // adress of AMG88xx
    int x,y;     // variables to got through the array
    signed short int internal_temp;
    signed short int pixel_temp[64];    // array for pixel temperatures
    
    int end=1;  // variable to end program
    printf("Pi Chef Stove Assistant Demo with AMG88xx\n");  // print start message
    if((file=open("/dev/i2c-1",O_RDWR))<0)    // open i2c-bus
    {
        perror("cannot open i2c-1");
        exit(1);
    }
    if(ioctl(file, I2C_SLAVE, addr)<0)    // open slave
    {
        perror("cannot open slave address");
        exit(1);
    }

    internal_temp=(signed short)(i2c_smbus_read_word_data(file, 0x0e)<<4);    // read internal temperature from sensor
    internal_temp=internal_temp/16;        // recalculate correct value
    printf("Internal Temp: %f C (0x%04X = %i)\n",(float)internal_temp*0.0625,internal_temp,internal_temp);    // print internal temperature

    cv::Mat cameraImageGray; // create opencv mat for grayscale camera image
    cv::Mat cameraImageBig(320,320,CV_8UC3);  // create opencv mat for camera with final resolution

    cap.open(0);        // open camera
    cap.set(CV_CAP_PROP_FRAME_WIDTH, 320);    // change camera width to 320 - we do not need more
    cap.set(CV_CAP_PROP_FRAME_HEIGHT, 240);    // change camera height to 240

    cv::VideoWriter outputVideo;    // create video output
    outputVideo.open("output.avi", CV_FOURCC('M','J','P','G'), 15, cv::Size(320,320), true);    // set video output to 15 fps and MJPG
    if (!outputVideo.isOpened())    // check if generation of video output was successful
    {
        perror("Could not open the output video for write\n");
        exit(1);
    }

    cv::Mat outSmall(8,8,CV_8UC1);        // create opencv mat for sensor data
    cv::Mat outSmallnorm(8,8,CV_8UC1);    // create opencv mat for normalized data
    cv::Mat outColor;    // create opencv mat for color output
    cv::Mat combined;    // create opencv mat for combined output

    cap >> cameraImage;    // copy camera input to opencv mat to get data to startup
    std::thread tcam(cameraThread);    // start extra thread to get camera input
    while(end==1)  // check end variable
    {
            cv::TickMeter t;  
            t.start();    // start timer  

        x=i2c_smbus_read_i2c_block_data(file,0x80,32,(__u8*)pixel_temp)// read first 32 byte / 16 temperature pixels from sensor
        x=i2c_smbus_read_i2c_block_data(file,0xa0,32,(__u8*)pixel_temp+32);       // read next 32 byte / 16 temperature pixels from sensor
        x=i2c_smbus_read_i2c_block_data(file,0xc0,32,(__u8*)pixel_temp+64);       // read next 32 byte / 16 temperature pixels from sensor
        x=i2c_smbus_read_i2c_block_data(file,0xe0,32,(__u8*)pixel_temp+96);       // read last 32 byte / 16 temperature pixels from sensor

        for(x=0;x<64;x++)
        {
            pixel_temp[x]=(signed short)(pixel_temp[x]<<4)/16;    // set pixel_temp to original value
        }

        for(x=0;x<8;x++)
        {
            for(y=0;y<8;y++)
            {
                outSmall.at(7-x,7-y)=pixel_temp[x*8+y];    // save data to opencv mat and rotate it
            }
        }

        cv::normalize(outSmall,outSmallnorm,255,0,cv::NORM_MINMAX);    // normalize Mat to values between 0 and 255
        cv::resize(outSmallnorm,outSmallnorm,cv::Size(320,320));    // resize Mat to 320 x 320 pixel
        cv::applyColorMap(outSmallnorm,outColor,cv::COLORMAP_JET);  // generate colored output with colormap
        cv::cvtColor(cameraImage,cameraImageGray,CV_RGB2GRAY);    // convert camera image to grayscale
        cv::cvtColor(cameraImageGray,cameraImageGray,CV_GRAY2RGB);    // make cameraImage 3 channels again
        cameraImageGray.copyTo(cameraImageBig(cv::Rect(0,40,320,240)));// copy camera ingae to mat with same resolution as temperature mat
        cv::addWeighted(cameraImageBig,0.5,outColor,0.5,0.0,combined);    // combine camera mat and temperature mat into one single image
        cv::imshow("combined",combined);  // display mat on screen
         outputVideo << combined;    // add frame to video

        t.stop();    // stop timer  
            printf("Time: %f ms\n", (double)t.getTimeMilli() / t.getCounter());  // print result of timer  
        
        char key = cv::waitKey(1);  // check keys for input
        if(key=='e') end=0;  // end if e was pressed
    }
    printf("ended regularly!\n");  // print end message
    close(file);
    return 0;
}

This program also measures the execution time of the main thread. The execution time of the main thread is about 60 ms. The refresh rate of the raspberry pi camera is 30 fps, so about 33 ms for each frame, which is way faster than the main thread. This explains why in a single thread application the camera buffer fills up and adds a delay to the camera input. With an extra thread the camera buffer is emptied as soon as a new image arrives and the main thread always gets the latest image and no delay is added.

The processor load on a Raspberry Pi 3 is about 60% on the first core, 25% on the second core and none on core 3 and 4.

Here is a remake of the video of last Sunday: