Related
Recommended

PYNQ-Z2 Dev Kit - CIFAR-10 Webcam continued....

ralphjy

In the previous blog PYNQ-Z2 Dev Kit - CIFAR-10 Convolutional Neural Network , I verified the 3 hardware classifiers against the reference "deer" test image.  Now I'm going to see how the classifiers perform with captured webcam images.  I expect the performance will be degraded because the webcam will produce lower quality images due to issues like image brightness and focus.  CIFAR-10 has a small training set (5000 images per class), so I'm going to use a solid background to help keep the image simple.  Intuitively, I would expect that with higher quantization level the classification accuracy would improve.

 

I decided that since CIFAR-10 has transportation classes (automobile, truck, ship, airplane) that I would try to classify a few Matchbox vehicles that had escaped being recycled.

 

Here is my test setup.  I was a bit disappointed in the auto-focus and auto-exposure of the webcam, but I haven't figured out whether I can manually control this particular camera with OpenCV.

image

 

 

First try is the firetruck.  I expected this one to be easy, but only the W2A2 classifier got it right.

image

 

Second try is the convertible.  All the classifiers thought this was an airplane.  I'm assuming that the odd colors may have been confusing.

image

 

Third try is a car that I thought should be easy but interestingly the classifiers with higher quantization very clearly classified it as an airplane and the binary classifier thought it was a ship.

image

 

 

As a head check I grabbed one of the CIFAR-10 test images and it correctly classified with increasing accuracy as the quantization increased.

image

 

At this point I'm confused as to the primary cause of the poor classification of the webcam images.  Is it the image quality or the actual image.  I suspect it is both.

 

I noticed that the example notebook includes a sample webcam capture of an elk figurine which should classify as a "deer".

image

 

I would consider this a much harder image to classify because of the background.  It's interesting that in two of the classifiers that "airplane" and "deer" are closely ranked.  I'm getting the impression that CIFAR-10 trained networks are a good educational tool, but would not work well in the real world.  I'm going to move on to look at other neural network examples.

  • It is one of the strange (?) things about using Artificial Neural Networks (ANN), particularly when using image systems or other data that is viewed by humans, is that they use all the data. When a human looks at the car picture they just see the car and use that part of the image for recognition. an ANN isn't intelligent and uses all the image, including the background so when recognising the car, or plane or whatever, it is actually just recognising the array of pixel values and working out the nearest trained image of pixels values it has been given.  So by making all the backgrounds similar (blank or anything else really) then it becomes the differences such as the car or plane and so on, that the ANN uses to make a match. The ANN will also use the position of the different pixels as well, so either the car or plane has to be precisely centred in every image, or, the training data must contain images of the car located in as many alternative locations within the image as possible. It can lead to a very large training data set, with the corresponding problem that the larger the training data set then the longer the training period, along with a greater possibility of not converging onto a working solution. Another  problem with ANNs is, depending on the type of ANN and training used, that because training starts with some randomised numbers that training is not repeatable. So you could train, get a working solution, decide to retrain without changing anything and then when retraining, it will just not converge. My approach was that if a working system was obtained, always make sure to keep a copy of it before retraining.

     

    Dubbie

  • in reply to beacon_dave

    Okay, I did as you suggested and now it indeed does fly.  Something tells me you've done this before image.   It does suggest that this network would not have worked as I would have tried to use it.

    image

  • in reply to beacon_dave

    An easy enough experiment to try.  Did not yield though.  Still favors airplane although now the binary classifier thinks it's a cat image.

     

    image

    Sorry misinterpreted your comment.  Let me give it another go with the training image....

  • in reply to genebren

    Just wondering if the solid background and the loose framing isn't helping matters here ?

     

    Perhaps try adding some whitespace around the sports car training image and see if it too starts to fly...