For this blog I will be showing how I'll identify the different objects on the shelf. As I have mentioned in the previous blogs, I am going to use an overhead camera in combination with specifically designed labels. The goal with this is to make a cheap and simple system which can easily later be refined as well as adding new products. This is a program showing the detection of a single label. First lets get to the labels. My intent was to design something that would be easy to recognize using the Raspberry as well as intuitive to the user. I didn't want to go with a typical QR code/barcode design, so this is the type of design I came up with.
Now to explain the design. The label consists of a bright pink outer rim (I went with that color as I feel it can rarely be seen on actual products so it is easier to recognize), I use it so i can identify where the label is exactly, after that there are the smaller circles (for now I implemented only two colors, but I will for sure implement at least one more for the container weight category), the combination of numbers of yellow and green circles gives me a code which I then can compare to a premade list and see what's on the label. The background is black so it doesn't make any additional noise when the program is detecting color, and of course in the center we have the design itself, the design can be a letter, a number, or a picture design as shown above. The goal is to use the least number of words/characters possible to make it more intuitive and not so language dependable. Ill be showing now step by step with the code how i am recognizing the labels. The list of all codes and connected names is not integrated yet, but this a nice little piece which shows the actual image processing. First of the libraries we are using:
import numpy as np import cv2 import picamera import picamera.array import time
Now capturing the image with the picamera:
with picamera.PiCamera() as cam:
raw = picamera.array.PiRGBArray(cam)
cam.start_preview()
time.sleep(3)
cam.capture(raw, format='rgb')
cam.stop_preview()
frame = raw.array
frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)In the preceding code we took a picture with the picamera and then converted to the HSV color space. This is the picture that the camera took:
Now we have to go and detect the outer pink rim in the picture, I've tested at multiple distances and it works great, the only problem I had sometimes was that the phone screen was to bright and the camera wasn't capturing the colors properly which will not be a problem with the labels. This next part of code shows the extraction of the pink circle:
image_mask = cv2.inRange(hsv,np.array([130,50,50]),np.array([160,255,255])) output = cv2.bitwise_and(frame, frame, mask=image_mask)
The first line detects the color in that range and makes a black and white mask, while the second line overlays the mask and the original making a pink circle. HSV color space is great for this kind of use because the color itself is dependant on just one parameter - hue (this also means that it can easily be adjusted for different colors, or for finer color detection). These are the images we get as output from this, first one is the mask, and the second one is the overlayed image.
Now the next thing we have to do is detect a circle. The reason behind this is so we can single out an item from a full shelf, and also eliminate the possibility of detecting other green and yellow circles around which would ruin the data. So this next part of the code will detect a circle:
blur = cv2.blur(image_mask,(5,5)) circles = cv2.HoughCircles(blur, method = cv2.cv.CV_HOUGH_GRADIENT, dp=1, minDist = 700, param1 = 50, param2 = 13, minRadius = 15, maxRadius = 300) image_mask = cv2.cvtColor(image_mask, cv2.COLOR_GRAY2BGR) if circles is not None: for i in circles[0,:]: cv2.circle(image_mask,(i[0],i[1]),i[2],(0,255,0),2) cv2.circle(image_mask,(i[o],i[1]),2,(0,0,255),3) x = int(i[0]) y = int(i[1]) r = int(i[2])
First we blur the mask a bit so we refine the edges, then we use the OpenCV function HoughCircles which as parameters takes the image (blur), method (CV_HOUGH_GRADIENT is the one that is integrated into OpenCV), dp (which is the inverse ratio of resolution), minDist (which is the minimal distance between the centers of detected circles), param1 and param2 (these are method specific parameters) and minRadius and maxRadius (which are the minimum and maximum radiuses of circles which will be detected), I played a little bit with this until i got it to detect just this circle, If you put minDist too small for example something like this can happen:
Or this:
But in the end i managed to set it up so it detects only one circle, the one we need. The x, y and r in the preceding code are the x and y coordinates of the center of the circle and r is the radius, i extracted them like this because we are going to use them in the upcoming part of the code. Here is an image when the code is working properly with the detection of the one circle that we need:
The circle detected isn't completely perfect but it doesn't need to be. There is just a tiny bit of noise outside the circle which is good, but as I have said before that can be adjusted making the program detected a narrower hue range. Now, because there will be more of these labels on the shelf at once it is crucial to single out only one and detect only its smaller circles. I did by cropping the image and then using it for the rest of the code, here is the code the crops the image using the mention x, y, and r that we extracted from the circle:
new = frame[(y-r):(y+r),(x-r):(x+r)]
This is the whole thing that crops an image and saves it into another one. We literally just take a sub-matrix of the old image that is the square surrounding the circle, and this is the output of that:
Now that we are here its time to detect the small circles with which we will know what is on the label. We will use a nearly identical procedure as before, just now we will count the number of circles detected because there are more than one. Here is a code showing the detection and counting of the green circles:
hsv1 = cv2.cvtColor(new, cv2.COLOR_BGR2HSV) image_mask1 = cv2.inRange(hsv1,np.array([40,50,50]),([80,255,255])) blur1 = cv2.blur(image_mask1,(5,5)) circles1 = cv2.HoughCircles(blur1, method = cv2.cv.CV_HOUGH_GRADIENT, dp=1, minDist = 20, param1 = 50, param2 = 13, minRadius = 10, maxRadius = 100) image_mask1 = cv2.cvtColor(image_mask1, cv2.COLOR_GRAY2BGR) k=0 if circles1 is not None: for i in circles1[0,:]: cv2.circle(image_mask1,(i[0],i[1]),i[2],(0,255,0),2) cv2.circle(image_mask1,(i[0],i[1]),2,(0,0,255),3) k=k+1 print(k)
The color range for green that I am using is 40-80 hue which covers and good but not that big of a range to make a problem. This part of the program first makes the black and white mask out of the image that we can see here:
Now we have to detect the circles themselves, we do this using the same function as before just this time with a much lower minimum distance and minimum radius so it would actually detect the circles, and we get something like this:
The counter k is for the number of circles detected and in this case it is 3. The only thing left now to do is do the exactly same thing for yellow, the code is completely the same so i won't clutter the blog anymore with it (the full code is in the attachment), the only thing that changes is a different counter and of course the hue range, which i found to be 15-35, and the resulting images from that part of the code are these:
And that other counter will show 4. The full code I used to test this out as well as get the images saved will be attached to the blog, as I've said the premade files with the values aren't implemented yet, so you will see an if and elif in the code which I used to see how it works. I will probably be adding another color to the spectrum to monitor the container itself, the plan is to have some if i can call them "official" containers which will have known mass, and I could detect those and subtract their mass from the ingredient itself making the whole thing a little but more accurate specially in the case when using heavier glass jars. The colors used here are not final, the one that will most certainly stay is the pink because its the easiest to detect (rarely used anywhere), for the other colors, I will try and find some that are a bit further apart in the hue spectrum so there is less noise and a smaller chance for error. The next step is implementing the files with this and of course getting all of this to work with the scales themselves. After that its making it user friendly and sending data over the internet to a phone. Its not a short journey but it will be a fun and interesting one for sure. Thanks for reading the whole blog. Hope you like it!
Milos
Top Comments