Group Actions
Engagement
  • Author Author: Fred27
  • Date Created: Date Created
  • Views 3638 views
  • Likes 6 likes
  • Comments 20 comments
Related
Recommended

Pynq OCR frustration

Fred27

I've been spending a while on what I think is going to be my Pynq embedded vision project. I say I think it will be because I'm having a frustratingly difficult time of it. it's not the FPGA stuff. That's difficult but I'm getting used to that. It's not the Pynq/Pythn side of things. It's image processing.

 

A friend of mine races kayaks and apparently the monitoring of kayaks crossing the finish like is manual and fraught with error. There are lots of "recounts" and nobody trusts the results at all. I decided to identify the numbers on the kayaks as they cross the finish line, so I started digging into OCR (Optical Character Recognition).

 

As nobody is racing kayaks at the moment I decided to make use of the slightly odd blue carpet that was fitted when we bought the house and fake it a bit. I have some video, but to start with I decided to use this image. I made the numbers reasonably easy to read and even used an OCR-B font. OCR-A will probably look reasonably familiar but is a bit too strange to convince anyone to use for racing!image

image

All image processing of this type like nice clean data to work with and so I grayscaled the data, used a threshold so I was just looking at black and white and applied a Gaussian blur to smooth any noise. All fairly standard stuff.

 

I also found that a "tophat" morphological operator removes the large elements, leaving just the smaller detail that we're interested in. So far so good. Here's a sample of the Python in my notebook:

import cv2

 

image = cv2.imread("kayak_ocr_b.jpg")

gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

gray = cv2.threshold(gray, 100, 255, cv2.THRESH_BINARY_INV)[1]

gray = cv2.GaussianBlur(gray,(3,3),0)

cv2.imwrite("gray_ocr_b.jpg",gray)

 

 

 

# apply a tophat (whitehat) morphological operator to find smaller light regions against a dark background

 

rectKernel = cv2.getStructuringElement(cv2.MORPH_RECT, (18, 6))

 

tophat = cv2.morphologyEx(gray, cv2.MORPH_TOPHAT, rectKernel)

cv2.imwrite("tophat_ocr_b.jpg",tophat)

This is what my image now looks like. This should be a little easier for image processing.

image

 

Tesseract OCR

image

If you Google OpenCV OCR to tend to find that you're led towards Tesseract and (for Python) PyTesseract. You simple point Tesseract at your image and tell it where to put the text output. There are of course a few options, which I tweaked to tell it that I was looking for separate pieces fo text rather than a whole page. The output was pretty awful and also took about 10 seconds on the Pynq Z2. I managed to get something if I gave it the cleanest screenshot of some on-screen text but it was still pretty bad.

 

i decided that considering how slow it was, it wasn't worth pursuing much further. Time to look at a different approach.

 

Matching

It seems that rather than going for all-out OCR it is possible to pattern match to find the shapes of the digits I need. If it's possible to find eyes and faces and other such variable objects it shouldn't be too hard with 10 fairly well-defined digits. I have got control of the font after all.

 

I found this interesting tutorial on scanning for credit card numbers. It seemed to be fairly similar to what I needed and it wasn't too difficult to adapt it to look for some separate blocks of digits rather than the 4 sets of four used by credit cards. I even managed to get it to work a bit when I had used an image program to clearly paste some OCR-A numbers over the ones in the photograph. It wasn't unbearably slow either.

 

However, as soon as I switched to something even slightly realistic - using OCR B instead of OCR A or printing the numbers on paper where they may be ever so slightly angled, it all fell apart. I've spent days trying to tweak it to get something to work. I could even get ludicrous results like this cropping up - from an earlier attempt when using copy and pasted OCR A.

image

Rant almost over

Sorry this has been a bit of a rant rather than a helpful blog post, but I believe it can sometimes be as important to document failure as success. If anyone does have some tips on OCR then feel free to post below. I'm really enjoying a lot about Pynq, but right now image processing it not part of that! image

Parents Comment Children
  • in reply to Jan Cumps

    IMHO, it seems – I don't know it specifically – that tesseract needs a homogeneous well-contrasted background to recognize the numbers (or whatever else it can recognize). The problem described is strictly related to the fact, as far as I know, that the background applying just the mentioned filters, is enhanced to a design pattern that is easily confused by the recognition algorithm.

    The best solution to faster remove the pattern in cases where the background is so different than the elements (black numbers on white background) I think a couple of adjustments should be applied to the image before the conversion.

     

    1. Identify all the objects with some CV function to identify the RGB contours of the objects. This will produce a large array of all the objects you see confusing the algorithm.
    2. Apply (maybe more than one time) the functions erode and dilate in sequence. This eliminates much of the background noise.
    3. If – as it seems to be – the average range of the objects to be recognized is almost bigger respect to the noise elements of the background, extract from the recognized objects a subset of ROI (Region of Interest) no smaller than the smaller size of the square with numbers you expect to find in the image.
    4. Apply the OCR recognition at this point.

     

    Enrico