Using OpenCV to detect a face or other specific object on an image isn't something new or groundbreaking. However many machine vision applications are based on such functionality and the best ones excel because they provide the best UI and user experience when used - things that not many backend developers often thought of or implemented.
In this example PYNQ project I'll just part of PYNQ-Z2 features to provide a sort of user interface presenting data based on face detection. The notebook can be found on: https://github.com/riklaunim/pynq-meme-generator
Usually face detection examples take an image and mark every detected face with a rectangle and call it a day. We want to do it bit differently. Say that our camera sees such image:
We want to detect the face, crop the image, and display it on a screen. This could be handy in surveillance system, temporary pass generators and more. Some inspection apps would behave similarly. So the example would be:
A camera and a display can be connected to a PYNQ-Z2 board but the display doesn't have to display image from the camera. We can use Pillow in Python to create an image file programatically and fill it with information based on processing we done with a frame from the camera:
So when camera sees a face, when pointed at a TV:
The HDMI output can be completely custom and listing detected faces:
Here Pillow is used to create a 1920 x 1080 image, colorized pink and then a cropped image of the face is placed on it. Final image is pushed to the HDMI output.
Face can be detected with simple:
def detect_faces_on_frame(frame):
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
return face_cascade.detectMultiScale(gray, 1.3, 5)
Then you can use Pillow to process it, similarly to how meme-generating websites work:
def draw_meme(image, face):
image = crop_face(image, face)
image = PIL.ImageOps.expand(image, border=20, fill='deeppink')
w, h = image.size
overlay = PIL.Image.new('RGB', (w, h + 60), (255, 20, 147))
overlay.paste(image, (0, 0))
font_size = 30
if w < 200:
font_size = 20
draw = PIL.ImageDraw.Draw(overlay)
font = PIL.ImageFont.truetype("/home/xilinx/jupyter_notebooks/ethan/data/COMIC.TTF", font_size)
draw.text((20, h),"PYNQ Hero!",(255,255,255),font=font)
return overlay
You will find all functions used here in the notebook (with pytest tests!). The key element is the "frame loop" where for every frame from camera we execute the detection and processing:
memes_displaying = False
while True:
frame = hdmi_in.readframe()
image = get_image_from_frame(frame)
faces = detect_faces_on_frame(frame)
created_memes = []
for face in faces:
meme = draw_meme(image, face)
created_memes.append(meme)
if created_memes:
print('Memes detected', len(created_memes))
output_image = display_memes_on_one_frame(created_memes)
output_frame = npframe.array(output_image)
outframe = hdmi_out.newframe()
outframe[:] = output_frame
hdmi_out.writeframe(outframe)
memes_displaying = True
else:
if not memes_displaying:
print('Displaying source')
hdmi_out.writeframe(frame)
sleep(1)
A simple "surveillance" system displaying detected faces on a high contrast background providing more clarity for a guard inspecting the result.
If you would have a touch display (like via USB2 interface) then you could connect it to the PYNQ-Z2 USB Host port, get a Python library for it and read events in a loop like the one above. If you know at which X/Y coordinates you drawn a given face or interface icon you can match it with touch event X/Y coordinates and execute an action, like for example face recognition via the obvious "face-recognition" Python package. It's very CPU intensive (and takes a lot of time to install on Z2) but it usable for a showcase. Assuming you have images of "known" personel you can then run unknown face against each of the known ones and see if it will match. The code is pretty simple:
class FaceDetectionError(Exception):
pass
def get_encoding(image_path):
face = face_recognition.load_image_file(image_path)
return face_recognition.face_encodings(face)[0]
def compare_faces(reference_encoding, analyzed_image_file):
try:
unknown_encoding = get_encoding(analyzed_image_file)
except IndexError:
raise FaceDetectionError()
else:
return face_recognition.compare_faces([reference_encoding], unknown_encoding)[0]
The "compare_faces" function returns True if faces match and False if they don't. It raises an exception if faces could not be compared.
The notebook has working pytest test cases that run the code versus an example data, I also go over the code on https://rk.edu.pl/en/managing-machine-vision-project-pynq/ as well. I've covered:
- Distributing notebooks via github repositories
- Testing within notebooks with ipytest
- Face comparison (no accelerated)
- Face detection meme generator
- Comic Sans in a machine vision project!