Demonstration and Outcome - Blog 6

Welcome to the final blog on building a Bottle Cap Inspection System using Machine Vision. In this project, we plan to create an automated visual inspection system that can detect defects in bottle caps, such as cracks, deformation, and missing parts. By using the power of the Ultra96-V2 FPGA from Xilinx, along with an HD camera and an actuator, we'll achieve accurate and efficient bottle cap inspection.

Hardware Overview:

The Ultra96-V2 is a development board based on the Xilinx Zynq UltraScale+ MPSoC architecture. It combines the programmable logic capabilities of an FPGA with the processing power of an Arm Cortex-A53 processor. With its rich set of peripherals and high-performance computing capabilities, the Ultra96-V2 is an excellent choice for our machine vision application.

We'll be using a compact camera module as our image source for capturing bottle cap images. The camera offers high-resolution imaging capabilities, along with features like auto-focus, low-light performance, and video recording. 

Getting started

We will be implementing the project concepts from the training module into practice, following a step-by-step approach. Download the uz96v2_sbc_base from http://avnet.me/avnet-u96v2_sbc_base-vitis-ai-1.3-image. After downloading the image file, program the image file to a 16GB micro SD card using Balena etcher.

The image file contains the hardware design (BOOT.BIN, dpu.xclbin), as well as the petalinux images (boot.scr, image.ub, rootfs.tar.gz).

PMIC Version Checking

Change directories to the location of the PMIC programmer and list the config files.

# cd ~/pmic-prog
# ls -l pmic-configs

You can see which revision you have by reading these two registers. 0x2A has the day and 0x2B has the month of release.

# ./pmic_prog read 6 0x13 0x002A
# ./pmic_prog read 6 0x13 0x002B
# ./pmic_prog read 6 0x14 0x002A
# ./pmic_prog read 6 0x14 0x002B

PMIC Reprogramming

To reprogram the 0x13 & 0x14 devices, enter the following command, making sure that the two highlighted numbers match!

# ./pmic_prog program 6 0x13 -i ./pmic-configs/U96V2_5401_0x13_191030.txt
# ./pmic_prog program 6 0x14 -i ./pmic-configs/U96V2_5401_0x14_191030.txt

Launch the dpu_sw_optimize.sh script

$ cd ~/dpu_sw_optimize/zynqmp
$ source ./zynqmp_dpu_optimize.sh

To find the version of the Linux, run lsb_release -a command

Define the DISPLAY environment variable

$ export DISPLAY=:0.0

Change the resolution of the DP monitor to a lower resolution, such as 640x480

$ xrandr --output DP-1 --mode 640x480

Launch the segmentation application

$ cd ~/Vitis-AI/demo/VART/segmentation
$ ./segmentation ./video/traffic.mp4 /usr/share/vitis_ai_library/models/fpn/fpn.xmodel

Algorithms for Bottle Cap Inspection 

The Camera captures images of bottle caps as they pass by. Image preprocessing techniques such as resizing, noise reduction, and contrast adjustment may be applied to improve the quality of the images. The algorithm identifies key features of the bottle cap, such as its shape, colour, and any distinguishing marks or patterns. Using machine learning or computer vision techniques, the system detects defects like cracks, deformities, or missing parts by comparing the features of each bottle cap with a reference. Based on the analysis, the system decides whether the cap is defective or not. If a defect is detected, the actuator is triggered to remove the faulty cap from the production line.

Simple Image processing using OpenCV

import cv2
import numpy as np

# Load the image from the Camera or any image source
image = cv2.imread('bottle_cap.jpg')  

# Convert the image to grayscale
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# Apply Gaussian blur to reduce noise
blurred = cv2.GaussianBlur(gray, (5, 5), 0)

# Use edge detection (Canny) to identify edges
edges = cv2.Canny(blurred, 50, 150)

# Find contours in the edge image
contours, _ = cv2.findContours(edges.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

# Initialize a list to store detected defects
defects = []

# Loop over the detected contours
for contour in contours:
    # Calculate the area of the contour
    area = cv2.contourArea(contour)
    
    # If the area is below a certain threshold, consider it a defect
    if area < 500:  # You may need to adjust this threshold based on your specific case
        defects.append(contour)
        
# Draw the detected defects on the original image
cv2.drawContours(image, defects, -1, (0, 0, 255), 2)

# Display the original image with defects highlighted
cv2.imshow('Defect Detection', image)

# Wait for a key press and then close the window
cv2.waitKey(0)
cv2.destroyAllWindows()

Training and testing the model with the collected dataset. 

Demonstration and Outcome

The Setup 

Detection and segmentation using a custom model

Challenges in detection

1. Variability: Bottle caps can vary in colour, size, and markings, making it challenging to create a one-size-fits-all detection algorithm.

2. Real-time Processing: Ensuring that the system processes images in real time, especially in high-speed production environments, can be demanding.

3. Training Data: Training the system's machine learning models requires a substantial dataset of both defective and non-defective caps.

4. Environmental Factors: Changes in lighting, dust, or camera angles can affect the accuracy of detection.

5. Integration: Integrating the system into an existing production line without disrupting operations can be complex.

Special-case conditions

• Overlapping caps
• Light conditions
• Partial visibility
• Other corner cases

Future work update on this project: Now that the detection phase is complete, my next task is to classify the items as either 'good cap,' 'loose cap,' or 'no cap.' This classification is used for the upcoming phase, which involves integrating these items into a simple conveyor system. 

Compatibility Note – e-Con USB3 Camera on Ultra96-V2 Some USB3 cameras, like the e-con Systems See3CAM_CU30_CHL_TC, may encounter compatibility issues on the Ultra96-V2 board. To resolve this, switch to USB2 mode by connecting these cameras through a USB2 hub or extender.

Summary

Our Bottle Cap Inspection System is a testament to the power of image processing and FPGA technology. It promises enhanced quality control, reduced waste, and increased efficiency in manufacturing processes, while also presenting various challenges that must be overcome for successful implementation.

Just clearing the draft! Thanks,rscasny for understanding my situation and constantly encouraging me.

Here are the links to my previous blogs

Blog-1 Taking the first steps - Xilinx Ultra96-V2 (PP3)

Blog 2 - UART Test - Serial Message on Ultra96-V2

Blog 4 - Interfacing LMS6DSL Click Mezzanine with Ultra96-V2 Using Vitis

Blog 5 - LCD click interface on Ultra96-V2

I hope that you find this interesting. Extending my sincere gratitude to Element14 and Avnet for hosting this incredible contest. It's been a fantastic journey of learning on Xilinx MPSoCs. Thanks for reading!