Group Actions
Engagement
  • Author Author: javagoza
  • Date Created: Date Created
  • Views 382 views
  • Likes 8 likes
  • Comments 8 comments
Related
Recommended

Final - FruitVision Scale

javagoza
Final - FruitVision Scale

Introduction: A Smart Solution for Fruit Identification and Weighing

For my entry in this challenge, I came up with a project to make the self-service weighing of fruits and vegetables in supermarkets easier. Right now, in many stores, customers have to manually pick out the product, write down its number, and then weigh it on a scale with lots of buttons. This process can take time and people can make mistakes.

To fix this, I thought of a smarter way that would automate the weighing and labeling, making it better for both customers and store operations. Using FPGA technology and Computer Vision, we can create a smart scale that automatically knows what type of fruit or vegetable is on it and generates a label with the weight and price or sends it directly to the cash register.

image

During the two-month challenge, I wrote five blog posts that delved into the technical intricacies of the smart scale. These posts explored the various components of the system and their integration.

I discussed different approaches to integrating load cells, including the development of hardware accelerators and controllers, as well as the use of various analog-to-digital converters, the one internal to the Zynq XADC and an external one. In addition, I delved into the implementation of deep learning models on the Zynq-7000 SoC, showing their potential for real-time fruit and vegetable classification.

Throughout the challenge, I received valuable feedback from the community, which helped to approach problems differently.

What is FruitVision Scale?

The FruitVision Scale is an intelligent system that automates the weighing and classification of fruits and vegetables. It utilizes computer vision techniques to identify the type of produce placed on the scale, and then measures its weight to calculate the total price. This innovative solution streamlines the checkout process in grocery stores, improving efficiency and accuracy.

image

How does FruitVision Scale works?

The FruitVision Scale uses the power of FPGAs and AI to accurately classify and weigh fruits. Here's a breakdown of how it works:

  1. Image Capture: A camera captures an image of the produce placed on the scale.
  2. Image Processing: Computer vision algorithms process the image
  3. Deep Learning Classification: The processed image is fed into a pre-trained deep learning model, which identifies the type of fruit or vegetable.
  4. Weight Measurement: Load cells measure the weight of the produce, and an FPGA-based driver processes the analog signal to obtain a precise digital weight reading.
  5. Data Integration and Pricing: The classification result and weight measurement are combined, and the system calculates the total price based on pre-defined pricing information.
  6. User Interface and POS Integration: The system displays the fruit type, weight, and price on a user-friendly interface. Additionally, it can communicate with a Point-of-Sale (POS) system to integrate the transaction into the store's inventory and sales records.

image

Technologies Involved:

  • FPGA: Provides hardware acceleration for real-time image processing and weight measurement.
  • Deep Learning: Enables accurate fruit classification using AI techniques.
  • Load Cells: Measure the weight of the fruit.
  • HX711 Amplifier: Converts the analog signal from the load cells into a digital signal.
  • PYNQ Framework: Simplifies the development process and allows for seamless integration of hardware and software components.

The Digilent Arty Z7 and AMD Zynq-7000 SoC

In the first blog I explored the hardware that underpins this smart fruit weighing and classification system focusing on the Digilent Arty Z7 development board and the AMD Zynq-7000 System-on-a-Chip (SoC).

The Digilent Arty Z7

The Digilent Arty Z7 is a development board centered around the AMD Zynq-7000 SoC. Some key features of the Arty Z7 that make it ideal for our project:

image

  • Dual Processing Power: The Arty Z7 boasts a dual-core ARM Cortex-A9 processor clocked at 650 MHz. This provides ample processing grunt for running complex algorithms and user interfaces.
  • Flexible I/O Options: The board is equipped with a rich set of input/output (I/O) options, including high-speed interfaces like Gigabit Ethernet, USB 2.0, and SDIO. This allows us to connect various peripherals like cameras, displays, and storage devices.
  • Universal Headers and Pmod Ports: The Arty Z7 integrates universally popular Arduino headers and multiple Pmod ports. These offer immense flexibility for connecting sensors, actuators, and other custom modules relevant to our project.
  • FPGA Fabric: But perhaps the most crucial aspect of the Zynq 7000 is its integrated Field-Programmable Gate Array (FPGA) fabric. This programmable logic allows us to create custom hardware circuits tailored to our specific needs, offering significant performance and efficiency benefits.

image


The AMD Zynq-7000 SoC

The AMD Zynq-7000 SoC is the heart of the Arty Z7. This chip tightly integrates the dual-core ARM processor mentioned earlier with the aforementioned FPGA fabric. This "heterogeneous" architecture offers a unique advantage:

  • Heterogeneous Processing Power: The Zynq-7000 allows us to use both processing paradigms. The ARM processor handles tasks like high-level decision making and user interface interaction, while the FPGA excels at real-time, computationally intensive operations like image processing and io communications.

image

PYNQ: A Rapid Prototyping Framework

In the second blog post, I explored a powerful software framework called PYNQ. PYNQ, which stands for "Python Productivity for Zynq," is an open-source platform designed to simplify the development of FPGA-based systems.

image

Benefits of PYNQ:

  • Python-based Development: PYNQ uses Python, a popular and user-friendly programming language, to accelerate the development process. This allows developers to focus on the core functionality of their applications without getting bogged down in complex hardware details.
  • Jupyter Notebook Integration: PYNQ seamlessly integrates with Jupyter Notebooks, providing an interactive environment for writing, testing, and visualizing code. This makes it easier to experiment and iterate on designs.
  • Hardware Abstraction Layer: PYNQ offers a hardware abstraction layer that hides the complexities of the underlying FPGA hardware. This allows developers to interact with the FPGA using high-level Python functions, significantly reducing development time and effort.

Getting Started with PYNQ on the Arty Z7


Getting started with PYNQ is very easy:

  • Download the PYNQ Image: Head over to the PYNQ website and download the appropriate PYNQ image for your Arty Z7 variant, PYNQ-Z1 shares the same schematics.
  • Flash the Image: Follow the instructions provided by PYNQ to flash the downloaded image onto the Arty Z7's SD card using a suitable tool. This essentially installs the PYNQ operating system onto the board.
  • Connect and Configure: Once flashed, connect the Arty Z7 to your computer using a USB cable and power it up. PYNQ provides a web interface that allows you to access the Jupyter Notebook environment.
  • Run Basic Examples: PYNQ comes pre-loaded with basic Python examples demonstrating functionalities like blinking LEDs and interacting with buttons on the Arty Z7. Running these examples helps verify that everything is set up correctly and introduces you to the PYNQ development environment.

image

Precision Weighing with Load Cells

In the previous blogs, I discussed the hardware platform and development tools. Then, I focused on a key feature of the FruitVision scale: accurate weight measurement. I took an in-depth look at load cells and how to easily integrate them with the Digilent Arty Z7 via PYNQ using the Zynq XADC analog-to-digital converter.

Understanding Load Cells

Load cells are transducers that convert mechanical force (weight) into an electrical signal. They are commonly used in weighing scales to measure the weight of objects. In our FruitVision Scale, load cells will be crucial for accurately determining the weight of the fruit.

image

A popular type of load cell is the strain gauge load cell. These sensors exhibit a change in electrical resistance when subjected to mechanical stress. By measuring this change, we can determine the applied force, or in our case, the weight of the fruit.

Integrating Load Cells with the Arty Z7

To integrate load cells with the Arty Z7, we need to follow these steps:

  1. Physical Connection: The load cells are physically mounted to the platform of the scale. Proper wiring and connection to the Arty Z7's input/output pins are crucial.
  2. Calibration: To ensure accurate weight measurements, the load cells need to be calibrated. This involves applying known weights and adjusting the system's software to correlate the electrical signal with the actual weight. PYNQ can be used to implement this calibration process.
  3. Software Implementation: Once calibrated, we need to write software to read the analog signals from the load cells. PYNQ provides libraries and functions to interact with the Zynq's analog-to-digital converter (ADC). The ADC converts the analog signal into a digital value, which can then be processed to determine the weight.

I needed to amplify the load cell signal with an AD8226 instrumental amplifier

image

FPGA-Based HX711 Driver

In the third blog, I talked about how important load cells are and how they fit into the FruitVision Scale. Based on feedback from that post, the fourth blog focused on creating a custom FPGA controller for the HX711 load cell amplifier.

The HX711

I used the HX711, a popular 24-bit analog-to-digital converter (ADC) made for weighing applications. It makes it easy to connect with load cells by providing a handy serial interface. In our project, the HX711 acts as a bridge between the load cells and the Arty Z7, turning the analog signal from the load cell into a digital format that the Zynq SoC can read.

image

Custom FPGA Controller

I developed a custom FPGA controller using the FPGA's programmable logic. This controller manages the low-level serial communication between the Arty Z7 and the HX711. By implementing this controller in hardware (FPGA), we can offer an alternative to the SPI implementations of the Zynq ARM processors for data acquisition and processing.

image

Real-time Data Processing

The FPGA provides real-time processing capabilities. The custom controller can perform basic filtering and noise reduction on the weight data from the HX711, improving the accuracy and reliability of measurements.

Integrating with PYNQ: Bridging Hardware and Software

image

Communicating with the FPGA from scratch can be tricky, but PYNQ makes it easier. It offers tools and libraries to interact with the FPGA using Python. This allows us to:

  • Configure FPGA Logic: We can use PYNQ to write Python code that sets up the FPGA for the custom HX711 controller overlay, simplifying the development process.
  • Read Weight Data: Once the controller is in place, we can write a Python driver so that PYNQ can provide functions to read weight data from the HX711, which can then be processed and displayed by Python code running on the Zynq processor.

image

Deep Learning on the Edge - The Power of AI for Fruit Recognition

Throughout this series, I talked about the hardware, tools, and weight measurement techniques for the FruitVision scale. In this final blog, I showed how artificial intelligence (AI) and deep learning can help us identify fruits in real-time.

imageimage

Deep Learning

Deep learning is a part of AI that’s inspired by how our brains work. It uses neural networks with many layers to learn patterns from data. For us, that data is fruit images, and the neural network will learn to tell different fruits apart.

image

Using deep learning for fruit classification means the FruitVision scale can automatically recognize the type of fruit on it. Here’s how it works in our project:

  • Training the Neural Network: We need a large collection of fruit images for the system to learn from. This dataset helps train the deep learning model to recognize different fruits.
  • Deploying on the Zynq-7000: After training, the model is deployed on the Zynq-7000 SoC in the Arty Z7. This allows the FruitVision scale to classify fruits in real-time, without needing an internet connection.

The Power of Edge Computing

Deploying deep learning on devices like the Zynq-7000 has several perks:

  • Low Latency: Processing happens directly on the device, so there’s no delay from sending data to a cloud server. This is crucial for real-time tasks like our FruitVision scale.
  • Privacy and Security: Fruit and surrounding images stay on the device, which is better for privacy compared to cloud-based solutions.
  • Offline Functionality: The FruitVision scale can work even without internet, which is handy in places with poor connectivity.

Tools for Deep Learning on Zynq

The Zynq-7000, with its mix of CPU and FPGA, is great for running deep learning models. Some useful tools are:

  • TensorFlow Lite: A lightweight version of TensorFlow, designed for devices with limited resources like the Zynq-7000.
  • PYNQ Machine Learning: Offers libraries and tools specifically for deploying machine learning models on Zynq-based systems.

I learnt various techniques to implement quantized neural networks, leveraging the strengths of both the Processing System (PS) and Programmable Logic (PL) of the Zynq-7000. By strategically partitioning the neural network across these two domains, we can optimize for performance and efficiency.

image

During my experimentation, I successfully installed and compiled TensorFlow Lite on the Zynq-7000. However, I encountered memory constraints during the inference phase, which limited its practical application in the FruitVision Scale. I plan to explore this further in future iterations, potentially by optimizing the model or using different techniques to manage memory usage.

Conclusion

The primary goal of this project was to assess the feasibility of deploying a FruitVision Scale solution utilizing the Zynq-7000 SoC. While I haven't been able to fully realize a complete prototype within the given timeframe, I have successfully established the necessary foundation and acquired the requisite knowledge to progress further.

I would like to express my sincere gratitude to the Element14 community and AMD for selecting me as a participant in this design challenge. Their generous sponsorship has made this opportunity possible. I would also like to thank all members of the community who have taken the time to read and contribute to the blog discussions.

FruitVision Scale Blogs Series