Sign Language Interpreter Blog 1: Project Overview and Kit Unboxing

19 Oct 2024

Introduction
I participated in The Eye on Intelligence challenge hosted by Element14 and luckily got selected as an official challenger. I have background in VLSI and like to play with Electronics items and boards in my free time.

For this challenge, I plan to make a sign language to text interpreter using Machine Learning (Using Deep Neural Network). I don't have much prior experience in running ML on FPGAs, so this is going to be new and probably a little bit challenging for me. I am sure that I am going to learn a lot while working on this project.

Hand Sign Language
Sign language is a visual language that uses hand motions, changes in hand shape, and track information to convey meaning. Generally, this technique is used for communication by people who have low or no hearing ability. All the 26 letters of English Alphabet can be conveyed using unique hand positions. There is one issue with this, not all the people in world can understand these hand signs. I believe this issue can be solved using machine learning.

Project Overview
To address the above mentioned problem, I plan to make a portable hand sign language interpreter. It will convert the hand signs into Alphabet characters one by one on the fly. The characters will be displayed on a screen. To implement this, I will be using an FPGA board. The plan is to use the parallel processing capability of the FPGA board for faster results. One HD webcam will be attached to the board which will be continuously sending visual input to the FPGA board. The board will process the visual data and classify the hand signs into characters using ML algorithms. The result will be then shown on the screen.

Here is a simple block diagram to understand the flow:

FPGA
What is an FPGA?
FPGA is a type of integrated circuit (IC) that can be programmed to perform specific functions after it has been manufactured. FPGA stands for Field Programmable Gate Array. These are referred to as 'Field Programmable' because they have the capability to be reconfigured for specific use cases. These can be reconfigured any number of times after manufacturing. When an FPGA is configured, the internal circuitry is coupled in such a way that the software program is implemented in hardware.
In short an FPGA is a chip that is reconfigurable.

FPGA Architecture:

Some one line definitions:
Logic Block - FPGAs are built from one basic 'logic-block', duplicated hundreds or thousands of time. A logic-block is basically a small lookup table (LUT), a D flip-flop and a 2-to-1 mux (to bypass the flip-flop if desired).
Interconnect - Each logic-block can be connected to other logic-blocks through interconnect to achieve complex functions.
Input/Output Block - The interconnect wires also go to the boundary of the device where I/O blocks are implemented and connected to the pins of the FPGAs.

Why use FPGA?
FPGAs can be more efficient for specific tasks, as they enable architecture customization that optimizes resource use. An important point to keep in mind is that FPGAs are not preferred in all the applications. The below table lists some differences between FPGA & ASIC.

FPGA	ASIC
Reconfigurable Circuit after manufacturing	Fixed circuit throughout product's life
Suitable for digital designs only	Analog circuit can also be implemented
Adaptable for many applications	Suitable for only one application
Larger size than ASICs	Smaller size than FPGAs
Less energy efficient (uses more power)	More energy efficient (uses less power)
Difficult to attain high frequency rates	Work at high frequency rates
Bugs can be fixed in the final product	Silicon bugs are very hard to fix

How to choose between FPGA & ASIC?
This depends on the application. If the application requires constant bug fixes, feature and design changes, and software flexibility, then FPGAs may be the right solution. But if the application requires high performance, smaller device footprint, and significantly lower power consumption, then ASICs are better choice.

The Kit
The 20 officially sponsored challengers were shipped a kit to use in their projects. I got the kit in a nicely packed box. The kit contains below items:

Arty Z7 FPGA Board
Multicomp Pro 12VDC 2AMP Power Adaptor
ZIQIAN HD 1080P webcam
Digilent ambient light sensor Pmod

Images:

Package Opened Package

In the coming sections , I will go through each item in the kit.

Arty Z7 FPGA Board
The Arty Z7 FPGA board came in a small box covered in multiple layers of foam for protection along with an ISSI pamphlet.

Front Arty Box Back Arty Box

Opened Arty Box

Front and Back of the board:

Front of Arty Z7 Back of Arty Z7

The Arty Z7 is a development platform designed around the ZYNQ-7000 architecture from Xilinx. This board is basically designed for hobbyists and makers. There are two variants of the board. The one which I got in the kit is Arty Z7 - 20 (Part Number: XC7Z020-1CLG400C) It has the following features:

Full support for Vivado, SDSoC, and Petalinux design environments
Powered by ZYNQ SOC Techonology
650MHz dual-core Cortex-A9 processors integrated with the programmable logic equivalent to Artix-7 FPGA.
512 MB DDR3 with 16-bit bus @ 1050 MBps
16 MB Quad SPI Flash
Buttons, RGB LEDs, Switches, HDMI (Input + Output), USB, Ethernet & Audio
Arduino and Pmod expansions
Programmable from JTAG, Quad-SPI flash, and microSD card

I have an Arty S7 board. I thought it will be interesting to compare both the boards based on their FPGA Specifications:

FPGA Specifications	Logic Cells	6-Input LUTs	Flip-Flops	Block RAM	DSP Slices	Clock Resources	Internal ADC
Arty Z7 - 20	13,300	53,200	106,400	630 KB	220	Zynq PLL with 4 outputs 4 PLLs 4 MMCMs 125 MHz external clock	Dual Channel, 1 MSPS
Arty S7 - 50	8,150	32,600	65,200	337.5 KB	120	100 MHz & 12MHz external clocks 5 PLLs 5 MMCMs	Dual Channel, 1 MSPS

Below table contains the connectivity and on-board IO information:

Connectivity
USB	USB-UART, USB-JTAG Programmer, USB Host
Ethernet	Gigabit Ethernet PHY
HDMI	Sink (input) Source (output)
Audio	PWM driven mono audio output with 3.5 mm jack
Pmod Connectors	2
Other Connectors	Arduino/chipKIT shield connector
Switches	2 slide switches
Buttons	4 Push buttons
LEDs	4 LEDs, 2 RGB LEDs

These ports can be seen in the below diagram:

Power Inputs:

USB
7 V - 15 V External source

Reference Manual: Link

Multicomp Pro Power Adaptor
The power adaptor came in a small box. The adaptor requires a two pin power cable cord to connect it to the power supply but it wasn't in the box.It would have been better if the box had the cord in it. I tried to look for it in the local market but couldn't find it. For now, I have decided to power the board from USB cable.

Adaptor Box Adaptor outside box

Webcam
Its a 1080P HD USB camera with Stereo Microphone. The webcam comes with a detachable tripod which is a good thing. It can be used with the tripod or it can be placed on the edge of the monitor.

Front camera box Back camera box

Opened camera Attached camera

I have attached a short video of it recording a video.

Before using it on Windows, I tried to use it on Ubuntu. It didn't record properly on Ubuntu. Maybe there is an issue in the camera app on Ubuntu as the webcam is working on Windows.

Ambient Light Sensor Pmod
An ambient light sensor (ALS) is a photodetector that measures the amount of light in a user's environment and can be used to adjust the brightness of a device's screen accordingly.

The Digilent PmodALS has a single ambient light sensor. The amount of light the ALS is exposed to determines the voltage level passed into the ADC, which converts it to 8-bits of data. A value of 0 indicates a low light level and a value of 255 indicates a high light level.

Features: