RoadTest: Enroll to Review the Microchip PolarFire SoC Video Kit
Author: dimiterk
Creation date:
Evaluation Type: Development Boards & Tools
Did you receive all parts the manufacturer stated would be included in the package?: True
What other parts do you consider comparable to this product?: ZYNQ 7000
What were the biggest problems encountered?: 1. VectorBlox Compiler ML ops support can be improved. 2. WIFI module not populated
Detailed Review:
In this review we will look at the Microchip PolarFire SoC Video Kit. This kit is aimed at image processing and video applications. Many thanks to @rscansy and Element14 for providing the opportunity to test the AI/ML capabilities of this new platform from Microchip.
The main agenda for this roadtest is as follows.
1. First the per-requisite software environment for Polarfire Video Soc development board was set up. This required installing multiple IDEs like Libero, Softconsole, SmartHLS and addressing the licensing steps. Graciously , Microchip offers a free one year Silver License for the MPFS250T part present on the Polarfire Kit.
2. The next step focuses on the installation and configuration of the Vectorblox SDK. In the blog entries below , the demo networks were first run on the board using an HDMI source as a video feed. The user can substitute this with an RPI or a camera that has a HDMI interface. A Lenet network was then written in Pytorch and ported to VectorBlox SDK. This proceeded smoothly with no issues. A HitNet Stereo network was then attempted together with a YOLOV8 network. Both of these networks contain ML operations not currently supported by Model Optimzer.
3. Lastly we'll look at the FPGA pipelines that comes with the kit. It should be noted that as of December 21 2023 the stereo pipeline has not been ported to the Polarfire SOC Video Kit.
| {gallery}Polarfire Video SOC |
|---|
 |
 |
 |
 |
The main features of the board are:
1. Stereo camera mezannine based on the IMX334 image sensor
2. Dual Gigabit Ethernet
3. PCI root endpoint
4. USB Host
5. Embedded Flash 6 Programmer
6. FMC VITA HPC connector
7. HDMI RX and TX connector
8. 1 MicroBUS connector for connecting embedded sensors
9. WIFI module - Not populated
The complete list of features is given on the product page at : https://www.microchip.com/en-us/development-tool/mpfs250-video-kit
Strangely enough the board comes with an unpopulated footprint for ATWIlC3000-MR110CA_NL.
The actual kit that was shipped did not contain a MIPI DSI display. However the dual stereo camera was included. The board setup is documented on the following git repo:
https://github.com/polarfire-soc/polarfire-soc-documentation/blob/master/reference-designs-fpga-and-development-kits/mpfs-video-kit-user-guide.md
The board out of the box comes pre-programmed with a Yocto image on the emmc. However this is an old image using the 2022.2 release and it has to be updated in order to use either the latest VectorBlox SDK or any of the latest H264 Video encoder reference designs.One important point is to synchronize between the buildroot image versions and the Libero reference designs as they must match. Otherwise one will observe kernel hanging during boot.
The emmc update procedure is shown on this page: https://github.com/polarfire-soc/polarfire-soc-documentation/blob/master/reference-designs-fpga-and-development-kits/updating-mpfs-kit.md#programming-a-linux-image
To get started simply attach the USB and open the 3rd serial port. The provided image does not have a GUI included on the Yocto Linux image
On the latest images 2023.1 and 2023.2, DHCP is enabled by default so plugging an Ethernet cable should be enough to connect to the network. ideally , Microchip could also have added the WIFI module , to allow wireless connectivity. It does not seem that this is currently supported on the Yocto image. The documentation does not provide any reason why the WIFI module was not populated. The board comes with a hefty power supply.
The following blog entries show how to :
1. Install the Libero environment and address the licensing issues - Libero Installation and licensing
2. Install the VectorBlox SDK and test the provided demo networks - VectorBlox SDK setup
3. Building a custom network from scratch and issues porting current State of the Art (SOTA) models - Custom networks with VectorBlox
The VectorBlox SDK allow to take a neural network model and perform:
a) Model optimization - this step converts the alredy trainend etwork into a common intermediate language (IR) VNNX and optimizes the network for inference by removing multiple layers. The open vino model optimizer (mo) is responsible for this step.
2. Quantization and calibration - In this step (mo) converts the optimized networks from FP32 to INT8 which enables the network to use less memory. This step has an effect on the accuracy.
3. Runtime generation - during this step a binary BLOB is written to embedded non-volatile storage (such as SPI flash memory)
This is used by the CoreVectorBlox IP which is a hardware accelerator included in the SDK. CoreVectorBlox IP uses the network blob and image stream and writes the output back to DDR.
The complete environment runs from Linux userspace. One notable item is that all post processing is done via C routines and python is only used for
One advantage of the Vectorblox network is that the embedded code can be modified and load to run multiple CNN BLOBs. this allows dynamically switching CNNs on the fly or to load CNNs sequentially for simultaneous inference. in practice however this requires optimizing the network for size. Due to limited size of NOR flash it's not possible to store relatively large networks.
Polarfire Video SOC comes with full Yocto Linux support. Microchip graciously enough provides not only the recipes to rebuild the image from scratch but also updated image builds for both eMMC and NOR flash. The official images contain a limited subset of software so while python is supported out of the box the user needs to install image processing packages like OpenCV or gstreamer. Compiling on device while doable for small program won't be as efficient for packages like ROS or OpenCV.
There are is a buildroot and a Yocto repo with the Polarfire meta layer.
https://github.com/linux4microchip/buildroot-external-microchip
https://github.com/polarfire-soc/meta-polarfire-soc-yocto-bsp/releases
The Hart Software Service (HSS) zero stage bootloader is critical in configuring the PolarFire SoC chip. This is used as a First Stage bootloader than can configure the device. It performs tasks such as retrieving the subsequent boot stage from eMMC and facilitating a seamless handover to the next boot stage, such as u-boot.
An important note is that Libero FPGA designs cannot be run with different Yocto images and have to match otherwise the boot process fails. The provided Yocto images do not comes with a desktop GUI enviornment so that needs to be added manually to the Yocto recipes and the images have to be rebuilt.
Microchip provides a couple of examples image processing pipelines.
TCL scripts are used for configuring IP the blocks, and a top level TCL scripts was used for stitching these IP blocks together. One important note is that tcl script are specific to a Libero version and as of December 2023 , Microchip does not provide an easy way to upgrade an old design from Libero 2022.1 to 2023.1.
In this section we'll look at the H264 video pipeline and the VectorBlox accelerator pipeline.
The H264 pipelines steps are as follows:
1. Receive images via MIPI CSI-2 camera bus
2. The image goes through a Bayer Interpolator and a gamma correction IP
3. next the RGB stream goes through image enhancement and image scaler
4. After it exits the Video processing pipeline it goes through an RGB to YCBCr chroma converter
5. Finally the YCbCr pixels are routed to the h264 encoder.
6. The output of the H264 encoder is then read via an APB3 bus and sent via Gigabit connection
The Vectoblox reference design makes use of the HDMI receiver and HDMI transmitter. It does not use the MIPI camera
The main steps are:
1. The image is stored in DDR4 memory from HDMI receiver by using a HDMI frame writer.
2. Before inference, the image is read back from DDR4 using an HDMI frame reader
3. The images are converted to RGB and written back to DDR4
4. The CoreVectorBlox engine runs inference on R, G, and B arrays and writes the results back to memory
5. Then the Mi-V (Soft RISC-V ) does the post-processing and sorts probabilities and creates coverage frames with bounding boxes and puts the results into DDR4
6. Finally the original image is read from DDR and it is alpha blended with the overlay frame, and sent to the HDMI display
In this road-test we looked at the Polarfire video SOC board from Microchip featuring the new MPFS250T Polarfire2 SOC with RISC-V. A couple of neural network were implemented and tested on board.
The Polarfire SOC Video Kit FPGA designs are continuously evolving with the Libero releases. There is quite a bit of material to digest however as of December 2023 the main limitations are:
a) Not straightforward to easily port HW designs between different Libero versions
b) Vectoblox SDK is not updated as frequently and tied to older releases of Libero
Other than that , the platform is optimal for implementing re-configurable AI/ML networks on flash based FPGAs for space applications.
References
[1] https://www.microchip.com/en-us/development-tool/mpfs250-video-kit
[2] https://github.com/Microchip-Vectorblox/VectorBlox-SDK
[3] https://github.com/polarfire-soc/polarfire-soc-documentation
[4] https://github.com/linux4microchip/buildroot-external-microchip
[5] https://coredocs.s3.amazonaws.com/Libero/2023_2/Tool/Libero_Installation_Licensing_Setup_User_Guide.pdf
[6] https://www.microchip.com/en-us/products/fpgas-and-plds/fpga-and-soc-design-tools/soc-fpga/softconsole
