RoadTest: Avnet UltraZed-EV Starter Kit
Evaluation Type: Development Boards & Tools
Did you receive all parts the manufacturer stated would be included in the package?: True
What other parts do you consider comparable to this product?: Xilinx ZCU104 and ZCU106 boards with the same ZU7EV UltraScale+ MPSoC with VCU.
What were the biggest problems encountered?: 1) Most of the reference designs using the VCU are for the ZCU104 and ZCU106, 2) Had difficulty getting GStreamer to work with VCU and RTSP streams, 3) The normal issues with design examples and tutorials using different tool versions, 4) There isn't a license voucher included to use the HDMI and some of the video IP
The UltraZed-EV SOM + Carrier card included in the UltraZed-EV Starter kit represent a high performance full-featured embedded video processing system. My roadtest will only test a subset of those features but it will still take quite a bit of detail to document it, so I've decided to do it as a series of blog posts which I'll summarize here. Please refer to the links to see the test details. My main focus of this roadtest was to utilize the VCU (hardware video codec) capability and add some machine learning using the DPU (Deep Learning processor) using Vitis-AI.
Links to posts for this roadtest:
The kit arrived the last week in June after a short shipping delay caused by a train derailment. The kit included the UltraZed-EV SOM, the Carrier Card, 12V power supply, ethernet cable, micro USB cable, 16GB SD card, SOM mounting and heatsink/fan assembly hardware. What the kit did not include - a license voucher and an RTC battery. I found out later that the kit no longer ships with a license voucher but you can get a 30 day evaluation license on request. Licenses are needed for the HDMI input/output IP and various video subsystem IP elements. I'll just use the PS DisplayPort interface which uses one of the GTH transceivers and does not require an additional license. I'll also not use the RTC in this roadtest although I'll probably get a battery later.
The out-of-box (OOB) demo comes pre-loaded in the onboard flash memory and provides simple GPIO tests using the user LEDs and switches. The demo ran without issue and we're off to a good start. The fan is a bit annoying if you're sitting next to the board but I'm used to the noise as it sounds similar to the Ultra96v2 fan.
Chronologically I wrote this as the seventh blog, but it probably should have been the second. You can read the blog for detail, but at a high level here is the plan:
The thing that got me interested in this roadtest was an Avnet demo of a port to the UltraZed-EV that Jason Moss had done from the VCU reference design (VCU TRD) for the ZCU106. The initial port used the 2018.3 tools. Jason provided excellent documentation describing the porting process and the test procedure to validate the design using GStreamer pipelines. The source code and a pre-built image are available on github. I initially started with the pre-built image and ran through the GStreamer test suite.
I had a minor issue with the RTP test because I didn't realize that the RTP client had to be running when the server started (I'm used to starting the server first when using RTSP).
I then proceeded to build the design starting from the source TCL files and ran through the test procedure again to validate the build.
Jason has released a 2019.2 version of the VCU TRD and I've run through the same test process using the pre-built and build from TCL designs. I'm currently using the 2019.2 design for my GStreamer development.
I've also learned that a 2020.1 design will be available soon but no ETA yet.
I've found that PYNQ with Jupyter notebooks to be a very useful prototyping tool and I thought that it would be nice to port it the UltraZed-EV. I started by upgrading the VCU TRD design to 2019.1 which is the current toolbase for PYNQv2.5. Then I followed a tutorial by Peter Ogden of Xilinx to port PYNQ to an unsupported board. The port using the UZ7EV_EVCC_2019_1.BSP (OOB design) worked after I fixed a minor issue specifying the correct root partition on the SD card. I ran through the GPIO tests (LEDs and switches) for the OOB design. I had a minor issue where the python libraries did not like using the concat element to aggregate interrupts rather than using an interrupt controller. I needed to go back and add an interrupt controller to the hardware design and that resolved the issue and the GPIO tests worked. I then tried using the upgraded VCU TRD BSP to add the VCU functionality. I ran through the same GPIO tests which worked, but I realized that the VCU drivers did not load and I needed to go back and modify my PetaLinux config. That seemed to be straightforward but then I started running into some odd boot related issues. At that point I decided that I needed to move on with the roadtest and come back to fight this battle another day.
I've recently discovered that there has been a process developed to upgrade PYNQv2.5 to use the DPU https://github.com/Xilinx/DPU-PYNQ . Of course, the process currently only supports the Ultra96, ZCU104, and ZCU111 boards . Since I want to incorporate the DPU functionality, I may just move forward and try to get this upgrade working with the VCU TRD when I have time.
In searching for an AI element to integrate into my project for object detection and classification, I came across a great series of tutorials that Mario Bergeron of Avnet has done on Hackster.io using Vitis AI.
I started with the Vitis AI 1.1 Flow for the DNNDK and as usual tried the pre-built image first. There are images available for the Ultra96v2, UZ7EV_EVCC, UZ3EG_IOCC, and UZ3EG_PCIEC boards.
There are example applications that are doing object detection and classification with images and video files stored on the SD card and also using a webcam input.
I tried the following examples:
The mobilenet example classified 1000 images in about 5.4s for an average frame rate of 183 fps which was quite impressive!
I have subsequently used the VART (Vitis AI Runtime) flow both from pre-built and built from hardware images, but I did not document that. The DNNDK and VART flows are similar on the build side but the VART flow requires a bit more setup for the target deployment and initialization of the runtime environment.
The VART flow is using the WAYLAND/Weston desktop rather than the X11/Matchbox desktop that was used with the DNNDK flow. One annoying aspect of the new desktop is that it doesn't allow locating the display window at a specific position on the desktop, so it moves around from run-to-run. Makes it painful for creating videos.
My plan is to use GStreamer to set up the video processing pipelines. I ran into a problem processing RTSP video streams from my IP cameras. Only one of my cameras properly executes a pipeline to decode and display an RTSP stream. The other cameras will initially display an image but it will not update. This has been a frustrating problem as the pipeline doesn't fail in an obvious way and this failure does not occur using a similar pipeline with Ubuntu Linux. I'm getting some troubleshooting help from the Avnet team and hopefully will get this figured out soon. There's always the possibility that maybe network or camera issues could be causing the problem. This blog will only be of interest to those familiar with GStreamer.
I wanted to get a measure of network performance not just of the UltraZed_EV interface but in the context of my network setup (GigE switch, etc). The test uses the iperf3 tool.
Running with TCP I was able to achieve a symmetrical 574 Mbps speed between the UltraZed-EV and the Win10 host computer connected through the Gigabit switch. The reference test configuration (point to point connection using the UltraZed-EV connected to a CentOS laptop host) was able to achieve 856-942 Mbps. I did not try a point to point connection as it would not be representative of my use configuration. I was less successful trying to run UDP at a high data rate but I discuss that in the blog.
I ran into an interesting issue testing SATA performance that was not related to the hardware but the PetaLinux OS. Plus there is also a minor issue with ease of use. The SATA interface on the board provides only a data interface connector, so that requires using an external power supply with the SATA drive. The ZCU102 board provides a power connector making an external power supply unnecessary.
I am using a 500GB SSD for this test. It came unformatted, so I formatted it as NTFS which I tend to use with larger capacity drives. I discovered that PetaLinux does not support NTFS except as Read Only. It supports FAT32 and EXT4. Windows did not give me the option to format a drive this large to FAT32 so I ended up formatting it to EXT4 using Ubuntu.
Here are the 3 sets of tests performed with their results (see blog post for test details):
4096000000 bytes (4.1 GB, 3.8 GiB) copied, 13.0229 s, 315 MB/s
Timing cached reads: 2166 MB in 2.00 seconds = 1082.55 MB/sec
Timing buffered disk reads: 1394 MB in 3.00 seconds = 464.11 MB/sec
12226 is the speed (in KBytes/sec) at which the dataset was written a single character at a time
406518 is the speed (in KBytes/sec) is the speed at which a file is written a block at a time
12239 is the speed (in KBytes/sec) at which the dataset was read a single character at a time
545003 is the speed (in KBytes/sec) at which a file is read a block at a time
My roadtest project is a subset of an Intelligent NVR that I'm designing. The project is a proof of concept that includes the following features:
Unfortunately, I'm unable to demonstrate a working project due to issues that I discuss in the blog post.
Instead I decided to demonstrate face detection/tracking using an RTSP stream with software decoding and the DPU using a VART Python based application.
This application is derived from yet another Mario Bergeron tutorial from Hackster.io that I discuss in the blog post.
I also discovered a Xilinx Embedded Platform Reference Design for the ZCU104 (8-Stream VCU + CNN Demo Design) that I may try to leverage for the Intelligent NVR.
The UltraZed-EV Starter kit is a very capable and versatile board set. I wish I could afford the GMSL cameras and interfaces. That being said, I think that it is a good cost vs performance/capability tradeoff vs the Xilinx ZCU104/ZCU106 boards.
I meant the statement as a general comment not trying to assign values to specific features. Certainly in aggregate the boards rank in features as you describe although the ZCU106 is more fully featured…
A very thorough road test. Great work.
You mention that the Ultra96v2 came with a year's licence and I just got one. I don't need the licence and would be happy to pass it over to you, but it seems to be…
Thanks for your detailed reply, Ralph. And yes i totally agree the biggest advantage of the UltraZed-EV is that it's mounted on a SOM, which makes it a lot more flexible than the ZCU104/6. Would be my go-to choice for sure, whenever I plan to build anything serious with the 7EV device.
A very thorough road test. Great work.
You mention that the Ultra96v2 came with a year's licence and I just got one. I don't need the licence and would be happy to pass it over to you, but it seems to be just a SDSoC licence which doesn't sound like it's what you need. If it is though, let me know.
I meant the statement as a general comment not trying to assign values to specific features. Certainly in aggregate the boards rank in features as you describe although the ZCU106 is more fully featured than the UltraZed-EV. Of course, the boards also rank in cost in the same order - in the Newark store - ZCU102 ($1393), UltraZed-EV ($1881), ZCU106 ($2823). It probably would have been a good addition to the roadtest to do a feature comparison of the boards. I guess at at a gross level I had indicated the boards were comparable primarily because I was focused on the VCU capability and they use the same UltraScale+ MPSoC. I think a unique advantage of the UltraZed-EV is that the ZU7EV is mounted on a SOM that could be leveraged onto a custom board.
Good, detailed road test. Thanks for sharing all the information.
Regarding the last line in your post "I think that it is a good cost vs performance/capability tradeoff vs the Xilinx ZCU104/ZCU106 boards.", do you feel this board has any performance/capability tradeoffs compared to the ZCU104/ZCU106? I myself feel the ZCU104 is more restrictive since it doesn't allow the user to use many of the Zu7EV peripherals, for example no SFP cages, no GTH SMA connectors, etc. ZCU106 seems comparable to the Ultrazed-EV though. Thoughts?
Good road test report.
Excellent review with a well explained series of blogs.
Another good road test Ralph. It sounds like an impressive board. One of the objectives of these road tests is to identify where products can be improved, so I always like hearing the problems you had to solve. For the standard road test question, Was everything in the box required?: Yes, I'm surprised you said yes. I would have been quite annoyed to find that the RTC wouldn't work as supplied, and annoyed about the HDMI licensing issues. A 30 day license is too limiting in my opinion, and isn't included until you ask.. I guess it's not required, since you could use the DisplayPort.