Having completed the 3 Avnet Ultra96 courses in the Path II Programmable training, it's now time to move on to creating a project with what we have learned. I know what I would like to do conceptually but I doubt that I can successfully implement it in the 5 week time frame we have for the project. Even though there are a wealth of examples in the Xilinx ecosystem, one thing I definitely learned from the courses are that there are huge issues with hardware and software version incompatibilities. That is to say that for a novice like myself that it will take a lot of time to figure out how to make things actually work.
At a high level of abstraction, I'd like to implement an embedded vision system. The input would be a H264/5 video stream from an IP camera or an NVR. The input would be an RTSP stream over the network (either wifi or ethernet). If I can get the base configuration working I'd like to extend it to multiple inputs and possibly an output to the NVR. I'd like to have some level of detection/identification capability. My thought is to use one of the quantized neural network implementations that I've tried on the PYNQ-Z2 (new overlays are available for the Ultra96). I thought that I could do the video decoding and encoding in the FPGA but the Ultra96 does not have a VCU (only the EV chips have one). I'm planning to use OpenCV for the video processing and there is an xfOpenCV library that accelerates many of the functions using the PL. The SDSoC tool is supposed to provide the capability for software/hardware function optimization but we did not cover that tool in the course. So, I'm struggling to figure out how to approach this. There are many examples available but many were developed with earlier tool versions and many are for the Ultra96-V1.
How to partition and connect the design is the first challenge and then to make it work. Obviously the XIlinx and Avnet folk have done much more complex designs 
.
I decided that I would try to break the problem into smaller pieces and try to verify that individual functions/components work before I try to integrate them. There are a lot of capabilities of the Ultra96 that I haven't tried (that's quite an understatement).
RTSP stream capture
I thought that this would be a good place to start. I've viewed RTSP streams from the cameras using VLC on Windows and Omxplayer on the RPi but I haven't tried it with OpenCV. I decided that I would use PYNQ to test it because you can try the code interactively. I used the Ultra96-V2 V2.5 PYNQ Image http://bit.ly/2MMrXcS that was just recently released. And I used the PYNQ-ComputerVision code that is available on github http://bit.ly/2MMrXcS .
The Jupyter notebooks that are provided in the ComputerVision repository use a webcam as the image source but it was reasonably straightforward to replace it with the RTSP stream. One thing that is somewhat tricky is that even though there are only a few hardware OEMs for the IP cameras, the vendors seem to try to differentiate their products with firmware so there is no standard syntax for the camera servers. Luckily most vendors provide API documentation but I have to use multiple formats to capture streams from the various cameras that I have. ONVIF is supposed to solve that problem but I don't think many cameras are fully compliant. The camera that I tried is an FDT 7903 which streams on port 554 and requires user and password authentication.
Here's the code changes required to go from webcam to ipcam: Image width and height are required for later processing
Setup and configure USB camera import cv2 camera = cv2.VideoCapture(0) width = 1920 height = 1080 camera.set(cv2.CAP_PROP_FRAME_WIDTH,width) camera.set(cv2.CAP_PROP_FRAME_HEIGHT,height)
Setup and configure RTSP capture
import cv2
camera = cv2.VideoCapture('rtsp://user:pw@ipaddr:554/12')
if camera.isOpened():
width = int(camera.get(cv2.CAP_PROP_FRAME_WIDTH)) # float cast to int
height = int(camera.get(cv2.CAP_PROP_FRAME_HEIGHT)) # float cast to int
print('width, height:', width, height)
Here is a capture from my driveway camera with subsequent Filter2D processing:
Display Port Output
Another capability that I had not used was the Display Port. The Ultra96 provides the Display Port interface through a mini DP connector. I used a Cable Creation Active mini DP to HDMI cable from Amazon to hook up my monitor: https://www.amazon.com/gp/product/B01FM50QJC/ref=ppx_yo_dt_b_asin_title_o04_s00?ie=UTF8&psc=1 .
I found a Display Port test project for the Ultra96-V1 on Hackster: https://www.hackster.io/eexma67/design-and-test-displayport-of-ultra96-board-from-scratch-ebee36 and decided that it would be good practice to port that to the V2. There were a few hiccups along the way. I've been happily building PetaLinux projects (lots of coffee breaks while it builds) and I was shocked when the build failed because I ran out of disk space 
. I guess each project consumes about 30GB and I have not been diligent about deleting them. When I tried out my new build, it happily booted Linux but when I tried the tricube display (spinning cube) - there was nothing displayed. It took me a while (not paying attention to the error output) to realize that my monitor was not correctly acknowledging the HDMI EDID and apparently that aborted the program. Rather than trying to figure out how to fix the software and having another coffee break I decided to try a different monitor. And happily that worked.
Here a short iPhone video of the test display on the monitor:
Now I need to put a QNN in the path......
