July 4, 2020
I had hoped to finish the VCU TRD design tests this week and move on to taking a closer look at the design and using the design tools to tweak it to verify that I have a good tool setup.
Unfortunately, even the simplest things sometimes turn out to be difficult and I have a couple of problems that I haven't resolved yet.
July 17, 2020
Well, here I am two weeks later. Some personal difficulties and other problems that I had not anticipated. But I guess relative to the issues our world is facing I shouldn't complain....
I've had a few bumps in the roadtest that I've been working through and haven't gotten around to documenting my progress. Here are the last two tests from the VCU TRD.
Use a USB Webcam with gstreamer v4l2src plugin and output to DisplayPort
For my testing I am using an eMeet C980 HD WebCam. I have it connected through a powered USB3 hub that I also use to connect a USB drive.
gstreamer pipeline:
[WEBCAM] ->
|--> [V4L2SRC] -> [QUEUE] -> [KMSSINK(DISPLAY PORT)]
Use v4l2-ctl to list available devices: v4l2-ctl --list-devices
Get the WebCam capabilities: v4l2-ctl -d /dev/video0 --list-formats-ext
Use the v4l2src element to interface the WebCam and gstreamer element properties to specify the stream resolution and format: v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=1280,height=720,framerate=10/1
Use the kmssink element to send decoded video to the DisplayPort: kmssink bus-id="fd4a0000.zynqmp-display" fullscreen-overlay=true
The complete pipeline: gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=1280,height=720,framerate=10/1 ! kmssink bus-id="fd4a0000.zynqmp-display" fullscreen-overlay=true
Use USB Webcam and VCU encoder to output to both the DisplayPort and an RTP Network stream simultaneously
This test uses a tee element to add a parallel pipeline to send out the video as an RTP network stream over Ethernet. The stream will be received and displayed using the VLC media player.
gstreamer pipeline:
[WEBCAM] ->
|--> [V4L2SRC] -> [TEE] -> [QUEUE] -> [VIDEOCONVERT] -> [OMXH264ENC] -> [QUEUE] -> [RTP] -> [UDPSINK]
|----> [QUEUE] -> [KMSSINK(DISPLAY PORT)]
Use the tee element to provide the parallel streams and add a data buffer in each path: tee name=tee_preview ! queue
The DisplayPort path is the same as before following the tee: tee_preview. ! queue ! kmssink bus-id="fd4a0000.zynqmp-display" fullscreen-overlay=true
The VCU Encoder pipeline is a bit more involved.
Convert format to a supported VCU colorspace and specify encoding parameters: videoconvert ! video/x-raw,format=NV16,width=1280,height=720,framerate=10/1
The videoconvert element is required because
The camera data format is YUYV (YUY2) which is 4:2:2
The H264 encoder only supports NV12 8-bit/10-bit YUV 4:2:0 and NV16 8-bit/10-bit YUV 4:2:2
Add the OpenMAX H264 encoder element and post encoding buffer: omxh264enc ! queue
Next setup the RTP Network Stream. The video data will be transmitted over the network using UDP to minimize latency.
Use the rtph264pay element to setup the H.264 payload (let gstreamer auto-negotiate properties): rtph264pay config-interval=10
Setup the Network Socket: udpsink host=192.168.1.103 port=5000
Use the udpsink element to setup a UDP socket to transfer the RTP stream
Specify the IP Address of the destination for the video
Specify the network port used
Here is the completed gstreamer pipeline:
gst-launch-1.0 -v v4l2src device=/dev/video0 ! \
video/x-raw,format=YUY2,width=1280,height=720,framerate=10/1 ! \
tee name=tee_preview ! queue ! \
videoconvert ! video/x-raw,format=NV16,width=1280,height=720,framerate=10/1 ! \
omxh264enc ! queue ! \
rtph264pay config-interval=10 ! \
udpsink host=192.168.1.103 port=5000 \
tee_preview. ! queue ! \
kmssink bus-id="fd4a0000.zynqmp-display" fullscreen-overlay=true
The first test that I did used the VLC Media player on a Windows 10 PC to receive the RTP stream. This requires creating a Session Description Protocol (SDP) file so that VLC can properly decode the RTP stream.
Here is a description of the contents of the SDP file:
Session description
v= (protocol version)
o= (originator and session identifier)
s= (session name)
i=* (session information)
u=* (URI of description)
e=* (email address)
p=* (phone number)
c=* (connection information -- not required if included in
all media)
b=* (zero or more bandwidth information lines)
One or more time descriptions ("t=" and "r=" lines; see below)
z=* (time zone adjustments)
k=* (encryption key)
a=* (zero or more session attribute lines)
Zero or more media descriptions
Time description
t= (time the session is active)
r=* (zero or more repeat times)
Media description, if present
m= (media name and transport address)
i=* (media title)
c=* (connection information -- optional if included at
session level)
b=* (zero or more bandwidth information lines)
k=* (encryption key)
a=* (zero or more media attribute lines)
The information needed to create the SDP file is contained in the GStreamer video stage information that is output in the verbose (-v) mode.
The details for constructing the SDP file are provided in the test documentation.
Here are contents of the SDP file used for this RTP stream:
v=0
o=- 3762381622 3353214112 IN IP4 192.168.1.109
s=USB Webcam OMXH264ENC RTP Stream 720p10
c=IN IP4 192.168.1.109
m=video 5000 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmpt:96 media=video; clock-rate=90000; encoding-name=H264; packetization-mode=1; sprop-parameter-sets=J3oAH70AzkAUAW7AUoKCgqAAAAMAIAAAAwKWAB8Au///gUAA\,KOtjywA\=; timestamp-offset=3550796446; ssrc=330783705; a-framerate=10
Short video of RTP stream to VLC:
There was quite a bit of latency (delay) in the VLC test so I thought that I would do a second test using GStreamer to receive the RTP stream.
For this test I am using an Ubuntu 16.04 Virtual Machine running on the same host PC that is used for VLC.
Here is the GStreamer Pipeline used: gst-launch-1.0 -v udpsrc port=5000 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96 ! rtph264depay ! h264parse ! queue leaky=1 ! avdec_h264 ! videoconvert ! autovideosink
The udpsrc element is used provide a socket to receive the RTP stream
The rtpdepay and h264parse elements are used to unpack and format the stream for the avdec_h264 decoder
The videoconvert element converts the colorspace for the display
Short video of RTP stream to GStreamer udpsrc:
Receiving the RTP stream using GStreamer had a lot less latency but it did have some data loss. I'll need to add some buffering which will increase the latency.
The encoder settings can be optimized by explicitly specifying the parameters and I tried that example. It will take me a while before I understand the trade-offs on setting those parameters.
omxh264enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=20000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true latency-mode=normal
Summary
This concludes the test suite for the 2018.3 VCU_TRD port. These tests were run using a prebuilt image that was provided. As I mentioned earlier, the documentation and design provided were excellent. Definitely helped to get a feel for the VCU and the UltraZed-EV.
GStreamer is essential to leveraging the video processing capabilities of the UltraZed-EV and I've underestimated the amount of time and effort using it and debugging it is going to take. I've primarily used GStreamer on the Raspberry Pi 3B+ with Raspbian and on Intel PCs with Ubuntu 16.04. Going through this testing exercise I've discovered how sensitive the pipeline to the specific hardware and OS that it is running on. That's probably the subject of another post but it was something that I had naively not fully anticipated.
I've been working on testing builds of the UltraZed-EV with Vivado and PetaLinux for different tool versions and also porting PYNQv2.5. I'll cover that in separate posts.
Links to previous posts for this roadtest: