Embedded Video Processing Using the UltraZed-EV

It was a great webinar surely! Thanks for answering my question on thermal/hyperspectral imaging. But what I really wanted to know is how this solution handles low resolution images, as is the case with the cameras for these imaging applications. Can it perform some computations or interpolations to make it look more resolved? Can it be used to perform heavy computing tasks like combining the video feeds of a visible cam, a depth cam and a thermal cam and apply ML/AI to it simultaneously in real time?

HI there,

I'm sorry I did not get at the core of what you wanted to know. First the VCU is only a CODEC processor.  It can encode and decode the video stream.  It does not really do much processing beyond what is allowable from the video standards that I listed.  There is a little bit of blur / quality / etc. features, but I suspect that is not what you are talking about.

I'm not sure which thermal imaging cores you are using, but I recall from ~10 years ago most were in the 320x240 range.  I think a few were just starting to get to 1024x768.  Based on this, I would have to say that you would either need to use an advanced configuration, or blend multiple streams together.  The VCU has limits on what it can easily do as it is a piece of hardened IP.  One of the limits is on the smaller side as everything has multiples of logic.  According to PG252 page 19, "AVC minimum picture resolution: 80×96; HEVC minimum picture resolution: 128×128"   This, as you probably saw in the webinar is smaller than the resolutions listed in the pull down menu and is handled outside the configurator.

I suspect that you are really asking, how you can use the VCU to make your video look better.  It can't.  It is only a codec.  What you would need to do is leverage the programmable logic.  This is exactly why the TRD is setup as it is.  As I demonstrated, the design uses the MINIMUM of the programmable logic and CPU, which lets you put in whatever you want between the video coming IN and the video going OUT.  Instead of taking the data stream DIRECTLY from the WEBCAM to the VCU, you could instead pre-process the data stream when it hits the programmable logic using either built in IP, home grown code, or even an HLS based accelerator (SDSoC / Vitis / Etc.).  From there, you can filter, interpolate or add whatever you wanted to the data stream (using OpenCV or similar).  This also means you can leverage things like Python on a PC to test out the OpenCV functions and really decrease the time it takes to come to a solution.  That is, the Python on a PC might not get the FPS you need, but can prove your algorithm.  Then leverage the PL (programmable logic) to get the performance - knowing that the algorithm is correct.

As for can it be used for heavy computing tasks, absolutely!  I have seen people do some REALLY amazing things with AI/ML on the Ultra96V2.  That part does not have even 1/2 of the resources that the UltraZed-EV has.  The amount of IO, type and amount of DDR memory, the  access type (UZ-EV is faster), the amount of logic cells, access to UltraRAM, etc. can all be leveraged to perform all of the things you asked for - and yes in real time.  It is an incredibly powerful and capable system.  For example, I've seen a customer take a slew of video streams into one of these over PCIe and the 7EV had room to spare for future expansion.  Really it just depends on how much of the system you need to utilize for your logic and processing.

As for blending the various cameras, I would say that is a non-issue.  We have a design for the PicoZed (much MUCH smaller than the Ultra96V2), which takes thermal and visible and blends them together and puts out video to HDMI.  That fit on a Zynq 7020.  As a gut feel, having actually written an interpolation engine for an FPGA from scratch for a 320x240 thermal camera, and knowing the rough resources used for that - I would have no issue telling my manager that "yep we can do that".

--Dan

HI there,

I'm sorry I did not get at the core of what you wanted to know. First the VCU is only a CODEC processor.  It can encode and decode the video stream.  It does not really do much processing beyond what is allowable from the video standards that I listed.  There is a little bit of blur / quality / etc. features, but I suspect that is not what you are talking about.

I'm not sure which thermal imaging cores you are using, but I recall from ~10 years ago most were in the 320x240 range.  I think a few were just starting to get to 1024x768.  Based on this, I would have to say that you would either need to use an advanced configuration, or blend multiple streams together.  The VCU has limits on what it can easily do as it is a piece of hardened IP.  One of the limits is on the smaller side as everything has multiples of logic.  According to PG252 page 19, "AVC minimum picture resolution: 80×96; HEVC minimum picture resolution: 128×128"   This, as you probably saw in the webinar is smaller than the resolutions listed in the pull down menu and is handled outside the configurator.

I suspect that you are really asking, how you can use the VCU to make your video look better.  It can't.  It is only a codec.  What you would need to do is leverage the programmable logic.  This is exactly why the TRD is setup as it is.  As I demonstrated, the design uses the MINIMUM of the programmable logic and CPU, which lets you put in whatever you want between the video coming IN and the video going OUT.  Instead of taking the data stream DIRECTLY from the WEBCAM to the VCU, you could instead pre-process the data stream when it hits the programmable logic using either built in IP, home grown code, or even an HLS based accelerator (SDSoC / Vitis / Etc.).  From there, you can filter, interpolate or add whatever you wanted to the data stream (using OpenCV or similar).  This also means you can leverage things like Python on a PC to test out the OpenCV functions and really decrease the time it takes to come to a solution.  That is, the Python on a PC might not get the FPS you need, but can prove your algorithm.  Then leverage the PL (programmable logic) to get the performance - knowing that the algorithm is correct.

As for can it be used for heavy computing tasks, absolutely!  I have seen people do some REALLY amazing things with AI/ML on the Ultra96V2.  That part does not have even 1/2 of the resources that the UltraZed-EV has.  The amount of IO, type and amount of DDR memory, the  access type (UZ-EV is faster), the amount of logic cells, access to UltraRAM, etc. can all be leveraged to perform all of the things you asked for - and yes in real time.  It is an incredibly powerful and capable system.  For example, I've seen a customer take a slew of video streams into one of these over PCIe and the 7EV had room to spare for future expansion.  Really it just depends on how much of the system you need to utilize for your logic and processing.

As for blending the various cameras, I would say that is a non-issue.  We have a design for the PicoZed (much MUCH smaller than the Ultra96V2), which takes thermal and visible and blends them together and puts out video to HDMI.  That fit on a Zynq 7020.  As a gut feel, having actually written an interpolation engine for an FPGA from scratch for a 320x240 thermal camera, and knowing the rough resources used for that - I would have no issue telling my manager that "yep we can do that".

--Dan

Thanks a lot for your detailed answer, it surely gave me a lot of clarity. I have been working on mathematically trying to improve resolution of a thermal camera by using programming logic and math (like bilinear interpolation, which has been done by someone on the internet). I saw in one of the videos on your site where 4 ML models were running on the same deep learning engine - It will really be helpful in places like India where the amount of humans or objects are very large, and if one actually wishes to run this system on a place like railway stations, which has multiple security cams, roughly 30 on a major station, this system can really find a place. They use nothing currently but manual surveillance there.

Whoa... hi Dan, you almost summarised everything through this comment. This webinar made me to think twice of GPU's fate against the U-96EV's VCU, specially for the ML based embedded systems.

-Ankur