Problem with PYNZ-Z2 and HDMI output from camera

Question

I'm currently making sure I'm in a good position to hit the ground running with the upcoming PYNQ-Z2 workshops. I bought the recommended kit including an Apeman A77 camera and thought I'd just have a quick poke around in advance. I ran through the "getting started" and the Pynq 2.5 image includes some simple HDMI examples. (I hope I'm not jumping too far ahead, but like to be prepared.)

Anyway, as I start to run the HDMI example I get a kernel died error (as shown). You can see on screen what's been run. This is just wiring HDMI out to show what's on HDMI in. The error is completely repeatable. Nothing noteworthy in /var/log/jupyter.log to give any clue.

As quick Google suggested that maybe the image is too high resolution, and when I checked the camera on my TV it outputs 1080p at 60fps regardless of the resolution of video being captured. Could 1080p be too much? Time to switch to a Raspberry Pi 4 where I can change resolution properly. The above example ran quite happily with the Pi at a lower resolution, but surprised me when it also ran using the Pi at 1920x1080 at 60fps too.

Is this camera (Apeman A77) compatible with the HDMI on the PYNQ-Z2? I'd have expected it to be as it was the recommended kit and the Crosstour one listed now is essentially the same model.

I'll try to do more digging into any subtle differences between the HDMI output of the Pi and the camera, but all I have is a TV that shows some info on the input. Anyone else have any ideas?

beacon_dave · Answer

Are you aware that the HDMI out IP provided in the PYNQ-Z2 the base overlay doesn't fully support 1080p video ?

https://pynq.readthedocs.io/en/v2.5.1/pynq_overlays/pynqz2/pynqz2_base_overlay.html#hdmi

HDMI Out¶

The HDMI out IP supports the following resolutions:
640x480
800x600
1280x720 (720p)
1280x1024
1920x1080 (1080p)*
*While the Pynq-Z2 cannot meet the official HDMI specification for 1080p, some HDMI devices at this resolution may work.

dimiterk · Answer

PYNQ Z2 implements the HDMI Codec for transmit and receive sides as IP blocks on the PL side wired via TMDS configured pins to buffers and then to the HDMI connectors. This is presumably done to save on cost as official development boards use external chipset codecs from ADI/Silabs which are configured via I2C. The downside of this is that it's using a couple of hundred LUT's from FPGA fabric. The HDMI INPUT on the above example is NOT connected directly to HDMI out port strictly speaking. According to the PYNQ Vivado HW design for this overlay, the HDMI input is piped to the processor DDR memory and via VDMA is then transferred to the HDMI out IP block. This allows breaking the loop so that one can implement image processing algorithms on the PS side using OpenCV. I don't have a PYNQ-Z2, but it looks like that notebook has not been updated for PYNQ V2.5.

adamtaylorcengfiet · Answer

In your case with the apeman camera you need to set the resolution to anything below 1080P quite easy to do via the menu, I usually run mine at 720P with no issues In the lab we have developed for this course we are using the VGA settings which is how the VDMA is configured.

Fred27 · Answer

Adam, - thanks for your input. Unfortunately changing the resolution that the camera records at does not change the resolution of the HDMI output. This is always fixed at 1080p on my Apeman A77 - whether it's recording in 720p or in 4K. I think I'll go back my original "Plan A" of using a Raspberry Pi and Pi camera as a HDMI video source. That's working fine.

Fred27 · Answer

I wasn't aware of those details. Maybe that's it - a borderline implementation of 1080p where output from the Pi 4 works and the camera doesn't. If that's the case it's frustrating that a recommended camera (although the exact model Apeman camera wasn't specified) is one that doesn't work.

If it's output that's a problem, maybe after processing the frames it could work? That simple example streams straight from HDMI in to HDMI out. I'll dig a bit more later, but right now I've promised to play football with the kids and put up a tent so we can camp in the garden tonight and pretend we're on holiday!

beacon_dave · Answer

I think that the base overlay supplied is pretty much a basic routing of the IO pins connected to the PL fabric to the input to the PS. So no real 'heavy lifting' being done with FPGA IP at this stage.

There appears to be a video processing overlay available however which starts to make use of the PL for IP blocks to provide video processing off-load, especially when using OpenCV on the PS side of things.

The two HDMI ports on the PYNQ-Z2 although labelled as input and output appear to be are wired directly to the FPGA fabric, so their true direction is presumably only defined by the configuration of the IP blocks. If the input IP meets the 1080p spec but the output IP doesn't, it perhaps suggests that it is some sort of marginal timing or memory limitation with the base overlay implementation.

The camera may be outputting component YPbPr colour space whereas the R-PI component RGB.

Perhaps good questions for Adam in the workshop sessions.

If the HDMI input IP in the base overly fully meets the HDMI spec for 1080p, then the recommended Apeman camera output may not be an issue. The workshop may never route the HDMI input directly to the HDMI output using the base overlay as you are doing here.

Another potential interesting issue with HDMI projects using FPGA devices is the storage of EDID information as this is supposed to be available from an HDMI sink device even in the powered off state. The HDMI source actively provides power to the sink device EEPROM in order to be able to return this EDID information. However with a FPGA device it typically needs to be powered up and the bitstream loaded to configure the fabric in order to create the memory to store the EDID information before it can be retrieved. So perhaps an external I2C EEPROM device still requires to be added to the design for this.

I recall hearing stories of families doing the camping in the back garden thing, yet ending up spending most of their time too'ing and fro'ing between the tent and the house.

beacon_dave · Answer

Can you disable the embedded audio on the A77 camera ? May be worth a try if you can.

ralphjy · Answer

Hi Fred, I saw this discussion but I haven't used my PYNQ-Z2 in a while and I'd never used it with an HDMI source. Since I've only tried PYNQ V2.5 with my Ultra96-v2 I thought I should try it out with the PYNQ-Z2 before the workshop. I have an Apeman A80 and I tried it out with that same base_overlay_video notebook that you tried. I have the camera resolution set for FHD P60 and it seems to work fine. Here's the image capture of my desktop from further down the notebook: The only time I killed the kernel was when I tried to execute the following cell before the base overlay had completed loading (I wasn't paying attention...). Just thought that I'd let you know that my camera is working. Ralph

beacon_dave · Answer

Status update. The Apeman A66 camera I ordered has since completed its tour of Europe (and the English Lake District) and finally ended up on my door step this afternoon. That's where the good news ends. Unfortunately the A66 doesn't appear to work with the PYNQ-Z2 as a live video source, so another one to avoid for this workshop. The best I was able to get it to do was to display some low quality still photographs from its microSD card via the PYNQ-Z2 and only after first establishing the video connection on the PYNQ-Z2 using a low-end CP-255D video scaler as an HDMI source first. It looks like the micro HDMI output on the A66 camera is only intended for playback of still photos from its microSD card. For live video, it would appear that you need to use the USB webcam mode. The A66 camera was able to detect the PYNQ-Z2 initialising the HDMI interface as it switched to external display mode whilst executing the code in the Jupyter notebook. In addition, the PYNQ-Z2 demo code wouldn't work with the HDMI solid state media player or the HDMI DVD player I had to hand either, although I didn't try connecting either of them after establishing the video connection with the scaler.

ralphjy · Answer

It seems odd that the A80 would work and the A66 and A77 wouldn't, but I guess they could have changed the interface. It seems really odd that the A7X cameras don't have the ability to shut off the OSD. I noticed that the A66 does have that setting. Are we missing something? I haven't seen any complaints about it but I also haven't seen any reviews using the HDMI out. I tried the OKCAM app on my Android tablet and it's painful to use, but it does work to operate the camera remotely. I also did verify that if I run the camera at a higher resolution than 1080p60 that I do kill the kernel. I'll consider myself fortunate that my camera seems to work. I'll keep my fingers crossed for the workshop. My tablet seems to have an HDMI output so I'm also going to give that a try.

beacon_dave · Answer

Yes, it is odd that there appears to be significant differences in the operation between the various camera models. The OSD option is there on the A66. There are 5 screens of option settings but nothing obvious to get live camera video out of the HDMI port. As soon as the PYNQ-Z2 is configured to use the HDMI ports, the camera's on-screen display briefly switches from live video mode to photo viewer mode then turns itself off. Once displaying on HDMI output, the camera buttons only allow you to cycle through the stored images; there doesn't appear to be any way to access the settings menu or put it back in camera mode. How does this behaviour compare with the A80 ? Does the A80 automatically go into photo viewer mode or does it remain in the mode the camera was in before connecting the display ? Once viewing on the HDMI display, do all the camera buttons on the A80 operate the same as they do when using the internal display ? Does the A80 have the USB webcam mode ? I'm just wondering if some HDMI features were lost when they implemented that feature. Something I need to try is what happens on the HDMI output when the camera is in USB webcam mode. You however need the camera to be connected to a host computer via USB before you get the settings for mass storage or webcam use. What doesn't really help is all the auto standby features of modern equipment. The camera needs to see the EDID or CEC signal from the display device to wake up its HDMI output. The display needs to see a valid sync signal from the source device to come out of standby. PYNQ expects to see a stable input signal to avoid killing the kernel. Ideally could do with a couple of PYNQ notebooks to test one half at a time, e.g. camera to PYNQ-Z2 and PYNQ-Z2 to display, with options for the PYNQ-Z2 to output EDID info from its HDMI input, or to pass the EDID through from the display via its HDMI output. I have a Kodak PlaySport action cam which has a mini HDMI output but unfortunately it has become rather temperamental and no longer reliably powers up. Shame as it was a robustly built camera otherwise.

ralphjy · Answer

A quick update: Thought I'd try seeing if there was an HDMI output when in USB Webcam mode - did not get past square one - the connectors are spaced too closely together to plug in simultaneously. I had a similar problem with my LapDock except the cables were both female and in this case they are both male. It was quite a pain to shave down the connectors and I don't have any cables to spare so I'm going to quit. Let me know if you get to manage to try this. Also tried my Android tablet (Dragon Touch K10). The HDMI out works fine with my Dell U2518 but using the setup in the PYNQ-Z2 the image was noisy (a small smattering of off color pixels). Also the display was rotated 90 degrees. It isn't rotated on the Dell. I guess at some point this will all make sense .

beacon_dave · Answer

Yes, I also discovered that the connectors are spaced rather close together for this test. In the meantime I've ordered up one of the latest recommended Crosstour action cameras, as one of the sellers appears to suggest that they will do live video continuously out of the micro HDMI port: however that was posted over a year ago, so if they have undergone the same firmware updates as the Apeman range of cameras appear to have, then I may still be no further forward. ( If that turns out to be the case then at least I'll be able to use the second camera to video the other camera not working... ) I also looked around at a number of other camera options but most appear to be a bit vague on this aspect. Some of the FPV forums appear to have similar issues in identifying suitable cameras. Some higher end helmet cameras and microscope type cameras appear to offer the functionality but at around 10x the cost. It's good to identify these sort of issues prior to the workshop however as a lot of time can be lost answering questions trying to get people's setups to work. As for the Android tablet, I've read about image orientation issues in machine vision applications before. Here is one article where I saw it mentioned: https://medium.com/@ageitgey/the-dumb-reason-your-fancy-computer-vision-app-isnt-working-exif-orientation-73166c7d39da

Fred27 · Answer

My A77 has a 2" screen. It does support live video out (with some OSD constantly overlaid on top) and my TV is happy to display this, just not the PYNQ-Z2. To be honest, I've given up on the camera as an input for these workshops. It's fine as a camera though.

ralphjy · Answer

My A80 has a 2" screen and has V3.1.0.1-A firmware. I saw a review that indicates there is an update to V3.1.0.2-A to fix a zooming issue when using EIS . The HDMI output works as I expected - the display screen blanks and the output goes to the HDMI port. The current mode setting does not change and all of the buttons work as normal - just using an external display. It will output live or recorded content with OSD on or off. The USB webcam also works (I haven't tried both simultaneously - using different computers - may try later). The only issue I have with the USB is that once I set the mode (Mass Storage vs Webcam) - I have to power cycle to change the mode. With the PYNQ-Z2 the only issue I've had is forgetting to disable the screensaver. The HDMI out shuts off when the screensaver kicks in and I didn't notice because the camera display was off.

beacon_dave · Answer

The Crosstour CT7000 camera arrived this afternoon but the PYNQ-Z2 base_overlay_video notebook example isn't working with it either. Can't get it to route through to the HDMI output or to get to capture a frame. As with the Apeman A66 it is detecting that there is an external device connected as it is switching to external display mode. This camera does support FPV mode however and will output live video to a directly connected HDMI display. Inserting a scaler between the camera and the PYNQ-Z2 and the base_overlay_video examples start to work but the scaler doesn't handle signal correctly from the camera. So 'close but no cigar' . There doesn't appear to be an option to disable the on screen display like there was with the Apeman A66, and there doesn't appear to be an option to change the output resolution of the live video.

Problem with PYNZ-Z2 and HDMI output from camera

Top Replies

HDMI Out¶