UltraZed USB 3 IDS Camera Speed Issue

Hi John,

First, have you seen this blog article I posted?

Solve USB 3.0 issues with this one weird trick

Second, do you have a local XFAE you are working with?  If so, you should contact them about this issue.  Please also have them reach out to me as I can help provide some guidance to them.

In the mean time, it is strange that you ask this at this time!  I just responded to another user where I explained that I had used a USB 3.0 camera with an Ultra96V2 (I realize not the same, but it is the same basic circuit).  I recall using that at high resolution, but I admit that I cannot say what the bit rate was.  Only that it was a CU30 (3.4MP) and was likely running at 1080P60.  I realize not apples to apples, but it just seemed strange two posts related to USB and video!

https://www.e-consystems.com/ar0330-lowlight-usb-cameraboard.asp

--Dan

Hi John,

First, have you seen this blog article I posted?

Solve USB 3.0 issues with this one weird trick

Second, do you have a local XFAE you are working with?  If so, you should contact them about this issue.  Please also have them reach out to me as I can help provide some guidance to them.

In the mean time, it is strange that you ask this at this time!  I just responded to another user where I explained that I had used a USB 3.0 camera with an Ultra96V2 (I realize not the same, but it is the same basic circuit).  I recall using that at high resolution, but I admit that I cannot say what the bit rate was.  Only that it was a CU30 (3.4MP) and was likely running at 1080P60.  I realize not apples to apples, but it just seemed strange two posts related to USB and video!

https://www.e-consystems.com/ar0330-lowlight-usb-cameraboard.asp

--Dan

Thanks for the info Dan.  Actually that blog article got me as far as I have gotten so far - getting a USB cable that works with the UltraZed (got the cable/adapter from Amazon).  With the right cable the camera was recognized as a USB 3.0 device.  Since I posted this earlier today, I ran the UltraZed performance test with a USB 3 toaster and a SSD on both a Linux desktop and the UltraZed.  Both performed about the same - around 300MB/s for read and write (I need only around 130MB/s).  So that tells me the USB 3 is performing at good speeds (at least for a SSD drive anyway) and the issue may be more camera/driver related.

Camera I'm using:

https://en.ids-imaging.com/store/ui-3140cp-rev-2.html

I will look through the article again for any other ideas just in case.  Thanks!

John,

Now you got me thinking...watch your processor loading.  I've seen where dropping everything on 1 core can make funny things happen as well.  IIRC, that is why our Webcam interface on the UltraZed-EV VCU TRD Port uses 2 or 3 threads to spread the load across the various processors.  I can't be certain how many threads, but I do recall we split the gstreamer path up to a few cores.

--Dan

Finally made some progress.  After changing the frame size to a power of 2 (512x512) and using a powered USB hub, we are able to meet framerate (500 fps).  However, we are seeing random failures with the USB.  It can happen after running for 15 minutes or it can happen after running for 2 minutes.  The error messages are:

[   83.289070] xhci-hcd xhci-hcd.0.auto: Stopped the command ring failed, maybe the host is dead

[   83.313647] xhci-hcd xhci-hcd.0.auto: Abort command ring failed

[   83.319478] xhci-hcd xhci-hcd.0.auto: HC died; cleaning up

[   83.319527] xhci-hcd xhci-hcd.0.auto: Error while assigning device slot ID

[   83.319531] xhci-hcd xhci-hcd.0.auto: Max number of devices this xHCI host supports is 64.

[   83.319536] usb usb1-port1: couldn't allocate usb_device

[   83.345402] usb 2-1: USB disconnect, device number 3

Hi John,

Do you have a local XFAE you are working with?  If so, you should contact them about this issue.  Please also have them reach out to me as I can help provide some guidance to them.

--Dan

Update.  I think the camera works fine on the UltraZed board.  Our goal was to capture and stream frames across the network.  But that would kill the USB controller after a while.  So as a test we have an app that just captures the camera images and does nothing with them.  This will run for days without issue.  We then created an app that just streams dummy frames (totally independent of the camera app).  This app will run alone for days without issue.  But when we run them together, within an hour or so, the USB controller dies with the message below.  We cannot figure out why using the network device causes the USB controller to die.

xhci-hcd xhci-hcd.0.auto: xHCI host controller not responding, assume dead

Hi there,

Wow, what a find.  I will ping a few people, but in the mean time, is there any suspicion on power draw / issues around the USB device?  I have no proof, and this might be a red herring, but my first gut feel with this new information is, could the device be getting marginal power?  Then eventually it just...hiccups?

I recall earlier you saying you resolved some issues by just adding a USB Hub.

Are you also leveraging the VCU?  PL DDR memory?

Have you checked the power estimator in Vivado?

Again, not trying to side-track you, but I was just musing about this.

--Dan

I know power definitely was an issue at first.  The camera people suggested a specific ("approved") USB hub for their camera (IDS uEye).  And this solved our initial issues.  I will have our Vivado expert look into your questions.  There must be some low-level resource (power, clocks, etc.) that the network device and USB device share for this to be an issue.  I have even tried the 2020.1 tools (Avnet Petalinux, Xilinx Linux/u-boot, with Ubuntu 18.04, etc.) in hopes there was a driver fix or something, but still same issue.  We are using the UltraZed EG board.

Thanks for your help!!

FYI, new data this morning related to power.

There are two processes.  The camera process just acquires frames and discards them.  The publisher process is streaming simulated images to N clients at once over the network (TCP).  Each process is totally independent of the other.

I connected a USB power monitor to the camera connection.

Power observations:

Neither process running: 5.09V, 0.280A

Only camera process: 5.02V, 0.544A

Start camera process after publisher with 5 clients: 5.02V, 0.494A, then suddenly 5.14V, 0.137A

Publisher with 1 client – best case will run for close to 18 hours

Publisher with 2 clients - best case will run for close to 8 hours

Publisher with 3 clients - best case will run for a couple hours

Publisher with 4 clients - best case will run for a few minutes

Publisher with 5 clients - best case will run for a few seconds

Hey John,

Dan asked me to take a look at this as well as I'm the power expert on the team. Based on what you saw on the power monitor, it would be reasonable to assume that there could be a decent size transient when the load jumps. There is an experiment you could to try to see if it is a lack of 5V power capacity from the IO carrier causing your issue. J6 on the carrier board has a jumper shorting pins 1-2. If you remove that jumper, it removes 5V power from the generating source and leaves the USB+ rail unpowered. This feature allows both Host and Device mode operation. If this jumper is removed, you could provide 5V from an external supply to pin 2, assuming you have an external supply you could use. That would allow you to use a high current 5V supply as an experiment to see if a power limitation was causing the issue. If the design ran without issue using an external high current source, that would point pretty hard at a power limitation as the source of your problem. Let me know if you think you may be able to try this and need any assistance in setup.

Thanks
Chris

Chris,

Thanks for the info.  We have a power supply connected to the board and one connected to the J6 pin 2.  We still see the same behavior even with the external 5V power.  There is no current draw though since the powered USB hub is powering the camera, however without the 5V on pin 2, the USB hub itself doesn't provide enough current for the camera to fully function.  The other thing we have tried is measuring power draw when writing to the network.  With nothing running (except typical Linux processes), the current level is around 0.66A.  As we connect a client (server on the board writes fake images), the current level increases by 0.01A so after 4 clients are connected it's at 0.70A.  At this point after a few seconds/minutes, the xCHI controller dies and the kernel panics and the board locks up (all the typical LEDs are still lit though).  We then tried to get the current draw at 0.70A by something other than the network.  Using the stress Linux tool, we were able to get the draw at 0.71A and so far haven't seen the issue - of course, we haven't ran it for several hours yet (maybe an hour), but with the network test it only takes a minute or less to see the issue.  So we were struggling to figure out if the sudden current drop was causing the issue or is it a symptom of the issue.  Since the stress test hasn't caused the issue thus far, it's looking like the sudden current drop is due to the xCHI controller and camera dying.

Let us know if you can think of any other tests we can perform.  Thanks for your help!