PCIe calibration issues on UZ-EV-CC board.

Update: I was able to generate a dts using this: https://support.xilinx.com/s/article/1157624?language=en_US

And this: github.com/.../device-tree-xlnx

Not to be confused with this: https://github.com/Xilinx/system-device-tree-xlnx

which of course I did. The bottom one is updated more recently, is that for the new SOCs from AMD?

Anyway, following the above directions generated the dtb based on 2020.1 (?), which is a bit out of date, since I'm using head of linux-xlnx which is kernel 6.1.0.

It does boot and the same number of devices work as the dtb I grabbed from peta-linux 2022.2.

But video and pcie still do not work although the error messages for video have changed a bit.

Based on the fact that they are trying to start, there must be something else wrong. It is possible that the u-boot (which is the u-boot that shipped with the board in 2017) may not be initializing the devices properly because they were not checked which means I have to wrangle u-boot next... ugh. I was hoping to avoid that. u-boot is so brick-able.

Here are the messages I'm getting from dmesg for video and pcie:

[    1.298239] nwl-pcie fd0e0000.pcie: host bridge /amba/pcie@fd0e0000 ranges:
[    1.298269] nwl-pcie fd0e0000.pcie:      MEM 0x00e0000000..0x00efffffff -> 0x00e0000000
[    1.298285] nwl-pcie fd0e0000.pcie:      MEM 0x0600000000..0x07ffffffff -> 0x0600000000
[    1.298390] nwl-pcie fd0e0000.pcie: Link is DOWN
[    1.298613] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
[    1.298622] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.298631] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xefffffff]
[    1.298638] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
[    1.298670] pci 0000:00:00.0: [10ee:d021] type 01 class 0x060400
[    1.298741] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    1.299966] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
[    1.303613] zynqmp-display fd4a0000.zynqmp-display: error -ENOENT: failed to get reset: -2
[    1.311918] zynqmp-display: probe of fd4a0000.zynqmp-display failed with error -2

If someone who has these things working could give me a pointer, it would really save me a lot of time.

I really don't like how petalinux overwrites the zyncmp.dtsi and others in the kernel source tree, this breaks the others who rely on it. There are so many ways to make that better.

I think I'll switch to the head version of device-tree-xlnx and see if that does a better job. After all the pcie calibration might still be broken on that version... Or perhaps calibration is broken in the stock u-boot which means I'm out of luck anyway.

Thanks in advance for any bones you might have on your table of ZyncMP tricks.

Update. I got u-boot to build and best of all u-boot-xlnx has the Ultrazed-ev-cc built in so I can just select it.

Unfortunately it's not using the correct dtb file (as far as I can tell, the one for the card carrier is 33KB and the one bundled with u-boot is 3.9 K) so I have to figure that out... Once I do, I'll program it onto the board and hope the backup boot flash works if I brick it. I can always boot sdcard (assuming I have one that works...).

Update. I got u-boot to build and best of all u-boot-xlnx has the Ultrazed-ev-cc built in so I can just select it.

Unfortunately it's not using the correct dtb file (as far as I can tell, the one for the card carrier is 33KB and the one bundled with u-boot is 3.9 K) so I have to figure that out... Once I do, I'll program it onto the board and hope the backup boot flash works if I brick it. I can always boot sdcard (assuming I have one that works...).

Update.

I booted example design 2022.2. There are reams of dmesg errors so I couldn't see what the system reported during boot up. But lspci -v showed the same thing as above.

I copied the dts source files for 2022.2 into my build of the xilinx kernel 6.1.0 and was able to build the dtb and the kernel and boot into linux.

That yielded the same results I've seen all along "Link is DOWN".

So I enabled pcie debug in the kernel.

Just a little more info:

[    1.286885] gpio gpiochip4: (zynqmp_gpio): not an immutable chip, please consider fixing it!
[    1.288177] nwl-pcie fd0e0000.pcie: host bridge /axi/pcie@fd0e0000 ranges:
[    1.288195] nwl-pcie fd0e0000.pcie: Parsing ranges property...
[    1.288213] nwl-pcie fd0e0000.pcie:      MEM 0x00e0000000..0x00efffffff -> 0x00e0000000
[    1.288229] nwl-pcie fd0e0000.pcie:      MEM 0x0600000000..0x07ffffffff -> 0x0600000000
[    1.288337] nwl-pcie fd0e0000.pcie: Link is DOWN
[    1.288565] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
[    1.288574] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.288583] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xefffffff]
[    1.288591] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
[    1.288599] pci_bus 0000:00: scanning bus
[    1.288626] pci 0000:00:00.0: [10ee:d021] type 01 class 0x060400
[    1.288698] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    1.288708] pci 0000:00:00.0: PME# disabled
[    1.289943] pci_bus 0000:00: fixups for bus
[    1.289951] pci 0000:00:00.0: scanning [bus 01-0c] behind bridge, pass 0
[    1.290007] pci_bus 0000:01: scanning bus
[    1.290049] pci_bus 0000:01: fixups for bus
[    1.290054] pci_bus 0000:01: bus scan returning with max=01
[    1.290063] pci 0000:00:00.0: scanning [bus 01-0c] behind bridge, pass 1
[    1.290074] pci_bus 0000:00: bus scan returning with max=0c
[    1.290087] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
[    1.294062] xilinx-zynqmp-dpdma fd4c0000.dma-controller: Xilinx DPDMA engine is probed
[    1.295061] zynqmp-display fd4a0000.display: vtc bridge property not present
[    1.297697] xilinx-dp-snd-codec fd4a0000.display:zynqmp_dp_snd_codec0: Xilinx DisplayPort Sound Codec probed
[    1.297954] xilinx-dp-snd-pcm zynqmp_dp_snd_pcm0: Xilinx DisplayPort Sound PCM probed
[    1.298192] xilinx-dp-snd-pcm zynqmp_dp_snd_pcm1: Xilinx DisplayPort Sound PCM probed
[    1.299219] xilinx-dp-snd-card fd4a0000.display:zynqmp_dp_snd_card: Xilinx DisplayPort Sound Card probed

Link is DOWN

I really don't know where to go from here. If this is going to turn out like getting pcie to work on Altera Cyclone V (where the link was up but the driver was failing to invalidate cache properly), then I'm going to have to start to add print statements to the driver to see where it is going wrong. But that may not help since the link is down.

I suspect that the link being down is the issue. I have a known working pcie x 1 SATA controller in the slot. Yet it's not linking. I don't have access to a scope. I guess I could try checking power, but I have to believe that someone already checked this port and verified that it at least works.

Xilinx tested their own board and the results are here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842417/Linux+ZynqMP+PS-PCIe+Root+Port+Driver

I checked the pcie descriptor in the device tree and it matches. It's actually a little better than the one listed in that link.

Any help would be appreciated. I will update this thread as I make progress so anyone in the future doesn't have to go through this.

Update: I realized that different BSP files have different features. So I found an nvme BSP and tried it.

It hangs on bootup at PCIe initialization:

[    2.871723] Key type asymmetric registered
[    2.875786] Asymmetric key parser 'x509' registered
[    2.880650] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    2.887985] io scheduler mq-deadline registered
[    2.892481] io scheduler kyber registered
[    2.897422] xilinx-xdma-pcie 500000000.axi-pcie: host bridge /amba_pl@0/axi-pcie@b0000000 ranges:
[    2.905299] xilinx-xdma-pcie 500000000.axi-pcie:   No bus range found for /amba_pl@0/axi-pcie@b0000000, using [b
us 00-ff]
[    2.916200] xilinx-xdma-pcie 500000000.axi-pcie:      MEM 0x00b0000000..0x00b0ffffff -> 0x00a0000000
[   23.904679] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[   23.904961] rcu:     1-...0: (2 ticks this GP) idle=baa/1/0x4000000000000000 softirq=37/37 fqs=2626  
[   23.913780]  (detected by 3, t=5254 jiffies, g=-1111, q=14)
[   23.919314] Task dump for CPU 1:
[   23.922515] task:swapper/0       state:R  running task     stack:    0 pid:    1 ppid:     0 flags:0x0000000a
[   23.932375] Call trace:
[   23.934804]  __switch_to+0x114/0x1a0
[   23.938345]  0x0

This actually gives me hope that this kernel, and PS configuration has the PCIe working and it's crashing because it expects there to be an nvme in the slot.

I'm going to use this kernel as the basis for the kernel I'm building and see if it helps.

I'm really stuck. I can't get pcie or USB to work.

The GPU is initializing. Go figure.

None of the downloadable images from 2017.1 to 2022.2 for the card carrier board + 7ev enable the USB or PCIe ports to work.

I have no idea how to debug it.

How am I supposed to debug my own stuff when the examples don't work.

Does anyone have a known working boot image that will successfully run lspci with a card inserted and USB with a usb device connected?