Actions
Forum Thread Details
  • State Verified Answer
  • Replies 15 replies
  • Subscribers 328 subscribers
  • Views 1548 views
  • Users 0 members are here
Related

DMA Kernel panics

Former Member

Hi,

over the past few days, I desperately tried to build my own bootloader kernel etc. without using Petalinux, because I already have my own ecosystem based on Yocto and meta-xilinx.

I started with all git HEADs for U-Boot, ATF and the Linux kernel and wanted to use U-Boot as an SPL. Which didn't work at all, because the newest git HEAD of U-Boot actually needs a PMU firmware in version 0.3 which is nowhere to be found. That's very annoying!

So I fell back to older releases of all the components which were 2016.4. I managed to get a fully working FSBL, U-Boot 2016.4 with ATF and Linux 2016.4. When including the PMU firmware in the boot image, the 4.6 kernel breaks because the SD-Card controller fails to initialize. It works without loading a custom PMU firmware. I can't use the current 4.9 kernel, because this in turn needs a PMU firmware with version 0.3. AAAHHH!

Okay, back to kernel 4.6 without a PMU firmware then. It booted up into my userland and I was quite happy. But after a while of using the system on this kernel, I get a bunch of kernel panics caused from different DMA operations happening in different components. Sometimes the SD-Card fails, sometimes the MACB NIC driver. Even the slightest bit of load on the network card causes a fatal kernel panic in the interrupt context.

Then I tried Petalinux 2016.4. I can't get this to work properly either. First the U-Boot complained about a messed up memory config in the device tree (https://www.xilinx.com/support/answers/68390.html) which I eventually managed to fix by editing the correct dts file which doesn't get overwritten by the generation tools. But then I can't properly boot the system, because the U-Boot confuses the first SHDC controller with the second one. Even though I boot from the external card, it always wants to load the kernel from the eMMC.

I basically gave up for now. Is there *any* combination of compoenents that work at all on this chip and which do not immediately crash?

Parents
  • 0

    Hi Andy1988,

     Is there any update on your work. I am facing the same crash issue. If you have solved the issue can you please share the solution. 

    Thanks

  • 0 in reply to suraish230

    Unfortunately not. I tried literally everything. Using Petalinux in different versions, using Yocto only with meta-xilinx, different kernel versions, different compiler suites. It shows the same behavior everywhere. As soon as you access the SDIO Controller, hell seems to breaks loose. I even tried to debug it using the Kernel address sanitizer to detect memory corruption, but as soon as I enable that, the bug is gone. This makes a it likely to be a caching issue. And just keeping KASAN enabled all the time is not really a nice thing even for development machines.

    Interestingly, I managed to get it a bit better by playing around with the clocks in Vivado and regenerating the FSBL. It happens less often now, but it is still pretty common. So this seems to be related to clocking, caching or something else. I have absolutely no idea how I am supposed to debug this.

    Unfortunately, I don't know what exactly I did to achieve this. I think I created a new Vivado project (I think in 2017.1) without applying the board preset and then playing around with the memory config until it was set up like in the 2016.4 or 2016.2 Ultrazed template. I also set up the PS-GTR clocks as needed, but didn't touch the other clocks. I really don't remember completely. I should've written down what I did.

    I put the board in the drawer for now. It is absolutely unusable in its current state.

    So far I also didn't manage to try out Petalinux 2017.1, because the u-boot always tried to load the kernel from the eMMC instead of the external SD-Card when booting, even though the boot mode clearly says it booted from the external SD. I wasn't in the mood yet to figure that one out. I don't even want to use Petalinux in the end.

    All in all: Meh. I'm open for suggestions though.

  • 0 in reply to Former Member

    Okay, small update: I just saw that Invincea updated their Tutorial for Vivado and Petalinux 2017.1 and I gave it a try. I just built everything as outlined by them using their scripts.

    It looks like I have a working system now. I didn't diff anything yet to find out what is happening. I'm just downloading a 5GiB file on the board and it is at 200MB now without a crash. I'll try a few times just to be sure, but it looks pretty good so far.

  • 0 in reply to Former Member

    I finally fixed the kernel panic issue. The mistake I made is a bit embarrassing though: I copied the device-tree from a ZCU102 and modified it to fit the Ultrazed-EG on an IOCC, but what I forgot to change was the memory node. The ZCU102 has 4GB RAM, the Ultrazed-EG only has 2GB. Of course this leads so memory corruption sooner or later. Especially if your test case is downloading a large file via HTTP to the SD which causes a lot of data to be written into the filesystem cache.

    Funnily enough I tested a generated device-tree as well, but I only put that on the SD-Card for u-boot to load. But u-boot has its own device-tree which it uses by itself. And that one still contained the 4GB memory config. When u-boot loads a device-tree from the SD and jumps into the kernel it patches the memory-node in the just loaded (and correct) tree to contain the wrong memory config. That's why I never suspected the device-tree to be the culprit.

    I still don't know why Petalinux 2016.4 didn't work for me and was so unstable because it only uses generated (and thereby hopefully correct) device trees, but at this point I don't care anymore.

    A big thanks goes out to Invincea Labs. Using their code to build a known-working Image and replacing components in my image one by one made it possible for me to pin down the issue to the BOOT.bin and finally to u-boot.

  • 0 in reply to Former Member

    Hi Andy1988,

     

    It's good to hear the blog post and github repo helped you out - I definitely had the same issues with 2016.4 but I'm not very experienced in kernel or device tree development so I just waited (hoped) the 2017.1 release would fix the kernel panics.  

     

    This issues with uBoot and porting the ZCU102 device tree over to the UltraZed is a big reason why I use petealinux to build uBoot and patched the petalinux device tree output.  It would be a nice addition to patch the petalinux device tree output after it's been built rather than using a static DTS.  The github repo is a work in progress  so if you or anyone else has some feedback I'd be happy to merge in anything helpful.  

Reply
  • 0 in reply to Former Member

    Hi Andy1988,

     

    It's good to hear the blog post and github repo helped you out - I definitely had the same issues with 2016.4 but I'm not very experienced in kernel or device tree development so I just waited (hoped) the 2017.1 release would fix the kernel panics.  

     

    This issues with uBoot and porting the ZCU102 device tree over to the UltraZed is a big reason why I use petealinux to build uBoot and patched the petalinux device tree output.  It would be a nice addition to patch the petalinux device tree output after it's been built rather than using a static DTS.  The github repo is a work in progress  so if you or anyone else has some feedback I'd be happy to merge in anything helpful.  

Children
  • 0 in reply to tchoyt

    I have a patch for u-boot-xlnx and linux-xlnx which adds device trees and other support for the Ultrazed on an IOCC. I reenabled a bunch of peripherals like the eMMC, QSPI and I2C1 in your FPGA image to use all of the peripherals on the board. U-Boot even reads the MAC address from the MAC-Address I2C EEPROM :) In theory I also added the psu_init_gpl.h/c files for the SPL, but that is absolutely untested. I've always been using the Xilinx FSBL. I also didn't add any peripherals in the PL like the GPIO or BRAM controller. The patch should work nicely with the Ultrazed-IOCC-BSP published by AVNET as well, though I didn't try that.

    I didn't publish the code yet, because USB3 and DisplayPort do not work yet (SATA works fine), because I don't know how to get them to work. For USB3 I'm waiting on AVNET's reference design as the clock chip needs to be reprogrammed and for DisplayPort I didn't have the time yet to play around with it.

    In case anybody is interested, I quickly threw the patches on Github:
    https://gist.github.com/G33KatWork/35b19227fc6e100dc875991d5f7859a9

    The linux patch applies on xilinx-v2017.1 and the u-boot patch on the current master.

    I also have a working meta-ultrazed yocto layer which is very simple. It just applies these two patches and provides a proper machine config copied from the ZCU102 and modified to use the right device tree, uboot defconfig etc. I didn't send the meta-xilinx guys a patch yet because I wanted to the DP and USB3 working first in Linux.

  • 0 in reply to Former Member

    Hi Andy,

    Thank you for continuing to work on this project.  Building BSPs is very new to me so your, and Tim's, work is very helpful.

    I am a little lost when you indicate you have patches.  Can you describe how to apply these patches?  I only see the directories referenced in the patches in my /usr/src so I don't see how they apply to Tim's git repo.