Actions
Forum Thread Details
  • Replies 9 replies
  • Subscribers 176 subscribers
  • Views 1319 views
  • Users 0 members are here
Related

Structure Needs Cleaning Errors

kvikrama

Hi All,

 

   We are using microZED SOM along with AVNET FMC carrier card in one of our industrial products.

We are running a linux based system. In lab conditions the setup works fine. In industrial environment we are experiencing "Structure Needs Cleaning" errors on linux libraries.

This forces us to keep re-flashing the SDCards. Have been looking for source of this problem for sometime.

SOM is Industrial grade (AES-Z7MB-7Z020-SOM-I-G/REV-H) .

 

Notes (Not sure if these can cause issues) :-

    1. The board takes a PMOD CAN module to communicate.

    2. DDR3 Chip Surface Temperature is 55 deg C.

 

A similar setup is working well in same environment just few feet away (another machine).

 

Some of the suspects that we tested and eliminated (might need revisit if any of you advice).

1. We suspected power supply noise and done many checks and filters to remove them. Looks like power supply noise is not the reason.

2. We suspected the linux kernel / application and did many regressions in our lab setup. All of then came clean.

3. We suspected quality of SDCard and moved to industrial SDCard. Didnt resolve the issue.

4. Suspected thermal issues and reduced the temperature on chip surfaces a lot (using RPi heatsink on Zynq 7020 and reducing the CPU speed to half). Didnt help either.

    On DDR3 we havent added heatsink as it's almost always 55 deg C on all our installations.

5. Thought machine grounding can be a problem and wired PMOD CAN ground to machine ground (In addition to grounding the FMC card's mounting holes). Didnt help.

 

Any help thought will be of great help.

  • Hello

    It sounds like there is something going on within your industrial environment that is corrupting writes on your file system.  Have you tried running some of the filesystem check tools to verify that this is the case and get a better understanding of the nature of the corruption?

    ubuntu - Cannot remove file: "Structure needs cleaning" - Unix & Linux Stack Exchange

     

    If you are not able to recreate the conditions within your lab, have you looked for transient events which might generate a burst of electromagnetic energy within the target environment?

     

    For example, in a previous design engineering role I was experiencing problems with spurious resets on my microcontroller board.  It turned out that these resets would only occur when a 50A@240VAC contactor was making and breaking contact some few feet away.  The solution I came up with was to protect the power input on my system by adding a transient protection device (in the USA we call this RC component a snubber) across the contactor switching device which was causing my system some reset problems.

    image

     

    There are a number of different suppliers out there for these types of components and I am not endorsing any particular one or telling you that this is the solution:

    Switching transient study and snubbers (eaton.com)

     

    I am not saying that EMI is for sure your issue either, I am just pointing out a problem that I discovered in my past job just by sitting and observing when my system was having problems.  Just take a look around your industrial site and spend some time there (as long as it is safe to do so) to get an understanding for what sort of EMI sources might be around your platform that could cause energy coupling onto your MicroZed system.

     

    Best Regards,

     

    -Kevin

  • in reply to zedhed

    Hi Kevin,

     

        There is an AC Induction motor one meter away, but electrically it's in a different power rail. Induction motor is not creating electrical noise.

    The microZED boards are housed in a large metallic enclosure (induction motor is just outside it). Is EMI interference possible in this case for the microZED board ??

  • in reply to kvikrama

    Hi

     

    I thought the same thing about my board design years ago and added RC protection to my reset lines to provide additional protection.

     

    Yes, we do test our MicroZed boards for EMI susceptibility for conducted and radiated emissions, but only to a certain level of energy.  At a high enough energy level, ANY board can be susceptible to EMI so I would look for ways that energy might be getting into your enclosure.  It sounds like if you have an EMI shield over the MicroZed system then perhaps there is still a conducted path for the energy to travel into your system through power/ground?

     

    Best Regards,

     

    -Kevin

  • We've seen issues with SD Cards, especially generic cheap ones but even some name-brand. I'd be curious to know exactly what you are using. We have had very good success with Delkin Utility-class cards. To see why we use them, check out this blog:

    Storage Insights #1 - Why Not All SD Cards are Created Equal

     

    You should also review this Xilinx Answer Record.

    https://support.xilinx.com/s/article/66779

     

    Is there any chance that power can be interrupted during operation? That can do very bad things to SD cards when Linux isn't shut down properly.

     

    If you do a simple web search, you will find a wide number of answers that are all over the board. Whether they apply to you depends on what you are doing:

    • improper dismounts
      • For example, If the partition you want to unmount has samba shares on it, you need to stop the smbd service first.
    • access from a foreign OS (ie: Windows driver)
    • disk (pre)failure
    • main memory failure

     

    Bryan

  • in reply to bhfletcher

    Hi Bryan,

     

      We are using Apacer card - AK6.112FHA.00103. We have tried other commercial cards as well, frequency of issue doesn't change.

  • in reply to kvikrama

    The Apacer AK6.112FHA.00103 card is a 3D TLC, which is going to be less reliable than an MLC card, like the Delkin Utility card.

    https://in.element14.com/delkin-devices/s416apg49-u3000-3/memory-card-microsdhc-mlc-16gb/dp/2806491

     

    I'm not saying that is the reason you see failures. However, it could contribute. It's likely interrupted power or some other improper dismount that is the likely culprit.

     

    Bryan

  • in reply to bhfletcher

    Hi

     

    I tend to agree with that improper shutdown can cause corruption, especially if you are in the middle of a write operation since the underlying Flash technology is going to be erasing a block of data back to 1s before writing the updated block info.  This is well documented for Linux systems and you can find information on steps you can take to protect your filesystem that may help here:

    filesystems - Linux file system corruption due to improper shutdown (fs ext4)? - Unix & Linux Stack Exchange

     

    Of course, spurious resets are by nature an improper shutdown of Linux.  Do you see any such evidence of unexpected resets happening on your system?

     

    Have you tried running some of the filesystem check tools to verify that this is the case and get a better understanding of the nature of the corruption?

    ubuntu - Cannot remove file: "Structure needs cleaning" - Unix & Linux Stack Exchange

     

    Which filesystem are you using on the SD card where you see the reported corruptions?  How frequently are you writing to the SD card?

     

    Best Regards,

     

    -Kevin

  • in reply to zedhed

    Hi Kevin,

     

         Application writes into SDCard is once in few minutes one image file (jpeg image). We are using EXT4 Filesystem.

    The errors are in linux libraries and not in the jpeg image files that are written into. 

     

    I am still not sure if the origin of the errors is SDCard itself or it's the DRAM memory that eventually shows up in SDCard.

  • in reply to kvikrama

    Hi

     

    If you are making known writes for only the JPGs, one thing you could try is to mount a separate EXT4 drive (try a USB thumb drive) using MicroZed USB host port and see if the corruption follows the write operations to the new drive.  That would be a clue that something interrupts the write operations over time of the issue moves to the USB thumb drive.

     

    You can also double check your EXT4 options since you should be able to recover from an improper shutdown with the right filesystem options enabled:

    ubuntu - ext4 filesystems frequently corrupting - Unix & Linux Stack Exchange

     

    Best Regards,

     

    -Kevin