Timeout Waiting for Hardware Interrupt on mmc0

I took a look at the Verified Peripherals wiki ( http://elinux.org/RPi_VerifiedPeripherals#SD_cards ) and saw Patriot 8GB Class 10 cards both on the Working and Problem lists (with your "mmc0: timeout waiting for hardware interrupt" error).  So you're not alone.

You might check the fine print on the cards to see if they're all the same exact model number, but I think it's more likely that RasPi is clocking the SD card faster than two of them can handle.  That could be as simple as a rounding error when RasPi calculates the SD card clock rate.  I don't know much about SD card classes and how a processor goes about deciding how fast to clock an SD card.  It could be that your PC is more conservative than RasPi about SD card timing, so it's fine but RasPi is a bit too optimistic about SD performance or has a rounding error in its clock rate calculation.  This is a SWAG based on ignorance, so add salt.

Question: does a misbehaving SD card fail in many RasPis or just one?  If it's the latter, it could be a subtle timing problem like clocking the SD card on the wrong edge, but I think that's unlikely or we would have heard much more keening and gnashing of teeth.

Coming from experience I know that any retail cards are a crap shoot as to what they really have inside of them, no matter what the outside label or marking is. Selling to OEMs there are tight constraints as to who is on the AVL but for the consumer retail market it's fair game as long as the card is the correct format (and even that's debatable!).

It ended up that 11/28 just wouldn't boot and would throw these errors on the RPi. The systems are already being assembled so I only have access to my one RPi.

I just returned them all to Fry's and ended up exchangin them for even scarier brand called Unirex (whom I've never heard of). The first one imaged fine and booted up good but I've 10 more to do before I breath easy.

I should have kept one to see if I could image it for the Pandaboard and Beagleboard and see if they had any problems but I didn't. I had placed a call into tech support of Patriot and asked them to call me back but of course they haven't.

I'm assumming that the RPi is using SPI mode to access the SD Card so they don't have to pay the SD licences fee? Is that the clock you're referring to?

George Ioakimedes wrote:

 

I'm assumming that the RPi is using SPI mode to access the SD Card so they don't have to pay the SD licences fee? Is that the clock you're referring to?

There's a clock signal no matter which mode you're using.

Yes but if it's using SPI that's about as common a bus signal as there can be in an embedded system (next to I2C) and with only 1 data pin (as opposed to the 4 bits) I would think the timing wouldn't be as critical.

So USB doesn't really work, and now SD Card access also doesn't really work, and I've read that there are problems with ethernet as well. I guess I'm getting what I paid for, eh?!

But seriously, are there really these issues and people haven't fixed them yet? I'm beginning to wonder more and more about the BCM2835 chip itself.

It's not using SPI, it's using all of the capabilities available at 3.3v. There's no way to change the interface voltage to 1.8v, so some modes are not available.

SD Card stuff really is a game of chance unfortunately and the Pi has had no end of problems in that respect. Most have now been fixed, but had you been trying this back in April you'd have been having a lot more problems.

Essentially which mode and clock speed is used is based on a descriptor that's read from the card using the slowest most basic mode that all cards must support. From the descriptor the system can then work out which version of card you have (i.e. SDHC, SDXC etc) and what speed modes it's capable of supporting. Buried in the drivers there's also some attempts at retraining at lower speeds and fallback to more basic modes. That doesn't always work, especially if the card isn't genuine and the descriptor lies.

For some values of "every other system has no problems" that may be due to other factors. For example, the onboard sdcard reader in my laptop only uses SDSC speeds. That makes it much slower, but also means it works with more cards.

Sadly, sd-card issues are not so easy to fix. Would you care to test the driver with every possible card out there, including counterfeit and simply faulty cards ?  The other option could be that the RPF castrate the driver in the 'official' images to only use the slowest possible modes, but I suspect that would bring more screams of annoyance from a different set of users.

Remember here that the original images had a very much worse sdcard driver and that the RPF weren't much interested at the time. It took some outside developers to fix the worst of the problems.

General advice from some months of watching the problem and testing some cards - avoid class 10 cards if possible, the faster modes are more prone to issues. Class 10 is also often optimised for a very different use case (large sequential writes) and a good class 6 card is often better in the Pi. YMMV of course depending on the card.

selsinork wrote:

 

It's not using SPI, it's using all of the capabilities available at 3.3v. There's no way to change the interface voltage to 1.8v, so some modes are not available.

By "all of the capabilities" do you mean it is using 4-bit mode? My understanding is that 1.8V is only for lower power consumption and that the mode of operation or available commands does not change. Of course 1.8V is only possible for SDHC/SDXC cards.

selsinork wrote:

 

Essentially which mode and clock speed is used is based on a descriptor that's read from the card using the slowest most basic mode that all cards must support. From the descriptor the system can then work out which version of card you have (i.e. SDHC, SDXC etc) and what speed modes it's capable of supporting. Buried in the drivers there's also some attempts at retraining at lower speeds and fallback to more basic modes. That doesn't always work, especially if the card isn't genuine and the descriptor lies.

I know nothing about their SD card driver but at initial boot up I would expect the driver to read the descriptor (CSD) and that is defined to happen at 400kHz. If the driver is written to use 4-bit mode then based on the descriptor read the clock speed will be determined. Do you think it is at this point in the boot process that these errors are getting thrown? It just seems odd to me that at this point it totally looses communication with the SD Card.

From my understanding the class rating of the cards just defines how fast the card can be written to and with Class 10 they need to be able to maintain a 10MB/sec write speed. I can understand during normal operation problems happening because not all cards may really be able to maintain certain speeds and if the card is being clocked at a rate that it says it can support but in reality it doesn't then problems can happen.

selsinork wrote:

 

Sadly, sd-card issues are not so easy to fix. Would you care to test the driver with every possible card out there, including counterfeit and simply faulty cards ?

My findings have nothing to do with counterfeit or faulty cards and I don't believe that a driver should be written to support those cards. I don't know the real reason why those 11 cards didn't work but failing so soon in the boot process makes me believe that the driver couldn't communicate with the card. To me that seems to be a major problem in the driver. Wouldn't you as an embedded engineer write a driver that reads the CSD, sets the system parameters (block size, speed, etc.) and then try to read the card, if the read fails try a slower speed? To totally crap out on a perfectly fine card seems pretty bad to me.

Now I would expect that USB card readers have paid the SD license and operate cards in 4-bit mode but maybe I'm wrong and they only operate in SPI mode so those other bits are never excercised and for all I know they could have opens on those lines but 11/28 cards have manufacturing defects?

I always try to learn something from problems and I just don't see what I learned here other than that nice little $35 board is fine for schools but not for all applications. I just don't get that with so many units sold that these types of problems are still happening though....

By "all of the capabilities" do you mean it is using 4-bit mode? My understanding is that 1.8V is only for lower power consumption and that the mode of operation or available commands does not change. Of course 1.8V is only possible for SDHC/SDXC cards.

Yes, err.. maybe, sometimes..

Lets take a step back for a moment, there are several distinct phases involved.

1. the minimal on-SoC GPU firmware tries to initialise the SD card and load bootcode.bin
2. bootcode.bin pulls in config.txt and sets up stuff defined there, then likely has a better look at the sdcard and tries to load start.elf, again into the GPU
3. start.elf loads cmdline.txt and finally kernel.img then boots linux on the arm

Only after all these things happen does the linux sd-card driver get a look in. At this point there's probably been three different drivers used already and the linux driver is the fourth.

So it may be that the first step uses a slow SPI-only mode and bit-bangs it, then as bigger and more intelligent bits of code get loaded the clock gets increased, 4bit mode gets turned on and DMA gets used. What's the point where it breaks ?  Can the previous driver leave the h/w in some screwed up state ?  Is there any communication up the layers to tell the new driver there's a problem etc..

1.8v does enable addidional modes - particularly for class 10 cards, here's a small snippet from the linux driver: https://github.com/bootc/linux/commit/ac1dba37f1e0ba1acdc19e12a70199c420101459  the point here is that you can have a card that supports a faster mode, a SoC that does, and a driver that does, but the particular hardware implementation on the R-Pi omits any way to switch voltage.

One of the early problems was that the default mmc clock set by one of the early boot steps was very non-optimal, resulting in the clock divisors forceing a very low clock speed on cards that were capable of much more.  So all the early bits would work fine and the linux kernel would load, slowly...  Then the problems would start.

Look at the comments to Grigori's fixes in early June for some context here https://github.com/grigorig/rpi-linux/commits/sdhci-perf

I know nothing about their SD card driver but at initial boot up I would expect the driver to read the descriptor (CSD) and that is defined to happen at 400kHz. If the driver is written to use 4-bit mode then based on the descriptor read the clock speed will be determined. Do you think it is at this point in the boot process that these errors are getting thrown? It just seems odd to me that at this point it totally looses communication with the SD Card.

Yeah, but which driver in which step uses what bit of the CSD ?  Some are probably making assumptions and hard-coding slow/safe speeds.. Later when the higher speeds get enabled is when you lose communication. Could that be a bug in the new driver ? Sure.  But it's hard to seperate the driver from the hardware and from the unknown quality card and work out where the problem lies.

From my understanding the class rating of the cards just defines how fast the card can be written to and with Class 10 they need to be able to maintain a 10MB/sec write speed. I can understand during normal operation problems happening because not all cards may really be able to maintain certain speeds and if the card is being clocked at a rate that it says it can support but in reality it doesn't then problems can happen.

I suspect there's some invalid assumptions on the part of both card manufacturers and driver writers. What happens when someone assumes that if you support mode 5 then you also must support 1-4, but in reality the thing only supports mode 1 and 5 ?  Any fallback to mode 4 when mode 5 fails is likely going to have problems too.

My findings have nothing to do with counterfeit or faulty cards and I don't believe that a driver should be written to support those cards.

Ok, but exactly how do you (or the driver) tell if the card is counterfeit or faulty ?  Read the details from the card, try to use them and it doesn't work. Now what ?  You don't know if the card is broken, lying to you or what..

I always try to learn something from problems and I just don't see what I learned here other than that nice little $35 board is fine for schools but not for all applications. I just don't get that with so many units sold that these types of problems are still happening though....

What I take from it is that I don't like SD cards   Thing is that for lots of people the sd card issues on the Pi have been fixed, maybe you were just incredibly unlucky and got a bad batch of cards, maybe your Pi is faulty......   While I believe the Pi is fine for lots of things, you do need to be aware that it has it's problems. Some fixable, some not. In light of that you have to weigh up the risks and decide what you're comfortable with. Maybe a BeagleBone or an OlinuXino is a better choice due to the issues you've had with the Pi, only you can decide that though.

George Ioakimedes wrote:

 

So USB doesn't really work, and now SD Card access also doesn't really work, and I've read that there are problems with ethernet as well. I guess I'm getting what I paid for, eh?!

Although "you get what you pay for" is a commonly heard line, it's not actually all that accurate in our context.  There is no shortage of examples of extremely cheap electronics components that work exactly to spec, even very complex SoCs.  With each passing year, the mountain of cheap but perfectly working electronics devices just grows and grows.

A more relevant observation (I think) is that using a device outside of its intended design space is risky.  The BCM2835 was not designed to be the primary SoC in a general purpose host computer.  Its small ARM core's primary function seems to be chip housekeeping such as servicing the minimalist hardware USB core, which decreased overall device cost.  This made some sense in the intended role, but maybe not in others.

We now know officially  that the BCM2835's USB design is inadequate for providing full USB host functionality, and it's an open question whether this can ever be remedied in the driver since Linux is really poor at realtime response even when the realtime kernel patches are applied.  Success remains to be proven.

So in summary, I'm not sure that you should blame the low cost of Pi for the faults you have encountered.  In terms of engineering, a more accurate diagnosis would appear to be that the wrong chip was chosen for the job at Pi design time.  We'll have to see how this issue pans out, but that will take a while.

Morgaine.

Not so much that you get what you pay for amounting to the low cost of the RPi but as to the "engineering" take on the problems. We're all somewhat assumming that some of these problems we've encountered can be attributed to the BCM2835 but if so what's the big deal for the Foundation to fess up and say hey the chip works great but these are the known problems and we're stuck with them.

I'm also somewhat surprised that I find very little actual engineering analysis put into some of these problems. Take for instance the SD card issue I've experienced. Are there any scope plots anywhere that show what speeds the RPi uses during the different phases of boot up, I didn't see any. That really seems odd to me especially since there are so many people working with these boards with all different levels of experience.

The SD card fails at such an early stage in the boot process and it is 100% repeatable so isn't that one of the easiest engineering problems to fix? I think you're probably right and I'm starting to believe that the BCM2835 has some known bugs and since Broadcom is so secrative they won't admit them and anyone who tries to tell them that is told to be quite...

Not so much that you get what you pay for amounting to the low cost of the RPi but as to the "engineering" take on the problems. We're all somewhat assumming that some of these problems we've encountered can be attributed to the BCM2835 but if so what's the big deal for the Foundation to fess up and say hey the chip works great but these are the known problems and we're stuck with them.

I'm also somewhat surprised that I find very little actual engineering analysis put into some of these problems. Take for instance the SD card issue I've experienced. Are there any scope plots anywhere that show what speeds the RPi uses during the different phases of boot up, I didn't see any. That really seems odd to me especially since there are so many people working with these boards with all different levels of experience.

The SD card fails at such an early stage in the boot process and it is 100% repeatable so isn't that one of the easiest engineering problems to fix? I think you're probably right and I'm starting to believe that the BCM2835 has some known bugs and since Broadcom is so secrative they won't admit them and anyone who tries to tell them that is told to be quite...

George Ioakimedes wrote:

 

The SD card fails at such an early stage in the boot process and it is 100% repeatable so isn't that one of the easiest engineering problems to fix? I think you're probably right and I'm starting to believe that the BCM2835 has some known bugs and since Broadcom is so secrative they won't admit them and anyone who tries to tell them that is told to be [quiet]...

While there are some RasPi problems that seem to be ignored, the SD card problem is not one of them.  The SD card problem is well known and RasPi is trying to solve it.  Do take a look at this RasPi forum thread started by dom: Request for Failing sdcards.  He may be interested in testing your failing cards.  Good engineers love reproducible bugs because they're already half solved.

George Ioakimedes wrote:

 

We're all somewhat assumming that some of these problems we've encountered can be attributed to the BCM2835 but if so what's the big deal for the Foundation to fess up and say hey the chip works great but these are the known problems and we're stuck with them.

Problems with the BCM2835 progressed beyond the stage of assumption with this post by gsh on the RPi forum --- http://www.raspberrypi.org/phpBB3/viewtopic.php?f=28&t=12097&hilit=elephant&start=75 .  Quoting from it:

I've had a little while to understand what is going wrong with the USB and traced it back to a problem with poor interrupt latency in the kernel.  This is causing the split transactions to be delayed beyond a 1ms frame which in turn means that the hub is dropping the packets it stores.

This would have come as no surprise to anyone even moderately acquainted with Linux kernels (nor indeed with any Unix-type system), because none of the *nix family of operating systems have realtime-capable kernels as standard functionality, and that is inherent in their design.

Linux acquired some basic realtime capability with special patches for that purpose, but although these allow certain realtime constraints to be met, their best possible realtime response latency is still very poor compared to operating systems expressly designed for realtime use.  (I've used these patched kernels personally for MIDI applications, and they are not very effective, loss of USB events still being common.)

It's because of this that we know that the ARM11 inside the BCM2835 was never intended to be used to power a general purpose computer, because its CPU power is required to be permanently available for servicing the SoC's USB controller in order to keep response latency below 1 ms so that USB's split transactions are not dropped.  As a matter of principle, this is impossible to guarantee under normal operation in a Linux kernel that is also scheduling significant user-space applications.

So, it's a lot more than a mere assumption that the BCM2835 was the wrong device to choose for the job.

Morgaine.

John Beetem wrote:

 

Do take a look at this RasPi forum thread started by dom: Request for Failing sdcards.  He may be interested in testing your failing cards.  Good engineers love reproducible bugs because they're already half solved.

I had seen that thread and it looked somewhat dead so I didn't bother posting to it. Unfortunately in this tight economy I can't afford to sit on unused stock so I already returned those cards. I kind of kick myself for not keeping one just for reference but in the heat of the moment I said screw it.

Morgaine Dinova wrote:

 

It's because of this that we know that the ARM11 inside the BCM2835 was never intended to be used to power a general purpose computer, because its CPU power is required to be permanently available for servicing the SoC's USB controller in order to keep response latency below 1 ms so that USB's split transactions are not dropped.  As a matter of principle, this is impossible to guarantee under normal operation in a Linux kernel that is also scheduling significant user-space applications.

 

So, it's a lot more than a mere assumption that the BCM2835 was the wrong device to choose for the job.

 

Morgaine.

I'd love to sit down with you guys one time over some beers but I have a feeling that we're spread across the world so I'll just drink one in your honor tonight!

Are you thinking that the ARM11 in general isn't enough or just one in the BCM2835? I ask because I don't see these issues being brought up on the Pandaboard or Beagleboard forums. I know that's not comparing Apples to Apples but at least I haven't heard of these latency issues and they're running Ubuntu as one of the standard platforms.

Are there any scope plots anywhere that show what speeds the RPi uses during the different phases of boot up, I didn't see any. That really seems odd to me especially since there are so many people working with these boards with all different levels of experience.

Well I have actually watched what goes on during boot with a scope. It turns out not to be particularly relevant as different cards cause different speeds to be used. As you've already surmised, you can't always tell from the label on the outside of the card who made the innards and what it's capable of. So any scope trace is only useful for the exact card it was taken with, even an identically branded one can be different, so there's no real help there.

The SD card fails at such an early stage in the boot process

actually, as I tried to explain, when you see that timeout message from linux you're a long way into the boot. It's nothing like as early as you seem to assume.

selsinork wrote:

 

The SD card fails at such an early stage in the boot process

 

actually, as I tried to explain, when you see that timeout message from linux you're a long way into the boot. It's nothing like as early as you seem to assume.

Going back and looking at the picture I snapped it's a bit misleading as I let that board run awhile before the picture was taken. From memory I would say that it thows the interrupt or other error message at about the 4th line displayed on the screen.

Do you really think that all cards operate at slightly different clock rates? I haven't read the spec in quite a while but I would think you have the initial 400kHz for reading the CSD and once that's read it sets a clock rate depending on what is reported. If I was writing the driver I would then perform some type of operation to see if communication is good (perhaps a register read and checking the CRC?). Since every command has an expected response you'll know right away if there's any problem with the communication. Even with bulk transfers there's CRC and error status responses.

You're right, I'm by no means a Linux guru so I couldn't tell you where in the boot process the RPi was complaining but in a very general sense the system didn't boot but yet that card worked fine in other systems and both chkdsk and fsck both reported not bad blocks.

When you were probing the cards did you verify that they actually use 4-bit mode out of curiosity? Just because the schematic shows all lines connected still doesn't mean that they're used.

George Ioakimedes wrote:

 

Are you thinking that the ARM11 in general isn't enough or just one in the BCM2835? I ask because I don't see these issues being brought up on the Pandaboard or Beagleboard forums. I know that's not comparing Apples to Apples but at least I haven't heard of these latency issues and they're running Ubuntu as one of the standard platforms.

The ARM11 is perfectly general.  It's the BCM2835's design that requires a CPU core to be dedicated to it if it is to maintain sub-1-ms latency in response to USB controller requests.  The ARM can't guarantee that response latency and be servicing the Linux process queue at the same time.  When it tries, a proportion of the time it can't meet the 1 ms maximum response time for split transactions, and so those get dropped.  The user sees this as lost USB events, which have totally plagued the Pi community as numerous RPF forum threads testify.

This shouldn't really have come as any surprise to Pi designers, because they're clearly very knowledgeable about both the Linux kernel and the BCM2835, and hence should have known that a normal Linux kernel is not designed for low-latency realtime response whereas the BCM2835 is designed to require it.

I'd love to sit down with you guys one time over some beers but I have a feeling that we're spread across the world so I'll just drink one in your honor tonight!

A glass of nice Shiraz for me please.

Going back and looking at the picture I snapped it's a bit misleading as I let that board run awhile before the picture was taken. From memory I would say that it thows the interrupt or other error message at about the 4th line displayed on the screen.

Ok,  let me try again. When you see any message from Linux, you're a long way into the boot process.  Where do you think the linux kernel was loaded from ?

It's also unfortunate that most current linux distros go out of their way to hide what's going on at boot, so your fourth line is even further in than you think.

So, there's an example from my Pi at http://pastebin.com/uVAmd6ji since it's a bit big to post here. Go down to line 133, that's where the sdcard driver finally gets loaded by linux. Your message can't appear before then and likely won't appear before line 144 as that's where the card is starting to be used.

Remember also that at least three other bits of code have booted off the card before linux runs and (in my case) something like 10-15Mb of data has been sucessfully read off the card before you see a single line on the screen.  An 'early' boot failure gives you a blank screen.

Do you really think that all cards operate at slightly different clock rates? I haven't read the spec in quite a while but I would think you have the initial 400kHz for reading the CSD and once that's read it sets a clock rate depending on what is reported.

Yes of course. You have two cards, the CSD in one says it can run at 10Mhz, the other says 20Mhz. You expect they'll run at the same speed ?  It is always possible that the early boot code (i.e. everything before linux) runs in some dumb, safe, mode (maybe only 400kHZ SPI mode), so it works, albeit slowly, until something increases the clock, enables 4 bit mode or whatever. 

The early bits are very difficult to capture reliably on a scope, the clock changes a lot as different things happen during boot. It's much easier to see what's happening once the boot is complete, but thats after the interesting stuff has finished.

Remind me another day and I'll see if I can get an OLS trace of the early boot to post here. Not sure that the OLS has enough memory to capture much, but it might at least give us a better idea.

When you were probing the cards did you verify that they actually use 4-bit mode out of curiosity? Just because the schematic shows all lines connected still doesn't mean that they're used.

What more can I say than the other bits are active - I don't have an SD-Card bus analyzer, so can't verify it for sure.

However, here's a little snippet from the linux debugfs showing how the driver thinks it's talking to the card in my Pi:

root@raspberry-pi:/sys/kernel/debug/mmc0# cat ios

clock:          50000000 Hz

vdd:            17 (2.9 ~ 3.0 V)

bus mode:       2 (push-pull)

chip select:    0 (don't care)

power mode:     2 (on)

bus width:      2 (4 bits)

timing spec:    2 (sd high-speed)

I currently have no reason to believe it's lying to me, especially when I can scope the lines and verify the clock and that all the bits are active.