Driver instability issue

I can confirm the issue that Paul reported; getting the exact same error message. Since the last update in this thread was on May 3rd, I wonder if any solution has been found so far to get rid of this error without rebooting?

Thomas

Hi all

I have just come back to this problem of 'running out of coherent memory for DMA' with the wolfson card (Hoping that someone else would fix the problem!!)

I confirm that it is NOT a problem with size of memory , but indeed a memory leak.

Specifically, the problem is that in bcm2708-dmaengine.c , the function bcm2708_dma_desc_free (which should free the memory used by the alsa sound driver for aplay/arecord etc) is NEVER called.

I have a simple patch that can be applied to bcm2708-dmaengine.c, for anyone that understands how to rebuild the RPi kernel (find details elsewhere).

The patch works by checking whether coherent memory has already been allocated, when further coherent memory is requested, and if so forces the old memory to be free'd. It presumes that the I2S driver is the only thing using this cyclic dma, and just 2 blocks (for playback and record) are managed.

(If bcm2708_dma_desc_free DOES start getting called properly in future, my code will print diags to remind to remove this patch)

I am still using a version of 3.12 kernel (but saw the problem with the standard 3.10 kernel) - I cannot see that anyone has fixed this issue already, but I have not confirmed that the problem has not 'gone away' already with later kernels.

It fixes the problem for me - hope it helps others.

Matt Willis

Oxford Creative Technology Ltd

Last night, somebody pointed me to this thread and told me that the problem is that bcm2835_dma_terminate_all doesn't free the resources bcm2835_dma_desc_free would free. However, it calls vchan_dma_desc_free_list, which would in turn call bcm2835_dma_desc_free, so that doesn't quite add up. I don't have the hardware to investigate this properly myself, but if somebody can provide some more information to go on, I could certainly fix it. Otherwise, I would have to write a test module to try to trigger the problem.

Adding printk statements to bcm2835_dma_desc_free and bcm2835_dma_terminate_all showed that bcm2835_dma_desc_free is called for scatter/gather DMA allocated by bcm2835_dma_prep_slave_sg but not for cyclic DMA allocated by bcm2835_dma_prep_dma_cyclic. I can see that bcm2835_dma_terminate_all sets c->desc to NULL before it calls vchan_dma_desc_free_list, so I think it's already leaked by that point. The issue seems to be that the control block in c->desc isn't freed before c->desc is set to NULL.

If you do manage to get hold of one, I can reproduce the issue with a Wolfson board using the Volumio distribution (I'm running with a custom kernel built from https://github.com/manish-wolfsonmicro, which is the subject of https://github.com/raspberrypi/linux/pull/705/files), put a bunch of files in the playlist and run "mpc next" in a loop.

bcm2835_dma_desc_free, determines desc using container_of, not by reading c->desc. Although  c->desc is set to NULL, vd is still contained by the structure previously pointed to by c->desc.

Does anyone know if there is still somebody working on this issue?

I also ran into the "ERROR: 4096 KiB atomic DMA coherent pool is too small!" error using a 3.16.5 kernel on a Raspberry Compute Module. It came up unexpectedly after a few minutes of playing around with aplay and gst-launch. First the system froze for about 2 minutes (but I could still ping it), then it came back to life with the error, plus I/O errors on the SD card interface because the system was in the middle of writing some file when the error occurred. File system was corrupted after a power cycle...

Today, I checked again with kernel 3.19.0 (revision f589118 from https://github.com/raspberrypi/linux.git) and tried something similar to what Paul Roe has done in post #10, but with aplay instead of arecord. Apparently there were some changes applied to the I2S driver, so I was hoping for better results than with 3.16.5, but I got disappointed.

Here is what I've tried with a simple one second long stereo file at 44.1 kHz:

 

$ for i in $(seq 10000); do aplay 1sec.wav || break; echo $(date) $i; done

Playing WAVE '1sec.wav' : Signed 16 bit Little Endian, Rate 44100 Hz, Stereo

Fri Feb 13 08:53:17 UTC 2015 1

....

This went on for about 20 minutes, then after a bit more than 900 iterations the loop broke.

Attempting to play the .wav with aplay yielded an error after ten seconds ("aplay: pcm_write:1947: write error: Input/output error").

Attempting to cat some random file not already in fscache also yielded an I/O error (maybe trying to load the cat executable already gave the error, not sure).

To my surprise, there was no atomic-DMA-coherent-pool-error, but found this in the kernel log after these actions:

[ 1732.219441] mmc0: Timeout waiting for hardware interrupt.

[ 1732.225168] mmcblk0: error -110 transferring data, sector 3013168, nr 32, cmd response 0x900, card status 0xb00

[ 1732.236730] mmcblk0: retrying using single block read

[ 1772.279443] mmc0: Timeout waiting for hardware interrupt.

[ 1772.285206] mmcblk0: error -110 transferring data, sector 3707576, nr 16, cmd response 0x900, card status 0xd00

[ 1772.296995] blk_update_request: I/O error, dev mmcblk0, sector 3707576

[ 1772.303841] blk_update_request: I/O error, dev mmcblk0, sector 3707584

[ 1772.311732] Aborting journal on device mmcblk0p7-8.

[ 1782.319444] mmc0: Timeout waiting for hardware interrupt.

[ 1782.325167] mmcblk0: error -110 transferring data, sector 3301472, nr 8, cmd response 0x900, card status 0xb00

[ 1782.336796] mmcblk0: retrying using single block read

[ 1792.359440] mmc0: Timeout waiting for hardware interrupt.

[ 1792.365216] mmcblk0: error -110 transferring data, sector 3706880, nr 8, cmd response 0x900, card status 0xd00

[ 1792.376864] blk_update_request: I/O error, dev mmcblk0, sector 3706880

[ 1792.383702] Buffer I/O error on dev mmcblk0p7, logical block 131072, lost sync page write

[ 1792.393305] JBD2: Error -5 detected when updating journal superblock for mmcblk0p7-8.

 

Well, the DMA-related error message was removed as part of a bigger change in 36d0fd2, so it cannot show up anymore.

Needless to say, attempting to cleanly shut down via shutdown -r now yielded more errors. I power-cycled the system after several minutes of waiting. There was no file system corruption this time (system booted, fsck.ext4 -v -n -f reported no errors), probably because there was no coincidental writing in the background when the I/O errors started to show up.

So, the original problem seems to persist: playing audio over I2S kills the system and may lead to file system corruption even in the latest kernel version.

Is there anyone who is working on this problem?

By the way, the patch from post #12 does not apply to the latest kernel anymore. As it contains the work "hack" many times, I am also not so sure that it should be used in production anyway. I didn't try it on older kernels.

Maybe unrelated, but I also found this in the kernel log:

[  602.269309] bcm2708_fb_blank(1) returns=1 p[1]=0x0

That's the video driver speaking here (linux/drivers/video/bcm2708_fb.c).

Hi,

I am seeing a similar problem on a custom rpi b board with rpi-proto codec connected via i2s.

Tried 3.12.37 PREEMPT and 3.18.6 with Device Tree support enabled.

Does anyone knows how to fix the issue?

Did you compile Kernel 3.19 with cirrus/wolfson drivers? When I compiled the driver I only got 3.18. https://github.com/CirrusLogic/rpi-linux/wiki/Building-the-code

How did you do it?

I got back to the topic and finally found the problem. It is a memory leak that was actually present in several dmaengine implementations, and the fixes got already merged into vanilla kernel in commit 0a48127 (so the fixes are in v4.0-rc7).

In the Raspberry Pi branch, a similar fix must be manually applied to file drivers/dma/bcm2708-dmaengine.c as no such fix has been applied to their driver yet.

Here is a patch for kernel 3.18:

--- a/drivers/dma/bcm2708-dmaengine.c
+++ b/drivers/dma/bcm2708-dmaengine.c
@@ -684,6 +684,7 @@ static int bcm2835_dma_terminate_all(struct bcm2835_chan *c)
        * c->desc is NULL and exit.)
        */
       if (c->desc) {
+              bcm2835_dma_desc_free(&c->desc->vd);
               c->desc = NULL;
               bcm2835_dma_abort(c->chan_base);

Will you or someone be able to upload and share this kernel with this modification ?

Not a lot of people have time or able to build a new kernel.

Thanks in advance.

Best Regards