Expanding shared memory on the BBB

RE: Expanding shared memory on the BBB

shabaz — Sun, 09 Mar 2014 15:37:22 GMT

These topics mentioned will be useful:

BBB - High speed data acquisition and web-based UI

In that topic and the current topic, the following URLs were discussed:

Allocating memory using modprobe uio_pruss:

https://groups.google.com/forum/m/#!topic/beagleboard/Gb6xL7V7Z00

and this URL shows how to retrieve the address:

http://hipstercircuits.com/beaglebone-pru-ddr-memory-access-the-right-way/

This method avoids the need of any other driver (but is not necessarily the best way for reasons described in the topics).

I noticed what looks like the same method is used here too:

https://bitbucket.org/intelligentagent/pypruss/src/f48e3d553687205d3f62420a95039a0d16dbee4d/pypruss/pypruss.c

RE: Expanding shared memory on the BBB

Former Member — Thu, 20 Jun 2013 17:48:50 GMT

So I found the old driver, unfortunately it's for a 2.4.x kernel. On a brighter note, it's only 168 lines and I'm guessing that 30% of that is comments, so I'll see if I can get it cleaned up and compiling on a recent kernel.

RE: Expanding shared memory on the BBB

Former Member — Thu, 20 Jun 2013 16:35:43 GMT

shabaz wrote:

Regarding drivers, I had found this link in the past and had bookmarked it to learn how to write drivers, so I'll read up and also browse some existing drivers to see how they work, and perhaps copy one to use as a base.

http://oreilly.com/openbook/linuxdrive3/book/index.html used to be available as a single pdf but I don't see it now, this is the closest http://lwn.net/Kernel/LDD3/

while it's obviously dated now it's still a good intro to a lot of the concepts.

I'm searching for the driver I had to do this sort of thing from 6-7 years ago. It probably won't build against a current kernel, but may be able to be cleaned up and reused to some degree - assuming I can find it !

RE: Expanding shared memory on the BBB

shabaz — Thu, 20 Jun 2013 16:17:20 GMT

selsinork wrote:

thinking about that further, while you may need to describe the size of the buffer in the dts somehow, you probably need to tell uBoot to load the kernel higher. I didn't dig too deeply, but possibly that can be done in the uEnv.txt that lives in /dev/mmcblk0p1 but if not then you likely need to rebuild uBoot just to change the kernel's load address in the script.

A driver will be the better way and it's not really all that complicated as essentially all it needs to do is kmalloc a contiguous buffer and let you mmap that from userspace. You could always add features by using ioctl's on a /dev entry or something in /sys to allow you to configure the buffer size at runtime if a simple fixed buffer of say 256k isn't enough.

I see, thank you for the excellent pointers. I'll check out uEnv.txt and the .dts tonight.

Regarding drivers, I had found this link in the past and had bookmarked it to learn how to write drivers, so I'll read up and also browse some existing drivers to see how they work, and perhaps copy one to use as a base.

RE: Expanding shared memory on the BBB

Former Member — Thu, 20 Jun 2013 16:00:59 GMT

shabaz wrote:

They just mmap the huge 256MB amount, but don't expect the user to try to access all that.

Sure, but the problem is that you're mmap'ing an area that includes kernel code and data, there's no actual reservation or proper allocation here, everything is a wing and a prayer

I'll check out the dts tonight, in case that contains it. If it doesn't, maybe I should request that the memory in future builds is allocated a bit more than 8kbyte, but others may not want that.

thinking about that further, while you may need to describe the size of the buffer in the dts somehow, you probably need to tell uBoot to load the kernel higher. I didn't dig too deeply, but possibly that can be done in the uEnv.txt that lives in /dev/mmcblk0p1 but if not then you likely need to rebuild uBoot just to change the kernel's load address in the script.

A driver will be the better way and it's not really all that complicated as essentially all it needs to do is kmalloc a contiguous buffer and let you mmap that from userspace. You could always add features by using ioctl's on a /dev entry or something in /sys to allow you to configure the buffer size at runtime if a simple fixed buffer of say 256k isn't enough.

RE: Expanding shared memory on the BBB

shabaz — Thu, 20 Jun 2013 14:31:45 GMT

Ah, I should have pasted a bit more of that code:

static int mem_fd;

mem_fd = open("/dev/mem", O_RDWR);

if (mem_fd < 0) {

printf("Failed to open /dev/mem (%s)\n", strerror(errno));

return -1;

and then they follow it with the ddrMem = mmap(.... portion pasted earlier.

They just mmap the huge 256MB amount, but don't expect the user to try to access all that.

I'll check out the dts tonight, in case that contains it. If it doesn't, maybe I should request that the memory in future builds is allocated a bit more than 8kbyte, but others may not want that. A driver or dts would be pretty neat, I wonder if that is planned by anyone if the functionality doesn't exist today. I'll try and query from Jonathan Bailey who is working on the GSoC PRU JTAG project, in case he needs to transfer more than 8kbyte too (although he could probably do it in chunks, which I can't with my use-case).

RE: Expanding shared memory on the BBB

Former Member — Thu, 20 Jun 2013 14:10:37 GMT

what file does mem_fd reference ? /dev/kmem ?

With a length of 0x0FFFFFFF you're trying to map a region of 256MB at address 0x8000000 which according to the am3359 TRM is ths start of RAM, so you'd expect to be overwriting something quite soon after that point.

There are a couple of posibilities, kernel commandline mem= option can limit the amount of ram available, so in theory you can reserve some space that linux doesn't know about at the top of memory. Not sure if this really helps as you may be unable to mmap that region from userspace. Oh, and mem= does that on x86, not sure if Arm will do the same. Also this reservation would be at the top of ram, i.e. you're setting the max memory to be 511Mb on your 512Mb board. So if your board ever gets 1GB, you've reserved 513MB !

The other is that in x86 there's a kernel option to reserve a chunk of memory at the bottom of the address space due to the bios using SMI mode and assuming cetrain things about the contents of the start of ram due to hangovers from dos days.

If this exists on Arm kernels you have a way to reserve some space that the kernel won't touch - kernel recompile will likely be needed though.

Hmm, seems the one I'm thinking of is called X86_RESERVE_LOW, so unfortunately not a generic feature although maybe it could be added. However if kernel surgery is the only way then maybe a proper device driver would be a better way longer term.

Is there anything in devicetree that could help reserve an area ? I'm assuming devicetree carries details of the board's memory map which might be tweaked ?

In any case, you're quickly getting to the point that your app couldn't expect to run on an un-modified BBB anyway, so only useful for people who are prepared to recompile kernels, drivers and such like.

RE: Expanding shared memory on the BBB

shabaz — Thu, 20 Jun 2013 13:22:28 GMT

The intention was to capture data at high speed from an ADC or a CCD, so I'd need to capture an entire frame at a time.

I can capture with the PRU and dump in any memory address, using (say):

MOV r1, 0x80001000

MOV r0, 0xaabbccdd

ST32 r0, r1 // store into address at r1

And for that r1 address, I'm free to use any location

(with the exception that if I want to address lower than

0x00080000 then I need to set a register (called PMAO) first.

At the ARM end, I have something like this (this is a snippet from TI's example app), and no malloc anywhere, so the memory must have been reserved at startup of the BBB:

#define DDR_BASEADDR 0x80000000

#define OFFSET_DDR 0x00001000

ddrMem = mmap(0, 0x0FFFFFFF, PROT_WRITE | PROT_READ, MAP_SHARED, mem_fd, DDR_BASEADDR);

DDR_regaddr1 = ddrMem + OFFSET_DDR;

If I try to get the PRU to write more than about 8-12kbyte, then I get crashes, because I'm trampling on some memory.

So, either I need to find a way to reserve more memory at startup of the BBB, or I had thought some way to be able to perform a malloc and send the address to the PRU to use may help, but you've just reminded me it may not be contiguous :-(

I'm wondering if maybe today the 8kbyte is just hard-coded in some linker file possibly, since the address is fixed. So this could be messy to have any convenient approach to have people be able to modify it. I think 8kbyte is too low, because it is easily possible to exceed that when doing high-speed data aquisition for more than a few milliseconds. Anything much higher than 128kbyte may be more unusual I suppose. I'll try doing a recursive grep for these addresses (might take a long time!), in case I can spot it, and I'll report back.

RE: Expanding shared memory on the BBB

Former Member — Thu, 20 Jun 2013 12:41:29 GMT

Can you explain a bit more about exactly what you're trying to do ? Maybe with a snippet of the problem code ?

mmap isn't limited to 8KB as such, but what you're trying to mmap could have limits, or what you're trying to do might not be sensible in a virtual memory environment.

in a bash shell, try typing ulimit -a which will show you process limits, to set these things from code you should start by looking at getrlimit & setrlimit, some of these can affect mmap

if you're trying to mmap 128k of memory from userspace and have that be contiguous physical memory for the pru to use then it's likely to be impossible from userspace as you're creating a region that's contiguous in virtual memory but which could be made up of pages scattered throuought physical memory. So the best you can do is a single page.

It's been a few years since I had to do this and it was on x86, so the details may well be different, but it required writing a kernel driver which would do the actual memory allocation in physical memory, it then provided a device node in /dev which could be mmap'ed from userspace along with a pointer describing the physical address this corresponded to.

Inside the kernel there are (or were) some functions to do the conversions - virt_to_phys, phys_to_virt, virt_to_bus, bus_to_virt - which ones you need to use are dependant on the CPU, MMU etc, so my recollection of how it all fits together on x86 is probably not quite right for Arm.

Other complications are that the virt addresses used by these are 'kernel' virtual addresses which will be different to userspace virtual addresses and whether you need phys or bus will depend on how Arm does things.

Interesting article here (although rather out of date, it should give you an idea of the complexities): http://www.tldp.org/LDP/khg/HyperNews/get/devices/addrxlate.html

shabaz wrote:
Or, is there any way for a Linux application to malloc at runtime and somehow query the MMU for the physical address? Is this even possible? I suspect, maybe this is not possible.. If it was, it would solve the problem because then I could pass that address to the PRU application.

I'd suggest reading the article and then doing some more digging. It seems likely that the things you can get direct access to from userspace are not going to be useful in the way you'd think.