Raspberry Pi server clusters

Was thinking same thing. Why just 1 Pi? How about 10.

One of my first enclosure mechanical designs is to house 10, slide in Pi's. Suitable airflow. Probably need some networking hub or router board, backplane for the GPIO , USB, JTAG and power, get rid of the audio and RCA connectors; they are a mechanical nuisance anyways. Light pipe in acrylic the LEDS for front monitoring. A brick of Pi's would be pretty powerful.

Yes indeed Richard, and such a server cluster would probably consume very little power too if we can disable enough subsystems, especially the graphics core.  (Another possibility is to use the VideoCore for GPGPU/OpenCL type functions, but at this point it's unknown whether that really holds any potential.)

Slide-in modules are a must when talking about 10+ boards, as wiring them together by hand will lose its appeal really fast.  I'm currently building a 3D printer which I hope to employ to print out suitable module holders and the rest of the structural framework.

PS. I only think in powers of 2, and 16 seems a nice number as a basic Rpi cluster node from which larger clusters can be built.

I was looking at building my own 3D printer, then I realized it was better to just spend the effort in quality design with a high quality, low cost 3D design syste and let the printer services do their job better and reasonably priced, like at Shapeways or ProPrinters. I do 3D CAD along with PCB so the overall design and manufacuring of computer and communications products is dear to me, (in reality, a multi-decade trans-century curse).

Now, if a guy could individually purchase the heart of the Raspberry, the BCM2835, plunk 10 down on a board, add circuitry for the LAN for all 10 and only 1 unit using the VideoCore, then a small form super computer could be formed based off that pretty nifty System on Chip processor the folks at Broadcom came up with. But then again, the overall price of the Raspberry as a demo unit for the Broadcom chip really trumps designing new system. Besides, there are many other ARM processors also suitable for Linux and thus clustering but I'm sure their individual cost plus assembly effort can't compete with simply stuffing a box full of already assembled Raspberrys.

I guess you could call this concept, the Raspberry Pi Raq

I like that idea, the "Raspberry Pi Raq".  Make the number of devices a power of 2 and I'll buy.

I wouldn't use the BCM2835 if I were designing boards for this though, I'd pick a more modern ARM device, particularly given that the VideoCore is not helpful in this application.

And, looking further into the future, I'd love to use the OpenRISC ASIC, which is projected to cost $5 in single units.  If only OpenCores would get themselves the financial backing of a major open source player like RedHat ...  Progress towards funding through personal donations doesn't seem to be working out too well.

Morgaine.

In case you are interested in running hadoop on your clusters, you probably want

a 64-bit architecture, because the recommended memory size for hadoop nodes

is 16-24GB for a balanced node, and 48-72GB for a compute-intensive node.

http://www.cloudera.com/blog/2010/03/clouderas-support-team-shares-some-basic-hardware-recommendations/

Nah, not interested much in Hadoop, I prefer Erlang for programming concurrent systems, and maybe Haskell would be a good match too as it's more or less the functional language of education.  What runs well on ARM will be one of the things I'm interested in discovering by making an Rpi cluster.

High performance isn't required (the Rpi would be rather weak as an HPC node), but as a host for general concurrent programming an Rpi cluster could be quite interesting.

@Richard: Referring back to my comment about using a more modern ARM if one were designing ARM cluster modules from scratch, one of my worries was that ARM's license fee for Cortex-A* cores would be too high.  After all, Broadcom probably chose an ARM1176 core for the BCM2835 purely to save money, we think.

Well that worry was probably ill-founded, judging by the short survey of ARM CPU costings in the first two paragraphs of the Allwinner A10's page:   http://rhombus-tech.net/allwinner_a10/

The license cost for Cortex-A8 must be very low indeed if the AM3358 price is $5 and the A10's price is $7.

Having mentioned the Allwinner A10 CPU, it immediately springs to mind that Rhombus Tech's EOMA-68 CPU card seems to be a plug-in cluster module all ready and waiting to go when it becomes available.  The BOM cost of $15 makes it highly relevant here, since it was the very low price of Rpi that created this great niche, and I'm sure that lots of other participants want to play the game too.

More references:

• http://elinux.org/Embedded_Open_Modular_Architecture/EOMA-68
• http://rhombus-tech.net/

Morgaine.

Interesting,

I am just now finishing a PCB design for an aerospace client that uses an X86-cpu mini pci module. Had to add alot of support and conversion circuitry such as LVDS to DVI , PCI Bridge and with loads of high speed, controlled impedance routing. A module like the EOMA would have greatly simplified if not for the need to support also VGA and special FPGA sync rx and tx drivers and serial ports in this seemingly archaic application. The mini-pci connector probably costs more than the EOMA itself will. Besides, they pay me to design their nightmares, not engineer my own obsolescence. I think I need to explore the details of the rhombus project deeper for real world applications.

In the meantime, the Raspberry is a quick turnkey product ready to tray and rack (if had access to about 8 or 16..^2 thing..) once things like power, booting, operating individually, plug and play and such needs to be handled. I imagine each Pi to have a snap-on tray with a front panel with the Eth and USB and LED indicators. The tray would slide into the rack. Back access would allow SD card and power plug access. Would be nice if the HDMI could be accessed also from the front for individual maintenance if needed. Dealing with all the Ethernet connections and cabling would be probably most challenging. Adding Wi-fi to each via USB might be cost prohibitive.

Yep, that mirrors my thoughts on the topic too, Richard.  Sticking the Pi into a bespoke plugin module is almost mandatory if one is going to use several, as hardwiring them together directly would be very unsatisfactory for a whole pile of reasons.  Designing a module with a little internal loom and replicating it is much more attractive.

And you mentioned LEDs .... No self-respecting project should ever skimp on LEDs, we need loads!!!!!

Morg.

In yesterday's webinar, Eben talks about clusters, at 56:25. 

He recommends that for building a cheap supercomputer,

you would be better off using x86 hardware (which is something

he has mentioned before).

What Eben said is quite right.  ARM isn't the world's fastest architecture, and within the ARM stable, the Pi's 700Mhz ARM1176JZF-S isn't going to win any speed races, so if one's goal is to build a supercomputer then to use a pile of Rpi boards would not be very effective.

But there are many more reasons for building clusters than just supercomputing.

One very common one is to run some types of Internet services which use little computing power per session but require multiple sessions to cater for a useful number of simultaneous Internet users.  For that, a cluster of multiple Rpi boards could be very effective if the application fits the constraints of a single Rpi well.

Also, ARM can beat most other processor families in performance per watt, so a cluster of Pi boards might well be able to hit a particular desired level of performance without requiring as much electrical power as say an Intel system.

Another application of clusters that isn't supercomputing is High Availability or resilient computing.  If your server cluster provides a number of independent hardware nodes and a heartbeat mechanism then graceful degradation can be achieved, reducing the overall system performance only slightly when one board dies.  This can be important even for home sites, allowing you to provide an effective Internet presence without losing sleep through midnight alerts, and being able to relax when on holiday.  Who knew, ARM clusters have social benefits.

And finally, actual applications aside, some people like myself want to work with clusters for the simple reason that, despite this being officially the age of multicore, concurrent software still hasn't really caught up with the multicore hardware that's been appearing for several years now.  Concurrency is still a research area even today some decades after I did my PhD in the subject, so I'd like to continue exploring the topic a bit more, and multiple Rpis would provide a perfectly suitable platform without breaking the bank.

And I'm sure that there are plenty of other good reasons for clustering Rpis as well, outside of supercomputing.

Morgaine.

Hi,

The idea of an ARM-based cluster is very interesting. I see a good applicationfield for for example SIP-based telephone systems. Scalability and high availability are here very important. Just as lower energy consumption.

Had not reflection of a value, a look at Q-Seven standard for example with this solution.

http://www.campro-cctv.com/products/13.Embeded/htm/Camera_01-0001.php

This board with Freescale ARM SoC has for 70 $ , PCI, Ethernet and everything that would be necessary for a cluster (imo).

And is available

And for more power freunds : http://www.iwavesystems.com/?q=node/239

The Q7 architecture has some advantages in my opinion, firstly for the development of the cluster and for the future for the use of other, more powerful modules.

Sorry for my poor English.

Zbig

Hi,

The idea of an ARM-based cluster is very interesting. I see a good applicationfield for for example SIP-based telephone systems. Scalability and high availability are here very important. Just as lower energy consumption.

Had not reflection of a value, a look at Q-Seven standard for example with this solution.

http://www.campro-cctv.com/products/13.Embeded/htm/Camera_01-0001.php

This board with Freescale ARM SoC has for 70 $ , PCI, Ethernet and everything that would be necessary for a cluster (imo).

And is available

And for more power freunds : http://www.iwavesystems.com/?q=node/239

The Q7 architecture has some advantages in my opinion, firstly for the development of the cluster and for the future for the use of other, more powerful modules.

Sorry for my poor English.

Zbig

I don't know the Q7 standard, but I do have some passive backplane PICMG rack units with dual redundant PSUs, and it did occur to me that filling their slots with ARM boards would be quite interesting and very low power.  Four Rpi mounted as daughterboards  on a carrier PCB that just takes power from a PICMG slot seems quite viable, and I have up to 20 slots available.

Hi Morgaine,

The people at the University of Munich have built and tested a cluster with ARM boards :http://www.nm.ifi.lmu.de/projects/ATV2CLUSTER/.

After doing this: http://www.mnm-team.org/projects/ATV2CLUSTER/atvcluster.pdf have read, I do not see a usable  solution for a cluster with 100MB Ethernet on an ARM board. Thereafter is a 1Gb Ethernet necessary for a reasonable load distribution.

BTW: Do you have any documentation about this PICMG backplane that you have ?.

Regards

Zbig

Thanks Zbig, those were interesting links.

I suspect that your comment about gigabit Ethernet is mostly with a view to using ARM nodes for HPC, which is not a very effective thing to do at the present level of ARM technology.  I certainly wouldn't consider using the Rpi for a supercomputing cluster, since its ARM core is rather slow and its GPU is not available for OpenCL tasks.  The result would be very poor.

But as I wrote earlier in this thread in answer to coder27, there are several good reasons for clustering ARM devices other than for performance, and that's my area of interest, not for HPC.

Re PICMG gear, my units are all from Advantech -- http://www.advantech.com/ , and their passive backplanes are here --http://www.advantech.com/products/Passive-Backplanes/sub_1-2JKOGC.aspx .  But there are tons of other players in the industrial computing sector offering similar systems (PICMG is a very long lived standard) -- Portwell, Peak, and Nexcom are other well known names that come to mind.

Morgaine.