Raspberry Pi server clusters

Morgaine, sounds very interesting and something I would like to play around with, however I am no-where near your level of expertise.

I do wonder how long it will take to be able to buy a cluster though when getting one seems hard enough

Hahahaha.

Was thinking same thing. Why just 1 Pi? How about 10.

One of my first enclosure mechanical designs is to house 10, slide in Pi's. Suitable airflow. Probably need some networking hub or router board, backplane for the GPIO , USB, JTAG and power, get rid of the audio and RCA connectors; they are a mechanical nuisance anyways. Light pipe in acrylic the LEDS for front monitoring. A brick of Pi's would be pretty powerful.

Yes indeed Richard, and such a server cluster would probably consume very little power too if we can disable enough subsystems, especially the graphics core.  (Another possibility is to use the VideoCore for GPGPU/OpenCL type functions, but at this point it's unknown whether that really holds any potential.)

Slide-in modules are a must when talking about 10+ boards, as wiring them together by hand will lose its appeal really fast.  I'm currently building a 3D printer which I hope to employ to print out suitable module holders and the rest of the structural framework.

PS. I only think in powers of 2, and 16 seems a nice number as a basic Rpi cluster node from which larger clusters can be built.

I was looking at building my own 3D printer, then I realized it was better to just spend the effort in quality design with a high quality, low cost 3D design syste and let the printer services do their job better and reasonably priced, like at Shapeways or ProPrinters. I do 3D CAD along with PCB so the overall design and manufacuring of computer and communications products is dear to me, (in reality, a multi-decade trans-century curse).

Now, if a guy could individually purchase the heart of the Raspberry, the BCM2835, plunk 10 down on a board, add circuitry for the LAN for all 10 and only 1 unit using the VideoCore, then a small form super computer could be formed based off that pretty nifty System on Chip processor the folks at Broadcom came up with. But then again, the overall price of the Raspberry as a demo unit for the Broadcom chip really trumps designing new system. Besides, there are many other ARM processors also suitable for Linux and thus clustering but I'm sure their individual cost plus assembly effort can't compete with simply stuffing a box full of already assembled Raspberrys.

I guess you could call this concept, the Raspberry Pi Raq

I like that idea, the "Raspberry Pi Raq".  Make the number of devices a power of 2 and I'll buy.

I wouldn't use the BCM2835 if I were designing boards for this though, I'd pick a more modern ARM device, particularly given that the VideoCore is not helpful in this application.

And, looking further into the future, I'd love to use the OpenRISC ASIC, which is projected to cost $5 in single units.  If only OpenCores would get themselves the financial backing of a major open source player like RedHat ...  Progress towards funding through personal donations doesn't seem to be working out too well.

Morgaine.

In case you are interested in running hadoop on your clusters, you probably want

a 64-bit architecture, because the recommended memory size for hadoop nodes

is 16-24GB for a balanced node, and 48-72GB for a compute-intensive node.

http://www.cloudera.com/blog/2010/03/clouderas-support-team-shares-some-basic-hardware-recommendations/

Nah, not interested much in Hadoop, I prefer Erlang for programming concurrent systems, and maybe Haskell would be a good match too as it's more or less the functional language of education.  What runs well on ARM will be one of the things I'm interested in discovering by making an Rpi cluster.

High performance isn't required (the Rpi would be rather weak as an HPC node), but as a host for general concurrent programming an Rpi cluster could be quite interesting.

@Richard: Referring back to my comment about using a more modern ARM if one were designing ARM cluster modules from scratch, one of my worries was that ARM's license fee for Cortex-A* cores would be too high.  After all, Broadcom probably chose an ARM1176 core for the BCM2835 purely to save money, we think.

Well that worry was probably ill-founded, judging by the short survey of ARM CPU costings in the first two paragraphs of the Allwinner A10's page:   http://rhombus-tech.net/allwinner_a10/

The license cost for Cortex-A8 must be very low indeed if the AM3358 price is $5 and the A10's price is $7.

Having mentioned the Allwinner A10 CPU, it immediately springs to mind that Rhombus Tech's EOMA-68 CPU card seems to be a plug-in cluster module all ready and waiting to go when it becomes available.  The BOM cost of $15 makes it highly relevant here, since it was the very low price of Rpi that created this great niche, and I'm sure that lots of other participants want to play the game too.

More references:

• http://elinux.org/Embedded_Open_Modular_Architecture/EOMA-68
• http://rhombus-tech.net/

Morgaine.

Interesting,

I am just now finishing a PCB design for an aerospace client that uses an X86-cpu mini pci module. Had to add alot of support and conversion circuitry such as LVDS to DVI , PCI Bridge and with loads of high speed, controlled impedance routing. A module like the EOMA would have greatly simplified if not for the need to support also VGA and special FPGA sync rx and tx drivers and serial ports in this seemingly archaic application. The mini-pci connector probably costs more than the EOMA itself will. Besides, they pay me to design their nightmares, not engineer my own obsolescence. I think I need to explore the details of the rhombus project deeper for real world applications.

In the meantime, the Raspberry is a quick turnkey product ready to tray and rack (if had access to about 8 or 16..^2 thing..) once things like power, booting, operating individually, plug and play and such needs to be handled. I imagine each Pi to have a snap-on tray with a front panel with the Eth and USB and LED indicators. The tray would slide into the rack. Back access would allow SD card and power plug access. Would be nice if the HDMI could be accessed also from the front for individual maintenance if needed. Dealing with all the Ethernet connections and cabling would be probably most challenging. Adding Wi-fi to each via USB might be cost prohibitive.