Build Challenge --- Super-Computer

I understand the speed difference in USB 3.0.. but there are multi gigabit ethernet things.. its called Fiber Optics.. however.. there is already a supercomputer for the budget. Check out Nvidia Tesla. using the extreme power of the GPU er.. multiple GPUs you can clearly get high performance supercomputing desktop..

my GPU (I currently have 2) is runing at just over 600 MHZ this does not seem very much because because my processor is running at 3333 MHZ and that is the transfer rate of USB 3.0..

here is why you may be wrong.

at 602 MHZ I am using 240 Processing cores PER GPU. Again, I have 2 GPUs

A tesla system can have 4 GPUs more powerful than mine

http://www.supermicro.com/products/system/4U/7046/SYS-7046GT-TRF.cfm?GPU=FC407 <<< check out this system

Supermicro has many suprecomputer systems tethered via gigabit Ethernet, (I saw them when I was in the navy) BUT.. somehow the motherboards the military choose did were not reliable and had to be replaced... alot..

but.. don't forget the data transfer rate of memory "unless you have DDR3" is fairly limited

my final thought is...  with a new Nvidia GTX 560 or 570 kids can in fact play crysis on max for around $1200

go to Ibuypower.com and see for youself

But fiber optics, generally speaking, isn't a household consumer item that "needs inventing" like a USB 3.0 switch could be, which is one of the things that Ben Heck puts down in his blog as a sort of  "pre-requisite" if you will for submitting ideas.  Although at roughly $800 for 1000 ft. it could work --- if you can afford fiber-optic router price ranges from $500 - $5,000.  A gigabit ethernet router costs around $160; USB 3.0, cheaper, but, heck, let's say it's not even 25% cheaper, could be $120 if done right.

In reference to your "multiple GPU" theory, there is a slight snag; running multiple GPU's in the same box can increase performance --- on programs that have been created to utilize those extra resources.

On most programs, they only use one or the other GPU, and while you are correct in asserting their dominion over the super-computing scene, there are a couple of programs available for free ( http://www.open-mpi.org/ and http://www.csm.ornl.gov/pvm/ ; bear in mind one or the other of these programs would need to be modified to call upon the USB bridgework, as they are currently adapted to gigabit ethernet alone ) that could effectively combine the resources (GPU's on each motherboard included) at a level that is above the hardware but below the program; as such any program run on this machine (this is an assumption, and we all know what happens to "u-and-me" when we assume) would run off of this collective pool of resources and, with a USB 3.0 setup, transmit data from the head node to the slave nodes much more effectively than in a system running gigabit ethernet.

AND, the point of the MAC addressing system is to ELIMINATE OVERHEAD that the USB 3.0 system uses, which would make for a theoretical amount of 5 GB/s (Might be less; let's be honest, nothing's perfect, and even if it were, it might still require SOME overhead), which is 1.6 GB's faster than what you're running at.  And, the supercomputing cluster advantage over your specific system (not including the system you linked, although I'll address my grievance about that in a moment) is that it has multiple motherboards, which could each contain multiple GPU's, and multiple processors with multiple cores which could all, if you were so inclined, run at 3.33 GHZ.  Four horses would pull harder than just the one, not-withstanding any GPU advantage/disadvantage.

As for the system you mention, I drooled when I looked at it.  However, it, much like the nVidia Tesla, is curiously devoid of a price-tag at first blush.  Perhaps someone has keener eyes than mine and can find out how much something like that would go for.  But again I suspect it might be a smidge higher than the $1250 computer cluster in the aforementioned link.  And again, four nVidia Tesla's running on gigabit ethernet (as the site suggests this system uses to link each of those "blades") would be fast, but not as fast as this same system with a USB 3.0 framework as I describe running the exact same hardware.

As for the limitations of data transfer rates for DDR3 memory and lower, you are, again, correct.  Even with DDR3 or higher, there is no assurance that the system can utilize this extra bandwidth due to Front-side Bus constraints.  However, Intel and AMD have both been researching their corporate posteriors off with their ( http://en.wikipedia.org/wiki/Intel_QuickPath_Interconnect ) and  ( http://en.wikipedia.org/wiki/HyperTransport ) technologies, respectively.  Hopefully they can improve FSB speeds in the same way USB speeds have been improved.  But one step at a time; get one or the other in FIRST, then do the other one.

To your final thought I say BAH!

( http://www.amd.com/us/products/desktop/graphics/ati-radeon-hd-5000/hd-5850/Pages/ati-radeon-hd-5850-overview.aspx )  Auto-detect in Crysis gives me a High setting with a Dell Studio XPS 8000 with no modding aside from said card and bigger power-supply.  But let's not get into the "my <insert brand> is better than your <insert brand>" debate because it's pointless.  Suffice it to say---I muddle through.

However, NONE of this conjecture, while fun and entertaining, may out do THIS technology ( http://en.wikipedia.org/wiki/Light_Peak ) in the relatively near future.

***Somehow, I am not surprised you were in the Navy --- most, if not all, the coolest entrepreneurial kinds of people were in some way, shape, or form militarily involved.  Me, I was just a military brat; close enough to no longer fit in with people, but far enough away to not stand at attention when my SO walks into the room. 

1. Why design a supercomputer to play a game that with a few tweaks and the right video card set up can be playered on most present computers? Thats like using a supernova to swat a fly.

2. Many companies now sell Personal supercomputers. Prices are high, in around 70K, but they are avaible.

3. when suggesting a idea, please consider price, Ben is not made out of money.

Your concerns are duly noted.

1.  The original purpose as stated in my original post is as follows. . . "Which would translate to more FLOPS a second.  Which would mean more bang for the buck on a researchers already anorexic budget.  Not only that but a device like this could potentially take the place of gigabit ethernet routers in the home." The critical element isn't so much the cluster, although, as I said $1250 bucks, while outside his $500 maximum spending limit (how often do projects stay within budget anyway?) is still a feasible project price-tag (if one was trying to make the USB 3.0/cluster combo their goal for this project); however, to focus solely on the USB 3.0 system, as the ONLY device needing inventing within this configuration, was my aim.  A facetious aside comment at the end of a much larger article hardly represents the fundamental aim inherent within the vast majority of the rest of my article.  I understand the dilemma as my post did tends towards ambiguity, and I apologize if my true motive was not readily apparent.

2. and 3.  My idea was originally intended as a viable replacement and upgrade for off-the-shelf gigabit ethernet routers put into what are known as Microwulf Clusters.  Researchers with a limited budget (not withstanding the millions in the consumer market currently using gigabit routers in home networks for a variety of reasons, not just home entertainment) who didn't feel like renting processing time on a larger super-computer, or purchasing that $70,000 dollar supercomputer you mention.  Researchers, like Ben, are also not made out of money, and this project, focussing solely on the USB 3.0 technology which does not exist as opposed to building a micro-super computer cluster which *does* exist is indeed well within his price-range (USB 3.0 hubs run about $30; adding in costs of the low-level programming that would go into this device, my estimate would put it at 120$, as previously stated in my 2nd post); the price-tag for the cluster was merely to illustrate the difference in price-range between professionaly made super-computers and their home-grown equivalents, reinforced here by comparing it to a $70K model.  He also mentions that his goal is to "help out" people who are in need, which it is my contention that such a device as I have suggested would.  Again, my apologies for not making this more distinct as a concept within my first two posts.

I hope I have addressed all of your concerns.

You know... the more I think about it the more this sounds better and better... but I don't think it can be a simple mod... I think this is a type of re-engineering. I seriously cannot wait to see an optical computer and light peak technology... I myself am trying hard to learn C++ (I am only in my first semester) but I really hope to be a part of a team to bring the next coolest thing to the market. I do have a great Idea... the first "box prototype" will be made of wood similar to those old modems for teletype systems.

If somehow we could trick USB 3.0 to communicate fro...

Wait...

SATA drives (mine at least) only support 3 GB per second. So... if USB 3.0 communicates at 3.5 GB per second this means that SATA 3 at 6 GB per second is overkill "which is better" BUT... what interrupted my thought is BUFFERS, you see, Data usually travels faster via light or bus than it can be saved on a hard disk which is why the HDD have built in cache.. So... a cluster of cheap systems networked via USB 3.0 would not be efficent enough to process data and save it to HD the parts you would need are... Motherboard supporting Intel’s new "Sandy Bridge" 6 core processors, USB 3.0 and at minimum SATA 2.0... But not just one of these systems... they have to be a cluster of them... because this is a supercomputing project... and last but not least... a solid state hard drive which.. According to Computerworld "the OCZ had a read time of 244MB/sec and a write time of 172MB/sec. The Seagate HDD had an average read rate of 98MB/sec and a write time of 87MB/sec." therefore even if we could somehow manage USB 3.0 cluster network on say 10 full power state of the art desktops the HDD can only transfer data at a MAX of 244MB a sec read and 172 MB a sec write.. So for now at least a 1000MB per second network cluster will do everything faster than it can actually save it.

HOWEVER, If and I do mean IF. If we can create software to load massive amounts of data on many many GB of RAM on state of the art memory modules running at 2000 MHZ "DDR3" and not use the hard drives for storage until AFTER all the data has been Processor crunched then perhaps USB 3.0 networking is worth the time.

You are correct, killerfishmonkey, this would be a type of re-engineering.  I was semi-hoping to adapt commercially available programs to my purpose that are already performing a similar purpose in gigabit ethernet. 

As for SATA being better, I looked into it a while back; while its transfer rate is better, Dev's have kept SATA to itself, as any of the sites that I've looked into say it cannot be used for networking computers together.  The only other technology that seems to come close, other than ethernet would be firewire.  However, firewire, while it could've exploded into something as big as USB, sadly didn't, and remains in its own niche as a technology that communicates with media equipment.

In reference to writing to hard-drives, one of the systems in that clustermonkey website uses only one HDD amongst four separate nodes.  I think this somewhat eliminates pitfalls when attempting to recall information because, as I understand it, the program installed on the "head node" will delegate specific portions of a program to the slave nodes, let them crunch it, and then call the information back using the RAM address, saving all of it to it's own RAM allotment (which leads me to believe that the head node should be an ungodly RAM-hog so it can overcome the pitfalls of any system designed with HDD's or SSD's on SATA) without the slave node having to save anything to a HDD.  Once back in the head node, the program inputs the values from the slave nodes into the "master program" if you will, which is then written to the HDD.  Cache's, from what I know of their use, are designed to hold values that are most frequently called; I would think (though don't quote me) that PVM or MPI would be able to allocate the onboard cache memory of each mother board so that, whatever segment of the program each slave node gets would have in its own cache system the values that would normally be called relevant to its own section of the program it was called to run. 

Although, to be honest, I truly and sincerely agree that SSD's are the wave of the future, and some of the cluster layouts in that clustermonkey website do actually call for a small SSD drive for uploading the operating system and other essential programs, then using a larger HDD for the rest.  Of course, as SSD's take off, hopefully we'll be seeing faster transfer rates.  However, I've been noticing in your posts that you seem to be interchanging MB/sec and Mb/sec to mean megabytes a second; the difference, while minute, is crucial, as it is not what I have been referring to in my posts, which is megabits/sec; it takes (usually) 8 bits to make one byte.  So, multiply the read and write speeds times 8, and you will have it's rough read / write time in mega (or giga) bits.  In looking back in my own posts, I have not kept up with the convention myself for writing Gbit/sec or Gb/sec vs. GB/sec.  My apologies, that was misleading of me.  The solid state drives near the top end of the spectrum advertise a read speed 355 MB/sec x 8 =  2 . 840 Gbit/sec which means this will *still* be our bottleneck---but maybe not for long, and it is still close enough to make a lot out of our system, and perhaps enable us to save more money by using even cheaper processors!  The same SSD advertises a 75 MB/sec write speed, so 75 x 8 = 600 Mb/sec.  Or take 3,600 Mb/s for USB 3.0 and divide by 8 which = 450 MB/s, which for some SSD's at about $600 would maintain that holy trinity of RAM - CPU - BUS speeds.

Now, using one that plugs into the PCI-E slot like http://www.ocztechnology.com/ocz-z-drive-r3-p84-pci-express-ssd.html this one advertises 1000 MB/sec read and 970 (sustained 900) write speeds meaning 8 Gb/sec and 7.2 Gb/sec respectively.  Which means an SSD like this which uses the bus speeds of the PCI-E (which is loooooads faster, especially used in conjunction with these particular SSD's, as we can plainly see) would actually take an even faster network, not slower.  This appears to be the top of the line at OCZ, and the slower read / write speeds of the Revodrives mean they are probably cheaper, but all of these SSD's are probably wayyyy outside the price range of the average consumer, or the average small-time researcher, as the case may be.  Again, hoping that demand fuels the need to drive down prices in the future, these will probably be the best options for our information storing needs, and maybe if we only buy ONE uber-expensive component, this particular system arrangement, with a $600 SSD drive will still be cheaper than the "ready-made" computer cluster/bladed systems.  Maybe like the one Maximum PC tested here http://www.maximumpc.com/tags/revodrive .

And yes, SATA 3 would be ideal for the HDD's that are currently wallet-friendly, and while any i Core processor incorporating sandy bridge into it (hopefully without the Cougar Point flaw) would produce eye-popping results, cheap, dual-core processors that the initial super-computing layout call for (bad read/write speeds included) produced tremendous benchmark results, performing at an astounding peak of 26.25 GFLOPS actually maxing out the systems available memory (8 GB's of RAM total; the horror. . .I have 6 on my one desktop alone) before it ran out of raw processing power.  Of course, the head node was built with slightly more robust system components, I think, incorporating a quad core into it as well as being the only motherboard connected to the peripherals and actually being capable of hooking into the internet, as, from what I'm told, a lot of super-computers are not.

killerfishmonkey, I must say it is truly a joy to be discussing this idea with you.  You keep throwing out other ideas for me to consider, and it's a fun challenge, as it tests what I've learned in all the research time I have put into this idea (which I have since lost track of).  I hope I have begun to persuade you of the viability of my idea, if for no other reason than to have someone else to shoot the breeze about it with.  And, I think my own super-computer cluster would be freakin' cool!  But that's beside the point.  I myself am a budding computer programmer (C# for me though, and getting into XNA programming) and I wish you the best of luck!

***IF MY ORIGINAL INTENDED AUDIENCE (Ben Heck) IS STILL READING THIS, I DO NOT INTEND FOR YOU TO SOFTWARE HACK ANYTHING: MPI AND PVM ARE OPEN SOURCE PROJECTS, WHICH I THINK MEANS THEY ARE *MEANT* TO BE MODDED!!!  THE USB 3.0 HUB CAN BE PROGRAMMED / MODDED USING FPGA's (FIeld Programmable Gate Array's) WHICH MEANS IT'S POSSIBLE NOT TO SOFTWARE-HACK THEM EITHER***

Forgive me System Admin, for I have committed the original sin of typing in all caps for dramatic effect. --- Say Five "Hail Hewlett-Packards" and all will be forgiven, my son.