Raspberry Pi server clusters

AMD is planning to make 64-bit ARMs for servers.

From ZDNet:

 

AMD has announced that it is teaming up with ARM to develop 64-bit ARM processors for servers to meet growing challenges for data centers. "AMD will transform the computing data center environment today," said AMD CEO and president Rory Read during a press conference on Monday afternoon, asserting that AMD will be the first company to offer both 64-bit ARM and x86 server processors.

More interesting news in this area:

• samsung-may-start-making-arm-server-chips  [slashdot]
• samsung_laying_groundwork_server_chips_analysts_say  [computerworld]

One thing that surprises me is that Intel aren't building up a server market presense based on multiple clustered Atom chips.  Indeed, Atom seems to be almost a stealth product for them, very low key, and that's pretty odd when the future clearly forecasts competition in power/watt from ARM.

Morgaine.

Morgaine Dinova wrote:

 

One thing that surprises me is that Intel aren't building up a server market presense based on multiple clustered Atom chips.  Indeed, Atom seems to be almost a stealth product for them, very low key, and that's pretty odd when the future clearly forecasts competition in [performance]/watt from ARM.

From my recollection of microprocessor history, Intel has never been into low power.  Intel's technological model is squeezing lots and lots of fast transistors onto a piece of element 14 and they are IMO better at doing this than anyone else.  This has allowed them to be lazy about architecture, since transistor performance has so far been able to win.  But all those fast transistors waste a lot of power and require mechanical cooling, which is one of the big reasons PowerPC has had much more success in industrial and automotive applications.  (ARM partners like TI and Freescale are now going after these applications.)

Another thing is profit margin.  Intel chips have always been really expensive.  Again from my recollection of uP history, I think the original quantity 1 price of the Intel 8080 was US$300 (not adjusted for inflation).  Yes, you could get an 8008 and its support chips for less, but the 8080 had much better performance and instruction set -- and it got a lot cheaper really fast.  However, whenever a new Intel uP comes out, it's generally in the same price range.  Compare to RasPi's US$5 SoC.

When a large company with large profit margins is faced with technology that can dramatically improve price/performance, they often retrench and sabotage internal efforts to take advantage of new technology.  Basically, they don't want internal products with improved price/performance to compete with the old ways of doing things which have enjoyed high profit margins.  Too often the large company stalls internal efforts so long that the company isn't ever able to recover.

My favorite example of this phenomenon is the IBM PCjr, which came out in 1984 -- the same year as the original Macintosh.  The PCjr could have taken advantage of newer Intel SoCs such as the 80186 and produced much better performance than the 1981 IBM PC, at much lower cost.  However, IBM didn't want the lower profit margin PCjr to take business away from the older PC, so they made sure the PCjr didn't compete by sabotaging its performance and giving it a toy keyboard.  Well, PCjr didn't compete with the PC -- or with anything else.

JMO/YMMV

Morgaine Dinova wrote:

 

One thing that surprises me is that Intel aren't building up a server market presense based on multiple clustered Atom chips.  Indeed, Atom seems to be almost a stealth product for them, very low key, and that's pretty odd when the future clearly forecasts competition in [performance]/watt from ARM.

From my recollection of microprocessor history, Intel has never been into low power.  Intel's technological model is squeezing lots and lots of fast transistors onto a piece of element 14 and they are IMO better at doing this than anyone else.  This has allowed them to be lazy about architecture, since transistor performance has so far been able to win.  But all those fast transistors waste a lot of power and require mechanical cooling, which is one of the big reasons PowerPC has had much more success in industrial and automotive applications.  (ARM partners like TI and Freescale are now going after these applications.)

Another thing is profit margin.  Intel chips have always been really expensive.  Again from my recollection of uP history, I think the original quantity 1 price of the Intel 8080 was US$300 (not adjusted for inflation).  Yes, you could get an 8008 and its support chips for less, but the 8080 had much better performance and instruction set -- and it got a lot cheaper really fast.  However, whenever a new Intel uP comes out, it's generally in the same price range.  Compare to RasPi's US$5 SoC.

When a large company with large profit margins is faced with technology that can dramatically improve price/performance, they often retrench and sabotage internal efforts to take advantage of new technology.  Basically, they don't want internal products with improved price/performance to compete with the old ways of doing things which have enjoyed high profit margins.  Too often the large company stalls internal efforts so long that the company isn't ever able to recover.

My favorite example of this phenomenon is the IBM PCjr, which came out in 1984 -- the same year as the original Macintosh.  The PCjr could have taken advantage of newer Intel SoCs such as the 80186 and produced much better performance than the 1981 IBM PC, at much lower cost.  However, IBM didn't want the lower profit margin PCjr to take business away from the older PC, so they made sure the PCjr didn't compete by sabotaging its performance and giving it a toy keyboard.  Well, PCjr didn't compete with the PC -- or with anything else.

JMO/YMMV

Hmmm, dunno.  Deliberate internal sabotage of developments that could lead to better technology at lower profit margins is one possible explanation, but it's singularly short-sighted when there is a possible external competitor coming over the horizon.  Assuming that Intel does do forward planning, the likelihood seems low to me.

It also doesn't seem very likely  for a second reason:  Atom exists, and works very well.  I don't know what the current state of play is, but a couple of years ago it was winning head-to-head reviews against all comers on performance per watt.  Nobody (sane) sabotages a winning product, surely?

That said, the performance of ARM has improved massively in the last few years, so perhaps the situation has changed, not in Intel's favour.

The CEO of ARM, Warren East, says in an interview at http://www.technologyreview.com/news/507116/moores-law-is-becoming-irrelevant/ :

"To me a PC is really just a smartphone in another form factor. [cut]  TVs are the same.

TVs are big smartphones. Computers are kind of medium smartphones."

I quote it mainly because it made me chuckle, and although it's to be expected that the ARM CEO would say such things, there's quite a lot of truth in it as well.  Computers are intrinsically the same, whatever the niche.  And as he says later in the interview, ARM certainly wasn't designed expressly for smartphones.

I just wish ARM would do something a little more explicit in the direction that their heads regularly speak about.  Without cluster interconnect becoming available as an optional but integral part of the ARM architecture so that we don't have a Tower of Babel of incompatible interconnects, ARM-based servers will have a hard time becoming ubiquitous.

Morgaine.

On our earlier topic of ARM versus Atom, this comparison of a new Cortex-A15 versus an Atom from 2011 is rather eye-opening --- http://www.anandtech.com/show/6422/samsung-chromebook-xe303-review-testing-arms-cortex-a15/ .

Executive summary:  ARM wins on idle, but consumption is in the same ballpark for both when running flat out.  The performance figures favour ARM in this comparison, although one should bear in mind that the Atom in question was an old one.

Morgaine Dinova wrote:

 

The CEO of ARM, Warren East, says in an interview at http://www.technologyreview.com/news/507116/moores-law-is-becoming-irrelevant/ :

 

"To me a PC is really just a smartphone in another form factor. [cut]  TVs are the same.

TVs are big smartphones. Computers are kind of medium smartphones."

 

I quote it mainly because it made me chuckle, and although it's to be expected that the ARM CEO would say such things, there's quite a lot of truth in it as well.  Computers are intrinsically the same, whatever the niche.  And as he says later in the interview, ARM certainly wasn't designed expressly for smartphones.

 

I just wish ARM would do something a little more explicit in the direction that their heads regularly speak about.  Without cluster interconnect becoming available as an optional but integral part of the ARM architecture so that we don't have a Tower of Babel of incompatible interconnects, ARM-based servers will have a hard time becoming ubiquitous.

 

Morgaine.

I hope that our Warren has his tongue embedded firmly in his cheek, or perhaps he's only concerned with his particular corner of the hardware world. Computet = smartphone = telly? Hmmm... perhaps in consumerland where it's only real tasks are to give access to media, "rich web content" (whatever that is), adverts, spam, oline shopping, more spam and then to become obsolete just in time for next gen. tech then maybe so. But, for folks like me who only really tolerate computers because they are good at doing hard sums very quickly then I fear he's talking cobblers.

If ARM is to become ubiquitous then it will have to offer a bit more than low power (in terms of Watts and flops) at bargain bucket prices. It's a bit of a chicken and egg scenario, where potential adopters don't bite unless they are confident about format longevity and future legacy support (a non-consideration with consumer devices, but essential in industry). Industrial software types may similarly balk at turning out high value, low volume product for a platform that's "not quite done yet" - especially as not all ARM hardware is created equal... The chip makers themselves probably aren't going to toss in features that are currently seen as niche in the hopes of attracting a few customers when consumer grade whatnot and low cost high volume embedded applications are ticking along quite nicely. Oh, I forgot the need for a fit-and-forget operating system that software and hardware manufacturers will have enough confidence in to universally support.

The trick will be to nudge things over that R0>1 tipping point, but there are a bazillion little details (and one big roadmap) to finalise first.

Jonathan Garrish wrote:

 

The trick will be to nudge things over that R0>1 tipping point, but there are a bazillion little details (and one big roadmap) to finalise first.

I'm glad you pointed out the little issue of roadmap, because it's a very important issue in industry despite having no importance whatsoever in the consumer gadgets sector.  Industrial and commercial players need to know that the ARM-based server that they'll buy tomorrow from Dell and others is going to have an evolutionary path for many years ahead before they'll start investing in non-x86 software.  Currently there is no indication of a concrete roadmap in that area from ARM whatsoever, AFAIK.

ARM likes dropping vague hints about servers and about ARM licensees delivering the goods through competition in the market, but very oddly they totally fail to realize that they have a crucial role to play in establishing the foundations upon which a server sector will be based.  There's a lot more to it than merely defining an ISA and telling licensees to get on with it.  That doesn't inspire confidence among prospective buyers at all.

To bring the aggregate performance of a server based on low-power ARM chips up to that of a modern Intel/AMD server requires a lot of cores, and ARM can't use an SMP architecture for this like Intel and AMD are currently doing.  Shared memory has extremely limited scalability, and a lot of cores would rapidly hit the ceiling even with fancy multi-level caching architectures (which introduce their own problems anyway, lots of them).

The scalable way for ARM to go is with a clustering approach instead, using on-chip interconnect hardware for parallel communication between cores on the same chip or on the same board without distinction.  This would allow server boards to scale to an arbitrary number of cores both on-chip and on the server motherboard.  It's not rocket science either, as the transputer pioneered that architecture back in the 80's.

But for that to happen, ARM needs to make the interconnect a standard feature that ARM licensees can add to their ARM SoCs, a standard feature supported by standard instructions so that we don't end up with the Tower of Babel I mentioned above.  And I don't see ARM doing anything like that yet.

Morgaine.

It's worth adding that such on-chip interconnect hardware would have tremendous impact far beyond the limited area of ARM server communications.  Just imagine the possibilities if your Cortex-A application processors could talk to your Cortex-M microcontrollers at gigabit rates on separate links instead of crawling along at SPI or I2C speeds on shared buses.  Suddenly a whole new class of applications becomes possible.

If the Transputer architecture was so great how is it that there are no transputers now?

The ideas live on in XMOS and while they aren't bust they are only achieving niche sucesss on a a very small scale.

The reality is that that parallel at the core level is far from sorted - it isn't rocket sicence (we know how to make rockets).

There are lots of core level experimental parallel schemes afoot, GPUs, Greenchip, XMOS, Propeller etc -- none of them seem to be that compelling (except perhaps GPUs).

So - since you pose the question - what are the possibilities of your Cortex A linked to Cortex M that you can't do right now with the Xilinx Zynq ?

Michael Kellett

Morgaine Dinova wrote:

I'm glad you pointed out the little issue of roadmap, because it's a very important issue in industry despite having no importance whatsoever in the consumer gadgets sector.  Industrial and commercial players need to know that the ARM-based server that they'll buy tomorrow from Dell and others is going to have an evolutionary path for many years ahead before they'll start investing in non-x86 software.  Currently there is no indication of a concrete roadmap in that area from ARM whatsoever, AFAIK.

One of the things is that for Intel the server market is an "evolution" of their existing market share. So they can plan ahead and have new processors for the server market in the pipeline.

ARM however, doesn't have a foothold in the server market. If their server-chip-experiment fails, they will end up with a lot of money down the drain, and they'll have to struggle to survive.

In that case, they won't continue throwing money at the dead project. So I understand that they cannot plan beyond their first server-chips.

The problem is that software is SO VERY important that most likely the architecture switch won't happen.

It has been shown time and time again that the installed-base-software-compatible processor wins. Add (slow) hardware X86 emulation support and suddenly you've got a much bigger chance of succeeding because you provide an upgrade path for those having older software. That's what made AMD64 succeed.

(When emulating another architecture, having hardware support for the basics helps a lot. We tried emulating x86 on an architecture our group designed back in the late 1980ies, It turns out 90% of the instructions was dealing with the difference in flag-setting of the emulated instructions compared to the native computer. Having that in hardware speeds things up enormously)....

Roger Wolff wrote:

 

The problem is that software is SO VERY important that most likely the architecture switch won't happen.

It has been shown time and time again that the installed-base-software-compatible processor wins. Add (slow) hardware X86 emulation support and suddenly you've got a much bigger chance of succeeding because you provide an upgrade path for those having older software. That's what made AMD64 succeed.

Pardon me while I fire up my IBM PC XT/370

Actually, these days ARM-based computing devices way outsell x86 computing devices when you include smart phones and tablets.  When Google's Dual Cortex-A15 Chromebook starts shipping in ernest for US$249 the inverted pendulum will swing even further in ARM's direction.  The x86 will still have its place for people who need higher performance or if software is not available on ARM (such as FPGA design), but most people will do just fine with ARM and ARM will take over those applications just like x86 won out over System/370 due to better price/performance and performance/watt.

JMO/YMMV

Michael Kellett wrote:

 

There are lots of core level experimental parallel schemes afoot, GPUs, Greenchip, XMOS, Propeller etc -- none of them seem to be that compelling (except perhaps GPUs).

 

So - since you pose the question - what are the possibilities of your Cortex A linked to Cortex M that you can't do right now with the Xilinx Zynq ?

There was a very good article in IEEE Spectrum last year by Peter Kogge on Next-Generation Supercomputers.  I found this to be the most interesting take-away:

The good news is that over the next decade, engineers should be able to get the energy requirements of a flop down to about 5 to 10 pJ.  The bad news is that even if we do that, it won't really help.  The reason is that the energy to perform an arithmetic operation is trivial in comparison with the energy needed to shuffle the data around, from one chip to another, from one board to another, and even from rack to rack.

This has been my experience as well: building high-throughput processing engines is easy.  The difficult part is getting operands to them so they can do useful work.  This is why DSPs have specialized high-speed multi-port memories, but they're small and only work for small data blocks that get reprocessed many times.  A GPU that acts as a SIMD pipeline is also very effective for some applications.  But you can't expect a general application to get much sustained performance without a lot of work to make it fit the parallelism of the hardware.  It's easy to get peak performance, defined by a wag as "a guarantee from the manufacturer that you won't go faster than this".

IMO the obvious solution is to design the parallel processor's architecture to match the parallelism of the application.  If the application is a good match to GPUs, use GPUs.  If it's a good match to FPGAs and their huge amount of processing (provided that you can get operands to the processing elements), use FPGAs.  However, there's a big non-technical problem: GPU and FPGA vendors won't give you direct access to their architectures, so work in using these incredibly powerful engines for parallel processing is advancing very slowly.  It's much easier just to network up a bunch of high-end x86 CPUs and pay the electric bill.

We could do a hell of a lot with a Xilinx Zynq -- if we could program the logic array directly.