element 14 for the win

If the Foundation had realised say six months ago the massive interest and demand, yes they could have done things better, but the project gradually snowballed and has become something of a monster to handle.

Oh I don't think that the Foundation misjudged the level of interest at all.  Not only did they realize it, but were totally terrified of it.  That's why they turned off all their Internet services except for their single static release page, in order not to contribute towards a collapse at their end.

Unfortunately they made two huge mistakes:

• They single-handedly created the conditions for a huge instantaneous spike in traffic, by naming the exact time of the announcement and telling people to set their alarm clocks.  If they had said "Availability and pre-ordering details will be announced sometime during the 24 hours of 29th February, at least a couple of days before the units arrive in stock" then people would have gone to work or to sleep normally, and the traffic would have been very gentle and spread out through the day.
• They shut down communications with the community entirely. Twitter did nothing but increase the sense of panic among bewildered fans, because everyone was witnessing everyone else having the same trouble as they, and any "official" tweets were rapidly swamped by the desperate masses --- it's simply the wrong mechanism for company presence.  Element 14 was a bit more useful, but unfortunately not many people came here to the Raspberry Pi group, and there was no official Foundation presence here acknowledging that the two commercial sites had collapsed and offering advice on what to do.  Perhaps they found it difficult to accept what was happening, or the sheer magnitude of the disaster.

The Foundation can't be held to blame for website inadequacies at Farnell/Newark and at RS though.  The companies were warned, and it's no secret that there were ~100k highly expectant people on the RasPi mailing list ready to pounce at 0600 GMT.  The fact that the webbies employed by two major industrial-strength distributors have no clue how to design resilient and scalable websites is sure to be causing waves inside the companies concerned, but that cannot be held against the Foundation.

There is NEVER an excuse for a well designed web service to crash or to hang through excess requests, particularly when advance warning is given so that more machines can be added to the load pools.  When back-end databases or front-end webservers serving dynamic content exceed their maximum rated throughput, the front end should simply return a static page explaining the problem and throttle requests by IP address a little, in proportion to the problem.  It's not rocket science.  It's just normal negative feedback, and without it a website is not a well engineered system.  It's a system designed to collapse, and it did.

Anyway, it's spilt milk.  On the positive side, one only learns by making mistakes, rarely from success.  There are a couple of things for people to learn here, and then focus on the future of RasPi, which I think will be glorious.

Morgaine.

RS are doing OK.

Remember that neither Farnell nor RS had any stock on Wednesday morning , so if Farnell sold anything then they sold a promise.

Quoting Raspberry Pi Foundation Weds at 6AM GMT:

"Although we are still waiting for units to arrive from China, you can start buying the Raspberry Pi today. We have entered into licensed manufacture partnerships with two British companies, Premier Farnell and RS Components. "

RS are now contacting those that registered interest to tell them that they will be able to order soon from the RS allocation. They did the right thing in my opinion, they didn't have stock to sell so they will only take payment when they do, and they will be able to do this in a more controlled and less frenzied manner.

It was obvious that the websites were going to get wiped out on Wednesday. The technology available at a component distributor is not ideal for these kind of releases. Ticketmaster would have been a better bet.

I say well done RS for their careful approach.

I don't like to point the finger at Raspberry_Pi, because their work so far with the publicity and the technical stuff has been brilliant. But that release chaos could have been done better.

In defence of Farnell/RS techies and their web servers, there may not have been a lot they could have done

Their systems and servers are probably designed to handle normal traffic levels for them plus a bit more for redundancy/growth. I am sure they do have a plan to scale up as they grow, but even with some warning, it can take time to implement this - servers have to be purchased, delivered, space found for them, racked, os + software installed, tested and finally put live

Also, the traffic they got on Tuesday was probably hundreds of times their usual traffic level. Even if they expected this and managed to get the (tens or hundreds) of new servers in on time to handle this load, what should they do with these servers once traffic goes back to normal levels. Most companies cant afford to have servers sitting around doing nothing.

Adding code to return static page under load is possible. But it takes development time to implement to write and test. If you have enough hardware to handle say 5 times the maximum expected load which you arent expected to reach for several years (by which the time the hardware will have been replaced and so you could handle more load), would you really want to to spend time doing this should never happen

Finally this is assuming their systems can scale infinitely. There are always decisions made that may stop this being the reality

For example

Their website normally takes x number of requests

This can be handled by 2 web servers

They have 4 web servers for resiliance

They have a choice of load balancers they can use - one that can scale up to 10 web servers that costs £1000, or one that can scale up to 100 web servers that costs £10000

Any sensible management would go for the £1000 load balancer as it provides enough for 5 times the current traffic level which wont be reached for several years, rather than go for the £10000 one that provides enough for 1 days traffic for a new product that nobody had heard of and they werent expecting to stock when this decision was made

In reality, there are probably only a handful of companies that are set up in the UK that could have handled the amount of traffic that was generated on the 29th (amazon, johnlewis.com, tesco.com etc) - none of which are an obvious place to sell raspberry pis or have the global logistics to ship internationally.

Only trouble with that theory was the NZ version of RS, you couldn't find ANYTHING when searching, even 12 or 13 hrs after the main event.

At least NZ.Element14 had it, and you could find it, and order it.

Hindsight is a wonderful thing, and sure there are lessons to be learnt, but remember the organisation (as far as I can see) is mostly Engineers, and practical people, rather than Vapourware and Spin Doctors. The fact the units got held up probably didn't help their situation.

The forums seem to have a huge amount of question askers whom couldn't read or search, so I'm not sure that any type of information about pre ordering, etc would help.

To be fair to the suppliers, this has to be the first time they have encountered this level of interest, and does it justify the expense for the additional hardware and support for a device they are making a few bucks on...I don't think so.

From the flame throwing I saw on the forums, anyone who got static pages from the suppliers would have been equally vocal, so no win there either.

I am sure that the leadership will certainly be adding it to their forthcoming reviews to prevent a re-occurrance.

Mark

Anthony Brown wrote:

 

In reality, there are probably only a handful of companies that are set up in the UK that could have handled the amount of traffic that was generated on the 29th (amazon, johnlewis.com, tesco.com etc) - none of which are an obvious place to sell raspberry pis or have the global logistics to ship internationally.

TicketMaster and See Tickets could have handled it. Either or both of these could have sold an individual voucher that would have been matched to your own ID (credit card). These vouchers would then have been redeemed in exchange for a PI.

The problem though, is that these companies are not known for their benevolence and would have wanted to make a charge on top of the Pi cost for their service to each voucher. Personally I would have paid an extra fiver without a problem, but that does go against the Raspberry Pi ethos.

From the flame throwing I saw on the forums, anyone who got static pages from the suppliers would have been equally vocal, so no win there either.

Static pages can be very informative, Mark.  The "static" doesn't refer to them never changing, but only to the fact that the file served is a pure HTML file and requires no server-side processing between being requested and being sent.

A background process could be generating updated versions of that static file periodically and it would still be "static" from the PoV of the webserver, which would be able to serve a colossal number of requests without making the web service keel over.  If such a page were to contain the number of Raspberry Pi boards still in stock or still left to be allocated on pre-order, we would have witnessed a completely different outcome, happy customers and happy distributors.

Good system design makes all the difference between total success and a very embarrassing collapse.

Unfortunately that's not the design M.O. of your average webbie.  Instead it's "Oooh, this web framework is damn sexy", followed by "What, we crashed? Not my fault that we had too many customers!"  That's the reality of web service design today.

Morgaine.

 

The "static" doesn't refer to them never changing, but only to the fact that the file served is a pure HTML file and requires no server-side processing between being requested and being sent.

 

Thanks for that, very informative to someone who is on the 'static' side.

Instead it's "Oooh, this web framework is damn sexy",

I used to avoid sites that seemed to have a lot of that.

That was before we got broadband..

Cheers

Anthony Brown wrote:

 

In defence of Farnell/RS techies and their web servers, there may not have been a lot they could have done

Yes I agree fully and tbh neither are a consumer orientated site and so their aren't used to having to handle heavy load at sale time. I've seen quite a few posts saying "just allocate the web servers more resources" or "just put in a few more web servers" its just not that simple.

Anthony Brown wrote:

 

In reality, there are probably only a handful of companies that are set up in the UK that could have handled the amount of traffic that was generated on the 29th (amazon, johnlewis.com, tesco.com etc) - none of which are an obvious place to sell raspberry pis or have the global logistics to ship internationally.

Yes and we've seen various retailers sites taken down when there have been the odd mad buying sessions such as the HP touchpad sell off last year. Its a shame that ebay couldn't have been used in some way as I'm sure they could take the load and have options for charities to sell without incurring much expense.

This launch was one that the Raspberry Pi Foundation will want to forget as soon as possible.

Sure. They introduced a product that was met with such overwhelming demand, that the web sites of the two distributors - in spite of their previous assurances that they could take the beating - virtually disappeared for hours. What a flop!

I am old enough to remember many computer products which met with overwhelming demand upon being launched, and customers had to wait for months before getting theirs. I don't remember anyone describing these events as colossal failures up to now.

My understanding is that the Pi foundation had to deal with suits from Farnel and RS, not with techies. This was a problem for two reasons: a) the suits probably had no clue that the Pi was going to be something big. ("Sure, we have this startup that claims their product will rock the world and our infrastructure will collapse under the weight of the demand. Ha!"). b) Even if the suits had had a clue that the demand would be enormous, they had no clue what to do about it TECHNICALLY. My guess is that the "webbies" whom you vilify had little if any idea aabout what was coming. If they had, they could have done a few simple things, which would not even involve infrastructure investments - such as providing the foundation with links to the exact pages where customers would place orders. They didn't do this, and customers had to search, navigate or float in the semi-dead sites, resulting in a tenfold increase of traffic.

This certainly wasn't the foundation's fault - it is obvious from the twitter feeds that they were unable to even communicate with the panicked distributors for hours on end. It was also definitely not the "webbies'" fault. What it was is a failure on the part of the suits running Farnell and RS. Hopefully they have learned something from the experience, although it is quite possible that the failure is at a more fundamental level - in the structure of these companies - and therefore much harder to fix.

Still, in the grand scale of things this is unimportant. The Pi was launched with a bang, the world took notice, and I 'm sure that within a few months all the problems will be ironed out. "A few months" may sound like a lot, but then again, Rome was not built in a day, neither was the Pi. Sure I would have liked to be able to order 10 Pis today and have them within a week or two, but that's not the way it is.

The Raspberry Pi isn't a flop, nobody said it was.  Quite the opposite, it's wildly successful, interest couldn't be higher.

The launch in contrast was an utter disaster of biblical proportions, making half the prospective buyers across the planet either interrupt their work day or their sleep, frustrating their attempts for several hours, denying them the ability to order or even pre-order, not providing them with information on the state of sales so they didn't know whether the 10k units had gone or they should continue trying, and creating a general sense of panic and huge disatisfaction as seen on the Twitter channel.

The launch wasn't just bad, it was a failure in every respect, failure in planning, and failure in execution.  The fact that the 10k units got sold when the number of interested parties was probably several hundred thousand isn't a sign of success.  That they would be sold rapidly was blatantly obvious to anyone with half a brain cell.  This kind of calamitous event was not required to get them sold.  Many others have suggested ways in which the initial batch could have been released in a controlled fashion instead of making server overload a near certainty by specifying an exact time.

Will this matter in the long run?  No, as long as all three parties learn from the experience and don't repeat it.  But learning requires accepting that failure occurred, and comments suggesting that all went well are quite surreal.

Morgaine.