element 14 for the win

Element14 and Farnell certainly did very much better than their commercial competitor, but to be fair, this launch was one that the Raspberry Pi Foundation will want to forget as soon as possible.  I wrote up a review of the day's painful events on G+ here:

https://plus.google.com/117463006296185212823/posts/RHZMDDKs3V5

I think what's important now is that the Foundation recognizes that things didn't go as planned, learns from its mistakes, and moves rapidly into Phase 2 to put this new-found industrial muscle to work.  Many hundreds of thousands of devices are needed even now in this initial hobbiest / development phase, and millions more once Rpi hits the educational bigtime.  There is much work to be done by all three partners.

And from our community end there is much work to be done as well, turning the Rpi ecosystem into something vibrant, innovative, educational, and inspiring.  But for that we need boards. :-)

Morgaine.

Element14 and Farnell certainly did very much better than their commercial competitor, but to be fair, this launch was one that the Raspberry Pi Foundation will want to forget as soon as possible.  I wrote up a review of the day's painful events on G+ here:

https://plus.google.com/117463006296185212823/posts/RHZMDDKs3V5

I think what's important now is that the Foundation recognizes that things didn't go as planned, learns from its mistakes, and moves rapidly into Phase 2 to put this new-found industrial muscle to work.  Many hundreds of thousands of devices are needed even now in this initial hobbiest / development phase, and millions more once Rpi hits the educational bigtime.  There is much work to be done by all three partners.

And from our community end there is much work to be done as well, turning the Rpi ecosystem into something vibrant, innovative, educational, and inspiring.  But for that we need boards. :-)

Morgaine.

"important now is that the Foundation recognizes that things didn't go as planned, learns from its mistakes" Well the only significant mistake the foundation made was to believe Farnell (E14) and to a lesser extent RS when they said there servers could cope with the rush!

Now that the sourcing of components, manufacturing, distributuion and finance have largely been off loaded they can concentrate much better on the fundamental aims of the charity.

If the Foundation had realised say six months ago the massive interest and demand, yes they could have done things better, but the project gradually snowballed and has become something of a monster to handle.

I now predict sales of 1,000,000 this year

Remember all* the members of the foundation are doing everything in their spare time.

*If I'm correct Liz may now be an employee, but she has been putting in 16 hour days 7 days a week, well above and behond the call of duty. I'm glad to see she has taken a break from twitter and the forums, for her sake and ours in the longer term.

If the Foundation had realised say six months ago the massive interest and demand, yes they could have done things better, but the project gradually snowballed and has become something of a monster to handle.

Oh I don't think that the Foundation misjudged the level of interest at all.  Not only did they realize it, but were totally terrified of it.  That's why they turned off all their Internet services except for their single static release page, in order not to contribute towards a collapse at their end.

Unfortunately they made two huge mistakes:

• They single-handedly created the conditions for a huge instantaneous spike in traffic, by naming the exact time of the announcement and telling people to set their alarm clocks.  If they had said "Availability and pre-ordering details will be announced sometime during the 24 hours of 29th February, at least a couple of days before the units arrive in stock" then people would have gone to work or to sleep normally, and the traffic would have been very gentle and spread out through the day.
• They shut down communications with the community entirely. Twitter did nothing but increase the sense of panic among bewildered fans, because everyone was witnessing everyone else having the same trouble as they, and any "official" tweets were rapidly swamped by the desperate masses --- it's simply the wrong mechanism for company presence.  Element 14 was a bit more useful, but unfortunately not many people came here to the Raspberry Pi group, and there was no official Foundation presence here acknowledging that the two commercial sites had collapsed and offering advice on what to do.  Perhaps they found it difficult to accept what was happening, or the sheer magnitude of the disaster.

The Foundation can't be held to blame for website inadequacies at Farnell/Newark and at RS though.  The companies were warned, and it's no secret that there were ~100k highly expectant people on the RasPi mailing list ready to pounce at 0600 GMT.  The fact that the webbies employed by two major industrial-strength distributors have no clue how to design resilient and scalable websites is sure to be causing waves inside the companies concerned, but that cannot be held against the Foundation.

There is NEVER an excuse for a well designed web service to crash or to hang through excess requests, particularly when advance warning is given so that more machines can be added to the load pools.  When back-end databases or front-end webservers serving dynamic content exceed their maximum rated throughput, the front end should simply return a static page explaining the problem and throttle requests by IP address a little, in proportion to the problem.  It's not rocket science.  It's just normal negative feedback, and without it a website is not a well engineered system.  It's a system designed to collapse, and it did.

Anyway, it's spilt milk.  On the positive side, one only learns by making mistakes, rarely from success.  There are a couple of things for people to learn here, and then focus on the future of RasPi, which I think will be glorious.

Morgaine.

In defence of Farnell/RS techies and their web servers, there may not have been a lot they could have done

Their systems and servers are probably designed to handle normal traffic levels for them plus a bit more for redundancy/growth. I am sure they do have a plan to scale up as they grow, but even with some warning, it can take time to implement this - servers have to be purchased, delivered, space found for them, racked, os + software installed, tested and finally put live

Also, the traffic they got on Tuesday was probably hundreds of times their usual traffic level. Even if they expected this and managed to get the (tens or hundreds) of new servers in on time to handle this load, what should they do with these servers once traffic goes back to normal levels. Most companies cant afford to have servers sitting around doing nothing.

Adding code to return static page under load is possible. But it takes development time to implement to write and test. If you have enough hardware to handle say 5 times the maximum expected load which you arent expected to reach for several years (by which the time the hardware will have been replaced and so you could handle more load), would you really want to to spend time doing this should never happen

Finally this is assuming their systems can scale infinitely. There are always decisions made that may stop this being the reality

For example

Their website normally takes x number of requests

This can be handled by 2 web servers

They have 4 web servers for resiliance

They have a choice of load balancers they can use - one that can scale up to 10 web servers that costs £1000, or one that can scale up to 100 web servers that costs £10000

Any sensible management would go for the £1000 load balancer as it provides enough for 5 times the current traffic level which wont be reached for several years, rather than go for the £10000 one that provides enough for 1 days traffic for a new product that nobody had heard of and they werent expecting to stock when this decision was made

In reality, there are probably only a handful of companies that are set up in the UK that could have handled the amount of traffic that was generated on the 29th (amazon, johnlewis.com, tesco.com etc) - none of which are an obvious place to sell raspberry pis or have the global logistics to ship internationally.

Anthony Brown wrote:

 

In reality, there are probably only a handful of companies that are set up in the UK that could have handled the amount of traffic that was generated on the 29th (amazon, johnlewis.com, tesco.com etc) - none of which are an obvious place to sell raspberry pis or have the global logistics to ship internationally.

TicketMaster and See Tickets could have handled it. Either or both of these could have sold an individual voucher that would have been matched to your own ID (credit card). These vouchers would then have been redeemed in exchange for a PI.

The problem though, is that these companies are not known for their benevolence and would have wanted to make a charge on top of the Pi cost for their service to each voucher. Personally I would have paid an extra fiver without a problem, but that does go against the Raspberry Pi ethos.

Anthony Brown wrote:

 

In defence of Farnell/RS techies and their web servers, there may not have been a lot they could have done

Yes I agree fully and tbh neither are a consumer orientated site and so their aren't used to having to handle heavy load at sale time. I've seen quite a few posts saying "just allocate the web servers more resources" or "just put in a few more web servers" its just not that simple.

Anthony Brown wrote:

 

In reality, there are probably only a handful of companies that are set up in the UK that could have handled the amount of traffic that was generated on the 29th (amazon, johnlewis.com, tesco.com etc) - none of which are an obvious place to sell raspberry pis or have the global logistics to ship internationally.

Yes and we've seen various retailers sites taken down when there have been the odd mad buying sessions such as the HP touchpad sell off last year. Its a shame that ebay couldn't have been used in some way as I'm sure they could take the load and have options for charities to sell without incurring much expense.

This launch was one that the Raspberry Pi Foundation will want to forget as soon as possible.

Sure. They introduced a product that was met with such overwhelming demand, that the web sites of the two distributors - in spite of their previous assurances that they could take the beating - virtually disappeared for hours. What a flop!

I am old enough to remember many computer products which met with overwhelming demand upon being launched, and customers had to wait for months before getting theirs. I don't remember anyone describing these events as colossal failures up to now.

My understanding is that the Pi foundation had to deal with suits from Farnel and RS, not with techies. This was a problem for two reasons: a) the suits probably had no clue that the Pi was going to be something big. ("Sure, we have this startup that claims their product will rock the world and our infrastructure will collapse under the weight of the demand. Ha!"). b) Even if the suits had had a clue that the demand would be enormous, they had no clue what to do about it TECHNICALLY. My guess is that the "webbies" whom you vilify had little if any idea aabout what was coming. If they had, they could have done a few simple things, which would not even involve infrastructure investments - such as providing the foundation with links to the exact pages where customers would place orders. They didn't do this, and customers had to search, navigate or float in the semi-dead sites, resulting in a tenfold increase of traffic.

This certainly wasn't the foundation's fault - it is obvious from the twitter feeds that they were unable to even communicate with the panicked distributors for hours on end. It was also definitely not the "webbies'" fault. What it was is a failure on the part of the suits running Farnell and RS. Hopefully they have learned something from the experience, although it is quite possible that the failure is at a more fundamental level - in the structure of these companies - and therefore much harder to fix.

Still, in the grand scale of things this is unimportant. The Pi was launched with a bang, the world took notice, and I 'm sure that within a few months all the problems will be ironed out. "A few months" may sound like a lot, but then again, Rome was not built in a day, neither was the Pi. Sure I would have liked to be able to order 10 Pis today and have them within a week or two, but that's not the way it is.

The Raspberry Pi isn't a flop, nobody said it was.  Quite the opposite, it's wildly successful, interest couldn't be higher.

The launch in contrast was an utter disaster of biblical proportions, making half the prospective buyers across the planet either interrupt their work day or their sleep, frustrating their attempts for several hours, denying them the ability to order or even pre-order, not providing them with information on the state of sales so they didn't know whether the 10k units had gone or they should continue trying, and creating a general sense of panic and huge disatisfaction as seen on the Twitter channel.

The launch wasn't just bad, it was a failure in every respect, failure in planning, and failure in execution.  The fact that the 10k units got sold when the number of interested parties was probably several hundred thousand isn't a sign of success.  That they would be sold rapidly was blatantly obvious to anyone with half a brain cell.  This kind of calamitous event was not required to get them sold.  Many others have suggested ways in which the initial batch could have been released in a controlled fashion instead of making server overload a near certainty by specifying an exact time.

Will this matter in the long run?  No, as long as all three parties learn from the experience and don't repeat it.  But learning requires accepting that failure occurred, and comments suggesting that all went well are quite surreal.

Morgaine.

It seems that there will be some time before the initial demand for Pis is satisfied. It is even quite possible that, due to the media attention, this demand will even increase.

It also seems that the Pi foundation plans to relase the hardware design (schematics, Gerber files etc).

If they do it soon enough, all those who believe they could have done it better than the foundation, will be able to try their hand at it - arrange with a Chinese factory to manufacture 20.000 Pis for you (or 200.000), and you can properly launch them, to show the foundation and the world how it is done. You may even get some pocket money out of it, if you really do it properly.

Then the foundation, relieved to find out that the production/launch/distribution of their brainchild is in good hands, will be able to concentrate on what it does best (to the best of my knowledge, they never claimed to be expert salesmen).

:-)

:-)

Interesting point, alfon1917.  Are Farnell and RS in competition with each other on Raspberry Pi, or do they adopt a common product policy in the manner of a cartel and don't try to out-do each other?

Either way, if neither of them commissions large batches to reduce the per-unit manufacturing cost and maintain stock, but instead if they both only trickle-feed the Rpi community by building to order using their "Register your interest" lists, then there will be ample room for a competitor to out-do them once the manufacturing information is available.

They will lose out badly to anyone who manufactures larger batches for stock, both on price and on convenience.  Nobody likes to wait 1-2 months, or however long it takes to order and manufacture a batch of boards to order, every time they want another board.

If you want an Arduino, a few clicks of your browser and it's on your doorstep the following day.  Farnell and RS are excellent at providing such next-day service (Farnell UK commonly holds over 1,000 Arduino Unos in stock), so if neither company does this for Raspberry Pi, there will clearly be something very amiss.