RG1 1.8v regulator

As shown in Troy Mackay's post on Jul 28 it seems to me that we could mod our PI's to work more as the chips were designed.

I think that this could make the PI more stable, from the looks of it.

I am not a designer Just an old tech, but I think we need to find a way to FIX what we now know is an error in the board.

So could someone do some practical testing to see if there is something that can be done to easly fix the current board, rather than wait for RPI foundation to fix it by waiting for a board redesign?

i will love to try out the hack Troy mackay has done also and then test again with that fix on the board, but i have looked into this and i most say it is very well done by Troy as i think it is to small for me todo and i done have an microscope as need for this.

so yes if some one can findout where to make an cut to split the LAN9512 1v8 from the lod 1v8 then i will try this also.

Tooms

Hi

If it has a different signal name it is not connected to the 1V8 plane.  Like most good CAD systems it does 'mostly' what it's told.

I've been travelling again and not been able to get to my lab but I'm now in 'full' duplex conversation with Microchip (ex:SMSC).

I see the issue with 'higher' temp but now I get the feeling you are looking to fix something else with this mod?  Can you confirm?  There is mention of the USB issue - point me at it please and anything else you think is connected.

Cheers

Pete

coder27 wrote:

 

Jamodio ought to be a hero on the RPi forum for finding what is probably

the most serious design error.  Instead, he is banned as a concern troll.

I don't think I have to be considered a hero, somebody else would have noticed and reported the error(s) if RPF had released the schematics and Gerbers before sending the boards to production.

There is another major issue with the quality of some of the components as I reported before. In particular the LDO voltage regulators that on the beta board were from NXP and now are from some Chinese knock off.

In my company we are moving away from buying some cheap components produced in China, it is well known in the electronics community that counterfeiting of components have been increasing substantially, like one of the more known cases being the Nichicon aluminium electrolytic blowing up on monitors and computer switching power supplies since the counterfeit part used a very low quality dialectic and the part does not meet the specs.

To tell the truth this is not a complex board in number of parts, signals, nets, and we engineers often make errors, that's why when we work professionally we have a review process before we commit to large scale production. In this case the process didn't exist or some people were not paying attention.

What really concerns me (after all I'AM a "concern troll") is that this type of situation serves as a learning experience so we don't repeat the same errors or wrong processes, but I doubt that if there is ever a new revision or version of this board, we'll have the opportunity to review it before the hype-pi 2.0 starts.

My .02

-J

In my company we are moving away from buying some cheap components produced in China, it is well known in the electronics community that counterfeiting of components have been increasing substantially, like one of the more known cases being the Nichicon aluminium electrolytic blowing up on monitors and computer switching power supplies since the counterfeit part used a very low quality dialectic and the part does not meet the specs.

I must have replaced thousands of those in PC's, but that particular problem seems to be solved - at least from what I see.

From the people I know who work for chinese manufacturers it certainly appears that a large part of going to china is to do with getting counterfeit parts - if not the whole device!

Part of the problem is quickly going to become - if it hasn't already - that your 'original' NXP LDO will have been made in the same factory, on the same equipment, and to the same design as the chinese one. Maybe both will use the NXP design, maybe the chinese one. As long as they're not totally flaky and outlast the warranty there's a part of me that thinks nobody will care - stuff dying sooner means you replace it sooner and someone makes another sale. The beancounters like that idea

Have we all seen this before ?

I have no idea if it's real or someone just did a mockup to illustrate a point, but certainly made me smile

jamodio wrote:

 

What really concerns me (after all I'AM a "concern troll") is that this type of situation serves as a learning experience so we don't repeat the same errors or wrong processes, but I doubt that if there is ever a new revision or version of this board, we'll have the opportunity to review it before the hype-pi 2.0 starts.

+1.

Feedback on issues is part of the engineering process.  That's alien to Liz's hype & fandom process.

PeteL wrote:

 

As always - comments welcome.

+1

I've been trying to quantify one particular area of faulty operation, RF mice on self-powered USB hubs .  With dozens of tested (device X hub) combinations, and all 8 of the data points for RF mice showing complete failure on Pi, it's a very black-and-white test bench for USB functionality.  I happen to have a hot-running LAN9512, so I may be able to provide relevant testing if the 1.8v issues are thought to couple to USB operation.

Morgaine.

J

Connecting the two 1V8’s is a sub optimal piece of design (you can look that up in the engineering translation dictionary - starts with a 'c' ends in ‘up’), and I didn't think that was of debate.

The design did go through extensive review within the 'inner circle' of supporters who 'know' and have worked with the chips before especially the BCM. Both prototypes and pre-prototypes had this same connection and that of course made it more difficult. I also suspect, had there not been some reason to go and look, it would have never been identified. Just for the record, your technical input is appreciated, but rather than just implying - you could have just asked? Maybe you did earlier and I missed it.

We tried to do the best job possible with the limited support and resources that the early phase project had. (We were going to make 5000-10,000 beta boards max). None of the early stress testing revealed the issue and only a small population (sub 0.01%) are reported to run unbearably hot (although more may, just not flagged up because it does not concern their owner - they are just having fun with it). Of the returns I have for analysis, I have still have not found a real ‘steamer’. These could be in part due to poor underfill or even a short/defect elsewhere on top of what we are asking the chip to do.

There is an implication in posts that this is responsible for something else to do with USB but to date no info is forthcoming? (Stop press - just seen that other thread - will go and look later).

It doesn't matter if you spend £1 or $10M things slip through, and there have been some very high profile events in that $10M category in all walks of engineering!

I remember saying at the outset that Pi would never be perfect - just doesn't happen in engineering there is always something to be optimised, improved. What we do need to do is measure, evaluate, garner input and decide what, if anything, needs to be done. And I do appreciate the work done by people on this thread already.

Pete

Question is - were the Rubycon's real !

P

Pete,

You wrote:

"There is an implication in posts that this is responsible for something else to do with USB but to date no info is forthcoming? (Stop press - just seen that other thread - will go and look later)."

I'm quite sure that there is no established connection at this point

between this 1.8v power issue and any USB or other functional problems.

Users with hot lan chips have been having USB problems, but so have

users with cooler lan chips, so the USB problems could very well be due to

other causes.

SMSC has said "don't do that", but they haven't said what will happen

if you do.  We've seen pretty convincing evidence that the lan chip

will get hot, but don't know if it will malfunction, or cause a malfunction

on the other devices connected to 1.8v, or whether its expected lifetime

will be shortened.

you wrote:

"only a small population (sub 0.01%) are reported to run unbearably hot (although more may, just not flagged up because it does not concern their owner - they are just having fun with it). "

I'm not sure what you are basing your statistics on.  Is it the return rate?

I suspect that the rate is higher than 0.01%.  If the rate was that low,

then certainly there would be no hesitation to announce that due to a

design defect, a very small number of boards have chips that run blisteringly

hot, and any user who is unlucky enough to have gotten one is welcome

to exchange it.

you wrote:

"We tried to do the best job possible with the limited support and resources that the early phase project had."

I don't think anyone would deny that you did a fantastic job with the limited

resources you had.  But I think jamodio's point is that releasing schematics

prior to production would not have cost anything, and could have resulted

in great savings by uncovering such errors before mass production.

I am not a hardware guy, but I am quite surprised to see that hardware

schematics aren't clear about the direction of power flow.  It is a bit

ironic that the beta board had a string of decoupling capacitors that

should have been connected to 1.8v, but wasn't, and the production

board has a similar string of decoupling capacitors that shouldn't have

been connected to 1.8v, but was.  But there is nothing in the schematics

to show which pins on the ICs have power going in, and which have

power coming out.  So it is very difficult to check the schematics to

find these kinds of errors, where components are either not connected

to any source of power, or are connected to more than one source.

At this point we are completely in the dark about what hardware revisions

are contemplated, other than Eben's mention of some unspecified pcb

change for FCC/CE residential compliance.  Hopefully that will change.

Eben said he wanted to fix the FCC/CE issue prior to the educational

release.  Since the 2012/2013 school year is about to start, the timing

seems really odd not to have fixed that by now.  The timing also seems

really odd not to have published the user's manual by now.  Amazon is

showing a projected date of 16 October.  So are you aiming for the

2013/2014 school year?  If so, I'm quite sure you will need 512MB ram

to be competitive.

First of all thanks for being here and for following up. I really appreciate the work you have done and I truly believe that some of the mishaps from the RPF are not your own making.

 

Connecting the two 1V8’s is a sub optimal piece of design (you can look that up in the engineering translation dictionary - starts with a 'c' ends in ‘up’), and I didn't think that was of debate.

Well, you may call it "sub optimal design" but it is actually an error, and in my dictionary it starts with 'f' and ends in 'up' ;-)

 

The design did go through extensive review within the 'inner circle' of supporters who 'know' and have worked with the chips before especially the BCM. Both prototypes and pre-prototypes had this same connection and that of course made it more difficult. I also suspect, had there not been some reason to go and look, it would have never been identified. Just for the record, your technical input is appreciated, but rather than just implying - you could have just asked? Maybe you did earlier and I missed it.

Obviously the process didn't work, and as you clearly know in the previous prototype it was reported that various power connections were missing, kind of a surprise since part of the 'inner circle' was apparently involved in the design of the BCM SoC chip. Perhaps the 'inner circle' has a very small radius and some of the supporters actually "don't know." Not just me but many other asked while before the boards went to production for schematics/gerbers and the only we obtained was a crop showing a psu section. I reported the problem as soon as the schematics were made public, and it didn't took too much know how, just reading the SMSC datasheet to figure what each pin was used for, something that we don't even have for the SoC part.

And about asking, I asked what else is on "the list", no response yet.

  We tried to do the best job possible with the limited support and resources that the early phase project had. (We were going to make 5000-10,000 beta boards max). None of the early stress testing revealed the issue and only a small population (sub 0.01%) are reported to run unbearably hot (although more may, just not flagged up because it does not concern their owner - they are just having fun with it). Of the returns I have for analysis, I have still have not found a real ‘steamer’. These could be in part due to poor underfill or even a short/defect elsewhere on top of what we are asking the chip to do.

The foundation should have put they arrogance away and ask for help and additional support, there has been a large group of people willing all the time to cooperate, and they are still out there but the RPF attitude has been always "what we did is perfect and we know everything." I'll not trust any number, percentages or analysis derived from them given that there are no public numbers about how many boards have been manufactured, how many have failed, how many have been sold. how many have been shipped, how many have been returned, etc, and there is no formal or reasonable system to track complains/failures/fixes.

There is an implication in posts that this is responsible for something else to do with USB but to date no info is forthcoming? (Stop press - just seen that other thread - will go and look later).

I'm not sure if there is a direct connection with several of the problems reported with USB. Certainly the entire power architecture does not help, but there are some hardware/firmware issues related to USB where things are not working as expected and somebody is now reading the Verilog files for that piece of silicon on the SoC discovering some limitations and other stuff hidden behind the obscurity of the drivers.

I know that everybody have tried to do their best, but recognition for a successful endeavour comes from producing positive results and not from the effort put to get them.

My .02

-J

So I guess it is not worth me publishing here what I know and what I can measure?

I do take issue with your final comment - it has been a success so far - with some measure of "success failure" thrown in.

I've been talking to users who just have one or two and are over the moon with them (faults issues and all) and they tell me that they have already learnt so much.

My overall positive view may yet be proved wrong, but as I said right at the outset it isn’t perfect, never will be - we just have to remember why we are clearing the swamp!

Off to watch the Olympics on TV

Pete

I am not a hardware guy, but I am quite surprised to see that hardware

schematics aren't clear about the direction of power flow.

In many years of working in the subcontract hardware assembly business I've seen hundreds of schematics from all sorts of companies, from the biggest names to tiny one man outfits and I've rarely seen anything indicating power direction.

This sort of thing wouldn't have been a problem in years gone by as it was rare for IC's to have internal regulators and often power was supplied from an off board supply so it was obvious.

Things change, technology gets more complex, leaving room for ambiguity and errors to creep in.

IME Petes schematics are pretty good, they lack some things you'd normally find on much larger schematics like a cross reference of refdes and signals to page and location, but for four pages most of that stuff isn't really necessary.

I am not a hardware guy, but I am quite surprised to see that hardware

schematics aren't clear about the direction of power flow.

In many years of working in the subcontract hardware assembly business I've seen hundreds of schematics from all sorts of companies, from the biggest names to tiny one man outfits and I've rarely seen anything indicating power direction.

This sort of thing wouldn't have been a problem in years gone by as it was rare for IC's to have internal regulators and often power was supplied from an off board supply so it was obvious.

Things change, technology gets more complex, leaving room for ambiguity and errors to creep in.

IME Petes schematics are pretty good, they lack some things you'd normally find on much larger schematics like a cross reference of refdes and signals to page and location, but for four pages most of that stuff isn't really necessary.

This sort of thing wouldn't have been a problem in years gone by as it was rare for IC's to have internal regulators and often power was supplied from an off board supply so it was obvious. Things change, technology gets more complex, leaving room for ambiguity and errors to creep in.

 

Selsinork,

  Thanks for the explanation.  It makes sense. 

Here I've assumed that at least since Intel's fdiv bug, hardware

guys have had the advantage over software guys because they

use mostly formal verification techniques where we rely mostly

on ad hoc testing.

  I couldn't figure out at first why it was difficult to verify that

all the components on the RPi board were properly connected

to power, which I assumed to be a pretty fundamental property

to be sure was verified, until I saw that power flow isn't

specified on the schematics.

  It makes one wonder what is meant by the claims that the RPi

design was carefully checked.  How can you check something

that isn't specified, especially when the datasheets for the ICs

are either non-existent or ambiguous themselves?

  As technology gets more complex, ambiguity has to be reduced

so checking can be increased, just to keep reliability from getting worse.

Here I've assumed that at least since Intel's fdiv bug, hardware

guys have had the advantage over software guys because they

use mostly formal verification techniques where we rely mostly

on ad hoc testing.

Don't get me wrong, formal verification has it's place and can help a lot. It also leads to two sometimes difficult problems

1. a simple error early in the design or even in a symbol/footprint supplied by a third party gets propagated through the design resulting in something that may not function at all, or as in this case works but is suboptimal. As the verification passes everyone has a high sense of confidence it'll work.
2. Over time it leads to a reliance of the verification technology and a de-skilling of the job - beancounters strike again.

In short there's no substitute for knowledge and experience. Experience of making the stupid mistakes and fixing them usually helps a lot.

If you're a software person, you're probably familiar with unit testing.  IME it suffers from variants of the same problems.

1. The spec is wrong, unit test to the letter of the spec and the result is wrong, but the test passes.
2. Often the person who writes the code also writes the unit test for it, if they have some misunderstanding of the requirement, the test will pass but the result is still wrong.

The same observation that there's no substitute for knowledge and experience holds for software just as much as for hardware.

A dodgy datasheet, or software spec, just means it's even harder to spot the error.

Hardware people do ad-hoc testing too, the tools are different and we literally get our fingers burnt - with a soldering iron

  As technology gets more complex, ambiguity has to be reduced

so checking can be increased, just to keep reliability from getting worse.

Agree, but as the complexity grows you're increasingly faced with a datasheet being 1000 pages instead of 50. It's trying to reduce ambiguity, but it's bringing up the needle in a haystack problem.

Then there's always the problem where increasing complexity gives you 17 different ways to do something, none of them obviously better or worse.

Since we're talking about testing now, and looking for ways to improve future versions of Pi, let's focus on that a bit since it's clear that testing needs to be revamped to a significant degree.  This observation stems from the fact that testing somehow managed to miss the dramatically widespread incompatibility with common class-compliant USB devices that users have encountered and reported, as well as not catching any USB event or data loss, nor the high operating temperature of some LAN9512 devices.

It seems that USB testing fell short in several respects:

• Sample size.  Even if devices had been chosen at random, with a sufficient sample size it should have become apparent that there is a large incompatibility problem and that data loss is common.
• Breath of devices tested.  Within the range of devices that fall within Pi's intended application area, a representative selection of device types and most common brands would have raised confidence that Pi works correctly with normal class-compliant devices that use only the generic drivers for their class.  It is very clear that Pi does not.
• Depth of functional testing.  This refers to performance, overhead and error rate characterisation.  USB supports various different types of data transfer, but the most common and important types are lossless, ie. these USB transfers are meant to be 100% reliable.  Users have found that USB data loss from mouse and keyboard are common, but this is hard for users to quantify.  Given the importance of these HID devices, the error rate should have been characterised at testing time.
• Physical tests.  Although only laterally related to USB testing, the wide variation in LAN9512 operating temperatures was not caught during Pi testing.  This suggests that either the test sample size was not large enough to reveal the problem or that it wasn't conceived that temperature measurement should be added to the set of tests that were performed.

Everything is a trade-off, and perfection can never be reached through testing.  However, basic correct operation is non-negotiable, so enough testing needs to be performed to guarantee that each of these categories are covered to a satisfactory degree.  That is the only way to have measureable confidence of engineering success upon product release.

Assuming that the premise is agreed, we then have to come up with a way to achieve this within cost and time constraints.  It has to be agreed first though.

Morgaine.

selsinork wrote:

 

If you're a software person, you're probably familiar with unit testing.  IME it suffers from variants of the same problems.

1. The spec is wrong, unit test to the letter of the spec and the result is wrong, but the test passes.
2. Often the person who writes the code also writes the unit test for it, if they have some misunderstanding of the requirement, the test will pass but the result is still wrong.

Expanding on (2), IMO you should always have different people designing and testing.  A good designer wants to make something work, a good tester wants to break it.  Getting both mind-sets in one person is a rarity, and inevitably leads to a conflict of interest.  Also, humans have difficulty perceiving what's really there versus what they want to see -- you should have someone else proofread your writing, and review your designs.

selsinork wrote:

 

Agree, but as the complexity grows you're increasingly faced with a datasheet being 1000 pages instead of 50. It's trying to reduce ambiguity, but it's bringing up the needle in a haystack problem.

There are good and bad tech reference manuals out there.  It all depends on whether the vendor sees a manual as (1) a way to get more design wins and require less tech support, or (2) an painful burden to be dispatched as cheaply as possible.  A 1000-page TRM isn't a problem if it's well-organized -- I can just print the chapters I need for my particular application.  Many SoCs have all sorts of functions that I don't need, so I can just ignore them provided that the SoC is well designed and there aren't unexpected interactions.  OTOH, some 3000-page TRMs are that way because the writer copied and pasted identical functions and then made small changes to each copy instead of thinking out clearly what would be most helpful to the designer and making one copy with a table of individual differences.

When a manufacturer is dodgy about documentation, it makes me wonder how well their products are designed, since a well-designed product generally starts with a well-written specification, which then provides the basis for a well-written TRM which doesn't cost much to complete.

John Beetem wrote:

 

When a manufacturer is dodgy about documentation, it makes me wonder how well their products are designed, since a well-designed product generally starts with a well-written specification, which then provides the basis for a well-written TRM which doesn't cost much to complete.

A TRM is written as much for the manufacturer's own staff as for the external audience.  If a company can't be bothered to describe to its own staff how its product works, it doesn't inspire any confidence that its staff actually knows how it works, so you can expect support and the company's support products to be poor or non-existent.

Orthogonal to the above, the days of monolithic TRMs or other large specification documents should have disappeared long ago.  Scalability refers not only to systems but to documentation as well.  Any non-trivial document needs to be hierarchical, distributed, and revisioned at each level of hierarchy, or it has no hope of being maintained to track the thing it describes.  Unfortunately today's prefered format for technical specs favours monolithic documentation.

Even if the delivery is "monolithic", internally inside the company the separation should take place.

With "monolithic delivery" I mean: "A single datasheet that describes the whole chip".

With separation I mean that the different chapters are maintained and written separately.

Care must be applied to ensure that when a module is functionally upgraded, the datasheets for the previous versions of that module don't get more complicated.

Having now read the last few pages of this thread....

The LAN chip is not designed to deliver power on the 1.8V. To stick to the specs the two signals need to be decoupled. i.e. a PCB redesign.

In practise the draw on the 1.8V is about 100mA. This means 3.3V-1.8V * 100mA = 150mW of power moves from the LAN chip to the regulator.

In practise, on the total power-draw of almost a Watt, the lan chip will be able to handle the 1/6th of a watt extra power. Maybe not over the full temperature range, so stuffing the 'pi into a mostly-closed enclosure may lead to unneccessary problems.

No recall of existing 'pi boards needs to be done. As said: The lan chip can handle it in practise, it is a question of conforming to the specifications, and being more robust.

If you are comfortable with using chips outside the official specifications, you can save a bit of cost if you leave off RG1. As it seems to work with RG1 removed, or with RG1 there is little current flowing from it. The lan chip seems to be able to cope.

Some people are expressing views that paralleled regulators will have one delivering ALL the current, and the other one none. This is not always the case, This would be the case if they are both ideal voltage sources (with an ideal diode in series). However in practise, both regulators would have a measureable output impedance. So if regulator 1 delivers 1.85V with an impedance of 1 ohm, and the other delivers 1.80V with an impedance of 0.1ohm, the first 50mA will be mostly delivered by the first regulator, but increasing the current draw to 100mA, the second regulator will provide about 50mA of that.

Roger Wolff wrote:

 

Some people are expressing views that paralleled regulators will have one delivering ALL the current, and the other one none. This is not always the case, This would be the case if they are both ideal voltage sources (with an ideal diode in series). However in practise, both regulators would have a measureable output impedance. So if regulator 1 delivers 1.85V with an impedance of 1 ohm, and the other delivers 1.80V with an impedance of 0.1ohm, the first 50mA will be mostly delivered by the first regulator, but increasing the current draw to 100mA, the second regulator will provide about 50mA of that.

The thing is, that isn't how regulators work, they aren't a voltage source with a resistive output.

Regulators, including the two linear regulators we're dealing with here pretty much always consist of a voltage reference with a feedback circuit controlling a variable resistance between the input power pin and the output power pin provided by either a FET or a bipolar transistor. If the voltage on the output pin is above the reference voltage by more than a tiny amount then the feedback circuit will shut off the output transistor and nearly all the power will be provided by the device with the higher reference voltage. Where it gets funny is when the two reference voltages are very close, both regulators provide part of the power and you can get into oscillations and all sorts of fun and games depending on the impedances around the circuit.

A TRM is written as much for the manufacturer's own staff as for the external audience.  If a company can't be bothered to describe to its own staff how its product works, it doesn't inspire any confidence that its staff actually knows how it works, so you can expect support and the company's support products to be poor or non-existent.

You've not worked at any of the places I have then

Reality tends to be that there'll be a warts-and-all document that only the design team in question will have access to. Someone will get tasked to produce a tidied up version which will then get sent to a dedicated team of technical writers who have no real knowledge of the device and somewhere in that loop it'll go to the lawyers before any sort of document can be made available internally, never mind publicly.

You'll also probably have four levels of document, publicly available, available after signing NDA, internally available and design team warts-and-all. So how much substance remains in the public doc ?

It's also highly likely they'll have some sort of records retention policy to make sure anything that could potentially be requested for a court case gets deleted ASAP.

IMHO, the lawyers probably have more input to the public doc than the design team does!

It might be nice to live in an ideal world where everything was done properly, the world we actually live in is more about protecting IP and maximising shareholder profit. In the eyes of the business people that means something completely different to what the engineers want.

To give you an idea, I've worked places where even employees use google to find the docs. Says a lot about the internal organisation..

It gets better, you'll probably find that nobody at SMSC/Microchip will know what the actual capabilities of the in-built regulator are. Their chip designers won't have designed a regulator from the bottom up, they will have included a regulator from a design library, it will have been specified to have a little bit of headroom over what the LAN9512 requires for its own operation. Beyond that we're into "experimental" territory.