PicoZed SOM failure - Improper power sequencing?

Improper sequencing can cause permanent damage to Zynq-7000, including the one on PicoZed. Xilinx documents Zynq-7000 “Power-On/-Off Sequence Requirements for PS eFUSE Integrity” in https://www.xilinx.com/support/answers/65240.html

Avnet's PicoZed FMC V2 Carrier follows the proper sequencing guidelines, so you should not experience issues when using the Avnet Carrier. However, if you have designed your own carrier and not followed the proper sequencing, then it is possible that you could design a carrier that is susceptible to eFuse damage.

INFORMATION FROM THE Zynq-7000 DATASHEET:

Before VCCPINT (SOM 1.0V Power Supply) reaches 0.80V at least one of the four following conditions is required during the power-off stage: the PS_POR_B (PG_MODULE) input is asserted to GND, the reference clock to the PS_CLK input is disabled, VCCPAUX (SOM 1.8V Power Supply) is lower than 0.70V, or VCCO_MIO0 is lower than 0.90V. The condition must be held until VCCPINT reaches 0.40V to ensure PS eFUSE integrity.

How to check a suspected failed system:

Use the XMD script attached to AR65240. It can read the PS eFUSE array for determining whether any PS eFUSE settings are different to the expected settings.

Carrier Card Design Requirement:

The PG_MODULE signal connects to the Zynq signal PS_POR_B. This open-drain signal should be utilized in a custom carrier card design in order to meet the Power-On and Power-Off requirements documented in the Zynq-7000 datasheet and further documented in Answer Record #65240 from Xilinx.

Xilinx Zynq-7000 Datasheet – PS Power-On/Off Power Supply Sequencing

https://www.xilinx.com/support/documentation/data_sheets/ds187-XC7Z010-XC7Z020-Data-Sheet.pdf

or

https://www.xilinx.com/support/documentation/data_sheets/ds191-XC7Z030-XC7Z045-data-sheet.pdf

Xilinx Answer Record #65240: https://www.xilinx.com/support/answers/65240.html

On Power-On, PS_POR_B (PG_MODULE) needs to be asserted and held LOW for a minimum of 100uS after PS supply voltages reaches minimum levels. Also, care must be taken to ensure that the PS_POR_B (PG_MODULE) signal is not de-asserted HIGH during the Secure Lock Down Window described in the Xilinx Zynq-7000 datasheets and Xilinx Answer Record #63149.

Xilinx Answer Record #63149: https://www.xilinx.com/support/answers/63149.html

  On Power-Off, PS_POR_B (PG_MODULE) needs to be asserted and held LOW prior to the PS power rails falling below acceptable thresholds identified in Xilinx Answer Record #65240 and maintain assertion through critical voltage levels as the PS power rails decay.

Hi Bryan:

Thank you for your response.

It appears that  I may be violating the carrier card guidance by allowing PWR_ENB to rise nearly coincidentally with the 5V VIN rail.  There is some time difference but the PWR enable is asserted before the 5V rail is stable.  Similarly, PWR_ENB is not removed before the VIN rail has fully dropped.   It would appear the POWER-OFF sequencing could cause the

The damage that I have found on about 2 of 18 PicoZed SOMs appears to be with regulator U11.  This this is the 1V regulator that is enabled by PWR_ENB. .  Even when I float the PWR_enable signal, the regulator does not produce 1.0V and the data sheet for the regulator states that the active current source should allow a floating enable to turn ON the regulator with a fixed under-voltage lockout threshold.

I have not isolated the 1.0V U11 output to know if the problem is with the regulator or its load.  Can the 1V rail of the Zynq be damaged and casing the regulator to current limit to a voltage below 1.0 V?  Have you seen any cases where U11 failed due to ESD or turn-on overshoot?

I will try and check the PS eFUSE array as suggested.

Thanks again,

Craig

Hi Bryan:

With respect to my last question, I am still unsure whether the apparent U11 regulator low voltage output of about 0.38V is due to regulator failure or due to excessive current drawn by the Zynq core's 1.0V input?  Could eFUSE damage cause the Zynq to draw excessive current?

In any case, it is clear to me that the original power-down sequencing of my carrier card was not correct.  The design of the carrier card seems to take advatange of an ON/OFF switch that assumes that 12V power is always available and that is not practical in our application as an instrument.  With respect tot he guidance you provided for Carrier Card Design repeated below:

Carrier Card Design Requirement:

The PG_MODULE signal connects to the Zynq signal PS_POR_B. This open-drain signal should be utilized in a custom carrier card design in order to meet the Power-On and Power-Off requirements documented in the Zynq-7000 datasheet and further documented in Answer Record #65240 from Xilinx.

Xilinx Zynq-7000 Datasheet – PS Power-On/Off Power Supply Sequencing

I reworked our design to detect loo of the main input power bus early before any other rails, such as the PicoZed SOM 5V VIN  rail, are disturbed and pull down the PG_MODULE line.

The new timing is shown for power-on and power-off with

Yellow = 5V input to PicoZed SOM

Blue = PWR_EN

Magenta = VCCIO_EN

Green = PZ_PG_MODULE

I believe this meets the requirements because with PG_MODULE low, the Zynq POR should be asserted before VIN falls and hence VCCIO_EN falls.  My only concern now is the timing of PWR_EN and VCCIO_EN - Is this adequate?

Just for clarification, the internal U11 1.0V regulator will turn ON when PWR_EN reaches 1.25V so even though the capture shows the blue trace slowly climbing in the power on case, the 1.0V rail is active good well before VCCIO_EN occurs.

Thanks for the support,

Craig

Hey Craig,

I designed the power solution for PicoZed so Bryan reached out to me to chime in as well. Our design follows the startup sequencing (assuming the carrier bank voltages are held off until VCCIO_EN is released) but we do no active power down sequencing. Pulling power from everything simultaneously results in everything dropping at roughly the same rate which will not cause damage. Make sure that VCCIO_EN is truly grounded when power is shut down (only if there is still an active input voltage somewhere in the system). The reason I say that is we did see situations at one point where powering down the SOM would not fully shut down the board. There was a parasitic voltage that held the VCCIO_EN signal just high enough that it did not shut down the bank voltages. If that happens with the core shut down, it can damage the device. That is why we provide extra guidance now on making sure VCCIO_EN is truly grounded to make sure those bank voltages turn off when the SOM shuts down. Please let me know if that is helpful, if I missed the mark on the question I'm happy to provide more support.

Thanks
Chris

Hi Chris:

I greatly appreciate your follow-up and hope that you would be willing to help me optimize my carrier card design.

To be candid, I am embarrassed that I did not, and may still have not, properly digested the power sequencing information provided in the Carrier Card Design document and subsequent guidance from you and Bryan. So I want to clearly understand how to modify my design that is now in early production.  If you will indulge me, I want to describe my concerns/questions and get your knowledge and guidance on how to proceed.

1. One of my unanswered questions has to do with the perceived failure of the PicoZed SOM itself.  We have seen damage to two or three PIcoZed SOMs out of about 16 in our production build and test. My analysis shows that a “bad” PicoZed SOM is one where the D3 LED does not come on because the 3.3V regulator [U13} is not ON.  Further investigation finds that the 1.0V regulator the one that feeds the Zynq core and is enabled by PWR_EN is not outputting 1.0V.The voltage is < 0.5V.  My initial hunch was that improper power sequencing caused damage to the SOM but it is not clear if the regulator is damaged or if the Zynq, itself, is damaged.  My understanding is that the PS eFUSE matrix could have been damaged by improper power-off sequencing but could that cause the Zynq to draw excessive power from the 1.0V rail?  We plan to install a new U11 regulator to see if the problem exists.  If it does, then it points to Zynq damage? What are your thoughts on this failure mode?  Have you seen a similar fault condition for U11 caused by improper power sequencing?
2. Bryan provided the following guidance in his forum reply:

Carrier Card Design Requirement:

The PG_MODULE signal connects to the Zynq signal PS_POR_B. This open-drain signal should be utilized in a custom carrier card design in order to meet the Power-On and Power-Off requirements documented in the Zynq-7000 datasheet and further documented in Answer Record #65240 from Xilinx.

I used this guidance to modify my power sequencing circuitry and used PG_MODULE to initiate a power down.  The main reset push button also pulls down PG_MODULE.  .  In my design, a 24 AC-DC supply is used to power our instrument.  The 24V bus feed several regulator modules including a 5V module. I include  an LTC2965 voltage monitor to detect when our main 24V input is below 20V.  On power -on, this monitor keeps the PG_MODULE low until well after the 5V rail is stable and +5V is supplied to the SOM and VCCIO_EN has been asserted good. On power-off, the monitor pulls PG_MODULE low well before any drop has occurred on the 5V SOM.  The older approach pulled VCCIO_EN before PG_MODULE was pulled low and while the 5V rail and PWR_EN was still active. 

In my carrier card design, the 5-channel regulator (ADP5052) design was lifted from your carrier card design so that when VCCIO_EN is asserted by the SOM, it enables the AV1.0V rail generated by the ADP5052 which then kicks off the chain of regulators.  The new power-up timing looks like this VCCIO_EN coming up before PG_MODULE does and I believe this is correct.  Note that I provided a delay in asserting PWR_EN (blue trace) by inserting a 10 nF capacitor  on the PWR_EN pin to ground to take advantage of the 1.9 uA internal pull-up current source in U11 but since U11 turns ON when PWR_EN reaches 1.25V, U11’s 1.0V output (not shown)  is definitely up before VCCIO_EN is asserted.  Is this power-on timing acceptable?

Yellow = 5V input to SOM

Blue = PWR_EN

Magenta = VCCIO_EN

Green = PZ_PG_MODULE

The new power-off timing looks like this:.  I believe this is acceptable but I am questioning if the fact that VCCIO_EN and PWR_EN dropping at nearly the same time is problematic based upon your email guidance?  Is this acceptable?

Thank you in advance for any help you can provide in answering my questions and confirming correct operation. 

Best,

Craig

Hi Chris:

I wanted to provide the original timing of the power-on and power-off sequencing.  This is the behavior that we have in our current units and what I am surmising has caused the two or three PicoZed SOM failures.  Perhaps there is another failure mode in play but I wanted you to see the difference when I controlled VCCIO_EN form 24V power good signal instead of PG_MODULE.

I am looking forward to your reply,

Craig

Yellow = +5V @ TP31VIN of PicoZed

Blue = PWR_ENB at TP68 (wire soldered) [JX1-5]

Magenta = VCCIO_EN @ TP50

Green = PZ_PG_MODULE @ top of SW2.  [JX2-11] This is the power_good signal from the PicoZed

Hey Craig,

Sorry for the delay. For your previous post, yes the new timing looks correct. The big risk with the efuse is if the board is powered down before it it put back in power reset. PG_MODULE is tied to that reset so if you pull that low then power down everything you should always be fine. Parasitic paths are what you have to worry about if parts of the system stay powered. As an example I had a situation where LEDS on the ethernet jack were being controlled (sinking current) directly through I/O pins which resulted in failures.

In your post from today the startup looks fine, on shutdown however PG_MODULE should be dropping almost identically with VCCIO_EN. PG_MODULE is tied to the power good signals on the bank regulators so the second they are disabled that PG should be going low. It looks like you have a 5-10ms delay before that PG is actually low. Power enable and VCCIO_EN dropping at the same time is ok. I think your failure is most likely due to the POR signal (PG_MODULE) going to the Zynq still being high when power goes away. Following your previous post shut down capture should eliminate that as a failure point. Let me know if I missed anything.
Thanks
Chris

Hi Chris:

Thanks for your response.

Just to clarify,  you indicated that the "OLD" timing, shown in my post made earlier today, could have resulted in the failure mode we have seen.  Is that true?    Can you confirm if you have seen the failure mode we are seeing with the U11, 1.0V regulator, outputting less than 0.5V?  Is this due to Zynq core damage (PS eFUSE or other?)  that causes excessive current to be drawn? Or, is this just a regulator failure? I will be replacing this regulator soon with a new part and will be able to infer if the damage is to the regulator Zynq but I am curious as to what you past experience has shown.

You also confirmed that the new timing shown in the post I made on 11/6 @5:04 PM looks correct so I will mover forward with implementing the small modification.

Thank you again for your guidance and support,

Craig

Hey Craig,

I honestly do not recall seeing any U11 failures. You could try lifting the output inductor and feed 1V in from a bench supply and see if it actually comes up or current limits. It could be either device, but I'd be more inclined to believe there was damage to the Zynq and the regulator is responding to that short / partial short. If 1V comes up properly from an external source then perhaps U11 is actually damaged and replacing it might do the trick.

Sorry for the delay, please let me know if you need any more support here.
Thanks

Chris

Hey Craig,

I honestly do not recall seeing any U11 failures. You could try lifting the output inductor and feed 1V in from a bench supply and see if it actually comes up or current limits. It could be either device, but I'd be more inclined to believe there was damage to the Zynq and the regulator is responding to that short / partial short. If 1V comes up properly from an external source then perhaps U11 is actually damaged and replacing it might do the trick.

Sorry for the delay, please let me know if you need any more support here.
Thanks

Chris

Hi Chris:

Yes, we essentially did what you described in two ways - First we replaced U11 with a new regulator and saw a low output voltage < 0.5V.  We also lifted L9 ad tried injecting 1.0V.  We saw excessive current so we have concluded that we did damage the Zynq.  It is still not clear if the damage is eFUSE related but from my understanding given this is the core rail there may be no way to know for sure?

I finally marked the solution above as the correct answer.  It does appear that after we made the sequencing modifications we have not seen any failures.  Let's hope this remains true and we see no future failures.  Thank you for all your support.

Best,

Craig

Hey Craig,

It may be efuse related, but as you mentioned without sending back to Xilinx for a failure analysis there is no way to know for sure. Let me apologize again for the delays between responses. I've been battling notifications getting blocked to my inbox and haven't done a good enough job of manually checking the forums. Please don't hesitate to reach out if you see any further issues.

Thanks

Chris