How can I make an embedded system robust?

It sounds like you are not doing the hardware yourself, but a vehicle environment can be very hard on hardware, so it needs to be considered when reliability is important. There are a lot of things to consider when making hardware robust for vehicle applications, including power supply, shock, vibration, temperature, humidity, operator mistreatment, etc. Let us know if advice is needed for that.

I'm not really a software guy, but here are some pointers anyway...

On the software side, try to build in self diagnostics for each subsystem and a graceful way to handle errors.

Add at least a temperature sensor and create a periodic non-volatile time-stamped log of temperature, system status, errors and system activity (and firmware rev number) - so when it fails you know when it failed, what the conditions were and what part of the program was running.

Ensure there are no possible conditions that could result in an endless loop - for example do not wait forever for a (possibly broken) sensor to respond.

Do not depend on data such as GPS data being properly formatted - garbled data needs to be taken in stride.

One aspect of fleet monitoring systems that should not be overlooked is acceptance by the operators. They need to pretty enthusiastic about what is being recorded and reported, otherwise equipment will just "mysteriously" cease to function. Protecting against reluctant operators is a whole other level of robustness.

Well for a start, an Arduino is probably not really suitable. There are microcontrollers and peripherals that are appropriate for automotive and industrial environments - such as TI's Hercules series.

That's some good advice, and yes, I'm not doing the hardware part, hopefully its done the right way and there are no issues with power or signal integrity. I like your suggestion about logging temperature, as high temperature could cause instability. I thought about logging, but I'm afraid the internal non-volatile memory could wear out very fast if I do so, as an alternative an SD card could be used, at least for the prototypes. On relying on the correct working of each subsystem is a very good idea also. For the endless loop condition I'll use a watchdog to make sure it doesn't get into such state. When you talk about GPS data being garbled, would that be because of signal integrity issues, or do you think GPS modules are not mature enough to work properly and generate properly formatted data?

How operators accept the system is completely out of my control, I suspect that if they think they are being controlled they may not like it at all and try to break it, I'll comment that to the part of the team concerned with those issues...

Was reading about the Hercules, I would think It's a bit of an overkill, being a CortexR with lots of safety mechanism, it also wouldn't make the system more robust unless other components are also equally robust, which I would think could increase the price quite a bit. Here, nobody would die if the system fails, and I would think a lower than 5% of failures per month of usage would be in the limit of whats acceptable. The Due is a CortexM3, why do you think such MCU isn't really suitable for the that task?

I'm not sure what all the reasons for GPS data to get messed up are, but I suspect they each manufacturer has their own algorithms to deal with satellite signal drop-out and RF interference. I know some of my GPS modules send out partial data sentences unexpectedly.

I use FRAM when I need to constantly write to a non-volatile memory. You can get an SPI FRAM module from Newark or Adafruit.

but what I'm really wondering is what would be the best practices to make the system as robust as possible so that when its delivered it doesn't fail

You need to include an external watchdog timer.

There are IC's out there that monitor the supply voltage and receive a regular 'pulse' from the processor.

Failure of either will cuase it to reset the micro.

Otherwise you need every aspect of the software to have a proper error routine as dougw mentioned.

You also need to allow for a wide voltage range (7-18v DC) along with a high supply noise in some periods.

Temperature is also going to be a factor with some internal temperatures exceeding 50 degC, or if you're in Australia even more.

Most silicon is only rated to 55 degC when it's running.

IMO three months to design, verify the hardware and then test is a bit short.

A working prototype might be possible to allow testing.

5% faliure/month is quite high. I would hope you were under 1%.

To give you a idea, my skifield trip systems have been operating for 8 years (approx 3 months/yr at 7 hrs/day) and there has been zero failure in either one.

The climate is less than ideal, and they get unplugged during the off season due to lightning.

I wasn't able to use a watchdog, but I did use dual processors and pulsed the output so a hang in either processor would trip the tow.

Version 3 which is the drawing board now, will have the watchdog timer and most likely use an Arduino with an external regulator.

Mark

neuromodulator  wrote:

 

Was reading about the Hercules, I would think It's a bit of an overkill, being a CortexR with lots of safety mechanism, it also wouldn't make the system more robust unless other components are also equally robust, which I would think could increase the price quite a bit. Here, nobody would die if the system fails, and I would think a lower than 5% of failures per month of usage would be in the limit of whats acceptable. The Due is a CortexM3, why do you think such MCU isn't really suitable for the that task?

If a failure causes major disaster and overheating causes instability, you may want to read about functional safety and safety controllers one more time. You will not get what you need with firmware. You need hardware support and a thourough risk analysis.

I was going to use the internal watchdog, (page 260, http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-11057-32-bit-Cortex-M3-Microcontroller-SAM3X-SAM3A_Datasheet.pdf). Which watchdog would you recommend?

Yeah , 5% is not that great, of course I want to reduce it as much as possible, in software I'll do all I can in that short time frame. But in terms of hardware I don't think it will be easy to justify duplicating the production costs to reduce the failure to half.

Thanks sharing your thoughts!

I was going to use the internal watchdog

I'm aware of the internal watchdogs, but I question whether they would operate if the micro gets screwed by either a brownout or something else.

It seems my thoughts aren't alone https://www.digikey.co.nz/en/articles/techzone/2012/may/a-designers-guide-to-watchdog-timers

TI have a great range of them plus others have them as well.

http://www.ti.com/lit/ds/symlink/tps3813.pdf

This is just one and gives you an idea.

Maxim https://www.maximintegrated.com/en/products/power/supervisors-voltage-monitors-sequencers/watchdog-timers.html

ST https://www.st.com/en/reset-and-supervisor-ics/watchdog-timers.html?querycriteria=productId=SC1854

Your hardware guy might have some ideas about what suits his hardware.

But in terms of hardware I don't think it will be easy to justify duplicating the production costs to reduce the failure to half.

I'm sure you mean doubling, but regardless the hardware design should not be a source of that high a failure rate.

I would be expecting a much less than 0.5% failure rate at production, and I'd be dissapointed if it was higher than 0.1%.

You can have a higher rate during prototype testing, but after that it should be closer to zero.

Good design means the product is suitable for the purpose, and in this day and age of consumer power and protection, they might have every right to demand their money back plus costs.

Imagine if the product overheats, and causes either a localised fire, or worse a major fire inside the vehicle that affected other vehicles.

Would your indemnity cover you for the owners costs.

Adding "Use at your own risk" is not sufficient in this day and age.

They could sue you not only for the physical aspects, but the emotional harm as well.

The insurance company is likely to chase you for costs, and that might also include any testing they did to show it was that product that caused it.

I was involved in a product our company built and it went through an independant lab for electical safety testing.

Part of that testing was subjecting it to high and low temperatures to ensure it didn't present a risk, and we were using another manufactured power supply, so it was things exploding they were looking for.

The point here is to show that rather than just make something and throw it out there, you really need to have documented proof that you've tested it for every scenario possible.

This also includes EMF/EMR testing to ensure it doesn't radiate .... just look at the Raspberry Pi debarkle before it was finally tested.

In the short timeframe given, I don't think any of this is possible.

We have a garage door manufacturer here in NZ that used to use asian sourced door controllers. While they accepted that some would fail during testing, it was the higher return rate after install that started costing them money.

In the end it became cheaper to make their own, and at the same time they improved their products reputation.

In advertising, bad products are 'shared' 10 times more than good products, so it makes sense to try not to be in the 'bad review' category.

IMO if I was asked to be involved in somethign like this and under these terms, I'd be walking away no matter how much I wanted to be involved.

The personal risk is just not worth it, and even if your lawyer can write a really good indemnity agreement, your name will still be associated with it.

Mark

I'll second that last paragraph - on this project 3 months might be just enough to get a working prototype together that was good enough for some initial trials. (But it is nothing like enough time to develop software to anything above what I would call cheap commercial standard.)

But there is a whole load of stuff in this project which suggests that it needs to be done to a much higher standard.

MK