Actions
Forum Thread Details
  • Replies 11 replies
  • Subscribers 475 subscribers
  • Views 1452 views
  • Users 0 members are here
Related

How can I make an embedded system robust?

neuromodulator

Hello,

 

I'm working on a fleet control project on a very tight schedule (which I didn't set). I'm the primary (probably the only) developer, and the project should be completed in around 3 months. The system is based on an arduino due, a simcom 5360 (3G + GPS), accelerometers, odb2 interface and bluetooth low energy, all of which would be integrated into a single PCB by the hardware guy. I don't think I will have any problem at using these peripherals, but what I'm really wondering is what would be the best practices to make the system as robust as possible so that when its delivered it doesn't fail. A crash and reset, wouldn't be a disaster, but having the hardware to fail and not do what is supposed to do (process and send telemetry) would be a major disaster. So my question is, what are some good practices/recommendations to make the system as robust as possible so that once its delivered it will keep working for months?

As of now what I've been doing is to code as much as possible in the PC, because its faster to compile and easier to debug. My plan is to create wrappers for some Arduino functions to be able to test as much as possible code on my PC. I'm doing exhaustive unit tests to all functions and considering corrupt serial data (I wouldn't like garbage to cause a hang or crash). I also would like to do some code coverage, but I need to find the right tools to do it, as visual studio community 2017 doesn't support it. I plan to program a server that will simulate different situations, including different network conditions to test if the client performs as it should. I also plan to use an ODB2 simulator to test at home different conditions. A watch dog is going to be used to make sure the loop is properly looping. And that is pretty much my current approach to make the system robust.

One thing I'm not completely sure, is what are the best ways to perform field tests. Ideally I would like to minimise them, as they are expensive and time consuming. What are some good practices to make the most out of them? If something fails in the field I would like to be able to track it to the source of the issue, as opposed to end up wondering what caused it and repeating field test over and over on different conditions.

An alternative solution to trying to make the system bug-free, could be to implement OTA updates, which on the espressif mcus is pretty straightforward, but here I'm not sure how I could do it. Any ideas?

 

Also any suggestions and comments would be gladly welcomed...

 

Thanks

Parents

  • It sounds like you are not doing the hardware yourself, but a vehicle environment can be very hard on hardware, so it needs to be considered when reliability is important. There are a lot of things to consider when making hardware robust for vehicle applications, including power supply, shock, vibration, temperature, humidity, operator mistreatment, etc. Let us know if advice is needed for that.

    I'm not really a software guy, but here are some pointers anyway...

    On the software side, try to build in self diagnostics for each subsystem and a graceful way to handle errors.

    Add at least a temperature sensor and create a periodic non-volatile time-stamped log of temperature, system status, errors and system activity (and firmware rev number) - so when it fails you know when it failed, what the conditions were and what part of the program was running.

    Ensure there are no possible conditions that could result in an endless loop - for example do not wait forever for a (possibly broken) sensor to respond.

    Do not depend on data such as GPS data being properly formatted - garbled data needs to be taken in stride.

    One aspect of fleet monitoring systems that should not be overlooked is acceptance by the operators. They need to pretty enthusiastic about what is being recorded and reported, otherwise equipment will just "mysteriously" cease to function. Protecting against reluctant operators is a whole other level of robustness.

  • in reply to dougw

    That's some good advice, and yes, I'm not doing the hardware part, hopefully its done the right way and there are no issues with power or signal integrity. I like your suggestion about logging temperature, as high temperature could cause instability. I thought about logging, but I'm afraid the internal non-volatile memory could wear out very fast if I do so, as an alternative an SD card could be used, at least for the prototypes. On relying on the correct working of each subsystem is a very good idea also. For the endless loop condition I'll use a watchdog to make sure it doesn't get into such state. When you talk about GPS data being garbled, would that be because of signal integrity issues, or do you think GPS modules are not mature enough to work properly and generate properly formatted data?

    How operators accept the system is completely out of my control, I suspect that if they think they are being controlled they may not like it at all and try to break it, I'll comment that to the part of the team concerned with those issues...

Reply
  • in reply to dougw

    That's some good advice, and yes, I'm not doing the hardware part, hopefully its done the right way and there are no issues with power or signal integrity. I like your suggestion about logging temperature, as high temperature could cause instability. I thought about logging, but I'm afraid the internal non-volatile memory could wear out very fast if I do so, as an alternative an SD card could be used, at least for the prototypes. On relying on the correct working of each subsystem is a very good idea also. For the endless loop condition I'll use a watchdog to make sure it doesn't get into such state. When you talk about GPS data being garbled, would that be because of signal integrity issues, or do you think GPS modules are not mature enough to work properly and generate properly formatted data?

    How operators accept the system is completely out of my control, I suspect that if they think they are being controlled they may not like it at all and try to break it, I'll comment that to the part of the team concerned with those issues...

Children
  • in reply to neuromodulator

    I'm not sure what all the reasons for GPS data to get messed up are, but I suspect they each manufacturer has their own algorithms to deal with satellite signal drop-out and RF interference. I know some of my GPS modules send out partial data sentences unexpectedly.

    I use FRAM when I need to constantly write to a non-volatile memory. You can get an SPI FRAM module from Newark or Adafruit.