C++ library for ST Teseo GPS - pt. 2: Dynamic GPS configuration (and some other things)

RE: C++ library for ST Teseo GPS - pt. 2: Dynamic GPS configuration (and some other things)

shabaz — Mon, 15 Jul 2024 14:49:32 GMT

I once worked on a system that was real-time but running on a large box that was essentially internally a server with storage, etc, rather than a microcontroller, and customers could switch on a logging mode (in the same production release, i.e. not a separate build) but were advised never to do that in their production systems (many of those customers kept a separate testbed to try things out). I can't recall now, but I think for a long time we only released builds built with debug, i.e. we took the performance hit, but eventually changed the build flags once we had more confidence.

Some ran the systems (as advised with logging switched off) like racing cars and to save costs wouldn't have purchased many of the systems to run in parallel, and if there was ever a crash, if they couldn't replicate on their testbed, then it was of course a nightmare to troubleshoot. And, of course, since it was such a severe thing, almost all of our other work would stop apart from focussing on the crash problem, trying to replicate it ourselves, and getting it solved.

One trick we ended up doing for a particular build for one customer was to give them a 'special' release that logged a minimal amount (events and the first dozen or so bytes of any messages) but instead of to disk which was slow, we decided to store into a circular RAM buffer of the last 1000 or 10000 events or whatever. That way, whenever there was a crash, there was a good chance that the RAM was not significantly corrupted, and we could read the contents and figure out what might have happened. That was not too difficult because the underlying OS would create the crash dump containing all the RAM contents, so we'd simply get the customer to ship across the (usually huge, many Gbytes) file (if it didn't contain anything they considered confidential), otherwise we would send them software to pull out what we needed. The RAM was easy to search since the buffer started with a magic sequence.

Over time we managed to weed out all the race conditions and improve reliability loads, but it was a stressful time since it's the worst customer experience when things crash. We improved guidelines (we created an online or Excel file calculator for the customer and salespeople to easily figure out how many of the systems each customer would require so as to not be running with massive amounts of events per sec), and improved performance testing (developing ways to hammer the system to simulate peak times, artificially loading up the CPU, etc), and then as it got more mature, it was a nice result that when the customer trends changed, it was possible to quickly port the code, and even virtualize it, without too many hiccups.

RE: C++ library for ST Teseo GPS - pt. 2: Dynamic GPS configuration (and some other things)

michaelkellett — Mon, 15 Jul 2024 09:13:40 GMT

Here's a thought, mainly applicable to high reliability hard real time systems,

I like the ideas, which were very prevalent in an aerospace project on which I worked:

1) if it executes ever it executes always

2) what you test is what you ship

To amplify a little:

rule 1 implies that if a stream of debug information is sent to serial port X the it should ideally be emitted at a constant rate. This is to avoid bugs of the kind where the extra processing load of debug printfs slows things down enough to avoid a fatal race condition elsewhere.

rule 2 implies that release code and test code are the same code

Compliance with these rules can be very hard to achieve - on some code I'm working on now a particular function runs in 20us with maximum optimisation, but debugging only works correctly at a much lesser optimisation setting, when the function takes 47us to run. So I must test the code with more less no debugging operating. My answer to that is to build instrumentation into the code (which is always there and will always execute according to rule 1).

I don't use asserts but if I did I think I would need to add an assert handler so I could leave them in the code.

One of the questions I have for the next code review is "what shall the code do when the scheduler detects that a task returned without error but took too long to do so". (Too long in this case is defined as greater than the maximum time set in the task definition - doesn't necessarily mean that anything bad happens if enough of the other tasks did better !).

RE: C++ library for ST Teseo GPS - pt. 2: Dynamic GPS configuration (and some other things)

Jan Cumps — Sun, 14 Jul 2024 09:10:44 GMT

I've added DoxyGen comments. A typical html doc page looks like this:

RE: C++ library for ST Teseo GPS - pt. 2: Dynamic GPS configuration (and some other things)

genebren — Sat, 13 Jul 2024 16:48:34 GMT

I do agree with your 'hurt' classifications and your use of asserts. I find with a lot of the embedded coding that I do, with a high degree of time sensitivity, that I don't often use a debugger during development and testing, so I rely on printf statements to provide me with some insight as to how the program is functioning or 'not' functioning.