Watchdog Timer for the Raspberry Pi

Like all PCs the Raspberry Pi is likley to hang up during operation.

Hi Seggy,

Just to flesh out things a bit, perhaps an application could be likely to hang, generally the Linux kernel is not likely to hang unless there is a very severe issue like a build issue, hardware fault or power related issue.

It is worth mentioning that if the board ever does lock up, this should be investigated; because for the hardware watchdog to have to ever take action on a Linux platform is "severe".

Possible reasons could be a poor power supply perhaps, or bad cooling strategy, and for these the hardware watchdog action is like a sticking plaster and does not fix the underlying issue(s).

The underlying issues should therefore be resolved if the watchdog timer actually has to ever take action.

Finally, if users are experiencing application hangs then these do not require a hardware watchdog timer.

The need for this watchdog circuit came out of a recent application the bare bones of which are rpesented below.

Remote RF linked temperature monitors (upto 8 possible)

RF receiver with serial link to RPi

Ethernet connection to upload data to cloud and e-mail alerts

Remote set-up with autostart (no keypad or monitor)

Even after plenty of testing the possibility of software (written in Python) crash is there. eg data collision resulting in corrupted data, e-mail failure due to server issues ........

I know the theoretical response is going to be "Test the system and iron out all bugs". However in practice this is a "safety net" while we do this and allows us to trial the system.

Hi Seggy,

The possibility of Python code crashing is a good example of a situation that does not require a hardware watchdog. You can have the application be killed and restarted. In other words, a process monitor i.e. a software watchdog.

A hardware restart due to a hardware watchdog kicking off because of a Python app not responding could make the situation worse (for example file system corruption).

Sounds good. Any guidance on how to do this? All my experience is with hardware - hence the use of Python and gut instincts leading me to a watchdog.

Actually the best answer would be to not do this, until the Python code is fully baked and hardened. Otherwise you're at risk of code restarting under your feet, making life very difficult in debugging.

A restart such as this needs to be an extremely rare thing, in my opinion to the point that you have sufficient time to investigate each single restart even if you have hundreds of customers using your software non-stop.

Your process monitor can be as simple or as sophisticated as required, there might be examples via google, ranging from scripts to complete applications (I've not checked). The only time I've written one was for a commercial app so I can't share it, but it was in C++ (the language doesn't matter). It is at heart a state machine that needs the ability to detect actual application status, desired status (running or stopped, etc - since sometimes you may wish the application not to run), be able to kill and restart the application (e.g. using a kill or pkill command).

Shabaz

In theory the python part should be the only part stopped.

Is it possible to write to a log to show that a forced restart was triggered, and then execute a graceful shutdown via the serial port.

This avoids the corruption roblem, which could definately cause more issues.

While I agree that software should really handle all the unexpected, and run off to a error handler to stop it locking up, as the Op has pointed out it is during testing that this 'supervisory' device and software could be employed.

The log should allow it to be run and then checked for restarts.

I'm thinking that this would be a useful add-on for the Pi.

Mark

Hi Mark,

Unfortunately I've more experience on other platforms, not Pi, and on those platforms I've never managed to make the Linux kernel die through user space apps (which is what Seggy would be doing, provided he's not writing his own drivers), and I wouldn't say I'm a super-careful programmer, just a "normal" programmer. A good system will generate logs as you say, capture any debug needed and then send alarms, and then do a graceful shutdown, and all this can be done as a process running on Linux (i.e. doesn't need external hardware, regardless of how hung/frozen the user app is). The apps I worked on were mainly sold to run on Solaris, which is famed for rock-solidness, but we had customers who ran it on Linux too and today I've no doubt about the reliability of Linux (which can be further enhanced through system-level redundancy, or load-sharing, etc).

An add-on watchdog isn't useless, it would be useful if the kernel is hanging (which could happen if there were brown-outs or hardware failure issues) but this is different to dealing with buggy Python application code. The watchdog won't perform the steps mentioned above, so it is like a extremely last resort, and if there were such issues I'd want to get to the bottom of each one again, i.e. resolve power issues, replace memory, etc. A hardware watchdog that kicks in for a user application failure (while the kernel is still running) could cause more harm than good, because (a) if the kernel is still running then a software process is more useful because it can capture logs as mentioned and (b) there is a risk of data corruption if other processes are busy writing to storage.

But a software solution too shouldn't kick in regularly either, if it is, then the application code needs a lot more effort..

In practice, even if it is not life-threatening and there is no agreed up-time, it still at most should only occur infrequently enough that there is plenty of time to debug each restart (across all your customers) caused in this manner, i.e. to minimise customer loss and bad sentiment, and still have time to develop the next great feature/project.

Hi Mark,

Unfortunately I've more experience on other platforms, not Pi, and on those platforms I've never managed to make the Linux kernel die through user space apps (which is what Seggy would be doing, provided he's not writing his own drivers), and I wouldn't say I'm a super-careful programmer, just a "normal" programmer. A good system will generate logs as you say, capture any debug needed and then send alarms, and then do a graceful shutdown, and all this can be done as a process running on Linux (i.e. doesn't need external hardware, regardless of how hung/frozen the user app is). The apps I worked on were mainly sold to run on Solaris, which is famed for rock-solidness, but we had customers who ran it on Linux too and today I've no doubt about the reliability of Linux (which can be further enhanced through system-level redundancy, or load-sharing, etc).

An add-on watchdog isn't useless, it would be useful if the kernel is hanging (which could happen if there were brown-outs or hardware failure issues) but this is different to dealing with buggy Python application code. The watchdog won't perform the steps mentioned above, so it is like a extremely last resort, and if there were such issues I'd want to get to the bottom of each one again, i.e. resolve power issues, replace memory, etc. A hardware watchdog that kicks in for a user application failure (while the kernel is still running) could cause more harm than good, because (a) if the kernel is still running then a software process is more useful because it can capture logs as mentioned and (b) there is a risk of data corruption if other processes are busy writing to storage.

But a software solution too shouldn't kick in regularly either, if it is, then the application code needs a lot more effort..

In practice, even if it is not life-threatening and there is no agreed up-time, it still at most should only occur infrequently enough that there is plenty of time to debug each restart (across all your customers) caused in this manner, i.e. to minimise customer loss and bad sentiment, and still have time to develop the next great feature/project.