Why I like a well-designed bootloader

Or, also known as: How I pretty much bricked, and then fixed a product I was roadtesting.

The following story is something I encountered during my Roadtest of the MKR WAN 1300. (See roadtests.) I've mentioned the programming error I encountered in passing and made mention of it in my roadtest, but I felt that

1. I didn't do it enough justice. The comment in the roadtest section was a short mention, and the roadtest review didn't lend itself for a truly appreciable bout of self-deprecation.
2. I really wanted to write a new blog post. I had this thing turning around in my mind for a while now. Might as well put it on 'paper'. (Well, the background is white after all..)

The story that follows might be slightly embellished and dialogues are paraphrased, names changed to protect,...  -well, nobody probably, and I might obfuscate some of my more colorful choices of language.

On to the story....

It was a weekday evening. My wife usually turns in early, since she gets up at insane hours. (Often is out of the door by the time I make my first coffee in the morning) I'm at my computer messing around with some LoRa experiments. I like listening to music while hacking, it blocks out other noise, like even the clicking of my keyboard itself. But this evening the music, while playing, is on a background because I'm also connected to a discord channel with 2 friends of mine. The discord channel is usually occupied by those of the friendgroup that do either a multiplayer PvP or co-op and often is a nice source of white-noise of banter. This evening, my friends are playing an RTS while I'm socializing during my coding.

I'm working on a way to reliably communicate between two MKR's (peer-to-peer LoRa) and listening to friends bickering about how on earth that AI is whooping their Co-op-ing ascii's. (Turns out that, configuring the game, the AI's cheatmode was accidentally activated. But I digress).

Have you ever done anything on an Arduino, that involved 2 ways of serial communication? I'm a C-programmer by profession and typically, when receiving data from a serial device, you setup an ISR (Interrupt Service Routine), receive bytes and put them in a 'ring-buffer'. 2 index-pointers point to the head and tail of the buffer. Then, when you're interested in what you received, you start reading the buffer from the tail, until you've got what you wanted, or arrive at the head. (Writing 'wraps around' at the end of the buffer, continues from the beginning and if in the process you arrive at the head again, .... Well, you know you need a bigger buffer.. ;-)   )

Ok, in an Arduino environment? Don't do that. The main reason is: The Arduino does it for you.

But if you thought that 'Serial.read()' returns a received character, or nothing if there was no character and you could build your own buffer, turns out that while technically this is probably true, (in the case of MKR boards) your program isn't the only thing using that serial connection.

The Arduino basic platforms like the nano and Uno have seperate FTDI chips that do quite a bit of control besides writing, specifically resetting the device. An uploader like Arduino IDE triggers DTR (connected to RESET of the controller) and then starts transmitting the code.

However: The MKR uses an ARM Cortex M0, which does its own USB communication. Long story short: Effectively it still reports as a serial device, but it's done pretty much entirely in software.

I'm still hazy on what technically happened entirely but:

• I coded the snafu containing the clobbering of Serial.read().
• Realized that something didn't work and tried to debug it with some code.
• Tried re-flashing this code but the IDE couldn't get the device in bootloader mode....

No matter what I did, the device would pretty much boot up and immediately the software would start trying to read the serial from the I/O. My friends were still cursing the AI's unfair competition and trying to help eachother.

Me: "Well, ***."

Short moment of silence. Then:

"Everything OK?"

Me: "I may have fscked up. I can't reprogram the device anymore, since the program currently on it is supposedly hogging the port also used for writing."

"No idea what you're talking about mate. But sounds ba- oh bugger! Planes!"

Maybe it's the serial port.... Remove and re-attach the device: No solution. Attach second device and switch back to first device: No solution.

Me "I'm so boned.."

"So are we!"

Me: "Thanks for the sympathy vote guys. I'm'a gonna have to try something at another computer. back in 20 minutes."

Very quickly, my hopes were dashed that it was indeed not my COM-ports messed up because my laptop wasn't able to program the units either.

Planes were low on my list of problems, unless they'd be the passing over my house. Even then: my more immediate worries were back on my workbench: How to get around this.

• The bootloader is typically the first thing firing up on the board.
• Pretty soon after that (not sure exactly how soon, but it's pretty fast) the application starts.
• Tried timing 'compile -> write' with a well-placed reset of the board, but to no avail.
• The program blocks the bootloader from re-flashing the device, but to fix that I had to remove / overwrite the program.

I considered lobotomizing the device over ISP and re-write bootloader on it from scratch. And while I had the peripheral for that in theory, it was cumbersome. I decided to google around for the symptoms and see if there was experience with this and how to fix it.

Turns out there's lots of experience, and after half an hour of pulling out my hair and head-desking against the kitchen table, I chanced upon a forum reply about the fact that the MKR boards have a type of bootloader that, if you press the reset button twice in rapid succession, the device stays in bootloader until you power-cycle the device.... The built-in led would fade in and out to indicate this state.

No way... Couldn't be that simple, could it?? It was getting late, even for me at this point, but I was damned if I wasn't going to solve this. Picking a mini-screwdriver, I tap the reset button twice in succession. Nothing...

Try again, this time with steadier hand. *click-click*. Led switches on, then fades out, then fades in.

Holy ***! we have a heartbeat! Immediately I push to program the blinky I cue'd up for this opportunity! IDE goes through the motions of compiling and writing. I goes through! The led switches off, stays off for a moment and I can practically hear my heartbeat in my ears!

The led starts blinking! I release the breath of air I didn't realize I was holding and throw my arms in the air to fully expend the complete blast of endorphin that floods out my brains!

I go back to my desktop and pick up my headset. The game's over, but they're still bantering.

Me: "Well, I saved the day. How did the game go?"

"Lets say our day went rather more south-wards than yours."

Me: "Heh. Well, I fixed what I've broken. I'm exhausted. I'm turning in."

"Ok, bye."

It's times like these I wonder why I torture myself with these kinds of problems. My hair, though full, turned gray early (though that might well be genetic) and as I prepare for bed and the endorphins wash out of my systems I feel... Older. Then I realized, that the alternative, is to not get into these situations, learn nothing new in the process and..... Do what? Some menial job with no future or way up or methods to advance myself? Live life in ignorance? Don't get me wrong: I know there are people in this situation. I know there are people that chóse that situation. And I don't look down on them.

Frankly, to them the lifestyle and path Í chose will be as much a hell as the other way around.

I fall asleep realizing that this step in the Roadtest wasn't ambitious enough, and dream about further steps to expand on it...