I managed to bring down a small datacenter once when setting up a new machine. I simply assigned the next available IP on the subnet the server was on, trusting the docs about what IPs are used and for what. Turns out, the IP I assigned conflicted with another machine's that was doing very important ingress failover and coordination related things, and brought down the entire uplink to the internet. Whoops.

Technically it wasn't just my fault, but a long chain of small mistakes that leads here, as it always happens in these sort of cases. The docs were obviously not updated correctly, the ingress code made some assumptions that seem correct in normal operation but aren't necessarily true in some wacky scenarios, and I didn't make sure that the IP I assigned was really unused.

Figuring out what was happening was a fun experience too, when your pings go through but things still can't communicate for some reason, and all sorts of weird things are happening on the network.

The worst part is that you can't really highlight a single mistake that caused it. Yes, the docs should've been updated, but it happens that someone forgets. It shouldn't be a problem, and it usually isn't. Sometimes you might enter the wrong IP, things go down, but there are failovers and redundancies for this case, that kick in. And there are also bugs in code that you have to live with, and we do. But sometimes the stars align in the exact way to line up all of these issues in a way that something catastrophic happens, so I'm not really sure what the ultimate conclusion is. At least we fixed a bug, I guess.

My boss would say,  mistakes dont happen if you're not working 

I had to select a power connector for a product, which had to be different from an existing one to make sure nobody could ever put in a wrong one. 

So I used a 3.5mm headphone plug (the mono single channel version), which had tip and sleeve only. Functionality was no a problem, it worked flawlessly. The problem was that unit was mounted with the connector hanging off the bottom of the housing most of the time. The cable was too heavy for the retaining force and the plug slipped out all the time. The product had passed CE/FCC certification already, so there was no cheap way to change the style of connector. Duct-tape and hot-glue for the fix (there was no win anymore) during installation. Nobody was happy (customer, service techs, management and myself). 

Lesson learned: It is never too early to think about usability and user experience (which are 2 separate things) of every interface of a product during development. Best, make it part of the requirements list, to be sure that those considerations make it into the test plan. 

My worst foul up in recent history was on the Shark Chase Tank robot building for the E14 Fun & Games competition. 

The chase tank was to autonomously hunt down the radio controlled Rubber Duck with Hammer.  The chase system used a PixyCam to object detect the rubber duck ( by color blob)  and it's placement in the field of view for steering sent to the controller via SPI. 

The original Pixycam 1.3 was used, I decided not to use an Arduino Uno  for the platform, rather use an Arduino Minima R4 to include some other features. The Arduino PixyCam SPI code was not compatible with Minima R4, so I reverted back to an Arduino Nano to save space and upgrade to screw terminals. 

Altho I compared pinouts, specs, and pin checked the hookup a dozen times, when it came to final assembly , I plugged in the SPI ribbon backwards. 

I said a sad sayonara to the PixyCam as that finally did it in. 

What did I learn ? .....time to upgrade the whole platform from blob detection to closed loop object detection tracking and steering !  

"Every bursted bubble has a glory!
Each abysmal failure makes a point!
Every glowing path that goes astray,
Shows you how to find a better way.
So every time you stumble never grumble.
Next time you'll bumble even less!
For up from the ashes, up from the ashes, grow the roses of success!
Grow the roses! Grow the roses! Grow the roses of success!
Oh yes!
Grow the roses! Those rosy roses!
From the ashes of disaster grow the roses of success! "

https://www.youtube.com/watch?v=jJxxvq3DNCU&list=RDjJxxvq3DNCU&start_radio=1

Im Dyslexic as ***, and while its useful sometimes, you can also wire an entire waste processing facility upside down and backwards. oof I dont like thinking about it at all. put in all the unpaid hours to fix it. 120,000 connections deep, realize oh yeah, that thing I do, Ive been doing it the whole time. ***. 

One of the biggest mistakes I made was during my college FYP, when I was helping a friend measure voltage and current with a multimeter. We first measured voltage, and when it was time to measure current, I forgot to change the probe to the correct multimeter terminal. The result was an instant kaboom with a white flash and smoke. Luckily, no one was hurt.

That moment really stayed with me. It taught me that in electronics, even a simple mistake can become a serious safety issue. Since then, I always double-check the multimeter mode, probe placement, and circuit setup before taking any measurement. It was a hard lesson, but it made me much more careful, safety-conscious, and disciplined in every project I work on.

Here is my story from a project in Sweden that I will never forget.

It was 11 PM in Stockholm. I was finally in bed after two hard weeks installing the ticket system for the Arlanda Express train. Everything was working perfectly.

Then, my phone rang.

My boss in Spain was very nervous. He had forgotten to change the "subnet mask" of the X.25 gateway for credit card payments. He was afraid of a big financial penalty. He asked us to go to the airport immediately, change the mask, and not tell anyone.

We were exhausted and did not think clearly. We forgot to ask the most important question: “If the mask is wrong, why are payments already working?”

Because of the pressure, we drove through the snow to the station, changed the configuration, and left. We did not perform any tests.

At 5 AM, as I got home, the phone rang again. It was the railway's Head of Technology. The station was open, the morning rush had started, and no credit card payments worked.

I explained that we had changed the subnet mask. He didn’t get angry; instead, he was surprised. He told us that he had also seen the problem earlier and had secretly changed his own system to match our "wrong" configuration so the launch would work.

Both of us made secret changes to "fix" the problem, and that broke everything.

What I Learned:

• Transparency is key: If we had communicated, we would have known the other side was already updated.
• Don’t work when exhausted: When you are tired, you miss the obvious.
• Always test: There is no such thing as a "small" change in a complex system.
• Question the "fix": If the system is working, ask why before you change anything.

robogary said:

My boss would say,  mistakes dont happen if you're not working 

"Wie niet werkt maakt ook geen fouten (Dutch)" the first thing I learned from one of my co-workers when starting as an engineer right after college.

This happened to me as well, but I first measured current, and then voltage, the other way around does not give much kaboom normally.

This reminds me of the story from old times: when I was a child, there was a communism where I live - which meant permanent lack of everything. I had a multimeter, but there was problem with fuses for current measurement protection - the ones I have managed to acquire were quickly blown (it was about 0.5A maximum I think). And as the fuse was in the ground patch, blown fuse also blocked voltage measurements.... I have learned that when multimeter was set for voltage measurements, signal could be connected to the ohmmeter contacts to avoid a patch with blown fuse - until the multimeter was set for something different and one element of resistor ladder has blown instead of the fuse...

Lessons learned - protection measurements have their place and sidestepping them should be avoided if possible