I recently designed a sensitive circuit similar to those used in the front end of high input impedance meters such as an electrometer. The design goal was to determine relative changes in capacitance on the order of 50fF. To keep the 'variable capacitor' from getting swamped out, the circuit took advantage of sensitive measurement techniques such as using a unity gain buffer to create a guard signal for shielding.
After a bit of troubleshooting and solving layout issues, the circuit had one last problem – 60Hz noise. But how was it coupling into my signal? Immediately I thought of the power supply feeding the sensitive circuitry and dove in head first on the noise hunt. At least until my co-worker walked out and accidentally turned off the lights. Right before I blurted out a lighthearted but snarky comment along the lines of “Just because my circuit doesn't work yet doesn't mean I don't exist!” I saw that the noise disappeared! No lights = no noise. It seemed the glow of the skin-bleaching overhead fluorescents was the noise source, not the board's power supply.
Probing around inadvertently gave me another accidental hint – the noise changes depending on where my hand is positioned over the circuit. At first I disregarded the effects of my hand as an antenna that only tuned the noise, similar to the iPhone 4 signal problems that occurred when a user's hand was too close to the antenna. Then I noticed it wasn't like normal noise tuning where the position of a hand might only tweak the noise on the output -- I was able to eradicate the noise if I completely covered the circuit. And cardboard could yield the same effect as my hand. So it wasn't even electrical – it was visual!
Visual noise pointed me in the direction of the phototransistors on the board. Sure enough, when they were covered the noise went away. I knew it couldn't be the PT's circuit since that part of the design was left untouched in this revision. But a glance at the new wiring harness showed the problem – a missing connection to the PT's emitters. The floating emitters were able to wreak havoc on my sensitive signal every time the base of the transistor got hit with light. A quick skywire to make the connection, and I was able to have a circuit that worked in the mood-crushing ambiance of overhead lights!
At the end of this problem, I was struck by two funny things. First, as engineers we are always looking for problems in difficult but interesting places such as the sensitive input node. Second, if I wasn't paying attention to the little things I would have missed big time saving hints, for example the scope when the lights went off and the nuances of my hand waiving.
My lesson that day? Mouth closed, mind and eyes open. Can't re-learn that one enough. What about you? What strange hints have you found by paying attention? Tell us about them in the comments!