I'm currently working on bringing up a board for a piece of hardware I've been working on for a little over a year. You can read more about the history of the project on my project website if you're interested. This post is a work in progress; I hope to update it as I progress to a complete, fully functioning board.
This particular batch of PCBs is the second revision for this project, so hopefully I've managed to correct all the glaring errors from the first revision without making any new ones!
I started by making a detailed plan for the bring-up order, with the codec power supplies first, then the difficult-to-solder QFN codecs, then the rest of the board components. Things went extremely well for the first step of the plan, which may be some sort of record. I had very little trouble mounting the two LDOs and their support passives, and both supplies were easily verified.
Next up, the itsy-bitsy QFN-package codecs. What could possibly go wrong? A lot, it turns out. First, I applied too much heat to the PCB attempting to prep the exposed ground pad in the center, peeling up a signal trace from the analog input section. I started over with a fresh PCB (AllPCB's minimum order is five pieces, and they frequently send more than that; I got seven pieces with this order), and this time I went straight to working on installing a codec. I used lots of flux, but I also attempted to fasten just two pins of the codec before going to work on the exposed pad. Unfortunately, this resulted in the part sliding slightly off-center.
After chatting with the senior engineer at work, I started again a third time, with a new strategy. Per his suggestion, I applied solder to the exposed pad on the part then removed it with solder wick to prepare the pad. I then soldered all the outer pins using my 70W Weller iron, a magnifier, and a lot of patience. I inspected the joints using a USB microscope, and re-worked the pins that needed attention. Afterwards, I tried applying solder to the back side of the PCB to fasten the exposed pad, but my first attempt appears to have been unsuccessful; I installed the bias and reference capacitors and applied power, but the part doesn't power up. It's sinking about 25uA of current from the +3.3V rail, but nothing from the +1.8V rail. My plan for the next step is to take the board in to work and try to attach the exposed pad with the Metcal iron I have there.
Update #1 (November 9, 2018)
I spent some time with the senior engineer today working on the third board; he showed me some techniques to use to hand-solder QFNs effectively, using the codec already on the third board as a teaching example. The codec still doesn't power up properly, but it's now sinking a lot more current (starts around 500uV, then slowly sinks down to about 270uV as though capacitors are charging) from the 3.3V rail. I still see nothing from the 1.8V rail, and the dev kit I have that goes with the codecs can't connect to the codec.
Next, I'm going to try attaching a codec to a fresh board using the new techniques I learned today. The senior engineer suggested removing any pads for pins that are floating (there are thirteen in this configuration, due to unused features of the codec) with an Xacto knife to make sure they don't peel up and create shorts due to excess heat; I may also try this bit of surgery beforehand to improve my chances of success.
Update #2 (November 10, 2018)
I tried out the method that was shown to me by the senior engineer at work yesterday, and it's worked fairly well. The basic idea is to get a big ball of hot solder built up on the end of the iron, then wipe the ball back and forth across the pins on one side of the QFN without dwelling too long in any one spot. Any solder bridges are removed with rosin wick afterwards. I had to use my finest soldering iron tip afterwards to touch up a couple pins, but overall I feel like the technique was effective.
Unfortunately, the codec dev board went belly-up for reasons that remain unclear (the device enumerates as USB, but the EVKit software does not detect it or connect to it), so I need to get another one before I can easily write and read back the codec registers to confirm full functionality. I have another small dev board based around a Microchip PIC32MX MCU that I'm planning to write simple firmware for to configure and turn on the part in an attempt to verify basic operation.
Update #3 (November 16, 2018)
The new codec dev board showed up today, and it revealed some very good news: The codec I installed in the last update powers up and connects to the codec dev board via I2C! I also got through the basic heartbeat example on my PIC32MX dev board this week, which should help speed things along when I get to installing the PIC32MX on the sound card. It's a bit hard to see, but in the picture below, the message in the EVKit software right above the right side of the Windows search bar reads: "MAX98090A Connected (0x20)"
Next, I'll need to install and verify the other three codecs, then transfer over the LDOs from the first PCB I tried to bring up. The next three codecs will likely be quicker than the first, but I still expect each one to take at least a couple hours. Unfortunately, my workspace is in my garage, which is a bit drafty (as many garages are.) Normally not a big issue, but at the present time the entire region is blanketed by thick smoke from the Camp Fire north of here and the air quality is currently one step below "Hazardous," so I don't know how much time I'll realistically be able to spend in the workshop until the air quality improves.
Update #4 (November 27, 2018)
I managed to get the three remaining codecs mounted and verified, though there were some difficulties along the way. The second codec went okay, but I decided to try a smaller soldering iron tip on the third codec. The smaller tip worked great for the wiping technique, but it knocked a number of the no-connection pads loose, and it took a lot of work to get them all removed. I preemptively removed all the no-connection pads on the fourth codec footprint prior to installation by heating them with the tip of the iron and lifting them off with an x-acto knife. While verifying the soldering connections, I discovered that a 3.3 V power connection to the codec had been orphaned by a ground plane change I made while moving an LDO at the end of the layout process. Luckily, there was a small 3.3V pour adjacent to the codec, so I was able to fix the problem with a small bit of wire. Hopefully this is the only new PCB layout error! After all four codecs were verified, I transferred the LDOs from the first PCB and verified their operation.
I've already started installing the support components for the PIC32MX MCU; once I have all the MCU section parts installed, I'll be starting the firmware for the project by implementing a simple "heartbeat" state machine health monitor.
Update #5 (December 2, 2018)
I've successfully installed and verified the MCU and implemented the heartbeat example. The MCU clock configuration seems to be correct, as the measured LED period closely matches the programmed value.
Next, I'll be installing the MCU support ICs: an I2C multiplexer to route traffic to the four audio codecs, and an I/O expander to drive the sixteen channel LEDs (four per stereo channel.) After I've installed and verified the parts, I'll be implementing I2C communication between the MCU and the support ICs before moving on to the next step.
Update #6 (December 16, 2018)
I've successfully mounted and verified the support ICs, bringing me to the first major milestone in the project: all digital parts are now installed and their functions have been validated. The remaining work is all analog sections and writing firmware. This last phase went fairly smoothly, with one exception: I had a very difficult time bringing up the I2C driver on the PIC32 MCU. The I2C output refused to work correctly when programmed directly to the MCU, but the driver and the functions I wrote all worked as intended while in hardware debug mode. I tried multiple slight variations of the code with portions commented out to isolate the problem to the I2C write operation. After some seemingly fruitless searching, I compared my current firmware configuration headers to those I generated for the previous hardware revision and discovered that my assertion about clock configuration in the previous update was wrong. I had inadvertently set a clock multiplier incorrectly, resulting in a final operating clock of 40 MHz instead of 48 MHz. This wasn't a problem for the timer driver used in the heartbeat state machine health monitor, but it was a problem for the I2C driver during write operation.
With this new information, some of the information I had found earlier while searching suddenly made sense: The problematic bus behavior I had observed was a result of the clock configuration issue. Because the I2C driver apparently expects the faster master clock, it was looking for the ACK/NACK bits at the wrong time (before the write was finished), resulting in failed write operations. I learned two important things from this: First, that the Microchip I2C driver relies on the master clock being set correctly; and second, that the hardware debugger configures the master clock for the full rated 48 MHz master clock operation of the PIC32 part in debug mode.
The next step will be to install the switching regulator and LDOs that supply the split rails for the analog sections. There are four different enable pins, so I'll need to implement a master system timer to make sure that the sequencing is correct. I'll use the same timer later on with the momentary switches for debounce and multifunction purposes.
Update #7 (January 21, 2019)
After some holiday-related delay, I started working on the power supply section for the analog signal ICs. I was able to get my original design working properly with very low output ripple (less than 3 mV peak-to-peak), but when I did a load test, I discovered that I had neglected to include the ideal transformer term in my power budget calculations, so the first design was WAY over the power budget. I've re-designed the feedback and compensation networks for the problematic power supply section to lower the rails from +/-9 V to +/-7.5 V, and made a few other changes in an attempt to reduce power consumption.
After the new parts arrive, I'll install them and update again.
Update #8 (February 3, 2019)
I've implemented the various power supply changes, and everything seems to be working properly. FFTs for both the positive and negative 7.4 V rails look fairly clean in the audio band, with only a few small noise spikes. Reducing the dropout between the LDO inputs/outputs for the analog section seems to have reduced the ripple a bit too. I did some averaged measurements with the FFT function of my oscilloscope (in the absence of any better noise measurement tools), and came up with roughly 10 uVrms for the positive rail and roughly 20 uVrms for the negative rail (to the nearest 10 uVrms.) I expected 11.1 uVrms from the positive rail and 33 uVrms from the negative rail, which seems reasonably close.
In addition to redesigning the analog PSU section, I also changed the yellow and red LEDs to lower-current parts, and replaced the input filtering ferrite bead for the +3.3 V LDO with a switching diode. The switching diode is dropping the input voltage to the LDO to 4.1 V - 4.2 V (depending on the current load), which is boosting the efficiency of the LDO. At present, the highest-power configuration draws 230 mA; slightly over my 220 mA re-design goal, but not by much. This should leave plenty of room in the power budget to add the USBStreamer (~200mA) and overhead for input/output drive currents. The next step will be to install the output boost stages and verify them.
Update #9 (February 9, 2019)
I've installed and verified the output stages. The frequency response is mostly flat (within +/-0.1 dB) in-band, and the matching between channels is pretty good. The gains are reasonably close to the designed value (13.2 dB.)
The next step will be to install the line-level input conditioning stages and verify them.
Update #10 (February 16, 2019)
I've installed and verified the line-level input stages, but not without a little excitement. During an earlier round of component mounting, I installed AC coupling capacitors at the inputs of the codecs, because I was already installing the various decoupling and biasing capacitors. Unfortunately, I goofed and used an outdated version of the values, so I had to pull up the previously installed AC coupling caps and replace them. I was using a large soldering tip with copper braid, and a couple pads at the inputs to the fourth codec stuck to the capacitors and tore up. After some careful x-acto knife scraping and a bit of surgery with some 28ga wire, I managed to re-build the connections, but it's definitely aggravating to have to do it in the first place. Lesson learned: When desoldering, go slow, and use a smaller iron tip with pads that connect to tiny (in this case 7mil) traces. The measured frequency response of all channels has the expected shape, and the matching between channels is again quite good. The designed passband gain was -15.4 dB; actual values are about 0.2 dB below this.
The typical analog input gain of the codec parts I am using (MAX98090) is 2.2 dB to 2.4 dB, with a signal inversion. Combined with the inversion of the line input stages, the total analog pathway gain is about -15.65 dB + 2.3 dB + 13.1 dB = -0.25 dB. Not quite unity, but pretty close. Next up is the last big analog section, the RIAA phono preamps. This board is getting very close to being completely populated!
Update #11 (February 22, 2019)
I've installed and verified the RIAA phono preamps, and luckily, the process was pretty straightforward. Channel matching is surprisingly good, with the deviation between channels at low frequencies maxing out at about 0.2 dB. The designed gain was 29.8 dB @ 1 kHz, actual gain is 29.4 dB to 29.5 dB @ 1 kHz.
Next up are the components to connect to the USBStreamer that provides the USB to I2C interface.
Update #12 (February 24, 2019)
I've installed all of the support components for the USBStreamer; everything is working as expected. With basically all of the hardware installed on the board, this is also the first opportunity to get an idea of the overall current draw, presently about 430 mA. I'll be able to get a better idea of the draw over the range of operating conditions once I've finished the firmware. I've also attached all the RCA jacks and channel switches.
Next up is finishing the firmware. I've already written part of it to test parts of the board during bring-up, so I just need to expand the existing state machine to incorporate all the required functions.
Update #13 (March 24, 2019)
I've managed to write firmware that handles all the basic tasks needed. Unfortunately, while I was testing the firmware I stumbled across a really big mistake I made during the design phase, one that I hadn't caught with any of my previous unit tests. This was pretty frustrating, as I thought that I was working on the final step with the firmware. I spent a couple days thinking about various solutions to the problem; I've settled on two that I think are the most reasonable given the constraints (I don't want to re-spin the PCB and pay for a whole new set of components, I want the most robust fix possible, and I want the fix to be easy and minimally invasive.)
The next step is obviously to fix the big problem I uncovered. I still have a few tests to run on one of my possible solutions; I'll include more details about the problem and the potential solutions in the next update.
Update #14 (March 31, 2019)
I've implemented and tested a solution to the big problem I encountered while working on the firmware. The outputs of the interface were originally badly distorted at full scale; examining a sine wave output on my scope showed an asymmetrically clipped waveform (yellow trace.) Probing inside the output capacitor (green trace) confirmed what I had feared as soon as I saw the high-side clipping: I had forgotten to address the output bias of the codecs before connecting to the output amplifier stages during design! The codec outputs operate from a 3.3V rail, and have a bias point of about 1.25V. The codec output with bias was being amplified into the positive supply rail.
After thinking about the problem for a little while, I decided that there were really only two choices: 1. Insert an AC coupling capacitor into each signal line between the codec and the output stage, and 2. Do something else. The traces from the codecs to the output stages are only 7 mil and there are no components inline, so inserting an AC coupling cap would involve surgery on tiny traces and soldering on scraped-off tiny traces; not exactly an appealing proposition. Duly motivated, I began searching for alternate solutions. After playing around with various ideas for a bit, I decided to address the problem by moving the bias point for the shunt resistor in the feedback network of the non-inverting op-amp output stage. It was attached to ground in the original design; the math showed that moving it to the codec reference would reduce the output bias to about the reference voltage. Not ideal, but better than nothing. (For more details about the math and a circuit design that completely eliminates DC bias at the output, see this blog post.) I made some forked wire assemblies from 28ga wire and soldered them between each codec's reference capacitor and the bottom of the shunt resistor. The results were slightly better than expected; most channels had an offset of about 1V after the modification, and seemed to be well within the linear range of the amplifier. The modification had the unsurprising added benefit of reducing power consumption by about 20mA, a welcome development.
The next step will be confirming the operation of the entire system. While working on correcting the output clipping issue, I discovered that the codec for channel 4 seems to be configured correctly, but it isn't outputting any audio, so I'll definitely be starting the systemwide check by addressing that problem.
Update #15 (April 29, 2019)
This update is a bit of a bummer. I've been working on validating system operation, and I'm just about entirely stuck. The channel 4 codec seems to refuse to accept or output I2S; I've tried everything short of replacing the IC, which isn't really an option on this particular PCB. Most other parts of the system are working properly, with the exception of the codec input switching behavior (the registers change, but the inputs don't appear to actually switch) and an anomaly involving codec inputs. The switching behavior is likely an issue of firmware command sequencing, so I'm not too concerned about that at the moment. The far more pressing issue is the HPF-like recurring pattern appearing at the inputs of codecs 2 and 3. This pattern is only present when viewed from the codec input through signal display software; the pattern is NOT observable as an electrical signal at the codec inputs using an oscilloscope. I tried removing one of the input capacitors to decouple the codec from the input stage, but nothing changed. Below is a screen grab of the signal display software; it is uncalibrated so the vertical scale isn't meaningful.
I've tried everything I can think of, to no avail. The firmware I've written will configure the dev board codec, and the inputs work properly, so it's not the firmware. The inputs of codec 1 on the audio interface PCB work properly, so it's not the PCB layout. At this point, it seems like the only difference between the two boards is that the codecs were hand-soldered on the audio interface PCB, and machine-soldered on the dev board. If that's the problem, it is unfortunately one I can't solve with the tools at my disposal, and therefore the end of both this board bring-up and the project. It wouldn't be a total loss, because I have learned a lot working on this, but it would have been really satisfying to get the whole thing working properly. I think I'll probably be steering clear of QFN parts on all future hobby projects.
Update #16 (October 26, 2019) - FINAL UPDATE
Sorry for the big delay in updates; I've been very busy lately. After talking with some more experienced engineers, I think that it is possible that the PCB layout is the culprit; specifically how the ground connections for the exposed pads on the codec parts are implemented. I'm not interested in spending the time and money on another PCB re-spin (too many other enticing projects to work on!), so I've decided to put this project to rest. I have one stereo I/O channel with it's switchable phono preamp fully functioning as long as the board is powered from a bench supply (I failed to design the board to handle the minimum USB voltage, so it isn't happy if it gets only 4.5V), so the board will live on as a bench tool of sorts.
Many thanks to all of you who followed along and/or offered helpful suggestions!