Big Bot Billy Bass
Introduction:
This project started a very long time ago when the first Amazon Alexa was released and my dad bought one for the house. A while later, I saw a video online of someone putting the Alex inside of the Billy Bass, making the fish act as the intelligence of Alexa. After I saw that video, my dad dug in his closet to show me his own Big Mouth Billy Bass from way back in the day. All of a sudden, it seemed possible! Now, as a college student, I wanted to try it. After my freshman year had passed as a college student, I got my hands on a Raspberry Pi Pico W, Particle Photon 2, and an ESP32. During the summer, I got busy and never even ended up getting started. 
A few months ago, I saw a post for this community competition and thought this was a perfect opportunity to actually do this project! So I began working.
Stage 1: Controlling Billy
Stage 1 of the project involved taking the Billy Bass apart and learning about the inside. Upon opening, I saw there were wires to 3 different motors, the head, the mouth, and the tail, 2 wires for the button, 2 for the motion sensor, the battery pair, the speaker positive and negative, and then 2 for the on/off switch. I started with the Photon and isolated testing the motors first, cutting and stripping the wires off of Billy to hooking them up to the pins of the Photon. In order to use the motors safely, I bought 2 DRV8835's from my schools electronics store. With these motor drivers, I wrote a somewhat simple test sketch to raise and lower each part of Billy individually. After that was successful, I upgraded to control by button input on the breadboard I was working on.
After this all was working, I moved onto the button and motion sensor from the toy. I analyzed the input with the analog inputs on the Photon and figured out thresholds and how to make the inputs work for my goals. The motion sensor put up a notable fight with its inconsistencies in data and deciding when to work and when to not. The button was much more reasonable and was much easier to implement.
Stage 2: Project Website
The next thing that was necessary for this project was a website to operate the fish. So after a long time of learning HTML and CSS, I created a pretty good looking website, if I do say so myself.
Basically how it works is there is a python server that is running on my computer that runs an AI model from OpenAI that is able to take what ever text input from the website and shoves it through the model for an answer. But, cool part, is that I made my own personality for Billy via a prompt I was able to use.Using this, you can type in a query, then it goes through an AI and outputted on the website.
Another important feature I wanted for this project is to keep the songs from the original toy. So, I added two buttons that would play the audio captured from the toy on the website. Now that that was working perfectly, it was time to go back to the Billy internals.
The next thing I worked on was animating Billy from the original toy. So, how do I do that? I went through the painstaking effort of recording the original animations and going through the slow motion video frame-by-frame and coding the animation using delay() functions and digitalWrite(); you can check it out in the GitHub!
After some minor tweaking afterwards, I got the animations looking great! Now, I am able to, by the press of a button, play the full animation of the original Billy Bass tune while listening to the same song from the website at the same time with the use of Particle functions in javascript!
Particle functions came in useful for this project when activating all of the things I wanted to do from the website interface, and even later, voice input.
Stage 3: Audio
Audio has its own stage because of how painful this was to get working. Online, I had a very hard time deciding which audio module to get, and I eventually decided the VS1053 MP3 decoder module would work the best for this project. However, I still had to decide where to get the module from. My first attempt was from Amazon, but that failed miserably after sinking probably 10-12 hours of trying and debugging and it literally just not working. Seriously, I was not getting any sound out of the VS1053, it was very frustrating. I soon gave up and went with a more expensive but less hassle Adafruit version of the same module, which came with its own SD card slot, which was perfect.
And low and behold, it worked almost immediately after playing with it for about an hour or so. And now I could use a pair of headphones to get sound from the module, starting small from the built in sine test feature. I uploaded the two original songs onto the SD card, and was able to play them from a simple test sketch provided from Adafruit. Then, I adapted it to work with the same button that played the motor animations discussed before. So now, I am able to recreate the entire system the toy had before, which was a major milestone for the project. With the press of the toy's button, I activated the animation and the audio playback. In addition, using the Particle functions, I am able to do the same thing from the website interface.
One of the next major endeavors was to stream the response the Billy AI gives from the website, which is theoretically possible with the VS1053. So with a lot of digging and trial and error, I eventually made it so I was able to stream data from a file, like the response.mp3 file created from the text input to the AI, through the VS1053 to output through the headphones I was using. Eventually, everything was working just as I wanted it to and I was ready to switch to the toy's speakers. So, I cut and stripped the speaker wires from the toy and put the speaker terminals going to the VS1053 output.
Now, the problem is that it is extremely quiet. So, using the knowledge I gained from lab this last semester, I designed and built my own variable gain, summing, power audio amplifier, which was surprisingly one of the most rewarding parts of the project to me. I hooked the output of the amplifier to the speakers and now it's much louder, AND I can adjust the volume! And just like that, I have a full system. Text input from the website goes to the server running an AI model that responds to the query, then the response is created in the form of an MP3 file, which is then streamed from the website to the VS1053 controlled by the Photon, which has it's signal amplified by my custom amplifier, then finally, outputted through the toy's speaker. A fully functional project!
INTERMEDIATE PRODUCT:
Stage 4: Microphone Input
Even though I have a fully functional project, I wanted to add one more element that would make this much more complete, a microphone. First, I had to select the right microphone for the job. Again, I had a very hard time deciding, and I eventually decided on the ICS 43434 I2S MEMS microphone. Unfortunately, this required a second brain surgery for Billy because this microphone require me to switch to the ESP32. The good news was that I had one lying around and that it would make the responses much faster.
So I fully replaced the Photon for the ESP32, which was much harder than I expected because I needed to learn how to manually use the WiFi features instead of relying on the Particle Functions the Photon had. Beyond that, I had to figure out a better way of playing audio and animations because the ESP32 had less support for the VS1053 than the Photon did. In addition, I had to reallocate the pins from the Photon, which was already very tight, but I was able to with a whopping zero pins to spare on the ESP32.
I used the microphone to write to a new WAV file of the SD card on the VS1053, which was then sent to the python server and entered into the same stream as before, except the input was transcribed using OpenAI's Whisper and put into the Billy AI pipeline. I spent about a month working on the software and hardware because of how much I had to redo from the Photon to the ESP32, all of which can be found on the GitHub repo's branches. I literally finished getting the entire output stream today, which is why this post is so late. But now, I have the website input, and now a voice input that optimally sends a created WAV file to the server, which transcribes it and enters that into the Billy AI, then the response is created in the form of an MP3 file and uploaded to the website, which is streamed through the same output sequence as before through the VS1053.
FINAL PRODUCT:
Conclusion:
I had so much fun and learned an insane amount about microelectronics and embedded systems during the three-ish months I have been making this project. There was much frustration, but the final result was so worth all the trouble it was. I literally built my own Alexa-like AI assistant with its own personality on an ESP32. I learned HTML, CSS, and Javascript to make my own website, I learned a little bit of python to run the server, which I was very unfamiliar with before this. I learned proper project organization, which is one of the biggest things I got out of this project. Future steps is maybe have a Raspberry Pi running the server locally with Ollama or something of the sort, then maybe even a PCB for everything so I can actually hang it on my wall! I probably will not get to any of those very soon, but it is exciting to think of where this project can go!
Again, I literally got it to a place I was happy with to submit maybe 4 hours ago at the time of submitting this, so excuse my late submission, please. It was before midnight in my local time if that makes it any better. Even if not, I am sharing for everyone to see what I did. 
Here is the GitHub repo for this project: https://github.com/KieferD17/Big-Bot-Billy-Bass/tree/Photon
Top Comments