Software woes
A feature that I wanted to add to my project is license plate recognition of the vehicle that just parked. No real need for it, but it looked straightforward and there are a few different project examples for an ESP32 based camera, so I decided to try it just for fun.
Well, as Murphy’s law would have it, the fun turned into a nightmare. I’ve had some problems from the very start with the TDK software, but for the most part I’ve been able to work around them. I haven’t figured out a way to get technical support from TDK (their contact form seems sales related, and they didn’t respond when I used it to request technical support). I probably just haven’t figured out their infrastructure and to be fair, they are really set up to support OEMs not individuals. I really wanted to get the previous version of their Demo GUI because of the features that are shown in their documentation but haven’t been able to find it.
The problem that I encountered was that the USSM Arduino library had a conflict with the latest ESP32 brdlib. The latest USSM library was released in August of 2023 and there have been 17 ESP32 brdlib releases since then. I backed off the ESP32 brdlib version to a 2023 release and that allowed me to get the sensor working with my board, so I continued with the project.
Unfortunately, as I’ve been trying to incorporate new features, I’ve run into ESP32 issues with other libraries. I ran into issues with DNS resolution of sites that use ipV6 as the primary address as it appears that the ipV4 fallback is not working in the esp32 core version I’m using. I then went down a rabbit hole that consumed days of my time. I first encountered the issue when I was trying to use “pool.ntp.org” to set the time on the RTC. Switching to a hard-coded ipaddr solved that quickly. I encountered the problem again when I tried to use “generativelanguage.googleapis.com” to use Gemini for the license plate recognition (LPR). Unfortunately, a hard-coded ipaddr did not resolve it because of the way certificates are handled. I then made the mistake of trying to use AI to help me fix the problem and I spent days making changes that ended up being a complex Band-Aid that ultimately didn’t work reliably.
It seemed that I had two choices – find a later version of the ESP32 brdlib that would fix the DNS issue and still work with the USSM library or switch to a different MCU. I’ve planned my project around the Xiao ESP32S3 Sense, so I chose to try to find a compatible brdlib version. That led me down another rabbit hole. When Arduino IDE upgraded to V2 it lost the ability to easily create a clean isolated “portable” environment. There are ways that “sort of” work, but that’s a long story… I didn’t want to mess up the environment that I use for a lot of other projects or to create another virtual machine, so I switched to using another computer. Suffice to say I lost lots of time, but I think that I’ve found a small intersection of versions that work. I got DNS working which allowed me to complete an LPR program which I’ll demonstrate here, but I am concerned that if I run into additional library issues during final integration that I may have to omit some program features. I wouldn’t recommend using the current USSM Arduino library with an ESP32 MCU.
License Plate Recognition (LPR)
Processing power and memory constraints on Edge AI boards like the Xiao ESP32S3 Sense prevent doing the complex, general-purpose OCR required for License Plate Recognition. A viable technique is to capture the license plate image using an esp32 camera and then send the image to a Cloud-Based OCR for processing.
Among several options that popped up in a search for LPR projects using an ESP32 and the Arduino IDE, I found this one on Hackster: AI-Powered ANPR using DFRobot's ESP32-S3 AI Cam & Gemini API
The details of the Hackster project are available at the link and I’ll provide more detail on my implementation in the final blog post.
The project setup is like mine, but it uses a DFRobot ESP32S3 AI Camera Module. The key differences are:
- Camera: OV3660 3MP vs OV2640 2MP
- Flash: 16MB vs 8MB
The fact that I have less image resolution and a smaller flash size shouldn’t be an issue. The biggest problem is that I am constrained by the ESP32 brdlib version that is compatible with the USSM Arduino library. This is where I ran into the DNS issue with Google’s Gemini API. Now I’ve found a version that seems to work and have implemented the LPR program.
Here is the basic program flow (modified from the Hackster project):
- Capture QVGA Image using OV2640 sensor on Xiao ESP32S3
- Send image via MQTT to Node-Red for display
- Encode image into Base64 format
- Send an HTTP request (with encoded image) to Gemini Vision API to detect and extract the license plate characters
- Send JSON doc with timestamp license plate data via MQTT to Node-Red for display
For my initial test I will use either a local pushbutton or a capture command sent via MQTT from Node-Red to initiate the flow. In the final project I will use detection of a parked vehicle to initiate it.
Serial Monitor output showing the program flow:
- WiFi connect
- Camera initialization
- Sync RTC
- Connect MQTT
- MQTT request image capture
- Capture image
- MQTT Publish Image
- Send data to Gemini
- Gemini response
- Parsed data
- MQTT Publish Time and PlateNumber
Node-Red Flow
- Receive Image Data and display
- Receive Time and PlateNumber and display
- Send Capture request when button pushed
Node-Red Dashboard
There are 3 groups on the dashboard:
- ESP32S3 Camera - displays the last captured image
- Gemini LPR - displays up to the last 10 plate data responses from Gemini, latest is on top
- Command - Capture button to send capture request
Video demonstrating dashboard operation:
- Images are various sizes of license plate images
- Initially you will see stale data in the Gemini table
- The mouse pointer will move over the Capture button to select it
- The image will update immediately
- The Gemini table will update after the the Gemini response which gives a sense of the processing time
- All images in this test are printed on white paper
All previous images were taken in my office. I then moved the setup down to the garage and positioned the camera at parking distance and height and captured the next image. The image is pretty dim but the correct information was extracted. I may try adjusting the brightness setting on the camera...
Next steps
- I have been working on adding a sound tube to reduce the multi-path reflections reaching the sensor. Results look promising. I'll discuss this in the next post.
- Lots of work to do to integrate the various software programs into a single application. Hope there are no surprises. Resource conflicts are the primary worry.
- I also have lots of hardware cleanup. I am working on a separate LED strip/buzzer module that I can mount 90cm above the main sensor unit. And I need the shelf mounting hardware, sound tube adapter, and a main enclosure.
- Then writing the final project blog.
Clock's ticking. If I end up with a fully functional unit, I'll consider that a success. Having nice enclosures will be icing on the cake 
.
