For the Element14 Project14 VisionThing theme I came up with a project which aimed to identify invasive insect species which could then be humanely destroyed without the need for pesticides or other chemicals, leaving other species unharmed. I called my project iBox. The target is the Box Tree Moth (Cydalima Perspectalis). My suggestion was very generously met with E14 sending me a Beaglebone AI. After receiving the BB-AI I realised it needed a USB-C power lead and therefore had a few more days waiting until I could buy one - that was good as in that period was able to observe fellow members posting about overheating problems. Therefore before powering up I made an aluminium fin arrangement which slotted into the existing heatsink's grooves (with a little heat transfer compound) and when clamped up gripped those existing fins. I then ran a small 12v DC fan across those new fins. My results detailed in the links below were very promising: the BB-AI stayed cool and I could progress the project.
I then spent many days trying to install OpenCV, until it was kindly pointed out that OpenCV2 was already part of the BB-AI install !
My next issues were how to use the board productively, I had been running this headless via SSH but that would negate the GUI display of results and also made it more difficult for me (a newbie to command line fun) to input files. I did run the BB-AI on HDMI with keyboard and mouse but then I would find the code development difficult. I was able to run the Texas Instruments TIDL examples but found this limiting over SSH as the who Vision Thing was based on graphics etc.
[Photo Source: SecurityAffairs.co]
2. Way Ahead
None of the above was directly helping me develop a solution to my iBox project and I felt like time was slipping away. Then I realised that it wasn't mandatory to use the BB-AI for this vision competition, although it was probably preferred as it was a new product. I therefore opted to move away from BB-AI to progress my iBox and instead I moved development onto my i5 Windows laptop. In this environment I was able to setup OpenCV4.1.1 along with Visual Studio C++ to make a great development combination which was easy for me to use, visually rewarding in the input and output and would allow me to learn some of the vision algorithms. What my approach doesn't address is the AI approach or use of the BB-AI, so my apologies there. As the product has just been released there is little available on the TIDL and other vision engines in the BB-AI.
I hoped this approach would:
- allow iBox to progress
- let me learn some cool things about vision projects and OpenCV
- let the BB-AI following develop and progress, allowing me to return to the BB-AI when there is more open source knowledge available
3. Installing OpenCV and Visual Studio
The starting point for my Windows install of OpenCV was their website https://opencv.org/releases/
I already had a free version of Visual Studio installed and then found this great article for combining the two: https://www.opencv-srf.com/2017/11/install-opencv-with-visual-studio.html
Initially I had several issues but they were down to a few simple errors where, in my haste, I had placed the new linker paths in the wrong boxes. Follow the previous link and everything should be fine. I do find for graphical work the Microsoft Visual Studio to be very easy to use, the debugging of the code is also very detailed and there are often several different approaches to seeing what has gone wrong; hovering over code, adding watch variables and breakpoints. The intellisense also helps new developers like me to instantly see when something is going to cause an issue later.
4. Code Examples
The best way of getting going is to copy and paste code examples from online articles. The ones I found that looked interesting were in Python but it was a simple matter to adjust them for C++.
Below is the input and output from one of my first examples using the canny processing algorithm of edge detection:
That may or may not be very interesting to you but to me it is a great step forward. Having the code working can now allow me to play around with the filters, explore the OpenCV documentation and try and work out what may be useful. I'm assuming I will undertake some form of automatic contrast/brightness adjust to maximise edge information and then extract that image. That could then be used to compare against known shapes of a moth. Taking a couple of images should provide a positive identification on the shape...but probably not species. So I can imagine some parallel calculations on the image will be required to maximise colours or patterns on the wings to get another identification on those. Weighting and adding those outputs should allow confirmation of the target moth: time to crank up the voltage !
5. Face Detection in OpenCV
I browsed the internet and pulled together some articles and tested out various snippets from each, these included:
And after some changing of my code which also found my suitable 'JPEG' from the internet search was actually a JFIF file format ! After adjusting and running my code I was able to then get this appear .
[photo source: http://i.huffpost.com/gen/2202994/images/o-DIVERSIFY-facebook.jpg ]
At least one of these people was identified as having a face although it shows something in the OpenCV script wasn't working too well. That was a first step anyway. Now I can try out some other photos and start to look at how the code determines what constitutes a face before moving on with iBox project.
The https://docs.opencv.org/3.0-beta/modules/face/doc/facerec/facerec_tutorial.html article is a great introduction to face detection, like it states, humans find this task so very easy. Sometimes we can even recognise people by the back of their head so there is much more than simple facial features being used. Reading that document made me think the above experiment failed as I didn't have any training data. I downloaded and unzipped the Yale Face Database code - if you are one of those 15 individuals then my thanks to you for taking part. I tried to follow the OpenCV training data tutorials and others that I found online but had issues configuring the data with the inbuilt tools.
6. Shape Detection i.e. Moth
Again I decided to move on as faces were not what I wanted anyway. I wanted to be able to extract and verify a shape. I found a few tutorials that identified geometric shapes within a photo and for a very short while considered the idea that a moth could be a collection of such shapes. However after some more internet searching I decided to work through these OpenCV tutorials:
I gathered my training data consisting of many photos of moths into a folder - these were any moths as at this point I want to just identify the shape. I plan that later I would be able to undertake a second stage analysis of the image to look at identifying marks and features/colours to get a high probability that the moth is actually a Box Moth. A high result for both tests would indicate that the moth could get destroyed and the plan would be to zap the insect with a high voltage.
And as the deadline for this VisionThing Project14 draws to a close that is where I still am.....reading, copying & pasting code to Visual Studio and wading through error codes. I'm learning along the way but I won't get much further with just a few days left and therefore decided to write this blog post up now rather than wait until the very last minute and end up with meaningless scribble. If I have a breakthrough over the weekend I will update this post.
7.1 What Was Good
I can say that iBox is still just an idea, but I believe it is achievable having started to explore the world of computer vision and detection.
I therefore managed to:
- Cool off my BB-AI and run it for many 24/7 sessions with no detrimental effect.
- Run BB-AI both headless via SSH and using its desktop to a HDMI monitor + USB keyboard and mouse
- Successfully run the Texas Instruments TIDL examples using command line and SCP to move images from laptop to BB-AI
- Install OpenCV on my laptop
- Integrate OpenCV into Visual Studio
- Run some of the basic OpenCV algorithms on a test photo
- Achieve face recognition (well on one person anyway)
7.2 What Went Less Well
The bits I was not so pleased with are:
- This took me a very long time to achieve so little
- I'm definitely not an 'Elliot Alderson' when it comes to coding - I much prefer graphical GUI and lots of help on every error
- I still had issues with Visual Studio and install paths - each project required them to be set again (needs me to look at this again)
- OpenCV is a huge set of algorithms and functions - I was a bit naive to think I could jump in having not done anything similar before
- AI and ML - I never even got to explore this area
I don't think the BB-AI heating issues really held me up, and unless I need to add a cape I will probably keep using mine with my adhoc heatsink.
8. Next Steps?
I've a long list of things I would now like to do, having 'dabbled' in this field and found it really interesting:
- Carry on with vision detection and OpenCV experimentation using my laptop
- Follow the BB-AI development and get ready to jump back in when things are at the 'right stage' for my understanding
- For now, keep picking those caterpillars off using tweezers and the Mk1 human eyeball
- 3D print a fan mount for my BB-AI
- Experiment with what else I can use the BB-AI for
- Revisit the Texas Instruments TIDL examples - documentation is constantly being updated
- Continue to learn about AI and ML techniques having made a start by reading this great article Artificial Intelligence I The Essentials of Artificial Intelligence
- Clone myself to do all of the above !
Luckily it is winter and in the Northern Hemisphere, that means short days = long dark evenings...the perfect time to do some learning
Thanks to tariq.ahmad for a great Project14 and of course supplying these interesting BB-AI boards. Also a big thank you to the Element 14 community members who were also a great help in the discussions and questions we all had - I could easily list a few dozen names, but won't in case I inadvertently leave someone off the list. This project challenge was IMO really stretching but at the same time cool and ground breaking.