Introduction
This will walk you through setting up your MaaXBoard with the hardware and software you must have, direct you through changes to make on an Edge impulse project and additional package installation steps, and result in running a script with an Edge Impulse model to see how it performs in terms of processing speed, power, and accuracy.
Debian vs Yocto
In addition, this benchmarking project was done with the Yocto image instead of the Debian image Monica used in her project. The Debian image is used for main desktop features for embedded systems. The Yocto image has more flexibility with custom features for embedded systems Below is an image from Kosta Zertsekel's article titled Yocto vs Ubuntu for Embedded that sums up the difference between the two.
This benchmark involves running the FOMO and MobileNet Edge Impulse models on the Yocto image for the MaaXBoard and seeing how it compares to Tenserflow models previously benchmarked on a Debian Image. A reference to this can be found on this Benchmarking Machine Learning on Avnet's MaaXBoard post by Monica Houston. After running the benchmark, you'll see results of power consumption (measured by the current), the accuracy of the models, and their inference time. From the results you can determine which model you would prefer to use for object detection projects.
Image
The models are tested on an image that consists of an apple and banana as the main objects. This is the same image used in Monica's Benchmarking Machine Learning project and Alasdair Allan's own benchmarking project. So I also used this image to keep my project as consistent as possible
Models
FOMO vs MobileNet
The Edge Impulse model that will the main focus of this benchmark is the Faster Object, More Objects (FOMO) model that Edge Impulse studio provides as opposed to the default MobileNetV2 SSD for a better benchmark. For object detection, the FOMO model is set to take a 224x224 input for the image size like the other models in Monica Houston's Benchmarking Machine Learning on MaaXBoard project (MobileNet v.2, Mobilenet v.1, EfficientNet v0) whereas the MobileNetV2 SSD model only takes a 320x320 input for image size. Compared to MobileNetV2 SSD, FOMO is 30x faster and captures the position of objects instead of the whole size in the image. This is stated in the Announcing FOMO (Faster Objects, More Objects) Edge Impulse blog. The default Edge Impulse model has also been tested and its results are also compared to the other models.
Dataset
For the Edge Impulse models, I took 181 pictures of backgrounds, apples and bananas.
Ex:
Benchmarking Guides
Here are step by step guides on how to install and run Edge Impulse on the MaaXBoard and benchmark the model.
Install and Run: http://avnet.me/install_ei_maaxboard
Benchmark: http://avnet.me/ei_maaxboard_benchmark
Results of Benchmarking for Edge Impulse
MobileNet Model Build (320x320) on MaaXBoard:
FOMO Model Build (224x224) on MaaXBoard:
With the image size reduced and the model not utilizing full-size bounding boxes, the FOMO model runs significantly faster but is still accurate.
Comparison of Results
Compared to the other models in Monica Houston's Hackster project Benchmarking Machine Learning on MaaXBoard (results shown below), the Edge Impulse model results are slower than a few other models in processing speed, but it has a pretty high accuracy.
- Note the results below are the result of models ran on the Yocto image instead of the Debian image. The accuracy scores are the same but the inference speed is slower compared to when the models were run on Monica's project.
Conclusion
Based on the results, the Edge Impulse models consumes more power compared to the other models. Depending on the image input size (320x320 vs 224x224) and the use of or omission of bounding boxes, the inference for object detection can be faster or slower compared to other models. In terms of accuracy, the Edge Impulse models did well! Though they were pretty accurate in object detection, the FOMO model had the faster inference time.
See how the models you make on Edge Impulse compares to other models on the MaaXBoard!