I'm doing this slightly out of sequence as I had intended to finish deploying the Keyword Model first. Unfortunately, I'm having some issues getting the C++ inference library integrated into my Modus application. Hopefully, I can get that figured out soon.
Modus Toolbox Machine Learning (MTB-ML) is a new addition to Modus Toolbox. I'd like to see if I can use it to implement some form of ML on the PSoC 62S2 board for my project.
Overview from the Users Guide:
MTB-ML 1.0 is a set of tools, libraries, and middleware that will help you build, evaluate and benchmark pre-trained ML models. MTB-ML libraries easily and efficiently run the inference on an Infineon MCU. These libraries and tools help you rapidly deploy Neural Network (NN) based classification-type ML applications.
Note Data collection and training algorithms are not part of the MTB-ML 1.0 release.
MTB-ML also provides a Configurator to import pre-trained machine learning models and generate an embedded model (as C-code or binary file). This generated model can be used with the MTB-ML library along with your application code for a target device. The tool also lets you fit the pre-trained model of choice and evaluate its performance.
So, there are a couple of issues that I'm facing
- There are currently only 2 examples that I can find in the Infineon GitHub repo:
- mtb-example-ml-profiler - uses MNIST model
- mtb-example-ml-gesture-classification - uses Magic_wand model
- The configurator appears to only use Keras H5 format models
My first step is to verify that I can get the gesture classification example working with my hardware. I looked at the MNIST example and that is using a data file rather than an image sensor input, so I thought I'd try something with a real sensor - in this case, the IMU, to classify gestures.
The first issue I ran into is that the example is only configured for the following 2 shields:
- CY8CKIT-028-TFT
- CY8CKIT-028-SENSE
That surprised me because the CY8CKIT-028-EPD shield that I'm using has the same BMI160 IMU as the other shields. It appears that the interrupt pins are allocated to different pins on the Arduino connector. I have not found where that is configured yet, but since I happen to have a CY8CKIT-028-TFT shield - I'll use it for this test. If I can find where to reconfigure the pins, I'll also try it with the EPD shield.
The sample application is available in Project Creator:
I was surprised that even though the shield needs to be specified in the Makefile, that the shield library is not used (but the BMI160 is specified). It appears that the shield name is only used in the Makefile and it changes the BMI160_CHIP_ID in the bmi160_defs.h file. I guess I'll have other things to figure out if I want to use the EPD shield later.
In the Quick Panel there is access to the ML Configurator that allows the selection of which model is used with the application.
Unfortunately, for ML 1.0, only Keras H5 format is available (the pulldown only has 1 option). I had hoped that the library would support Tensorflow Lite or ONNX models as I've never tried using (or finding) Keras models.
Here is the file tree for the application:
The application uses a "Magic Wand" model that has been trained to classify 3 gestures: square, circle and side-to-side.
Here is a quick demo of it working:
It likes exaggerated movements (the Readme indicates on the order of a foot). The results are printed to the serial UART. If confidence is >97%, a class is reported.
In general it works well. A couple of times after it correctly classifies "circle", it incorrectly classifies "square" - I think because I have some small movement at the end of the motion (old age 
).
So, if you have a pre-trained Keras model and a few distinct classes it looks like it is working well. I'm somewhat concerned about my being able to train models for voice and vision in the timeframe of this challenge. I'm thinking that I may have to "pivot" and use models and hardware that I'm familiar with to implement the ML part of my project and use Modus Toolbox and the PSoC 62S2 to develop the communication and control part. I'll need to think on that for a while...
Update 10/04/2021
I was mistaken, this example does work with the EPD shield. I must have done something wrong earlier. You do need to leave the setting in the Makefile as "CY8CKIT-028-TFT".
I guess the interrupt pins must not be required for this example.