Tennis Picker@The Core#4 The ML configuration and Obeject Detection

#4 The ML configuration and Obeject Detection

1 Machine Learning in PSoC 62Sx

Infineon provide ModusToolbox as a set of tools to develop applications for Infineon devices, including GUIs, command-line programs, software libraries, and third-party software.  Machine learning solution is part of the ModusToolbox ecosystem that helps to  build, evaluate and benchmark pre-trained ML models. The ML libraries easily and efficiently run the inference on an Infineon MCU.

ML configurator is one GUI tool to import pre-trained machine learning models and generate an embedded model (as C-code or binary file). This generated model can be used with the ML library along with your application code for a target device. ML configurator also helps to fit the pre-trained model of choice and evaluate its performance.

Machine learning can run on any PSoC 6 MCU. The recommended hardware/kit platform is the CY8CKIT-062S2-43012. This CY8CKIT-062S2 Kits can be targeted as well, but the  memory size and performance requirements to run the inference engine are bottlenet when it comes to designing the NN, as migration ML magic wand project below.

2 Object Detection

Object detectors can identify which of a known set of objects might be present and provide information about their positions within the given image or a video stream. Refer to  Tensorflow object_detector page for more details. An object detector is trained to detect the presence and location of multiple classes of objects.

There have been dozens of valid object detection model such as VGG, mobile Net, and yolo. But all of them are oversizing for embedded MCUs. That is why there are little resources on such applications.

Tensorflow provide SSD_mobileNet tflite as compressed format, but is still too large for PSoC 62S4 with 256 KB Flash and 128 KB SRAM.

The idea of moving the rover accoring to the size and location of detected Box is the core of this Tennis Picker project. And That is obviously challengous.

3 Migrating ML magicwand Project

Before the SSD model project, go to sample magic wand project is fast-tracking. This is normal ML example in many arduino boards library. In Modus Toolbox IDE, it is called gesture classification. Detect moving gesture with 6-D gyro-sensors.

First, import the PSoC 62S2-kit example project and build it,

The SRAM memory is 1M and flash of 256k.

Then, start library manager

to add bsp

select the PSoC 62S4 as active BSP the update the library,

Build the project, it passes, only because the Flash can hold the codes, but not fit for SRAM with 115968 Byte. It is sure to crash where downloading into the MCU.

But that is not the point for now. Modus toolbox provide ML configurator to do modification and validation.

4 Machine Learning Configurator

Laounch the ML configurator from Modus Toolbox directly, Then setting parameters of NN model, browse and import ML model in h5 format

Start validation on desktop, with sample random numbers. This is QEMU behind this simulation, it works fine.

ML configurator can evaluate memory size.

and optimize size of memory as reference

It can download into board for validation as well, the board feed true value by means of serial port, the ML engine runs in desktop computer

It fails since there is no such data get from board, BIM60 is not comes with PSoC 62S4, only valid in PSoC 62S2 Kits.

Anyway, the ML configurator is valid all-in-one tool for ML project.

5 ML project in General

To make it simple, here is ML project development flow,

The hardware of CY8CKIT-062S4 PSoC 62S4 pioneer kit support the software structure,

ML Configurator has two options for inference engine:
• TFLM inference engine
• Infineon inference engine

It is recommand to work with IFX, with reference to APIs of MTB Machine Learning Inference Engine Library. For TFLM , refer to TFLM  APIs in github.

Here is core ML engine in for-Loop of Gesture Task ,

for(;;)
    {
        /* Wait until there is 128 samples from the accelerometer and the
         * gyroscope in the circular buffer */
        cy_rtos_waitbits_event(&sensor_event, &sensor_event_bits, true, true, CY_RTOS_NEVER_TIMEOUT);

        uint16_t cur = 0;

        /* Cast the data from an int16 to a float for pre-processing */
        int16_t temp_buffer[SENSOR_BATCH_SIZE][SENSOR_NUM_AXIS];
        cy_fifo_read(&sensor_fifo, &temp_buffer, SENSOR_BATCH_SIZE * SENSOR_DATA_WIDTH);
        cast_int16_to_float(&temp_buffer[0][0], &data_feed[0][0], SENSOR_BATCH_SIZE*SENSOR_NUM_AXIS);

        /* Third order butter-worth filter */
        while(cur < SENSOR_NUM_AXIS)
        {
            /* Initialize and run the filter */
            iir_filter_init(&butter_lp_fil, coeff_b, coeff_a, n_order);
            iir_filter(&butter_lp_fil, &data_feed[0][0], SENSOR_BATCH_SIZE, cur, SENSOR_NUM_AXIS);
            cur++;
        }

        /* A min max normalization to get all data between -1 and 1 */
        normalization_min_max(&data_feed[0][0], SENSOR_BATCH_SIZE, SENSOR_NUM_AXIS, MIN_DATA_SAMPLE, MAX_DATA_SAMPLE);

        /* Get the model output size */
        int model_output_size = mtb_ml_model_get_output_size(magic_wand_obj);

        /* Convert to int16 based on the q format */
        MTB_ML_DATA_T data_feed_int[SENSOR_BATCH_SIZE][SENSOR_NUM_AXIS];
        mtb_ml_utils_convert_flt_to_int16(&data_feed[0][0], &data_feed_int[0][0], SENSOR_BATCH_SIZE*SENSOR_NUM_AXIS, QFORMAT_VALUE);

        /* Feed the Model */
        input_reference = (MTB_ML_DATA_T *) data_feed_int;
        result_buffer = (MTB_ML_DATA_T *) malloc(model_output_size * sizeof(MTB_ML_DATA_T));
        mtb_ml_model_run(magic_wand_obj, input_reference, result_buffer);
        control(result_buffer, model_output_size);
    }

All input data be prepared are read for one and only code to ML get feedback,

mtb_ml_model_run(magic_wand_obj, input_reference, result_buffer);

This makes every one access Machine Learning easily, even without  previous experience thanks to toolset provide by Infineon Modus Toolbox.

6 TFLM or IFX and memory challenges

In all, this CY8CKIT-062S4 PSoC 62S4 pioneer kit is fit for Machine Learning project, but challengous in memory size. Another part consumes memory most is the camera library, that make the situation complex and hard to solve.

But there are still wayout. I shall figure it out on external memory.