There are a few different tool flows that are available to program the Maixduino. I previously used the Arduino IDE which allows reuse of existing Arduino libraries for common peripherals.
It appears that much of the Neural Network programming for the K210 KPU (knowledge processing unit) has been done using MaixPy (microPython). So, I'm going to look at a tool flow that allows me to upload the MaixPy firmware and the network models and the Python programs.
The memory utilization appears to be the tricky part of the setup. The Maix-1 processor module on the Maixduino has 16MB of flash memory and 8MB of RAM plus the 4GB SD card. I still have lots to figure out, but it appears that real-time processing is constrained by the RAM size and maximum network model size is constrained by the flash memory size. I haven't used microPython before but apparently the flash memory is mounted as the directory '/flash' and the SD card is mounted as '/sd'. A boot.py file is executed upon boot and /sd/boot.py has priority over /flash/boot.py.
Here is an excerpt from the spec:
The kflash utility is used to upload the MaixPy firmware and the network models (or other binary files). The examples that I came across all are using the flash memory. I'll need to figure out the limitations of using the SD storage.
Below is the kflash GUI. The MaixPy binary file is loaded at address 0x0000. The other binaries are loaded at an offset addresses beyond the end of the MaixPy binary. Here I am loading the "full" version of the MaixPy binary which allows for running remotely from an IDE. Unfortunately, I discovered that the size of this binary would not allow me to run the neural network examples. I ended up loading the "minimum" version and not using the IDE.
The version of MaixPy that was pre-loaded on the board was v0.1.1 and I updated it to the latest stable release v0.3.2.
The next utility required was the uPyLoader which enables file transfers between my Windows file system (on the left) and the Maixduino file system (on the right). This tool also allows for remotely executing Python programs. And it has a terminal capability for command line execution.
Image Classifier
The neural network example that I decided to try is a yolo2 image classifier that has been trained with 20 different classes. I uploaded the 20class.kmodel at offset 0x500000 and then transferred and executed the 20class.py program. The program captures frames from the camera and displays the image and bounding box with the class and frame rate superimposed. It is nice that using the provided libraries that the code to run the model is very compact.
20class.py
import sensor,image,lcd,time
import KPU as kpu
lcd.init(freq=15000000)
sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.set_vflip(1)
sensor.run(1)
clock = time.clock()
classes = ['aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor']
task = kpu.load(0x500000)
anchor = (1.889, 2.5245, 2.9465, 3.94056, 3.99987, 5.3658, 5.155437, 6.92275, 6.718375, 9.01025)
a = kpu.init_yolo2(task, 0.5, 0.3, 5, anchor)
while(True):
clock.tick()
img = sensor.snapshot()
code = kpu.run_yolo2(task, img)
print(clock.fps())
if code:
for i in code:
a=img.draw_rectangle(i.rect())
a = lcd.display(img)
for i in code:
lcd.draw_string(i.x(), i.y(), classes[i.classid()], lcd.RED, lcd.WHITE)
lcd.draw_string(i.x(), i.y()+12, '%f1.3'%i.value(), lcd.RED, lcd.WHITE)
else:
a = lcd.display(img)
a = kpu.deinit(task)
I recorded a short video of the Maixduino classifying objects. Unfortunately, I still have not figured out how to get a decent recording from these small LCD displays so the images are not very good. I used a photo of a woman and a dog to test the classifier. You can see that if the woman's full face is not in the frame that she is mis-classified as a dog or a cat instead of a person.
Here are some frames from the video. Barely legible. The frame rates are between 0.5 to 1 FPS which is not great but acceptable.
I definitely need to print myself a case before going forward because the unit is too difficult to use with a loose camera and display. And I need to learn how to train and configure my own models so that I can customize the unit for the specific objects that I want to detect. Maybe also add facial recognition.
