Blog #9: Gesture Controlled Tetris

8 Dec 2022

Hello everyone. I welcome you to my next blog post as part of Maxim Integrated (now ADI) sponsored competition Experimenting with Gesture Sensor Design Challenge. In this 9^th blog post I will show my first outcome as part of this competition. I will show my first completed gesture-controlled game. This blog post does not contain very much technical details because I described all technical parts in my previous blogs. This blog shows project utilizes and composes all skills which I already learned and shown in all previous blogs. Here are links to all previous blogs:

Gesture Controlled Tetris Online Game

The first game which I implemented for control using Gestures instead of mouse and keyboard is Tetris. As I already mentioned in previous blogs, I implemented Tetris as an online game utilizing modern JavaScript API for communicating with MAX25405EVKIT over UART interface. All these “ingrediencies” I combined to the online game now. UART API, I described in blog #5, Web Serial API I described in blog #6. Combining these two parts with bunch of JavaScript (TypeScript) code implementing Tetris logic resulted to functional game.

See it in the action

You can see the game in the action on the following video. I was holding my phone in one hand and play the tetris using second hand. It is not very comfortable. Later when the situation become more stressful, and I was loosing I slightly moved camera outside of the view, but for most of time most gestures are visible including response on the Tetris screen.

Try it yourself

If you have MAX25405 you can play it. I bought https://gesture-tetris.fun/ domain and deployed the game here. It is online game. You do not need to install anything. Just follow instructions on game page and connect the sensor evaluation kit to the computer.

Source Code

Source code of the game are now available on the Github: https://github.com/misaz/WebGestureTetris. The main part is in src folder. Code it is split to 16 classes. Most important classes are App, Tetris, FixedCells, Part and TransformSimulator. Other very important class is Max25405EvkitController which handles UART communication with the evaluation kit.

Gesture Sensor Performance

At the video you have seen that it is possible to play Tetris with gesture sensor. Gesture detection worked in most cases very well. I noticed three types of errors:

Gesture not detected: This happened several times when playing. I did some gesture in the front of sensor, but algorithm did not determine the gesture.
Reverse swipe gesture glitch: This happened about one or two times when playing. When I was moving hand from one side to other (for example from left to right) sensor for a short period time reported gesture in reverse direction. This resulted to moving block in a opposite direction than I want or move it one position to side and then immediately back. This issue happened occasionally but sometimes it happened for good reason. Before and after issuing gesture, you need to move hand to the start point. This movement is of course also sensed by the sensor. For example, when you need to move block to right several time, after every swipe gesture (for example from left to tight) you need to move your hand back (right to left) but wait. This movement is also swipe gesture. You need to move your hand back outside of field view, but it must be outside “enough”. After realising this aspect, I imagined that in many situations you will need to reduce sensor range. It sounds curios at the first look, because usually we want to maximize performance rather reducing it but, in this situation, it makes sense. Range resulting from default settings of MAX25405 used by EVKIT firmware is optimal for this use (and most other) cases, I think.
Wrong gesture detected: Very rarely sensor detected completely different gesture then I expected. I remember moment when rotation gesture was detected as a swipe up. But this happened very occasionally. About once per game, I think.
“Long” State Persistence: For detecting gesture firmware needs to hold some state information and firmware have to detect when the gesture starts and ends. When the game get into stressful situation and I tried to execute several consecutive swipes, none of them was detected because firmware most probably noticed all three gestures as a single gesture and detection against these combined data of course failed.

In overall I am satisfied with sensor performance. Issues mentioned above occurred only occasionally. As you have seen user experience from playing Tetris is very good. If you have the Kit, you can try it yourself.

Summary and next steps

This experiment shown that sensor with only 60 pixels can detect gestures with very high accuracy and this concept work. Technically most of the code is responsible for Tetris stuff (like movement, timing, checking collisions, rotations, boundaries and so on) and in fact sensor part was very simple because in this first project I used full EVKIT including MAX32620 microcontroller with stock firmware developed by Maxim. This provided firmware was responsible for the most advanced part of the solution – gesture detection. It is very simple but very powerful way for developing with MAX25405 Gesture Sensor.

If you remember my first blog, I promised that after I complete this project, I will try developing custom firmware using library provided by Maxim. But I had to redesign the idea because deeper look to the Firmware Framework which is available at Maxim website shown that this project is incomplete, it is not finished and implemented parts are affected by bugs like but which I was debugging (see my Blog #7 for more details). According to all these I decided to do not use it. Instead, I will develop my own library for reading and writing MAX25405 registers and then I will try to implement some basic gesture detection algorithm. After I develop both library and simple algorithm, I will try to implement second game. I think, it would be pacman. Because I will run it on the device, I will use different technique for showing the game. I will not use UART and WebSerial anymore and instead I will use physical LCD display connected to the microcontroller. In fact, I already implemented one Pacman using LCD display as part of my Bridgetek BT817 RoadTest review. I will reuse application logic from this project, but I still have lot of things to reimplement. I will switch the platform. As part of previous RoadTest Review I used STM32 but because this contest is sponsored by Maxim Integrated, I will use their MAX78000 microcontroller. In fact, I already have completed first experiments with this setup, but I am delayed with writing blogs because it takes lot of time to write blog like this.

This is end of this part. Thank you very much for spending time by reading this blog post. If you have any comment or question about my project or MAX25405 Gesture Sensor, then feel free to ask in comment below. I welcome every feedback. Have a nice day and stay tuned for next blog posts.

Next blog: Blog #10: C Library for Low-Level MAX25405 Control