Gesture Mouse Experiment #3: Polling the Gesture Sensor EV Kit using the default Serial API

29 Oct 2022

Introduction

Maxim Integrated has an application note available online, which describes the serial Application Programming Interface (API) for the Max25405 Gesture Sensor EV kit.

Even though I have read through the app note a couple of times, it never quite dawned on me what the following statement about polling really meant:

Note that this command can be sent asynchronously with respect to the sample rate of the sensor, at a rate determined by the host application. But to avoid perceived lag in the gesture response, it should be sent as close to the sensor frame rate as possible, or at minimum 10 times per second. (Note that there is no benefit to sending this command faster than the sample rate of the sensor).

So it was time for another experiment to help me understand behaviour (or firmware logic), and there's no better way to learn by doing than through a mini project.

Arduino UNO connected to MAX32620FTHR / MAX25405 Serial API via USB Host Shield

I wanted to move away from using my Linux laptop to communicate with the MAX32620FTHR board, but as I was still using the original firmware I had to come up with an alternative. I could of course have readily used a Raspberry Pi, but this seemed too close to my Linux laptop.

So I decided to use a USB Host Shield and a humble Arduino UNO.

Thankfully there’s a great library available for the MAX3421 USB Host Shield, called USB Host Shield Library 2.0, which has plenty of useful examples to start from.

The example that will work with the MAX32620FTHR board without modification, is called “acm_terminal”.

This established my serial link via USB cable allowing me to send a command to the MAX32620FTHR and to then receive the response sent back.

I started by sending a “ping” command to the MAX32620FTHR, which then confirms if you have a valid communication link by sending an “ack” response.

I then decided to send a “ver” command to check that the Arduino UNO’s RX buffer works properly as you get a nice long text string containing the firmware version information.

This prompted a few code mods in the example to improve performance.

Finally, I send the “poll” command to retrieve the gesture results. I then repeated the poll request at a set interval to monitor the gesture sensor firmware behaviour.

Even though I only focused on the first three data fields (gesture result, gesture state, N samples), the firmware behaviour was still surprising.

Here are the key observations:

The last completed gesture response detected latches until the next poll request. This is great if you just want to capture a gesture event type. You can also send a gesture mask command to only capture a specific gesture type and ignore the rest. However, if gesture is still seen as busy it wait until next poll to reveal the result.
The gesture state has two states, one of which only changes for Clockwise (CW) and Counterclockwise (CCW) gestures in order to confirm it is ongoing.
The number of samples taken does not clear until a new gesture is seen. I was not sure of the purpose for this type of logic.
I got the click gesture to work.

So, admittedly, I hadn’t fully appreciated just how simple it really is to get a gesture result from the default Serial API firmware.

Clearly I was expecting more complexity but then again I was being influenced by the somewhat unfamiliar, if not a little messy, firmware framework code.

Demonstration

To demonstrate how it works, I decided to add in a few bells and whistles and included a TFT display to showcase a range of gestures, including the click gesture.

First, here is a video with a polling request every 5 seconds (notice the lag and apologies for the length of video - it is in real time after all):

And second, here is a video with a polling request every 500 milliseconds (notice the click gestures and the CW/CCW rotations - note that rotation counting is based on polling update rates, so not accurate):

And finally here is my Arduino code for the Arduino UNO:

/*
 Copyright (c) 2022 C Gerrish (https://github.com/Gerriko)
 This example sends commands and reads the response from the MAX32620FTHR Serial API for
 the MAX25405 Gesture Sensor

 It uses the USB Host Shield Library 2.0 utilising the CDC ASM functionality.
 It also uses SSD1283A library:
// modified by Jean-Marc Zingg to be an example for the SSD1283A library (from GxTFT library)
// original source taken from https://github.com/Bodmer/TFT_HX8357

 Permission is hereby granted, free of charge, to any person obtaining a
 copy of this software and associated documentation files (the "Software"),
 to deal in the Software without restriction, including without limitation
 the rights to use, copy, modify, merge, publish, distribute, sublicense,
 and/or sell copies of the Software, and to permit persons to whom the
 Software is furnished to do so, subject to the following conditions:

 The above copyright notice and this permission notice shall be included in
 all copies or substantial portions of the Software.

 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
 DEALINGS IN THE SOFTWARE.
 */
 
#include <cdcacm.h>
#include <usbhub.h>

#include <SPI.h>
#include <LCDWIKI_GUI.h>        //Core graphics library
#include <SSD1283A.h>           //Hardware-specific library

#define TFTSPI_CSPIN      6
#define TFTSPI_LEDPIN     5
#define TFTSPI_DCPIN      4
#define TFTSPI_RSTPIN     3
#define BUTTONPIN         2

#define RELAYPIN          14

#define BLACK         0x0000
#define BLUE          0x001F
#define RED           0xF800
#define GREEN         0x07E0
#define CYAN          0x07FF
#define MAGENTA       0xF81F
#define YELLOW        0xFFE0
#define WHITE         0xFFFF

const char PING_CMD[] = {'p','i','n','g','\r','\n'};
const char VER_CMD[] = {'v','e','r','\r','\n'};
const char POLL_CMD[] = {'p','o','l','l','\r','\n'};

const uint32_t POLLING_INTERVAL = 500;
const uint8_t RAD_OFFSET = 36;


class ACMAsyncOper : public CDCAsyncOper
{
public:
    uint8_t OnInit(ACM *pacm);
};

uint8_t ACMAsyncOper::OnInit(ACM *pacm)
{
    uint8_t rcode;
    // Set DTR = 1 RTS=1
    rcode = pacm->SetControlLineState(3);

    if (rcode)
    {
        ErrorMessage<uint8_t>(PSTR("SetControlLineState"), rcode);
        return rcode;
    }

    LINE_CODING	lc;
    lc.dwDTERate	= 115200;
    lc.bCharFormat	= 0;
    lc.bParityType	= 0;
    lc.bDataBits	= 8;

    rcode = pacm->SetLineCoding(&lc);

    if (rcode)
        ErrorMessage<uint8_t>(PSTR("SetLineCoding"), rcode);

    return rcode;
}

USB           Usb;
ACMAsyncOper  AsyncOper;
ACM           GestureUSB(&Usb, &AsyncOper);

SSD1283A_GUI  tft(/*CS=*/ TFTSPI_CSPIN, /*DC=*/ TFTSPI_DCPIN, /*RST=*/ TFTSPI_RSTPIN, /*LED=*/ TFTSPI_LEDPIN); //hardware spi,cs,cd,reset,led


bool GestureSent = false;
String CmdString = "";

int numx = 0;
int numy = 0;

void setup()
{
  CmdString.reserve(164);
  Serial.begin( 115200 );
  
#if !defined(__MIPSEL__)
  while (!Serial); // Wait for serial port to connect - used on Leonardo, Teensy and other boards with built-in USB CDC serial connection
#endif
  
  Serial.println(F("\r\nStart USB Host Gesture Controller"));

  if (Usb.Init() == -1) {
      Serial.println(F("OSCOKIRQ failed to assert"));
      while(1) yield();
  }

  tft.init();
  tft.fillScreen(BLACK);
  tft.setCursor(6, 10);
  tft.setTextColor(WHITE);  
  tft.setTextSize(1);
  tft.println(F("Arduino UNO:"));
  tft.println(F(" USB Host Gesture\r\n Controller"));
  tft.println(F(" version 1.0.0")) ;
  tft.setTextSize(2);

  numx = (tft.width()/2)-5;
  numy = (tft.height()/2)-5;

  delay( 200 );
}

void loop()
{
  Usb.Task();
  
  if( GestureUSB.isReady()) {
    uint8_t rcode;
    
    if (!GestureSent) {
      Serial.println(F("Send ping command"));
      /* sending to the MAX25x05 Gesture Sensor */
      rcode = GestureUSB.SndData(6, (uint8_t*)&PING_CMD);
      if (rcode) ErrorMessage<uint8_t>(PSTR("SndData"), rcode);
      else GestureSent = true;
      delay(5);    // short delay to wait for response

      if (!checkAck()) GestureSent = false;     // try again
    }
    else {
      Serial.println(F("Get Gesture Sensor API version"));
      rcode = GestureUSB.SndData(5, (uint8_t*)&VER_CMD);
      if (rcode) ErrorMessage<uint8_t>(PSTR("SndData"), rcode);
      delay(10);    // short delay to wait for response

      if (checkVer()) {
        // Now version known we can use poll command
        uint32_t t_poll = 0;

        // Now loop continuously
        while (1) {
          if (!t_poll) t_poll = millis();
          else {
            if ((millis() - t_poll) > POLLING_INTERVAL) {
              t_poll = millis();
              Serial.println(F("Polling..."));
              /* sending to the MAX25x05 Gesture Sensor */
              rcode = GestureUSB.SndData(6, (uint8_t*)&POLL_CMD);
              if (rcode) ErrorMessage<uint8_t>(PSTR("SndData"), rcode);
              else GestureSent = true;
              delay(5);    // short delay to wait for response
        
              if (!checkPoll()) GestureSent = false;     // try again

            }
          }
        }
        
      }
    }
  }
}

// Check if an ack is received
bool checkAck() {
  uint32_t t_out = millis();
  uint8_t rcode;
  uint8_t  buf[64];
  uint16_t rcvd = 0;
  while (1) {
    if ((millis() - t_out) > 5000) return false;      // if no response after 5 seconds return false
    if( GestureUSB.isReady()) {
      /* reading the gesture shield */
      /* buffer size must be greater or equal to max.packet size */
      /* it it set to 64 (largest possible max.packet size) here, can be tuned down
      for particular endpoint */
      memset(buf, '\0', 64);
      rcvd = 64;
      rcode = GestureUSB.RcvData(&rcvd, buf);
      if (rcode && rcode != hrNAK) ErrorMessage<uint8_t>(PSTR("Ret"), rcode);
      
      if( rcvd ) {
        // Check contents
        for(uint16_t i=0; i < rcvd; i++ ) {
          Serial.print((char)buf[i]); //printing on the screen
          if (buf[i] == '\n') {
            // Check response received
            if (CmdString.startsWith("ack")) {
              Serial.println(F("Gesture Sensor found"));
              return true;
            }
            CmdString = "";
          }
          else {
            if (buf[i] != '\r') CmdString += (char)buf[i];
          }
          
        }
      }
    }
  }
  
  return true;
}

// Check if an version response is received
bool checkVer() {
  uint32_t t_out = millis();
  uint8_t rcode;
  uint8_t  buf[64];
  uint16_t rcvd = 0;
  while (1) {
    if ((millis() - t_out) > 3000) return false;      // if no response after 3 seconds return false
    if( GestureUSB.isReady()) {
      /* reading the gesture shield */
      /* buffer size must be greater or equal to max.packet size */
      /* it it set to 64 (largest possible max.packet size) here, can be tuned down
      for particular endpoint */
      memset(buf, '\0', 64);
      rcvd = 64;
      rcode = GestureUSB.RcvData(&rcvd, buf);
      if (rcode && rcode != hrNAK) ErrorMessage<uint8_t>(PSTR("Ret"), rcode);
      
      if( rcvd ) {
        // Check contents
        for(uint16_t i=0; i < rcvd; i++ ) {
          Serial.print((char)buf[i]); //printing on the screen
          if (buf[i] == '\n') {
            CmdString = "";
            Serial.println("");
            delay(10);
            return true;         
          }
        }
      }
    }
  }
  
  return true;
}

// Check if an poll response is received
bool checkPoll() {
  uint32_t t_out = millis();
  uint8_t rcode;
  uint8_t  buf[64];
  uint16_t rcvd = 0;
  while (1) {
    if ((millis() - t_out) > 3000) return false;      // if no response after 3 seconds return false
    if( GestureUSB.isReady()) {
      /* reading the gesture shield */
      /* buffer size must be greater or equal to max.packet size */
      /* it it set to 64 (largest possible max.packet size) here, can be tuned down
      for particular endpoint */
      memset(buf, '\0', 64);
      rcvd = 64;
      rcode = GestureUSB.RcvData(&rcvd, buf);
      if (rcode && rcode != hrNAK) ErrorMessage<uint8_t>(PSTR("Ret"), rcode);
      
      if( rcvd ) {
        // Check contents
        for(uint16_t i=0; i < rcvd; i++ ) {
          //Serial.print((char)buf[i]); //printing on the screen
          if (buf[i] == '\n') {
            // Check polling response received
            // The first csv data field provides gesture response
            CmdString.trim();
            if (CmdString.length() > 10) {
              uint8_t Pos1 = CmdString.indexOf(",");
              if (Pos1 <=0) {
                CmdString = "";
                return true;
              }
              uint8_t Pos2 = CmdString.indexOf(",", Pos1+1);
              if (Pos2 <=0) {
                CmdString = "";
                return true;
              }
              uint8_t Pos3 = CmdString.indexOf(",", Pos2+1);
              if (Pos3 <=0) {
                CmdString = "";
                return true;
              }
              // Get Gesture Result
              int GesResult = CmdString.substring(0, Pos1).toInt();
              // Get Gesture State
              int GesState = CmdString.substring(Pos1+1, Pos2).toInt();
              // Get Gesture Samples
              int GesSamples = CmdString.substring(Pos2+1, Pos3).toInt();
              
              if (GesState) {
                Serial.println("Gesture in Progress: " + String(GesState, DEC));
                UpdateTFTdisplay(GesResult, true);
                switch (GesResult) {
                   case 2: Serial.println("Gesture ROTATE_CW");
                    break;
                  case 3: Serial.println("Gesture ROTATE_CCW");
                    break;
                }
                Serial.println("Samples taken: " + String(GesSamples, DEC));               
              }
              else {
                UpdateTFTdisplay(GesResult, false);
                switch (GesResult) {
                  case 1: Serial.println("Gesture CLICK");
                    break;
                  case 2: Serial.println("Gesture ROTATE_CW");
                    break;
                  case 3: Serial.println("Gesture ROTATE_CCW");
                    break;
                  case 4: Serial.println("Gesture SWIPE_LEFT");
                    break;
                  case 5: Serial.println("Gesture SWIPE_RIGHT");
                    break;
                  case 6: Serial.println("Gesture SWIPE_UP");
                    break;
                  case 7: Serial.println("Gesture SWIPE_DOWN");
                    break;
                  case 10: Serial.println("Gesture LINGER_ON_REGION");
                    break;
                  case 8:
                  case 9:
                  case 11: Serial.println("RESERVED");
                    break;
                  case 12: Serial.println("Gesture ERROR");
                    break;                
                }
              }
            }
            //Serial.print("Str: "); Serial.println(CmdString);
            CmdString = "";
            return true;
          }
          else {
            if (buf[i] != '\r') CmdString += (char)buf[i];
          }
          
        }
      }
    }
  }
  
  return true;
}

void showCntrVal(bool CW) {
  static int cntr = 0;

  tft.setCursor(numx, numy);
  tft.setTextColor(BLACK);
  if (CW) {
    cntr++;
    if (cntr == 1) tft.println("9");
    else tft.println(String(cntr-1, DEC));
  }
  else {
    cntr--;
    if (cntr == 9) tft.println("1");
    else tft.println(String(cntr+1, DEC));
  }
  if (CW && cntr > 9) cntr = 1;
  else if (!CW && cntr < 1) cntr = 9;   
  tft.setCursor(numx, numy);
  tft.setTextColor(YELLOW);
  tft.println(String(cntr, DEC));
  
}

void UpdateTFTdisplay(int num, bool showCntr) {

  tft.fillScreen(BLACK);
  switch(num) {
    case 1: clickRects(RED, WHITE);
      break;
    case 2: if (showCntr) showCntrVal(true);
      drawCircle(true);
      break;
    case 3: if (showCntr) showCntrVal(false);
      drawCircle(false);
      break;
    case 4: drawRectangles(false);
      break;
    case 5: drawRectangles(true);
      break;
    case 6: displayText("UP");
      break;
    case 7: displayText("DOWN");
      break;
      
  }
}

void displayText(String displTxt) {
  tft.setTextSize(3);
  if (displTxt.length() < 4) tft.setCursor(40, (tft.height()/2)-10);
  else  tft.setCursor(20, (tft.height()/2)-10);
  tft.setTextColor(GREEN);
  tft.println(displTxt);
  tft.setTextSize(2);
  
}

void drawCircle(bool CW) {
  OutlineCircle(0, WHITE);
  for (uint8_t i = 0; i<8; i++) {
    FilledCircles(5, BLUE);
    if (i) {
      if (CW) MarkedCircle(5, i);
      else MarkedCircle(5, 8-i);
    }
    else MarkedCircle(5, i);
    yield();
    delay(100);
  }
  
}

void drawRectangles(bool LR) {
  
  for (uint8_t i = 0; i<8; i++) {
    
    if (i) {
      if (LR) MarkedRect(10, 20, i);
      else MarkedRect(10, 20, 8-i);
    }
    else FilledRects(10, 20, BLUE);
    yield();
    delay(100);
  }
}

void OutlineCircle(uint8_t radius, uint16_t color) {

  const int x = tft.width()/2, y = tft.height()/2;

  if (radius == 0 || radius > x) radius = x-RAD_OFFSET;
  tft.drawCircle(x, y, radius, color);
  
}

void FilledCircles(uint8_t radius, uint16_t color) {
  const float FOURFIVE_RADS = 0.785398;
  const int COORD45 = int((float(tft.width()-(RAD_OFFSET*2))/2.0)*sin(0.785398));

  tft.fillCircle(tft.width()/2, RAD_OFFSET, radius, color);
  tft.fillCircle(tft.width()/2, tft.height()-RAD_OFFSET, radius, color);
  tft.fillCircle(RAD_OFFSET, tft.height()/2, radius, color);
  tft.fillCircle(tft.width()-RAD_OFFSET, tft.height()/2, radius, color);
  tft.fillCircle(tft.width()/2+COORD45, tft.height()/2+COORD45, radius, color);
  tft.fillCircle(tft.width()/2-COORD45, tft.height()/2-COORD45, radius, color);
  tft.fillCircle(tft.width()/2+COORD45, tft.height()/2-COORD45, radius, color);
  tft.fillCircle(tft.width()/2-COORD45, tft.height()/2+COORD45, radius, color);
}

void MarkedCircle(uint8_t radius, uint8_t num) {
  const float FOURFIVE_RADS = 0.785398;
  const int COORD45 = int((float(tft.width()-(RAD_OFFSET*2))/2.0)*sin(0.785398));
  const uint16_t MARKCOLOR = WHITE;

  switch (num) {
    case 0: tft.fillCircle(tft.width()/2, RAD_OFFSET, radius, MARKCOLOR);
      break;
    case 1: tft.fillCircle(tft.width()/2+COORD45, tft.height()/2-COORD45, radius, MARKCOLOR);
      break;
    case 2: tft.fillCircle(tft.width()-RAD_OFFSET, tft.height()/2, radius, MARKCOLOR);
      break;
    case 3: tft.fillCircle(tft.width()/2+COORD45, tft.height()/2+COORD45, radius, MARKCOLOR);
      break;
    case 4: tft.fillCircle(tft.width()/2, tft.height()-RAD_OFFSET, radius, MARKCOLOR);
      break;
    case 5: tft.fillCircle(tft.width()/2-COORD45, tft.height()/2+COORD45, radius, MARKCOLOR);
      break;
    case 6: tft.fillCircle(RAD_OFFSET, tft.height()/2, radius, MARKCOLOR);
      break;
    case 7: tft.fillCircle(tft.width()/2-COORD45, tft.height()/2-COORD45, radius, MARKCOLOR);
      break;
  }  
}


void FilledRects(uint8_t w, uint8_t h, uint16_t color) {

    tft.fillRect(10, tft.height()/2-10, w, h, color);
    tft.fillRect(30, tft.height()/2-10, w, h, color);
    tft.fillRect(50, tft.height()/2-10, w, h, color);
    tft.fillRect(70, tft.height()/2-10, w, h, color);
    tft.fillRect(90, tft.height()/2-10, w, h, color);
    tft.fillRect(110, tft.height()/2-10, w, h, color);
    tft.fillRect(130, tft.height()/2-10, w, h, color);
}

void MarkedRect(uint8_t w, uint8_t h, uint8_t num) {
  const uint16_t MARKCOLOR = WHITE;

  switch(num) {
    case 1: tft.fillRect(10, tft.height()/2-10, w, h, MARKCOLOR);
      break;
    case 2: tft.fillRect(30, tft.height()/2-10, w, h, MARKCOLOR);
      break;
    case 3: tft.fillRect(50, tft.height()/2-10, w, h, MARKCOLOR);
      break;
    case 4: tft.fillRect(70, tft.height()/2-10, w, h, MARKCOLOR);
      break;
    case 5: tft.fillRect(90, tft.height()/2-10, w, h, MARKCOLOR);
      break;
    case 6: tft.fillRect(110, tft.height()/2-10, w, h, MARKCOLOR);
      break;
    case 7: tft.fillRect(130, tft.height()/2-10, w, h, MARKCOLOR);
      break;
  }
}

void clickRects(uint16_t color, uint16_t colorTxt) {
  unsigned long start;
  int           n, i, i2,
                cx = tft.width()  / 2,
                cy = tft.height() / 2;

  n     = min(tft.width(), tft.height());
  start = micros();
  for (i = 2; i < n; i += 6) {
    i2 = i / 2;
    tft.drawRect(cx - i2, cy - i2, i, i, color);
  }
  tft.setTextSize(3);
  tft.setCursor(18, (tft.height()/2)-10);
  tft.setTextColor(colorTxt);
  tft.println("CLICK");
  tft.setTextSize(2);

}

Top Comments

rsjawale24 over 2 years ago in reply to BigG +1

Yes I remember now. I have already flashed my MCU without backup of the stock firmware so that's not an option for me anymore. I still want to make the firmware framework work for the UART Serial communication…

rsjawale24 over 2 years ago in reply to BigG

Yes I remember now. I have already flashed my MCU without backup of the stock firmware so that's not an option for me anymore.

I still want to make the firmware framework work for the UART Serial communication. Because once UART is implemented it can pretty much be used with any MCU.
- Cancel
- Vote Up +1 Vote Down
- Sign in to reply
- More
- Cancel
BigG over 2 years ago in reply to DAB

Thanks DAB

Yep, even though Arduino is an old work horse it still has a few tricks up its sleeve.
- Cancel
- Vote Up 0 Vote Down
- Sign in to reply
- More
- Cancel
BigG over 2 years ago in reply to rsjawale24

Thanks. Yes, I had mentioned this USB host option in your previous forum post. At the time I hadn't understood polling so didn't consider using testing, but there you go. So if you have a USB Host shield, it is certainly an option for you and should make things a little bit easier.
- Cancel
- Vote Up 0 Vote Down
- Sign in to reply
- More
- Cancel
rsjawale24 over 2 years ago

Great!

So all this while there was an option of using the USB host shield with Arduino. I didn't think of this at all! I was working with UART only.
- Cancel
- Vote Up 0 Vote Down
- Sign in to reply
- More
- Cancel
dougw over 2 years ago

Using a USB host on an UNO is a great idea.
- Cancel
- Vote Up 0 Vote Down
- Sign in to reply
- More
- Cancel