Apple’s new gesture customization framework allows users to add their own gestures for wearable sensors. (Image Credit: Xu et. Al)
Apple researchers developed a new framework for users to add their own gestures to wearable sensors without causing other gesture sets to experience performance loss. Overall, this framework improves gesture memorability, interaction efficiency, and accessibility for those with special needs. In order to achieve this, any gesture-compatible system must provide a rapid and minimal data collection process. It should also “go beyond model fine-tuning” since two issues typically arise from this event. For starters, it causes the fine-tuned model to overfit newer classes, affecting the generalizability. At the same time, it forces predefined classes to deteriorate drastically.
The team’s framework supports three to five user examples, providing in-situ feedback while “maintaining recognition performance on the original gesture set.” It also incorporates transfer learning, incremental learning, and few-shot learning techniques. They recruited 500 participants for this study, which involved using accelerometer gyroscope data that trained a convolution neural network (CNN), providing it with predefined gesture set recognition capabilities. Afterward, the researchers trained a lightweight model for custom gestures, which didn’t require old model parameter adjustments.
Their results reveal that the wrist-worn gesture recognition model recognizes four gestures with 95.7% accuracy and a 95.8% F1 score in a cross-user configuration. It also has a 0.6 times per hour false positive rate when tested on daily behavior non-gesture data.
The team also applied the pre-trained model’s first half as a feature embedding extractor and developed a parallel output after the embedded layer. This made it possible to enable new model training without causing pre-trained model performance loss. They then used a series of data augmentation, data synthesis, and adversarial training techniques that extracted “the most utility from user samples” while improving model performance.
For reliability, the team designed the model around an interactive customization experience. In this case, it provides interactive feedback when a user’s gesture input nearly matches the existing set, shows inconsistency compared to predefined samples, cannot be told apart from daily activities, or provides sub-optimal recognition performance. Users can then choose to provide more samples for the system.
People with personalized accessibility conditions or situational impairments can use this new gesture framework. According to the paper, “In situations where the original gesture set can be inappropriate or inaccessible, our framework can support the creation of gestures that best cater to users’ preferences and abilities; our interactive feedback mechanisms ensure that end-users get to decide what level of robustness and accuracy helps them achieve their device usage goals.”
Have a story tip? Message me at: http://twitter.com/Cabe_Atwell