M-Gesture: Person-Independent Real-Time In-Air Gesture Recognition Using Commodity Millimeter Wave Radar

• Published in: IEEE internet of things journal Vol. 9; no. 5; pp. 3397 - 3415
• Main Authors: Liu, Haipeng, Zhou, Anfu, Dong, Zihe, Sun, Yuyang, Zhang, Jiahe, Liu, Liang, Ma, Huadong, Liu, Jianhua, Yang, Ning
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01-03-2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Commodities; Feature extraction; Gesture recognition; Human computer interaction; Internet of Things; millimeter wave (mmWave) sensing; Millimeter waves; Neural networks; Radar tracking; Real time; Real-time systems; Segmentation; Sensors
• ISSN: 2327-4662; 2327-4662
• DOI: 10.1109/JIOT.2021.3098338

Millimeter wave (mmWave) sensing promises to enable contactless and high-precision "in-air" gesture-based human-computer interaction (HCI). While previous works have demonstrated its feasibility, they require tedious gesture collecting for person-independent recognition and they operate in an off-line mode without considering practical issues, such as segmenting gesture and recognition latency. In this work, we propose M-Gesture , a person-independent real-time mmWave gesture recognition solution. We first build a compact gesture model with a custom-designed neural network to distill the unique features underlying each gesture, while suppressing personalized discrepancy across different users without extra collection and retraining. Furthermore, we design a system status transition (SST) to decide when a gesture begins and ends, which enables automatic gesture segmentation and hence real-time recognition. We prototype M-Gesture on a commodity mmWave sensor and demonstrate its advantages using two practical applications: 1) a contactless music player and 2) camera. Extensive experiments and user studies show that M-Gesture has an accuracy of 99% and a short response latency within 25 ms. Moreover, we also collect and release a comprehensive mmWave gesture data set consisting of 54 620 instances from 144 persons, which may have an independent value of facilitating future research.