A wearable system for sign language recognition enabled by a convolutional neural network

• Published in: Nano energy Vol. 116; p. 108767
• Main Authors: Liu, Yuxuan, Jiang, Xijun, Yu, Xingge, Ye, Huaidong, Ma, Chao, Wang, Wanyi, Hu, Youfan
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01-11-2023
• Subjects: Convolutional neural networks; Sign language recognition; Stretchable strain sensor; Wearable device; Sign language recognition; Stretchable strain sensor; Wearable device; Convolutional neural networks
• ISSN: 2211-2855
• DOI: 10.1016/j.nanoen.2023.108767

Sign language recognition is of great significance to connect the hearing/speech impaired and non-sign language communities. Compared to isolated word recognition, sentence recognition is more practical in real-world scenarios, but is also more complicated because continuous, high-quality sign data with distinct features must be collected and isolated signs must be identified with high accuracy. Here, we propose a wearable sign language recognition system enabled by a convolutional neural network (CNN) that integrates stretchable strain sensors and inertial measurement units attached to the body to perceive hand postures and movement trajectories. Forty-eight Chinese sign language words commonly used in daily life were collected and used to train the CNN model, and an isolated sign language word recognition accuracy of 95.85% was achieved. For sentence-level sign language recognition, we proposed a method that combines multiple sliding windows and uses correlation analysis to improve the CNN recognition performance, achieving a correct rate of 84% for 50 sign language sentence samples, showing good extendibility.
[Display omitted]
•A wearable system that attached to the body to perceive hand postures and movement trajectories was demonstrated.•A high accuracy of 95.85% was achieved in isolated sign language word recognition for Chinese sign language.•Combining multiple sliding windows and correlation analysis, a correct rate of 84% for sign language sentence was achieved.