Automatic Assessment of Upper Extremity Function and Mobile Application for Self-Administered Stroke Rehabilitation

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 32; pp. 652 - 661
• Main Authors: Kim, Dong-Wook, Park, Ji Eun, Kim, Min-Jung, Byun, Seung Hwan, Jung, Chung In, Jeong, Ha Mok, Woo, Sang Rok, Lee, Kwon Haeng, Lee, Myoung Hwa, Jung, Jung-Woo, Lee, Dayeon, Ryu, Byung-Ju, Yang, Seung Nam, Baek, Seung Jun
• Format: Journal Article
• Language: English
• Published: United States IEEE 01-01-2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Applications programs; Deep learning; Environmental factors; Extremities; Grasping; Hemiplegia; Home environment; Hospitals; Medical personnel; Mobile applications; Mobile communications networks; Mobile computing; Model accuracy; motion assessment; Optical flow; Optical flow (image analysis); Patients; Rehabilitation; self-administered rehabilitation; Stroke; Stroke (medical condition); Task analysis; Training; upper extremity; Video data; Videos
• ISSN: 1534-4320; 1558-0210
• DOI: 10.1109/TNSRE.2024.3358497

Rehabilitation training is essential for a successful recovery of upper extremity function after stroke. Training programs are typically conducted in hospitals or rehabilitation centers, supervised by specialized medical professionals. However, frequent visits to hospitals can be burdensome for stroke patients with limited mobility. We consider a self-administered rehabilitation system based on a mobile application in which patients can periodically upload videos of themselves performing reach-to-grasp tasks to receive recommendations for self-managed exercises or progress reports. Sensing equipment aside from cameras is typically unavailable in the home environment. A key contribution of our work is to propose a deep learning-based assessment model trained only with video data. As all patients carry out identical tasks, a fine-grained assessment of task execution is required. Our model addresses this difficulty by learning RGB and optical flow data in a complementary manner. The correlation between the RGB and optical flow data is captured by a novel module for modality fusion using cross-attention with Transformers. Experiments showed that our model achieved higher accuracy in movement assessment than existing methods for action recognition. Based on the assessment model, we developed a patient-centered, solution-based mobile application for upper extremity exercises for hemiplegia, which can recommend 57 exercises with three levels of difficulty. A prototype of our application was evaluated by potential end-users and achieved a good quality score on the Mobile Application Rating Scale (MARS).