Telerehabilitation System Based on OpenPose and 3D Reconstruction with Monocular Camera

• Published in: Journal of robotics and mechatronics Vol. 35; no. 3; pp. 586 - 600
• Main Authors: Osawa, Keisuke, You, Yu, Sun, Yi, Wang, Tai-Qi, Zhang, Shun, Shimodozono, Megumi, Tanaka, Eiichiro
• Format: Journal Article
• Language: English
• Published: Tokyo Fuji Technology Press Co. Ltd 01-06-2023
• Subjects: Body parts; Cameras; Image reconstruction; Motion capture; Physicians; Populations; Position errors; Real time; Rehabilitation; Training; Virtual humans
• ISSN: 0915-3942; 1883-8049
• DOI: 10.20965/jrm.2023.p0586

Owing to aging populations, the number of elderly people with limb dysfunction affecting their daily lives will continue to increase. These populations have a great need for rehabilitation training to restore limb functions. However, the current numbers of rehabilitation hospitals and doctors are limited. Moreover, people often cannot go to a hospital owing to external conditions (e.g., the impacts of COVID-19). Thus, an urgent need exists for telerehabilitation system for allowing patients to have training at home. The purpose of this study is to develop an easy-to-use system for allowing target users to experience rehabilitation training at home and to remotely receive real-time guidance from doctors. The proposed system only needs a monocular camera to capture 3D motions. First, the 2D key joints of the human body are detected; then, a simple baseline network is used to reconstruct 3D key joints from the 2D key joints. The 2D detection only has an average angle error of 1.7% compared to that of a professional motion capture system. In addition, the 3D reconstruction has a mean per-joint position error of only 67.9 mm compared to the real coordinates. After acquiring the user’s 3D motions, the system synchronizes the 3D motions to a virtual human model in Unity, providing the user with a more intuitive and interactive experience. Generally, many telerehabilitation systems require professional motion capture cameras and wearable equipment, and the training target is a single body part. In contrast, the proposed system is low-cost and easier to use and only requires a monocular camera and computer to achieve real-time and intuitive telerehabilitation (even though the training target is the entire body). Furthermore, the system provides a similarity evaluation of the motions based on the dynamic time warping; this can provide more accurate and direct feedback to users. In addition, a series of evaluation experiments verify the system’s usability, convenience, feasibility, and accuracy, with the ultimate conclusion that the system can be used in practical rehabilitation applications.