Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen

• Published in: Sensors (Basel, Switzerland) Vol. 22; no. 14; p. 5347
• Main Authors: Wehbi, Mohamad, Luge, Daniel, Hamann, Tim, Barth, Jens, Kaempf, Peter, Zanca, Dario, Eskofier, Bjoern M.
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 18-07-2022; MDPI
• Subjects: Character recognition; convolutional neural network; digital pen; Digitization; Handwriting; Handwriting recognition; inertial measurement unit; Neural networks; Pens; Reconstruction; Segmentation; sensor-based deep learning; Sensors; Styli; trajectory reconstruction
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s22145347

Efficient handwriting trajectory reconstruction (TR) requires specific writing surfaces for detecting movements of digital pens. Although several motion-based solutions have been developed to remove the necessity of writing surfaces, most of them are based on classical sensor fusion methods limited, by sensor error accumulation over time, to tracing only single strokes. In this work, we present an approach to map the movements of an IMU-enhanced digital pen to relative displacement data. Training data is collected by means of a tablet. We propose several pre-processing and data-preparation methods to synchronize data between the pen and the tablet, which are of different sampling rates, and train a convolutional neural network (CNN) to reconstruct multiple strokes without the need of writing segmentation or post-processing correction of the predicted trajectory. The proposed system learns the relative displacement of the pen tip over time from the recorded raw sensor data, achieving a normalized error rate of 0.176 relative to unit-scaled tablet ground truth (GT) trajectory. To test the effectiveness of the approach, we train a neural network for character recognition from the reconstructed trajectories, which achieved a character error rate of 19.51%. Finally, a joint model is implemented that makes use of both the IMU data and the generated trajectories, which outperforms the sensor-only-based recognition approach by 0.75%.