HpEIS: Learning Hand Pose Embeddings for Multimedia Interactive Systems

• Main Authors: Xu, Songpei, Ge, Xuri, Kaul, Chaitanya, Murray-Smith, Roderick
• Format: Journal Article
• Language: English
• Published: 11-10-2024
• Subjects: Computer Science - Computer Vision and Pattern Recognition; Computer Science - Human-Computer Interaction
• DOI: 10.48550/arxiv.2410.08779

We present a novel Hand-pose Embedding Interactive System (HpEIS) as a
virtual sensor, which maps users' flexible hand poses to a two-dimensional
visual space using a Variational Autoencoder (VAE) trained on a variety of hand
poses. HpEIS enables visually interpretable and guidable support for user
explorations in multimedia collections, using only a camera as an external hand
pose acquisition device. We identify general usability issues associated with
system stability and smoothing requirements through pilot experiments with
expert and inexperienced users. We then design stability and smoothing
improvements, including hand-pose data augmentation, an anti-jitter
regularisation term added to loss function, stabilising post-processing for
movement turning points and smoothing post-processing based on One Euro
Filters. In target selection experiments (n=12), we evaluate HpEIS by measures
of task completion time and the final distance to target points, with and
without the gesture guidance window condition. Experimental responses indicate
that HpEIS provides users with a learnable, flexible, stable and smooth mid-air
hand movement interaction experience.