Automated work efficiency analysis for smart manufacturing using human pose tracking and temporal action localization

• Published in: Journal of visual communication and image representation Vol. 73; p. 102948
• Main Authors: Sun, Hao, Ning, Guanghan, Zhao, Zhiqun, Huang, Zhongchao, He, Zhihai
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01-11-2020
• Subjects: Deep learning; Dynamic time warping; Generative adversarial networks; Human pose estimation; Smart manufacturing; Temporal activity localization; Deep learning; Temporal activity localization; Generative adversarial networks; Human pose estimation; Dynamic time warping; Smart manufacturing
• ISSN: 1047-3203; 1095-9076
• DOI: 10.1016/j.jvcir.2020.102948

In this paper, we aim to develop an automatic system to monitor and evaluate worker’s efficiency for smart manufacturing based on human pose tracking and temporal action localization. First, we explore the generative adversarial networks (GANs) to achieve significantly improved estimation of human body joints. Second, we formulate the automated worker efficiency analysis into a temporal action localization problem in which the action video performed by the worker is matched against a reference video performed by a teacher. We extract invariant spatio-temporal features from the human body pose sequences and perform cross-video matching using dynamic time warping. Our proposed human pose estimation method achieves state-of-the-art performance on the benchmark dataset. Our automated work efficiency analysis is able to achieve action localization with an average IoU (intersection over union) score large than 0.9. This represents one of the first systems to provide automated worker efficiency evaluation.
•Highly desirable to develop an automated system for worker efficiency evaluation.•Generative adversarial learning improves human pose estimation.•Action-level cross-video DTW matching helps to locate worker’s activity.•Establish a method for automated worker efficiency analysis.