Dual-attention Network for View-invariant Action Recognition

• Published in: Complex & intelligent systems Vol. 10; no. 1; pp. 305 - 321
• Main Authors: Kumie, Gedamu Alemu, Habtie, Maregu Assefa, Ayall, Tewodros Alemu, Zhou, Changjun, Liu, Huawen, Seid, Abegaz Mohammed, Erbad, Aiman
• Format: Journal Article
• Language: English
• Published: Cham Springer International Publishing 01-02-2024; Springer Nature B.V; Springer
• Subjects: Attention transfer; Body parts; Complexity; Computational Intelligence; Computer science; Computers; Cross-attention; Data Structures and Information Theory; Design; Dual-attention; Engineering; Human action recognition; Human activity recognition; Human body; Intelligent systems; Invariants; Learning; Modules; Original Article; Representations; Self-attention; Semantics; Surveillance; View-invariant representation; Human action recognition; Cross-attention; Dual-attention; Attention transfer; View-invariant representation; Self-attention
• ISSN: 2199-4536; 2198-6053
• DOI: 10.1007/s40747-023-01171-8

View-invariant action recognition has been widely researched in various applications, such as visual surveillance and human–robot interaction. However, view-invariant human action recognition is challenging due to the action occlusions and information loss caused by view changes. Modeling spatiotemporal dynamics of body joints and minimizing representation discrepancy between different views could be a valuable solution for view-invariant human action recognition. Therefore, we propose a
D
ual-
A
ttention
Net
work (DANet) aims to learn robust video representation for view-invariant action recognition. The DANet is composed of relation-aware spatiotemporal self-attention and spatiotemporal cross-attention modules. The relation-aware spatiotemporal self-attention module learns representative and discriminative action features. This module captures local and global long-range dependencies, as well as pairwise relations among human body parts and joints in the spatial and temporal domains. The cross-attention module learns view-invariant attention maps and generates discriminative features for semantic representations of actions in different views. We exhaustively evaluate our proposed approach on the NTU-60, NTU-120, and UESTC large-scale challenging datasets with multi-type evaluation metrics including Cross-Subject, Cross-View, Cross-Set, and Arbitrary-view. The experimental results demonstrate that our proposed approach significantly outperforms state-of-the-art approaches in view-invariant action recognition.