Spatio-temporal Relation Modeling for Few-shot Action Recognition

• Published in: 2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2022) pp. 19926 - 19935
• Main Authors: Thatipelli, Anirudh, Narayan, Sanath, Khan, Salman, Anwer, Rao Muhammad, Khan, Fahad Shahbaz, Ghanem, Bernard
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-01-2022
• Series: IEEE Conference on Computer Vision and Pattern Recognition
• Subjects: Action and event recognition; Recognition: detection; Aggregates; Benchmark testing; categorization; Codes; Computational modeling; Computer vision; Pattern recognition; Representation learning; retrieval; Transfer/low-shot/long-tail learning
• ISBN: 9781665469470; 1665469463; 1665469471; 9781665469463
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52688.2022.01933

We propose a novel few-shot action recognition framework, STRM, which enhances class-specific feature discrim-inability while simultaneously learning higher-order temporal representations. The focus of our approach is a novel spatio-temporal enrichment module that aggregates spatial and temporal contexts with dedicated local patch-level and global frame-level feature enrichment sub-modules. Local patch-level enrichment captures the appearance-based characteristics of actions. On the other hand, global framelevel enrichment explicitly encodes the broad temporal context, thereby capturing the relevant object features over time. The resulting spatio-temporally enriched representations are then utilized to learn the relational matching between query and support action sub-sequences. We further introduce a query-class similarity classifier on the patch-level enriched features to enhance class-specific feature dis-criminability by reinforcing the feature learning at different stages in the proposed framework. Experiments are performed on four few-shot action recognition benchmarks: Kinetics, SSv2, HMDB51 and UCF101. Our extensive ablation study reveals the benefits of the proposed contributions. Furthermore, our approach sets a new state-of-the-art on all four benchmarks. On the challenging SSv2 benchmark, our approach achieves an absolute gain of 3.5% in classification accuracy, as compared to the best existing method in the literature. Our code and models are available at https://github.com/Anirudh257/strm.