Class-Aware Sounding Objects Localization via Audiovisual Correspondence

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 44; no. 12; pp. 9844 - 9859
• Main Authors: Hu, Di, Wei, Yake, Qian, Rui, Lin, Weiyao, Song, Ruihua, Wen, Ji-Rong
• Format: Journal Article
• Language: English
• Published: United States IEEE 01-12-2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Annotations; audiovisual correspondence; Class-aware sounding object localization; Correspondence; Dictionaries; distribution alignment; Localization; Location awareness; Object recognition; Representations; Semantics; Sounding; Task analysis; Videos; Visualization
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2021.3137988

Audiovisual scenes are pervasive in our daily life. It is commonplace for humans to discriminatively localize different sounding objects but quite challenging for machines to achieve class-aware sounding objects localization without category annotations, i.e., localizing the sounding object and recognizing its category. To address this problem, we propose a two-stage step-by-step learning framework to localize and recognize sounding objects in complex audiovisual scenarios using only the correspondence between audio and vision. First, we propose to determine the sounding area via coarse-grained audiovisual correspondence in the single source cases. Then visual features in the sounding area are leveraged as candidate object representations to establish a category-representation object dictionary for expressive visual character extraction. We generate class-aware object localization maps in cocktail-party scenarios and use audiovisual correspondence to suppress silent areas by referring to this dictionary. Finally, we employ category-level audiovisual consistency as the supervision to achieve fine-grained audio and sounding object distribution alignment. Experiments on both realistic and synthesized videos show that our model is superior in localizing and recognizing objects as well as filtering out silent ones. We also transfer the learned audiovisual network into the unsupervised object detection task, obtaining reasonable performance.