SiamEFT: adaptive-time feature extraction hybrid network for RGBE multi-domain object tracking

• Published in: Frontiers in neuroscience Vol. 18; p. 1453419
• Main Authors: Liu, Shuqi, Wang, Gang, Song, Yong, Huang, Jinxiang, Huang, Yiqian, Zhou, Ya, Wang, Shiqiang
• Format: Journal Article
• Language: English
• Published: Switzerland Frontiers Media S.A 08-08-2024
• Subjects: hybrid network; neuromorphic computing; Neuroscience; object tracking; RGB and Event; spatio-temporal; spiking neural networks; RGB and Event; hybrid network; spiking neural networks; object tracking; neuromorphic computing; spatio-temporal
• ISSN: 1662-4548; 1662-453X; 1662-453X
• DOI: 10.3389/fnins.2024.1453419

Integrating RGB and Event (RGBE) multi-domain information obtained by high-dynamic-range and temporal-resolution event cameras has been considered an effective scheme for robust object tracking. However, existing RGBE tracking methods have overlooked the unique spatio-temporal features over different domains, leading to object tracking failure and inefficiency, especally for objects against complex backgrounds. To address this problem, we propose a novel tracker based on adaptive-time feature extraction hybrid networks, namely Siamese Event Frame Tracker (SiamEFT), which focuses on the effective representation and utilization of the diverse spatio-temporal features of RGBE. We first design an adaptive-time attention module to aggregate event data into frames based on adaptive-time weights to enhance information representation. Subsequently, the SiamEF module and cross-network fusion module combining artificial neural networks and spiking neural networks hybrid network are designed to effectively extract and fuse the spatio-temporal features of RGBE. Extensive experiments on two RGBE datasets (VisEvent and COESOT) show that the SiamEFT achieves a success rate of 0.456 and 0.574, outperforming the state-of-the-art competing methods and exhibiting a 2.3-fold enhancement in efficiency. These results validate the superior accuracy and efficiency of SiamEFT in diverse and challenging scenes.