Unsafe Maneuver Classification From Dashcam Video and GPS/IMU Sensors Using Spatio-Temporal Attention Selector

• Published in: IEEE transactions on intelligent transportation systems Vol. 23; no. 9; pp. 15605 - 15615
• Main Authors: Simoncini, Matteo, de Andrade, Douglas Coimbra, Taccari, Leonardo, Salti, Samuele, Kubin, Luca, Schoen, Fabio, Sambo, Francesco
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-09-2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accidents; Algorithms; attention; Classification; Computer networks; dashcam; Data mining; deep learning; Detectors; Feature extraction; Frames (data processing); GPS; IMU; Machine learning; Maneuvers; Object detection; Object recognition; road scene understanding; Roads; Task analysis; Unsafe maneuver classification; XAI
• ISSN: 1524-9050; 1558-0016
• DOI: 10.1109/TITS.2022.3142672

In this paper, we propose a novel deep learning architecture to classify unsafe driving maneuvers from dashcam and IMU data. Such architecture processes the output of an object detection algorithm in combination with raw video frames and GPS/IMU data. At the core of the architecture there is a novel Spatio-Temporal Attention Selector (STAS) module, which (1) extracts features describing the evolution of each object in the scene over time and (2) leverages multi-head dot product attention to select the relevant ones, i.e. , the dangerous ones or the ones in danger, to perform classification. We also introduce a simple but effective methodology to increase the benefit of fine-tuning the backbone network. Our method is shown to achieve higher performance than other approaches in the literature applying attention over single frames.