Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review

• Published in: IEEE transactions on intelligent transportation systems Vol. 23; no. 2; pp. 722 - 739
• Main Authors: Cui, Yaodong, Chen, Ren, Chu, Wenbo, Chen, Long, Tian, Daxin, Li, Ying, Cao, Dongpu
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-02-2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Autonomous vehicles; Autonomy; Camera-LiDAR fusion; Cameras; Convolution; Data integration; Data processing; Deep learning; depth completion; Feature extraction; Geometry; Image segmentation; Laser radar; Lidar; object detection; Object recognition; semantic segmentation; Semantics; sensor fusion; Three-dimensional displays; tracking
• ISSN: 1524-9050; 1558-0016
• DOI: 10.1109/TITS.2020.3023541

Autonomous vehicles were experiencing rapid development in the past few years. However, achieving full autonomy is not a trivial task, due to the nature of the complex and dynamic driving environment. Therefore, autonomous vehicles are equipped with a suite of different sensors to ensure robust, accurate environmental perception. In particular, the camera-LiDAR fusion is becoming an emerging research theme. However, so far there has been no critical review that focuses on deep-learning-based camera-LiDAR fusion methods. To bridge this gap and motivate future research, this article devotes to review recent deep-learning-based data fusion approaches that leverage both image and point cloud. This review gives a brief overview of deep learning on image and point cloud data processing. Followed by in-depth reviews of camera-LiDAR fusion methods in depth completion, object detection, semantic segmentation, tracking and online cross-sensor calibration, which are organized based on their respective fusion levels. Furthermore, we compare these methods on publicly available datasets. Finally, we identified gaps and over-looked challenges between current academic researches and real-world applications. Based on these observations, we provide our insights and point out promising research directions.