Camera-Radar Fusion for 3-D Depth Reconstruction

• Published in: 2020 IEEE Intelligent Vehicles Symposium (IV) pp. 265 - 271
• Main Authors: Niesen, Urs, Unnikrishnan, Jayakrishnan
• Format: Conference Proceeding
• Language: English
• Published: IEEE 19-10-2020
• Subjects: Cameras; Image reconstruction; Laser radar; Radar; Radar imaging; Radar measurements; Sensors
• ISSN: 2642-7214
• DOI: 10.1109/IV47402.2020.9304559

We introduce and study the problem of camera-radar fusion for 3-D depth reconstruction. This problem is motivated by autonomous driving applications, in which we can expect to have access to both front-facing camera and radar sensors. These two sensors are complementary in several respects: the camera is a passive sensor measuring azimuth and elevation; the radar is an active sensor measuring azimuth and range. Fusing their measurements is therefore beneficial. Our fusion solution uses a modified encoder-decoder deep convolutional neural network. We train and evaluate this network on over 100 000 samples collected in highway environments. Our results demonstrate an improvement in reconstruction accuracy and robustness from fusing the two sensors.