U-Net-based RGB and LiDAR image fusion for road segmentation

• Published in: Signal, image and video processing Vol. 17; no. 6; pp. 2837 - 2843
• Main Authors: Candan, Arda Taha, Kalkan, Habil
• Format: Journal Article
• Language: English
• Published: London Springer London 01-09-2023; Springer Nature B.V
• Subjects: Altitude; Cameras; Computer Imaging; Computer Science; Computer vision; Image Processing and Computer Vision; Image segmentation; Lidar; Multimedia Information Systems; Original Paper; Pattern Recognition and Graphics; Signal,Image and Speech Processing; Vision; Deep learning; Sensor fusion; Computer vision; Autonomous driving
• ISSN: 1863-1703; 1863-1711
• DOI: 10.1007/s11760-023-02502-5

Drivable road detection is a fundamental problem for autonomous vehicles. RGB cameras and LiDAR are the mostly used data sources in road detection. While cameras provide lots of useful visual information, LiDARs can provide precise altitude information without being affected by the ambient light. However, these sensors create images at different space and this causes a challenging fusion task when they are intended to be used together. In this study, a U-Net-based novel fusion set is developed to fuse the RGB and LiDAR images for road detection. The LiDAR images are pre-processed and transferred to the 2D image space before fusion. Then, U-NET model, which is effectively used in image segmentation applications, is adapted for three different fusion techniques: early fusion, late fusion and cross-fusion. Models are evaluated on the KITTI road detection dataset, and the developed early fusion model which fuses the RGB and altitude difference image achieved the highest MaxF score on road detection. The obtained results are also at a competitive level with state-of-the-art models.