A Preliminary Study of Deep Learning Sensor Fusion for Pedestrian Detection

Most pedestrian detection methods focus on bounding boxes based on fusing RGB with lidar. These methods do not relate to how the human eye perceives objects in the real world. Furthermore, lidar and vision can have difficulty detecting pedestrians in scattered environments, and radar can be used to...

Full description

Saved in:

Bibliographic Details
Published in:	Sensors (Basel, Switzerland) Vol. 23; no. 8; p. 4167
Main Authors:	Plascencia, Alfredo Chávez, García-Gómez, Pablo, Perez, Eduardo Bernal, DeMas-Giménez, Gerard, Casas, Josep R, Royo, Santiago
Format:	Journal Article
Language:	English
Published:	Switzerland MDPI AG 21-04-2023 MDPI
Subjects:	autonomous driving Autonomous vehicles Calibration Cameras Classification Color imagery Computer architecture convolutional neural networks: sensor calibration Datasets Deep Learning Employee motivation Feasibility Humans Kalman filters Matrix methods Methods Multisensor fusion Neural networks Neural Networks, Computer Optical radar Pedestrians Pixels Remote sensing Semantic segmentation Semantics sensor fusion Sensors Vision systems Vision, Ocular convolutional neural networks: sensor calibration sensor fusion autonomous driving
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	Most pedestrian detection methods focus on bounding boxes based on fusing RGB with lidar. These methods do not relate to how the human eye perceives objects in the real world. Furthermore, lidar and vision can have difficulty detecting pedestrians in scattered environments, and radar can be used to overcome this problem. Therefore, the motivation of this work is to explore, as a preliminary step, the feasibility of fusing lidar, radar, and RGB for pedestrian detection that potentially can be used for autonomous driving that uses a fully connected convolutional neural network architecture for multimodal sensors. The core of the network is based on SegNet, a pixel-wise semantic segmentation network. In this context, lidar and radar were incorporated by transforming them from 3D pointclouds into 2D gray images with 16-bit depths, and RGB images were incorporated with three channels. The proposed architecture uses a single SegNet for each sensor reading, and the outputs are then applied to a fully connected neural network to fuse the three modalities of sensors. Afterwards, an up-sampling network is applied to recover the fused data. Additionally, a custom dataset of 60 images was proposed for training the architecture, with an additional 10 for evaluation and 10 for testing, giving a total of 80 images. The experiment results show a training mean pixel accuracy of 99.7% and a training mean intersection over union of 99.5%. Also, the testing mean of the IoU was 94.4%, and the testing pixel accuracy was 96.2%. These metric results have successfully demonstrated the effectiveness of using semantic segmentation for pedestrian detection under the modalities of three sensors. Despite some overfitting in the model during experimentation, it performed well in detecting people in test mode. Therefore, it is worth emphasizing that the focus of this work is to show that this method is feasible to be used, as it works regardless of the size of the dataset. Also, a bigger dataset would be necessary to achieve a more appropiate training. This method gives the advantage of detecting pedestrians as the human eye does, thereby resulting in less ambiguity. Additionally, this work has also proposed an extrinsic calibration matrix method for sensor alignment between radar and lidar based on singular value decomposition.
Bibliography:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23
ISSN:	1424-8220 1424-8220
DOI:	10.3390/s23084167