RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera-Radar Fused Object 3D Localization

• Published in: IEEE journal of selected topics in signal processing Vol. 15; no. 4; pp. 954 - 967
• Main Authors: Wang, Yizhou, Jiang, Zhongyu, Li, Yudong, Hwang, Jenq-Neng, Xing, Guanbin, Liu, Hui
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-06-2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: advanced driver assistance system; Algorithms; Annotations; autonomous driving; Cameras; cross-modal supervision; Datasets; deep CNN; Driving conditions; Field of view; Localization; M-Net; Millimeter waves; Object detection; Object recognition; Perception; Radar; Radar data; Radar detection; Radar imaging; radar object annotation; Radar object detection; Radio frequency; Real time; Robustness (mathematics); Sensors; temporal deformable convolution; temporal inception CNN; Three-dimensional displays
• ISSN: 1932-4553; 1941-0484
• DOI: 10.1109/JSTSP.2021.3058895

Various autonomous or assisted driving strategies have been facilitated through the accurate and reliable perception of the environment around a vehicle. Among the commonly used sensors, radar has usually been considered as a robust and cost-effective solution even in adverse driving scenarios, e.g., weak/strong lighting or bad weather. Instead of considering fusing the unreliable information from all available sensors, perception from pure radar data becomes a valuable alternative that is worth exploring. In this paper, we propose a deep radar object detection network, named RODNet, which is cross-supervised by a camera-radar fused algorithm without laborious annotation efforts, to effectively detect objects from the radio frequency (RF) images in real-time. First, the raw signals captured by millimeter-wave radars are transformed to RF images in range-azimuth coordinates. Second, our proposed RODNet takes a snippet of RF images as the input to predict the likelihood of objects in the radar field of view (FoV). Two customized modules are also added to handle multi-chirp information and object relative motion. The proposed RODNet is cross-supervised by a novel 3D localization of detected objects using a camera-radar fusion (CRF) strategy in the training stage. Due to no existing public dataset available for our task, we create a new dataset, named CRUW,<xref ref-type="fn" rid="fn1"> 1 1
The dataset and code are available at https://www.cruwdataset.org/ .
which contains synchronized RGB and RF image sequences in various driving scenarios. With intensive experiments, our proposed cross-supervised RODNet achieves 86% average precision and 88% average recall of object detection performance, which shows the robustness in various driving conditions.