Benchmarking Object Detection Robustness against Real-World Corruptions

• Published in: International journal of computer vision Vol. 132; no. 10; pp. 4398 - 4416
• Main Authors: Liu, Jiawei, Wang, Zhijie, Ma, Lei, Fang, Chunrong, Bai, Tongtong, Zhang, Xufan, Liu, Jia, Chen, Zhenyu
• Format: Journal Article
• Language: English
• Published: New York Springer US 01-10-2024; Springer Nature B.V
• Subjects: Artificial Intelligence; Benchmarks; Computer Imaging; Computer Science; Corruption; Damage assessment; Damage detection; Detectors; Effectiveness; Image Processing and Computer Vision; Object recognition; Pattern Recognition; Pattern Recognition and Graphics; Robustness; Safety critical; Sensors; Special Issue on Robust Vision; State-of-the-art reviews; Systematic review; Vision; Computer vision; Data augmentation; Robustness; Object detection
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-024-02096-6

With the rapid recent development, deep learning based object detection techniques have been applied to various real-world software systems, especially in safety-critical applications like autonomous driving. However, few studies are conducted to systematically investigate the robustness of state-of-the-art object detection techniques against real-world image corruptions and yet few benchmarks of object detection methods in terms of robustness are publicly available. To bridge this gap, we initiate to create a public benchmark of COCO-C and BDD100K-C, composed of sixteen real-world corruptions according to the real damages in camera sensors and image pipeline. Based on that, we further perform a systematic empirical study and evaluation of twelve representative object detectors covering three different categories of architectures (
i.e
., two-stage, one-stage, transformer architectures) to identify the current challenges and explore future opportunities. Our key findings include (1) the proposed real-world corruptions pose a threat to object detectors, especially for the corruptions involving colour changes, (2) a detector with a high mAP may still be vulnerable to real-world corruptions, (3) if there are potential cross-scenarios applications, the one-stage detectors are recommended, (4) when object detection architectures suffer from real-world corruptions, the effectiveness of existing robustness enhancement methods is limited, and (5) two-stage and one-stage object detection architectures are more likely to miss detect objects compared with transformer-based methods against the proposed corruptions. Our results highlight the need for designing robust object detection methods against real-world corruption and the need for more effective robustness enhancement methods for existing object detectors.