Joint-ID: Transformer-based Joint Image Enhancement and Depth Estimation for Underwater Environments

• Published in: IEEE sensors journal Vol. 24; no. 3; p. 1
• Main Authors: Yang, Geonmo, Kang, Gilhwan, Lee, Juhui, Cho, Younggun
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-02-2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Datasets; Decoding; Depth Estimation; Distortion; Estimation; Feature extraction; Image color analysis; Image degradation; Image enhancement; Image quality; Multi-task learning; Neural networks; Oceanography; Pipeline design; Robot learning; Robotics; Synthetic data; Transformer; Transformers; Turbidity; Underwater; Underwater robots; Visual perception
• ISSN: 1530-437X; 1558-1748
• DOI: 10.1109/JSEN.2023.3338730

Underwater imaging is a challenging task due to factors such as scattering, absorption, and turbulence, which degrade image quality and limit visibility. In this paper, we propose a novel approach for enhancing underwater images that leverages the benefits of joint learning for simultaneous image enhancement and depth estimation. We introduce Joint-ID, a transformer-based neural network that can obtain high-perceptual image quality and depth information from raw underwater images. Our approach formulates a multi-modal objective function that addresses invalid depth, lack of sharpness, and image degradation based on color and local texture. We design an end-to-end training pipeline that enables joint restoration and depth estimation in a shared hierarchical feature space. Additionally, we propose a synthetic dataset with various distortions and scene depths for multi-task learning. We evaluate Joint-ID on synthetic and standard datasets, as well as real underwater images with diverse spectra and harsh turbidity, demonstrating its effectiveness for underwater image enhancement and depth estimation. Furthermore, we demonstrate the ability of Joint-ID to perform feature matching and saliency detection for visually guided underwater robots. Our proposed method has the potential to improve the visual perception of underwater environments and benefit applications such as oceanography and underwater robotics. Supplements are available at: https://sites.google.com/view/joint-id/home.