Synthetic-to-Real Domain Adaptation Joint Spatial Feature Transform for Stereo Matching

• Published in: IEEE signal processing letters Vol. 29; pp. 60 - 64
• Main Authors: Li, Xing, Fan, Yangyu, Rao, Zhibo, Lv, Guoyun, Liu, Shiya
• Format: Journal Article
• Language: English
• Published: New York IEEE 2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptation; Datasets; Domain adaptation; Domains; edge cues; Fuses; Generators; Image edge detection; Image reconstruction; Matching; Spatial data; spatial feature transform; stereo matching; Task analysis; Training; Transforms
• ISSN: 1070-9908; 1558-2361
• DOI: 10.1109/LSP.2021.3125264

Most deep learning-based state-of-the-art stereo matching methods significantly depend on large-scale datasets. However, it is implausible to collect sufficient real-world samples with dense and clear ground-truth disparity maps in practice. Although synthetic datasets' appearance has alleviated the demand for extensive real data, there is a domain shift between synthetic and real sets. To tackle this problem, we propose an individually trained synthetic-to-real domain adaptation (SDA) network that maps synthetic images into the real domain. Specifically, our approach translates the data style from synthetic domain to real domain while maintaining the content and the spatial information. First, edge cues are leveraged to guide domain adaptation in preserving the spatial consistency between input and the generated image. Second, we combine the spatial feature transform (SFT) layer to effectively fuse features from the edge map and the source image. Extensive experiments demonstrate that: 1) when only trained on synthetic data and generalized to real data, our model evidently outperforms many state-of-the-art domain adaptation methods; 2) our translated synthetic datasets (TSD) help to improve the generalization capability of any stereo matching CNNs. Codes and data will be available at https://github.com/Archaic-Atom/SDA_network .