Transformer Meets GAN: Cloud-Free Multispectral Image Reconstruction via Multisensor Data Fusion in Satellite Images

• Published in: IEEE transactions on geoscience and remote sensing Vol. 61; pp. 1 - 13
• Main Authors: Li, Congyu, Liu, Xinxin, Li, Shutao
• Format: Journal Article
• Language: English
• Published: New York IEEE 2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptive optics; Cloud computing; Cloud cover; Cloud removal; Clouds; Data integration; generative adversarial network (GAN); Generative adversarial networks; image fusion; Image processing; Image quality; Image reconstruction; Multisensor fusion; optical image; Optical imaging; Optical sensors; Radar imaging; Remote sensing; Removal; Restoration; SAR (radar); Satellite imagery; Satellites; Synthetic aperture radar; synthetic aperture radar (SAR) image; transformer; Transformers
• ISSN: 0196-2892; 1558-0644
• DOI: 10.1109/TGRS.2023.3326545

Cloud-free image reconstruction is of great significance for improving the quality of optical satellite images that are vulnerable to bad weather. When cloud cover makes it impossible to obtain information under the cloud, auxiliary data are indispensable to guide the reconstruction of the cloud-contaminated area. In addition, the areas that require continuous observation are mostly regions with complex features, which puts higher demands on the restoration of texture, color, and other details in data reconstruction. In this article, we propose a transformer-based generative adversarial network for cloud-free multispectral image reconstruction (TransGAN-CFR) via multisensor data fusion in satellite images. Synthetic aperture radar (SAR) images that are not affected by clouds are used as auxiliary data and paired with cloudy optical images into the generative adversarial network (GAN) generator. To take advantage of the deep-shallow features and global-local geographical proximity in remote sensing images, the proposed generator uses a hierarchical encoder-decoder structure, in which the transformer blocks adopt a nonoverlapping window multihead self-attention (WMSA) mechanism and a modified feedforward network (FFN) through depthwise convolutions and the gating mechanism. Besides, we introduce a triplet loss function specifically designed for cloud removal tasks to provide the generated cloud-less image with greater proximity to the ground truth. Compared with seven state-of-the-art deep-learning-based cloud removal models, our network can yield more natural cloud-free images with better visual performance and more accurate results in quantitative evaluation on the SEN12MS-CR dataset.