High Dynamic Range Imaging with Context-aware Transformer

• Published in: 2023 International Joint Conference on Neural Networks (IJCNN) pp. 1 - 8
• Main Authors: Zhou, Fangfang, Fu, Zhengming, Zhang, Dan
• Format: Conference Proceeding
• Language: English
• Published: IEEE 18-06-2023
• Subjects: Attention; CNN; Convolutional neural networks; Feature extraction; HDR deghosting; High dynamic range; Image synthesis; Imaging; Neural networks; Transformer; Transformers
• ISSN: 2161-4407
• DOI: 10.1109/IJCNN54540.2023.10191491

Avoiding the introduction of ghosts when synthesising LDR images as high dynamic range (HDR) images is a challenging task. Convolutional neural networks (CNNs) are effective for HDR ghost removal in general, but are challenging to deal with the LDR images if there are large movements or oversaturation/undersaturation. Existing dual-branch methods combining CNN and Transformer omit part of the information from non-reference images, while the features extracted by the CNN-based branch are bound to the kernel size with small receptive field, which are detrimental to the deblurring and the recovery of oversaturated/undersaturated regions. In this paper, we propose a novel hierarchical dual Transformer method for ghost-free HDR (HDT-HDR) images generation, which extracts global features and local features simultaneously. First, we use a CNN-based head with spatial attention mechanisms to extract features from all the LDR images. Second, the LDR features are delivered to the Hierarchical Dual Transformer (HDT). In each Dual Transformer (DT), the global features are extracted by the window-based Transformer, while the local details are extracted using the channel attention mechanism with deformable CNNs. Finally, the ghost free HDR image is obtained by dimensional mapping on the HDT output. Abundant experiments demonstrate that our HDT-HDR achieves the state-of-the-art performance among existing HDR ghost removal methods.