PanoStyle: Semantic, Geometry-Aware and Shading Independent Photorealistic Style Transfer for Indoor Panoramic Scenes

• Published in: 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) pp. 1545 - 1556
• Main Authors: Tukur, M., Rehman, A. Ur, Pintore, G., Gobbetti, E., Schneider, J., Agus, M.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 02-10-2023
• Subjects: generative adversarial networks; Geometry; indoor panoramas; Measurement; Semantics; Shape; style transfer; Superresolution; Training; Visualization
• ISSN: 2473-9944
• DOI: 10.1109/ICCVW60793.2023.00170

While current style transfer models have achieved impressive results for the application of artistic style to generic images, they face challenges in achieving photorealistic performances on indoor scenes, especially the ones represented by panoramic images. Moreover, existing models overlook the unique characteristics of indoor panoramas, which possess particular geometry and semantic properties. To address these limitations, we propose the first geometry-aware and shading-independent, photorealistic and semantic style transfer method for indoor panoramic scenes. Our approach extends semantic-aware generative adversarial architecture capabilities by introducing two novel strategies to account the geometric characteristics of indoor scenes and to enhance performance. Firstly, we incorporate strong geometry losses that use layout and depth inference at the training stage to enforce shape consistency between generated and ground truth scenes. Secondly, we apply a shading decomposition scheme to extract the albedo and normalized shading signal from the original scenes, and we apply the style transfer on albedo instead of full RGB images, thereby preventing shading-related bleeding issues. On top of that, we apply super-resolution to the resulting scenes to improve image quality and yield fine details. We evaluate our model's performance on public domain synthetic data sets. Our proposed architecture outperforms state-of-the-art style transfer models in terms of perceptual and accuracy metrics, achieving a 26.76% lower ArtFID, a 6.95% higher PSNR, and a 25.23% higher SSIM. The visual results show that our method is effective in producing realistic and visually pleasing indoor scenes.