GENERATIVE ADVERSARIAL NETWORKS FOR SINGLE PHOTO 3D RECONSTRUCTION

• Published in: International archives of the photogrammetry, remote sensing and spatial information sciences. Vol. XLII-2/W9; pp. 403 - 408
• Main Authors: Kniaz, V. V., Remondino, F., Knyaz, V. A.
• Format: Journal Article Conference Proceeding
• Language: English
• Published: Gottingen Copernicus GmbH 01-01-2019; Copernicus Publications
• Subjects: Archaeology; Cultural heritage; Cultural resources; Generative adversarial networks; Image reconstruction; Temples; Three dimensional models; Training; Two dimensional models
• ISSN: 2194-9034; 1682-1750; 2194-9034
• DOI: 10.5194/isprs-archives-XLII-2-W9-403-2019

Fast but precise 3D reconstructions of cultural heritage scenes are becoming very requested in the archaeology and architecture. While modern multi-image 3D reconstruction approaches provide impressive results in terms of textured surface models, it is often the need to create a 3D model for which only a single photo (or few sparse) is available. This paper focuses on the single photo 3D reconstruction problem for lost cultural objects for which only a few images are remaining. We use image-to-voxel translation network (Z-GAN) as a starting point. Z-GAN network utilizes the skip connections in the generator network to transfer 2D features to a 3D voxel model effectively (Figure 1). Therefore, the network can generate voxel models of previously unseen objects using object silhouettes present on the input image and the knowledge obtained during a training stage. In order to train our Z-GAN network, we created a large dataset that includes aligned sets of images and corresponding voxel models of an ancient Greek temple. We evaluated the Z-GAN network for single photo reconstruction on complex structures like temples as well as on lost heritage still available in crowdsourced images. Comparison of the reconstruction results with state-of-the-art methods are also presented and commented.