Unifying Correspondence, Pose and NeRF for Generalized Pose-Free Novel View Synthesis

• Published in: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 20196 - 20206
• Main Authors: Hong, Sunghwan, Jung, Jaewoo, Shin, Heeseong, Yang, Jiaolong, Kim, Seungryong, Luo, Chong
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16-06-2024
• Subjects: Accuracy; Cameras; Geometry; Neural radiance field; Pose estimation; Three-dimensional displays; Training
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52733.2024.01909

This work delves into the task of pose-free novel view synthesis from stereo pairs, a challenging and pioneering task in 3D vision. Our innovative framework, unlike any before, seamlessly integrates 2D correspondence matching, camera pose estimation, and NeRF rendering, fostering a synergistic enhancement of these tasks. We achieve this through designing an architecture that utilizes a shared representation, which serves as a foundation for enhanced 3D geometry understanding. Capitalizing on the inherent in-terplay between the tasks, our unified framework is trained end-to-end with the proposed training strategy to improve overall model accuracy. Through extensive evaluations across diverse indoor and outdoor scenes from two real-world datasets, we demonstrate that our approach achieves substantial improvement over previous methodologies, es-pecially in scenarios characterized by extreme viewpoint changes and the absence of accurate camera poses. The project page and code will be made available at: https://ku-cvlab.github.io/CoPoNeRF/.