Make3D: Learning 3D Scene Structure from a Single Still Image

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 31; no. 5; pp. 824 - 840
• Main Authors: Saxena, A., Min Sun, Ng, A.Y.
• Format: Journal Article
• Language: English
• Published: Los Alamitos, CA IEEE 01-05-2009; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Computer vision; depth cues; Exact sciences and technology; Image Enhancement - methods; Image Interpretation, Computer-Assisted - methods; Image segmentation; Imaging, Three-Dimensional - methods; Information Storage and Retrieval - methods; Internet; Large-scale systems; Layout; Learning; learning depth; Machine learning; Magnetorheological fluids; Markov random fields; monocular vision; Pattern analysis; Pattern Recognition, Automated - methods; Pattern recognition. Digital image processing. Computational geometry; Photography - methods; Planes; Rendering; Rendering (computer graphics); Reproducibility of Results; scene analysis; Sensitivity and Specificity; Software; Sun; Supervised learning; Three dimensional; vision and scene understanding; Scene Analysis; Computer vision; Virtual reality; Machine learning; Vision and Scene Understanding; Statistical; Image-based rendering; Depth cues; Markov process; Probabilistic approach; Image processing; Image interpretation; Monocular vision; Image rendering; Orientation; Scale model; Modeling; depth cues; Scaling law; Supervised learning; Scene analysis; vision and scene understanding; Fixed image; learning depth; Tridimensional image; Internet; Localization; Pattern analysis; Artificial intelligence
• ISSN: 0162-8828; 1939-3539
• DOI: 10.1109/TPAMI.2008.132

We consider the problem of estimating detailed 3D structure from a single still image of an unstructured environment. Our goal is to create 3D models that are both quantitatively accurate as well as visually pleasing. For each small homogeneous patch in the image, we use a Markov random field (MRF) to infer a set of "plane parametersrdquo that capture both the 3D location and 3D orientation of the patch. The MRF, trained via supervised learning, models both image depth cues as well as the relationships between different parts of the image. Other than assuming that the environment is made up of a number of small planes, our model makes no explicit assumptions about the structure of the scene; this enables the algorithm to capture much more detailed 3D structure than does prior art and also give a much richer experience in the 3D flythroughs created using image-based rendering, even for scenes with significant nonvertical structure. Using this approach, we have created qualitatively correct 3D models for 64.9 percent of 588 images downloaded from the Internet. We have also extended our model to produce large-scale 3D models from a few images.