3D Shape and Indirect Appearance by Structured Light Transport

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 38; no. 7; pp. 1298 - 1312
• Main Authors: O'Toole, Matthew, Mather, John, Kutulakos, Kiriakos N.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01-07-2016; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cameras; coded exposure; coded illumination; Computer vision; direct/global separation; dynamic 3D shape capture; epipolar constraints; Holiday decorations; inter-reflections; Light; light transport; multi-path interference; Optical imaging; structured light 3D scanning; subsurface scattering; Three-dimensional displays; Transmission line matrix methods; inter-reflections; subsurface scattering; coded illumination; direct/global separation; coded exposure; dynamic 3D shape capture; primal-dual coding; epipolar constraints; structured light 3D scanning; Light transport; multi-path interference
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2016.2545662

We consider the problem of deliberately manipulating the direct and indirect light flowing through a time-varying, general scene in order to simplify its visual analysis. Our approach rests on a crucial link between stereo geometry and light transport: while direct light always obeys the epipolar geometry of a projector-camera pair, indirect light overwhelmingly does not. We show that it is possible to turn this observation into an imaging method that analyzes light transport in real time in the optical domain, prior to acquisition. This yields three key abilities that we demonstrate in an experimental camera prototype: (1) producing a live indirect-only video stream for any scene, regardless of geometric or photometric complexity; (2) capturing images that make existing structured-light shape recovery algorithms robust to indirect transport; and (3) turning them into one-shot methods for dynamic 3D shape capture.