What Motion Reveals about Shape with Unknown Material Behavior

Abstract:

Image formation is an outcome of a complex interaction between object geometry, lighting and camera, as governed by the reflectance of the underlying material. Psychophysical studies show that motion of the object, light source or camera are important cues for shape perception from image sequences. However, due to the complex and often unknown nature of the bidirectional reflectance distribution function (BRDF) that determines material behavior, computer vision algorithms have traditionally relied on simplifying assumptions such as brightness constancy or Lambertian reflectance. We take a step towards overcoming those limitations by answering a fundamental question: what does motion reveal about unknown shape and material? In each case of light source, object or camera motion, we show that physical properties of BRDFs yield PDE invariants that precisely characterize the extent of shape recovery under a given imaging condition. Conventional optical flow, multiview stereo and photometric stereo follow as special cases. This leads to the surprising result that motion can decipher shape even with complex, unknown material behavior and unknown lighting. Further, we show that contrary to intuition, joint recovery of shape, material and lighting using motion cues is often well-posed and tractable, requiring the solution of only sparse linear systems.

At the beginning of the talk, I will also describe our recent work on 3D scene understanding for autonomous driving. Using a single camera, we demonstrate real-time structure from motion performance on par with stereo, on the challenging KITTI benchmark. Combined with top-performing methods in object detection and tracking, we demonstrate 3D object localization with high accuracy comparable to LIDAR. We demonstrate high-level applications such as scene recognition that form the basis for collaborations on collision avoidance and danger prediction with automobile manufacturers.

Brief Bio:

Manmohan Chandraker received a B.Tech. in Electrical Engineering at the Indian Institute of Technology, Bombay and a PhD in Computer Science at the University of California, San Diego. Following a postdoctoral scholarship at the University of California, Berkeley, he joined NEC Labs America in Cupertino, where he conducts research in computer vision. His principal research interests are modern optimization methods for geometric 3D reconstruction, 3D scene understanding and recognition for autonomous driving and shape recovery in the presence of complex illumination and material behavior. His work has received the Marr Prize Honorable Mention for Best Paper at ICCV 2007, the 2009 CSE Dissertation Award for Best Thesis at UC San Diego, a nomination for the 2010 ACM Dissertation Award and the Best Paper Award at CVPR 2014, besides appearing in Best Paper Special Issues of IJCV 2009, IEEE PAMI 2011 and 2014.