Intrinsic Scene Properties from a Single RGB-D Image

Jonathan T. Barron, Jitendra Malik; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 17-24

Abstract


In this paper we extend the "shape, illumination and reflectance from shading" (SIRFS) model [3, 4], which recovers intrinsic scene properties from a single image. Though SIRFS performs well on images of segmented objects, it performs poorly on images of natural scenes, which contain occlusion and spatially-varying illumination. We therefore present Scene-SIRFS, a generalization of SIRFS in which we have a mixture of shapes and a mixture of illuminations, and those mixture components are embedded in a "soft" segmentation of the input image. We additionally use the noisy depth maps provided by RGB-D sensors (in this case, the Kinect) to improve shape estimation. Our model takes as input a single RGB-D image and produces as output an improved depth map, a set of surface normals, a reflectance image, a shading image, and a spatially varying model of illumination. The output of our model can be used for graphics applications, or for any application involving RGB-D images.

Related Material


[pdf]
[bibtex]
@InProceedings{Barron_2013_CVPR,
author = {Barron, Jonathan T. and Malik, Jitendra},
title = {Intrinsic Scene Properties from a Single RGB-D Image},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2013}
}