Self-calibrating Deep Photometric Stereo Networks
Guanying Chen1Kai Han2Boxin Shi3,4Yasuyuki Matsushita5Kwan-Yee K. Wong1
1The University of Hong Kong   2University of Oxford
3Peking University   4Peng Cheng Laboratory   5Osaka University

Code [PyTorch]     Paper [CVPR 2019 Oral]     Journal Extension [TPAMI 2020]    
Supplementary [PDF]     Poster [LaTex]


This paper proposes an uncalibrated photometric stereo method for non-Lambertian scenes based on deep learning. Unlike previous approaches that heavily rely on assumptions of specific reflectances and light source distributions, our method is able to determine both shape and light directions of a scene with unknown arbitrary reflectances observed under unknown varying light directions. To achieve this goal, we propose a two-stage deep learning architecture, called SDPS-Net, which can effectively take advantage of intermediate supervision, resulting in reduced learning difficulty compared to a single-stage model. Experiments on both synthetic and real datasets show that our proposed approach significantly outperforms previous uncalibrated photometric stereo methods.



The network architecture of SDPS-Net is composed of (a) Lighting Calibration Network and (b) Normal Estimation Network. Kernel sizes for all convolutional layers are 3 × 3, and values above the layers indicate the number of feature channels.

Training Dataset

We adopted the publicly available synthetic PS Blobby and Sculpture datasets for training. Blobby and Sculpture datasets provide surfaces with complex normal distributions and diverse materials from MERL dataset. Effects of cast shadow and inter-reflection were considered during rendering using the physically based raytracer Mitsuba. There are 85,212 samples in total. Each sample was rendered under 64 distinct light directions sampled from the upper-hemisphere with uniform light intensity, resulting in 5,453,568 images (85,212 X 64). The rendered images have a dimension of 128 X 128.

Results on Real Dataset

1. DiLiGenT Main Dataset.

Up to Down: Sample Input, Object, GT Normal, Results of SDPS-Net, and Error Map

2. Light Stage Data Gallery.

Up to Down: Sample Input, Object, Results of SDPS-Net and UPS-FCN

3. Gourd&Apple Dataset.

Code and Model

Code, models and datasets are available at Github!


We thank the anonymous reviewers for their constructive comments. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X Pascal GPU. Kai Han is supported by EPSRC Programme Grant Seebibyte EP/M013774/1. Boxin Shi is supported in part by National Science Foundation of China under Grant No. 61872012. Yasuyuki Matsushita is supported by the New Energy and Industrial Technology Development Organization (NEDO).

Webpage template borrowed from Split-Brain Autoencoders, CVPR 2017.