  Many learning-based approaches have difficulty scaling to unseen data, as the
generality of its learned prior is limited to the scale and variations of the
training samples. This holds particularly true with 3D learning tasks, given
the sparsity of 3D datasets available. We introduce a new learning framework
for 3D modeling and reconstruction that greatly improves the generalization
ability of a deep generator. Our approach strives to connect the good ends of
both learning-based and optimization-based methods. In particular, unlike the
common practice that fixes the pre-trained priors at test time, we propose to
further optimize the learned prior and latent code according to the input
physical measurements after the training. We show that the proposed strategy
effectively breaks the barriers constrained by the pre-trained priors and could
lead to high-quality adaptation to unseen data. We realize our framework using
the implicit surface representation and validate the efficacy of our approach
in a variety of challenging tasks that take highly sparse or collapsed
observations as input. Experimental results show that our approach compares
favorably with the state-of-the-art methods in terms of both generality and
accuracy.
