  Recently, learning-based approaches for 3D model reconstruction have
attracted attention owing to its modern applications such as Extended
Reality(XR), robotics and self-driving cars. Several approaches presented good
performance on reconstructing 3D shapes by learning solely from images, i.e.,
without using 3D models in training. Challenges, however, remain in texture
generation due to the gap between 2D and 3D modals. In previous work, the grid
sampling mechanism from Spatial Transformer Networks was adopted to sample
color from an input image to formulate texture. Despite its success, the
existing framework has limitations on searching scope in sampling, resulting in
flaws in generated texture and consequentially on rendered 3D models. In this
paper, to solve that issue, we present a novel sampling algorithm by optimizing
the gradient of predicted coordinates based on the variance on the sampling
image. Taking into account the semantics of the image, we adopt Frechet
Inception Distance (FID) to form a loss function in learning, which helps
bridging the gap between rendered images and input images. As a result, we
greatly improve generated texture. Furthermore, to optimize 3D shape
reconstruction and to accelerate convergence at training, we adopt part
segmentation and template learning in our model. Without any 3D supervision in
learning, and with only a collection of single-view 2D images, the shape and
texture learned by our model outperform those from previous work. We
demonstrate the performance with experimental results on a publically available
dataset.
