Paper:
We present a novel method for photo-realistic re-rendering of reconstructed objects. The digital reproduction of object appearances is of paramount importance nowadays. Augmented and virtual reality relies on such 3D content. It enables virtual showrooms, virtual tours & sightseeing, the digital inspection of historical artifacts and many other applications. Classical approaches use methods to reconstruct the geometry of an object and textures to capture the appearance properties. Instead, we propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis. A core component of our work is the handling of view-dependent effects. Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object. As input data we are using an RGB video of the object. This video is used to reconstruct a proxy geometry of the object via multi-view stereo. Based on this 3D proxy, the appearance of a captured view can be warped into a new target view. This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts. To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects. Based on these estimations, we are able to convert observed images to diffuse images. These diffuse images can be projected into other views. In the target view, our pipeline reinserts the new view-dependent effects. To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results. Using this image-guided approach, the network does not have to allocate capacity on "remembering" object appearance, instead it learns how to combine the appearance of captured images. We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data. Results are best seen in the supplemental video: https://www.dropbox. com/sh/8pi37qlfqoukthd/AAB0sdQT79noBRtKBKWCywtJa?dl=0.

Review:
Authors propose a novel neural image-guided rendering method, a hybrid between classical image-based rendering and machine learning. They learn EffectsNet in a self-supervised manner to capture the view-dependent component using a reprojection loss. To render a novel view they subtract a view-dependent component from K closest train views, reproject the diffuse component to the novel view, add view-dependent component, use a CompositionNet to render a final image. They use both synthetic and real data for the experiments and compare to state-of-the-art image-based rendering methods of Hedman et al.    
I think the method discussed in the paper is a nice contribution to the neural rendering area. The paper is written clearly and the provided amount of technical details is enough to reproduce the method. 
I only have some concerns about the experiments. 
1) The method relies on the depth maps and the main contribution of the paper is related to the rendering of view-dependent effects. However, in realistic scenarios, the presence of the view-dependent effects leads to severe degradation of photogrammetry pipeline: imprecise camera poses and geometry. To my understating, imprecise geometry and camera poses should affect the method heavily. I think, the paper lacks a thorough evaluation on more realistic data, having only a single example in Fig. 14. 
2) Authors compare to DeepBlending of Hedman et al. and although the proposed model achieves the best quantitative result, to my mind, the qualitative difference is not very noticeable, while it is known that MSE error does not correlate with human perception good enough. To justify the presented method it would be great to see experiments with a more clear difference to Hedman et al. baseline. 
3) Authors of Pix2Pix (Isola et al., 2016) do not provide experimental results on novel view-point synthesis. Although it is completely fine to refer to Pix2Pix as to the closest and simplest neural-based baseline I do not find it convincing to compare the proposed method to Pix2Pix without any other baselines as in the figures 5, 11, 12. Pix2pix had never been claimed to be developed or validated for this particular problem so it is a kind of "artificial" baseline. Moreover, I believe a stronger "artificial" baseline can be easily constructed. In particular, it is hard to believe that for the "Vase" scene, which has almost no view-dependent effects, a network that takes a world space position map cannot learn a direct mapping from (x,y,z) to (r, g, b). That is why, I believe, a stronger "artificial" baseline could be easily constructed by some kind of tweaking of Pix2Pix, say, turning off GAN or changing neural networks' architecture slightly.  
Considering the issues mentioned above my I tend to recommend this paper for rejection. 
Questions:
1) Video shows examples of novel view synthesis for a predefined camera trajectory. How this trajectory is created for both synthetic and real objects? How does it differ from the train trajectory?
Some minor comments:
1) In Fig. 2 it reads "Input color", whereas the RGB color image is not used as input to EffectsNet. It confuses on the first read. In "... based on the current view and the respective depth map", it is also easy to think that "current view" refers to RGB image.
2) The model from Figure 8 is never used.