Original: I MAGE - GUIDED N EURAL O BJECT R ENDERING
Revised: I MAGE - GUIDED N EURAL O BJECT R ENDERING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a novel method for photo-realistic re-rendering of reconstructed objects. The digital reproduction of object appearances is of paramount importance nowadays. Augmented and virtual reality relies on such 3D content.
Revised: We propose a learned image guided rendering technique that combines the bene ﬁts of image based rendering and GAN based image synthesis.

Original: It enables virtual showrooms, virtual tours & sightseeing, the digital inspection of historical artifacts and many other applications. Classical approaches use methods to reconstruct the geometry of an object and textures to capture the appearance properties.
Revised: The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g. virtual showrooms, virtual tours & sightseeing, the digital inspection of historical artifacts).

Original: Instead, we propose a learned image-guided rendering technique that combines the beneﬁts of image-based rendering and GAN-based image synthesis.

Original: A core component of our work is the handling of view-dependent effects.
Revised: A core component of our work is the handling of view-dependent effects.

Original: Specifically, we directly train an object-speciﬁc deep neural network to synthesize the view-dependent appearance of an object.
Revised: Speciﬁcally, we directly train an object-speciﬁc deep neural network to synthesize the view-dependent appearance of an object.

Original: As input data we are using an RGB video of the object.
Revised: As input data we are using an RGB video of the object.

Original: This video is used to reconstruct a proxy geometry of the object via multi-view stereo.
Revised: This video is used to reconstruct a proxy geometry of the object via multi view stereo.

Original: Based on this 3D proxy, the appearance of a captured view can be warped into a new target view.
Revised: Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.

Original: This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts.
Revised: This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts.

Original: To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects.
Revised: To this end we propose EffectsNet a deep neural network that predicts view-dependent effects.

Original: Based on these estimations, we are able to convert observed images to diffuse images.
Revised: Based on these estimations we are able to convert observed images to diffuse images.

Original: These diffuse images can be projected into other views.
Revised: These diffuse images can be projected into other views.

Original: In the target view, our pipeline reinserts the new view-dependent effects.
Revised: In the target view, our pipeline reinserts the new view dependent effects.

Original: To composite multiple reprojected images to a ﬁnal output, we learn a composition network that outputs photo-realistic results.
Revised: To composite multiple reprojected images to a ﬁnal output, we learn a composition network that outputs photo realistic results.

Original: Using this image-guided approach, the network does not have to allocate capacity on “remembering” object appearance, instead it learns how to combine the appearance of captured images.
Revised: Using this image guided approach, the network does not have to allocate capacity on “remembering” object appearance, instead it learns how to combine the appearance of captured images.

Original: We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data.
Revised: We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data.

Original: Results are best seen in the supplemental video: https://www.dropbox. com/sh/8pi37qlfqoukthd/AAB0sdQT79noBRtKBKWCywtJa?dl=0 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, large progress has been made in 3D shape reconstruction of objects from photographs or depth streams.
Revised: In recent years, large progress has been made in 3D shape reconstruction of objects from photographs or depth streams.

Original: However, highly realistic re-rendering of such objects, e.g., in a virtual environment, is still very challenging.
Revised: However, highly realistic re-rendering of such objects, e.g., in a virtual environment, is still very challenging.

Original: The reconstructed surface models and color information often exhibit inaccuracies or are comparably coarse (e.g., Izadi et al.
Revised: The reconstructed surface models and color information often exhibit inaccuracies or are comparably coarse (e.g., Izadi et al.

Original: Many objects also exhibit strong view-dependent appearance effects, such as specularities.
Revised: Many objects also exhibit strong view-dependent appearance effects, such as specularities.

Original: These effects not only frequently cause errors already during image-based shape reconstruction, but are also hard to reproduce when re-rendering an object from novel viewpoints.
Revised: These effects not only frequently cause errors already during image-based shape reconstruction, but are also hard to reproduce when re-rendering an object from novel viewpoints.

Original: Static diffuse textures are frequently reconstructed for novel viewpoint synthesis, but these textures lack view-dependent appearance effects.
Revised: Static diffuse textures are frequently reconstructed for novel viewpoint synthesis, but these textures lack view-dependent appearance effects.

Original: Imagebased rendering (IBR) introduced variants of view-dependent texturing that blend input images on the shape (Buehler et al., 2001; Heigl et al., 1999; Carranza et al., 2003; Zheng et al., 2009).
Revised: Imagebased rendering (IBR) introduced variants of view-dependent texturing that blend input images on the shape (Buehler et al., 2001; Heigl et al., 1999; Carranza et al., 2003; Zheng et al., 2009).

Original: This enables at least coarse approximation of view-dependent effects.
Revised: This enables at least coarse approximation of view-dependent effects.

Original: However, these approaches often produce ghosting artifacts due to view blending on inaccurate geometry, or artifacts at occlusion boundaries.
Revised: However, these approaches often produce ghosting artifacts due to view blending on inaccurate geometry, or artifacts at occlusion boundaries.

Original: Some algorithms reduce these artifacts by combining view blending and optical ﬂow correction (Eisemann et al., 2008;
Revised: Some algorithms reduce these artifacts by combining view blending and optical ﬂow correction (Eisemann et al., 2008;

Original: Casas et al., 2015; Du et al., 2018), or by combining viewdependent blending with view-speciﬁc geometry (Chaurasia et al., 2013; Hedman et al., 2016) or geometry with soft 3D visibility like Penner & Zhang (2017).
Revised: Casas et al., 2015; Du et al., 2018), or by combining viewdependent blending with view-speciﬁc geometry (Chaurasia et al., 2013; Hedman et al., 2016) or geometry with soft 3D visibility like Penner & Zhang (2017).

Original: Hedman et al.
Revised: Hedman et al.

Original: (2018) reduces these artifacts using a deep neural network which is predicting per-pixel blending weights.
Revised: (2018) reduces these artifacts using a deep neural network which is predicting per-pixel blending weights.

Original: In contrast, our approach explicitly handles view-dependent effects to output photo-realistic images and videos.
Revised: In contrast, our approach explicitly handles view-dependent effects to output photo-realistic images and videos.

Original: It is a neural rendering approach that combines image-based rendering and the advances in deep learning.
Revised: It is a neural rendering approach that combines image-based rendering and the advances in deep learning.

Original: As input, we capture a short video of an object to reconstruct the geometry using multi-view stereo.
Revised: As input, we capture a short video of an object to reconstruct the geometry using multi-view stereo.

Original: Given this 3 D reconstruction and the set of images of the video, we are able to train our pipeline in a self-supervised manner.
Revised: Given this 3 D reconstruction and the set of images of the video, we are able to train our pipeline in a self-supervised manner.

Original: The core of our approach is a neural network called EffectsNet which is trained in a Siamese way to estimate view-dependent effects, for example, specular highlights or reﬂections.
Revised: The core of our approach is a neural network called EffectsNet which is trained in a Siamese way to estimate view-dependent effects, for example, specular highlights or reﬂections.

Original: This allows us to remove view-dependent effects from the input images, resulting in images that contain view-independent appearance information of the object.
Revised: This allows us to remove view-dependent effects from the input images, resulting in images that contain view-independent appearance information of the object.

Original: This view-independent information can be projected into a novel view using the reconstructed geometry, where new view-dependent effects can be added.
Revised: This view-independent information can be projected into a novel view using the reconstructed geometry, where new view-dependent effects can be added.

Original: CompositionNet , a second network, composites the projected K nearest neighbor images to a ﬁnal output.
Revised: CompositionNet , a second network, composites the projected K nearest neighbor images to a ﬁnal output.

Original: Since CompositionNet is trained to generate photo-realistic output images, it is resolving reprojection errors as well as ﬁlling regions where no image content is available.
Revised: Since CompositionNet is trained to generate photo-realistic output images, it is resolving reprojection errors as well as ﬁlling regions where no image content is available.

Original: We demonstrate the effectiveness of our algorithm using synthetic and real data, and compare to classical computer graphics and learned approaches.
Revised: We demonstrate the effectiveness of our algorithm using synthetic and real data, and compare to classical computer graphics and learned approaches.

Original: To summarize, we propose a novel neural image-guided rendering method, a hybrid between classical image-based rendering and machine learning.
Revised: To summarize, we propose a novel neural image-guided rendering method, a hybrid between classical image-based rendering and machine learning.

Original: The core contribution is the explicit handling of view-dependent effects in the source and the target views using EffectsNet that can be learned in a self-supervised fashion.
Revised: The core contribution is the explicit handling of view-dependent effects in the source and the target views using EffectsNet that can be learned in a self-supervised fashion.

Original: The composition of the reprojected views to a ﬁnal output image without the need of hand-crafted blending schemes is enabled using our network called CompositionNet .
Revised: The composition of the reprojected views to a ﬁnal output image without the need of hand-crafted blending schemes is enabled using our network called CompositionNet .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multi-view 3D Reconstruction Our approach builds on a coarse geometric proxy that is obtained using multi-viewimage-based 3D reconstruction based on COLMAP (Sch¨onberger & Frahm, 2016).
Revised: Multi-view 3D Reconstruction Our approach builds on a coarse geometric proxy that is obtained using multi-view 3D reconstruction based on COLMAP (Sch ¨ onberger & Frahm, 2016).

Original: In the last decade, there has been a lot of progress in the ﬁeld of image-based 3D reconstruction.
Revised: In the last decade, there has been a lot of progress in the ﬁeld of image-based 3D reconstruction.

Original: Large-scale 3D models have been automatically obtained from images downloaded from the internet (Agarwal et al., 2011).
Revised: Large-scale 3D models have been automatically obtained from images downloaded from the internet (Agarwal et al., 2011).

Original: Camera poses and intrinsic calibration parameters are estimated based on structure-from-motion (Jebara et al., 1999; Schnberger & Frahm, 2016), which can be implemented based on a global bundle adjustment step (Triggs et al., 2000).
Revised: Camera poses and intrinsic calibration parameters are estimated based on structurefrom-motion (Jebara et al., 1999; Schnberger & Frahm, 2016), which can be implemented based on a global bundle adjustment step (Triggs et al., 2000).

Original: Afterwards, based on the camera poses and calibration, a dense three-dimensional pointcloud of the scene can be obtained using a multi-view stereo reconstruction approach (Seitz et al., 2006; Goesele et al., 2007; Geiger et al., 2011).
Revised: Afterwards, based on the camera poses and calibration, a dense three-dimensional pointcloud of the scene can be obtained using a multi-view stereo reconstruction approach (Seitz et al., 2006; Goesele et al., 2007; Geiger et al., 2011).

Original: Finally, a triangulated surface mesh is obtained, for example using Poisson surface reconstruction (Kazhdan et al., 2006).
Revised: Finally, a triangulated surface mesh is obtained, for example using Poisson surface reconstruction (Kazhdan et al., 2006).

Original: Even specular objects can bewell reconstructed (Godard et al., 2015).
Revised: Even specular objects can be reconstructed (Godard et al., 2015).

Original: Learning-based Image Synthesis
Revised: Learning-based Image Synthesis

Original: Deep learning methods can improve quality in many realistic image synthesis tasks.
Revised: Deep learning methods can improve quality in many realistic image synthesis tasks.

Original: Historically, many of these approaches have been based on generator networks following an encoder-decoder architecture (Hinton & Salakhutdinov, 2006; Kingma &
Revised: Historically, many of these approaches have been based on generator networks following an encoder-decoder architecture (Hinton & Salakhutdinov, 2006; Kingma &

Original: Welling, 2013), such as a U-Net (Ronneberger et al., 2015a) with skip connections.
Revised: Welling, 2013), such as a U-Net (Ronneberger et al., 2015a) with skip connections.

Original: Very recently, adversarially trained networks (Goodfellow et al., 2014; Isola et al., 2017; Mirza & Osindero, 2014;
Revised: Very recently, adversarially trained networks (Goodfellow et al., 2014; Isola et al., 2017; Mirza & Osindero, 2014;

Original: Radford et al., 2016) have shown some of the best result quality for various image synthesis tasks.
Revised: Radford et al., 2016) have shown some of the best result quality for various image synthesis tasks.

Original: For example, generative CNN models to synthesize body appearance (Esser et al., 2018), body articulation (Chan et al., 2018), body pose and appearance (Zhu et al., 2018; Liu et al., 2018), and face rendering (Kim et al., 2018; Thies et al., 2019) have been proposed.
Revised: For example, generative CNN models to synthesize body appearance (Esser et al., 2018), body articulation (Chan et al., 2018), body pose and appearance (Zhu et al., 2018; Liu et al., 2018), and face rendering (Kim et al., 2018; Thies et al., 2019) have been proposed.

Original: The DeepStereo approach of
Revised: The DeepStereo approach of

Original: Flynn et al.
Revised: Flynn et al.

Original: (2016) trains a neural network for view synthesis based on a large set of posed images.
Revised: (2016) trains a neural network for view synthesis based on a large set of posed images.

Original: Tulsiani et al.
Revised: Tulsiani et al.

Original: (2018) employ view synthesis as a proxy task to learn a layered scene representation.
Revised: (2018) employ view synthesis as a proxy task to learn a layered scene representation.

Original: View synthesis can be learned directly from light ﬁeld data as shown by Kalantari et al.
Revised: View synthesis can be learned directly from light ﬁeld data as shown by Kalantari et al.

Original: Appearance Flow (Zhou et al., 2016) learns an image warp based on a dense ﬂow ﬁeld to map information from the input to the target view.
Revised: Appearance Flow (Zhou et al., 2016) learns an image warp based on a dense ﬂow ﬁeld to map information from the input to the target view.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2018) learn to extrapolate stereo views from imagery captured by a narrow-baseline stereo camera.
Revised: (2018) learn to extrapolate stereo views from imagery captured by a narrow-baseline stereo camera.

Original: Park et al.
Revised: Park et al.

Original: (2017) explicitly decouple the view synthesis problem into an image warping and inpainting task.
Revised: (2017) explicitly decouple the view synthesis problem into an image warping and inpainting task.

Original: In the results, we also show that CNNs trained for image-to-image translation (Isola et al.
Revised: In the results, we also show that CNNs trained for image-to-image translation (Isola et al.

Original: (2016)) could be applied to novel view synthesis, also with assistance of a shape proxy.
Revised: (2016)) could be applied to novel view synthesis, also with assistance of a shape proxy.

Original: Image-based Rendering Our approach is related to image-based rendering (IBR) algorithms that cross-project input views to the target via a geometry proxy, and blend the re-projected views (Buehler et al., 2001; Heigl et al., 1999; Carranza et al., 2003; Zheng et al., 2009).
Revised: Image-based Rendering Our approach is related to image-based rendering (IBR) algorithms that cross-project input views to the target via a geometry proxy, and blend the re-projected views (Buehler et al., 2001; Heigl et al., 1999; Carranza et al., 2003; Zheng et al., 2009).

Original: Many previous
Revised: Many previous

Original: IBR approaches exhibit ghosting artifacts due to view blending on inaccurate geometry, or exhibit artifacts at occlusion boundaries.
Revised: IBR approaches exhibit ghosting artifacts due to view blending on inaccurate geometry, or exhibit artifacts at occlusion boundaries.

Original: Some methods try to reduce these artifacts by combining view blending and optical ﬂow correction (Eisemann et al., 2008;
Revised: Some methods try to reduce these artifacts by combining view blending and optical ﬂow correction (Eisemann et al., 2008;

Original: Casas et al., 2015; Du et al., 2018), by using view-speciﬁc geometry proxies (Chaurasia et al., 2013; Hedman et al., 2016), or by encoding uncertainty in geometry as soft 3D visibility (Penner & Zhang, 2017).
Revised: Casas et al., 2015; Du et al., 2018), by using view-speciﬁc geometry proxies (Chaurasia et al., 2013; Hedman et al., 2016), or by encoding uncertainty in geometry as soft 3D visibility (Penner & Zhang, 2017).

Original: Hedman et al.
Revised: Hedman et al.

Original: (2018) propose a hybrid approach between IBR and learning-based image synthesis.
Revised: (2018) propose a hybrid approach between IBR and learning-based image synthesis.

Original: They use a CNN to learn a view blending function for image-based rendering with view-dependent shape proxies.
Revised: They use a CNN to learn a view blending function for image-based rendering with view-dependent shape proxies.

Original: Our approach is also a learned IBR method. In contrast to theirs, our approach learns to combine input views and to explicitly separate view-dependent effects which leads to superior reproduction of view-dependent appearance.
Revised: In contrast, our learned IBR method learns to combine input views and to explicitly separate view-dependent effects which leads to better reproduction of view-dependent appearance.

Original: Intrinsic Decomposition Intrinsic decomposition tackles the ill-posed problem of splitting an image into a set of layers that correspond to physical quantities such as surface reﬂectance, diffuse shading, and/or specular shading.
Revised: Intrinsic Decomposition Intrinsic decomposition tackles the ill-posed problem of splitting an image into a set of layers that correspond to physical quantities such as surface reﬂectance, diffuse shading, and/or specular shading.

Original: The decomposition of monocular video into reﬂectance and shading is classically approached based on a set of hand-crafted priors (Bonneel et al., 2014; Ye et al., 2014; Meka et al., 2016).
Revised: The decomposition of monocular video into reﬂectance and shading is classically approached based on a set of hand-crafted priors (Bonneel et al., 2014; Ye et al., 2014; Meka et al., 2016).

Original: Other approaches speciﬁcally tackle the problem of estimating (Lin et al., 2002) or removing specular highlights (Yang et al., 2015).
Revised: Other approaches speciﬁcally tackle the problem of estimating (Lin et al., 2002) or removing specular highlights (Yang et al., 2015).

Original: Highlight removal is of particular interest for images of faces (Li et al., 2017).

Original: A diffuse/specular separation can also be obtained based on a set of multi-view images captured under varying illumination (Takechi & Okabe, 2017).
Revised: A diffuse/specular separation can also be obtained based on a set of multi-view images captured under varying illumination (Takechi & Okabe, 2017).

Original: The learning-based approach of Wu et al.
Revised: The learning-based approach of Wu et al.

Original: (2018) converts a set of multi-view images of a specular object into corresponding diffuse images.
Revised: (2018) converts a set of multi-view images of a specular object into corresponding diffuse images.

Original: An extensive overview is give in the survey paper of Bonneel et al.
Revised: An extensive overview is given in the survey paper of Bonneel et al.

Original:  O VERVIEW
Revised:  O VERVIEW

Original: We propose a learning-based image-guided rendering approach that enables novel view synthesis for arbitrary objects.
Revised: We propose a learning-based image-guided rendering approach that enables novel view synthesis for arbitrary objects.

Original: Input to our approach is a set of N images
Revised: Input to our approach is a set of N images

Original: I =
Revised: I =

Original: {I k } Nk =1 of an object with constant illumination.
Revised: {I k } Nk =1 of an object with constant illumination.

Original: In a preprocess, we obtain camera pose estimates and a coarse proxy geometry using the COLMAP structure-from-motion approach (Sch¨onberger & Frahm (2016); Sch¨onberger et al.
Revised: In a preprocess, we obtain camera pose estimates and a coarse proxy geometry using the COLMAP structure-from-motion approach (Sch¨onberger & Frahm (2016); Sch¨onberger et al.

Original: We use the reconstruction and the camera poses to render synthetic depth maps D k for all input images I k to obtain the training corpus T
Revised: We use the reconstruction and the camera poses to render synthetic depth maps D k for all input images I k to obtain the training corpus T

Original: = { ( I k , D k ) } Nk =1 , see Fig. 8.
Revised: = { ( I k , D k ) } N k =1 , see Fig. 8.

Original: Based on this input, our learning-based approach generates novel views based on the stages that are depicted in
Revised: Based on this input, our learning-based approach generates novel views based on the stages that are depicted in

Original: Fig.
Revised: Fig.

Original: First, we employ a coverage-based look-up to select a small number n (cid:28) N of ﬁxed views from a subset of the training corpus.
Revised: First, we employ a coverage-based look-up to select a small number n (cid:28) N of ﬁxed views from a subset of the training corpus.

Original: In our experiments, we are using a number of n =frames, which we call reference images.
Revised: In our experiments, we are using a number of n =frames, which we call reference images.

Original: Per target view, we select K = 4 nearest views from these reference images.
Revised: Per target view, we select K = 4 nearest views from these reference images.

Original: Our EffectsNet predicts the view-dependent effects for these views and, thus, the corresponding view-independent components can be obtained via subtraction (Sec. 5).
Revised: Our EffectsNet predicts the view-dependent effects for these views and, thus, the corresponding view-independent components can be obtained via subtraction (Sec. 5).

Original: The view-independent component is explicitly warped to the target view using geometry-guided crossprojection (Sec. 6).
Revised: The view-independent component is explicitly warped to the target view using geometry-guided crossprojection (Sec. 6).

Original: Next, the view-dependent effects of the target view are predicted and added on top of the warped views.
Revised: Next, the view-dependent effects of the target view are predicted and added on top of the warped views.

Original: Finally, our CompositionNet is used to optimally combine all warped views to generate the ﬁnal output (Sec. 6).
Revised: Finally, our CompositionNet is used to optimally combine all warped views to generate the ﬁnal output (Sec. 6).

Original: In the following, we discuss details, show how our approach can be trained based on our training corpus (Sec. 4), and extensively evaluate our proposed approach (see Sec.
Revised: In the following, we discuss details, show how our approach can be trained based on our training corpus (Sec. 4), and extensively evaluate our proposed approach (see Sec.

Original: 7 and the appendix).
Revised: 7 and the appendix).

Original:  T RAINING D ATA
Revised:  T RAINING D ATA

Original: Our approach is trained in an object-speciﬁc manner, from scratch each time.
Revised: Our approach is trained in an object-speciﬁc manner, from scratch each time.

Original: The training corpus
Revised: The training corpus

Original: T = { ( I k , D k ) }
Revised: T = { ( I k , D k ) }

Original: Nk =1 consists of N images I k and depth maps D k per object with constant light.
Revised: Nk =1 consists of N images I k and depth maps D k per object with constant light.

Original: Synthetic Training Data To generate photo-realistic synthetic imagery we employ the Mitsuba
Revised: Synthetic Training Data To generate photo-realistic synthetic imagery we employ the Mitsuba

Original: Renderer (Jakob, 2010) to simulate global illumination effects.
Revised: Renderer (Jakob, 2010) to simulate global illumination effects.

Original: For each of the N views, we raytrace a color image I k and its corresponding depth map D k .
Revised: For each of the N views, we raytrace a color image I k and its corresponding depth map D k .

Original: We extract a dense and smooth temporal camera path based on a spiral around the object.
Revised: We extract a dense and smooth temporal camera path based on a spiral around the object.

Original: The camera is oriented towards the center of the object.
Revised: The camera is oriented towards the center of the object.

Original: All images have a resolution of 512 × 512 and are rendered using pathtracing withsamplesper pixel and a maximum path length of 10 .
Revised: All images have a resolution of 512 × 512 and are rendered using path withsamples path length size training is

Original: The size of the training sequence is 920 , the test set contains 177 images.

Original: Real World Training Data Our real world training data is captured using a Nikon D5300 at a resolution of 1920 × 1080 pixels.
Revised: Real World Training Data Our real world training data is captured using a Nikon D5300 at a resolution of 1920 × 1080 pixels.

Original: Since we rely on a sufﬁciently large set of images, we record videos of the objects at a frame rate of 30 Hz.
Revised: Since we rely on a sufﬁciently large set of images, we record videos of the objects at a frame rate of 30 Hz.

Original: Based on COLMAP (Sch ¨ onberger & Frahm, 2016;
Revised: Based on COLMAP (Sch ¨ onberger & Frahm, 2016;

Original: Sch ¨ onberger et al., 2016), we reconstruct the camera path and a dense point cloud.
Revised: Sch ¨ onberger et al., 2016), we reconstruct the camera path and a dense point cloud.

Original: We manually isolate the target object from other reconstructed geometry and run a Poisson reconstruction (Kazhdan & Hoppe, 2013) step to extract the surface.
Revised: We manually isolate the target object from other reconstructed geometry and run a Poisson reconstruction (Kazhdan & Hoppe, 2013) step to extract the surface.

Original: We use this mesh to generate synthetic depth maps D k corresponding to the images
Revised: We use this mesh to generate synthetic depth maps D k corresponding to the images

Original: I k
Revised: I k

Original: (see Fig.
Revised: (see Fig.

Original: Finally, both, the color and depth images are cropped and re-scaled to a resolution of 512 × 512 pixels.
Revised: Finally, both, the color and depth images are cropped and re-scaled to a resolution of 512 × 512 pixels.

Original: The training corpus ranges from 1000 toframes, depending on the sequence.
Revised: The training corpus ranges from 1000 toframes, depending on the sequence.

Original:  EffectsNet
Revised:  EffectsNet

Original: The main contributions of our work is a convolutional neural network that learns the disentanglement of view-dependent and view-independent effects in a self-supervised manner (see Fig. 2).
Revised: A main contribution of our work is a convolutional neural network that learns the disentanglement of view-dependent and view-independent effects in a self-supervised manner (see Fig. 2).

Original: Since our training data consists of a series of images taken from different viewing directions, assuming constant illumination, the reﬂected radiance of two corresponding points in two different images only differs by the view-dependent effects.
Revised: Since our training data consists of a series of images taken from different viewing directions, assuming constant illumination, the reﬂected radiance of two corresponding points in two different images only differs by the view-dependent effects.

Original: Our self-supervised training procedure is based on a
Revised: Our self-supervised training procedure is based on a

Original: Siamese network that gets a pair of randomly selected images from the training set as input.
Revised: Siamese network that gets a pair of randomly selected images from the training set as input.

Original: The task of the network is to extract view-dependent lighting effects from an image, based on  geometric information from the proxy geometry.
Revised: The task of the network is to extract view-dependent lighting effects from an image, based on the geometric information from the proxy geometry.

Original: Network Inputs: Using a ﬁxed projection layer, we back-project the input depth image D i to world space using the intrinsic and extrinsic camera parameters that are known from the photogrammetric reconstruction.
Revised: Network Inputs: Using a ﬁxed projection layer, we back-project the input depth image D i to world space using the intrinsic and extrinsic camera parameters that are known from the photogrammetric reconstruction.

Original: Based on this position map we generate normal maps via ﬁnite differences as well as a map of the reﬂected viewing directions.
Revised: Based on this position map we generate normal maps via ﬁnite differences as well as a map of the reﬂected viewing directions.

Original: These inputs are inspired by the Phong illumination model (Phong, 1975) and are stacked along the dimension of the channels.
Revised: These inputs are inspired by the Phong illumination model (Phong, 1975) and are stacked along the dimension of the channels.

Original: Note, the network input is only dependent on the geometry and the current camera parameters, i.e., the view.
Revised: Note, the network input is only dependent on the geometry and the current camera parameters, i.e., the view.

Original: Thus, it can not only be applied toexisting depth images, but also to new target views.
Revised: Thus, it can also be applied to new target views based on the rendered depth of the proxy geometry.

Original: Network Architecture:
Revised: Network Architecture:

Original: Our network Φ is an encoder-decoder network with skip connections, similar to U-Net (Ronneberger et al., 2015b).
Revised: Our network Φ is an encoder-decoder network with skip connections, similar to U-Net (Ronneberger et al., 2015b).

Original: The skip connections can directly propagate lowlevel features to the decoder.
Revised: The skip connections can directly propagate lowlevel features to the decoder.

Original: The encoder is based on 6 convolution layers (kernel size 4 and stride 2 ).
Revised: The encoder is based on 6 convolution layers (kernel size 4 and stride 2 ).

Original: The convolution layers output 32 , 32 , 64 , 128 , 256 and 512 -dimensional feature maps, respectively.
Revised: The convolution layers output 32 , 32 , 64 , 128 , 256 and 512 -dimensional feature maps, respectively.

Original: We use the ReLU activation function and normalize activations based on batchnorm.
Revised: We use the ReLU activation function and normalize activations based on batchnorm.

Original: The decoder mirrors the encoder.
Revised: The decoder mirrors the encoder.

Original: We use transposed convolutions (kernel size 4 and stride 2 ) with the same number of feature channels as in the respective encoder layer.
Revised: We use transposed convolutions (kernel size 4 and stride 2 ) with the same number of feature channels as in the respective encoder layer.

Original: As ﬁnal layer we use a 4 × 4 -convolution with a stride of 1 that outputs a 3 -dimensional tensor that is fed to a Sigmoid to generate an image of the view-dependent illumination effects.
Revised: As ﬁnal layer we use a 4 × 4 -convolution with a stride of 1 that outputs a 3 -dimensional tensor that is fed to a Sigmoid to generate an image of the view-dependent illumination effects.

Original: Self-supervised Training: Since we assume constant illumination, the diffuse light reﬂected by a surface point is the same in every image, thus, the appearance of a surface point only changes by the view-dependent components.
Revised: Self-supervised Training: Since we assume constant illumination, the diffuse light reﬂected by a surface point is the same in every image, thus, the appearance of a surface point only changes by the view-dependent components.

Original: We train our network in a self-supervised manner based on a Siamese network that predicts the view-dependent effects of two random views such that the difference of the diffuse aligned images is minimal (see Fig. 2).
Revised: We train our network in a self-supervised manner based on a Siamese network that predicts the view-dependent effects of two random views such that the difference of the diffuse aligned images is minimal (see Fig. 2).

Original: To this end, we use the re-projection ability (see
Revised: To this end, we use the re-projection ability (see

Original: Sec. 6) to align pairs of input images, from which the view-dependent effects have been removed (original image minus view-dependent effects), and train the network to minimize the resulting differences in the overlap region of the two images.
Revised: Sec. 6) to align pairs of input images, from which the view-dependent effects have been removed (original image minus view-dependent effects), and train the network to minimize the resulting differences in the overlap region of the two images.

Original: Given a randomly selected training pair ( I p , I q ) and let Φ Θ ( X t ) , t ∈ { p, q } denote the output of the two Siamese towers.
Revised: Given a randomly selected training pair ( I p , I q ) and let Φ Θ ( X t ) , t ∈ { p, q } denote the output of the two Siamese towers.

Original: Then, our self-supervised loss for this training sample can be expressed as:
Revised: Then, our self-supervised loss for this training sample can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: Here, ◦ denotes the Hadamard product, Θ are the parameters of the encoder-decoder network Φ , which is shared between the two towers.
Revised: Here, ◦ denotes the Hadamard product, Θ are the parameters of the encoder-decoder network Φ , which is shared between the two towers.

Original: M is a binary mask that is set to one if a surface point is visible in both views and zero otherwise.
Revised: M is a binary mask that is set to one if a surface point is visible in both views and zero otherwise.

Original: We regularize the estimated view-dependent effects to be small w.r.t.
Revised: We regularize the estimated view-dependent effects to be small w.r.t.

Original: an (cid:96) 1 -norm.
Revised: an (cid:96) 1 -norm.

Original: This regularizer is weighted with 0 .
Revised: This regularizer is weighted with 0 .

Original: 01 in our experiments.
Revised: 01 in our experiments.

Original: The crossprojection W pq from image p to image q is based on the geometric proxy.
Revised: The crossprojection W pq from image p to image q is based on the geometric proxy.

Original:  I MAGE - GUIDED R ENDERING P IPELINE
Revised:  I MAGE - GUIDED R ENDERING P IPELINE

Original: To generate a novel target view, we select a subset of K = 4 images based on a coverage-based nearest neighbor search in the set of reference views ( n = 20 ).
Revised: To generate a novel target view, we select a subset of K = 4 images based on a coverage-based nearest neighbor search in the set of reference views ( n = 20 ).

Original: We use EffectsNet to estimate the view-dependent effects of these views, to compute diffuse images.
Revised: We use EffectsNet to estimate the view-dependent effects of these views, to compute diffuse images.

Original: Each diffuse image is crossprojected to the target view, based on the depth maps of the proxy geometry.
Revised: Each diffuse image is crossprojected to the target view, based on the depth maps of the proxy geometry.

Original: Since the depth map of the target view is known, we are able to predict the view-dependent effects in the target image space.
Revised: Since the depth map of the target view is known, we are able to predict the view-dependent effects in the target image space.

Original: After adding these new effects to the reprojected diffuse images, we give these images as input to our composition network CompositionNet (see Sec. 6).
Revised: After adding these new effects to the reprojected diffuse images, we give these images as input to our composition network CompositionNet (see Sec. 6).

Original: CompositionNet fuses the information of the nearest neighbor images into a single output image.
Revised: CompositionNet fuses the information of the nearest neighbor images into a single output image.

Original: In the following, we describe the coveragebased sampling and the cross-projection, and we show how to use our EffectsNet to achieve a robust re-projection of the view-dependent effects.
Revised: In the following, we describe the coveragebased sampling and the cross-projection, and we show how to use our EffectsNet to achieve a robust re-projection of the view-dependent effects.

Original: Coverage-based View Selection
Revised: Coverage-based View Selection

Original: The selection of the K nearest neighbor frames is based on surface coverage w.r.t.
Revised: The selection of the K nearest neighbor frames is based on surface coverage w.r.t.

Original: the target view.
Revised: the target view.

Original: The goal is to have maximum coverage of the target view to ensure that texture information for the entire visible geometry is cross-projected.
Revised: The goal is to have maximum coverage of the target view to ensure that texture information for the entire visible geometry is cross-projected.

Original: View selection is cast as an iterative process based on a greedy selection strategy that locally maximizes surface coverage.
Revised: View selection is cast as an iterative process based on a greedy selection strategy that locally maximizes surface coverage.

Original: To this end, we start with 64 × 64 sample points on a uniform grid on the target view.
Revised: To this end, we start with 64 × 64 sample points on a uniform grid on the target view.

Original: In each iteration step, we search the view that has the largest overlap with the currently ‘uncovered’ region in the target view.
Revised: In each iteration step, we search the view that has the largest overlap with the currently ‘uncovered’ region in the target view.

Original: We determine this view by cross-projecting the samples from the target view to the captured images, based on the reconstructed proxy geometry and camera parameters.
Revised: We determine this view by cross-projecting the samples from the target view to the captured images, based on the reconstructed proxy geometry and camera parameters.

Original: A sample point in the target view is considered as covered, if it is also visible from the other view point, where visibility is determined based on an occlusion check.
Revised: A sample point in the target view is considered as covered, if it is also visible from the other view point, where visibility is determined based on an occlusion check.

Original: Each sample point that is covered by the ﬁnally selected view is invalidated for the next iteration steps.
Revised: Each sample point that is covered by the ﬁnally selected view is invalidated for the next iteration steps.

Original: This procedure is repeated until the K best views have been selected.
Revised: This procedure is repeated until the K best views have been selected.

Original: To keep processing time low, we restrict this search to a small subset of the input images.
Revised: To keep processing time low, we restrict this search to a small subset of the input images.

Original: This set of reference images is taken from the training corpus and contains n = 20 images.
Revised: This set of reference images is taken from the training corpus and contains n =images.

Original: We chose these views also based on a coverage-based selection scheme based on all frames of the training corpus.
Revised: We chose these views also based on the coverage-based selection scheme described above. I.e., we choose the views with most (unseen) coverage among all views in an iterative manner.

Original: Note that this selection is done in a pre-processing step and is independent of the test phase.
Revised: Note that this selection is done in a pre-processing step and is independent of the test phase.

Original: Proxy-based Cross-projection
Revised: Proxy-based Cross-projection

Original: We model the cross-projection W pq from image p to image q based on the reconstructed geometric proxy and the camera parameters.
Revised: We model the cross-projection W pq from image p to image q based on the reconstructed geometric proxy and the camera parameters.

Original: Let K p ∈ R 4 × 3 denote the matrix of intrinsic parameters and T p =
Revised: Let K p ∈ R 4 × 3 denote the matrix of intrinsic parameters and T p =

Original: [ R p | t p ] ∈ R 4 × 4 the matrix of extrinsic parameters of view p .
Revised: [ R p | t p ] ∈ R 4 × 4 the matrix of extrinsic parameters of view p .

Original: A similar notation holds for view q .
Revised: A similar notation holds for view q .

Original: Then, a homogeneous 2D screen space point s p
Revised: Then, a homogeneous 2D screen space point s p

Original: =
Revised: =

Original: ( u, v, d ) T ∈ Rin view p , with depth being d , can be mapped to screen space of view q by: s
Revised: ( u, v, d ) T ∈ Rin view p , with depth being d , can be mapped to screen space of view q by: s

Original: q = W pq
Revised: q = W pq

Original: ( s p ) , with
Revised: ( s p ) , with

Original: W pq ( s p ) =
Revised: W pq ( s p ) =

Original: K q T q T − 1 p K − 1 p s p .
Revised: K q T q T − 1 p K − 1 p s p .

Original: We employ this mapping to cross-project color information from a source view to a novel target view.
Revised: We employ this mapping to cross-project color information from a source view to a novel target view.

Original: To this end, we map every valid pixel (with a depth estimate) from the target view to the source view.
Revised: To this end, we map every valid pixel (with a depth estimate) from the target view to the source view.

Original: The color information from the source view is sampled based on bilinear interpolation.
Revised: The color information from the source view is sampled based on bilinear interpolation.

Original: Projected points that are occluded in the source view or are not in the view frustum are invalidated.
Revised: Projected points that are occluded in the source view or are not in the view frustum are invalidated.

Original: Occlusion is determined by a depth test w.r.t.
Revised: Occlusion is determined by a depth test w.r.t.

Original: the source depth map.
Revised: the source depth map.

Original: Applying the cross-projection to the set of all nearest neighbor images, we get multiple images that match the novel target view point.
Revised: Applying the cross-projection to the set of all nearest neighbor images, we get multiple images that match the novel target view point.

Original: View-dependent Effects Image-based rendering methods often have problems with the reprojection of view-dependent effects (see Sec. 7).
Revised: View-dependent Effects Image-based rendering methods often have problems with the reprojection of view-dependent effects (see Sec. 7).

Original: In our image-guided pipeline, we solve this problem using EffectsNet .
Revised: In our image-guided pipeline, we solve this problem using EffectsNet .

Original: Before re-projection, we estimate the view-dependent effects from the input images and subtract them.
Revised: Before re-projection, we estimate the view-dependent effects from the input images and subtract them.

Original: By this, view-dependent effects are excluded from warping.
Revised: By this, view-dependent effects are excluded from warping.

Original: Viewdependent effects are then re-inserted after re-projection, again using EffectsNet based on the target view depth map.
Revised: Viewdependent effects are then re-inserted after re-projection, again using EffectsNet based on the target view depth map.

Original: CompositionNet : Image compositing The warped nearest views are fused using a deep neural network called CompositionNet .
Revised: CompositionNet : Image compositing The warped nearest views are fused using a deep neural network called CompositionNet .

Original: Similar to the EffectsNet , our CompositionNet is an encoderdecoder network with skip connections.
Revised: Similar to the EffectsNet , our CompositionNet is an encoderdecoder network with skip connections.

Original: The network input is a tensor that stacks the K warped views, the corresponding warp ﬁelds as well as the target position map along the dimension of the channels and the output is a three channel RGB image.
Revised: The network input is a tensor that stacks the K warped views, the corresponding warp ﬁelds as well as the target position map along the dimension of the channels and the output is a three channel RGB image.

Original: The encoder is based on 6 convolution layers (kernel size 4 and stride 2 ) with 64 , 64 , 128 , 128 , 256 and 256 -dimensional feature maps, respectively.
Revised: The encoder is based on 6 convolution layers (kernel size 4 and stride 2 ) with 64 , 64 , 128 , 128 , 256 and 256 -dimensional feature maps, respectively.

Original: The activation functions are leaky ReLUs (negative slope of 0.2) in the encoder and ReLUs in the decoder.
Revised: The activation functions are leaky ReLUs (negative slope of 0.2) in the encoder and ReLUs in the decoder.

Original: In both cases, we normalize all activations based on batchnorm.
Revised: In both cases, we normalize all activations based on batchnorm.

Original: The decoder mirrors the encoder.
Revised: The decoder mirrors the encoder.

Original: We use transposed convolutions (kernel size 4 and stride 2 ) with the same number of feature channels as in the respective encoder layer.
Revised: We use transposed convolutions (kernel size 4 and stride 2 ) with the same number of feature channels as in the respective encoder layer.

Original: As ﬁnal layer we use a 4 × 4 -convolution with a stride of 1 and a Sigmoid activation function that outputs the ﬁnal image.
Revised: As ﬁnal layer we use a 4 × 4 -convolution with a stride of 1 and a Sigmoid activation function that outputs the ﬁnal image.

Original: We are using an (cid:96) 1 -loss and an additional adversarial loss to measure the difference between the predicted output images and the ground truth data.
Revised: We are using an (cid:96) 1 -loss and an additional adversarial loss to measure the difference between the predicted output images and the ground truth data.

Original: The adversarial loss is based on the conditional
Revised: The adversarial loss is based on the conditional

Original: PatchGAN loss that is also used in Pix2Pix (Isola et al., 2016).
Revised: PatchGAN loss that is also used in Pix2Pix (Isola et al., 2016).

Original: In our experiments, we are weighting the adversarial loss with a factor of 0 .
Revised: In our experiments, we are weighting the adversarial loss with a factor of 0 .

Original: 01 and the (cid:96) 1 -loss with a factor of 1 . 0 .
Revised: 01 and the (cid:96) 1 -loss with a factor of 1 . 0 .

Original: Training For each object, both networks are trained independently using the Adam optimizer (Kingma & Ba, 2014) built into Tensorﬂow (Abadi et al., 2015).
Revised: Training Per object, both networks are trained independently using the Adam optimizer (Kingma & Ba, 2014) built into Tensorﬂow (Abadi et al., 2015).

Original: Each network is trained for 64 epochs with a learning rate of 0 .
Revised: Each network is trained for 64 epochs with a learning rate of 0 .

Original: 001 and the default parameters β 1 = 0 .
Revised: 001 and the default parameters β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , (cid:15) = 1 · e − 8 .
Revised: 999 , (cid:15) = 1 · e − 8 .

Original:  R ESULTS
Revised:  R ESULTS

Original: The main contribution of our work is to combine the beneﬁts of IBR and 2D GANs, a hybrid that is able to generate temporally-stable view changes including view-dependent effects.
Revised: The main contribution of our work is to combine the beneﬁts of IBR and 2D GANs, a hybrid that is able to generate temporally-stable view changes including view-dependent effects.

Original: We analyze our approach both qualitatively and quantitatively, and show comparisons to IBR as well as to 2D GAN-based methods.
Revised: We analyze our approach both qualitatively and quantitatively, and show comparisons to IBR as well as to 2D GAN-based methods.

Original: For all experiments we used K = 4 views per frame selected from n = 20 reference views.
Revised: For all experiments we used K = 4 views per frame selected from n =reference views.

Original: An overview of all image reconstruction errors is given in Tab. 1 in the appendix.
Revised: An overview of all image reconstruction errors is given in Tab. 1 in the appendix.

Original: The advantages of our approach can best be seen in the supplemental video, especially, the temporal coherence ( https://www.dropbox.com/sh/8pi37qlfqoukthd/
Revised: The advantages of our approach can best be seen in the supplemental video, especially, the temporal coherence.

Original: AAB0sdQT79noBRtKBKWCywtJa?dl=0 ).

Original: Using synthetic data we are quantitatively analyzing the performance of our image-based rendering approach.
Revised: Using synthetic data we are quantitatively analyzing the performance of our image-based rendering approach.

Original: We refer to the Appendix A.2.1 for a detailed ablation study w.r.t.
Revised: We refer to the Appendix A.2.1 for a detailed ablation study w.r.t.

Original: the training corpus and comparisons to classical and learned image-based rendering techniques.
Revised: the training corpus and comparisons to classical and learned image-based rendering techniques.

Original: Since the EffectsNet is a core component of our algorithm, we compare our technique with and without EffectsNet (see
Revised: Since the EffectsNet is a core component of our algorithm, we compare our technique with and without EffectsNet (see

Original: Fig. 3) using synthetic data.
Revised: Fig. 3) using synthetic data.

Original: The full pipeline results in smoother specular highlights and sharper details.
Revised: The full pipeline results in smoother specular highlights and sharper details.

Original: On the test set the MSE without EffectsNet is 2 .
Revised: On the test set the MSE without EffectsNet is 2 .

Original: 6876 versus 2 .
Revised: 6876 versus 2 .

Original: 3864 with EffectsNet .
Revised: 3864 with EffectsNet .

Original: The following experiments are conducted on real data.
Revised: The following experiments are conducted on real data.

Original: Fig.
Revised: Fig.

Original: 4 shows the effectiveness of our EffectsNet to estimate the specular effects in an image.
Revised: 4 shows the effectiveness of our EffectsNet to estimate the specular effects in an image.

Original: The globe has a specular surface and reﬂects the ceiling lights.
Revised: The globe has a specular surface and reﬂects the ceiling lights.

Original: These specular highlights are estimated and removed from the original image of the object which results in a diffuse image of the object.
Revised: These specular highlights are estimated and removed from the original image of the object which results in a diffuse image of the object.

Original: In Fig.
Revised: In Fig.

Original: 5 we show a comparison to Pix2Pix trained on position maps.
Revised: 5 we show a comparison to Pix2Pix trained on position maps.

Original: Similar to the synthetic experiments in the appendix, our method results in higher quality.
Revised: Similar to the synthetic experiments in the appendix, our method results in higher quality.

Original: We also compare to the state-of-the-art image-based rendering approach of Hedman et al.
Revised: We also compare to the state-of-the-art image

Original: The idea of this technique is to use a neural network to predict blending weights for a image-based rendering composition like InsideOut (Hedman et al., 2016).
Revised: The idea of this technique is to use a neural network to predict blending weights for a image-based rendering composition like InsideOut (Hedman et al., 2016).

Original: Note that this method uses the per frame reconstructed depth maps of 20 reference frames in addition to the fused 3D mesh.
Revised: Note that this method uses the per frame reconstructed depth maps of 20 reference frames in addition to the fused 3D mesh.

Original: As can be seen in Fig.
Revised: As can be seen in Fig.

Original: 6, the results are of high-quality, achieving an MSE of 45 .
Revised: 6, the results are of high-quality, achieving an MSE of 45 .

Original: 07 for DeepBlending and an MSE of 51 .
Revised: 07 for DeepBlending and an MSE of 51 .

Original: 17 for InsideOut.
Revised: 17 for InsideOut.

Original: Our object-speciﬁc rendering results in an error of 25 . 24 .
Revised: Our object-speciﬁc rendering results in an error of 25 . 24 .

Original: Both methods of Hedman et al.
Revised: Both methods of Hedman et al.

Original: , do not explicitly handle the correction of view-dependent effects.
Revised: , do not explicitly handle the correction of view-dependent effects.

Original: In contrast, our approach uses EffectsNet to remove the view-dependent effect in the source views (thus, enabling the projection to a different view) and to add new view-dependent effect in the target view.
Revised: In contrast, our approach uses EffectsNet to remove the view-dependent effect in the source views (thus, enabling the projection to a different view) and to add new view-dependent effect in the target view.

Original: This can be seen in the bottom row of Fig.
Revised: This can be seen in the bottom row of Fig.

Original: 6, where we computed the quotient between reconstruction and ground truth showing the shading difference.
Revised: 6, where we computed the quotient between reconstruction and ground truth showing the shading difference.

Original:  L IMITATIONS
Revised:  L IMITATIONS

Original: Our approach is trained in an object speciﬁc manner.
Revised: Our approach is trained in an object speciﬁc manner.

Original: This is a limitation of our method, but also ensures the optimal results that can be generated using our architecture.
Revised: This is a limitation of our method, but also ensures the optimal results that can be generated using our architecture.

Original: Since the multi-view stereo reconstruction of an object is an ofﬂine algorithm that takes about an hour, we think that training the object speciﬁc networks ( EffectsNet and CompositionNet ) that takes a similar amount of time is practically feasible.
Revised: Since the multi-view stereo reconstruction of an object is an ofﬂine algorithm that takes about an hour, we think that training the object speciﬁc networks ( EffectsNet and CompositionNet ) that takes a similar amount of time is practically feasible.

Original: The training of these networks can be seen as reconstruction reﬁnement that also includes the appearance of the object.
Revised: The training of these networks can be seen as reconstruction reﬁnement that also includes the appearance of the object.

Original: At test time our approach runs at interactive rates, the inference time of EffectsNet is 50 Hz , while CompositionNet runs at 10 Hz on an Nvidia 1080Ti.
Revised: At test time our approach runs at interactive rates, the inference time of EffectsNet is 50 Hz, while CompositionNet runs at 10 Hz on an Nvidia 1080Ti.

Revised: Note that our approach fails, when the stereo reconstruction fails.

Original: Similar to other learning based approaches, the method is relying on a reasonable large training dataset.
Revised: Similar to other learning based approaches, the method is relying on a reasonable large training dataset.

Original: In the appendix, we conducted an ablation study regarding the dataset size, where our approach gracefully degenerates while a pure learning-based approach shows strong artifacts.
Revised: In the appendix, we conducted an ablation study regarding the dataset size, where our approach gracefully degenerates while a pure learning-based approach shows strong artifacts.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a novel image-guided rendering approach that outputs photo-realistic images of an object.
Revised: In this paper, we propose a novel image-guided rendering approach that outputs photo-realistic images of an object.

Original: We demonstrate the effectiveness of our method in a variety of experiments.
Revised: We demonstrate the effectiveness of our method in a variety of experiments.

Original: The comparisons to competing methods show on-par or even better results, especially, in the presence of view-dependent effects that can be handled using our EffectsNet .
Revised: The comparisons to competing methods show on-par or even better results, especially, in the presence of view-dependent effects that can be handled using our EffectsNet .

Original: We hope to inspire follow-up work in self-supervised re-rendering using deep neural networks.
Revised: We hope to inspire follow-up work in self-supervised re-rendering using deep neural networks.

Original: A. A DDITIONAL R ESULTS
Revised: A CKNOWLEDGEMENTS

Revised: We thank Artec3D 1 for providing scanned 3D models and Angela Dai for the video voice over.

Revised: This work is funded by a Google Research Grant, supported by the ERC Starting Grant Scan2CAD (804724), ERC Starting Grant CapReal (335545), and the ERC Consolidator Grant 4DRepLy (770784), the Max Planck Center for Visual Computing and Communication (MPC-VCC), a TUMIAS Rudolf M¨oßbauer Fellowship (Focus Group Visual Computing), and a Google Faculty Award.

Revised: In addition, this work is funded by Sony and the German Research Foundation (DFG)

Revised: Grant Making

Revised: Machine Learning on Static and Dynamic 3D Data Practical , and supported by Nvidia.

Revised: A. A DDITIONAL R ESULTS

Original: An overview of all image reconstruction errors of all sequences used in this paper is given in Tab. 1.
Revised: An overview of all image reconstruction errors of all sequences used in this paper is given in Tab. 1.

Original: MSE values are reported w.r.t.
Revised: MSE values are reported w.r.t.

Original: a color range of [0 , 255] .
Revised: a color range of [0 , 255] .

Original: Note that the synthetic data contain ground truth depth maps, while for the real data, we are relying on reconstructed geometry.
Revised: Note that the synthetic data contain ground truth depth maps, while for the real data, we are relying on reconstructed geometry.

Original: This is also reﬂected in the photo-metric error (higher error due to misalignment).
Revised: This is also reﬂected in the photo-metric error (higher error due to misalignment).

Original: Sequence
Revised: Sequence

Original: N-IBR
Revised: N-IBR

Original: IBR (Debevec et al.
Revised: IBR (Debevec et al.

Original: Pix2Pix (Isola et al.
Revised: Pix2Pix (Isola et al.

Original: Ours
Revised: Ours

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 1, Car [Table] [Table]
Revised: 1, Car 72.

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 3, Statue 
Revised: 3, Statue 44.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 11, Vase [Table] [Table] [Table]
Revised: 11, Vase 38.

Original: Fig.
Revised: Fig.

Original: 12, Bust
Revised: 12, Bust

Original: [Table]
Revised: [Table]

Original: Fig. 4, Globe 152.
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: ea

Original: [Table]
Revised: [Table]

Revised: Fig. 4, Globe 152.

Original: Fig.
Revised: Fig.

Original: 14, Shoe 
Revised: 14, Shoe 98.

Original: [Table] [Table] [Table] [Table]
Revised: Fig

Original: [Table]
Revised: [Table]

Original: A.2. A DDITIONAL E XPERIMENTS ON S YNTHETIC D ATA
Revised: A.2. A DDITIONAL E XPERIMENTS ON S YNTHETIC D ATA

Original: Using synthetic data we are quantitatively analyzing the performance of our image-based rendering approach.
Revised: Using synthetic data we are quantitatively analyzing the performance of our image-based rendering approach.

Original: EffectsNet In Fig.
Revised: EffectsNet In Fig.

Original: 9, we show a qualitative comparison of our predicted diffuse texture to the ground truth.
Revised: 9, we show a qualitative comparison of our predicted diffuse texture to the ground truth.

Original: The ﬁgure shows the results for a Phong rendering sequence.
Revised: The ﬁgure shows the results for a Phong rendering sequence.

Original: As can be seen, the estimated diffuse image is close to the ground truth.
Revised: As can be seen, the estimated diffuse image is close to the ground truth.

Original: Comparison to Image-based Rendering We compare our method to two baseline image-based rendering approaches.
Revised: Comparison to Image-based Rendering We compare our method to two baseline image-based rendering approaches.

Original: A na¨ıve IBR method that uses the nearest neighbor of our method and computes a per pixel average of the re-projected views, and the IBR method of Debevec et al.
Revised: A na¨ıve IBR method that uses the nearest neighbor of our method and computes a per pixel average of the re-projected views, and the IBR method of Debevec et al.

Original: (1998) that uses all reference views and a per triangle view selection.
Revised: (1998) that uses all reference views and a per triangle view selection.

Original: In contrast to our method, the classical
Revised: In contrast to our method, the classical

Original: IBR techniques are not reproducing view-dependent effects as realistically and smoothly which can be seen in Fig.
Revised: IBR techniques are not reproducing view-dependent effects as realistically and smoothly which can be seen in Fig.

Original: As can be seen the na¨ıve IBR method also suffers from occluded regions.
Revised: As can be seen the na¨ıve IBR method also suffers from occluded regions.

Original: Our method is able to in-paint these regions.
Revised: Our method is able to in-paint these regions.

Original: Comparison to Learned Image Synthesis To demonstrate the advantage of our approach, we also compare to an image-to-image translation baseline ( Pix2Pix (Isola et al., 2016)).
Revised: Comparison to Learned Image Synthesis To demonstrate the advantage of our approach, we also compare to an image-to-image translation baseline ( Pix2Pix (Isola et al., 2016)).

Original: Pix2Pix is trained to translate position images into color images of the target object.
Revised: Pix2Pix is trained to translate position images into color images of the target object.

Original: While it is not designed for this speciﬁc task, we want to show that it is in fact able to produce individual images that look realistic (Fig. 11); however, it is unable to produce a temporally-coherent video.
Revised: While it is not designed for this speciﬁc task, we want to show that it is in fact able to produce individual images that look realistic (Fig. 11); however, it is unable to produce a temporally-coherent video.

Original: On our test set with 190 images, our method has a MSE of 1 . 12 while Pix2Pix results in a higher MSE of 18 . 66 .
Revised: On our test set with 190 images, our method has a MSE of 1 . 12 while Pix2Pix results in a higher MSE of 18 . 66 .

Original: Pix2Pix trained on pure depth maps from our training set results in a MSE of 36 . 63 since the input does not explicitly contain view information.
Revised: Pix2Pix trained on pure depth maps from our training set results in a MSE of 36 . 63 since the input does not explicitly contain view information.

Original: Evaluation of Training Corpus Size In Fig.
Revised: Evaluation of Training Corpus Size In Fig.

Original: 12 we show the inﬂuence of the training corpus size on the quality of the results.
Revised: 12 we show the inﬂuence of the training corpus size on the quality of the results.

Original: While our method handles the reduction of the training data size well, the performance of Pix2Pix drastically decreases leading to a signiﬁcantly higher MSE.
Revised: While our method handles the reduction of the training data size well, the performance of Pix2Pix drastically decreases leading to a signiﬁcantly higher MSE.

Original: When comparing these results to the results in Fig 11 it becomes evident that Pix2Pix has a signiﬁcantly lower error on the bust sequence than on the vase sequence.
Revised: When comparing these results to the results in Fig 11 it becomes evident that Pix2Pix has a signiﬁcantly lower error on the bust sequence than on the vase sequence.

Original: The vase has much more details than the bust and, thus, is harder to reproduce.
Revised: The vase has much more details than the bust and, thus, is harder to reproduce.

Original: A.2. A DDITIONAL E XPERIMENTS ON R EAL D ATA
Revised: A.2. A DDITIONAL E XPERIMENTS ON R EAL D ATA

Original: Comparison to Texture-based Rendering Nowadays, most reconstruction frameworks like
Revised: Comparison to Texture-based Rendering Nowadays, most reconstruction frameworks like

Original: COLMAP (Sch¨onberger & Frahm, 2016), KinectFusion (Izadi et al., 2011), or VoxelHashing (Nießner et al., 2013) output a mesh with per-vertex colors or with a texture, which is the de facto standard in computer graphics.
Revised: COLMAP (Sch¨onberger & Frahm, 2016), KinectFusion (Izadi et al., 2011), or VoxelHashing (Nießner et al., 2013) output a mesh with per-vertex colors or with a texture, which is the de facto standard in computer graphics.

Original: Fig.
Revised: Fig.

Original: 13 shows a side-by-side comparison of our method and the rendering using per-vertex colors as well as using a static texture.
Revised: 13 shows a side-by-side comparison of our method and the rendering using per-vertex colors as well as using a static texture.

Original: Since both the vertex colors as well as the texture are static, these approaches are not able to capture the view-dependent effects.
Revised: Since both the vertex colors as well as the texture are static, these approaches are not able to capture the view-dependent effects.

Original: Thus, view-dependent effects are baked into the vertex colors or texture and stay ﬁxed (seen close-ups in Fig. 13).
Revised: Thus, view-dependent effects are baked into the vertex colors or texture and stay ﬁxed (seen close-ups in Fig. 13).

Original: Comparison to Image-based Rendering Fig.
Revised: Comparison to Image-based Rendering Fig.

Original: 14 shows a comparison of our method to imagebased rendering.
Revised: 14 shows a comparison of our method to imagebased rendering.

Original: The IBR method of Debevec et al.
Revised: The IBR method of Debevec et al.

Original: (1998) uses a per triangle based view selection which leads to artifacts, especially, in regions with specular reﬂections.
Revised: (1998) uses a per triangle based view selection which leads to artifacts, especially, in regions with specular reﬂections.

Original: Our method is able to reproduce these specular effects.
Revised: Our method is able to reproduce these specular effects.

Original: Note, you can also see that our result is sharper than the ground truth (motion blur), because the network reproduces the appearance of the training corpus and most images do not contain motion blur.
Revised: Note, you can also see that our result is sharper than the ground truth (motion blur), because the network reproduces the appearance of the training corpus and most images do not contain motion blur.