Original: I S
Revised: I S

Original: A TTENTION
Revised: A TTENTION

Original: A LL T HAT N E RF N EEDS ?
Revised: A LL T HAT N E RF N EEDS ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present Generalizable NeRF Transformer ( GNT ), a transformer-based architecture that reconstructs Neural Radiance Fields (NeRFs) and learns to renders novel views on the fly from source views.
Revised: We present Generalizable NeRF Transformer ( GNT ), a transformer-based architecture that reconstructs Neural Radiance Fields (NeRFs) and learns to renders novel views on the fly from source views.

Original: While prior works on NeRFs optimize a scene representation by inverting a handcrafted rendering equation, GNT achieves neural representation and rendering that generalizes across scenes using transformers at two stages.
Revised: While prior works on NeRFs optimize a scene representation by inverting a handcrafted rendering equation, GNT achieves neural representation and rendering that generalizes across scenes using transformers at two stages.

Original: (1) The view transformer leverages multi-view geometry as an inductive bias for attention-based scene representation, and predicts coordinatealigned features by aggregating information from epipolar lines on the neighboring views.
Revised: (1) The view transformer leverages multi-view geometry as an inductive bias for attention-based scene representation, and predicts coordinatealigned features by aggregating information from epipolar lines on the neighboring views.

Original: The ray transformer renders novel views using attention to decode the features from the view transformer along the sampled points during ray marching.
Revised: The ray transformer renders novel views using attention to decode the features from the view transformer along the sampled points during ray marching.

Original: Our experiments demonstrate that when optimized on a single scene, GNT can successfully reconstruct NeRF without an explicit rendering formula due to the learned ray renderer.
Revised: Our experiments demonstrate that when optimized on a single scene, GNT can successfully reconstruct NeRF without an explicit rendering formula due to the learned ray renderer.

Original: When trained on multiple scenes, GNT consistently achieves state-of-the-art performance when transferring to unseen scenes and outperform all other methods by ~ 10% on average.
Revised: When trained on multiple scenes, GNT consistently achieves state-of-the-art performance when transferring to unseen scenes and outperform all other methods by ~ 10% on average.

Original: Our analysis of the learned attention maps to infer depth and occlusion indicate that attention enables learning a physicallygrounded rendering.
Revised: Our analysis of the learned attention maps to infer depth and occlusion indicate that attention enables learning a physicallygrounded rendering.

Original: Our results show the promise of transformers as a universal modeling tool for graphics.
Revised: Our results show the promise of transformers as a universal modeling tool for graphics.

Original: Code will be made publicly available.
Revised: Code will be made publicly available.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural Radiance Field (NeRF) (Mildenhall et al., 2020) and its follow-up works (Barron et al., 2021;
Revised: Neural Radiance Field (NeRF) (Mildenhall et al., 2020) and its follow-up works (Barron et al., 2021;

Original: Zhang et al., 2020; Chen et al., 2022) have achieved remarkable success on novel view synthesis, generating photo-realistic, high-resolution, and view-consistent scenes.
Revised: Zhang et al., 2020; Chen et al., 2022) have achieved remarkable success on novel view synthesis, generating photo-realistic, high-resolution, and view-consistent scenes.

Original: Two key ingredients in NeRF are, (1) a coordinate-based neural network that maps each spatial position to its corresponding color and density, and (2) a differentiable volumetric rendering pipeline that composes the color and density of points along each ray cast from the image plane to generate the target pixel color.
Revised: Two key ingredients in NeRF are, (1) a coordinate-based neural network that maps each spatial position to its corresponding color and density, and (2) a differentiable volumetric rendering pipeline that composes the color and density of points along each ray cast from the image plane to generate the target pixel color.

Original: Optimizing a
Revised: Optimizing a

Original: NeRF can be regarded as an inverse imaging problem that fits a neural network to satisfy the observed views.
Revised: NeRF can be regarded as an inverse imaging problem that fits a neural network to satisfy the observed views.

Original: Such training leads to a major limitation of NeRF, making it a time-consuming optimization process for each scene (Chen et al., 2021a;
Revised: Such training leads to a major limitation of NeRF, making it a time-consuming optimization process for each scene (Chen et al., 2021a;

Original: Wang et al., 2021b; Yu et al., 2021).
Revised: Wang et al., 2021b; Yu et al., 2021).

Original: Recent works Neuray (Liu et al., 2022), IBRNet (Wang et al., 2021b), and PixelNerf (Yu et al., 2021) go beyond the coordinate-based network and rethink novel view synthesis as a cross-view imagebased interpolation problem.
Revised: Recent works Neuray (Liu et al., 2022), IBRNet (Wang et al., 2021b), and PixelNerf (Yu et al., 2021) go beyond the coordinate-based network and rethink novel view synthesis as a cross-view imagebased interpolation problem.

Original: Unlike the vanilla NeRF that tediously fits each scene, these methods synthesize a generalizable 3D representation by aggregating image features extracted from seen views according to the camera and geometry priors.
Revised: Unlike the vanilla NeRF that tediously fits each scene, these methods synthesize a generalizable 3D representation by aggregating image features extracted from seen views according to the camera and geometry priors.

Original: However, despite showing large performance gains, they unexceptionally decode the feature volume to a radiance field, and rely on classical volume rendering (Max, 1995; Levoy, 1988) to generate images.
Revised: However, despite showing large performance gains, they unexceptionally decode the feature volume to a radiance field, and rely on classical volume rendering (Max, 1995; Levoy, 1988) to generate images.

Original: Note that the volume rendering equation adopted in NeRF over-simplifies the optical modeling of solid surface (Yariv et al., 2021; Wang et al., 2021a), reflectance (Chen et al., 2021c; Verbin et al., 2021; Chen et al., 2022), inter-surface scattering and other effects.
Revised: Note that the volume rendering equation adopted in NeRF over-simplifies the optical modeling of solid surface (Yariv et al., 2021; Wang et al., 2021a), reflectance (Chen et al., 2021c; Verbin et al., 2021; Chen et al., 2022), inter-surface scattering and other effects.

Original: This implies that radiance fields along with volume rendering in NeRF are not a universal imaging model, which may have limited the generalization ability of NeRFs as well.
Revised: This implies that radiance fields along with volume rendering in NeRF are not a universal imaging model, which may have limited the generalization ability of NeRFs as well.

Original: In this paper, we first consider the problem of transferable novel view synthesis as a two-stage information aggregation process: the multi-view image feature fusion, followed by the samplingbased rendering integration.
Revised: In this paper, we first consider the problem of transferable novel view synthesis as a two-stage information aggregation process: the multi-view image feature fusion, followed by the samplingbased rendering integration.

Original: Our key contributions comes from using transformers (Vaswani et al., 2017) for both these stages.
Revised: Our key contributions comes from using transformers (Vaswani et al., 2017) for both these stages.

Original: Transformers have had resounding success in language modeling (Devlin et al., 2018) and computer vision (Dosovitskiy et al., 2020) and their “self-attention” mechanism can be thought of as a universal trainable aggregation function.
Revised: Transformers have had resounding success in language modeling (Devlin et al., 2018) and computer vision (Dosovitskiy et al., 2020) and their “self-attention” mechanism can be thought of as a universal trainable aggregation function.

Original: In our case, for volumetric scene representation, we train a view transformer , to aggregate pixel-aligned image features (Saito et al., 2019) from corresponding epipolar lines to predict coordinate-wise features.
Revised: In our case, for volumetric scene representation, we train a view transformer , to aggregate pixel-aligned image features (Saito et al., 2019) from corresponding epipolar lines to predict coordinate-wise features.

Original: For rendering a novel view, we develop a ray transformer that composes the coordinate-wise point features along a traced ray via attention mechanism.
Revised: For rendering a novel view, we develop a ray transformer that composes the coordinate-wise point features along a traced ray via attention mechanism.

Original: These two form the Generalizable NeRF Transformer ( GNT ).
Revised: These two form the Generalizable NeRF Transformer ( GNT ).

Original: GNT simultaneously learns to represent scenes from source view images and perform scene adaptive ray-based rendering using the learned attention mechanism.
Revised: GNT simultaneously learns to represent scenes from source view images and perform scene adaptive ray-based rendering using the learned attention mechanism.

Original: Remarkably, GNT predicts novel views using the captured images without fitting per-scene.
Revised: Remarkably, GNT predicts novel views using the captured images without fitting per-scene.

Original: Our promising results endorse that transformers are strong, scalable and a versatile learning backbone for graphical rendering (Tewari et al., 2020).
Revised: Our promising results endorse that transformers are strong, scalable and a versatile learning backbone for graphical rendering (Tewari et al., 2020).

Original: Our key contributions are:
Revised: Our key contributions are:

Original: A view transformer to aggregate multi-view image features complying with epipolar geometry and to infer coordinate-aligned features.
Revised: A view transformer to aggregate multi-view image features complying with epipolar geometry and to infer coordinate-aligned features.

Original: A ray transformer for a learned ray-based rendering to predict target color.
Revised: A ray transformer for a learned ray-based rendering to predict target color.

Original: Experiments to demonstrate that GNT’s fully transformer-based architecture achieves stateof-the-art results on complex scenes and cross-scene generalization.
Revised: Experiments to demonstrate that GNT’s fully transformer-based architecture achieves stateof-the-art results on complex scenes and cross-scene generalization.

Original: Analysis of the attention module showing that GNT learns to be depth and occlusion aware.
Revised: Analysis of the attention module showing that GNT learns to be depth and occlusion aware.

Original: Overall, our combined Generalizable NeRF Transformer (GNT) demonstrates that many of the inductive biases that were thought necessary for view synthesis (e.g. persistent 3D model, hard-coded rendering equation) can be replaced with attention/transformer mechanisms.
Revised: Overall, our combined Generalizable NeRF Transformer (GNT) demonstrates that many of the inductive biases that were thought necessary for view synthesis (e.g. persistent 3D model, hard-coded rendering equation) can be replaced with attention/transformer mechanisms.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Transformers (Vaswani et al., 2017) have emerged as a ubiquitous learning backbone that captures long-range correlation for sequential data.
Revised: Transformers (Vaswani et al., 2017) have emerged as a ubiquitous learning backbone that captures long-range correlation for sequential data.

Original: It has shown remarkable success in language understanding (Devlin et al., 2018;
Revised: It has shown remarkable success in language understanding (Devlin et al., 2018;

Original: Dai et al., 2019; Brown et al., 2020), computer vision (Dosovitskiy et al., 2020;
Revised: Dai et al., 2019; Brown et al., 2020), computer vision (Dosovitskiy et al., 2020;

Original: Liu et al., 2021), speech (Gulati et al., 2020) and even protein structure (Jumper et al., 2021) amongst others.
Revised: Liu et al., 2021), speech (Gulati et al., 2020) and even protein structure (Jumper et al., 2021) amongst others.

Original: In computer vision, (Dosovitskiy et al., 2020) were successful in demonstrating Vision
Revised: In computer vision, (Dosovitskiy et al., 2020) were successful in demonstrating Vision

Original: Trainsformers (ViT) for image classification.
Revised: Trainsformers (ViT) for image classification.

Original: Subsequent works extended ViT to other vision tasks, including object detection (Carion et al., 2020), segmentation (Chen et al., 2021b;
Revised: Subsequent works extended ViT to other vision tasks, including object detection (Carion et al., 2020), segmentation (Chen et al., 2021b;

Original: Wang et al., 2021c), video processing (Zhou et al., 2018a; Arnab et al., 2021), and 3D instance processing (Guo et al., 2021; Lin et al., 2021).
Revised: Wang et al., 2021c), video processing (Zhou et al., 2018a; Arnab et al., 2021), and 3D instance processing (Guo et al., 2021; Lin et al., 2021).

Original: In this work we apply transformers for view synthesis by learning to reconstruct neural radiance fields and render novel views.
Revised: In this work we apply transformers for view synthesis by learning to reconstruct neural radiance fields and render novel views.

Original: Neural Radiance Fields (NeRF) introduced by Mildenhall et al.
Revised: Neural Radiance Fields (NeRF) introduced by Mildenhall et al.

Original: (2020) synthesizes consistent and photorealistic novel views by fitting each scene as a continuous 5D radiance field parameterized by an
Revised: (2020) synthesizes consistent and photorealistic novel views by fitting each scene as a continuous 5D radiance field parameterized by an

Original: MLP.
Revised: MLP.

Original: Since then, several works have improved NeRFs further.
Revised: Since then, several works have improved NeRFs further.

Original: For example, Mip-NeRF Barron et al.
Revised: For example, Mip-NeRF Barron et al.

Original: (2021; 2022) efficiently addresses scale of objects in unbounded scenes, Nex (Wizadwongsa et al., 2021) models large view dependent effects, others (Oechsle et al., 2021; Yariv et al., 2021; Wang et al., 2021a) improve the surface representation, extend to dynamic scenes (Park et al., 2021a;b;
Revised: (2021; 2022) efficiently addresses scale of objects in unbounded scenes, Nex (Wizadwongsa et al., 2021) models large view dependent effects, others (Oechsle et al., 2021; Yariv et al., 2021; Wang et al., 2021a) improve the surface representation, extend to dynamic scenes (Park et al., 2021a;b;

Original: Pumarola et al., 2021) , introduce lighting and reflection modeling (Chen et al., 2021c; Verbin et al., 2021), or leverage depth to regress from few views (Xu et al., 2022; Deng et al., 2022).
Revised: Pumarola et al., 2021) , introduce lighting and reflection modeling (Chen et al., 2021c; Verbin et al., 2021), or leverage depth to regress from few views (Xu et al., 2022; Deng et al., 2022).

Original: Our work aims to avoid per-scene training, similar to PixelNeRF (Yu et al., 2021), IBRNet (Wang et al., 2021b),
Revised: Our work aims to avoid per-scene training, similar to PixelNeRF (Yu et al., 2021), IBRNet (Wang et al., 2021b),

Original: MVSNeRF (Chen et al., 2021a), and NeuRay (Liu et al., 2022) which train a cross-scene multi-view aggregator and reconstruct the radiance field with a one-shot forward pass.
Revised: MVSNeRF (Chen et al., 2021a), and NeuRay (Liu et al., 2022) which train a cross-scene multi-view aggregator and reconstruct the radiance field with a one-shot forward pass.

Original: Transformer Meets Radiance Fields.
Revised: Transformer Meets Radiance Fields.

Original: Most similar to our work are NeRF methods that apply transformers for novel view synthesis and generalize across scenes.
Revised: Most similar to our work are NeRF methods that apply transformers for novel view synthesis and generalize across scenes.

Original: IBRNet (Wang et al., 2021b) processes sampled points on the ray using an MLP to predict color values and density features which are then input to a transformer to predict density.
Revised: IBRNet (Wang et al., 2021b) processes sampled points on the ray using an MLP to predict color values and density features which are then input to a transformer to predict density.

Original: Recently, NeRFormer (Reizenstein et al., 2021) and
Revised: Recently, NeRFormer (Reizenstein et al., 2021) and

Original: Wang et al.
Revised: Wang et al.

Original: (2022) use attention module to aggregate source views to construct feature volume with epipolar geometry constraints.
Revised: (2022) use attention module to aggregate source views to construct feature volume with epipolar geometry constraints.

Original: However, a key difference with our work is that, all of them decode the latent feature representation to point-wise color and density, and rely on classic volume rendering to form the image, while our ray transformer learns to render the target pixel directly.
Revised: However, a key difference with our work is that, all of them decode the latent feature representation to point-wise color and density, and rely on classic volume rendering to form the image, while our ray transformer learns to render the target pixel directly.

Original: Other works that use transformers but differ significantly from our approach and in application include (Lin et al., 2022) which generates novel views from just a single image via a vision transformer and
Revised: Other works that use transformers but differ significantly from our approach and in application include (Lin et al., 2022) which generates novel views from just a single image via a vision transformer and

Original: SRT (Sajjadi et al., 2022b) which treats images and camera parameters in a latent space and trains
Revised: SRT (Sajjadi et al., 2022b) which treats images and camera parameters in a latent space and trains

Original: Ray Transformer
Revised: Ray Transformer

Original: [Figure]
Revised: [Figure]

Original: P oo li ng
Revised: P oo li ng

Original: Pred.
Revised: Pred.

Original: GT
Revised: GT

Original: Photometric Loss
Revised: Photometric Loss

Original: [Figure]
Revised: [Figure]

Original: Target Views
Revised: Target Views

Original: SourceViews
Revised: SourceViews

Original: E N C OD E R D E C OD E R
Revised: E N C OD E R D E C OD E R

Original: ImageEncoder ImageFeature Pos.
Revised: ImageEncoder ImageFeature Pos.

Original: Embed
Revised: Embed

Original: Feature Encoder
Revised: Feature Encoder

Original: [Figure]
Revised: [Figure]

Original: Representation
Revised: Representation

Original: Neural Rendering
Revised: Neural Rendering

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: a transformer that directly maps camera pose embedding to the corresponding image without any physical constraints.
Revised: a transformer that directly maps camera pose embedding to the corresponding image without any physical constraints.

Original: An alternative route formulates view synthesis as rendering a sparsely observed 4D light field, rather than following NeRF’s 5D scene representation and volumetric rendering.
Revised: An alternative route formulates view synthesis as rendering a sparsely observed 4D light field, rather than following NeRF’s 5D scene representation and volumetric rendering.

Original: The recently proposed NLF (Suhail et al., 2021) uses an attention based framework to display light fields with view consistency, where the first transformer summarizes information on epipolar lines independently and then fuses epipolar features using a second transformer.
Revised: The recently proposed NLF (Suhail et al., 2021) uses an attention based framework to display light fields with view consistency, where the first transformer summarizes information on epipolar lines independently and then fuses epipolar features using a second transformer.

Original: This differs from GNT, where we aggregate across views, and are hence able to generalize across scenes which NLF fails to do.
Revised: This differs from GNT, where we aggregate across views, and are hence able to generalize across scenes which NLF fails to do.

Original: Lately, GPNR (Suhail et al., 2022), which was developed concurrently to our work, generalizes
Revised: Lately, GPNR (Suhail et al., 2022), which was developed concurrently to our work, generalizes

Original: NLF (Suhail et al., 2021) by also enabling cross-view communication through attention mechanism.
Revised: NLF (Suhail et al., 2021) by also enabling cross-view communication through attention mechanism.

Original:  M ETHOD : M AKE A TTENTION A LL THAT N E RF N EEDS
Revised:  M ETHOD : M AKE A TTENTION A LL THAT N E RF N EEDS

Original: Overview.
Revised: Overview.

Original: Given a set of N input views with known camera parameters { ( I i ∈
Revised: Given a set of N input views with known camera parameters { ( I i ∈

Original: R H
Revised: R H

Original: × W × 3 , P i ∈ R 3 ×) }
Revised: × W × 3 , P i ∈ R 3 ×) }

Original: Ni =1 , our goal is to synthesize novel views from arbitrary angles and also generalize to new scenes.
Revised: Ni =1 , our goal is to synthesize novel views from arbitrary angles and also generalize to new scenes.

Original: Our method can be divided into two stages: (1) construct the 3D representation from source views on the fly in the feature space, (2) re-render the feature field at the specified angle to synthesize novel views.
Revised: Our method can be divided into two stages: (1) construct the 3D representation from source views on the fly in the feature space, (2) re-render the feature field at the specified angle to synthesize novel views.

Original: Unlike PixelNeRF, IBRNet, MVSNeRF and Neuray that borrow classic volume rendering for view synthesis after the first multi-view aggregation stage, we propose transformers to model both stages.
Revised: Unlike PixelNeRF, IBRNet, MVSNeRF and Neuray that borrow classic volume rendering for view synthesis after the first multi-view aggregation stage, we propose transformers to model both stages.

Original: Our pipeline is depicted in Fig.
Revised: Our pipeline is depicted in Fig.

Original: First, the view transformer aggregates coordinate-aligned features from source views.
Revised: First, the view transformer aggregates coordinate-aligned features from source views.

Original: To enforce multi-view geometry, we inject the inductive bias of epipolar constraints into the attention mechanism.
Revised: To enforce multi-view geometry, we inject the inductive bias of epipolar constraints into the attention mechanism.

Original: After obtaining the feature representation of each point on the ray, the ray transformer composes point-wise features along the ray to form the ray color.
Revised: After obtaining the feature representation of each point on the ray, the ray transformer composes point-wise features along the ray to form the ray color.

Original: This pipeline constitutes GNT and it is trained end-to-end.
Revised: This pipeline constitutes GNT and it is trained end-to-end.

Original: 3. E PIPOLAR G EOMETRY C ONSTRAINED S CENE R EPRESENTATION
Revised: 3. E PIPOLAR G EOMETRY C ONSTRAINED S CENE R EPRESENTATION

Original: NeRF represents 3D scene as a radiance field F : ( x , θ ) (cid:55)→ ( c , σ ) , where each spatial coordinate x ∈ R 3 together with the viewing direction θ ∈ [ − π, π ] 2 is mapped to a color c ∈ R 3 plus density σ ∈ R + tuple.
Revised: NeRF represents 3D scene as a radiance field F : ( x , θ ) (cid:55)→ ( c , σ ) , where each spatial coordinate x ∈ R 3 together with the viewing direction θ ∈ [ − π, π ] 2 is mapped to a color c ∈ R 3 plus density σ ∈ R + tuple.

Original: Vanilla NeRF parameterizes the radiance field using an MLP, and recovers the scene in a backward optimization fashion, inherently limiting NeRF from generalizing to new scenes.
Revised: Vanilla NeRF parameterizes the radiance field using an MLP, and recovers the scene in a backward optimization fashion, inherently limiting NeRF from generalizing to new scenes.

Original: Generalizable NeRFs (Yu et al., 2021;
Revised: Generalizable NeRFs (Yu et al., 2021;

Original: Wang et al., 2021b; Chen et al., 2021a) construct the radiance field in a feed-forward scheme, directly encoding multi-view images into 3D feature space, and decoding it to a color-density field.
Revised: Wang et al., 2021b; Chen et al., 2021a) construct the radiance field in a feed-forward scheme, directly encoding multi-view images into 3D feature space, and decoding it to a color-density field.

Original: In our work, we adopt the similar feed-forward fashion to convert multi-view images into 3D representation, but instead of using physical variables (e.g., color and density), we model a 3D scene as a coordinate-aligned feature field F : ( x , θ ) (cid:55)→ f ∈ R d , where d is the dimension of the latent
Revised: In our work, we adopt the similar feed-forward fashion to convert multi-view images into 3D representation, but instead of using physical variables (e.g., color and density), we model a 3D scene as a coordinate-aligned feature field F : ( x , θ ) (cid:55)→ f ∈ R d , where d is the dimension of the latent

Original: [Figure]
Revised: [Figure]

Original: space.
Revised: space.

Original: We formulate the feed-forward scene representation as follows:
Revised: We formulate the feed-forward scene representation as follows:

Original: [Equation]
Revised: [Equation]

Original: where V ( · ) is a function invariant to the permutation of input images to aggregate different views { I i , · · · , I N } into a coordinate-aligned feature field, and extracts features at a specific location.
Revised: where V ( · ) is a function invariant to the permutation of input images to aggregate different views { I i , · · · , I N } into a coordinate-aligned feature field, and extracts features at a specific location.

Original: We use transformers as a set aggregation function (Lee et al., 2019).
Revised: We use transformers as a set aggregation function (Lee et al., 2019).

Original: However, plugging in attention to globally attend to every pixel in the source images (Sajjadi et al., 2022b;a) is memory prohibitive and lacks multi-view geometric priors.
Revised: However, plugging in attention to globally attend to every pixel in the source images (Sajjadi et al., 2022b;a) is memory prohibitive and lacks multi-view geometric priors.

Original: Hence, we use epipolar geometry as an inductive bias that restricts each pixel to only attend to pixels that lie on the corresponding epipolar lines of the neighboring views.
Revised: Hence, we use epipolar geometry as an inductive bias that restricts each pixel to only attend to pixels that lie on the corresponding epipolar lines of the neighboring views.

Original: Specifically, we first encode each view to be a feature map F i =
Revised: Specifically, we first encode each view to be a feature map F i =

Original: ImageEncoder( I i ) ∈
Revised: ImageEncoder( I i ) ∈

Original: R H
Revised: R H

Original: × W × d .
Revised: × W × d .

Original: We expect the image encoder to extract not only shading information, but also material, semantics, and local/global complex light transport via its multi-scale architecture (Ronneberger et al., 2015).
Revised: We expect the image encoder to extract not only shading information, but also material, semantics, and local/global complex light transport via its multi-scale architecture (Ronneberger et al., 2015).

Original: To obtain the feature representation at a position x , we first project x to every source image, and interpolate the feature vector on the image plane.
Revised: To obtain the feature representation at a position x , we first project x to every source image, and interpolate the feature vector on the image plane.

Original: We then adopt a transformer (dubbed view transformer ) to combine all the feature vectors.
Revised: We then adopt a transformer (dubbed view transformer ) to combine all the feature vectors.

Original: Formally, this process can be written as below:
Revised: Formally, this process can be written as below:

Original: [Equation]
Revised: [Equation]

Original: where View-Transformer( · ) is a transformer encoder (see Appendix A)
Revised: where View-Transformer( · ) is a transformer encoder (see Appendix A)

Original: , Π i ( x ) projects x ∈ Ronto the i -th image plane by applying extrinsic matrix, and F i ( z , θ ) ∈
Revised: , Π i ( x ) projects x ∈ Ronto the i -th image plane by applying extrinsic matrix, and F i ( z , θ ) ∈

Original: R d computes the feature vector at position z ∈ R 2 via bilinear interpolation on the feature grids.
Revised: R d computes the feature vector at position z ∈ R 2 via bilinear interpolation on the feature grids.

Original: We use the transformer’s positional encoding γ ( · ) to concatenate the extracted feature vector with point coordinate, viewing direction, and relative directions of source views with respect to the target view (similar to Wang et al.
Revised: We use the transformer’s positional encoding γ ( · ) to concatenate the extracted feature vector with point coordinate, viewing direction, and relative directions of source views with respect to the target view (similar to Wang et al.

Original: (2021b)).
Revised: (2021b)).

Original: The detailed implementation of view transformer is depicted in Fig.
Revised: The detailed implementation of view transformer is depicted in Fig.

Original: We defer our elaboration on its memory-efficient design to Appendix B. We argue that view transformer can detect occlusion through the pixel values like a stereo matching algorithm and selectively aggregate visible views (see details in Appendix E).
Revised: We defer our elaboration on its memory-efficient design to Appendix B. We argue that view transformer can detect occlusion through the pixel values like a stereo matching algorithm and selectively aggregate visible views (see details in Appendix E).

Original: 3. A TTENTION D RIVEN V OLUMETRIC R ENDERING
Revised: 3. A TTENTION D RIVEN V OLUMETRIC R ENDERING

Original: Volume rendering (App.
Revised: Volume rendering (App.

Original: Eq. 7), which simulates outgoing radiance from a volumetric field has been regarded as a key knob of NeRF’s success.
Revised: Eq. 7), which simulates outgoing radiance from a volumetric field has been regarded as a key knob of NeRF’s success.

Original: NeRF renders the color of a pixel by integrating the color and density along the ray cast from that pixel.
Revised: NeRF renders the color of a pixel by integrating the color and density along the ray cast from that pixel.

Original: Existing works, including NeRF (Sec. 2), all use handcrafted and simplified versions of this integration.
Revised: Existing works, including NeRF (Sec. 2), all use handcrafted and simplified versions of this integration.

Original: However, one can regard volume rendering as a weighted aggregation of all the point-wise output, in which the weights are globally dependent on the other points for occlusion modeling.
Revised: However, one can regard volume rendering as a weighted aggregation of all the point-wise output, in which the weights are globally dependent on the other points for occlusion modeling.

Original: This aggregation can be learned by a transformer such that point-wise colors can be mapped to token features, and attention scores correspond to transmittance (the blending weights).
Revised: This aggregation can be learned by a transformer such that point-wise colors can be mapped to token features, and attention scores correspond to transmittance (the blending weights).

Original: This is how we model the ray transformer which is illustrated in Fig.
Revised: This is how we model the ray transformer which is illustrated in Fig.

Original: 2b
Revised: 2b

Original: To render the color of a ray r =
Revised: To render the color of a ray r =

Original: ( o , d ) , we can compute a feature representation f i =
Revised: ( o , d ) , we can compute a feature representation f i =

Original: F ( x i , θ ) ∈ R d for each point x i sampled on r .
Revised: F ( x i , θ ) ∈ R d for each point x i sampled on r .

Original: In addition to this, we also add position encoding of spatial location and view direction into f i .
Revised: In addition to this, we also add position encoding of spatial location and view direction into f i .

Original: We obtain the rendered color by feeding the sequence of { f 1 , · · · , f M } into the ray transformer, perform mean pooling over all the predicted tokens, and map the pooledfeature vector to RGB via an MLP:
Revised: We obtain the rendered color by feeding the sequence of { f 1 , · · · , f M } into the ray transformer, perform mean pooling over all the predicted tokens, and map the pooled

Original: [Equation]
Revised: [Table]

Original: ◦ Mean ◦ Ray-Transformer( F ( o + t 1 d , θ ) , · · · , F ( o + t M d , θ )) ,
Revised: feature vector to RGB via an MLP:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: where t 1 , · · · , t M are uniformly sampled between near and far plane.
Revised: where t 1 , · · · , t M are uniformly sampled between near and far plane.

Original: Ray-Transformer is a standard transformer encoder
Revised: Ray-Transformer is a standard transformer encoder

Original: [Table]
Revised: [Table]

Original: and its pseudocode implementation is provided in Appendix B. Rendering on feature space utilizes rich geometric, optical and semantic information, which are intractable to be modeled explicitly.
Revised: and its pseudocode implementation is provided in Appendix B. Rendering on feature space utilizes rich geometric, optical and semantic information [Table] which are intractable to be modeled explicitly.

Original: We argue that our ray transformer can automatically adjust the attention distribution to control the sharpness of reconstructed surface
Revised: We argue that our ray transformer can automatically adjust the attention distribution to control the sharpness of reconstructed surface

Original: [Table]
Revised: [Table]

Original: and bake desirable lighting effects from the illumination and material features.
Revised: and bake desirable lighting effects from the illumination and material features.

Original: Moreover, by exerting the expressiveness of image encoder, ray transformer can also overcome the limitation of ray casting and epipolar geometry to simulate complex light transport (e [Table] g., refraction, reflection, etc.).
Revised: Moreover, by exerting the expressiveness of image encoder, ray transformer can also overcome the limitation of ray casting and epipolar geometry to simulate complex light transport (e.g., refraction, reflection, etc.).

Original: Interestingly, despite all in latent space, we can also infer some explicit physical properties (such as depth) from ray transformer.
Revised: Interestingly, despite all in latent space, we can also infer some explicit physical properties (such as depth) from ray transformer.

Original: See Appendix
Revised: See Appendix

Original: E for depth cuing.
Revised: E for depth cuing.

Original: We also involve discussion on the extension to auto-regressive rendering and attention-based coarse-to-fine sampling in Appendix C.
Revised: We also involve discussion on the extension to auto-regressive rendering and attention-based coarse-to-fine sampling in Appendix C.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct extensive experiments to compare GNT with several state-of-the-art (SOTA) methods for novel view synthesis.
Revised: We conduct extensive experiments to compare GNT with several state-of-the-art (SOTA) methods for novel view synthesis.

Original: We first provide qualitative and quantitative results in the per-scene training setting, followed by generalization to unseen scenes.
Revised: We first provide qualitative and quantitative results in the per-scene training setting, followed by generalization to unseen scenes.

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: Source and Target View Sampling.
Revised: Source and Target View Sampling.

Original: Following IBRNet, we construct pairs of source and target views for training by first selecting a target view, and then identifying a pool of k × N nearby views, from which N views are randomly sampled as source views.
Revised: Following IBRNet, we construct pairs of source and target views for training by first selecting a target view, and then identifying a pool of k × N nearby views, from which N views are randomly sampled as source views.

Original: This sampling strategy simulates various view densities during training and therefore helps the network generalize better.
Revised: This sampling strategy simulates various view densities during training and therefore helps the network generalize better.

Original: During training, the values for k and N are uniformly sampled at random from (1 , 3) and (8 , 12) respectively.
Revised: During training, the values for k and N are uniformly sampled at random from (1 , 3) and (8 , 12) respectively.

Original: Training / Inference Details.
Revised: Training / Inference Details.

Original: We train both the feature extraction network and GNT end-to-end on datasets of multi-view posed images using the Adam optimizer to minimize the mean-squared error between predicted and ground truth RGB pixel values.
Revised: We train both the feature extraction network and GNT end-to-end on datasets of multi-view posed images using the Adam optimizer to minimize the mean-squared error between predicted and ground truth RGB pixel values.

Original: The base learning rates for the feature extraction network and GNT are 10 − 3 and 5 × 10 − 4 respectively, which decay exponentially over training steps.
Revised: The base learning rates for the feature extraction network and GNT are 10 − 3 and 5 × 10 − 4 respectively, which decay exponentially over training steps.

Original: For all our experiments, we train for 250,000 steps with 4096 rays sampled in each iteration.
Revised: For all our experiments, we train for 250,000 steps with 4096 rays sampled in each iteration.

Original: Unlike most NeRF methods, we do not use separate coarse, fine networks and therefore to bring GNT to a comparable experimental setup, we sample 192 coarse points per ray across all experiments (unless otherwise specified).
Revised: Unlike most NeRF methods, we do not use separate coarse, fine networks and therefore to bring GNT to a comparable experimental setup, we sample 192 coarse points per ray across all experiments (unless otherwise specified).

Original: Metrics.
Revised: Metrics.

Original: We use three widely adopted metrics: Peak Signal-to-Noise Ratio (PSNR), Structural
Revised: We use three widely adopted metrics: Peak Signal-to-Noise Ratio (PSNR), Structural

Original: Similarity Index Measure (SSIM) (Wang et al., 2004), and the Learned Perceptual Image Patch
Revised: Similarity Index Measure (SSIM) (Wang et al., 2004), and the Learned Perceptual Image Patch

Original: Similarity (LPIPS) (Zhang et al., 2018).
Revised: Similarity (LPIPS) (Zhang et al., 2018).

Original: We report the averages of each metric over different views in each scene (for single scene experiments) and across multiple scenes in each dataset (for generalization experiments).
Revised: We report the averages of each metric over different views in each scene (for single scene experiments) and across multiple scenes in each dataset (for generalization experiments).

Original: We additionally report the geometric mean of 10 − PSNR/ 10 , √ 1 − SSIM and LPIPS, which provides a summary of the three metrics for easier comparison (Barron et al., 2021).
Revised: We additionally report the geometric mean of 10 − PSNR/ 10 , √ 1 − SSIM and LPIPS, which provides a summary of the three metrics for easier comparison (Barron et al., 2021).

Original: 4. S INGLE S CENE R ESULTS
Revised: 4. S INGLE S CENE R ESULTS

Original: Datasets.
Revised: Datasets.

Original: To evaluate the single scene view generation capacity of GNT, we perform experiments on datasets containing synthetic rendering of objects and real images of complex scenes.
Revised: To evaluate the single scene view generation capacity of GNT, we perform experiments on datasets containing synthetic rendering of objects and real images of complex scenes.

Original: In these  experiments, we use the same resolution and train/test splits as NeRF (Mildenhall et al., 2020).
Revised: In these [Figure] experiments, we use the same resolution and train/test splits as NeRF (Mildenhall et al

Original: (a) Local Light Field Fusion (LLFF) dataset : Introduced by Mildenhall et al.
Revised: Local Light Field Fusion (LLFF) dataset : Introduced by Mildenhall et al.

Original: (2019), it consists of 8 forward facing captures of real-world scenes using a smartphone.
Revised: (2019), it consists offorward facing captures of real-world scenes using a smartphone.

Original: We report average scores across {Orchids, Horns, Trex, Room, Leaves, Fern, Fortress} and the metrics are summarized in Tab. 1b.
Revised: We report average scores across {Orchids, Horns, Trex, Room, Leaves, Fern, Fortress} and the metrics are summarized in Tab. 1b.

Original: [Figure] ) NeRF Synthetic Dataset : The synthetic dataset introduced by ( et al., 2020) consists of 8, 360°scenes of objects with complicated geometry and realistic material.
Revised: NeRF Synthetic Dataset : The synthetic dataset introduced by (Mildenhall et al., 2020) consists of 8, 360°scenes of objects with complicated geometry and realistic material.

Original: Each scene consists of images rendered from viewpoints randomly sampled on a hemisphere around the object.
Revised: Each scene consists of images rendered from viewpoints randomly sampled on a hemisphere around the object.

Original: Similar to the experiments on the LLFF dataset, we report the average metrics across all eight scenes in Tab. 1a
Revised: Similar to the experiments on the LLFF dataset, we report the average metrics across all eight scenes in Tab. 1a.

Original: Discussion.
Revised: Discussion.

Original: We compare our GNT with LLFF , NeRF , MipNeRF , NeX , and NLF.
Revised: We compare our GNT with LLFF , NeRF , MipNeRF , NeX , and NLF.

Original: Compared to other methods, we utilize a smaller batch size (specifically GNT samples 4096 rays per batch while NLF samples as much as 16384 rays per batch) and only sample coarse points fed into the network in one single forward pass unlike most methods which use a two stage coarse-fine sampling strategy.
Revised: Compared to other methods, we utilize a smaller batch size (specifically GNT samples 4096 rays per batch while NLF samples as much as 16384 rays per batch) and only sample coarse points fed into the network in one single forward pass unlike most methods which use a two stage coarse-fine sampling strategy.

Original: These hyperparameters have a strong correlation with the rendered image quality, leaving our method at a disadvantage.
Revised: These hyperparameters have a strong correlation with the rendered image quality, leaving our method at a disadvantage.

Original: Despite these differences, GNT still manages to outperform most methods and performs on-par when compared to SOTA NLF method on both LLFF and Synthetic datasets.
Revised: Despite these differences, GNT still manages to outperform most methods and performs on-par when compared to SOTA NLF method on both LLFF and Synthetic datasets.

Original: We provide a scene-wise breakdown of results on both these datasets in Appendix D (Tab.
Revised: We provide a scene-wise breakdown of results on both these datasets in Appendix D (Tab.

Original: In complex scenes like Drums, Ship, and Leaves, GNT manages to outperform other methods more substantially by 2 . 49 dB, 0 .
Revised: In complex scenes like Drums, Ship, and Leaves, GNT manages to outperform other methods more substantially by 2 . 49 dB, 0 .

Original: 82 dB and 0 .
Revised: 82 dB and 0 .

Original: 10 dB respectively.
Revised: 10 dB respectively.

Original: This indicates the effectiveness of our attention-driven volumetric rendering to model complex conditions.
Revised: This indicates the effectiveness of our attention-driven volumetric rendering to model complex conditions.

Original: Interestingly, even in the worse performing scenes by PSNR (e.g. T-Rex), GNT achieves best perceptual metric scores across all scenes in the LLFF dataset (i.e LPIPS ~ 27% ↓ ).
Revised: Interestingly, even in the worse performing scenes by PSNR (e.g. T-Rex), GNT achieves best perceptual metric scores across all scenes in the LLFF dataset (i.e LPIPS ~ 27% ↓ ).

Original: This could be because PSNR fails to measure structural distortions, blurring, has high sensitivity towards brightness, and hence does not effectively measure visual quality.
Revised: This could be because PSNR fails to measure structural distortions, blurring, has high sensitivity towards brightness, and hence does not effectively measure visual quality.

Original: Similar inferences are discussed in Lin et al.
Revised: Similar inferences are discussed in Lin et al.

Original: (2022) regarding discrepancies in
Revised: (2022) regarding discrepancies in

Original: PSNR scores and their correlation to rendered image quality.
Revised: PSNR scores and their correlation to rendered image quality.

Original: Figs.
Revised: Figs.

Original: 3 provides qualitative comparisons on the Orchids, Drums scene respectively and we can clearly see that GNT recovers the edge details of objects (in the case of Orchids) and models complex lighting effect like specular reflection (in the case of Drums) more accurately.[Figure]
Revised: 3 provides qualitative comparisons on the Orchids, Drums scene respectively and we can clearly see that GNT recovers the edge details of objects (in the case of Orchids) and models complex lighting effect like specular reflection (in the case of Drums) more accurately.

Original: [Table]

Original: GNT can learn to adapt to refraction and reflection in scenes.

Original: Encouraged by the promise shown by GNT in modelling reflections in the Drums scene

Original: [Table]

Original: we further evaluate GNT on the complex Shiny dataset (Wizadwongsa et al., 2021)

Original: [Table]

Original: that contains several challenging view-dependent effects, such as the rainbow reflections on a CD

Original: [Table]

Original: and the refraction through liquid bottles.

Original: Technically, the full formulation of volume rendering (radiative transfer equation, as used in modern volume path tracers) is capable of handling all these effects.

Original: However, standard NeRFs use a simplified formulation which does not simulate all physical effects, and hence easily fail to capture these effects.

Original: Fig.

Original: 4 presents rendering results on the two example scenes of Lab and CD.

Original: Comparing to the baseline (NeX), GNT is able to reconstruct complex refractions through test tube

Original: [Table]

Original: and the interference patterns on the disk with higher quality , indicating the strong flexibility and adaptivity of our learnable renderer.

Original: That “serendipity" is intriguing to us, since the presence of refraction and scattering means that any technique that only uses samples along a single ray will not be able to properly simulate the full light transport.

Original: We conjecture GNT’s success in modeling those challenging physical scenes to be the full transformer architectures for not only rays but also views: the view encoder aggregates multi-ray information and can in turn tune the ray encoding itself.

Original: 4. G ENERALIZATION TO U NSEEN S CENES
Revised: 4. G ENERALIZATION TO U NSEEN S CENES

Original: Datasets.
Revised: Datasets.

Original: GNT leverages multi-view features complying with epipolar geometry, enabling generalization to unseen scenes.
Revised: GNT leverages multi-view features complying with epipolar geometry, enabling generalization to unseen scenes.

Original: We follow the experimental protocol in IBRNetto evaluate the cross-scene generalization of GNT and use the following datasets for training/evaluation respectively.
Revised: We follow the experimental protocol in IBRNetto evaluate the cross-scene generalization of GNT and use the following datasets for training/evaluation resectively.

Original: (a) Training Datasets consists of both real, and synthetic data.
Revised: (a) Training Datasets consists of both real, and synthetic data.

Original: For synthetic data, we use object renderings of 1023 models from Google Scanned Object (Downs et al., 2022).
Revised: For synthetic data, we use object renderings of 1023 models from Google Scanned Object (Downs et al For real data , we make [Table]

Original: For real data, we make use of RealEstate10K (Zhou et al., 2018b), 100 scenes from the Spaces dataset (Flynn et al., 2019), and 102 real scenes from handheld cellphone captures (Mildenhall et al., 2019; Wanget al., 2021b).
Revised: [Table] [Table] [Table] use of RealEstate10K (Zhou et al., 2018b), 100 scenes from the Spaces dataset (Flynn et al., 2019), and 102 real scenes from handheld cellphone captures (Mildenhall et al., 2019; Wang

Original: (b) Evaluation Datasets include the previously discussed Synthetic (Mildenhall et al., 2020), LLFF datasets (Mildenhall et al., 2019).

Original: Please note that the LLFF scenes present in the validation set are not included in the handheld cellphone captures in the training set.

Original: Discussion.
Revised: Discussion.

Original: We compare our method with PixelNeRF (Yu et al., 2021), MVSNeRF (Chen et al., 2021a), IBRNet (Wang et al., 2021b), and NeuRay (Liu et al., 2022).
Revised: We compare our method with PixelNeRF (Yu et al., 2021), MVSNeRF (Chen et al., 2021a), IBRNet (Wang et al., 2021b), and NeuRay (Liu et al., 2022).

Original: As seen from Tab. 2, our method outperforms SOTA by ~ 17% ↓ , ~ 9% ↓ average scores in the LLFF, Synthetic datasets respectively.
Revised: As seen from Tab. 2a, our method outperforms SOTA by ~ 17% ↓ , ~ 9% ↓ average scores in the LLFF, Synthetic datasets respectively.

Original: This indicates the effectiveness of our proposed view transformer to extract generalizable scene representations.
Revised: This indicates the effectiveness of our proposed view transformer to extract generalizable scene representations.

Original: Similar to the single scene experiments, we observe significantly better perceptual metric scores 3% ↑ SSIM, 27% ↓ LPIPS in both datasets.
Revised: Similar to the single scene experiments, we observe significantly better perceptual metric scores 3% ↑ SSIM, 27% ↓ LPIPS in both datasets.

Original: We show qualitative results in Fig.where GNT renders novel views with clearly better visual quality when compared to other methods.
Revised: We show qualitative results in Fig.where GNT renders novel views with clearly better visual quality when compared to other methods.

Original: Specifically, as seen from the second row in Fig. 5, GNT is able to handle regions that are sparsely visible in the source views and generates images of comparable visual quality as NeuRay even with no explicit supervision for occlusion.
Revised: Specifically, as seen from the second row in Fig. 5, GNT is able to handle regions that are sparsely visible in the source views and generates images of comparable visual quality as NeuRay even with no explicit supervision for occlusion.

Original: The combination of our enhanced scene representation (view transformer) and learnable renderer (ray transformer) contribute to these improved results.
Revised: The combination of our enhanced scene representation (view transformer) and learnable renderer (ray transformer) contribute to these improved results.

Revised: GNT can learn to adapt to refraction and reflection in scenes.

Original: 4. A BLATION S TUDIES
Revised: Encouraged by the promise shown by GNT in modelling reflections in the Drums scene via pre-scene training, we further directly evaluate pre-trained GNT on the Shiny dataset (Wizadwongsa et al., 2021), that contains several challenging view-dependent effects, such as the rainbow reflections on a CD, and the refraction through liquid bottles. Technically, the full formulation of volume rendering (radiative transfer equation, as used in modern volume path tracers) is capable of handling all these effects. However, standard NeRFs use a simplified formulation which does not simulate all physical effects, and hence easily fail to capture these effects. Tab. 2b presents the numerical results of GNT when generalizing to Shiny dataset.

Revised: Notably, our GNT outperforms state-of-the-art GPNR (Suhail et al., 2022) by 3dB in PSNR and ~40% in average metric.

Revised: Compared with per-scene optimized NeRFs, GNT outperforms many of them and even approaches to the best performer NLF (Suhail et al., 2021) without any extra training.

Original: To validate the significance of our architectural designs in GNT, we conduct the following ablation studies on the Drums scene from the Blender dataset.
Revised: This further supports our argument that cross-scene training can help learn a better renderer. Fig. 4 exhibits rendering results on the two example scenes of Lab and CD. Comparing to the baseline (NeX), GNT is able to reconstruct complex refractions through test tube, and the interference patterns on the disk with higher quality , indicating the strong flexibility and adaptivity of our learnable renderer.

Revised: That “serendipity" is intriguing to us, since the presence of refraction and scattering means that any technique that only uses samples along a single ray will not be able to properly simulate the full light transport.

Revised: We conjecture GNT’s success in modeling those challenging physical scenes to be the full transformer architectures for not only rays but also views: the view encoder aggregates multi-ray information and can in turn tune the ray encoding itself.

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: 4. A BLATION S TUDIES

Revised: We conduct the following ablation studies on the Drums scene to validate our architectural designs.

Original: One-Stage Transformer :
Revised: One-Stage Transformer :

Original: We convert the point-wise epipolar features into one single sequence and pass it through a “one-stage transformer” network with standard dot-product self-attention layers, without considering our two-stage pipeline: view and ray aggregation.
Revised: We convert the point-wise epipolar features into one single sequence and pass it through a “one-stage transformer” network with standard dot-product self-attention layers, without considering our two-stage pipeline: view and ray aggregation.

Original: [Table]
Revised: [Table]

Original: Epipolar Agg.
Revised: Epipolar Agg.

Original: → View Agg.
Revised: → View Agg.

Original: : Moving to a two-stage transformer, we train a network which first aggregates features from the points along the epipolar lines followed by feature aggregation across epipolar lines on different reference views (in contrast to
Revised: : Moving to a two-stage transformer, we train a network which first aggregates features from the points along the epipolar lines followed by feature aggregation across epipolar lines on different reference views (in contrast to

Original: GNT’s first view aggregation then ray aggregation).
Revised: GNT’s first view aggregation then ray aggregation).

Original: This two-stage aggregation resembles the strategies adopted in NLF (Suhail et al., 2021).
Revised: This two-stage aggregation resembles the strategies adopted in NLF (Suhail et al., 2021).

Original: Dot Product-Attention View Transformer :
Revised: Dot Product-Attention View Transformer :

Original: Next we train a network that uses the standard dotproduct attention in the view transformer blocks, in contrast to our proposed memory-efficient subtraction-based attention (see Appendix B).
Revised: Next we train a network that uses the standard dotproduct attention in the view transformer blocks, in contrast to our proposed memory-efficient subtraction-based attention (see Appendix B).

Original: w/ Volumetric Rendering :
Revised: w/ Volumetric Rendering :

Original: Last but not least, we train a network to predict per-point RGB and density values from the point features output by view aggregator, and compose them using the volumetric rendering equation, instead of our  attention-driven volumetric renderer.
Revised: Last but not least, we train a network to predict per-point RGB and density values from the point features output by view aggregator, and compose them using the volumetric rendering equation, instead of our learned attention-driven volumetric renderer.

Original: We report the performance of the above investigation in Tab. 3.
Revised: We report the performance of the above investigation in Tab. 3.

Original: We verify that our design of two-stage transformer is superior than one-stage aggregation or an alternative two-stage pipeline (Suhail et al., 2021) since our renderer strictly complies with the multi-view geometry.
Revised: We verify that our design of two-stage transformer is superior than one-stage aggregation or an alternative two-stage pipeline (Suhail et al., 2021) since our renderer strictly complies with the multi-view geometry.

Original: Compared with dot-product attention, subtraction-based attention achieves slightly higher overall scores.
Revised: Compared with dot-product attention, subtraction-based attention achieves slightly higher overall scores.

Original: This also indicates the performance of GNT does not heavily rely on the choice of attention operation.
Revised: This also indicates the performance of GNT does not heavily rely on the choice of attention operation.

Original: What matters is bringing in attention mechanism for cross-point interaction.
Revised: What matters is bringing in attention mechanism for cross-point interaction.

Original: For practical usage, we also consider the memory efficiency in our view transformer.
Revised: For practical usage, we also consider the memory efficiency in our view transformer.

Original: Our ray transformer outperforms the classic volumetric rendering implies the advantage of adopting a data-driven renderer.
Revised: Our ray transformer outperforms the classic volumetric rendering implies the advantage of adopting a data-driven renderer.

Original: 4. I NTERPRETATION ON A TTENTION M APS
Revised: 4. I NTERPRETATION ON A TTENTION M APS

Original: The use of transformers as a core element in GNT enables interpretation by analyzing the attention weights.
Revised: The use of transformers as a core element in GNT enables interpretation by analyzing the attention weights.

Original: As discussed earlier, view transformer finds correspondence between the queried points, andneighboring views which enables it to pay attention to more “visible” views or be occlusion-aware.
Revised: As discussed earlier, view transformer finds correspondence between the queried points, and

Original: [Figure]
Revised: [Figure]

Revised: neighboring views which enables it to pay attention to more “visible” views or be occlusion-aware.

Original: Similarly, the ray transformer captures point-to-point interactions which enables it to model the relative importance of each point or be depth-aware.
Revised: Similarly, the ray transformer captures point-to-point interactions which enables it to model the relative importance of each point or be depth-aware.

Original: We validate our hypothesis by visualization.
Revised: We validate our hypothesis by visualization.

Revised: [Figure]

Revised: transformer learns to in the target view to its corresponding regions in the source views which are least occluded

Original: View Attention.
Revised: View Attention.

Original: To visualize the view-wise attention maps learnt by our model, we use the attention matrix from Eq.
Revised: To visualize the view-wise attention maps learnt by our model, we use the attention matrix from Eq.

Original: 9 and collapse the channel dimension by performing mean pooling.
Revised: 9 and collapse the channel dimension by performing mean pooling.

Original: We then identify the view-number which is assigned maximum attention with respect to each point and then compute the most repeating view number across points along a ray (by computing mode).
Revised: We then identify the view-number which is assigned maximum attention with respect to each point and then compute the most repeating view number across points along a ray (by computing mode).

Original: These “view importance” values denote the view which has maximum correspondence with the target pixel’s color.
Revised: These “view importance” values denote the view which has maximum correspondence with the target pixel’s color.

Original: Fig.
Revised: Fig.

Original: 6 visualizes the source view correspondence with every pixel in the target view.
Revised: 6 visualizes the source view correspondence with every pixel in the target view.

Original: Given a region in the target view, GNT attempts to pay maximum attention to a source view which is least occluded in the same region.
Revised: Given a region in the target view, GNT attempts to pay maximum attention to a source view which is least occluded in the same region.

Original: For example: In Fig. 6, the truck’s bucket is most visible from view number 8, hence the regions corresponding to the same are orange colored, while regions towards the front of the lego are are most visible from number 7 (yellow).
Revised: For example: In Fig. 6, the truck’s bucket is most visible from view number 8, hence the regions corresponding to the same are orange colored, while regions towards the front of the lego are are most visible from view

Original: Ray Attention.
Revised: Ray Attention.

Original: To visualize the attention maps across points in a ray, we use the attention matrix from Eq. 4 and collapse the head dimension by performing mean pooling.
Revised: To visualize the attention maps across points in a ray, we use the attention matrix from Eq. 4 and collapse the head dimension by performing mean pooling.

Original: From the derived matrix, we select a point and extract its relative importance with every other point.
Revised: From the derived matrix, we select a point and extract its relative importance with every other point.

Original: We then compute a depth map from these learned point-wise correspondence values by multiplying it with the marching distance of each point and sum-pooling along a ray.
Revised: We then compute a depth map from these learned point-wise correspondence values by multiplying it with the marching distance of each point and sum-pooling along a ray.

Original: Fig.
Revised: Fig.

Original: 7 plots the depth maps computed from the learned attention values in the ray transformer block.
Revised: 7 plots the depth maps computed from the learned attention values in the ray transformer block.

Original: We can clearly see that the obtained depth maps have a physical meaning i.e pixels closer to the view directions are blue while the ones farther away are red.
Revised: We can clearly see that the obtained depth maps have a physical meaning i.e pixels closer to the view directions are blue while the ones farther away are red.

Original: Therefore, with no explicit supervision, GNT learns to physically ground its attention maps.
Revised: Therefore, with no explicit supervision, GNT learns to physically ground its attention maps.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present Generalizable NeRF Transformer (GNT), a pure transformer based architecture that efficiently reconstructs NeRFs on the fly.
Revised: We present Generalizable NeRF Transformer (GNT), a pure transformer based architecture that efficiently reconstructs NeRFs on the fly.

Original: The view transformer of GNT leverages epipolar geometry as an inductive bias for scene representation.
Revised: The view transformer of GNT leverages epipolar geometry as an inductive bias for scene representation.

Original: The ray transformer renders novel views by ray marching and decoding the sequences of sampled point features using attention mechanism.
Revised: The ray transformer renders novel views by ray marching and decoding the sequences of sampled point features using attention mechanism.

Original: Extensive experiments demonstrate that GNT can improve both single-scene and cross-scene training results, and demonstrates “out of the box” promise for refraction and reflection scenes.
Revised: Extensive experiments demonstrate that GNT can improve both single-scene and cross-scene training results, and demonstrates “out of the box” promise for refraction and reflection scenes.

Original: We also show by visualization that depth and occlusion can be inferred from attention maps.
Revised: We also show by visualization that depth and occlusion can be inferred from attention maps.

Original: This implies that pure attention can be “universal modeling tool” for physically-grounded rendering process.
Revised: This implies that pure attention can be “universal modeling tool” for physically-grounded rendering process.

Original: Interesting future directions include relaxing the epipolar constraints to simulate more complicated light transport.
Revised: Interesting future directions include relaxing the epipolar constraints to simulate more complicated light transport.

Original: A P RELIMINARIES
Revised: A P RELIMINARIES

Original: Self-Attention and Transformer.
Revised: Self-Attention and Transformer.

Original: Multi-Head Self-Attention (MHA) is the key ingredient of transformers (Vaswani et al., 2017).
Revised: Multi-Head Self-Attention (MHA) is the key ingredient of transformers (Vaswani et al., 2017).

Original: Data is first tokenized into sequences and a pairwise score is computed to weight the relation of each token with all the others in a given input context.
Revised: Data is first tokenized into sequences and a pairwise score is computed to weight the relation of each token with all the others in a given input context.

Original: Formally, let X ∈ R N × d represent some sequential data with N tokens of d -dimension.
Revised: Formally, let X ∈ R N × d represent some sequential data with N tokens of d -dimension.

Original: A self-attention layer transforms the feature matrix as below:
Revised: A self-attention layer transforms the feature matrix as below:

Original: [Equation]
Revised: [Equation]

Original: where A ∈ R N × N is called the attention matrix, softmax( · ) operation normalizes the attention matrix row-wise, and α ( · ) represents a pair-wise relation function, most commonly the dot-product α
Revised: where A ∈ R N × N is called the attention matrix, softmax( · ) operation normalizes the attention matrix row-wise, and α ( · ) represents a pair-wise relation function, most commonly the dot-product α

Original: ( X i , X j ) =
Revised: ( X i , X j ) =

Original: f Q ( X i )
Revised: f Q ( X i )

Original: ⊤ f K ( X j ) /γ , where f Q ( · ) , f K ( · ) , f V ( · ) are called query, key, and value mapping functions.
Revised: ⊤ f K ( X j ) /γ , where f Q ( · ) , f K ( · ) , f V ( · ) are called query, key, and value mapping functions.

Original: In a standard transformer, they are chosen as fully-connected layers.
Revised: In a standard transformer, they are chosen as fully-connected layers.

Original: This self-attention is akin to an aggregation operation.
Revised: This self-attention is akin to an aggregation operation.

Original: Multi-Head Self-Attention (MHA) sets a group of self-attention blocks, and adopts a linear layer to project them onto the output space:
Revised: Multi-Head Self-Attention (MHA) sets a group of self-attention blocks, and adopts a linear layer to project them onto the output space:

Original: [Equation]
Revised: [Equation]

Original: Following an MHA block, one standard layer of transformer also adopts a Feed-Forward Network (FFN) to do point-wise feature transformation, as well as skip connection and layer normalization to stablize training.
Revised: Following an MHA block, one standard layer of transformer also adopts a Feed-Forward Network (FFN) to do point-wise feature transformation, as well as skip connection and layer normalization to stablize training.

Original: The whole transformer block can be formulated as below:
Revised: The whole transformer block can be formulated as below:

Original: [Equation]
Revised: [Equation]

Original: Neural Radiance Field.
Revised: Neural Radiance Field.

Original: NeRFs (Mildenhall et al., 2020) converts multi-view images into a radiance field and interpolates novel views by re-rendering the radiance field from a new angle.
Revised: NeRFs (Mildenhall et al., 2020) converts multi-view images into a radiance field and interpolates novel views by re-rendering the radiance field from a new angle.

Original: Technically,
Revised: Technically,

Original: NeRF models the underlying 3D scene as a continuous radiance field F : ( x , θ ) (cid:55)→ ( c , σ ) parameterized by a Multi-Layer Perceptron (MLP) Θ , which maps a spatial coordinate x ∈ R 3 together with the viewing direction θ ∈ [ − π, π ] 2 to a color c ∈ R 3 plus density σ ∈ R + tuple.
Revised: NeRF models the underlying 3D scene as a continuous radiance field F : ( x , θ ) (cid:55)→ ( c , σ ) parameterized by a Multi-Layer Perceptron (MLP) Θ , which maps a spatial coordinate x ∈ R 3 together with the viewing direction θ ∈ [ − π, π ] 2 to a color c ∈ R 3 plus density σ ∈ R + tuple.

Original: To form an image,
Revised: To form an image,

Original: NeRF performs the ray-based rendering, where it casts a ray r =
Revised: NeRF performs the ray-based rendering, where it casts a ray r =

Original: ( o , d ) from the optical center o ∈ R 3 through each pixel (towards direction d ∈ R 3 ), and then leverages volume rendering (Max, 1995) to compose the color and density along the ray between the near-far planes:
Revised: ( o , d ) from the optical center o ∈ R 3 through each pixel (towards direction d ∈ R 3 ), and then leverages volume rendering (Max, 1995) to compose the color and density along the ray between the near-far planes:

Original: [Equation]
Revised: [Equation]

Original: where r ( t ) =
Revised: where r ( t ) =

Original: o + t d , t n and t f are the near and far planes respectively.
Revised: o + t d , t n and t f are the near and far planes respectively.

Original: In practice, the Eq.
Revised: In practice, the Eq.

Original: 7 is numerically estimated using quadrature rules (Mildenhall et al., 2020).
Revised: 7 is numerically estimated using quadrature rules (Mildenhall et al., 2020).

Original: Given images captured from surrounding views with known camera parameters, NeRF fits the radiance field by maximizing the likelihood of simulated results.
Revised: Given images captured from surrounding views with known camera parameters, NeRF fits the radiance field by maximizing the likelihood of simulated results.

Original: Suppose we collect all pairs of rays and pixel colors as the training set
Revised: Suppose we collect all pairs of rays and pixel colors as the training set

Original: D = { ( r i , (cid:98) C i ) } Ni =1 , where N is the total number of rays sampled, and (cid:98) C i denotes the ground-truth color of the i -th ray, then we train the implicit representation Θ via the following loss function:
Revised: D = { ( r i , (cid:98) C i ) } Ni =1 , where N is the total number of rays sampled, and (cid:98) C i denotes the ground-truth color of the i -th ray, then we train the implicit representation Θ via the following loss function:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B I MPLEMENTATION D ETAILS
Revised: B I MPLEMENTATION D ETAILS

Original: Memory-Efficient Cross-View Attention.
Revised: Memory-Efficient Cross-View Attention.

Original: Computing attention between every pair of inputs has
Revised: Computing attention between every pair of inputs has

Original: O ( N 2 ) memory complexity, which is computational prohibitive when sampling thousands of points at the same time.
Revised: O ( N 2 ) memory complexity, which is computational prohibitive when sampling thousands of points at the same time.

Original: Nevertheless, we note that view transformer only needs to read out one token as the fused results of all the views.
Revised: Nevertheless, we note that view transformer only needs to read out one token as the fused results of all the views.

Original: Therefore, we propose to only place one read-out token X 0 ∈ R d in the query sequence, and let it iteratively summarize features from other data points.
Revised: Therefore, we propose to only place one read-out token X 0 ∈ R d in the query sequence, and let it iteratively summarize features from other data points.

Original: This reduces the complexity for each layer up to O ( N ) .
Revised: This reduces the complexity for each layer up to O ( N ) .

Original: We initialize the read-out token as the element-wise maxpooling of all the inputs: X 0 = max( F 1 (Π 1 ( x ) , θ ) , · · · , F N (Π N ( x ) , θ )) .
Revised: We initialize the read-out token as the element-wise maxpooling of all the inputs: X 0 = max( F 1 (Π 1 ( x ) , θ ) , · · · , F N (Π N ( x ) , θ )) .

Original: Rather than adopting a standard dot-product attention, we choose subtraction operation as the relation function.
Revised: Rather than adopting a standard dot-product attention, we choose subtraction operation as the relation function.

Original: Subtraction attention has been shown more effective for positional and geometric relationship reasoning (Zhao et al., 2021; Fan et al., 2022).
Revised: Subtraction attention has been shown more effective for positional and geometric relationship reasoning (Zhao et al., 2021; Fan et al., 2022).

Original: Compared with dot-product that collapses the feature dimension into a scalar, subtraction attention computes different attention scores for every channel of the value matrix, which increases diversity in feature interactions.
Revised: Compared with dot-product that collapses the feature dimension into a scalar, subtraction attention computes different attention scores for every channel of the value matrix, which increases diversity in feature interactions.

Original: Moreover, we augment the attention map and value matrix with { ∆ d } Ni =1 to provide relative spatial context.
Revised: Moreover, we augment the attention map and value matrix with { ∆ d } Ni =1 to provide relative spatial context.

Original: Technically, we utilize a linear layer W P to lift ∆ d i to the hidden dimension.
Revised: Technically, we utilize a linear layer W P to lift ∆ d i to the hidden dimension.

Original: We illustrate view transformer in Fig.
Revised: We illustrate view transformer in Fig.

Original: 2a.
Revised: 2a.

Original: To be more specific, the modified attention adopted in our view transformer can be formulated as:
Revised: To be more specific, the modified attention adopted in our view transformer can be formulated as:

Original: View-Attn( X ) =
Revised: View-Attn( X ) =

Original: diag(softmax( A + ∆ ⊤ ) f V
Revised: diag(softmax( A + ∆ ⊤ ) f V

Original: ( X + ∆ )) , where A j = f Q ( X 0 )
Revised: ( X + ∆ )) , where A j = f Q ( X 0 )

Original: − f K ( X j ) , (9) where A j ∈ R d denotes the j -th column of A , ∆ =
Revised: − f K ( X j ) , (9) where A j ∈ R d denotes the j -th column of A , ∆ =

Original: [ ∆ d· · · ∆ d N ] ⊤ W P ∈ R N × d , f Q , f K , and f V are parameterized by an MLP.
Revised: [ ∆ d· · · ∆ d N ] ⊤ W P ∈ R N × d , f Q , f K , and f V are parameterized by an MLP.

Original: We note that by applying diag( · ) , we read out the updated query token X 0 .
Revised: We note that by applying diag( · ) , we read out the updated query token X 0 .

Original: Network Architecture.
Revised: Network Architecture.

Original: To extract features from the source views, we use a U-Net like architecture with a ResNet34 encoder, followed by two up-sampling layers as decoder.
Revised: To extract features from the source views, we use a U-Net like architecture with a ResNet34 encoder, followed by two up-sampling layers as decoder.

Original: Each view transformer block contains a single-headed cross-attention layer while the ray transformer block contains a multi-headed self-attention layer with four heads.
Revised: Each view transformer block contains a single-headed cross-attention layer while the ray transformer block contains a multi-headed self-attention layer with four heads.

Original: The outputs from these attention layers are passed onto corresponding feedforward blocks with a Rectified Linear Unit (RELU) activation and a hidden dimension of 256 .
Revised: The outputs from these attention layers are passed onto corresponding feedforward blocks with a Rectified Linear Unit (RELU) activation and a hidden dimension of 256 .

Original: A residual connection is applied between the pre-normalized inputs (LayerNorm) and outputs at each layer.
Revised: A residual connection is applied between the pre-normalized inputs (LayerNorm) and outputs at each layer.

Original: For all our single scene experiments, we alternatively stack 4 view and ray transformer blocks while our larger generalization experiments use 8 blocks each.
Revised: For all our single scene experiments, we alternatively stack 4 view and ray transformer blocks while our larger generalization experiments use 8 blocks each.

Original: All transformer blocks (view and ray) are of dimension 64 .
Revised: All transformer blocks (view and ray) are of dimension 64 .

Original: Following Vaswani et al.
Revised: Following Vaswani et al.

Original: ; Mildenhall et al.
Revised: ; Mildenhall et al.

Original: Zhong et al.
Revised: Zhong et al.

Original: (2021), we convert the low-dimensional coordinates to a high-dimensional representation using Fourier components, where the number of frequencies is selected as 10 for all our experiments.
Revised: (2021), we convert the low-dimensional coordinates to a high-dimensional representation using Fourier components, where the number of frequencies is selected as 10 for all our experiments.

Original: The derived view and position embeddings are each of dimension 63 .
Revised: The derived view and position embeddings are each of dimension 63 .

Original: X o → coordinate
Revised: X o → coordinate

Original: ( N rays , N pts , D )
Revised: ( N rays , N pts , D )

Original: X j → epipolar view features ( N rays , N pts , N views , D )
Revised: X j → epipolar view features ( N rays , N pts , N views , D )

Original: ∆ d
Revised: ∆ d

Original: → relative directions of source views wrt target views ( N rays , N pts , N views , 3) f Q , f K , f V , f P , f A , f O → functions that parameterize MLP layers
Revised: → relative directions of source views wrt target views ( N rays , N pts , N views , 3) f Q , f K , f V , f P , f A , f O → functions that parameterize MLP layers

Original: Q = f Q ( X o )
Revised: Q = f Q ( X o )

Original: K = f K ( X j )
Revised: K = f K ( X j )

Original: V =
Revised: V =

Original: f V ( X j )
Revised: f V ( X j )

Original: P = f P (∆ d )
Revised: P = f P (∆ d )

Original: A = K − Q [: , : , None , :] + P
Revised: A = K − Q [: , : , None , :] + P

Original: A = softmax( A, dim = − 2)
Revised: A = softmax( A, dim = − 2)

Original: O =
Revised: O =

Original: (( V + P ) · A ) .
Revised: (( V + P ) · A ) .

Original: sum(dim
Revised: sum(dim

Original: = 2)
Revised: = 2)

Original: O =
Revised: O =

Original: f O ( O )
Revised: f O ( O )

Original: Pseudocode.
Revised: Pseudocode.

Original: We provide a simple and efficient pytorch pseudo-code to implement the attention operations in the view, ray transformer blocks in Alg. 1, 2 respectively.
Revised: We provide a simple and efficient pytorch pseudo-code to implement the attention operations in the view, ray transformer blocks in Alg. 1, 2 respectively.

Original: We do not indicate the feedforward and layer normalization operations for simplicity.
Revised: We do not indicate the feedforward and layer normalization operations for simplicity.

Original: As seen in Alg.
Revised: As seen in Alg.

Original: 3, we reuse the epipolar view features X j to derive keys, and values across view transformer blocks.
Revised: 3, we reuse the epipolar view features X j to derive keys, and values across view transformer blocks.

Original: Therefore, one could further improve efficiency by computing them only once while also sharing the network weights across view transformer blocks or simply put f view i ( . ) represents the same function across different values of i .
Revised: Therefore, one could further improve efficiency by computing them only once while also sharing the network weights across view transformer blocks or simply put f view i ( . ) represents the same function across different values of i .

Original: This can be considered analogous to an unfolded recurrent neural network that updates itself iteratively but using the same weights.
Revised: This can be considered analogous to an unfolded recurrent neural network that updates itself iteratively but using the same weights.

Original: C T ENTATIVE E XTENSIONS C. A UTO -R EGRESSIVE D ECODING
Revised: C T ENTATIVE E XTENSIONS C. A UTO -R EGRESSIVE D ECODING

Original: The final rendered color is obtained by mean-pooling the outputs from the ray transformer block, and mapping the pooled feature vector to RGB via an MLP layer.
Revised: The final rendered color is obtained by mean-pooling the outputs from the ray transformer block, and mapping the pooled feature vector to RGB via an MLP layer.

Original: It can be understood that the target pixel’s color is strongly dependent on the closest point from the ray origin and weakly related to the
Revised: It can be understood that the target pixel’s color is strongly dependent on the closest point from the ray origin and weakly related to the

Original: Algorithm 2 Ray Attention: PyTorch like Pseudocode
Revised: Algorithm 2 Ray Attention: PyTorch like Pseudocode

Original: X o → coordinate aligned features ( N rays , N pts , D ) x
Revised: X o → coordinate aligned features ( N rays , N pts , D ) x

Original: → point coordinates (after position encoding) ( N rays , N pts , D )
Revised: → point coordinates (after position encoding) ( N rays , N pts , D )

Original: d → target view direction (after position encoding) ( N rays , N pts , D ) f Q , f K , f V , f P , f A , f O → functions that parameterize MLP layers
Revised: d → target view direction (after position encoding) ( N rays , N pts , D ) f Q , f K , f V , f P , f A , f O → functions that parameterize MLP layers

Original: X o = f P (concat( X o , d, x ))
Revised: X o = f P (concat( X o , d, x ))

Original: Q = f Q ( X o )
Revised: Q = f Q ( X o )

Original: K = f K ( X o )
Revised: K = f K ( X o )

Original: V =
Revised: V =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: o )
Revised: o )

Original: [Equation]
Revised: [Equation]

Original: Algorithm
Revised: Algorithm

Original: X j → epipolar view features ( N rays , N pts , N views , D ) x
Revised: X j → epipolar view features ( N rays , N pts , N views , D ) x

Original: → point coordinates (after position encoding) ( N rays , N pts , D )
Revised: → point coordinates (after position encoding) ( N rays , N pts , D )

Original: d → target view direction (after position encoding) ( N rays , N pts , D ) ∆ d → relative directions of source views wrt target views ( N rays , N pts , N views , 3) f view i , f ray i
Revised: d → target view direction (after position encoding) ( N rays , N pts , D ) ∆ d → relative directions of source views wrt target views ( N rays , N pts , N views , 3) f view i , f ray i

Original: → functions that parameterize view transforms, ray transformers at layer i respectively
Revised: → functions that parameterize view transforms, ray transformers at layer i respectively

Original: f rgb → functions that parameterize MLP layers i =
Revised: f rgb → functions that parameterize MLP layers i =

Original: X o = max N views i =1 ( X j ) while i < N layers do
Revised: X o = max N views i =1 ( X j ) while i < N layers do

Original: X o = f view i ( X o , X j , ∆ d )
Revised: X o = f view i ( X o , X j , ∆ d )

Original: X o = f ray
Revised: X o = f ray

Original: i
Revised: i

Original: ( X o , d, x ) i
Revised: ( X o , d, x ) i

Original: =
Revised: =

Original: i +end while
Revised: i +end while

Original: O =
Revised: O =

Original: Norm( X o )
Revised: Norm( X o )

Original: [Equation]
Revised: [Equation]

Original: farthest.
Revised: farthest.

Original: Revisiting Eq.
Revised: Revisiting Eq.

Original: 7, volumetric rendering attempts to compose point-wise color depending on the other points in a far to near fashion.
Revised: 7, volumetric rendering attempts to compose point-wise color depending on the other points in a far to near fashion.

Original: Motivated by this, we propose an auto-regressive decoder to better simulate the rendering process.
Revised: Motivated by this, we propose an auto-regressive decoder to better simulate the rendering process.

Original: Transformers have shown great success in auto-regressive decoding, more specifically in NLP (Vaswani et al., 2017).
Revised: Transformers have shown great success in auto-regressive decoding, more specifically in NLP (Vaswani et al., 2017).

Original: We borrow a similar strategy and replace the simpler MLP based color prediction with a series of transformer blocks - with self, cross attention layers.
Revised: We borrow a similar strategy and replace the simpler MLP based color prediction with a series of transformer blocks - with self, cross attention layers.

Original: In the first pass, the decoder is queried with positional encoding of the farthest point ( γ ( x N ) ) to generate an output feature representation of the same.
Revised: In the first pass, the decoder is queried with positional encoding of the farthest point ( γ ( x N ) ) to generate an output feature representation of the same.

Original: In the next step, the output token is concatenated with the second farthest point ( γ ( x N − 1 ) ) to query the decoder.
Revised: In the next step, the output token is concatenated with the second farthest point ( γ ( x N − 1 ) ) to query the decoder.

Original: This process repeats until all the points in the ray are queried in a far to near fashion.
Revised: This process repeats until all the points in the ray are queried in a far to near fashion.

Original: In the final pass, the encoded view direction ( γ ( d ) ) is concatenated with the per-point output features in the previous passes to query the decoder and the output token corresponding to the view direction is extracted.
Revised: In the final pass, the encoded view direction ( γ ( d ) ) is concatenated with the per-point output features in the previous passes to query the decoder and the output token corresponding to the view direction is extracted.

Original: The extracted token is mapped to RGB via an MLP layer.
Revised: The extracted token is mapped to RGB via an MLP layer.

Original: This entire process is summarized in Fig.
Revised: This entire process is summarized in Fig.

Original: The auto-regressive procedure closely resembles the volumetric rendering equation which iteratively blends and overrides the previous color when marching along a ray from far to near.
Revised: The auto-regressive procedure closely resembles the volumetric rendering equation which iteratively blends and overrides the previous color when marching along a ray from far to near.

Original: Transformer-based decoders used in language iteratively predict output tokens only during inference, i.e. they are trained in a non-autoregressive fashion due to the availability of ground truth output tokens in each step.
Revised: Transformer-based decoders used in language iteratively predict output tokens only during inference, i.e. they are trained in a non-autoregressive fashion due to the availability of ground truth output tokens in each step.

Original: Transferring the same to neural rendering is not possible, as we do not have access to ground truth color for each point sampled along the ray.
Revised: Transferring the same to neural rendering is not possible, as we do not have access to ground truth color for each point sampled along the ray.

Original: Hence, we require a loop to
Revised: Hence, we require a loop to

Original: S e l f A tt en t i on
Revised: S e l f A tt en t i on

Original: C r o ss
Revised: C r o ss

Original: A tt en t i on
Revised: A tt en t i on

Original: Ray Features
Revised: Ray Features

Original: S e l f A tt en t i on
Revised: S e l f A tt en t i on

Original: C r o ss
Revised: C r o ss

Original: A tt en t i on
Revised: A tt en t i on

Original: Ray Transformer γ(x N ) γ(x N-1 )
Revised: Ray Transformer γ(x N ) γ(x N-1 )

Original: γ(d) γ(x N ) O(x 1 )
Revised: γ(d) γ(x N ) O(x 1 )

Original: O N
Revised: O N

Original: O N O
Revised: O N O

Original: NO 1 O d
Revised: NO 1 O d

Original: O N
Revised: O N

Original: O N O N-1 O N O N-
Revised: O N O N-1 O N O N-

Original: [Figure]
Revised: [Figure]

Original: Auto Regressive
Revised: Auto Regressive

Original: [Table]
Revised: [Table]

Original: auto-regressively decode features even during training.
Revised: auto-regressively decode features even during training.

Original: This reduces the computational efficiency of the proposed strategy especially as the number of points sampled along the ray increases.
Revised: This reduces the computational efficiency of the proposed strategy especially as the number of points sampled along the ray increases.

Original: Therefore, we introduce a cacheing mechanism to store the per-layer outputs of the previous tokens and only compute the attention of a new token in the current pass.
Revised: Therefore, we introduce a cacheing mechanism to store the per-layer outputs of the previous tokens and only compute the attention of a new token in the current pass.

Original: This does not overcome the iterative loop during each forward pass but avoid redundant computations, which helps improve the decoding speed drastically when compared to the naive strategy.
Revised: This does not overcome the iterative loop during each forward pass but avoid redundant computations, which helps improve the decoding speed drastically when compared to the naive strategy.

Original: Due to computational constraints, we are only able to train GNT + AutoReg with much fewer rays sampled per iteration (500) when compared to other methods as discussed in Sec.
Revised: Due to computational constraints, we are only able to train GNT + AutoReg with much fewer rays sampled per iteration (500) when compared to other methods as discussed in Sec.

Original: Tab. 4 discusses single scene optimization results on the LLFF dataset and we can clearly see that the GNT + AutoReg improves the overall performance when compared to existing baselines, and improving the PSNR scores in complex scenes (Orchids) when compared to our own method without the decoder.
Revised: Tab. 4 discusses single scene optimization results on the LLFF dataset and we can clearly see that the GNT + AutoReg improves the overall performance when compared to existing baselines, and improving the PSNR scores in complex scenes (Orchids) when compared to our own method without the decoder.

Original: However, this is not consistent across all scenes and metrics.
Revised: However, this is not consistent across all scenes and metrics.

Original: This could be because of the fewer number of rays sampled and we expect our results would improve when scaled to similar settings.
Revised: This could be because of the fewer number of rays sampled and we expect our results would improve when scaled to similar settings.

Original: Nevertheless, this shows that the learnable decoder predicts per-point RGB features effectively without any supervision from the volumetric rendering equation.
Revised: Nevertheless, this shows that the learnable decoder predicts per-point RGB features effectively without any supervision from the volumetric rendering equation.

Original: C. A TTENTION G UIDED C OARSE - TO -F INE S AMPLING
Revised: C. A TTENTION G UIDED C OARSE - TO -F INE S AMPLING

Original: GNT’s ray transformer learns point-to-point correspondences which helps model visibility and occlusion or more formally point-wise density σ .
Revised: GNT’s ray transformer learns point-to-point correspondences which helps model visibility and occlusion or more formally point-wise density σ .

Original: Motivated by this hypothesis, we estimate the depth maps from the extracted attention maps and qualitatively analyze the same in Sec. 4.5.
Revised: Motivated by this hypothesis, we estimate the depth maps from the extracted attention maps and qualitatively analyze the same in Sec. 4.5.

Original: Therefore, we could conclude that the learned point-wise importance values can be considered equivalent to pointwise density or σ .
Revised: Therefore, we could conclude that the learned point-wise importance values can be considered equivalent to pointwise density or σ .

Original: To further test our claim, we attempt to use the learned point-wise correspondence values to sample “fine” points, which are then queried to GNT to render a higher-quality image.
Revised: To further test our claim, we attempt to use the learned point-wise correspondence values to sample “fine” points, which are then queried to GNT to render a higher-quality image.

Original: Due to the set-like property of attention, we directly query the fine points to the same network without using a separately trained “fine” network unlike other NeRF methods (Mildenhall et al., 2020; Barron et al., 2021;
Revised: Due to the set-like property of attention, we directly query the fine points to the same network without using a separately trained “fine” network unlike other NeRF methods (Mildenhall et al., 2020; Barron et al., 2021;

Original: Wang et al., 2021b; Liu et al., 2022).
Revised: Wang et al., 2021b; Liu et al., 2022).

Original: Please note that we follow the same training strategy from Sec.
Revised: Please note that we follow the same training strategy from Sec.

Original: 4.1 and only sample coarse-fine points during evaluation.
Revised: 4.1 and only sample coarse-fine points during evaluation.

Original: Tab.
Revised: Tab.

Original: 4 compares “GNT
Revised: 4 compares “GNT

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: + Fine” against other methods, and we can clearly see that it outperforms other SOTA methods in complex scenes like Orchids, performing even better than our own method without fine sampling.
Revised: + Fine” against other methods, and we can clearly see that it outperforms other SOTA methods in complex scenes like Orchids, performing even better than our own method without fine sampling.

Original: However, the performance improvements are not significant across all scenes and we attribute this to the lack of training with the coarse-fine sampling strategy and expect our results to improve further.
Revised: However, the performance improvements are not significant across all scenes and we attribute this to the lack of training with the coarse-fine sampling strategy and expect our results to improve further.

Original: In Fig.
Revised: In Fig.

Original: 9, we visualize the estimated depth values obtained from the learned attention maps during both coarse, and fine stages.
Revised: 9, we visualize the estimated depth values obtained from the learned attention maps during both coarse, and fine stages.

Original: We can clearly see that the fine depth map is able to better estimate differences between nearby pixels which results in a higher resolution output.
Revised: We can clearly see that the fine depth map is able to better estimate differences between nearby pixels which results in a higher resolution output.

Original: D A DDITIONAL R ESULTS
Revised: D A DDITIONAL R ESULTS

Original: Tables 5 and 6 include a breakdown of the quantitative results presented in the main paper into per-scene metrics.
Revised: Tables 5 and 6 include a breakdown of the quantitative results presented in the main paper into per-scene metrics.

Original: Our method quantitatively surpasses original NeRF and achieve on-par results with state-of-the-art methods.
Revised: Our method quantitatively surpasses original NeRF and achieve on-par results with state-of-the-art methods.

Original: Although we slightly underperform NLP (Suhail et al., 2021) on some scenes, we argue that the comparison is not fair because NLP requires much larger batch size and longer iterations.
Revised: Although we slightly underperform NLF (Suhail et al., 2021) on some scenes, we argue that the comparison is not fair because NLF requires much larger batch size and longer iterations.

Original: We also include videos to demonstrate our results in supplementary materials.
Revised: We also include videos to demonstrate our results in supplementary materials.

Original: E D EFERRED D ISCUSSION
Revised: E D EFERRED D ISCUSSION

Original: Discussion on Occlusion Awareness.
Revised: Discussion on Occlusion Awareness.

Original: Conceptually, view transformer attempts to find correspondence between the queried points and source views.
Revised: Conceptually, view transformer attempts to find correspondence between the queried points and source views.

Original: The learned attention amounts to a likelihood score that a pixel on the source view is an image of the same point in the 3D space, i.e., no points lies between the target point and the pixel.
Revised: The learned attention amounts to a likelihood score that a pixel on the source view is an image of the same point in the 3D space, i.e., no points lies between the target point and the pixel.

Original: NeuRay (Liu et al., 2022) leverages the cost volume from MVSNet (Yao et al., 2018) to predict per-pixel visibility and shows that introducing occlusion
Revised: NeuRay (Liu et al., 2022) leverages the cost volume from MVSNet (Yao et al., 2018) to predict per-pixel visibility and shows that introducing occlusion

Original: [Table]
Revised: [Table]

Original: NeRF 0.
Revised: NeRF 0.

Original: MipNeRF 0.
Revised: MipNeRF 0.

Original: NLF 0.
Revised: NLF 0.

Original: GNT 0.
Revised: GNT 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Models
Revised: Models

Original: Lego
Revised: Lego

Original: Chair
Revised: Chair

Original: Drums
Revised: Drums

Original: Ficus
Revised: Ficus

Original: Hotdog
Revised: Hotdog

Original: Materials
Revised: Materials

Original: Mic
Revised: Mic

Original: Ship
Revised: Ship

Original: LLFF 0.
Revised: LLFF 0.

Original: NeRF 0.
Revised: NeRF 0.

Original: MipNeRF 0.
Revised: MipNeRF 0.

Original: NLF 0.
Revised: NLF 0.

Original: GNT 0.
Revised: GNT 0.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: NeRF 0.
Revised: NeRF 0.

Original: NeX 0.
Revised: NeX 0.

Original: NLF 0.
Revised: NLF 0.

Original: GNT 0.
Revised: GNT 0.

Original: [Table]
Revised: [Table]

Original: Models
Revised: Models

Original: Room
Revised: Room

Original: Fern
Revised: Fern

Original: Leaves
Revised: Leaves

Original: Fortress
Revised: Fortress

Original: Orchids
Revised: Orchids

Original: Flower
Revised: Flower

Original: T-Rex
Revised: T-Rex

Original: Horns
Revised: Horns

Original: LLFF 0.
Revised: LLFF 0.

Original: NeRF 0.
Revised: NeRF 0.

Original: NeX 0.
Revised: NeX 0.

Original: NLF 0.
Revised: NLF 0.

Original: GNT 0.
Revised: GNT 0.

Original: [Table]
Revised: [Table]

Original: Avg
Revised: Avg

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: information is beneficial for multi-view aggregation in generalizable NeRF.
Revised: information is beneficial for multi-view aggregation in generalizable NeRF.

Original: We argue that instead of explicitly regressing the visibility, purely relying on epipolar geometry constrained attention can automatically learn how to infer occlusion, given prior works in Multi-View Stereo (MVS)
Revised: We argue that instead of explicitly regressing the visibility, purely relying on epipolar geometry constrained attention can automatically learn how to infer occlusion, given prior works in Multi-View Stereo (MVS)

Original: (Yang et al., 2022; Ding et al., 2021).
Revised: (Yang et al., 2022; Ding et al., 2021).

Original: In view transformer, the U-Net provides multi-scale features to the transformer, and the attention block acts as a matching algorithm, which selects the pixels from neighboring views that maximize view consistency.
Revised: In view transformer, the U-Net provides multi-scale features to the transformer, and the attention block acts as a matching algorithm, which selects the pixels from neighboring views that maximize view consistency.

Original: We defer empirical discussion to Sec. 4.5.
Revised: We defer empirical discussion to Sec. 4.5.

Original: Discussion on Depth Cuing.
Revised: Discussion on Depth Cuing.

Original: The ray transformer iteratively aggregates features according to the attention value.
Revised: The ray transformer iteratively aggregates features according to the attention value.

Original: This attention value can be regarded as the importance of each point to form the image, which reflects visibility and occlusion reasoned by point-to-point interaction.
Revised: This attention value can be regarded as the importance of each point to form the image, which reflects visibility and occlusion reasoned by point-to-point interaction.

Original: Therefore, we can interpret the average attention score for each point as the accumulated weights in volume rendering.
Revised: Therefore, we can interpret the average attention score for each point as the accumulated weights in volume rendering.

Original: In this sense, we can infer the depth map from the attention map by averaging the marching distance t i with the attention value.
Revised: In this sense, we can infer the depth map from the attention map by averaging the marching distance t i with the attention value.

Original: This implies our ray transformer learns geometry-aware 3D semantics on both feature space and attention map, which helps it generalize well across scenes.
Revised: This implies our ray transformer learns geometry-aware 3D semantics on both feature space and attention map, which helps it generalize well across scenes.

Original: We defer visualization and analysis to Sec. 4.5.
Revised: We defer visualization and analysis to Sec. 4.5.

Original: NLF (Suhail et al., 2021) proposes a similar two-stage rendering transformer, but it first extracts features on the epipolar lines and then aggregates epipolar features to get the pixel color.
Revised: NLF (Suhail et al., 2021) proposes a similar two-stage rendering transformer, but it first extracts features on the epipolar lines and then aggregates epipolar features to get the pixel color.

Original: We doubt this strategy may fail to generalize as epipolar features lack communication with each other and thus cannot induce geometry grounded semantics.
Revised: We doubt this strategy may fail to generalize as epipolar features lack communication with each other and thus cannot induce geometry grounded semantics.

Revised: F L IMITATIONS

Revised: Although our method achieves strong single-scene performance and achieves SOTA cross-scene generalization, we discuss certain limitations of our method.

Revised: The view transformer relies on epipolar constraints - i.e can only aggregate information from valid epipolar lines.

Revised: Therefore, non-epipolar scenes and complex light transport might not be captured by the view transformer.

Revised: Although we adopt a feature extractor with large receptive fields to encode global light transport and our view transformer empirically works well on complex lighting effects, what is captured by the image encoder remains unclear.

Revised: Moreover, epipolar correspondence for the boundary pixels sometimes are missing, which causes some minor artifacts (see Fig. 11).