Original: I S A TTENTION A LL T HAT N E RF N EEDS ?
Revised: I S A TTENTION A LL T HAT N E RF N EEDS ?

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present Generalizable NeRF Transformer ( GNT ), a transformer-based architecture that reconstructs Neural Radiance Fields (NeRFs) and learns to renders novel views on the fly from source views.
Revised: We present Generalizable NeRF Transformer ( GNT ), a transformer-based architecture that reconstructs Neural Radiance Fields (NeRFs) and learns to render novel views on the fly from source views.

Original: While prior works on NeRFs optimize a scene representation by inverting a handcrafted rendering equation, GNT achieves neural representation and rendering that generalizes across scenes using transformers at two stages.
Revised: While prior works on NeRFs optimize a scene representation by inverting a handcrafted rendering equation, GNT achieves neural representation and rendering that generalizes across scenes using transformers at two stages.

Original: (1) The view transformer leverages multi-view geometry as an inductive bias for attention-based scene representation, and predicts coordinatealigned features by aggregating information from epipolar lines on the neighboring views.
Revised: (1) The view transformer leverages multi-view geometry as an inductive bias for attention-based scene representation, and predicts coordinatealigned features by aggregating information from epipolar lines on the neighboring views.

Original: The ray transformer renders novel views using attention to decode the features from the view transformer along the sampled points during ray marching.
Revised: The ray transformer renders novel views using attention to decode the features from the view transformer along the sampled points during ray marching.

Original: Our experiments demonstrate that when optimized on a single scene, GNT can successfully reconstruct NeRF without an explicit rendering formula due to the learned ray renderer.
Revised: Our experiments demonstrate that when optimized on a single scene, GNT can successfully reconstruct NeRF without an explicit rendering formula due to the learned ray renderer.

Original: When trained on multiple scenes, GNT consistently achieves state-of-the-art performance when transferring to unseen scenes and outperform all other methods by ~ 10% on average.
Revised: When trained on multiple scenes, GNT consistently achieves state-of-the-art performance when transferring to unseen scenes and outperform all other methods by ~ 10% on average.

Original: Our analysis of the learned attention maps to infer depth and occlusion indicate that attention enables learning a physicallygrounded rendering.
Revised: Our analysis of the learned attention maps to infer depth and occlusion indicate that attention enables learning a physicallygrounded rendering.

Original: Our results show the promise of transformers as a universal modeling tool for graphics.
Revised: Our results show the promise of transformers as a universal modeling tool for graphics.

Original: Code will be made publicly available.
Revised: Please refer to our project page for video results: https://vita-group.github.io/GNT/

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural Radiance Field (NeRF) (Mildenhall et al., 2020) and its follow-up works (Barron et al., 2021;
Revised: Neural Radiance Field (NeRF) (Mildenhall et al., 2020) and its follow-up works (Barron et al., 2021;

Original: Zhang et al., 2020; Chen et al., 2022) have achieved remarkable success on novel view synthesis, generating photo-realistic, high-resolution, and view-consistent scenes.
Revised: Zhang et al., 2020; Chen et al., 2022) have achieved remarkable success on novel view synthesis, generating photo-realistic, high-resolution, and view-consistent scenes.

Original: Two key ingredients in NeRF are, (1) a coordinate-based neural network that maps each spatial position to its corresponding color and density, and (2) a differentiable volumetric rendering pipeline that composes the color and density of points along each ray cast from the image plane to generate the target pixel color.
Revised: Two key ingredients in NeRF are, (1) a coordinate-based neural network that maps each spatial position to its corresponding color and density, and (2) a differentiable volumetric rendering pipeline that composes the color and density of points along each ray cast from the image plane to generate the target pixel color.

Original: Optimizing a
Revised: Optimizing a

Original: NeRF can be regarded as an inverse imaging problem that fits a neural network to satisfy the observed views.
Revised: NeRF can be regarded as an inverse imaging problem that fits a neural network to satisfy the observed views.

Original: Such training leads to a major limitation of NeRF, making it a time-consuming optimization process for each scene (Chen et al., 2021a;
Revised: Such training leads to a major limitation of NeRF, making it a time-consuming optimization process for each scene (Chen et al., 2021a;

Original: Wang et al., 2021b; Yu et al., 2021).
Revised: Wang et al., 2021b; Yu et al., 2021).

Original: Recent works Neuray (Liu et al., 2022), IBRNet (Wang et al., 2021b), and PixelNerf (Yu et al., 2021) go beyond the coordinate-based network and rethink novel view synthesis as a cross-view imagebased interpolation problem.
Revised: Recent works Neuray (Liu et al., 2022), IBRNet (Wang et al., 2021b), and PixelNerf (Yu et al., 2021) go beyond the coordinate-based network and rethink novel view synthesis as a cross-view imagebased interpolation problem.

Original: Unlike the vanilla NeRF that tediously fits each scene, these methods synthesize a generalizable 3D representation by aggregating image features extracted from seen views according to the camera and geometry priors.
Revised: Unlike the vanilla NeRF that tediously fits each scene, these methods synthesize a generalizable 3D representation by aggregating image features extracted from seen views according to the camera and geometry priors.

Original: However, despite showing large performance gains, they unexceptionally decode the feature volume to a radiance field, and rely on classical volume rendering (Max, 1995; Levoy, 1988) to generate images.
Revised: However, despite showing large performance gains, they unexceptionally decode the feature volume to a radiance field, and rely on classical volume rendering (Max, 1995; Levoy, 1988) to generate images.

Original: Note that the volume rendering equation adopted in NeRF over-simplifies the optical modeling of solid surface (Yariv et al., 2021; Wang et al., 2021a), reflectance (Chen et al., 2021c; Verbin et al., 2021; Chen et al., 2022), inter-surface scattering and other effects.
Revised: Note that the volume rendering equation adopted in NeRF over-simplifies the optical modeling of solid surface (Yariv et al., 2021; Wang et al., 2021a), reflectance (Chen et al., 2021c; Verbin et al., 2021; Chen et al., 2022), inter-surface scattering and other effects.

Original: This implies that radiance fields along with volume rendering in NeRF are not a universal imaging model, which may have limited the generalization ability of NeRFs as well.
Revised: This implies that radiance fields along with volume rendering in NeRF are not a universal imaging model, which may have limited the generalization ability of NeRFs as well.

Original: In this paper, we first consider the problem of transferable novel view synthesis as a two-stage information aggregation process: the multi-view image feature fusion, followed by the samplingbased rendering integration.
Revised: In this paper, we first consider the problem of transferable novel view synthesis as a two-stage information aggregation process: the multi-view image feature fusion, followed by the samplingbased rendering integration.

Original: Our key contributions comes from using transformers (Vaswani et al., 2017) for both these stages.
Revised: Our key contributions come from using transformers (Vaswani et al., 2017) for both these stages.

Original: Transformers have had resounding success in language modeling (Devlin et al., 2018) and computer vision (Dosovitskiy et al., 2020) and their “self-attention” mechanism can be thought of as a universal trainable aggregation function.
Revised: Transformers have had resounding success in language modeling (Devlin et al., 2018) and computer vision (Dosovitskiy et al., 2020) and their “self-attention” mechanism can be thought of as a universal trainable aggregation function.

Original: In our case, for volumetric scene representation, we train a view transformer , to aggregate pixel-aligned image features (Saito et al., 2019) from corresponding epipolar lines to predict coordinate-wise features.
Revised: In our case, for volumetric scene representation, we train a view transformer , to aggregate pixel-aligned image features (Saito et al., 2019) from corresponding epipolar lines to predict coordinate-wise features.

Original: For rendering a novel view, we develop a ray transformer that composes the coordinate-wise point features along a traced ray via  attention mechanism.
Revised: For rendering a novel view, we develop a ray transformer that composes the coordinate-wise point features along a traced ray via the attention mechanism.

Original: These two form the Generalizable NeRF Transformer ( GNT ).
Revised: These two form the Generalizable NeRF Transformer ( GNT ).

Original: GNT simultaneously learns to represent scenes from source view images and  perform scene adaptive ray-based rendering using the learned attention mechanism.
Revised: GNT simultaneously learns to represent scenes from source view images and to perform sceneadaptive ray-based rendering using the learned attention mechanism.

Original: Remarkably, GNT predicts novel views using the captured images without fitting per-scene.
Revised: Remarkably, GNT predicts novel views using the captured images without fitting per scene.

Original: Our promising results endorse that transformers are strong, scalable anda versatile learning backbone for graphical rendering (Tewari et al., 2020).
Revised: Our promising results endorse that transformers are strong, scalable, and versatile learning backbones for graphical rendering (Tewari et al., 2020).

Original: Our key contributions are:
Revised: Our key contributions are:

Original: A view transformer to aggregate multi-view image features complying with epipolar geometry and to infer coordinate-aligned features.
Revised: A view transformer to aggregate multi-view image features complying with epipolar geometry and to infer coordinate-aligned features.

Original: A ray transformer for a learned ray-based rendering to predict target color.
Revised: A ray transformer for a learned ray-based rendering to predict target color.

Original: Experiments to demonstrate that GNT’s fully transformer-based architecture achieves stateof-the-art results on complex scenes and cross-scene generalization.
Revised: Experiments to demonstrate that GNT’s fully transformer-based architecture achieves stateof-the-art results on complex scenes and cross-scene generalization.

Original: Analysis of the attention module showing that GNT learns to be depth and occlusion aware.
Revised: Analysis of the attention module showing that GNT learns to be depth and occlusion aware.

Original: Overall, our combined Generalizable NeRF Transformer (GNT) demonstrates that many of the inductive biases that were thought necessary for view synthesis (e.g. persistent 3D model, hard-coded rendering equation) can be replaced with attention/transformer mechanisms.
Revised: Overall, our combined Generalizable NeRF Transformer (GNT) demonstrates that many of the inductive biases that were thought necessary for view synthesis ( e g. persistent 3D model hard-

Revised: coded

Revised: rendering equation) can be replaced with attention/transformer mechanisms.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Transformers (Vaswani et al., 2017) have emerged as a ubiquitous learning backbone that captures long-range correlation for sequential data.
Revised: Transformers (Vaswani et al., 2017) have emerged as a ubiquitous learning backbone that captures long-range correlation for sequential data.

Original: It has shown remarkable success in language understanding (Devlin et al., 2018;
Revised: It has shown remarkable success in language understanding (Devlin et al., 2018;

Original: Dai et al., 2019; Brown et al., 2020), computer vision (Dosovitskiy et al., 2020;
Revised: Dai et al., 2019; Brown et al., 2020), computer vision (Dosovitskiy et al., 2020;

Original: Liu et al., 2021), speech (Gulati et al., 2020) and even protein structure (Jumper et al., 2021) amongst others.
Revised: Liu et al., 2021), speech (Gulati et al., 2020) and even protein structure (Jumper et al., 2021) amongst others.

Original: In computer vision, (Dosovitskiy et al., 2020) were successful in demonstrating Vision
Revised: In computer vision, (Dosovitskiy et al., 2020) were successful in demonstrating Vision

Original: Trainsformers (ViT) for image classification.
Revised: Transformers (ViT) for image classification.

Original: Subsequent works extended ViT to other vision tasks, including object detection (Carion et al., 2020), segmentation (Chen et al., 2021b;
Revised: Subsequent works extended ViT to other vision tasks, including object detection (Carion et al., 2020), segmentation (Chen et al., 2021b;

Original: Wang et al., 2021c), video processing (Zhou et al., 2018a; Arnab et al., 2021), and 3D instance processing (Guo et al., 2021; Lin et al., 2021).
Revised: Wang et al., 2021c), video processing (Zhou et al., 2018a; Arnab et al., 2021), and 3D instance processing (Guo et al., 2021; Lin et al., 2021).

Original: In this work we apply transformers for view synthesis by learning to reconstruct neural radiance fields and render novel views.
Revised: In this work, we apply transformers for view synthesis by learning to reconstruct neural radiance fields and render novel views.

Original: Neural Radiance Fields (NeRF) introduced by Mildenhall et al.
Revised: Neural Radiance Fields (NeRF) introduced by Mildenhall et al.

Original: (2020) synthesizes consistent and photorealistic novel views by fitting each scene as a continuous 5D radiance field parameterized by an
Revised: (2020) synthesizes consistent and photorealistic novel views by fitting each scene as a continuous 5D radiance field parameterized by an

Original: MLP.
Revised: MLP.

Original: Since then, several works have improved NeRFs further.
Revised: Since then, several works have improved NeRFs further.

Original: For example, Mip-NeRF Barron et al.
Revised: For example, Mip-NeRF Barron et al.

Original: (2021; 2022) efficiently addresses scale of objects in unbounded scenes, Nex (Wizadwongsa et al., 2021) models large view dependent effects, others (Oechsle et al., 2021; Yariv et al., 2021; Wang et al., 2021a) improve the surface representation, extend to dynamic scenes (Park et al., 2021a;b;
Revised: (2021; 2022) efficiently addresses scale of objects in unbounded scenes, Nex (Wizadwongsa et al., 2021) models large view dependent effects, others (Oechsle et al., 2021; Yariv et al., 2021; Wang et al., 2021a) improve the surface representation, extend to dynamic scenes (Park et al., 2021a;b;

Original: Pumarola et al., 2021) , introduce lighting and reflection modeling (Chen et al., 2021c; Verbin et al., 2021), or leverage depth to regress from few views (Xu et al., 2022; Deng et al., 2022).
Revised: Pumarola et al., 2021) , introduce lighting and reflection modeling (Chen et al., 2021c; Verbin et al., 2021), or leverage depth to regress from few views (Xu et al., 2022; Deng et al., 2022).

Original: Our work aims to avoid per-scene training, similar to PixelNeRF (Yu et al., 2021), IBRNet (Wang et al., 2021b),
Revised: Our work aims to avoid per-scene training, similar to PixelNeRF (Yu et al., 2021), IBRNet (Wang et al., 2021b),

Original: MVSNeRF (Chen et al., 2021a), and NeuRay (Liu et al., 2022) which train a cross-scene multi-view aggregator and reconstruct the radiance field with a one-shot forward pass.
Revised: MVSNeRF (Chen et al., 2021a), and NeuRay (Liu et al., 2022) which train a cross-scene multi-view aggregator and reconstruct the radiance field with a one-shot forward pass.

Original: Transformer Meets Radiance Fields.
Revised: Transformer Meets Radiance Fields.

Original: Most similar to our work are NeRF methods that apply transformers for novel view synthesis and generalize across scenes.
Revised: Most similar to our work are NeRF methods that apply transformers for novel view synthesis and generalize across scenes.

Original: IBRNet (Wang et al., 2021b) processes sampled points on the ray using an MLP to predict color values and density features which are then input to a transformer to predict density.
Revised: IBRNet (Wang et al., 2021b) processes sampled points on the ray using an MLP to predict color values and density features which are then input to a transformer to predict density.

Original: Recently, NeRFormer (Reizenstein et al., 2021) and
Revised: Recently, NeRFormer (Reizenstein et al., 2021) and

Original: Wang et al.
Revised: Wang et al.

Original: (2022) use attention module to aggregate source views to construct feature volume with epipolar geometry constraints.
Revised: (2022) use attention module to aggregate source views to construct feature volume with epipolar geometry constraints.

Original: However, a key difference with our work is that, all of them decodethe latent feature representation to point-wise color and density, and rely on classic volume rendering to form the image, while our ray transformer learns to render the target pixel directly. Other works that use transformers but differ significantly from our approach and in application include (Lin et al., 2022) which generates novel views from just a single image via a vision transformer and SRT (Sajjadi et al., 2022b) which treats images and camera parameters in a latent space and trains
Revised: However, a key difference with our work is that, all of them decode

Original: Ray Transformer [Figure]
Revised: Ray Transformer View Transformer

Original: P oo li ng
Revised: P oo li ng

Original: Pred.
Revised: Pred.

Original: GT
Revised: GT

Original: Photometric Loss
Revised: Photometric Loss

Original: [Figure]
Revised: [Figure]

Original: Target Views
Revised: Target Views

Original: SourceViews
Revised: SourceViews

Original: E N C OD E R D E C OD E R
Revised: E N C OD E R D E C OD E R

Original: ImageEncoder ImageFeature Pos.
Revised: ImageEncoder ImageFeature Pos.

Original: Embed
Revised: Embed

Original: Feature Encoder
Revised: Feature Encoder

Original: [Figure] Representation
Revised: Scene Representation

Original: Neural Rendering
Revised: Neural Rendering

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original:  a transformer that directly maps camera pose embedding to the corresponding image without any physical constraints.
Revised: the latent feature representation to point-wise color and density, and rely on classic volume rendering to form the image, while our ray transformer learns to render the target pixel directly. Other works, that use transformers but differ significantly in methodology or application, include (Lin et al., 2022) which generates novel views from just a single image via a vision transformer and SRT (Sajjadi et al., 2022b) which treats images and camera parameters in a latent space and trains a transformer that directly maps camera pose embedding to the corresponding image without any physical constraints.

Original: An alternative route formulates view synthesis as rendering a sparsely observed 4D light field, rather than following NeRF’s 5D scene representation and volumetric rendering.
Revised: An alternative route formulates view synthesis as rendering a sparsely observed 4D light field, rather than following NeRF’s 5D scene representation and volumetric rendering.

Original: The recently proposed NLF (Suhail et al., 2021) uses an attention based framework to display light fields with view consistency, where the first transformer summarizes information on epipolar lines independently and then fuses epipolar features using a second transformer.
Revised: The recently proposed NLF (Suhail et al., 2021) uses an attention-based framework to display light fields with view consistency, where the first transformer summarizes information on epipolar lines independently and then fuses epipolar features using a second transformer.

Original: This differs from GNT, where we aggregate across views, and are hence able to generalize across scenes which NLF fails to do.
Revised: This differs from GNT, where we aggregate across views, and are hence able to generalize across scenes which NLF fails to do.

Original: Lately, GPNR (Suhail et al., 2022), which was developed concurrently to our work, generalizes NLF (Suhail et al., 2021) by also enabling cross-view communication through  attention mechanism.
Revised: Lately, GPNR (Suhail et al., 2022), which was developed concurrently with our work, generalizes NLF (Suhail et al., 2021) by also enabling cross-view communication through the attention mechanism.

Original: M ETHOD : M AKE A TTENTION A LL THAT N E RF N EEDS
Revised: [Equation] A TTENTION A LL THAT N E RF N EEDS

Original: Overview.
Revised: Overview.

Original: Given a set of N input views with known camera parameters { ( I i ∈
Revised: Given a set of N input views with known camera parameters { ( I i ∈

Original: R H
Revised: R H

Original: × W × 3 , P i ∈ R 3 ×) }
Revised: × W × 3 , P i ∈ R 3 ×) }

Original: Ni =1 , our goal is to synthesize novel views from arbitrary angles and also generalize to new scenes.
Revised: Ni =1 , our goal is to synthesize novel views from arbitrary angles and also generalize to new scenes.

Original: Our method can be divided into two stages: (1) construct the 3D representation from source views on the fly in the feature space, (2) re-render the feature field at the specified angle to synthesize novel views.
Revised: Our method can be divided into two stages: (1) construct the 3D representation from source views on the fly in the feature space, (2) re-render the feature field at the specified angle to synthesize novel views.

Original: Unlike PixelNeRF, IBRNet, MVSNeRF and Neuray that borrow classic volume rendering for view synthesis after the first multi-view aggregation stage, we propose transformers to model both stages.
Revised: Unlike PixelNeRF, IBRNet, MVSNeRF and Neuray that borrow classic volume rendering for view synthesis after the first multi-view aggregation stage, we propose transformers to model both stages.

Original: Our pipeline is depicted in Fig.
Revised: Our pipeline is depicted in Fig.

Original: First, the view transformer aggregates coordinate-aligned features from source views.
Revised: First, the view transformer aggregates coordinate-aligned features from source views.

Original: To enforce multi-view geometry, we inject the inductive bias of epipolar constraints into the attention mechanism.
Revised: To enforce multi-view geometry, we inject the inductive bias of epipolar constraints into the attention mechanism.

Original: After obtaining the feature representation of each point on the ray, the ray transformer composes point-wise features along the ray to form the ray color.
Revised: After obtaining the feature representation of each point on the ray, the ray transformer composes point-wise features along the ray to form the ray color.

Original: This pipeline constitutes GNT and it is trained end-to-end.
Revised: This pipeline constitutes GNT and it is trained end-to-end.

Original: 3. E PIPOLAR G EOMETRY C ONSTRAINED S CENE R EPRESENTATION
Revised: 3. E PIPOLAR G EOMETRY C ONSTRAINED S CENE R EPRESENTATION

Original: NeRF represents 3D scene as a radiance field F : ( x , θ ) (cid:55)→ ( c , σ ) , where each spatial coordinate x ∈ R 3 together with the viewing direction θ ∈ [ − π, π ] 2 is mapped to a color c ∈ R 3 plus density σ ∈ R + tuple.
Revised: NeRF represents 3D scene as a radiance field F : ( x , θ ) (cid:55)→ ( c , σ ) , where each spatial coordinate x ∈ R 3 together with the viewing direction θ ∈ [ − π, π ] 2 is mapped to a color c ∈ R 3 plus density σ ∈ R + tuple.

Original: Vanilla NeRF parameterizes the radiance field using an MLP, and recovers the scene in a backward optimization fashion, inherently limiting NeRF from generalizing to new scenes.
Revised: Vanilla NeRF parameterizes the radiance field using an MLP, and recovers the scene in a backward optimization fashion, inherently limiting NeRF from generalizing to new scenes.

Original: Generalizable NeRFs (Yu et al., 2021;
Revised: Generalizable NeRFs (Yu et al., 2021;

Original: Wang et al., 2021b; Chen et al., 2021a) construct the radiance  field in a feed-forward scheme, directly encoding multi-view images into 3D feature space, and decoding it to a color-density field.
Revised: Wang et al., 2021b; Chen et al., 2021a) construct the radiance [Figure] field in a feed-forward scheme, directly encoding multi-view images into 3D feature space, and decoding it to a color-density field.

Original: In our work, we adopt the similar feed-forward fashion to convert multi-view images into 3D representation, but instead of using physical variables (e.g., color and density), we model a 3D scene as a coordinate-aligned feature field F : ( x , θ ) (cid:55)→ f ∈ R d , where d is the dimension of the latent[Figure] space.
Revised: In our work, we adopt the similar feed-forward fashion to convert multi-view images into 3D representation, but instead of using physical variables (e.g., color and density), we model a 3D scene as a coordinate-aligned feature field F : ( x , θ ) (cid:55)→ f ∈ R d , where d is the dimension of the latent space.

Original: We formulate the feed-forward scene representation as follows:
Revised: We formulate the feed-forward scene representation as follows:

Original: [Equation]
Revised: [Equation]

Original: where V ( · ) is a function invariant to the permutation of input images to aggregate different views { I i , · · · , I N } into a coordinate-aligned feature field, and extracts features at a specific location.
Revised: where V ( · ) is a function invariant to the permutation of input images to aggregate different views { I i , · · · , I N } into a coordinate-aligned feature field, and extracts features at a specific location.

Original: We use transformers as a set aggregation function (Lee et al., 2019).
Revised: We use transformers as a set aggregation function (Lee et al., 2019).

Original: However, plugging in attention to globally attend to every pixel in the source images (Sajjadi et al., 2022b;a) is memory prohibitive and lacks multi-view geometric priors.
Revised: However, plugging in attention to globally attend to every pixel in the source images (Sajjadi et al., 2022b;a) is memory prohibitive and lacks multi-view geometric priors.

Original: Hence, we use epipolar geometry as an inductive bias that restricts each pixel to only attend to pixels that lie on the corresponding epipolar lines of the neighboring views.
Revised: Hence, we use epipolar geometry as an inductive bias that restricts each pixel to only attend to pixels that lie on the corresponding epipolar lines of the neighboring views.

Original: Specifically, we first encode each view to be a feature map F i =
Revised: Specifically, we first encode each view to be a feature map F i =

Original: ImageEncoder( I i ) ∈
Revised: ImageEncoder( I i ) ∈

Original: R H
Revised: R H

Original: × W × d .
Revised: × W × d .

Original: We expect the image encoder to extract not only shading information, but also material, semantics, and local/global complex light transport via its multi-scale architecture (Ronneberger et al., 2015).
Revised: We expect the image encoder to extract not only shading information, but also material, semantics, and local/global complex light transport via its multi-scale architecture (Ronneberger et al., 2015).

Original: To obtain the feature representation at a position x , we first project x to every source image, and interpolate the feature vector on the image plane.
Revised: To obtain the feature representation at a position x , we first project x to every source image, and interpolate the feature vector on the image plane.

Original: We then adopt a transformer (dubbed view transformer ) to combine all the feature vectors.
Revised: We then adopt a transformer (dubbed view transformer ) to combine all the feature vectors.

Original: Formally, this process can be written as below:
Revised: Formally, this process can be written as below:

Original: [Equation]
Revised: [Equation]

Original: where View-Transformer( · ) is a transformer encoder (see Appendix A)
Revised: where View-Transformer( · ) is a transformer encoder (see Appendix A)

Original: , Π i ( x ) projects x ∈ Ronto the i -th image plane by applying extrinsic matrix, and F i ( z , θ ) ∈
Revised: , Π i ( x ) projects x ∈ Ronto the i -th image plane by applying extrinsic matrix, and F i ( z , θ ) ∈

Original: R d computes the feature vector at position z ∈ R 2 via bilinear interpolation on the feature grids.
Revised: R d computes the feature vector at position z ∈ R 2 via bilinear interpolation on the feature grids.

Original: We use the transformer’s positional encoding γ ( · ) to concatenate the extracted feature vector with point coordinate, viewing direction, and relative directions of source views with respect to the target view (similar to Wang et al.
Revised: We use the transformer’s positional encoding γ ( · ) to concatenate the extracted feature vector with point coordinate, viewing direction, and relative directions of source views with respect to the target view (similar to Wang et al.

Original: (2021b)).
Revised: (2021b)).

Original: The detailed implementation of view transformer is depicted in Fig.
Revised: The detailed implementation of view transformer is depicted in Fig.

Original: We defer our elaboration on its memory-efficient design to Appendix B. We argue that  view transformer can detect occlusion through the pixel values like a stereo matching algorithm and selectively aggregate visible views (see details in Appendix E).
Revised: We defer our elaboration on its memory-efficient design to Appendix B. We argue that the view transformer can detect occlusion through the pixel values like a stereo-matching algorithm and selectively aggregate visible views (see details in Appendix E).

Original: 3. A TTENTION D RIVEN V OLUMETRIC R ENDERING
Revised: 3. A TTENTION D RIVEN V OLUMETRIC R ENDERING

Original: Volume rendering (App.
Revised: Volume rendering (App.

Original: Eq. 7), which simulates outgoing radiance from a volumetric field has been regarded as a key knob of NeRF’s success.
Revised: Eq. 7), which simulates outgoing radiance from a volumetric field has been regarded as a key knob of NeRF’s success.

Original: NeRF renders the color of a pixel by integrating the color and density along the ray cast from that pixel.
Revised: NeRF renders the color of a pixel by integrating the color and density along the ray cast from that pixel.

Original: Existing works, including NeRF (Sec. 2), all use handcrafted and simplified versions of this integration.
Revised: Existing works, including NeRF (Sec. 2), all use handcrafted and simplified versions of this integration.

Original: However, one can regard volume rendering as a weighted aggregation of all the point-wise output, in which the weights are globally dependent on the other points for occlusion modeling.
Revised: However, one can regard volume rendering as a weighted aggregation of all the point-wise output, in which the weights are globally dependent on the other points for occlusion modeling.

Original: This aggregation can be learned by a transformer such that point-wise colors can be mapped to token features, and attention scores correspond to transmittance (the blending weights).
Revised: This aggregation can be learned by a transformer such that point-wise colors can be mapped to token features, and attention scores correspond to transmittance (the blending weights).

Original: This is how we model the ray transformer which is illustrated in Fig.
Revised: This is how we model the ray transformer which is illustrated in Fig.

Original: 2b
Revised: 2b

Revised: [Table]

Original: To render the color of a ray r =
Revised: To render the color of a ray r =

Original: ( o , d ) , we can compute a feature representation f i =
Revised: ( o , d ) [Table] we can compute a feature representation f i =

Original: F ( x i , θ ) ∈ R d for each point x i sampled on r .
Revised: F ( x i , θ ) ∈ R d for each point x i sampled on r .

Original: In addition to this, we also add position encoding of spatial location and view direction into f i .
Revised: In addition to this, we also add position encoding of spatial location and view direction into f i .

Original: We obtain the rendered color by feeding the sequence of { f 1 , · · · , f M } into the ray transformer, perform mean pooling over all the predicted tokens, and map the pooled
Revised: We obtain the rendered color by feeding the sequence of { f 1 , · · · , f M } into the ray transformer, perform mean pooling over all the predicted tokens

Original: [Table]
Revised: [Table]

Original:  feature vector to RGB via an MLP:
Revised: and map the pooled feature vector to RGB via an MLP:

Original: [Equation]
Revised: [Equation]

Original: where t 1 , · · · , t M are uniformly sampled between near and far plane.
Revised: where t 1 , · · · , t M are uniformly sampled between near and far planes.

Original: Ray-Transformer is a standard transformer encoder [Table] and its pseudocode implementation is provided in Appendix B. Rendering on feature space utilizes rich geometric, optical and semantic information [Table]
Revised: Ray-Transformer is a standard transformer encoder, and its pseudocode implementation is provided in Appendix B. Rendering on feature space utilizes rich geometric, optical, and semantic information, which are intractable to be modeled explicitly.

Original: which are intractable to be modeled explicitly.

Original: We argue that our ray transformer can automatically adjust the attention distribution to control the sharpness of  reconstructed surface [Table] and bake desirable lighting effects from the illumination and material features.
Revised: We argue that our ray transformer can automatically adjust the attention distribution to control the sharpness of the reconstructed surface, and bake desirable lighting effects from the illumination and material features.

Original: Moreover, by exerting the expressiveness of  image encoder,  ray transformer can also overcome the limitation of ray casting and epipolar geometry to simulate complex light transport (e.g., refraction, reflection, etc.).
Revised: Moreover, by exerting the expressiveness of the image encoder, the ray transformer can also overcome the limitation of ray casting and epipolar geometry to simulate complex light transport (e.g., refraction, reflection, etc.).

Original: Interestingly, despite all in latent space, we can also infer some explicit physical properties (such as depth) from ray transformer.
Revised: Interestingly, despite all in latent space, we can also infer some explicit physical properties (such as depth) from ray transformer.

Original: See Appendix E for depth cuing.
Revised: See Appendix E for depth cueing.

Original: We also involve discussion on the extension to auto-regressive rendering and attention-based coarse-to-fine sampling in Appendix C.
Revised: We also involve discussion on the extension to auto-regressive rendering and attention-based coarse-to-fine sampling in Appendix C.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conductextensive experiments to compare GNT with several state-of-the-art(SOTA) methods for novel view synthesis.
Revised: We conduct experiments to compare GNT against state-of-the-art methods for novel view synthesis.

Original: We first provide qualitative and quantitative results in the per-scene training setting, followed by generalization to unseen scenes.
Revised: Our experiment settings include both per-scene optimization and cross-scene generalization.

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: Source and Target View Sampling.
Revised: Source and Target View Sampling.

Original: Following IBRNet, we construct pairs of source and target views for training by first selecting a target view, and then identifying a pool of k × N nearby views, from which N views are randomly sampled as source views.
Revised: Following IBRNet, we construct pairs of source and target views for training by first selecting a target view, and then identifying a pool of k × N nearby views, from which N views are randomly sampled as source views.

Original: This sampling strategy simulates various view densities during training and therefore helps the network generalize better.
Revised: This sampling strategy simulates various view densities during training and therefore helps the network generalize better.

Original: During training, the values for k and N are uniformly sampled at random from (1 , 3) and (8 , 12) respectively.
Revised: During training, the values for k and N are uniformly sampled at random from (1 , 3) and (8 , 12) respectively.

Original: Training / Inference Details.
Revised: Training / Inference Details.

Original: We train both the feature extraction network and GNT end-to-end on datasets of multi-view posed images using the Adam optimizer to minimize the mean-squared error between predicted and ground truth RGB pixel values.
Revised: We train both the feature extraction network and GNT end-to-end on datasets of multi-view posed images using the Adam optimizer to minimize the mean-squared error between predicted and ground truth RGB pixel values.

Original: The base learning rates for the feature extraction network and GNT are 10 − 3 and 5 × 10 − 4 respectively, which decay exponentially over training steps.
Revised: The base learning rates for the feature extraction network and GNT are 10 − 3 and 5 × 10 − 4 respectively, which decay exponentially over training steps.

Original: For all our experiments, we train for 250,000 steps with 4096 rays sampled in each iteration.
Revised: For all our experiments, we train for 250,000 steps with 4096 rays sampled in each iteration.

Original: Unlike most NeRF methods, we do not use separate coarse, fine networks and therefore to bring GNT to a comparable experimental setup, we sample 192 coarse points per ray across all experiments (unless otherwise specified).
Revised: Unlike most NeRF methods, we do not use separate coarse, fine networks and therefore to bring GNT to a comparable experimental setup, we sample 192 coarse points per ray across all experiments (unless otherwise specified).

Original: Metrics.
Revised: Metrics.

Original: We use three widely adopted metrics: Peak Signal-to-Noise Ratio (PSNR), Structural 
Revised: We use three widely adopted metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM)

Original: Similarity Index Measure (SSIM) (Wang et al., 2004), and the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018).
Revised: (Wang et al., 2004), and the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018).

Original: We report the averages of each metric over different views in each scene (for single scene experiments) and across multiple scenes in each dataset (for generalization experiments).
Revised: We report the averages of each metric over different views in each scene (for single-scene experiments) and across multiple scenes in each dataset (for generalization experiments).

Original: We additionally report the geometric mean of 10 − PSNR/ 10 , √ 1 − SSIM and LPIPS, which provides a summary of the three metrics for easier comparison (Barron et al., 2021). 
Revised: We additionally report the geometric mean of 10 − PSNR/ 10 , √ 1 − SSIM and LPIPS, which provides a summary of the three metrics for easier comparison (Barron et al., 2021). [Figure]

Original: 4. S INGLE S CENE R ESULTS
Revised: 4. S INGLE S CENE R ESULTS

Original: Datasets.
Revised: Datasets.

Original: To evaluate the single scene view generation capacity of GNT, we perform experiments on datasets containing synthetic rendering of objects and real images of complex scenes.
Revised: To evaluate the single scene view generation capacity of GNT, we perform experiments on datasets containing synthetic rendering of objects and real images of complex scenes.

Original: In these[Figure] experiments, we use the same resolution and train/test splits as NeRF (Mildenhall et al
Revised: In these experiments, we use the same resolution and train/test splits as NeRF (Mildenhall et al., 2020).

Original: Local Light Field Fusion (LLFF) dataset : Introduced by Mildenhall et al.
Revised: Local Light Field Fusion (LLFF) dataset : Introduced by Mildenhall et al.

Original: (2019), it consists offorward facing captures of real-world scenes using a smartphone.
Revised: (2019), it consists offorward facing captures of real-world scenes using a smartphone.

Original: We report average scores across {Orchids, Horns, Trex, Room, Leaves, Fern, Fortress} and the metrics are summarized in Tab. 1b.
Revised: We report average scores across {Orchids, Horns, Trex, Room, Leaves, Fern, Fortress} and the metrics are summarized in Tab. 1b.

Original: NeRF Synthetic Dataset : The synthetic dataset introduced by (Mildenhall et al., 2020) consists of 8, 360°scenes of objects with complicated geometry and realistic material.
Revised: NeRF Synthetic Dataset : The synthetic dataset introduced by (Mildenhall et al., 2020) consists of 8, 360°scenes of objects with complicated geometry and realistic material.

Original: Each scene consists of images rendered from viewpoints randomly sampled on a hemisphere around the object.
Revised: Each scene consists of images rendered from viewpoints randomly sampled on a hemisphere around the object.

Original: Similar to the experiments on the LLFF dataset, we report the average metrics across all eight scenes in Tab. 1a.
Revised: Similar to the experiments on the LLFF dataset, we report the average metrics across all eight scenes in Tab. 1a.

Original: Discussion.
Revised: Discussion.

Original: We compare our GNT with LLFF , NeRF , MipNeRF , NeX , and NLF.
Revised: We compare our GNT with LLFF , NeRF , MipNeRF , NeX , and NLF.

Original: Compared to other methods, we utilize a smaller batch size (specifically GNT samples 4096 rays per batch while NLF samples as much as 16384 rays per batch) and only sample coarse points fed into the network in one single forward pass unlike most methods which use a two stage coarse-fine sampling strategy.
Revised: Compared to other methods, we utilize a smaller batch size (specifically GNT samples 4096 rays per batch while NLF samples as much as 16384 rays per batch) and only sample coarse points fed into the network in one single forward pass, unlike most methods that use a two-stage coarse-fine sampling strategy.

Original: These hyperparameters have a strong correlation with the rendered image quality, leaving our method at a disadvantage.
Revised: These hyperparameters have a strong correlation with the rendered image quality, leaving our method at a disadvantage.

Original: Despite these differences, GNT still manages to outperform most methods and performs on-par when compared to SOTA NLF method on both LLFF and Synthetic datasets.
Revised: Despite these differences, GNT still manages to outperform most methods and performs on par when compared to SOTA NLF method on both LLFF and Synthetic datasets.

Original: We provide a scene-wise breakdown of results on both these datasets in Appendix D (Tab.
Revised: We provide a scene-wise breakdown of results on both these datasets in Appendix D (Tab.

Original: In complex scenes like Drums, Ship, and Leaves, GNT manages to outperform other methods more substantially by 2 . 49 dB, 0 .
Revised: In complex scenes like Drums, Ship, and Leaves, GNT manages to outperform other methods more substantially by 2 . 49 dB, 0 .

Original: 82 dB and 0 .
Revised: 82 dB and 0 .

Original: 10 dB respectively.
Revised: 10 dB respectively.

Original: This indicates the effectiveness of our attention-driven volumetric rendering to model complex conditions.
Revised: This indicates the effectiveness of our attention-driven volumetric rendering to model complex conditions.

Original: Interestingly, even in the worse performing scenes by PSNR (e.g. T-Rex), GNT achieves best perceptual metric scores across all scenes in the LLFF dataset (i.e LPIPS ~ 27% ↓ ).
Revised: Interestingly, even in the worse performing scenes by PSNR (e.g. T-Rex), GNT achieves best perceptual metric scores across all scenes in the LLFF dataset (i.e LPIPS ~ 27% ↓ ).

Original: This could be because PSNR fails to measure structural distortions, blurring, has high sensitivity towards brightness, and hence does not effectively measure visual quality.
Revised: This could be because PSNR fails to measure structural distortions, blurring, has high sensitivity towards brightness, and hence does not effectively measure visual quality.

Original: Similar inferences are discussed in Lin et al.
Revised: Similar inferences are discussed in Lin et al.

Original: (2022) regarding discrepancies in
Revised: (2022) regarding discrepancies in

Original: PSNR scores and their correlation to rendered image quality.
Revised: PSNR scores and their correlation to rendered image quality.

Original: Figs.
Revised: Fig.

Original: 3 provides qualitative comparisons on the Orchids, Drums scene respectively and we can clearly see that GNT recovers the edge details of objects (in the case of Orchids) and models complex lighting effect like specular reflection (in the case of Drums) more accurately.
Revised: 3 provides qualitative comparisons on the Orchids, Drums scene respectively and we can clearly see that GNT recovers the edge details of objects (in the case of Orchids) and models complex lighting effect like specular reflection (in the case of Drums) more accurately.

Original: 4. G ENERALIZATION TO U NSEEN S CENES
Revised: 4. G ENERALIZATION TO U NSEEN S CENES

Original: Datasets.
Revised: Datasets.

Original: GNT leverages multi-view features complying with epipolar geometry, enabling generalization to unseen scenes.
Revised: GNT leverages multi-view features complying with epipolar geometry, enabling generalization to unseen scenes.

Original: We follow the experimental protocol in IBRNetto evaluate the cross-scene generalization of GNT and use the following datasets for training/evaluation resectively.
Revised: We follow the experimental protocol in IBRNetto evaluate the cross-scene generalization of GNT and use the following datasets for training and evaluation, respectively. [Table]

Original:  (a) Training Datasets consists of both real, and synthetic data.
Revised: [Table] (a) Training Datasets consists of both real, and synthetic data.

Original: For synthetic data, we use object renderings of 1023 models from Google Scanned Object (Downs et al For real data , we make [Table]
Revised: For synthetic data, we use object renderings of 1023 models from Google Scanned Object (Downs et al., 2022).

Original: [Table] [Table] [Table] use of RealEstate10K(Zhou et al., 2018b), 100 scenes from the Spaces dataset (Flynn et al., 2019), and 102 real scenes from handheld cellphone captures (Mildenhall et al., 2019; Wang
Revised: For real data, we make use of RealEstate10K

Original: Discussion.
Revised: Discussion.

Original: We compare our method with PixelNeRF (Yu et al., 2021), MVSNeRF (Chen et al., 2021a), IBRNet (Wang et al., 2021b), and NeuRay (Liu et al., 2022).
Revised: We compare our method with PixelNeRF (Yu et al., 2021), MVSNeRF (Chen et al., 2021a), IBRNet (Wang et al., 2021b), and NeuRay (Liu et al., 2022).

Original: As seen from Tab.
Revised: As seen from Tab.

Original: 2a, our method outperforms SOTA by ~ 17% ↓ , ~ 9% ↓ average scores in the LLFF, Synthetic datasets respectively.
Revised: 2a, our method outperforms SOTA by ~ 17% ↓ , ~ 9% ↓ average scores in the LLFF, Synthetic datasets respectively.

Original: This indicates the effectiveness of our proposed view transformer to extract generalizable scene representations.
Revised: This indicates the effectiveness of our proposed view transformer to extract generalizable scene representations.

Original: Similar to the single scene experiments, we observe significantly better perceptual metric scores 3% ↑ SSIM, 27% ↓ LPIPS in both datasets.
Revised: Similar to the single scene experiments, we observe significantly better perceptual metric scores 3% ↑ SSIM, 27% ↓ LPIPS in both datasets.

Original: We show qualitative results in Fig.where GNT renders novel views with clearly better visual quality when compared to other methods.
Revised: We show qualitative results in Fig.where GNT renders novel views with clearly better visual quality when compared to other methods.

Original: Specifically, as seen from the second row in Fig. 5, GNT is able to handle regions that are sparsely visible in the source views and generates images of comparable visual quality as NeuRay even with no explicit supervision for occlusion.
Revised: Specifically, as seen from the second row in Fig. 5, GNT is able to handle regions that are sparsely visible in the source views and generates images of comparable visual quality as NeuRay even with no explicit supervision for occlusion.

Original: The combination of our enhanced scene representation (view transformer) and learnable renderer (ray transformer) contribute to these improved results.
Revised: We also provide an additional comparison against SRT (Sajjadi et al., 2022b) in Appendix D (Tab. 5), where our GNT significantly generalizes better.

Original: GNT can learn to adapt to refraction and reflection in scenes.
Revised: GNT can learn to adapt to refraction and reflection in scenes.

Original: Encouraged by the promise shown by GNT in modelling reflections in the Drums scene via pre-scene training, we further directly evaluate pre-trained GNT on the Shiny dataset (Wizadwongsa et al., 2021), that contains several challenging view-dependent effects, such as the rainbow reflections on a CD, and the refraction through liquid bottles.
Revised: Encouraged by the promise shown by GNT in modeling reflections in the Drums scene via pre-scene training, we further directly evaluate pre-trained GNT on the Shiny dataset (Wizadwongsa et al., 2021), which contains several challenging view-dependent effects, such as the rainbow reflections on a CD, and the refraction through liquid bottles.

Original: Technically, the full formulation of volume rendering (radiative transfer equation, as used in modern volume path tracers) is capable of handling all these effects.
Revised: Technically, the full formulation of volume rendering (radiative transfer equation, as used in modern volume path tracers) is capable of handling all these effects.

Original: However, standard NeRFs use a simplified formulation which does not simulate all physical effects, and hence easily fail to capture these effects.
Revised: However, standard NeRFs use a simplified formulation which does not simulate all physical effects, and hence easily fail to capture these effects.

Original: Tab. 2b presents the numerical results of GNT when generalizing to Shiny dataset.
Revised: Tab. 2b presents the numerical results of GNT when generalizing to Shiny dataset.

Original: Notably, our GNT outperforms state-of-the-art GPNR (Suhail et al., 2022) by 3dB in PSNR and ~40% in average metric.
Revised: Notably, our GNT outperforms state-of-the-art GPNR (Suhail et al., 2022) by 3dB in PSNR and ~40% in average metric.

Original: Compared with per-scene optimized NeRFs, GNT outperforms many of them and even approaches to the best performer NLF (Suhail et al., 2021) without any extra training.
Revised: Compared with per-scene optimized NeRFs, GNT outperforms many of them and even approaches to the best performer NLF (Suhail et al., 2021) without any extra training.

Original: This further supports our argument that cross-scene training can help learn a better renderer.
Revised: This further supports our argument that cross-scene training can help learn a better renderer.

Original: Fig.
Revised: Fig.

Original: 4 exhibits rendering results on the two example scenes of Lab and CD.
Revised: 4 exhibits rendering results on the two example scenes of Lab and CD.

Original: Comparing to the baseline (NeX), GNT is able to reconstruct complex refractions through test tube, and the interference patterns on the disk with higher quality , indicating the strong flexibility and adaptivity of our learnable renderer.
Revised: Compared to the baseline (NeX), GNT is able to reconstruct complex refractions through test tube, and the interference patterns on the disk with higher quality, indicating the strong flexibility and adaptivity of our learnable renderer.

Original: That “serendipity" is intriguing to us, since the presence of refraction and scattering means that any technique that only uses samples along a single ray will not be able to properly simulate the full light transport.
Revised: That “serendipity" is intriguing to us, since the presence of refraction and scattering means that any technique that only uses samples along a single ray will not be able to properly simulate the full light transport.

Original: We conjecture GNT’s success in modeling those challenging physical scenes to be the full transformerarchitectures for not only rays but also views: the view encoder aggregates multi-ray information and can in turn tune the ray encoding itself.
Revised: We conjecture GNT’s success in modeling those challenging physical scenes to be the full transformer

Original: [Figure]
Revised: [Figure]

Revised: architectures for not only rays but also views: the view encoder aggregates multi-ray information and can in turn tune the ray encoding itself.

Original: [Figure]
Revised: [Figure]

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: We conduct the following ablation studies on the Drums scene to validate our architectural designs.
Revised: We conduct the following ablation studies on the Drums scene to validate our architectural designs.

Original: One-Stage Transformer :
Revised: One-Stage Transformer :

Original: We convert the point-wise epipolar features into one single sequence and pass it through a “one-stage transformer” network with standard dot-product self-attention layers, without considering our two-stage pipeline: view and ray aggregation. 
Revised: We convert the point-wise epipolar features into one single sequence and pass it through a “one-stage transformer” network with standard dot-product self-attention layers, without considering our two-stage pipeline: view and ray aggregation. [Figure]

Original: [Table]
Revised: [Table]

Original: Epipolar Agg.
Revised: Epipolar Agg.

Original: → View Agg.
Revised: → View Agg.

Original: : Moving to a two-stage transformer, we train a network which first aggregates features from the points along the epipolar lines followed by feature aggregation across epipolar lines on different reference views (in contrast to
Revised: : Moving to a two-stage transformer, we train a network that first aggregates features from the points along the epipolar lines followed by feature aggregation across epipolar lines on different reference views (in contrast to

Original: GNT’s first view aggregation then ray aggregation).
Revised: GNT’s first view aggregation then ray aggregation).

Original: This two-stage aggregation resembles the strategies adopted in NLF (Suhail et al., 2021).
Revised: This two-stage aggregation resembles the strategies adopted in NLF (Suhail et al., 2021).

Original: Dot Product-Attention View Transformer :
Revised: Dot Product-Attention View Transformer :

Original: Next we train a network that uses the standard dotproduct attention in the view transformer blocks, in contrast to our proposed memory-efficient subtraction-based attention (see Appendix B).
Revised: Next we train a network that uses the standard dotproduct attention in the view transformer blocks, in contrast to our proposed memory-efficient subtraction-based attention (see Appendix B).

Original: w/ Volumetric Rendering :
Revised: w/ Volumetric Rendering :

Original: Last but not least, we train a network to predict per-point RGB and density values from the point features output by  view aggregator, and compose them using the volumetric rendering equation, instead of our learned attention-driven volumetric renderer.
Revised: Last but not least, we train a network to predict per-point RGB and density values from the point feature output by the view aggregator, and compose them using the volumetric rendering equation, instead of our learned attention-driven volumetric renderer.

Original: We report the performance of the above investigation in Tab. 3.
Revised: We report the performance of the above investigation in Tab. 3.

Original: We verify that our design of two-stage transformer is superior than one-stage aggregation or an alternative two-stage pipeline (Suhail et al., 2021) since our renderer strictly complies with the multi-view geometry.
Revised: We verify that our design of two-stage transformer is superior to one-stage aggregation or an alternative two-stage pipeline (Suhail et al., 2021) since our renderer strictly complies with the multi-view geometry.

Original: Compared with dot-product attention, subtraction-based attention achieves slightly higher overall scores.
Revised: Compared with dot-product attention, subtraction-based attention achieves slightly higher overall scores.

Original: This also indicates the performance of GNT does not heavily rely on the choice of attention operation.
Revised: This also indicates the performance of GNT does not heavily rely on the choice of attention operation.

Original: What matters is bringing in  attention mechanism for cross-point interaction.
Revised: What matters is bringing in the attention mechanism for cross-point interaction.

Original: For practical usage, we also consider the memory efficiency in our view transformer.
Revised: For practical usage, we also consider the memory efficiency in our view transformer.

Original: Our ray transformer outperforms the classic volumetric rendering implies the advantage of adopting a data-driven renderer. 
Revised: Our ray transformer outperforms the classic volumetric rendering, implying the advantage of adopting a data-driven renderer. [Figure]

Original: 4. I NTERPRETATION ON A TTENTION M APS
Revised: I NTERPRETATION ON A TTENTION M APS

Original:  The use of transformers as a core element in GNT enables interpretation by analyzing the attention weights.
Revised: are occluded The use of transformers as a core element in GNT enables interpretation by analyzing the attention weights.

Original: As discussed earlier, view transformer finds correspondence between the queried points, and[Figure] neighboring views which enables it to pay attention to more “visible” views or be occlusion-aware.
Revised: As discussed earlier, view transformer finds correspondence between the queried points, and neighboring views which enables it to pay attention to more “visible” views or be occlusion-aware.

Original: Similarly, the ray transformer captures point-to-point interactions which enables it to model the relative importance of each point or be depth-aware.
Revised: Similarly, the ray transformer captures point-to-point interactions which enable it to model the relative importance of each point or be depth-aware.

Original: We validate our hypothesis by visualization.
Revised: We validate our hypothesis by visualization.

Original: [Figure]

Original: transformer learns to in the target view to its corresponding regions in the source views which are least occluded

Original: View Attention.
Revised: View Attention.

Original: To visualize the view-wise attention maps learnt by our model, we use the attention matrix from Eq.
Revised: To visualize the view-wise attention maps learned by our model, we use the attention matrix from Eq.

Original: 9 and collapse the channel dimension by performing mean pooling.
Revised: 9 and collapse the channel dimension by performing mean pooling.

Original: We then identify the view-number which is assigned maximum attention with respect to each point and then compute the most repeating view number across points along a ray (by computing mode).
Revised: We then identify the view number which is assigned maximum attention with respect to each point and then compute the most repeating view number across points along a ray (by computing mode).

Original: These “view importance” values denote the view which has maximum correspondence with the target pixel’s color.
Revised: These “view importance” values denote the view which has maximum correspondence with the target pixel’s color.

Original: Fig.
Revised: Fig.

Original: 6 visualizes the source view correspondence with every pixel in the target view.
Revised: 6 visualizes the source view correspondence with every pixel in the target view.

Original: Given a region in the target view, GNT attempts to pay maximum attention to a source view which is least occluded in the same region.
Revised: Given a region in the target view, GNT attempts to pay maximum attention to a source view that is least occluded in the same region.

Original: For example: In Fig. 6, the truck’s bucket is most visible from view number 8, hence the regions corresponding to the same are orange colored, while regions towards the front of the lego areare most visible from view 
Revised: For example: In Fig. 6, the truck’s bucket is most visible from view number 8, hence the regions corresponding to the same are orange colored, while regions towards the front of the lego are most visible from view number 7 (yellow).

Original: Ray Attention.
Revised: Ray Attention.

Original: To visualize the attention maps across points in a ray, we use the attention matrix from Eq. 4 and collapse the head dimension by performing mean pooling.
Revised: To visualize the attention maps across points in a ray, we use the attention matrix from Eq. 4 and collapse the head dimension by performing mean pooling.

Original: From the derived matrix, we select a point and extract its relative importance with every other point.
Revised: From the derived matrix, we select a point and extract its relative importance with every other point.

Original: We then compute a depth map from these learned point-wise correspondence values by multiplying it with the marching distance of each point and sum-pooling along a ray.
Revised: We then compute a depth map from these learned point-wise correspondence values by multiplying it with the marching distance of each point and sum-pooling along a ray.

Original: Fig.
Revised: Fig.

Original: 7 plots the depth maps computed from the learned attention values in the ray transformer block.
Revised: 7 plots the depth maps computed from the learned attention values in the ray transformer block.

Original: We can clearly see that the obtained depth maps have a physical meaning i.e pixels closer to the view directions are blue while the ones farther away are red.
Revised: We can clearly see that the obtained depth maps have a physical meaning i.e pixels closer to the view directions are blue while the ones farther away are red.

Original: Therefore, with no explicit supervision, GNT learns to physically ground its attention maps.
Revised: Therefore, with no explicit supervision, GNT learns to physically ground its attention maps.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present Generalizable NeRF Transformer (GNT), a pure transformer based architecture that efficiently reconstructs NeRFs on the fly.
Revised: We present Generalizable NeRF Transformer (GNT), a pure transformer-based architecture that efficiently reconstructs NeRFs on the fly.

Original: The view transformer of GNT leverages epipolar geometry as an inductive bias for scene representation.
Revised: The view transformer of GNT leverages epipolar geometry as an inductive bias for scene representation.

Original: The ray transformer renders novel views by ray marching and decoding the sequences of sampled point features using  attention mechanism.
Revised: The ray transformer renders novel views by ray marching and decoding the sequences of sampled point features using the attention mechanism.

Original: Extensive experiments demonstrate that GNT can improve both single-scene and cross-scene training results, and demonstrates “out of the box” promise for refraction and reflection scenes.
Revised: Extensive experiments demonstrate that GNT improves both single-scene and cross-scene training results, and demonstrates “out of the box” promise for refraction and reflection scenes.

Original: We also show by visualization that depth and occlusion can be inferred from attention maps.
Revised: We also show by visualization that depth and occlusion can be inferred from attention maps.

Original: This implies that pure attention can be  “universal modeling tool” for  physically-grounded rendering process.
Revised: This implies that pure attention can be a “universal modeling tool” for the physically-grounded rendering process.

Original: Interesting future directions include relaxing the epipolar constraints to simulate more complicated light transport. 
Revised: Future directions include relaxing the epipolar constraints to simulate more complicated light transport. A CKNOWLEDGMENTS

Revised: We thank Pratul Srinivasan for his comments on a draft of this work.

Original: A P RELIMINARIES
Revised: A P RELIMINARIES

Original: Self-Attention and Transformer.
Revised: Self-Attention and Transformer.

Original: Multi-Head Self-Attention (MHA) is the key ingredient of transformers (Vaswani et al., 2017).
Revised: Multi-Head Self-Attention (MHA) is the key ingredient of transformers (Vaswani et al., 2017).

Original: Data is first tokenized into sequences and a pairwise score is computed to weight the relation of each token with all the others in a given input context.
Revised: Data is first tokenized into sequences and a pairwise score is computed to weight the relation of each token with all the others in a given input context.

Original: Formally, let X ∈ R N × d represent some sequential data with N tokens of d -dimension.
Revised: Formally, let X ∈ R N × d represent some sequential data with N tokens of d -dimension.

Original: A self-attention layer transforms the feature matrix as below:
Revised: A self-attention layer transforms the feature matrix as below:

Original: [Equation]
Revised: [Equation]

Original: where A ∈ R N × N is called the attention matrix, softmax( · ) operation normalizes the attention matrix row-wise, and α ( · ) represents a pair-wise relation function, most commonly the dot-product α
Revised: where A ∈ R N × N is called the attention matrix, softmax( · ) operation normalizes the attention matrix row-wise, and α ( · ) represents a pair-wise relation function, most commonly the dot-product α

Original: ( X i , X j ) =
Revised: ( X i , X j ) =

Original: f Q ( X i )
Revised: f Q ( X i )

Original: ⊤ f K ( X j ) /γ , where f Q ( · ) , f K ( · ) , f V ( · ) are called query, key, and value mapping functions.
Revised: ⊤ f K ( X j ) /γ , where f Q ( · ) , f K ( · ) , f V ( · ) are called query, key, and value mapping functions.

Original: In a standard transformer, they are chosen as fully-connected layers.
Revised: In a standard transformer, they are chosen as fully-connected layers.

Original: This self-attention is akin to an aggregation operation.
Revised: This self-attention is akin to an aggregation operation.

Original: Multi-Head Self-Attention (MHA) sets a group of self-attention blocks, and adopts a linear layer to project them onto the output space:
Revised: Multi-Head Self-Attention (MHA) sets a group of self-attention blocks, and adopts a linear layer to project them onto the output space:

Original: [Equation]
Revised: [Equation]

Original: Following an MHA block, one standard layer of transformer also adopts a Feed-Forward Network (FFN) to do point-wise feature transformation, as well as skip connection and layer normalization to stablize training.
Revised: Following an MHA block, one standard layer of transformer also adopts a Feed-Forward Network (FFN) to do point-wise feature transformation, as well as skip connection and layer normalization to stablize training.

Original: The whole transformer block can be formulated as below:
Revised: The whole transformer block can be formulated as below:

Original: [Equation]
Revised: [Equation]

Original: Neural Radiance Field.
Revised: Neural Radiance Field.

Original: NeRFs (Mildenhall et al., 2020) converts multi-view images into a radiance field and interpolates novel views by re-rendering the radiance field from a new angle.
Revised: NeRFs (Mildenhall et al., 2020) converts multi-view images into a radiance field and interpolates novel views by re-rendering the radiance field from a new angle.

Original: Technically,
Revised: Technically,

Original: NeRF models the underlying 3D scene as a continuous radiance field F : ( x , θ ) (cid:55)→ ( c , σ ) parameterized by a Multi-Layer Perceptron (MLP) Θ , which maps a spatial coordinate x ∈ R 3 together with the viewing direction θ ∈ [ − π, π ] 2 to a color c ∈ R 3 plus density σ ∈ R + tuple.
Revised: NeRF models the underlying 3D scene as a continuous radiance field F : ( x , θ ) (cid:55)→ ( c , σ ) parameterized by a Multi-Layer Perceptron (MLP) Θ , which maps a spatial coordinate x ∈ R 3 together with the viewing direction θ ∈ [ − π, π ] 2 to a color c ∈ R 3 plus density σ ∈ R + tuple.

Original: To form an image,
Revised: To form an image,

Original: NeRF performs the ray-based rendering, where it casts a ray r =
Revised: NeRF performs the ray-based rendering, where it casts a ray r =

Original: ( o , d ) from the optical center o ∈ R 3 through each pixel (towards direction d ∈ R 3 ), and then leverages volume rendering (Max, 1995) to compose the color and density along the ray between the near-far planes:
Revised: ( o , d ) from the optical center o ∈ R 3 through each pixel (towards direction d ∈ R 3 ), and then leverages volume rendering (Max, 1995) to compose the color and density along the ray between the near-far planes:

Original: [Equation]
Revised: [Equation]

Original: where r ( t ) =
Revised: where r ( t ) =

Original: o + t d , t n and t f are the near and far planes respectively.
Revised: o + t d , t n and t f are the near and far planes respectively.

Original: In practice, the Eq.
Revised: In practice, the Eq.

Original: 7 is numerically estimated using quadrature rules (Mildenhall et al., 2020).
Revised: 7 is numerically estimated using quadrature rules (Mildenhall et al., 2020).

Original: Given images captured from surrounding views with known camera parameters, NeRF fits the radiance field by maximizing the likelihood of simulated results.
Revised: Given images captured from surrounding views with known camera parameters, NeRF fits the radiance field by maximizing the likelihood of simulated results.

Original: Suppose we collect all pairs of rays and pixel colors as the training set
Revised: Suppose we collect all pairs of rays and pixel colors as the training set

Original: D = { ( r i , (cid:98) C i ) } Ni =1 , where N is the total number of rays sampled, and (cid:98) C i denotes the ground-truth color of the i -th ray, then we train the implicit representation Θ via the following loss function:
Revised: D = { ( r i , (cid:98) C i ) } Ni =1 , where N is the total number of rays sampled, and (cid:98) C i denotes the ground-truth color of the i -th ray, then we train the implicit representation Θ via the following loss function:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B I MPLEMENTATION D ETAILS
Revised: B I MPLEMENTATION D ETAILS

Original: Memory-Efficient Cross-View Attention.
Revised: Memory-Efficient Cross-View Attention.

Original: Computing attention between every pair of inputs has
Revised: Computing attention between every pair of inputs has

Original: O ( N 2 ) memory complexity, which is computational prohibitive when sampling thousands of points at the same time.
Revised: O ( N 2 ) memory complexity, which is computational prohibitive when sampling thousands of points at the same time.

Original: Nevertheless, we note that view transformer only needs to read out one token as the fused results of all the views.
Revised: Nevertheless, we note that view transformer only needs to read out one token as the fused results of all the views.

Original: Therefore, we propose to only place one read-out token X 0 ∈ R d in the query sequence, and let it iteratively summarize features from other data points.
Revised: Therefore, we propose to only place one read-out token X 0 ∈ R d in the query sequence, and let it iteratively summarize features from other data points.

Original: This reduces the complexity for each layer up to O ( N ) .
Revised: This reduces the complexity for each layer up to O ( N ) .

Original: We initialize the read-out token as the element-wise maxpooling of all the inputs: X 0 = max( F 1 (Π 1 ( x ) , θ ) , · · · , F N (Π N ( x ) , θ )) .
Revised: We initialize the read-out token as the element-wise maxpooling of all the inputs: X 0 = max( F 1 (Π 1 ( x ) , θ ) , · · · , F N (Π N ( x ) , θ )) .

Original: Rather than adopting a standard dot-product attention, we choose subtraction operation as the relation function.
Revised: Rather than adopting a standard dot-product attention, we choose subtraction operation as the relation function.

Original: Subtraction attention has been shown more effective for positional and geometric relationship reasoning (Zhao et al., 2021; Fan et al., 2022).
Revised: Subtraction attention has been shown more effective for positional and geometric relationship reasoning (Zhao et al., 2021; Fan et al., 2022).

Original: Compared with dot-product that collapses the feature dimension into a scalar, subtraction attention computes different attention scores for every channel of the value matrix, which increases diversity in feature interactions.
Revised: Compared with dot-product that collapses the feature dimension into a scalar, subtraction attention computes different attention scores for every channel of the value matrix, which increases diversity in feature interactions.

Original: Moreover, we augment the attention map and value matrix with { ∆ d } Ni =1 to provide relative spatial context.
Revised: Moreover, we augment the attention map and value matrix with { ∆ d } Ni =1 to provide relative spatial context.

Original: Technically, we utilize a linear layer W P to lift ∆ d i to the hidden dimension.
Revised: Technically, we utilize a linear layer W P to lift ∆ d i to the hidden dimension.

Original: We illustrate view transformer in Fig.
Revised: We illustrate view transformer in Fig.

Original: 2a.
Revised: 2a.

Original: To be more specific, the modified attention adopted in our view transformer can be formulated as:
Revised: To be more specific, the modified attention adopted in our view transformer can be formulated as:

Original: View-Attn( X ) =
Revised: View-Attn( X ) =

Original: diag(softmax( A + ∆ ⊤ ) f V
Revised: diag(softmax( A + ∆ ⊤ ) f V

Original: ( X + ∆ )) , where A j = f Q ( X 0 )
Revised: ( X + ∆ )) , where A j = f Q ( X 0 )

Original: − f K ( X j ) , (9) where A j ∈ R d denotes the j -th column of A , ∆ =
Revised: − f K ( X j ) , (9) where A j ∈ R d denotes the j -th column of A , ∆ =

Original: [ ∆ d· · · ∆ d N ] ⊤ W P ∈ R N × d , f Q , f K , and f V are parameterized by an MLP.
Revised: [ ∆ d· · · ∆ d N ] ⊤ W P ∈ R N × d , f Q , f K , and f V are parameterized by an MLP.

Original: We note that by applying diag( · ) , we read out the updated query token X 0 . 
Revised: We note that by applying diag( · ) , we read out the updated query token X 0 . See Alg.

Revised: 1 for the implementation in practice.

Original: Network Architecture.
Revised: Network Architecture.

Original: To extract features from the source views, we use a U-Net like architecture with a ResNet34 encoder, followed by two up-sampling layers as decoder.
Revised: To extract features from the source views, we use a U-Net-like architecture with a ResNet34 encoder, followed by two up-sampling layers as decoder.

Original: Each view transformer block contains a single-headed cross-attention layer while the ray transformer block contains a multi-headed self-attention layer with four heads.
Revised: Each view transformer block contains a single-headed cross-attention layer while the ray transformer block contains a multi-headed self-attention layer with four heads.

Original: The outputs from these attention layers are passed onto corresponding feedforward blocks with a Rectified Linear Unit (RELU) activation and a hidden dimension of 256 .
Revised: The outputs from these attention layers are passed onto corresponding feedforward blocks with a Rectified Linear Unit (RELU) activation and a hidden dimension of 256 .

Original: A residual connection is applied between the pre-normalized inputs (LayerNorm) and outputs at each layer.
Revised: A residual connection is applied between the pre-normalized inputs (LayerNorm) and outputs at each layer.

Original: For all our single scene experiments, we alternatively stack 4 view and ray transformer blocks while our larger generalization experiments use 8 blocks each.
Revised: For all our single-scene experiments, we alternatively stack 4 view and ray transformer blocks while our larger generalization experiments use 8 blocks each.

Original: All transformer blocks (view and ray) are of dimension 64 .
Revised: All transformer blocks (view and ray) are of dimension 64 .

Original: Following Vaswani et al.
Revised: Following Vaswani et al.

Original: ; Mildenhall et al.
Revised: ; Mildenhall et al.

Original: Zhong et al.
Revised: Zhong et al.

Original: (2021), we convert the low-dimensional coordinates to a high-dimensional representation using Fourier components, where the number of frequencies is selected as 10 for all our experiments.
Revised: (2021), we convert the low-dimensional coordinates to a high-dimensional representation using Fourier components, where the number of frequencies is selected as 10 for all our experiments.

Original: The derived view and position embeddings are each of dimension 63 .
Revised: The derived view and position embeddings are each of dimension 63 .

Original: X o → coordinate

Original: ( N rays, N pts , D )
Revised: rays pts , D )

Original: X j → epipolar view features ( N rays , N pts , N views , D )
Revised: X j → epipolar view features ( N rays , N pts , N views , D )

Original: ∆ d
Revised: ∆ d

Original: → relative directions of source views wrt target views ( N rays , N pts , N views , 3) f Q , f K , f V , f P , f A , f O → functions that parameterize MLP layers
Revised: → relative directions of source views wrt target views ( N rays , N pts , N views , 3) f Q , f K , f V , f P , f A , f O → functions that parameterize MLP layers

Original: Q = f Q ( X o )
Revised: Q = f Q ( X 0 )

Original: K = f K ( X j )
Revised: K = f K ( X j )

Original: V =
Revised: V =

Original: f V ( X j )
Revised: f V ( X j )

Original: P = f P (∆ d )
Revised: P = f P ( ∆ d )

Original: A = K − Q [: , : , None , :] + P
Revised: A = K − Q [: , : , None , :] + P

Original: A = softmax( A, dim = − 2)
Revised: A = softmax( A , dim = − 2)

Original: O =
Revised: O =

Original: (( V + P ) · A ) .
Revised: (( V + P ) · A ) .

Original: sum(dim
Revised: sum(dim

Original: = 2)
Revised: = 2)

Original: O =
Revised: O =

Original: f O ( O )
Revised: f O ( O )

Original: Pseudocode.
Revised: Pseudocode.

Original: We provide a simple and efficient pytorch pseudo-code to implement the attention operations in the view, ray transformer blocks in Alg. 1, 2 respectively.
Revised: We provide a simple and efficient pytorch pseudo-code to implement the attention operations in the view, ray transformer blocks in Alg. 1, 2 respectively.

Original: We do not indicate the feedforward and layer normalization operations for simplicity.
Revised: We do not indicate the feedforward and layer normalization operations for simplicity.

Original: As seen in Alg.
Revised: As seen in Alg.

Original: 3, we reuse the epipolar view features X j to derive keys, and values across view transformer blocks.
Revised: 3, we reuse the epipolar view features X j to derive keys, and values across view transformer blocks.

Original: Therefore, one could further improve efficiency by computing them only once while also sharing the network weights across view transformer blocks or simply put f view i ( . ) represents the same function across different values of i .
Revised: Therefore, one could further improve efficiency by computing them only once while also sharing the network weights across view transformer blocks or simply put f view i ( . ) represents the same function across different values of i .

Original: This can be considered analogous to an unfolded recurrent neural network that updates itself iteratively but using the same weights.
Revised: This can be considered analogous to an unfolded recurrent neural network that updates itself iteratively but using the same weights.

Original: C T ENTATIVE E XTENSIONS C. A UTO -R EGRESSIVE D ECODING
Revised: C T ENTATIVE E XTENSIONS C. A UTO -R EGRESSIVE D ECODING

Original: The final rendered color is obtained by mean-pooling the outputs from the ray transformer block, and mapping the pooled feature vector to RGB via an MLP layer.
Revised: The final rendered color is obtained by mean-pooling the outputs from the ray transformer block, and mapping the pooled feature vector to RGB via an MLP layer.

Original: It can be understood that the target pixel’s color is strongly dependent on the closest point from the ray origin and weakly related to the
Revised: It can be understood that the target pixel’s color is strongly dependent on the closest point from the ray origin and weakly related to the

Original: Algorithm 2 Ray Attention: PyTorch like Pseudocode
Revised: Algorithm 2 Ray Attention: PyTorch-like Pseudocode

Original: X o{+
+} → coordinate aligned features ( N rays , N pts , D ) x
Revised: X {+
+} → coordinate aligned features ( N rays , N pts , D ) x

Original: → point coordinates (after position encoding) ( N rays , N pts , D )
Revised: → point coordinates (after position encoding) ( N rays , N pts , D )

Original: d → target view direction (after position encoding) ( N rays , N pts , D ) f Q , f K , f V , f P , f A , f O → functions that parameterize MLP layers
Revised: d → target view direction (after position encoding) ( N rays , N pts , D ) f Q , f K , f V , f P , f A , f O → functions that parameterize MLP layers

Original: X o{+
+} = f P (concat( X o , d, x ))
Revised: X {+
+} = f P (concat( X o , d, x ))

Original: Q = f Q ( X o )
Revised: Q = f Q ( X 0 )

Original: K = f K ( X o )
Revised: K [Equation] X [Equation]

Original: V=
Revised: V

Original: [Equation]
Revised: [Equation]

Revised: A

Original: [Equation]
Revised: [Equation]

Original: o )
Revised: O

Original: [Equation]
Revised: [Equation]

Original: Algorithm
Revised: O [Equation]

Original:  X j → epipolar view features ( N rays , N pts , N views , D ) x
Revised: O ) X j → epipolar view features ( N rays , N pts , N views , D ) x

Original: → point coordinates (after position encoding) ( N rays , N pts , D )
Revised: → point coordinates (after position encoding) ( N rays , N pts , D )

Original: d → target view direction (after position encoding) ( N rays , N pts , D ) ∆ d → relative directions of source views wrt target views ( N rays , N pts , N views , 3) f  viewi , f  rayi → functions that parameterize view transforms, ray transformers at layer i respectively f rgb → functions that parameterize MLP layers i =
Revised: d → target view direction (after position encoding) ( N rays , N pts , D ) ∆ d → relative directions of source views wrt target views ( N rays , N pts , N views , 3) f ( l ) view , f ( l ) ray → functions that parameterize view transforms, ray transformers at layer l respectively f rgb → functions that parameterize MLP layers l =

Original: X o = max N views i =1 ( X j ) while i < N layers do
Revised: X 0 = max N views i =1 ( X j ) while l < N layers do

Original: X o{+
+} = f  viewi ( X o , X j , ∆ d )X o = f ray
Revised: X {+
+} = f ( l ) view ( X 0 , X j , ∆ d )

Original: i{+X 
+}
Revised: {+X 
+}

Original:  ( X o , d, x ) i = i +end while
Revised: = f ( l ) ray ( X o , d , x ) l = l +end while

Original: O =
Revised: O =

Original: Norm( X o )
Revised: Norm( X 0 )

Revised: RGB

Original: [Equation]
Revised: [Equation]

Revised: mean N points i =

Revised: [Equation]

Original:  farthest.
Revised: ) farthest.

Original: Revisiting Eq.
Revised: Revisiting Eq.

Original: 7, volumetric rendering attempts to compose point-wise color depending on the other points in a far to near fashion.
Revised: 7, volumetric rendering attempts to compose point-wise color depending on the other points in a far to near fashion.

Original: Motivated by this, we propose an auto-regressive decoder to better simulate the rendering process.
Revised: Motivated by this, we propose an auto-regressive decoder to better simulate the rendering process.

Original: Transformers have shown great success in auto-regressive decoding, more specifically in NLP (Vaswani et al., 2017).
Revised: Transformers have shown great success in auto-regressive decoding, more specifically in NLP (Vaswani et al., 2017).

Original: We borrow a similar strategy and replace the simpler MLP based color prediction with a series of transformer blocks - with self, cross attention layers.
Revised: We borrow a similar strategy and replace the simpler MLP-based color prediction with a series of transformer blocks - with self, cross attention layers.

Original: In the first pass, the decoder is queried with positional encoding of the farthest point ( γ ( x N ) ) to generate an output feature representation of the same.
Revised: In the first pass, the decoder is queried with positional encoding of the farthest point ( γ ( x N ) ) to generate an output feature representation of the same.

Original: In the next step, the output token is concatenated with the second farthest point ( γ ( x N − 1 ) ) to query the decoder.
Revised: In the next step, the output token is concatenated with the second farthest point ( γ ( x N − 1 ) ) to query the decoder.

Original: This process repeats until all the points in the ray are queried in a far to near fashion.
Revised: This process repeats until all the points in the ray are queried in a far-to-near fashion.

Original: In the final pass, the encoded view direction ( γ ( d ) ) is concatenated with the per-point output features in the previous passes to query the decoder and the output token corresponding to the view direction is extracted.
Revised: In the final pass, the encoded view direction ( γ ( d ) ) is concatenated with the per-point output features in the previous passes to query the decoder and the output token corresponding to the view direction is extracted.

Original: The extracted token is mapped to RGB via an MLP layer.
Revised: The extracted token is mapped to RGB via an MLP layer.

Original: This entire process is summarized in Fig.
Revised: This entire process is summarized in Fig.

Original: The auto-regressive procedure closely resembles the volumetric rendering equation which iteratively blends and overrides the previous color when marching along a ray from far to near.
Revised: The auto-regressive procedure closely resembles the volumetric rendering equation which iteratively blends and overrides the previous color when marching along a ray from far to near.

Original: Transformer-based decoders used in language iteratively predict output tokens only during inference, i.e. they are trained in a non-autoregressive fashion due to the availability of ground truth output tokens in each step.
Revised: Transformer-based decoders used in language iteratively predict output tokens only during inference, i.e. they are trained in a non-autoregressive fashion due to the availability of ground truth output tokens in each step.

Original: Transferring the same to neural rendering is not possible, as we do not have access to ground truth color for each point sampled along the ray.
Revised: Transferring the same to neural rendering is not possible, as we do not have access to the groundtruth color for each point sampled along the ray.

Original: Hence, we require a loop to
Revised: Hence, we require a loop to

Original: S e l f A tt en t i on
Revised: S e l f A tt en t i on

Original: C r o ss
Revised: C r o ss

Original: A tt en t i on
Revised: A tt en t i on

Original: Ray Features
Revised: Ray Features

Original: S e l f A tt en t i on
Revised: S e l f A tt en t i on

Original: C r o ss
Revised: C r o ss

Original: A tt en t i on
Revised: A tt en t i on

Original: Ray Transformer γ(x N ) γ(x N-1 )
Revised: Ray Transformer γ(x N ) γ(x N-1 )

Original: γ(d) γ(x N ) O(x 1 )
Revised: γ(d) γ(x N ) O(x 1 )

Original: O N
Revised: O N

Original: O N O
Revised: O N O

Original: NO 1 O d
Revised: NO 1 O d

Original: O N
Revised: O N

Original: O N O N-1 O N O N- [Figure]
Revised: O N O N-1 O N O NTarget Ray Color

Original: Auto Regressive
Revised: Auto Regressive

Original: [Table]
Revised: [Table]

Original: auto-regressively decode features even during training.
Revised: auto-regressively decode features even during training.

Original: This reduces the computational efficiency of the proposed strategy especially as the number of points sampled along the ray increases.
Revised: This reduces the computational efficiency of the proposed strategy, especially as the number of points sampled along the ray increases.

Original: Therefore, we introduce a cacheing mechanism to store the per-layer outputs of the previous tokens and only compute the attention of a new token in the current pass.
Revised: Therefore, we introduce a caching mechanism to store the per-layer outputs of the previous tokens and only compute the attention of a new token in the current pass.

Original: This does not overcome the iterative loop during each forward pass but avoid redundant computations, which helps improve the decoding speed drastically when compared to the naive strategy.
Revised: This does not overcome the iterative loop during each forward pass but avoids redundant computations, which helps improve the decoding speed drastically when compared to the naive strategy.

Original: Due to computational constraints, we are only able to train GNT + AutoReg with much fewer rays sampled per iteration (500) when compared to other methods as discussed in Sec.
Revised: Due to computational constraints, we are only able to train GNT + AutoReg with much fewer rays sampled per iteration (500) when compared to other methods as discussed in Sec.

Original: Tab. 4 discusses single scene optimization results on the LLFF dataset and we can clearly see that the GNT + AutoReg improves the overall performance when compared to existing baselines, and improving the PSNR scores in complex scenes (Orchids) when compared to our own method without the decoder.
Revised: Tab. 4 discusses single scene optimization results on the LLFF dataset and we can clearly see that the GNT + AutoReg improves the overall performance when compared to existing baselines, and improving the PSNR scores in complex scenes (Orchids) when compared to our own method without the decoder.

Original: However, this is not consistent across all scenes and metrics.
Revised: However, this is not consistent across all scenes and metrics.

Original: This could be because of the fewer number of rays sampled and we expect our results would improve when scaled to similar settings.
Revised: This could be because of the fewer number of rays sampled and we expect our results would improve when scaled to similar settings.

Original: Nevertheless, this shows that the learnable decoder predicts per-point RGB features effectively without any supervision from the volumetric rendering equation.
Revised: Nevertheless, this shows that the learnable decoder predicts per-point RGB features effectively without any supervision from the volumetric rendering equation.

Original: C. A TTENTION G UIDED C OARSE - TO -F INE S AMPLING
Revised: C. A TTENTION G UIDED C OARSE - TO -F INE S AMPLING

Original: GNT’s ray transformer learns point-to-point correspondences which helps model visibility and occlusion or more formally point-wise density σ .
Revised: GNT’s ray transformer learns point-to-point correspondences which helps model visibility and occlusion or more formally point-wise density σ .

Original: Motivated by this hypothesis, we estimate the depth maps from the extracted attention maps and qualitatively analyze the same in Sec. 4.5.
Revised: Motivated by this hypothesis, we estimate the depth maps from the extracted attention maps and qualitatively analyze the same in Sec. 4.5.

Original: Therefore, we could conclude that the learned point-wise importance values can be considered equivalent to pointwise density or σ .
Revised: Therefore, we could conclude that the learned point-wise importance values can be considered equivalent to pointwise density or σ .

Original: To further test our claim, we attempt to use the learned point-wise correspondence values to sample “fine” points, which are then queried to GNT to render a higher-quality image.
Revised: To further test our claim, we attempt to use the learned point-wise correspondence values to sample “fine” points, which are then queried to GNT to render a higher-quality image.

Original: Due to the set-like property of attention, we directly query the fine points to the same network without using a separately trained “fine” network unlike other NeRF methods (Mildenhall et al., 2020; Barron et al., 2021;
Revised: Due to the set-like property of attention, we directly query the fine points to the same network without using a separately trained “fine” network unlike other NeRF methods (Mildenhall et al., 2020; Barron et al., 2021;

Original: Wang et al., 2021b; Liu et al., 2022).
Revised: Wang et al., 2021b; Liu et al., 2022).

Original: Please note that we follow the same training strategy from Sec.
Revised: Please note that we follow the same training strategy from Sec.

Original: 4.1 and only sample coarse-fine points during evaluation.
Revised: 4.1 and only sample coarse-fine points during evaluation.

Original: Tab.
Revised: Tab.

Original: 4 compares “GNT
Revised: 4 compares “GNT

Original: [Figure]
Revised: [Figure]

Original: [Figure] + Fine” against other methods, and we can clearly see that it outperforms other SOTA methods in complex scenes like Orchids, performing even better than our own method without fine sampling.
Revised: + Fine” against other methods, and we can clearly see that it outperforms other SOTA methods in complex scenes like Orchids, performing even better than our own method without fine sampling.

Original: However, the performance improvements are not significant across all scenes and we attribute this to the lack of training with the coarse-fine sampling strategy and expect our results to improve further.
Revised: However, the performance improvements are not significant across all scenes and we attribute this to the lack of training with the coarse-fine sampling strategy and expect our results to improve further.

Original: In Fig.
Revised: In Fig.

Original: 9, we visualize the estimated depth values obtained from the learned attention maps during both coarse, and fine stages.
Revised: 9, we visualize the estimated depth values obtained from the learned attention maps during both coarse, and fine stages.

Original: We can clearly see that the fine depth map is able to better estimate differences between nearby pixels which results in a higher resolution output.
Revised: We can clearly see that the fine depth map is able to better estimate differences between nearby pixels which results in a higher resolution output.

Original: D A DDITIONAL R ESULTS 
Revised: D A DDITIONAL R ESULTS AND A NALYSIS

Original:  Tables5 and 6  include a breakdown of the quantitative results presented in the main paper into per-scene metrics.
Revised: Breakdown of Table 1. Tables 6 and 7 include a breakdown of the quantitative results presented in the main paper into per-scene metrics.

Original: Our method quantitatively surpasses original NeRF and achieve on-par results with state-of-the-art methods.
Revised: Our method quantitatively surpasses original NeRF and achieve on-par results with state-of-the-art methods.

Original: Although we slightly underperform NLF (Suhail et al., 2021) on some scenes, we argue that the comparison is not fair because NLF requires much larger batch size and longer iterations.
Revised: Although we slightly underperform NLF (Suhail et al., 2021) on some scenes, we argue that the comparison is not fair because NLF requires much larger batch size and longer iterations.

Original: We also include videos to demonstrate our results in supplementary materials.
Revised: We also include videos to demonstrate our results in the project page.

Revised: Comparison with SRT (Sajjadi et al., 2022b).

Revised: SRT (Sajjadi et al., 2022b) is another pure transformer based generalizable view synthesis baseline.

Revised: In contrast to GNT, SRT barely utilizes attention blocks to interpolate views without any explicit geometry priors.

Revised: We directly evaluate our cross-scene trained GNT in Sec.

Revised: 4.3 on NMR dataset (Kato et al., 2018) without further tuning.

Original: E D EFERRED D ISCUSSION
Revised: In addition to SRT, we also include other generalizable novel view synthesis methods LFN (Sitzmann et al., 2021) and PixelNeRF (Yu et al., 2021) which are compared with SRT in Sajjadi et al. (2022b).

Revised: All the results are presented in Tab. 5.

Revised: Overall, we find GNT can largely outperform all baselines in all the metrics.

Revised: We note that the pre-training data of SRT include the samples from NMR dataset (Kato et al., 2018), which is way more massive than GNT’s pre-training datasets and has a narrower domain gap to the evaluation set.

Revised: After all, our superior performance indicates our GNT can generalize better than SRT.

Revised: We argue this might be because multi-view geometry is a strong inductive bias for novel view interpolation.

Revised: That being said, a pixel on the novel view should be roughly consistent with its epipolar correspondence.

Revised: Enforcing such constraints explicitly can significantly improve trainability and data efficiency.

Revised: Nevertheless, we admit relieving multi-view geometry and learning a data-driven light transport prior from scratch can potentially render more sophisticated optics, which we leave for future exploration.

Revised: Comparison with GPNR (Suhail et al., 2022).

Revised: The concurrent work GPNR (Suhail et al., 2022) also utilizes fully attention-based architecture for neural rendering.

Revised: Below, we summarize several key differences: Embeddings: GPNR leverages three forms of positional encoding (including light field embeddings) to encode the information of location, camera pose, view direction, etc.

Revised: In contrast,

Revised: GNT merely utilizes image features (with point coordinates).

Revised: In this sense, GNT enjoys a neat design space and can potentially suggest such handcrafted feature engineering may not be necessary as they can be learned through cross-scene training.

Revised: Aggregation schemes: GPNR has three-stage aggregation:

Revised: 1) visual feature transformer to exchange information between patches across views,

Revised: [Table]

Revised: [Figure]

Revised: 2) epipolar aggregator transformer combines features along the epipolar line for each reference view, reference view aggregator transformer fuses the epipolar information across multiple views.

Revised: Such aggregation scheme extends NLF (Suhail et al., 2021) paper, which indicates GPNR rendering pipeline is more likely to simulate a light field-based rendering.

Revised: Instead, GNT leverages the two-stage aggregations, which naturally correspond to the online scene representation and ray-based rendering in generalizable NeRF.

Revised: In this sense, the rendering pipeline of GNT looks more like radiance field-based rendering.

Revised: Performance:

Revised: We have a direct comparison with GPNR on cross-scene generalization experiments (Tab. 2).

Revised: Our results show that GNT can outperform GPNR on all the tested datasets with a simpler architecture and lighter feature engineering.

Revised: Especially on the Shiny dataset (Tab. 2b),

Revised: GNT significantly outperforms GPNR by ~3dB in PSNR.

Revised: [Figure]

Revised: E D EFERRED D ISCUSSION

Original: Discussion on Occlusion Awareness.
Revised: Discussion on Occlusion Awareness.

Original: Conceptually, view transformer attempts to find correspondence between the queried points and source views.
Revised: Conceptually, view transformer attempts to find correspondence between the queried points and source views.

Original: The learned attention amounts to a likelihoodscore that a pixel on the source view is an image of the same point in the 3D space, i.e., no points lies between the target point and the pixel. NeuRay (Liu et al., 2022) leverages the cost volume from MVSNet (Yao et al., 2018) to predict per-pixel visibility and shows that introducing occlusion
Revised: The learned attention amounts to a likelihood

Original: [Table]
Revised: [Table]

Original: NeRF 0.
Revised: NeRF 0.

Original: MipNeRF 0.
Revised: MipNeRF 0.

Original: NLF 0.
Revised: NLF 0.

Original: GNT 0.
Revised: GNT 0.

Original: [Table] [Table]
Revised: LPIPS ↓

Original: Models
Revised: Models

Original: Lego
Revised: Lego

Original: Chair
Revised: Chair

Original: Drums
Revised: Drums

Original: Ficus
Revised: Ficus

Original: Hotdog
Revised: Hotdog

Original: Materials
Revised: Materials

Original: Mic
Revised: Mic

Original: Ship
Revised: Ship

Original: LLFF 0.
Revised: LLFF 0.

Original: NeRF 0.
Revised: NeRF 0.

Original: MipNeRF 0.
Revised: MipNeRF 0.

Original: NLF 0.
Revised: NLF 0.

Original: GNT 0.
Revised: GNT 0.

Original: [Table] [Table]
Revised: Avg ↓

Original: [Table]
Revised: [Table]

Original: NeRF 0.
Revised: NeRF 0.

Original: NeX 0.
Revised: NeX 0.

Original: NLF 0.
Revised: NLF 0.

Original: GNT 0.
Revised: GNT 0.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: Models
Revised: Models

Original: Room
Revised: Room

Original: Fern
Revised: Fern

Original: Leaves
Revised: Leaves

Original: Fortress
Revised: Fortress

Original: Orchids
Revised: Orchids

Original: Flower
Revised: Flower

Original: T-Rex
Revised: T-Rex

Original: Horns
Revised: Horns

Original: LLFF 0.
Revised: LLFF 0.

Original: NeRF 0.
Revised: NeRF 0.

Original: NeX 0.
Revised: NeX 0.

Original: NLF 0.
Revised: NLF 0.

Original: GNT 0.
Revised: GNT 0.

Original: [Table]
Revised: [Table]

Original: Avg
Revised: d

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: score that a pixel on the source view is an image of the same point in the 3D space, i.e., no points lies between the target point and the pixel.

Original:  information is beneficial for multi-view aggregation in generalizable NeRF.
Revised: NeuRay (Liu et al., 2022) leverages the cost volume from MVSNet (Yao et al., 2018) to predict per-pixel visibility and shows that introducing occlusion information is beneficial for multi-view aggregation in generalizable NeRF.

Original: We argue that instead of explicitly regressing the visibility, purely relying on epipolar geometry constrained attention can automatically learn how to infer occlusion, given prior works in Multi-View Stereo (MVS)
Revised: We argue that instead of explicitly regressing the visibility, purely relying on epipolar geometry-constrained attention can automatically learn how to infer occlusion, given prior works in Multi-View Stereo (MVS)

Original: (Yang et al., 2022; Ding et al., 2021).
Revised: (Yang et al., 2022; Ding et al., 2021).

Original: In view transformer, the U-Net provides multi-scale features to the transformer, and the attention block acts as a matching algorithm, which selects the pixels from neighboring views that maximize view consistency.
Revised: In view transformer, the U-Net provides multi-scale features to the transformer, and the attention block acts as a matching algorithm, which selects the pixels from neighboring views that maximize view consistency.

Original: We defer empirical discussion to Sec. 4.5.
Revised: We defer empirical discussion to Sec. 4.5.

Original: Discussion on Depth Cuing.
Revised: Discussion on Depth Cuing.

Original: The ray transformer iteratively aggregates features according to the attention value.
Revised: The ray transformer iteratively aggregates features according to the attention value.

Original: This attention value can be regarded as the importance of each point to form the image, which reflects visibility and occlusion reasoned by point-to-point interaction.
Revised: This attention value can be regarded as the importance of each point to form the image, which reflects visibility and occlusion reasoned by point-to-point interaction.

Original: Therefore, we can interpret the average attention score for each point as the accumulated weights in volume rendering.
Revised: Therefore, we can interpret the average attention score for each point as the accumulated weights in volume rendering.

Original: In this sense, we can infer the depth map from the attention map by averaging the marching distance t i with the attention value.
Revised: In this sense, we can infer the depth map from the attention map by averaging the marching distance t i with the attention value.

Original: This implies our ray transformer learns geometry-aware 3D semantics on both feature space and attention map, which helps it generalize well across scenes.
Revised: This implies our ray transformer learns geometry-aware 3D semantics on both feature space and attention map, which helps it generalize well across scenes.

Original: We defer visualization and analysis to Sec. 4.5.
Revised: We defer visualization and analysis to Sec. 4.5.

Original: NLF (Suhail et al., 2021) proposes a similar two-stage rendering transformer, but it first extracts features on the epipolar lines and then aggregates epipolar features to get the pixel color.
Revised: NLF (Suhail et al., 2021) proposes a similar two-stage rendering transformer, but it first extracts features on the epipolar lines and then aggregates epipolar features to get the pixel color.

Original: We doubt this strategy may fail to generalize as epipolar features lack communication with each other and thus cannot induce geometry grounded semantics.
Revised: We doubt this strategy may fail to generalize as epipolar features lack communication with each other and thus cannot induce geometry-grounded semantics.

Original: F L IMITATIONS
Revised: F L IMITATIONS

Original: Although our method achieves strong single-scene performance and achieves SOTA cross-scene generalization, we discuss certain limitations of our method.
Revised: Although our method achieves strong single-scene performance and achieves SOTA cross-scene generalization, we discuss certain limitations of our method.

Original: The view transformer relies on epipolar constraints - i.e can only aggregate information from valid epipolar lines.
Revised: The view transformer relies on epipolar constraints so that it can only aggregate information from valid epipolar lines.

Original: Therefore, non-epipolar scenes and complex light transport might not be captured by the view transformer.
Revised: Therefore, non-epipolar scenes and complex light transport might not be captured by the view transformer.

Original: Although we adopt a feature extractor with large receptive fields to encode global light transport and our view transformer empirically works well on complex lighting effects, what is captured by the image encoder remains unclear.
Revised: Although we adopt a feature extractor with large receptive fields to encode global light transport and our view transformer empirically works well on complex lighting effects, what is captured by the image encoder remains unclear.

Original: Moreover, epipolar correspondence for the boundary pixels sometimes are missing, which causes some minor artifacts (see Fig. 11).
Revised: Moreover, epipolar correspondence for the boundary pixels sometimes are missing, which causes some minor artifacts (see Fig. 11).