L ASER : L ATENT S ET R EPRESENTATIONS FOR 3D G ENERATIVE M ODELING
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural Radiance Field ( N e RF ) provides unparalleled fidelity of novel view synthesis—rendering a 3D scene from an arbitrary viewpoint.
N e RF requires training on a large number of views that fully cover a scene, which limits its applicability.
While these issues can be addressed by learning a prior over scenes in various forms, previous approaches have been either applied to overly simple scenes or struggling to render unobserved parts.
We introduce Latent Set Representations for N e RF - VAE (L ASER -NV)—a generative model which achieves high modelling capacity, and which is based on a set-valued latent representation modelled by normalizing flows.
Similarly to previous amortized approaches, L ASER -NV learns structure from multiple scenes and is capable of fast, feed-forward inference from few views.
To encourage higher rendering fidelity and consistency with observed views, L ASER -NV further incorporates a geometry-informed attention mechanism over the observed views.
L ASER -NV further produces diverse and plausible completions of occluded parts of a scene while remaining consistent with observations.
L ASER -NV shows state-of-the-art novel-view synthesis quality when evaluated on
ShapeNet and on a novel simulated City dataset, which features high uncertainty in the unobserved regions of the scene.
I NTRODUCTION
Probabilistic scene modelling aims to learn stochastic models for the structure of 3D scenes, which are typically only partially observed (Eslami et al., 2018; Kosiorek et al., 2021; Burgess et al., 2019).
Such models need to reason about unobserved parts of a scene in way that is consistent with the observations and the data distribution.
Scenes are usually represented as latent variables, which are ideally compact and concise, yet expressive enough to describe complex data.
Such 3D scenes can be thought of as projections of light rays onto an image plane.
Neural Radiance
Field ( N e RF , (Mildenhall et al., 2020)) exploits this structure explicitly.
It represents a scene as a radiance field (a.k.a.
a scene function), which maps points in space (with the corresponding camera viewing direction) to color and mass density values.
We can use volumetric rendering to project these radiance fields onto any camera plane, thus obtaining an image.
Unlike directly predicting images with a CNN, this rendering process respects 3D geometry principles.
N e RF represents scenes as parameters of an MLP , and is trained to minimize the reconstruction error of observations from a single scene—resulting in unprecedented quality of novel view synthesis.
For generative modelling, perhaps the most valuable property of N e RF is the notion of 3D geometry embedded in the rendering process, which does not need to be learned, and which promises strong generalisation to camera poses outside the training distribution.
However, since N e RF ’s scene representations are high dimensional MLP parameters, they are not easily amenable to generative modelling (Dupont et al., 2022).
N e RF - VAE (Kosiorek et al., 2021) embeds N e RF in a generative model by conditioning the scene function on a latent vector that is inferred from a set of ‘context views’.
It then uses N e RF ’s rendering mechanism to generate outputs.
While N e RF - VAE admits efficient inference of a compact latent representation, its outputs lack visual fidelity.
This is not surprising, given its simple latent structure, and the inability to directly incorporate observed features.
In addition, N e RF - VAE does not produce varied samples of unobserved parts of a scene.
A number of recent deterministic methods (Yu et al., 2021; Trevithick & Yang, 2021; Wang et al., 2021) uses local image features to directly condition radiance fields in 3D. This greatly improves reconstruction quality of observed parts of the scene.
However, these methods are still unable to produce plausible multimodal predictions for the unobserved parts of a scene.
In this work, we address N e RF - VAE ’s shortcomings by proposing Latent Set Representations for N e RFVAE (L ASER -NV).
To increase modelling capacity, L ASER -NV uses an arbitrarily-sized set of latent variables (instead of just one vector) modelled with normalizing flows.
To further enable producing samples which are consistent with observed parts, we make the generative model conditional on a set of context views (as opposed to conditioning only the approximate posterior).
L ASER -NV offers superior visual quality with the ability to synthesise multiple varied novel views compatible with observations.
Figure 1 shows L ASER -NV’s key components and abilities.
We include a gif in supp.
material showing fly-throughs of additional prior samples, see Section 4.5 for details.
Our contributions are as follows:
• We introduce a novel set-valued latent representation modelled by purpose-built permutationinvariant normalizing flows conditioned on context views.
We show that increasing the number of latent set elements improves modelling performance, providing a simple way to trade off computation for quality without adding new model parameters.
We also verify that increasing latent dimensionality in N e RF - VAE offers no such benefits.
In contrast with deterministic scene models in the literature, our probabilistic treatment over the latent set allows covering multiple models when predicting novel views.
• We develop a novel attention mechanism to condition the scene function on the set-valued latent as well as additional local features computed from context views.
We show that including local features further improves visual quality.
• We evaluate L ASER -NV on three datasets: a category-agnostic ShapeNet dataset, MultiShapeNet, and on a novel “City” dataset that contains a large simulated urban area and poses significant challenges as a benchmark for novel view synthesis due to high uncertainty in the unobserved parts of the scene.
Our model overcomes some of the main limitations of N e RF - VAE and also outperforms deterministic N e RF models on novel view synthesis in the face of uncertainty.
B ACKGROUND : N E RF & N E RF-VAE
Neural Radiance Field ( N e RF , (Mildenhall et al., 2020)) represents a 3D scene as a scene function
F ( x , d ) of a point coordinate x ∈ R 3 and direction d ∈ R 3 , and outputs the colour c ∈ R 3 and volume density σ ≥ 0 .
This function is parameterized using a neural network (typically a fully connected MLP).
To obtain a pixel colour, the scene function is evaluated at many points along the corresponding ray in the volumetric rendering process.
This volumetric rendering integral is in practice approximated using numerical integration; see (Mildenhall et al., 2020; Blinn, 1982) for details.
We use ˆ I
= render ( F ( · ) , C ) to denote the image rendering process that outputs the image ˆ
I ∈ R H
× W × 3 for the rays of a camera C and a scene function F .
The camera C =
( K , R , t ) is specified by its intrinsic parameters K ∈ R 3 × 3 and extrinsic parameters given by a rotation R ∈ R 3 × 3 and translation t ∈ R 3 .
In its original formulation, the parameters of NeRF are optimized by minimizing an error function (typically the mean-squared error) between rendered images and the ground-truth images across many views of a single scene.
While N e RF learns high-fidelity scene representations when many views are available, it does not learn a prior over scenes, does not provide compact scene representations, and does not admit efficient inference (i.e. fast estimation of the scene parameters from a few input views).
To address these issues, N e RF - VAE (Kosiorek et al., 2021) embeds N e RF as a decoder in a variational auto-encoder ( VAE , (Kingma & Welling, 2014; Rezende et al., 2014)).
Each scene is represented by a latent vector z (instead of the set of parameters of an MLP as in NeRF) which can be either sampled from a
Gaussian prior p ( z ) or from the approximate posterior q ( z | C ) inferred from a set of context views and associated cameras C .
The latent z is used to condition an MLP that parameterizes the scene function, F θ ( · , z ) : ( x i , d i )
(cid:55)→ c i , σ i .
Here, parameters θ are shared between scenes, while z is scene specific.
N e RF - VAE uses a Gaussian likelihood p ( I | z , C ) with the image rendered from the sampled scene function as mean, and a fixed standard deviation.
L ATENT S ET R EPRESENTATIONS FOR N e RF - VAE
LASER-NV is a conditional generative model, conditioned on a set of input views V :=
{ I n , C n } Nn =1 .
Given these views the model infers a latent representation that can be used for novel view synthesis.
In order to represent large-scale and complex scenes we use La tent Se t R epresentations ( LASER s), a set of latents of the form Z := { z k } Kk =1 , where K is a hyperparameter.
In the N e RF - VAE , the scene function MLP is directly conditioned on the inferred latent.
In contrast, L ASER -NV’s scene function is conditioned on the latent set Z and features H :=
{ H n } Nn =1 extracted from N input views, and integrates information from both.
This results in the form F θ ( · , Z , H ) : ( x i , d i )
(cid:55)→ c
i , σ i , which we detail further in Section 3.2.
We obtain the input features H by separately encoding the input views using a convolutional neural net (CNN) that maintains spatial structure.
These feature maps are subsequently used in the conditional prior, the posterior, and the resulting scene function, all of which we describe next.
Further architectural details are provided in Appendix A.
3. C ONDITIONAL P RIOR AND P OSTERIOR
Given a set of context views V :=
{ I n , C n } Nn =1 consisting of images I n and corresponding cameras
C n , L ASER -NV defines a conditional prior p ( Z|V ) over a latent set of size K , parameterized by a permutation-invariant normalizing flow— a distribution model that allows for both fast and exact sampling and efficient density evaluation (Rezende & Mohamed, 2015;
Papamakarios et al., 2021).
Flows are defined by an invertible mapping f of random variables, starting from a base distribution p 0 (cid:0) Z (0) (cid:1) .
When f is composed of multiple invertible mappings f 1 , . .
. , f m , the resulting density is given by
[Equation]
Our flow design closely follows that of Wirnsberger et al.
(2020; 2021), who parameterize the transformations in each layer using transformers, preserving permutation invariance.
Specifically, we propose a novel design to condition the flow distribution on the features H extracted from the context views.
To achieve that, we use split coupling layers (Dinh et al., 2017), where for each layer, we first apply self-attention to the set of latents, followed by cross-attention to attend the context features.
The output of the second transformer defines the parameters for an invertible affine transformation of the random variables.
Implementation details of our normalizing flow is in Appendix A.1.
Since L ASER -NV is trained as a VAE, we define an approximate posterior q ( Z|U ∪ V ) , where
U =
{ I t , C t } Mt =1 are target views that the model is trained to reconstruct, denoted as posterior context.
The posterior and conditional prior are both distributions over the latent space, where the posterior is conditioned on the union of prior and posterior context.
We model the posterior distribution using the same architecture as the conditional prior, but with a separate set of learned parameters.
3. D ECODER & S CENE F UNCTION
When evaluating the scene function, i. e. computing colour and density for a point x i , d i , L ASER -NV has access to both the latent set Z as well as the features H extracted from the context views V .
We now describe how to integrate those two, which consists of querying the latent set for each point, and using 3D projection to extract relevant features.
Querying the Latent Set While the latent set representation Z contains information regarding the entire scene, when querying for a specific point x i , we would like to extract only relevant information from Z .
We propose using a transformer cross-attention model to compute a single feature vector from Z .
In the first layer, queries are computed from the positional encoding of x i , and keys and values are computed from the elements of Z .
In subsequent layers, queries are computed from previous layers’ output.
We denote the output of the latent querying as latent features ˜
z
i , each corresponding to a query point x i .
Local Features When synthesizing a novel view given a few input views, a number of existing
N e RF -based methods uses 3D projection to determine which features from the context views are useful for explaining a particular 3D point (Yu et al., 2021; Trevithick & Yang, 2021; Wang et al., 2021).
L ASER -NV incorporates this idea, and in particular follows the design of pixelNeRF (Yu et al., 2021) as it is shown to achieve strong results in a wide range of domains.
We use the encoded input views { H n } Nn =1 described above, along with their corresponding camera matrices { C n } Nn =1 .
Given a query 3D point x i and direction d i , we obtain the point p ni ∈ R 2 in the image space of the n th context view using the known intrinsic parameters.
The corresponding local features are then bilinearly interpolated h ni
= H n [ p ni ] .
We refer to h i as local features since they are spatially localized based on the projected coordinates, as opposed to being a global aggregation of all context features.
We finally obtain ˆ h ni for each context view by separately processing the features h ni using a residual MLP that has as an additional conditioning on the image-space projected points and directions.
Integrating Latents and Local Features In contrast to pixelNeRF, which does not have latents,
L ASER -NV integrates the both local and the latent features.
It does so with a transformer:
[Equation]
The attention queries are computed from latent features ˜
z
i (see above), and attend to the combined set of local features and to ˜ z i .
Note that, for a point x i , we use only those context views from which this point is visible.
Using a transformer to process encoded features from multiple views with epipolar constraints has been proposed in prior work on Multi-view Stereo, see e.g. (He et al., 2020; Xi et al., 2022; Wang et al., 2022; Ding et al., 2022).
The final step in evaluating the scene function is to use f i to condition a residual MLP similar to
N e RF - VAE resulting in L ASER -NV’s scene function F θ ( · , Z , H ) : ( x i , d i )
(cid:55)→ c
i , σ i which is used to volume-render image outputs.
[Equation]
Similarly to N e RF - VAE , our objective function is given by the evidence lower-bound:
[Equation]
where I t and C t are the images and cameras from the posterior context views U .
We use a Gaussian likelihood term for p ( I t | C t , Z , V ) with a fixed variance.
E XPERIMENTS 4. D ATASETS
We first evaluate L ASER -NV on a novel dataset based on a set of synthetic, procedurally generated, environments consisting of urban outdoor scenes.
This dataset contains posed images of a virtual city with objects of multiple scales, such as house blocks, roads, buildings, street lights, etc (but no transient objects such as cars or people).
As such, it is an extremely challenging scene representation benchmark, especially when representations are inferred from few views only—resulting in partial observability and high uncertainty.
In this situation, when evaluated on unobserved parts of a scene, we want a model to produce predictions that are varied, self-consistent, and aligned with the data distribution.
Each scene consists of a neighborhood block, and a randomly chosen designated point of interest (road intersection or park) is selected within.
Images are then rendered by placing the camera at random nearby positions and viewpoints from the point of interest.
The dataset has 100K training scenes, and 3200 test scenes.
We provide further details and dataset samples in Appendix B.
To test L ASER -NV’s ability to produce near-deterministic scene completions in a low-uncertainty setting, we compare it against a number of previous methods on the ShapeNet NMR dataset (Kato et al., 2018), which consists of several views of single objects from 13 different ShapeNet categories.
Finally, we evaluate on the more challenging MultiShapeNet-Hard (MSN-Hard) (Sajjadi et al., 2021), which tests the models capabilities in the presence of a large number of cluttered objects of varying size with detailed textures and realistic backgrounds.
4. M ODELS & E VALUATION
On the City dataset, we compare L ASER -NV with Multi-view Conditional NeRF ( MVC - N e RF ) and
N e RF - VAE ; see Figure 7 in Appendix A for model diagrams.
M VC - N e RF (Figure 7c) is a variant of
L ASER -NV (Figure 7a) which does not use L ASER -NV’s latent set representations but instead only relies on deterministically rendering new views using local features.
It therefore closely resembles the design of pixelNeRF but uses the same architecture for the image encoder and scene function F θ as L ASER -NV to ensure they are comparable.
On ShapeNet, we further show quantitative results of a number of non-NeRF novel view synthesis methods: Scene Representation Networks (SRN,
Sitzmann et al.
Differentiable Volumetric Rendering (DVR, Niemeyer et al.
(2020)), and
Scene Representation Transformers (SRT, Sajjadi et al.
During training, L ASER -NV uses a random subset of 4 of the 23 target views as posterior context views 1 .
MVC - N e RF and N e RF - VAE are trained to directly predict the 23 views given a single context view.
At test time, L ASER -NV samples the latents from its conditional prior whereas N e RF - VAE uses its posterior (given that its prior is unconditional).
Importantly, all models in this evaluation use one input view and predicttarget views.
At evaluation time, we feed each model with a set of context views, and evaluate reconstructions of those same views, and predictions of novel views of the same scene.
We evaluate reconstruction ability via likelihood estimates, and standard image similarity metrics PSNR, and SSIM (Wang et al., 2004).
For novel view predictions we report the Fréchet Inception Distance (FID, Heusel et al.
between the marginal distribution of samples and the evaluation dataset 2 .
This metric
Image similarity metrics such as PSNR and SSIM are not useful when there exists a large multimodal space of possible predictions as is the case in the City.
highlights the incapability of deterministic models to generate varied and plausible samples in face of uncertainty.
Note that since N e RF - VAE is unconditional, we use the learned approximate posterior as the conditional distribution.
In the City, we thus train a separate model (one to use for reconstruction metrics that has 8 posterior context views and another one with 2 views for predictions).
For ShapeNet results all models use one context view.
On MSN-Hard, we train L ASER -NV with 5 prior context views and 3 target views.
We report the test
PSNR of 1 novel view given 5 input views, and compare with the results reported in Sajjadi et al.
The methods of comparison include two amortized non-NeRF models based on light fields,
Scene Representation Transformers (SRT) and Object SRT (OSRT Sajjadi et al.
; and two amortized NeRF methods, pixelNeRF and Volumetric OSRT (VOSRT).
4. A BLATIONS
We further ablate L ASER -NV to isolate contributions of individual model components.
To quantify the effects of conditioning the prior we compare L ASER -NV to a conditional version of N e RF - VAE .
Note that the original N e RF - VAE used an unconditional prior (Kosiorek et al., 2021).
We also compare
L ASER -NV to a version that does not condition the scene function with local context features, cf.
Equation (2) and Figure 7b in Appendix A, denoted by L ASER -NV ( NO GEOM .).
L ASER -NV,
L ASER -NV ( NO GEOM .), N e RF - VAE variants, and MVC - N e RF share architectures for the image encoder and the scene function F θ .
4. T RAINING AND IMPLEMENTATION DETAILS
We use the hierarchical sampling technique of (Mildenhall et al., 2020) when rendering a pixel colour, which involves learning two separate sets of scene function parameters as well as optimizing a corresponding colour likelihood term for each step.
For experiments with the City, we leverage ground-truth depth maps using the method proposed by Stelzner et al.
(2021) in order to train all the models with fewer scene function evaluations per ray.
Not using ground-truth depth significantly increases training time while only marginally lowering reconstruction PSNR, see Appendix D for a discussion and Fig.
16 in Appendix B for visualisations.
Importantly, when evaluating our models we revert to volumetric rendering.
All generative models are trained by optimizing the evidence lower bound; where we anneal the KL term in Eq.
All experiments use the Adam (Kingma & Ba, 2014) optimizer.
We provide full details of model architectures in Appendix A and training procedures in
Appendix E.
4. R ESULTS
Results on the City dataset are shown in Table 1.
In terms of reconstructing observed input views, both L ASER -NV and MVC - N e RF , due to the use of explicit geometrical knowledge, result in excellent reconstructions.
N e RF - VAE , however, needs to reconstruct input views purely from its latent representation, resulting in low fidelity reconstructions.
For prediction of unseen views, the output distribution of L ASER -NV is closer to the actual evaluation data (lower FID), in contrast to MVC - N e RF which outputs a single prediction, and N e RF - VAE which produces low quality outputs that do not vary much.
We show representative example reconstructions and predictions in Fig.
We also include a gif 3 in supp.
material showing fly-throughs of model samples.
Results on ShapeNet are shown in Table 2.
Both MVC - N e RF and L ASER -NV outperform existing published methods when predicting novel views of these single-object scenes.
We further observe low quality predictions of N e RF - VAE in example predictions in Fig.
Finally, Fig.
12 shows a hand-selected example where the input view has ambiguity and how L ASER -NV can sample plausible variations.
As in the City experiment, L ASER -NV produces plausibly varied predictions of the object in contrast to the other methods.
Results on MSN-Hard are shown in Table 3.
L ASER -NV clearly outpeforms the NeRF-based models, and improves over OSRT and the original SRT.
Only a modified SRT shows higher PSNR (25.93) over L ASER -NV (24.45).
Note that SRT does not learn to estimate densities nor depth maps and
[Table]
is deterministic.
Fig.
3 shows a prediction example using L ASER -NV and further examples are in
Appendix C which demonstrate high fidelity colours and densities.
4. T RAIN - AND T EST - TIME SCALING & D ATA E FFICIENCY
Latent set representations allow trading computation for increased model capacity.
We train L ASERNV on the City dataset with increasing latent set sizes K , and compare it to a conditional N e RF - VAE also trained with increasing latent dimensions 4 .
Figure 4 shows reconstruction log-likelihoods as a
[Table] [Table]
function of latent capacity.
While L ASER -NV consistently improves its reconstructions with larger latent sets, cond.
N e RF - VAE quickly saturates at 256 latent dimensions, at worse reconstructions compared to L ASER -NV.
Further, in Appendix D, we provide an analysis of test-time scaling—we show that increasing the latent set size at test time does not result in clear change of performance.
Finally, in Figure 14 we show that L ASER -NV performs better at test time than N e RF - VAE and
MVC - N e RF for different training dataset sizes.
R ELATED WORK
Neural scene representations Inferring neural scene representations using a deep generative model of scenes is an active research area, and we mention some of the most closely related approaches below.
N e RF - VAE learns a generative model of 3D scenes which is only shown to work in relatively simple scenes.
When applied to more complicated data (see Fig. 2), it produces blurry reconstructions and novel views.
We posit that this is caused by insufficient capacity of its latent representation z .
Generative Query Network (Eslami et al., 2018) similarly learns a distribution over scenes, but its decoder is a CNN.
Subsequent work (Rosenbaum et al., 2018) improves over its representational capacity by using an attentive mechanism when conditioning the decoder from context in an analogous way to how L ASER -NV improves over N e RF - VAE .
By implementing a NeRF in our decoder we gain the benefits of more robust generalization capabilities to novel viewpoints compared to the convolutional counterparts (Kosiorek et al., 2021).
Simone (Kabra et al., 2021) is a convolutional generative model that performs high quality novel view synthesis and while also performing unsupervised segmentation of input videos.
The authors in (Dupont et al., 2022) propose directly modelling the underlying radiance fields as functions, which they refer to as ‘functa’.
Both our method and theirs learn a prior over scene functions with a model that has a large representational capacity.
Bautista et al.
(2022) propose a two-stage method to learn a generative model of 3D scenes.
As L ASER -NV, their model can conditionally sample radiance fields.
However, they focus on scenes with dense views, whereas L ASER -NV works with sparse views.
[Figure]
Deterministic novel view synthesis NeRF relies on optimizing the parameters of an MLP scene function, separately for every scene.
A number of methods in the literature propose amortizing the mapping of input views to the scene function using multi-view geometry, see e.g. IBRNet (Wang et al., 2021), General Radiance Fields (GRF) (Trevithick & Yang, 2021) and pixelNeRF (Yu et al., 2021).
These methods extract features of the context views by using projective geometry to select which features to attend to, and are shown to generalize well and carry out high-fidelity predictions.
PixelNeRF is designed to work well in a solely view-centered coordinate system.
MVSNerf (Chen et al., 2021) lifts context features to a 3D representation with strong novel-view synthesis performance.
Scene Representation Transformers (SRT) (Sajjadi et al., 2021) use global features obtained from context views using a generalization of the Vision Transformer (Dosovitskiy et al., 2020), which bears similarities to L ASER -NV.
SRT bypasses the use of 3D geometry entirely, yet shows remarkable performance on real world data.
NeRF for large-scale scenes A number of NeRF methods have been developed to scale up to city-wide scenes.
They work by optimizing multiple separate scene functions for separate blocks of a scene, which are then dynamically used when rendering (Tancik et al., 2022; Turki et al., 2021).
Alternatively they design a scene function that represents the scene at multiple levels of scale (Martel et al., 2021; Xiangli et al., 2021).
Similarly, Mip-NeRF (Barron et al., 2021) allows prefiltering the scene function inputs to better capture coarse and fine details.
These methods are orthogonal to ours since they do not learn a conditional NeRF model sparse views nor address the uncertainty in predictions.
C ONCLUSION
We propose L ASER -NV, a conditional generative model of neural radiance fields capable of efficient inference of large and complex scenes under partial observability conditions.
While recent advances in neural scene representation and neural rendering lead to unprecedented novel view synthesis capabilities, producing sharp and multi-modal completions of unobserved parts of the scene remains a challenging problem.
We experimentally show that L ASER -NV can model scenes of different scale and uncertainty structure, and isolate the usefulness of each contribution through ablation studies.
While NeRF serves as a strong inductive bias for learning 3D structure from images, L ASER -NV also inherits some of its drawbacks.
Volumetric rendering with neural radiance fields is computationally costly and can limit the model from real-time rendering.
It remains to be seen whether fast NeRF implementations (Müller et al., 2022) can be of help.
Furthermore, accurate GT camera information is required for learning and novel view synthesis; (Moreau et al., 2022) recently made progress on localization methods using NeRF.
While learning a generative scene model of real scenes remains an open problem, we believe this work is an important step in that direction.
Finally, an interesting direction for future research is incorporating object-centric structure and dynamics into L ASER -NV, which might be useful for for downstream reasoning tasks (Sajjadi et al., 2022; Ding et al., 2021).
A M ODEL ARCHITECTURES A. N ORMALIZING FLOWS FOR L ASER
We wish to model our latent set representation Z = { z 1 , . .
, z K } using a flexible distribution that is efficient both to evaluate as well as sample from.
Furthermore, the distribution should ideally be invariant to re-ordering of the elements of the set.
Normalizing flows (Rezende & Mohamed, 2015) allow specifying a distribution with precisely these properties.
Formally, it is given by:
[Equation]
where p 0 ( · ) is a base distribution, f m ( · ) are invertible transformations applied to an intermediate random variable, and Z is the final LASER after M flow layers.
V is an additional set of context vectors that condition the flow transformation.
The resulting probability density of Z is
[Equation]
The need to compute the Jacobian of intermediate flow transformations puts constraints on the design of those transformations, as they need to be computationally efficient.
We use a split-coupling flow similar to RealNVP (Dinh et al., 2017), and we follow a similar architecture as the ones used in
Wirnsberger et al.
in adapting it to sets.
The method Flow Scans (Bender et al., 2020) first introduced the general idea of adapting split-coupling to work on sets, and Wirnsberger et al.
(2020; 2021) use a transformer for increased flexibility.
A split-coupling transformation partitions the set Z into two sets Z 1 , Z 2 by splitting each element of the original set across channels (we split in half).
At a high level, the transformation of one part is computed as a function of the other part.
Finally, the original set is recovered by elementwise concatenation of both transformed parts.
More formally, let g be an invertible function, a split-coupling transform
f sc m that maps Z ( m − 1) → Z ( m ) is given by three operations:
[Equation]
where z k, 1 indicates dimensions of split 1 of element k , and V is conditioning context.
Note how the parameters of the transformation g of one partition are derived from the other partition.
To do so in a permutation-equivariant manner, the functions t m are a composition of two transformers: a self-attention module is applied followed by a cross-attention conditioned on context features, defined in Appendix A.2.
More specifically, when transforming split 1 as a function of split 2, we have:
[Equation]
We finally apply the function g , in our case the affine function (Dinh et al., 2017), from the parameters obtained for every element in ψ m − 1:
[Equation]
for k
[Equation] [Equation]
where W mscale and W mbias are linear weights.
When composing our complete flow with M layers of split-coupling flows, we find that including an additional linear invertible transformation after each f m to be helpful.
To do so, we use the Linear Permutation Equivariant ( f lpe m ), proposed in the Flow
Scans work, the purpose of which is to allow capturing interdependencies in the dimensions of the set elements.
The only difference is that we parameterize each sub-block of the linear transformation using a full unconstrained matrix (instead of a diagonal matrix).
In summary, our flow chains together the sequence of mappings f sc 1 , f lpe 1 , . .
, f sc M , f lpe M .
Thus, sampling from this flow involves sampling from the base distribution followed by applying the chain of functions.
Our normalizing flow differs in a number of ways compared with Wirnsberger et al.
First, our flow layers are additionally parameterized by a cross-attention module in order to condition the distribution on the context set; second, we don’t use rational-quadratic splines, instead we found the simpler affine coupling layer to work well.
Notice that the size of the resulting set depends only on the hyper-parameter K , that is the number of elements sampled from the base distribution.
It does not depend on the context size, i.e. features extracted from input images.
In our experiments, we use a diagonal Gaussian base distribution, and set the number of flow layers to be M = 8 for the prior flow, and M = 4 for the posterior flows.
Our self-attention and cross-attention are composed of L = 1 layers, and 256 hidden units.
Furthermore, we invert the chain of mappings (which is possible due to invertibility) for the prior flow as it we found it to be more numerically stable during training.
A. A TTENTION MODULES
We use transformer attention Vaswani et al.
(2017) in various components of L ASER -NV.
We distinguish two types of attention modules: self-attention, where keys, queries and values are computed from the same input, and cross-attention, where some context is used to compute keys and values, and queries are computed from a different input).
An attention module can be composed of multiple attention blocks, where each attention block is shown in Fig.
We use the standard multihead dot-product attention with softmax weights.
We define SelfAttn ( y , L ) to be a self-attention module with input y and L blocks 5 and, similarly, CrossAttn ( y , c , L ) to be a cross-attention module with context c .
Note how SelfAttn and CrossAttn do not include positional encoding that are often used in transformers.
A. I MAGE ENCODER
To encode the context images, we use a convolutional neural network based on the ResNet architecture from Vahdat & Kautz (2020) (also described in Kosiorek et al.
to independently encode context elements into h × w feature maps.
The residual blocks that compose the encoder are shown in Fig.
For all models and datasets we use L = 4 .
Each context view consists of an image in RGB space with concatenated information of the camera position and direction.
For N context views, our last layer outputs Nhw vectors that form the conditioning set used to parameterize our distributions over Z .
Furthermore, the feature maps at different layers are also combined into a feature stack to use as the context local features as described in Appendix A.4.
[Figure]
Fig.
7 provides a high-level overview of the information flows going into the scene function of each model, and we describe implementation details of their components below.
[Figure]
Latent attention
A query point x i attends to Z as per Fig.
7 (a) using a cross-attn ( γ ( x i ) , Z , L ) , where γ ( x i ) is the circular encoding function of positions (see Appendix A.4).
We use the same architecture across datasets, setting L = 3 layers, 512 hidden units, and 4 attention heads.
Local features from multi-view geometry We follow PixelNERF’s design of stacking the features of the encoded images at different levels of resolution, which allows the conditioning features to capture context structure at different scales.
For input images of resolution W × H , we stack the features of the outputs from all layers after the first layer:
[Equation]
where i = 2 l .
When stacking, we upscale the feature maps of all the feature maps of layers l ∈ { 2 . . .
L } to the size of the feature map with largest resolution ( l = 1) using bilinear interpolation.
The resulting set of stacked feature maps (one for each context view) is then used to condition the scene function.
As we described in Section 3.2, our multi-view geometry module is in part based on pixelNeRF’s processing of context views.
Particularly, we process local features with exactly the f 1 ResNet detailed in Yu et al.
(2021): positions and directions are projected to the view-space using the cameras intrinsic parameters, and then processed with 3 residual blocks of 512 hidden units and ReLu activation.
When computing the projection, we also define the binary variable v ni based on whether the projected coordinates are within the image plane.
The output features are integrated with our latent features via M θ which is a cross-attention module.
Note that context features with v ni = 0 are not attended to (we modify the pre-softmax logits of those context features by adding a large negative number to the logits).
In our experiments M θ has 3 layers and 512 hidden units.
Scene function output MLP We now describe the final component of the scene functions of all models, as shown in Fig.
This component follows the residual MLP architecture of the scene function of N e RF - VAE .
Namely, it takes as input the circular encoded positions γ ( x i ) and directions γ ( d i ) , and at every layer it linearly projects the conditioning features (i.e. z for N e RF - VAE , or f i for
L ASER -NV) and sums it to the activations.
The MLP is composed of two parts: the first has 4 layer of 256 hidden units each and outputs the densities, and the second part has an additional 4 layers and outputs the colours.
All layers use a swish activation function.
The circular encoding function γ is given by
[Equation]
where we use different constants L min and L max for positions and directions and for each dataset (listed in Appendix E).
A. N e RF - VAE AND CONDITIONAL N e RF - VAE
Our implementation N e RF - VAE follows the same overall architecture as that of Kosiorek et al.
, but we use the image encoder described in Fig.
6 instead.
The approximate posterior is computed as a diagonal Gaussian distribution, the means and log-variances of which are given by an MLP e as a function of the pooled feature maps H (pooled with a simple averaging over feature maps).
The MLP e has two layers of 256 hidden units and ReLu activation function.
The scene function is composed of an MLP that processes the latent vector z (one linear layer of 256 units, followed by a
ReLu non-linearity), followed by the output MLP described in Appendix A.4.
A. B ACKGROUND SCENE FUNCTION
The volumetric rendering integral of a ray r is given by
[Equation]
If the ray is infinite (camera’s far plane t f → ∞ ) the colour weights T ( t ) σ ( r ( t )) would always sum to one.
For a truncated ray (finite t f ), this happens only if the ray passes through a solid surface.
This heuristic allows to model distant backgrounds by putting them at the end of the integration range, but it doesn’t work if the camera position changes significantly compared to the ray range t f − t n .
If not normalized, the final weight T ( t f ) tell us about the probability of light hitting a particle beyond the ray.
In that case, we can model the background as an infinite dome around the scene and model it with a light-field that depends only on the camera direction.
That is, we compute the colour as
[Equation]
For the City dataset, in order to model the sky and sun, f bg ( Z , d i ) is a learned neural network.
We use a cross-attention module with 2 layers to attend to Z .
For N e RF - VAE and cond.
N e RF - VAE we instead use a 2 layered MLP with ReLu activations.
For the ShapeNet f
bg is a constant colour (white).
We address the challenging backgrounds of MSN-Hard with a more expressive attentive background scene function that conditions on both the latents Z as well as context image features – this is in a similar fashion to how the NeRF scene function of L ASER -NV integrates its latents with attention to local context features.
Given an encoded ray direction γ ( d i ) and context feature maps (cid:83)
Nn =1 H n , a cross-attention module attends to the latents: ˆ h bgi =
f 1 bg ( γ ( d i ) , Z ) , and the output is used to attend to the set of context features (cid:83) Nn =1 H n with a second cross-attention module h bgi = f
bg
( ˆ h bgi , (cid:83) Nn =1 H n ) .
Finally, an MLP f mlpbg computes the background colour by taking as input the obtained background features and additionally conditions on the encoded directions by adding a linear projection of the directions to the activations of every layer.
This MLP is composed of 2 layers of 512 hidden units each with a ReLu nonlinearity, and outputs the RGB background colours using a sigmoid nonlinearity.
B C ITY DATASET
The aim of the City dataset is to allow generating visually complex scenes with large variability in its structure and appearance.
Each generated city consists of a large area spanning approximately 270 m × 270 m , which consists of a 2x2 grid of blocks, where each block is composed of 4 lots.
A lot can be a building or a green patch (park).
The city 2x2 grid is surrounded by an outer ring of buildings.
Roads separate all blocks in the scene.
Each building is randomly generated with variation in size, appearance (textures, materials, colours), architectural styles (residential, industrial, commercial).
Fig.
8 shows examples of generated city scenes.
Once a city is built, we specify a range of points of interest, i.e. road intersections and parks and use each of these points as the starting placement for a scene (data point) of our dataset.
Each scene is comprised of posed images rendered from randomly chosen viewpoints near the point of interest.
More specifically, we render 8 randomly chosen nearby points, and 4 random viewpoints for each point, making a total of 32 generated views per scene.
Each generated view is annotated with the ground-truth render, camera parameters and depth maps.
Our generated training dataset contains 100 , 000 scenes generated in total from approximately 2 ,uniquely generated cities.
We test on 3 , 200 scenes generated in total from 10 uniquely generated cities.
C A DDITIONAL VISUALIZATIONS
Capturing detail at different scale In Fig.
10 we can see at a higher resolution how our model is expressive enough to capture texture and geometry details at different scales (e.g. far-away buildings and nearby objects such as the lamp post and the colonnade).
Novel view synthesis with ShapeNet NMR In Fig.
11 we compare predictions for a number of objects that reveal how N e RF - VAE produces blurry reconstructions compared to MVC - N e RF and
L ASER -NV.
When an image does not reveal the full shape of an object, as shown in Fig.
12, L ASERNV is able to generate multiple plausible variations (in contrast to a deterministic which generates a single blurry prediction).
MSN-Hard predictions Figure 13 shows L ASER -NV’s predictions on MSN-Hard including depth maps estimated from N e RF ’s densities.
D A DDITIONAL RESULTS
Data efficiency We evaluate how data hungry L ASER -NV is compared to N e RF - VAE and MVCN e RF .
In Figure 14 we report the reconstruction colour log-probability estimated on a validation set of the City dataset.
For L ASER -NV, we observe little degradation between using 50K scenes and 12.5K unique scenes.
For N e RF - VAE we notice a larger drop when reducing the size from 25K to 12.5 scenes, showing clearer signs of overfitting.
Finally, since MVC - N e RF severely underfits the City dataset for all training set sizes, we do not see any clear performance drop.
Test-Time Scaling of the Latent Set We further want to understand how different latent set sizes at test time impact performance.
Figure 15 shows reconstruction performance for different values of
K at test time, for various values of used K during training.
As we can see, L ASER -NV does not significantly leverage a larger latent set size than trained with.
As expected, we do see performance degrading when using smaller sets at test time, and this effect becomes stronger for models trained with larger capacity.
Training L ASER -NV without ground-truth depth We train all our models using ground-truth depth on the City dataset.
This reduces memory and computation requirements during training.
We note, however, that it is also possible to train L ASER -NV without using ground truth depths.
Once trained, we find that both the model trained with depths and the model trained without achieves slightly higher average colour log-likelihood (3.87) compared to L ASER -NV (of 3.83) (importanceweighted estimate using 10 samples).
We hypothesize that this is because the model trained without depths is trained solely using the rendering method used at test time.
As any NeRF-related method, L ASER -NV can output estimated depth maps easily.
We show example reconstructions and inferred depth maps of a trained L ASER -NV on the City in Fig.
Note how the model is capable of capturing fine detail in textures and shapes (e.g. lamp post).
Computational efficiency of L ASER -NV vs N e RF - VAE
For a model conditioned on 10 input views, inference (inferring the latent distribution) and rendering of a single image for L ASER -NV (with 24 slots) take 0.39 GFLOPs and 1.80 GFLOPs respectively.
N e RF - VAE instead takes 0.37 GFLOPs and 0.15 GFLOPs.
The main reason for more expensive rendering in L ASER -NV is the introduction of PixelNeRF-like conditioning.
A model with a flow posterior but without such conditioning uses 0.47 GFLOPs to render a single image.
Attention, however expensive, is evaluated only once per scene (to compute the prior/posterior).
The PixelNeRF features are cheap per point, but they are evaluated for every 3D point used in rendering, therefore dominating the overall cost.
Note that the above numbers do not reflect training latency.
At training time we can guide the rendering process by using depths, which allows us to evaluate the scene function only twice for every pixel.
We also subsample images and we never reconstruct the whole target view(s).
This makes training faster, and means that L ASER -NV is only about 4x slower per training iteration than N e RF - VAE .
Having said that, we noticed that N e RF - VAE does not benefit from longer training (its performance saturates), while the performance increase from increasing the model size (to equalize the FLOPs) is negligible.
Robustness to camera noise We carry out an experiment on MSN-Hard in which we train and test with camera poses with incremental levels of Gaussian noise.
We replicate the set up in Sajjadi et al.
and compare their reported results using SRT and PixelNeRF to L ASER -NV.
Results in Fig.show that while L ASER -NV outperforms the other models when there is no noise, its performance drops more quickly compared to SRT, in a similar fashion to PixelNeRF.
However, L ASER -NV is more robust than PixelNeRF for all levels of noise.
We believe that the use of NeRF and local features makes these models less robust to noisy cameras.
That said, there are recent methods that allow mitigating noise sensitivity with NeRF (Lin et al., 2021) that can be incorporated.
The PSNR values of L ASER -NV* are preliminary training values models that have not trained to convergence (340K steps out of 1000K).
E T RAINING DETAILS
Hierarchical rendering Instead of optimizing separate sets of parameters for the coarse and fine rendering passes, we find that it suffices to only keep separate parameters of F θ .
This allows sharing parameters (and reduce computation) when querying the scene function on the coarse sampled points with respect to latent attention and multi-view geometry components of models.
Training with depth (City) Because we have RGB-D data in the City dataset, we can use depth as a supervised signal following the method in Stelzner et al.
This method involves replacing the pixel colour likelihood, for a given ray r , with the ray colour and depth likelihood p ( t ( r ) , c ( r ) | V , Z ) , where t ( r ) is the ray’s ground-truth depth.
This alternative likelihood requires only two scene function evaluations per ray.
In practice, we find that we get the best quality renders when combining both loglikelihood terms to the loss (each weighted by 0.5).
At every training set, similar to how N e RF - VAE is trained, the likelihood terms are estimated by only reconstructing a subset of rays (i.e. pixels) of all the target images.
We find that using 64 rays for estimating the (expensive) image log-likelihood, and 512 to estimate the ray colour and depth likelihood term, works well in the City.
Additionally, we restrict output densities to be between 0 and 10 by using a weighted sigmoid in the output of the scene function.
Additional regularization The loss function for ShapeNet experiments includes a density regularization term based on the L1-norm of the densities, with a weight of 0 . 01 .
For all experiments, we scale the model parameter updates (Pascanu et al., 2013, Sec. 3.2), such that loss gradient norms have a maximum value of 10 for improved stability.
Hyper-parameters We use the following hyper-parameters across all models.
Some of which were selected for each dataset based on a preliminary exploration using the training dataset and a validation split.
Hyperparameter
City
[Table]
NMR
MSN-
[Table]
Learning rate 0.
Batch size   # of rays per scene   # of context views   # of target views   # of posterior context views   # of coarse points   # of fine points
Camera t near 0.
Camera t far 3.
Positions circular encoding L min - - -
Positions circular encoding L max
[Table]