INFERNO: I NFERRING O BJECT -C ENTRIC 3D S CENE R EPRESENTATIONS WITHOUT S UPERVISION
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose INFERNO, a method to infer object-centric representations of visual scenes without relying on annotations.
Our method learns to decompose a scene into multiple objects, with each object having a structured representation that disentangles its shape, appearance and 3D pose.
To impose this structure we rely on recent advances in neural 3D rendering.
Each object representation deﬁnes a localized neural radiance ﬁeld that is used to generate 2D views of the scene through a differentiable rendering process.
Our model is subsequently trained by minimizing a reconstruction loss between inputs and corresponding rendered scenes.
We empirically show that INFERNO discovers objects in a scene without supervision.
We also validate the interpretability of the learned representations by manipulating inferred scenes and showing the corresponding effect in the rendered output.
Finally, we demonstrate the usefulness of our 3D object representations in a visual reasoning task using the CATER dataset.
I NTRODUCTION
Inferring objects and their 3D geometry in a scene is a fundamental ability of biological visual systems (Kahneman et al., 1992; Roelfsema et al., 1998; Spelke et al., 1993).
Replicating this ability in machine is a promising step towards visual reasoning valuable to several applications involving object manipulation, navigation or forecasting.
Recent works (Jiang et al., 2019;
Locatello et al., 2020; Burgess et al., 2019) have shown that neural networks can learn object-centric representations from low-level perceptual features.
They learn to recognize the objects in a visual scene from a singe image without relying on supervision.
However, most of those approaches only consider the 2D structure of images and ignore the underlying 3D geometry of the visual scenes.
On the other hand, Neural Radiance Fields (NeRFs)
(Mildenhall et al., 2020) have demonstrated that differentiable renderers can be combined with gradient-based optimization to learn high-ﬁdelity 3D scene reconstructions.
NeRFs have been subsequently used to learn 3D-aware generative models, including compositional scene models (Niemeyer & Geiger, 2021).
In this work, we leverage these recent advances in object-centric representation learning (Locatello et al., 2020) and 3D modelling through implicit functions (Mildenhall et al., 2020; Niemeyer &
Geiger, 2021) and propose INFERNO, a model which infers a structured representation of objects and their 3D poses from a single image.
Each object is represented by latent variables characterizing its shape and appearance, together with an explicit representation of their 3D poses (translation, scale and rotation).
The object representations are then decoded using implicit functions that are localized in the scene according to the objects poses and combined together to generate a 2D output view.
Our model does not need supervision and instead is ﬁtted through minimizing a reconstruction loss, akin to an auto-encoder.
Disentangling the object appearance and pose in a scene representation allows for the model to manipulate a visual scene.
In particular, we demonstrate that INFERNO learns interpretable object poses, which we can modify and render to alter the pose of an object in a scene.
We also validate that our approach learns meaningful representations for object discovery and visual reasoning.
More speciﬁcally, we show that our approach obtains competitive performance on the CLEVR6 object discovery benchmark (Johnson et al., 2017; Greff et al., 2019) as well as for the snitch localization visual reasoning task of CATER (Girdhar & Ramanan, 2019).
In summary our contributions are the following:
• We propose a model able to infer and render 3D scene representations composed of multiple objects, each of them modeled by an implicit function and explicitly localized in the scene.
• We show that the representations learned by the model are interpretable and amenable to manipulations.
• We demonstrate that the inferred representations are useful for downstream tasks by showing competitive performance in object discovery and reasoning tasks.
M ETHOD
Shape
Appearance
Pose
Shape
Appearance
Pose
Shape
Appearance
Pose
Shape
Appearance
Camera
INFERENCE
RENDERING
[Figure]
Background
Low Resolution Rendering
Reconstruction
[Figure]
We propose INFERNO (Infer NeRF Objects).
The goal of our method is to infer object-centric 3D scene representations from single 2D views.
Given an image x ∈ R H × W × 3 , we learn an inference function f θ that maps images to scene representations s = f θ ( x ) =
( o 1 , o 2 , ..., o K , o bg , c ) .
Scenes are composed of K objects o i , a background object o bg and a camera location c .
Each object is composed of three tensors o i =
( o shapei , o appi , o posei ) .
The object shape o shapei ∈ R D shape and object appearance o app ∈ R D shape are tensors that respectively describe the 3D shape occupancy and color of an object with an implicit function.
The object location o posei ∈ R 4 × 4 is an afﬁne matrix that describes the object 3D pose (i.e. scale, translation and rotation) in the scene.
The background object o bg = ( bg shape , bg app ) only models 3D shape and color, and its location is ﬁxed, encompassing the back-of-scene cube.
We also deﬁne a camera matrix c ∈ R 3 × 4 , that deﬁnes the location of the scene camera and determines a 2D projection of the 3D scene.
To optimize our inference function we formulate an optimization problem in which we minimize a reconstruction loss over a dataset, similar to an auto-encoder.
We deﬁne a rendering function g γ that takes as input a scene representation and generates a 2D view of that scene ˆ x = g γ ( f θ ( x )) .
We assume a isotropic Gaussian likelihood model with unit covariance and optimize the probability of the data under our model, which is equivalent to minimizing the mean squared error of inputs and reconstructions:
[Equation]
In the following sections we describe in more detail our inference mechanism, our rendering pipeline and their implementation.
2. R ENDERING P IPELINE
We represent 3D objects as Neural Radiance Fields (NeRFs)
(Mildenhall et al., 2020) with a similar setup as that of GIRAFFE (Niemeyer & Geiger, 2021).
A NeRF is a function g τ that deﬁnes a 3D shape implicitly.
It takes as input a 3D location l
= ( x, y, z ) and a 2D viewing direction d = ( ψ, φ ) and outputs an occupancy value σ and a color value
a = ( r, g, b ) .
NeRFs are usually implemented using fully connected neural networks.
Additionally, the inputs are usually embedded into a higherdimensional space using positional encodings γ that embed locations and viewing directions into higher dimensional spaces R P l and R P d , respectively.
[Equation]
To represent multiple 3D shapes with the same NeRF function, we can augment it with latent variables that determine which shape is being modeled (Schwarz et al., 2020).
NeRFs are usually augmented with two random variables: one random variable µ ∈ R D shape deﬁnes the shape of the entity being modeled, while υ ∈ R D app models its appearance.
In practice, this specialization is enforced by making the occupancy output a function of only the shape latent, while the color output is conditioned on the appearance latent.
[Equation]
In INFERNO, we share a single parametrization of a NeRF function across all objects.
Each object speciﬁc shape and appearance is deﬁned by the shape and appearance latent variables, which correspond to the object attributes o shape , o app .
The background is deﬁned as another NeRF with separate parameters.
The background NeRF also has shape and appearance latent variables to model different backgrounds.
The pose of an object o i in the scene is determined by the afﬁne transformation matrix o posei .
We denote the coordinate system of the NeRF function of an object as the object space, and the coordinate system of the scene (and the background NeRF) as scene space.
Given an object pose, we can convert points from the scene space to the object space by applying the o pose transformation matrix on those points, and we can transform points from object space to scene space by computing the inverse of the object pose matrix.
To render a scene, we cast rays from each pixel in the 2D plane deﬁned by a given camera to the 3D scene.
We evaluate NeRFs at different points along a given ray, and integrate their occupancy and color outputs to determine pixel values.
Rays might traverse multiple object NeRFs in addition to the background NeRF.
To determine the occupancy and color of points described by multiple
NeRFs, we ﬁrst query each NeRF at those particular points.
To query the object NeRFs, we ﬁrst need to transform the points from scene space to the particular object space.
Then, we compose the results of each NeRF with a pooling function C , which in our case is a weighted average:
[Equation]
Since rendering with NeRFs as originally proposed is computationally expensive, we render output views at a ﬁxed low resolution.
Low resolution scenes are then upscaled to the desired output resolution using a convolutional neural network, keeping the entire rendering pipeline differentiable.
Additionally, low resolution scenes are rendered with additional channels, allowing for more detailed upscalings beyond those possible when just rendering low resolution RGB outputs.
For more details about NeRF rendering and compositional NeRF models refer to (Mildenhall et al., 2020; Niemeyer & Geiger, 2021).
2. I NFERENCE
Given the rendering pipeline, the goal of our method is to infer representations that reconstruct a given scene.
Our inference mechanism computes image features through a neural network encoder and then extracts K object and a background slots.
These slots are then mapped to our structured scene representation through learned neural networks.
To extract image features for each object and background slots, we use Slot Attention (Locatello et al., 2020).
Slot Attention is a mechanism that maps a set of K entities, called slots, to image features without annotations.
It extracts image features
I ∈ R H
× W × D from a given input using a resolution-preserving convolutional encoder.
These features are then attended to by a set of K randomly sampled slots π j ∼ N ( µ, σ ) of dimension D , where µ and σ are learnable parameters.
We denote by π the matrix concatenating all the sampled slots.
Slots attend spatial chunks of the input features I through soft-attention u = TQ T , where
T = k ( I ) and Q = q ( π ) are the embeddings of the inputs and slots respectively.
The attention weights are normalized through a softmax operating on the slots axis, which makes slots compete among themselves and discourages multiple slots from attending the same input region w = softmax ( u ) .
The weighted average of I according to the attention weights is then computed and fed to a GRU network, to update each slot value: π j = GRU ( w j ∗
I, π j ) ∀ j ∈ 1 , K .
Multiple rounds of soft attention are performed to iteratively reﬁne the slots.
For more details about Slot-Attention, please refer to (Locatello et al., 2020).
Object slots are unstructured tensors that result from aggregating image features.
We map these slots to our structured scene representation through small fully-connected networks that operate on individual objects.
More concretely, we map object slots to their 3D pose in the scene through a 2-layer MLP.
To infer the object shape and appearance tensors we also use 2-layer MLPs, but we make them conditional on the predicted object pose through conditional normalization (Dumoulin et al., 2018).
R ELATED W ORK 3. 3D S HAPE R EPRESENTATIONS
There are different ways to represent 3D geometry such as voxels or meshes (Rematas & Ferrari, 2020; Gkioxari et al., 2019).
For example, the GAN models of Nguyen-Phuoc et al.
(2019; 2020) successfully use voxel-based representations to render images.
Voxel-based methods have trouble scaling up to high resolutions as the size of a voxel representation scales cubically with the resolution.
Recently, the use of functions that implicitly model 3D volumes has gained popularity (Park et al., 2019;
Mescheder et al., 2019; Sitzmann et al., 2019; 2020b;a; Kosiorek et al., 2021;
Pumarola et al., 2021; Yu et al., 2021a).
Implicit representations have better scaling properties, as usually the output resolution does not directly affect the dimensionality of the learned function.
NeRFs (Mildenhall et al., 2020) generate scenes by learning a function that outputs the occupancy and color of points in a scene when viewed from a particular direction.
By casting rays through a plane and aggregating the output values NeRFs can generate 2D views of an implicitly modeled 3D scenes.
NeRFs have obtained superior reconstructions compared to other implict methods, and our model uses NeRFs to represent multiple objects and the background of a scene.
Most methods using NeRFs represent scenes monolithically as a single entity.
GIRAFFE (Niemeyer & Geiger, 2021) is a GAN-based method that represents multiple objects in a scene using NeRFs as part of their generator.
Their factored representations are amenable to object manipulations.
Our model uses a rendering pipeline inspired by GIRAFFE.
However, we focus on recovering scene representations from existing images, while GIRAFFE does not have an inference mechanism.
ObjSURF (Stelzner et al., 2021) and UORF (Yu et al., 2021b) infer scene representations composed of multiple objects, each object represented with a differently instantiated NeRF.
Different from our work, they focus on novel view generation.
These methods do not explicitly infer the pose of the different objects in the scene.
Both methods use ground-truth camera locations and multiple scene views during training, and ObjSURF additionally requires depth annotations.
3. O BJECT -C ENTRIC S CENE M ODELS
Segmenting objects in a scene is a landmark computer vision task with an extensive literature.
Recently, there is a line of work on object-centric generative models of scenes (Kosiorek et al., 2021;
Burgess et al., 2019; Locatello et al., 2020; Lin et al., 2020; Greff et al., 2019).
These models learn to generate scenes as a composition of multiple objects and a background.
When equipped with an inference mechanism, these models learn to segment objects in a scene without annotations, driven by their compositional generative process.
MONet (Burgess et al., 2019) implements a multi-object
VAE that segments object sequentially by infering latents corresponding over parts of the scene not yet attended to iteratively.
IODINE (Greff et al., 2019) uses a similar multi-object VAE and performs multiple rounds of inference to settle on a scene decomposition.
Slot-Attention (Locatello et al., 2020) maps a set of entities, called slots, to image features through multiple rounds of soft attention.
The slots compete among themselves to attend to features, making each slot attend to a region of the input image.
When driven with alpha-compositing decoder, slot attention learns to segment objects in a scene.
Our model uses a variant of Slot Attention to decide on which part of a 2D view should each object in our scene attend to, but we infer 3D-aware representations for each object.
Object-centric scene models have also been implemented as world models, with the goal of simulating dynamics (Lin et al., 2020).
By decomposing the scene into objects, these methods can simulate dynamics at an object level, which are usually simpler and shared among objects.
Contrary to most previous approaches which segment 2D shapes, our method infers object-centric scene representations in 3D space.
E XPERIMENTS
In this section we showcase the capabilities of INFERNO with three main experiments.
First, we demonstrate the interpretability of the scene representations through manipulating scenes and verifying the corresponding effects in the rendered outputs.
Then we show that it learns to identify and segment the objects in a scene without supervision.
Finally, we highlight the usefulness of such representations for downstream tasks by applying our model on the CATER snitch localization task.
4. T RAINING S ETUP
We use the same training setup for all experiments unless otherwise mentioned.
We train our models for 400K iterations using the Adam optimizer Kingma & Ba (2014) with a learning rate of 1 × 10 − 4 .
We use a batch size of 128 and we use up to 16 nVidia V100 GPUs.
We use learning rate warmup (Goyal et al., 2017), which was found to be helpful to avoid optimization issues with
Slot Attention.
We use a weight decay rate λ = 1 × 10 − 6 .
We use three iterations of slot attention during training and evaluation.
We set up the number of objects in a scene as the maximum possible number of objects in a dataset, i.e. ﬁve objects for CLEVR2345, 6 for CLEVR6 and 10 for CATER.
Please refer to Appendix B for more details on the experimental setting.
4. S CENE I NFERENCE
In this section we demonstrate the properties of our scene representation.
Our model infers 3D object-centric scene representations from single 2D views.
These representations disentangle the 3D appearance and pose of objects, which allows for semantic manipulations of the scene not possible otherwise.
These manipulations can be validated by rendering the modiﬁed scene representations.
Additionally, we verify that decomposing the scene into multiple objects leads to better reconstructions.
First, we verify the quality of INFERNO’s generations by comparing them two baselines: i) a version of our model that does not consider multiple objects, and ii) the GAN method of GIRAFFE (Niemeyer & Geiger, 2021).
We perform this comparison on the CLEVR-2345 dataset
[Figure]
Removal
Scale
Forward
Right
[Table]
introduced by GIRAFFE, which contains CLEVR images with 2 to 5 objects.
For reconstruction, we compare models using reconstruction metrics including mean-squared error, PSNR and SSIM
We rely on the population metric Frechet Inception Distance (FID) to evaluate the generation quality.
In INFERNO, we generate novel scenes by inferring representations for ground-truth images and then manipulating them.
To compare to GIRAFFE, we manipulate scenes by adding additional objects and swapping object shapes and appearances across scenes.
Manipulations are described in more detail in the Appendix.
We highlight that GIRAFFE is an unconditional model, while
INFERNO generates novel scenes conditioned on existing ones.
We also investigate different interpretable manipulations of scene representations and visualize the effects in the corresponding output renderings.
We also conduct this experiment on the CLEVR2345 dataset.
We validate that our model is able to render out-of-distribution scenes not corresponding to training examples, such as scenes having 1 or 6 objects
[Table]
and verify that the pose manipulations have semantically coherent effects.
consider it to state of the- ground-truth images our model obtains better
We consider object manipulations to generate novel scenes from existing ones scene slightly increases our FID identities we
[Table]
Table 1 shows the reconstruction metrics obtained by our model and baseline on the CLEVR dataset.
Note that GIRAFFE is a GAN-method that does not have an inference mechanism, and therefore it cannot reconstruct scenes.
We observe that the NeRF-AE baseline obtains higher reconstruction error than our regular model, as our object-centric method can make better use of its capacity.
In Table 2 we compare INFERNO with GIRAFFE using the FID metric.
Our model reconstructions have better FID than the generations of the GIRAFFE.
Additionally, our model can perform inference and manipulations on existing scenes.
We use that capability to generate novel scenes by manipulating existing ones.
Our model is able to generate novel scenes with additional objects or with altered object shapes and appearances, with better FID than GIRAFFE.
In Figure 2 we show some examples of the scene manipulations possible with our model.
Given a scene representation, we can remove or add objects, rearrange object poses, translate the objects to new locations or change the object scales.
While some of these manipulations can be performed with regular object-centric models, modiﬁcations to the scale and location of the objects are hard to implement without explicitly modeling 3D object pose.
4. O BJECT D ISCOVERY
[Table]
Unsupervised object discovery consists in learning to segment the objects in a scene without using annotations.
We test our model on CLEVR6 benchmark, a variant of the CLEVR dataset with scenes of up to 6 objects and annotated with 2D object masks.
We choose this dataset to compare to previous work in unsupervised object discovery.
Note that this setup evaluates 2D segmentation masks, although our model naturally provides 3D segmentations.
We evaluate the quality of the segmentations using the Adjusted Random Index (ARI) metric (Rand, 1971), which is a measure of clustering similarity.
In line with previous work, we compute only the foreground ARI, which does not take into account the background segmentation mask.
In particular, we consider object as a different clusters and compare the cluster assignment of each foreground pixel in the original image to its prediction.
To determine which pixels correspond to each object in our model we make use of the input segmentation masks predicted by our inference mechanism.
Table 3 reports the ARI metric of our model and different baselines in this task.
We observe that INFERNO is competitive with state-of-the-art methods, surpassing MONet and having slightly lower
[Figure]
ARI than Slot-Attention and IODINE.
However, our model learns to segment objects in 3D, while the other baselines extract 2D segmentation masks.
Figure 3 shows some examples of the discovered object masks.
We can see that the model learns to segment different objects and properly discards object slots when the number of objects in the scene is lower than needed.
We also include an example that our model fails to segment properly - multiple objects are segmented by the same object slot, and a single object is represented in multiple parts by different slots.
4. S NITCH L OCALIZATION
The goal of this experiment is to show that the representation learned by our model is useful for the snitch localization task from the CATER dataset (Girdhar & Ramanan, 2019).
We focus on the
CATER task involving videos with a static camera.
The objective is to predict the ﬁnal position of an object (the snitch) in a video.
The scenes show multiple objects that move over time, one being the snitch object.
The snitch can be occluded and moved around simultaneously by other objects, requiring object tracking and reasoning about dynamics to solve the task.
We follow the experimental setup of Ding et al.
First, we train INFERNO to reconstruct images from the CATER dataset.
Once INFERNO is trained, we discard the rendering pipeline of our model, and instead feed the scene representations to a 12-layer transformer model to predict the ﬁnal snitch position.
Each object in our representation is given as an input element to the transformer.
We add a learned positional encoding to the object representations based on their frame index.
Objects in the same frame have the same positional encoding.
The transformer last output is given to a MLP head that predicts the logits for the 36 possible output positions.
We minimize the sum of the cross-entropy and a L1 loss between the predicted and the true snitch ﬁnal position.
Following (Ding et al., 2020), we also consider the use of auxiliary SSL loss after pretraining.
The SSL loss randomly masks one object per-frame and tries to predict its representation at the corresponding output.
The model minimizes the L2 distance between the predicted object
[Table]
representation and the observed one.
The SSL loss is only backpropagated through the transformer and not the inference network.
During training [Table] we randomly samples 40 frames from a video and predict the snitch localization from this one crop.
At test time [Table] we randomly sample 10 temporal crops of 40 frames each and average the prediction over the 10 crops as our ﬁnal prediction.
Refer to Appendix B for more details about the experiment setup.
Table 4 reports CATER Top-1 and Top-5 accuracies for different methods.
We ﬁrst compare our model that is pretrained to reconstruct image on CATER with a randomly initialized encoder.
Using a randomly initialized encoder does not perform well.
It indicates that INFERNO pretraining, and not the encoder architecture, is key to learn useful representation for the task.
We also observe that the additional SSL loss provides regularization and slightly improves the performance of our model.
We next compare our approach with Aloe (Ding et al., 2020), an object-centric baseline, R3D and
R3D
NL, 3D convolutional models proposed by (Girdhar & Ramanan, 2019) and architectures using strong inductive biases toward object tracking such as OPNet (Shamsian et al., 2020) or Hopper (Zhou et al., 2021).
Our model outperforms the R3D and R3D NL models that rely on unstructured representation.
This result suggests that the object-centric representation learned by INFERNO is useful for the visual reasoning task.
Our model also outperforms Aloe, an object-centric method using 2D object representation, when both methods do not use additional SSL loss.
However Aloe beneﬁts more from the use of an additional SSL loss.
Overall, INFERNO achieves performances close to the state-of-art approaches.
We ﬁnally evaluate the performance of a slot-attention baseline (Locatello et al., 2020) in Table 4.
The slot-attention baseline ﬁrst pretrains a slot-attention encoder, with a similar architecture than our model, by reconstructing CATER frames using a mask decoder.
It then ﬁne-tunes the encoder using the same procedure than our model to solve the snitch localization task.
We observe that our model signiﬁcantly outperforms the slot-attention model which focus on the 2D geometry of the scene.
This result supports the advantage of 3D-aware representation for solving the CATER task.
C ONCLUSIONS
We propose INFERNO, a model for inferring object-centric 3D scene representations.
Our model is able to discover objects in a scene without annotations, and the inferred scene representations are interpretable and amenable to manipulations.
Further, the scene representation is useful for visual reasoning downstream tasks such as the snitch localization task in CATER.
A A DDITIONAL M ODEL D ETAILS
Our model is composed of ﬁve main modules: encoder, slot attention, slot to object mapping, NeRF decoder, neural network upscaler.
In this section we provide additional details about each of these modules as well as describe their architecture
[Table]
[Table]
Encoder The goal of the encoder is to extract image features.
We use an encoder with no downsampling, as it is typically used with Slot Attention.
Details about the encoder architecture are described in Table
[Table] [Table]
The output of soft-attention goes through a linear layer + GRU
Slot Attention We employ Slot Attention to map image features to object slots.
Slots are sampled randomly from a Gaussian distribution with learned parameters.
We use different distributions for the background slot and the object slots.
During training we employ three iterations of slot attention to reﬁne the image features to slot assignments
[Table] [Table]
The background and object slots are mapped to scene parameters using a series of MLP.
For each object slot, we ﬁrst map the object to its pose parameters.
We use a 2-layer
MLP with LayerNorm to map a slot to its pose.
The size of the hidden dimension is the same than the output dimension size.
We parametrize object pose as a 7-dimensional tensor.
We use three dimensions for the object location along each axis, three dimensions for the scale of the object and a single dimension to express a rotation of the object along the X axis.
These parameters are then mapped to their corresponding 4 × 4 afﬁne transformation matrix for each object.
Note that in practice we are not modeling rotations in the experimental section.
Once we have inferred the object poses, we infer object shapes and appearances.
These are inferred individually for each object using a common 2-layer MLP.
The MLPs are conditional on the object pose using Conditional Layer Normalization, that makes the learned parameters of LayerNorm be a function of a condition.
Speciﬁcally, we map the 7-dimensional pose tensor to LayerNorm parameters with a single linear layer with no activation or normalization.
Shapes and appearances are deﬁned by the output of the MLPs, which produce two 128-dimensional tensors.
For the background object we only infer shape and appearance, and deﬁne its pose to be that of the scene cube.
The shape and appearance of the background are inferred through another 2-layer MLP with ReLU and LayerNorm.
Note that our scene representation also admits a camera pose.
In our experiments we ﬁx the camera location to look at the scene from a standard location (centered and 33 degrees above the Z plane).
Other concurrent approaches (Yu et al., 2021b; Stelzner et al., 2021) use ground-truth camera locations to generate novel views, while we focus on recovering scene representations without the use of ground-truth annotations.
[Table]
NeRF MLPs With the object shape and appearance tensors we can render them following
GRAF (Schwarz et al., 2020).
We use one NeRF for the objects and one NeRF MLP for the background.
The details about each NeRF architecture can be found in Table 8.
Note that, to render objects according to their pose, we query their NeRF MLP in a canonical object space by transforming input coordinates in scene space to object space using the object pose.
To reduce the computational complexity of rendering with many NeRFs, we render scenes at a ﬁxed resolution of 16x16.
Instead of rendering RGB pixels, we render feature images with 128 channels.
The output of the rendering is then upscaled and mapped to RGB views with a neural network upscaler.
[Table]
Neural Upscaler
The neural upscaler takes the low resolution output of the NeRF MLPs and upscales it to the full output resolution.
Additionally, it maps the rendered image to RGB space.
This module is implemented using a convolutional neural network.
We always render the NeRF output at 16px.
Consequently, we add additional layers to the neural upscaler depending on the desired output resolution.
For most experiments we use a resolution of 64px, while for the Scene
Inference experiments on CLEVR2345 we use a resolution of 128px.
The architecture of the neural upscaler can be found in Table 9.
B E XPERIMENT D ETAILS B. S CENE M ANIPULATION
Dataset For generating manipulated scenes we consider the CLEVR2345 dataset (Niemeyer &
Geiger, 2021).
Images in the CLEVR2345 dataset contain from 2 to 5 objects.
We use the original train and test splits.
Images are resized to 128x128 pixels and RGB values are normalized in the [0, 1] range.
Training We use a batch size of 128 and train our model for 400k iterations.
We use Adam with a learning rate of 1 × 10 − 4 and weight decay 1 × 10 − 6 .
We use 5 objects and rely on the model to not use additional slots if the scene shows less than 5 objects.
We use the neural upscaler with additional layers to upscale to 128px.
Additionally, for this experiment we use an additional LPIPS loss.
We use the LPIPS metric computed by an AlexNet network, and we add this loss to our regular MSE loss.
We weight the LPIPS loss by a factor of 100 , so that it has a comparable order of magnitude to the MSE loss.
Manipulations Manipulations are done as follows:
• Substraction
: We randomly delete up to two of the object slots.
• Addition: We randomly add an object slot from another scene.
• Scale: We reduce the scale (in all XYZ axis) of one of the objects in the substraction scene.
• Forward: We manipulate the pose vector of one object in the substraction scene and move it forward on the Z axis.
Right: We manipulate the pose vector of one object in the substraction scene and move it forward on the X axis.
Additionally, we consider the Swap transformation for Table 1.
This transformations modiﬁes a scene by replacing the object shape and appearance vectors with those of an object from another scene.
Metrics To compare reconstructions we use Mean-Squared Error, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM) and the LPIPS metrics.
MSE measure the average squared difference between pixel values.
[Equation]
PSNR is a metric commonly used in signal processing.
[Equation]
SSIM (Wang et al., 2004) provides scores more aligned with human perception, specially under the presence of image noise.
Scores are computed convolutionally by applying a kernel over images, which are then contrasted.
LPIPS (Zhang et al., 2018) computes differences in neural network activations for two images.
It is a perceptual metric that has been shown to have higher correlation to human perception than other metrics not based on neural networks.
To compare populations of generated images we use the Frechet Inception Distance (Heusel et al., 2017).
The Frechet Inception Distance embeds images into a neural network space and then ﬁts a
Gaussian distribution to the generated and ground-truth activation statistics.
The score is obtained by then computing the Frechet distance between the two.
Note that other metrics such as Inception
Score are not applicable for the CLEVR2345 since there are no well-deﬁned classes.
B. O BJECT D ISCOVERY
Dataset For object discovery we consider the CLEVR6 dataset.
We use the original CLEVR dataset and extract the images from TFRecord ﬁles available at this URL.
We use the original training/test split, using the ﬁrst 70% images for training and the remaining ones for test.
We take a crop between pixels [29, 221] and [64, 256], for the height and width respectively, and then resize the crop to 64px.
We normalize the value of the images between [0, 1].
To generate the CLEVR dataset, we keep only those images that have at maximum 6 objects according to the annotation ﬁles.
Training We use a batch size of 128 and train our model for 400k iterations.
We use Adam with a learning rate of 1 × 10 − 4 and weight decay 1 × 10 − 6 .
We use 6 objects and rely on the model to not use additional slots when needed.
Metrics We follow previous work (Greff et al., 2019) and use the Adjusted Rand Index (ARI) (Rand, 1971) to evaluate cluster assignments in object discovery.
ARI scores range from(random assigment) to 1 (perfect match).
As in previous works, we do not consider a segmentation mask for the background.
B. V ISUAL R EASONING ON CATER
For the visual reasoning task, we consider the CATER dataset and uses 5K videos that do not have camera motion.
All the videos are resized to a 64x64 resolution.
B.3. P RETRAINING
We ﬁrst pretrain our INFERNO to reconstruct individual frames from the CATER dataset.
We bootstrap a model trained on CLEVR6 for object discovery for 400k iterations and train it on the
CATER dataset for an additional 100k iterations to speed-up training, as the iteration time of a model with 10 objects is larger.
We use a batch size of 128 and we use Adam with a learning rate of 1 × 10 − 4 and weight decay 1 × 10 − 6 .
When training on CLEVR6 we use 6 object slots, while when training on CATER we use
object
slots
. B.3. F INETUNING
After pretraining, we ﬁnetune the INFERNO encoder to the supervised task of snitch localization.
We discard the rendering pipeline of our model, and instead feed the inferred object slots representations to a transformer that aims at predicting the ﬁnal position of the snitch.
To predict the snitch, we consider a 12 layers transformer with the hidden dimension of 128 which takes the slot representation as input.
The transformer treats each object as input element.
A learned positional embedding is added each slots representation based on their frames index, i.e. the position of the objects is the same within a frame.
The ﬁnal output to the transformer is given to a 1 layer
MLP head with an hidden dimension of 128.
It outputs 36 logits that correspond to possible snitch location.
We minimize the sum of the cross-entropy between the predicted position and the true target and a the l1 loss between the prediction and target.
We optionally use a SSL loss similar to Ding et al.
The SSL loss randomly masks one object per-frame and tries to predict its representation at the corresponding output.
The model minimizes the L2 distance between the predicted object representation and the observed one.
The SSL loss is only backpropagated through the transformer and not the encoder.
We weight the SSL loss by a factor of 1 .
e − 3 .
We use an Adam optimizer to minimize the loss.
The initial learning rate is set to the 1 .
e − 4 and gradually decreased to 1 .
e − 6 using a cosine learning rate decay.
Similarly, we use a initial weight decay of 1 .
e − 5 that we increases to 1 .
e − 3 using a cosine schedule.
Our model is ﬁnetuned for 500 epochs.
We don’t make use of learning rate decay.
During training, we randomly samples 40 frames from a video and predict the snitch localization from this one crop.
At test time, we randomly sample 10 temporal crop of 40 frames each and average the prediction over the 10 crops as our ﬁnal prediction.
C A DDITIONAL V ISUALIZATIONS
Input
Recon.
Input
Recon.
Input
Recon.
We have included additional manipulations of one scene in GIF format in the supplementary material.
We show: i) each object rendered individually, ii) object identity swaps with other scenes, iii) object translations along one axis, iv) translations in diagonal (two axis), and v) objects moving in a circle.
[Figure]
Add
Remove
D N OVEL V IEW S YNTHESIS
In this section we synthesize novel views of a scene by modifying the camera pose.
For each example we show the input image, our reconstruction, then two images as a result of moving the camera ± 15 ◦ in the azimuth axis, and two images as a result of zooming in the scene.
While our model is trained without ground-truth camera poses and with single views of scenes, it is able to generalize to small camera pose modiﬁcations and render novel views of a scene
Input
Recon.
Rot.
+15 deg.
Rot.
-15 deg.
Zoom x1.
Zoom x
E N E RF OUTPUT
In this section we show the raw outputs of the NeRF function.
These outputs show scene views rendered at low resolution, which are then upscaled with a neural network.
While these generations have reduced details due to their resolution, they clearly show the different objects and their location in the scene.
NeRFs are rendered at low resolution to ease the computational costs, as the time and memory requirements of rendering with a NeRF are linearly correlated with the number of casted rays/pixels in the output image.
Input
Recon.
NeRF out
Input
Recon.
NeRF out
F S HAPE - A PPEARANCE D ISENTANGLEMENT
In this section we show examples of the disentanglement of shape and appearance of objects.
To demonstrate this property of our representations, we ﬁrst reconstruct a given scene, and we then randomly change the appearance vector of one of the objects in the scene by the appearance vectors of other objects in the same scene.
While the shading and color of the object changes, the overall shape remains the same.
Input
Recon.
App.
App.
G O CCUPANCY M APS
In this section we show the occupancy maps for each rendered object.
The occupancy maps are obtained by integrating rays going from the image plane to the scene according to the NeRF density, but without taking into account the RGB output.
We observe that each object is rendering a part of the scene corresponding to a single object instance.
NeRF outputs are rendered at low resolution and then upscaled to full resolution with a neural network.
To perform unsupervised object discovery we use the outputs of slot attention, since they do not require any upsampling and thus can provide more accurate segmentations.
[Figure]