Original: L EARNING G LOBAL S PATIAL I NFORMATION FOR M ULTI -V IEW O BJECT -C ENTRIC M ODELS
Revised: L EARNING G LOBAL S PATIAL I NFORMATION FOR M ULTI -V IEW O BJECT -C ENTRIC M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, several studies have been working on multi-view object-centric models, which predict unobserved views of a scene and infer object-centric representations from several observation views.
Revised: Recently, several studies have been working on multi-view object-centric models, which predict unobserved views of a scene and infer object-centric representations from several observation views.

Original: In general, multi-object scenes can be uniquely determined if both the properties of individual objects and the spatial arrangement of objects are speciﬁed; however, existing multi-view object-centric models only infer object-level representations and lack spatial information.
Revised: In general, multi-object scenes can be uniquely determined if both the properties of individual objects and the spatial arrangement of objects are speciﬁed; however, existing multi-view object-centric models only infer object-level representations and lack spatial information.

Original: This insufﬁcient modeling can degrade novel-view synthesis quality and make it difﬁcult to generate novel scenes.
Revised: This insufﬁcient modeling can degrade novel-view synthesis quality and make it difﬁcult to generate novel scenes.

Original: We can model both spatial information and object representations by introducing hierarchical probabilistic model, which contains a global latent variable on top of object-level latent variables.
Revised: We can model both spatial information and object representations by introducing hierarchical probabilistic model, which contains a global latent variable on top of object-level latent variables.

Original: However, how to execute inference and training with that hierarchical multi-view object-centric model is unclear.
Revised: However, how to execute inference and training with that hierarchical multi-view object-centric model is unclear.

Original: Therefore, we introduce several crucial components which help inference and training with the proposed model.
Revised: Therefore, we introduce several crucial components which help inference and training with the proposed model.

Original: We show that the proposed method achieves good inference quality and can also generate novel scenes.
Revised: We show that the proposed method achieves good inference quality and can also generate novel scenes.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Extracting representations of individual objects from direct perception (e.g., visual input) is called object-centric representation learning.
Revised: Extracting representations of individual objects from direct perception (e.g., visual input) is called object-centric representation learning.

Original: It is considered to be effective in many aspects such as robotics (Devin et al., 2018;
Revised: It is considered to be effective in many aspects such as robotics (Devin et al., 2018;

Original: Veerapaneni et al., 2019; Kulkarni et al., 2019), reasoning (Ding et al., 2021), and sample efﬁciency (Watters et al., 2019a).
Revised: Veerapaneni et al., 2019; Kulkarni et al., 2019), reasoning (Ding et al., 2021), and sample efﬁciency (Watters et al., 2019a).

Original: Among them, the deep generative model (DGM)-based method is one of the promising directions, as it can infer rich information about objects in a fully unsupervised manner, and some of its models can also generate novel scenes (Greff et al., 2017;
Revised: Among them, the deep generative model (DGM)-based method is one of the promising directions, as it can infer rich information about objects in a fully unsupervised manner, and some of its models can also generate novel scenes (Greff et al., 2017;

Original: Locatello et al., 2020; Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2020;
Revised: Locatello et al., 2020; Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2020;

Original: Eslami et al., 2016; Crawford & Pineau, 2019; Jiang & Ahn, 2020).
Revised: Eslami et al., 2016; Crawford & Pineau, 2019; Jiang & Ahn, 2020).

Original: Recently, several studies have been working on multi-view object-centric models, which predict unobserved views of a scene: novel view synthesis, and infer object-centric representations from several observation views (Nanbo et al., 2020; Chen et al., 2021).
Revised: Recently, several studies have been working on multi-view object-centric models, which predict unobserved views of a scene: novel view synthesis, and infer object-centric representations from several observation views (Nanbo et al., 2020; Chen et al., 2021).

Original: This can be also regarded as an object-centric expansion of the Generative Query Network (GQN) (Eslami et al., 2018).
Revised: This can be also regarded as an object-centric expansion of the Generative Query Network (GQN) (Eslami et al., 2018).

Original: In order to represent multi-object scenes, the properties of individual objects and the spatial arrangement of objects should be speciﬁed.
Revised: In order to represent multi-object scenes, the properties of individual objects and the spatial arrangement of objects should be speciﬁed.

Original: However, existing multi-view object-centric methods only explicitly model representations of individual objects, and each of them includes their own spatial information separately.
Revised: However, existing multi-view object-centric methods only explicitly model representations of individual objects, and each of them includes their own spatial information separately.

Original: Namely, relationship between objects is not represented in this case.
Revised: Namely, relationship between objects is not represented in this case.

Original: This modeling can degrade novel view synthesis and segmentation, because the model need to solve occlusions and spatial ambiguity virtually without the prior knowledge about objects’ spatial relationship.
Revised: This modeling can degrade novel view synthesis and segmentation, because the model need to solve occlusions and spatial ambiguity virtually without the prior knowledge about objects’ spatial relationship.

Original: This can be serious especially when the number of available observation views is limited.
Revised: This can be serious especially when the number of available observation views is limited.

Original: In addition, this modeling can also lead to inability to generate physically plausible novel scenes, because they need to place objects independently, which often brings about collisions and misplacement of objects.
Revised: In addition, this modeling can also lead to inability to generate physically plausible novel scenes, because they need to place objects independently, which often brings about collisions and misplacement of objects.

Original: Spatial information of the whole scene can be represented by introducing another latent variable besides object-level latent variables.
Revised: Spatial information of the whole scene can be represented by introducing another latent variable besides object-level latent variables.

Original: This variable can be independent of object-level latent variables, or can be modeled as a global latent on top of object-level latent variables.
Revised: This variable can be independent of object-level latent variables, or can be modeled as a global latent on top of object-level latent variables.

Original: In the former formulation, those latent variables need to be disentangled during training, which can often be challenging.
Revised: In the former formulation, those latent variables need to be disentangled during training, which can often be challenging.

Original: From another point of view, it is common assumption in cognitive science that humans recognize the whole and parts, or objects, separately (Schmidt, 2009).
Revised: From another point of view, it is common assumption in cognitive science that humans recognize the whole and parts, or objects, separately (Schmidt, 2009).

Original: In feature integration theory (Treisman & Gelade, 1980), visual feature maps become “master map of locations” combined with positions of 𝑥 !
Revised: In feature integration theory (Treisman & Gelade, 1980), visual feature maps become “master map of locations” combined with positions of 𝑥 !

Original: 𝑥 " 𝑥 #
Revised: 𝑥 " 𝑥 #

Original: P r o d u c t
Revised: P r o d u c t

Original: o f E x p e r t s ( P o
Revised: o f E x p e r t s ( P o

Original: E )
Revised: E )

Original: 𝑧 $
Revised: 𝑧 $

Original: Sequential Encoder 𝑧 !
Revised: Sequential Encoder 𝑧 !

Original: 𝑧 "% 𝑧 &% …
Revised: 𝑧 "% 𝑧 &% …

Original: Spatial Gaussian Mixture 𝑣 ’ query viewpoint 𝑥$ ’ 𝑧 !
Revised: Spatial Gaussian Mixture 𝑣 ’ query viewpoint 𝑥$ ’ 𝑧 !

Original: ( 𝑧 "( 𝑧 &( …
Revised: ( 𝑧 "( 𝑧 &( …

Original: Iterative Amortized Inference 𝑞 !
Revised: Iterative Amortized Inference 𝑞 !

Original: ( 𝑧 " ∣ 𝑥 )
Revised: ( 𝑧 " ∣ 𝑥 )

Original: S t r u c t u r e d
Revised: S t r u c t u r e d

Original: P r i o r s ha r ed s ha r ed
Revised: P r i o r s ha r ed s ha r ed

Original: E n c ode r
Revised: E n c ode r

Original: E n c ode r 𝐾 slots
Revised: E n c ode r 𝐾 slots

Original: [Figure]
Revised: [Figure]

Original: objects in visual space.
Revised: objects in visual space.

Original: Then, attention binds those features to recognize individual objects.
Revised: Then, attention binds those features to recognize individual objects.

Original: Modeling the spatial arrangement of objects as a global latent variable closely adheres to this point of view due to its hierarchical property.
Revised: Modeling the spatial arrangement of objects as a global latent variable closely adheres to this point of view due to its hierarchical property.

Original: For the above reasons, we believe that introducing a global latent variable is more natural step for improving object-centric generative models.
Revised: For the above reasons, we believe that introducing a global latent variable is more natural step for improving object-centric generative models.

Original: In this paper, we build a new multi-view object-centric model which has a latent variable for global spatial information to improve inference: novel view synthesis and segmentation.
Revised: In this paper, we build a new multi-view object-centric model which has a latent variable for global spatial information to improve inference: novel view synthesis and segmentation.

Original: In addition, our model can also generate physically plausible novel scenes.
Revised: In addition, our model can also generate physically plausible novel scenes.

Original: The problem is that how to execute inference and training with the proposed model is unclear.
Revised: The problem is that how to execute inference and training with the proposed model is unclear.

Original: In regards to this point, we propose several crucial components: Global Encoder, Sequential Encoder and Structured Prior.
Revised: In regards to this point, we propose several crucial components: Global Encoder, Sequential Encoder and Structured Prior.

Original: Firstly, the model needs to infer global representation from variable number of input views, thus we introduced an encoder using Product of Experts (PoE)
Revised: Firstly, the model needs to infer global representation from variable number of input views, thus we introduced an encoder using Product of Experts (PoE)

Original: (Hinton, 2002), which we refer to as Global Encoder.
Revised: (Hinton, 2002), which we refer to as Global Encoder.

Original: Secondly, the structured, complex posterior of object-level latent variables need to be sampled including spatial relationship between objects in order to generated physically plausible novel scenes.
Revised: Secondly, the structured, complex posterior of object-level latent variables need to be sampled including spatial relationship between objects in order to generated physically plausible novel scenes.

Original: Therefore, we introduced a learnable prior called Structured Prior, which is implemented by Transformer (Vaswani et al., 2017).
Revised: Therefore, we introduced a learnable prior called Structured Prior, which is implemented by Transformer (Vaswani et al., 2017).

Original: Lastly, we introduced an encoder using auto-regressive model called Sequential Encoder.
Revised: Lastly, we introduced an encoder using auto-regressive model called Sequential Encoder.

Original: This helps stable inference and training with the hierarchical probabilistic model of the proposed method.
Revised: This stabilize inference and training with the hierarchical probabilistic model of the proposed method.

Original: The proposed model has representation about the whole, or spatial information of a scene and has representations about a part of a scene (i.e., individual properties of objects); hence, we call this model the Whole-Part Representation Learning Model for Object-Centric 3D Scene Inference and
Revised: The proposed model has representation about the whole, or spatial information of a scene and has representations about a part of a scene (i.e., individual properties of objects); hence, we call this model the Whole-Part Representation Learning Model for Object-Centric 3D Scene Inference and

Original: Sampling ( WeLIS ).
Revised: Sampling ( WeLIS ).

Original: Contributions:
Revised: Contributions:

Original: • We introduce a multi-view object-centric model with a global latent variable which represents the spatial conﬁguration of objects in a scene.
Revised: • We introduce a multi-view object-centric model with a global latent variable which represents the spatial conﬁguration of objects in a scene.

Original: • We introduce essential components for introducing a global latent variable as well as stable learning and inference for our multi-view object-centric model.
Revised: • We introduce essential components for introducing a global latent variable as well as stable learning and inference for our multi-view object-centric model.

Original: • We show that the proposed model performs well in terms of inference quality, and can generate novel scenes and corresponding segmentation masks properly, unlike existing methods.
Revised: • We show that the proposed model performs well in terms of inference quality, and can generate novel scenes and corresponding segmentation masks properly, unlike existing methods.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Object-Centric Models :
Revised: Object-Centric Models :

Original: The objective of object-centric representation learning is to obtain representations of each object from visual input.
Revised: The objective of object-centric representation learning is to obtain representations of each object from visual input.

Original: It is applicable to many domains such as robotics (Devin et al., 2018; Florence et al., 2019;
Revised: It is applicable to many domains such as robotics (Devin et al., 2018; Florence et al., 2019;

Original: Veerapaneni et al., 2019), reasoning (Ding et al., 2021) and reinforcement learning (Watters et al., 2019a).
Revised: Veerapaneni et al., 2019), reasoning (Ding et al., 2021) and reinforcement learning (Watters et al., 2019a).

Original: One of the promising directions of object-centric representation learning is variational autoencoder (VAE)-based (Kingma & Welling, 2013) methods, as they can obtain rich information about objects that can also be disentangled
Revised: One of the promising directions of object-centric representation learning is variational autoencoder (VAE)-based (Kingma & Welling, 2013) methods, as they can obtain rich information about objects that can also be disentangled

Original: [Table]
Revised: [Table]

Original: and some of them can generate novel scenes outside of the training distribution (Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2020; Eslami et al., 2016; Crawford & Pineau, 2019; Jiang & Ahn, 2020)
Revised: and some of them can generate novel scenes outside of the empirical distribution (Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2020; Eslami et al., 2016; Crawford & Pineau, 2019; Jiang & Ahn, 2020)

Original: [Table]
Revised: [Table]

Original: VAE-based object-centric models are often referred to as “scene interpretation models” due to their inference ability.
Revised: VAE-based object-centric models are often referred to as “scene interpretation models” due to their inference ability.

Revised: Some models are capable of object-centric generation (Dai et al., 2019; NguyenPhuoc et al., 2020; Niemeyer & Geiger, 2021), however, these methods focus on generation and not for inference.

Original: Multi-View Object-Centric Models : Lately, several studies have been working on multi-view scene interpretation models (Chen, 2012; Nanbo et al., 2020).
Revised: Multi-View Object-Centric Models : Lately, several studies have been working on multi-view scene interpretation models (Nanbo et al., 2020; Chen et al., 2021).

Original: These methods predict images of a 3D scene from arbitrary viewpoints in an object-centric way, using several observation views as input.
Revised: These methods predict images of a 3D scene from arbitrary viewpoints in an object-centric way, using several observation views as input.

Original: This problem setting can be regarded as an object-centric expansion of GQN family (Eslami et al., 2018;
Revised: This problem setting can be regarded as an object-centric expansion of GQN family (Eslami et al., 2018;

Original: Tobin et al., 2019; Kosiorek et al., 2021).
Revised: Tobin et al., 2019; Kosiorek et al., 2021).

Original: The above mentioned single-view setting is referred to as multi-object-single-view (MOSV), and the multi-view setting is deﬁned as multiobject-multi-view (MOMV) in (Nanbo et al., 2020).
Revised: The above mentioned single-view setting is referred to as multi-object-single-view (MOSV), and the multi-view setting is deﬁned as multiobject-multi-view (MOMV) in (Nanbo et al., 2020).

Original: Table 1 compares methods related to the MOMV setting.
Revised: Table 1 compares methods related to the MOMV setting.

Original: Novel view synthesis stands for prediction of unobserved viewpoints, and novel scene generation stands for random generation of novel scenes.
Revised: Novel view synthesis stands for prediction of unobserved viewpoints, and novel scene generation stands for random generation of novel scenes.

Original: Among these models, ROOTS and MulMON (Chen, 2012; Nanbo et al., 2020) are the only methods that work on apure MOMV setting.uORF(Yu
Revised: Among these models, ROOTS and MulMON (Nanbo et al., 2020; Chen et al., 2021) are the only generative models that work on a MOMV setting.

Original: et al., 2021) and ObSuRF (Stelzner et al., 2021)can also perform object-centric inference and novel view synthesis, but they are single-view settings, and ObSuRF requires a dense depth map for training.
Revised: In addition to this, O3V (Henderson & Lampert, 2020) have a similar problem setting, but it takes ﬁxed number of viewpoints as input. ObSuRF (Stelzner et al., 2021) and uORF(Yu

Original: Using only single images is useful when observation is limited, but it makes dealing with occlusion and spatial ambiguity difﬁcult. Moreover, observing a scene from multiple viewpoints is not difﬁcult in many application scenarios such as robot arm manipulation and RL agents, as they can use multiple cameras or walk around to get multiple views.
Revised: et al., 2021) also work on MOMV setting, but these are not generative models, and are based on Slot Attention (Locatello et al., 2020) and NeRF (Mildenhall et al., 2020). Therefore these methods can not generate novel samples and does not obtain low dimensional representation space, or latent space.

Original: In this paper, we introduce a new MOMV model with a global latent variable, which mainly represents the spatial arrangement of objects in a scene.
Revised: In this paper, we introduce a new MOMV model with a global latent variable, which can potentially represents the spatial arrangement of objects in a scene.

Original: The proposed model is the only MOMV model that can generate novel scenes.
Revised: The proposed model is the only MOMV model that can generate novel scenes.

Original: In this paper, we set MulMON as our baseline model due to the following reasons.
Revised: In this paper, we set MulMON as our baseline model due to the following reasons.

Original: Firstly, MulMON uses segmentation masks to represent objects’ regions unlike ROOTS which uses bounding boxes.
Revised: Firstly, MulMON uses segmentation masks to represent objects’ regions unlike ROOTS which uses bounding boxes.

Original: Bounding box-based methods, which originate from (Eslami et al., 2016) tend to be more sensitive to object size and shape (Engelcke et al., 2021).
Revised: Bounding box-based methods, which originate from (Eslami et al., 2016) tend to be more sensitive to object size and shape (Engelcke et al., 2021).

Original: Secondly, MulMON is more robust in terms of the number of observation views required.
Revised: Secondly, MulMON is more robust in terms of the number of observation views required.

Original: Although we built our model based on MulMON, a similar approach can be applied to ROOTS and other methods.
Revised: Although we built our model based on MulMON, a similar approach can be applied to ROOTS and other methods.

Original: Introducing the global latent variable itself has been done in some studies in various areas (Vasco et al., 2020; Akuzawa et al., 2021).
Revised: Introducing the global latent variable itself has been done in some studies in various areas (Vasco et al., 2020; Akuzawa et al., 2021).

Original: In particular, GNM (Jiang & Ahn, 2020) introduces a global latent variable to bounding box-based single-view object-centric model for random generation.
Revised: In particular, GNM (Jiang & Ahn, 2020) introduces a global latent variable to bounding box-based single-view object-centric model for random generation.

Original: However, how to execute stable inference and training with the proposed MOMV model is unclear, and is thus the points of investigation in this paper.
Revised: However, how to execute stable inference and training with the proposed MOMV model is unclear, and is thus the points of investigation in this paper.

Original:  M ETHOD
Revised:  M ETHOD

Original: Our objective was to introduce a MOMV model that has both object-level representations and the spatial arrangement of the scene to obtain further inference quality and novel scene generation ability.
Revised: Our objective was to introduce a MOMV model that has both object-level representations and the spatial arrangement of the scene to obtain further inference quality and novel scene generation ability.

Original: We achieved this by introducing a global latent variable for global spatial information on top of object latent variables, which is formulated as hierarchical probabilistic model.
Revised: We achieved this by introducing a global latent variable for global spatial information on top of object latent variables, which is formulated as hierarchical probabilistic model.

Original: In this paper, we introduce several crucial components to realize stable inference and training with this MOMV model.
Revised: In this paper, we introduce several crucial components to realize stable inference and training with this MOMV model.

Original: An overview of the proposed model is shown in Figure 1.
Revised: An overview of the proposed model is shown in Figure 1.

Original: In this section, we ﬁrst explain the problem setting in this paper, and then the probabilistic model of our proposed model.
Revised: In this section, we ﬁrst explain the problem setting in this paper, and then the probabilistic model of our proposed model.

Original: Lastly, the training method and additional details of the introduced components are explained.
Revised: Lastly, the training method and additional details of the introduced components are explained.

Original: 3. P ROBLEM S ETTING
Revised: 3. P ROBLEM S ETTING

Original: The target of this research is 3D scenes with multiple objects, and thus a dataset with N data points can be deﬁned as D
Revised: The target of this research is 3D scenes with multiple objects, and thus a dataset with N data points can be deﬁned as D

Original: = { X 1 , · · · , X N } ,
Revised: = { X 1 , · · · , X N } ,

Original: where X i = { ( x 1 i , v 1 i ) , ..., ( x Mi , v Mi ) } , which corresponds to one scene with M views, and v is a viewpoint vector with arbitrary dimensions, which are mostly three.
Revised: where X i = { ( x 1 i , v 1 i ) , ..., ( x Mi , v Mi ) } , which corresponds to one scene with M views, and v is a viewpoint vector with arbitrary dimensions, which are mostly three.

Original: The task is to predict queried unobserved views of the scene given several observations { ( x 1 i , v 1 i ) , · · · , ( x N obs i , v N obs i ) } , and infer K representations each of which correspond to an object in the scene Z
Revised: The task is to predict queried unobserved views of the scene given several observations { ( x 1 i , v 1 i ) , · · · , ( x N obs i , v N obs i ) } , and infer K representations each of which correspond to an object in the scene Z

Original: = { z 1 , · · · , z K } .
Revised: = { z 1 , · · · , z K } .

Original: In practice, the M views in X i is split into N obs observations and N qry queries in the training process.
Revised: In practice, the M views in X i is split into N obs observations and N qry queries in the training process.

Original: N obs can be varied in every iteration for generalization.
Revised: N obs can be varied in every iteration for generalization.

Original: 3. P ROBABILISTIC M ODEL
Revised: 3. P ROBABILISTIC M ODEL

Original: Generative Model : As mentioned above, we introduced the hierarchical probabilistic model with global latent z g on top of object-level latent variables z
Revised: Generative Model : As mentioned above, we introduced the hierarchical probabilistic model with global latent z g on top of object-level latent variables z

Original: = { z 1 , · · · , z K } .
Revised: = { z 1 , · · · , z K } .

Original: In the following sections, object-level latent variables are abbreviated as z , as required.
Revised: In the following sections, object-level latent variables are abbreviated as z , as required.

Original: The marginal likelihood is thus:
Revised: The marginal likelihood is thus:

Original: [Equation]
Revised: [Equation]

Original: where γ = { θ, φ } .
Revised: where γ = { θ, φ } .

Original: The generative model p θ ( x | z , v ) is a spatial Gaussian mixture model derived from prior works (Nanbo et al., 2020; Burgess et al., 2019) that can be written as:
Revised: The generative model p θ ( x | z , v ) is a spatial Gaussian mixture model derived from prior works (Nanbo et al., 2020; Burgess et al., 2019) that can be written as:

Original: [Equation]
Revised: [Equation]

Original: where D ∈ N is the number of image pixels, and K is the number of mixture components which are often referred to as “slots”.
Revised: where D ∈ N is the number of image pixels, and K is the number of mixture components which are often referred to as “slots”.

Original: The decoder has parameters θ and implemented by a broadcast decoder (Watters et al., 2019b).
Revised: The decoder has parameters θ and implemented by a broadcast decoder (Watters et al., 2019b).

Original: z k and v are ﬁrst concatenated and projected onto a single vector f k by MLP and then fed into the decoder.
Revised: z k and v are ﬁrst concatenated and projected onto a single vector f k by MLP and then fed into the decoder.

Original: The decoder generates K components { x 1 , · · · , x K } and corresponding segmentation masks from { f 1 , · · · , f K } .
Revised: The decoder generates K components { x 1 , · · · , x K } and corresponding segmentation masks from { f 1 , · · · , f K } .

Original: The second term in the r.h.s of Equation 1 p φ ( z | z g ) is a learnable prior which we refer to as the Structured Prior, and the third term is a standard Gaussian distribution.
Revised: The second term in the r.h.s of Equation 1 p φ ( z | z g ) is a learnable prior which we refer to as the Structured Prior, and the third term is a standard Gaussian distribution.

Original: The Structured Prior is explained in Section 3.3.
Revised: The Structured Prior is explained in Section 3.3.

Original: Inference Model : We used amortized variational inference (Kingma & Welling, 2013) and computed variational approximate posteriors because the marginal likelihood in Equation 1 is intractable.
Revised: Inference Model : We used amortized variational inference (Kingma & Welling, 2013) and computed variational approximate posteriors because the marginal likelihood in Equation 1 is intractable.

Original: We factorized the variational posteriors as:
Revised: We factorized the variational posteriors as:

Original: [Equation]
Revised: [Equation]

Original: where X = { ( x 1 , v 1 ) , . . .
Revised: where X = { ( x 1 , v 1 ) , . . .

Original: , ( x N obs , v N obs ) } .
Revised: , ( x N obs , v N obs ) } .

Original: The ﬁrst RHS term and second RHS term are approximate posteriors of z and z g respectively.
Revised: The ﬁrst RHS term and second RHS term are approximate posteriors of z and z g respectively.

Original: We refer to these encoders as Sequential Encoder and
Revised: We refer to these encoders as Sequential Encoder and

Original: Global Encoder.
Revised: Global Encoder.

Original: The Global Encoder adopts PoE (Hinton, 2002) to deal with a variable number of input observation views N obs .
Revised: The Global Encoder adopts PoE (Hinton, 2002) to deal with a variable number of input observation views N obs .

Original: In particular, we used PoE with prior expert (Wu & Goodman, 2018).
Revised: In particular, we used PoE with prior expert (Wu & Goodman, 2018).

Original: In addition to this, we applied normalizing ﬂow (NF) (Rezende & Mohamed, 2015) to this inference model.
Revised: In addition to this, we applied normalizing ﬂow (NF) (Rezende & Mohamed, 2015) to this inference model.

Original: Although this is optional, the posterior should be expressive in order to represent spatial conﬁguration of complex 3D multi-object scenes.
Revised: Although this is optional, the posterior should be expressive in order to represent spatial conﬁguration of complex 3D multi-object scenes.

Original: Therefore, the variational approximate posterior is:
Revised: Therefore, the variational approximate posterior is:

Original: [Equation]
Revised: [Equation]

Original: where T is the number of transformations of NF, and N obs is the number of available observation views.
Revised: where T is the number of transformations of NF, and N obs is the number of available observation views.

Original: We used planar ﬂow for its simplicity, but any methods can be applied and can improve the results potentially.
Revised: We used planar ﬂow for its simplicity, but any methods can be applied and can improve the results potentially.

Original: Thus, the transform f ψ in the above equation is
Revised: Thus, the transform f ψ in the above equation is

Original: [Equation]
Revised: [Equation]

Original: The Sequential Encoder was implemented in auto-regressive manner using LSTM (Hochreiter &
Revised: The Sequential Encoder was implemented in auto-regressive manner using LSTM (Hochreiter &

Original: Schmidhuber, 1997).
Revised: Schmidhuber, 1997).

Original: We tried several architectures, but non-auto-regressive models were unstable as long as we have tested.
Revised: We tried several architectures, but non-auto-regressive models were unstable as long as we have tested.

Original: Firstly, this auto-regressive part infers K latent variables z 01 , . . .
Revised: Firstly, this auto-regressive part infers K latent variables z 01 , . . .

Original: , z 0 K
Revised: , z 0 K

Original: and then, these are updated using iterative amortized inference (IAI) (Marino et al., 2018) as MulMON and IODINE (Greff et al., 2019) do.
Revised: and then, these are updated using iterative amortized inference (IAI) (Marino et al., 2018) as MulMON and IODINE (Greff et al., 2019) do.

Original: However, we omitted image-sized inputs as in (Emami et al., 2021).
Revised: However, we omitted image-sized inputs as in (Emami et al., 2021).

Original: Thus, the Sequential Encoder can be deﬁned as:
Revised: Thus, the Sequential Encoder can be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: ) where ∇ z ik L iIAI is a gradient of negative log-likelihood in that iteration, and f ψ is a reﬁnement network that is implemented by MLP followed by LSTM.
Revised: ) where ∇ z ik L iIAI is a gradient of negative log-likelihood in that iteration, and f ψ is a reﬁnement network that is implemented by MLP followed by LSTM.

Original: The posterior can be updated arbitrary times, and we set L = 5 through this paper.
Revised: The posterior can be updated arbitrary times, and we set L = 5 through this paper.

Original: The details of L iIAI will be explained in the next section.
Revised: The details of L iIAI will be explained in the next section.

Original: Training : All the parameters in this model can be trained end-to-end by maximizing evidence lower bound (ELBO) on the log-marginal likelihood log p ( x 1 , · · · , x N ) .
Revised: Training : All the parameters in this model can be trained end-to-end by maximizing evidence lower bound (ELBO) on the log-marginal likelihood log p ( x 1 , · · · , x N ) .

Original: The ELBO of the proposed model is
Revised: The ELBO of the proposed model is

Original: [Equation]
Revised: [Equation]

Original: Here, KL stands for Kullback–Leibler divergence.
Revised: Here, KL stands for Kullback–Leibler divergence.

Original: Then, introducing two coefﬁcient β 1 and β 2 for balancing KL terms, the objective can be written as
Revised: Then, introducing two coefﬁcient β 1 and β 2 for balancing KL terms, the objective can be written as

Original: [Equation]
Revised: [Equation]

Original: Moreover, due to the iterative process of IAI, negative log-likelihood is computed as
Revised: Moreover, due to the iterative process of IAI, negative log-likelihood is computed as

Original: [Equation]
Revised: [Equation]

Original: where, L is the number of iterations in IAI.
Revised: where, L is the number of iterations in IAI.

Original: As shown in Equation 9, log-likelihoods in the later steps of iterative reﬁnement are emphasized as in MulMON and IODINE.
Revised: As shown in Equation 9, log-likelihoods in the later steps of iterative reﬁnement are emphasized as in MulMON and IODINE.

Original: In addition to this, we used GECO (Rezende & Viola, 2018) as in some existing methods (Engelcke et al., 2020; Emami et al., 2021).
Revised: In addition to this, we used GECO (Rezende & Viola, 2018) as in some existing methods (Engelcke et al., 2020; Emami et al., 2021).

Original: GECO reinterprets the maximization of ELBO as a constrained
Revised: GECO reinterprets the maximization of ELBO as a constrained

Original: R M SE m I

Original: [Figure]
Revised: [Figure]

Original: CLEVR MV

Original: CLEVR AUG
Revised: CLEVR AUG

Original: GQN Jaco
Revised: GQN Jaco

Original: OursOurs: w/o Flow MulMON (bassline)
Revised: OursOurs: w/o Flow MulMON (bassline)

Revised: [Figure]

Original: 𝑁
Revised: 𝑁

Original: [Figure]
Revised: [Figure]

Original: 𝑁 : the number ofobservations 0.35 0.
Revised: the number of

Original: [Figure]
Revised: [Figure]

Revised: 𝑁

Original: [Figure]
Revised: [Figure]

Original: 𝑁 0.03 0.
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: optimization problem that takes KL terms as the main objective and log-likelihood as the constraint.
Revised: optimization problem that takes KL terms as the main objective and log-likelihood as the constraint.

Original: This optimization is done by the method of Lagrange multipliers, namely, the coefﬁcient of KL terms β is regarded as a Lagrange multiplier, and automatically tweaked during training.
Revised: This optimization is done by the method of Lagrange multipliers, namely, the coefﬁcient of KL terms β is regarded as a Lagrange multiplier, and automatically tweaked during training.

Original: GECO is not necessary for our model, but proper balancing of β further improves random generation quality.
Revised: GECO is not necessary for our model, but proper balancing of β further improves random generation quality.

Original: The reinterpreted optimization objective thus becomes
Revised: The reinterpreted optimization objective thus becomes

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: where R ∈ R is the maximum allowed negative log-likelihood, which is a hyperparameter in GECO instead of β .
Revised: where R ∈ R is the maximum allowed negative log-likelihood, which is a hyperparameter in GECO instead of β .

Original: Although all the parameters can be trained end-to-end, we found that end-to-end training including the parameters of Structured Prior φ often leads to overﬁtting and  degrades novel scene generation quality.
Revised: Although all the parameters can be trained end-to-end, we found that end-to-end training including the parameters of Structured Prior φ is slightly unstable and often degrades novel scene generation.

Original: Thus, we introduced a separate training strategy,  which  is explained in the next section.
Revised: Thus, we introduced a separate training strategy, in which φ is updated after the training of the main model. Detail of this method is explained in the next section.

Original: 3. S TRUCTURED P RIOR
Revised: 3. S TRUCTURED P RIOR

Original: In this section, we explain the detail of the Structured Prior and its separate training strategy.
Revised: In this section, we explain the detail of the Structured Prior and its separate training strategy.

Original: First of all, the Structured Prior is deﬁned as p φ ( z L 1 , · · · , z LK | z g ) .
Revised: First of all, the Structured Prior is deﬁned as p φ ( z L 1 , · · · , z LK | z g ) .

Original: The latent variables, or slots  updated by IAI 
Revised: As we mentioned in Equation 5, the latent variables of each slots are iteratively updated by IAI in the inference process.

Original:  z L 1 , · · · , z LK are not independent, because spatial correlation always exists between objects.
Revised: However, it makes sampling from the updated posterior z L 1 , · · · , z LK difﬁcult.

Original: For example, objects cannot possess the same spatial volume, and they need to be grounded on the ﬂoor. Therefore, sampling each latent variable’s z Lk independently is not sufﬁcient, and generation in that way lacks physical plausibility.
Revised: Though IAI updates latent variables using reconstruction error and auxiliary inputs, generative model needs to blindly capture this process without using them.

Original: To avoid this, we modeled this dependency between slots using Transformer (Vaswani et al., 2017).
Revised: Therefore, in order to model this process, we implemented Structured Prior as Transformer (Vaswani et al., 2017).

Original: Firstly, initial K variables z 

Revised: Firstly, initial K variables z

Original: k are computed from z g using the Sequential Encoder q ψ ( z 0 | z g ) .
Revised: k are computed from z g using the Sequential Encoder q ψ ( z 0 | z g ) .

Original: Then, Transformer takes the K latent variables as input, and approximates updated ones z Lk .
Revised: Then, Transformer takes the K latent variables as input, and approximates updated ones z Lk .

Original: Note that the parameter of Sequential Encoder is shared with inference.
Revised: Note that the parameter of Sequential Encoder is shared with inference.

Original: Separate Training : As we mentioned above, we introduced a separate training method for theparameters of Structured Prior φ .
Revised: Separate Training : As we mentioned above, we introduced a separate training method for the Structured Prior.

Original: In a ﬁrst stage, WeLIS is trained without the parameter of Structured Prior φ .
Revised: In a ﬁrst stage, WeLIS is trained without the parameters of Structured Prior φ .

Original: Namely, φ is frozen and β 2 in Equation 7 is set to zero or the second term of Equation 11 is excluded if we use GECO.

Original: Then, as a second stage, φ is updated while other parameters are frozen.

Original: Gen.
Revised: Gen.

Original: Slots 𝑘 =𝑘 = 7 𝑘 =𝑘 =𝑘 =𝑘 =𝑘 =
Revised: Slots 𝑘 =𝑘 = 7 𝑘 =𝑘 =𝑘 =𝑘 =𝑘 =

Original: Seg.
Revised: Seg.

Original: W eL
Revised: W eL

Original: I
Revised: I

Original: S ( O u r s )
Revised: S ( O u r s )

Original: M u l M
Revised: M u l M

Original: O N ( B a s e li ne ) viewviewviewviewviewview
Revised: O N ( B a s e li ne ) viewviewviewviewviewview

Original: [Table] [Table]
Revised: Namely, φ is frozen and β 2 in Equation 7 is set to zero or the second term of Equation 11 is excluded if we use GECO. Then, as a second stage, φ is updated while other parameters are frozen.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluated our model with three different datasets derived from the MulMON paper. CLEVR
Revised: We evaluated our model with three different datasets derived from the MulMON paper (Nanbo et al., 2020).

Original:  Multi-View (CLEVR MV) is a multi-view version of the CLEVR dataset (Johnson et al., 2017) [Table] and CLEVR Augmented (CLEVR AUG) extends CLEVR MV with more complex and various objects for additional difﬁculty.
Revised: CLEVR Multi-View (CLEVR MV) is a multi-view version of the CLEVR dataset (Johnson et al., 2017), and CLEVR Augmented (CLEVR AUG) extends CLEVR MV with more complex and various objects for additional difﬁculty.

Original: GQN Jaco is a simulation based robot arm dataset introduced originally in (Eslami et al., 2018).
Revised: GQN Jaco is a simulation based robot arm dataset introduced originally in (Eslami et al., 2018).

Original: All the datasets are used in 64 × 64 resolution [Table] and we trained all the models with Adam optimizer for 300 k iterations.
Revised: All the datasets are used in 64 × 64 resolution 1 , and we trained all the models with Adam optimizer for 300 k iterations.

Original: We used ofﬁcial implementation of MulMON for reproduction.
Revised: We used ofﬁcial implementation of MulMON for reproduction.

Original: Up to six views are given as observation during training.
Revised: Up to six views are given as observation during training.

Original: First, we evaluated inference quality quantitatively and show that WeLIS outperforms MulMON in many cases.
Revised: First, we evaluated inference quality quantitatively and show that WeLIS outperforms MulMON in many cases.

Original: Then, we show WeLIS can also generate novel scenes, in contrast to baseline models.
Revised: Then, we show WeLIS can also generate novel scenes, in contrast to baseline models.

Original: Lastly, we show ablation results and also investigate how global latent variable z g works.
Revised: Lastly, we show ablation results and also investigate how global latent variable z g works.

Original: In addition, additional samples and the results of downstream task is shown in Appendix B.
Revised: In addition, additional ablation studies, samples, and the results of downstream task are shown in Appendix B.

Original: 4. N OVEL V IEW S YNTHESIS AND S EGMENTATION
Revised: 4. N OVEL V IEW S YNTHESIS AND S EGMENTATION

Original: In this section, we evaluate novel view synthesis and segmentation quality.
Revised: In this section, we evaluate novel view synthesis and segmentation quality.

Original: Figure 2 shows quantitative evaluation of three models: WeLIS, WeLIS without NF (ablation) and MulMON.
Revised: Figure 2 shows quantitative evaluation of three models: WeLIS, WeLIS without NF (ablation) and MulMON.

Original: Novel[Figure] view synthesis is evaluated by Root Mean Squared Error (RMSE), and  is evaluated by mean Intersection (mIoU).
Revised: Novel view synthesis is evaluated by Root Mean Squared Error (RMSE), and segmentation is evaluated by mean Intersection over Union(mIoU).

Original: Random four seeds are used to calculate these scores.
Revised: Random four seeds are used to calculate these scores.

Original: As
Revised: As

Original: GQN Jaco does not provide ground truth segmentation, we instead show novel view synthesis results in Figure 3.
Revised: GQN Jaco does not provide ground truth segmentation, we instead show novel view synthesis results in Figure 3.

Original: We can see that WeLIS outperforms MulMON in CLEVR AUG, GQN Jaco and CLEVR MV with a small number of observation views N .
Revised: We can see that WeLIS outperforms MulMON in CLEVR AUG, GQN Jaco and CLEVR MV with a small number of observation views N .

Original: WeLIS is relatively stable in terms of N , which means that WeLIS is good at inferring whole scene from limited, partial observation. 
Revised: WeLIS is relatively stable in terms of N , which means that WeLIS is good at inferring whole scene from limited, partial observation. Introducing global [Figure]

Original: Introducing global latent in addition to object-level latent variables leads to more precise modeling of the world, thus we consider that it helps inference with small number of observations.
Revised: [Table] [Table] [Table] latent in addition to object-level latent variables leads to more precise modeling of the world, thus we consider that it helps inference with small number of observations.

Original: As for GQN Jaco (Figure 3),
Revised: As for GQN Jaco (Figure 3),

Original: WeLIS captures the balls consistently with clean segmentation.
Revised: WeLIS captures the balls consistently with clean segmentation.

Original: We also evaluated WeLIS without NF as ablation study.
Revised: We also evaluated WeLIS without NF as ablation study.

Original: We found that NF is important for stable inference and training.
Revised: We found that NF is important for stable inference and training.

Original: Without it, the model often fall into local minima that does not properly represent one object per slot, as you can see from the relatively large standard error in Figure 2.
Revised: Without it, the model often fall into local minima that does not properly represent one object per slot, as you can see from the relatively large standard error in Figure 2.

Original: If the training successfully ﬁnished, the model achieved decent performance close to full model.
Revised: If the training successfully ﬁnished, the model achieved decent performance close to full model.

Original: 4. N OVEL S CENE G ENERATION
Revised: 4. N OVEL S CENE G ENERATION

Original: In this section, we evaluate novel scene generation results.
Revised: In this section [Table] we evaluate novel scene generation results.

Original: As shown in Figure 4, WeLIS successfully generates objects and segmentation masks, and as a result, generates physically plausible novel scenes.
Revised: As shown in Figure 4, WeLIS successfully generates objects and segmentation masks [Table] and as a result, generates physically plausible novel scenes.

Original: On the other hand, MulMON generates blurred and spatially ambiguous images.
Revised: On the other hand, MulMON generates blurred and spatially ambiguous images.

Original: Top row of Figure 5 shows more samples.
Revised: Top row of Figure 5 shows more samples.

Original: 4.2. A BLATION S TUDIES
Revised: 4.2. A BLATION S TUDIES

Original: Bottom row of Figure 5 shows ablation studies of Structured Prior and NF.
Revised: Bottom row of Figure 5 shows ablation studies of Structured Prior and NF.

Original: Firstly, we look into
Revised: Firstly, we look into

Original: Structured Prior.
Revised: Structured Prior.

Original: In this experiment, the results from full model (top left) and ablation study of
Revised: In this experiment, the results from full model (top left) and ablation study of

Original: Structured Prior (bottom left) are generated from same z g .
Revised: Structured Prior (bottom left) are generated from same z g .

Original: Generated images without Structured
Revised: Generated images without Structured

Original: Prior are severely blurred, however, we can see that the spatial arrangement of objects itself is similar to full model.
Revised: Prior are severely blurred, however, we can see that the spatial arrangement of objects itself is similar to full model.

Original: This indicates that Structured Prior only contributes to object-level generation quality.
Revised: This indicates that Structured Prior only contributes to object-level generation quality.

Original: Secondly, bottom right images in Figure 5 shows ablation result of NF.
Revised: Secondly, bottom right images in Figure 5 shows ablation result of NF.

Original: As you can see, the quality is almost same with full model even without NF, which indicates that NF is not essential for generation.
Revised: As you can see, the quality is almost same with full model even without NF, which indicates that NF is not essential for generation.

Original: However, as we mentioned in Section 4.1, NF is important for stable inference and training, thus it is also necessary for generation in practice.

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: [Figure]

Original: D
Revised: D

Original: [Figure] [Figure] B C
Revised: However, as we mentioned in Section 4.1, NF is important for stable inference and training, thus it is also necessary for generation in practice.

Original: D 4.2. Q UANTITATIVE E VALUATIONS
Revised: 4.2. Q UANTITATIVE E VALUATIONS

Original: We also quantitatively evaluated random generation by Fr ´ echet Inception Distance(FID) score in
Revised: We also quantitatively evaluated random generation by Fr ´ echet Inception Distance(FID) score in

Original: Table 2.
Revised: Table 2.

Original: WeLIS outperforms MulMON in every dataset, and as shown in above section, WeLIS without NF achieved similar quality to full model, as long as the training succeeded.
Revised: WeLIS outperforms MulMON in every dataset, and as shown in above section, WeLIS without NF achieved similar quality to full model, as long as the training succeeded.

Original: Scores without
Revised: Scores without

Original: NF in CLEVR AUG
Revised: NF in CLEVR AUG

Original: and GQN Jaco are degraded because the training fails more frequently as datasets become difﬁcult.
Revised: and GQN Jaco are degraded because the training fails more frequently as datasets become difﬁcult.

Original: Poor generation quality of MulMON comes from mainly two reasons: insufﬁcient modeling of spatial arrangement and difﬁculty in sampling actual posterior.
Revised: Poor generation quality of MulMON comes from mainly two reasons: insufﬁcient modeling of spatial arrangement and difﬁculty in sampling actual posterior.

Original: In WeLIS, former one is dealt with by introducing global latent z g which represents spatial information of a whole scene, and latter one is dealt with by a Structured Prior which estimates complex posterior inferred from IAI.
Revised: In WeLIS, former one is dealt with by introducing global latent z g which represents spatial information of a whole scene, and latter one is dealt with by a Structured Prior which estimates complex posterior inferred from IAI.

Original: Effectiveness of introducing z g is evaluated in Section 4.3 and of Structured Prior is already investigated in
Revised: Effectiveness of introducing z g is evaluated in Section 4.3 and of Structured Prior is already investigated in

Original: Figure 5.
Revised: Figure 5.

Original: 4. I NSPECTION OF G LOBAL L ATENT S PACE
Revised: 4. I NSPECTION OF G LOBAL L ATENT S PACE

Original: To look into the representation obtained by global latent z g , we show random generation results from four different z g
Revised: To look into the representation obtained by global latent z g , we show random generation results from four different z g

Original: (A-D) and four neighborhoods z g + (cid:15) of them in Figure 6.
Revised: (A-D) and four neighborhoods z g + (cid:15) of them in Figure 6.

Original: Here, (cid:15) is a small perturbation randomly sampled from standard normal distribution.
Revised: Here, (cid:15) is a small perturbation randomly sampled from standard normal distribution.

Original: First column in each group shows images generated from four different z g , and other four columns are generated with different perturbation.
Revised: First column in each group shows images generated from four different z g , and other four columns are generated with different perturbation.

Original: Thus, each four group represents a “cluster” around z g .
Revised: Thus, each four group represents a “cluster” around z g .

Original: We can see that generated scenes from same z g have similar spatial structure, while their objects vary.
Revised: We can see that generated scenes from same z g have similar spatial structure, while their objects vary.

Original: This indicates that the latent space of z g obtains spatial conﬁguration of a scene as we expected.
Revised: This indicates that the latent space of z g obtains spatial conﬁguration of a scene as we expected.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced new multi-view object-centric model: WeLIS, which is able to perform multi-view object-centric inference and sampling in 3D scenes.
Revised: We introduced new multi-view object-centric model: WeLIS, which is able to perform multi-view object-centric inference and sampling in 3D scenes.

Original: We introduced several components to WeLIS, in order to adapt a global latent variable and to enable inference and training with that model.
Revised: We introduced several components to WeLIS, in order to adapt a global latent variable and to enable inference and training with that model.

Original: The global latent variable improved inference quality especially when the observations are limited (Figure 2) and also enabled the model to generate physically plausible novel scenes (Figure 5).
Revised: The global latent variable improved inference quality especially when the observations are limited (Figure 2) and also enabled the model to generate physically plausible novel scenes (Figure 5).

Original: We also conducted some ablation studies and investigated the representation obtained by global latent space (Figure 6).
Revised: We also conducted some ablation studies and investigated the representation obtained by global latent space (Figure 6).

Original: One of the future directions of this research is to sophisticate inference methods: normalizing ﬂow and IAI.
Revised: One of the future directions of this research is to sophisticate inference methods: normalizing ﬂow and IAI.

Original: There is a room for improvement with optimal choice of normalizing ﬂow.
Revised: There is a room for improvement with optimal choice of normalizing ﬂow.

Original: In addition, we used simpliﬁed IAI without image-sized input which can reduce computational cost, however there is a possibility that this sacriﬁced the ability to distinguish similar, ambiguous objects such as rounded cubes and spheres in CLEVR MV dataset.
Revised: In addition, we used simpliﬁed IAI without image-sized input which can reduce computational cost, however there is a possibility that this sacriﬁced the ability to distinguish similar, ambiguous objects such as rounded cubes and spheres in CLEVR MV dataset.

Original:  R EPRODUCIBILITY S TATEMENT Architecture :
Revised:  R EPRODUCIBILITY S TATEMENT Architecture :

Original: Detail of the network architecture is provided in Appendix A. Public version of the source code will be released soon
Revised: Detail of the network architecture is provided in Appendix A. Public version of the source code will be released soon.

Original: . Optimization :
Revised: Optimization :

Original: We derived the training scheme from MulMON in which all models are trained for 300k gradient steps with Adam optimizer.
Revised: We derived the training scheme from MulMON in which all models are trained for 300k gradient steps with Adam optimizer.

Original: 300k gradient steps is composed of 1000 epochs × 300 scenes.
Revised: 300k gradient steps is composed of 1000 epochs × 300 scenes.

Original: The 300 scenes are randomly picked in each iteration.
Revised: The 300 scenes are randomly picked in each iteration.

Original: Learning rate decay in GQN and MulMON is also applied to our model.
Revised: Learning rate decay in GQN and MulMON is also applied to our model.

Original: A N ETWORK A RCHITECTURE A. G LOBAL E NCODER
Revised: A N ETWORK A RCHITECTURE A. G LOBAL E NCODER

Original: [Table]
Revised: [Table]

Original: Product of Experts
Revised: Product of Experts

Original: Global Encoder is composed of CNN, viewpoint embedding, feedforward network for reparameterization and PoE. Output from CNN and viewpoint embedding is concatenated and fed into feedforward network.
Revised: Global Encoder is composed of CNN, viewpoint embedding, feedforward network for reparameterization and PoE. Output from CNN and viewpoint embedding is concatenated and fed into feedforward network.

Original: We used Planar Flow in this encoder.
Revised: We used Planar Flow in this encoder.

Original: The dimension of the ﬂow is same as the dimension of z g , and the transforms are applied 16 times in this model.
Revised: The dimension of the ﬂow is same as the dimension of z g , and the transforms are applied 16 times in this model.

Original: A. S EQUENTIAL E NCODER
Revised: A. S EQUENTIAL E NCODER

Original: [Table]
Revised: [Table]

Original: Sequential Encoder is composed of LSTM and linear layer.
Revised: Sequential Encoder is composed of LSTM and linear layer.

Original: Input of LSTM is concatenation of z g and z ’s parameters inferred in last iteration.
Revised: Input of LSTM is concatenation of z g and z ’s parameters inferred in last iteration.

Original: Initial input is concatenation of z g and zero vector.
Revised: Initial input is concatenation of z g and zero vector.

Original: A. R EFINEMENT N ETWORK
Revised: A. R EFINEMENT N ETWORK

Original: [Table]
Revised: [Table]

Original: Reﬁnement Network takes log-likelihood, gradient of the parameters (mean and variance) of z i −and the parameters of z g as input, where i means iteration step.
Revised: Reﬁnement Network takes log-likelihood, gradient of the parameters (mean and variance) of z i −and the parameters of z g as input, where i means iteration step.

Original: [Table]
Revised: [Table]

Original: Structured prior is implemented by Transformer.
Revised: Structured prior is implemented by Transformer.

Original: We used learnable positinal embedding, and activation is GeLU (Hendrycks & Gimpel, 2016) as in BERT (Devlin et al., 2019).
Revised: We used learnable positinal embedding, and activation is GeLU (Hendrycks & Gimpel, 2016) as in BERT (Devlin et al., 2019).

Original: We did not use dropout here.
Revised: We did not use dropout here.

Original: [Table]
Revised: [Table]

Original: We used same decoder as MulMON.
Revised: We used same decoder as MulMON.

Original: v and z are concatenated.
Revised: v and z are concatenated.

Original: Concatenated vector is input to projector (former part of the table), then the output is fed into broadcast decoder (latter part of the table).
Revised: Concatenated vector is input to projector (former part of the table), then the output is fed into broadcast decoder (latter part of the table).

Original: B A DDITIONAL R ESULTS B. Q UANTITATIVE E VALUATION ON O BSERVATION
Revised: B A DDITIONAL R ESULTS B. Q UANTITATIVE E VALUATION ON O BSERVATION

Original: [Figure]
Revised: [Figure]

Original: OursOurs: w/o Flow MulMON (bassline)
Revised: OursOurs: w/o Flow MulMON (bassline)

Original: [Figure]
Revised: [Figure]

Original: 𝑁 : the number of observations
Revised: 𝑁 : the number of observations

Original: [Figure]
Revised: [Figure]

Original: Figure 2 in the main text showed quantitative evaluation for query views.
Revised: Figure 2 in the main text showed quantitative evaluation for query views.

Original: In this section, we show the results for observation views (Figure 7).
Revised: In this section, we show the results for observation views (Figure 7).

Original: Namely, this ﬁgure is about reconstruction and segmentation quality.
Revised: Namely, this ﬁgure is about reconstruction and segmentation quality.

Original: The performance of MulMON with small N is relatively not good.
Revised: The performance of MulMON with small N is relatively not good.

Original: This is because
Revised: This is because

Original: MulMON needs to process observation views iteratively, thus the model requires at least several observation views to achieve its best performance.
Revised: MulMON needs to process observation views iteratively, thus the model requires at least several observation views to achieve its best performance.

Original: On the other hand, WeLIS is stable against N thanks to the parallel processing of Global Encoder.
Revised: On the other hand, WeLIS is stable against N thanks to the parallel processing of Global Encoder.

Original: B. N OVEL V IEW S YNTHESIS AND S EGMENTATION B.2. E VALUATION M ETRICS AND Q UALITATIVE D IFFERENCE
Revised: B. N OVEL V IEW S YNTHESIS AND S EGMENTATION B.2. E VALUATION M ETRICS AND Q UALITATIVE D IFFERENCE

Original: WeLIS (RMSE: 0.089 mIoU: 0.58)
Revised: WeLIS (RMSE: 0.089 mIoU: 0.58)

Original: MulMON (RMSE: 0.093 mIoU: 0.55)
Revised: MulMON (RMSE: 0.093 mIoU: 0.55)

Original: [Figure]
Revised: [Figure]

Original: Figure 8 shows additional quantitative samples of novel view synthesis.
Revised: Figure 8 shows additional quantitative samples of novel view synthesis.

Original: This ﬁgure helps us to understand how much quantitative difference of RMSE and mIoU actually correspond to the qualitative difference.
Revised: This ﬁgure helps us to understand how much quantitative difference of RMSE and mIoU actually correspond to the qualitative difference.

Original: In addition, we can see that WeLIS represents background with a ﬁxed slot (yellow), while MulMON represents it with different slots (various colors).
Revised: In addition, we can see that WeLIS represents background with a ﬁxed slot (yellow), while MulMON represents it with different slots (various colors).

Original: This is because the auto-regressive encoder is no longer permutation invariant.
Revised: This is because the auto-regressive encoder is no longer permutation invariant.

Original: It can be beneﬁcial that ﬁxed slot represents background, however, disentanglement might be sacriﬁced with this.
Revised: It can be beneﬁcial that ﬁxed slot represents background, however, disentanglement might be sacriﬁced with this.

Original: It is difﬁcult to conclude which one is better currently, because this kind of properties should be evaluated by various downstream tasks.
Revised: It is difﬁcult to conclude which one is better currently, because this kind of properties should be evaluated by various downstream tasks.

Original: B. D OWNSTREAM T ASK
Revised: B. A BLATION S TUDY

Revised: B

Original: Models
Revised: [Figure]

Original: Datasets Observations
Revised: Figure 9 shows ablation result of Global Encoder (PoE), GECO, separate training of Structured

Original: Accuracy MulMON CLEVR MV 64.
Revised: Prior, and Normalizing Flow.

Original: WeLIS( z g ) 73. WeLIS( z ) 63. WeLIS( z g + z ) 71.
Revised: Regarding PoE, we replaced PoE with simple average, which is often used in many researches such as NeRF-VAE (Kosiorek et al., 2021).

Original: MulMON CLEVR MV [Table] WeLIS( z g ) 76.
Revised: This result indicates that each component contributes to the scores and training stability.

Original: WeLIS( z ) 53.
Revised: N OVEL S

Original: WeLIS( z g + z )
Revised: CENE G ENERATION

Original: MulMON [Table] [Table] WeLIS( z g ) 76.
Revised: We tested two different architectures for Structured Prior.

Original: WeLIS( z ) 58. WeLIS( z g + z ) 76. MulMON CLEVR AUG
Revised: Both variants use MLP, and one of them updates each slot independently, while the other one considers interaction between slots.

Original: [Table] [Table]
Revised: As we show in Table 8, there is no signiﬁcant difference between these variants.

Original: WeLIS( z g ) 63. WeLIS( z ) 74. WeLIS( z g + z )
Revised: This result indicates that z g alone properly models object placement, and Structured Prior does not have to model dependency between slots.

Original: [Table]
Revised: [Table]

Original: MulMON

Original: CLEVR AUG 70.

Original: WeLIS( z g ) 68. WeLIS( z ) 78. WeLIS( z g + z ) 78. MulMON
Revised: B. D OWNSTREAM T ASK

Original: CLEVR AUG 71. WeLIS( z g ) 69. WeLIS( z ) 79. WeLIS( z g + z ) 79.
Revised: [Table]

Original: We evaluated the latent space of z g qualitatively in Figure 6.
Revised: We evaluated the latent space of z g qualitatively in Figure 6.

Original: In this section, we evaluate the obtained representation by downstream task.
Revised: In this section, we evaluate the obtained representation by downstream task.

Original: The task is to predict how many objects are in the observed scenes.
Revised: The task is to predict how many objects are in the observed scenes.

Original: We used linear classiﬁer in all the models.
Revised: We used linear classiﬁer in all the models.

Original: WeLIS can use both z and z g for classiﬁcation task, thus we tested all three patterns: z only, z g only and both z and z g .
Revised: WeLIS can use both z and z g for classiﬁcation task, thus we tested all three patterns: z only, z g only and both z and z g .

Original: [Figure]
Revised: [Figure]

Original: We show

Original: These