Original: C OMPOSITIONAL L AW P ARSING WITH L ATENT R ANDOM F UNCTIONS
Revised: C OMPOSITIONAL L AW P ARSING WITH L ATENT R ANDOM F UNCTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Human cognition has compositionality.
Revised: Human cognition has compositionality.

Original: We understand a scene by decomposing the scene into different concepts (e.g. shape and position of an object) and learning the respective laws of these concepts which may be either natural (e.g. laws of motion) or man-made (e.g. laws of a game).
Revised: We understand a scene by decomposing the scene into different concepts (e.g., shape and position of an object) and learning the respective laws of these concepts, which may be either natural (e.g., laws of motion) or man-made (e.g., laws of a game).

Original: The automatic parsing of these laws indicates the model’s ability to understand the scene, which makes law parsing play a central role in many visual tasks.
Revised: The automatic parsing of these laws indicates the model’s ability to understand the scene, which makes law parsing play a central role in many visual tasks.

Original: In this paper, we propose a deep latent variable model for Compositional LAw Parsing (CLAP). CLAP achieves the human-like compositionality ability through an encoding-decoding architecture to represent concepts in the scene as latent variables, and further employ conceptspeciﬁc random functions, instantiated with Neural Processes, in the latent space to capture the law on each concept.
Revised: This paper proposes a deep latent variable model for Compositional LAw Parsing (CLAP), which achieves the human-like compositionality ability through an encoding-decoding architecture to represent concepts in the scene as latent variables. CLAP employs concept-speciﬁc latent random functions that can be instantiated with Neural Processes to capture the law of concepts.

Original: Our experimental results demonstrate that CLAP outperforms thecompared baseline methods in multiple visual tasks including intuitive physics, abstract visual reasoning, and scene representation.
Revised: Our experimental results demonstrate that CLAP outperforms the baseline methods in multiple visual tasks such as intuitive physics, abstract visual reasoning, and scene representation.

Original: In addition, CLAP can learn concept-speciﬁc laws in a scene without supervision and one can edit laws by modifying the corresponding latent random functions, validating its interpretability and manipulability.
Revised: The law manipulation experiments illustrate CLAP’s interpretability by modifying speciﬁc latent random functions on samples. For example, CLAP learns position-changing physical laws and appearance constancy from the moving balls scene, making it possible to exchange laws between samples or compose existing laws into novel laws.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Compositionality is an important feature of human cognition (Lake et al., 2017).
Revised: Compositionality is an important feature of human cognition (Lake et al., 2017).

Original: Humans can decompose a scene into individual concepts to learn the respective laws of these concepts, which can be either natural (e.g. laws of motion) or man-made (e.g. laws of a game).
Revised: Humans can decompose a scene into individual concepts to learn the respective laws of these concepts, which can be either natural (e.g. laws of motion) or man-made (e.g. laws of a game).

Original: When observing a scene of a moving ball, one tends to parse the changing patterns of its appearance and position separately:
Revised: When observing a scene of a moving ball, one tends to parse the changing patterns of its appearance and position separately:

Original: The appearance stays consistent over time, while the position changes according to the laws of motion.
Revised: The appearance stays consistent over time, while the position changes according to the laws of motion.

Original: By composing the laws of the ball’s appearance and position, one can understand the changing pattern and predict the status of the moving ball.
Revised: By composing the laws of the ball’s appearance and position, one can understand the changing pattern and predict the status of the moving ball.

Original: Although compositionality has inspired a number of models in visual understanding such as representing handwritten characters through hierarchical decomposition of characters (Lake et al., 2011; 2015) and representing a multi-object scene with object-centric representations (Eslami et al., 2016;
Revised: Although compositionality has inspired a number of models in visual understanding such as representing handwritten characters through hierarchical decomposition of characters (Lake et al., 2011; 2015) and representing a multi-object scene with object-centric representations (Eslami et al., 2016;

Original: Kosiorek et al., 2018; Greff et al., 2019; Locatello et al., 2020), automatic parsing of laws in a scene is still a great challenge.
Revised: Kosiorek et al., 2018; Greff et al., 2019; Locatello et al., 2020), automatic parsing of laws in a scene is still a great challenge.

Original: For example, to understand the rules in abstract visual reasoning such as
Revised: For example, to understand the rules in abstract visual reasoning such as

Original: Raven’s Progressive Matrices (RPMs) test, the comprehension of attribute-speciﬁc representations and the underlying relationships among them is crucial for a model to predict the missing images (Santoro et al., 2018; Steenbrugge et al., 2018; Wu et al., 2020).
Revised: Raven’s Progressive Matrices (RPMs) test, the comprehension of attribute-speciﬁc representations and the underlying relationships among them is crucial for a model to predict the missing images (Santoro et al., 2018; Steenbrugge et al., 2018; Wu et al., 2020).

Original: To understand the laws of motion in intuitive physics, a model needs to grasp the changing patterns of different attributes (e.g. appearance and position) of each object in a scene to predict the future (Agrawal et al., 2016;
Revised: To understand the laws of motion in intuitive physics, a model needs to grasp the changing patterns of different attributes (e.g. appearance and position) of each object in a scene to predict the future (Agrawal et al., 2016;

Original: Kubricht et al., 2017; Ye et al., 2018).
Revised: Kubricht et al., 2017; Ye et al., 2018).

Original: However, these methods usually employ neural networks to directly model changing patterns of the scene in a black box, and can hardly abstract the laws of individual concepts in an explicit, interpretable and even manipulatable way.
Revised: However, these methods usually employ neural networks to directly model changing patterns of the scene in a black box, and can hardly abstract the laws of individual concepts in an explicit, interpretable and even manipulatable way.

Original: A possible solution to enable the abovementioned ability for a model is exploiting a function to represent a law of concept in terms of the representation of the concept itself.
Revised: A possible solution to enable the abovementioned ability for a model is exploiting a function to represent a law of concept in terms of the representation of the concept itself.

Original: To represent a law that may depict arbitrary changing patterns, an expressive and ﬂexible random function is required.
Revised: To represent a law that may depict arbitrary changing patterns, an expressive and ﬂexible random function is required.

Original: The Gaussian Process (GP) is a classical family of random functions that achieves the diversity of function space through different kernel functions (Williams & Rasmussen, 2006).
Revised: The Gaussian Process (GP) is a classical family of random functions that achieves the diversity of function space through different kernel functions (Williams & Rasmussen, 2006).

Original: Recently proposed random functions (Garnelo et al., 2018b; Eslami et al., 2018; Kumar et al., 2018; Garnelo et al., 2018a; Singh et al., 2019; Kim et al., 2019; Louizos et al., 2019; Lee et al., 2020; Foong et al., 2020) describe function spaces with the powerful nonlinear ﬁtting ability of neural networks.
Revised: Recently pro- posed random functions (Garnelo et al., 2018b; Eslami et al., 2018; Kumar et al., 2018; Garnelo et al., 2018a; Singh et al., 2019; Kim et al., 2019; Louizos et al., 2019; Lee et al., 2020; Foong et al., 2020) describe function spaces with the powerful nonlinear ﬁtting ability of neural networks.

Original: These random functions have been used to capture the changing patterns in a scene, such as mapping timestamps to frames to describe the physical law of moving objects in a video (Kumar et al., 2018;
Revised: These random functions have been used to capture the changing patterns in a scene, such as mapping timestamps to frames to describe the physical law of moving objects in a video (Kumar et al., 2018;

Original: Singh et al., 2019; Fortuin et al., 2020).
Revised: Singh et al., 2019; Fortuin et al., 2020).

Original: However, these applications of random functions take images as inputs and the captured laws account for all pixels instead of expected individual concepts.
Revised: However, these applications of random functions take images as inputs and the captured laws account for all pixels instead of expected individual concepts.

Original: In this paper, we propose a deep latent variable model for Compositional LAw Parsing (CLAP).
Revised: In this paper, we propose a deep latent variable model for Compositional LAw Parsing (CLAP).

Original: CLAP achieves the human-like compositionality ability through an encoding-decoding architecture (Hamrick et al., 2018) to represent concepts in the scene as latent variables, and further employ concept-speciﬁc random functions in the latent space to capture the law on each concept.
Revised: CLAP achieves the human-like compositionality ability through an encoding-decoding architecture (Hamrick et al., 2018) to represent concepts in the scene as latent variables, and further employ concept-speciﬁc random functions in the latent space to capture the law on each concept.

Original: By means of the plug-in of different random functions, CLAP gains generality and ﬂexibility applicable to various law parsing tasks.
Revised: By means of the plug-in of different random functions, CLAP gains generality and ﬂexibility applicable to various law parsing tasks.

Original: We introduce CLAP-NP as an example that instantiates latent random functions with recently proposed Neural Processes (Garnelo et al., 2018b).
Revised: We introduce CLAP-NP as an example that instantiates latent random functions with recently proposed Neural Processes (Garnelo et al., 2018b).

Original: Our experimental results demonstrate that the proposed CLAP outperforms the compared baseline methods in multiple visual tasks including intuitive physics, abstract visual reasoning, and scene representation.
Revised: Our experimental results demonstrate that the proposed CLAP outperforms the compared baseline methods in multiple visual tasks including intuitive physics, abstract visual reasoning, and scene representation.

Original: In addition, the experiment on exchanging latent random functions on a speciﬁc concept and the experiment on composing latent random functions of the given samples to generate new samples both well demonstrate the interpretability and even manipulability of the proposed method for compositional law parsing.
Revised: In addition, the experiment on exchanging latent random functions on a speciﬁc concept and the experiment on composing latent random functions of the given samples to generate new samples both well demonstrate the interpretability and even manipulability of the proposed method for compositional law parsing.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Since CLAP focuses on parsing compositional laws which are modeled by latent random functions, compositional scene representation and random functions are related to this paper.
Revised: Since CLAP focuses on parsing compositional laws which are modeled by latent random functions, compositional scene representation and random functions are related to this paper.

Original: Compositional 
Revised: Compositional Scene Representation

Original: Scene Representation

Original: Inspired by the compositionality in human cognition, compositional scene representation models understand a scene by object-centric representations.
Revised: Inspired by the compositionality in human cognition, compositional scene representation models understand a scene by object-centric representations.

Original: To this end, a class of models initialize object-centric representations ﬁrst and update them by iterative processes such as Neural Expectation Maximization (Greff et al., 2017), iterative amortized inference (Greff et al., 2019), iterative cross-attention
Revised: To this end, a class of models initialize object-centric representations ﬁrst and update them by iterative processes such as Neural Expectation Maximization (Greff et al., 2017), iterative amortized inference (Greff et al., 2019), iterative cross-attention

Original: (Locatello et al., 2020), and two-stage inference (Emami et al., 2021).
Revised: (Locatello et al., 2020), and two-stage inference (Emami et al., 2021).

Original: Another class of models extracts object-centric representations from a scene without iterative inference processes.
Revised: Another class of models extracts object-centric representations from a scene without iterative inference processes.

Original: AIR (Eslami et al., 2016) parses object-centric presence, appearance, and position representations as latent variables.
Revised: AIR (Eslami et al., 2016) parses object-centric presence, appearance, and position representations as latent variables.

Original: SPACE (Lin et al., 2020b) divides a scene image into small regions and parses region-speciﬁc representations in parallel.
Revised: SPACE (Lin et al., 2020b) divides a scene image into small regions and parses region-speciﬁc representations in parallel.

Original: GNM (Jiang & Ahn, 2020) models the layout of scenes.
Revised: GNM (Jiang & Ahn, 2020) models the layout of scenes.

Original: Recently, many models aim at parsing compositional scene representations in videos (Kosiorek et al., 2018; Jiang et al., 2019; Lin et al., 2020a).
Revised: Recently, many models aim at parsing compositional scene representations in videos (Kosiorek et al., 2018; Jiang et al., 2019; Lin et al., 2020a).

Original: Random Functions
Revised: Random Functions

Original: The GP (Williams & Rasmussen, 2006) is a classical family of random functions (Williams & Rasmussen, 2006) that regards the outputs of a function as a random variable of multivariate Gaussian distribution.
Revised: The GP (Williams & Rasmussen, 2006) is a classical family of random functions (Williams & Rasmussen, 2006) that regards the outputs of a function as a random variable of multivariate Gaussian distribution.

Original: GQN (Eslami et al., 2018) represents 3D scenes as functions by a well-designed representation network and generation network.
Revised: GQN (Eslami et al., 2018) represents 3D scenes as functions by a well-designed representation network and generation network.

Original: Conditional Neural Process (CNP) (Garnelo et al., 2018a) uses a deterministic representation to capture the global structure of functions. To introduce stochasticity in function parsing, NP (Garnelo et al., 2018b) adopts a Gaussian distributed global random variable g . ANP (Kim et al., 2019) overcomes the underﬁtting problem in NP by combining a deterministic and stochastic computational branch with the attention mechanism.
Revised: NP (Garnelo et al., 2018b) captures function stochasticity by a Gaussian distributed latent variable instead of a deterministic representation in CNP (Garnelo et al., 2018a).

Original: Depending on the way of randomness modeling, context aggregation and output prediction, we can constructdifferent types of function spaces (Louizos et al., 2019; Singh et al., 2019; Lee et al., 2020; Foong et al., 2020).
Revised: Considering different ways of stochasticity modeling, context aggregation, and output prediction for NP, we can construct function spaces with different characteristics (Kim et al., 2019; Louizos et al., 2019; Singh et al., 2019; Lee et al., 2020; Foong et al., 2020).

Original: In addition, some of the few-shot learning methods parse functions from data by learning global variables (Wu et al., 2018; Gordon et al., 2019) or adaptive kernels (Tossou et al., 2019; Patacchiola et al., 2020).
Revised: Some few-shot learning methods parse functions from data by learning global variables (Wu et al., 2018; Gordon et al., 2019) or adaptive kernels (Tossou et al., 2019; Patacchiola et al., 2020).

Original: Another class of methods incorporates latent variable models and ordinary differential equations (Norcliffe et al., 2021) or stochastic differential equations (Li et al., 2020; Hasan et al., 2021) to describe random functions by the integral over latent states.
Revised: We can also combine latent variable models with ODEs (Norcliffe et al., 2021) or SDEs (Li et al., 2020; Hasan et al., 2021) to describe random functions by the integral over latent states.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: A random function is a distribution over function space F . By sampling from a random function, we get a function f : X (cid:55)→ Y that maps the input sequence  x 
Revised: A random function describes a distribution over function space F , from which we can sample a function f mapping the input sequence ( x 1 , ..., x N ) to output sequence ( y 1 , ..., y N ) .

Original: = ( x 1 , ..., x N ) to output sequence y =
Revised: The important problem for a random function is ﬁnding the underlying function over some given points and predicting function values at novel positions accordingly. In detail, given a set of context points

Original: [Figure]
Revised: [Figure]

Original: ( y 1 , ..., y N ) . Suppose that D  = { ( x n , y n ) } Nn =1 is a set of points from f ∈ F that y n = f( x n ) .
Revised: D S = { ( x s , y s ) | s ∈ S } , we should ﬁnd the most possible function that y s = f

Original: By a context index set C and target index set T , we divide D into two subsets D C
Revised: ( x s ) for these context points, then use f ( x t ) to predict y t for target points D T

Original: = { ( xc , y c ) | c ∈ C } and D T = { ( x t , y t ) | t ∈ T }where D = D C ∪ D T and D C ∩ D T = ∅ .
Revised: = { ( x t , y t ) | t ∈ T } .

Original: For a random function, an important problem is to ﬁnd the corresponding function f according to D C and predict target outputs y t = f (  x t ) .
Revised: The abovementioned prediction problem can be regarded as estimating a conditional probability p ( y T | x T , D S ) .

Original: Neural Process (NP)
Revised: Neural Process (NP)

Original: The key of the problem is to estimate p ( y T | y C , x C , x T ) .

Original: NP (Garnelo et al., 2018b) introduces Gaussian distributedglobal latent variable g to represent the stochasticity of functions.
Revised: NP (Garnelo et al., 2018b) captures function stochasticity by a Gaussian distributed latent variable g and lets p ( y T | x T , D S )

Original: In this case, the estimation of p ( y T | y C , x C , x T ) consists of two steps since p ( y T | y C , x C , x T ) =

Original:  (cid:82) (cid:81) t ∈ T p ( y t | g , x t ) p ( g | y C , x C )
Revised: = (cid:82) (cid:81) t ∈ T p ( y t | g , x t ) q ( g | D S )

Original: d g .
Revised: d g .

Original: In the ﬁrst step, an aggregator module extracts the information from context points in D C to estimate p ( g |  y C , x C ) = N ( g |  µ , diag [Equation]
Revised: In this case, p ( y T | x T , D S ) consists of a generative process p ( y t | g , x t ) and inference process q ( g | D S ) , which can be jointly optimized by variational inference. To estimate p ( y T | x T , D S ) , NP ﬁrst extracts the feature of each context point in D S and sum them up to acquire the mean µ and deviation σ of g : µ , σ = T f (cid:32)| S | (cid:88) s ∈ S

Original: σ 2 )) where
Revised: T a ( y s , x s ) (cid:33) .

Original: [Equation]
Revised: [Equation]

Original: In the second step, NP samples the global latent variable g , which is combined with x t to independently predict y t  by  p ( y t | g , x t ) =
Revised: Then NP samples the global latent variable g from q ( g | D S ) . NP can independently predict y t at the target position x t by the generative process p ( y t | g , x t ) =

Original: N ( y t |T p ( g , x t ) , σ 2 I ) .
Revised: N ( y t |T p ( g , x t ) , σ 2 I ) .

Original: Neural networks T a , T f and T p increase the model capacity of NP to learn data-speciﬁc function spaces.
Revised: Neural networks T a , T f and T p increase the model capacity of NP to learn data-speciﬁc function spaces.

Original:  M ODEL
Revised:  M ODEL

Original: Figure 1 is an overview of CLAP.
Revised: Figure 1 is an overview of CLAP.

Original: First, an encoder converts context images to individual concepts in the latent space.
Revised: First, an encoder converts context images to individual concepts in the latent space.

Original: Then CLAP parses concept-speciﬁc random functions on the context visual concepts, which are exploited to predict the target concepts.
Revised: Then CLAP parses concept-speciﬁc random functions on the context visual concepts, which are exploited to predict the target concepts.

Original: Finally, a decoder maps the target concepts to target images.
Revised: Finally, a decoder maps the target concepts to target images.

Original: The concept-speciﬁc latent random functions achieve the compositional law parsing in CLAP.
Revised: The concept-speciﬁc latent random functions achieve the compositional law parsing in CLAP.

Original: In the following part, we will introduce the generative process, variational inference, and parameter learning of CLAP.
Revised: In the following part, we will introduce the generative process, variational inference, and parameter learning of CLAP.

Original: At the end of this section, we propose CLAP-NP, an instance that instantiates the latent random function with NP.
Revised: At the end of this section, we propose CLAP-NP, an instance that instantiates the latent random function with NP.

Original: 4. L ATENT R ANDOM F UNCTIONS
Revised: 4. L ATENT R ANDOM F UNCTIONS

Original: Given D C and D T , the objective of CLAP is to maximizelog-likelihood log p ( y T | y C , x ) .
Revised: Given D S and D T , the objective of CLAP is to maximize log p ( y T | y S , x ) .

Original: In the framework of conditional latent variable models, we introduce a variational posterior q ( h | y , x ) for the latent variable h , and approximate log-likelihood by evidence lower bound (ELBO) (Sohn et al., 2015):
Revised: In the framework of conditional latent variable models, we introduce a variational posterior q ( h | y , x ) for the latent variable h , and approximate log-likelihood by evidence lower bound (ELBO) (Sohn et al., 2015):

Original: [Equation]
Revised: [Equation]

Revised: The latent variable h includes the local information of images

Revised: z = { z n } Nn =1 where z n is the low-dimensional representation of x n .

Revised: Moreover, we decompose z n into independent concepts { z cn | c ∈ C } (e.g., an image of a single-object scene may have concepts such as object appearance, illumination, camera orientation, etc.)

Revised: where C refers to the name set of these concepts.

Revised: Assuming that these concepts satisfy their respective change patterns, we employ independent latent random

Original: 𝐴 𝐶  𝒚 !
Revised: 𝐶 𝑆 𝒚 !

Original: 𝒙 !
Revised: 𝒙 !

Original: 𝒛 !"
Revised: 𝒛 !"

Original: 𝒛 #" 𝒚 # 𝒙 # 𝑇
Revised: 𝒛 #" 𝒚 # 𝒙 # 𝑇

Original: Latent Random Function 𝑓 !
Revised: Latent Random Function 𝑓 !

Original: (a) CLAP 𝒙 !
Revised: (a) CLAP 𝒙 !

Original: 𝒛 !"
Revised: 𝒛 !"

Original: 𝒛 #" 𝒙 #
Revised: 𝒛 #" 𝒙 #

Original: NP 𝒈 " 𝒯 !"
Revised: NP 𝒈 " 𝒯 !"

Original: (b) NP as the latent random function The latent variable h includes the local information of images z = { z n } Nn =1 where z n is the low-dimensional representation of x n . Moreover, we decompose z n into independent concepts { z an | a ∈ A } (e.g., an image of a single-object scene may have concepts such as object appearance, illumination, camera orientation, etc.)
Revised: (b) NP as the latent random function functions to capture the law of each concept.

Original: where A refers to the name set of these concepts.

Original: Assuming that these concepts satisfy their respective change patterns, we employ independent latent random functions to capture the law of each concept.

Original: Denote the latent random function on the concept a as f a , the speciﬁc form of f a depends on the way we model it.
Revised: Denote the latent random function on the concept c as f c , the speciﬁc form of f c depends on the way we model it.

Original: As Figure 2b shows, if adopting NP as the latent random function, we have f a ( x t ) =
Revised: As Figure 2b shows, if adopting NP as the latent random function, we have f c ( x t ) =

Original: T ap ( g a , x t ) where g a is a Gaussian-distributed latent variable to control the randomness of f a .
Revised: T cp ( g c , x t ) where g c is a Gaussian-distributed latent variable to control the randomness of f c .

Original: Within the graphical model of CLAP in Figure 2a, the prior and posterior in L are factorized into
Revised: Within the graphical model of CLAP in Figure 2a, the prior and posterior in L are factorized into

Original: [Equation]
Revised: [Equation]

Original: Decomposing f into concept-speciﬁc laws { f a | a ∈ A } in the above process is critical for CLAP’s compositionality.
Revised: Decomposing f into concept-speciﬁc laws { f c | c ∈ C } in the above process is critical for CLAP’s compositionality.

Original: The compositionality makes it possible to parse only several concept-speciﬁc laws rather than the entire law on z , which reduces the complexity of law parsing while increasing the interpretability of laws.
Revised: The compositionality makes it possible to parse only several concept-speciﬁc laws rather than the entire law on z , which reduces the complexity of law parsing while increasing the interpretability of laws.

Original: The interpretability allows us to edit laws, for example, exchanging the law on some concepts and composing the laws of existing samples.
Revised: The interpretability allows us to edit laws, for example, exchanging the law on some concepts and composing the laws of existing samples.

Original: 4. P ARAMETER L EARNING
Revised: 4. P ARAMETER L EARNING

Original: Based on  the detailed priors and posteriors in Equation
Revised: Based on Equation 3, we factorize ELBO as (See Appendix B.3 for the detailed derivation)

Original: [Equation]
Revised: [Equation]

Original: we factorize ELBO as (See Appendix B. for the detailed derivation)

Original: [Equation]
Revised: [Equation]

Original: Reconstruction Term
Revised: Reconstruction Term

Original: L r is the sum of the log-likelihood log p ( y t | z t ) of each target image.
Revised: L r is the sum of the log-likelihood log p ( y t | z t ) of each target image.

Original: p ( y t | z t ) is modelled by a decoder that converts concepts to the distribution parameter of target images.
Revised: p ( y t | z t ) is modelled by a decoder that converts concepts to the distribution parameter of target images.

Original: CLAP maximizes L r to connect the latent space with the image space, enabling the decoder to generate high-quality images.
Revised: CLAP maximizes L r to connect the latent space with the image space, enabling the decoder to generate high-quality images.

Original: Target Regularizor
Revised: Target Regularizor

Original: L t consists of Kullback–Leibler (KL) divergences between the posterior q ( z at | y t ) and the prior p ( z at | f a , z aC , x C , x t ) of target concepts.
Revised: L t consists of Kullback–Leibler (KL) divergences between the posterior q ( z ct | y t ) and the prior p ( z ct | f c , z cS , x S , x t ) of target concepts.

Original: In q ( z at | y t ) , the distribution of target concepts is computed directly from the target image by the encoder, so it usually contains more accurate information.
Revised: In q ( z ct | y t ) , the distribution of target concepts is computed directly from the target image by the encoder, so it usually contains more accurate information.

Original: And in p ( z at | f a , z aC , x C , x t ) , the distribution of the target concept is estimated from the context.
Revised: And in p ( z c t | f c , z c S , x S , x t ) , the distribution of the target concept is estimated from the context.

Original: The minimization of L t ﬁlls the gap between the posterior and prior and ensures the accuracy of predictors.
Revised: The minimization of L t ﬁlls the gap between the posterior and prior and ensures the accuracy of predictors.

Original: Function Regularizor
Revised: Function Regularizor

Original: L f consists of KL divergences between the posterior q ( f a | z a , x ) and the prior p ( f a | z aC , x C ) .
Revised: L f consists of KL divergences between the posterior q ( f c | z c , x ) and the prior p ( f c | z cS , x S ) .

Original: We use L f as a measure of function consistency to ensure that model can obtain similar distributions for f a with any subset z aC ⊆
Revised: We use L f as a measure of function consistency to ensure that model can obtain similar distributions for f c with any subset z cS ⊆

Original: z a and x C ⊆ x as inputs.
Revised: z c and x S ⊆ x as inputs.

Original: To this end, CLAP shares concept-speciﬁc function parsers to model q ( f a | z a , x ) and p ( f a | z aC , x C ) .
Revised: To this end, CLAP shares concept-speciﬁc function parsers to model q ( f c | z c , x ) and p ( f c | z cS , x S ) .

Original: And we need to design the architecture of function parsers so that they can take different subsets as input.
Revised: And we need to design the architecture of function parsers so that they can take different subsets as input.

Original: Context Regularizor
Revised: Context Regularizor

Original: L c consists of KL divergences between the posterior q ( z ac | y c ) and the prior p ( z ac | y c ) of context concepts.
Revised: L s consists of KL divergences between the posterior q ( z cs | y s ) and the prior p ( z cs | y s ) of context concepts.

Original: To avoid the mismatch between the posterior and prior and reduce model parameters, the posterior and prior are parameterized with the same encoder.
Revised: To avoid the mismatch between the posterior and prior and reduce model parameters, the posterior and prior are parameterized with the same encoder.

Original: In this case, we have p ( z ac | y c )
Revised: In this case, we have p ( z cs | y s )

Original: =
Revised: =

Original: q ( z ac | y c ) and further L c = 0 .
Revised: q ( z cs | y s ) and further L s = 0 .

Original: So we remove L c from Equation 4.
Revised: So we remove L s from Equation 4.

Original: Correct decomposition of concepts is essential for model performance.
Revised: Correct decomposition of concepts is essential for model performance.

Original: However, CLAP is trained without additional supervision to guide the learning of concepts.
Revised: However, CLAP is trained without additional supervision to guide the learning of concepts.

Original: Although CLAP explicitly deﬁnes independent concepts in priors and posteriors, it is not sufﬁcient to automatically decompose concepts for some complex data.
Revised: Although CLAP explicitly deﬁnes independent concepts in priors and posteriors, it is not sufﬁcient to automatically decompose concepts for some complex data.

Original: To solve the problem, CLAP can introduce extra inductive biases by designing proper encoder and decoder to artiﬁcially decompose concepts, e.g., use spatial transformation layers to decompose spatial and non-spatial concepts (Skafte & Hauberg, 2019).
Revised: To solve the problem, CLAP can introduce extra inductive biases by designing proper encoder and decoder to artiﬁcially decompose concepts, e.g., use spatial transformation layers to decompose spatial and non-spatial concepts (Skafte & Hauberg, 2019).

Original: We can also set hyperparameters to control the regularizers (Higgins et al., 2017), or add a total correlation (TC) regularizer R TC to help concept learning (Chen et al., 2018).
Revised: We can also set hyperparameters to control the regularizers (Higgins et al., 2017), or add a total correlation (TC) regularizer R TC to help concept learning (Chen et al., 2018).

Original: Here we choose to add extra regularizers to ELBO of CLAP, and ﬁnally, the optimization objective becomes
Revised: Here we choose to add extra regularizers to ELBO of CLAP, and ﬁnally, the optimization objective becomes

Original: [Equation]
Revised: [Equation]

Original:  where β t , β f , β TC are hyperparameters that regulate the importance of regularizers in the training process.
Revised: 5) where β t , β f , β TC are hyperparameters that regulate the importance of regularizers in the training process.

Original: We can compute L ∗ by Stochastic Gradient Variational Bayes (SGVB) estimator (Kingma & Welling, 2013) and update parameters by gradient descent optimizers (Kingma & Ba, 2015).
Revised: We can compute L ∗ by Stochastic Gradient Variational Bayes (SGVB) estimator (Kingma & Welling, 2013) and update parameters by gradient descent optimizers (Kingma & Ba, 2015).

Original: 4. NP AS L ATENT R ANDOM F UNCTION
Revised: 4. NP AS L ATENT R ANDOM F UNCTION

Original: In this subsection, we propose CLAP-NP as an example that instantiates the latent random function with NP.
Revised: In this subsection, we propose CLAP-NP as an example that instantiates the latent random function with NP.

Original: According to NP, we employ a latent variable g a to control the randomness of the latent random function f a for each concept in CLAP-NP.
Revised: According to NP, we employ a latent variable g c to control the randomness of the latent random function f c for each concept in CLAP-NP.

Original: Generative Process
Revised: Generative Process

Original: CLAP-NP ﬁrst encodes each context image x c to the concepts:
Revised: CLAP-NP ﬁrst encodes each context image x s to the concepts:

Original: [Equation]
Revised: [Equation]

Original: To stabilize the learning of concepts, the encoder outputs only the mean of Gaussian distribution, with the standard deviation as a hyperparameter.
Revised: To stabilize the learning of concepts, the encoder outputs only the mean of Gaussian distribution, with the standard deviation as a hyperparameter.

Original: Using the process in Equation 1 for each concept, a concept-speciﬁc function parser aggregates the contextual information and transform it to the mean µ ag and standard deviation σ ag of g a .
Revised: Using the process in Equation 1 for each concept, a concept-speciﬁc function parser aggregates the contextual information and transform it to the mean µ cg and standard deviation σ cg of g c .

Original: The concept-speciﬁc target predictor takes g a ∼ N ( µ ag , diag ( σ 2 g )) and x t as inputs and output the mean µ az,t of z t , leaving the standard deviation as a hyperparameter.
Revised: The concept-speciﬁc target predictor takes g c ∼ N ( µ cg , diag ( σ 2 g )) and x t as inputs and output the mean µ cz,t of z t , leaving the standard deviation as a hyperparameter.

Original: To keep independence between concepts, CLAP-NP introduces identical but independent function parsers and target predictors.
Revised: To keep independence between concepts, CLAP-NP introduces identical but independent function parsers and target predictors.

Original: Once all of the concepts z at ∼ N ( µ az,t , σ 2 z I ) for a ∈ A are generated, we concatenate and decode them into target images:
Revised: Once all of the concepts z ct ∼ N ( µ cz,t , σ 2 z I ) for c ∈ C are generated, we concatenate and decode them into target images:

Original: [Equation]
Revised: [Equation]

Original: To control the noise in sampled images, we introduce a hyperparameter σ x as the standard deviation.
Revised: To control the noise in sampled images, we introduce a hyperparameter σ x as the standard deviation.

Original: As Figure 2 shows, we can rewrite the prior p ( h | y C , x ) in CLAP-NP as
Revised: As Figure 2 shows, we can rewrite the prior p ( h | y S , x ) in CLAP-NP as

Original: [Equation]
Revised: [Equation]

Original: Inference and Learning
Revised: Inference and Learning

Original: In the variational inference, other than that the randomness of f a is replaced by g a , the posterior of CLAP-NP is the same as that in Equation 3:
Revised: In the variational inference, other than that the randomness of f c is replaced by g c , the posterior of CLAP-NP is the same as that in Equation 3:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Figure]

Original: In the ﬁrst stage, we compute the mean of both z C and z T by the encoder.
Revised: In the ﬁrst stage, we compute the mean of both z S and z T by the encoder.

Original: Because CLAP shares the encoder in the prior and posterior, in the training process we can use the results in Equation 6 of the generative process instead of recalculating the mean of z C . 
Revised: Because CLAP shares the encoder in the prior and posterior, in the training process we can use the results in Equationof the generative process instead of recalculating the mean of z S . Then we compute the distribution [Table] 650.4 ± 12.5 GQN 414.3 ± 7.

Original: Then we compute the distribution parameters of g a by the shared concept-speciﬁc function parsers in generative process where the input in the variational inference are z a and x instead of z aC and x C .
Revised: 823.0 ± 15. 55.6 ± 0. 331.8 ± 2. 772.1 ± 3.

Revised: 1148.9 ± 3.2 CLAP-NP 119.5 ± 17.

Revised: 264.5 ± 27.

Revised: 58.8 ± 2.

Revised: 153.2 ± 6.

Revised: 105.5 ± 0.

Revised: 213.9 ± 1.

Revised: parameters of g c by the shared concept-speciﬁc function parsers in generative process where the input in the variational inference are z c and x instead of z cS and x S .

Original: With the above p ( h | y C , x ) and q ( h | y , x ) , some subterms in the ELBO of CLAP-NP become 
Revised: With the above p ( h | y S , x ) and q ( h | y , x ) , some subterms in the ELBO of CLAP-NP become L

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate the model’s abilityto parse the laws of individual concepts on different types of data, we use three datasets in the experiments: (1) bouncing ball (abbreviated as BoBa ) dataset (Lin et al., 2020a) to validate the ability of intuitive physics; (2) Continuous Raven’s Progressive Matrix (CRPM) dataset (Shi et al., 2021) to validate the ability of abstract visual reasoning; (3) MPI3D dataset (Gondal et al., 2019) to validate the ability of scene representation.
Revised: To evaluate the model’s ability of compositional law parsing on different types of data, we use three datasets in the experiments: (1) bouncing ball (abbreviated as BoBa ) dataset (Lin et al., 2020a) to validate the ability of intuitive physics; (2) Continuous Raven’s Progressive Matrix (CRPM) dataset (Shi et al., 2021) to validate the ability of abstract visual reasoning; (3) MPI3D dataset (Gondal et al., 2019) to validate the ability of scene representation.

Original: We use NP (Garnelo et al., 2018b) and GQN (Eslami et al., 2018) ascompared baselines.
Revised: We use NP (Garnelo et al., 2018b), GP with the deep kernel (Wilson et al., 2016), and GQN (Eslami et al., 2018) as baselines.

Original: Because CLAP-NP instantiates the latent random function with NP, we include NP  as one of the baselines. Although there are similar processes to model the randomness of functions  in NP andCLAP-NP, they have signiﬁcant differences: NP models the changing patterns of images directly; while CLAP-NP models the changing patterns of concepts in the latent space.
Revised: We consider NP and GP as baselines because they are representative random functions that construct the function space in different ways, and CLAP-NP instantiates the latent random function with NP.

Original: GQN is selected as a baseline since it models the distribution of image-valued functions, which highly ﬁts our experimental settings.
Revised: GQN is adopted since it models the distribution of image-valued functions, which highly ﬁts our experimental settings.

Original: Considering that NP is not used to model image-valued functions,  weneed to add an encoder and decoder to NP’s aggregator and predictor to help it handle high-dimensional images. 
Revised: NP is not used to model image-valued functions, so we add an encoder and decoder to NP’s aggregator and predictor to help it handle high-dimensional images. For GP, we use a pretrained encoder and decoder to reduce the dimension of raw images.

Original: See Appendix C.1 and C.2 for a detailed introduction to datasets, hyperparameters, and architectures of models.
Revised: See Appendix C.1 and C.2 for a detailed introduction to datasets, hyperparameters, and architectures of models.

Original: For quantitative experiments, we take Mean Squared Error (MSE) and Selection Accuracy (SA) as evaluation metrics.
Revised: For quantitative experiments, we take Mean Squared Error (MSE) and Selection Accuracy (SA) as evaluation metrics.

Original: Since non-compositional law parsing will increase the risk of producing 
Revised: Since non-compositional law parsing will increase the risk of producing undesired changes on constant concepts and further cause pixel deviations in predictions, we adopt MSE as an indicator of semantic error. To compute the concept-level deviations between predictions G T

Revised: N P

Revised: GQN

Original: [Figure]
Revised: C L A P - N P G P NP GQN

Revised: CLAP

Original: [Figure]
Revised: [Figure]

Revised: NP

Revised: Prediction

Revised: [Figure]

Revised: Prediction

Revised: Prediction

Revised: Prediction

Revised: Prediction

Revised: Prediction

Revised: Prediction

Revised: Prediction

Revised: Prediction

Revised: Prediction

Revised: Prediction

Revised: Prediction 2 P r e d i c ti m a g e s

Revised: P r e d i c t a r o w Test sample

Revised: GP

Revised: Prediction

Original:  We provide two predictions of the same context to illustrate a special case row  removed from the given matrix, all predictions with the progressive law of size and constant law of color are correct.
Revised: Prediction Prediction [Figure] We provide two predictions of the same context to illustrate a special case: when a row of images is removed from the given matrix, all predictions with the progressive law of size and constant law of color are correct.

Original: In this special case, only CLAP-NP captures such prediction uncertainty.
Revised: In this special case, only CLAP-NP captures such prediction uncertainty.

Original: undesired changes on constant concepts and further cause pixel deviations in predictions, we adopt MSE as an indicator of semantic error. To compute the concept-level deviations between predictions and ground truths, which we think can be an indicator of semantic error, we introduce SA as another metric to calculate the distance on the representation space.
Revised: and ground truths, which we think can be an indicator of semantic error, we introduce SA as another metric to calculate the distance on the representation space.

Original: We compute SA by picking out correct ground truths from a set of candidates because the representation spaces vary greatly among models.
Revised: We compute SA by picking out correct ground truths from a set of candidates because the representation spaces vary greatly among models.

Original: For a sample x , we ﬁrst combine its ground truth x T with those from other N s samples to establish the candidate set { c i } [-N s +
-]
Revised: For a sample x , we ﬁrst combine its ground truth x T with those from other K samples to establish the candidate set { d k } [-N s +
-]K +1 k

Original: i =1 .
Revised: =1 .

Original: Then these candidates and the prediction ˜ x T ∼ p ( x T | x C ) are encoded into representations to calculate concept-level distances (cid:107) Encoder (˜ x T ) − Encoder ( c i ) (cid:107) 22 .
Revised: Then these candidates and the prediction ˜ x T ∼ p ( x T | x S ) are encoded into representations to calculate concept-level distances (cid:107) Encoder (˜ x T ) − Encoder ( d k ) (cid:107) 22 .

Original: Finally, the candidate with the smallest distance is chosen, and SA is the selection accuracy across all test samples.
Revised: Finally, the candidate with the smallest distance is chosen, and SA is the selection accuracy across all test samples.

Original: In the following experiments, we use η = N T / N to denote the training or test conﬁguration where N T of N images in each sample are prediction targets.
Revised: In the following experiments, we use η = N T / N to denote the training or test conﬁguration where N T of N images in each sample are prediction targets.

Original: 5. I NTUITIVE P HYSICS
Revised: 5. I NTUITIVE P HYSICS

Original: To evaluate the intuitive physics of models, which is the underlying knowledge for humans to understand the evolution of the physical world (Kubricht et al., 2017), we use BoBa dataset where physical laws of collisions between colored bouncing balls are described as functions mapping timestamps to frames.
Revised: To evaluate the intuitive physics of models, which is the underlying knowledge for humans to understand the evolution of the physical world (Kubricht et al., 2017), we use BoBa dataset where physical laws of collisions between colored bouncing balls are described as functions mapping timestamps to frames.

Original: BoBa-1 and BoBa-2 arerespectively one-ball and two-ball instances of BoBa.
Revised: BoBa-1 and BoBa-2 are one-ball and two-ball instances of BoBa.

Original: In the prediction of target frames, models should learn both constant laws of appearance and physical laws of collisions.
Revised: When predicting target frames, models should learn both constant laws of appearance and physical laws of collisions.

Original: We set up two test conﬁgurations η = 4/12 and η = 6/12 for BoBa.
Revised: We set up two test conﬁgurations η = 4/12 and η = 6/12 for BoBa.

Original: The quantitative results in Table 1 and Figure 3 illustrate CLAP-NP’s intuitive physics on both BaBo-1 and BaBo-2.
Revised: The quantitative results in Table 1 and Figure 3 illustrate CLAP-NP’s intuitive physics on both BoBa-1 and BoBa-2.

Original: Figure 4 visualizes the predictions of BaBo-2 where NP can only generate blurred images, and GQNhas the potential to generate non-existent balls or lose existent balls. This is probably because GQN can hardly understand the law of appearance.
Revised: Figure 4 visualizes the predictions of BoBa-2 where NP can only generate blurred images while GP and GQN can hardly understand the appearance constancy of balls. That is, balls in the predictions of GP have different colors, and GQN generates non-existent balls and ignores existing balls.

Original: Instead, CLAP-NP can separately capture theconstant law of appearance  and physical laws of moving balls, which can beneﬁt the law parsing in scenesthat contain balls with novel shapes or colors (see Appendix D for experiment results).
Revised: Instead, CLAP-NP can separately capture the appearance constancy and collision laws. This beneﬁts the law parsing in scenes with novel appearances and physical laws (Appendix D.7).

Original: 5. A BSTRACT V ISUAL R EASONING
Revised: 5. A BSTRACT V ISUAL R EASONING

Original: Abstract visual reasoning is a manifestation of human’s ﬂuid intelligence (Cattell, 1963).
Revised: Abstract visual reasoning is a manifestation of human’s ﬂuid intelligence (Cattell, 1963).

Original: And RPM (Raven & Court, 1998) is a famous nonverbal test widely used to estimate the model’s abstract visual reasoning ability.
Revised: And RPM (Raven & Court, 1998) is a famous nonverbal test widely used to estimate the model’s abstract visual reasoning ability.

Original: RPM is a 3 × 3 image matrix, and the changing rule of the entire matrix consists[Figure] of multiple attribute-speciﬁc subrules.
Revised: RPM is a 3 × 3 image matrix, and the changing rule of the entire matrix consists of multiple attribute-speciﬁc subrules.

Original: In this experiment, we evaluate models on four instances of
Revised: In this experiment, we evaluate models on four instances of

Original: CRPM dataset and two test conﬁgurations η = 2/9 and η = 3/9.
Revised: CRPM dataset and two test conﬁgurations η = 2/9 and η = 3/9.

Original: In Table 1 and Figure 3, CLAP-NP achieves the best results with the metrics MSE and SA.
Revised: In Table 1 and Figure 3, CLAP-NP achieves the best results with the metrics MSE and SA.

Original: Figure 5 displays the predictions on CRPMDT.
Revised: Figure 5 displays the predictions on CRPMDT.

Original: In the results, NP may produce ambiguous-edged outer triangles,  and GQN generates very similar target images at all missing positions.
Revised: In the results, NP produces ambiguous-edged outer triangles, GP predicts targets with incorrect laws, and GQN generates multiple similar images.

Original: It is worth stressing that the answer to a CRPM can be not unique when we remove the entire line of images from it.
Revised: It is worth stressing that the answer to a CRPM is [Figure] not unique when we remove the entire line of images from it.

Original: We display examples of this situation in the second row of Figure 5, which indicates that CLAP-NP canindeed understand progressive and constant laws on different attributes, rather than simple context-to-target mappings.
Revised: We display examples of this situation in the second row of Figure 5, indicating that CLAP-NP can understand progressive and constant laws on different attributes rather than simple context-to-target mappings.

Original: 5. S CENE R EPRESENTATION
Revised: 5. S CENE R EPRESENTATION

Original: In scene representation, we use MPI3D dataset where ascene contains a robotic arm attached to the bottom of a 3D object  with different shapes, colors, and sizes.
Revised: In scene representation, we use MPI3D dataset where a robotic arm in scenes manipulates the altitude and azimuth of the object to produce images with different laws.

Original: The robotic arm will manipulate the altitude and azimuth of the object to produce image sequences by different laws.

Original: Such laws of scene images can be regarded asrandom functions that input altitudes or azimuths of the 3D object and output corresponding RGB images.
Revised: The laws of scenes can be regarded as functions that project object altitudes or azimuths to RGB images.

Original: Since the 3D object is moved along altitude or azimuth, the key to predicting correct target imagesin this task is to observe context images and ﬁnd out which law the scene follows.
Revised: In this task, the key to predicting correct target images is to determine which law the scene follows by observing given context images.

Original: All modelsare trained underthe conﬁguration η = 1 ∼ 4/ and tested via four conﬁgurations with more target and scene images ( η = 4/8, 6/8, 10/40, 20/40).
Revised: We train models under η = 1 ∼ 4/8 and test them with more target or scene images ( η = 4/8, 6/8, 10/40, 20/40).

Original: The MSE and SA scores in Table 1 and Figure 3 demonstrate that CLAP-NP outperforms NP in all conﬁgurations and GQN in η = 6/8, 10/40, 20/40.
Revised: The MSE and SA scores in Table 1 and Figure 3 demonstrate that CLAPNP outperforms NP in all conﬁgurations and GQN in η = 6/8, 10/40, 20/40.

Original: It is observed that CLAP-NP has a slightly higher SA but lower MSE scores than GQN when η = 4/8.
Revised: CLAP-NP has a slightly higher SA but lower MSE scores than GQN when η = 4/8.

Original: A possible reason is that the network of GQN has good inductive biases and more parameters to generate clearer images and achieve better MSE scores.
Revised: A possible reason is that the network of GQN has good inductive biases and more parameters to generate clearer images and achieve better MSE scores.

Original: But a such improvement in the pixel-level reconstruction has limited inﬂuence on theconcept-level representation learning, as well as the SA scores of GQN.
Revised: However, such improvement in the pixel-level reconstruction has limited inﬂuence on the representation space, as well as the SA scores of GQN.

Original: GQN can hardly generalize thelearned random functions to novel situations with more images, which leads toan obvious performance declining when η = 6/8, 10/40, 20/40.
Revised: GQN can hardly generalize the random functions to novel situations with more images, leading to performance loss when η = 6/8, 10/40, 20/40.

Original: However, compositionality allows
Revised: GP’s performance on MPI3D indicates that it is suitable for scenes with more context images (e.g., η =10/40). With only sparse points, the prediction uncertainty of GP will lead to a signiﬁcant performance loss.

Original:  CLAP-NP to parse concept-speciﬁc subrules, and in this case, it is more convenient for CLAP-NP to learn random functions with generalizability.
Revised: However, compositionality allows CLAP-NP to parse concept-speciﬁc subrules, which improves the model’s generalization ability in novel test conﬁgurations.

Original: 5. C OMPOSITIONALITY OF L AWS
Revised: 5. C OMPOSITIONALITY OF L AWS

Original: It is possible to exactly editing a law if one could ﬁnd out which latent random functions (LRFs) encode it.
Revised: It is possible to exactly editing a law if one could ﬁnd out which latent random functions (LRFs) encode it.

Original: Here we adopt two methods to ﬁnd the correspondences between laws and LRFs.
Revised: Here we adopt two methods to ﬁnd the correspondences between laws and LRFs.

Original: As Figure 6 shows, we can visualize the traversal in the latent space by perturbing and decoding concepts, and acquire the correspondences by observing changing patterns of these concepts.
Revised: As Figure 6 shows, we can visualize the traversal in the latent space by perturbing and decoding concepts, and acquire the correspondences by observing changing patterns of these concepts.

Original: Another way is to compute variance declines between laws and LRFs (Kim & Mnih, 2018).
Revised: Another way is to compute variance declines between laws and LRFs (Kim & Mnih, 2018).

Original: Assume that there are
Revised: Assume that there are

Original: M types of laws { l m } M m =1 .
Revised: M types of laws { l m } Mm =1 .

Original: For each law, we can sample an instance by ˜ l m ∼ p ( l m ) and generate a batch of samples B m ∼ p
Revised: For each law, we can sample an instance by ˜ l m ∼ p ( l m ) and generate a batch of samples B m ∼ p

Original: ( B m | ˜ l m ) that follows the law.
Revised: ( B m | ˜ l m ) that follows the law.

Original: First, we generate a batch of samples without ﬁxing any laws and parse LRFs { f a ; a ∈ A } for these samples to estimate the global variances
Revised: First, we generate a batch of samples without ﬁxing any laws and parse LRFs { f c ; c ∈ C } for these samples to estimate the global variances

Original: Var( f a ) .
Revised: Var( f c ) .

Original: Then we generate a series of batches from p ( B m | ˜ l m ) in which all the samples follow the ﬁx law ˜
Revised: Then we generate a series of batches from p ( B m | ˜ l m ) in which all the samples follow the ﬁx law ˜

Original: l m ∼ p ( l m ) and estimate the in-batch variances Var B m ( f a ) .
Revised: l m ∼ p ( l m ) and estimate the in-batch variances Var B m ( f c ) .

Original: Finally the variance decline between l m and f a is computed by s m,a = E p ( l m ) (cid:2) E p ( B m | l m )
Revised: Finally the variance decline between l m and f c is computed by s m,c = E p ( l m ) (cid:2) E p ( B m | l m )

Original: [(Var( f a ) − Var B m ( f a )) /
Revised: [(Var( f c ) − Var B m ( f c )) /

Original: Var( f a )] (cid:3) .
Revised: Var( f c )] (cid:3) .

Original: [Figure] In Figure 6, we display the variance declines of BoBa-2, CRPM-T, and CRPM-C, which will guide the manipulation of laws in the following experiments.
Revised: In Figure 6, we display the variance declines of BoBa-2, CRPM-T, and CRPM-C, which will guide the manipulation of laws in the following experiments.

Original: Latent Random Function Exchange
Revised: Latent Random Function Exchange

Original: The compositional modeling of laws allows us to swap the law of samples, which is shown at the top of Figure 7.
Revised: The compositional modeling of laws allows us to swap the law of samples, which is shown at the top of Figure 7.

Original: If we want to exchange the law of appearance on two samples of BaBo-2, we ﬁrst refer to the variance declines and ﬁnd that LRFs in the ﬁrst and last dimensions encode the law.
Revised: If we want to exchange the law of appearance on two samples of BoBa-2, we ﬁrst refer to the variance declines and ﬁnd that LRFs in the ﬁrst and last dimensions encode the law.

Original: Then we infer the LRFs of two samples (in CLAP-NP is to obtain the global latent variables) and swap LRFs of the ﬁrst and last dimensions.
Revised: Then we infer the LRFs of two samples (in CLAP-NP is to obtain the global latent variables) and swap LRFs of the ﬁrst and last dimensions.

Original: Finally, we execute the  generative process to regenerate the edited samples, and further, make predictions and interpolations on them.
Revised: Finally, we execute the [Figure] generative process to regenerate the edited samples, and further, make predictions and interpolations on them.

Original: Figure 7 visualizes the law manipulation process, which well illustrates the motivation of our model on compositional law parsing.
Revised: Figure 7 visualizes the law manipulation process, which well illustrates the motivation of our model on compositional law parsing.

Original: Latent Random Function Composition
Revised: Latent Random Function Composition

Original: For samples of CRPM, we manipulate laws by combining the LRFs from existing samples to generate samples with novel laws.
Revised: For samples of CRPM, we manipulate laws by combining the LRFs from existing samples to generate samples with novel laws.

Original: Take the  law composition on CRPM-T as an example, which is shown at the bottom of Figure 7, the ﬁrst step is to infer the
Revised: Figure 7 shows the results of law composition on CRPM-T.

Original:  LRFs foreach given sample.
Revised: The ﬁrst step is parsing LRFs for given samples.

Original: By querying the variance declines in Figure 6, we can ﬁnd these LRFs respectively correspond to the law of size, rotation, and color.
Revised: By querying the variance declines in Figure 6, we can ﬁnd these LRFs respectively correspond to the law of size, rotation, and color.

Original: Since we have ﬁgured out the relations between laws and LRFs, the next step is to pick laws from three different samples respectively and compose them into an unseen mixture of laws.
Revised: Since we have ﬁgured out the relations between laws and LRFs, the next step is to pick laws from three different samples respectively and compose them into an unseen mixture of laws.

Original: Through the generative process, we will obtain a sample with novel changing patterns.
Revised: Finally, the mixed laws are decoded to generate a sample with novel changing patterns.

Original: The processof law composition is similar to the way a human designs RPMs (i.e., sets up separate sub-laws for each attribute and combines them into a complete PRM).
Revised: This process is similar to the way a human designs RPMs (i.e., sets up separate sub-laws for each attribute and combines them into a complete PRM).

Original: And from another perspective, law composition provides an option for us to generate novel samples from existing samples (Lake et al., 2015).
Revised: From another perspective, law composition provides an option for us to generate novel samples from existing samples (Lake et al., 2015).

Original:  C ONCLUSION 
Revised:  C ONCLUSION AND L IMITATION

Original: Inspired by the compositionality in human cognition, we propose a deep latent variable model for
Revised: Inspired by the compositionality in human cognition, we propose a deep latent variable model for

Original: Compositional LAw Parsing (CLAP).
Revised: Compositional LAw Parsing (CLAP).

Original: CLAP decomposes high-dimensional images into independent visual concepts in the latent space and employs latent random functions to capture the conceptchanging laws, by which CLAP achieves the compositionality of laws.
Revised: CLAP decomposes high-dimensional images into independent visual concepts in the latent space and employs latent random functions to capture the conceptchanging laws, by which CLAP achieves the compositionality of laws.

Original: To instantiate CLAP, we propose CLAP-NP that uses NPs as latent random functions.
Revised: To instantiate CLAP, we propose CLAP-NP that uses NPs as latent random functions.

Original: The experimental results demonstrate the beneﬁts of our model in intuitive physics, abstract visual reasoning, and scene representation.
Revised: The experimental results demonstrate the beneﬁts of our model in intuitive physics, abstract visual reasoning, and scene representation.

Original: Through the experiments on latent random function exchange and composition, we further qualitatively evaluate the interpretability and manipulability of the learned functions in CLAP. 
Revised: Through the experiments on latent random function exchange and composition, we further qualitatively evaluate the interpretability and manipulability of the learned functions in CLAP. Limitations.

Revised: (1) Complexity of datasets.

Original: A A PPENDIX
Revised: Because compositional law parsing is an underexplored task, we ﬁrst validate the effectiveness of SVCL on datasets with relatively clear and simple rules to avoid the inﬂuence of unknown confounders in complicated datasets. (2) Setting the number of LRFs. For scenes with multiple complex rules, we can empirically set an appropriate upper bound or directly put a large upper bound on the number of LRFs. However, using too many LRFs increases model parameters, and the redundant concepts decrease CLAP’s computational efﬁciency.

Revised: See Appendix E for detailed limitations and future works.

Original: In Supplementary Materials, (1) we ﬁrst provide the details of ELBO, then (2) introduce the datasets, model architecture, hyperparameters, and computational resources adopted in our experiments, and ﬁnally (3) provide additional experimental results.
Revised: In Supplementary Materials, (1) we ﬁrst provide the details of ELBO, then (2) introduce the datasets, model architecture, hyperparameters, and computational resources adopted in our experiments, and ﬁnally (3) provide additional experimental results.

Original: In particular, we provide additional results of editing and manipulating latent random functions, which validate our motivation and contribution.
Revised: In particular, we provide additional results of editing and manipulating latent random functions, which validate our motivation and contribution.

Original: B D ETAILS OF ELBO B. ELBO OF C ONDITIONAL L ATENT V ARIABLE M ODELS
Revised: B D ETAILS OF ELBO B. ELBO OF C ONDITIONAL L ATENT V ARIABLE M ODELS

Original: [Equation]
Revised: [Equation]

Original: B. P RIOR AND P OSTERIOR F ACTORIZATION
Revised: B. P RIOR AND P OSTERIOR F ACTORIZATION

Original: p 
Revised: p ( y T | h ) = (cid:89) t ∈ T p

Original: [Equation]
Revised: [Equation]

Original:  | 
Revised: y t | z t ) p ( h | y S

Original: [Equation]
Revised: [Equation]

Original: t ∈ T
Revised: x ) = (cid:89) c ∈ C p

Original: [Equation]
Revised: [Equation]

Original: p
Revised: f

Original: [Equation]
Revised: [Equation]

Original: |
Revised: z

Original: [Equation]
Revised: [Equation]

Original: (cid:89) a ∈ A
Revised: z cT | y S

Original: [Equation]
Revised: [Equation]

Original: x 
Revised: x ) = (cid:89) c ∈ C p ( z cT | f

Original: [Equation]
Revised: [Equation]

Original: a ∈ A p
Revised: cS

Original: [Equation]
Revised: [Equation]

Original: xC , x T ) p ( f a | z aC , x C ) p ( z aC | y C )
Revised: x

Original: = (cid:89) a ∈ A (cid:32) p ( f a | z aC , x C ) (cid:89) t ∈ T p
Revised: [Equation]

Original: ( z at | f a , z aC , x C , x t )  (cid:89) c 
Revised: x [Equation] ) = (cid:89) c ∈ C [Equation]

Original: ∈ C p ( z ac | y c ) (cid:33) (12) q ( h | y, x ) =
Revised: [Equation] ( h [Equation] y

Original: (cid:89) a ∈
Revised: [Equation] x )

Original: A q ( f a , z aC , z
Revised: [Equation] c [Equation] C

Original: aT | y , x ) = (cid:89)
Revised: [Equation] ) = (cid:89)

Original: a ∈ A q ( f a | z a , x ) q ( z aC | y C ) q ( z aT | y T ) =
Revised: [Equation]

Original:  (cid:89) a ∈ A (cid:32) q
Revised: = (cid:89) [Equation] [Equation]

Original: ( f a | z a , x ) (cid:89) c ∈ C q ( z ac | y c ) (cid:89) t ∈ T q ( z at | y t ) (cid:33) (13)
Revised: [Equation] [Equation] (cid:33) (13)

Original: B. ELBO OF CLAP
Revised: B. ELBO OF CLAP

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original:  [Equation]
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: term

Original: [Equation]
Revised: [Equation]

Original: r

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: L f

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: B.3. R ECONSTRUCTION T ERM
Revised: B.3. R ECONSTRUCTION T ERM

Original: [Equation]
Revised: [Equation]

Original:  where ˜ z t ∼ q ( z t | y t ) (15)
Revised: z [Equation] , where ˜ z t ∼ q ( z t | y t ) (15)

Original: B.3. T ARGET R EGULARIZER
Revised: B.3. T ARGET R EGULARIZER

Original: [Equation]
Revised: [Equation]

Original: Because of the function consistency in samples
Revised: Because of the function consistency in samples

Original: [Equation]
Revised: [Equation]

Original: function posteriors q ( f a | ˜z a
Revised: function posteriors q ( f c | ˜

Original: [Equation]

Original:  x ) derived from arbitrary sampled ˜
Revised: z c , x ) derived from arbitrary sampled ˜

Original: z a ∼ q ( z a | y ) need to be consistent.
Revised: z c ∼ q ( z c | y ) need to be consistent.

Original: So we let f a condition on only the observations x
Revised: So we let f c condition on only the observations x

Original: and y to simplify the posterior distribution
Revised: and y to simplify the posterior distribution

Original: [Equation]
Revised: [Equation]

Original: that is, we replace q ( f a | ˜
Revised: that is, we replace q ( f c | ˜

Original: z a , x ) with q ( f a | y , x ) :
Revised: z c , x ) with q ( f c | y , x ) :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In this way
Revised: In this way

Original: [Equation]
Revised: [Equation]

Original: we convert the computation of the log-likelihoods on q ( z a t | y t ) and p ( z at | f a , z a C , x C , x t ) to the KL divergences between them.
Revised: we convert the computation of the log-likelihoods on q ( z c t | y t ) and p ( z ct | f c , z c S , x S , x t ) to the KL divergences between them.

Original: Then a Monte Carlo estimator can be used to approximate L t where ˜ z aC is sampled by ˜ z aC ∼ q ( z aC | y C ) to compute the outer expectation E q ( z aC | y C )
Revised: Then a Monte Carlo estimator can be used to approximate L t where ˜ z cS is sampled by ˜ z cS ∼ q ( z cS | y S ) to compute the outer expectation E q ( z cS | y S )

Original: [ ∗ ] ; and for inner expectation E q ( f a | y , x ) [ ∗ ] , because q ( f a | y , x ) = (cid:82) q 
Revised: [ ∗ ] ; and for inner expectation E q ( f c | y , x ) [ ∗ ] , because q ( f c | y , x ) = (cid:82) q ( f c |

Original: ( f a | z a , x ) q ( z a | x ) d z a , we are able to ﬁrst sample 
Revised: z c , x ) q ( z c | x ) d z c , we are able to ﬁrst sample ˜

Original: ˜ z aT ∼ q ( z aT | y T ) and obtain ˜ f a ∼ q ( f a | ˜ z aC , ˜ z
Revised: z cT ∼ q ( z cT | y T ) and obtain ˜ f c ∼ q ( f c | ˜ z cS , ˜ z

Original: aT , x ) .
Revised: cT , x ) .

Original: By means of ˜ f 
Revised: By means of ˜ f c and ˜ z

Original: a and ˜ z a sampled from the posterior, we have L t ≈ KL ( q ( z at | y t ) (cid:107) p ( z at | ˜ f a , ˜ z
Revised: c sampled from the posterior, we have L t ≈ KL ( q ( z ct | y t ) (cid:107) p ( z ct | ˜ f c , ˜ z

Original: a C , x C , x t )) .
Revised: c S , x S , x t )) .

Original: B.3. F UNCTION R EGULARIZER
Revised: B.3. F UNCTION R EGULARIZER

Original: [Equation]
Revised: [Equation]

Original: To estimate L f in the same way as the target regularizer, ˜ z a is sampled by ˜ 
Revised: To estimate L f in the same way as the target regularizer, ˜ z c is sampled by ˜ z

Original: z a ∼ q ( z a | y ) and we have L f ≈ KL ( q ( f a | ˜
Revised: c ∼ q ( z c | y ) and we have L f ≈ KL ( q ( f c | ˜

Original: z a , x ) (cid:107) p ( f a | ˜ z aC , x C )) .
Revised: z c , x ) (cid:107) p ( f c | ˜ z cS , x S )) .

Original: B.3. C ONTEXT R EGULARIZER
Revised: B.3. C ONTEXT R EGULARIZER

Original: [Equation]
Revised: [Equation]

Original: B. T OTAL C ORRELATION
Revised: B. T OTAL C ORRELATION

Original: Let I = { y 1 , ..., y K } be the set with images from all samples in the dataset, and
Revised: Let I = { y 1 , ..., y K } be the set with images from all samples in the dataset, and

Original: ˜ z
Revised: ˜ z

Original: k ∼ q ( z k | y k ) are the sampled latent variables for each image.
Revised: k ∼ q ( z k | y k ) are the sampled latent variables for each image.

Original: To apply the Total Correlation (TC) of βTCVAE
Revised: To apply the Total Correlation (TC) of βTCVAE

Original: (Chen et al., 2018) to CLAP, we make a tiny modiﬁcation that the aggregated posterior ¯ q ( z ) =
Revised: (Chen et al., 2018) to CLAP, we make a tiny modiﬁcation that the aggregated posterior ¯ q ( z ) =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =1 q ( z k | y k ) p ( y k ) is decomposed along with concepts instead of the dimension of latent variables.
Revised: Kk =1 q ( z k | y k ) p ( y k ) is decomposed along with concepts instead of the dimension of latent variables.

Original: According to β -TCVAE, there is
Revised: According to β -TCVAE, there is

Original: [Equation]
Revised: [Equation]

Original: where β -TCVAE provides Minibatch Weighted Sampling (MWS) to approximate Equation 20 by a minibatch { y m | m ∈ M } ⊆
Revised: where β -TCVAE provides Minibatch Weighted Sampling (MWS) to approximate Equation 20 by a minibatch { y m | m ∈ M } ⊆

Original: I of the dataset:
Revised: I of the dataset:

Original: [Equation]
Revised: [Equation]

Original: C D ATASETS AND E XPERIMENTAL S ETUP C. D ETAILS OF D ATASETS
Revised: C D ATASETS AND E XPERIMENTAL S ETUP C. D ETAILS OF D ATASETS

Original: In this paper, three types of datasets are adopted: BoBa, CRPM, and MPI3D. Figure 8 displays two samples for each instance of the datasets, and Table 2 describes the train, validation, and test conﬁgurations of the instances.
Revised: In this paper, three types of datasets are adopted: BoBa, CRPM, and MPI3D. Figure 8 displays two samples for each instance of the datasets, and Table 2 describes the train, validation, and test conﬁgurations of the instances.

Original: Bouncing Ball
Revised: Bouncing Ball

Original: (BoBa)
Revised: (BoBa)

Original: BoBa contains many videos of bouncing balls.
Revised: BoBa contains many videos of bouncing balls.

Original: Depending on the number of balls in the scene, the dataset provides two instances BoBa-1 and BoBa-2.
Revised: Depending on the number of balls in the scene, the dataset provides two instances BoBa-1 and BoBa-2.

Original: In the videos of Figure 8a and 8b, the motion of balls follows the law of physical collisions; and the appearance of balls (color, size, amount, etc.) is constant over time.
Revised: In the videos of Figure 8a and 8b, the motion of balls follows the law of physical collisions; and the appearance of balls (color, size, amount, etc.) is constant over time.

Original: The models need to capture the probability space on functions f : R (cid:55)→ R 3 × 64 × 64 that map timestamps to video frames.
Revised: The models need to capture the probability space on functions f : R (cid:55)→ R 3 × 64 × 64 that map timestamps to video frames.

Original: Referring to Table 2, we provide 10,000 videos of bouncing balls for training, 1,000 for validation and hyperparameter selection, and 2,000 to test the intuitive physics of the models.
Revised: Referring to Table 2, we provide 10,000 videos of bouncing balls for training, 1,000 for validation and hyperparameter selection, and 2,000 to test the intuitive physics of the models.

Original: In the training and validation phase, we randomly select 1 to 4 frames from video frames as the target images to predict; in the testing phase, we provide two conﬁgurations (referred to as Test-4 and Test-6) to evaluate the model’s performance when there are 4 and 6 target images in videos.
Revised: In the training and validation phase, we randomly select 1 to 4 frames from video frames as the target images to predict; in the testing phase, we provide two conﬁgurations (referred to as Test-4 and Test-6) to evaluate the model’s performance when there are 4 and 6 target images in videos.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  Progressive Matrix (CRPM)
Revised: Continuous Raven’s Progressive Matrix (CRPM)

Original: CRPM consists of 3 × 3 image matrices where the images contain one or two centered triangles or circles.
Revised: CRPM consists of 3 × 3 image matrices where the images contain one or two centered triangles or circles.

Original: CRPM provides four instances with different image types: CRPM-T, CRPM-DT, CRPM-C, and CRPM-DC.
Revised: CRPM provides four instances with different image types: CRPM-T, CRPM-DT, CRPM-C, and CRPM-DC.

Original: Images in a matrix follow attribute-speciﬁc changing rules (e.g., rules in the size, grayscale, and rotation of the triangle).
Revised: Images in a matrix follow attribute-speciﬁc changing rules (e.g., rules in the size, grayscale, and rotation of the triangle).

Original: The ﬁrst sample in Figure 8c shows that, for images within the row, the sizes of triangles increase progressively, the grayscales change progressively from dark to light, and the rotations keep constant.
Revised: The ﬁrst sample in Figure 8c shows that, for images within the row, the sizes of triangles increase progressively, the grayscales change progressively from dark to light, and the rotations keep constant.

Original: To predict the missing target images in the matrix, the models need to learn the probability space on functions f : {− 1 , 0 , 1 } 2 (cid:55)→ R 64 × 64 that map 2D grid coordinates to grid images.
Revised: To predict the missing target images in the matrix, the models need to learn the probability space on functions f : {− 1 , 0 , 1 } 2 (cid:55)→ R 64 × 64 that map 2D grid coordinates to grid images.

Original: Tableshows that we provide 10,000 image matrices for training, 1,000 for tuning the model hyperparameters, and 2,000 to test the abstract visual reasoning ability of models.
Revised: Tableshows that we provide 10,000 image matrices for training, 1,000 for tuning the model hyperparameters, and 2,000 to test the abstract visual reasoning ability of models.

Original: In the training and validation  process, we randomly select 1 or 2 images as target images; in the testing process, we use Testand Test-3 to evaluate the performance of the models to predict 2 and 3 target images.
Revised: In the training and validation [Table] process, we randomly select 1 or 2 images as target images; in the testing process, we use Testand Test-3 to evaluate the performance of the models to predict 2 and 3 target images.

Original: MPI3D
Revised: MPI3D

Original: MPI3D (Gondal et al., 2019) dataset contains a series of single-object scenes, each with 40 different scene images.
Revised: MPI3D (Gondal et al., 2019) dataset contains a series of single-object scenes, each with 40 different scene images.

Original: There are two underlying rules to change the object in a scene: change the altitude of the object (sample 1 in Figure 8g); change the azimuth of the object (sample 2 in
Revised: There are two underlying rules to change the object in a scene: change the altitude of the object (sample 1 in Figure 8g); change the azimuth of the object (sample 2 in

Original: Figure 8g).
Revised: Figure 8g).

Original: The other attributes (e.g., object color, camera height, etc.) do not change within scene images.
Revised: The other attributes (e.g., object color, camera height, etc.) do not change within scene images.

Original: It is essential for the models to grasp different changing patterns for target image prediction,[Table] and the key is to describe the distribution over functions f : R (cid:55)→ R 3 × 64 × 64 that map object altitudes or azimuths to scene images.
Revised: It is essential for the models to grasp different changing patterns for target image prediction, and the key is to describe the distribution over functions f : R (cid:55)→ R 3 × 64 × 64 that map object altitudes or azimuths to scene images.

Original: As Table 2 shows, we provide 16127 scenes for training, 2305 for tuning the hyperparameters, and 4608 for testing.
Revised: As Table 2 shows, we provide 16127 scenes for training, 2305 for tuning the hyperparameters, and 4608 for testing.

Original: In terms of training and validation, we ﬁrst randomly select 8 images from 40 scene images to represent the scene, and then randomly select 1 to 4 images from the 8 images again as the target images, leaving the remaining images as the context.
Revised: In terms of training and validation, we ﬁrst randomly select 8 images from 40 scene images to represent the scene, and then randomly select 1 to 4 images from the 8 images again as the target images, leaving the remaining images as the context.

Original: For testing, we supply Test-4, Test-6, Test-10, and Test-20, where Test-4 and Testevaluate the performance of the models to predict 4 and 6 target images out of 8 images; Test-10 and
Revised: For testing, we supply Test-4, Test-6, Test-10, and Test-20, where Test-4 and Testevaluate the performance of the models to predict 4 and 6 target images out of 8 images; Test-10 and

Original: Test-20 evaluate the performance to predict 10 and 20 target images out of 40 images.
Revised: Test-20 evaluate the performance to predict 10 and 20 target images out of 40 images.

Original: One can fetch
Revised: One can fetch

Original: MPI3D from the repository 1 under the Creative Commons Attribution 4.0 International License.
Revised: MPI3D from the repository 1 under the Creative Commons Attribution 4.0 International License.

Original: C. M ODEL A RCHITECTURE AND H YPERPARAMETERS 
Revised: C. M ODEL A RCHITECTURE AND H YPERPARAMETERS CLAP

Original: CLAP-NP
Revised: -NP

Original: In this subsection, we will ﬁrst describe the architecture of the encoder, decoder, concept-speciﬁc function parsers, and concept-speciﬁc target predictors network in CLAP-NP.
Revised: In this subsection, we will ﬁrst describe the architecture of the encoder, decoder, concept-speciﬁc function parsers, and concept-speciﬁc target predictors network in CLAP-NP.

Original: Then we will list hyperparameters in the training phase.
Revised: Then we will list hyperparameters in the training phase.

Original: • Encoder : for all datasets, CLAP-NP uses the same convolutional blocks to downsample highdimensional images into parameters of concepts, which can be described as – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 64 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 128 BatchNorm, ReLU – 4 × 4 Conv, 256 BatchNorm, ReLU – 1 × 1 Conv,– ReshapeBlock,– Fully Connected, | A |
Revised: • Encoder : for all datasets, CLAP-NP uses the same convolutional blocks to downsample highdimensional images into parameters of concepts, which can be described as – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 64 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 128 BatchNorm, ReLU – 4 × 4 Conv, 256 BatchNorm, ReLU – 1 × 1 Conv,– ReshapeBlock,– Fully Connected, | C |

Original: The ReshapeBlock ﬂattens the output tensor of size512 × 1 × 1 from the last convolutional layer to the vector of size 512 .
Revised: The ReshapeBlock ﬂattens the output tensor of size × 1 × 1 from the last convolutional layer to the vector of size 512 .

Original: The output of the encoder is the mean of | A | visual concepts.
Revised: The output of the encoder is the mean of | C | visual concepts

Original: • Decoder : in BoBa and CRPM datasets, CLAP
Revised: • Decoder : in BoBa and CRPM datasets, CLAP-NP uses several deconvolutional layers to upsample concepts to images, which is – ReshapeBlock, | C | × 1 ×– 1 × 1 Deconv, 128 BatchNorm, ReLU – 4 × 4 Deconv, 64 BatchNorm2d, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU

Original: NP uses several deconvolutional layers to
Revised: [Table]

Original: upsample concepts to images, which is – ReshapeBlock, | A | × 1 ×– 1 × 1 Deconv, 128 BatchNorm, ReLU – 4 × 4 Deconv, 64 BatchNorm2d, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, N C Sigmoid where N C is the number of image channels and Sigmoid is the activation function used to generate pixel values ranging in (0 , 1) .[Table]
Revised: – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, N Channel Sigmoid where N Channel is the number of image channels and Sigmoid is the activation function used to generate pixel values ranging in (0 , 1) .

Original: • Function Parser : CLAP-NP provides identical but independent function parsers for each concept.
Revised: • Function Parser : CLAP-NP provides identical but independent function parsers for each concept.

Original: Each function parser consists of a multi-layer perception (MLP) T c to encode concepts, a MLP T i to encode function inputs, a MLP T a to aggregate context points and a MLP T f to predict 
Revised: Each function parser consists of a multi-layer perception (MLP) T s to encode concepts, a MLP T i to encode function inputs, a MLP T a to aggregate context points, and a MLP T f to predict the mean and standard deviation of the global latent variable.

Original: the mean and standard deviation of the global latent variable.

Original: The architecture of T c and

Original:  T i is – Fully Connected, 32 ReLU – Fully Connected,
Revised: The architecture of T s and T i is – Fully Connected, 32 ReLU – Fully Connected,

Original: By concatenating the encoded concepts and the encoded function inputs, we get the context function points.
Revised: By concatenating the encoded concepts and the encoded function inputs, we get the context function points.

Original: The architecture of T a is
Revised: The architecture of T a is

Original: – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected,
Revised: – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected,

Original: Finally, the architecture of T f is
Revised: Finally, the architecture of T f is

Original: – Fully Connected, 256 ReLU – Fully Connected, 2 d g where d g is the size of the global latent variable g a .
Revised: – Fully Connected, 256 ReLU – Fully Connected, 2 d g where d g is the size of the global latent variable g c .

Original: The outputs of T f consist of the mean and standard deviation of the latent variable g a . • Target Predictor : the concept-speciﬁc target predictor T p is a MLP that maps g a and the encoded function inputinto the target concept, whose architecture is – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected,
Revised: The outputs of T f consist of the mean and standard deviation of the latent variable g c . • Target Predictor : the concept-speciﬁc target predictor T p is a MLP that maps g c and the encoded function input

Revised: into

Revised: the

Revised: target

Original:  To generate more complex scene images in MPI3D, CLAP-NP uses a deeper decoder where the hidden size is [512, 256, 128, 64, 32].
Revised: concept, whose architecture is – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected, To generate more complex scene images in MPI3D, CLAP-NP uses a deeper decoder where the hidden size is [512, 256, 128, 64, 32].

Original: And we use T f with the hidden size [128, 128] and output size 64, T f with the hidden size [64], and T p with the hidden size [128, 128].
Revised: And we use T f with the hidden size [128, 128] and output size 64, T f with the hidden size [64], and T p with the hidden size [128, 128].

Original: Hyperparameters for
Revised: Hyperparameters for

Original: CLAP-NP on different datasets are shown in Table 3.
Revised: CLAP-NP on different datasets are shown in Table 4.

Original: And in all datasets, we set learning rate as 3 × 10 − 4 , 
Revised: And in all datasets, we set learning rate as 3 × 10 − 4 , batch size as 512, and σ x = 0 . 1 .

Revised: For MPI3D, we anneal β t and β f in the ﬁrst 400 epochs.

Original: After each training epoch, we use the validation set to compute the evidence lower bound (ELBO) of the current model and save the model with the largest ELBO as the trained model.
Revised: After each training epoch, we use the validation set to compute the evidence lower bound (ELBO) of the current model and save the model with the largest ELBO as the trained model.

Original: For all datasets,
Revised: For all datasets,

Original: CLAP-NP uses the Adam (Kingma Ba 2015) optimizer to update parameters.
Revised: CLAP-NP uses the Adam (Kingma & Ba, 2015) optimizer to update parameters.

Original: Neural Process (NP)
Revised: Neural Process (NP)

Original: Different from (Garnelo et al., 2018b), to apply NP in high-dimensional images, we add an encoder and decoder which are the same as those in CLAP-NP to convert highdimensional images into low-dimensional representations.
Revised: Different from (Garnelo et al., 2018b), to apply NP in high-dimensional images, we add an encoder and decoder which are the same as those in CLAP-NP to convert highdimensional images into low-dimensional representations.

Original: And the representations are combined with the function inputs to form the context by an aggregator T a .
Revised: And the representations are combined with the function inputs to form the context by an aggregator T a .

Original: Then the function parser T f obtains the mean and standard deviation of the global latent variable.
Revised: Then the function parser T f obtains the mean and standard deviation of the global latent variable.

Original: Finally, the decoder generates  target images from the target function inputs and the global latent variable.
Revised: Finally, the decoder generates [Table] target images from the target function inputs and the global latent variable.

Original: The architectures of T a and T f are
Revised: The architectures of T a and T f are

Original: • Aggregator – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU[Table] – Fully Connected,• Function Parser – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU – Fully Connected, 2 d g
Revised: • Aggregator – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU – Fully Connected,• Function Parser – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU – Fully Connected, 2 d g

Original: Hyperparameters for NP on different datasets are shown in Table 4 where d r is the representation size of the encoder.
Revised: Hyperparameters for NP on different datasets are shown in Table 5 where d r is the representation size of the encoder.

Original: And for all datasets, NP adopts the learning rate 1 × 10 − 4 batch size 512, and σ x = 0 . 1 .
Revised: And for all datasets, NP adopts the learning rate 1 × 10 − 4 batch size 512, and σ x = 0 . 1 .

Original: For all datasets, NP uses the Adam optimizer for parameter updates.
Revised: For all datasets, NP uses the Adam optimizer for parameter updates.

Original: Generative Query Network (GQN)
Revised: Generative Query Network (GQN)

Original: To implement GQN (Eslami et al., 2018), we use a PyTorch implementation from the repository 2 with the following changes to the default conﬁguration: set learning rate to 0.0005, batch size to 256, the representation type to pool, the number of iterations to 4, and share the ConvLSTM core among iterations. 
Revised: To implement GQN (Eslami et al., 2018), we use a PyTorch implementation from the repository 2 with the following changes to the default conﬁguration: set learning rate to 0.0005, batch size to 256, the representation type to pool, the number of iterations to 4, and share the ConvLSTM core among iterations. Gaussian Process (GP)

Revised: We use a pretrained autoencoder to reduce the dimensionality of images, by which GP needs to capture the changing patterns in the latent space and thus can handle image-valued functions.

Revised: We adopt the same encoder and decoder as in CLAP-NP and NP for a fair comparison.

Revised: The mean function of GP is set to a constant function.

Original: C. C OMPUTATIONAL R ESOURCE
Revised: By comparing the RBF kernel, periodic kernel, and deep kernel, we ﬁnally chose the deep kernel as the kernel function that k ( x i , x j ) = σ 2 exp( −(cid:107)T k ( x i ) −T k ( x j ) (cid:107) / 2 (cid:96) 2 ) (Wilson et al., 2016). The architecture of the neural network T k in the deep kernel is a multilayer perceptron with hidden sizes [1000, 1000, 500, 50], output size 2, and ReLU activation functions, which is the same as in DKL (Wilson et al., 2016). The hyperparameters of GP are tuned on each sample through the log-likelihood of context points and the RMSprop optimizer.

Revised: We give the learning rate and epoch to adjust hyperparameters in Table 6.

Revised: To model the correlations between dimensions, we use multi-task Gaussian Process prediction (Bonilla et al., 2007).

Revised: C. C OMPUTATIONAL R ESOURCE

Original: We train our model and baselines on the server with Intel(R)
Revised: We train our model and baselines on the server with Intel(R)

Original: Xeon(R)
Revised: Xeon(R)

Original: Gold 6133 CPUs, 24GB
Revised: Gold 6133 CPUs, 24GB

Original: NVIDIA GeForce RTX 3090 GPUs, 512GB RAM, and Ubuntu 18.04 OS.
Revised: NVIDIA GeForce RTX 3090 GPUs, 512GB RAM, and Ubuntu 18.04 OS.

Original: All models are implemented with the PyTorch (Paszke et al., 2019) framework.
Revised: All models are implemented with the PyTorch (Paszke et al., 2019) framework.

Original: To avoid the randomness in dataset splitting, we ﬁx the random seed to 0. 
Revised: To avoid the randomness in dataset splitting, we ﬁx the random seed to 0. [Figure] [Table]

Original: D A DDITIONAL E XPERIMENTAL R ESULTS 
Revised: D A DDITIONAL E XPERIMENTAL R ESULTS D. I NTUITIVE P HYSICS

Original: Intuitive Physics

Original: In Figure 9 we display additional prediction results on BoBa-1 and BoBa2.
Revised: In Figure 9 we display additional prediction results on BoBa-1 and BoBa-2.

Original: For each instance, there are samples that correspond to Test-4 (left) and Test-6 (right), where
Revised: For each instance, there are samples that correspond to Test-4 (left) and Test-6 (right) [Table]

Original:  CLAP-NP outperforms other baselines.
Revised: where CLAP-NP outperforms other baselines.

Original: NP generates blurred target images on both BoBa-1 and BoBa-2 datasets.
Revised: NP generates blurred target images on both BoBa-1 and BoBa-2 datasets.

Original: It indicates that NP has difﬁculty modeling changing patterns on bouncing balls.
Revised: It indicates that NP has difﬁculty modeling changing patterns on bouncing balls.

Original: GQN can produce clear images but may generate non-existent balls and lose existing balls.
Revised: GQN can produce clear images but may generate non-existent balls and lose existing balls.

Original: In the ﬁrst sample of BoBa-1, the predictions of GQN deviate from the ground truths signiﬁcantly in the position of balls.
Revised: In the ﬁrst sample of BoBa-1, the predictions of GQN deviate from the ground truths signiﬁcantly in the position of balls.

Original: Overall.
Revised: Overall.

Original: CLAP-NP performs well in modeling scene consistency and predicting motion trajectory.
Revised: CLAP-NP performs well in modeling scene consistency and predicting motion trajectory.

Original: Abstract Visual Reasoning
Revised: D. A BSTRACT V ISUAL R EASONING

Original: With the additional experimental results in Figure 10, we can intuitively see that CLAP-NP has a better understanding of the changing rules than baselines.
Revised: With the additional experimental results in Figure [Table] we can intuitively see that CLAP-NP has a better understanding of the changing rules than baselines.

Original: For the second sample of each dataset, a row of images is removed to interpret the abstract visual reasoning ability of the model.
Revised: For the second sample of each dataset [Table] a row of images is removed to interpret the abstract visual reasoning ability of the model.

Original: As samples in Figure 10 shows, when we remove a row of images, the answer is not unique: the predictions are correct as long as their sizes increase progressively and their colors and rotations keep constant.
Revised: As samples in Figure 10 shows, when we remove a row of images, the answer is not unique: the predictions are correct as long as their sizes increase progressively and their colors and rotations keep constant.

Original: CLAP-NP represents such randomness in predictions by means of the probabilistic generative process, making it possible to generate different correct answers.
Revised: CLAP-NP represents such randomness in predictions by means of the probabilistic generative process, making it possible to generate different correct answers.

Original: In terms of the prediction quality, although generating blurred images, NP has the basic reasoning ability about the progressive rule of the outer triangle size in Figure 10b.
Revised: In terms of the prediction quality, although generating blurred images, NP has the basic reasoning ability about the progressive rule of the outer triangle size in Figure 10b.

Original: GQN generates clear images, however, the generated images probably deviate from the underlying changing rules on matrices.
Revised: GQN generates clear images, however, the generated images probably deviate from the underlying changing rules on matrices.

Original: Table 5 and Table 11 illustrate the outperforming results of CLAP-NP in abstract visual reasoning by quantitative experiments.
Revised: Table 7 and Table 11 illustrate the outperforming results of CLAP-NP in abstract visual reasoning by quantitative experiments.

Original: Scene Representation
Revised: D. S CENE R EPRESENTATION

Original: Figure 12 shows the prediction results within 8 scene images.
Revised: Figure 12 shows the prediction results within 8 scene images.

Original: Both GQN and CLAP-NP generate clear prediction results when the number of target images is 4; if the number 
Revised: Both GQN and CLAP-NP generate clear prediction results when the number of target images is 4; if the number of target images is increased to 6, the prediction results of GQN have signiﬁcant deviations, while CLAP-NP maintains the accuracy of predictions.

Original: [Figure] NP 
Revised: NP generates ambiguous results in all experiments.

Original:  GQN 
Revised: Figure 13 shows a more complicated situation: we provide 40 scene images and set 20 of them as target images. In this case, GQN and NP can hardly generate clear foreground objects; while CLAP-NP produces relatively accurate predictions with only a little decrease in generative quality.

Original:  CLAP-NP 
Revised: This experiment aims to test the generalization of the laws learned by the models, and the results above illustrate that NP can hardly represent scenes with functions, GQN has difﬁculty generalizing the scene representation ability to different conﬁgurations, and the laws learned by CLAP-NP can be generalized to novel scenes with more available images.

Original: Prediction Prediction P r e d i c t 2 i m a g e s P r e d i c t a r o w Test sample
Revised: D. C ONCEPT D ECOMPOSITION

Original: Prediction

Original: Prediction Prediction Prediction Prediction
Revised: Concept decomposition is the foundation for CLAP to understand concept-speciﬁc laws.

Original: Prediction Prediction Prediction Prediction
Revised: In this experiment, we traverse each concept in the latent space and visualize the concepts through a decoder to illustrate the LRF learned for each concept.

Original: Prediction(a) CRPM-T NP GQN CLAP-NP
Revised: First of all, we decompose a batch of images into

Original: Prediction Prediction P r e d i c t 2 i m a g e s P r e d i c t a r o w Test sample
Revised: [Figure]

Original: Prediction Prediction Prediction Prediction
Revised: CLAP

Original: Prediction

Original: Prediction Prediction Prediction Prediction
Revised: [Figure]

Original: Prediction(b) CRPM-DT

Original: NP
Revised: NP

Original: GQN CLAP-NP Prediction Prediction
Revised: [Figure]

Original: P r e d i c t 2 i m a g e s
Revised: P r e d i c t 2 i m a g e s

Original: P r e d i c t a ro w Test sample Prediction Prediction Prediction
Revised: P r e d i c t a r

Original: Prediction Prediction Prediction Prediction
Revised: [Figure]

Original: Prediction Prediction Prediction(c) CRPM-C NP
Revised: a)

Original: GQN CLAP-NP Prediction Prediction
Revised: [Figure]

Original: P r e d i c t 2 i m a g e s
Revised: P r e d i c t 2 i m a g e s

Original: P r e d i c t a r o wTest sample
Revised: P r e d i c t a r o w

Original: Prediction
Revised: [Figure]

Original: Prediction Prediction Prediction Prediction
Revised: i c t 2 i m

Original: Prediction
Revised: [Figure]

Original: Prediction Prediction Prediction Prediction
Revised: g

Original: [Figure]
Revised: [Figure]

Original: NP
Revised: NP

Original: GT
Revised: GT

Original: GQN

Original: CLAP-NP

Original: NP

Original: GT

Original: GQN
Revised: GQN

Original: [Figure]

Original: -NP

Original: NP

Original: GT

Original: GQN

Original: CLAP-NPNP
Revised: CLAP-NP

Original: GT

Original: GQN

Original: CLAP-NP

Original: NP
Revised: NP

Original: GT
Revised: GT

Original: GQN
Revised: GQN

Original: CLAP-NP

Original: NP

Original: GT

Original: GQN

Original: CLAP-NP
Revised: CLAP-NP

Original: [Figure] of target images is increased to 6, the prediction results of GQN have signiﬁcant deviations, while CLAP-NP maintains the accuracy of predictions. NP generates ambiguous results in all experiments.
Revised: GP

Original: Figure 13 shows a more complicated situation: we provide 40 scene images and set 20 of them as target images.

Original: In this case, GQN and NP can hardly generate clear foreground objects; while CLAPNP produces relatively accurate predictions with only a little decrease in generative quality.

Original: This experiment aims to test the generalization of the laws learned by the models, and the results above illustrate that NP can hardly represent scenes with functions, GQN has difﬁculty generalizing the scene representation ability to different conﬁgurations, and the laws learned by CLAP-NP can be generalized to novel scenes with more available images.

Original: Concept decomposition is the foundation for CLAP to understand concept-speciﬁc laws. In this experiment, we traverse each concept in the latent space and visualize the concepts through a decoder to illustrate the LRF learned for each concept. First of all, we decompose a batch of images into concepts by the encoder to estimate the range of concepts in the latent space.
Revised: GP concepts by the encoder to estimate the range of concepts in the latent space.

Original: To traverse one concept, we ﬁx the other concepts and linearly interpolate it from the minimum value to the maximum value to generate a sequence of interpolation results, which are decoded into images for visualization.
Revised: To traverse one concept, we ﬁx the other concepts and linearly interpolate it from the minimum value to the maximum value to generate a sequence of interpolation results, which are decoded into images for visualization.

Original: Each row of Figure 14 represents the latent traversal results of one concept.
Revised: Each row of Figure 14 represents the latent traversal results of one concept.

Original: In BoBa-1, CLAP-NP learns LRFs on concepts of color, horizontal position, and vertical position in an unsupervised manner.
Revised: In BoBa-1, CLAP-NP learns LRFs on concepts of color, horizontal position, and vertical position in an unsupervised manner.

Original: This is similar to the way we understand the motion of balls: the color keeps constant over time; the horizontal and vertical positions conform to the physical laws in their respective directions.
Revised: This is similar to the way we understand the motion of balls: the color keeps constant over time; the horizontal and vertical positions conform to the physical laws in their respective directions.

Original: For CRPM, CLAP-NP correctly parses images into concepts that correspond to the attribute-speciﬁc rules in matrices.
Revised: For CRPM, CLAP-NP correctly parses images into concepts that correspond to the attribute-speciﬁc rules in matrices.

Original: Concept decomposition in real environments is a challenge for models.
Revised: Concept decomposition in real environments is a challenge for models.

Original: For MPI3D, CLAP-NP parses the LRFs on the object axis (Dimension 1, 5, 6, 7) and other static scene attributes (Dimension 2, 3, 4).
Revised: For MPI3D, CLAP-NP parses the LRFs on the object axis (Dimension 1, 5, 6, 7) and other static scene attributes (Dimension 2, 3, 4).

Original: Latent Random Function Composition
Revised: D. L ATENT R ANDOM F UNCTION C OMPOSITION

Original: By means of concept decomposition, CLAP-NP can parse the latent random function of each concept to represent concept-speciﬁc laws, by which the law of a sample can be modiﬁed in terms of concepts.
Revised: By means of concept decomposition, CLAP-NP can parse the latent random function of each concept to represent concept-speciﬁc laws, by which the law of a sample can be modiﬁed in terms of concepts.

Original: Figure 15 and 16 display the way to edit the laws.
Revised: Figure 15 and 16 display the way to edit the laws.

Original: We compose the concept-speciﬁc LRFs of existing samples to generate samples that have novel changing patterns.
Revised: We compose the concept-speciﬁc LRFs of existing samples to generate samples that have novel changing patterns.

Original: For example, in the third sample of Figure 15a, we generate a sample with the progressive law of size, constant law of rotation, and constant law of color from three existing samples.
Revised: For example, in the third sample of Figure 15a, we generate a sample with the progressive law of size, constant law of rotation, and constant law of color from three existing samples.

Original: The results on four instances of CRPM indicate that CLAP-NP understands the changing patterns on matrices in an interpretable and manipulable way, which embodies the compositionality of laws.
Revised: The results on four instances of CRPM indicate that CLAP-NP understands the changing patterns on matrices in an interpretable and manipulable way, which embodies the compositionality of laws.

Original: Latent Random Function Exchange
Revised: D. L ATENT R ANDOM F UNCTION E XCHANGE

Original: Another way to edit laws is to exchange LRFs of some concepts between samples.
Revised: Another way to edit laws is to exchange LRFs of some concepts between samples.

Original: In BoBa, we swap the law of collisions and appearance between samples, and the results are shown in Figure 17.
Revised: In BoBa, we swap the law of collisions and appearance between samples, and the results are shown in Figure 17.

Original: According to the concept decomposition results in Figure 14, we exchange the LRFs on Dimension 1 (Dimensions 2 and 3) to modify the law of collisions (the law of appearance) in BoBa-1.
Revised: According to the concept decomposition results in Figure 14, we exchange the LRFs on Dimension 1 (Dimensions 2 and 3) to modify the law of collisions (the law of appearance) in BoBa-1.

Original: And we exchange the LRFs on Dimensions 1 and 6 (Dimensions 2, 3, 4, and 5) to modify the law of collisions (the law of appearance) in BoBa-2.
Revised: And we exchange the LRFs on Dimensions 1 and 6 (Dimensions 2, 3, 4, and 5) to modify the law of collisions (the law of appearance) in BoBa-2.

Original: To exchange laws in MPI3D, as shown in Figure 18, we swap LRFs on Dimensions 1, 5, 6, and 7 to change the law of the object axis.
Revised: To exchange laws in MPI3D, as shown in Figure 18, we swap LRFs on Dimensions 1, 5, 6, and 7 to change the law of the object axis.

Original: By modifying laws on example 1 of Figure 18, the changing pattern of the ﬁrst sample becomes the horizontal rotation, and the second sample inherits the law of vertical rotation from the ﬁrst sample, whichillustrates CLAP-NP’s ability to generate new samples. The experiments on law exchange evidence CLAP-NP’s interpretability and manipulability from another perspective. Generalization on Unseen Concepts We extend BoBa-2 to generate two novel datasets: BoBa2-color and BoBa-2-shape.
Revised: By modifying laws on example 1 of Figure 18, the changing pattern of the ﬁrst sample becomes the horizontal rotation, and the second sample inherits the law of vertical rotation from the ﬁrst sample, which

Original: Without changing the law of motion, we draw balls in BoBa-2-color with unseen colors and replace the balls in BoBa-2-shape with squares.

Original: After training on BoBa-2,

Original: CLAP-NP is tested by BoBa-2-color and BoBa-2-shape to evaluate whether the compositionality of laws helps generalization to unseen concepts. The experimental results in Figure 19 show that CLAP-NP correctly predicts the position of objects on both datasets because the compositionality makes the law of motion unrelated to the law of color or shape. However, the color and shape of objects deviate from those of real ones.
Revised: [Figure]

Original: This phenomenon is intuitive because the encoder and decoder are trained based on BoBa-2, whose latent space does not include unseen colors or shapes.

Original: But the correct prediction of trajectory indicates CLAP-NP’s ability to generalize the law of motion to objects with unseen colors and shapes. Failure Cases In this experiment, we display some failure cases. For BoBa, most failure cases occur when there are continuous target images (1st sample of BoBa in Figure 20) or too many target images (2nd sample of BoBa in Figure 20).
Revised: [Figure]

Original: For CRPM, CLAP can predict target images with diversity when we remove the third row from a matrix, but it sometimes generates target images that break the rules. For example, the color of images keeps invariant in the ﬁrst two rows, but CLAP generates images in the third row with changing grayscales (samples of CRPM in Figure 20). For MPI3D, when the object size has an obvious change (1st sample of MPI3D in Figure 20) or the centric object is tiny (2nd sample of MPI3D in Figure 20), the predictions will deviate or become unclear.
Revised: [Figure]

Original: + + = { Size 1, Rotation 1,
Revised: + + = { Size 1, Rotation 1,

Original: Color 1 } { Size 2, Rotation 2,
Revised: Color 1 } { Size 2, Rotation 2,

Original: Color 2 } { Size 3, Rotation 3,
Revised: Color 2 } { Size 3, Rotation 3,

Original: Color 3 } { Size 1, Rotation 2,
Revised: Color 3 } { Size 1, Rotation 2,

Original: Color 3 } + + = + + =
Revised: Color 3 } + + = + + =

Original: { Size 1, Rotation 1,
Revised: { Size 1, Rotation 1,

Original: Color 1 } { Size 2, Rotation 2,
Revised: Color 1 } { Size 2, Rotation 2,

Original: Color 2 } { Size 3, Rotation 3, Color 3 } { Size 1, Rotation 2,
Revised: Color 2 } { Size 3, Rotation 3, Color 3 } { Size 1, Rotation 2,

Original: Color 3 } { Size 1, Rotation 1,
Revised: Color 3 } { Size 1, Rotation 1,

Original: Color 1 } { Size 2, Rotation 2,
Revised: Color 1 } { Size 2, Rotation 2,

Original: Color 2 } { Size 3, Rotation 3,
Revised: Color 2 } { Size 3, Rotation 3,

Original: Color 3 } { Size 1, Rotation 2,
Revised: Color 3 } { Size 1, Rotation 2,

Original: Color 3 } (a) CRPM-T + + =
Revised: Color 3 } (a) CRPM-T + + =

Original: { Inner Color 1, Inner Rotation 1,
Revised: { Inner Color 1, Inner Rotation 1,

Original: Outer Size 1 } { Inner Color 2, Inner Rotation 2,
Revised: Outer Size 1 } { Inner Color 2, Inner Rotation 2,

Original: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,
Revised: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,

Original: Outer Size 3 } + + = + + = { Inner Color 1, Inner Rotation 1,
Revised: Outer Size 3 } + + = + + = { Inner Color 1, Inner Rotation 1,

Original: Outer Size 1 } { Inner Color 2, Inner Rotation 2,
Revised: Outer Size 1 } { Inner Color 2, Inner Rotation 2,

Original: Outer Size 2 } { Inner Color 3, Inner Rotation 3,
Revised: Outer Size 2 } { Inner Color 3, Inner Rotation 3,

Original: Outer Size 3 } { Inner Color 1, Inner Rotation 2, Outer Size 3 } { Inner Color 1, Inner Rotation 1,
Revised: Outer Size 3 } { Inner Color 1, Inner Rotation 2, Outer Size 3 } { Inner Color 1, Inner Rotation 1,

Original: Outer Size 1 } { Inner Color 2, Inner Rotation 2,
Revised: Outer Size 1 } { Inner Color 2, Inner Rotation 2,

Original: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,
Revised: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,

Original: Outer Size 3 }
Revised: Outer Size 3 }

Original: [Figure]
Revised: [Figure]

Original: + =
Revised: + =

Original: [Figure]
Revised: [Figure]

Original: + =
Revised: + =

Original: [Figure]
Revised: [Figure]

Original: + =
Revised: + =

Original: [Figure]
Revised: [Figure]

Original: + + =
Revised: + + =

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Law
Revised: Law

Original: [Figure]
Revised: [Figure]

Original: Law 2 }
Revised: Law 2 }

Revised: [Figure]

Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: illustrates CLAP-NP’s ability to generate new samples.

Revised: The experiments on law exchange evidence

Original: [Figure]
Revised: CLAP-NP’s interpretability and manipulability from another perspective. D. G ENERALIZATION ON U NSEEN C ONCEPTS We extend BoBa-2 to generate four datasets with novel laws: Novel-colors (BoBa-2-NC), Novelshape (BoBa-2-NS), Without-ball-collisions (BoBa-2-WBO), and With-gravity (BoBa-2-WG). We draw balls on BoBa-2-NC with unseen colors and BoBa-2-NS with squares without changing the laws of positions.

Revised: For BoBa-2-WBO, we disable collisions between balls and reserve collisions between balls and borders; for BoBa-2-WG, we add vertical gravity to scenes.

Revised: After training on BoBa-2, CLAP-NP is directly tested by four datasets to evaluate whether the compositionality of laws helps generalization to unseen concepts.

Revised: The experimental results in Figure 19 show that

Revised: CLAP-NP can still correctly predict the object positions on datasets with novel colors and shapes.

Revised: And the color and shape of objects deviate from those of real ones because the encoder and decoder are trained on BoBa-2, whose latent space does not include unseen colors or shapes.

Revised: The same situation can be observed on BoBa-2-WBO and BoBa-2-WG.

Revised: When confronted with unseen physical laws, CLAP-NP learns correct appearance constancy but the biased change of positions.

Revised: To better illustrate the generalization ability in scenes with novel laws, we quantitatively evaluate CLAP-NP and baseline models by MSE ad SA scores on the four datasets.

Revised: Table 8 shows MSE scores and SA scores with different numbers of candidates.

Revised: CLAP-NP achieves the best MSE and SA scores in most situations, which well illustrates CLAP-NP’s generalization ability in scenes with novel laws.

Revised: D. N UMBER OF L ATENT R ANDOM F UNCTIONS

Revised: This experiment explores the inﬂuence of setting too large or too small numbers of latent random functions (LRFs) on CLAP-NP.

Revised: Figure 20 shows the latent traversal results on MPI3D, BoBa-2, and

Revised: CRPM-DT.

Revised: If we set too few LRFs, CLAP-NP learns mixed laws among dimensions instead of the expected compositional laws.

Revised: Moreover, too few LRFs can inﬂuence CLAP-NP’s generation ability (e.g., set only two LRFs for BoBa-2).

Revised: Setting too many LRFs has no signiﬁcant inﬂuence on CLAPNP’s performance because there will be redundant dimensions in CLAP-NP that do not encode any information (e.g., dimensions 1, 3, and 6 on CRPM-DT).

Revised: However, due to the independent modeling

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: of concept-speciﬁc LRFs

Original: [Figure]
Revised: [Table]

Revised: setting a large number of LRFs will reduce the computational speed of

Revised: CLAP-NP and increase the number of model parameters.

Revised: D. P REDICTION S TRATEGY

Revised: CLAP uses the one-shot strategy to predict all target images at one time.

Revised: However, the rollout strategy can be another choice when predicting the following target images through a few context images.

Revised: For example, in the test conﬁguration η = 20 / 25 , we ﬁrst predict the 6th to 10th images from the given ﬁrst ﬁve images, then combine them (1st to 10th images) to predict the following ﬁve images

Original: [Figure]
Revised: [Table] and repeat this process until ﬁnishing the prediction of all target images. Table 9 shows the MSE scores on CRPM-DT, BoBa- [Table]

Revised: and MPI3D where CLAP-NP predicts targets by the rollout and one-shot strategy.

Revised: On BoBa-2 and MPI3D, the rollout strategy slightly improves the prediction accuracy.

Revised: CLAP is trained by predicting target images from randomly selected context images.

Revised: If a target image is distant from all context images, the prediction will have high uncertainty.

Revised: In this experiment, we only give the ﬁrst few images of the sample as context images while requiring the model to predict a large number of target images.

Revised: At this time, the rollout strategy only predicts a few target images in close positions each time, while the one-shot strategy has to predict the distant target images at one time.

Revised: Therefore, the rollout strategy can reduce uncertainty to improve the accuracy of the predictions.

Revised: D. F AILURE C ASES

Revised: In this experiment, we display some failure cases.

Revised: For BoBa, most failure cases occur when there are continuous target images (1st sample of BoBa in Figure 21) or too many target images (2nd sample of BoBa in Figure 21).

Revised: For CRPM, CLAP can predict target images with diversity when we remove the third row from a matrix, but it sometimes generates target images that break the rules.

Revised: For example, the color of images keeps invariant in the ﬁrst two rows, but CLAP generates images in the third row with changing grayscales (samples of CRPM in Figure 21).

Revised: For MPI3D, when the object size has an obvious change (1st sample of MPI3D in Figure 21) or the centric object is tiny (2nd sample of MPI3D in Figure 21), the predictions will deviate or become unclear.

Original: [Figure]
Revised: [Figure]

Revised: E L IMITATIONS

Revised: We conclude our limitations in two aspects.

Revised: (1) Complexity of datasets.

Revised: Because compositional law parsing is an underexplored task, we ﬁrst validate the effectiveness of SVCL on datasets with relatively clear and simple rules to avoid the inﬂuence of unknown confounders in complicated datasets.

Revised: We believe that the compositionality of laws also exists in more complex scenarios (e.g., learning physical laws in realistic scenes).

Revised: Furthermore, some vision tasks may beneﬁt from compositional law parsing.

Revised: For example, we can perform a controllable video generation process based on law modiﬁcation or make more interpretable predictions by analyzing dominant laws in videos.

Revised: Discovering such compositional law parsing ability in more complex situations can be a valuable topic in future works.

Revised: (2) Setting the number of LRFs.

Revised: CLAP places an upper bound on the number of LRFs.

Revised: For scenes with multiple complex rules, we can empirically set an appropriate upper bound or directly put a large upper bound for the number of LRFs.

Revised: However, a large bound will linearly increase the number of model parameters, and the redundant concepts will waste computing resources, decreasing computational efﬁciency.

Revised: Therefore, exploring a mechanism in the future to dynamically adjust the number of functions will be meaningful.

Revised: When applying CLAP to more complex scenes, we may introduce more inductive biases for better concept decomposition (e.g., use a more task-speciﬁc encoder or decoder).

Revised: We design CLAP as an unsupervised model independent of task-speciﬁc annotations from tasks, which means that it can be extended to take advantage of these annotations.

Revised: Thus we can ﬁnd ways to integrate CLAP with additional supervision information (e.g., supervise SVCL with the labels of changing factors for each scene) to help concept learning.