Original: C OMPOSITIONAL L AW P ARSING WITH L ATENT R ANDOM F UNCTIONS
Revised: C OMPOSITIONAL L AW P ARSING WITH L ATENT R ANDOM F UNCTIONS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Human cognition has compositionality.
Revised: Human cognition has compositionality.

Original: We understand a scene by decomposing the scene into different concepts (e.g., shape and position of an object) and learning the respective laws of these concepts, which may be either natural (e g., laws of motion) or man-made (e.g., laws of a game)
Revised: We understand a scene by decomposing the scene into different concepts (e.g., shape and position of an object) and learning the respective laws of these concepts, which may be either natural (e.g., laws of motion) or man-made (e.g., laws of a game).

Original: The automatic parsing of these laws indicates the model’s ability to understand the scene which makes law parsing play a central role in many visual tasks
Revised: The automatic parsing of these laws indicates the model’s ability to understand the scene, which makes law parsing play a central role in many visual tasks.

Original: This paper proposes a deep latent variable model for Compositional LAw Parsing (CLAP), which achieves the human -like compositionality ability through an encoding decoding architecture to represent concepts in the scene as latent variables.
Revised: This paper proposes a deep latent variable model for Compositional LAw Parsing (CLAP), which achieves the human-like compositionality ability through an encoding-decoding architecture to represent concepts in the scene as latent variables.

Original: CLAP employs concept-speciﬁc latent random functionsthat can be instantiated with Neural Processes to capture the law of concepts
Revised: CLAP employs conceptspeciﬁc latent random functions instantiated with Neural Processes to capture the law of concepts.

Original: Our experimental results demonstrate that CLAP outperforms the baseline methods in multiple visual tasks such as intuitive physics abstract visual reasoning, and scene representation
Revised: Our experimental results demonstrate that CLAP outperforms the baseline methods in multiple visual tasks such as intuitive physics, abstract visual reasoning, and scene representation.

Original: The law manipulation experiments illustrate CLAP’s interpretability by modifying speciﬁc latent random functions on samples.
Revised: The law manipulation experiments illustrate CLAP’s interpretability by modifying speciﬁc latent random functions on samples.

Original: For example, CLAP learns the laws of position-changing and appearance constancy from the moving balls in a scene, making it possible to exchange laws between samples or compose existing laws into novel laws.
Revised: For example, CLAP learns the laws of position-changing and appearance constancy from the moving balls in a scene, making it possible to exchange laws between samples or compose existing laws into novel laws.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Compositionality is an important feature of human cognition (Lake et al., 2017).
Revised: Compositionality is an important feature of human cognition (Lake et al., 2017).

Original: Humans can decompose a scene into individual concepts to learn the respective laws of these concepts, which can be either natural (e.g. laws of motion) or man-made (e.g. laws of a game).
Revised: Humans can decompose a scene into individual concepts to learn the respective laws of these concepts, which can be either natural (e.g. laws of motion) or man-made (e g. laws of a game).

Original: When observing a scene of a moving ball, one tends to parse the changing patterns of its appearance and position separately:
Revised: When observing a scene of a moving ball, one tends to parse the changing patterns of its appearance and position separately:

Original: The appearance stays consistent over time, while the position changes according to the laws of motion.
Revised: The appearance stays consistent over time, while the position changes according to the laws of motion.

Original: By composing the laws of the ball’s appearance and position, one can understand the changing pattern and predict the status of the moving ball.
Revised: By composing the laws of the ball’s appearance and position, one can understand the changing pattern and predict the status of the moving ball.

Original: Although compositionality has inspired a number of models in visual understanding such as representing handwritten characters through hierarchical decomposition of characters (Lake et al., 2011; 2015) and representing a multi-object scene with object-centric representations (Eslami et al., 2016;
Revised: Although compositionality has inspired a number of models in visual understanding such as representing handwritten characters through hierarchical decomposition of characters (Lake et al., 2011; 2015) and representing a multi-object scene with object-centric representations (Eslami et al., 2016;

Original: Kosiorek et al., 2018; Greff et al., 2019; Locatello et al., 2020), automatic parsing of laws in a scene is still a great challenge.
Revised: Kosiorek et al., 2018; Greff et al., 2019; Locatello et al., 2020), automatic parsing of laws in a scene is still a great challenge.

Original: For example, to understand the rules in abstract visual reasoning such as
Revised: For example, to understand the rules in abstract visual reasoning such as

Original: Raven’s Progressive Matrices (RPMs) test, the comprehension of attribute-speciﬁc representations and the underlying relationships among them is crucial for a model to predict the missing images (Santoro et al., 2018; Steenbrugge et al., 2018; Wu et al., 2020).
Revised: Raven’s Progressive Matrices (RPMs) test, the comprehension of attribute-speciﬁc representations and the underlying relationships among them is crucial for a model to predict the missing images (Santoro et al., 2018; Steenbrugge et al., 2018; Wu et al., 2020).

Original: To understand the laws of motion in intuitive physics, a model needs to grasp the changing patterns of different attributes (e.g. appearance and position) of each object in a scene to predict the future (Agrawal et al., 2016;
Revised: To understand the laws of motion in intuitive physics, a model needs to grasp the changing patterns of different attributes (e.g. appearance and position) of each object in a scene to predict the future (Agrawal et al., 2016;

Original: Kubricht et al., 2017; Ye et al., 2018).
Revised: Kubricht et al., 2017; Ye et al., 2018).

Original: However, these methods usually employ neural networks to directly model changing patterns of the scene in a black box, and can hardly abstract the laws of individual concepts in an explicit, interpretable and even manipulatable way.
Revised: However, these methods usually employ neural networks to directly model changing patterns of the scene in black-box fashion, and can hardly abstract the laws of individual concepts in an explicit, interpretable and even manipulatable way.

Original: A possible solution to enable the abovementioned ability for a model is exploiting a function to represent a law of concept in terms of the representation of the concept itself.
Revised: A possible solution to enable the abovementioned ability for a model is exploiting a function to represent a law of concept in terms of the representation of the concept itself.

Original: To represent a law that may depict arbitrary changing patterns, an expressive and ﬂexible random function is required.
Revised: To represent a law that may depict arbitrary changing patterns, an expressive and ﬂexible random function is required.

Original: The Gaussian Process (GP) is a classical family of random functions that achieves the diversity of function space through different kernel functions (Williams & Rasmussen, 2006).
Revised: The Gaussian Process (GP) is a classical family of random functions that achieves the diversity of function space through different kernel functions (Williams & Rasmussen, 2006).

Original: Recently pro- posed random functions (Garnelo et al., 2018b; Eslami et al., 2018; Kumar et al., 2018; Garnelo et al., 2018a; Singh et al., 2019; Kim et al., 2019; Louizos et al., 2019; Lee et al., 2020; Foong et al., 2020) describe function spaces with the powerful nonlinear ﬁtting ability of neural networks.
Revised: Recently proposed random functions (Garnelo et al., 2018b; Eslami et al., 2018; Kumar et al., 2018; Garnelo et al., 2018a; Singh et al., 2019; Kim et al., 2019; Louizos et al., 2019; Lee et al., 2020; Foong et al., 2020) describe function spaces with the powerful nonlinear ﬁtting ability of neural networks.

Original: These random functions have been used to capture the changing patterns in a scene, such as mapping timestamps to frames to describe the physical law of moving objects in a video (Kumar et al., 2018;
Revised: These random functions have been used to capture the changing patterns in a scene, such as mapping timestamps to frames to describe the physical law of moving objects in a video (Kumar et al., 2018;

Original: Singh et al., 2019; Fortuin et al., 2020).
Revised: Singh et al., 2019; Fortuin et al., 2020).

Original: However, these applications of random functions take images as inputs and the captured laws account for all pixels instead of expected individual concepts.
Revised: However, these applications of random functions take images as inputs and the captured laws account for all pixels instead of expected individual concepts.

Original: In this paper, we propose a deep latent variable model for Compositional LAw Parsing (CLAP).
Revised: In this paper, we propose a deep latent variable model for Compositional LAw Parsing (CLAP) 1 .

Original: CLAP achieves the human-like compositionality ability through an encoding-decoding architecture (Hamrick et al., 2018) to represent concepts in the scene as latent variables, and further employ concept-speciﬁc random functions in the latent space to capture the law on each concept.
Revised: CLAP achieves the human-like compositionality ability through an encoding-decoding architecture (Hamrick et al., 2018) to represent concepts in the scene as latent variables, and further employ concept-speciﬁc random functions in the latent space to capture the law on each concept.

Original: By means of the plug-in of different random functions, CLAP gains generality and ﬂexibility applicable to various law parsing tasks.
Revised: By means of the plug-in of different random functions, CLAP gains generality and ﬂexibility applicable to various law parsing tasks.

Original: We introduce CLAP-NP as an example that instantiates latent random functions with recently proposed Neural Processes (Garnelo et al., 2018b).
Revised: We introduce CLAP-NP as an example that instantiates latent random functions with recently proposed Neural Processes (Garnelo et al., 2018b).

Original: Our experimental results demonstrate that the proposed CLAP outperforms the compared baseline methods in multiple visual tasks including intuitive physics, abstract visual reasoning, and scene representation.
Revised: Our experimental results demonstrate that the proposed CLAP outperforms the compared baseline methods in multiple visual tasks including intuitive physics, abstract visual reasoning, and scene representation.

Original: In addition, the experiment on exchanging latent random functions on a speciﬁc concept and the experiment on composing latent random functions of the given samples to generate new samples both well demonstrate the interpretability and even manipulability of the proposed method for compositional law parsing.
Revised: In addition, the experiment on exchanging latent random functions on a speciﬁc concept and the experiment on composing latent random functions of the given samples to generate new samples both well demonstrate the interpretability and even manipulability of the proposed method for compositional law parsing.

Original:  R ELATED W ORK 
Revised:  R ELATED W ORK Compositional Scene Representation

Original: Since CLAP focuses on parsing compositional laws which are modeled by latent random functions, compositional scene representation and random functions are related to this paper.

Original: Compositional scene representation models  can understand a scene through object-centric representations.
Revised: Compositional scene representation models (Yuan et al., 2022a) can understand a scene through object-centric representations (Yuan et al., 2019b;a; 2021;

Revised: Emami et al., 2021; Yuan et al., 2022b).

Original: Some of them initialize and update objectcentric representations by iterative  processes such as Neural Expectation Maximization (Greff et al., 2017), iterative amortized inference (Greff et al., 2019),  iterative cross-attention
Revised: Some models initialize and update object-centric representations through iterative computational processes like neural expectation maximization (Greff et al., 2017), iterative amortized inference (Greff et al., 2019), and iterative cross-attention

Original: (Locatello et al., 2020), and two-stage inference (Emami et al., 2021).
Revised: (Locatello et al., 2020).

Original: Another class of models learns object-centric representations without iterative inference processes. AIR (Eslami et al., 2016) parses object-centric representations as latent variables. SPACE (Lin et al., 2020b) divides the image into regions and learns region-speciﬁc representations in parallel.
Revised: Other models adopt non-iterative computational processes to parse disentangled attributes for objects as latent variables (Eslami et al., 2016) or extract representations from evenly divided regions in parallel (Lin et al., 2020b).

Original: GNM (Jiang & Ahn, 2020) models the layout of scenes. Recently, many models aim at learning compositional scene representations in videos (Kosiorek et al., 2018; Jiang et al., 2019; Lin et al., 2020a).
Revised: Recently, many models focus on capturing layouts of scenes (Jiang & Ahn, 2020) or learning consistent object-centric representations from videos (Kosiorek et al., 2018; Jiang et al., 2019; Lin et al., 2020a).

Original: Unlike CLAP, compositional scene representation models learn how objects in a scene are composed but cannot explicitly represent underlying laws and understand how these laws constitute the changing pattern of scenes.
Revised: Unlike CLAP, compositional scene representation models learn how objects in a scene are composed but cannot explicitly represent underlying laws and understand how these laws constitute the changing pattern of the scene.

Original: Random Functions
Revised: Random Functions

Original: The GP (Williams & Rasmussen, 2006) is a classical family of random functions(Williams & Rasmussen, 2006) that regards the outputs of a function as a random variable of multivariate Gaussian distribution.
Revised: The GP (Williams & Rasmussen, 2006) is a classical family of random functions that regards the outputs of a function as a random variable of multivariate Gaussian distribution.

Original: GQN (Eslami et al., 2018) represents 3D scenes as functions by a well-designed representation network and generation network.
Revised: To incorporate neural networks with random functions, some models encode functions through global representations (Wu et al., 2018; Eslami et al., 2018; Gordon et al., 2019).

Original: NP (Garnelo et al., 2018b) captures function stochasticity by a Gaussian distributed latent variable instead of a deterministic representation in CNP (Garnelo et al., 2018a).
Revised: NP (Garnelo et al., 2018b) captures function stochasticity with a Gaussian distributed latent variable.

Original: Considering different ways of stochasticity modeling, context aggregation, and output prediction for NP, we can construct function spaces with different characteristics (Kim et al., 2019;
Revised: According to the way of stochasticity modeling, one can construct random functions with different characteristics (Kim et al., 2019;

Original: Louizos et al., 2019;Singh et al., 2019; Lee et al., 2020; Foong et al., 2020).
Revised: Louizos et al., 2019; Lee et al., 2020; Foong et al., 2020).

Original: Some few-shot learning methods parse functionsfrom data by learningglobal variables (Wu et al., 2018; Gordon et al., 2019) or adaptive kernels (Tossou et al., 2019;
Revised: And other models develop random functions by learning adaptive kernels (Tossou et al., 2019;

Original: Patacchiola et al., 2020). We can also combine latent variable models with ODEs  (Norcliffe et al., 2021) or SDEs (Li et al., 2020; Hasan et al., 2021) to describe random functions by the integral over latent states.
Revised: Patacchiola et al., 2020) or computing the integral of ODEs or SDEs on latent states (Norcliffe et al., 2021; Li et al., 2020; Hasan et al., 2021).

Original: Random functions provide explicit representations for laws but cannot model the compositionality of laws.
Revised: Random functions provide explicit representations for laws but cannot model the compositionality of laws.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: A random function describes a distribution over function space F , from which we can sample a function f mapping the input sequence ( x 1 , ..., x N ) to output sequence ( y 1 , ..., y N ) .
Revised: A random function describes a distribution over function space F , from which we can sample a function f mapping the input sequence ( x 1 , ..., x N ) to output sequence ( y 1 , ..., y N ) .

Original: The impor-
Revised: The impor-

Original: [Figure]
Revised: [Figure]

Original: tant problem for a random function is ﬁnding the underlying function over some given points and predicting function values at novel positions accordingly.
Revised: tant role of a random function is uncovering the underlying function over some given points and predicting function values at novel positions accordingly.

Original: In detail, given a set of context points
Revised: In detail, given a set of context points

Original: D S
Revised: D S

Original: = { ( x s , y s ) | s ∈ S } , we should ﬁnd the most possible function that y s = f
Revised: = { ( x s , y s ) | s ∈ S } , we should ﬁnd the most possible function that y s = f

Original: ( x s ) for these context points, then use f ( x t ) to predict y t for target points D T
Revised: ( x s ) for these context points, then use f ( x t ) to predict y t for target points D T

Original: = { ( x t , y t ) | t ∈ T } .
Revised: = { ( x t , y t ) | t ∈ T } .

Original: The abovementioned prediction problem can be regarded as estimating a conditional probability p ( y T | x T , D S ) .
Revised: The abovementioned prediction problem can be regarded as estimating a conditional probability p ( y T | x T , D S ) .

Original: Neural Process (NP)
Revised: Neural Process (NP)

Original: NP (Garnelo et al., 2018b) captures function stochasticity by a Gaussian distributed latent variable g and lets p ( y T | x T , D S )
Revised: NP (Garnelo et al., 2018b) captures function stochasticity with a Gaussian distributed latent variable g and lets p ( y T | x T , D S )

Original: = (cid:82) (cid:81) t ∈ T p ( y t | g , x t ) q ( g | D S )
Revised: = (cid:82) (cid:81) t ∈ T p ( y t | g , x t ) q ( g | D S )

Original: d g .
Revised: d g .

Original: In this case, p ( y T | x T , D S ) consists of a generative process p ( y t | g , x t ) and  inference process q ( g | D S ) , which can be jointly optimized by variational inference.
Revised: In this case, p ( y T | x T , D S ) consists of a generative process p ( y t | g , x t ) and an inference process q ( g | D S ) , which can be jointly optimized through variational inference.

Original: To estimate p ( y T | x T , D S ) , NPﬁrst extracts the feature of each context point in D S and sum them up to acquire the mean µ and deviation σ of g : µ
Revised: To estimate p ( y T | x T , D S ) , NP extracts the feature of each context point in D S and sum them up to acquire the mean µ and deviation σ of g : µ

Original: [Equation]
Revised: [Equation]

Original: Then NP samples the global latent variable g from q ( g | D S ) .
Revised: Then NP samples the global latent variable g from q ( g | D S ) .

Original: NP can independently predict y t at the target position x t by the generative process p ( y t | g , x t ) =
Revised: NP can independently predict y t at the target position x t through the generative process p ( y t | g , x t ) =

Original: N ( y t |T p ( g , x t ) , σ 2  I ) .
Revised: N ( y t |T p ( g , x t ) , σ 2 y I ) .

Original: Neural networks T a , T f and T p increase the model capacity of NP to learn data-speciﬁc function spaces.
Revised: Neural networks T a , T f and T p increase the model capacity of NP to learn data-speciﬁc function spaces.

Original:  M ODEL
Revised:  M ODEL

Original: Figure 1 is an overview of CLAP.
Revised: Figure 1 is the overview of CLAP.

Original: First, an encoder converts context images to individual concepts in the latent space.
Revised: The encoder converts context images to individual concepts in the latent space.

Original: Then CLAP parses concept-speciﬁc random functions on the  context visual concepts, which are exploited to predict the  target concepts.
Revised: Then CLAP parses concept-speciﬁc random functions on the concept representations of context images (context concepts) to predict the concept representations of target images (target concepts).

Original: Finally, a decoder maps the target concepts to target images.
Revised: Finally, a decoder maps the target concepts to target images.

Original: The concept-speciﬁc latent random functions achieve the compositional law parsing in CLAP.
Revised: The concept-speciﬁc latent random functions achieve the compositional law parsing in CLAP.

Original: In the following part, we will introduce the generative process, variational inference, and parameter learning of CLAP.
Revised: We will introduce the generative process, variational inference, and parameter learning of CLAP.

Original: At the end of this section, we propose CLAP-NP, an instance that instantiates the latent random function with NP.
Revised: At the end of this section, we propose CLAP-NP that instantiates the latent random function with the Neural Process (NP).

Original: 4. L ATENT R ANDOM F UNCTIONS
Revised: 4. L ATENT R ANDOM F UNCTIONS

Original: Given D S and D T , the objective of CLAP is to maximize log p ( y T | y S , x ) .
Revised: Given D S and D T , the objective of CLAP is maximizing log p ( y T | y S , x ) .

Original: In the framework of conditional latent variable models, we introduce a variational posterior q ( h | y , x ) for the latent variable h, and approximate  log-likelihood by evidence lower bound (ELBO) (Sohn et al., 2015):
Revised: In the framework of conditional latent variable models, we introduce a variational posterior q ( h | y , x ) for the latent variable h and approximate the log-likelihood with evidence lower bound (ELBO) (Sohn et al., 2015):

Original: [Equation]
Revised: [Equation]

Original: The latent variable h includes the  local information of images
Revised: The latent variable h includes the global latent random functions f and local information of images

Original: z = { z n } Nn =1 where z n is the low-dimensional representation of x n .
Revised: z = { z n } Nn =1 where z n is the low-dimensional representation of y n .

Original: Moreover, we decompose z n into independent concepts { z cn | c ∈ C } (e.g., an image of a single-object scene may have conceptssuch as object appearance,
Revised: Moreover, we decompose z n into independent concepts { z cn | c ∈ C } (e.g., an image of a single-object scene may have concepts

Original: 𝐶 𝑆 𝒚 !
Revised: 𝐶 𝑆 𝒚 !

Original: 𝒙 !
Revised: 𝒙 !

Original: 𝒛 !"
Revised: 𝒛 !"

Original: 𝒛 #" 𝒚 # 𝒙 # 𝑇
Revised: 𝒛 #" 𝒚 # 𝒙 # 𝑇

Original: Latent Random Function 𝑓 !
Revised: Latent Random Function 𝑓 !

Original: (a) CLAP 𝒙 !
Revised: (a) CLAP 𝒙 !

Original: 𝒛 !"
Revised: 𝒛 !"

Original: 𝒛 #" 𝒙 #
Revised: 𝒛 #" 𝒙 #

Original: NP 𝒈 " 𝒯 !"
Revised: NP 𝒈 " 𝒯 !"

Original: (b) NP as the latent random function illumination, camera orientation, etc.)

Original: where C refers to the name set of these concepts.

Original: Assuming that these concepts satisfy their respective change patterns, we employ independent latent random functions to capture the law of each concept.

Original: Denote the latent random function on the concept c as f c , the speciﬁc form of f c depends on the way we model it. As Figure 2b shows, if adopting NP as the latent random function, we have f c ( x t ) = T c p ( g c , x t ) where g c is a Gaussian-distributed latent variable to control the randomness of f c . Within the graphical model of CLAP in Figure 2a, the prior and posterior in L are factorized into
Revised: (b) NP as the latent random function

Original: [Equation]
Revised: [Equation]

Original: q [Equation] Decomposing f into concept-speciﬁc laws { f c | c ∈ C } in the above process is critical for CLAP’s compositionality.
Revised: Decomposing f into concept-speciﬁc laws { f c | c ∈ C } in the above process is critical for CLAP’s compositionality.

Original: The compositionality makes it possible to parse only several concept-speciﬁc laws rather than the entire law on z , which reduces the complexity of law parsing while increasing the interpretability of laws.
Revised: The compositionality makes it possible to parse only several concept-speciﬁc laws rather than the entire law on z , which reduces the complexity of law parsing while increasing the interpretability of laws.

Original: The interpretability allows us to edit laws, for example, exchanging the law on some concepts and composing the laws of existing samples.
Revised: The interpretability allows us to manipulate laws, for example, exchanging the law on some concepts and composing the laws of existing samples.

Original: 4. P ARAMETER L EARNING
Revised: 4. P ARAMETER L EARNING

Original: Based on Equation 3, we factorize  ELBO as (See Appendix B.3 for the detailed derivation)
Revised: Based on Equations 2 and 3, we factorize the ELBO as (See Appendix B.3 for the detailed derivation)

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Reconstruction Term
Revised: Reconstruction Term

Original: [Equation] is the sum ofthe log-likelihood log p ( y t | z t ) of each target image. p ( y t | z t ) is modelled by a decoderthat converts concepts to  the distribution parameter of target images.
Revised: L r is the sum of log p ( y t | z t ) , which is modeled with a decoder to reconstruct target images from the representation of concepts.

Original: CLAP maximizes L r to connect the latent space with the image space, enabling the decoder to generate high-quality images.
Revised: CLAP maximizes L r to connect the latent space with the image space, enabling the decoder to generate high-quality images.

Original: Target Regularizor
Revised: Target Regularizor

Original: L t consists of Kullback–Leibler (KL) divergences between the posterior q ( z ct | y t ) and the prior p ( z ct | f c , z cS , x S , x t ) of target concepts.
Revised: L t consists of Kullback–Leibler (KL) divergences between the posterior q ( z c t | y t ) and the prior p ( z c t | f c , z c S , x S , x t ) of target concepts.

Original: In q ( z ct | y t ), the distribution of target concepts iscomputed directly from the target image by the encoder, so it usually contains more accurate information. And in p ( z ct | f c , z cS , x S , x t ), the distribution of the target concept is estimated from the context.
Revised: The parameter of q ( z c t | y t ) is directly computed using the encoder to contain more accurate information about the target image while p ( z ct | f c , z cS , x S , x t ) is estimated from the context.

Original: The minimization of L t ﬁlls the gap between the posterior and prior and ensures the accuracy of predictors.
Revised: The minimization of L t ﬁlls the gap between the posterior and prior to ensure the accuracy of predictors.

Original: Function Regularizor
Revised: Context Regularizor

Revised: L s consists of KL divergences between the posterior q ( z cs | y s ) and the prior p ( z cs | y s ) of context concepts.

Revised: To avoid mismatch between the posterior and prior and reduce model parameters, the posterior and prior are parameterized with the same encoder.

Revised: In this case, we have p ( z cs | y s )

Revised: =

Revised: q ( z cs | y s ) and further L s = 0 .

Revised: So we remove L s from Equation 4.

Revised: Function Regularizor

Original: L f consists of KL divergences between the posterior q ( f c |
Revised: L f consists of KL divergences between the posterior q ( f c |

Original: z c , x ) and the prior p ( f c | z cS , x S ) .
Revised: z c , x ) and the prior p ( f c | z cS , x S ) .

Original: We use L f as a measure of function consistency to ensure that model can obtain similar distributions for f c with any subset z cS ⊆
Revised: We use L f as a measure of function consistency to ensure that model can obtain similar distributions for f c with any subset z cS ⊆

Original: z c and x S ⊆ x as inputs.
Revised: z c and x S ⊆ x as inputs.

Original: To this end, CLAP shares concept-speciﬁc function parsers to model q ( f c |
Revised: To this end, CLAP shares concept-speciﬁc function parsers to model q ( f c |

Original: z c , x ) and p ( f c | z cS , x S ) .
Revised: z c , x ) and p ( f c | z cS , x S ) .

Original: And we need to design the architecture of function parsers so that they can take different subsets as input.
Revised: And we need to design the architecture of function parsers so that they can take different subsets as input.

Original: Context Regularizor

Original: L s consists of KL divergences between the posterior q ( z cs | y s ) and the prior p ( z cs | y s ) of context concepts.

Original: To avoid the mismatch between the posterior and prior and reduce model parameters, the posterior and prior are parameterized with the same encoder.

Original: In this case, we have p ( z cs | y s )

Original: =

Original: q ( z cs | y s ) and further L s = 0 .

Original: So we remove L s from Equation 4.

Original: Correct decomposition of concepts is essential for model performance.
Revised: Correct decomposition of concepts is essential for model performance.

Original: However, CLAP is trained without additional supervision to guide the learning of concepts.
Revised: However, CLAP is trained without additional supervision to guide the learning of concepts.

Original: Although CLAP explicitly deﬁnes independent concepts in priors and posteriors, it is not sufﬁcient to automatically decompose concepts for some complex data.
Revised: Although CLAP explicitly deﬁnes independent concepts in priors and posteriors, it is not sufﬁcient to automatically decompose concepts for some complex data.

Original: To solve the problem, CLAP can introduce extra inductive biases  by designing proper encoder and decoder to artiﬁcially decompose concepts, e.g., use spatial transformation layers to decompose spatial and non-spatial concepts (Skafte & Hauberg, 2019).
Revised: To solve the problem, we can introduce extra inductive biases to promote concept decomposition by designing proper network architectures, e.g., using spatial transformation layers to decompose spatial and non-spatial concepts (Skafte & Hauberg, 2019).

Original: We can also set hyperparameters to control the regularizers (Higgins et al., 2017), or add a total correlation (TC) regularizer R TC to help concept learning (Chen et al., 2018).
Revised: We can also set hyperparameters to control the regularizers (Higgins et al., 2017), or add a total correlation (TC) regularizer R TC to help concept learning (Chen et al., 2018).

Original: Here we choose to add extra regularizers to  ELBO of CLAP, and ﬁnally, the optimization objective becomes
Revised: Here we choose to add extra regularizers to the ELBO of CLAP, and ﬁnally, the optimization objective becomes

Original: [Equation]
Revised: [Equation]

Original: (5) where β t , β f , β TC are hyperparameters that regulate the importance of regularizers in the training process.
Revised: [Equation] where β t , β f , β TC are hyperparameters that regulate the importance of regularizers in the training process.

Original: Wecan compute L ∗ by Stochastic Gradient Variational Bayes (SGVB) estimator (Kingma & Welling, 2013) and update parameters by gradient descent optimizers (Kingma & Ba, 2015).
Revised: We compute L ∗ through Stochastic Gradient Variational Bayes (SGVB) estimator (Kingma & Welling, 2013) and update parameters using gradient descent optimizers (Kingma & Ba, 2015).

Original: 4. NP AS L ATENT R ANDOM F UNCTION
Revised: 4. NP AS L ATENT R ANDOM F UNCTION

Original: In this subsection, we propose CLAP-NP as an example that instantiates the latent random function with NP.
Revised: In this subsection, we propose CLAP-NP as an example that instantiates the latent random function with NP.

Original: According to NP, we employ a latent variable g c to control the randomness of the latent random function f c for each concept in CLAP-NP.
Revised: According to NP, we employ a latent variable g c to control the randomness of the latent random function f c for each concept in CLAP-NP.

Original: Generative Process
Revised: Generative Process

Original: CLAP-NP ﬁrst encodes each context image x s to the concepts:
Revised: CLAP-NP ﬁrst encodes each context image y s to the concepts:

Original: [Equation]
Revised: [Equation]

Original: To stabilize the learning of concepts, the encoder outputs only the mean of Gaussian distribution, with the standard deviation as a hyperparameter.
Revised: To stabilize the learning of concepts, the encoder outputs only the mean of Gaussian distribution, with the standard deviation as a hyperparameter.

Original: Using the process in Equation 1 for each concept, a concept-speciﬁc function parser aggregates  the contextual informationand transform it to the mean µ cg and standard deviation σ cg of g c .
Revised: Using the process in Equation 1 for each concept, a concept-speciﬁc function parser aggregates and transforms the contextual information to the mean µ cg and standard deviation σ cg of g c .

Original: The concept-speciﬁc target predictor  takes g c ∼ N ( µ cg , diag ( σ 2 g )) and x t as inputs and output the mean µ cz,t of z t , leaving the standard deviation as a hyperparameter.
Revised: As Figure 2b shows, the concept-speciﬁc target predictor T cp takes g c ∼ N ( µ cg , diag ( σ 2 g )) and x t as inputs to predict the mean µ cz,t of z t , leaving the standard deviation as a hyperparameter.

Original: To keep independence between concepts, CLAP-NP introduces identical but independent function parsers and target predictors.
Revised: To keep independence between concepts, CLAP-NP introduces identical but independent function parsers and target predictors.

Original: Once all of the concepts z c t ∼ N ( µ c z,t , σ 2 z I ) for c ∈ C are generated, we concatenate and decode them into target images:
Revised: Once all of the concepts z ct ∼ N ( µ cz,t , σ 2 z I ) for c ∈ C are generated, we concatenate and decode them into target images:

Original: [Equation]
Revised: [Equation]

Original: To control the noise in sampled images, we introduce a hyperparameter σ x as the standard deviation.
Revised: To control the noise in sampled images, we introduce a hyperparameter σ y as the standard deviation.

Original: As Figure 2 shows, we can rewrite the prior p ( h | y S , x ) in CLAP-NP as
Revised: As Figure 2a shows, we can rewrite the prior p ( h | y S , x ) in CLAP-NP as

Original: [Equation]
Revised: [Equation]

Original: Inference and Learning
Revised: Inference and Learning

Original: In the variational inference, other than that the randomness of f c is replaced by g c , the posterior of CLAP-NP is the same as that in Equation 3:
Revised: In the variational inference, other than that the randomness of f c is replaced by g c , the posterior of CLAP-NP is the same as that in Equation 3:

Original: [Equation]
Revised: [Equation]

Original: In the ﬁrst stage, we compute the mean of both z S and z T by the encoder.
Revised: In the ﬁrst stage [Equation] we compute the means of both z S and z T using the encoder in CLAP.

Original: BecauseCLAP shares the encoder  in the prior and posterior,in the training process we can use the results in Equation 
Revised: Because the encoder is shared in the prior and posterior, we obtain z c s through Equation 6 instead of recalculating

Original: [Table]
Revised: [Table]

Revised: .6 ± 12.

Revised: 279.3 ± 8.

Original: 650.4 ± 12.5 GQN 414.3 ± 7.
Revised: 650.4 ± 12.5 GQN 414.3 ± 7.

Original: 823.0 ± 15.
Revised: 823.0 ± 15.

Original: 55.6 ± 0.
Revised: 55.6 ± 0.

Original: 331.8 ± 2.
Revised: 331.8 ± 2.

Original: 772.1 ± 3.
Revised: 772.1 ± 3.

Original: 1148.9 ± 3.2 CLAP-NP 119.5 ± 17.
Revised: 1148.9 ± 3.2 CLAP-NP 119.5 ± 17.

Original: 264.5 ± 27.
Revised: 264.5 ± 27.

Original: 58.8 ± 2.
Revised: 58.8 ± 2.

Original: 153.2 ± 6.
Revised: 153.2 ± 6.

Original: 105.5 ± 0.
Revised: 105.5 ± 0.

Original: 213.9 ± 1.of the generative process instead of recalculating the mean of z S .
Revised: 213.9 ± 1. z s ∼ q ( z cs | y s ) .

Original: Then we compute the distribution parameters of g c by the shared concept-speciﬁc function parsers in generative process where the inputin the variational inference are z c and xinstead of z cS and x S .
Revised: Then we compute the distribution parameters of g c through the shared conceptspeciﬁc function parsers and the input z c and x .

Original: With the above p ( h | y S , x ) and q ( h | y , x ) , some subterms in the ELBO of CLAP-NP become 
Revised: With the above generative and inference processes, some subterms in the ELBO of CLAP-NP become L

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate the model’s ability of compositional law parsing on different types of data, we use three datasets in the experiments: (1) bouncing ball (abbreviated as BoBa ) dataset (Lin et al., 2020a) to validate the ability of intuitive physics; (2) Continuous Raven’s Progressive Matrix (CRPM) dataset (Shi et al., 2021) to validate the ability of abstract visual reasoning; (3) MPI3D dataset (Gondal et al., 2019) to validate the ability of scene representation.
Revised: To evaluate the model’s ability of compositional law parsing on different types of data, we use three datasets in the experiments: (1) Bouncing Ball (abbreviated as BoBa ) dataset (Lin et al., 2020a) to validate the ability of intuitive physics; (2) Continuous Raven’s Progressive Matrix (CRPM) dataset (Shi et al., 2021) to validate the ability of abstract visual reasoning; (3) MPI3D dataset (Gondal et al., 2019) to validate the ability of scene representation.

Original: We use NP (Garnelo et al., 2018b), GP with the deep kernel (Wilson et al., 2016), and GQN (Eslami et al., 2018) as baselines.
Revised: We adopt NP (Garnelo et al., 2018b), GP with the deep kernel (Wilson et al., 2016), and GQN (Eslami et al., 2018) as baselines.

Original: We consider NP and GP as baselines because they are representative random functions that construct the function space in different ways, and CLAP-NP instantiates the latent random function with NP.
Revised: We consider NP and GP as baselines because they are representative random functions that construct the function space in different ways, and CLAP-NP instantiates the latent random function with NP.

Original: GQN is adopted since it models the distribution of image-valued functions, which highly ﬁts our experimental settings.
Revised: GQN is adopted since it models the distribution of image-valued functions, which highly ﬁts our experimental settings.

Original: NP is not used to model image-valued functions, so we add an encoder and decoder to NP’s aggregator and predictor to help it handle high-dimensional images.
Revised: NP is not used to model image-valued functions, so we add an encoder and decoder to NP’s aggregator and predictor to help it handle high-dimensional images.

Original: For GP, we use a pretrained encoder and decoder to reduce the dimension of raw images.
Revised: For GP, we use a pretrained encoder and decoder to reduce the dimension of raw images.

Original: See Appendix C.1 and
Revised: See Appendix D.1 and

Original: C.2 for a detailed introduction to datasets, hyperparameters, and architectures of models.
Revised: D.2 for a detailed introduction to datasets, hyperparameters, and architectures of models.

Original: For quantitative experiments, we take Mean Squared Error (MSE) and Selection Accuracy (SA) as evaluation metrics.
Revised: For quantitative experiments, we adopt Mean Squared Error (MSE) and Selection Accuracy (SA) as evaluation metrics.

Original: Since non-compositional law parsing will increase the risk of producing undesired changes on constant concepts and further cause pixel deviations in predictions, we adopt 
Revised: Since non-compositional law parsing will increase the risk of producing undesired changes on constant concepts and further cause pixel deviations in predictions, we adopt MSE as an indicator of prediction accuracy. To compute the concept-level deviations between predictions

Original: G T
Revised: G T

Original: N P
Revised: N P

Original: GQN
Revised: GQN

Original: C L A P - N P
Revised: C L A P - N P

Original: G P
Revised: G P

Original: NP

Original: GQN

Original: CLAP

Original: [Figure]
Revised: [Figure]

Original: NP Prediction Prediction Prediction
Revised: P r e d i c ti m a g e s

Original: Prediction

Original: Prediction

Original: Prediction

Original: Prediction Prediction Prediction Prediction
Revised: P r e d i c t a r o

Original: Prediction

Original: Prediction 2 P r e d i c ti m a g e s P r e d i c t a r o w Test sample GP Prediction
Revised: [Figure]

Original: Prediction Prediction Prediction We provide two predictions of the same context to illustrate a special case: when a row of images is removed from the given matrix, all predictions with the progressive law of size and constant law of color are correct.
Revised: We provide two predictions of the same context to illustrate a special case: when a row of images is removed from the given matrix, all predictions with the progressive law of size and constant law of color are correct.

Original: In this special case, only CLAP-NP captures such prediction uncertainty.
Revised: In this special case, only CLAP-NP captures such prediction uncertainty.

Original: MSE as an indicator of semantic error. To compute the concept-level deviations between predictions and ground truths, which we think can be an indicator of semantic error, we introduce SA as another metric to calculate the distance on the representation space.
Revised: and ground truths, which we think can be an indicator of semantic error, we introduce SA as another metric to calculate the distance on the representation space.

Original: We compute SA by picking out correct ground truths from a set of candidates because the representation spaces vary greatly among models.
Revised: Since the scale of representation-level distances is tightly according to the representation space of models, we turn to get SA by selecting target images from a set of candidates.

Original: For a sample x ,  weﬁrst combine its ground truth x T with those from otherK samples to establish the candidate set { d k } K +1 k =1 .
Revised: For ( x , y ) , we combine the target images y T with K targets randomly selected from other samples to establish a candidate set.

Original: Then these candidates and the prediction ˜ x T ∼ p ( x T | x S  )  are encoded into representations to calculate concept-level distances (cid:107) Encoder (˜ x T ) − Encoder ( d k ) (cid:107) 22 .
Revised: Then the candidates and the prediction ˜ y T ∼ p ( y T | y S , x ) of the models are encoded into representations to calculate concept-level distances.

Original: Finally, the candidate with the smallest distance is chosen, and SA is the selection accuracy across all test samples.
Revised: Finally, the candidate with the smallest distance is chosen, and SA is the selection accuracy across all test samples.

Original: In the following experiments, we use η = N T / N to denote the training or test conﬁguration where N T of N images in each sample are prediction targets.
Revised: In the following experiments, we use η = N T / N to denote the training or test conﬁguration where N T of N images in the sample are target images to predict.

Original: 5. I NTUITIVE P HYSICS
Revised: 5. I NTUITIVE P HYSICS

Original: To evaluate the intuitive physics of models, which is the underlying knowledge for humans to understand the evolution of the physical world (Kubricht et al., 2017), we use BoBa dataset where physical laws of collisions between colored bouncing balls are described as functions mapping timestamps to frames.
Revised: To evaluate the intuitive physics of models, which is the underlying knowledge for humans to understand the evolution of the physical world (Kubricht et al., 2017), we use BoBa dataset where physical laws of collisions and motions are described as functions mapping timestamps to frames.

Original: BoBa-1 and BoBa-2 are one-ball and two-ball instances of BoBa.
Revised: BoBa-1 and BoBa-2 are one-ball and two-ball instances of BoBa.

Original: When predicting target frames, models should learn both constant laws of appearance and physical laws of collisions.
Revised: When predicting target frames, models should learn both constant laws of appearance and physical laws of collisions.

Original: We set up two test conﬁgurations η = 4/12 and η = 6/12 for BoBa.
Revised: We set up two test conﬁgurations η = 4/12 and η = 6/12 for BoBa.

Original: The quantitative results in Table 1 and Figure 3 illustrate CLAP-NP’s intuitive physics on both BoBa-1 and BoBa-2.
Revised: The quantitative results in Table 1 and Figure 3 illustrate CLAP-NP’s intuitive physics on both BoBa-1 and BoBa-2.

Original: Figure 4 visualizes the predictions of BoBa-2 where NP can only generate blurred images while GP and GQN can hardly understand the appearance constancy of balls.
Revised: Figure 4 visualizes the predictions of BoBa-2 where NP can only generate blurred images while GP and GQN can hardly understand the appearance constancy of balls.

Original: That is, balls in the predictions of GP have different colors, and GQN generates non-existent balls and ignores existing balls.
Revised: That is, balls in the predictions of GP have different colors, and GQN generates non-existent balls and ignores existing balls.

Original: Instead, CLAP-NP can separately capture the appearance constancy and collision laws. This beneﬁts the law parsing in scenes with novel appearances and physical laws (Appendix D.7).
Revised: Instead, CLAP-NP can separately capture the appearance constancy and collision laws, which beneﬁts the law parsing in scenes with novel appearances and physical laws (Appendix E.7).

Original: 5. A BSTRACT V ISUAL R EASONING
Revised: 5. A BSTRACT V ISUAL R EASONING

Original: Abstract visual reasoning is a manifestation of human’s ﬂuid intelligence (Cattell, 1963).
Revised: Abstract visual reasoning is a manifestation of human’s ﬂuid intelligence (Cattell, 1963).

Original: And RPM (Raven & Court, 1998) is a famous nonverbal test widely used to estimate the model’s abstract visual reasoning ability.
Revised: And RPM (Raven & Court, 1998) is a famous nonverbal test widely used to estimate the model’s abstract visual reasoning ability.

Original: RPM is a 3 × 3 image matrix, and the changing rule of the entire matrix consists of multiple attribute-speciﬁc subrules.
Revised: RPM is a 3 × 3 image matrix, and the changing rule of the entire matrix consists of multiple attribute-speciﬁc subrules.

Original: In this experiment, we evaluate models on four instances of 
Revised: In this experiment, we evaluate models on four instances of the

Original: CRPM dataset and two test conﬁgurations η = 2/9 and η = 3/9.
Revised: CRPM dataset and two test conﬁgurations η = 2/9 and η = 3/9.

Original: In Table 1 and Figure 3, CLAP-NP achieves the best results with the metrics  MSE and SA.
Revised: In Table 1 and Figure 3, CLAP-NP achieves the best results with the metrics in both MSE and SA.

Original: Figure 5 displays the predictions on CRPMDT.
Revised: Figure 5 displays the predictions on CRPM-DT.

Original: In the results, NP produces ambiguous-edged outer triangles, GP predicts targets with incorrect 
Revised: In the results, NP produces ambiguous-edged outer triangles, GP predicts targets with incorrect laws, and GQN generates multiple similar images. It is worth stressing that the answer to

Original: [Figure]
Revised: [Figure]

Original: laws, and GQN generates multiple similar images. It is worth stressing that the answer to a CRPM is not unique when we remove the entire line of images from it.
Revised: an RPM is not unique when we remove the row from it.

Original: We display examples of this situation in the second row of Figure 5, indicating that CLAP-NP can understand progressive and constant laws on different attributes rather than simple context-to-target mappings.
Revised: We display examples of this situation in the second row of Figure 5, indicating that CLAP-NP can understand progressive and constant laws on different attributes rather than simple context-to-target mappings.

Original: 5. S CENE R EPRESENTATION
Revised: 5. S CENE R EPRESENTATION

Original: In scene representation, we use MPI3D dataset where a robotic arm in scenes manipulates the altitude and azimuth of the object to produce images with different laws.
Revised: In scene representation, we use MPI3D dataset where a robotic arm in scenes manipulates the altitude and azimuth of the object to produce images with different laws.

Original: The laws of scenes can be regarded as functions that project object altitudes or azimuths to RGB images.
Revised: The laws of scenes can be regarded as functions that project object altitudes or azimuths to RGB images.

Original: In this task, the key to predicting correct target images is to determine which law the scene follows by observing given context images.
Revised: In this task, the key to predicting correct target images is to determine which law the scene follows by observing given context images.

Original: We train models under η = 1 ∼ 4/8 and test them with more target or scene images ( η = 4/8, 6/8, 10/40, 20/40).
Revised: We train models under η = 1/8 ∼ 4/8 and test them with more target or scene images ( η = 4/8, 6/8, 10/40, 20/40).

Original: The MSE and SA scores in Table 1 and Figure 3 demonstrate that CLAPNP outperforms NP in all conﬁgurations and GQN in η = 6/8, 10/40, 20/40.
Revised: The MSE and SA scores in Table 1 and Figure 3 demonstrate that CLAP-NP outperforms NP in all conﬁgurations and GQN in η = 6/8, 10/40, 20/40.

Original: CLAP-NP has a slightly higher SA but lower MSE scores than GQN when η = 4/8.
Revised: CLAP-NP has a slightly higher SA but lower MSE scores than GQN when η = 4/8.

Original: A possible reason is that the network of
Revised: A possible reason is that the network of

Original: GQN has good inductive biases and more parameters to generate clearer images and achieve better
Revised: GQN has good inductive biases and more parameters to generate clearer images and achieve better

Original: MSE scores.
Revised: MSE scores.

Original: However, such improvement in the pixel-level reconstruction has limited inﬂuence on the representation space, as well as the SA scores of GQN.
Revised: However, such improvement in the pixel-level reconstruction has limited inﬂuence on the representation space, as well as the SA scores of GQN.

Original: GQN can hardly generalize the random functions to novel situations with more images, leading to performance loss when η = 6/8, 10/40, 20/40.
Revised: GQN can hardly generalize the random functions to novel situations with more images, leading to performance loss when η = 6/8, 10/40, 20/40.

Original: GP’s MSE scores on MPI3D indicate that it is suitable for scenes with more context images (e.g., η =10/40).
Revised: GP’s MSE scores on MPI3D indicate that it is suitable for scenes with more context images (e.g., η = 10/40).

Original: With only sparse points, the prediction uncertainty of GP leads to a signiﬁcant decrease in performance.
Revised: With only sparse points, the prediction uncertainty of GP leads to a signiﬁcant decrease in performance.

Original: However, compositionality allows CLAP-NP to parse concept-speciﬁc subrules, which improves the model’s generalization ability in novel test conﬁgurations.
Revised: However, compositionality allows CLAP-NP to parse concept-speciﬁc subrules, which improves the model’s generalization ability in novel test conﬁgurations.

Original: 5. C OMPOSITIONALITY OF L AWS
Revised: 5. C OMPOSITIONALITY OF L AWS

Original: It is possible to exactly editing a law if one could ﬁnd out which latent random functions (LRFs) encode it.
Revised: It is possible to exactly manipulate a law if one knows which latent random functions (LRFs) encode it.

Original: Here we adopt two methods to ﬁnd the correspondences between laws and LRFs.
Revised: We adopt two methods to ﬁnd the correspondence between laws and LRFs.

Original: As Figure 6 shows, we can visualize the traversal in the latent space by perturbing and decoding concepts, and acquire the correspondences by observing changing patterns of these concepts.
Revised: Figure 6 visualizes the changing patterns by perturbing and decoding concepts, enabling us to understand the meaning of LRFs.

Original: Another way is to compute variance declines between laws and LRFs (Kim & Mnih, 2018).
Revised: Another way is to compute variance declines between laws and LRFs (Kim & Mnih, 2018).

Original: Assume thatthere are M types of laws { l m } Mm =1 . For each law, we can sample an instance by ˜ l m ∼ p ( l m ) andgenerate a batch of samples B m ∼ p ( B m | ˜ l m ) that follows the law.
Revised: Assume that we have ground truth laws L and a generator to produce sample batches by ﬁxing the arbitrary law l ∈ L .

Original: First, we generate a batch of samples without ﬁxing any laws and parse LRFs{ f c ; c ∈ C } for these samples to estimate the global variances Var( f c ) .
Revised: First, we generate a batch of samples without ﬁxing any laws and parse LRFs for these samples to estimate the variance ¯ v c of each concept-speciﬁc LRF.

Original: Then we generate a series of batches from p ( B m | ˜ l m ) in which all the samples follow the ﬁx law ˜ l m ∼ p ( l m ) and estimate the in-batch variances Var B m ( f c ) .
Revised: By ﬁxing the law l , we generate some batches {B bl } N B b =1 and estimate the in-batch variance { v c,bl ; c ∈ C } of LRFs for B bl .

Original: Finally the variance decline between  lm and f c iscomputed by s m,c = E p ( l m ) (cid:2)
Revised: Finally, the variance decline between the law l and the concept c is

Original: E p ( B m | l m ) [(Var( f c ) − Var B m ( f c )) / Var( f c )] (cid:3) .
Revised: [Equation]

Original: In Figure 6, we display the variance declines of BoBa-2, CRPM-T, and CRPM-C, which will guide the manipulation of laws in the following experiments.
Revised: In Figure 6, we display the variance declines on BoBa-2, CRPM-T, and CRPM-C, which will guide the manipulation of laws in the following experiments.

Revised: And Figure 7 visualizes the law manipulation process, which well illustrates the motivation of our model on compositional law parsing.

Original: Latent Random Function Exchange
Revised: Latent Random Function Exchange

Original: The compositional modeling of laws allows us to swap the law of samples, which is shown at the top of Figure 7.
Revised: The top of Figure 7 shows how we swap the laws of samples with the aid of compositional law parsing.

Original: If we want to exchange the law of appearance on two samples of BoBa-2, we ﬁrst refer to the variance declinesand ﬁnd that LRFs in  the ﬁrst and last dimensions encode the law. Then we infer the LRFsof two samples (in CLAP-NP is to obtain
Revised: To exchange the law of appearance between samples, we ﬁrst refer to the variance declines in Figure 6. For BoBa-2, the laws are represented with 6 LRFs

Original: [Figure]
Revised: [Figure]

Original:  the global latent variables) and swap  LRFs of  the ﬁrst and last dimensions.
Revised: where the ﬁrst and last LRFs encode the law of appearance while the others indicate the motion of balls. Thus we infer the LRFs (global rule latent variables in CLAP-NP) of two samples and swap the LRFs of two samples on the ﬁrst and last dimensions.

Original: Finally, weexecute the generative process to regenerate the edited samples, and further, make predictions and interpolations on them.
Revised: Finally, we regenerate the edited samples using the generative process, and further, make predictions and interpolations on them.

Original: Figure 7 visualizes the law manipulation process, which well illustrates the motivation of our model on compositional law parsing.

Original: Latent Random Function Composition
Revised: Latent Random Function Composition

Original: For samples of CRPM, we manipulate laws by combining the LRFs from existing samples to generate samples with novel laws.
Revised: For samples of CRPM, we manipulate laws by combining the LRFs from existing samples to generate samples with novel laws.

Original: Figure 7 shows the results of law composition on CRPM-T.
Revised: Figure 7 shows the results of law composition on CRPM-T.

Original: The ﬁrst step is parsing LRFs for given samples.
Revised: The ﬁrst step is parsing LRFs for given samples.

Original: By querying the variance declines in Figure 6, we can ﬁnd these LRFs respectively correspond to the law of size, rotation, and color.
Revised: By querying the variance declines in Figure 6, we can ﬁnd these LRFs respectively correspond to the laws of size, rotation, and color.

Original: Since we have ﬁgured out the relations between laws and LRFs, the next step is to pick laws from three different samples respectively and compose them into an unseen mixture of laws.
Revised: Since we have ﬁgured out the relations between laws and LRFs, the next step is to pick laws from three different samples respectively and compose them into an unseen combination of laws.

Original: Finally, the mixed laws are decoded to generate a sample with novel changing patterns.
Revised: Finally, the composed laws are decoded to generate a sample with novel changing patterns.

Original: This process is similar to the way a human designs RPMs (i.e., sets up separate sub-laws for each attribute and combines them into a complete PRM).
Revised: This process is similar to the way a human designs RPMs (i.e., set up separate sub-laws for each attribute and combine them into a complete PRM).

Original: From another perspective, law composition provides an option for us to generate novel samples from existing samples (Lake et al., 2015).
Revised: From another perspective, law composition provides an option for us to generate novel samples from existing samples (Lake et al., 2015).

Original:  C ONCLUSION AND L IMITATIONS
Revised:  C ONCLUSION AND L IMITATIONS

Original: Inspired by the compositionality in human cognition, we propose a deep latent variable model for
Revised: Inspired by the compositionality in human cognition, we propose a deep latent variable model for

Original: Compositional LAw Parsing (CLAP).
Revised: Compositional LAw Parsing (CLAP).

Original: CLAP decomposes high-dimensional images into independent visual concepts in the latent space and employs latent random functions to capture the conceptchanging laws, by which CLAP achieves the compositionality of laws.
Revised: CLAP decomposes high-dimensional images into independent visual concepts in the latent space and employs latent random functions to capture the conceptchanging laws, by which CLAP achieves the compositionality of laws.

Original: To instantiate CLAP, we propose CLAP-NP that uses NPs as latent random functions.
Revised: To instantiate CLAP, we propose CLAP-NP that uses NPs as latent random functions.

Original: The experimental results demonstrate the beneﬁts of our model in intuitive physics, abstract visual reasoning, and scene representation.
Revised: The experimental results demonstrate the beneﬁts of our model in intuitive physics, abstract visual reasoning, and scene representation.

Original: Through the experiments on latent random function exchange and composition, we further qualitatively evaluate the interpretability and manipulability of the  learned functions in CLAP.
Revised: Through the experiments on latent random function exchange and composition, we further qualitatively evaluate the interpretability and manipulability of the laws learned by CLAP.

Original: Limitations.
Revised: Limitations.

Original: (1) Complexity of datasets.
Revised: (1) Complexity of datasets.

Original: Because compositional law parsing is an underexplored task, we ﬁrst validate the effectiveness of SVCL on datasets with relatively clear and simple rules to avoid the inﬂuence of unknown confounders in complicated datasets.
Revised: Because compositional law parsing is an underexplored task, we ﬁrst validate the effectiveness of CLAP on datasets with relatively clear and simple laws to avoid the inﬂuence of unknown confounders in complicated datasets.

Original: (2) Setting the number of LRFs.
Revised: (2) Setting the number of LRFs.

Original: For scenes with multiple complex rules, we can empirically set an appropriate upper bound or directly put a large upper bound on the number of LRFs.
Revised: For scenes with multiple complex laws, we can empirically set an appropriate upper bound or directly put a large upper bound on the number of LRFs.

Original: However, using too many LRFs increases model parameters, and the redundant concepts decrease CLAP’s computational efﬁciency.
Revised: However, using too many LRFs increases model parameters, and the redundant concepts decrease CLAP’s computational efﬁciency.

Original: See Appendix E for detailed limitations and future works.
Revised: See Appendix F for detailed limitations and future works.

Revised: A CKNOWLEDGMENTS

Revised: This work was supported in part by the National Natural Science Foundation of China (No.62176060), STCSM projects (No.20511100400, No.22511105000), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0103) and the Program for Professor of Special

Revised: Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In Supplementary Materials,(1) we ﬁrst provide the details of ELBO, then (2) introduce the datasets, model architecture, hyperparameters, and computational resources adopted in our experiments, and ﬁnally (3) provide additional experimental results.
Revised: In Supplementary Materials, we (1) provide the details of ELBO, then (2) introduce the datasets, model architecture, hyperparameters, and computational resources adopted in our experiments, and ﬁnally (3) provide additional experimental results.

Original: In particular, we provide additional results of editing and manipulating latent random functions, which validate our motivation and contribution.
Revised: In particular, we provide additional results of editing and manipulating latent random functions, which validate our motivation and contribution.

Original: B D ETAILS OF ELBO B. ELBO OF C ONDITIONAL L ATENT V ARIABLE M ODELS
Revised: B D ETAILS OF ELBO B. ELBO OF C ONDITIONAL L ATENT V ARIABLE M ODELS

Original: [Equation]
Revised: [Equation]

Original: B. P RIOR AND P OSTERIOR F ACTORIZATION
Revised: B. P RIOR AND P OSTERIOR F ACTORIZATION

Original: p( y T | h ) = (cid:89) t ∈ T p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: y t |z t ) p ( h | y S [Equation] x ) = (cid:89) c ∈ C p
Revised: y T |

Original: [Equation]
Revised: [Equation]

Original: f [Equation] z
Revised: t ∈ T p

Original: [Equation]
Revised: [Equation]

Original: z cT | y S
Revised: y t | z t

Original: [Equation]
Revised: [Equation]

Original: x ) = (cid:89) c ∈ C p ( z cT | f [Equation] cS [Equation]
Revised: p ( h | y S

Original: x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: ) = (cid:89) c ∈ C 
Revised: (cid:89) c ∈ C p

Original: [Equation]

Original: [Equation]

Original: ( h

Original: [Equation]
Revised: [Equation]

Original: y
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x ) [Equation] c 
Revised: = (cid:89) c ∈ C p

Original: [Equation]
Revised: [Equation]

Original: C [Equation] )= (cid:89) [Equation]
Revised: x T ) p ( f c | z cS , x S ) p ( z cS | y S )

Original: = (cid:89) [Equation] [Equation] [Equation]
Revised: = (cid:89) c ∈ C (cid:32) p ( f c | z cS , x S ) (cid:89) t ∈ T p ( z ct | f c , z cS , x S , x t ) (cid:89) s ∈ S p

Original: [Equation] (cid:33) (13)
Revised: ( z cs | y s ) (cid:33) q

Original: B. ELBO OF CLAP
Revised: ( h | y , x ) = (cid:89) c ∈ C q ( f c , z cS , z cT | y , x ) = (cid:89) c ∈ C q ( f c | z c , x ) q ( z cS | y S ) q

Original: [Equation] [Table]
Revised: ( z cT | y T ) =

Original: [Equation] [Table] [Equation] [Table]
Revised: (cid:89) c ∈ C (cid:32) q

Original: [Equation] [Table] [Equation] [Table]
Revised: ( f c |

Original: [Equation] [Table] [Equation] [Table]
Revised: z c , x ) (cid:89) s ∈ S q ( z cs | y s ) (cid:89)

Original: [Equation] [Table] [Equation] [Table]
Revised: t ∈ T q ( z ct | y t ) (cid:33) (13)

Original: [Equation] [Table] [Equation] [Table]
Revised: B. ELBO OF CLAP

Original: [Equation] [Table] [Equation] [Table]
Revised: L = E q ( h | y , x ) (cid:2) log (cid:89) t ∈ T p ( y t | z t ) (cid:3) − E q ( h | y , x ) (cid:34) log (cid:81)

Original: [Equation] [Table] [Equation]
Revised: c

Original: [Table] [Equation] [Table] [Equation]
Revised: ∈ C (cid:0) q ( f c |

Original: [Table] [Equation] [Table] [Equation]
Revised: z c , x ) (cid:81) s

Original: [Table] [Equation] [Table] [Equation]
Revised: ∈ S q ( z cs | y s ) (cid:81) t ∈ T q ( z ct | y t ) (cid:1) (cid:81) c ∈ C (cid:0)(cid:81) t ∈ T p ( z c t | f c , z c S , x S , x t ) p ( f c | z c S , x S ) (cid:81) s ∈ S p ( z c s | y s ) (cid:1) (cid:35) (14)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: B.3. R ECONSTRUCTION T ERM
Revised: B.3. R ECONSTRUCTION T ERM

Original: [Equation]
Revised: [Equation]

Original:  z 
Revised: ∈ [Equation] log p ( y t | ˜ z t )

Original: [Equation]
Revised: [Equation]

Original: , where ˜ z t ∼ q ( z t | y t ) (15)
Revised: where ˜ z t ∼ q ( z t | y t ) [Equation]

Original: B.3. T ARGET R EGULARIZER
Revised: B.3. T ARGET R EGULARIZER

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Because of the function consistency in samples
Revised: Because of the function consistency in samples

Original: [Equation]
Revised: [Equation]

Original: function posteriors q ( f c | ˜
Revised: function posteriors q ( f c | ˜

Original: z c , x ) derived from arbitrary sampled ˜
Revised: z c , x ) derived from arbitrary sampled ˜

Original: z c ∼ q ( z c | y ) need to be consistent.
Revised: z c ∼ q ( z c | y ) need to be consistent.

Original: So we let f c condition on only the observations x
Revised: So we let f c condition on only the observations x

Original: and y to simplify the posterior distribution [Equation] that is, we replace q ( f c | ˜
Revised: and y to simplify the posterior distribution, that is, we replace q ( f c | ˜

Original: z c , x ) with q ( f c | y , x ) :
Revised: z c , x ) with q ( f c | y , x ) :

Original: [Equation]
Revised: [Equation]

Original: [Equation] In this way [Equation] we convert the computation of the log-likelihoods on q ( z c t | y t ) and p ( z ct | f c , z c S , x S , x t ) to the KL divergences between them.
Revised: In this way, we convert the computation of the log-likelihoods on q ( z ct | y t ) and p ( z ct | f c , z cS , x S , x t ) to the KL divergences between them.

Original: Then a Monte Carlo estimator can be used to approximate
Revised: Then a Monte Carlo estimator can be used to approximate

Original: L t where ˜ z cS is sampled by ˜
Revised: L t where ˜ z cS is sampled by ˜

Original: z cS ∼ q ( z cS | y S ) to compute the outer expectation E q ( z cS | y S )
Revised: z cS ∼ q ( z cS | y S ) to compute the outer expectation E q ( z cS | y S )

Original: [ ∗ ] ; and for inner expectation E q ( f c | y , x ) [ ∗ ] , because q ( f c | y , x ) =
Revised: [ ∗ ] ; and for inner expectation E q ( f c | y , x ) [ ∗ ] , because q ( f c | y , x ) =

Original: (cid:82) q ( f c |
Revised: (cid:82) q ( f c |

Original: z c , x ) q ( z c | x ) d z c , we are able to ﬁrst sample ˜
Revised: z c , x ) q ( z c | y ) d z c , we are able to ﬁrst sample ˜

Original: z cT ∼ q ( z cT | y T ) and obtain ˜ f
Revised: z cT ∼ q ( z cT | y T ) and obtain ˜ f

Original: c ∼ q ( f c | ˜ z cS , ˜ z
Revised: c ∼ q ( f c | ˜ z cS , ˜ z

Original: cT , x ) .
Revised: cT , x ) .

Original: By means of ˜ f c and ˜ z
Revised: By means of ˜ f c and ˜ z

Original: c sampled from the posterior, we have L t ≈ KL ( q ( z ct | y t ) (cid:107) p ( z ct | ˜ f c ,˜ z
Revised: c sampled from the posterior, we have L t ≈ KL ( q ( z ct | y t ) (cid:107) p ( z ct | ˜ f c ,

Original: c S , x S , x t )) .
Revised: ˜ z cS , x S , x t )) .

Revised: B.3. C ONTEXT R EGULARIZER

Revised: [Equation]

Revised: [Equation]

Original: B.3. F UNCTION R EGULARIZER
Revised: B.3. F UNCTION R EGULARIZER

Original: [Equation]
Revised: [Equation]

Original: To estimate L f in the same way as the target regularizer,
Revised: To estimate L f in the same way as the target regularizer,

Original: ˜
Revised: ˜

Original: z c is sampled by ˜ z c ∼ q ( z c | y ) and we have L f ≈ KL ( q ( f c | ˜
Revised: z c is sampled from q ( z c | y ) and we have L f ≈ KL ( q ( f c | ˜

Original: z c , x ) (cid:107) p ( f c | ˜
Revised: z c , x ) (cid:107) p ( f c | ˜

Original: z cS , x S )) .
Revised: z cS , x S )) .

Original: B.3. C ONTEXT R EGULARIZER

Original: [Equation]

Original: B. T OTAL C ORRELATION
Revised: B. T OTAL C ORRELATION

Original: Let I = { y 1 , ..., y K } be the set with images from all samples in the dataset, and
Revised: Let I = { y k } Kk =1 denote the set of all sample images in the dataset, and

Original: ˜ z
Revised: ˜ z

Original: k ∼ q ( z k | y k ) are thesampled latent variables for each image.
Revised: k ∼ q ( z k | y k ) are the latent representations of concepts for the k th image.

Original: To apply the Total Correlation (TC)of βTCVAE (Chen et al., 2018) to CLAP, we make a tiny modiﬁcation that the aggregated posterior ¯ q ( z ) = (cid:80)
Revised: To apply the Total Correlation (TC) (Chen et al., 2018) in CLAP, we decompose the aggregated posterior ¯ q ( z ) = (cid:80)

Original: Kk =1 q ( z k | y k ) p ( y k )is decomposed along with concepts instead of the dimension of latent variables.
Revised: K k = 1 q ( z k | y k ) p ( y k )

Original: According to β -TCVAE, there is
Revised: according to the concepts, that is

Original: [Equation]
Revised: [Equation]

Original: where β -TCVAE provides Minibatch Weighted Sampling (MWS) to approximate Equation 20 by a minibatch { y m| m ∈ M }  ⊆
Revised: We adopt Minibatch Weighted Sampling (Chen et al., 2018) to approximate R TC on the minibatch { y m } Mm =1 ⊆

Original: I of the dataset:
Revised: I of the dataset and the corresponding representations of concepts { ˜ z m } Mm =1 :

Original: [Equation]
Revised: [Equation]

Revised: C P RELIMINARIES

Revised: Generative Query Network (GQN)

Revised: GQN (Eslami et al., 2018) regards the mappings from camera poses to scene images as random functions.

Revised: GQN adopts deterministic neural scene representations r and latent variables z to capture the conﬁguration and stochasticity of scenes in the conditional probability p ( y T | x T , y C , x C )

Revised: =

Revised: (cid:82) p ( y T , z | x T , r ) d z .

Revised: The representation network T a extracts the representation for each context point and summarizes them as the global scene representation r = (cid:80) c ∈ C T a ( y c , x c ) .

Revised: The generation network T p predicts the mean of target scene images via µ =

Revised: T p ( r , z , x t ) , and the target scene images are sampled from N ( y t | µ , σ 2 I ) .

Revised: Gaussian Process (GP)

Revised: GP (we only discuss noise-free GP here) models the probability distribution p ( y | x ) as the multivariate Gaussian y ∼ N ( 0 , K ) where K ij is computed through the kernel function κ ( x i , x j ) .

Revised: In this case, the probability p ( y T | x T , y C , x C ) is a multivariate Gaussian, and the parameters have closed-form solutions.

Revised: Kernel functions are the key to constructing different types of random functions.

Revised: The models can use a basic kernel like the RBF kernel, combine different kernels to develop complex kernels, or learn the function space for different data adaptively through neural networks (Wilson et al., 2016).

Original: C D ATASETS AND E XPERIMENTAL S ETUP C. D ETAILS OF D ATASETS
Revised: D D ATASETS AND E XPERIMENTAL S ETUP D. D ETAILS OF D ATASETS

Original: In this paper, three types of datasets are adopted: BoBa, CRPM, and MPI3D. Figure 8 displays two samples for each instance ofthe datasets, and Table 2 describes the train, validation, and test conﬁgurations of the instances. 
Revised: In this paper, three types of datasets are adopted: BoBa, CRPM, and MPI3D. Figure 8 displays two samples for each instance of datasets, and Table 2 describes the train, validation, and test conﬁgurations of all instances. [Figure] [Table] [Table]

Original: Bouncing Ball (BoBa)
Revised: Bouncing Ball (BoBa)

Original: BoBa containsmany videos of bouncing balls.
Revised: BoBa contains videos of bouncing balls.

Original: Depending on the number of balls in the scene, the dataset provides two instances BoBa-1 and BoBa-2.
Revised: Depending on the number of balls in the scene, the dataset provides two instances BoBa-1 and BoBa-2.

Original: In the videos of Figure 8a and 8b, the motion of balls follows the law of physical collisions; and the appearance of balls (color, size, amount, etc.) is constant over time.
Revised: In the videos of Figure 8a and 8b, the motion of balls follows the law of physical collisions; and the appearance of balls (color, size, amount, etc.) is constant over time.

Original: The models need to capture the probability space on functions f : R (cid:55)→ R 3 × 64 × 64 that map timestamps to video frames.
Revised: The models need to capture the probability space on functions f : R (cid:55)→ R 3 × 64 × 64 that map timestamps to video frames.

Original: Referring to Table 2, we provide 10,000 videos of bouncing balls for training, 1,000 for validation and hyperparameter selection, and 2,000 to test the intuitive physics of the models.
Revised: Referring to Table 2, we provide 10,000 videos of bouncing balls for training, 1,000 for validation and hyperparameter selection, and 2,000 to test the intuitive physics of the models.

Original: In the training and validation phase, we randomly select 1 to 4 frames from video frames as the target images to predict; in the testing phase, we provide two conﬁgurations (referred to as Test-4 and Test-6) to evaluate the model’s performance when there are 4 and 6 target images in videos.[Figure] [Table]
Revised: In the training and validation phase, we randomly select 1 to 4 frames from video frames as the target images to predict; in the testing phase, we provide two conﬁgurations (referred to as Test-4 and Test-6) to evaluate the model’s performance when there are 4 or 6 target images in videos.

Original: [Table]
Revised: [Table]

Original: Continuous Raven’s Progressive Matrix (CRPM)
Revised: Continuous Raven’s Progressive Matrix (CRPM)

Original: CRPM consists of 3 × 3 image matrices where the images contain one or two centered triangles or circles.
Revised: CRPM consists of 3 × 3 image matrices where the images contain one or two centered triangles or circles.

Original: CRPM provides four instances with different image types: CRPM-T, CRPM-DT, CRPM-C, and CRPM-DC.
Revised: CRPM provides four instances with different image types: CRPM-T, CRPM-DT, CRPM-C, and CRPM-DC.

Original: Images in a matrix follow attribute-speciﬁc changing rules (e.g., rules in the size, grayscale, and rotation of the triangle).
Revised: Images in a matrix follow attribute-speciﬁc changing rules (e.g., rules in the size, grayscale, and rotation of the triangle).

Original: The ﬁrst sample in Figure 8c shows that, for images within the row, the sizes of triangles increase progressively, the grayscales change progressively from dark to light, and the rotations keep constant.
Revised: The ﬁrst sample in Figure 8c shows that, in the row of the matrix, the sizes of triangles increase progressively, the grayscales change progressively from dark to light, and the rotations keep constant.

Original: To predict the missing target images in the matrix, the models need to learn the probability space on functions f : {− 1 , 0 , 1 } 2 (cid:55)→ R 64 × 64 that map 2D grid coordinates to grid images.
Revised: To predict the missing target images in the matrix, the models need to learn the probability space on functions f : {− 1 , 0 , 1 } 2 (cid:55)→ R 64 × 64 that map 2D grid coordinates to grid images.

Original: Tableshows that we provide 10,000 image matrices for training, 1,000 for tuning the model hyperparameters, and 2,000 to test the abstract visual reasoning ability of models.
Revised: Table 2 shows that we provide 10,000 image matrices for training, 1,000 for tuning the model hyperparameters, and 2,000 to test the abstract visual reasoning ability of models.

Original: In the training and validation[Table] process, we randomly select 1 or 2 images as target images; in the testing process, we use Testand Test-3 to evaluate the performance of the models to predict 2 and 3 target images.
Revised: In the training and validation process, we randomly select 1 or 2 images as target images; in the testing process, we use Test-2 and Testto evaluate the performance of the models to predict 2 or 3 target images.

Original: MPI3D
Revised: MPI3D

Original: MPI3D (Gondal et al., 2019) dataset contains a series of single-object scenes, each with 40 different scene images.
Revised: MPI3D (Gondal et al., 2019) dataset contains a series of single-object scenes, each with 40 different scene images.

Original: There are two underlying rules to change the object in a scene: change the altitude of the object (sample 1 in Figure 8g); change the azimuth of the object (sample 2 in
Revised: There are two underlying rules to change the object in a scene: change the altitude of the object (sample 1 in Figure 8g); change the azimuth of the object (sample 2 in

Original: Figure 8g).
Revised: Figure 8g).

Original: The other attributes (e.g., object color, camera height, etc.) do not change within scene images.
Revised: The other attributes (e.g., object color, camera height, etc.) do not change within scene images.

Original: It is essential for the models to grasp different changing patterns for target image prediction, and the key is to describe the distribution over functions f : R (cid:55)→ R 3 × 64 × 64 that map object altitudes or azimuths to scene images.
Revised: It is essential for the models to grasp different changing patterns for target image prediction, and the key is to describe the distribution over functions f : R (cid:55)→ R 3 × 64 × 64 that map object altitudes or azimuths to scene images.

Original: As Table 2 shows, we provide 16127 scenes for training, 2305 for tuning the hyperparameters, and 4608 for testing.
Revised: As Table 2 shows, we provide 16127 scenes for training, 2305 for tuning the hyperparameters, and 4608 for testing.

Original: In terms of training and validation, we ﬁrst randomly select 8 images from 40 scene images to represent the scene, and then randomly select 1 to 4 images from the 8 images again as the target images, leaving the remaining images as the context.
Revised: In terms of training and validation, we ﬁrst randomly select 8 images from 40 scene images to represent the scene, and then randomly select 1 to 4 images from the 8 images again as the target images, leaving the remaining images as the context.

Original: For testing, we supply Test-4, Test-6, Test-10, and Test-20, where Test-4 and Testevaluate the performance of the models to predict 4 and 6 target images out of 8 images; Test-10 and
Revised: For testing, we supply Test-4, Test-6, Test-10, and Test-20, where Test-4 and Testevaluate the performance of the models to predict 4 or 6 target images out of 8 images; Test-10 and

Original: Test-20 evaluate the performance to predict 10 and 20 target images out of 40 images.
Revised: Test-20 evaluate the performance to predict 10 or 20 target images out of 40 images.

Original: One can fetch
Revised: One can fetch

Original: MPI3D from the repository 1 under the Creative Commons Attribution 4.0 International License.
Revised: MPI3D from the repository 2 under the Creative Commons Attribution 4.0 International License.

Original: C. M ODEL A RCHITECTURE AND H YPERPARAMETERS CLAP
Revised: D. M ODEL A RCHITECTURE AND H YPERPARAMETERS CLAP-NP

Original: -NP

Original: In this subsection, we will ﬁrst describe the architecture of the encoder, decoder, concept-speciﬁc function parsers, and concept-speciﬁc target predictorsnetwork in CLAP-NP.
Revised: In this subsection, we will ﬁrst describe the architecture of the encoder, decoder, concept-speciﬁc function parsers, and concept-speciﬁc target predictors in CLAP-NP.

Original: Then we will list hyperparameters in the training phase.
Revised: Then we will list hyperparameters in the training phase.

Original: • Encoder : for all datasets, CLAP-NP uses the same convolutional blocks to downsample highdimensional images into parameters of concepts, which can be described as – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 64 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 128 BatchNorm, ReLU – 4 × 4 Conv, 256 BatchNorm, ReLU – 1 × 1 Conv,– ReshapeBlock,– Fully Connected, | C |
Revised: • Encoder : for all datasets, CLAP-NP uses the same convolutional blocks to downsample highdimensional images into the representations of concepts, which can be described as – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 64 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 128 BatchNorm, ReLU – 4 × 4 Conv, 256 BatchNorm, ReLU – 1 × 1 Conv,– ReshapeBlock,

Original:  The ReshapeBlock ﬂattens the output tensor of size  × 1 × 1 from the last convolutional layer to the vector of size 512 .
Revised: – Fully Connected, | C | The ReshapeBlock ﬂattens the output tensor of size 512 × 1 × 1 from the last convolutional layer to the vector of size 512 .

Original: The output of the encoder is the mean of | C | visual concepts
Revised: The output of the encoder is the mean of | C | concepts.

Original: • Decoder :in BoBa and CRPM datasets, CLAP-NP uses several deconvolutional layers to upsample  concepts to images, which is – ReshapeBlock, | C | × 1 ×– 1 × 1 Deconv, 128 BatchNorm, ReLU – 4 × 4 Deconv, 64 BatchNorm2d, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU[Table] – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, N Channel Sigmoid where N Channel is the number of image channels and Sigmoid is the activation function used to generate pixel values ranging in (0 , 1) .
Revised: • Decoder : CLAP-NP uses several deconvolutional layers to upsample the representations of concepts to images, which is – ReshapeBlock, | C | × 1 ×– 1 × 1 Deconv, 128 BatchNorm, ReLU – 4 × 4 Deconv, 64 BatchNorm2d, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, N Channel Sigmoid where N Channel is the number of image channels and Sigmoid is the activation function used to generate pixel values ranging in (0 , 1) .

Original: • Function Parser : CLAP-NP provides identical but independent function parsers for each concept.
Revised: • Function Parser : CLAP-NP provides identical but independent function parsers for concepts. Each function parser consists

Revised: of the network T s to encode the representations of concepts, T i to encode the inputs

Revised: of

Revised: latent random

Original: Each function parser consists of a multi-layer perception (MLP) T s to encode concepts, a MLP T i to encode function inputs, a MLP T a to aggregate context points, anda MLP T f to predict the mean and standard deviation of the global latent variable.
Revised: functions, T a to aggregate context information and T f to estimate the distribution of the global latent variable.

Original: The architecture of T s and T i is – Fully Connected, 32 ReLU – Fully Connected,
Revised: The architecture of T s and T i is – Fully Connected, 32 ReLU – Fully Connected,

Original: By concatenating the encoded concepts and theencoded function inputs, we get the context function points.
Revised: By concatenating the representations of concepts and the function inputs we get the context points in the latent random function.

Original: The architecture of T a is
Revised: The architecture of T a is

Original: – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected, Finally, the architecture of T f is – Fully Connected, 256 ReLU – Fully Connected, 2 d g where d g is the size of the global latent variable g c .
Revised: – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected,Finally, the architecture of T f is – Fully Connected, 256 ReLU – Fully Connected, 2 d g where d g is the size of the global latent variable g c .

Original: The outputs of T f consist of the mean and standard deviation ofthe latent variable g c .
Revised: The outputs of T f consist of the mean and standard deviation of g c .

Original: • Target Predictor : the concept-speciﬁc target predictor T pis a MLP that maps g c and the encodedfunction input into
Revised: • Target Predictor : the concept-speciﬁc target predictor T p maps g c and the encoded

Original:  the target concept, whose architecture is
Revised: function inputs into the target concept, whose architecture is

Original: – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected,
Revised: – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected,

Original: To generate more complex scene images in MPI3D, CLAP-NP uses a deeper decoder where the hidden size is [512, 256, 128, 64, 32].
Revised: To generate more complex scene images in MPI3D, CLAP-NP uses a deeper decoder with hidden sizes [512, 256, 128, 64, 32].

Original: And we use T f withthe hidden size [128, 128] and output size 64, T f withthe hidden size [64], and T p withthe hidden size [128, 128].
Revised: And we use T f with hidden sizes [128, 128] and output size 64, T f with hidden size [64], and T p with hidden sizes [128, 128].

Original: Hyperparameters for CLAP-NP on different datasets are shown in Table 4.
Revised: Hyperparameters for CLAP-NP on different datasets are shown in Table 4.

Original: And in all datasets, we set  learning rate as 3 × 10 − 4 , batch size as 512, and σ x = 0 . 1 .
Revised: In all datasets, we set the learning rate as 3 × 10 − 4 , batch size as 512, and σ y = 0 . 1 .

Original: For MPI3D, we anneal β t and β f in the ﬁrst 400 epochs.
Revised: For MPI3D, we anneal β t and β f in the ﬁrst 400 epochs.

Original: After each training epoch, we use the validation set to compute the evidence lower bound (ELBO) of the current model and save the model with the largest ELBO as the trained model.
Revised: After each training epoch, we use the validation set to compute the evidence lower bound (ELBO) of the current model and save the model with the largest ELBO as the trained model.

Original: For all datasets, CLAP-NP uses the Adam (Kingma & Ba, 2015) optimizer to update parameters.
Revised: For all datasets, CLAP-NP uses the Adam (Kingma & Ba, 2015) optimizer to update parameters.

Original: Neural Process (NP)
Revised: Neural Process (NP)

Original: Different from (Garnelo et al., 2018b),to apply NP in high-dimensional images, we add an encoder and decoder which are the same as those in CLAP-NP  to convert highdimensional images into low-dimensional representations.
Revised: To deal with high-dimensional images in NP (Garnelo et al., 2018b), we apply the encoder and decoder of CLAP-NP in it to convert high-dimensional images into lowdimensional representations.

Original: And the representations are combined with the function inputs to form the context by an aggregator T a .
Revised: The aggregator T a extract context information from the representations and function inputs.

Original: Then the function parser T f obtains the mean and standard deviation of the global latent variable.
Revised: Then the function parser T f converts the context representation into the mean and standard deviation of the global latent variable.

Original: Finally, the decoder generates[Table] target images from the target function inputs and the global latent variable.
Revised: Finally, the decoder generates target images from the target function inputs and the global latent variable.

Original: The architectures of T a and T f are
Revised: The architectures of T a and T f are

Original: • Aggregator  – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU – Fully Connected,• Function Parser – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU – Fully Connected, 2 d g
Revised: • Aggregator [Table] [Table] – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU – Fully Connected,• Function Parser – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU – Fully Connected, 2 d g

Original: Hyperparameters for NP on different datasets are shown in Table 5 where d r is the representation size of the encoder.
Revised: NP’s hyperparameters on different datasets are shown in Table 5 where d r is the representation size of the encoder.

Original: And for all datasets, NP adopts the learning rate 1 × 10 − 4 batch size 512, and σ x = 0 . 1 .
Revised: In all datasets, NP adopts the learning rate 1 × 10 − 4 batch size 512, and σ y = 0 . 1 .

Original: For all datasets, NP uses the Adam optimizer for parameter updates.
Revised: The parameters of NP are updated with the Adam optimizer (Kingma & Ba, 2015).

Original: Generative Query Network (GQN)
Revised: Generative Query Network (GQN)

Original: To implement GQN (Eslami et al., 2018), we use a PyTorch implementation from the repository 2 with the following changes to the default conﬁguration: set learning rate to 0.0005, batch size to 256, the representation type to pool, the number of iterations to 4, and share the ConvLSTM core among iterations.
Revised: To implement GQN (Eslami et al., 2018), we use a PyTorch implementation from the repository 3 with the following changes to the default conﬁguration to control the computational resource: set learning rate to 0.0005, batch size to 256, the representation type to pool, the number of iterations to 4, and share the ConvLSTM core among iterations.

Original: Gaussian Process (GP)
Revised: Gaussian Process (GP)

Original: We use a pretrained autoencoder to reduce the dimension of images and let GP capture the changing patterns in the low-dimensional space.
Revised: We use a pretrained autoencoder to reduce the dimension of images and let GP capture the changing patterns in the low-dimensional space.

Original: We adopt the same encoder and decoder as in CLAP-NP and NP for a fair comparison.
Revised: We adopt the same encoder and decoder as in CLAP-NP and NP for a fair comparison.

Original: The mean function of GP is set to a constant function.
Revised: The mean function of GP is set to a constant function.

Original: After comparing the RBF kernel, periodic kernel, and deep kernel, we chose the deep kernel as the kernel function that k ( x i , x j ) =
Revised: After comparing the RBF kernel, periodic kernel, and deep kernel, we chose the deep kernel as the kernel function that κ ( x i , x j ) =

Original: σ 2 exp( −(cid:107)T k ( x i ) − T k ( x j ) (cid:107) / 2 (cid:96) 2 ) (Wilson et al., 2016).
Revised: σ 2 exp( −(cid:107)T k ( x i ) − T k ( x j ) (cid:107) / 2 (cid:96) 2 ) (Wilson et al., 2016).

Original: Neural network T k is a multilayer perceptron with hidden sizes [1000, 1000, 500, 50], output size 2, and ReLU activation functions, which is the same as in DKL (Wilson et al., 2016).
Revised: Neural network T k is a multilayer perceptron with hidden sizes [1000, 1000, 500, 50], output size 2, and ReLU activation functions, which is the same as in DKL (Wilson et al., 2016).

Original: The hyperparameters of GP are tuned on each sample through the log-likelihood of context points and the RMSprop optimizer.
Revised: The hyperparameters of GP are tuned on each sample through the log-likelihood of context points and the RMSprop optimizer.

Original: We give the learning rate and epoch to adjust hyperparameters in Table 6 and use multi-task Gaussian Process prediction (Bonilla et al., 2007) to model the correlations between dimensions. 
Revised: We give the learning rate and epoch to adjust hyperparameters in Table 6 and use multi-task Gaussian Process prediction (Bonilla et al., 2007) to model the correlations between dimensions. [Figure] GP [Figure]

Revised: GT

Revised: [Figure]

Original: C. C OMPUTATIONAL R ESOURCE
Revised: GP GP [Figure] D. C OMPUTATIONAL R ESOURCE

Original: We train our model and baselines on the server with Intel(R)
Revised: We train our model and baselines on the server with Intel(R)

Original: Xeon(R)
Revised: Xeon(R)

Original: Gold 6133 CPUs, 24GB
Revised: Gold 6133 CPUs, 24GB

Original: NVIDIA GeForce RTX 3090 GPUs, 512GB RAM, and Ubuntu 18.04 OS.
Revised: NVIDIA GeForce RTX 3090 GPUs, 512GB RAM, and Ubuntu 18.04 OS.

Original: All models are implemented with the PyTorch (Paszke et al., 2019) framework.
Revised: All models are implemented with the PyTorch (Paszke et al., 2019) framework.

Original: To avoid the randomness in dataset splitting, we ﬁx the random seed to 0.

Original: [Figure]

Original: NP

Original: [Figure]

Original: [Table]

Original: D A DDITIONAL E XPERIMENTAL R ESULTS D. I NTUITIVE P HYSICS
Revised: E A DDITIONAL E XPERIMENTAL R ESULTS E. I NTUITIVE P HYSICS

Original: In Figure 9 we display additional prediction results on BoBa-1 and BoBa-2.
Revised: In Figure 9 we display additional prediction results on BoBa-1 and BoBa-2.

Original: For each instance, there are samples that correspond to Test-4 (left) and Test-6 (right) [Table] where CLAP-NP outperforms other baselines.
Revised: For each instance, we provide samples using the conﬁgurations Test-4 (left) or Test-6 (right), where CLAP-NP outperforms other baselines.

Original: NP generates blurred target images on both BoBa-1 and BoBa-2 datasets.
Revised: NP generates blurred target images on both BoBa-1 and BoBa-2 datasets.

Original: It indicates that NP has difﬁculty modeling changing patterns on bouncing balls.
Revised: It indicates that NP has difﬁculty modeling changing patterns on bouncing balls.

Original: GQN can produce clear images but may generate non-existent balls and lose existing balls.
Revised: GQN can produce clear images but may generate non-existent balls and lose existing balls.

Original: In the ﬁrst sample of BoBa-1, the predictions of GQN deviate from the ground truths signiﬁcantly in the position of balls.
Revised: In the ﬁrst sample of BoBa-1, the predictions of GQN deviate from the ground truths signiﬁcantly in the position of balls.

Original: Overall.

Original: CLAP-NP performs well in modeling scene consistency and predicting motion trajectory. 
Revised: CLAP-NP performs well in modeling scene consistency and predicting motion trajectory. [Figure] GQN CLAP

Revised: [Figure]

Revised: NP

Original: D. A BSTRACT V ISUAL R EASONING
Revised: [Figure] P r e d i c t 2 i m a g e s P r e d i c t a r o w [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: Prediction

Revised: Prediction

Revised: Prediction

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: (a) CRPM-T NP

Revised: GQN

Revised: CLAP

Revised: [Figure]

Revised: NP

Original: With the additional experimental results in Figure
Revised: [Figure] P r e d i c t 2 i m a g e s P r e d i c t a r o [Figure]

Revised: r e d i c t 2 i m a g e

Original: [Table]
Revised: [Figure]

Original:  we can intuitively see that CLAP-NP has a better understanding of the changing rules than baselines.
Revised: P r e d i c t a r [Figure] E. A BSTRACT V ISUAL R EASONING With the additional experimental results in Figure 10, we can intuitively see that CLAP-NP has a better understanding of the changing rules than baselines.

Original: For the second sample of each dataset [Table] a row of images is removed to interpret the abstract visual reasoning ability of the model.
Revised: For the second sample of each dataset, a row of images is removed to interpret the abstract visual reasoning ability of the model.

Original: As samples in Figure 10 shows, when we remove a row of images, the answer is not unique: the predictions are correct as long as their sizes increase progressively and their colors and rotations keep constant.
Revised: As samples in Figure 10 shows, when we remove a row from the matrix, the answer is not unique: the predictions are correct as long as their sizes increase progressively and their colors and rotations keep constant.

Original: CLAP-NP represents such randomness in predictions by means of the probabilistic generative process, making it possible to generate different correct answers.
Revised: CLAP-NP represents such randomness in predictions by means of the probabilistic generative process, making it possible to generate different correct answers.

Original: In terms of the prediction quality, although generating blurred images, NP has the basic reasoning ability about the progressive rule of  the outer triangle size in Figure 10b.
Revised: In terms of the prediction quality, although generating blurred images, NP has the basic reasoning ability about the progressive rule of [Table] the outer triangle size in Figure 10b.

Original: GQN generates clear images, however, the generated images probably deviate from the underlying changing rules on matrices.
Revised: GQN generates clear images, however, the generated images probably deviate from the underlying changing rules on matrices.

Original: Table 7 and Table 11 illustrate the outperforming results of CLAP-NP in abstract visual reasoning by quantitative experiments.
Revised: Table 7 and Table 11 illustrate the outperforming results of CLAP-NP in abstract visual reasoning by quantitative experiments [Table]

Original: D. S CENE R EPRESENTATION
Revised: E. S CENE R EPRESENTATION

Original: Figure 12 shows the prediction results within 8 scene images.
Revised: Figure 12 shows the prediction results within 8 scene images.

Original: Both GQN and CLAP-NP generate clear prediction results when the number of target images is 4; if the number of target images is increased to 6, the prediction results of GQN have signiﬁcant deviations, while CLAP-NP maintains the accuracy of predictions.
Revised: Both GQN and CLAP-NP generate clear prediction results when the number of target images is 4; if the number of target images is increased to 6, the prediction accuracy of GQN has a signiﬁcant decline, while CLAP-NP maintains the accuracy.

Original: NP generates ambiguous results in all experiments.
Revised: NP generates ambiguous results in all experiments.

Original: Figure 13 shows a more complicated situation: we provide 40 scene images and set 20 of them as target images.
Revised: Figure 13 shows a more complicated situation: we provide 40 scene images and set 20 of them as target images.

Original: In this case, GQN and NP can hardly generate clear foreground objects; while CLAP-NP produces relatively accurate predictions with only a little decrease in generative quality.
Revised: In this case, GQN and NP can hardly generate clear foreground objects; while CLAP-NP produces relatively accurate predictions with only a little decrease in generative quality.

Original: This experiment aims to test the generalization of the laws learned by the models, and the results above illustrate that NP can hardly represent scenes with functions, GQN has difﬁculty generalizing the scene representation ability to different conﬁgurations, and the laws learned by CLAP-NP can be generalized to novel scenes with more available images.
Revised: This experiment aims to test the generalization of the laws learned by the models [Table] and the results above illustrate that NP can hardly represent scenes with functions, GQN has difﬁculty generalizing the scene representation ability to different conﬁgurations, and the laws learned by CLAP-NP can be generalized to novel scenes with the compositional modeling of laws.

Original: D. C ONCEPT D ECOMPOSITION
Revised: E. C ONCEPT D ECOMPOSITION

Original: Concept decomposition is the foundation for CLAP to understand concept-speciﬁc laws.
Revised: Concept decomposition is the foundation for CLAP to understand concept-speciﬁc laws.

Original: In this experiment, we traverse each concept in the latent space and visualize the concepts through a decoder to illustrate the LRF learned for each concept. First of all, we decompose a batch of images into [Figure] CLAP
Revised: In this experiment, we traverse each concept in the latent space and visualize the concepts through the decoder to illustrate the LRFs.

Original: [Figure]

Original: NP

Original: [Figure]

Original: P r e d i c t 2 i m a g e s

Original: P r e d i c t a r

Original: [Figure]

Original: a)

Original: [Figure]

Original: P r e d i c t 2 i m a g e s

Original: P r e d i c t a r o w

Original: [Figure]

Original: i c t 2 i m

Original: [Figure]

Original: g

Original: [Figure]

Original: NP

Original: GT

Original: GQN

Original: CLAP-NP

Original: NP

Original: GT

Original: GQN CLAP-NP GP GP concepts by the encoder to estimate the range of concepts in the latent space.
Revised: First of all, we decompose a batch of images into concepts by the encoder to estimate the range of concepts in the latent space.

Original: To traverse one concept, we ﬁx the other concepts and linearly interpolate it from the minimum value to the maximum value to generate a sequence of interpolation results, which are decoded into images for visualization.
Revised: To traverse one concept, we ﬁx the other concepts and linearly interpolate it from the minimum value to the maximum value to generate a sequence of interpolation results, which are decoded into images for visualization.

Original: Each row of Figure 14 represents thelatent traversal results of one concept.
Revised: Each row of Figure 14 represents the traversal results of one concept.

Original: In BoBa-1, CLAP-NP learns LRFs on concepts of color, horizontal position, and vertical position in an unsupervised manner.
Revised: In BoBa-1, CLAP-NP learns LRFs on concepts of color, horizontal position, and vertical position in an unsupervised manner.

Original: This is similar to the way we understand the motion of balls: the color keeps constant over time; the horizontal and vertical positions conform to the physical laws in their respective directions.
Revised: This is similar to the way we understand the motion of balls: the color keeps constant over time; the horizontal and vertical positions conform to the physical laws in their respective directions.

Original: For CRPM, CLAP-NP correctly parses images into concepts that correspond to the attribute-speciﬁc rules in matrices.
Revised: For CRPM, CLAP-NP correctly parses images into concepts that correspond to the attribute-speciﬁc rules in matrices.

Original: Concept decomposition in real environments is a challenge for models.
Revised: Concept decomposition in real environments is a challenge for models.

Original: For MPI3D, CLAP-NP parses the LRFs on the object axis (Dimension 1, 5, 6, 7) and other static scene attributes (Dimension 2, 3, 4). 
Revised: For MPI3D, CLAP-NP parses the LRFs on the object appearance (Dimension 1, 5, 6, 7) and other static scene attributes (Dimension 2, 3, 4). NP GT

Revised: GQN

Revised: CLAP-NP

Original: D. L ATENT R ANDOM F UNCTION C OMPOSITION
Revised: NP GT GQN CLAP-NP

Original:  By means of concept decomposition, CLAP-NP can parse the latent random function of each concept to represent concept-speciﬁc laws, by which the law of a sample can be modiﬁed in terms of concepts.
Revised: GP GP E. L ATENT R ANDOM F UNCTION C OMPOSITION By means of concept decomposition, CLAP-NP can parse the LRF on each concept to represent concept-speciﬁc laws, by which the law of a sample can be modiﬁed in terms of concepts.

Original: Figure 15 and 16 display the way to edit the laws.
Revised: Figureand 16 display the way to manipulate the laws.

Original: We compose the concept-speciﬁc LRFs of existing samples to generate samples that have novel changing patterns.
Revised: We compose the concept-speciﬁc LRFs of existing samples to generate samples that have novel changing patterns.

Original: For example, in the third sample of Figure 15a, we generate a sample with the progressive law of size, constant law of rotation, and constant law of color from three existing samples.
Revised: For example, in the third sample of Figure 15a, we generate a sample with the progressive law of size, constant law of rotation, and constant law of color from three existing samples.

Original: The results on four instances of CRPM indicate that CLAP-NP understands the changing patterns on matrices in an interpretable and manipulable way, which embodies the compositionality of laws.
Revised: The results on four instances of CRPM indicate that CLAP-NP understands the changing patterns on matrices in an interpretable and manipulable way, which embodies the compositionality of laws.

Original: D. L ATENT R ANDOM F UNCTION E XCHANGE
Revised: E. L ATENT R ANDOM F UNCTION E XCHANGE

Original: Another way to edit laws is to exchange LRFs of some concepts between samples.
Revised: Another way to manipulate laws is to exchange LRFs of some concepts between samples.

Original: In BoBa, we swap the law ofcollisions and appearance  between samples, and the results are shown in Figure 17.
Revised: In BoBa, we swap the law of appearance or object motion between samples, and the results are shown in Figure 17.

Original: According to the concept decomposition results in Figure 14, we exchange the LRFs  on Dimension 1 (Dimensions 2 and 3) to modify the law of collisions (the law of appearance) in BoBa-1.
Revised: According to the concept decomposition results in Figure 14, we exchange the LRFs between samples on Dimension 1 (Dimensions 2 and 3) to modify the motion of balls (the law of appearance) in BoBa-1.

Original: And we exchange the LRFs  on Dimensions 1 and 6 (Dimensions 2, 3, 4, and 5) to modify the law of collisions (the law of appearance) in BoBa-2.
Revised: And we exchange the LRFs between samples on Dimensions 1 and(Dimensions 2, 3, 4, and 5) to modify the motion of balls (the law of appearance) in BoBa-2.

Original: To exchange laws in MPI3D, as shown in Figure 18, 
Revised: To exchange laws in MPI3D, as shown in Figure 18, we swap LRFs between samples on Dimensions 1, 5, 6, and 7 to change the law of the object motion. By modifying laws on example 1 of Figure 18, the changing pattern of the ﬁrst sample becomes the horizontal rotation, and the second sample inherits the law of vertical rotation from the ﬁrst sample, which illustrates CLAP-NP’s ability to generate new samples. The experiments on law exchange evidence CLAP-NP’s interpretability and manipulability from another perspective. E. G ENERALIZATION ON U NSEEN C ONCEPTS

Original: [Figure]
Revised: We extend BoBa-2 to generate four datasets with novel laws: Novel-colors (BoBa-2-NC), Novelshape (BoBa-2-NS), Without-ball-collisions (BoBa-2-WBO), and With-gravity (BoBa-2-WG). We draw balls on BoBa-2-NC with unseen colors and replace the balls on BoBa-2-NS with squares without changing the laws of motion. For BoBa-2-WBO, we disable collisions between balls and re- NP

Revised: GT

Original: [Figure] 
Revised: [Figure] NP NP GT

Revised: GQN

Original: [Figure]
Revised: [Figure]

Revised: NP

Revised: [Figure]

Revised: GT

Revised: [Figure]

Revised: [Figure]

Original: + + = { Size 1, Rotation 1,
Revised: + + = { Size 1, Rotation 1,

Original: Color 1 } { Size 2, Rotation 2,
Revised: Color 1 } { Size 2, Rotation 2,

Original: Color 2 } { Size 3, Rotation 3,
Revised: Color 2 } { Size 3, Rotation 3,

Original: Color 3 } { Size 1, Rotation 2,
Revised: Color 3 } { Size 1, Rotation 2,

Original: Color 3 } + + = + + =
Revised: Color 3 } + + = + + =

Original: { Size 1, Rotation 1,
Revised: { Size 1, Rotation 1,

Original: Color 1 } { Size 2, Rotation 2,
Revised: Color 1 } { Size 2, Rotation 2,

Original: Color 2 } { Size 3, Rotation 3, Color 3 } { Size 1, Rotation 2,
Revised: Color 2 } { Size 3, Rotation 3, Color 3 } { Size 1, Rotation 2,

Original: Color 3 } { Size 1, Rotation 1,
Revised: Color 3 } { Size 1, Rotation 1,

Original: Color 1 } { Size 2, Rotation 2,
Revised: Color 1 } { Size 2, Rotation 2,

Original: Color 2 } { Size 3, Rotation 3,
Revised: Color 2 } { Size 3, Rotation 3,

Original: Color 3 } { Size 1, Rotation 2,
Revised: Color 3 } { Size 1, Rotation 2,

Original: Color 3 } (a) CRPM-T + + =
Revised: Color 3 } (a) CRPM-T + + =

Original: { Inner Color 1, Inner Rotation 1,
Revised: { Inner Color 1, Inner Rotation 1,

Original: Outer Size 1 } { Inner Color 2, Inner Rotation 2,
Revised: Outer Size 1 } { Inner Color 2, Inner Rotation 2,

Original: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,
Revised: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,

Original: Outer Size 3 } + + = + + = { Inner Color 1, Inner Rotation 1,
Revised: Outer Size 3 } + + = + + = { Inner Color 1, Inner Rotation 1,

Original: Outer Size 1 } { Inner Color 2, Inner Rotation 2,
Revised: Outer Size 1 } { Inner Color 2, Inner Rotation 2,

Original: Outer Size 2 } { Inner Color 3, Inner Rotation 3,
Revised: Outer Size 2 } { Inner Color 3, Inner Rotation 3,

Original: Outer Size 3 } { Inner Color 1, Inner Rotation 2, Outer Size 3 } { Inner Color 1, Inner Rotation 1,
Revised: Outer Size 3 } { Inner Color 1, Inner Rotation 2, Outer Size 3 } { Inner Color 1, Inner Rotation 1,

Original: Outer Size 1 } { Inner Color 2, Inner Rotation 2,
Revised: Outer Size 1 } { Inner Color 2, Inner Rotation 2,

Original: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,
Revised: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,

Original: Outer Size 3 }
Revised: Outer Size 3 }

Original: [Figure]
Revised: [Figure]

Original: + =
Revised: + =

Original: [Figure]
Revised: [Figure]

Original: + =
Revised: + =

Original: [Figure]
Revised: [Figure]

Original: + =
Revised: + =

Revised: [Figure]

Revised: =

Original: [Figure]
Revised: [Figure]

Original: + + =
Revised: + + =

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: +

Original: [Figure]
Revised: [Figure]

Original:  Law 
Revised: {Collision Law 1,Appearance Law 1 } {Collision Law 2,Appearance Law 2 } {Collision Law 1,Appearance Law 2 } {Collision Law 2,Appearance Law 1 } Exchange LRFs between samples on Dim # {Collision Law 1,Appearance Law 1 } {Collision Law 2,Appearance Law 2 } {Collision Law 2,Appearance Law 1 } {Collision Law 1,Appearance Law 2} Exchange LRFs between samples on Dim #2 and #

Original: [Figure]
Revised: [Figure]

Original:  Law 2 } 
Revised: {Collision Law 1,Appearance Law 1 } {Collision Law 2,Appearance Law 2 } {Collision Law 2,

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure] [Figure]
Revised: Law 1 } {Collision Law

Original: [Figure]
Revised: [Figure]

Original: we swap LRFs  on Dimensions 1, 5, 6, and 7 to change the law of the object axis. By modifying laws on example 1 of Figure 18, the changing pattern of the ﬁrst sample becomes the horizontal rotation, and the second sample inherits the law of vertical rotation from the ﬁrst sample, which illustrates CLAP-NP’s ability to generate new samples.
Revised: Appearance Law 2} Exchange LRFs between samples on Dim #2,#3,#4 and # {Collision Law 1,Appearance Law 1 } {Collision Law 2,Appearance Law 2 } {Collision Law 1,Appearance Law 2 } {Collision Law 2,Appearance Law 1 }

Original: The experiments onlaw exchange evidence
Revised: Exchange LRFs between samples on

Original: CLAP-NP’s interpretability and manipulability from another perspective.
Revised: Dim #1 and # [Figure]

Original: We extend BoBa-2 to generate four datasets with novel laws: Novel-colors (BoBa-2-NC), Novelshape (BoBa-2-NS), Without-ball-collisions (BoBa-2-WBO), and With-gravity (BoBa-2-WG).
Revised: [Figure] [Figure]

Original: We draw balls on BoBa-2-NC with unseen colors and replace the balls on BoBa-2-NS with squares without changing the laws of positions.
Revised: [Table]

Original: For BoBa-2-WBO, we disable collisions between balls and reserve collisions between balls and borders; for BoBa-2-WG, we add vertical gravity to scenes.
Revised: serve collisions between balls and borders; for BoBa-2-WG, we add vertical gravity to scenes.

Original: After training on BoBa-2, CLAP-NP is tested on four datasets without retraining to evaluate whether the compositionality of laws improves the model’s generalization ability to scenes with unseen concepts or laws.
Revised: After training on BoBa-2, CLAP-NP is tested on four datasets without retraining to evaluate whether the compositionality of laws improves the model’s generalization ability in scenes with unseen concepts or laws.

Original: Figure 19 shows that CLAP-NP can predict the correct object positions on BoBa-2-NC and BoBa-2-NS.
Revised: Figure 19 shows that CLAP-NP can predict the correct object positions on BoBa-2-NC and BoBa-2-NS.

Original: CLAP-NP predicts inaccurate color and shape of objects because the encoder and decoder are trained on BoBa-2, and the latent space does not encode the unseen colors or shapes.
Revised: CLAP-NP predicts inaccurate color and shape of objects because the encoder and decoder are trained on BoBa-2, and the latent space does not encode the unseen colors or shapes.

Original: We observe similar results on BoBa-2-WBO and BoBa-2-WG that CLAP-NP learns the correct law of appearanceconstancy but predicts incorrect positions of balls when there are unseen physical laws in scenes.
Revised: We observe similar results on BoBa-2-WBO and BoBa-2-WG that CLAP-NP learns the correct law of appearance but predicts incorrect positions of balls when there are unseen physical laws in scenes.

Original: To better evaluate the generalization ability in scenes with novel laws, we quantitatively evaluate CLAP-NP and baseline models with MSE ad SA scores on the four datasets.
Revised: To better evaluate the generalization ability in scenes with novel laws, we quantitatively evaluate CLAP-NP and baseline models with MSE ad SA scores on the four datasets.

Original: The results in Table 8 indicate that CLAP-NP achieves the best MSE and SA scores in most situations, which illustrates CLAP-NP’s generalization ability in scenes with novel  laws.
Revised: The results in Table 8 indicate that CLAP-NP achieves the best MSE and SA scores in most situations, which illustrates CLAP-NP’s generalization ability in scenes with novel concepts or laws.

Original: D. N UMBER OF L ATENT R ANDOM F UNCTIONS
Revised: E. N UMBER OF L ATENT R ANDOM F UNCTIONS

Original: This experiment explores the inﬂuence of setting too large or too small numbers of latent random functions (LRFs) in CLAP-NP.
Revised: This experiment explores the inﬂuence of setting too many or too few LRFs in CLAP-NP.

Original: Figure 20 shows the latent traversal results on MPI3D, BoBa-2, and CRPM-DT.
Revised: Figureshows the latent traversal results on MPI3D, BoBa-2, and CRPM-DT.

Original: If we set too few LRFs, CLAP-NP learns mixed laws among dimensions instead of the expected compositional laws. Moreover, too few LRFs can inﬂuenceCLAP-NP’s generation [Figure] [Table]
Revised: If we set too few LRFs, CLAPNP encodes different laws in one LRF instead of learning compositional laws, which will inﬂuence

Original: [Figure]

Original: [Table] ability (e [Table] g., set only two LRFs for BoBa-2).
Revised: CLAP-NP’s generation ability (e.g., set only two LRFs for BoBa-2).

Original: Setting too many LRFs has no signiﬁcant inﬂuence on CLAP-NP’s performance because there will be redundant dimensions inCLAP-NP that do not encode information (e
Revised: Setting too many LRFs has no signiﬁcant inﬂuence on CLAP-NP’s performance because there will be redundant dimensions in

Original: [Table] g., dimensions 1, 3, and 6 on CRPM-DT).
Revised: CLAP-NP that do not encode information (e.g., dimensions 1, 3, and 6 on CRPM-DT).

Original: However, due to the independent modeling of concept-speciﬁc LRFs, setting a large number of LRFs will reduce the computational speed of CLAP-NP and increase the number of model parameters.
Revised: However, due to the independent modeling of concept-speciﬁc LRFs, setting a large number of LRFs will reduce the computational speed of CLAP-NP and increase the number of model parameters.

Original: D. P REDICTION S TRATEGY
Revised: E. P REDICTION S TRATEGY

Original: CLAP uses the one-shot strategy that predicts all target images at one time.
Revised: CLAP uses the one-shot strategy that predicts all target images at one time.

Original: However, the rollout strategy can be another choice to predict the following target images through the few context images  at the beginning.
Revised: However, the rollout strategy can be another choice to predict the following target images through the few context images [Figure] [Table] at the beginning.

Original: For example, in the test conﬁguration η = 20 / 25 , we ﬁrst predict the 6th to 10th images from the ﬁrst ﬁve context images, then combine them (1st to 10th images) to predict the following ﬁve images, and repeat this process until all target images are predicted.
Revised: For example, in the test conﬁguration η = 20 / 25 , we ﬁrst predict the 6th to 10th images from the ﬁrst ﬁve context images, then combine them (1st to 10th images) to predict the following ﬁve images, and repeat this process until all target images are predicted.

Original: Table 9 shows the MSE scores on CRPM-DT, BoBa-2, and MPI3D where CLAP-NP predicts targets with the rollout and one-shot strategy, respectively.
Revised: Tableshows the MSE scores on CRPM-DT, BoBa-2, and MPI3D where CLAP-NP predicts targets with the rollout and one-shot strategy, respectively.

Original: On BoBa-2 and MPI3D, the rollout strategy slightly improves the prediction accuracy.
Revised: On BoBa-2 and MPI3D, the rollout strategy slightly improves the prediction accuracy.

Original: If a target image is far away from all the context images, the prediction may have high uncertainty.
Revised: Generally, if a target image is far away from all the context images, the prediction may have high uncertainty.

Original: The rollout strategy only predicts a few target imagesat close positions each time, while the one-shot strategy has to predict all the target images at one time.
Revised: The model only predicts a few target images close to the context each time with the rollout strategy, while the one-shot strategy requires the model to predict all the target images at one time.

Original: Therefore, the rollout strategy is more likely to reduce uncertainty and improve prediction accuracy.
Revised: Therefore, the rollout strategy is more likely to have lower computational uncertainty and higher prediction accuracy.

Original: D. F AILURE C ASES
Revised: E. F AILURE C ASES

Original: In this experiment, we display some failure cases.
Revised: In this experiment, we display some failure cases.

Original: For BoBa, most failure cases occur when there are continuous target images (1st sample of BoBa in Figure 21) or too many target images (2nd sample of BoBa in Figure 21).
Revised: For BoBa, most failure cases occur when there are continuous target images (1st sample of BoBa in Figure 21) or too many target images (2nd sample of BoBa in Figure 21).

Original: For CRPM, CLAP can predict target images with diversity when we remove the third row from a matrix, but it sometimes generates target images that break the rules.
Revised: For CRPM, CLAP can predict target images with diversity when we remove a row from a matrix, but it sometimes generates target images that break the rules.

Original: For example, the color of images keeps invariant in the ﬁrst two rows, but CLAP generates images in the third row with changing grayscales (samples of CRPM in Figure 21).
Revised: For example, the color of images keeps invariant in the ﬁrst two rows, but CLAP generates images in the third row with changing grayscales (samples of CRPM in Figure 21).

Original: For MPI3D, when the object size has an obvious change (1st sample of MPI3D in Figure 21) or the centric object is tiny (2nd sample of MPI3D in Figure 21), the predictions will deviate orbecome unclear.[Figure]
Revised: For MPI3D, when the object size has an obvious change (1st sample of MPI3D in Figure 21) or the centric object is tiny (2nd sample of MPI3D in Figure 21), the predictions can be incorrect or unclear.

Original: E L IMITATIONS
Revised: F L IMITATIONS

Original: We conclude our limitations in two aspects.
Revised: We conclude our limitations in two aspects.

Original: (1) Complexity of datasets.
Revised: (1) Complexity of datasets.

Original: Because compositional law parsing is an underexplored task, we ﬁrst validate the effectiveness of SVCL on datasets with relatively clear and simple rules to avoid the inﬂuence of unknown confounders in complicated datasets.
Revised: Because compositional law parsing is an underexplored task, we ﬁrst validate the effectiveness of CLAP on datasets with relatively clear and simple rules to avoid the inﬂuence of unknown confounders in complicated datasets.

Original: We believe that the compositionality of laws also exists in more complex scenarios (e.g., learning physical laws in realistic scenes). Furthermore, some vision tasks may beneﬁt from compositional law parsing.
Revised: We believe that the compositionality of laws also exists in more complex scenarios (e.g., learning physical laws in realistic scenes) and some vision tasks may beneﬁt from compositional law parsing.

Original: For example, we can perform a controllable video generation process based on law modiﬁcation or make more interpretable predictions by analyzing dominant laws in videos.
Revised: For example, we can perform a controllable video generation process based on law modiﬁcation or make more interpretable predictions by analyzing dominant laws in videos.

Original: Discovering such compositional law parsing ability in more complex situations can be a valuable topic in future works.
Revised: Discovering such compositional law parsing ability in more complex situations can be a valuable topic in future works.

Revised: [Figure]

Original: (2) Setting the number of LRFs.
Revised: (2) Setting the number of LRFs.

Original: CLAP places an upper bound on the number of LRFs.
Revised: CLAP places an upper bound on the number of LRFs.

Original: For scenes with multiple complex rules, we can empirically set an appropriate upper bound or directly put a large upper bound for the number of LRFs.
Revised: For scenes with multiple complex rules, we can empirically set an appropriate upper bound or directly put a large upper bound for the number of LRFs.

Original: However, a large bound will linearly increase the number of model parameters, and the redundant concepts will waste computing resources, decreasing computational efﬁciency.
Revised: However, a large bound will linearly increase the number of model parameters, and the redundant concepts will waste computing resources, decreasing computational efﬁciency.

Original: Therefore, exploring a mechanism in the future to dynamically adjust the number of functions will be meaningful.
Revised: Therefore, exploring a mechanism in the future to dynamically adjust the number of functions will be meaningful.

Original: When applying CLAP to more complex scenes, we may introduce more inductive biases for better concept decomposition (e.g., use a more task-speciﬁc encoder or decoder).
Revised: When applying CLAP to more complex scenes, we may introduce more inductive biases for better concept decomposition (e g., use a more task-speciﬁc encoder or decoder).

Original: We design CLAP as an unsupervised model independent of task-speciﬁc annotations from tasks, which means that it can be extended to take advantage of these annotations.
Revised: We design CLAP as an unsupervised model, which means that it can be extended to take advantage of task-speciﬁc annotations.

Original: Thus we can ﬁnd ways to integrate CLAP with additional supervision information (e.g., supervise SVCL with thelabels of changing factors for each scene) to help concept learning.
Revised: Thus we can ﬁnd ways to integrate CLAP with additional supervision information (e g., supervise CLAP with the changing factors of scenes) to help concept learning in a speciﬁc task