Original: C OMPOSITIONAL L AW P ARSING WITH L ATENT R ANDOM F UNCTIONS
Revised: C OMPOSITIONAL L AW P ARSING WITH L ATENT R ANDOM F UNCTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Human cognition has compositionality.
Revised: Human cognition has compositionality.

Original: We understand a scene by decomposing the scene into different concepts (e.g., shape and position of an object) and learning the respective laws of these concepts, which may be either natural (e g., laws of motion) or man-made (e.g., laws of a game)
Revised: We understand a scene by decomposing the scene into different concepts (e.g., shape and position of an object) and learning the respective laws of these concepts, which may be either natural (e g., laws of motion) or man-made (e.g., laws of a game)

Original: The automatic parsing of these laws indicates the model’s ability to understand the scene which makes law parsing play a central role in many visual tasks
Revised: The automatic parsing of these laws indicates the model’s ability to understand the scene which makes law parsing play a central role in many visual tasks

Original: This paper proposes a deep latent variable model for
Revised: This paper proposes a deep latent variable model for

Original: Compositional
Revised: Compositional

Original: LAw Parsing (CLAP), which achieves the human
Revised: LAw Parsing (CLAP), which achieves the human

Original: -like compositionality ability through an encoding decoding architecture to represent concepts in the scene as latent variables.
Revised: -like compositionality ability through an encoding decoding architecture to represent concepts in the scene as latent variables.

Original: CLAP employs concept-speciﬁc latent random functions that can be instantiated with
Revised: CLAP employs concept-speciﬁc latent random functions that can be instantiated with

Original: Neural Processes to capture the law of concepts
Revised: Neural Processes to capture the law of concepts

Original: Our experimental results demonstrate that
Revised: Our experimental results demonstrate that

Original: CLAP outperforms the baseline methods in multiple visual tasks such as intuitive physics abstract visual reasoning, and scene representation
Revised: CLAP outperforms the baseline methods in multiple visual tasks such as intuitive physics abstract visual reasoning, and scene representation

Original: The law manipulation experiments illustrate CLAP’s interpretability by modifying speciﬁc latent random functions on samples.
Revised: The law manipulation experiments illustrate CLAP’s interpretability by modifying speciﬁc latent random functions on samples.

Original: For example, CLAP learns the laws of position-changing and appearance constancy from the moving balls in a scene, making it possible to exchange laws between samples or compose existing laws into novel laws.
Revised: For example, CLAP learns the laws of position-changing and appearance constancy from the moving balls in a scene, making it possible to exchange laws between samples or compose existing laws into novel laws.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Compositionality is an important feature of human cognition (Lake et al., 2017).
Revised: Compositionality is an important feature of human cognition (Lake et al., 2017).

Original: Humans can decompose a scene into individual concepts to learn the respective laws of these concepts, which can be either natural (e.g. laws of motion) or man-made (e.g. laws of a game).
Revised: Humans can decompose a scene into individual concepts to learn the respective laws of these concepts, which can be either natural (e.g. laws of motion) or man-made (e.g. laws of a game).

Original: When observing a scene of a moving ball, one tends to parse the changing patterns of its appearance and position separately:
Revised: When observing a scene of a moving ball, one tends to parse the changing patterns of its appearance and position separately:

Original: The appearance stays consistent over time, while the position changes according to the laws of motion.
Revised: The appearance stays consistent over time, while the position changes according to the laws of motion.

Original: By composing the laws of the ball’s appearance and position, one can understand the changing pattern and predict the status of the moving ball.
Revised: By composing the laws of the ball’s appearance and position, one can understand the changing pattern and predict the status of the moving ball.

Original: Although compositionality has inspired a number of models in visual understanding such as representing handwritten characters through hierarchical decomposition of characters (Lake et al., 2011; 2015) and representing a multi-object scene with object-centric representations (Eslami et al., 2016;
Revised: Although compositionality has inspired a number of models in visual understanding such as representing handwritten characters through hierarchical decomposition of characters (Lake et al., 2011; 2015) and representing a multi-object scene with object-centric representations (Eslami et al., 2016;

Original: Kosiorek et al., 2018; Greff et al., 2019; Locatello et al., 2020), automatic parsing of laws in a scene is still a great challenge.
Revised: Kosiorek et al., 2018; Greff et al., 2019; Locatello et al., 2020), automatic parsing of laws in a scene is still a great challenge.

Original: For example, to understand the rules in abstract visual reasoning such as
Revised: For example, to understand the rules in abstract visual reasoning such as

Original: Raven’s Progressive Matrices (RPMs) test, the comprehension of attribute-speciﬁc representations and the underlying relationships among them is crucial for a model to predict the missing images (Santoro et al., 2018; Steenbrugge et al., 2018; Wu et al., 2020).
Revised: Raven’s Progressive Matrices (RPMs) test, the comprehension of attribute-speciﬁc representations and the underlying relationships among them is crucial for a model to predict the missing images (Santoro et al., 2018; Steenbrugge et al., 2018; Wu et al., 2020).

Original: To understand the laws of motion in intuitive physics, a model needs to grasp the changing patterns of different attributes (e.g. appearance and position) of each object in a scene to predict the future (Agrawal et al., 2016;
Revised: To understand the laws of motion in intuitive physics, a model needs to grasp the changing patterns of different attributes (e.g. appearance and position) of each object in a scene to predict the future (Agrawal et al., 2016;

Original: Kubricht et al., 2017; Ye et al., 2018).
Revised: Kubricht et al., 2017; Ye et al., 2018).

Original: However, these methods usually employ neural networks to directly model changing patterns of the scene in a black box, and can hardly abstract the laws of individual concepts in an explicit, interpretable and even manipulatable way.
Revised: However, these methods usually employ neural networks to directly model changing patterns of the scene in a black box, and can hardly abstract the laws of individual concepts in an explicit, interpretable and even manipulatable way.

Original: A possible solution to enable the abovementioned ability for a model is exploiting a function to represent a law of concept in terms of the representation of the concept itself.
Revised: A possible solution to enable the abovementioned ability for a model is exploiting a function to represent a law of concept in terms of the representation of the concept itself.

Original: To represent a law that may depict arbitrary changing patterns, an expressive and ﬂexible random function is required.
Revised: To represent a law that may depict arbitrary changing patterns, an expressive and ﬂexible random function is required.

Original: The Gaussian Process (GP) is a classical family of random functions that achieves the diversity of function space through different kernel functions (Williams & Rasmussen, 2006).
Revised: The Gaussian Process (GP) is a classical family of random functions that achieves the diversity of function space through different kernel functions (Williams & Rasmussen, 2006).

Original: Recently pro- posed random functions (Garnelo et al., 2018b; Eslami et al., 2018; Kumar et al., 2018; Garnelo et al., 2018a; Singh et al., 2019; Kim et al., 2019; Louizos et al., 2019; Lee et al., 2020; Foong et al., 2020) describe function spaces with the powerful nonlinear ﬁtting ability of neural networks.
Revised: Recently pro- posed random functions (Garnelo et al., 2018b; Eslami et al., 2018; Kumar et al., 2018; Garnelo et al., 2018a; Singh et al., 2019; Kim et al., 2019; Louizos et al., 2019; Lee et al., 2020; Foong et al., 2020) describe function spaces with the powerful nonlinear ﬁtting ability of neural networks.

Original: These random functions have been used to capture the changing patterns in a scene, such as mapping timestamps to frames to describe the physical law of moving objects in a video (Kumar et al., 2018;
Revised: These random functions have been used to capture the changing patterns in a scene, such as mapping timestamps to frames to describe the physical law of moving objects in a video (Kumar et al., 2018;

Original: Singh et al., 2019; Fortuin et al., 2020).
Revised: Singh et al., 2019; Fortuin et al., 2020).

Original: However, these applications of random functions take images as inputs and the captured laws account for all pixels instead of expected individual concepts.
Revised: However, these applications of random functions take images as inputs and the captured laws account for all pixels instead of expected individual concepts.

Original: In this paper, we propose a deep latent variable model for Compositional LAw Parsing (CLAP).
Revised: In this paper, we propose a deep latent variable model for Compositional LAw Parsing (CLAP).

Original: CLAP achieves the human-like compositionality ability through an encoding-decoding architecture (Hamrick et al., 2018) to represent concepts in the scene as latent variables, and further employ concept-speciﬁc random functions in the latent space to capture the law on each concept.
Revised: CLAP achieves the human-like compositionality ability through an encoding-decoding architecture (Hamrick et al., 2018) to represent concepts in the scene as latent variables, and further employ concept-speciﬁc random functions in the latent space to capture the law on each concept.

Original: By means of the plug-in of different random functions, CLAP gains generality and ﬂexibility applicable to various law parsing tasks.
Revised: By means of the plug-in of different random functions, CLAP gains generality and ﬂexibility applicable to various law parsing tasks.

Original: We introduce CLAP-NP as an example that instantiates latent random functions with recently proposed Neural Processes (Garnelo et al., 2018b).
Revised: We introduce CLAP-NP as an example that instantiates latent random functions with recently proposed Neural Processes (Garnelo et al., 2018b).

Original: Our experimental results demonstrate that the proposed CLAP outperforms the compared baseline methods in multiple visual tasks including intuitive physics, abstract visual reasoning, and scene representation.
Revised: Our experimental results demonstrate that the proposed CLAP outperforms the compared baseline methods in multiple visual tasks including intuitive physics, abstract visual reasoning, and scene representation.

Original: In addition, the experiment on exchanging latent random functions on a speciﬁc concept and the experiment on composing latent random functions of the given samples to generate new samples both well demonstrate the interpretability and even manipulability of the proposed method for compositional law parsing.
Revised: In addition, the experiment on exchanging latent random functions on a speciﬁc concept and the experiment on composing latent random functions of the given samples to generate new samples both well demonstrate the interpretability and even manipulability of the proposed method for compositional law parsing.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Since CLAP focuses on parsing compositional laws which are modeled by latent random functions, compositional scene representation and random functions are related to this paper.
Revised: Since CLAP focuses on parsing compositional laws which are modeled by latent random functions, compositional scene representation and random functions are related to this paper.

Original: Compositional Scene Representation
Revised: Compositional Scene Representation

Original: Inspired by the compositionality in human cognition, compositional scene representation models  understand a scene by object-centric representations.
Revised: Compositional scene representation models can understand a scene through object-centric representations.

Original: To this end, a class of models initializeobject-centric representations ﬁrst and update them by iterative processes such as Neural Expectation Maximization (Greff et al., 2017), iterative amortized inference (Greff et al., 2019), iterative cross-attention
Revised: Some of them initialize and update objectcentric representations by iterative processes such as Neural Expectation Maximization (Greff et al., 2017), iterative amortized inference (Greff et al., 2019), iterative cross-attention

Original: (Locatello et al., 2020), and two-stage inference (Emami et al., 2021).
Revised: (Locatello et al., 2020), and two-stage inference (Emami et al., 2021).

Original: Another class of models extracts object-centric representationsfrom a scene without iterative inference processes.
Revised: Another class of models learns object-centric representations without iterative inference processes.

Original: AIR (Eslami et al., 2016) parses object-centricpresence, appearance, and position representations as latent variables.
Revised: AIR (Eslami et al., 2016) parses object-centric representations as latent variables.

Original: SPACE (Lin et al., 2020b) divides a scene image intosmall regions and parses region-speciﬁc representations in parallel.
Revised: SPACE (Lin et al., 2020b) divides the image into regions and learns region-speciﬁc representations in parallel.

Original: GNM (Jiang & Ahn, 2020) models the layout of scenes.
Revised: GNM (Jiang & Ahn, 2020) models the layout of scenes.

Original: Recently, many models aim at parsing compositional scene representations in videos (Kosiorek et al., 2018; Jiang et al., 2019; Lin et al., 2020a).
Revised: Recently, many models aim at learning compositional scene representations in videos (Kosiorek et al., 2018; Jiang et al., 2019; Lin et al., 2020a).

Revised: Unlike CLAP, compositional scene representation models learn how objects in a scene are composed but cannot explicitly represent underlying laws and understand how these laws constitute the changing pattern of scenes.

Original: Random Functions
Revised: Random Functions

Original: The GP (Williams & Rasmussen, 2006) is a classical family of random functions (Williams & Rasmussen, 2006) that regards the outputs of a function as a random variable of multivariate Gaussian distribution.
Revised: The GP (Williams & Rasmussen, 2006) is a classical family of random functions (Williams & Rasmussen, 2006) that regards the outputs of a function as a random variable of multivariate Gaussian distribution.

Original: GQN (Eslami et al., 2018) represents 3D scenes as functions by a well-designed representation network and generation network.
Revised: GQN (Eslami et al., 2018) represents 3D scenes as functions by a well-designed representation network and generation network.

Original: NP (Garnelo et al., 2018b) captures function stochasticity by a Gaussian distributed latent variable instead of a deterministic representation in CNP (Garnelo et al., 2018a).
Revised: NP (Garnelo et al., 2018b) captures function stochasticity by a Gaussian distributed latent variable instead of a deterministic representation in CNP (Garnelo et al., 2018a).

Original: Considering different ways of stochasticity modeling, context aggregation, and output prediction for NP, we can construct function spaces with different characteristics (Kim et al., 2019;
Revised: Considering different ways of stochasticity modeling, context aggregation, and output prediction for NP, we can construct function spaces with different characteristics (Kim et al., 2019;

Original: Louizos et al., 2019; Singh et al., 2019; Lee et al., 2020; Foong et al., 2020).
Revised: Louizos et al., 2019; Singh et al., 2019; Lee et al., 2020; Foong et al., 2020).

Original: Some few-shot learning methods parse functions from data by learning global variables (Wu et al., 2018; Gordon et al., 2019) or adaptive kernels (Tossou et al., 2019;
Revised: Some few-shot learning methods parse functions from data by learning global variables (Wu et al., 2018; Gordon et al., 2019) or adaptive kernels (Tossou et al., 2019;

Original: Patacchiola et al., 2020).
Revised: Patacchiola et al., 2020).

Original: We can also combine latent variable models with ODEs (Norcliffe et al., 2021) or SDEs (Li et al., 2020; Hasan et al., 2021) to describe random functions by the integral over latent states.
Revised: We can also combine latent variable models with ODEs (Norcliffe et al., 2021) or SDEs (Li et al., 2020; Hasan et al., 2021) to describe random functions by the integral over latent states.

Revised: Random functions provide explicit representations for laws but cannot model the compositionality of laws.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: A random function describes a distribution over function space F , from which we can sample a function f mapping the input sequence ( x 1 , ..., x N ) to output sequence ( y 1 , ..., y N ) .
Revised: A random function describes a distribution over function space F , from which we can sample a function f mapping the input sequence ( x 1 , ..., x N ) to output sequence ( y 1 , ..., y N ) .

Original: The important problem for a random function is ﬁnding the underlying function over some given points and predicting function values at novel positions accordingly.
Revised: The impor- [Figure] tant problem for a random function is ﬁnding the underlying function over some given points and predicting function values at novel positions accordingly.

Original: In detail, given a set of context points
Revised: In detail, given a set of context points

Original: [Figure]

Original: D S
Revised: D S

Original: = { ( x s , y s ) | s ∈ S } , we should ﬁnd the most possible function that y s = f
Revised: = { ( x s , y s ) | s ∈ S } , we should ﬁnd the most possible function that y s = f

Original: ( x s ) for these context points, then use f ( x t ) to predict y t for target points D T
Revised: ( x s ) for these context points, then use f ( x t ) to predict y t for target points D T

Original: = { ( x t , y t ) | t ∈ T } .
Revised: = { ( x t , y t ) | t ∈ T } .

Original: The abovementioned prediction problem can be regarded as estimating a conditional probability p ( y T | x T , D S ) .
Revised: The abovementioned prediction problem can be regarded as estimating a conditional probability p ( y T | x T , D S ) .

Original: Neural Process (NP)
Revised: Neural Process (NP)

Original: NP (Garnelo et al., 2018b) captures function stochasticity by a Gaussian distributed latent variable g and lets p ( y T | x T , D S )
Revised: NP (Garnelo et al., 2018b) captures function stochasticity by a Gaussian distributed latent variable g and lets p ( y T | x T , D S )

Original: = (cid:82) (cid:81) t ∈ T p ( y t | g , x t ) q ( g | D S )
Revised: = (cid:82) (cid:81) t ∈ T p ( y t | g , x t ) q ( g | D S )

Original: d g .
Revised: d g .

Original: In this case, p ( y T | x T , D S ) consists of a generative process p ( y t | g , x t ) and inference process q ( g | D S ) , which can be jointly optimized by variational inference.
Revised: In this case, p ( y T | x T , D S ) consists of a generative process p ( y t | g , x t ) and inference process q ( g | D S ) , which can be jointly optimized by variational inference.

Original: To estimate p ( y T | x T , D S ) , NP ﬁrst extracts the feature of each context point in D S and sum them up to acquire the mean µ and deviation σ of g : µ, σ = T f (cid:32)| S | (cid:88) s ∈ S T a ( y s , x s ) (cid:33) .
Revised: To estimate p ( y T | x T , D S ) , NP ﬁrst extracts the feature of each context point in D S and sum them up to acquire the mean µ and deviation σ of g : µ

Original: [Equation]
Revised: [Equation]

Original: Then NP samples the global latent variable g from q ( g | D S ) .
Revised: Then NP samples the global latent variable g from q ( g | D S ) .

Original: NP can independently predict y t at the target position x t by the generative process p ( y t | g , x t ) =
Revised: NP can independently predict y t at the target position x t by the generative process p ( y t | g , x t ) =

Original: N ( y t |T p ( g , x t ) , σ 2 I ) .
Revised: N ( y t |T p ( g , x t ) , σ 2 I ) .

Original: Neural networks
Revised: Neural networks

Original: T a , T f and T p increase the model capacity of NP to learn data-speciﬁc function spaces.
Revised: T a , T f and T p increase the model capacity of NP to learn data-speciﬁc function spaces.

Original:  M ODEL
Revised:  M ODEL

Original: Figure 1 is an overview of CLAP.
Revised: Figure 1 is an overview of CLAP.

Original: First, an encoder converts context images to individual concepts in the latent space.
Revised: First, an encoder converts context images to individual concepts in the latent space.

Original: Then CLAP parses concept-speciﬁc random functions on the context visual concepts, which are exploited to predict the target concepts.
Revised: Then CLAP parses concept-speciﬁc random functions on the context visual concepts, which are exploited to predict the target concepts.

Original: Finally, a decoder maps the target concepts to target images.
Revised: Finally, a decoder maps the target concepts to target images.

Original: The concept-speciﬁc latent random functions achieve the compositional law parsing in CLAP.
Revised: The concept-speciﬁc latent random functions achieve the compositional law parsing in CLAP.

Original: In the following part, we will introduce the generative process, variational inference, and parameter learning of CLAP.
Revised: In the following part, we will introduce the generative process, variational inference, and parameter learning of CLAP.

Original: At the end of this section, we propose CLAP-NP, an instance that instantiates the latent random function with NP.
Revised: At the end of this section, we propose CLAP-NP, an instance that instantiates the latent random function with NP.

Original: 4. L ATENT R ANDOM F UNCTIONS
Revised: 4. L ATENT R ANDOM F UNCTIONS

Original: Given D S and D T , the objective of CLAP is to maximize log p ( y T | y S , x ) .
Revised: Given D S and D T , the objective of CLAP is to maximize log p ( y T | y S , x ) .

Original: In the framework of conditional latent variable models, we introduce a variational posterior q ( h | y , x ) for the latent variable h , and approximate log-likelihood by evidence lower bound (ELBO) (Sohn et al., 2015):
Revised: In the framework of conditional latent variable models, we introduce a variational posterior q ( h | y , x ) for the latent variable h , and approximate log-likelihood by evidence lower bound (ELBO) (Sohn et al., 2015):

Original: [Equation]
Revised: [Equation]

Original: The latent variable h includes the local information of images
Revised: The latent variable h includes the local information of images

Original: z = { z n } Nn =1 where z n is the low-dimensional representation of x n .
Revised: z = { z n } Nn =1 where z n is the low-dimensional representation of x n .

Original: Moreover, we decompose z n into independent concepts { z cn | c ∈ C } (e.g., an image of a single-object scene may have concepts such as object appearance,illumination, camera orientation, etc.) where C refers to the name set of these concepts. Assuming that these concepts satisfy their respective change patterns, we employ independent latent random
Revised: Moreover, we decompose z n into independent concepts { z cn | c ∈ C } (e.g., an image of a single-object scene may have concepts such as object appearance,

Original: 𝐶 𝑆 𝒚 !
Revised: 𝐶 𝑆 𝒚 !

Original: 𝒙 !
Revised: 𝒙 !

Original: 𝒛 !"
Revised: 𝒛 !"

Original: 𝒛 #" 𝒚 # 𝒙 # 𝑇
Revised: 𝒛 #" 𝒚 # 𝒙 # 𝑇

Original: Latent Random Function 𝑓 !
Revised: Latent Random Function 𝑓 !

Original: (a) CLAP 𝒙 !
Revised: (a) CLAP 𝒙 !

Original: 𝒛 !"
Revised: 𝒛 !"

Original: 𝒛 #" 𝒙 #
Revised: 𝒛 #" 𝒙 #

Original: NP 𝒈 " 𝒯 !"
Revised: NP 𝒈 " 𝒯 !"

Original: (b) NP as the latent random function  functions to capture the law of each concept.
Revised: (b) NP as the latent random function illumination, camera orientation, etc.) where C refers to the name set of these concepts. Assuming that these concepts satisfy their respective change patterns, we employ independent latent random functions to capture the law of each concept.

Original: Denote the latent random function on the concept c as f c , the speciﬁc form of f c depends on the way we model it.
Revised: Denote the latent random function on the concept c as f c , the speciﬁc form of f c depends on the way we model it.

Original: As Figure 2b shows, if adopting NP as the latent random function, we have f c ( x t ) =
Revised: As Figure 2b shows, if adopting NP as the latent random function, we have f c ( x t ) =

Original: T cp ( g c , x t ) where g c is a Gaussian-distributed latent variable to control the randomness of f c .
Revised: T c p ( g c , x t ) where g c is a Gaussian-distributed latent variable to control the randomness of f c .

Original: Within the graphical model of CLAP in Figure 2a, the prior and posterior in L are factorized into
Revised: Within the graphical model of CLAP in Figure 2a, the prior and posterior in L are factorized into

Original: [Equation]
Revised: [Equation]

Original:  Decomposing f into concept-speciﬁc laws { f c | c ∈ C } in the above process is critical for CLAP’s compositionality.
Revised: q [Equation] Decomposing f into concept-speciﬁc laws { f c | c ∈ C } in the above process is critical for CLAP’s compositionality.

Original: The compositionality makes it possible to parse only several concept-speciﬁc laws rather than the entire law on z , which reduces the complexity of law parsing while increasing the interpretability of laws.
Revised: The compositionality makes it possible to parse only several concept-speciﬁc laws rather than the entire law on z , which reduces the complexity of law parsing while increasing the interpretability of laws.

Original: The interpretability allows us to edit laws, for example, exchanging the law on some concepts and composing the laws of existing samples.
Revised: The interpretability allows us to edit laws, for example, exchanging the law on some concepts and composing the laws of existing samples.

Original: 4. P ARAMETER L EARNING
Revised: 4. P ARAMETER L EARNING

Original: Based on Equation 3, we factorize ELBO as (See Appendix B.3 for the detailed derivation)
Revised: Based on Equation 3, we factorize ELBO as (See Appendix B.3 for the detailed derivation)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Reconstruction Term
Revised: Reconstruction Term

Original: L r is the sum of the log-likelihood log p ( y t | z t ) of each target image.
Revised: [Equation] is the sum of the log-likelihood log p ( y t | z t ) of each target image.

Original: p ( y t | z t ) is modelled by a decoder that converts concepts to the distribution parameter of target images.
Revised: p ( y t | z t ) is modelled by a decoder that converts concepts to the distribution parameter of target images.

Original: CLAP maximizes L r to connect the latent space with the image space, enabling the decoder to generate high-quality images.
Revised: CLAP maximizes L r to connect the latent space with the image space, enabling the decoder to generate high-quality images.

Original: Target Regularizor
Revised: Target Regularizor

Original: L t consists of Kullback–Leibler (KL) divergences between the posterior q ( z ct | y t ) and the prior p ( z ct | f c , z cS , x S , x t ) of target concepts.
Revised: L t consists of Kullback–Leibler (KL) divergences between the posterior q ( z ct | y t ) and the prior p ( z ct | f c , z cS , x S , x t ) of target concepts.

Original: In q ( z ct | y t ) , the distribution of target concepts is computed directly from the target image by the encoder, so it usually contains more accurate information.
Revised: In q ( z ct | y t ) , the distribution of target concepts is computed directly from the target image by the encoder, so it usually contains more accurate information.

Original: And in p ( z c t | f c , z c S , x S , x t ) , the distribution of the target concept is estimated from the context.
Revised: And in p ( z ct | f c , z cS , x S , x t ) , the distribution of the target concept is estimated from the context.

Original: The minimization of L t ﬁlls the gap between the posterior and prior and ensures the accuracy of predictors.
Revised: The minimization of L t ﬁlls the gap between the posterior and prior and ensures the accuracy of predictors.

Original: Function Regularizor
Revised: Function Regularizor

Original: L f consists of KL divergences between the posterior q ( f c |
Revised: L f consists of KL divergences between the posterior q ( f c |

Original: z c , x ) and the prior p ( f c | z cS , x S ) .
Revised: z c , x ) and the prior p ( f c | z cS , x S ) .

Original: We use L f as a measure of function consistency to ensure that model can obtain similar distributions for f c with any subset z cS ⊆
Revised: We use L f as a measure of function consistency to ensure that model can obtain similar distributions for f c with any subset z cS ⊆

Original: z c and x S ⊆ x as inputs.
Revised: z c and x S ⊆ x as inputs.

Original: To this end, CLAP shares concept-speciﬁc function parsers to model q ( f c |
Revised: To this end, CLAP shares concept-speciﬁc function parsers to model q ( f c |

Original: z c , x ) and p ( f c | z cS , x S ) .
Revised: z c , x ) and p ( f c | z cS , x S ) .

Original: And we need to design the architecture of function parsers so that they can take different subsets as input.
Revised: And we need to design the architecture of function parsers so that they can take different subsets as input.

Original: Context Regularizor
Revised: Context Regularizor

Original: L s consists of KL divergences between the posterior q ( z cs | y s ) and the prior p ( z cs | y s ) of context concepts.
Revised: L s consists of KL divergences between the posterior q ( z cs | y s ) and the prior p ( z cs | y s ) of context concepts.

Original: To avoid the mismatch between the posterior and prior and reduce model parameters, the posterior and prior are parameterized with the same encoder.
Revised: To avoid the mismatch between the posterior and prior and reduce model parameters, the posterior and prior are parameterized with the same encoder.

Original: In this case, we have p ( z cs | y s )
Revised: In this case, we have p ( z cs | y s )

Original: =
Revised: =

Original: q ( z cs | y s ) and further L s = 0 .
Revised: q ( z cs | y s ) and further L s = 0 .

Original: So we remove L s from Equation 4.
Revised: So we remove L s from Equation 4.

Original: Correct decomposition of concepts is essential for model performance.
Revised: Correct decomposition of concepts is essential for model performance.

Original: However, CLAP is trained without additional supervision to guide the learning of concepts.
Revised: However, CLAP is trained without additional supervision to guide the learning of concepts.

Original: Although CLAP explicitly deﬁnes independent concepts in priors and posteriors, it is not sufﬁcient to automatically decompose concepts for some complex data.
Revised: Although CLAP explicitly deﬁnes independent concepts in priors and posteriors, it is not sufﬁcient to automatically decompose concepts for some complex data.

Original: To solve the problem, CLAP can introduce extra inductive biases by designing proper encoder and decoder to artiﬁcially decompose concepts, e.g., use spatial transformation layers to decompose spatial and non-spatial concepts (Skafte & Hauberg, 2019).
Revised: To solve the problem, CLAP can introduce extra inductive biases by designing proper encoder and decoder to artiﬁcially decompose concepts, e.g., use spatial transformation layers to decompose spatial and non-spatial concepts (Skafte & Hauberg, 2019).

Original: We can also set hyperparameters to control the regularizers (Higgins et al., 2017), or add a total correlation (TC) regularizer R TC to help concept learning (Chen et al., 2018).
Revised: We can also set hyperparameters to control the regularizers (Higgins et al., 2017), or add a total correlation (TC) regularizer R TC to help concept learning (Chen et al., 2018).

Original: Here we choose to add extra regularizers to ELBO of CLAP, and ﬁnally, the optimization objective becomes
Revised: Here we choose to add extra regularizers to ELBO of CLAP, and ﬁnally, the optimization objective becomes

Original: [Equation]
Revised: [Equation]

Original: 5) where β t , β f , β TC are hyperparameters that regulate the importance of regularizers in the training process.
Revised: (5) where β t , β f , β TC are hyperparameters that regulate the importance of regularizers in the training process.

Original: We can compute L ∗ by Stochastic Gradient Variational Bayes (SGVB) estimator (Kingma & Welling, 2013) and update parameters by gradient descent optimizers (Kingma & Ba, 2015).
Revised: We can compute L ∗ by Stochastic Gradient Variational Bayes (SGVB) estimator (Kingma & Welling, 2013) and update parameters by gradient descent optimizers (Kingma & Ba, 2015).

Original: 4. NP AS L ATENT R ANDOM F UNCTION
Revised: 4. NP AS L ATENT R ANDOM F UNCTION

Original: In this subsection, we propose CLAP-NP as an example that instantiates the latent random function with NP.
Revised: In this subsection, we propose CLAP-NP as an example that instantiates the latent random function with NP.

Original: According to NP, we employ a latent variable g c to control the randomness of the latent random function f c for each concept in CLAP-NP.
Revised: According to NP, we employ a latent variable g c to control the randomness of the latent random function f c for each concept in CLAP-NP.

Original: Generative Process
Revised: Generative Process

Original: CLAP-NP ﬁrst encodes each context image x s to the concepts:
Revised: CLAP-NP ﬁrst encodes each context image x s to the concepts:

Original: [Equation]
Revised: [Equation]

Original: To stabilize the learning of concepts, the encoder outputs only the mean of Gaussian distribution, with the standard deviation as a hyperparameter.
Revised: To stabilize the learning of concepts, the encoder outputs only the mean of Gaussian distribution, with the standard deviation as a hyperparameter.

Original: Using the process in Equation 1 for each concept, a concept-speciﬁc function parser aggregates the contextual information and transform it to the mean µ cg and standard deviation σ cg of g c .
Revised: Using the process in Equation 1 for each concept, a concept-speciﬁc function parser aggregates the contextual information and transform it to the mean µ cg and standard deviation σ cg of g c .

Original: The concept-speciﬁc target predictor takes g c ∼ N ( µ cg , diag ( σ 2 g )) and x t as inputs and output the mean µ cz,t of z t , leaving the standard deviation as a hyperparameter.
Revised: The concept-speciﬁc target predictor takes g c ∼ N ( µ cg , diag ( σ 2 g )) and x t as inputs and output the mean µ cz,t of z t , leaving the standard deviation as a hyperparameter.

Original: To keep independence between concepts, CLAP-NP introduces identical but independent function parsers and target predictors.
Revised: To keep independence between concepts, CLAP-NP introduces identical but independent function parsers and target predictors.

Original: Once all of the concepts z ct ∼ N ( µ cz,t , σ 2 z I ) for c ∈ C are generated, we concatenate and decode them into target images:
Revised: Once all of the concepts z c t ∼ N ( µ c z,t , σ 2 z I ) for c ∈ C are generated, we concatenate and decode them into target images:

Original: [Equation]
Revised: [Equation]

Original: To control the noise in sampled images, we introduce a hyperparameter σ x as the standard deviation.
Revised: To control the noise in sampled images, we introduce a hyperparameter σ x as the standard deviation.

Original: As Figure 2 shows, we can rewrite the prior p ( h | y S , x ) in CLAP-NP as
Revised: As Figure 2 shows, we can rewrite the prior p ( h | y S , x ) in CLAP-NP as

Original: [Equation]
Revised: [Equation]

Original: Inference and Learning
Revised: Inference and Learning

Original: In the variational inference, other than that the randomness of f c is replaced by g c , the posterior of CLAP-NP is the same as that in Equation 3:
Revised: In the variational inference, other than that the randomness of f c is replaced by g c , the posterior of CLAP-NP is the same as that in Equation 3:

Original: [Equation]
Revised: [Equation]

Original: In the ﬁrst stage, we compute the mean of both z S and z T by the encoder.
Revised: In the ﬁrst stage, we compute the mean of both z S and z T by the encoder.

Original: Because CLAP shares the encoder in the prior and posterior, in the training process we can use the results in Equationof the generative process instead of recalculating the mean of z S . Then we compute the distribution
Revised: Because CLAP shares the encoder in the prior and posterior, in the training process we can use the results in Equation

Original: [Table]
Revised: [Table]

Original: 650.4 ± 12.5 GQN 414.3 ± 7.
Revised: 650.4 ± 12.5 GQN 414.3 ± 7.

Original: 823.0 ± 15.
Revised: 823.0 ± 15.

Original: 55.6 ± 0.
Revised: 55.6 ± 0.

Original: 331.8 ± 2.
Revised: 331.8 ± 2.

Original: 772.1 ± 3.
Revised: 772.1 ± 3.

Original: 1148.9 ± 3.2 CLAP-NP 119.5 ± 17.
Revised: 1148.9 ± 3.2 CLAP-NP 119.5 ± 17.

Original: 264.5 ± 27.
Revised: 264.5 ± 27.

Original: 58.8 ± 2.
Revised: 58.8 ± 2.

Original: 153.2 ± 6.
Revised: 153.2 ± 6.

Original: 105.5 ± 0.
Revised: 105.5 ± 0.

Original: 213.9 ± 1. 
Revised: 213.9 ± 1. of the generative process instead of recalculating the mean of z S .

Original:  parameters of g c by the shared concept-speciﬁc function parsers in generative process where the input in the variational inference are z c and x instead of z cS and x S .
Revised: Then we compute the distribution parameters of g c by the shared concept-speciﬁc function parsers in generative process where the input in the variational inference are z c and x instead of z cS and x S .

Original: With the above p ( h | y S , x ) and q ( h | y , x ) , some subterms in the ELBO of CLAP-NP becomeL
Revised: With the above p ( h | y S , x ) and q ( h | y , x ) , some subterms in the ELBO of CLAP-NP become

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate the model’s ability of compositional law parsing on different types of data, we use three datasets in the experiments: (1) bouncing ball (abbreviated as BoBa ) dataset (Lin et al., 2020a) to validate the ability of intuitive physics; (2) Continuous Raven’s Progressive Matrix (CRPM) dataset (Shi et al., 2021) to validate the ability of abstract visual reasoning; (3) MPI3D dataset (Gondal et al., 2019) to validate the ability of scene representation.
Revised: To evaluate the model’s ability of compositional law parsing on different types of data, we use three datasets in the experiments: (1) bouncing ball (abbreviated as BoBa ) dataset (Lin et al., 2020a) to validate the ability of intuitive physics; (2) Continuous Raven’s Progressive Matrix (CRPM) dataset (Shi et al., 2021) to validate the ability of abstract visual reasoning; (3) MPI3D dataset (Gondal et al., 2019) to validate the ability of scene representation.

Original: We use NP (Garnelo et al., 2018b), GP with the deep kernel (Wilson et al., 2016), and GQN (Eslami et al., 2018) as baselines.
Revised: We use NP (Garnelo et al., 2018b), GP with the deep kernel (Wilson et al., 2016), and GQN (Eslami et al., 2018) as baselines.

Original: We consider NP and GP as baselines because they are representative random functions that construct the function space in different ways, and CLAP-NP instantiates the latent random function with NP.
Revised: We consider NP and GP as baselines because they are representative random functions that construct the function space in different ways, and CLAP-NP instantiates the latent random function with NP.

Original: GQN is adopted since it models the distribution of image-valued functions, which highly ﬁts our experimental settings.
Revised: GQN is adopted since it models the distribution of image-valued functions, which highly ﬁts our experimental settings.

Original: NP is not used to model image-valued functions, so we add an encoder and decoder to NP’s aggregator and predictor to help it handle high-dimensional images.
Revised: NP is not used to model image-valued functions, so we add an encoder and decoder to NP’s aggregator and predictor to help it handle high-dimensional images.

Original: For GP, we use a pretrained encoder and decoder to reduce the dimension of raw images.
Revised: For GP, we use a pretrained encoder and decoder to reduce the dimension of raw images.

Original: See Appendix C.1 and
Revised: See Appendix C.1 and

Original: C.2 for a detailed introduction to datasets, hyperparameters, and architectures of models.
Revised: C.2 for a detailed introduction to datasets, hyperparameters, and architectures of models.

Original: For quantitative experiments, we take Mean Squared Error (MSE) and Selection Accuracy (SA) as evaluation metrics.
Revised: For quantitative experiments, we take Mean Squared Error (MSE) and Selection Accuracy (SA) as evaluation metrics.

Original: Since non-compositional law parsing will increase the risk of producing undesired changes on constant concepts and further cause pixel deviations in predictions, we adopt
Revised: Since non-compositional law parsing will increase the risk of producing undesired changes on constant concepts and further cause pixel deviations in predictions, we adopt

Original: MSE as an indicator of semantic error.

Original: To compute the concept-level deviations between predictions

Original: G T
Revised: G T

Original: N P
Revised: N P

Original: GQN
Revised: GQN

Original: C L A P - N P
Revised: C L A P - N P

Original: G P
Revised: G P

Original: NP
Revised: NP

Original: GQN
Revised: GQN

Original: CLAP
Revised: CLAP

Original: [Figure]
Revised: [Figure]

Original: NP
Revised: NP

Original: Prediction
Revised: Prediction

Original: [Figure]

Original: Prediction
Revised: Prediction

Original: Prediction Prediction 
Revised: Prediction Prediction Prediction

Original: Prediction PredictionPrediction
Revised: Prediction Prediction

Original: Prediction Prediction
Revised: Prediction Prediction

Original: Prediction Prediction
Revised: Prediction Prediction

Original: Prediction 2 P r e d i c ti m a g e s
Revised: Prediction 2 P r e d i c ti m a g e s

Original: P r e d i c t a r o w Test sample
Revised: P r e d i c t a r o w Test sample

Original: GP
Revised: GP

Original: Prediction Prediction
Revised: Prediction Prediction

Revised: Prediction

Original: Prediction
Revised: Prediction

Original: [Figure]

Original: We provide two predictions of the same context to illustrate a special case: when a row of images is removed from the given matrix, all predictions with the progressive law of size and constant law of color are correct.
Revised: We provide two predictions of the same context to illustrate a special case: when a row of images is removed from the given matrix, all predictions with the progressive law of size and constant law of color are correct.

Original: In this special case, only CLAP-NP captures such prediction uncertainty.
Revised: In this special case, only CLAP-NP captures such prediction uncertainty.

Original:  and ground truths, which we think can be an indicator of semantic error, we introduce SA as another metric to calculate the distance on the representation space.
Revised: MSE as an indicator of semantic error. To compute the concept-level deviations between predictions and ground truths, which we think can be an indicator of semantic error, we introduce SA as another metric to calculate the distance on the representation space.

Original: We compute SA by picking out correct ground truths from a set of candidates because the representation spaces vary greatly among models.
Revised: We compute SA by picking out correct ground truths from a set of candidates because the representation spaces vary greatly among models.

Original: For a sample x , we ﬁrst combine its ground truth x T with those from other K samples to establish the candidate set { d k } K +1 k
Revised: For a sample x , we ﬁrst combine its ground truth x T with those from other K samples to establish the candidate set { d k } K +1 k

Original: =1 .
Revised: =1 .

Original: Then these candidates and the prediction ˜ x T ∼ p ( x T | x S ) are encoded into representations to calculate concept-level distances (cid:107) Encoder (˜ x T ) − Encoder ( d k ) (cid:107) 22 .
Revised: Then these candidates and the prediction ˜ x T ∼ p ( x T | x S ) are encoded into representations to calculate concept-level distances (cid:107) Encoder (˜ x T ) − Encoder ( d k ) (cid:107) 22 .

Original: Finally, the candidate with the smallest distance is chosen, and SA is the selection accuracy across all test samples.
Revised: Finally, the candidate with the smallest distance is chosen, and SA is the selection accuracy across all test samples.

Original: In the following experiments, we use η = N T / N to denote the training or test conﬁguration where N T of N images in each sample are prediction targets.
Revised: In the following experiments, we use η = N T / N to denote the training or test conﬁguration where N T of N images in each sample are prediction targets.

Original: 5. I NTUITIVE P HYSICS
Revised: 5. I NTUITIVE P HYSICS

Original: To evaluate the intuitive physics of models, which is the underlying knowledge for humans to understand the evolution of the physical world (Kubricht et al., 2017), we use BoBa dataset where physical laws of collisions between colored bouncing balls are described as functions mapping timestamps to frames.
Revised: To evaluate the intuitive physics of models, which is the underlying knowledge for humans to understand the evolution of the physical world (Kubricht et al., 2017), we use BoBa dataset where physical laws of collisions between colored bouncing balls are described as functions mapping timestamps to frames.

Original: BoBa-1 and BoBa-2 are one-ball and two-ball instances of BoBa.
Revised: BoBa-1 and BoBa-2 are one-ball and two-ball instances of BoBa.

Original: When predicting target frames, models should learn both constant laws of appearance and physical laws of collisions.
Revised: When predicting target frames, models should learn both constant laws of appearance and physical laws of collisions.

Original: We set up two test conﬁgurations η = 4/12 and η = 6/12 for BoBa.
Revised: We set up two test conﬁgurations η = 4/12 and η = 6/12 for BoBa.

Original: The quantitative results in Table 1 and
Revised: The quantitative results in Table 1 and

Original: Figure 3 illustrate CLAP-NP’s intuitive physics on both BoBa-1 and BoBa-2.
Revised: Figure 3 illustrate CLAP-NP’s intuitive physics on both BoBa-1 and BoBa-2.

Original: Figure 4 visualizes the predictions of BoBa-2 where NP can only generate blurred images while GP and GQN can hardly understand the appearance constancy of balls.
Revised: Figure 4 visualizes the predictions of BoBa-2 where NP can only generate blurred images while GP and GQN can hardly understand the appearance constancy of balls.

Original: That is, balls in the predictions of GP have different colors, and GQN generates non-existent balls and ignores existing balls.
Revised: That is, balls in the predictions of GP have different colors, and GQN generates non-existent balls and ignores existing balls.

Original: Instead, CLAP-NP can separately capture the appearance constancy and collision laws.
Revised: Instead, CLAP-NP can separately capture the appearance constancy and collision laws.

Original: This beneﬁts the law parsing in scenes with novel appearances and physical laws (Appendix D.7).
Revised: This beneﬁts the law parsing in scenes with novel appearances and physical laws (Appendix D.7).

Original: 5. A BSTRACT V ISUAL R EASONING
Revised: 5. A BSTRACT V ISUAL R EASONING

Original: Abstract visual reasoning is a manifestation of human’s ﬂuid intelligence (Cattell, 1963).
Revised: Abstract visual reasoning is a manifestation of human’s ﬂuid intelligence (Cattell, 1963).

Original: And RPM (Raven & Court, 1998) is a famous nonverbal test widely used to estimate the model’s abstract visual reasoning ability.
Revised: And RPM (Raven & Court, 1998) is a famous nonverbal test widely used to estimate the model’s abstract visual reasoning ability.

Original: RPM is a 3 × 3 image matrix, and the changing rule of the entire matrix consists of multiple attribute-speciﬁc subrules.
Revised: RPM is a 3 × 3 image matrix, and the changing rule of the entire matrix consists of multiple attribute-speciﬁc subrules.

Original: In this experiment, we evaluate models on four instances of
Revised: In this experiment, we evaluate models on four instances of

Original: CRPM dataset and two test conﬁgurations η = 2/9 and η = 3/9.
Revised: CRPM dataset and two test conﬁgurations η = 2/9 and η = 3/9.

Original: In Table 1 and Figure 3, CLAP-NP achieves the best results with the metrics MSE and SA.
Revised: In Table 1 and Figure 3, CLAP-NP achieves the best results with the metrics MSE and SA.

Original: Figure 5 displays the predictions on CRPMDT.
Revised: Figure 5 displays the predictions on CRPMDT.

Original: In the results, NP produces ambiguous-edged outer triangles, GP predicts targets with incorrect  laws, and GQN generates multiple similar images.
Revised: In the results, NP produces ambiguous-edged outer triangles, GP predicts targets with incorrect [Figure] laws, and GQN generates multiple similar images.

Original: It is worth stressing that the answer to a CRPM is[Figure] not unique when we remove the entire line of images from it.
Revised: It is worth stressing that the answer to a CRPM is not unique when we remove the entire line of images from it.

Original: We display examples of this situation in the second row of Figure 5, indicating that CLAP-NP can understand progressive and constant laws on different attributes rather than simple context-to-target mappings.
Revised: We display examples of this situation in the second row of Figure 5, indicating that CLAP-NP can understand progressive and constant laws on different attributes rather than simple context-to-target mappings.

Original: 5. S CENE R EPRESENTATION
Revised: 5. S CENE R EPRESENTATION

Original: In scene representation, we use MPI3D dataset where a robotic arm in scenes manipulates the altitude and azimuth of the object to produce images with different laws.
Revised: In scene representation, we use MPI3D dataset where a robotic arm in scenes manipulates the altitude and azimuth of the object to produce images with different laws.

Original: The laws of scenes can be regarded as functions that project object altitudes or azimuths to RGB images.
Revised: The laws of scenes can be regarded as functions that project object altitudes or azimuths to RGB images.

Original: In this task, the key to predicting correct target images is to determine which law the scene follows by observing given context images.
Revised: In this task, the key to predicting correct target images is to determine which law the scene follows by observing given context images.

Original: We train models under η = 1 ∼ 4/8 and test them with more target or scene images ( η = 4/8, 6/8, 10/40, 20/40).
Revised: We train models under η = 1 ∼ 4/8 and test them with more target or scene images ( η = 4/8, 6/8, 10/40, 20/40).

Original: The MSE and SA scores in Table 1 and Figure 3 demonstrate that CLAPNP outperforms NP in all conﬁgurations and GQN in η = 6/8, 10/40, 20/40.
Revised: The MSE and SA scores in Table 1 and Figure 3 demonstrate that CLAPNP outperforms NP in all conﬁgurations and GQN in η = 6/8, 10/40, 20/40.

Original: CLAP-NP has a slightly higher SA but lower MSE scores than GQN when η = 4/8.
Revised: CLAP-NP has a slightly higher SA but lower MSE scores than GQN when η = 4/8.

Original: A possible reason is that the network of
Revised: A possible reason is that the network of

Original: GQN has good inductive biases and more parameters to generate clearer images and achieve better
Revised: GQN has good inductive biases and more parameters to generate clearer images and achieve better

Original: MSE scores.
Revised: MSE scores.

Original: However, such improvement in the pixel-level reconstruction has limited inﬂuence on the representation space, as well as the SA scores of GQN.
Revised: However, such improvement in the pixel-level reconstruction has limited inﬂuence on the representation space, as well as the SA scores of GQN.

Original: GQN can hardly generalize the random functions to novel situations with more images, leading to performance loss when η = 6/8, 10/40, 20/40.
Revised: GQN can hardly generalize the random functions to novel situations with more images, leading to performance loss when η = 6/8, 10/40, 20/40.

Original: GP’s MSE scores on MPI3D indicate that it is suitable for scenes with more context images (e.g., η =10/40).
Revised: GP’s MSE scores on MPI3D indicate that it is suitable for scenes with more context images (e.g., η =10/40).

Original: With only sparse points, the prediction uncertainty of GP leads to a signiﬁcant decrease in performance.
Revised: With only sparse points, the prediction uncertainty of GP leads to a signiﬁcant decrease in performance.

Original: However, compositionality allows CLAP-NP to parse concept-speciﬁc subrules, which improves the model’s generalization ability in novel test conﬁgurations.
Revised: However, compositionality allows CLAP-NP to parse concept-speciﬁc subrules, which improves the model’s generalization ability in novel test conﬁgurations.

Original: 5. C OMPOSITIONALITY OF L AWS
Revised: 5. C OMPOSITIONALITY OF L AWS

Original: It is possible to exactly editing a law if one could ﬁnd out which latent random functions (LRFs) encode it.
Revised: It is possible to exactly editing a law if one could ﬁnd out which latent random functions (LRFs) encode it.

Original: Here we adopt two methods to ﬁnd the correspondences between laws and LRFs.
Revised: Here we adopt two methods to ﬁnd the correspondences between laws and LRFs.

Original: As Figure 6 shows, we can visualize the traversal in the latent space by perturbing and decoding concepts, and acquire the correspondences by observing changing patterns of these concepts.
Revised: As Figure 6 shows, we can visualize the traversal in the latent space by perturbing and decoding concepts, and acquire the correspondences by observing changing patterns of these concepts.

Original: Another way is to compute variance declines between laws and LRFs (Kim & Mnih, 2018).
Revised: Another way is to compute variance declines between laws and LRFs (Kim & Mnih, 2018).

Original: Assume that there are
Revised: Assume that there are

Original: M types of laws { l m } Mm =1 .
Revised: M types of laws { l m } Mm =1 .

Original: For each law, we can sample an instance by ˜ l m ∼ p ( l m ) and generate a batch of samples B m ∼ p
Revised: For each law, we can sample an instance by ˜ l m ∼ p ( l m ) and generate a batch of samples B m ∼ p

Original: ( B m | ˜ l m ) that follows the law.
Revised: ( B m | ˜ l m ) that follows the law.

Original: First, we generate a batch of samples without ﬁxing any laws and parse LRFs { f c ; c ∈ C } for these samples to estimate the global variances
Revised: First, we generate a batch of samples without ﬁxing any laws and parse LRFs { f c ; c ∈ C } for these samples to estimate the global variances

Original: Var( f c ) .
Revised: Var( f c ) .

Original: Then we generate a series of batches from p ( B m | ˜ l m ) in which all the samples follow the ﬁx law ˜
Revised: Then we generate a series of batches from p ( B m | ˜ l m ) in which all the samples follow the ﬁx law ˜

Original: l m ∼ p ( l m ) and estimate the in-batch variances Var B m ( f c ) .
Revised: l m ∼ p ( l m ) and estimate the in-batch variances Var B m ( f c ) .

Original: Finally the variance decline between l m and f c is computed by s m,c = E p ( l m ) (cid:2)
Revised: Finally the variance decline between l m and f c is computed by s m,c = E p ( l m ) (cid:2)

Original: E p ( B m | l m )
Revised: E p ( B m | l m )

Original: [(Var( f c ) − Var B m ( f c )) /
Revised: [(Var( f c ) − Var B m ( f c )) /

Original: Var( f c )] (cid:3) .
Revised: Var( f c )] (cid:3) .

Original: In Figure 6, we display the variance declines of BoBa-2, CRPM-T, and CRPM-C, which will guide the manipulation of laws in the following experiments.
Revised: In Figure 6, we display the variance declines of BoBa-2, CRPM-T, and CRPM-C, which will guide the manipulation of laws in the following experiments.

Original: Latent Random Function Exchange
Revised: Latent Random Function Exchange

Original: The compositional modeling of laws allows us to swap the law of samples, which is shown at the top of Figure 7.
Revised: The compositional modeling of laws allows us to swap the law of samples, which is shown at the top of Figure 7.

Original: If we want to exchange the law of appearance on two samples of BoBa-2, we ﬁrst refer to the variance declines and ﬁnd that LRFs in the ﬁrst and last dimensions encode the law.
Revised: If we want to exchange the law of appearance on two samples of BoBa-2, we ﬁrst refer to the variance declines and ﬁnd that LRFs in the ﬁrst and last dimensions encode the law.

Original: Then we infer the LRFs of two samples (in CLAP-NP is to obtainthe global latent variables) and swap LRFs of the ﬁrst and last dimensions. Finally, we execute the
Revised: Then we infer the LRFs of two samples (in CLAP-NP is to obtain

Original: [Figure]
Revised: [Figure]

Original:  generative process to regenerate the edited samples, and further, make predictions and interpolations on them.
Revised: the global latent variables) and swap LRFs of the ﬁrst and last dimensions. Finally, we execute the generative process to regenerate the edited samples, and further, make predictions and interpolations on them.

Original: Figure 7 visualizes the law manipulation process, which well illustrates the motivation of our model on compositional law parsing.
Revised: Figure 7 visualizes the law manipulation process, which well illustrates the motivation of our model on compositional law parsing.

Original: Latent Random Function Composition
Revised: Latent Random Function Composition

Original: For samples of CRPM, we manipulate laws by combining the LRFs from existing samples to generate samples with novel laws.
Revised: For samples of CRPM, we manipulate laws by combining the LRFs from existing samples to generate samples with novel laws.

Original: Figure 7 shows the results of law composition on CRPM-T.
Revised: Figure 7 shows the results of law composition on CRPM-T.

Original: The ﬁrst step is parsing LRFs for given samples.
Revised: The ﬁrst step is parsing LRFs for given samples.

Original: By querying the variance declines in Figure 6, we can ﬁnd these LRFs respectively correspond to the law of size, rotation, and color.
Revised: By querying the variance declines in Figure 6, we can ﬁnd these LRFs respectively correspond to the law of size, rotation, and color.

Original: Since we have ﬁgured out the relations between laws and LRFs, the next step is to pick laws from three different samples respectively and compose them into an unseen mixture of laws.
Revised: Since we have ﬁgured out the relations between laws and LRFs, the next step is to pick laws from three different samples respectively and compose them into an unseen mixture of laws.

Original: Finally, the mixed laws are decoded to generate a sample with novel changing patterns.
Revised: Finally, the mixed laws are decoded to generate a sample with novel changing patterns.

Original: This process is similar to the way a human designs RPMs (i.e., sets up separate sub-laws for each attribute and combines them into a complete PRM).
Revised: This process is similar to the way a human designs RPMs (i.e., sets up separate sub-laws for each attribute and combines them into a complete PRM).

Original: From another perspective, law composition provides an option for us to generate novel samples from existing samples (Lake et al., 2015).
Revised: From another perspective, law composition provides an option for us to generate novel samples from existing samples (Lake et al., 2015).

Original:  C ONCLUSION AND L IMITATION
Revised:  C ONCLUSION AND L IMITATIONS

Original: Inspired by the compositionality in human cognition, we propose a deep latent variable model for
Revised: Inspired by the compositionality in human cognition, we propose a deep latent variable model for

Original: Compositional LAw Parsing (CLAP).
Revised: Compositional LAw Parsing (CLAP).

Original: CLAP decomposes high-dimensional images into independent visual concepts in the latent space and employs latent random functions to capture the conceptchanging laws, by which CLAP achieves the compositionality of laws.
Revised: CLAP decomposes high-dimensional images into independent visual concepts in the latent space and employs latent random functions to capture the conceptchanging laws, by which CLAP achieves the compositionality of laws.

Original: To instantiate CLAP, we propose CLAP-NP that uses NPs as latent random functions.
Revised: To instantiate CLAP, we propose CLAP-NP that uses NPs as latent random functions.

Original: The experimental results demonstrate the beneﬁts of our model in intuitive physics, abstract visual reasoning, and scene representation.
Revised: The experimental results demonstrate the beneﬁts of our model in intuitive physics, abstract visual reasoning, and scene representation.

Original: Through the experiments on latent random function exchange and composition, we further qualitatively evaluate the interpretability and manipulability of the learned functions in CLAP.
Revised: Through the experiments on latent random function exchange and composition, we further qualitatively evaluate the interpretability and manipulability of the learned functions in CLAP.

Original: Limitations.
Revised: Limitations.

Original: (1) Complexity of datasets.
Revised: (1) Complexity of datasets.

Original: Because compositional law parsing is an underexplored task, we ﬁrst validate the effectiveness of SVCL on datasets with relatively clear and simple rules to avoid the inﬂuence of unknown confounders in complicated datasets.
Revised: Because compositional law parsing is an underexplored task, we ﬁrst validate the effectiveness of SVCL on datasets with relatively clear and simple rules to avoid the inﬂuence of unknown confounders in complicated datasets.

Original: (2) Setting the number of LRFs.
Revised: (2) Setting the number of LRFs.

Original: For scenes with multiple complex rules, we can empirically set an appropriate upper bound or directly put a large upper bound on the number of LRFs.
Revised: For scenes with multiple complex rules, we can empirically set an appropriate upper bound or directly put a large upper bound on the number of LRFs.

Original: However, using too many LRFs increases model parameters, and the redundant concepts decrease CLAP’s computational efﬁciency.
Revised: However, using too many LRFs increases model parameters, and the redundant concepts decrease CLAP’s computational efﬁciency.

Original: See Appendix E for detailed limitations and future works.
Revised: See Appendix E for detailed limitations and future works.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In Supplementary Materials, (1) we ﬁrst provide the details of ELBO, then (2) introduce the datasets, model architecture, hyperparameters, and computational resources adopted in our experiments, and ﬁnally (3) provide additional experimental results.
Revised: In Supplementary Materials, (1) we ﬁrst provide the details of ELBO, then (2) introduce the datasets, model architecture, hyperparameters, and computational resources adopted in our experiments, and ﬁnally (3) provide additional experimental results.

Original: In particular, we provide additional results of editing and manipulating latent random functions, which validate our motivation and contribution.
Revised: In particular, we provide additional results of editing and manipulating latent random functions, which validate our motivation and contribution.

Original: B D ETAILS OF ELBO B. ELBO OF C ONDITIONAL L ATENT V ARIABLE M ODELS
Revised: B D ETAILS OF ELBO B. ELBO OF C ONDITIONAL L ATENT V ARIABLE M ODELS

Original: [Equation]
Revised: [Equation]

Original: B. P RIOR AND P OSTERIOR F ACTORIZATION
Revised: B. P RIOR AND P OSTERIOR F ACTORIZATION

Original: p ( y T | h ) =
Revised: p ( y T | h ) =

Original: (cid:89) t ∈ T p
Revised: (cid:89) t ∈ T p

Original: [Equation]
Revised: [Equation]

Original: y t | z t ) p ( h | y S
Revised: y t | z t ) p ( h | y S

Original: [Equation]
Revised: [Equation]

Original: x ) = (cid:89) c ∈ C p
Revised: x ) = (cid:89) c ∈ C p

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: z cT | y S
Revised: z cT | y S

Original: [Equation]
Revised: [Equation]

Original: x ) = (cid:89) c ∈ C p ( z cT | f
Revised: x ) = (cid:89) c ∈ C p ( z cT | f

Original: [Equation]
Revised: [Equation]

Original: cS
Revised: cS

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: ) = (cid:89) c ∈ C
Revised: ) = (cid:89) c ∈ C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( h
Revised: ( h

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: x )
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: ) = (cid:89)
Revised: ) = (cid:89)

Original: [Equation]
Revised: [Equation]

Original: = (cid:89)
Revised: = (cid:89)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:33) (13)
Revised: (cid:33) (13)

Original: B. ELBO OF CLAP
Revised: B. ELBO OF CLAP

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation]
Revised: [Table] [Equation]

Original: [Table] [Equation]
Revised: [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original:  B.3. R ECONSTRUCTION T ERM
Revised: [Equation] [Table] [Equation] B.3. R ECONSTRUCTION T ERM

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: , where ˜ z t ∼ q ( z t | y t ) (15)
Revised: , where ˜ z t ∼ q ( z t | y t ) (15)

Original: B.3. T ARGET R EGULARIZER
Revised: B.3. T ARGET R EGULARIZER

Original: [Equation]
Revised: [Equation]

Original: Because of the function consistency in samples
Revised: Because of the function consistency in samples

Original: [Equation]
Revised: [Equation]

Original: function posteriors q ( f c | ˜
Revised: function posteriors q ( f c | ˜

Original: z c , x ) derived from arbitrary sampled ˜
Revised: z c , x ) derived from arbitrary sampled ˜

Original: z c ∼ q ( z c | y ) need to be consistent.
Revised: z c ∼ q ( z c | y ) need to be consistent.

Original: So we let f c condition on only the observations x
Revised: So we let f c condition on only the observations x

Original: and y to simplify the posterior distribution
Revised: and y to simplify the posterior distribution

Original: [Equation]
Revised: [Equation]

Original: that is, we replace q ( f c | ˜
Revised: that is, we replace q ( f c | ˜

Original: z c , x ) with q ( f c | y , x ) :
Revised: z c , x ) with q ( f c | y , x ) :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In this way
Revised: In this way

Original: [Equation]
Revised: [Equation]

Original: we convert the computation of the log-likelihoods on q ( z c t | y t ) and p ( z ct | f c , z c S , x S , x t ) to the KL divergences between them.
Revised: we convert the computation of the log-likelihoods on q ( z c t | y t ) and p ( z ct | f c , z c S , x S , x t ) to the KL divergences between them.

Original: Then a Monte Carlo estimator can be used to approximate
Revised: Then a Monte Carlo estimator can be used to approximate

Original: L t where ˜ z cS is sampled by ˜
Revised: L t where ˜ z cS is sampled by ˜

Original: z cS ∼ q ( z cS | y S ) to compute the outer expectation E q ( z cS | y S )
Revised: z cS ∼ q ( z cS | y S ) to compute the outer expectation E q ( z cS | y S )

Original: [ ∗ ] ; and for inner expectation E q ( f c | y , x ) [ ∗ ] , because q ( f c | y , x ) =
Revised: [ ∗ ] ; and for inner expectation E q ( f c | y , x ) [ ∗ ] , because q ( f c | y , x ) =

Original: (cid:82) q ( f c |
Revised: (cid:82) q ( f c |

Original: z c , x ) q ( z c | x ) d z c , we are able to ﬁrst sample ˜
Revised: z c , x ) q ( z c | x ) d z c , we are able to ﬁrst sample ˜

Original: z cT ∼ q ( z cT | y T ) and obtain ˜ f
Revised: z cT ∼ q ( z cT | y T ) and obtain ˜ f

Original: c ∼ q ( f c | ˜ z cS , ˜ z
Revised: c ∼ q ( f c | ˜ z cS , ˜ z

Original: cT , x ) .
Revised: cT , x ) .

Original: By means of ˜ f c and ˜ z
Revised: By means of ˜ f c and ˜ z

Original: c sampled from the posterior, we have L t ≈ KL ( q ( z ct | y t ) (cid:107) p ( z ct | ˜ f c , ˜ z
Revised: c sampled from the posterior, we have L t ≈ KL ( q ( z ct | y t ) (cid:107) p ( z ct | ˜ f c , ˜ z

Original: c S , x S , x t )) .
Revised: c S , x S , x t )) .

Original: B.3. F UNCTION R EGULARIZER
Revised: B.3. F UNCTION R EGULARIZER

Original: [Equation]
Revised: [Equation]

Original: To estimate L f in the same way as the target regularizer,
Revised: To estimate L f in the same way as the target regularizer,

Original: ˜
Revised: ˜

Original: z c is sampled by ˜ z
Revised: z c is sampled by ˜ z

Original: c ∼ q ( z c | y ) and we have L f ≈ KL ( q ( f c | ˜
Revised: c ∼ q ( z c | y ) and we have L f ≈ KL ( q ( f c | ˜

Original: z c , x ) (cid:107) p ( f c | ˜
Revised: z c , x ) (cid:107) p ( f c | ˜

Original: z cS , x S )) .
Revised: z cS , x S )) .

Original: B.3. C ONTEXT R EGULARIZER
Revised: B.3. C ONTEXT R EGULARIZER

Original: [Equation]
Revised: [Equation]

Original: B. T OTAL C ORRELATION
Revised: B. T OTAL C ORRELATION

Original: Let I = { y 1 , ..., y K } be the set with images from all samples in the dataset, and
Revised: Let I = { y 1 , ..., y K } be the set with images from all samples in the dataset, and

Original: ˜ z
Revised: ˜ z

Original: k ∼ q ( z k | y k ) are the sampled latent variables for each image.
Revised: k ∼ q ( z k | y k ) are the sampled latent variables for each image.

Original: To apply the Total Correlation (TC) of βTCVAE
Revised: To apply the Total Correlation (TC) of βTCVAE

Original: (Chen et al., 2018) to CLAP, we make a tiny modiﬁcation that the aggregated posterior ¯ q ( z ) =
Revised: (Chen et al., 2018) to CLAP, we make a tiny modiﬁcation that the aggregated posterior ¯ q ( z ) =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =1 q ( z k | y k ) p ( y k ) is decomposed along with concepts instead of the dimension of latent variables.
Revised: Kk =1 q ( z k | y k ) p ( y k ) is decomposed along with concepts instead of the dimension of latent variables.

Original: According to β -TCVAE, there is
Revised: According to β -TCVAE, there is

Original: [Equation]
Revised: [Equation]

Original: where β -TCVAE provides Minibatch Weighted Sampling (MWS) to approximate Equation 20 by a minibatch { y m | m ∈ M } ⊆
Revised: where β -TCVAE provides Minibatch Weighted Sampling (MWS) to approximate Equation 20 by a minibatch { y m | m ∈ M } ⊆

Original: I of the dataset:
Revised: I of the dataset:

Original: [Equation]
Revised: [Equation]

Original: C D ATASETS AND E XPERIMENTAL S ETUP C. D ETAILS OF D ATASETS
Revised: C D ATASETS AND E XPERIMENTAL S ETUP C. D ETAILS OF D ATASETS

Original: In this paper, three types of datasets are adopted: BoBa, CRPM, and MPI3D. Figure 8 displays two samples for each instance of the datasets, and Table 2 describes the train, validation, and test conﬁgurations of the instances.
Revised: In this paper, three types of datasets are adopted: BoBa, CRPM, and MPI3D. Figure 8 displays two samples for each instance of the datasets, and Table 2 describes the train, validation, and test conﬁgurations of the instances.

Original: Bouncing Ball
Revised: Bouncing Ball

Original: (BoBa)
Revised: (BoBa)

Original: BoBa contains many videos of bouncing balls.
Revised: BoBa contains many videos of bouncing balls.

Original: Depending on the number of balls in the scene, the dataset provides two instances BoBa-1 and BoBa-2.
Revised: Depending on the number of balls in the scene, the dataset provides two instances BoBa-1 and BoBa-2.

Original: In the videos of Figure 8a and 8b, the motion of balls follows the law of physical collisions; and the appearance of balls (color, size, amount, etc.) is constant over time.
Revised: In the videos of Figure 8a and 8b, the motion of balls follows the law of physical collisions; and the appearance of balls (color, size, amount, etc.) is constant over time.

Original: The models need to capture the probability space on functions f : R (cid:55)→ R 3 × 64 × 64 that map timestamps to video frames.
Revised: The models need to capture the probability space on functions f : R (cid:55)→ R 3 × 64 × 64 that map timestamps to video frames.

Original: Referring to Table 2, we provide 10,000 videos of bouncing balls for training, 1,000 for validation and hyperparameter selection, and 2,000 to test the intuitive physics of the models.
Revised: Referring to Table 2, we provide 10,000 videos of bouncing balls for training, 1,000 for validation and hyperparameter selection, and 2,000 to test the intuitive physics of the models.

Original: In the training and validation phase, we randomly select 1 to 4 frames from video frames as the target images to predict; in the testing phase, we provide two conﬁgurations (referred to as Test-4 and Test-6) to evaluate the model’s performance when there are 4 and 6 target images in videos.
Revised: In the training and validation phase, we randomly select 1 to 4 frames from video frames as the target images to predict; in the testing phase, we provide two conﬁgurations (referred to as Test-4 and Test-6) to evaluate the model’s performance when there are 4 and 6 target images in videos.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Continuous Raven’s Progressive Matrix (CRPM)
Revised: Continuous Raven’s Progressive Matrix (CRPM)

Original: CRPM consists of 3 × 3 image matrices where the images contain one or two centered triangles or circles.
Revised: CRPM consists of 3 × 3 image matrices where the images contain one or two centered triangles or circles.

Original: CRPM provides four instances with different image types: CRPM-T, CRPM-DT, CRPM-C, and CRPM-DC.
Revised: CRPM provides four instances with different image types: CRPM-T, CRPM-DT, CRPM-C, and CRPM-DC.

Original: Images in a matrix follow attribute-speciﬁc changing rules (e.g., rules in the size, grayscale, and rotation of the triangle).
Revised: Images in a matrix follow attribute-speciﬁc changing rules (e.g., rules in the size, grayscale, and rotation of the triangle).

Original: The ﬁrst sample in Figure 8c shows that, for images within the row, the sizes of triangles increase progressively, the grayscales change progressively from dark to light, and the rotations keep constant.
Revised: The ﬁrst sample in Figure 8c shows that, for images within the row, the sizes of triangles increase progressively, the grayscales change progressively from dark to light, and the rotations keep constant.

Original: To predict the missing target images in the matrix, the models need to learn the probability space on functions f : {− 1 , 0 , 1 } 2 (cid:55)→ R 64 × 64 that map 2D grid coordinates to grid images.
Revised: To predict the missing target images in the matrix, the models need to learn the probability space on functions f : {− 1 , 0 , 1 } 2 (cid:55)→ R 64 × 64 that map 2D grid coordinates to grid images.

Original: Tableshows that we provide 10,000 image matrices for training, 1,000 for tuning the model hyperparameters, and 2,000 to test the abstract visual reasoning ability of models.
Revised: Tableshows that we provide 10,000 image matrices for training, 1,000 for tuning the model hyperparameters, and 2,000 to test the abstract visual reasoning ability of models.

Original: In the training and validation
Revised: In the training and validation

Original: [Table]
Revised: [Table]

Original: process, we randomly select 1 or 2 images as target images; in the testing process, we use Testand Test-3 to evaluate the performance of the models to predict 2 and 3 target images.
Revised: process, we randomly select 1 or 2 images as target images; in the testing process, we use Testand Test-3 to evaluate the performance of the models to predict 2 and 3 target images.

Original: MPI3D
Revised: MPI3D

Original: MPI3D (Gondal et al., 2019) dataset contains a series of single-object scenes, each with 40 different scene images.
Revised: MPI3D (Gondal et al., 2019) dataset contains a series of single-object scenes, each with 40 different scene images.

Original: There are two underlying rules to change the object in a scene: change the altitude of the object (sample 1 in Figure 8g); change the azimuth of the object (sample 2 in
Revised: There are two underlying rules to change the object in a scene: change the altitude of the object (sample 1 in Figure 8g); change the azimuth of the object (sample 2 in

Original: Figure 8g).
Revised: Figure 8g).

Original: The other attributes (e.g., object color, camera height, etc.) do not change within scene images.
Revised: The other attributes (e.g., object color, camera height, etc.) do not change within scene images.

Original: It is essential for the models to grasp different changing patterns for target image prediction, and the key is to describe the distribution over functions f : R (cid:55)→ R 3 × 64 × 64 that map object altitudes or azimuths to scene images.
Revised: It is essential for the models to grasp different changing patterns for target image prediction, and the key is to describe the distribution over functions f : R (cid:55)→ R 3 × 64 × 64 that map object altitudes or azimuths to scene images.

Original: As Table 2 shows, we provide 16127 scenes for training, 2305 for tuning the hyperparameters, and 4608 for testing.
Revised: As Table 2 shows, we provide 16127 scenes for training, 2305 for tuning the hyperparameters, and 4608 for testing.

Original: In terms of training and validation, we ﬁrst randomly select 8 images from 40 scene images to represent the scene, and then randomly select 1 to 4 images from the 8 images again as the target images, leaving the remaining images as the context.
Revised: In terms of training and validation, we ﬁrst randomly select 8 images from 40 scene images to represent the scene, and then randomly select 1 to 4 images from the 8 images again as the target images, leaving the remaining images as the context.

Original: For testing, we supply Test-4, Test-6, Test-10, and Test-20, where Test-4 and Testevaluate the performance of the models to predict 4 and 6 target images out of 8 images; Test-10 and
Revised: For testing, we supply Test-4, Test-6, Test-10, and Test-20, where Test-4 and Testevaluate the performance of the models to predict 4 and 6 target images out of 8 images; Test-10 and

Original: Test-20 evaluate the performance to predict 10 and 20 target images out of 40 images.
Revised: Test-20 evaluate the performance to predict 10 and 20 target images out of 40 images.

Original: One can fetch
Revised: One can fetch

Original: MPI3D from the repository 1 under the Creative Commons Attribution 4.0 International License.
Revised: MPI3D from the repository 1 under the Creative Commons Attribution 4.0 International License.

Original: C. M ODEL A RCHITECTURE AND H YPERPARAMETERS CLAP
Revised: C. M ODEL A RCHITECTURE AND H YPERPARAMETERS CLAP

Original: -NP
Revised: -NP

Original: In this subsection, we will ﬁrst describe the architecture of the encoder, decoder, concept-speciﬁc function parsers, and concept-speciﬁc target predictors network in CLAP-NP.
Revised: In this subsection, we will ﬁrst describe the architecture of the encoder, decoder, concept-speciﬁc function parsers, and concept-speciﬁc target predictors network in CLAP-NP.

Original: Then we will list hyperparameters in the training phase.
Revised: Then we will list hyperparameters in the training phase.

Original: • Encoder : for all datasets, CLAP-NP uses the same convolutional blocks to downsample highdimensional images into parameters of concepts, which can be described as – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 64 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 128 BatchNorm, ReLU – 4 × 4 Conv, 256 BatchNorm, ReLU – 1 × 1 Conv,– ReshapeBlock,– Fully Connected, | C |
Revised: • Encoder : for all datasets, CLAP-NP uses the same convolutional blocks to downsample highdimensional images into parameters of concepts, which can be described as – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 64 BatchNorm, ReLU – 4 × 4 Conv, stride 2, padding 1, 128 BatchNorm, ReLU – 4 × 4 Conv, 256 BatchNorm, ReLU – 1 × 1 Conv,– ReshapeBlock,– Fully Connected, | C |

Original: The ReshapeBlock ﬂattens the output tensor of size
Revised: The ReshapeBlock ﬂattens the output tensor of size

Original: × 1 × 1 from the last convolutional layer to the vector of size 512 .
Revised: × 1 × 1 from the last convolutional layer to the vector of size 512 .

Original: The output of the encoder is the mean of | C | visual concepts
Revised: The output of the encoder is the mean of | C | visual concepts

Original: • Decoder : in BoBa and CRPM datasets, CLAP-NP uses several deconvolutional layers to upsample concepts to images, which is – ReshapeBlock, | C | × 1 ×– 1 × 1 Deconv, 128 BatchNorm, ReLU – 4 × 4 Deconv, 64 BatchNorm2d, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU
Revised: • Decoder : in BoBa and CRPM datasets, CLAP-NP uses several deconvolutional layers to upsample concepts to images, which is – ReshapeBlock, | C | × 1 ×– 1 × 1 Deconv, 128 BatchNorm, ReLU – 4 × 4 Deconv, 64 BatchNorm2d, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU

Original: [Table]
Revised: [Table]

Original: – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, N Channel Sigmoid where N Channel is the number of image channels and Sigmoid is the activation function used to generate pixel values ranging in (0 , 1) .
Revised: – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU – 4 × 4 Deconv, stride 2, padding 1, N Channel Sigmoid where N Channel is the number of image channels and Sigmoid is the activation function used to generate pixel values ranging in (0 , 1) .

Original: • Function Parser : CLAP-NP provides identical but independent function parsers for each concept.
Revised: • Function Parser : CLAP-NP provides identical but independent function parsers for each concept.

Original: Each function parser consists of a multi-layer perception (MLP) T s to encode concepts, a MLP T i to encode function inputs, a MLP T a to aggregate context points, and a MLP T f to predict the mean and standard deviation of the global latent variable.
Revised: Each function parser consists of a multi-layer perception (MLP) T s to encode concepts, a MLP T i to encode function inputs, a MLP T a to aggregate context points, and a MLP T f to predict the mean and standard deviation of the global latent variable.

Original: The architecture of T s and T i is – Fully Connected, 32 ReLU – Fully Connected,
Revised: The architecture of T s and T i is – Fully Connected, 32 ReLU – Fully Connected,

Original: By concatenating the encoded concepts and the encoded function inputs, we get the context function points.
Revised: By concatenating the encoded concepts and the encoded function inputs, we get the context function points.

Original: The architecture of T a is
Revised: The architecture of T a is

Original: – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected,
Revised: – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected,

Original: Finally, the architecture of T f is
Revised: Finally, the architecture of T f is

Original: – Fully Connected, 256 ReLU – Fully Connected, 2 d g
Revised: – Fully Connected, 256 ReLU – Fully Connected, 2 d g

Original: where d g is the size of the global latent variable g c .
Revised: where d g is the size of the global latent variable g c .

Original: The outputs of T f consist of the mean and standard deviation of the latent variable g c .
Revised: The outputs of T f consist of the mean and standard deviation of the latent variable g c .

Original: • Target Predictor : the concept-speciﬁc target predictor T p is a MLP that maps g c and the
Revised: • Target Predictor : the concept-speciﬁc target predictor T p is a MLP that maps g c and the

Original: encoded
Revised: encoded

Original: function input
Revised: function input

Original: into
Revised: into

Original: the
Revised: the

Original: target
Revised: target

Original: concept,
Revised: concept,

Original: whose architecture is
Revised: whose architecture is

Original: – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected,
Revised: – Fully Connected, 256 ReLU – Fully Connected, 256 ReLU – Fully Connected,

Original: To generate more complex scene images in MPI3D, CLAP-NP uses a deeper decoder where the hidden size is [512, 256, 128, 64, 32].
Revised: To generate more complex scene images in MPI3D, CLAP-NP uses a deeper decoder where the hidden size is [512, 256, 128, 64, 32].

Original: And we use T f with the hidden size [128, 128] and output size 64, T f with the hidden size [64], and T p with the hidden size [128, 128].
Revised: And we use T f with the hidden size [128, 128] and output size 64, T f with the hidden size [64], and T p with the hidden size [128, 128].

Original: Hyperparameters for
Revised: Hyperparameters for

Original: CLAP-NP on different datasets are shown in Table 4.
Revised: CLAP-NP on different datasets are shown in Table 4.

Original: And in all datasets, we set learning rate as 3 × 10 − 4 , batch size as 512, and σ x = 0 . 1 .
Revised: And in all datasets, we set learning rate as 3 × 10 − 4 , batch size as 512, and σ x = 0 . 1 .

Original: For MPI3D, we anneal β t and β f in the ﬁrst 400 epochs.
Revised: For MPI3D, we anneal β t and β f in the ﬁrst 400 epochs.

Original: After each training epoch, we use the validation set to compute the evidence lower bound (ELBO) of the current model and save the model with the largest ELBO as the trained model.
Revised: After each training epoch, we use the validation set to compute the evidence lower bound (ELBO) of the current model and save the model with the largest ELBO as the trained model.

Original: For all datasets,
Revised: For all datasets,

Original: CLAP-NP uses the Adam (Kingma & Ba, 2015) optimizer to update parameters.
Revised: CLAP-NP uses the Adam (Kingma & Ba, 2015) optimizer to update parameters.

Original: Neural Process (NP)
Revised: Neural Process (NP)

Original: Different from (Garnelo et al., 2018b), to apply NP in high-dimensional images, we add an encoder and decoder which are the same as those in CLAP-NP to convert highdimensional images into low-dimensional representations.
Revised: Different from (Garnelo et al., 2018b), to apply NP in high-dimensional images, we add an encoder and decoder which are the same as those in CLAP-NP to convert highdimensional images into low-dimensional representations.

Original: And the representations are combined with the function inputs to form the context by an aggregator T a .
Revised: And the representations are combined with the function inputs to form the context by an aggregator T a .

Original: Then the function parser T f obtains the mean and standard deviation of the global latent variable.
Revised: Then the function parser T f obtains the mean and standard deviation of the global latent variable.

Original: Finally, the decoder generates
Revised: Finally, the decoder generates

Original: [Table]
Revised: [Table]

Original: target images from the target function inputs and the global latent variable.
Revised: target images from the target function inputs and the global latent variable.

Original: The architectures of T a and T f are
Revised: The architectures of T a and T f are

Original: • Aggregator – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU – Fully Connected,• Function Parser – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU – Fully Connected, 2 d g
Revised: • Aggregator – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU – Fully Connected,• Function Parser – Fully Connected, 512 ReLU – Fully Connected, 512 ReLU – Fully Connected, 2 d g

Original: Hyperparameters for NP on different datasets are shown in Table 5 where d r is the representation size of the encoder.
Revised: Hyperparameters for NP on different datasets are shown in Table 5 where d r is the representation size of the encoder.

Original: And for all datasets, NP adopts the learning rate 1 × 10 − 4 batch size 512, and σ x = 0 . 1 .
Revised: And for all datasets, NP adopts the learning rate 1 × 10 − 4 batch size 512, and σ x = 0 . 1 .

Original: For all datasets, NP uses the Adam optimizer for parameter updates.
Revised: For all datasets, NP uses the Adam optimizer for parameter updates.

Original: Generative Query Network (GQN)
Revised: Generative Query Network (GQN)

Original: To implement GQN (Eslami et al., 2018), we use a PyTorch implementation from the repository 2 with the following changes to the default conﬁguration: set learning rate to 0.0005, batch size to 256, the representation type to pool, the number of iterations to 4, and share the ConvLSTM core among iterations.
Revised: To implement GQN (Eslami et al., 2018), we use a PyTorch implementation from the repository 2 with the following changes to the default conﬁguration: set learning rate to 0.0005, batch size to 256, the representation type to pool, the number of iterations to 4, and share the ConvLSTM core among iterations.

Original: Gaussian Process (GP)
Revised: Gaussian Process (GP)

Original: We use a pretrained autoencoder to reduce the dimensionality of images, by which GPneeds to capture the changing patterns in the latent space and thus can handle image-valued functions.
Revised: We use a pretrained autoencoder to reduce the dimension of images and let GP capture the changing patterns in the low-dimensional space.

Original: We adopt the same encoder and decoder as in CLAP-NP and NP for a fair comparison.
Revised: We adopt the same encoder and decoder as in CLAP-NP and NP for a fair comparison.

Original: The mean function of GP is set to a constant function.
Revised: The mean function of GP is set to a constant function.

Original: By comparing the RBF kernel, periodic kernel, and deep kernel, weﬁnally chose the deep kernel as the kernel function that k ( x i , x j ) =
Revised: After comparing the RBF kernel, periodic kernel, and deep kernel, we chose the deep kernel as the kernel function that k ( x i , x j ) =

Original: σ 2 exp( −(cid:107)T k ( x i ) −T k ( x j ) (cid:107) / 2 (cid:96) 2 ) (Wilson et al., 2016).
Revised: σ 2 exp( −(cid:107)T k ( x i ) − T k ( x j ) (cid:107) / 2 (cid:96) 2 ) (Wilson et al., 2016).

Original: The architecture of the neural network T kin the deep kernel is a multilayer perceptron with hidden sizes [1000, 1000, 500, 50], output size 2, and ReLU activation functions, which is the same as in DKL (Wilson et al., 2016).
Revised: Neural network T k is a multilayer perceptron with hidden sizes [1000, 1000, 500, 50], output size 2, and ReLU activation functions, which is the same as in DKL (Wilson et al., 2016).

Original: The hyperparameters of GP are tuned on each sample through the log-likelihood of context points and the RMSprop optimizer.
Revised: The hyperparameters of GP are tuned on each sample through the log-likelihood of context points and the RMSprop optimizer.

Original: We give the learning rate and epoch to adjust hyperparameters in Table 6. To model the correlations between dimensions, we use multi-task Gaussian Process prediction (Bonilla et al., 2007).
Revised: We give the learning rate and epoch to adjust hyperparameters in Table 6 and use multi-task Gaussian Process prediction (Bonilla et al., 2007) to model the correlations between dimensions.

Original: C. C OMPUTATIONAL R ESOURCE
Revised: C. C OMPUTATIONAL R ESOURCE

Original: We train our model and baselines on the server with Intel(R)
Revised: We train our model and baselines on the server with Intel(R)

Original: Xeon(R)
Revised: Xeon(R)

Original: Gold 6133 CPUs, 24GB
Revised: Gold 6133 CPUs, 24GB

Original: NVIDIA GeForce RTX 3090 GPUs, 512GB RAM, and Ubuntu 18.04 OS.
Revised: NVIDIA GeForce RTX 3090 GPUs, 512GB RAM, and Ubuntu 18.04 OS.

Original: All models are implemented with the PyTorch (Paszke et al., 2019) framework.
Revised: All models are implemented with the PyTorch (Paszke et al., 2019) framework.

Original: To avoid the randomness in dataset splitting, we ﬁx the random seed to 0.
Revised: To avoid the randomness in dataset splitting, we ﬁx the random seed to 0.

Original: [Figure]
Revised: [Figure]

Revised: NP

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: D A DDITIONAL E XPERIMENTAL R ESULTS D. I NTUITIVE P HYSICS
Revised: D A DDITIONAL E XPERIMENTAL R ESULTS D. I NTUITIVE P HYSICS

Original: In Figure 9 we display additional prediction results on BoBa-1 and BoBa-2.
Revised: In Figure 9 we display additional prediction results on BoBa-1 and BoBa-2.

Original: For each instance, there are samples that correspond to Test-4 (left) and Test-6 (right)
Revised: For each instance, there are samples that correspond to Test-4 (left) and Test-6 (right)

Original: [Table]
Revised: [Table]

Original: where CLAP-NP outperforms other baselines.
Revised: where CLAP-NP outperforms other baselines.

Original: NP generates blurred target images on both BoBa-1 and BoBa-2 datasets.
Revised: NP generates blurred target images on both BoBa-1 and BoBa-2 datasets.

Original: It indicates that
Revised: It indicates that

Original: NP has difﬁculty modeling changing patterns on bouncing balls.
Revised: NP has difﬁculty modeling changing patterns on bouncing balls.

Original: GQN can produce clear images but may generate non-existent balls and lose existing balls.
Revised: GQN can produce clear images but may generate non-existent balls and lose existing balls.

Original: In the ﬁrst sample of BoBa-1, the predictions of GQN deviate from the ground truths signiﬁcantly in the position of balls.
Revised: In the ﬁrst sample of BoBa-1, the predictions of GQN deviate from the ground truths signiﬁcantly in the position of balls.

Original: Overall.
Revised: Overall.

Original: CLAP-NP performs well in modeling scene consistency and predicting motion trajectory.
Revised: CLAP-NP performs well in modeling scene consistency and predicting motion trajectory.

Original: D. A BSTRACT V ISUAL R EASONING
Revised: D. A BSTRACT V ISUAL R EASONING

Original: With the additional experimental results in Figure
Revised: With the additional experimental results in Figure

Original: [Table]
Revised: [Table]

Original: we can intuitively see that CLAP-NP has a better understanding of the changing rules than baselines.
Revised: we can intuitively see that CLAP-NP has a better understanding of the changing rules than baselines.

Original: For the second sample of each dataset
Revised: For the second sample of each dataset

Original: [Table]
Revised: [Table]

Original: a row of images is removed to interpret the abstract visual reasoning ability of the model.
Revised: a row of images is removed to interpret the abstract visual reasoning ability of the model.

Original: As samples in Figure 10 shows, when we remove a row of images, the answer is not unique: the predictions are correct as long as their sizes increase progressively and their colors and rotations keep constant.
Revised: As samples in Figure 10 shows, when we remove a row of images, the answer is not unique: the predictions are correct as long as their sizes increase progressively and their colors and rotations keep constant.

Original: CLAP-NP represents such randomness in predictions by means of the probabilistic generative process, making it possible to generate different correct answers.
Revised: CLAP-NP represents such randomness in predictions by means of the probabilistic generative process, making it possible to generate different correct answers.

Original: In terms of the prediction quality, although generating blurred images, NP has the basic reasoning ability about the progressive rule of the outer triangle size in Figure 10b.
Revised: In terms of the prediction quality, although generating blurred images, NP has the basic reasoning ability about the progressive rule of the outer triangle size in Figure 10b.

Original: GQN generates clear images, however, the generated images probably deviate from the underlying changing rules on matrices.
Revised: GQN generates clear images, however, the generated images probably deviate from the underlying changing rules on matrices.

Original: Table 7 and Table 11 illustrate the outperforming results of CLAP-NP in abstract visual reasoning by quantitative experiments.
Revised: Table 7 and Table 11 illustrate the outperforming results of CLAP-NP in abstract visual reasoning by quantitative experiments.

Original: D. S CENE R EPRESENTATION
Revised: D. S CENE R EPRESENTATION

Original: Figure 12 shows the prediction results within 8 scene images.
Revised: Figure 12 shows the prediction results within 8 scene images.

Original: Both GQN and CLAP-NP generate clear prediction results when the number of target images is 4; if the number of target images is increased to 6, the prediction results of GQN have signiﬁcant deviations, while CLAP-NP maintains the accuracy of predictions.
Revised: Both GQN and CLAP-NP generate clear prediction results when the number of target images is 4; if the number of target images is increased to 6, the prediction results of GQN have signiﬁcant deviations, while CLAP-NP maintains the accuracy of predictions.

Original: NP generates ambiguous results in all experiments.
Revised: NP generates ambiguous results in all experiments.

Original: Figure 13 shows a more complicated situation: we provide 40 scene images and set 20 of them as target images.
Revised: Figure 13 shows a more complicated situation: we provide 40 scene images and set 20 of them as target images.

Original: In this case, GQN and NP can hardly generate clear foreground objects; while CLAP-NP produces relatively accurate predictions with only a little decrease in generative quality.
Revised: In this case, GQN and NP can hardly generate clear foreground objects; while CLAP-NP produces relatively accurate predictions with only a little decrease in generative quality.

Original: This experiment aims to test the generalization of the laws learned by the models, and the results above illustrate that NP can hardly represent scenes with functions, GQN has difﬁculty generalizing the scene representation ability to different conﬁgurations, and the laws learned by CLAP-NP can be generalized to novel scenes with more available images.
Revised: This experiment aims to test the generalization of the laws learned by the models, and the results above illustrate that NP can hardly represent scenes with functions, GQN has difﬁculty generalizing the scene representation ability to different conﬁgurations, and the laws learned by CLAP-NP can be generalized to novel scenes with more available images.

Original: D. C ONCEPT D ECOMPOSITION
Revised: D. C ONCEPT D ECOMPOSITION

Original: Concept decomposition is the foundation for CLAP to understand concept-speciﬁc laws.
Revised: Concept decomposition is the foundation for CLAP to understand concept-speciﬁc laws.

Original: In this experiment, we traverse each concept in the latent space and visualize the concepts through a decoder to illustrate the LRF learned for each concept.
Revised: In this experiment, we traverse each concept in the latent space and visualize the concepts through a decoder to illustrate the LRF learned for each concept.

Original: First of all, we decompose a batch of images into
Revised: First of all, we decompose a batch of images into

Original: [Figure]
Revised: [Figure]

Original: CLAP
Revised: CLAP

Original: [Figure]
Revised: [Figure]

Original: NP
Revised: NP

Original: [Figure]
Revised: [Figure]

Original: P r e d i c t 2 i m a g e s
Revised: P r e d i c t 2 i m a g e s

Original: P r e d i c t a r
Revised: P r e d i c t a r

Original: [Figure]
Revised: [Figure]

Original: a)
Revised: a)

Original: [Figure]
Revised: [Figure]

Original: P r e d i c t 2 i m a g e s
Revised: P r e d i c t 2 i m a g e s

Original: P r e d i c t a r o w
Revised: P r e d i c t a r o w

Original: [Figure]
Revised: [Figure]

Original: i c t 2 i m
Revised: i c t 2 i m

Original: [Figure]
Revised: [Figure]

Original: g
Revised: g

Original: [Figure]
Revised: [Figure]

Original: NP
Revised: NP

Original: GT
Revised: GT

Original: GQN
Revised: GQN

Original: CLAP-NP
Revised: CLAP-NP

Original: NP
Revised: NP

Original: GT
Revised: GT

Original: GQN
Revised: GQN

Original: CLAP-NP
Revised: CLAP-NP

Original: GP
Revised: GP

Original: GP concepts by the encoder to estimate the range of concepts in the latent space.
Revised: GP concepts by the encoder to estimate the range of concepts in the latent space.

Original: To traverse one concept, we ﬁx the other concepts and linearly interpolate it from the minimum value to the maximum value to generate a sequence of interpolation results, which are decoded into images for visualization.
Revised: To traverse one concept, we ﬁx the other concepts and linearly interpolate it from the minimum value to the maximum value to generate a sequence of interpolation results, which are decoded into images for visualization.

Original: Each row of Figure 14 represents the latent traversal results of one concept.
Revised: Each row of Figure 14 represents the latent traversal results of one concept.

Original: In BoBa-1,
Revised: In BoBa-1,

Original: CLAP-NP learns LRFs on concepts of color, horizontal position, and vertical position in an unsupervised manner.
Revised: CLAP-NP learns LRFs on concepts of color, horizontal position, and vertical position in an unsupervised manner.

Original: This is similar to the way we understand the motion of balls: the color keeps constant over time; the horizontal and vertical positions conform to the physical laws in their respective directions.
Revised: This is similar to the way we understand the motion of balls: the color keeps constant over time; the horizontal and vertical positions conform to the physical laws in their respective directions.

Original: For CRPM, CLAP-NP correctly parses images into concepts that correspond to the attribute-speciﬁc rules in matrices.
Revised: For CRPM, CLAP-NP correctly parses images into concepts that correspond to the attribute-speciﬁc rules in matrices.

Original: Concept decomposition in real environments is a challenge for models.
Revised: Concept decomposition in real environments is a challenge for models.

Original: For MPI3D, CLAP-NP parses the LRFs on the object axis (Dimension 1, 5, 6, 7) and other static scene attributes (Dimension 2, 3, 4).
Revised: For MPI3D, CLAP-NP parses the LRFs on the object axis (Dimension 1, 5, 6, 7) and other static scene attributes (Dimension 2, 3, 4).

Original: D. L ATENT R ANDOM F UNCTION C OMPOSITION
Revised: D. L ATENT R ANDOM F UNCTION C OMPOSITION

Original: By means of concept decomposition, CLAP-NP can parse the latent random function of each concept to represent concept-speciﬁc laws, by which the law of a sample can be modiﬁed in terms of concepts.
Revised: By means of concept decomposition, CLAP-NP can parse the latent random function of each concept to represent concept-speciﬁc laws, by which the law of a sample can be modiﬁed in terms of concepts.

Original: Figure 15 and 16 display the way to edit the laws.
Revised: Figure 15 and 16 display the way to edit the laws.

Original: We compose the concept-speciﬁc
Revised: We compose the concept-speciﬁc

Original: LRFs of existing samples to generate samples that have novel changing patterns.
Revised: LRFs of existing samples to generate samples that have novel changing patterns.

Original: For example, in the third sample of Figure 15a, we generate a sample with the progressive law of size, constant law of rotation, and constant law of color from three existing samples.
Revised: For example, in the third sample of Figure 15a, we generate a sample with the progressive law of size, constant law of rotation, and constant law of color from three existing samples.

Original: The results on four instances of CRPM indicate that CLAP-NP understands the changing patterns on matrices in an interpretable and manipulable way, which embodies the compositionality of laws.
Revised: The results on four instances of CRPM indicate that CLAP-NP understands the changing patterns on matrices in an interpretable and manipulable way, which embodies the compositionality of laws.

Original: D. L ATENT R ANDOM F UNCTION E XCHANGE
Revised: D. L ATENT R ANDOM F UNCTION E XCHANGE

Original: Another way to edit laws is to exchange LRFs of some concepts between samples.
Revised: Another way to edit laws is to exchange LRFs of some concepts between samples.

Original: In BoBa, we swap the law of collisions and appearance between samples, and the results are shown in Figure 17.
Revised: In BoBa, we swap the law of collisions and appearance between samples, and the results are shown in Figure 17.

Original: According to the concept decomposition results in Figure 14, we exchange the LRFs on Dimension 1 (Dimensions 2 and 3) to modify the law of collisions (the law of appearance) in BoBa-1.
Revised: According to the concept decomposition results in Figure 14, we exchange the LRFs on Dimension 1 (Dimensions 2 and 3) to modify the law of collisions (the law of appearance) in BoBa-1.

Original: And we exchange the LRFs on Dimensions 1 and 6 (Dimensions 2, 3, 4, and 5) to modify the law of collisions (the law of appearance) in BoBa-2.
Revised: And we exchange the LRFs on Dimensions 1 and 6 (Dimensions 2, 3, 4, and 5) to modify the law of collisions (the law of appearance) in BoBa-2.

Original: To exchange laws in MPI3D, as shown in Figure 18,we swap LRFs on Dimensions 1, 5, 6, and 7 to change the law of the object axis. By modifying laws on example 1 of Figure 18, the changing pattern of the ﬁrst sample becomes the horizontal rotation, and the second sample inherits the law of vertical rotation from the ﬁrst sample, which
Revised: To exchange laws in MPI3D, as shown in Figure 18,

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: + + = { Size 1, Rotation 1,
Revised: + + = { Size 1, Rotation 1,

Original: Color 1 } { Size 2, Rotation 2,
Revised: Color 1 } { Size 2, Rotation 2,

Original: Color 2 } { Size 3, Rotation 3,
Revised: Color 2 } { Size 3, Rotation 3,

Original: Color 3 } { Size 1, Rotation 2,
Revised: Color 3 } { Size 1, Rotation 2,

Original: Color 3 } + + = + + =
Revised: Color 3 } + + = + + =

Original: { Size 1, Rotation 1,
Revised: { Size 1, Rotation 1,

Original: Color 1 } { Size 2, Rotation 2,
Revised: Color 1 } { Size 2, Rotation 2,

Original: Color 2 } { Size 3, Rotation 3, Color 3 } { Size 1, Rotation 2,
Revised: Color 2 } { Size 3, Rotation 3, Color 3 } { Size 1, Rotation 2,

Original: Color 3 } { Size 1, Rotation 1,
Revised: Color 3 } { Size 1, Rotation 1,

Original: Color 1 } { Size 2, Rotation 2,
Revised: Color 1 } { Size 2, Rotation 2,

Original: Color 2 } { Size 3, Rotation 3,
Revised: Color 2 } { Size 3, Rotation 3,

Original: Color 3 } { Size 1, Rotation 2,
Revised: Color 3 } { Size 1, Rotation 2,

Original: Color 3 } (a) CRPM-T + + =
Revised: Color 3 } (a) CRPM-T + + =

Original: { Inner Color 1, Inner Rotation 1,
Revised: { Inner Color 1, Inner Rotation 1,

Original: Outer Size 1 } { Inner Color 2, Inner Rotation 2,
Revised: Outer Size 1 } { Inner Color 2, Inner Rotation 2,

Original: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,
Revised: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,

Original: Outer Size 3 } + + = + + = { Inner Color 1, Inner Rotation 1,
Revised: Outer Size 3 } + + = + + = { Inner Color 1, Inner Rotation 1,

Original: Outer Size 1 } { Inner Color 2, Inner Rotation 2,
Revised: Outer Size 1 } { Inner Color 2, Inner Rotation 2,

Original: Outer Size 2 } { Inner Color 3, Inner Rotation 3,
Revised: Outer Size 2 } { Inner Color 3, Inner Rotation 3,

Original: Outer Size 3 } { Inner Color 1, Inner Rotation 2, Outer Size 3 } { Inner Color 1, Inner Rotation 1,
Revised: Outer Size 3 } { Inner Color 1, Inner Rotation 2, Outer Size 3 } { Inner Color 1, Inner Rotation 1,

Original: Outer Size 1 } { Inner Color 2, Inner Rotation 2,
Revised: Outer Size 1 } { Inner Color 2, Inner Rotation 2,

Original: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,
Revised: Outer Size 2 } { Inner Color 3, Inner Rotation 3, Outer Size 3 } { Inner Color 1, Inner Rotation 2,

Original: Outer Size 3 }
Revised: Outer Size 3 }

Original: [Figure]
Revised: [Figure]

Original: + =
Revised: + =

Original: [Figure]
Revised: [Figure]

Original: + =
Revised: + =

Original: [Figure]
Revised: [Figure]

Original: + =
Revised: + =

Original: [Figure]
Revised: [Figure]

Original: + + =
Revised: + + =

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Law
Revised: Law

Original: [Figure]
Revised: [Figure]

Original: Law 2 }
Revised: Law 2 }

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  illustrates CLAP-NP’s ability to generate new samples.
Revised: we swap LRFs on Dimensions 1, 5, 6, and 7 to change the law of the object axis. By modifying laws on example 1 of Figure 18, the changing pattern of the ﬁrst sample becomes the horizontal rotation, and the second sample inherits the law of vertical rotation from the ﬁrst sample, which illustrates CLAP-NP’s ability to generate new samples.

Original: The experiments on law exchange evidence
Revised: The experiments on law exchange evidence

Original: CLAP-NP’s interpretability and manipulability from another perspective.
Revised: CLAP-NP’s interpretability and manipulability from another perspective.

Original: D. G ENERALIZATION ON U NSEEN C ONCEPTS
Revised: D. G ENERALIZATION ON U NSEEN C ONCEPTS

Original: We extend BoBa-2 to generate four datasets with novel laws: Novel-colors (BoBa-2-NC), Novelshape (BoBa-2-NS), Without-ball-collisions (BoBa-2-WBO), and With-gravity (BoBa-2-WG).
Revised: We extend BoBa-2 to generate four datasets with novel laws: Novel-colors (BoBa-2-NC), Novelshape (BoBa-2-NS), Without-ball-collisions (BoBa-2-WBO), and With-gravity (BoBa-2-WG).

Original: We draw balls on BoBa-2-NC with unseen colors and  BoBa-2-NS with squares without changing the laws of positions.
Revised: We draw balls on BoBa-2-NC with unseen colors and replace the balls on BoBa-2-NS with squares without changing the laws of positions.

Original: For BoBa-2-WBO, we disable collisions between balls and reserve collisions between balls and borders; for BoBa-2-WG, we add vertical gravity to scenes.
Revised: For BoBa-2-WBO, we disable collisions between balls and reserve collisions between balls and borders; for BoBa-2-WG, we add vertical gravity to scenes.

Original: After training on BoBa-2, CLAP-NP isdirectly tested by four datasets  to evaluate whether the compositionality of laws helps generalization  to  unseen concepts.
Revised: After training on BoBa-2, CLAP-NP is tested on four datasets without retraining to evaluate whether the compositionality of laws improves the model’s generalization ability to scenes with unseen concepts or laws.

Original: The experimental results in Figure 19 show that CLAP-NP canstill correctly predict the  object positions on datasets with novel colors and shapes.
Revised: Figure 19 shows that CLAP-NP can predict the correct object positions on BoBa-2-NC and BoBa-2-NS.

Original: And the color and shape of objectsdeviate from those of real ones because the encoder and decoder are trained on BoBa-2, whose latent space does not include unseen colors or shapes.
Revised: CLAP-NP predicts inaccurate color and shape of objects because the encoder and decoder are trained on BoBa-2, and the latent space does not encode the unseen colors or shapes.

Original: The same situation can be observed on BoBa-2-WBO and BoBa-2-WG. In the scene with unseen physical laws, CLAP-NP learns the correct law of appearance constancy but predicts incorrect positions of balls.
Revised: We observe similar results on BoBa-2-WBO and BoBa-2-WG that CLAP-NP learns the correct law of appearance constancy but predicts incorrect positions of balls when there are unseen physical laws in scenes.

Original: To better illustrate the generalization ability in scenes with novel laws, we quantitatively evaluate CLAP-NP and baseline models by MSE ad SA scores on the four datasets.
Revised: To better evaluate the generalization ability in scenes with novel laws, we quantitatively evaluate CLAP-NP and baseline models with MSE ad SA scores on the four datasets.

Original:  Table 8 shows MSE scores and SA scores with different numbers of candidates. CLAP-NP achieves the best MSE and SA scores in most situations, illustrating CLAP-NP’s generalization ability in scenes with novel laws.
Revised: The results in Table 8 indicate that CLAP-NP achieves the best MSE and SA scores in most situations, which illustrates CLAP-NP’s generalization ability in scenes with novel laws.

Original: D. N UMBER OF L ATENT R ANDOM F UNCTIONS
Revised: D. N UMBER OF L ATENT R ANDOM F UNCTIONS

Original: This experiment explores the inﬂuence of setting too large or too small numbers of latent random functions (LRFs) on CLAP-NP.
Revised: This experiment explores the inﬂuence of setting too large or too small numbers of latent random functions (LRFs) in CLAP-NP.

Original: Figure 20 shows the latent traversal results on MPI3D, BoBa-2, and
Revised: Figure 20 shows the latent traversal results on MPI3D, BoBa-2, and

Original: CRPM-DT.
Revised: CRPM-DT.

Original: If we set too few LRFs, CLAP-NP learns mixed laws among dimensions instead of the expected compositional laws.
Revised: If we set too few LRFs, CLAP-NP learns mixed laws among dimensions instead of the expected compositional laws.

Original: Moreover, too few LRFs can inﬂuence CLAP-NP’s generationability (e.g., set only two LRFs for BoBa-2). Setting too many LRFs has no signiﬁcant inﬂuence on CLAPNP’s performance because there will be redundant dimensions in CLAP-NP that do not encode any
Revised: Moreover, too few LRFs can inﬂuence CLAP-NP’s generation

Original: [Figure] [Table] 
Revised: [Figure] [Table] [Figure]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: information (e
Revised: ability (e

Original: [Table]
Revised: [Table]

Original: g.,  dimensions 1,
Revised: g., set only two LRFs for BoBa-2). Setting too many LRFs has no signiﬁcant inﬂuence on CLAP-NP’s performance because there will be redundant dimensions in CLAP-NP that do not encode information (e

Original: [Table]
Revised: [Table]

Original:  and 6 on CRPM-DT).
Revised: g., dimensions 1, 3, and 6 on CRPM-DT).

Original: However, due to the independent modeling of concept-speciﬁc LRFs, setting a large number of LRFs will reduce the computational speed of CLAP-NP and increase the number of model parameters.
Revised: However, due to the independent modeling of concept-speciﬁc LRFs, setting a large number of LRFs will reduce the computational speed of CLAP-NP and increase the number of model parameters.

Original: D. P REDICTION S TRATEGY
Revised: D. P REDICTION S TRATEGY

Original: CLAP uses the one-shot strategy to predict all target images at one time.
Revised: CLAP uses the one-shot strategy that predicts all target images at one time.

Original: However, the rollout strategy can be another choice when predicting the following target images through a few context images.
Revised: However, the rollout strategy can be another choice to predict the following target images through the few context images at the beginning.

Original: For example, in the test conﬁguration η = 20 / 25 , we ﬁrst predict the 6th to 10th images from thegiven ﬁrst ﬁve  images, then combine them (1st to 10th images) to predict the following ﬁve images [Table] and repeat this process untilpredicting all target images.
Revised: For example, in the test conﬁguration η = 20 / 25 , we ﬁrst predict the 6th to 10th images from the ﬁrst ﬁve context images, then combine them (1st to 10th images) to predict the following ﬁve images, and repeat this process until all target images are predicted.

Original: Table 9 shows the MSE scores on CRPM-DT, BoBa-2, and MPI3D where CLAP-NP predicts targets with the rollout and one-shot strategy, respectively.
Revised: Table 9 shows the MSE scores on CRPM-DT, BoBa-2, and MPI3D where CLAP-NP predicts targets with the rollout and one-shot strategy, respectively.

Original: On BoBa-2 and MPI3D, the rollout strategy slightly improves the prediction accuracy.
Revised: On BoBa-2 and MPI3D, the rollout strategy slightly improves the prediction accuracy.

Original: If a target image is far away from all the context images, the prediction is likely to have high uncertainty.
Revised: If a target image is far away from all the context images, the prediction may have high uncertainty.

Original: In this experiment, we give the ﬁrst few images as the contexts and let the model predict the following target images.

Original: The rollout strategy only predicts a few target images at close positions each time, while the one-shot strategy has to predict all the target images at one time.
Revised: The rollout strategy only predicts a few target images at close positions each time, while the one-shot strategy has to predict all the target images at one time.

Original: Therefore, the rollout strategy is more likely to reduce uncertainty and improve prediction accuracy.
Revised: Therefore, the rollout strategy is more likely to reduce uncertainty and improve prediction accuracy.

Original: D. F AILURE C ASES
Revised: D. F AILURE C ASES

Original: In this experiment, we display some failure cases.
Revised: In this experiment, we display some failure cases.

Original: For BoBa, most failure cases occur when there are continuous target images (1st sample of BoBa in Figure 21) or too many target images (2nd sample of BoBa in Figure 21).
Revised: For BoBa, most failure cases occur when there are continuous target images (1st sample of BoBa in Figure 21) or too many target images (2nd sample of BoBa in Figure 21).

Original: For CRPM, CLAP can predict target images with diversity when we remove the third row from a matrix, but it sometimes generates target images that break the rules.
Revised: For CRPM, CLAP can predict target images with diversity when we remove the third row from a matrix, but it sometimes generates target images that break the rules.

Original: For example, the color of images keeps invariant in the ﬁrst two rows, but CLAP generates images in the third row with changing grayscales (samples of CRPM in Figure 21).
Revised: For example, the color of images keeps invariant in the ﬁrst two rows, but CLAP generates images in the third row with changing grayscales (samples of CRPM in Figure 21).

Original: For MPI3D, when the object size has an obvious change (1st sample of MPI3D in Figure 21) or the centric object is tiny (2nd sample of MPI3D in Figure 21), the predictions will deviate or become unclear.
Revised: For MPI3D, when the object size has an obvious change (1st sample of MPI3D in Figure 21) or the centric object is tiny (2nd sample of MPI3D in Figure 21), the predictions will deviate or become unclear.

Original: [Figure]
Revised: [Figure]

Original: E L IMITATIONS
Revised: E L IMITATIONS

Original: We conclude our limitations in two aspects.
Revised: We conclude our limitations in two aspects.

Original: (1) Complexity of datasets.
Revised: (1) Complexity of datasets.

Original: Because compositional law parsing is an underexplored task, we ﬁrst validate the effectiveness of SVCL on datasets with relatively clear and simple rules to avoid the inﬂuence of unknown confounders in complicated datasets.
Revised: Because compositional law parsing is an underexplored task, we ﬁrst validate the effectiveness of SVCL on datasets with relatively clear and simple rules to avoid the inﬂuence of unknown confounders in complicated datasets.

Original: We believe that the compositionality of laws also exists in more complex scenarios (e.g., learning physical laws in realistic scenes).
Revised: We believe that the compositionality of laws also exists in more complex scenarios (e.g., learning physical laws in realistic scenes).

Original: Furthermore, some vision tasks may beneﬁt from compositional law parsing.
Revised: Furthermore, some vision tasks may beneﬁt from compositional law parsing.

Original: For example, we can perform a controllable video generation process based on law modiﬁcation or make more interpretable predictions by analyzing dominant laws in videos.
Revised: For example, we can perform a controllable video generation process based on law modiﬁcation or make more interpretable predictions by analyzing dominant laws in videos.

Original: Discovering such compositional law parsing ability in more complex situations can be a valuable topic in future works.
Revised: Discovering such compositional law parsing ability in more complex situations can be a valuable topic in future works.

Original: (2) Setting the number of LRFs.
Revised: (2) Setting the number of LRFs.

Original: CLAP places an upper bound on the number of LRFs.
Revised: CLAP places an upper bound on the number of LRFs.

Original: For scenes with multiple complex rules, we can empirically set an appropriate upper bound or directly put a large upper bound for the number of LRFs.
Revised: For scenes with multiple complex rules, we can empirically set an appropriate upper bound or directly put a large upper bound for the number of LRFs.

Original: However, a large bound will linearly increase the number of model parameters, and the redundant concepts will waste computing resources, decreasing computational efﬁciency.
Revised: However, a large bound will linearly increase the number of model parameters, and the redundant concepts will waste computing resources, decreasing computational efﬁciency.

Original: Therefore, exploring a mechanism in the future to dynamically adjust the number of functions will be meaningful.
Revised: Therefore, exploring a mechanism in the future to dynamically adjust the number of functions will be meaningful.

Original: When applying CLAP to more complex scenes, we may introduce more inductive biases for better concept decomposition (e.g., use a more task-speciﬁc encoder or decoder).
Revised: When applying CLAP to more complex scenes, we may introduce more inductive biases for better concept decomposition (e.g., use a more task-speciﬁc encoder or decoder).

Original: We design CLAP as an unsupervised model independent of task-speciﬁc annotations from tasks, which means that it can be extended to take advantage of these annotations.
Revised: We design CLAP as an unsupervised model independent of task-speciﬁc annotations from tasks, which means that it can be extended to take advantage of these annotations.

Original: Thus we can ﬁnd ways to integrate CLAP with additional supervision information (e.g., supervise SVCL with the labels of changing factors for each scene) to help concept learning.
Revised: Thus we can ﬁnd ways to integrate CLAP with additional supervision information (e.g., supervise SVCL with the labels of changing factors for each scene) to help concept learning.