Original: L ANGUAGE -M EDIATED , O BJECT -C ENTRIC R EPRESENTATION L EARNING
Revised: L ANGUAGE -M EDIATED , O BJECT -C ENTRIC R EPRESENTATION L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present Language-mediated, Object-centric Representation Learning (LORL),  learning disentangled, object-centric scene representations from vision and language.
Revised: We present Language-mediated, Object-centric Representation Learning (LORL), for learning disentangled, object-centric scene representations from vision and language.

Original: LORL builds upon recent advances in unsupervised object segmentation, notably MONet and Slot Attention.
Revised: LORL builds upon recent advances in unsupervised object segmentation, notably MONet and Slot Attention.

Original: Just like these algorithms, LORL also learns an object-centric representation by reconstructing the input image.
Revised: Just like these algorithms, LORL also learns an object-centric representation by reconstructing the input image.

Original: But LORL further learns to associate the learned representations to concepts, i.e., words for object categories, properties, and spatial relationships, from language input.
Revised: But LORL further learns to associate the learned representations to concepts, i.e., words for object categories, properties, and spatial relationships, from language input.

Original: These objectcentric concepts derived from language facilitate the learning of object-centric representations.
Revised: These objectcentric concepts derived from language facilitate the learning of object-centric representations.

Original: LORL can be integrated with various unsupervised segmentation algorithms that are language-agnostic.
Revised: LORL can be integrated with various unsupervised segmentation algorithms that are language-agnostic.

Original: Experiments show that LORL consistently improves the performance of MONet and Slot Attention on two datasets via the help of language.
Revised: Experiments show that LORL consistently improves the performance of MONet and Slot Attention on two datasets via the help of language.

Original: We also show that concepts learned by LORL aid downstream tasks such as referential expression interpretation.
Revised: We also show that concepts learned by LORL aid downstream tasks such as referring expression comprehension.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Cognitive studies show that human infants develop object individuation skill from diverse sources of information: spatial-temporal information, object property information, and language (Xu, 1999; 2007; Westermann & Mareschal, 2014).
Revised: Cognitive studies show that human infants develop object individuation skill from diverse sources of information: spatial-temporal information, object property information, and language (Xu, 1999; 2007; Westermann & Mareschal, 2014).

Original: Speciﬁcally, young infants develop object-based attention that disentangles the motion and location of objects from their visual appearance features.
Revised: Speciﬁcally, young infants develop object-based attention that disentangles the motion and location of objects from their visual appearance features.

Original: Later on, they can leverage the knowledge acquired through word learning to solve the problem of object individuation: words provide clues about object identity and type.
Revised: Later on, they can leverage the knowledge acquired through word learning to solve the problem of object individuation: words provide clues about object identity and type.

Original: The general picture from cognitive science is that object perception and language co-develop in support of one another (Bloom, 2002).
Revised: The general picture from cognitive science is that object perception and language co-develop in support of one another (Bloom, 2002).

Original: Our long-term goal is to endow machines with similar abilities.
Revised: Our long-term goal is to endow machines with similar abilities.

Original: In this paper, we focus on how language may support object segmentation.
Revised: In this paper, we focus on how language may support object segmentation.

Original: Many recent works have studied the problem of unsupervised object representation learning, though without language.
Revised: Many recent works have studied the problem of unsupervised object representation learning, though without language.

Original: As an example, factorized, object-centric scene representations have been used in various kinds of prediction (Goel et al., 2018), reasoning (Yi et al., 2018; Mao et al., 2019), and planning tasks (Veerapaneni et al., 2020), but they have not considered the role of language and how it may help object representation learning.
Revised: As an example, factorized, object-centric scene representations have been used in various kinds of prediction (Goel et al., 2018), reasoning (Yi et al., 2018; Mao et al., 2019), and planning tasks (Veerapaneni et al., 2020), but they have not considered the role of language and how it may help object representation learning.

Original: As a concrete example, consider the input images shown in Fig.
Revised: As a concrete example, consider the input images shown in Fig.

Original: 1 and the paired questions.
Revised: 1 and the paired questions.

Original: From language, we can learn to associate concepts, such as black, pan , and legs , with the referred object’s visual appearance.
Revised: From language, we can learn to associate concepts, such as black, pan , and legs , with the referred object’s visual appearance.

Original: Further, language provides cues about how an input scene should be segmented into individual objects: a wrong parsing of the input scene will lead to an incorrect answer to the question.
Revised: Further, language provides cues about how an input scene should be segmented into individual objects: a wrong parsing of the input scene will lead to an incorrect answer to the question.

Original: We can learn from such failure that the handle belongs to the frying pan (Fig. 1a) and the chair has four legs (Fig. 1b).
Revised: We can learn from such failure that the handle belongs to the frying pan (Fig. 1a) and the chair has four legs (Fig. 1b).

Original: Motivated by these observations, we propose a computational framework, Language-mediated, Objectcentric Representation Learning (LORL), that learns object-centric representations and associates them to their visual appearance (masks) in images, and to concepts—words for object properties such as color, shape, and material—as provided in language.
Revised: Motivated by these observations, we propose a computational framework, Language-mediated, Objectcentric Representation Learning (LORL), that learns object-centric representations and associates them to their visual appearance (masks) in images, and to concepts—words for object properties such as color, shape, and material—as provided in language.

Original: Here the language input can be either descriptive sentences or question-answer pairs.
Revised: Here the language input can be either descriptive sentences or question-answer pairs.

Original: LORL requires no annotations on object masks, categories, or properties during the learning process.
Revised: LORL requires no annotations on object masks, categories, or properties during the learning process.

Original: LORL has four modules.
Revised: LORL has four modules.

Original: The ﬁrst is an image encoder, learning to encode an image into factorized, object-centric representations.
Revised: The ﬁrst is an image encoder, learning to encode an image into factorized, object-centric representations.

Original: The second is an image decoder, learning to reconstruct masks for individual objects from the learned representations by reconstructing the input.
Revised: The second is an image decoder, learning to reconstruct masks for individual objects from the learned representations by reconstructing the input.

Original: These two modules share the same formulation as recent unsupervised object segmentation research, and may be realized as methods such as MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020).
Revised: These two modules share the same formulation as recent unsupervised object segmentation research, and may be realized as methods such as MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020).

Original: The third module in LORL is a pre-trained semantic parser that translates the input sentence into a semantic, executable program, where each concept (i.e., words for object properties such as ‘red’) is associated with a vector space embedding.
Revised: The third module in LORL is a pre-trained semantic parser that translates the input sentence into a semantic, executable program, where each concept (i.e., words for object properties such as ‘red’) is associated with a vector space embedding.

Original: Finally, the last module, a neural-symbolic program executor, takes the object-centric representation from Module 1, intermediate representations from
Revised: Finally, the last module, a neural-symbolic program executor, takes the object-centric representation from Module 1, intermediate representations from

Original: Module 2, and concept embeddings and the semantic program from Module 3 as input, and outputs an answer if the language input is a question, or TRUE/FALSE if it’s a descriptive sentence.
Revised: Module 2, and concept embeddings and the semantic program from Module 3 as input, and outputs an answer if the language input is a question, or TRUE/FALSE if it’s a descriptive sentence.

Original: The correctness of the executor’s output and the quality of reconstructed images (as output of Module 2) are the two supervisory signals we use to jointly train Modules 1, 2, and 4.
Revised: The correctness of the executor’s output and the quality of reconstructed images (as output of Module 2) are the two supervisory signals we use to jointly train Modules 1, 2, and 4.

Original: We compare our model to state-of-the-art unsupervised segmentation methods, MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020).
Revised: We compare our model to state-of-the-art unsupervised segmentation methods, MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020).

Original: Our evaluation is  on two datasets:
Revised: Our evaluation is based on two datasets:

Original: ShopVRB (Nazarczuk & Mikolajczyk, 2020) contains images of daily objects and question-answer pairs; PartNet (Mo et al., 2019) contains images of furniture with hierarchical structure, supplemented by descriptive sentences we collected ourselves.
Revised: ShopVRB (Nazarczuk & Mikolajczyk, 2020) contains images of daily objects and question-answer pairs; PartNet (Mo et al., 2019) contains images of furniture with hierarchical structure, supplemented by descriptive sentences we collected ourselves.

Original: We show that LORL consistently outperforms baselines on unsupervised object segmentation, much more likely to group different parts of a single object into a single mask.
Revised: We show that LORL consistently outperforms baselines on unsupervised object segmentation, much more likely to group different parts of a single object into a single mask.

Original: We further analyze the object-centric representations learned by LORL.
Revised: We further analyze the object-centric representations learned by LORL.

Original: When visualizing them in 2D, we found that conceptually similar objects (e.g. objects of similar shapes) appear to be clustered in the embedding space.
Revised: When visualizing them in 2D, we found that conceptually similar objects (e.g. objects of similar shapes) appear to be clustered in the embedding space.

Original: Moreover, experiments demonstrate that the learned concepts can be used in new tasks, such as visual grounding of referential expressions, without any additional ﬁne-tuning.
Revised: Moreover, experiments demonstrate that the learned concepts can be used in new tasks, such as visual grounding of referring expressions, without any additional ﬁne-tuning.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Unsupervised object representation learning.
Revised: Unsupervised object representation learning.

Original: Given an input image, unsupervised object representation learning methods segment objects in the scene and build an object-centric representation for them.
Revised: Given an input image, unsupervised object representation learning methods segment objects in the scene and build an object-centric representation for them.

Original: A mainstream approach has focused on using compositional generative scene models that decompose the scene as a mixture of component images (Greff et al., 2016; Eslami et al., 2016; Greff et al., 2017; Burgess et al., 2019; Engelcke et al., 2020; Greff et al., 2019; Locatello et al., 2020).
Revised: A mainstream approach has focused on using compositional generative scene models that decompose the scene as a mixture of component images (Greff et al., 2016; Eslami et al., 2016; Greff et al., 2017; Burgess et al., 2019; Engelcke et al., 2020; Greff et al., 2019; Locatello et al., 2020).

Original: In general, these models use an encoder-decoder architecture: the image encoder encodes the input image into a set of latent object representations, which are fed into the image decoder to reconstruct the image.
Revised: In general, these models use an encoder-decoder architecture: the image encoder encodes the input image into a set of latent object representations, which are fed into the image decoder to reconstruct the image.

Original: Speciﬁcally, Greff et al.
Revised: Speciﬁcally, Greff et al.

Original: ; Burgess et al.
Revised: ; Burgess et al.

Original: ; Engelcke et al.
Revised: ; Engelcke et al.

Original: (2020) use recurrent encoders that iteratively localize and encode objects in the scene.
Revised: (2020) use recurrent encoders that iteratively localize and encode objects in the scene.

Original: Another line of research (Eslami et al., 2016; Crawford & Pineau, 2019; Kosiorek et al., 2018; Stelzner et al., 2019; Lin et al., 2020) leverages object locality to attend to different local patches of the image.
Revised: Another line of research (Eslami et al., 2016; Crawford & Pineau, 2019; Kosiorek et al., 2018; Stelzner et al., 2019; Lin et al., 2020) leverages object locality to attend to different local patches of the image.

Original: These models often use a pixel-level reconstruction loss.
Revised: These models often use a pixel-level reconstruction loss.

Original: In contrast, we propose to explore how language, in addition to visual observations, may contribute to object-centric representation learning.
Revised: In contrast, we propose to explore how language, in addition to visual observations, may contribute to object-centric representation learning.

Original: There have also been work that uses other types of supervisions, such as dynamic prediction (Kipf et al., 2020; Bear et al., 2020) and multi-view consistency (Prabhudesai et al., 2020).
Revised: There have also been work that uses other types of supervisions, such as dynamic prediction (Kipf et al., 2020; Bear et al., 2020) and multi-view consistency (Prabhudesai et al., 2020).

Original: In this paper, we focus on unsupervised learning of object-centric representations from static images and language.
Revised: In this paper, we focus on unsupervised learning of object-centric representations from static images and language.

Original: Visual concept learning.
Revised: Visual concept learning.

Original: Learning visual concepts from language and other forms of supervisions provides useful representations for various downstream tasks, such as cross-modal retrieval (Wu et al., 2019), visual-question answering (Yi et al., 2018; Mao et al., 2019), and scene manipulation (Prabhudesai et al., 2020).
Revised: Learning visual concepts from language and other forms of supervisions provides useful representations for various downstream tasks, such as cross-modal retrieval (Wu et al., 2019), visual-question answering (Yi et al., 2018; Mao et al., 2019), and scene manipulation (Prabhudesai et al., 2020).

Original: Previous work has been focusing on various types of representations (Ren et al., 2016; Wu et al., 2017b), training algorithms (Faghri et al., 2018) and supervisions (Johnson et al., 2016).
Revised: Previous work has been focusing on various types of representations (Ren et al., 2016; Wu et al., 2017), training algorithms (Faghri et al., 2018) and supervisions (Johnson et al., 2016; Yang et al., 2018).

Original: In this paper, we focus on learning visual concepts that can be grounded on object-centric representations.
Revised: In this paper, we focus on learning visual concepts that can be grounded on object-centric representations.

Original: Recent works on object-centric grounding of visual concepts (Wu et al., 2017a; Mao et al., 2019; Prabhudesai et al., 2020) have shown great success in achieving high performance in downstream tasks and strong generalization from a small amount of data.
Revised: Recent works on object-centric grounding of visual concepts (Wu et al., 2017; Mao et al., 2019; Prabhudesai et al., 2020) have shown great success in achieving high performance in downstream tasks and strong generalization from a small amount of data.

Original: However, these methods assume pre-trained object detectors to generate object proposals in the scene.
Revised: However, these methods assume pre-trained object detectors to generate object proposals in the scene.

Original: In contrast, our LORL learns to individuate objects and associates concepts with the learned object-centric representations without any annotations on object segmentation masks or properties.
Revised: In contrast, our LORL learns to individuate objects and associates concepts with the learned object-centric representations without any annotations on object segmentation masks or properties.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Before delving into our framework for language-mediated object-centric representation learning, we ﬁrst discuss a general framework that uniﬁes multiple concurrent unsupervised object representation learning methods and a neuro-symbolic framework for learning visual concepts from language.
Revised: Before delving into our framework for language-mediated object-centric representation learning, we ﬁrst discuss a general framework that uniﬁes multiple concurrent unsupervised object representation learning methods and a neuro-symbolic framework for learning visual concepts from language.

Original: 3. U NSUPERVISED O BJECT -C ENTRIC R EPRESENTATION LEARNING
Revised: 3. U NSUPERVISED O BJECT -C ENTRIC R EPRESENTATION LEARNING

Original: Given an image I , a typical unsupervised object representation learning model will decompose the scene into a series of slot proﬁles { ( z 1 , x 1 , m 1 ) , . . .
Revised: Given an image I , a typical unsupervised object representation learning model will decompose the scene into a series of slot proﬁles { ( z 1 , x 1 , m 1 ) , . . .

Original: , ( z K , x K , m K ) } , where each slot proﬁle is expected to represent an object (or nothing, as the number of slots may be greater than the actual number of objects in the scene).
Revised: , ( z K , x K , m K ) } , where each slot proﬁle is expected to represent an object (or nothing, as the number of slots may be greater than the actual number of objects in the scene).

Original: Here o i is the object feature, x i is the object image, and m i is the object mask specifying its location in the scene.
Revised: Here z i is the object feature, x i is the object image, and m i is the object mask specifying its location in the scene.

Original: In our paper, we focus on two recent models, MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020).
Revised: In our paper, we focus on two recent models, MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020).

Original: MONet uses a recurrent spatial attention network (Ronneberger et al., 2015) to segment out objects in the scene, and adopts a variational autoencoder (Kingma & Welling, 2014) to encode objects as well as reconstructing object images for self-supervision.
Revised: MONet uses a recurrent spatial attention network (Ronneberger et al., 2015) to segment out objects in the scene, and adopts a variational autoencoder (Kingma & Welling, 2014) to encode objects as well as reconstructing object images for self-supervision.

Original: At a very high level, its objective function is calculated as
Revised: At a very high level, its objective function is calculated as

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term is a pixel-wise L2 reconstruction loss and the second term computes the KL divergence between the distribution of z k ’s and a prior Gaussian distribution.
Revised: where the ﬁrst term is a pixel-wise L2 reconstruction loss and the second term computes the KL divergence between the distribution of z k ’s and a prior Gaussian distribution.

Original: Slot Attention uses a transformer-like attention network (Vaswani et al., 2017) to extract object features, and decode them with convolutional neural networks to component images and object masks.
Revised: Slot Attention uses a transformer-like attention network (Vaswani et al., 2017) to extract object features, and decode them with convolutional neural networks to component images and object masks.

Original: The model is trained by the same reconstruction loss in the form of L2-norm:
Revised: The model is trained by the same reconstruction loss in the form of L2-norm:

Original: [Equation]
Revised: [Equation]

Original: 3. N EURO -S YMBOLIC C ONCEPT L EARNING
Revised: 3. N EURO -S YMBOLIC C ONCEPT L EARNING

Original: The neuro-symbolic concept learner (NS-CL; Mao et al., 2019) learns visual concepts by looking at images and reading paired questions and answers.
Revised: The neuro-symbolic concept learner (NS-CL; Mao et al., 2019) learns visual concepts by looking at images and reading paired questions and answers.

Original: NS-CL takes a set of segmented objects in a given image as its input, extracts their visual features with a ResNet (He et al., 2015), translates the input question into an executable program by a semantic parser, and executes the program based on the object-centric representation to answer the question.
Revised: NS-CL takes a set of segmented objects in a given image as its input, extracts their visual features with a ResNet (He et al., 2015), translates the input question into an executable program by a semantic parser, and executes the program based on the object-centric representation to answer the question.

Original: The key idea of NS-CL is to explicitly represent individual concepts in natural language (colors, shapes, spatial relationships, etc.)
Revised: The key idea of NS-CL is to explicitly represent individual concepts in natural language (colors, shapes, spatial relationships, etc.)

Original: as vector space embeddings, and associate them with the object embeddings.
Revised: as vector space embeddings, and associate them with the object embeddings.

Original: NS-CL answers the input question by executing the program based on the object-centric representation.
Revised: NS-CL answers the input question by executing the program based on the object-centric representation.

Original: For example, in order to query the name of the white object  Fig. 2, NS-CL ﬁrst ﬁlters out the object by computing the cosine similarity between the concept white and individual object representations, which produces a “mask” vector where each entry denotes the probability that an object has been selected.
Revised: For example, in order to query the name of the white object in Fig. 2, NS-CL ﬁrst ﬁlters out the object by computing the cosine similarity between the concept white and individual object representations, which produces a “mask” vector where each entry denotes the probability that an object has been selected.

Original: The output “mask” on the objects will be fed into the next module and the execution will
Revised: The output “mask” on the objects will be fed into the next module and the execution will

Original: [Figure]
Revised: [Figure]

Original: continue.
Revised: continue.

Original: The last query operation will produce the answer to the question.
Revised: The last query operation will produce the answer to the question.

Original: The vector embeddings of individual objects and the concepts will be jointly trained based on language supervision
Revised: The vector embeddings of individual objects and the concepts will be jointly trained based on language supervision

Original:  L ANGUAGE - MEDIATED , O BJECT - CENTRIC R EPRESENTATION L EARNING
Revised:  L ANGUAGE - MEDIATED , O BJECT - CENTRIC R EPRESENTATION L EARNING

Original: Marrying the ideas of unsupervised object-centric representation learning and neuro-symbolic concept learning, we are able to learn an object-centric representation using both visual and language supervision.
Revised: Marrying the ideas of unsupervised object-centric representation learning and neuro-symbolic concept learning, we are able to learn an object-centric representation using both visual and language supervision.

Original: Fig.
Revised: Fig.

Original: 2 illustrates the framework of Language-mediated, Object-centric Representation
Revised: 2 illustrates the framework of Language-mediated, Object-centric Representation

Original: Learning (LORL).
Revised: Learning (LORL).

Original: As mentioned in Section 1, LORL comprises four modules: an image encoder, an image decoder, a semantic parser, and a neuro-symbolic program executor.
Revised: As mentioned in Section 1, LORL comprises four modules: an image encoder, an image decoder, a semantic parser, and a neuro-symbolic program executor.

Original: Image encoder.
Revised: Image encoder.

Original: Given an input image, we ﬁrst use the image encoder (Fig. 2a) to individuate objects in the scene and extract an object-centric scene representation.
Revised: Given an input image, we ﬁrst use the image encoder (Fig. 2a) to individuate objects in the scene and extract an object-centric scene representation.

Original: It takes the input image as its input, individuates objects in the scene, and produces a collection of latent slot embeddings { z i } .
Revised: It takes the input image as its input, individuates objects in the scene, and produces a collection of latent slot embeddings { z i } .

Original: Image decoder.
Revised: Image decoder.

Original: The decoder (Fig. 2b) takes the object-centric representation produced by the image encoder and produces a 3-tuple for each individual slot ( x k , m k , s k ) , where x k reconstructs the RGB image of the slot, m k reconstructs the mask, and s k ∈ [0 , 1] is a scalar indicating the objectness of the slot.
Revised: The decoder (Fig. 2b) takes the object-centric representation produced by the image encoder and produces a 3-tuple for each individual slot ( x k , m k , s k ) , where x k reconstructs the RGB image of the slot, m k reconstructs the mask, and s k ∈ [0 , 1] is a scalar indicating the objectness of the slot.

Original: That is, whether k -th slot corresponds to a single object in the scene.
Revised: That is, whether k -th slot corresponds to a single object in the scene.

Original: Here, we have extended the general framework we described in Section 3.1 with an objectness indicator.
Revised: Here, we have extended the general framework we described in Section 3.1 with an objectness indicator.

Original: It serves dual purposes.
Revised: It serves dual purposes.

Original: First, it weights each reconstructed component image while generating the reconstructed image.
Revised: First, it weights each reconstructed component image while generating the reconstructed image.

Original: Mathematically, the reconstructed image I (cid:48) is computed as: I (cid:48) =
Revised: Mathematically, the reconstructed image I (cid:48) is computed as: I (cid:48) =

Original: (cid:80)
Revised: (cid:80)

Original: Kk
Revised: Kk

Original: =1 s k · ( m k x k ) .
Revised: =1 s k · ( m k x k ) .

Original: Second, it mediates the output of all filter operations in the program executor.
Revised: Second, it mediates the output of all filter operations in the program executor.

Original: In this paper, we will experiment with two image encoder-decoder options: MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020).
Revised: In this paper, we will experiment with two image encoder-decoder options: MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020).

Original: They are both compatible with the general framework described above.
Revised: They are both compatible with the general framework described above.

Original: For both models, we use a single linear layer to predict the objectness score for each slot on top of the second-last layer of their image decoders.
Revised: For both models, we use a single linear layer to predict the objectness score for each slot on top of the second-last layer of their image decoders.

Original: Semantic parser.
Revised: Semantic parser.

Original: A pre-trained semantic parser (Fig. 2c) will translate the input question into an executable program composed of primitive operations, such as filter , which ﬁlters out objects with certain concepts and query , which query the attribute of the input object.
Revised: A pre-trained semantic parser (Fig. 2c) will translate the input question into an executable program composed of primitive operations, such as filter , which ﬁlters out objects with certain concepts and query , which queries the attribute of the input object.

Original: We use  the same domain-speciﬁc language (DSL) for representing programs as CLEVR (Johnson et al., 2017, see also Appendix A for details).
Revised: We use roughly the same domain-speciﬁc language (DSL) for representing programs as CLEVR (Johnson et al., 2017a, see also Appendix A for details).

Original: All concepts  appear in the program, such as white , are associated with distinct, learnable concept embedding vectors.
Revised: All concepts that appear in the program, such as white , are associated with distinct, learnable concept embedding vectors.

Original: Neuro-symbolic program executor.
Revised: Neuro-symbolic program executor.

Original: The program executor (Fig. 2d) takes the object-centric representation from the image encoder { z k } , the objectness score { s k } from the image decoder, the concept embeddings and the program generated by the semantic parser as its input.
Revised: The program executor (Fig. 2d) takes the object-centric representation from the image encoder { z k } , the objectness score { s k } from the image decoder, the concept embeddings and the program generated by the semantic parser as its input.

Original: It executes the program based on the visual and concept representations to answer the question.
Revised: It executes the program based on the visual and concept representations to answer the question.

Original: Q: The small plate is what color?
Revised: Q: The small plate is what color?

Original: P:scene, filter[small], filter[plate], query[color]
Revised: P:scene, filter[small], filter[plate], query[color]

Original: A: purple
Revised: A: purple

Original: Q: What number of ceramic things are there?
Revised: Q: What number of ceramic things are there?

Original: P: scene, filter[ceramic], count
Revised: P: scene, filter[ceramic], count

Original: A:
Revised: A:

Original: D: There is a part left of the cyan seat, its color is red.
Revised: D: There is a part left of the cyan seat, its color is red.

Original: P: scene, filter[cyan, seat], relate[left], equal[red]
Revised: P: scene, filter[cyan, seat], relate[left], equal[red]

Original: D: The number of legs there is 4.
Revised: D: The number of legs there is 4.

Original: P: scene, filter[leg], count, equal[4]
Revised: P: scene, filter[leg], count, equal[4]

Original: D: The color of the back is brown.
Revised: D: The color of the back is brown.

Original: P: scene, filter[back], query[color], equal[brown]
Revised: P: scene, filter[back], query[color] [Figure] equal[brown]

Original: [Figure]
Revised: [Figure]

Original: The original program executor in NS-CL (Section 3.2) assumes a pre-trained object detector for generating object proposals.
Revised: The original program executor in NS-CL (Section 3.2) assumes a pre-trained object detector for generating object proposals.

Original: In LORL, we associate each object representation with an objectness score s k .
Revised: In LORL, we associate each object representation with an objectness score s k .

Original: Recall that an filter operation in NS-CL produces a mask vector indicating whether an object has been selected.
Revised: Recall that a filter operation in NS-CL produces a mask vector indicating whether an object has been selected.

Original: Here, we mediate the output of an filter(c) operation as min( s k , filter(c) ) .
Revised: Here, we mediate the output of an filter(c) operation as min( s k , filter(c) ) .

Original: Intuitively, a slot will be selected only if, ﬁrst, it has concept c and, second, it corresponds to a single object in the scene.
Revised: Intuitively, a slot will be selected only if, ﬁrst, it has concept c and, second, it corresponds to a single object in the scene.

Original: Training paradigm.
Revised: Training paradigm.

Original: During training, we jointly optimize the image encoder, the image decoder, and the concept embeddings.
Revised: During training, we jointly optimize the image encoder, the image decoder, and the concept embeddings.

Original: LORL is trained by minimizing the loss L :
Revised: LORL is trained by minimizing the loss L :

Original: [Equation]
Revised: [Equation]

Original: For MONet-based image encoder-decoder, we use Equation 1 as the perception loss L perception , while for Slot Attention-based encoder-decoder, we use Equation 2.
Revised: For MONet-based image encoder-decoder, we use Equation 1 as the perception loss L perception , while for Slot Attention-based encoder-decoder, we use Equation 2.

Original: The neuro-symbolic program executor produces a distribution over candidate answers to the input question.
Revised: The neuro-symbolic program executor produces a distribution over candidate answers to the input question.

Original: We use the cross-entropy loss between the predicted answer and the ground truth answer as L reasoning .
Revised: We use the cross-entropy loss between the predicted answer and the ground truth answer as L reasoning .

Original: We use a three-stage training paradigm for LORL.
Revised: We use a three-stage training paradigm for LORL.

Original: First, we train the model with only visual inputs with L perception for N 1 epochs.
Revised: First, we train the model with only visual inputs with L perception for N 1 epochs.

Original: Next, we ﬁx the image encoder and the image decoder, and optimize the concept embeddings with the loss term L reasoning for N 2 epochs.
Revised: Next, we ﬁx the image encoder and the image decoder, and optimize the concept embeddings with the loss term L reasoning for N 2 epochs.

Original: During this second stage, the image encoder and the decoder can already produce descent object segmentation results.
Revised: During this second stage, the image encoder and the decoder can already produce descent object segmentation results.

Original: Finally, we jointly optimize all three modules for N 3 epochs.
Revised: Finally, we jointly optimize all three modules for N 3 epochs.

Original: We provide detailed information about the hyperparameters for different models in Appendix B.
Revised: We provide detailed information about the hyperparameters for different models in Appendix B.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We ﬁrst evaluate whether the representations learned by LORL lead to better image segmentation with the help of language.
Revised: We ﬁrst evaluate whether the representations learned by LORL lead to better image segmentation with the help of language.

Original: We then evaluate how these representations may be used for instance retrieval, visual reasoning, and referential expression interpretation.
Revised: We then evaluate how these representations may be used for instance retrieval, visual reasoning, and referring expression comprehension.

Original: 5. I MAGE S EGMENTATION
Revised: 5. I MAGE S EGMENTATION

Original: Data.
Revised: Data.

Original: We use two datasets for image segmentation evaluation.
Revised: We use two datasets for image segmentation evaluation.

Original: The ﬁrst, Shop-VRB-Simple, is based on Shop-VRB (Nazarczuk & Mikolajczyk, 2020), a dataset of complex household objects and question-answer pairs.
Revised: The ﬁrst, Shop-VRB-Simple, is based on Shop-VRB (Nazarczuk & Mikolajczyk, 2020), a dataset of complex household objects and question-answer pairs.

Original: The second is based on chairs in PartNet (Mo et al., 2019), a dataset where the objects are different parts of a chair.
Revised: The second is based on chairs in PartNet (Mo et al., 2019), a dataset where the objects are different parts of a chair.

Original: Fig.
Revised: Fig.

Original: 3 shows some examples from the two datasets.
Revised: 3 shows some examples from the two datasets.

Original: Shop-VRB is a visual reasoning dataset, similar to CLEVR (Johnson et al., 2017), but with complex household objects of different sizes, weights, materials, colors, shapes, and mobility.
Revised: Shop-VRB is a visual reasoning dataset, similar to CLEVR (Johnson et al., 2017a), but with complex household objects of different sizes, weights, materials, colors, shapes, and mobility.

Original: Because the original Shop-VRB dataset includes very small and highly transparent objects and complex backgrounds, which current unsupervised representation learning models cannot handle, we generate 10K images with a clean background ourselves using large objects from the dataset.
Revised: Because the original Shop-VRB dataset includes very small and highly transparent objects and complex backgrounds, which current unsupervised representation learning models cannot handle, we generate 10K images with a clean background ourselves using large objects from the dataset.

Original: We also pair every image with 9 questions, resulting in 90K questions in total.
Revised: We also pair every image with 9 questions, resulting in 90K questions in total.

Original: The test split has 960 imagesand 8.6K questions.
Revised: The test split has 960 images

Original: We name this variant Shop-VRB-Simple and we ﬁnd that it better highlights the quality of learned object-centric representations for various models.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: While the previous literature on unsupervised object segmentation mainly focuses on settings where objects are spatially disentangled, we also explore how language may help when objects of interest are different parts of a global shape.
Revised: While the previous literature on unsupervised object segmentation mainly focuses on settings where objects are spatially disentangled, we also explore how language may help when objects of interest are different parts of a global shape.

Original: To this end, we collect a new dataset, PartNet-Chairs, using chair shapes from PartNet.
Revised: To this end, we collect a new dataset, PartNet-Chairs, using chair shapes from PartNet.

Original: Every image here shows a chair, where each part of the chair (legs, seat, back, arms) is randomly assigned a color.
Revised: Every image here shows a chair, where each part of the chair (legs, seat, back, arms) is randomly assigned a color.

Original: We select six different chair shapes with one or four legs and zero or two arms.
Revised: We select six different chair shapes with one or four legs and zero or two arms.

Original: We generate 5K images for training.
Revised: We generate 5K images for training.

Original: Each image is paired with 8 descriptive sentences generated from human-written templates, resulting in 40K examples in total.
Revised: Each image is paired with 8 descriptive sentences generated from human-written templates, resulting in 40K examples in total.

Original: The test split has 960 images.
Revised: The test split has 960 images.

Original: Each sentence describes the name and color of parts.
Revised: Each sentence describes the name and color of parts.

Original: We provide all templates in
Revised: We provide all templates in

Original: Appendix C. We are interested in whether object-centric representation learning models may separate these parts and whether and how language may help in this scenario.
Revised: Appendix C. We are interested in whether object-centric representation learning models may separate these parts and whether and how language may help in this scenario.

Original: Baselines.
Revised: Baselines.

Original: We use MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020) as the image modules (Modules 1 and 2), and evaluate how the incorporation of language may improve their performance.
Revised: We use MONet (Burgess et al., 2019) and Slot Attention (Locatello et al., 2020) as the image modules (Modules 1 and 2), and evaluate how the incorporation of language may improve their performance.

Original: Because MONet is color-sensitive, and on Shop-VRB-Simple, many objects have diverse colors and specular reﬂection, it does not produce meaningful results there.
Revised: Because MONet is color-sensitive, and on Shop-VRB-Simple, many objects have diverse colors and specular reﬂection, it does not produce meaningful results there.

Original: Thus we only show results with MONet on PartNet-Chairs.
Revised: Thus we only show results with MONet on PartNet Chairs.

Original: We show results with Slot Attention on both datasets.
Revised: We show results with Slot Attention on both datasets.

Original: Metrics.
Revised: Metrics.

Original: We use three metrics for evaluation.
Revised: We use three metrics for evaluation.

Original: Following Greff et al.
Revised: Following Greff et al.

Original: (2019), we ﬁrst use the Adjusted
Revised: (2019), we ﬁrst use the Adjusted

Original: Rand Index (ARI; Rand, 1971; Hubert & Arabie, 1985).
Revised: Rand Index (ARI; Rand, 1971; Hubert & Arabie, 1985).

Original: It treats segmentation as a clustering problem: each mask is the cluster index that the pixels within belong to.
Revised: It treats segmentation as a clustering problem: each mask is the cluster index that the pixels within belong to.

Original: ARI is computed as the similarity between the predicted and ground truth clusters, and ranges from 0 (random) to 1 (perfect match).
Revised: ARI is computed as the similarity between the predicted and ground truth clusters, and ranges from 0 (random) to 1 (perfect match).

Original: In practice, we found this pixel-wise metric is sensitive to the size of objects: a model that infrequently makes mistakes on large objects will have lower ARI than one that frequently mis-segments small objects.
Revised: In practice, we found this pixel-wise metric is sensitive to the size of objects: a model that infrequently makes mistakes on large objects will have lower ARI than one that frequently mis-segments small objects.

Original: Thus, we in addition design two object-centric metrics:
Revised: Thus, we in addition design two object-centric metrics:

Original: • Ground Truth Split Ratio (GT Split) measures the ratio of objects (GT masks) that are covered by more than one prediction mask.
Revised: • Ground Truth Split Ratio (GT Split) measures the ratio of objects (GT masks) that are covered by more than one prediction mask.

Original: • Prediction Split Ratio (Pred Split) measures the ratio of prediction masks that cover more than one object (GT mask).
Revised: • Prediction Split Ratio (Pred Split) measures the ratio of prediction masks that cover more than one object (GT mask).

Original: Concretely, we ﬁrst assign each pixel to the prediction mask with the maximum value at the pixel.
Revised: Concretely, we ﬁrst assign each pixel to the prediction mask with the maximum value at the pixel.

Original: We say a prediction mask covers an object if it covers at least 20% of the object’s pixels.
Revised: We say a prediction mask covers an object if it covers at least 20% of the object’s pixels.

Original: The GT and
Revised: The GT and

Original: Pred Split ratios are thus deﬁned as:
Revised: Pred Split ratios are thus deﬁned as:

Original: GTSplit = # of objects that are covered by > 1 masks # of objects that are covered by > 0 masks ; PredSplit = # of masks that cover > 1 objects # of masks that cover > 0 objects .

Original: [Table]
Revised: [Table]

Original: Burgess

Original: [Table]
Revised: [Table]

Original: Locatello et [Table] [Table]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Ideally, there is a one-to-one correspondence between objects and predicted masks, with both GT
Revised: Ideally, there is a one-to-one correspondence between objects and predicted masks, with both GT

Original: Split and Pred Split being 0.
Revised: Split and Pred Split being 0.

Original: Results.
Revised: Results.

Original: The quantitative results  are summarized in Table 1.
Revised: The quantitative results on SHOP-VRB-Simple are summarized in Table 1.

Original: For this table and all following tables, we show the mean and standard error on each metric over 3 runs. 
Revised: We show the mean and standard error on each metric over 3 runs. Since our semantic parsing module is trained on paired question-program pairs, it achieves nearly perfect accuracy ( > 99 . 9% ) on test questions.

Revised: Thus, in later sections, we will focus on evaluating object segmentation, concept grounding, and downstream task performances.

Original: Our LORL helps Slot Attention achieve better segmentation results in all three metrics.
Revised: Our LORL helps Slot Attention achieve better segmentation results in all three metrics.

Original: From visualizations in Fig.
Revised: From visualizations in Fig.

Original: 4, we ﬁnd that the original Slot Attention model struggles with metallic objects; but with LORL, it performs much better in those cases.
Revised: 4, we ﬁnd that the original Slot Attention model struggles with metallic objects; but with LORL, it performs much better in those cases.

Original: To further explore how LORL helps Slot Attention on failure cases, we calculate the Ground Truth
Revised: To further explore how LORL helps Slot Attention on failure cases, we calculate the Ground Truth

Original: Split Ratio for each object category, and ﬁnd that Slot Attention most often fail to segment coffee makers, blenders, and toasters as a whole.
Revised: Split Ratio for each object category, and ﬁnd that Slot Attention most often fail to segment coffee makers, blenders, and toasters as a whole.

Original: These objects have complex sub-parts and their appearance changes quickly when the viewpoint changes.
Revised: These objects have complex sub-parts and their appearance changes quickly when the viewpoint changes.

Original: With the help of language, LORL improves consistently over its ablative variants across all the three metrics we have, reducing the GT split ratio by 50% on average, as shown in Table 1b.
Revised: With the help of language, LORL improves consistently over its ablative variants across all the three metrics we have, reducing the GT split ratio by 50% on average, as shown in Table 1b.

Original: On PartNet-Chairs, our model also helps both MONet and Slot Attention improve with a large margin, as shown in Table 2.
Revised: On PartNet-Chairs, our model also helps both MONet and Slot Attention improve with a large margin, as shown in Table 2.

Original:  MONet in general performs well on this dataset, though it still sometimes merges different parts with the same color into a single mask.
Revised: The results are averaged over 4 runs. MONet in general performs well on this dataset, though it still sometimes merges different parts with the same color into a single mask.

Original: An example can be found in Fig.
Revised: An example can be found in Fig.

Original: 5, column 3, where the blue arm and the blue bottom in the input image are put into the same mask by MONet.
Revised: 5, column 3, where the blue arm and the blue bottom in the input image are put into the same mask by MONet.

Original: Such an issue is alleviated in LORL + MONet.
Revised: Such an issue is alleviated in LORL + MONet.

Original: Fig.
Revised: Fig.

Original: 5 also includes examples to show how LORL helps Slot Attention.
Revised: 5 also includes examples to show how LORL helps Slot Attention.

Original: As shown in Table 2, the improvement on Slot Attention is larger and more consistent, compared with the improvement on MONet.
Revised: As shown in Table 2, the improvement on Slot Attention is larger and more consistent, compared with the improvement on MONet.

Original: We hypothesize that this is because the two models adopt different approaches for aligning object features and masks.
Revised: We hypothesize that this is because the two models adopt different approaches for aligning object features and masks.

Original: While MONet uses separate modules for segmentation and object representation learning, Slot Attention obtains masks by directly decoding  object representations.
Revised: While MONet uses separate modules for segmentation and object representation learning, Slot Attention obtains masks by directly decoding [Table] [Table] object representations.

Original: Having a shared representation might have allowed Slot Attention to gain more from language supervision.
Revised: Having a shared representation might have allowed Slot Attention to gain more from language supervision.

Original: 5. I NSTANCE R ETRIEVAL
Revised: 5. I NSTANCE R ETRIEVAL

Original: We now analyze the learned object representations on Shop-VRB-Simple.
Revised: We now analyze the learned object representations on Shop-VRB-Simple.

Original: We ﬁrst use them for instance retrieval: for each model, we randomly select a segmented object and use its learned[Table] [Table] representation to search for its k nearest neighbors in the feature space.
Revised: We ﬁrst use them for instance retrieval: for each model, we randomly select a segmented object and use its learned representation to search for its k nearest neighbors in the feature space.

Original: Then, for each selected object, we compute how many of the k nearest neighbors belong to the same category.
Revised: Then, for each selected object, we compute how many of the k nearest neighbors belong to the same category.

Original: During searching, we only consider object representations whose corresponding mask, after decoding, has at least an
Revised: During searching, we only consider object representations whose corresponding mask, after decoding, has at least an

Original: Intersection over Union (IoU) of at least 0 .
Revised: Intersection over Union (IoU) of at least 0 .

Original: 75 with a ground truth object mask.
Revised: 75 with a ground truth object mask.

Original: We sample 1, object features from each model for evaluation.
Revised: We sample 1, object features from each model for evaluation.

Original: Table 3 includes results with k = 1 , 3 , 5 , suggesting that the object representations learned by LORL + Slot Attention are better for retrieval, compared with features learned by Slot Attention alone without language.
Revised: Table 3 includes results with k = 1 , 3 , 5 , suggesting that the object representations learned by LORL + Slot Attention are better for retrieval, compared with features learned by Slot Attention alone without language.

Original: This is because Slot Attention often confuses categories that are visually similar but conceptually different, such as baking tray and chopping board .
Revised: This is because Slot Attention often confuses categories that are visually similar but conceptually different, such as baking tray and chopping board .

Original: 5. V ISUAL R EASONING
Revised: 5. V ISUAL R EASONING

Original: As another analysis we also evaluate how LORL + Slot Attention performs on visual question answering on the Shop-VRB-Simple dataset.
Revised: As another analysis we also evaluate how LORL + Slot Attention performs on visual question answering on the Shop-VRB-Simple dataset.

Original: Here we compare with an ablated version of our model, where we only train the model for the ﬁrst two stages, as stated at the end of Section 4.
Revised: Here we compare with an ablated version of our model, where we only train the model for the ﬁrst two stages, as stated at the end of Section 4.

Original: We do not train the model for the third stage—jointly optimizing or ﬁne-tuning all three trainable modules.
Revised: We do not train the model for the third stage—jointly optimizing or ﬁne-tuning all three trainable modules.

Original: We name this ablation LORL + SA (No FT).
Revised: We name this ablation LORL + SA (No FT).

Original: Through this analysis, we hope to understand the importance of joint training of the vision modules (Modules 1 and 2) and the reasoning module (Module 4).
Revised: Through this analysis, we hope to understand the importance of joint training of the vision modules (Modules 1 and 2) and the reasoning module (Module 4).

Original: Table 4 shows that joint training is crucial for visual reasoning.
Revised: Table 4 shows that joint training is crucial for visual reasoning.

Original: This resonates with the previous result, where visually similar objects are clustered together in the latent space, impeding the usefulness of the information encoded.
Revised: This resonates with the previous result, where visually similar objects are clustered together in the latent space, impeding the usefulness of the information encoded.

Original: 5. R EFERENTIAL E XPRESSION I NTERPRETATION
Revised: 5. R EFERRING E XPRESSION C OMPREHENSION

Original: Finally we evaluate the representations learned by LORL on referential expression interpretation, where given an expression referring to a set of objects in the scene, like “The white plates”, the model is expected to return all of the corresponding object masks.
Revised: Finally we evaluate the representations learned by LORL on referring expression comprehension, where given an expression referring to a set of objects in the scene, like “The white plates”, the model is expected to return all of the corresponding object masks.

Original: After learning all needed concepts from question-answer pairs, LORL can naturally handle referential expression without any further training, if we assume a pre-trained semantic parser.
Revised: After learning all needed concepts from question-answer pairs, LORL can naturally handle referring expression without any further training, if we assume a pre-trained semantic parser.

Original: We choose the IEP-Ref model (Liu et al., 2019) as our baseline.
Revised: We choose the IEP-Ref model (Liu et al., 2019) as our baseline.

Original: It uses a module approach and receives direct segmentation supervision.
Revised: It uses a module approach and receives direct segmentation supervision.

Original: On Shop-VRB, we adapt the code provided by Liu et al.
Revised: On Shop-VRB, we adapt the code provided by Liu et al.

Original: (2019) to generate 17K training examples, only for IEP-Ref, and 1.7K testing referential expressions for evaluating both LORL and IEP-Ref.
Revised: (2019) to generate 17K training examples, only for IEP-Ref, and 1.7K testing referring expressions for evaluating both LORL and IEP-Ref.

Original: Measured in Recall@0.5 (the ratio of the recalled objects based on an IoU threshold of 0.5), IEP-Ref performs better than LORL, but the margin is small (90.1% vs .
Revised: Measured in Recall@0.5 (the ratio of the recalled objects based on an IoU threshold of 0.5), IEP-Ref performs better than LORL, but the margin is small (90.1% vs .

Original: Note that while IEP-Ref has been trained on 17K training examples with ground truth object segmentations, LORL does not require any training data on referential expression interpretation.
Revised: Note that while IEP-Ref has been trained on 17K training examples with ground truth object segmentations, LORL does not require any training data on referring expression comprehension.

Original: The relatively comparable results are strong evidence that the representations learned by LORL also transfer to a new task.
Revised: The relatively comparable results are strong evidence that the representations learned by LORL also transfer to a new task.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed Language-mediated, Object-centric Representation Learning (LORL), aprincipled framework for learning object-centric representations from vision and language.
Revised: We have proposed Language-mediated, Object-centric Representation Learning (LORL), a framework for learning object-centric representations from vision and language.

Original: Experiments on Shop-VRB-Simple and PartNet-Chairs show that language signiﬁcantly contributes to learning better representations.
Revised: Experiments on ShopVRB-Simple and PartNet-Chairs show that language signiﬁcantly contributes to learning better representations.

Original: This behavior is consistent across two unsupervised image segmentation models.
Revised: This behavior is consistent across two unsupervised image segmentation models.

Original: Through systematic studies, we have also shown how LORL learns object representations that encode conceptual information, useful for downstream tasks such as retrieval, visual reasoning, and referential expression interpretation.
Revised: Through systematic studies, we have also shown how LORL learns object representations that encode conceptual information, useful for downstream tasks such as retrieval, visual reasoning, and referring expression comprehension.

Original: A D OMAIN -S PECIFIC L ANGUAGE (DSL)
Revised: A D OMAIN -S PECIFIC L ANGUAGE (DSL)

Original: LORL extends the domain-speciﬁc language of the CLEVR dataset (Johnson et al., 2017) to accommodate descriptive sentences.
Revised: LORL extends the domain-speciﬁc language of the CLEVR dataset (Johnson et al., 2017a) to accommodate descriptive sentences.

Original: Speciﬁcally, we add an extra primitive operation:
Revised: Speciﬁcally, we add an extra primitive operation:

Original: Equal(X, y) .
Revised: Equal(X, y) .

Original: It takes two inputs.
Revised: It takes two inputs.

Original: In our case, the ﬁrst argument
Revised: In our case, the ﬁrst argument

Original: X is the output of a Query , Exist , or Count operation.
Revised: X is the output of a Query , Exist , or Count operation.

Original: All three operations output a distribution over possible answers.
Revised: All three operations output a distribution over possible answers.

Original: The second argument y is either a word or number, such as TRUE , white , or 4.
Revised: The second argument y is either a word or number, such as TRUE , white , or 4.

Original: The Equal operation computes the probability of X=y .
Revised: The Equal operation computes the probability of X=y .

Original: To learn language-mediated, object-centric representations, we train our model LORL to maximize the output probability.
Revised: To learn language-mediated, object-centric representations, we train our model LORL to maximize the output probability.

Original: B H YPERPARAMETERS
Revised: B H YPERPARAMETERS

Original: For optimization hyperparameters, we largely adopt original settings in Burgess et al.
Revised: For optimization hyperparameters, we largely adopt original settings in Burgess et al.

Original: and
Revised: and

Original: Locatello et al.
Revised: Locatello et al.

Original: Table 5 summarizes the hyperparameters for the loss weights ( α and β ), the number of training epochs of different stages ( N 1 , N 2 , N 3 ), and the batch size.
Revised: Table 5 summarizes the hyperparameters for the loss weights ( α and β ), the number of training epochs of different stages ( N 1 , N 2 , N 3 ), and the batch size.

Original: We early-stop the training when QA performance converges.
Revised: We early-stop the training when QA performance converges.

Revised: We were skipping the second training phase on PartNetChairs because the ﬁrst training phase (vision-only) yields very poor segmentation performance on this dataset.

Revised: Establishing a meaningful grounding of concepts could be hard in this case.

Revised: If we keep the second training phase for LORL + Slot Attention on PartNet-Chairs, the model converges slower in the third training phase (15 more epochs in our experiments), but the ﬁnal performance remains the same.

Original: [Table]
Revised: [Table]

Original: Learning rate scheduling For Slot Attention models, during the ﬁrst training stage (perceptiononly), we use the learning rate schedule described in the original paper on both datasets.
Revised: Learning rate scheduling For Slot Attention models, during the ﬁrst training stage (perceptiononly), we use the learning rate schedule described in the original paper on both datasets.

Original: Speciﬁcally, the initial learning rate is 4 × 10 − 4 , and is ﬁxed 10K iterations.
Revised: Speciﬁcally, the initial learning rate is 4 × 10 − 4 , and is ﬁxed 10K iterations.

Original: After that, we decay the learning rate by 0 .
Revised: After that, we decay the learning rate by 0 .

Original: 5 for every 100K iterations.
Revised: 5 for every 100K iterations.

Original: On PartNet-Chairs, after the ﬁrst stage, Slot Attention models continue to use the same learning rate scheduling.
Revised: On PartNet-Chairs, after the ﬁrst stage, Slot Attention models continue to use the same learning rate scheduling.

Original: For Shop-VRB-Simple, we switch to a ﬁxed learning rate of 0 .
Revised: For Shop-VRB-Simple, we switch to a ﬁxed learning rate of 0 .

Original: 001 during N 2 phase, which takes 20 epochs.
Revised: 001 during N 2 phase, which takes 20 epochs.

Original: After 20 epochs, we decrease the learning rate to 2 × 10 − 4 .
Revised: After 20 epochs, we decrease the learning rate to 2 × 10 − 4 .

Original: We further decrease the learning rate to 2 × 10 − 5 after another 65 epochs.
Revised: We further decrease the learning rate to 2 × 10 − 5 after another 65 epochs.

Original: We use the Adam optimizer (Kingma & Ba, 2015) for Slot Attention models.
Revised: We use the Adam optimizer (Kingma & Ba, 2015) for Slot Attention models.

Original: For MONet models, we use RMSProp with a learning rate of 0 .
Revised: For MONet models, we use RMSProp with a learning rate of 0 .

Original: 01 during the ﬁrst stage, and use 0 .
Revised: 01 during the ﬁrst stage, and use 0 .

Original: 001 for the second and the third stage.
Revised: 001 for the second and the third stage.

Original: Meanwhile, we also follow NS-CL (Mao et al., 2019) to use curriculum learning.
Revised: Meanwhile, we also follow NS-CL (Mao et al., 2019) to use curriculum learning.

Original: Speciﬁcally, in the second training stage, we limit the number of objects in the scene to be 3.
Revised: Speciﬁcally, in the second training stage, we limit the number of objects in the scene to be 3.

Original: In the third training stage, we gradually increase the number of objects in the scene and the complexity of the questions.
Revised: In the third training stage, we gradually increase the number of objects in the scene and the complexity of the questions.

Original: C P ART N ET -C HAIRS D ATASET
Revised: C P ART N ET -C HAIRS D ATASET

Original: Table 6 illustrates all templates that we use to generate descriptive sentences on the PartNet-Chairs dataset.
Revised: Table 6 illustrates all templates that we use to generate descriptive sentences on the PartNet-Chairs dataset.

Original: We also show the corresponding latent programs for each template.
Revised: We also show the corresponding latent programs for each template.

Original: [Table]
Revised: In PartNet-Chairs, all programs for descriptive sentences end with an Equal operator, which evaluates to true iff. both of its arguments are equal to each other. We train our model with paired and , by to argument.

Revised: Template

Revised: Example Sentence

Original: Example Program
Revised: Example Program

Original: There is/are <number> <color>?
Revised: There is/are <number> <color>?

Original: ( <part name> /part).
Revised: ( <part name> /part).

Original: There are 4 legs.
Revised: There are 4 legs.

Original: Scene, Filter[Leg],
Revised: Scene, Filter[Leg],

Original: Count, Equal[4]
Revised: Count, Equal[4]

Original: The number of <color> ?
Revised: The number of <color> ?

Original: ( <part name> /part)
Revised: ( <part name> /part)

Original: there is <number> .
Revised: there is <number> .

Original: The number of legs there is 4.
Revised: The number of legs there is 4.

Original: Scene, Filter[Leg],
Revised: Scene, Filter[Leg],

Original: Count, Equal[4]
Revised: Count, Equal[4]

Original: There is a <color>?
Revised: There is a <color>?

Original: ( <part name> /part).
Revised: ( <part name> /part).

Original: There is a brown seat.
Revised: There is a brown seat.

Original: Scene, Filter[Brown,
Revised: Scene, Filter[Brown,

Original: Seat], Exist,
Revised: Seat], Exist,

Original: Equal[True]
Revised: Equal[True]

Original: There is no <color>?
Revised: There is no <color>?

Original: ( <part name> /part).
Revised: ( <part name> /part).

Original: There is no brown seat.
Revised: There is no brown seat.

Original: Scene, Filter[Brown,
Revised: Scene, Filter[Brown,

Original: Seat], Exist,
Revised: Seat], Exist,

Original: Equal[False]
Revised: Equal[False]

Original: A <color>?
Revised: A <color>?

Original: ( <part name> /part) is visible.
Revised: ( <part name> /part) is visible.

Original: A brown seat is visible.
Revised: A brown seat is visible.

Original: Scene, Filter[Brown,
Revised: Scene, Filter[Brown,

Original: Seat], Exist,
Revised: Seat], Exist,

Original: Equal[True]
Revised: Equal[True]

Original: No <color>?
Revised: No <color>?

Original: ( <part name> /part) is visible.
Revised: ( <part name> /part) is visible.

Original: No brown seat is visible
Revised: No brown seat is visible

Original: Scene, Filter[Brown,
Revised: Scene, Filter[Brown,

Original: Seat], Exist,
Revised: Seat], Exist,

Original: Equal[False]
Revised: Equal[False]

Original: The name of the <color> part is <part name> .
Revised: The name of the <color> part is <part name> .

Original: The name of the yellow part is arm.
Revised: The name of the yellow part is arm.

Original: Scene,
Revised: Scene,

Original: Filter[Yellow],
Revised: Filter[Yellow],

Original: Query[PartName,
Revised: Query[PartName,

Original: Equal[Arm]
Revised: Equal[Arm]

Original: The <color> part is called <part name> .
Revised: The <color> part is called <part name> .

Original: The yellow part is called arm
Revised: The yellow part is called arm

Original: Scene,
Revised: Scene,

Original: Filter[Yellow],
Revised: Filter[Yellow],

Original: Query[PartName,
Revised: Query[Part Name,

Original: Equal[Arm]
Revised: Equal[Arm]

Original: There is a <color> part; its name is <part name> .
Revised: There is a <color> part; its name is <part name> .

Original: There is a yellow part; its name is arm.
Revised: There is a yellow part; its name is arm.

Original: Scene,
Revised: Scene,

Original: Filter[Yellow],
Revised: Filter[Yellow],

Original: Query[PartName,
Revised: Query[PartName,

Original: Equal[Arm]
Revised: Equal[Arm]

Original: There is a <color> thing called <part name> .
Revised: There is a <color> thing called <part name> .

Original: There is a yellow part called arm.
Revised: There is a yellow part called arm.

Original: Scene,
Revised: Scene,

Original: Filter[Yellow],
Revised: Filter[Yellow],

Original: Query[PartName,
Revised: Query[PartName,

Original: Equal[Arm]
Revised: Equal[Arm]

Original: The color of the <part name> is <color> .
Revised: The color of the <part name> is <color> .

Original: The color of the back is purple.
Revised: The color of the back is purple.

Original: Scene, Filter[Back],
Revised: Scene, Filter[Back],

Original: Query[Color],
Revised: Query[Color],

Original: Equal[Purple]
Revised: Equal[Purple]

Revised: Template

Revised: Example Sentence

Original:  The <part name> has <color> color.
Revised: Latent Program The <part name> has <color> color.

Original: The back has purple color.
Revised: The back has purple color.

Original: Scene, Filter[Back],
Revised: Scene, Filter[Back],

Original: Query[Color],
Revised: Query[Color],

Original: Equal[Purple]
Revised: Equal[Purple]

Original: Template

Original: Example Sentence

Original: Latent Program

Original: There is a <part name> ; its color is <color> .
Revised: There is a <part name> ; its color is <color> .

Original: There is a back; its color is purple.
Revised: There is a back; its color is purple.

Original: Scene, Filter[Back],
Revised: Scene, Filter[Back],

Original: Query[Color],
Revised: Query[Color],

Original: Equal[Purple]
Revised: Equal[Purple]

Original: The <part name> is <color> .
Revised: The <part name> is <color> .

Original: The back is purple.
Revised: The back is purple.

Original: Scene, Filter[Back],
Revised: Scene, Filter[Back],

Original: Query[Color],
Revised: Query[Color],

Original: Equal[Purple]
Revised: Equal[Purple]

Original: There is/are <number> <color>?
Revised: There is/are <number> <color>?

Original: ( <part name> /part) <
Revised: ( <part name> /part) <

Original: relation> the <color>?
Revised: relation> the <color>?

Original: ( <part name> /part).
Revised: ( <part name> /part).

Original: There is a green seat and a yellow leg is behind it.
Revised: There is a green seat and a yellow leg is behind it.

Original: Scene, Filter[Green,
Revised: Scene, Filter[Green,

Original: Seat], Unique,
Revised: Seat], Unique,

Original: Relate[Behind],
Revised: Relate[Behind],

Original: Filter[Yellow,Leg],
Revised: Filter[Yellow,Leg],

Original: Exist,Equal[True]
Revised: Exist,Equal[True]

Original: There is/are <number> <color>?
Revised: There is/are <number> <color>?

Original: ( <part name> /part) <
Revised: ( <part name> /part) <

Original: relation> the <color>?
Revised: relation> the <color>?

Original: ( <part name> /part).
Revised: ( <part name> /part).

Original: There is a green seat and no yellow leg is behind it.
Revised: There is a green seat and no yellow leg is behind it.

Original: Scene, Filter[Green,
Revised: Scene, Filter[Green,

Original: Seat], Unique,
Revised: Seat], Unique,

Original: Relate[Behind],
Revised: Relate[Behind],

Original: Filter[Yellow,Leg],
Revised: Filter[Yellow,Leg],

Original: Exist,Equal[False] <number> <color>?
Revised: Exist,Equal[False] <number> <color>?

Original: ( <part name> /part) is/are <relation> the <color>?
Revised: ( <part name> /part) is/are <relation> the <color>?

Original: ( <part name> /part).
Revised: ( <part name> /part).

Original: 0 leg is behind the brown leg.
Revised: 0 leg is behind the brown leg.

Original: Scene, Filter[Brown,
Revised: Scene, Filter[Brown,

Original: Leg], Unqiue,
Revised: Leg], Unqiue,

Original: Relate[behind],
Revised: Relate[behind],

Original: Filter[Leg],Count,
Revised: Filter[Leg],Count,

Original: Equal[0]
Revised: Equal[0]

Original: There is a <color>?
Revised: There is a <color>?

Original: ( <part name> /part); <number> <color> <part name> is/are <relation> it.
Revised: ( <part name> /part); <number> <color> <part name> is/are <relation> it.

Original: There is a brown leg; 0 leg is behind it.
Revised: There is a brown leg; 0 leg is behind it.

Original: Scene, Filter[Brown,
Revised: Scene, Filter[Brown,

Original: Leg], Unqiue,
Revised: Leg], Unqiue,

Original: Relate[behind],
Revised: Relate[behind],

Original: Filter[Leg],Count,
Revised: Filter[Leg],Count,

Original: Equal[0]
Revised: Equal[0]

Original: The number of <color>?
Revised: The number of <color>?

Original: ( <part name> /part)
Revised: ( <part name> /part)

Original: that is/are <relation> <color>?
Revised: that is/are <relation> <color>?

Original: ( <part name> /part)is <number> .
Revised: ( <part name> /part)is <number> .

Original: The number of leg that is behind the brown leg is 0.
Revised: The number of leg that is behind the brown leg is 0.

Original: Scene, Filter[Brown,
Revised: Scene, Filter[Brown,

Original: Leg], Unqiue,
Revised: Leg], Unqiue,

Original: Relate[behind],
Revised: Relate[behind],

Original: Filter[Leg], Count,
Revised: Filter[Leg],Count,

Original: Equal[0]
Revised: Equal[0]

Original: There is a <color>?
Revised: There is a <color>?

Original: ( <part name> /part) <
Revised: ( <part name> /part) <

Original: relation> the <color>?
Revised: relation> the <color>?

Original: ( <part name> /part).
Revised: ( <part name> /part).

Original: There is a blue back behind the seat.
Revised: There is a blue back behind the seat.

Original: Scene, Filter[Seat],
Revised: Scene, Filter[Seat],

Original: Unqiue,
Revised: Unqiue,

Original: Filter[Blue, Back],
Revised: Filter[Blue, Back],

Original: Relate[behind],
Revised: Relate[behind],

Original: Exist,Equal[True] 
Revised: Exist,Equal[True] Template

Revised: Example Sentence

Original:  There is no <color>?<part name> <relation> the <color>?
Revised: Latent Program There is no <color>?<part name> <relation> the <color>?

Original: ( <part name> /part).
Revised: ( <part name> /part).

Original: There is no blue back to the right of the seat.
Revised: There is no blue back to the right of the seat.

Original: Scene, Filter[Seat],
Revised: Scene, Filter[Seat],

Original: Unqiue,
Revised: Unqiue,

Original: Relate[right],
Revised: Relate[right],

Original: Filter[Blue, Back],
Revised: Filter[Blue,Back],

Original: Exist, Equal[False]
Revised: Exist,Equal[False]

Original: Template

Original: Example Sentence

Original: Latent Program

Original: There is a <color>?
Revised: There is a <color>?

Original: ( <part name> /part) and a <color>?
Revised: ( <part name> /part) and a <color>?

Original: ( <part name> /part) is <relation> it.
Revised: ( <part name> /part) is <relation> it.

Original: There is a seat and a blue back is to the right of it.
Revised: There is a seat and a blue back is to the right of it.

Original: Scene, Filter[Seat],
Revised: Scene, Filter[Seat],

Original: Unqiue,
Revised: Unqiue,

Original: Relate[Right],
Revised: Relate[Right],

Original: Filter[Blue,Back],
Revised: Filter[Blue,Back],

Original: Exist,Equal[True]
Revised: Exist,Equal[True]

Original: There is a <color>?
Revised: There is a <color>?

Original: ( <part name> /part); there is no <color>?
Revised: ( <part name> /part); there is no <color>?

Original: <part name> <relation> it.
Revised: <part name> <relation> it.

Original: There is a green seat; there is no blue back to the right of it.
Revised: There is a green seat; there is no blue back to the right of it.

Original: Scene, Filter[Seat],
Revised: Scene, Filter[Seat],

Original: Unqiue,
Revised: Unqiue,

Original: Relate[Right],
Revised: Relate[Right],

Original: Filter[Blue,Back],
Revised: Filter[Blue,Back],

Original: Exist,Equal[False]
Revised: Exist,Equal[False]

Original: The color of the <part name> (that is)?
Revised: The color of the <part name> (that is)?

Original: <relation> the <color>?
Revised: <relation> the <color>?

Original: ( <part name> /part) is <color> .
Revised: ( <part name> /part) is <color> .

Original: The color of the leg left of the red arm is blue.
Revised: The color of the leg left of the red arm is blue.

Original: Scene, Filter[Red,
Revised: Scene, Filter[Red,

Original: Arm], Unique,
Revised: Arm], Unique,

Original: Relate[Left],
Revised: Relate[Left],

Original: Filter[Leg],
Revised: Filter[Leg],

Original: Query[Color],
Revised: Query[Color],

Original: Relate[behind],
Revised: Relate[behind],

Original: Equal[Blue]
Revised: Equal[Blue]

Original: There is a <part name> <relation> the <color>?
Revised: There is a <part name> <relation> the <color>?

Original: ( <part name> /part); its color is <color>
Revised: ( <part name> /part); its color is <color>

Original: There is a leg to the left of the red arm; its color is blue.
Revised: There is a leg to the left of the red arm; its color is blue.

Original: Scene, Filter[Red,
Revised: Scene, Filter[Red,

Original: Arm], Unique,
Revised: Arm], Unique,

Original: Relate[Left],
Revised: Relate[Left],

Original: Filter[Leg],
Revised: Filter[Leg],

Original: Query[Color],
Revised: Query[Color],

Original: Equal[Blue]
Revised: Equal[Blue]

Original: The name of the <color>?
Revised: The name of the <color>?

Original: ( <part name> /part)
Revised: ( <part name> /part)

Original: (that is)?
Revised: (that is)?

Original: <relation> the <color>?
Revised: <relation> the <color>?

Original: ( <part name> /part) is <part name> .
Revised: ( <part name> /part) is <part name> .

Original: The name of the cyan part that is in front of the yellow seat is leg.
Revised: The name of the cyan part that is in front of the yellow seat is leg.

Original: Scene,
Revised: Scene,

Original: Filter[Yellow,
Revised: Filter[Yellow,

Original: Seat],Unique,
Revised: Seat],Unique,

Original: Relate[Front],
Revised: Relate[Front],

Original: Filter[Cyan],
Revised: Filter[Cyan],

Original: Query[PartName,
Revised: Query[Part Name,

Original: Equal[Leg]
Revised: Equal[Leg]

Original: The <color>?
Revised: The <color>?

Original: ( <part name> /part)
Revised: ( <part name> /part)

Original: (that is)?
Revised: (that is)?

Original: <relation> the <color>?
Revised: <relation> the <color>?

Original: ( <part name> /part) is called <part name> .
Revised: ( <part name> /part) is called <part name> .

Original: The cyan part that is in front of the yellow seat is called leg.
Revised: The cyan part that is in front of the yellow seat is called leg.

Original: Scene,
Revised: Scene,

Original: Filter[Yellow,
Revised: Filter[Yellow,

Original: Seat], Unique,
Revised: Seat],Unique,

Original: Relate[Front],
Revised: Relate[Front],

Original: Filter[Cyan],
Revised: Filter[Cyan],

Original: Query[PartName,
Revised: Query[PartName,

Original: Equal[Leg]
Revised: Equal[Leg]

Revised: D I MPLEMENTATION D ETAILS

Revised: In this section, we provide additional implementation details of our experimental setups and metrics.

Revised: [Figure]

Revised: [Figure]

Revised: GT and Pred split ratios.

Revised: In this paper, we have introduced two new metrics for evaluating the performance of unsupervised object segmentation: namely, the GT split ratio and and the Pred split ratio.

Revised: A simple example of how we can compute GT/

Revised: Pred split ratios is shown in Fig.

Revised: At a high level, the GT split ratio computes the percentage of objects that are split into multiple parts in model segmentation.

Revised: Meanwhile, Pred split ratio computes the percentage of objects that are merged into a single object in model segmentation.

Revised: We introduce these two new metrics because the ARI score is evaluated at the pixel level and does not account for the variance of object sizes.

Revised: By contrast, GT split and Pred split metrics are computed at the object level.

Revised: This difference is illustrated in Fig.

Revised: For concrete examples, in the Fig.

Revised: 6 (a.1), two objects, the chopping board and the thermos, are wrongly segmented.

Revised: In Fig.

Revised: 6 (a.2), only one object mis-segmented.

Revised: However, the ARI score of the ﬁrst image is much higher because the coffee maker has a large size.

Revised: GT split ratio is evaluated on the object level and thus favor the second one.

Revised: Similarly, in the Fig.

Revised: 6 (b.1) , the four legs are merged into two masks, while in Fig.

Revised: 6 (b.2), the seat and the back of the chair are merged into a single object.

Revised: However, the ﬁrst segmentation result has a signiﬁcantly higher ARI score because the chair legs only contribute to a small area in the image.

Revised: In this paper, we propose to jointly use ARI scores and the proposed GT/Pred split ratio to evaluate segmentation masks.

Revised: Throughout the paper, we have been using IoU=0.2 as the threshold while computing the GT/Pred split ratios.

Revised: Table 7 summarizes the results with different IoU thresholds.

Revised: Our model consistently outperforms the baseline.

Revised: Referring expression comprehension.

Revised: In this experiment, the data is generated using the code adapted from Liu et al.

Revised: It contains two types of expressions, the ﬁrst one directly refers to an

Revised: [Table]

Revised: [Table]

Revised: object by its properties: for example, “the white plate”.

Revised: The second type of sentences refers to the object by relating it with another object: for example, “the object that is in front of the mug.”

Revised: The output of the model is the masks of all referred objects.

Revised: The dataset is composed of the same set of concepts and same DSL as in the Shop-VRB-Simple.

Revised: We use the IEP-Ref model proposed in Liu et al.

Revised: as our baseline.

Revised: It is adapted from its prior work IEP (Johnson et al., 2017b).

Revised: IEP-Ref ﬁrst translates the referring expression into a sequence of primitive operations, which are implemented as different modular networks.

Revised: The model takes the image feature extracted by a CNN as input and executes the program by chaining these component networks.

Revised: It outputs a segmentation mask on objects being referred to.

Revised: During training, groundtruth segmentation masks are needed.

Revised: For all methods, including ours and the baseline, we assume a pretrained semantic parser.

Revised: Since the neuro-symbolic program executor outputs a distribution over all objects indicating whether they are selected, we directly multiply its output with the object segmentation masks to get the ﬁnal output.

Revised: E A DDITIONAL E XPERIMENTS

Revised: The following section presents a collection of ablation studies on different modules of LORL, as well as a few extensions.

Revised: Objectness score.

Revised: To validate the effectiveness of the proposed objectness score module, we hereby compare two models: the original Slot Attention model and the Slot Attention model augmented with the proposed objectness score module.

Revised: Both models are trained using images only (there is no language in the loop), on the Shop-VRB-Simple dataset.

Revised: The Table 8 summarizes the result.

Revised: The objectness module alone does not contribute to the segmentation performance.

Revised: Question type.

Revised: In this section, we investigate how different types of questions affect the performance of our model.

Revised: We use the Shop-VRB-Simple dataset for evaluation.

Revised: There are three types of questions in the dataset, counting (e.g., how many plates are there?)

Revised: , existence (e.g., is there a toaster?), and query (e.g., what is the color of the mug?).

Revised: We train LORL +SA with only one single type of questions (the number of total questions is the same).

Revised: Results are summarized in Table 9.

Revised: In general, training on all three types of questions improves the segmentation accuracy.

Revised: The largest gain comes from the query question.

Revised: Interestingly, the best result is achieved when trained on the original dataset, where the ratio of counting, existence, and query questions is 1:1:7.

Revised: Note that all these models are trained with the same number of questions and thus they are directly comparable with each other.

Revised: [Table]

Revised: [Table]

Revised: Data efﬁciency.

Revised: In addition, we provide another analysis by comparing models trained with different number of question-answer pairs.

Revised: The results are shown in Table 10.

Revised: Adding more language data consistently improves the result.

Revised: All results are based on the LORL +SA model trained on the

Revised: Shop-VRB-Simple dataset.

Revised: Visual reasoning baselines.

Revised: To establish a baseline for visual reasoning tasks, we present the results of two visual reasoning approaches IEP (Johnson et al., 2017b) and NS-CL

Revised: (Mao et al., 2019) for reference on the Shop-VRB-Simple dataset, as shown in Table 11.

Revised: All models are trained with the same set of question-answer pairs.

Revised: Note that NSCL has the access to a pretrained object detection module, while LORL and IEP do not.

Revised: Our model outperforms IEP, which is trained with exactly the same amount of supervision as ours.

Revised: It also achieves a comparable result as NS-CL.

Revised: Integration with SPACE.

Revised: SPACE

Revised: (Lin et al., 2020) is another popular method for unsupervised object-centric representation learning.

Revised: SPACE uses parallel spatial attention to decompose the input scene into a collection of objects, and it is also compatible with the proposed framework LORL.

Revised: We include additional results of LORL +SPACE on the CLEVR dataset.

Revised: Shown in the Table 12,

Revised: LORL +SPACE shows a signiﬁcant advantage over the vanilla SPACE model.

Revised: Additionally, we ﬁnd that SPACE shows poor segmentation results on Shop-VRB-Simple and ParNet-Chairs, no matter whether it is integrated with LORL.

Revised: For example, it frequently segments complex objects into too many fragments on Shop-VRB-Simple.

Revised: We conjecture that this is because SPACE was designed for segmenting objects of similar sizes.

Revised: Baseline using language supervision.

Revised: We also conducted an additional baseline model that uses language supervision in a different way.

Revised: Speciﬁcally, based on the Slot Attention model, we use a

Revised: GRU to directly encode question and answer, and concatenate it with the image feature to obtain the object representation.

Revised: On Shop-VRB-Simple, this model does not show improvement over the Slot

Revised: Attention baseline: ARI=76.4%, GT

Revised: Split=29.2%, Pred Split=13.6%.

Revised: This suggests the effectiveness of our proposed framework.

Revised: Concept quantiﬁcation.

Revised: Although LORL without the objectness score can achieve a comparable result in terms of QA accuracy, objectness score is crucial if we want to evaluate how models discover

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: objects in images.

Revised: Here, we show that, on the Shop-VRB-Simple dataset, our model LORL +SA shows signiﬁcant improvement in recovering a holistic scene representation.

Revised: Speciﬁcally, we extract a scene graph for each scene, where each node corresponds to a detected object.

Revised: We represent each node i as a set of concepts

Revised: C i associated with the object (e.g., { large, brown, wooden, chopping board } ).

Revised: We associate a concept with a detected object if its cosine similarity with the object representation is greater than 0.

Revised: We heuristically remove nodes that are not associated with any concepts (by treating them as “background” objects) or have objectness scores that are smaller than 0.5.

Revised: This results in a scene graph, where each node corresponds to a detected object.

Revised: In the following, we compare it against the groundtruth scene graph.

Revised: For each pair of groundtruth node i and detected node j , we compute the concept IoU score based on their associated concepts C i and C j as:

Revised: [Equation]

Revised: Next, we perform a maximum weight matching between the detected scene graph and the groundtruth scene graph with the Hungarian algorithm.

Revised: We use the IoU score as the weight for every edge and remove edges whose IoU score is smaller than a given threshold.

Revised: Finally, based on the macthing, we can compute the precision and recall of the detected scene graph.

Revised: We show the average precision and recall over the entire test set images in Table 13.

Revised: The results suggest that objectness score signiﬁcantly improves the precision of the extracted concepts.