FALCON: F AST V ISUAL C ONCEPT L EARNING BY I NTEGRATING I MAGES , L INGUISTIC DESCRIPTIONS , AND C ONCEPTUAL R ELATIONS
Ziqi Wang UIUC
MIT BCS, CBMM, CSAIL
A
BSTRACT
We present a meta-learning framework for learning new visual concepts quickly, from just one or a few examples, guided by multiple naturally occurring data streams: simultaneously looking at images, reading sentences that describe the objects in the scene and interpreting supplemental sentences that relate the novel concept with other concepts.
The learned concepts support downstream applications, such as answering questions by reasoning about unseen images.
Our model, namely FALCON, represents individual visual concepts, such as colors and shapes, as axis-aligned boxes in a high-dimensional space (the “ box embedding space ”).
Given an input image and its paired sentence, our model ﬁrst resolves the referential expression in the sentence and associate the novel concept with particular objects in the scene.
Next, our model interprets supplemental sentences to relate the novel concept with other known concepts, such as “ X has property Y ” or “ X is a kind of
Y ”.
Finally, it infers an optimal box embedding for the novel concept that jointly 1) maximizes the likelihood of the observed instances in the image and 2) satisﬁes the relationships between the novel concepts and the known ones.
We demonstrate the effectiveness of our model on both synthetic and real-world datasets
I NTRODUCTION
Humans build a cumulative knowledge repository of visual concepts throughout their lives from a diverse set of inputs: by looking at images, reading sentences that describe the properties of the concept, etc.
Importantly, adding a novel concept to the knowledge repository requires only a small amount of data, such as a few images about the concept and a short textual description (Bloom, 2000;
Swingley, Daniel, 2010; Carey & Bartlett, 1978).
Take Fig.
1 as an example: from just a single image that contains many objects (Fig. 1a), as well as a short descriptive sentence that describes a new concept red : “the object left of the yellow cube is red”, humans can effortlessly ground the novel word “red” with the visual appearance of the object being referred to (Bloom, 2000).
Supplemental sentences such as “red is a kind of color” (Fig. 1b) may provide additional information: “red” objects should be classiﬁed based on their hue.
This further supports us to generalize the learned concept “red” to objects of various shapes, sizes, and materials.
Finally, the acquired concept can be used ﬂexibly in other tasks such as question answering (Fig. 1c).
Our goal is to build machines that can learn concepts that are associated with the physical world in an incremental manner and ﬂexibly use them to answer queries.
To learn a new concept, for example, the word red in Fig.
1a, the system should 1) interpret the semantics of the descriptive sentence composed of other concepts, such as left , yellow , and cube , 2) instantiate a representation for the novel concept with the visual appearance of the referred object, 3) mediate the concept representation based on the supplemental sentences that describe the property of the concept or relate it with other concepts, and 4) use the learned concept ﬂexibly in different scenarios.
A framework that can solve these challenges will allow us to build machines that can better learn from human and communicate with human.
To address these challenges, in this paper, we present a uniﬁed framework, FALCON (FAst Learning of novel visual CONcepts).
FALCON maintains a collection of embedding vectors for individual ∗ Equal contribution.
[Figure]
visual concepts, which naturally grows in an incremental way as it learns more concepts.
A neurosymbolic concept learning and reasoning framework learns new concepts by looking at images and reading paired sentences, and use them to answer incoming queries.
Concretely, FALCON represents individual visual concepts, such as colors and shapes, as axis-aligned boxes in a high-dimensional space (the “ box embedding space ” (Vilnis et al., 2018)), while objects in different images will be embedded into the same latent space as points.
We say object X has property Y if the embedding vector X is inside the embedding box of Y .
Given an input image and its paired sentence, our model ﬁrst resolves the referential expression in the sentence using the previously seen concepts (e.g., left , yellow , and cube ) and associate the novel concept with particular objects in the scene.
Next, our model interprets supplemental sentences to relate the novel concept with other known concepts (e.g., yellow ).
To infer the box embedding of the novel concept, we train a neural network to predict the optimal box embedding for the novel concept that jointly 1) maximizes the data likelihood of the observed examples, and 2) satisﬁes the relationships between the novel concepts and the known ones.
This module is trained with a meta-learning procedure.
Our paper makes the following contributions.
First, we present a uniﬁed neuro-symbolic framework for fast visual concept learning from diverse data streams.
Second, we introduce a new concept embedding prediction module that learns to integrate visual examples and conceptual relations to infer a novel concept embedding.
Finally, we build a protocol for generating meta-learning test cases for evaluating fast visual concept learning, by augmenting existing visual reasoning datasets and knowledge graphs.
By evaluation on both synthetic and natural image datasets, we show that our model learns more accurate representations for novel concepts compared with existing baselines for fast concept learning.
Systematical studies also show that our model can efﬁciently use the supplemental concept descriptions to resolve ambiguities in the visual examples.
We also provide discussions about the design of different modules in our system.
R ELATED W ORKS
Visual concept learning and visual reasoning.
Visual reasoning aims to reason about object properties and their relationships in given images, usually evaluated as the question-answering accuracy (Johnson et al., 2017a; Hudson & Manning, 2018;
Mascharka et al., 2018;
Hu et al., 2018).
Recently, there has been an increasing amount of work has been focusing on using neuro-symbolic frameworks to bridge visual concept learning and visual reasoning (Yi et al., 2018; Mao et al., 2019;
Li et al., 2020).
The high-level idea is to disentangle concept learning: association of linguistic units with visual representations, and reasoning: the ability to count objects or make queries.
Han et al.
(2019) recently shown how jointly learning concepts and metaconcepts can help each other.
Our work is an novel approach towards making use of the metaconcepts in a meta-learning setting aiming at boost the learning of novel concepts based on known ones.
Few-shot visual learning.
Recent work has studied learning to classify visual scene with very limited labeled examples (Vinyals et al., 2016;
Sung et al., 2018;
Snell et al., 2017) or even without any example (Wang et al., 2018;
Kampffmeyer et al., 2019; Tian et al., 2020).
For few-shot learning, existing work proposes to compare the similarity, such as cosine similarity (Vinyals et al., 2016) and
Euclidean distance (Snell et al., 2017), between examples, while (Sung et al., 2018) introduces a learnable module to predict such similarities.
In addition, (Gidaris & Komodakis, 2018) learns a weight generator to predict the classiﬁer for classes with very few examples.
(Ravi & Larochelle,
[Figure]
2017; Finn et al., 2017; Nichol et al., 2018) address this problem by learning the initialization for gradient-based optimization.
(Santoro et al., 2016) used external memory to facilitate while (Munkhdalai & Yu, 2017) for is inspired
, but
Geometric embeddings.
In contrast to representing concepts in vector spaces (Kiros et al., 2014), a geometric embedding framework associates each concept with a geometric entity such as a Gaussian distribution (Vilnis & McCallum, 2015), the intersection of hyperplanes (Vendrov et al., 2016; Vilnis et al., 2018), and a hyperbolic cone (Ganea et al., 2018).
Among them, box embeddings (Vilnis et al., 2018) which map each concept to a hyper-box in the high-dimensional space, have been popular for concept representation: (Li, Xiang and Vilnis, Luke and Zhang, Dongxu and Boratko, Michael and
McCallum, Andrew, 2019) proposed a smooth training objective, and (Ren et al., 2020) uses box embeddings for reasoning over knowledge graphs.
In this paper, we extend the box embedding from knowledge graphs to visual domains, and compare it with other concept embeddings.
FALCON
The proposed model, FALCON, learns visual concepts by simultaneously looking at images, reading sentences that describe the objects in the scene, and interpreting supplemental sentences that describe the properties of the novel concepts.
FALCON learns to learn novel concepts quickly and in a continual manner.
We start with a formal deﬁnition of our fast and continual concept learning task.
Problem formulation.
Each concept learning task is a 4-tuple ( c, X c , D c , T c ) .
Denote c as the novel concept to be learned (e.g., red ).
Models learn to recognize red objects by looking at paired images x i and sentences y i : X c = { ( x i , y i ) } .
Optionally, supplementary sentences D c = { d i } describe the concept c by relating it to other known concepts.
After learning, the model will be tested on downstream tasks.
In this paper, we speciﬁcally focus on visual reasoning: the ability to answer questions about objects in the testing set T c , which is represented as pairs of images and questions.
There are two possible options to approach this problem.
One is manually specifying rules to compute the representation for the new concept.
In this paper, we focus on a meta-learning approach: to build a system that can learn to learn new concept.
Our training data is data tuples for a set of training concepts (base concepts, C base ).
After training, the system is evaluated on a collection of novel concepts ( C test ).
That is, we will provide our system with X c and D c for a novel concept c , and test it on visual reasoning data T c .
Thus, the system works in a continual learning fashion: the description of a new concept depends on a previously learned concept.
Overview.
Fig.
2 gives an overview of our proposed model, FALCON.
Our key idea is to represent each concept as an axis-aligned box in a high-dimensional space (the “box embedding space”,
Section 3.1).
Given example images x i and descriptions y i (Fig. 2a), FALCON interprets the referential expression in y i as a symbolic program (Fig. 2a2).
An neuro-symbolic reasoning module executes the inferred program to locate the object being referred to (Fig. 2a3), see Section 3.2.
Meanwhile, supplementary descriptions D c (Fig. 2b) will be translated into relational representations of concepts (Fig. 2b1), i.e., how the new concept c relates to other known concepts.
Based on the examples of the novel concept and its relationships with other concepts, we formulate the task of novel concept learning as learning to infer the best concept embedding for c in the box embedding space, evaluated on downstream tasks (Fig. 2c).
Once the model learned, we will be using the same neuro-symbolic module for answering questions in the testing set T c .
3. V ISUAL R EPRESENTATIONS AND E MBEDDING S PACES
Given the input image, our model extracts an object-based representation for the scene (Fig. 2a1).
This is done by ﬁrst using a pretrained Mask-RCNN (He et al., 2017) to detect objects in the image.
The mask for each object, paired with the original image is sent to a ResNet-34 (He et al., 2016) model to extract its visual representation, as a feature vector.
For each pair of objects, we will concatenate their feature vectors to form the relational representation between this pair of objects.
The feature vector of each object will be mapped into a geometric embedding space with a fullyconnected layer.
Each object-based concepts (such as object categories, colors) will be mapped into the same embedding space.
Similarly, the relational representation of object pairs, as well as relational concepts (such as spatial relationships) will be mapped into another space.
Here, we focus on the box embedding space (Vilnis et al., 2018) since it naturally models the entailment relationships between concepts.
It is also worth noting that other embedding spaces are also compatible with our framework, and we will empirically compare them in Section 4.
Box embedding.
We slightly extend the box embedding model for knowledge graphs (Vilnis et al., 2018) into visual domains.
Throughout this section, we will be using object-based concepts as an example.
The idea can be easily applied to relational concepts as well.
Each object-based concept is a tuple of two vectors: e c =
(Cen( e c ) , Oﬀ( e c )) , denoting center and offset of the box embedding, both in R d , where d is the embedding dimension.
Intuitively, each concept corresponds to an axis-aligned box (hypercube) in the embedding space, centered at Cen( e c ) and have edge length 2 · Oﬀ( e c ) .
Each object o is also a box: (Cen( e o ) , δ ) , centered at Cen( e o ) with a ﬁxed edge length of δ = 10 − 6 .
We constrain that all boxes should reside within [ − 12 , 12 ] d .
For each box embedding (either an embedding of an object, or a concept), we deﬁne helper functions
Min( e ) =
Cen( e ) − Oﬀ( e ) and Max( e ) =
Cen( e )+Oﬀ( e ) to denote the lower- and upper-bound of the a box e .
The denotational probability is the volume of the box: Pr[ e ] = (cid:81) i (Max( c )
i − Min( c ) i ) , where subscript i denotes the i -th dimension of vectors.
Obvsiouly, Pr[ e ] ∈ [0 , 1] .
Moreover, we deﬁne the joint distribution of two boxes e 1 and e 2 as the volume of their intersection: Pr[ e 1 ∩ e 2 ] = (cid:81) i max (( M ( e 1 , e 2 ) i − m ( e 1 , e 2 ) i ) , 0) , where M ( e 1 , e 2 ) =
Max( e 1 ) ∨ Max( e 2 ) , m ( e 1 , e 2 ) =
Min( e 1 ) ∧ Min( e 2 )) .
∧ and ∨ are element-wise max and min operators.
Finally, we deﬁne the conditional probability
Pr[ e 1 | e 2 ] =
Pr[ e 1 ∩ e 2 ] / Pr[ e 2 ] .
This notation is useful in classifying objects: we will use Pr[ e red | e obj ] as the classiﬁcation conﬁdence of obj being red .
Similarly, it also handles entailment of concepts: Pr[ e c (cid:48)
| e c ] readily denotes the probability that concept c entails concept c (cid:48) .
This well suits our need for modeling entailment relations between concepts: in Fig. 2, the supplemental sentence implies “ white-eyed vireo entails vireo ”. In box embedding, this means: any instance inside the box “white-eyed vireo” should also be inside the box “vireo.” We leave detailed discussions and comparisons with other embedding spaces in Appendix Section C.4.
The formulation above has non-continuous gradient at the edges of the box embeddings, following (Li, Xiang and Vilnis, Luke and Zhang, Dongxu and Boratko, Michael and McCallum, Andrew, 2019), we “smooth” the computation by replacing all max( · , 0) operations with the softplus operator:
softplus( x ) =
τ
log(1 + exp( x/τ )) , where τ is a scalar hyperparameter.
3. N EURO -S YMBOLIC R EASONING
The neuro-symbolic reasoning module processes all linguistic inputs by translating them into a sequence of queries in the concept embedding space, organized by a program layout.
Speciﬁcally, all natural language inputs, including the sentences
y i paired with the image examples, the supplementary sentences d i , and questions in T c will be parsed into symbolic programs, using the same grammar as (Han et al., 2019).
Visually grounded questions including y i and questions in T c will be parsed into programs with hierarchical layouts.
They contain primitive operations such as Filter for locating objects with a speciﬁc color or shape, and Query for query the attribute of a speciﬁed object.
Appendix Section C contains a formal deﬁnition of grammar.
Meanwhile, supplementary sentences d i will be translated as conjunctions of fact tuples: e.g., (red, is, color), which denotes the extra information that the new concept red is a kind of color .
We use pre-trained semantic parsers to translate the natural language inputs into such symbolic representations.
Next, we locate the object being referred to in each ( x i , y i ) pair.
This is done by executing the program on the extracted object representations and concept embeddings.
Speciﬁcally, we apply the neuro-symbolic program executor proposed in (Mao et al., 2019).
It contains deterministic functional modules for each primitive operations.
The execution result is a distribution p j over all objects j in image x
i , interpreted as the probability that the j -th object is being referred to.
The high-level idea of the neuro-symbolic program executor is to smooth the boolean value in deterministic program execution into scores ranging from 0 to 1.
For example, in Fig. 2, the execution result of the ﬁrst Filter[ white-eyed vireo ] operation yields to a vector v , with each entry v i denoting the probability that the i -th object is red.
The scores are computed by computing the probability Pr[ e white − eyed − vireo | e i ] for all object embeddings
e i in the box embedding space.
The output is fully differentiable w.r.t.
the concept embeddings and the object embeddings.
Similarly, the concept embeddings can be used to answer downstream visual reasoning questions.
3. L EARNING TO L EARN C ONCEPTS
The embedding prediction module takes the located objects { o ( c ) i } of the novel concept c in the image, as well as the relationship between c and other concepts as input.
It outputs a box embedding for the new concept e c .
Here, the relationship between the concepts is represented as a collection of 3-tuples ( c, c (cid:48) , rel ) , where c (cid:48) is an known concept and rel denotes the relationship between c and c (cid:48) .
At a high level, the inference process has two steps.
First, we infer N candidate box embeddings based on their relationship with other concepts, as well as the example objects.
Next, we select the concept embedding candidate e c with the highest data likelihood: (cid:81) Mi
=1 Pr[ e c | o ( c ) i ] , where i indexes over M example objects.
The data likelihood is computed by the denotational probabilities.
Following we discuss two candidate models for generating candidate concept embeddings.
Graph neural networks.
A natural way to model relational structures between concepts and example objects is to use graph neural networks (Hamilton et al., 2017).
Speciﬁcally, we represent each concept as a node in the graph, whose node embeddings are initialized as the concept embedding e c .
For each relation tuple ( c, c (cid:48) , rel ) , we connect c and c (cid:48) .
The edge embedding is initialized based on their relationships rel .
We denote the derived graph as G concept (Fig. 2 (b1)).
We also handle example objects in a similar way: for each example object o ( c ) i , we initialize a node with its visual representation, and connect it with c .
We denote the derived graph as G example (Fig. 2 (b2)).
To generate candidate concept embeddings, we draw N initial guesses
e i, 0 c , i = 1 , · · · N i.i.d.
from the a prior distribution p θ .
For each candidate e i, 0 c , we feed it to two graph neural networks sequentially:
[Equation]
Note that GNN 1 and GNN 2 are different GNNs.
We use the output e c as the concept embedding for the novel concept c .
We denote this model as FALCON-G.
Recurrent neural networks.
A variant of our model, FALCON-R, replaces the graph neural network in FALCON-G with recurrent neural networks.
Speciﬁcally, we process each edge in the graph in a sequential order.
We ﬁrst sample e i, 0 c from p ( θ ) and use it to initialize the hidden state of an RNN cell RNN 1 .
Next, for each edge ( c, c (cid:48) , rel ) in G concept , we concatenate the concept embedding of c (cid:48) and the edge embedding for rel as the input.
They are fed into RNN 1 sequentially.
We use the last hidden state of the RNN as e i, 1 c .
We process all edges in G example in the same manner, but with a different RNN cell RNN 2 .
Since all edges in a graph are unordered, we use random orders for all edges in both graphs.
The implementation details for both the GNN and the RNN model can be found in our supplemental material.
Training objective.
We evaluate the inferred e c on downstream tasks.
Speciﬁcally, given an input question and its paired image, we execute the latent program recovered from the question based on the image, and compare it with the groundtruth answer to compute the loss.
Note that since the output of the neuro-symbolic program executor is fully differentiable w.r.t.
the visual representation and concept embeddings, we can use gradient descent to optimize our model.
The overall training objective is comprised of two terms: one for question answering L QA and the other for the prior distribution matching: L meta =
L QA − λ prior log p θ ( e c ) is the ﬁnal loss function.
3. T RAINING AND T ESTING P ARADIGM
The training and testing pipeline for FALCON consists of three stages, namely the pre-training stage, the meta-training stage, and the meta-testing stage.
We process a dataset by splitting all concepts into three groups: C base , C val , and C test , corresponding to the base concepts, validation concepts, and test concepts.
Recall that each concept in the dataset is associated with a 4-tuple ( c, X c , D c , T c ) .
In pre-training stage, our model is trained on all concepts c in C base using T c , to obtain all concept embeddings for c ∈ C base .
This step is the same as the concept learning stage of (Mao et al., 2019), except that we are using a geometric embedding space.
In the meta-training stage, we ﬁx the concept embedding for all c ∈ C base , and train the GNN modules (in FALCON-G) or the RNN modules (in
FALCON-R) with our meta-learning objective L meta .
In this step, we randomly sample task tuples ( c, X c , D c , T c ) from concepts in C base , and use concepts in C val for validation and model selection.
In the meta-testing stage, we evaluate models by its downstream task performance on concepts drawn from C test .
For more details including hyperparameters, please refer to Appendix Section B.
E XPERIMENTS 4. D ATASET
We procedurally generate meta-training and testing examples.
This allows us to have ﬁne-grained control over the dataset distribution.
At a high-level, our idea is to generate synthetic descriptive sentences and questions based on the object-level annotations of images and external knowledge bases such as the bird taxonomy.
We compare FALCON with end-to-end and other concept-based method on two datasets: CLEVR (Johnson et al., 2017a) and CUB (Wah et al., 2011).
CUB Dataset.
The CUB-200-2011 dataset (CUB; Wah et al., 2011) contains 12K images ofcategories of birds.
We split these 200 classes into 100 base, 50 validation and 50 test classes following Hilliard et al.
We expand the concept set by including their hypernyms in the bird taxonomy (Sullivan et al., 2009), resulting in 211 base, 83 validation, and 72 test concepts. We consider the two concept relations in CUB: hypernym and cohypernym (concepts that share the same hypernym).
Examples could be found in Fig.
CLEVR dataset.
We also evaluate our method on the CLEVR dataset (Johnson et al., 2017a).
Each image contains 3D shapes placed on a tabletop with various properties – shapes, colors, types of material and sizes.
These attribute concepts are grouped into 10 base, 2 validation and 3 test concepts.
We create four splits of the CLEVR dataset with different concept split.
We generate the referential expressions following (Liu et al., 2019) and question-answer pairs following (Johnson et al., 2017a).
Examples could be found in Fig.
We consider only one relation between concepts: hypernym .
For example, red is a kind of color .
GQA dataset.
Our last dataset is thee GQA dataset (Hudson & Manning, 2019), which contains image associated with a scene graph annotating the visual concepts in the scene.
We select a subset of 105 most frequent concepts.
These concepts are grouped into 89 base concepts, 6 validation and 10 test concepts.
Examples generated in this dataset can be found in Fig.
We consider only one relation between concepts: hypernym , e.g. standing is a kind of pose .
We consider two kinds of approaches: end-to-end and concept-centric .In end-to-end approaches, example images and other descriptive sentences ( X c and D c ) are treated as an additional input to the system when responding to each test query in T c . In this sense, the system is not “learning” the novel concept c as an internal representation.
Instead, it reasons about images and texts that is related to c and has been seen before, in order to process the queries. Their implementation details can be found in Appendix Section A. [Figure]
LSTM "
MLP 𝑜 "(!) 𝑒 !
AVG 𝑒 !
𝑜 "(!)
GNN !
𝐺 !
GNN " 𝑒 ! 𝑜 "(!)
GNN !
RNN " 𝑜 "(!)
RNN !
𝑒 !
a) NSCL+LSTM b) NSCL+GNN c) FALCON-G d) FALCON-R 𝐺 !
() 𝐺 !
MAC (Hudson & Manning, 2018), is a state-of-the-art end-to-end method for visual reasoning.
It predicts the answer based on dual attention over images and texts.
We also consider a group of concept-centric baselines, clariﬁed in 3.
Like FALCON, they all maintain a set of concept embeddings in their memory.
The same neuro-symbolic program execution model is used to answer questions in T c .
This isolates the inference of novel concept embeddings.
NSCL+LSTM , the ﬁrst baseline in this group uses the same semantic parsing and program execution module to derive the example object embeddings o ( c ) i as FALCON.
Then, it uses an LSTM to encode all o ( c ) i into a single vector, and uses a different LSTM to encode the natural language sentences in
D c into another vector.
These two vectors are used to compute the concept embedding e c .
NSCL+GNN is similar to NSCL+LSTM except that supplemental sentences D c are ﬁrst parsed as conceptual graphs.
Next, we initialize the concept embedding in the graph by taking the “average” of visual examples.
We then use the same GNN module as FALCON on the conceptual graph
G concept to derive e c .
Note that NSCL+GNN only use a single GNN corresponding to G concept , while FALCON-G uses two GNNs–one for concept graphs and the other for example graphs.
We test concept-centric models on two more embedding spaces besides the box embedding.
Hyperplane : object o and concept c correspond to vectors in R d , denoted as e o and e c , respectively.
We deﬁne Pr[ e o | e c ] = σ ( e To e c /τ − γ ) , where σ is the sigmoid function, τ = 0 .
125 , and γ = 2 d .
Hypercone : object o and concept c correspond to vectors in R d , denoted as e o and e c , respectively.
We deﬁne Pr[ e o | e c ] = σ (( (cid:104)
e o , e c (cid:105) − γ ) /τ ) , where (cid:104)· , ·(cid:105) is the cosine similarity function, and τ = 0 .
1 , γ = 0 .
2 are scalar hyperparameters.
This embedding space has been proved useful in concept learning literature (Gidaris & Komodakis, 2018; Mao et al., 2019).
4. R ESULTS ON CUB
We evaluate model performance on downstream question-answering pairs in T c for all test concepts c ∈ C test .
There is only one example image for each concept ( | X c | = 1 ).
Visual reasoning questions on CUB are simple queries about the bird category in this image.
Thus, it has minimal visual reasoning complexity and focuses more on the fast learning of novel concepts.
The results are summarized in Table 1, with qualitative examples in Fig.
In general, our model outperforms all baselines, suggesting the effectiveness of our concept embedding prediction.
FALCON-R with box embeddings achieves the highest performance.
We conjecture that this is because box embeddings can better capture hierarchies and partial order relationships between concepts, such as hypernyms .
NSCL+LSTM and GNN models have inferior performance with box embeddings, probably because their neural modules cannot handle complex geometric structures like box containment.
Unlike them, we use a sample-update-evaluate procedure to predict box embeddings.
By comparing concept-centric models based on the same embedding space (e.g., box embeddings), we see that translating the descriptive sentences D c into concept graphs signiﬁcantly improves the performance.
Thus, NSCL+LSTM has an inferior performance than other methods.
NSCL-GNN and FALCON-G are very similar to each other, except that in NSCL-GNN, the initial embedding of the concept e c is initialized based on the object examples, while in FALCON-G, we use multiple random samples to initialize the embedding and select the best embedding based on the likelihood function.
Moreover, FALCON-G and FALCON-R has similar performance, and it is worth noting that FALCON-R has the advantage that it can perform online learning.
That is, edges in G concept and G example of concept c can be gradually introduced to the system to improve the accuracy of the corresponding concept embedding e c . [Table] [Table] [Figure]
We summarize our key ﬁndings as follows.
First, compared with end-to-end neural baselines (CNN+LSTM, MAC), explicit concept representation (NSCL, FALCON) is helpful (Table 3).
Second, comparing modules for predicting concept embeddings from supplemental sentences D c : LSTMs over texts D c (NSCL+LSTM) v.s. GNNs over concept graphs G concept (NSCL+GNN, FALCON-G), we see that, explicitly representing supplemental sentences as graphs and using GNNs to encode the information yields the larger improvement.
Third, comparing modules for predicting concept embeddings from visual examples o ( c ) i : heuristically taking the average (NSCL+GNN) v.s. GNNs on example graphs (FALCON-G), we see that using example graphs is better.
These ﬁndings will be further supported by results on the other two datasets.
Case study: few-shot learning.
Few-shot learning (i.e., learning new concepts from just a few examples) is a special case of our fast concept learning task, where D c = ∅ , i.e., there is no supplemental sentences to relate the novel concept c with other concepts.
Results for this few-shot learning setting are summarized in Table 2.
Our model FALCON outperforms other baselines with a signiﬁcant margin, showing the effectiveness of our sample-update-select procedure for inferring concept embeddings.
In this setting, NSCL+GNN computes the novel concept embedding solely based on the “average” of visual examples, and thus being a Prototypical Network (Snell et al., 2017).
Note that it cannot work with the box embedding space since the offset of the box cannot be inferred from just one input example.
Extension: continual learning.
Another advantage of concept-centric models is that it supports the cumulative growth of the learned concept set.
That is, newly learned concepts during the “metatesting” stage can serve as supporting concepts in future concept learning.
This is not feasible for end-to-end approaches as they require all concepts referred to in the supplemental sentence in the base concept set C base .
We extend concept-centric approaches to this continual learning setting, where test concepts C test are learned incrementally, and new concepts might relate to another concept in
C test that has been learned before.
In this setup, FALCON-G achieves the highest accuracy (79.67%), outperforming the strongest baseline NSCL+LSTM by 4.46%.
Details are in Appendix Section C.1.
Ablation: the number of base concepts.
We examine how the number of base concepts contributes to the model performance.
We design a new split of the CUB dataset containing 50 training species (130 base concepts), compared to 100 training species (211 base concepts) of the original split.
Our
FALCON-G model’s accuracy drops from 81.33% to 76.32% when given fewer base concepts. This demonstrates that transfering knowledge from concepts already learned is helpful. See Appendix D.1. Ablation: the number of related concepts in supplemental sentences.
We provide an ablation
[Table]
on the effects of the number of related concepts in supplemental sentences.
use supplemental sentences containing 0%(no supp.
sentence), 25%, 50%, 75%, and 100%(the original setting) of all related concepts.
Our model FALCON-G yields an accuracy of 76.37%, 80.20%, 80.78%, 81.20%, 81.33%, respectively.
This result shows that more related concepts included in supplemental sentences lead to higher accuracy.
Details are in Appendix D.2.
We use a similar data generation protocol for the CLEVR dataset as CUB.
While CLEVR contains concepts with simpler appearance variants, it has a more complicated linguistic interface.
More importantly, all concepts in CLEVR are disentangled and thus allow us to produce compositional object appearances.
We will exploit this feature to conduct a systematical study on concept learning from biased data.
Our framework FALCON provides a principled solution to learning from biased data, which has been previously studied in (Geirhos et al., 2019) and (Han et al., 2019).
Our main results are summarized in Table 3.
Fig.
4 provides qualitative examples.
All conceptcentric models use the box embedding space.
We also compare model performance on the “few-shot” learning setting, where no supplemental sentences are provided.
Due to the compositional nature of CLEVR objects, learning novel concepts from just a single example is hard, and almost always ambiguous: the model can choose to classify the examples based on either its shape, or color, or material.
As a result, all models have a relatively poor performance when there are no supplementary sentences specifying the kind of novel concept (color/shape/etc.).
Case study: learning from biased data.
We provide a systematical analysis on concept learning from biased data.
In this setup, each test concept c is associated with another test concept c (cid:48)
(both unseen during training).
We impose the constraint that all example objects of concept c in X c also has concept c (cid:48) .
Thus, c and c (cid:48) is theoretically ambiguous given only image examples.
The ambiguity can be resolved by relating the concept c with other known concepts.
Overall, our model,
FALCON-G achieves the highest accuracy (88.29%) in this setting (MAC: 61.93%, NSCL+GNN 84.83%).
Moreover, our model shows the most signiﬁcant performance improvement after reading supplemental sentences.
Details can be found in Appendix Section C.2.
4. R ESULTS ON GQA
Following a similar data generation protocol as the previous datasets, we generate questions on the
GQA dataset that ask whether a linguistically grounded object in the picture is associated with a novel concept.
Since each GQA picture contains a large number of concepts in different image regions, it provides a more complex linguistic interface than the CUB dataset and more diverse visual appearances than the CLEVR dataset.
We exploit this advantage to test our model’s performance on real-world images and questions in the GQA dataset.
Our main results are shown in Table 11a in the Appendix Section C.3.
Fig.
4 provides qualitative examples.
Overall, our model achieves the highest accuracy (55.96%) in the setting with supplementary sentences (MAC: 54.99%, NSCL+LSTM: 55.41%).
Due to the visual complexity in the GQA dataset, the exact decision boundary of a concept is hard to determine.
Details are in Appendix Section C.3.
C ONCLUSION
We have presented FALCON, a meta-learning framework for fast learning of visual concepts.
Our model learns from multiple naturally occurring data streams: simultaneously looking at images, reading sentences that describe the image, and interpreting supplemental sentences about relationships between concepts.
Our high-level idea is to incorporate neuro-symbolic learning frameworks to interpret natural language and associate visual representations with latent concept embeddings.
We improve the learning efﬁciency by meta-learning concept embedding inference modules that integrates visual and textual information.
The acquired concepts directly transfer to other downstream tasks, evaluated by the visual question answering accuracy in this paper. Acknowledgements.
This work is in part supported by ONR MURI N00014-16-1-2007, the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award CCF-1231216), the MIT Quest for Intelligence, MIT–IBM AI Lab.
Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of our sponsors
A M ODEL D ETAILS A. R EMARKS ON THE OVERALL MODEL STRUCTURE
FALCON follows a modularized, concept-centric design pattern: the system maintains a set of concept embeddings, which naturally grows as it learns new concepts.
We use separate modules for computing novel concept embeddings (Section 3.3) and processing queries (Section 3.2).
These modules directly compute the novel concept embeddings, essentially classiﬁers on objects and their relationships, and use them to answer complex linguistic queries.
Here, all images and texts processed by the system will be “compressed” as these concept embeddings in a latent space.
There are other possible designs, such as storing all training images and texts directly, and re-processing all training data for every single query in T c .
Despite being inefﬁcient, as we will show in experiments, such design also shows inferior performance compared with FALCON.
A. C ONCEPT E MBEDDING
Throughout the paper, We use a hidden dimension of d = 100 for the box embedding space (note that the vector representation for this box embedding space is 200-d since we have both centers and offsets), and d = 512 for the hyperplane and the hypercone embedding space.
For the box embedding space, we set the smoothing hyperparameter to τ = 0 . 2 , following Li, Xiang and Vilnis, Luke and
Zhang, Dongxu and Boratko, Michael and McCallum, Andrew (2019).
A. P RIOR D ISTRIBUTION
In both FALCON-G and FALCON-R, we assume a prior p θ on concept embeddings, from which we sample the initial guess e i, 0 c .
For box embedding space, we model the prior distribution of e c so that its center and offset along all d dimensions can be generated by the same Dirichlet distribution parameterized by θ .
In particular, we sample d independent samples ( x 0 p , x 1 p , x 2 p ) p ∈{ 0 , 1 , ··· d − 1 } from a Dirichlet distribution parameterized by θ =
( θ 0 , θ 1 , θ 2 ) .
Then an initial guess for concept embedding of c is given as Cen( c ) p = 12 ( x 1 p − x 0 p ) , and Oﬀ( c ) p = 12 x 2 p .
For hyperplane and hypercone embedding space, we model the prior as the same Gaussian distribution on all d dimensions, namely e c ∼ N ( 0 , θ 0 I d ) , where I is the identity matrix and θ 0 a scalar parameter.
In both embedding space choices, θ can be learnt from back-propagation from L meta =
L QA − λ prior log p θ ( e c ) , and we set the hyperparameter λ prior = 1 .
A. FALCON-G
In FALCON-G, there two graphical neural networks, GNN 1 and GNN 2 , which operates on the concept graph G concept and example graph G example , respectively.
Both of them are composed by stacking two graphical neural network (GNN) layers.
For convenience, we denote the input embedding of node i at layer (cid:96) ∈ { 0 , 1 } as h ( (cid:96) )
i .
At layer (cid:96) , we ﬁrst generate the message from node j to its neighbour i by:
[Equation]
where k denote the relation between j and i , where MP is a message passing function, which we will formally deﬁne later.
Next, for each node i , we aggregate the message passed to i by computing an element-wise maximum (i.e., max pooling) of all neighbouring j ’s as h ( (cid:96) )
N G ( i ) , where N G ( i ) is the set of all neighboring nodes.
Finally, we update node embeddings for i based on its original embedding h ( (cid:96) )
i and aggregated messages h
( (cid:96) )
N G ( i ) for node i .
Mathematically,
[Equation]
Concretely, we implement the message passing model MP k based on speciﬁc concept embedding spaces.
For box embedding spaces, we ﬁrst split h ( (cid:96) )
j as two vectors, denoted as Cen( h ( (cid:96) ) j ) and
Oﬀ( h ( (cid:96) ) j ) , respectively.
Similarly, we split h ( (cid:96) )
i into Cen( h ( (cid:96) )
i ) and Oﬀ( h ( (cid:96) ) i ) .
For each hidden dimension p ∈ { 0 , 1 , 2 , · · · , d − 1 } , we concatenate Cen p ( h ( (cid:96) )
j ) , Oﬀ p ( h ( (cid:96) ) j ) , Cen p ( h ( (cid:96) ) i ) , and Oﬀ p ( h ( (cid:96) ) i ) as a 4-d vector, and applies a two-layer perceptron (MLP) on it.
This MLP is shared across all hidden dimensions p .
For different binary relations k , we use different MLPs.
In our experiment, the aforementioned MLP has an output dimension of 10 and a hidden dimension of 20.
For the hyperplane and the hypercone embeddings, we simply concatenate h ( (cid:96) )
j and h ( (cid:96) )
i , and send it into a MLP associated with relation k .
The MLP has 1024 hidden units and 512 output units.
The update model UPDATE is implemented in a similar manner.
For the box embedding space, we concatenate the hidden vector associated with each hidden dimension p of h ( (cid:96) )
i and h ( (cid:96) )
N G ( i ) as a 12-d vector.
We send it to a feed-forward neural network with 20 hidden units and 4 output units, which is shared across d dimensions.
If we denote the four output units as o ( (cid:96) ) ip , δ ( (cid:96) ) ip , o (cid:48) ( (cid:96) ) ip , δ (cid:48) ( (cid:96) ) ip , we use them to calculate the new box embedding h ( l +1) i as
[Equation]
where σ is the sigmoid function.
For hyperplane and hypercone embeddings spaces, we implement
UPDATE by concatenate h ( (cid:96) )
i and h ( (cid:96) )
N G ( i ) and send it to a feed-forward neural network withhidden units and 1024 output units.
If we denote its output as o ( (cid:96) ) ip , δ ( (cid:96) ) ip , each having a length of 512, then the update model can be written as
[Equation]
where (cid:12) denotes element-wise multiplication.
A. FALCON-R
In FALCON-R, concepts graph and example graphs are handled jointly in a uniﬁed framework.
For each each object example o ( c ) i associated with c , we treat their relationship as a “concept-example” relation.
Thus, we can combine two graphs into a single graph.
We denote the resulting graph as G ∗ .
We ﬂatten the graph G ∗ by randomly order all relational tuples ( c, c (cid:48) , rel ) in
[Table]
Return all objects in the scene.
Filter
Filter out the set of objects from input objects that have the speciﬁc object-level concept.
Relate
Filter out the set of objects that have the speciﬁc relational concept with as input object.
AERelate
Filter out the set of objects that have the same attribute value as the input object.
[Table]
Return the intersection of two object sets.
[Table]
Return the union of two object sets.
Query
Query the attribute of the input object.
AEQuery
Query if two objects have the same attribute value (e.g., color).
Exist
Query if an object set is empty.
Count
Return the size of an object set.
CountLessThan
Query if the size of the ﬁrst object set is smaller than the size of the second object set.
CountGreaterThan
Query if the size of the ﬁrst object set is greater than the size of the second object set.
CountEqual
Query if the size of the ﬁrst set is the same as the size of the second object set.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Snippet 1: A snippet of one-step concept graph propagation based on the box embedding space.
The code is written in PyTorch Paszke et al.
The code snippet updates the node embedding h[i] by ﬁrst computing the messages passing from node j to node i in the ﬁrst for-loop, aggregating the messages to node i by taking the element-wise max, and ﬁnally updating the node embedding of i .
A. S EMANTIC PARSER
In semantic parsing module, the sentence is translated into programs of primitive operations that have hierarchical layouts.
For CLEVR dataset, the complete list of operation we provided is given in Table 4.
For CUB dataset, since there is only one bird in the image, we only provide operations Scene ,
Filter and Exist listed in 4.
Examples of the output of semantic parsing module are provided in
Table 5.
Our semantic parser follows the general framework of Seq2seq (Sutskever et al., 2014) and our implementation follows NS-VQA (Yi et al., 2018) which has been developed for VQA tasks.
We ﬁrst tokenize the input sentence into a sequence of word ids.
Then, we feed the sequence into a
Seq2Seq model and generate a sequence of ( op i , concept i ) ,
where i = 1 , 2 , 3 , ..., L out where L out is the output sequence length.
op i ’s are operations deﬁned in the DSL, and they form a ”ﬂattened” program based on which we will recover a hierarchical program.
Some operations such as ﬁlter has a concept input, in which case we use the corresponding concept i .
The semantic parser is trained on 10% of all generated data.
A. N EURO -S YMBOLIC P ROGRM E XECUTION
We mostly follow the original design of NSCL as in Mao et al.
Mao et al.
The neuro-symbolic executor executes the program based on both the scene representation and the concept embeddings.
Dataset
Input
Output
CLEVR (Description Sentences)
The red objects that are both in front of the cyan object and behind the cube is a sphere.
Filter(Intersection(
Relate(Filter(Scene(), cyan),front),
Relate(Filter(Scene(), cube),behind) ),red)
CLEVR (Questions)
There is a rubber object to the left of the metal object;
What is its color?
Query(Filter(
Relate(Filter(Scene(), metal),left), rubber),color)
CUB (Description Sentences)
There is a frigatebird.
Scene()
CUB (Questions)
Is there a frigatebird?
Exist(Filter(Scene(), frigatebird))
GQA (Description Sentences)
The clouds are a white object.
Filter(Scene(),clouds)
GQA (Questions)
Is the standing man a red object?
Exist(Filter(Filter(
Filter(Scene(),standing), man),red))
Lying in the core of execution is the representation of object sets, which are immediate results during the execution.
Each set of objects (such as the output of a Filter operation is a vector of length N , where N is the number of objects in the scene.
Each entry is a score ranging from 0 to 1.
We use the same implementation as Mao et al.
(2019), with two modiﬁcations:
First, the probability of classifying object o as concept c , Pr [ e o | e c ] ( ObjClassify in the NSCL paper), is implemented based on different embedding spaces: box, hypercone and hyperplane.
The original NSCL paper uses the hypercone representation.
Second, the probability that two objects having the same attribute (named AEClassify in the NSCL paper) are treated as special kinds of relational concepts: has same color , has same shape , etc.
Thus, the AERelate operation is implemented in the same way as Relate .
A. B ASELINE I MPLEMENTATION
CNN+LSTM In CNN+LSTM, we use a ResNet-34 network with a global average pooling layer to extract the image representation.
Each image feature is a feature vector of dimension 512.
We concatenate the description sentences, supplemental sentences (if any), and test questions with a separator to form the text input.
We use a bi-directional LSTM to encode the text into a 512dimensional vector (we use the output state of the last token).
We use a two-layer LSTM, each with a hidden state of dimension 512.
The image representation of both images (the example image in
X c , and the test image in T c ), together with the text encoding, are sent into a multi-layer perceptron (MLP) to predict the ﬁnal answer.
The MLP has two layers, with a hidden dimension of 512.
MAC The MAC network takes an image representation (encoded by CNNs) and a text representation (encoded by RNNs) as its input, and directly outputs a distribution over possible answers.
Following the original paper Hudson & Manning (2018), we use the ﬁrst four residual blocks of a ResNet-101 He et al.
(2016) to extract image representations, which is a 2D feature map with hidden dimension 512.
The image representations of the example image and the test image an stacked channel-wise into a 2D feature map of hidden dimension 1024.
The Similar to CNN+LSTM, we concatenate the description sentences, supplemental sentences (if any), and test questions with a separator to form the text input.
For all other parts, we use the same setting as the original paper Hudson & Manning (2018), such as the hidden dimension for all units in MAC.
NSCL+LSTM
In NSCL+LSTM, we use the same semantic parsing module and program execution module to derive the example object embeddings o ( c ) i .
These object embeddings are encoded by a bi-directional LSTM into a 512-dimensional vector.
We use a separate bi-directional LSTM to encode the G concept into a 512-dimensional vector.
Embeddings of related concept c (cid:48) , concatenated with an embedding for the relation rel , are fed in to the LSTM one at a time.
Two LSTM both use as output the output state associated with the last input, and have two layers and 512 hidden dimension.
These output of two LSTMs are concatenated and sent to a linear transform to predict the concept embedding e c .
The output of the linear transform e c will be used by the neuro-symbolic program execution module to predict the ﬁnal answer to queries.
NSCL+LSTM is an ablative model of FALCON-R.
It keeps similar designs as FALCON-R except it places visual examples and linguistic guidance on different streams, whereas FALCON-R uses a single recurrent mechanism to capture both the visual and linguistic information.
NSCL+GNN In NSCL+GNN, the supplemental sentence D c is ﬁrst parsed into conceptual graph
G concept .
All nodes corresponding to related concepts c (cid:48) are initialized by the concept embeddings of c (cid:48) .
The node corresponding to concept c is initialized from the “average” of all visual examples.
For both hyperplane and hypercone embedding spaces, the average is computed by taking the mean of (L2-normalized) example object embeddings.
For box embedding spaces, it is computed as the speciﬁc boundary (i.e., the minimal bounding box that covers all examples) of visual example embeddings.
For meta-learning setting without supplmentary sentences, we use the average box size of all concepts in the training set to initialize the box size for novel concepts, so the NSCL+GNN model would become a special example of the Prototypical Network.
The GNN has two layers, and has the same design as our mode FALCON-G. NSCL+GNN is an ablative model of FALCON-G. It keeps all designs of FALCON-G except that it directly computes the “average” of visual examples.
In contrast, in FALCON-G, the example object information are gathered with a example graph neural network encoder.
B T RAINING D ETAILS
Our training paradigm consists of three stages: pre-training, meta-training and meta-testing.
Fig.gives an illustrative overview of these three stages.
Recall that we have processed the dataset by splitting all concepts into three groups: C base , C val , and C test , and each concept in the dataset is associated with a 4-tuple ( c, X c , D c , T c ) .
In pre-training stage, our model is trained on all concepts c in base concept set C base .
The concept embedding of c and the visual representations are trained using question-answer pairs from T c .
This step is the same as the concept learning stage of Mao et al.
(2019), except that we are using a geometric embedding space.
In particular, based on the classiﬁcation conﬁdence scores of object properties and their relations, the model outputs a distribution over possible answers for each recovered program.
On the CLEVR dataset, we use the same curriculum learning setting as Mao et al.
(2019), based on the number of objects in the image and the complexity of the questions.
In the meta-training stage, we ﬁx the concept embedding for all concepts c ∈ C base , and train the GNN modules (in
FALCON-G) or the RNN modules (in FALCON-R) with our meta-learning objective L meta .
In this step, we randomly sample task tuples ( c, X c , D c , T c ) of all concepts in C base , and use concepts in
C val for validation and model selection.
In the meta-testing state, we evaluate each model by its downstream task performance on concepts drawn from C test .
One issue with end-to-end methods (CNN+LSTM and MAC) is that they assume that the concept vocabulary is known.
Upon seeing a new concept, they do not have the mechanism for inducing a new word embedding for this concept c .
To address this issue, in the meta-training and meta-testing stages, the appearance of the novel concept c in all texts will be replaced by a special token <UNK> .
Our model is trained with the following hyperparameters.
During the pre-training stage, the model is trained for 50000 iterations with a batch size of 10, using an Adam optimizer Kingma & Ba (2014) with learning rate 0.0001.
During the meta-training stage, the model is trained for
[Figure]
iterations with a batch size of 1 (one novel concept per iteration).
We train the model with the same optimizer setup (Adam with a learning rate of 0.001).
In both stages, we evaluate the model on their corresponding validation set every 1000 iterations.
We decrease the learning rate by a factor of 0 .
1 when the validation accuracy fails to improve for 5 consecutive validation evaluations,
i
[Figure]
e. the iteration with the best validation accuracy is more than 5 validations ago.
C E XPERIMENT D ETAILS C. T HE CUB D ATASET
In the CUB dataset, descriptive sentences are generated from the referential expressions produced by the templates in Table 6a, containing only concepts from C base .
We provided an approximate of 1.8k descriptive sentences, 1055 of which for C base , 415 for C val and 360 for C test .
Supplemental sentences are generated based on the templates in Table 6b.
In supplemental sentences for concept c ∈ C base , we randomly remove a related concept c (cid:48) from the sentence with a probability of 0.2.
We generate 54k testing questions based on the templates in Table 6c, 31.5k of them paired with concepts in C train , 12.5k of them paired with concepts in C val and 10k of them paired with concepts in C test .
Descriptive sentences, supplemental sentences and testing questions are zipped together to form the input for fast concept learning task.
The same template generates Pre-training questions as testing questions.
We generated around 11.8k question-image pairs for pre-training.
During training, we apply standard data augmentation techniques on the images, including horizontal ﬂip, random resizing and cropping.
Extension: continual learning.
In the continual learning setting, concepts that has been learned in the meta-testing instances can be used as supporting concepts for other concepts.
We implement this idea as the following.
Since CUB is constructed based on a bird taxonomy, we can deﬁne the distance between two concepts by the length of their shortest path on the bird taxonomy.
We deﬁne the hop number of each concept as their distance with the closest concepts c (cid:48) ∈ C base .
In the meta-testing
[Table]
There
[Table]
There
[Table]
white
[Table]
<CONCEPT>
[Table]
the word representing
[Table]
<CONCEPT-1> , <CONCEPT-2> · · · <CONCEPT-N> are a kind of <HYPERNYM> .
[Table]
Black-capped vireo, philadelphia vireo, warbling vireo, white-eyed vireo are a kind of vireos
[Table]
<CONCEPT-i> are words representing c and c (cid:48) that share a cohypernym relation with c . <
HYPERNYM> denotes c ’s hypernym.
[Table]
test question [Table]
[Table]
Is there a/an <CONCEPT> .
Example
Is there a white-eyed vireo.
[Table]
<CONCEPT> is the word representing concept c
[Table]
[Table] [Table]
[Table]
FALCON-G and FALCON-R outperform all baselines.
The table b serves as comparison with table a. FALCON-G and FALCON-R make the best use supplemental sentences to help concept learning.
stage, we test models with concepts of increasing hop numbers.
For example, we ﬁrst teach the model with 1-hop concepts, which directly relates to concepts they have seen in C base .
Next, we add 2-hop concepts.
During the learning of 2-hop concepts, the concept embedding of those 1-hop concepts are ﬁxed.
The detailed result for concept-centric methods for continual concept learning can be found in
Table 7a.
The accuracy is averaged over all 72 test concepts in C test .
For reference, we also show the results of applying these models to the fast concept learning tasks without supplemental sentences in Table 7b (the few-shot learning setting).
This results are computed over all 72 concepts in C test , although we do not use any supplemental sentences.
Our model makes the best use of supplemental sentences in inducing novel concept embeddings.
C. T HE CLEVR D ATASET .
CLEVR dataset contains 15 individual concepts, 8 of them associates with colors, 3 associates with shapes, 2 associated with the materials of objects and 2 associated with the sizes of objects.
Each split is generated by selecting 2 colors as validation concepts C val , and another 2 colors and 1 shape as testing concepts C test .
[Table]
The descriptive
[Table]
template
in CLEVR.
[Table]
There is a <REFEXP> ; it is <CONCEPT> .
Example
There is a small object to the right of the yellow object; it is a cyan object.
[Table]
<REFEXP> is a referential expression generated by CLEVR-Ref+
Liu et al.
<CONCEPT> is the word for concept c .
[Table]
The supplemental
sentence template
in CLEVR.
[Table]
<CONCEPT-1> , <CONCEPT-2> · · · <CONCEPT-N> are <ATTRIBUTE> .
[Table]
[Table]
<CONCEPT
[Table]
i> are words representing c and its related concepts c (cid:48) .
<ATTRIBUTE> is the word for an attribute of a object, one of shape , color , material and size . [Table]
test question generation template in CLEVR
[Table]
Does the small blue thing have the same material as the gray object?
[Table]
are question generated from the original CLEVR paper
Johnson
[Table]
[Table]
On CLEVR, descriptive sentences are
Shown in Table 8a, we ﬁrst generate a referential expression for the target object [Table] and associate it with the novel concept c .
The referential expression contains only concepts from C base .
In total we have generated 7.5k pairs of descriptive sentences and images, 5k of which are for C base , 1k for C val , and another 1.5k for C test .
supplemental sentences sentences are generated based on template in
Table 8b by combining the concept c and its related concepts c (cid:48) .
We generate 225k pairs of testing questions and images based on the templates in Table 8c.
150k of them are associated with concepts in C train , 30k for C val , and another 45k for in C test .
Testing question contains only one occurrence of concept c and possibly many occurrences of base concepts.
An example of the generated texts can be found in Fig.
We generated 402k questions for pre-training.
Learning from biased data.
We supplement a concrete example to illustrate how our model can learn novel visual concepts from biased data.
Shown in Fig.
6, in the ﬁrst task, the learner is trained on a novel concept cylinder .
However, all training examples of cylinders are also purple .
Thus, without any supplemental information about whether the novel concept is a color or a shape, the learning task itself is ambiguous.
Our model is capable of resolve such kind of ambiguity by interpreting supplemental sentences.
We have also included the full experimental result of this setup in Table 9.
Handling non-prototypical concepts.
On CLEVR, we have studied a speciﬁc type of conceptconcept relationship: hypernym .
It is worth noting that there is a difference between the hypernym relations in CUB dataset and hypernym relations in CLEVR dataset.
Speciﬁcally, the hypernym of a bird category is also a bird category.
However, the hypernym of a concept in CLEVR (e.g., color,
[Figure] [Figure]
shape, material, etc.) is no longer concepts that can be visually grounded.
They are non-prototypical concepts because they can not be associated with a prototypical visual instance.
We have studied two different ways to handle such kind of hypernym relation.
They are different in terms of their way of constructing the concept graph G concept , namely Type I and Type II.
Illustrated in Fig.
7, in Type I graph, the novel concept c
= purple is connected with other concepts that share the same hypernym with c .
In type II G concept , the non-prototypical concept color is treated as a node in the concept graph.
All concepts that are “colors”, including c is connected to this new node color .
Our experiment shows that, in the fast concept learning test with continual learning setting, our best model (FALCON-G + Box Embedding) achieves a test accuracy of 88.40% using the type
I concept graph.
With the type II concept graph, it can only reach the accuracy of 71.34%.
Thus, throughout the paper, we will be using concept graphs of type I.
C. T HE GQA DATASET
In the GQA dataset, descriptive sentences are generated from the referential expressions produced by the templates in Table 10a, containing only concepts from C base .
We provided an approximate
[Table]
[Table]
of 3.9k descriptive sentences, 2.3k of which for C base , 600 for C val and 1k for C test .
Supplemental sentences are generated based on the templates in Table 6b.
In supplemental sentences for concept c ∈ C base , we randomly remove a related concept c (cid:48) from the sentence with a probability of 0.2.
We generate 117k testing questions based on the templates in Table 6c, 13.8k of them paired with concepts in C train , 18k of them paired with concepts in C val and 30k of them paired with concepts in
C test .
We generated around 13.4k question-image pairs for pre-training.
We use the object-based features produced by a pretrained Faster-RCNN model Ren et al.
(2015), which is provided in the dataset.
The main result is listed in Table 11a and 11b.
We also compare the model performance on the “few-shot” learning setting, where no supplemental sentences are provided.
Out experiments shows that our model (FALCON-G or FALCON-R with box embedding spaces) outperforms other baselines in both settings.
C. T HE CONNECTIONS BETWEEN THE BOX EMBEDDING AND OUR FORMULATION
We justify the choice of the box embedding space from two aspects.
First, intuitively some supplemental sentences in our setting can be modeled as concept entailment relationships, which can be well modeled by the box embedding space.
Second, empirically, the box embedding space, among three embedding space choices studied in the paper, is the best choice for representing entailment.
Modeling concept entailments (intuition).
In order to learn a new concept c , the model receives supplemental sentences relating c with known concepts c (cid:48) . A speciﬁc type of relationship is the hypernym relation. The fact that c is a hypernym of c (cid:48) implies that “all objects that belong to concept c (cid:48) also belong to concept c ”. In the box embedding space, such relationship can be represented as a constraint that “the box of c is inside the box c (cid:48) ”.
Thus, the geometric structures of the box embedding space naturally ﬁts the concept entailment relations in our data.
There are two things to be noted.
First, there are other non-entailment relations that can not be explicitly written down as a constraint.
For example, the “same kind” relationship between concept red and concept blue : they are both colors.
Second, instead of explicitly imposing these constraints while inferring the concept embedding for c , we use learned models to directly predict the embedding of c -
but we hope neural models to leverage such geometric structures.
Comparison with other box embedding spaces (empirical support).
We have already compared the “box embedding” space with other, commonly-used approaches, such as the “hyperplane space” (i.e., the score is computed by a dot-product), and the “hypercone space” (i.e., the score is computed by cosine-similarity) using downstream task performance measures.
The box embedding space outperforms the other two (Table 1 and 2 of the main paper).
Other than downstream task performances, to further validate our intuition, we conduct further analysis on whether the predicted box embeddings of concepts actually satisfy the entailment relations.
Speciﬁcally, using the CUB dataset, we use trained FALCON-G models based on different embedding spaces to predict the concept embeddings in the validation and the test concept set.
For concepts in the training set, we use the learned concept embeddings in the pre-training stage.
Next, we use all concept embeddings and the underlying conditional probability measure for each space to compute Pr[ c (cid:48) | c ] for all pairs of concepts ( c, c (cid:48) ) .
We denote Pr[ c (cid:48) | c ] as the prediction score for “ c entails c (cid:48) ”
(i.e., the hypernym relation: c (cid:48) is a hypernym of c ).
In particular, we use a family of classiﬁers entails t ( c, c (cid:48) )
=
[Pr c
(cid:48)
| c
> t ]
, where
1 [ · ] is the indicator function and t a threshold parameter.
The conditional probability Pr[ c (cid:48)
| c ] is deﬁned as the following.
Denote e c and e c (cid:48) as the concept embeddings for c and c (cid:48) .
• For the box embedding space, Pr[ c (cid:48) | c ] =
Pr[ e c (cid:48) ∩ e c ] / Pr
[ e c ] , where Pr[ e c (cid:48) ∩ e c ] and Pr[ e c ] have been deﬁned in Section 3.1.
• For the hyperplane embedding space, Pr[ c (cid:48) | c ] = σ ( e Tc e c (cid:48) /τ − γ ) , where σ is the sigmoid function, τ = 0 .
125 , and γ = 2 d are the same set of scalar hyperparameters used for classiﬁes objects.
d is the embedding dimension.
• For the hypercone embedding space, Pr[ c (cid:48) | c ] = σ (( (cid:104) e c (cid:48) , e c (cid:105) − γ ) /τ ) , where (cid:104)· , ·(cid:105) is the cosine similarity function, and τ = 0 .
1 , γ = 0 .
2 are the same set of scalar hyperparameters used for classifying objects.
By varying the threshold t , we compute the AUC-ROC score for FALCON-G based on three different embedding spaces, shown in Table 12.
This results suggest that the predicted concept embeddings from FALCON-G (Box) indeed better capture the hypernym relations between concepts.
Note that the choice of speciﬁc hyperparameters: τ , γ , etc., will not affect the AUC-ROC score.
We also want to highlight that the hyperplane and hypercone embedding spaces do not naturally support a conditional probability measure for two concepts, and thus are not optimized for capturing concept entailments, yielding a lower AUC-ROC score.
[Table]
D A BLATION D ETAILS
This section discusses two ablation studies on how the number of base concepts in pretraining and the effect of the number of related concepts in supplemental sentences. [Table] [Table]
[Table]
D. T HE E FFECT OF THE N UMBER OF B ASE C ONCEPTS
In this section, we will conduct an ablation study on how the number of base concepts can contribute to the model performance.
We design a new split of the CUB dataset derived from 50 training species (130 base concepts), 50 validation species (81 concepts), and 100 test species (155 concepts).
In our original experimental setup, we have used 100 training concepts (211 base concepts).
We perform the same fast concept learning experiments on this new split, using our model FALCONG, a concept-centric baseline NSCL+GNN, and an end-to-end baseline MAC.
All concept-centric models use box embeddings spaces.
Our results are summarized in Table 13.
Since we leverage the relationship between the novel concept and known concepts during the learning of the novel concept, all methods have an accuracy drop when there are fewer base concepts.
This demonstrates that transfering knowledge from concepts already learned is helpful.
Moreover, our model FALCON-G still has the highest accuracy when the number of base concepts is reduced.
D. T HE E FFECT OF T HE N UMBER OF R ELATED C ONCEPTS IN S UPPLEMENTAL S ENTENCES
In this section, we design an ablation study on how the number of supplemental sentences would affect model performance.
Since all information captured in the supplemental sentences are the names of related concepts and their relations with the novel concept, we provide an ablation on the effect of the number of related concepts in supplemental sentences.
In the following experiments, we evaluate several models on the fast concept learning tasks on the
CUB dataset.
For each model, we use supplemental sentences containing 0% (no supp.
sentence), 25%, 50%, 75%, and 100% (the setting described in the main paper) of all related concepts.
Again, we compare our model FALCON-G, a concept-centric baseline NSCL+GNN, and an end-to-end baseline MAC.
All concept-centric models use box embeddings spaces.
The results are summarized in Table 14 and Fig.
We have the following observations.
More related concepts included in the supplemental sentence generally lead to higher accuracy, across all models.
In both ceoncept-centric models (FALCON-G and NSCL+GNN), the most signiﬁcant improvement in test accuracy occurs between 0% and 25%.
This suggests that these models can beneﬁt from even just an incomplete set of related concept information.
Our model, FALCON-G, performs consistently the best across all percentages of related concepts.
E F AILURE M ODE D ETAILS
This section discusses how failure may occur in the detection module, the semantic parser, and the concept learning module, which may or may not result in a wrong answer.
E. F AILURE MODE IN DETECTION MODULE
There are a few circumstances where the detection module from the feature extractor produces an error.
Here we list two cases where the detection module may fail, illustrated in Fig.
In one scenario, the model may fail to detect one object from the scene, as in Fig. 15a.
Such errors usually occur among small and partially occluded objects.
In the latter scenario, the model may recognize one object mask that does not correspond to any object, creating a false positive, as in Fig. 15b.
However, the example object may be referenced correctly, as the error does not affect the evaluation of the program in the neural symbolic execution phase.
In all scenarios, the detection error leads to an incorrect object or a semantically ambiguous object being referenced during program evaluation.
Such error may propagate to the meta-learning stage, change the novel concept’s embedding, and eventually result in a correct or wrong answer.
E. F AILURE MODE IN SEMANTIC PARSER
There are a few circumstances where the semantic parser produces an error.
Here we list three cases where the semantic parser may fail, illustrated in Table 16.
First, the model may produce a synthetically wrong program.
For example, there are invalid concept arguments in the output program.
Following Johnson et al.
(2017b)
, the parser applies heuristics to the output, including removing the invalid subprograms where concept arguments are illegal, and produces a feasible program, as in Table 16a.
In other circumstances, the model produces a synthetically correct but semantically wrong program.
We will evaluate the program as it is produced by the Seq2seq.
Such evaluation will either lead to the right answer or lead to the wrong answer.
It should be no surprise that a semantically incorrect program may lead to a wrong answer since the concept arguments are wrong, as in Table 16b.
A semantically wrong program may also lead to the correct answer if the question has redundancy in concept quantiﬁers that refers to the correct object, as in Table 16c.
Currently, there is no mechanism implemented trying to recover from the semantic errors.
In the future, we
Descriptive sentence : The shiny block that are behind small rubber thing is a red object.
Expected example object : The metallic red cube.
Predicted example object : The metallic grey cube.
Explanation : The relational concept behind is not accurate enough; it is so extensive that even the grey cube is behind the yellow cube.
[Figure]
Question : How many red cylinders are to the right of the blue sphere?
Expected answer : 1.
(The red cylinder)
Predicted answer : 0.
Explanation :
The meta-learned concept cylinder is not accurate enough; it is too exclusive that even the red cylinder is not considered a cylinder .
[Figure]
hope to incorporate methods such as REINFORCE (as in Yi et al.
and Mao et al.
to leverage visual information to reﬁne the semantic parser.
E. F AILURE MODE IN CONCEPT LEARNING MODULE
Besides errors in the semantic parser and the detection module, the concept learning module may also lead to potential errors due to imperfect visual groundings.
Here we list two cases where the concept learning module may fail, illustrated in Fig.
In the ﬁrst scenario, the model encounters an error when ﬁltering for a relational concept.
The concept in the program is too extensive and ﬁlters out more candidates than desired.
Such error results in a wrong object example, as in Fig. 9a.
In another scenario, the model encounters an error when ﬁltering for a novel concept recently meta-learned.
The concept in the program is too exclusive and ﬁlters out fewer candidates than desired.
Such error results in a wrong answer in the ﬁnal result, as in Fig.
9b.
In general, errors may take place in the concept learner when ﬁltering for a base concept, a relational concept, or the novel concept just meta-learned when the visual grounding is imperfect.
Such errors may result in different ﬁlter results with more or fewer objects being included.
Further operation on the wrong ﬁlter output may eventually end up with a correct or wrong answer.
Descriptive sentence : The blue matte thing to the right of the big
cyan object is a cube. Expected example object : The blue cube around
the upper right corner of the cyan cube.
Predicted example object : The purple metal cylinder.
(cid:55)
Detection error :
The blue cube is not detected by the detection module.
Descriptive sentence : The white small object is a plate.
Expected example object : The white plate.
Predicted example object : The lower side of the plate.
(cid:55)
Detection error :
The entirety of the white plate is not detected as one object
Descriptive sentence :The metallic thing that is right of the tiny sphere is a cube.
Expected example object : The green cube. Predicted example object : The green cube.
(cid:51)
Detection error :
The union of the blue cube and the gray cube is detected as one object by the detection module.
Descriptive sentence :The large long object is a silver object.
Expected example object : The long knife. Predicted
[Table]
Detection error : Shadows of the knife and the apple are detected as objects by the detection module
[Table]
CountEqual(Filter(Filter(Relate(Filter(Filter(Scene(),small), ball), front), small), ball), Filter(Filter(Scene(), big), cyan)) CountEqual(Filter(Filter(Relate(Filter(Filter(Scene(),small), ball), front), small), Filter), Filter(Filter(Scene(), big), cyan))
[Table]
CountEqual(Filter(Relate(Filter(Filter(Scene(), small), ball), front), small), Filter(Filter(Scene(), big), cyan)
[Table]
Exists(Filter(Filter(Relate(Filter(Filter(Scene(),big), brown), left), metal), cyan))
[Table]
Exists(Filter(Filter(Relate(Filter(Filter(Scene(),big), purple), left), metal), cyan))
[Table]
[Table]
Exists(Filter(Filter(Relate(Filter(Filter(Scene(),big), brown), right), big), cyan))
[Table]
Exists(Filter(Filter(Relate(Filter(Filter(Scene(),big), purple), right), big), cyan))
[Table]