ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time
Abstract
Humans have the remarkable ability to recognize and acquire novel visual concepts in a zero-shot manner.
Given a high-level, symbolic description of a novel concept in terms of previously learned visual concepts and their relations, humans can recognize novel concepts without seeing any examples.
Moreover, they can acquire new concepts by parsing and communicating symbolic structures using learned visual concepts and relations.
Endowing these capabilities in machines is pivotal in improving their generalization capability at inference time.
In this work, we introduce Zero-shot Concept Recognition and Acquisition (ZeroC), a neuro-symbolic architecture that can recognize and acquire novel concepts in a zero-shot way.
ZeroC represents concepts as graphs of constituent concept models (as nodes) and their relations (as edges).
To allow inference time composition, we employ energy-based models (EBMs) to model concepts and relations.
We design
ZeroC architecture so that it allows a one-to-one mapping between a symbolic graph structure of a concept and its corresponding EBM, which for the ﬁrst time, allows acquiring new concepts, communicating its graph structure, and applying it to classiﬁcation and detection tasks (even across domains) at inference time.
We introduce algorithms for learning and inference with ZeroC.
We evaluate ZeroC on a challenging grid-world dataset which is designed to probe zero-shot concept recognition and acquisition, and demonstrate its capability.
(a) training of models !
# % , % - %(b) inference:given a hierarchical concept graph,derive its model from trained models 3(4,5,6 789:;< )= !
+!
+!
+∑ AB,,-,2 !
(#,% A ,& '()* ) !
/ & '()* & '()* & '()* % , % - %(c) detection at inference: combined mask % : k= k= k= k= k= k= k= k= k= k=
Introduction
Humans learn in diverse ways.
Besides learning from demonstrations of a novel concept, humans can also learn concepts on a high-level.
Consider learning the “rectangle” concept, for example.
Suppose that one has never seen such a concept, but has already mastered the concept of “line” and relations of “parallel” and “perpendicular”; s/he can easily master the “rectangle” concept when told that a “rectangle” consists of two pairs of “lines”, the lines within the pairs are “parallel,” and the lines between the pairs are “perpendicular”.
Then s/he can directly use this newly mastered concept to recognize “rectangles” in novel images.
Such zero-shot concept recognition capability is still beyond the reach of deep learning models, which require many examples (as in typical supervised learning), or many tasks drawn from the same distribution (as in few-shot learning) to learn a novel concept.
Moreover, along with the above symbolic to instance direction, humans can easily do the reverse .
Suppose again that we haven’t seen the “rectangle” concept, but have already mastered concepts of “line” and relations of “parallel” and “perpendicular”.
When seeing a novel image which contains an instance of a rectangle, humans can easily decompose it into its constituent concepts of “lines” and their relational graph structure (from instance to symbolic).
Moreover, humans can then communicate with each other about this new concept, allowing the transfer of knowledge in a high-level and in a succinct way.
Such zero-shot concept acquisition capability at inference time is beyond the reach of today’s machine learning and
AI systems.
Above all, endowing the above two capabilities of zero-shot concept recognition and acquisition capability to machines at inference time, will allow them to tackle more diverse tasks.
Past efforts have attempted to address aspects of the above problem of recognizing and acquiring novel concepts at inference time in a zero-shot manner.
An essential component is compositionality, i.e. the ability to compose new concepts from elementary ones.
[1] and [2] introduce techniques for compositional generation in the context of energy-based models (EBMs) and variational autoencoders (VAEs), respectively.
However, these works only consider composing factors of variation ( e.g. , color, position, smiling, young).
In order to recognize concepts that consist of constituent concepts as parts, the structure of objects and their relationships are equally important.
For instance, given 4 “line” concepts, without specifying their relational graph structure, they can easily form a trapezoid or a generic parallelogram.
[3] designed a transformer-based relational architecture for learning explicit relations, and showed that it can generalize to new objects for the relation.
However, this architecture cannot generalize to concepts with more complex relational graph structure.
It remains an open problem of how to design an architecture that has the capability to compose novel concepts at inference time, based on its internal relational graph structure of prior learned concepts and relations.
A separate line of works [4, 5, 6, 7, 8] tackle zero-shot classiﬁcation, but are limited to novel composition of label features for concepts, and ignore relation features as well as the graph structure formed by concepts and relations.
Here we introduce our ZeroC architecture, which is a model designed towards zero-shot concept recognition and acquisition.
It models a visual concept or relation as an Energy-Based Model (EBM) with 3 inputs: an image, a mask (or two masks for relation) denoting a set of pixels within the image, and a concept/relation label string.
It returns a low scalar energy value if the mask(s) correctly indicates the concept or relation within the image, and a high energy value otherwise.
After training, the concept and relation EBMs are able to perform classiﬁcation of concept/relation labels given an image, and detection of concept/relation mask(s) given an image and concept/relation label.
The core contribution of this work is ZeroC’s architecture and inference algorithms that allow a one-to-one correspondence between a speciﬁcation of a relational graph structure of a novel concept and its corresponding composed concept model that can also perform similar downstream classiﬁcation and detection, which enables zero-shot concept recognition and acquisition at inference time.
To perform zero-shot concept recognition at inference time, ZeroC can simply compose a new concept EBM, by summing over the constituent concept and relation EBMs according to the speciﬁed graph structure.
This new concept EBM has the same input-output format as its constituent concept
EBMs, allowing the same capability of classiﬁcation and detection.
Note that through the above process, ZeroC have mastered a concept without seeing an image, only symbolic instruction, thus it is called “zero-shot”.
To perform zero-shot concept acquisition, we introduce an algorithm that given an instance (image) of a novel concept, parse it into a relational graph of prior learned concepts and relations.
Such relational graph can then be transferred, even cross-domain, allowing models independently trained in a different domain to be able to zero-shot classify and detect this concept in its own domain.
There are currently no specialized benchmarks to test such zero-shot concept recognition and acquisition capabilities.
Inspired by the Abstraction and Reasoning Corpus (ARC) [9], we have created a synthetic, grid-world based dataset with tasks that capture the essence of the above capabilities that are deceptively simple for humans, but very hard for neural models.
We demonstrate that our model, trained to classify and detect elementary visual concepts and relations, is able to classify and detect novel concepts at inference time, being given a zero-shot symbolic graph structure.
We also compare with a state-of-the-art zero-shot learning model CADA-VAE [8].
Due to that it is not designed for such graph-based composition, it signiﬁcantly underperforms our ZeroC.
Furthermore, we show that two independently trained models, one on 2D images and the other on 3D images, are able to acquire hierarchical concepts from each other by communicating the graph structure and then perform classiﬁcation and detection of these hierarchical concepts in their respective domains.
Method
In this section, we describe the ZeroC architecture for recognizing and acquiring novel visual concepts at inference time 2 .
We give an overview of our approach, describe how it performs classiﬁcation and detection, how it supports zero-shot concept recognition, how independently-trained models can acquire hierarchical concepts from each other and, ﬁnally, introduce its learning method.
2.1 An Overview of ZeroC Architecture
The key components of ZeroC are concepts and relations.
Each concept consists of a graph and an energy-based model.
The concept graph describes the concept as a composition of its constituent concepts and relations between them.
The base concepts that do not have any constituent concepts are called elementary concepts.
Their graph is a singleton.
The concept energy-based model is used to recognize the concept in the input data.
Each relation is represented in ZeroC also with a graph and an energy-based model.
The relation graph is simply an edge that connects the two related concepts.
A hierarchical concept is a concept composed of constituent concepts as nodes and relations as edges according to a graph structure.
During training, energy-based models are learned from images and concept or relation labels.
During inference, the learned models are used to recognize concepts seen during training or more complex hierarchical concepts which were not seen before.
For concepts seen during training, their learned models are applied.
For new hierarchical concepts not seen during training ( i.e. , without learned model), their models are derived from the graph of the new hierarchical concept and energy-based models of their constituent concepts and relations.
Concepts and relations in ZeroC can be viewed as templates for objects and their connections, which then get grounded during inference with a speciﬁc image, where those objects and relations are assigned actual values.
ZeroC also handles object masks, which indicate object locations in the image.
2.2 Zero-shot Concept Recognition and Acquisition
Formally, we model a concept with E X,M,C ( x, m, c ) 3 , which maps an image x ∈ R D , a mask m ∈ [0 , 1] D and a concept label 4 c to a scalar energy.
Similarly, we model relations with
E X,M 1 ,M 2 ,R ( x, m 1 , m 2 , r ) , where m 1 and m 2 are a pair of masks indicating two objects in the image x , and r is a relation label between the objects.
The concept and relation models have a probabilistic interpretation.
For example, the energy function E X,M,C ( x, m, c ) corresponds to the joint probability of
[Equation]
where Z C =
(cid:80) c ∈ C (cid:82)
e − E X,M,C ( x,m,c ) dx dm is a normalizing constant.
Therefore, if the mask m is actually masking an object that belongs to concept c in image x , then P X,M,C ( x, m, c ) will be high and E X,M,C ( x, m, c ) will be low, and vice versa.
Essentially, the models deﬁne an energy landscape for their respective multi-modal inputs that give low energy if the mask correctly identiﬁes its corresponding concept.
Next, we show how ZeroC performs detection, classiﬁcation, and models hierarchically composed concepts.
We use the term HC-EBM to denote concept models and R-EBM for relation models.
Detection.
We want to infer the location mask m of the concept c , given image x .
Probabilistically, we are computing P M | X,C ( m | x, c ) .
To perform detection, we employ Stochastic Gradient Langevin
Dynamics (SGLD) to sample masks on the energy landscape [10]:
[Equation]
where ˜ m k is the inferred mask at the k th iteration, k = 1 , 2 , ...K .
Applying [11], as
K → + ∞ and λ → 0 , we generate samples from the distribution of P M | X,C ( m | x, c ) =
Z x,c exp ( − E X,M,C ( x, m, c )) , where Z x,c = (cid:82) exp ( − E X,M,C ( x, m, c )) dm is a normalizing constant.
In practice, we use a ﬁnite K to generate samples 5 ˜ m Kn , n = 1 , 2 , ...
N given x, c .
Classiﬁcation.
We want to determine whether the concept c appears in a given image x , i.e. compute
P C | X ( c | x ) .
We need to marginalize over the mask m :
[Equation]
We again use SGLD [10] in Eq.
(1) to generate N samples ˜ m Kn , n ∈ [ N ] = { 1 , 2 , ...N } given x, c , and approximate the above integral using maximum a posteriori (MAP) estimation:
[Equation]
In this paper, we use i to index different concepts, j to index relations, and n to index example images.
In practice, we only need to ﬁnd the concept with the highest value in the numerator.
Zero-shot recognition of novel concepts.
To master a novel hierarchical concept and directly use it for classiﬁcation and detection only given its relational graph structure, we need a way to compose the previous concept and relation energy based models.
Here we introduce the hierarchical composition rule, using an English letter “F” as an illustrative example 6 .
The concept F has one constituent concept, a "line", and two relations, "parallel" and "perpendicular".
The models for constituents are learned during training.
During inference, these models are combined, using the concept graph, into a combined energy model for the letter
F. Essentially, the models of all the recognized constituent concepts and relations are added together.
Note that although F contains three lines, only one "line" model is needed.
The concept graph acts as a template and recognized line instances (objects) are matched with nodes in the template to obtain actual models.
The "line" model is instantiated three times with speciﬁc values for the three identiﬁed lines and the three models plus their corresponding relations models are used to derive the hierarchical model.
In addition to the models, we also need to combine the masks of all the recognized constituent objects.
Formally, we deﬁne the following composition rule for HC-EBMs plus the masks to be combined.
Deﬁnition 2.1.
Hierarchical Composition Rule: Let a hierarchical concept c have graph G =
( C , R ) , where C are constituent concept nodes and R are relation edges.
During inference, these nodes and edges are matched to recognized concept objects and their relationships, which provides their models and masks.
The combined model E X,M,C ( x, m, c ) is then a sum of the models for all the nodes and edges in the graph and the mask M is the maximum of all the masks:
[Equation]
i } , { M j 1 } , { M j 2 }}
2.3 Zero-shot Concept Acquisition at Inference Time
Here, we introduce techniques for acquiring novel concept graph, which is useful for sharing high-level knowledge transfer between independently trained models.
Such capability may have implications in future scenarios where a hypothetical self-driving car communicating the structure of a novel road sign to other cars on the road, or a 2D vision model learning a novel object and transferring it to a more accurate 3D depth model for inference.
To achieve this, we need to establish an equivalence relation between concepts in different domains ( e.g. in 2D and 3D images).
Deﬁnition 2.2.
Structural Equivalence for HC-EBM:
Two HC-EBMs are structurally equivalent, if their graphs are isomorphic, and their constituent HC-EBMs are recursively structurally equivalent.
For example, HC-EBMs for a “rectangle” concept in 3D and 2D images are structurally equivalent if they have the same decomposition into two “parallel-line” HC-EBMs connected by a “connect”
R-EBM, and the “parallel-lines” have the same decomposition into two “line” HC-EBMs connected by a “parallel” R-EBM.
Even though the HC-EBMs are grounded in different domains, they have the same abstract structure, so they represent the same hierarchical concept.
With structural equivalence, independently-trained HC-EBMs from different domains can acquire hierarchical concepts via the graph structure.
The algorithm works as follows (see Appendix A.5): the image is parsed to decompose it into a graph of concepts and relations previously learned by
HC-EBM and R-EBM, then the graph is used in a different domain to compose its HC-EBM for the new concept.
The key step of this process is the ﬁrst, parsing step.
The parsing step can be seen as an inverse of the Hierarchical Composition Rule and is critical in allowing the model to recognize novel hierarchical concepts in its domain using prior-learned concepts and relations from other domains, thus facilitating downstream knowledge transfer between domains.
Alg. 1 provides further details.
Steps 1-2 infer the concept instances in image x using
SGLD (Eq. 1).
Here the energy E ( p ) ( x, M C , c ) is the summation of HC-EBMs on independent masks, each mask m i l representing a potential concept instance belonging to concept c i .
E (overlap) = max (cid:16) 0 , (cid:80) m in ∈M C
m i n − 1 (cid:17) penalizes overlapping masks.
Since the image probably contains fewer concept instances than given, some masks are empty with all-zero values; Step 3 removes these.
Step 4 classiﬁes relations (using Eq. 2) between pairs of detected concept instances.
Step 5 combines the detected concept instances and their relations to build the graph G for this new hierarchical concept c .
2.4 Learning
Since a standard EBM-training method yields poor performance as shown later in our experiments, we describe here our approach to train the models.
Further details of our training algorithm are given in Alg. 2 in Appendix A.1.
We are optimizing the following objective:
[Equation]
The expression of each term is given in Appendix A.2.
The ﬁrst loss L (Improved) is the objective proposed in [12], a state-of-the-art EBM training technique.
However, to enable the challenging zero-shot concept recognition and acquisition, we need more suitable inductive biases.
The next three terms inject the right inductive bias for the task.
L (pos-std) makes sure that the positive energy have similar level, so the composed concept EBM can identify all constituent concepts and relations, without one constituent EBM having too low energy and only recognizing it.
The empty-mask regularization L (em) makes sure that the when the mask is empty, its energy is between the positive and negative energy, which we prove in Appendix A.3 that it is the necessary condition for correctly discover the underlying graph in Sec.
The L (neg) additionally provide additional negative examples, encouraging discovering full concept instead of a part of it (see Appendix A.2 for details).
Experiments
In this section, we set out to answer the following two questions: (a) given a speciﬁcation of graph structure for a novel hierarchical concept, can the composed concept model successfully perform classiﬁcation and detection tasks?
(b) Can HC-EBM acquire a novel concept from HC-EBM trained from another domain?
We also evaluated our model in a challenging setting by comparing with other existing approaches in a controlled and systematic way.
Since there are no suitable dataset, we designed a HierarchicalConcept corpus, a dataset based on grid-world images.
The dataset comes with testing images for the classiﬁcation and detection tasks which contain more complex, hierarchical concepts, composed from simpler concepts in the training set (See Appendix A.9 for details).
The concept instances on our images have varying locations, size, and relative positions, making them more difﬁcult than ﬁxed-location ﬁxed-size inference tasks such as in [3].
We also provide 2D and 3D versions of images for demonstrating concept acquisition across domains.
[Table]
3.1 Zero-shot Classiﬁcation and Detection of Novel Concepts
To test recognition of novel concepts [Table] we designed two datasets consisting of different concept and relation types.
The HD-Letter hierarchical dataset consists of concept instances of “line” and relation instances of “parallel”, “perp-edge” (perpendicular and touching edge), “perp-mid” (perpendicular and touching middle), together with distractor objects.
Examples are provided in the form of 3-tuples ( image, mask, concept ) for concepts, and ( image, mask 1 , mask 2 , relation ) for relations.
ZeroC models are trained via objective Eq.
(See Appendix A.9 for examples of training datasets).
At inference time, the models need to perform classiﬁcation and detection on novel images with more complicated English characters of “E-shape”, “F-shape” and “A-shape”, given their structure graphs with up to 4 nodes and 6 edges.
For detection, the images also contain a few distractors (concepts unrelated to the ones the model is predicting).
The model is asked to return a mask indicating which pixels belong to an instance of the speciﬁed novel concept.
Although looking simple, this is actually a very challenging task.
because in order to solve the problem of classifying/detecting hierarchical concepts, the model needs to solve a subgraph isomorphism problem, which is NP-hard.
For example, take the problem of detecting “E” shape in an image with distractor of “T” and “Rectangle” (Fig.
2 (a) ﬁrst subﬁgure), a graph with 10 nodes (lines), 45 possible edges.
Note that during training time, a model has only learned “line” concept, and relations “parallel”, “perp-edge” (perpendicular, touching edge), and “perp-mid”, but not the overall “E” shape.
At inference time and given the “E” concept graph with 4 nodes (lines), 6 edges (relations), the model needs to ﬁnd the “E” concept subgraph within the large image graph.
This involves C 410 × 4!
=
5040 possible mask assignments.
Additionally, a model may not perfectly detect the masks for low-level concepts.
For more complex concepts and relations, we designed the HD-Concept hierarchical dataset, which consists of training concepts of “E-shape” and “rectangle”, and training relations of “inside”, “enclose” and “non-overlap”.
The hierarchical concepts to be classiﬁed and detected are three characters which we term Concept1,
Concept2 and Concept3, as the ground-truth masks in Fig.
2 (a) indicate.
The three hierarchical concepts have the same multiset of concepts, but a different relation structures.
As the closest setting to our classiﬁcation task is zero-shot classiﬁcation, we compare ZeroC to a state-of-the-art zero-shot learning algorithm CADA-VAE [8], which we adapted to our setting, using the set of concepts and relations as feature embeddings to represent the concept graph (see Appendix
A.6 for details).
Additionally, we evaluated a “Statistics” baseline which samples random pixels on the image based on the global statistics of pixel occurrence, and a “Heuristics” baseline that randomly chooses a same-color connected object.
We use a CNN-based architecture for ZeroC, as given in
Appendix A.4.
We add comparisons with ablation of different aspects of ZeroC, including without
HC-EBM terms or R-EBM terms in the composition rule of Def.
2.1 (Eq. 3), and without different regularization terms of the objective in Eq. 4.
Fig.
2 shows a demonstration of this experiment.
From performance results in Table 1, we see that ZeroC achieves classiﬁcation accuracy of 84.5% and 70.5% on HD-Letter and HD-Concept, respectively, both higher than CADA-VAE.
The gap is more signiﬁcant in HD-Letter, where there is a larger distribution shift for hierarchical concepts (as can be seen in Appendix A.9), which the graph for F graph for E graph for A
E=-1.
combined mask
TotalEnergy
E=-0.
E=0.
Individual masks graph for Concept
E=0.
combined mask
TotalEnergy
E=0.
E=-0.
Individual masks graph for Concept graph for Concept (b) Classification: (a) Detection: correct!
correct!
graph for A
IoU: 0.
graph for E
IoU: 1.
inferred ground-truth graph for F
IoU: 0.
inferred ground-truth
Concept
IoU: 0.
Concept
IoU: 0.
inferred ground-truth
Concept
IoU: 0.
[Table]
joint embedding of the image and features learned by CADA-VAE is insufﬁcient to handle.
The reason for the low accuracy is that CADA-VAE is not able to address these out-of-domain distribution shifts (See Appendix A.7 for details).
During inference, its embedding (multi-hot vector) for the graph structure can contain up to 10 hots (4 lines, 6 edges), while during training, it is only up to 1-hot.
This example also shows the intrinsic difﬁculty of the task. In addition, Table 1 and Fig. 2 (a) show that ZeroC is able to detect the hierarchical concepts in the presence of distractors, and that its performance is signiﬁcantly better than “Statistics” and “Heuristics” baselines.
The results of our ablation studies in Table 1 show that both the HC-EBM and R-EBM terms in Eq.are key to successful classiﬁcation and detection.
This is likely because without relations and using only concept terms, ZeroC may lose its ability to distinguish between different hierarchical concepts; for example, both “E-shape” and “A-shape” have 4 lines but different relation structures.
Moreover, we see that both L (pos-std) and L (neg) improve classiﬁcation and detection performance.
3.2 Acquiring Novel Hierarchical Concepts Across Domains
We show that ZeroC can acquire novel hierarchical concepts across models and domains (Fig. 3).
We train a ZeroC model in domain ZeroC 1 , where the images are 2D and one-hot color-coded, containing the the same training concepts and relation types as in HD-Letter in Sec. 3.1.
Independently , we train another model in a different domain ZeroC 2 , where the images contain the same set of concepts and relations, but are viewed in 3D from a certain camera angle, have larger size, and use RGB colors.
At inference time, each test task consists of a tuple of three images showing the hierarchical concepts in the ﬁrst domain to ZeroC 1 .
ZeroC 1 is only allowed to send symbolic information, up to a few bits, to
ZeroC 2 .
Then ZeroC 2 performs classiﬁcation and detection tasks on three example images in the second domain.
In addition, we also evaluate whether ZeroC 1 can parse the concept graph of the hierarchical concept correctly, using the metric of graph edit distance
d edit and graph-isomorphism accuracy w.r.t.
the ground-truth.
Note that the graph-isomorphism accuracy is a stringent metric,
ZeroCparse cccdiscovered graph d edit =
d edit =
d edit =
2D concept demonstration channel ccc
E=1. E=1.
E=0. E=0.
E=0. E=0.
E=0. E=0.
E=1.
3D image
ZeroCinferred: which is only 1 if the parsed graph is isomorphic to the ground-truth, and 0 otherwise 7 . We compare with a strong baseline of Mask R-CNN [13] for object detection together with relation classiﬁcation.
The relation classiﬁcation network is a CNN which takes as input a pair of embeddings produced by the Mask R-CNN, and predicts the label of the relation (see Appendix A.8 for details).
We did extensive tuning of the Mask R-CNN approach to obtain its best performance.
We also evaluate the performance of ZeroC 2 given the ground-truth structure graph, as an ideal scenario of perfect parsing, providing an upper bound for the performance.
Table 2 shows that ZeroC 1 achieves 72.7% graph isomorphism accuracy for parsing the hierarchical concepts in the ﬁrst domain, signiﬁcantly higher than Mask R-CNN
+ relation classiﬁcation.
After sending the concept graphs in the second domain, ZeroC 2 achieves a classiﬁcation accuracy of 60.7% and detection IoU of 94.4%.
Without this information and relying on global statistics, the classiﬁcation accuracy of ZeroC 2 is 33.3%.
This demonstrates that ZeroC 2 is able to acquire novel hierarchical concepts at inference time, from an independently trained model from a different domain .
The ablation study shows that L (em) , L (neg) and L (pos) all contribute to more accurate parsing.
The standard technique of EBM training [12] are insufﬁcient to achieve good parsing, leading to less accurate transfer of hierarchical concepts.
We also see in the ablations that even without perfect parsing from ZeroC 1 , the reduction of classiﬁcation accuracy for ZeroC 2 is small, showing that it is able to classify under noisy speciﬁcation of the concept graph.
We discuss further on the generality (Appendix A.13), scalability (Appendix A.14) and computational complexity (Appendix A.15) in respective Appendix sections.
Related Work
Our work relates to visual compositionality, concept and relation learning, and zero-shot learning.
Visual compositionality.
Compositionality is a key for addressing diverse tasks given ﬁnite basic knowledge.
Some approaches use composition EBMs for generation [1], and a VAE-based architecture for bi-directional symbol-image generation that can also learn logical recombination of concepts [2].
These two works focus on composing factors of variation , e.g. color, position, smiling, young.
In contrast, our work focuses on concepts that abstract objects, where the internal hierarchical relational structure is key.
Moreover, while the above works focus on generation, our work focuses on the tasks of classiﬁcation and detection.
A novel Bayesian-based method for few-shot learning on the
Omniglot dataset shows that compositionality is pivotal for improved performance [14].
They achieve compositionality via hierarchical MCMC sampling on hand-coded priors of elementary concepts and relations.
In comparison, our work is neural network-based and only requires demonstrations of elementary concepts and relations, reducing hand-coded priors.
Another approach introduced a modular neural network, which uses composition of neural modules for visual question answering [15].
Their method can be seen as composing transformations on a representation.
In comparison, our composition is achieved via composition of energy landscapes.
While they focus on question answering on scene graphs, we focus on classiﬁcation and detection of hierarchical concepts.
Concept and relation learning.
There has been exciting progress in concept learning and relation learning.
Works in concept learning generally represent concepts in latent space via prototypes [16, 17], or via latent embedding such as SCAN [2], InfoGAN [18] and Neuro-Symbolic Concept
Learner [19].
[20] introduced EBMs to represent concepts with a demonstration in simple state space.
[21] further introduced unsupervised learning of local and global concepts with EBMs.
Regarding relation learning, non-local neural networks [22], Relation Networks [23], Neural Relational Inference [24] and C-SWM [25] use latent complete graphs as inductive biases to represent potential relations.
PrediNet [3] explicitly represents propositions, relations, and objects with a transformer-based architecture, and demonstrates that it can learn relations that generalize to novel shapes of objects and column patterns.
In comparison to the above works, ZeroC explicitly learns both concepts and relations, which has the unique capability to recognize and acquire hierarchical concepts at inference.
Zero-shot learning.
Zero-shot learning methods [4, 5, 6, 7, 8] typically learn a joint embedding space between image and feature labels, and perform classiﬁcation at inference time on images with novel classes, based on how the novel classes correspond to a set of features.
Our generalization task also requires generalizing to new concepts without seeing the image (zero-shot), but has the important distinctions that at inference time, our concepts to be inferred lie at a higher hierarchy than that in training, and furthermore use the structural concept graph.
In comparison, in standard zero-shot learning, concepts in training and inference lie at the same hierarchy level, and only generalize to new combinations of features (constituent concepts) while neglecting relation structures.
Conclusion
In this paper, we introduce ZeroC, a new framework for zero-shot concept recognition and acquisition at inference time.
Our experiments show that in a challenging grid-world domain, ZeroC is able to recognize complex, hierarchical concepts composed of English characters in a grid-world in a zero-shot manner, being given a high-level, symbolic speciﬁcation of their structures, and after being trained with simpler concepts.
In addition, we demonstrate that an independently trained ZeroC is able to transfer hierarchical concepts across different domains at inference.
Although this work is evaluated only in grid-world visual domain, we are the ﬁrst to address this difﬁcult challenge. We are also excited to see its potential application in broader domains, e.g. in AI for scientiﬁc discovery, where it may infer novel patterns and concepts from data in a zero-shot manner. We hope that this work will make a useful step in the development of composable neural systems, capable of zero-shot concept recognition and acquisition and hence suitable for more diverse tasks.
Acknowledgments and Disclosure of Funding
We thank Rui Yan and Blaž Škrlj for discussions and for providing feedback on our manuscript.
We also gratefully acknowledge the support of DARPA under Nos.
HR00112190039 (TAMI),
N660011924033 (MCS); ARO under Nos.
W911NF-16-1-0342 (MURI), W911NF-16-1- (DURIP); NSF under Nos.
OAC-1835598 (CINES), OAC-1934578 (HDR), CCF-1918940 (Expeditions), NIH under No. 3U54HG010426-04S1 (HuBMAP), Stanford Data Science Initiative, Wu Tsai
Neurosciences Institute, Amazon, Docomo, GSK, Hitachi, Intel, JPMorgan Chase, Juniper Networks,
KDDI, NEC, and Toshiba.
The content is solely the responsibility of the authors and does not necessarily represent the ofﬁcial views of the funding entities.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
In conclusion and Appendix A.10.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] In Appendix A.11.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.
If you ran experiments (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The Appendix includes full details on model architecture, training and evaluation to reproduce the experimental results.
Code and data will be released upon publication of the paper.
(b) Did you specify all the training details ( e.g. , data splits, hyperparameters, how they were chosen)?
[Yes] Full details to reproduce the experiments are included in the Appendix.
(c) Did you report error bars ( e.g. , with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used ( e.g. , type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix of respective experiment section.
If you are using existing assets ( e.g. , code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] We use our own created benchmark.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] In Appendix A.9, we describe the engine and training/testing tasks.
Full dataset will be released upon publication.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix
In the Appendix, we provide details that complement the main text.
Speciﬁcally, in Appendix A.1 and
A.2, we detail the learning algorithm of HC-EBMs and the learning objective these models optimize.
In Appendix A.3, we prove a necessary conditions for ZeroC to correctly discover the underlying concept graph.
In Appendix A.4, we provide the exact neural architecture used to parameterize
HC-EBMs.
In Appendix A.5, we present further explanation on how hierarchical concepts acquired in one domain may be transferred to different domain, as well as a note on time complexity of inference.
In Appendix A.6 and A.7, we present implementation details on the CADA-VAE baseline for classifying hierarchical concepts and an analysis of its limited performance.
In Appendix A.8, we similarly present implementation details on the Mask R-CNN + Relation Classiﬁcation baseline for acquiring concepts.
In Appendix A.9, we detail the generation process of our 2D and 3D grid-world datasets.
In Appendix A.10, we explain some limitations of our current work and propose several future directions to explore.
In Appendix A.11, we discuss our work’s broader social impact.
In
Appendix A.12, we present additional experimental results on a CLEVR dataset to demonstrate the potential of our framework to generalize to more real-world settings.
In Appendix A.13, we further discuss the generality of our framework to different sets of elementary concepts and relations and to other datasets.
In Appendix A.14, we explain the scalability of the framework with respect to task complexity, inference time complexity, and image complexity.
In Appendix A.15, we summarize
ZeroC’s computational complexity and our empirical observations.
A.1 Learning algorithm
Here we give the learning algorithm for HC-EBMs, which can be elementary or hierarchical.
A similar algorithm applies to R-EBMs, by simply replacing the m by m 1 , m 2 .
Here we omit the subscript of X, M, C for clarity.
In our experiments, we perform hyperparameter search over the coefﬁcients, to optimize the average classiﬁcation accuracy and detection IoU on the validation set that has the same type of concepts/relations (not validating on the inference tasks in Section 3).
For training HC-EBMs, we use α pos-std = 0 .
1 , α neg = 0 .
05 and α em = 0 .
For training R-EBMs, we use α pos-std = 1 , α neg = 0 .and
α em = 0 .
We use learning rate 10 − 4 , number of SGLD steps K = 60 for training and K =for inference.
During inference, we use an ensemble size of 64 to perform MAP estimation as in Eq.
(2), and use ensemble size of 256 for detection.
A.2 Learning objective
Here we detail the learning objective for ZeroC and provide explanations, complementing Sec. 2.4.
The L improved n is the objective proposed in [12], neglecting the entropy regularization term.
In addition, as explained in Sec. 2.4, we introduce three regularizations to inject the right inductive biases to address the tasks of zero-shot concept recognition and acquisition:
[Equation]
The superscript “ + ” denotes positive examples from ground-truth, “ − ” denotes negative examples generated from SGLD, “(em)” denotes an empty mask, and “(neg)” denotes algorithmically generated negative examples.
The additional regularizations are L (pos-std) n for reducing energy variance for positive examples, L (em) n for empty-mask regularization, and L (neg) n which is algorithmically generated negative examples.
Note that unlike standard image-only EBMs, the training needs to take multiple modalities into account: image x , mask m and concept c .
We perform conditional SGLD w.r.t.
each modality to generate negative examples.
E.g. , when generating negative examples for masks m − n , we provide ground-truth of image and concept label x − n ← x + n , c − n ← c + n .
Thus, ( x − n , m − n , c − n ) =
( x + n , m − n , c + n ) is a negative tuple even though x − n and c − n are from ground-truth.
For each minibatch we randomly sample the modality the conditional SGLD is performed on (steps 4-21 in Alg. 2).
Require: data dist.
p D ( x, m, c ) , HC-EBM E θ ,
Require: step size λ , number of steps K , random mask generator U a , random embedding generator
U c , coefﬁcients α pos-std , α neg , and α em .
1: B ← ∅ 2: while not converged do 3: ( x + n , m + n , c + n ) ∼ p D //
Generate samples from Langevin dynamics: 4: rand ∼ U [0 , 1] 5: if rand ∈ [0 , 1 / 4) do 6: ( x n , ˜ m 0 n , c + n ) ∼ B with 20% probability and ˜ m 0 n ∼ U a otherwise 7: ˜ m
Kn ← SGLD ˜ m
( E θ ; x n , ˜ m 0 n , c + n ) 8: ( x − n , m − n , c − n ) ← ( x n , ˜ m
Kn , c + n ) 9: elseif rand ∈ [1 / 4 , 1 / 2) do 10: ( x n , m + n , ˜ c 0 n ) ∼ B with 20% probability and ˜ c 0 n ∼ U c otherwise 11: ˜ c
Kn ← SGLD ˜ c ( E θ ; x n , m n , ˜ c 0 n ) 12: ( x − n , m − n , c − n ) ← ( x n , m + n , ˜ c Kn ) 13: elseif rand ∈ [1 / 2 , 3 / 4) do 14: ( x n , ˜ m 0 n , ˜ c 0 n ) ∼ B with 20% probability and ˜ m 0 n ∼ U m , ˜ c
0 n ∼ U c otherwise 15: ˜ m Kn , ˜ c
Kn ← SGLD ˜ m, ˜ c
( E θ ; x n , ˜ m 0 n , ˜ c 0 n ) 16: ( x − n , m − n , c − n ) ← ( x n , ˜ m Kn , ˜ c Kn ) 17: else do 18: (˜ x n , ˜ m 0 n , ˜ c 0 n ) ∼ B with 20% probability and ˜ x 0 n ∼ U
x
otherwise 19: ˜ x
Kn ← SGLD ˜
x
( E θ ; ˜ x 0 n , m n , c n ) 20: (˜ x − n , m − n , c − n ) ← (˜ x Kn , m n , c n ) 21: end if // Optimize objective for E θ wrt θ with Eq. 4: 22: ∆ θ ← ∇ θ 1 N (cid:80) n (cid:16) L (Improved) n + α pos-std L (pos-std) n + α em L (em) n +
[Equation]
L (neg) n (cid:17) 23:
Update θ based on ∆ θ using Adam optimizer 24:
B ← B ∪ ( x − n , m − n , c − n ) 25: end while 26: return E θ
Require: energy-based model E θ with concept (relation) embedding c ( r )
Require: SGLD target ˜ q , choose from ˜ m
˜ c
( ˜ m c )
Require:
Input x
Require: step size of steps K
Reducing variance of energy for positive examples.
The use of only contrastive divergence in typical EBM training, i.e. pushing down energy for positive examples and pushing up energy for negative examples, is insufﬁcient, since the composed HC-EBM needs to discover the masks of all its constituent models.
For example, with 2 objects in the image x + with concepts c +
and c
+2 , respectively, we can use E ( x + , m 1 , c + 1 ) + E ( x + , m 2 , c + 2 ) according to our composition rule (Def.
2.1) to detect their respective masks.
However, L (improved) only encourages ( x + , m +1 , c +1 ) to be lower than ( x + , m − 1 , c +1 ) locally, but it can still be higher than ( x + , m − 2 , c +2 ) for a negative mask m − 2 for concept c +2 .
Then the composed energy model will favor discovering m − 2 instead of m +1 .
Thus, we add L (pos-std) to encourage similar energy for positive examples, thus lower than the energy for negative examples.
x mmx mx mx mx m
Training concepts: line
Training relations: parallel/perp-mid/perp-edge
Inference: classifying E-shape/F-shape/A-shape, given only x and the each of its concept graph
Inference: detecting E-shape from distractor, given x and E-shape’s concept graph
Inference: detecting F-shape from distractor, given x and F-shape’s concept graph
Inference: detecting A-shape from distractor, given x and A-shape’s concept graph
[Figure]
line line c line line line line line perp-edge parallel perp-edge
[Figure]
r perp
[Figure]
mid perp
[Figure]
edge perp
[Figure]
edge perp
[Figure]
edge perp
[Figure]
mid
[Figure]
We see that the concepts in inference is more complex than those in training.
E.g. for detecting “Eshape” in inference, a model will need to compose up to 4 nodes (of “line” concepts) and 6 edges of “parallel”, “perp-mid”, “perp-edge” relations.
Empty-mask regularization.
To properly perform parsing (Alg. 1) when there are more energy terms than actual instances, we require that the redundant masks are empty instead of being some random negative masks.
in Appendix A.3, we prove that the necessary condition for Alg. 1 to correctly discover the underlying concept graph is that the energy of the empty mask lies between the energy of positive examples and negative examples.
Intuitively, given the image x + n and concept c + n , the energy E ( x + n , m (em) n , c + n ) with empty mask m (em) n should be between the positive energy
E ( x + n , m + n , c + n ) and negative energy E ( x + n , m − n , c + n ) for the empty mask to appear before random negative masks appear.
Therefore, in L (em) n , we encourage E ( x + n , m (em) n , c + n ) to be near the average of positive and negative energy.
If there is redundant energy terms during parsing, the corresponding masks will favorably become all-zero instead of some random negative mask.
Algorithmically generated negative examples.
To encourage each mask to discover a single concept instance instead of a partial instance, we randomly corrupt the tuple ( x + n , m + n , c + n ) and push up the corresponding energy.
Reason to neglect the entropy regularization.
The entropy term in [12] serves to increase the diversity of the generated examples.
And the computation of entropy requires many examples.
This is ﬁne in [12] since the EBM there has the form of E(x) which only needs to generate images unconditionally , and the entropy can be estimated using all previous generated images x. In our work, our EBM are E ( x, m, c ) and E ( x, m 1 , m 2 , c ) , and we need to generate the mask conditionally , e.g. generate mask m conditioned on the image x and label c .
The entropy term would need to be a conditional entropy of m given x and c , where the pool of mask m should be different for each individual image x and label c .
This requires, e.g. for each x , c , we generate over 100 masks to estimate the entropy which is computationally expensive, while currently we only need to samplemask.
Moreover, typically there are limited correct masks for a concept in an image, and encouraging diversity may not help the model identify the correct mask.
In fact, we have tried empirically with keeping the entropy term and it results in a much worse accuracy, likely due to the above reason.
A.3 Proof for necessary condition for correctly parsing the graph
Here we provide the proof that the energy for an empty mask needs to lie between the positive energy and negative energy, justifying the introduction of L (em) n in Eq.
Speciﬁcally, we prove:
Theorem A.1.
Let E + =
E θ ( x + n , m + n , c + n ) be the “positive energy” for all positive examples of ( x + n , m + n , c + n ) , and E − =
E θ ( x − n , m − n , c − n ) be the “negative energy” for any negative examples 8 of ( x + n , m + n , c + n ) .
Let E (em) be the energy for an ( x + n , m (0) , c + n ) where m (0) = 0 is an empty mask.
A necessary condition for correctly discovering the underlying concept graph with Alg.
1 is that
[Equation]
Here E (overlap) is the energy added in Alg.
1 to penalize the concept EBMs to discover overlapping concepts.
Proof.
Suppose that in the image there are in total m ≥ 1 objects, and there are in total n ≥concept EBMs.
We want that only m EBMs have their masks enabled and all the rest n − m masks are empty (if n ≥ m ), i.e. this conﬁguration should have the lowest total energy.
In other words suppose that instead there are k masks that ﬁnds the objects (if will not overlap until k > m , and n − k remains empty or have negative masks, then it should have a higher energy:
[Equation]
These two expressions should hold for any k, m .
Setting k
= m in Eq.
(8), we have
[Equation]
em
[Equation]
< E −
From Eq.
(7), if n ≥ k > m , and after re-arranging terms, we have
E (em) <
E + + E (overlap
[Equation]
From Eq.
(7), if n ≥ m > k , and after re-arranging terms, we have
[Equation]
Combining the above three conditions, we have
[Equation]
) which concludes the proof.
This justiﬁes the L (em) n
[Equation]
E (em) | in Eq.
(4), where it encourages that the E (em) stays between E + and E − , and penalizes the deviation of E (em) to 12 ( E + + E − ) .
A.4 Network architecture of ZeroC
For all experiments in the paper, we use the same architecture of concept and relation EBMs, with the only difference being the number of input channels (10 for 2D images and 3 for 3D images).
We provide in Table 5 the architecture of a HC-EBM, which consists of several ResBlocks (Table 3).
Adding the residual to the ﬁnal output is denoted as Skip().
When downsampling is performed, the residual is the output of two fully-connected layers applied to a ﬂattened input image; otherwise, the residual is the input image.
Chunk() splits an input vector into two equal-sized halves and expands both halves along a 2d grid with the same dimensions as the input image.
The C_Embed() architecture is detailed in Table 4, with c_dim denoting the dimension of the concept embedding.
[Table]
[Table]
A.5 Acquiring Hierarchical Concepts
We present details about the algorithm for acquiring hierarchical concepts across models and domains
[Table]
The algorithm works as follows (see also Alg. 4)
We ﬁrst parse the image by decomposing it x 1 into a graph G 1 of concepts and relations previously learned by HC-EBM 1 and R-EBM 1 .
Next, we copy G 1 to G 2 in domain 2.
Finally, we compose a new HC-EBM 2 ( c ) using G 2 , HC-EBM 2 and R-EBM 2 (Hierarchical Composition Rule, Def. 2.1).
The most complex step of this algorithm is parsing, the ﬁrst step, which is described in detail in the paper (see Alg. 1).
Note on time complexity.
Note that in Alg. 1, although the relation EBM needs to perform classiﬁcation for each pair of discovered objects, which scales as n 2 where n is the number of objects, the
[Table]
Require: HC-EBM 1 and R-EBM 1 in domain 1, HC-EBM 2 and R-EBM 2 in domain 2, with prior-learned concepts or relations in their respective domain.
Require:
Image x 1 in domain 1, containing unseen hierarchical concept c .
// Using Hierarchical Composition Rule (Def.
.1) process is actually very fast, since we can batch all the pairs into a single minibatch, and can get the classiﬁcation result with a single SGLD run, which has the same runtime as doing inference with a relation EBM on a single image.
A.6 Implementation Details for CADA-VAE
CADA-VAE [8] learns a common space latent space for image features and class embeddings, by aligning modality speciﬁc variational autoencoders.
Alignment is encouraged by adding two regularization terms to the standard VAE loss.
This enables discriminative latent features to be sampled for unseen classes and a softmax classiﬁer to be trained on top of such features.
To train VAE’s in encoding and decoding features in the image modality, we require a pretrained backbone speciﬁc to our domain.
We obtain this pretrained model by training train a network for predicting object masks and either the concept or relation labels in a self-supervised manner [26].
Similar to the original work, we use a ResNet-12 as the backbone, which consists of 3 residual blocks of 64, 160, 320 ﬁlters, each with 3 × 3 convolutions.
A 2 × 2 max pooling operation is applied after each of the ﬁrst 3 blocks.
Following the blocks, we have two mask prediction heads that have the same architecture.
The architecture is symmetric to the ResNet-12 backbone, with 3 blocks of 320, 160, 64 ﬁlters, each with 3 × 3 convolutions and an upsampling layer.
For classiﬁcation, a global average pooling is applied after the last block.
Additionally, a 4 neuron fully-connected layer is added after the ﬁnal classiﬁcation layer.
After training the network end-to-end, we use the ResNet backbone as a pretrained feature extractor.
Class embeddings for the Hierarchical Concept corpus consist of slots for each atomic concept and relation.
The number of slots per concept / relation is equal to the maximum number of times it can appear in a hierarchical concept.
A single slot assignment (setting the value of a slot from 0 to 1) corresponds to an instance of the slot’s concept / relation.
Multiple slots are assigned if more than one instance of the matching concept / relation is found in a class.
During training, where the ground truth label exists for only one concept / relation instance, we randomly sample at each minibatch to determine which of the ground truth slot to assign.
In this way, the class embedding VAE should learn encodings that are invariant to permutations of assigned and non-assigned slots.
#$" ! " #$" ! "
%$ & '()# & '()# * + * , *
[Figure]
yes no no
Domain 1: 2D,10 color channels
Domain 2: 3D, RGB
[Figure]
A.7 Investigation of CADA-VAE performance on HD-Letter dataset
In Table 1, we see that the classiﬁcation accuracy of CADA-VAE is 18.0%, even lower than the “Statistics” method of 46.5%.
Here we investigate the reason.
Firstly, we want to see if the image encoder in CADA-VAE has enough power to differentiate the different novel concept classes (“Eshape”, “Fshape” or “Ashape”) during inference.
We perform t-SNE on the embeddings of test images, with each color denoting a different class (the labels are unseen to the algorithm).
Fig.
6 shows the visualization.
We see that the encoder is able to roughly separate the unseen images into different clusters that roughly correspond to different unseen concept classes.
This shows that the image encoder of CADA-VAE has enough power and is not the reason for its low performance in HD-Letter.
The reason then lies in the distribution shift of the class embedding, which is a multi-hot vector indicating the presence of individual concepts (“line”) or relations (“parallel”, “perp-mid”, “perp-edge”).
During training, the class encoder has only seen the class embeddings where only one concept or relation is present.
However, during inference, a hierarchical concept ( e.g. “Eshape”) may have up to 4 concept instances ( e.g. 4 lines) and 6 relations, thus the class embedding will have many hots activated.
This constitutes a large distribution shift for the class embedding, so that CADA-VAE does not know how to interpret it.
Note that this cannot be easily ﬁxed with alternative class embedding schemes.
No matter how we specify features for the classes in training and inference, the nature of our challenging HD-Letter dataset will result in a large distribution shift for the class embedding (up to 4 concept instance + 6 relation in inference vs.concept/ 1 relation in training).
This shows the limitation of CADA-VAE and similar zero-shot learning methods, where they are not equipped to handle zero-shot learning of hierarchical concepts that are more complex due to the composition of learned concepts and relations.
In contrast, our ZeroC naturally supports such composition into concepts with more complex structure, which enables zero-shot recognition of hierarchical concepts at inference time.
A.8 Implementation Details for Mask R-CNN + Relation Classiﬁcation
Firstly, one may note that the performance of Mask R-CNN’s graph parsing accuracy (35.5%) is seemingly low.
In fact, this performance is very good.
Note that we use the stringent metric of isomorphism accuracy: the acc is 1 only if the inferred graph is isomorphic to ground-truth, and 0 otherwise.
This metric presents a major challenge even for “simple” shapes.
In the example of parsing the graph for “E” shape with 4 nodes (lines), 6 edges (their relations), even if each node and edge classiﬁcation acc is 0.9, the isomorphism acc is 0 .
9 10 = 0 .
349 , and 0.8 individual acc
would
[Figure]
result in 0 .
8 10 = 0 .
Thus to reach 35.5% isomorphism accuracy, a model has to have very high accuracy for classifying individual concepts and relations.
We perform intensive hyperparameter tuning for the baseline Mask R-CNN
+ relation classiﬁcation.
We ﬁrst pretrain a Mask R-CNN [13] on the Hierarchical Concept corpus to minimize object segmentation and object classiﬁcation losses.
We then ﬁx the Mask R-CNN weights and use the pretrained networks to output all objects in an image.
Given the Mask R-CNN output for an image, the relation head is trained to minimize the relation classiﬁcation loss with respect to a ground truth pair of objects and their corresponding relation.
Mask R-CNN : Instance segmentation is the task of precisely detecting objects through bounding-box localization and precisely segmenting each object instance while correctly predicting its corresponding class.
We make use of a popular choice for instance segmentation, Mask R-CNN, which consists of a branch for predicting segmentation masks within Region of Interests (RoI) in parallel with a branch for classiﬁcation and bounding box regression.
The architecture for the segmentation branch consists of four convolutional upsampling blocks, each with 256 ﬁlters.
A vanilla Mask R-CNN architecture with a ResNet50 and Feature Proposal Network (FPN) [27] backbone achieves strong object localization and segmentation results on the COCO dataset [28] when trained from scratch.
As COCO is much larger in scale, in terms of image resolution, number of object instances, and number of object classes, we modify the vanilla architecture to suit the
Hierarchical Concept corpus’s smaller image size.
We use a ResNet18-FPN backbone for feature extraction, with the FPN having anchor boxes with side lengths of 4, 8, 16, 32.
During training, we consider the top 2000 object proposals and reduce these to the top 1000 through NMS with a IoU threshold of 0.7.
During inference, the top 20 object proposals are kept before and after NMS.
We train on one GPU with a batch size of 2 for 176k iterations (equivalent to 8 epochs), with a learning rate of 0.0025.
To tune the standard Mask R-CNN architecture that works on much larger image resolutions, we decreased the size of the backbone ResNet, decreased the size of anchor boxes, and increased the scales for the RoIPooler relative to the input image.
We found that decreasing the number of proposals before NMS did not improve performance, nor did decreasing the size of the mask upsampling head.
We also tuned the learning rate across values in the set {0.005, 0.001, 0.0025, 0.05}, with 0.
yielding the best performance.
Our Mask R-CNN performance for detecting lines on a subset of 1000 training images is 97.4 mIoU. We found that this performance is not signiﬁcantly affected by increases in learning rate warmup length or by increases in the number of iterations before learning rate decay.
Relation Head: The architecture of the relation classiﬁer consists of three residual blocks, followed by a fully-connected network with two hidden layer.
Each residual block consists of 3x3 convolutions with spectral normalization, followed by downsampling.
The relation head predicts the relation between all pairs of object masks outputted by the Mask R-CNN.
To train the relation head, we obtain
[Table]
line an object represents a solid line rectangle an object represents a hollow rectangle rectangleSolid an object represents a solid rectangle
L-shape an object represents a shape that is “L” like
C-shape an object represents a shape that is “C” like
A-shape an object represents a shape that is “A” like
E-shape an object represents a shape that is “E” like
T-shape an object represents a shape that is “T” like rand-Shape an object that is randomly constructed
[Table]
Description inside object x is inside of object y enclose objects x is enclosed by object y parallel objects x is parallel with object y perp-mid object x is perpendicular with object y, and touch object y in its middle.
perp-edge object x is perpendicular with object y, and touch object y in its edge.
non-overlap objects x and y are not overlapped in both x and y axes.
the predicted object masks that are closest (in terms of IoU) to the pair of ground truth masks and compute the loss on the predicted relation for the corresponding Mask R-CNN object masks.
We tuned the learning rate of the relation head across values in the set {5e-5, 2.5e-5, 1e-5} and found that 2.5e-5 gave the best performance.
We found that ﬁxing the Mask R-CNN weights was essential to stable training of the relation head.
The accuracy of our relation classiﬁer is 94.5% on the training set.
A.9 The Hierarchical Concept Corpus
In this section, we describe our Hierarchical-Concept corpus for our experiments and how it is generated.
Our data generation framework is designed for generating large-scale datasets for concept and relation learning in a grid-world setting.
Speciﬁcally, it samples pixel-level arbitrary objects, and places on to grid-world with predeﬁned relations between objects.
The task format is inspired by the
Abstract Reasoning Corpus (ARC) proposed by [9].
Figure 4 to Figure 8 shows examples for concepts and relations used in our experiments, for the
HD-Letter and HD-Concept datasets, and a dataset of 2D to 3D concept transfer that tests acquiring hierarchical concepts between domains at inference time.
For both HD-Letter and HD-Concept, there are 44000 examples of concepts, split 10:1 for train and validation, and 44000 examples for relations, split 10:1 for train and validation.
At inference time for hierarchical concepts, the classiﬁcation task has 200 examples, and detection task has 600 examples, 200 for each of the hierarchical concepts.
The 2D to 3D transfer dataset has 200 tasks.
We deﬁne our primitive objects and relations in Table 6.
To generate the datasets 9 , our engine consists of the following components:
Concept (Object)
As shown in Table 6, we deﬁne several shape primitives such as line, reactangle,
L-shape, and random shape.
Our data generation framework allows conﬁgurations such as color, width, height and orientation.
We allow nine different colors, and four different orientations at maximum.
In addition, we have composite objects as well.
For instance, the “Lshape” consists of two lines with a ﬁxed relation between them.
We evaluate our models with these composite objects during inference time to evaluate their performance on recognizing composite objects based on primitive concepts.
Relation As shown in Table 6, we deﬁne several relation primitives such as “inside”, “enclose”, and “parallel”.
We use these relations to deﬁne the spatial relations between objects.
For example, “inside” means one object is inside of the other object.
A pair of objects may formulate multiple relations between them.
Likewise, a pair of objects may be unrelated as well giving there is no primitive relation between them.
Canvas Each of our examples in the corpus consists a canvas, where objects are positioned with relations between them.
Our canvas is a n × n grid world, where each pixel in the grid world is a colored pixel.
Our data generation framework can place objects onto the canvas with desired relations.
For example, our framework can generate two objects where one of them is inside of the other.
In the process, our framework samples a rectangle, and sample another object to be placed inside the rectangle.
In the meantime, the framework allows multiple object pairs to be deﬁned when placing objects.
In addition, the framework allows objects to be speciﬁed with pre-deﬁned attributes including shape, color, width, height and orientation.
We also allow other conﬁgurations such as whether we allow objects to touch each other, and whether objects have uniﬁed color.
Generation Process and Artifacts To generate training and evaluation sets for our experiments, we specify conﬁgurations of the canvas and repeatedly randomly generate canvas till we have enough examples.
In addition, we allow distractor sampling, where we specify the number and the shape of distractors.
Then, our data generation framework places distractors at random on the canvas.
Our framework parses the canvas and the relations between objects after all objects are placed onto the canvas.
Notice that the set of the relations between objects after the placement is a superset of the pre-deﬁned conﬁguration when creating the canvas.
For example, if we sample two objects sharing the same color along with another random distractor.
The relation between the distractor and one of the object can be free-formed.
We disallow non-ﬂatten conﬁguration when creating canvas.
For example, if there are three objects to be placed, we disallow circular relations speciﬁed between them ( i.e. , there exist a relation between each two objects).
We disallow two objects touching each other share the same color.
To generate 3D images, we ﬁrst generate the 2D images, and then use a standard tool of povray 10 to build 3D scenes based on the contour of the 2D image.
We make sure that in the 2D to 3D transfer dataset, there is no overlap between the generated 2D images and the images used to generate 3D images by using a distinct seed.
Design of dataset HD-Concept The main goal of this dataset is to test whether our method can detect different relational graph structure given the same number of concept nodes.
Concretely, given 2 “rectangle” concept instances and one “E-shape” instance, there are limited ways to form a compositional concept with different relational graph structures: Concept1 is where “E-shape” is not inside any of the “rectangle”.
Concept2 is “E-shape” and “rectangle” 1 are both inside the other “rectangle” 2 , and “E-shape” not inside “rectangle” 1 .
Concept3 is that “Eshape” is enclosed by one “rectangle” which is also enclosed by another “rectangle”.
All three compositional concepts have the same number of constituent concepts but different relation graphs.
Example images datasets are shown in Fig.
4, Fig.
7 and Fig.
A.10 Limitations of current work
In this work, we have demonstrated the zero-shot concept recognition and acquisition capability of ZeroC, with experiments in a grid-world domain.
We focus on grid-world since it provides a systematic and challenging testbed to evaluate the above two capabilities of models, similar to many other pioneering works that evaluate their models in grid-world that captures the essence of the problem ( e.g. RL in early days [29], PrediNet [3], program synthesis evaluated with Karel dataset [30], BabyAI [31], Machine Learning Theory of Mind [32], etc.)
Nevertheless, the fact that our main experiment is in grid-world is a limitation of current work.
In the main Sec 3.2, we train ZeroC 2 with 3D images, and the fact that ZeroC 2 is able to zero-shot classify and detect in its domain demonstrate that ZeroC is able to handle more complex 3D images.
Moreover, in Appendix A.12, we perform additional zero-shot classiﬁcation with a variant of CLEVR, which shows that our ZeroC is able to handle more realistic 3D images, and out-perform strong baseline of CADA-VAE.
Another limitation of current work is that we have only considered one hierarchy of composition, without considering more hierarchies.
Although more hierarchies is in principle possible with the
ZeroC method, it is out of scope of this work, since in this work we focus on demonstrating whether zero-shot recognition and acquisition of concepts are possible with our model.
It is an exciting future work to build on current work, to explore zero-shot recognition with multiple levels of hierarchies.
Our ZeroC architecture naturally supports continued expansion or compression of the EBM pool, as newly learned compositional concept EBMs can be dynamically added to the pool.
Independently trained EBMs on new concepts/relations can also be added to the pool and composed together with existing EBMs.
This is an exciting future direction, but is out-of-scope of the present paper, since here we focus on proposing the framework and demonstrate the zero-shot recognition and acquisition capability of ZeroC.
A.11 Broader social impact
Here we discuss the broader social impact of our work, including its potential positive and negative aspects.
On the positive side, the capability of ZeroC enables will improve the generalization capability of deep learning models, allowing them to acquire concepts and address more diverse tasks at inference time.
This provides a possible method to address the long-standing problem that deep learning models has limited generalization capability and mainly learn via examples.
Our ZeroC also improves interpretability of models, since we can know exactly the structure of the concepts the model learns, allowing us to know how the model makes such decision.
We see no obvious negative social impact of this work.
In its current state, ZeroC’s capability is still very limited, not nearly addressing any of the tasks as good as human level.
It will only improve the interpretability and versatility of the models, which can be used to better address challenging tasks in society.
A.12 Additional experiments
Here we perform additional experiments with a CLEVR dataset to explore whether ZeroC can generalize to more realistic images.
The models are trained with basic concepts of “Red”, “Cube”, “Large” and relations of “SameColor”, “SameShape”, “SameSize” (See Fig. 9), where the models are provided with tuples of ( image , mask ( s ) , label ) for concept or relation.
During inference, given only symbolic speciﬁcation of three hierarchical concepts, the model needs to zero-shot classify whether an image contains an instance of such concept.
For example, HConcept1 is deﬁned as three objects where one object is red, the second object has the same color as the ﬁrst object and the third object has the same shape as the second object.
The other two hierarchical concepts have a similar structure of speciﬁcation.
We used 100K images for training and 200 for inference (evaluation).
The table below shows the results of our model and comparison with CADA-VAE and the statistics baseline.
We see that ZeroC achieves signiﬁcantly higher performance on the task than the baselines, and able to zero-shot classify more realistic images.
[Table]
A.13 Generality of ZeroC approach
The ZeroC framework is quite general.
The generality lies in the following two folds:
Generality of learning the elementary concepts and relations The EBMs in the ZeroC framework can learn general primitives as long as labeled data is provided to demonstrate the concept or the relation, even if that concept or relation is a range that contains some intrinsic variation.
For example, to learn the “acute angle” relation between two lines (with varying angles), ZeroC only needs a dataset that contains many (image, mask1, mask2, “accute-angle”) tuples where the mask and mask2 identify the two lines in the image that form an acute angle, with different examples containing different angles.
In other words, as long as the dataset contains enough data that identiﬁes a concept/relation in a certain range, the ZeroC can learn such primitives.
This is also shown in the
HD-Concept dataset in Section 3.1, where “inside”, “outside”, “non-overlap” relation primitives are learned, and each relation has intrinsic variation.
For example, the two masks for “inside” relation can have different positions, relative positions, and sizes.
Generality to different datasets and scenarios Our architecture is general to learn diverse concepts and relations.
For all our experiments, we use the same network architecture (Appendix A.4), for the (1) dataset HD-Letter, (2) dataset HD-Concept (that contains more complex concepts and relations e.g. “rectangles”, “Eshape”, “inside”, “enclose”), (3) Section 3.2 “Acquiring Novel Hierarchical
Concepts Across Domains”, and (4) CLEVR experiment in Appendix A.12.
This shows that the algorithm is very general, not tuned toward speciﬁc concepts or scenarios.
The architectures only differ in the number of input channels (since the 2D images have 10 channels and 3D images have
RGB channels).
For future work, we can also experiment with using a single model to learn concepts or relations across datasets, which is out-of-scope of current work.
A.14 Scalability of ZeroC
Scalability to task complexity We have demonstrated that even for larger images like the 3D image (32x32x3) in Section 3.2 and CLEVR in Appendix A.12 (64x64x3), our approach achieves reasonable accuracy, signiﬁcantly outperforming baselines.
This shows the scalability of our methods to larger images and realistic use cases.
In addition, downsampling the images to lower resolution can be performed to reduce the complexity of the inference and learning.
Scalability in terms of time complexity In terms of time complexity, the SGLD inference algorithm (Alg. 3) uses a ﬁxed number of iteration steps K .
We ﬁnd
that K = 60 is enough for reasonable detection accuracy for larger images.
For larger images, a single step of SGLD make take slightly longer to run due to the larger image size.
For parsing hierarchical concept from image (Alg. 1), the detection of all concept instances can be obtained for a single SGLD run, and for the classiﬁcation of relations, we can concatenate all pairs of concept masks into a single minibatch and feed into the relation-EBM, which only requires one relation-EBM forward run, which is instant.
Thus we see that the time complexity is fairly constant for increasing number of objects in the images and larger image size, measured in terms of a single forward or SGLD step.
Potential application to real world images In order to scale to recognizing real-world objects, the
ZeroC architecture and pipeline will be able to do that in principle.
The main bottleneck is presented by labeled data, as we need to have detailed labels for many elementary concepts and relations that constitute real world objects.
For example, the CUB-200-2011 dataset [33] provides annotations for elementary concepts for birds, e.g. beak, belly, tail, etc.
This dataset lacks relation annotations, so is unsuitable for our pipeline.
A suitable dataset for our pipeline can be like an augmented dataset to the above CUB-200-2011 dataset that also has relation annotations like “connect-to”, “up”, “down”, “extend”, etc.
With such annotations, we can learn both concept EBMs and relation EBMs and compose together to recognize compositional concepts like different species of birds.
A.15 Computational complexity of ZeroC
The computational complexity of the inference is detailed in Appendix A.14.
In summary, the run time remains fairly constant increasing number of objects in the images and larger image size, measured in terms of a single forward or SGLD step.
Since learning take the inference as an inner loop, it also remains fairly constant for the same model structure.
For increasing EBM model size, the increased number of parameters will deﬁnitely require more time to train, as is typical for deep learning models.
Empirically, we observe that for the same model architecture, it takes similar number of epochs to learn reasonably for different datasets. Training
: rectangle/E-shape
Inference: Detecting Concept3 from distractor, given x and concept3’s concept graph
Inference: Detecting Concept2 from distractor, given x and concept2’s concept graph
Training relations: non-overlap/inside/enclose
Inference: Classifying Concept1/Concept2/Concept3, given only x and the each of its concept graph
Inference: Detecting Concept1 from distractor, given x and concept1’s concept graph x mx mmx mx mx mx m
E-shape
E-shape E-shape
E-shape
E-shape rectangle rectangle
E-shape non-overlap non-overlap inside non-overlap inside inside inside non-overlap c r concept concept concept concept concept concept concept concept
[Figure]
2 D c o n c e p t 3 D
i m a g e
2 D c o n c e p t 3 D
i m a g e
2 D c o n c e p
t 3 D
i m a g e
2 D c o n c e p
t 3 D
i m a g e
2 D c o n c e p
t 3 D
i m a g e
2 D c o n c e p
t 3 D
i m a g e
2 D c o n c e p
t 3 D
i m a g e
2 D c o n c e p
t 3 D
i m a g e task#1 task#
[Figure]
Red:
Cube:
Large:
SameColor:
SameShape:
SameSize: t r a i n
I n f e r e n c e G i v en
C l a ss i f y ?
(gt: HConcept2)
Hconcept2 =
LargeSameSize
SameColor
HConcept1 =
Red
SameShape SameColor
Hconcept3 =
Cube
SameSize SameShape ?
(gt: HConcept1) (gt: HConcept3) ?