ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time
Abstract
Humans have the remarkable ability to recognize and acquire novel visual concepts in a zero-shot manner.
Given a high-level, symbolic description of a novel concept in terms of previously learned visual concepts and their relations, humans can recognize novel concepts without seeing any examples.
Moreover, they can acquire new concepts by parsing and communicating symbolic structures using learned visual concepts and relations.
Endowing these capabilities in machines is pivotal in improving their generalization capability at inference time.
In this work, we introduce Zero-shot Concept Recognition and Acquisition (ZeroC), a neuro-symbolic architecture that can recognize and acquire novel concepts in a zero-shot way.
ZeroC represents concepts as graphs of constituent concept models (as nodes) and their relations (as edges).
To allow inference time composition, we employ energy-based models (EBMs) to model concepts and relations.
We design
ZeroC architecture so that it allows a one-to-one mapping between a symbolic graph structure of a concept and its corresponding EBM, which for the ﬁrst time, allows acquiring new concepts, communicating its graph structure, and applying it to classiﬁcation and detection tasks (even across domains) at inference time.
We introduce algorithms for learning and inference with ZeroC.
We evaluate ZeroC on a challenging grid-world dataset which is designed to probe zero-shot concept recognition and acquisition, and demonstrate its capability.
(a) training of models !
# % , % - %(b) inference:given a hierarchical concept graph,derive its model from trained models 3(4,5,6 789:;< )= !
+!
+!
+∑ AB,,-,2 !
(#,% A ,& '()* ) !
/ & '()* & '()* & '()* % , % - %(c) detection at inference: combined mask % : k= k= k= k= k= k= k= k= k= k=
Introduction
Humans learn in diverse ways.
Besides learning from demonstrations of a novel concept, humans can also learn concepts on a high-level.
Consider learning the “rectangle” concept, for example.
Suppose that one has never seen such a concept, but has already mastered the concept of “line” and relations of “parallel” and “perpendicular”; s/he can easily master the “rectangle” concept when told that a “rectangle” consists of two pairs of “lines”, the lines within the pairs are “parallel,” and the lines between the pairs are “perpendicular”.
Then s/he can directly use this newly mastered concept to recognize “rectangles” in novel images.
Such zero-shot concept recognition capability is still beyond the reach of deep learning models, which require many examples (as in typical supervised learning), or many tasks drawn from the same distribution (as in few-shot learning) to learn a novel concept.
Moreover, along with the above symbolic to instance direction, humans can easily do the reverse .
Suppose again that we haven’t seen the “rectangle” concept, but have already mastered concepts of “line” and relations of “parallel” and “perpendicular”.
When seeing a novel image which contains an instance of a rectangle, humans can easily decompose it into its constituent concepts of “lines” and their relational graph structure (from instance to symbolic).
Moreover, humans can then communicate with each other about this new concept, allowing the transfer of knowledge in a high-level and in a succinct way.
Such zero-shot concept acquisition capability at inference time is beyond the reach of today’s machine learning and
AI systems.
Above all, endowing the above two capabilities of zero-shot concept recognition and acquisition capability to machines at inference time, will allow them to tackle more diverse tasks.
Past efforts have attempted to address aspects of the above problem of recognizing and acquiring novel concepts at inference time in a zero-shot manner.
An essential component is compositionality, i.e. the ability to compose new concepts from elementary ones.
[1] and [2] introduce techniques for compositional generation in the context of energy-based models (EBMs) and variational autoencoders (VAEs), respectively.
However, these works only consider composing factors of variation ( e.g. , color, position, smiling, young).
In order to recognize concepts that consist of constituent concepts as parts, the structure of objects and their relationships are equally important.
For instance, given 4 “line” concepts, without specifying their relational graph structure, they can easily form a trapezoid or a generic parallelogram.
[3] designed a transformer-based relational architecture for learning explicit relations, and showed that it can generalize to new objects for the relation.
However, this architecture cannot generalize to concepts with more complex relational graph structure.
It remains an open problem of how to design an architecture that has the capability to compose novel concepts at inference time, based on its internal relational graph structure of prior learned concepts and relations.
A separate line of works [4, 5, 6, 7, 8] tackle zero-shot classiﬁcation, but are limited to novel composition of label features for concepts, and ignore relation features as well as the graph structure formed by concepts and relations.
Here we introduce our ZeroC architecture, which is a model designed towards zero-shot concept recognition and acquisition.
It models a visual concept or relation as an Energy-Based Model (EBM) with 3 inputs: an image, a mask (or two masks for relation) denoting a set of pixels within the image, and a concept/relation label string.
It returns a low scalar energy value if the mask(s) correctly indicates the concept or relation within the image, and a high energy value otherwise.
After training, the concept and relation EBMs are able to perform classiﬁcation of concept/relation labels given an image, and detection of concept/relation mask(s) given an image and concept/relation label.
The core contribution of this work is ZeroC’s architecture and inference algorithms that allow a one-to-one correspondence between a speciﬁcation of a relational graph structure of a novel concept and its corresponding composed concept model that can also perform similar downstream classiﬁcation and detection, which enables zero-shot concept recognition and acquisition at inference time.
To perform zero-shot concept recognition at inference time, ZeroC can simply compose a new concept EBM, by summing over the constituent concept and relation EBMs according to the speciﬁed graph structure.
This new concept EBM has the same input-output format as its constituent concept
EBMs, allowing the same capability of classiﬁcation and detection.
Note that through the above process, ZeroC have mastered a concept without seeing an image, only symbolic instruction, thus it is called “zero-shot”.
To perform zero-shot concept acquisition, we introduce an algorithm that given an instance (image) of a novel concept, parse it into a relational graph of prior learned concepts and relations.
Such relational graph can then be transferred, even cross-domain, allowing models independently trained in a different domain to be able to zero-shot classify and detect this concept in its own domain.
There are currently no specialized benchmarks to test such zero-shot concept recognition and acquisition capabilities.
Inspired by the Abstraction and Reasoning Corpus (ARC) [9], we have created a synthetic, grid-world based dataset with tasks that capture the essence of the above capabilities that are deceptively simple for humans, but very hard for neural models.
We demonstrate that our model, trained to classify and detect elementary visual concepts and relations, is able to classify and detect novel concepts at inference time, being given a zero-shot symbolic graph structure.
We also compare with a state-of-the-art zero-shot learning model CADA-VAE [8].
Due to that it is not designed for such graph-based composition, it signiﬁcantly underperforms our ZeroC.
Furthermore, we show that two independently trained models, one on 2D images and the other on 3D images, are able to acquire hierarchical concepts from each other by communicating the graph structure and then perform classiﬁcation and detection of these hierarchical concepts in their respective domains.
Method
In this section, we describe the ZeroC architecture for recognizing and acquiring novel visual concepts at inference time 2 .
We give an overview of our approach, describe how it performs classiﬁcation and detection, how it supports zero-shot concept recognition, how independently-trained models can acquire hierarchical concepts from each other and, ﬁnally, introduce its learning method.
2.1 An Overview of ZeroC Architecture
The key components of ZeroC are concepts and relations.
Each concept consists of a graph and an energy-based model.
The concept graph describes the concept as a composition of its constituent concepts and relations between them.
The base concepts that do not have any constituent concepts are called elementary concepts.
Their graph is a singleton.
The concept energy-based model is used to recognize the concept in the input data.
Each relation is represented in ZeroC also with a graph and an energy-based model.
The relation graph is simply an edge that connects the two related concepts.
A hierarchical concept is a concept composed of constituent concepts as nodes and relations as edges according to a graph structure.
During training, energy-based models are learned from images and concept or relation labels.
During inference, the learned models are used to recognize concepts seen during training or more complex hierarchical concepts which were not seen before.
For concepts seen during training, their learned models are applied.
For new hierarchical concepts not seen during training ( i.e. , without learned model), their models are derived from the graph of the new hierarchical concept and energy-based models of their constituent concepts and relations.
Concepts and relations in ZeroC can be viewed as templates for objects and their connections, which then get grounded during inference with a speciﬁc image, where those objects and relations are assigned actual values.
ZeroC also handles object masks, which indicate object locations in the image.
2.2 Zero-shot Concept Recognition and Acquisition
Formally, we model a concept with E X,M,C ( x, m, c ) 3 , which maps an image x 2 R D , a mask m 2 [0 , 1] D and a concept label 4 c to a scalar energy.
Similarly, we model relations with
E X,M 1 ,M 2 ,R ( x, m 1 , m 2 , r ) , where m 1 and m 2 are a pair of masks indicating two objects in the image x , and r is a relation label between the objects.
The concept and relation models have a probabilistic interpretation.
For example, the energy function E X,M,C ( x, m, c ) corresponds to the joint probability of
[Equation]
where Z C =
[Equation]
e  E X,M,C ( x,m,c ) dx dm is a normalizing constant.
Therefore, if the mask m is actually masking an object that belongs to concept c in image x , then P X,M,C ( x, m, c ) will be high and E X,M,C ( x, m, c ) will be low, and vice versa.
Essentially, the models deﬁne an energy landscape for their respective multi-modal inputs that give low energy if the mask correctly identiﬁes its corresponding concept.
Next, we show how ZeroC performs detection, classiﬁcation, and models hierarchically composed concepts.
We use the term HC-EBM to denote concept models and R-EBM for relation models.
Detection.
We want to infer the location mask m of the concept c , given image x .
Probabilistically, we are computing P M | X,C ( m | x, c ) .
To perform detection, we employ Stochastic Gradient Langevin
Dynamics (SGLD) to sample masks on the energy landscape [10]:
[Equation]
where ˜ m k is the inferred mask at the k th iteration, k = 1 , 2 , ...K .
Applying [11], as
K ! + 1 and  ! 0 , we generate samples from the distribution of P M | X,C
( m | x, c ) = Z x,c exp (  E X,M,C ( x, m, c )) , where Z x,c = R exp (  E X,M,C ( x, m, c )) dm is a normalizing constant.
In practice, we use a ﬁnite K to generate samples 5 ˜ m Kn , n = 1 , 2 , ...
N given x, c .
Classiﬁcation.
We want to determine whether the concept c appears in a given image x , i.e. compute
P C | X ( c | x ) .
We need to marginalize over the mask m :
[Equation]
We again use SGLD [10] in Eq.
(1) to generate N samples ˜ m Kn , n 2 [ N ] = { 1 , 2 , ...N } given x, c , and approximate the above integral using maximum a posteriori (MAP) estimation:
[Equation]
In this paper, we use i to index different concepts, j to index relations, and n to index example images.
In practice, we only need to ﬁnd the concept with the highest value in the numerator.
Zero-shot recognition of novel concepts.
To master a novel hierarchical concept and directly use it for classiﬁcation and detection only given its relational graph structure, we need a way to compose the previous concept and relation energy based models.
Here we introduce the hierarchical composition rule, using an English letter “F” as an illustrative example 6 .
The concept F has one constituent concept, a "line", and two relations, "parallel" and "perpendicular".
The models for constituents are learned during training.
During inference, these models are combined, using the concept graph, into a combined energy model for the letter
F. Essentially, the models of all the recognized constituent concepts and relations are added together.
Note that although F contains three lines, only one "line" model is needed.
The concept graph acts as a template and recognized line instances (objects) are matched with nodes in the template to obtain actual models.
The "line" model is instantiated three times with speciﬁc values for the three identiﬁed lines and the three models plus their corresponding relations models are used to derive the hierarchical model.
In addition to the models, we also need to combine the masks of all the recognized constituent objects.
Formally, we deﬁne the following composition rule for HC-EBMs plus the masks to be combined.
Deﬁnition 2.1.
Hierarchical Composition Rule: Let a hierarchical concept c have graph G = (
C , R ) , where C are constituent concept nodes and R are relation edges.
During inference, these nodes and edges are matched to recognized concept objects and their relationships, which provides their models and masks.
The combined model E X,M,C ( x, m, c ) is then a sum of the models for all the nodes and edges in the graph and the mask M is the maximum of all the masks:
[Equation]
2.3 Zero-shot Concept Acquisition at Inference Time
Here, we introduce techniques for acquiring novel concept graph, which is useful for sharing high-level knowledge transfer between independently trained models.
Such capability may have implications in future scenarios where a hypothetical self-driving car communicating the structure of a novel road sign to other cars on the road, or a 2D vision model learning a novel object and transferring it to a more accurate 3D depth model for inference.
To achieve this, we need to establish an equivalence relation between concepts in different domains ( e.g. in 2D and 3D images).
Deﬁnition 2.2.
Structural Equivalence for HC-EBM:
Two HC-EBMs are structurally equivalent, if their graphs are isomorphic, and their constituent HC-EBMs are recursively structurally equivalent.
For example, HC-EBMs for a “rectangle” concept in 3D and 2D images are structurally equivalent if they have the same decomposition into two “parallel-line” HC-EBMs connected by a “connect”
R-EBM, and the “parallel-lines” have the same decomposition into two “line” HC-EBMs connected by a “parallel” R-EBM.
Even though the HC-EBMs are grounded in different domains, they have the same abstract structure, so they represent the same hierarchical concept.
With structural equivalence, independently-trained HC-EBMs from different domains can acquire hierarchical concepts via the graph structure.
The algorithm works as follows (see Appendix A.5): the image is parsed to decompose it into a graph of concepts and relations previously learned by
HC-EBM and R-EBM, then the graph is used in a different domain to compose its HC-EBM for the new concept.
The key step of this process is the ﬁrst, parsing step.
The parsing step can be seen as an inverse of the Hierarchical Composition Rule and is critical in allowing the model to recognize novel hierarchical concepts in its domain using prior-learned concepts and relations from other domains, thus facilitating downstream knowledge transfer between domains.
Alg. 1 provides further details.
Steps 1-2 infer the concept instances in image x using
SGLD (Eq. 1).
Here the energy E ( p ) ( x, M C , c ) is the summation of HC-EBMs on independent masks, each mask m i l representing a potential concept instance belonging to concept c i .
E (overlap) = max [Equation]
penalizes overlapping masks.
Since the image probably contains fewer concept instances than given, some masks are empty with all-zero values; Step 3 removes these.
Require: HC-EBM E ( C ) with prior-learned concepts { c 1 , c 2 , ...
c I } , R-EBM model
E ( R ) with prior-learned relations { r 1 , r 2 , ... r J } . Require: Image x , containing unseen hierarchical concept c . Require: Maximum number of instances N i for concept c
I .
P
+
1 E (
( M C
{ m
} , i
N i
= 1 , ...
I , is the set of all masks. G Step 4 classiﬁes relations (using Eq. 2) between pairs of detected concept instances. Step 5 combines the detected concept instances and their relations to build the graph G for this new hierarchical concept c .
2.4 Learning
Since a standard EBM-training method yields poor performance as shown later in our experiments, we describe here our approach to train the models.
Further details of our training algorithm are given in Alg. 2 in Appendix A.1.
We are optimizing the following objective:
[Equation]
The expression of each term is given in Appendix A.2.
The ﬁrst loss L (Improved) is the objective proposed in [12], a state-of-the-art EBM training technique.
However, to enable the challenging zero-shot concept recognition and acquisition, we need more suitable inductive biases.
The next three terms inject the right inductive bias for the task.
L (pos-std) makes sure that the positive energy have similar level, so the composed concept EBM can identify all constituent concepts and relations, without one constituent EBM having too low energy and only recognizing it.
The empty-mask regularization L (em) makes sure that the when the mask is empty, its energy is between the positive and negative energy, which we prove in Appendix A.3 that it is the necessary condition for correctly discover the underlying graph in Sec.
The L (neg) additionally provide additional negative examples, encouraging discovering full concept instead of a part of it (see Appendix A.2 for details).
Experiments
In this section, we set out to answer the following two questions: (a) given a speciﬁcation of graph structure for a novel hierarchical concept, can the composed concept model successfully perform classiﬁcation and detection tasks?
(b) Can HC-EBM acquire a novel concept from HC-EBM trained from another domain?
We also evaluated our model in a challenging setting by comparing with other existing approaches in a controlled and systematic way.
Since there are no suitable dataset, we designed a HierarchicalConcept corpus, a dataset based on grid-world images.
The dataset comes with testing images for the classiﬁcation and detection tasks which contain more complex, hierarchical concepts, composed from simpler concepts in the training set (See Appendix A.9 for details).
The concept instances on our images have varying locations, size, and relative positions, making them more difﬁcult than ﬁxed-location ﬁxed-size inference tasks such as in [3].
We also provide 2D and 3D versions of images for demonstrating concept acquisition across domains.
[Table]
3.1 Zero-shot Classiﬁcation and Detection of Novel Concepts
To test recognition of novel concepts
[Table]
we designed two datasets consisting of different concept and relation types.
The HD-Letter hierarchical dataset consists of concept instances of “line” and relation instances of “parallel”, “perp-edge” (perpendicular and touching edge), “perp-mid” (perpendicular and touching middle), together with distractor objects.
Examples are provided in the form of 3-tuples ( image, mask, concept ) for concepts, and ( image, mask 1 , mask 2 , relation ) for relations.
ZeroC models are trained via objective Eq.
(See Appendix A.9 for examples of training datasets).
At inference time, the models need to perform classiﬁcation and detection on novel images with more complicated English characters of “E-shape”, “F-shape” and “A-shape”, given their structure graphs with up to 4 nodes and 6 edges.
For detection, the images also contain a few distractors (concepts unrelated to the ones the model is predicting).
The model is asked to return a mask indicating which pixels belong to an instance of the speciﬁed novel concept.
Although looking simple, this is actually a very challenging task.
because in order to solve the problem of classifying/detecting hierarchical concepts, the model needs to solve a subgraph isomorphism problem, which is NP-hard.
For example, take the problem of detecting “E” shape in an image with distractor of “T” and “Rectangle” (Fig.
2 (a) ﬁrst subﬁgure), a graph with 10 nodes (lines), 45 possible edges.
Note that during training time, a model has only learned “line” concept, and relations “parallel”, “perp-edge” (perpendicular, touching edge), and “perp-mid”, but not the overall “E” shape.
At inference time and given the “E” concept graph with 4 nodes (lines), 6 edges (relations), the model needs to ﬁnd the “E” concept subgraph within the large image graph.
This involves C 410 ⇥ 4!
=
5040 possible mask assignments.
Additionally, a model may not perfectly detect the masks for low-level concepts.
For more complex concepts and relations, we designed the HD-Concept hierarchical dataset, which consists of training concepts of “E-shape” and “rectangle”, and training relations of “inside”, “enclose” and “non-overlap”.
The hierarchical concepts to be classiﬁed and detected are three characters which we term Concept1,
Concept2 and Concept3, as the ground-truth masks in Fig.
2 (a) indicate.
The three hierarchical concepts have the same multiset of concepts, but a different relation structures.
As the closest setting to our classiﬁcation task is zero-shot classiﬁcation, we compare ZeroC to a state-of-the-art zero-shot learning algorithm CADA-VAE [8], which we adapted to our setting, using the set of concepts and relations as feature embeddings to represent the concept graph (see Appendix
A.6 for details).
Additionally, we evaluated a “Statistics” baseline which samples random pixels on the image based on the global statistics of pixel occurrence, and a “Heuristics” baseline that randomly chooses a same-color connected object.
We use a CNN-based architecture for ZeroC, as given in
Appendix A.4.
We add comparisons with ablation of different aspects of ZeroC, including without
HC-EBM terms or R-EBM terms in the composition rule of Def.
2.1 (Eq. 3), and without different regularization terms of the objective in Eq. 4.
Fig.
2 shows a demonstration of this experiment.
From performance results in Table 1, we see that ZeroC achieves classiﬁcation accuracy of 84.5% and 70.5% on HD-Letter and HD-Concept, respectively, both higher than CADA-VAE.
The gap is more signiﬁcant in HD-Letter, where there is a larger distribution shift for hierarchical concepts (as can be seen in Appendix A.9), which the graph for F graph for E graph for A
E=-1.
combined mask
TotalEnergy
E=-0.
E=0.
Individual masks graph for Concept
E=0.
combined mask
TotalEnergy
E=0.
E=-0.
Individual masks graph for Concept graph for Concept (b) Classification: (a) Detection: correct!
correct!
graph for A
IoU: 0.
graph for E
IoU: 1.
inferred ground-truth graph for F
IoU: 0.
inferred ground-truth
Concept
IoU: 0.
Concept
IoU: 0.
inferred ground-truth
Concept
IoU: 0.
[Table]
joint embedding of the image and features learned by CADA-VAE is insufﬁcient to handle.
The reason for the low accuracy is that CADA-VAE is not able to address these out-of-domain distribution shifts (See Appendix A.7 for details).
During inference, its embedding (multi-hot vector) for the graph structure can contain up to 10 hots (4 lines, 6 edges), while during training, it is only up to 1-hot.
This example also shows the intrinsic difﬁculty of the task.
In addition, Table 1 and Fig. 2 (a) show that ZeroC is able to detect the hierarchical concepts in the presence of distractors, and that its performance is signiﬁcantly better than “Statistics” and “Heuristics” baselines.
The results of our ablation studies in Table 1 show that both the HC-EBM and R-EBM terms in Eq.are key to successful classiﬁcation and detection.
This is likely because without relations and using only concept terms, ZeroC may lose its ability to distinguish between different hierarchical concepts; for example, both “E-shape” and “A-shape” have 4 lines but different relation structures.
Moreover, we see that both L (pos-std) and L (neg) improve classiﬁcation and detection performance.
3.2 Acquiring Novel Hierarchical Concepts Across Domains
We show that ZeroC can acquire novel hierarchical concepts across models and domains (Fig. 3).
We train a ZeroC model in domain ZeroC 1 , where the images are 2D and one-hot color-coded, containing the the same training concepts and relation types as in HD-Letter in Sec. 3.1.
Independently , we train another model in a different domain ZeroC 2 , where the images contain the same set of concepts and relations, but are viewed in 3D from a certain camera angle, have larger size, and use RGB colors.
At inference time, each test task consists of a tuple of three images showing the hierarchical concepts in the ﬁrst domain to ZeroC 1 .
ZeroC 1 is only allowed to send symbolic information, up to a few bits, to
ZeroC 2 .
Then ZeroC 2 performs classiﬁcation and detection tasks on three example images in the second domain.
In addition, we also evaluate whether ZeroC 1 can parse the concept graph of the hierarchical concept correctly, using the metric of graph edit distance
d edit and graph-isomorphism accuracy w.r.t.
the ground-truth.
Note that the graph-isomorphism accuracy is a stringent metric,
ZeroCparse cccdiscovered graph d edit = d edit =
d edit =
2D concept demonstration channel ccc
E=1.
E=1. E=0. E=0.
E=0. E=0. E=0.
E=0. E=1.
3D image
ZeroCinferred: which is only 1 if the parsed graph is isomorphic to the ground-truth, and 0 otherwise 7 .
We compare with a strong baseline of Mask R-CNN [13] for object detection together with relation classiﬁcation.
The relation classiﬁcation network is a CNN which takes as input a pair of embeddings produced by the Mask R-CNN, and predicts the label of the relation (see Appendix A.8 for details).
We did extensive tuning of the Mask R-CNN approach to obtain its best performance.
We also evaluate the performance of ZeroC 2 given the ground-truth structure graph, as an ideal scenario of perfect parsing, providing an upper bound for the performance.
Table 2 shows that ZeroC 1 achieves 72.7% graph isomorphism accuracy for parsing the hierarchical concepts in the ﬁrst domain, signiﬁcantly higher than Mask R-CNN
+ relation classiﬁcation.
After sending the concept graphs in the second domain, ZeroC 2 achieves a classiﬁcation accuracy of 60.7% and detection IoU of 94.4%.
Without this information and relying on global statistics, the classiﬁcation accuracy of ZeroC 2 is 33.3%.
This demonstrates that ZeroC 2 is able to acquire novel hierarchical concepts at inference time, from an independently trained model from a different domain .
The ablation study shows that L (em) , L (neg) and L (pos) all contribute to more accurate parsing.
The standard technique of EBM training [12] are insufﬁcient to achieve good parsing, leading to less accurate transfer of hierarchical concepts.
We also see in the ablations that even without perfect parsing from ZeroC 1 , the reduction of classiﬁcation accuracy for ZeroC 2 is small, showing that it is able to classify under noisy speciﬁcation of the concept graph.
We discuss further on the generality (Appendix A.13), scalability (Appendix A.14) and computational complexity (Appendix A.15) in respective Appendix sections.
Related Work
Our work relates to visual compositionality, concept and relation learning, and zero-shot learning.
Visual compositionality.
Compositionality is a key for addressing diverse tasks given ﬁnite basic knowledge.
Some approaches use composition EBMs for generation [1], and a VAE-based architecture for bi-directional symbol-image generation that can also learn logical recombination of concepts [2].
These two works focus on composing factors of variation , e.g. color, position, smiling, young.
In contrast, our work focuses on concepts that abstract objects, where the internal hierarchical relational structure is key.
Moreover, while the above works focus on generation, our work focuses on the tasks of classiﬁcation and detection.
A novel Bayesian-based method for few-shot learning on the
Omniglot dataset shows that compositionality is pivotal for improved performance [14].
They achieve compositionality via hierarchical MCMC sampling on hand-coded priors of elementary concepts and relations.
In comparison, our work is neural network-based and only requires demonstrations of elementary concepts and relations, reducing hand-coded priors.
Another approach introduced a modular neural network, which uses composition of neural modules for visual question answering [15].
Their method can be seen as composing transformations on a representation.
In comparison, our composition is achieved via composition of energy landscapes.
While they focus on question answering on scene graphs, we focus on classiﬁcation and detection of hierarchical concepts.
Concept and relation learning.
There has been exciting progress in concept learning and relation learning.
Works in concept learning generally represent concepts in latent space via prototypes [16, 17], or via latent embedding such as SCAN [2], InfoGAN [18] and Neuro-Symbolic Concept
Learner [19].
[20] introduced EBMs to represent concepts with a demonstration in simple state space.
[21] further introduced unsupervised learning of local and global concepts with EBMs.
Regarding relation learning, non-local neural networks [22], Relation Networks [23], Neural Relational Inference [24] and C-SWM [25] use latent complete graphs as inductive biases to represent potential relations.
PrediNet [3] explicitly represents propositions, relations, and objects with a transformer-based architecture, and demonstrates that it can learn relations that generalize to novel shapes of objects and column patterns.
In comparison to the above works, ZeroC explicitly learns both concepts and relations, which has the unique capability to recognize and acquire hierarchical concepts at inference.
Zero-shot learning.
Zero-shot learning methods [4, 5, 6, 7, 8] typically learn a joint embedding space between image and feature labels, and perform classiﬁcation at inference time on images with novel classes, based on how the novel classes correspond to a set of features.
Our generalization task also requires generalizing to new concepts without seeing the image (zero-shot), but has the important distinctions that at inference time, our concepts to be inferred lie at a higher hierarchy than that in training, and furthermore use the structural concept graph.
In comparison, in standard zero-shot learning, concepts in training and inference lie at the same hierarchy level, and only generalize to new combinations of features (constituent concepts) while neglecting relation structures.
Conclusion
In this paper, we introduce ZeroC, a new framework for zero-shot concept recognition and acquisition at inference time.
Our experiments show that in a challenging grid-world domain, ZeroC is able to recognize complex, hierarchical concepts composed of English characters in a grid-world in a zero-shot manner, being given a high-level, symbolic speciﬁcation of their structures, and after being trained with simpler concepts.
In addition, we demonstrate that an independently trained ZeroC is able to transfer hierarchical concepts across different domains at inference.
Although this work is evaluated only in grid-world visual domain, we are the ﬁrst to address this difﬁcult challenge.
We are also excited to see its potential application in broader domains, e.g. in AI for scientiﬁc discovery, where it may infer novel patterns and concepts from data in a zero-shot manner.
We hope that this work will make a useful step in the development of composable neural systems, capable of zero-shot concept recognition and acquisition and hence suitable for more diverse tasks.
Acknowledgments and Disclosure of Funding
We thank Rui Yan and Bla ž Š krlj for discussions and for providing feedback on our manuscript.
We also gratefully acknowledge the support of DARPA under Nos.
HR00112190039 (TAMI),
N660011924033 (MCS); ARO under Nos.
W911NF-16-1-0342 (MURI), W911NF-16-1- (DURIP); NSF under Nos.
OAC-1835598 (CINES), OAC-1934578 (HDR), CCF-1918940 (Expeditions), NIH under No. 3U54HG010426-04S1 (HuBMAP), Stanford Data Science Initiative, Wu Tsai
Neurosciences Institute, Amazon, Docomo, GSK, Hitachi, Intel, JPMorgan Chase, Juniper Networks,
KDDI, NEC, and Toshiba.
The content is solely the responsibility of the authors and does not necessarily represent the ofﬁcial views of the funding entities.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
In conclusion and Appendix A.10.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] In Appendix A.11.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.
If you ran experiments
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The Appendix includes full details on model architecture, training and evaluation to reproduce the experimental results.
Code and data will be released upon publication of the paper.
(b) Did you specify all the training details ( e g. , data splits, hyperparameters, how they were chosen)?
[Yes] Full details to reproduce the experiments are included in the Appendix.
(c) Did you report error bars ( e.g. , with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used ( e.g. , type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix of respective experiment section.
If you are using existing assets ( e.g. , code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] We use our own created benchmark.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] In Appendix A.9, we describe the engine and training/testing tasks.
Full dataset will be released upon publication.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[-For example, with 2 objects in the image x + with concepts c +
-]
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]