ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time
Abstract
Humans have the remarkable ability to recognize and acquire novel visual concepts in a zero-shot manner.
Given a high-level, symbolic description of a novelconcept in terms of previously learned visual concepts and their relations, humanscan recognize novel concepts without seeing any examples.
Moreover, they canacquire new concepts by parsing and communicating symbolic structures usinglearned visual concepts and relations.
Endowing these capabilities in machinesis pivotal in improving their generalization capability at inference time.
In thiswork, we introduce Zero-shot Concept Recognition and Acquisition (ZeroC), aneuro-symbolic architecture that can recognize and acquire novel concepts in azero-shot way.
ZeroC represents concepts as graphs of constituent concept models(as nodes) and their relations (as edges).
To allow inference time composition, weemploy energy-based models (EBMs) to model concepts and relations.
We design
ZeroC architecture so that it allows a one-to-one mapping between a symbolicgraph structure of a concept and its corresponding EBM, which for the ﬁrst time,allows acquiring new concepts, communicating its graph structure, and applying itto classiﬁcation and detection tasks (even across domains) at inference time.
Weintroduce algorithms for learning and inference with ZeroC.
We evaluate ZeroCon a challenging grid-world dataset which is designed to probe zero-shot conceptrecognition and acquisition, and demonstrate its capability.
Introduction
Humans learn in diverse ways.
Besides learning from demonstrations of a novel concept, humans canalso learn concepts on a high-level.
Consider learning the “rectangle” concept, for example.
Supposethat one has never seen such a concept but has already mastered the concept of “line” and relationsof “parallel” and “perpendicular”; s/he can easily master the “rectangle” concept when told that a“rectangle” consists of two pairs of “lines”, the lines within the pairs are “parallel,” and the linesbetween the pairs are “perpendicular”.
Then s/he can directly use this newly mastered concept torecognize “rectangles” in novel images.
Such zero-shot concept recognition capability is still beyond the reach of deep learning models,which require many examples (as in typical supervised learning), or many tasks drawn from the same distribution (as in few-shot learning) to learn a novel concept.
Moreover, along with the above symbolic to instance direction, humans can easily do the reverse .
Suppose again that we haven’t seen the “rectangle” concept, but have already mastered concepts of “line” and relations of “parallel” and“perpendicular”.
When seeing a novel image which contains an instance of a rectangle, humans can easily decompose it into its constituent concepts of “lines” and their relational graph structure (from instance to symbolic).
Moreover, humans can then communicate with each other about this new
(a) training of models !
# % , % - %(b) inference:given a hierarchical concept graph,derive its model from trained models 3(4,5,6 789:;< )= !
+!
+!
+∑ AB,,-,2 !
(#,% A ,& '()* ) !
/ & '()* & '()* & '()* % , % - %(c) detection at inference: combined mask % : k= k= k= k= k= k= k= k= k= k= concept, allowing the transfer of knowledge in a high-level and in a succinct way.
Such zero-shotconcept acquisition capability at inference time is beyond the reach of today’s machine learning and
AI systems.
Above all, endowing the above two capabilities of zero-shot concept recognition andacquisition capability to machines at inference time, will allow them to tackle more diverse tasks.
Past efforts have attempted to address aspects of the above problem of recognizing and acquiringnovel concepts at inference time in a zero-shot manner.
An essential component is compositionality,i.e. the ability to compose new concepts from elementary ones.
[1] and [2] introduce techniques forcompositional generation in the context of energy-based models (EBMs) and variational autoencoders(VAEs), respectively.
However, these works only consider composing factors of variation ( e.g. , color,position, smiling, young).
In order to recognize concepts that consist of constituent concepts as parts,the structure of objects and their relationships are equally important.
For instance, given 4 “line”concepts, without specifying their relational graph structure, they can easily form a trapezoid or ageneric parallelogram.
[3] designed a transformer-based relational architecture for learning explicitrelations, and showed that it can generalize to new objects for the relation.
However, this architecturecannot generalize to concepts with more complex relational graph structure.
It remains an open problem of how to design an architecture that has the capability to compose novelconcepts at inference time, based on its internal relational graph structure of prior learned conceptsand relations.
A separate line of works [4, 5, 6, 7, 8] tackle zero-shot classiﬁcation, but are limited tonovel composition of label features for concepts, and ignore relation features as well as the graphstructure formed by concepts and relations.
Here we introduce our ZeroC architecture, which is a model designed towards zero-shot conceptrecognition and acquisition.
It models a visual concept or relation as an Energy-Based Model (EBM)with 3 inputs: an image, a mask (or two masks for relation) denoting a set of pixels within theimage, and a concept/relation label string.
It returns a low scalar energy value if the mask(s) correctlyindicates the concept or relation within the image, and a high energy value otherwise.
After training,the concept and relation EBMs are able to perform classiﬁcation of concept/relation labels given animage, and detection of concept/relation mask(s) given an image and concept/relation label.
The corecontribution of this work is ZeroC’s architecture and inference algorithms that allow a one-to-onecorrespondence between a speciﬁcation of a relational graph structure of a novel concept and itscorresponding composed concept model that can also perform similar downstream classiﬁcation anddetection, which enables zero-shot concept recognition and acquisition at inference time.
To perform zero-shot concept recognition at inference time, ZeroC can simply compose a newconcept EBM, by summing over the constituent concept and relation EBMs according to the speciﬁedgraph structure.
This new concept EBM has the same input-output format as its constituent concept EBMs, allowing the same capability of classiﬁcation and detection.
Note that through the aboveprocess, ZeroC have mastered a concept without seeing an image, only symbolic instruction, thus it is called “zero-shot”.
To perform zero-shot concept acquisition, we introduce an algorithm thatgiven an instance (image) of a novel concept, parse it into a relational graph of prior learned conceptsand relations.
Such relational graph can then be transferred, even cross-domain, allowing modelsindependently trained in a different domain to be able to zero-shot classify and detect this concept inits own domain.
There are currently no specialized benchmarks to test such zero-shot concept recognition and acquisition capabilities.
Inspired by the Abstraction and Reasoning Corpus (ARC) [9], we have createda synthetic, grid-world based dataset with tasks that capture the essence of the above capabilitiesthat are deceptively simple for humans, but very hard for neural models.
We demonstrate that ourmodel, trained to classify and detect elementary visual concepts and relations, is able to classifyand detect novel concepts at inference time, being given a zero-shot symbolic graph structure.
Wealso compare with a state-of-the-art zero-shot learning model CADA-VAE [8].
Due to that it is notdesigned for such graph-based composition, it signiﬁcantly underperforms our ZeroC.
Furthermore,we show that two independently trained models, one on 2D images and the other on 3D images, areable to acquire hierarchical concepts from each other by communicating the graph structure and thenperform classiﬁcation and detection of these hierarchical concepts in their respective domains.
Method
In this section, we describe the ZeroC architecture for recognizing and acquiring novel visual conceptsat inference time.
We give an overview of our approach, describe how it performs classiﬁcation anddetection, how it supports zero-shot concept recognition, how independently-trained models canacquire hierarchical concepts from each other and, ﬁnally, introduce its learning method.
2.1 An Overview of ZeroC Architecture
The key components of ZeroC are concepts and relations.
Each concept consists of a graph and anenergy-based model.
The concept graph describes the concept as a composition of its constituentconcepts and relations between them.
The base concepts that do not have any constituent conceptsare called elementary concepts.
Their graph is a singleton.
The concept energy-based model is usedto recognize the concept in the input data.
Each relation is represented in ZeroC also with a graph andan energy-based model.
The relation graph is simply an edge that connects the two related concepts.
A hierarchical concept is a concept composed of constituent concepts as nodes and relations as edgesaccording to a graph structure.
During training, energy-based models are learned from images and concept or relation labels.
Duringinference, the learned models are used to recognize concepts seen during training or more complexhierarchical concepts which were not seen before.
For concepts seen during training, their learnedmodels are applied.
For new hierarchical concepts not seen during training ( i.e. , without learnedmodel), their models are derived from the graph of the new hierarchical concept and energy-basedmodels of their constituent concepts and relations.
Concepts and relations in ZeroC can be viewed astemplates for objects and their connections, which then get grounded during inference with a speciﬁcimage, where those objects and relations are assigned actual values.
ZeroC also handles object masks,which indicate object locations in the image.
2.2 Zero-shot Concept Recognition and Acquisition
Formally, we model a concept with E X,M,C ( x, m, c ) 1 , which maps an image x 2 R D , a mask m 2 [0 , 1] D and a concept label 2 c to a scalar energy.
Similarly, we model relations with
E X,M 1 ,M 2 ,R ( x, m 1 , m 2 , r ) , where m 1 and m 2 are a pair of masks indicating two objects in the image x , and r is a relation label between the objects.
The concept and relation models have a probabilistic interpretation.
For example, the energy function E X,M,C ( x, m, c ) corresponds to the joint probability of
[Equation]
where Z C =
P
[Equation]
e  E X,M,C ( x,m,c ) dx dm is a normalizing constant.
Therefore, if the maskm is actually masking an object that belongs to concept c in image x , then P X,M,C ( x, m, c ) willbe high and E X,M,C ( x, m, c ) will be low, and vice versa.
Essentially, the models deﬁne an energylandscape for their respective multi-modal inputs that give low energy if the mask correctly identiﬁesits corresponding concept.
Next, we show how ZeroC performs detection, classiﬁcation, and models hierarchically composedconcepts.
We use the term HC-EBM to denote concept models and R-EBM for relation models.
Detection.
We want to infer the location mask m of the concept c , given image x .
Probabilistically,we are computing P M | X,C ( m | x, c ) .
To perform detection, we employ Stochastic Gradient Langevin
Dynamics (SGLD) to sample masks on the energy landscape [10]:
[Equation]
where ˜ m k is the inferred mask at the k th iteration, k = 1 , 2 , ...K .
Applying [11], as
K !
+ 1 and  ! 0 , we generate samples from the distribution of P M | X,C
( m | x, c )
=
1 Z x,c exp (  E X,M,C ( x, m, c )) , where Z x,c =
R exp (  E X,M,C ( x, m, c )) dm is a normalizing constant.
In practice, we use a ﬁnite K to generate samples
˜ m K n , n = 1 , 2 , ...
N given x, c .
Classiﬁcation.
We want to determine whether the concept c appears in a given image x , i.e. compute
P C | X ( c | x ) .
We need to marginalize over the mask m :
[Equation]
We again use SGLD [10] in Eq.
(1) to generate N samples ˜ m Kn , n 2 [ N ] = { 1 , 2 , ...N } given x, c ,and approximate the above integral using maximum a posteriori (MAP) estimation:
[Equation]
In practice, we only need to ﬁnd the concept with the highest value in the numerator.
Zero-shot recognition of novel concepts.
To master a novel hierarchical concept and directly use itfor classiﬁcation and detection only given its relational graph structure, we need a way to compose theprevious concept and relation energy based models.
Here we introduce the hierarchical compositionrule, using an English letter “F” as an illustrative example 4 .
The concept F has one constituent concept, a "line", and two relations, "parallel" and "perpendicular".
The models for constituents are learned during training.
During inference, these models are combined,using the concept graph, into a combined energy model for the letter
F. Essentially, the models of allthe recognized constituent concepts and relations are added together.
Note that although F containsthree lines, only one "line" model is needed.
The concept graph acts as a template and recognized lineinstances (objects) are matched with nodes in the template to obtain actual models.
The "line" modelis instantiated three times with speciﬁc values for the three identiﬁed lines and the three models plustheir corresponding relations models are used to derive the hierarchical model.
In addition to themodels, we also need to combine the masks of all the recognized constituent objects.
Formally, we deﬁne the following composition rule for HC-EBMs plus the masks to be combined.
Deﬁnition 2.1.
Hierarchical Composition Rule: Let a hierarchical concept c have graph G = 145 (
C , R ) , where C are constituent concept nodes and R are relation edges.
During inference, these 146 nodes and edges are matched to recognized concept objects and their relationships, which providestheir models and masks.
The combined model E X,M,C ( x, m, c ) is then a sum of the models for allthe nodes and edges in the graph and the mask M is the maximum of all the masks:
P
Ii
P
+
M
[Equation]
2.3 Zero-shot Concept Acquisition at Inference Time
Here, we introduce techniques for acquiring novel concept graph, which is useful for sharinghigh-level knowledge transfer between independently trained models.
Such capability may haveimplications in future scenarios where a hypothetical self-driving car communicating the structureof a novel road sign to other cars on the road, or a 2D vision model learning a novel object andtransferring it to a more accurate 3D depth model for inference.
To achieve this, we need to establishan equivalence relation between concepts in different domains ( e.g. in 2D and 3D images).
Deﬁnition 2.2.
Structural Equivalence for HC-EBM:
Two HC-EBMs are structurally equivalent,if their graphs are isomorphic, and their constituent HC-EBMs are recursively structurally equivalent.
For example, HC-EBMs for a “rectangle” concept in 3D and 2D images are structurally equivalentif they have the same decomposition into two “parallel-line” HC-EBMs connected by a “connect”
R-EBM, and the “parallel-lines” have the same decomposition into two “line” HC-EBMs connectedby a “parallel” R-EBM.
Even though the HC-EBMs are grounded in different domains, they have thesame abstract structure, so they represent the same hierarchical concept.
With structural equivalence, independently-trained HC-EBMs from different domains can acquirehierarchical concepts via the graph structure.
The algorithm works as follows (see Appendix A.5):the image is parsed to decompose it into a graph of concepts and relations previously learned by
HC-EBM and R-EBM, then the graph is used in a different domain to compose its HC-EBM for thenew concept.
The key step of this process is the ﬁrst, parsing step.
The parsing step can be seen as an inverse of the Hierarchical Composition Rule and is criticalin allowing the model to recognize novel hierarchical concepts in its domain using prior-learnedconcepts and relations from other domains, thus facilitating downstream knowledge transfer betweendomains.
Alg.
1 provides further details.
Steps 1-2 infer the concept instances in image x using
SGLD (Eq. 1).
Here the energy E ( p ) ( x, M C , c ) is the summation of HC-EBMs on independentmasks, each mask m i l representing a potential concept instance belonging to concept c i .
E (overlap) =max
[Equation]
⌘ penalizes overlapping masks.
Since the image probably containsfewer concept instances than given, some masks are empty with all-zero values; Step 3 removes these.
Step 4 classiﬁes relations (using Eq. 2) between pairs of detected concept instances.
Step 5 combinesthe detected concept instances and their relations to build the graph G for this new hierarchicalconcept c .
2.4 Learning
Since a standard EBM-training method yields poor performance as shown later in our experiments,we describe here our approach to train the models.
Further details of our training algorithm are given
[Table]
in Alg. 2 in Appendix A.1.
We are optimizing the following objective:
[Equation]
The expression of each term is given in Appendix A.2.
The ﬁrst loss L (Improved) is the objectiveproposed in [12], a state-of-the-art EBM training technique.
However, to enable the challengingzero-shot concept recognition and acquisition, we need more suitable inductive biases.
The nextthree terms inject the right inductive bias for the task.
L (pos-std) makes sure that the positive energyhave similar level, so the composed concept EBM can identify all constituent concepts and relations,without one constituent EBM having too low energy and only recognizing it.
The empty-maskregularization L (em) makes sure that the when the mask is empty, its energy is between the positiveand negative energy, which we prove in Appendix A.3 that it is the necessary condition for correctlydiscover the underlying graph in Sec.
The L (neg) additionally provide additional negativeexamples, encouraging discovering full concept instead of a part of it (see Appendix A.2 for details).
Experiments
In this section, we set out to answer the following two questions: (a) given a speciﬁcation of graphstructure for a novel hierarchical concept, can the composed concept model successfully performclassiﬁcation and detection tasks?
(b) Can HC-EBM acquire a novel concept from HC-EBM trainedfrom another domain?
We also evaluated our model in a challenging setting by comparing with other existing approachesin a controlled and systematic way.
Since there are no suitable dataset, we designed a HierarchicalConcept corpus, a dataset based on grid-world images.
The dataset comes with testing images forthe classiﬁcation and detection tasks which contain more complex, hierarchical concepts, composedfrom simpler concepts in the training set (See Appendix A.9 for details).
The concept instanceson our images have varying locations, size, and relative positions, making them more difﬁcult thanﬁxed-location ﬁxed-size inference tasks such as in [3].
We also provide 2D and 3D versions of imagesfor demonstrating concept acquisition across domains.
3.1 Zero-shot Classiﬁcation and Detection of Novel Concepts
To test recognition of novel concepts, we designed two datasets consisting of different concept andrelation types.
The HD-Letter hierarchical dataset consists of concept instances of “line” and relationinstances of “parallel”, “perp-edge” (perpendicular and touching edge), “perp-mid” (perpendicularand touching middle), together with distractor objects.
Examples are provided in the form of 3-tuples( image, mask, concept ) for concepts, and ( image, mask 1 , mask 2 , relation ) for relations.
ZeroCmodels are trained via objective Eq.
(See Appendix A.9 for examples of training datasets).
Atinference time, the models need to perform classiﬁcation and detection on novel images with morecomplicated English characters of “E-shape”, “F-shape” and “A-shape”, given their structure graphswith up to 4 nodes and 6 edges.
For detection, the images also contain a few distractors (concepts graph for F graph for E graph for A
E=-1.
combined mask
TotalEnergy
E=-0.
E=0.
Individual masks graph for Concept
E=0.
combined mask
TotalEnergy
E=0.
E=-0.
Individual masks graph for Concept graph for Concept (b) Classification: (a) Detection: correct!
correct!
graph for A
IoU: 0.
graph for E
IoU: 1.
inferred ground-truth graph for F
IoU: 0.
inferred ground-truth
Concept
IoU: 0.
Concept
IoU: 0.
inferred ground-truth
Concept
IoU: 0.
unrelated to the ones the model is predicting).
The model is asked to return a mask indicating whichpixels belong to an instance of the speciﬁed novel concept.
Although looking simple, this is actuallya very challenging task.
because in order to solve the problem of classifying/detecting hierarchicalconcepts, the model needs to solve a subgraph isomorphism problem, which is NP-hard.
For example,take the problem of detecting “E” shape in an image with distractor of “T” and “Rectangle” (Fig.2 (a) ﬁrst subﬁgure), a graph with 10 nodes (lines), 45 possible edges.
Note that during trainingtime, a model has only learned “line” concept, and relations “parallel”, “perp-edge” (perpendicular,touching edge), and “perp-mid”, but not the overall “E” shape.
At inference time and given the “E”concept graph with 4 nodes (lines), 6 edges (relations), the model needs to ﬁnd the “E” conceptsubgraph within the large image graph.
This involves C 410 ⇥ 4!
=
5040 possible mask assignments.
Additionally, a model may not perfectly detect the masks for low-level concepts.
For more complex concepts and relations, we designed the HD-Concept hierarchical dataset, whichconsists of training concepts of “E-shape” and “rectangle”, and training relations of “inside”, “enclose”and “non-overlap”.
The hierarchical concepts to be classiﬁed and detected are three characters whichwe term Concept1,
Concept2 and Concept3, as the ground-truth masks in Fig.
2 (a) indicate.
Thethree hierarchical concepts have the same multiset of concepts, but a different relation structures.
As the closest setting to our classiﬁcation task is zero-shot classiﬁcation, we compare ZeroC to astate-of-the-art zero-shot learning algorithm CADA-VAE [8], which we adapted to our setting, usingthe set of concepts and relations as feature embeddings to represent the concept graph (see Appendix
A.6 for details).
Additionally, we evaluated a “Statistics” baseline which samples random pixels onthe image based on the global statistics of pixel occurrence, and a “Heuristics” baseline that randomlychooses a same-color connected object.
We use a CNN-based architecture for ZeroC, as given in
Appendix A.4.
We add comparisons with ablation of different aspects of ZeroC, including without
HC-EBM terms or R-EBM terms in the composition rule of Def.
2.1 (Eq. 3), and without differentregularization terms of the objective in Eq. 4.
Fig.
2 shows a demonstration of this experiment.
From performance results in Table 1, we seethat ZeroC achieves classiﬁcation accuracy of 84.5% and 70.5% on HD-Letter and HD-Concept,respectively, both higher than CADA-VAE.
The gap is more signiﬁcant in HD-Letter, where thereis a larger distribution shift for hierarchical concepts (as can be seen in Appendix A.9), which thejoint embedding of the image and features learned by CADA-VAE is insufﬁcient to handle.
Thereason for the low accuracy is that CADA-VAE is not able to address these out-of-domain distributionshifts (See Appendix A.7 for details).
During inference, its embedding (multi-hot vector) for thegraph structure can contain up to 10 hots (4 lines, 6 edges), while during training, it is only up to1-hot.
This example also shows the intrinsic difﬁculty of the task.
In addition, Table 1 and Fig.
2 (a)show that ZeroC is able to detect the hierarchical concepts in the presence of distractors, and that itsperformance is signiﬁcantly better than “Statistics” and “Heuristics” baselines.
[Table]
The results of our ablation studies in Table 1 show that both the HC-EBM and R-EBM terms in Eq.
3are key to successful classiﬁcation and detection.
This is likely because without relations and usingonly concept terms, ZeroC may lose its ability to distinguish between different hierarchical concepts;for example, both “E-shape” and “A-shape” have 4 lines but different relation structures.
Moreover,we see that both L (pos-std) and L (neg) improve classiﬁcation and detection performance.
3.2 Acquiring Novel Hierarchical Concepts Across Domains
We show that ZeroC can acquire novel hierarchical concepts across models and domains (Fig. 3).
Wetrain a ZeroC model in domain ZeroC 1 , where the images are 2D and one-hot color-coded, containingthe the same training concepts and relation types as in HD-Letter in Sec. 3.1.
Independently , we trainanother model in a different domain ZeroC 2 , where the images contain the same set of concepts andrelations, but are viewed in 3D from a certain camera angle, have larger size, and use RGB colors.
Atinference time, each test task consists of a tuple of three images showing the hierarchical concepts inthe ﬁrst domain to ZeroC 1 .
ZeroC 1 is only allowed to send symbolic information, up to a few bits, to
ZeroC 2 .
Then ZeroC 2 performs classiﬁcation and detection tasks on three example images in thesecond domain.
In addition, we also evaluate whether ZeroC 1 can parse the concept graph of thehierarchical concept correctly, using the metric of graph edit distance
d edit and graph-isomorphismaccuracy w.r.t.
the ground-truth.
Note that the graph-isomorphism accuracy is a stringent metric,which is only 1 if the parsed graph is isomorphic to the ground-truth, and 0 otherwise.
We comparewith a strong baseline of Mask R-CNN [13] for object detection together with relation classiﬁcation.
The relation classiﬁcation network is a CNN which takes as input a pair of embeddings producedby the Mask R-CNN, and predicts the label of the relation (see Appendix A.8 for details).
We didextensive tuning of the Mask R-CNN approach to obtain its best performance.
We also evaluate theperformance of ZeroC 2 given the ground-truth structure graph, as an ideal scenario of perfect parsing,providing an upper bound for the performance.
Table 2 shows that ZeroC 1 achieves 72.7% graph isomorphism accuracy for parsing the hierarchicalconcepts in the ﬁrst domain, signiﬁcantly higher than Mask R-CNN
+ relation classiﬁcation.
Aftersending the concept graphs in the second domain, ZeroC 2 achieves a classiﬁcation accuracy of
60.7% and detection IoU of 94.4%.
Without this information and relying on global statistics, theclassiﬁcation accuracy of ZeroC 2 is 33.3%.
This demonstrates that ZeroC 2 is able to acquire novelhierarchical concepts at inference time, from an independently trained model from a different domain .
The ablation study shows that L (em) , L (neg) and L (pos) all contribute to more accurate parsing.
Thestandard technique of EBM training [12] are insufﬁcient to achieve good parsing, leading to lessaccurate transfer of hierarchical concepts.
We also see in the ablations that even without perfectparsing from ZeroC 1 , the reduction of classiﬁcation accuracy for ZeroC 2 is small, showing that it isable to classify under noisy speciﬁcation of the concept graph.
Related Work
Our work relates to visual compositionality, concept and relation learning, and zero-shot learning.
Visual compositionality.
Compositionality is a key for addressing diverse tasks given ﬁnite basic knowledge.
Some approaches use composition EBMs for generation [1], and a VAE-basedarchitecture for bi-directional symbol-image generation that can also learn logical recombination ofconcepts [2].
These two works focus on composing factors of variation , e.g. color, position, smiling,young.
In contrast, our work focuses on concepts that abstract objects, where the internal hierarchicalrelational structure is key.
Moreover, while the above works focus on generation, our work focuses onthe tasks of classiﬁcation and detection.
A novel Bayesian-based method for few-shot learning on the
Omniglot dataset shows that compositionality is pivotal for improved performance [14].
They achievecompositionality via hierarchical MCMC sampling on hand-coded priors of elementary conceptsand relations.
In comparison, our work is neural network-based and only requires demonstrationsof elementary concepts and relations, reducing hand-coded priors.
Another approach introduceda modular neural network, which uses composition of neural modules for visual question answering [15].
Their method can be seen as composing transformations on a representation.
In comparison,our composition is achieved via composition of energy landscapes.
While they focus on questionanswering on scene graphs, we focus on classiﬁcation and detection of hierarchical concepts.
Concept and relation learning.
There has been exciting progress in concept learning and relationlearning.
Works in concept learning generally represent concepts in latent space via prototypes[16, 17], or via latent embedding such as SCAN [2], InfoGAN [18] and Neuro-Symbolic Concept
Learner [19].
[20] introduced EBMs to represent concepts with a demonstration in simple state space.[21] further introduced unsupervised learning of local and global concepts with EBMs.
Regardingrelation learning, non-local neural networks [22], Relation Networks [23], Neural Relational Inference[24] and C-SWM [25] use latent complete graphs as inductive biases to represent potential relations.
PrediNet [3] explicitly represents propositions, relations, and objects with a transformer-basedarchitecture, and demonstrates that it can learn relations that generalize to novel shapes of objectsand column patterns.
In comparison to the above works, ZeroC explicitly learns both concepts andrelations, which has the unique capability to recognize and acquire hierarchical concepts at inference.
Zero-shot learning.
Zero-shot learning methods [4, 5, 6, 7, 8] typically learn a joint embeddingspace between image and feature labels, and perform classiﬁcation at inference time on images withnovel classes, based on how the novel classes correspond to a set of features.
Our generalization taskalso requires generalizing to new concepts without seeing the image (zero-shot), but has the importantdistinctions that at inference time, our concepts to be inferred lie at a higher hierarchy than that intraining, and furthermore use the structural concept graph.
In comparison, in standard zero-shotlearning, concepts in training and inference lie at the same hierarchy level, and only generalize tonew combinations of features (constituent concepts) while neglecting relation structures.
Conclusion
In this paper, we introduce ZeroC, a new framework for zero-shot concept recognition and acquisitionat inference time.
Our experiments show that in a challenging grid-world domain, ZeroC is ableto recognize complex, hierarchical concepts composed of English characters in a grid-world in azero-shot manner, being given a high-level, symbolic speciﬁcation of their structures, and after beingtrained with simpler concepts.
In addition, we demonstrate that an independently trained ZeroC isable to transfer hierarchical concepts across different domains at inference.
Although this work isevaluated only in grid-world domain, we are the ﬁrst to address this difﬁcult challenge, and hopethat this work will make a useful step in the development of composable neural systems, capable ofzero-shot concept recognition and acquisition and hence suitable for more diverse tasks.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] In conclusion and AppendixA.10.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] InAppendix A.11.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The Appendix
includes full details on model architecture, training and evaluation to reproduce the
experimental results.
Code and data will be released upon publication of the paper.
(b) Did you specify all the training details ( e g. , data splits, hyperparameters, how theywere chosen)?
[Yes] Full details to reproduce the experiments are included in theAppendix.(c)
Did you report error bars ( e g. , with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used ( e.g. ,
type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix of respective experiment section.
If you are using existing assets ( e.g. , code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] We use our own
created benchmark.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
In Appendix A.9, we describe the engine and training/testing tasks.
Full dataset will be
released upon publication.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]