Zero-Shot Compositional Concept Learning (2021)

Abstract: In this paper, we study the problem of recognizing compositional attribute-object concepts within the zero-shot learning (ZSL) framework. We propose an episode-based cross-attention (EpiCA) network which combines merits of cross-attention mechanism and episode-based training strategy to recognize novel compositional concepts. Firstly, EpiCA bases on cross-attention to correlate conceptvisual information and utilizes the gated pooling layer to build contextualized representations for both images and concepts. The updated representations are used for a more indepth multi-modal relevance calculation for concept recognition. Secondly, a two-phase episode training strategy, especially the ransductive phase, is adopted to utilize unlabeled test examples to alleviate the low-resource learning problem. Experiments on two widelyused zero-shot compositional learning (ZSCL) benchmarks have demonstrated the effectiveness of the model compared with recent approaches on both conventional and generalized ZSCL settings.

## introduction humans can recognize novel concepts through composing previously learnt knowledge -known as compositional generalization ability @xcite . as a key critical capacity to build modern ai systems, this paper investigates the problem of zero-shot compositional learning (zscl) focusing on recognizing novel compositional attribute-object pairs appeared in the images. for example in figure 1 , suppose the training set has images with compositional concepts sliced-tomato, sliced-cake, ripe-apple, peeled-apple, etc. given a new image, our goal is to assign a novel compositonal concept slicedapple to the image by composing the element concepts, sliced and apple, learned from the training data. although sliced and apple have appeared concept of sliced sliced tomato sliced cake sliced bread concept of apple diced apple peeled apple ripe apple sliced apple diced pizza … train phase: test phase: localize, learn and compose regional visual features compose the learnt regional visual features figure 1 : given the concepts of sliced and apple in the training phase, our target is to recognize the novel compositional concept slice apple which doesn't appear in the training set by decomposing, grounding and composing concept-related visual features. with other objects or attributes, the combination of this attribute-object pair is not observed in the training set. this is a challenging problem, because objects with different attributes often have a significant diversity in their visual features. while red apple has similar visual features as the apple prototype, sliced apple presents rather different visual features as shown in fig 1 . similarly, same attributes can have different visual effects depending on the modified objects. for example, old has different visual effect in objects of old town compared to objects of old car. despite recent progress @xcite , previous works still suffer several limitations: (1) most existing methods adopt metric learning framework by projecting concepts and images into shared latent space, and focus on regularizing the structure of the latent space by adding principled constraints without considering the relationship between concepts and visual features. our work brings a new perspective, the relevancebased framework inspired by @xcite , to conduct compositional concept learning. (2)previous works represent concept and image by the same vector regardless of the context it occurs. however, cross concept-visual representation often provides more grounded information to help in recognizing objects and attributes which will consequently help in learning their compositions. motivated by the above discussions, we propose an episode-based cross attention (epica) network to capture multi-modal interactions and exploit the visual clues to learn novel compositional concepts. specifically, within each episode, we first adopt cross-attention encoder to fuse the conceptvisual information and discover possible relationships between image regions and element concepts which corresponds to the localizing and learning phase in fig. 1 . second, gated pooling layer is introduced to obtain the global representation by selectively aggregating the salient element features corresponding to fig. 1 's composing phase. finally, relevance score is calculated based on the updated features to update epica. the contribution of this work can be summarized as follows: 1) different from previous work, epica has the ability to learn and ground the attributes and objects in the image by cross-attention mechanism. 2) episode-based training strategy is introduced to train the model. moreover, we are among the first works to employ the transductive training to select confident unlabelled examples to gain knowledge about novel compositional concepts. 3) empirical results show that our framework achieves competitive results on two benchmarks in conventional zscl setting. in the more realistic generalized zscl setting, our framework significantly outperforms sota and achieves over 2× improved performance on several metrics. ## related work compositional concept learning. as a specific zero-shot learning (zsl) problem, zero-shot compositional learning (zscl) tries to learn complex concepts by composing element concepts. previous solutions can mainly be categorized as: (1) classifier-based methods train classifiers for element concepts and combine the element classifiers to recognize compositional concepts @xcite @xcite . the related works to epica are relnet @xcite and cvczsl @xcite . compared with these methods, we have two improvements including an explicit way to construct episodes which is more consistent with the test scenario and a cross-attention module to fuse and ground more detailed information between the concept space and the visual space. ## conclusion in this paper, we propose epica which combines episode-based training and cross-attention mechanism to exploit the alignment between concepts and images to address zscl problems. it has led to competitive performance on two benchmark datasets. in generalized zscl setting, epica achieves over 2× performance gain compared to the sota on several evaluation metrics. however, zscl remains a challenging problem. future work that explores cognitively motivated learning models and incorporates information about relations between objects as well as attributes will be interesting directions to pursue.