M eta R e V ision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition (2023)

Abstract: Humans have the ability to learn novel compositional concepts by recalling primitive concepts acquired from past experience and generalizing these primitive concepts to novel compositions. Inspired by the above human’s compositional learning procedure, in this paper, we propose MetaReVision, a retrievalenhanced meta-learning model to solve the visually grounded compositional concept learning problem. The proposed MetaReVision consists of a retrieval module and a meta-learning module which are designed to incorporate retrieved primitive concepts as supporting set to meta-train visual-language models for grounded compositional concept recognition. Through meta-learning from episodes constructed by the retriever, MetaReVision learns a generic compositional representation that can be fast updated to recognize novel composi tional concepts. We create CompCOCO and CompFlickr to benchmark the grounded compositional concept learning. Our experimental results show MetaReVision outperforms other competitive baselines and the retrieval module does plays an important role in this compositional learning process.

## introduction learning to compose from previous experience is an important integral part of human intelligence @xcite . generally, compositional learning refers to the ability to learn a set of basic primitives and generalize these primitives in a novel scenario different from training time @xcite . it includes various learning aspects, such as systematic generalization, productivity and substitutivity @xcite . in this work, we focus on systematic generalization within the multi-modal setting and propose a multimodal compositional problem: grounded compositional concept learning (gccl). as shown in figure 1 , in the gccl setting, the models are trained with primitive concepts, such as red and a large red bus is driving down the street. to address these limitations, this paper takes a closer look at the compositionality in vlm with an attempt to improve its ability. more specifically, we create two grounded compositional concept learning datasets, compflickr and compcoco curated from mscoco @xcite and flickr30k @xcite , for vlms' token-level compositional analysis. moreover, we present metarevision, meta-learning with retrieval for visually grounded compositional concept acquisition, a retrieval-enhanced meta-learning framework for compositional concept acquisition, which introduces retriever into gccl. the retrieval mechanism plays a crucial role in human learning. it facilitates long-term retention, understanding enhancement, and knowledge transfer during the learning process, which have been discussed by a large body of studies in cognitive science @xcite . to mimic such human's retrieving behavior @xcite , metarevision retrieves relevant primitive concepts from a preconstructed concept database and provides them as support evidence to do meta-learning for compositional concept learning. metarevision follows a learn-retrieval-compose framework. it shares the compositional learning burden between vlms and the retriever. through meta-1 element concepts are also called primitive concepts in our setting. we use them interchangeably in this work. contributions of this work can be summarized as follows: retrieval-enhanced learning. retrieving related instances from a database, either the training set or external knowledge base, has been widely applied in tasks such as language modeling @xcite , reinforcement learning @xcite and language tasks such as ner @xcite . instead of distilling all training information into the model's parameters through gradient updates, retrieval-enhanced learning introduces a retriever to find related instances and based on these instances conduct further learning. for example, knn-lm @xcite extends the pre-trained language model by linearly interpolating its next word distribution with a retrieval module. this design shows effective domain adaptation ability. wang et al. finds external contexts for the target instance by retrieving a set of semantically relevant texts to fine-tune the crf module to address the ner problem. these studies highlight the significance of actively recalling information from a database to enhance learning outcomes. the general scheme of such methods is to combine a parametric model with a non-parametric retrieval system @xcite . different from these settings, in gccl, we train our own concept retriever and show retrieval's importance in compositional learning. ## grounded compositional concept learning (gccl) we start by introducing the settings of grounded compositional concept learning (gccl) and further introduce the benchmarks we curated for this problem in this section. ## meta-learning with retrieval for gccl (metarevision) metarevision mainly consists of two modules: the retrieval model and the meta-learner as shown in figure 3 . the retrieval module learns to find similar element concepts from the training data. the meta-learner organizes the retrieved items as a pseudo task to meta-tune vlms for compositional learning. in this part, we will discuss the base vlms, retrieval module, and meta-learning module in detail and answer two key questions in metarevision's design: 1) how to retrieve related items, 2) how to utilize the retrieved items in the context of meta-learning. ## experiments in this section, we introduce the gccl's datasets, demonstrate the implementing details of metarevision, and compare its results with other baselines. ultimately, we empirically analyze the retriever importance in mearevision. ## acknowledgement this project is supported by national science foundation (nsf) career award 2028626 and partially supported by the office of naval research (onr) grant n00014-20-1-2005. any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the national science foundation nor the office of naval research. we thank all reviewers for their thoughtful comments and suggestions. ## limitations the limitations of the proposed metarevision include 1) grounding limitation. currently, we rely on vlm's attention mechanism to do grounding. we do not have an explicit grounding design to align the textual concepts and visual regions. this could be an interesting direction for future gccl works. 2) sota generative model comparisons. currently, we can not directly apply sota generative models, such as blip-2 and minigpt, on gccl due to the following reasons. one reason is the gccl problem setting. in gccl, it is not easy to transform the supporting items, including multiple images and captions, into contextual input for these generative models. another reason is controlled evaluation which means that these huge generative models may have already seen the novel compositions during training and it is not a fair comparison with other models. 3) updating retriever. we construct our element concept db in advance and not updating this db during the meta-learning time. training both the learner and the retriever in an end-to-end manner could improve the performance for gccl and other retrieval-enhanced models.