Visually Grounded Continual Learning of Compositional Phrases (2020)

Abstract: Humans acquire language continually with much more limited access to data samples at a time, as compared to contemporary NLP systems. To study this human-like language acquisition ability, we present VisCOLL, a visually grounded language learning task, which simulates the continual acquisition of compositional phrases from streaming visual scenes. In the task, models are trained on a paired image-caption stream which has shifting object distribution; while being constantly evaluated by a visually-grounded masked language prediction task on held-out test sets. VisCOLL compounds the challenges of continual learning (i.e., learning from continuously shifting data distribution) and compositional generalization (i.e., generalizing to novel compositions). To facilitate research on VisCOLL, we construct two datasets, COCO-shift and Flickr-shift, and benchmark them using different continual learning methods. Results reveal that SoTA continual learning approaches provide little to no improvements on VisCOLL, since storing examples of all possible compositions is infeasible. We conduct further ablations and analysis to guide future work.

## introduction modern nlp systems, including ones that build on pre-trained language models @xcite , excel on a wide variety of tasks. these systems rely on offline (batch) training and have drawn recent criticism due to their inability to adapt to new contexts @xcite ). in contrast, humans acquire language from evolving environments, require a small memory footprint @xcite , and can generalize their knowledge to newer tasks @xcite . it has been suggested that humans ground perceptual 1 code and data: https://github.com/ink-usc/ viscoll a green with a bunch of mask mask hanging from it. experience to semantically interpret symbols @xcite @xcite . model model the challenge, we propose vis-coll, a visually-grounded continual learning setup, to acquire compositional phrases from streaming visual-linguistic data. models receive a stream of paired image-caption data which has a shifting object distribution. as the end task, we employ masked token prediction of captions given the associated image, as illustrated in fig. 1(a) . this evaluates a model's learned knowledge on composing phrases with the given context. viscoll captures two inter-related challenges. first, unlike previous continual learning works on image classification @xcite , viscoll requires predicting, for example, a noun with a verb or an adjectivewhich results in a significantly large search space. as a result of this increased search space, memory based continual methods @xcite cannot expect to store prototypes of each visited compositions. second, the increased search space makes it infeasible to view all possible combinations of atomic words at train time. therefore, to succeed on viscoll, models should generalize to novel compositions at test time (also called composition generalization) @xcite . in this work, we extensively study the challenges associated with viscoll. to facilitate the research, we construct a continuously shifting data distribution to closely resemble real-word datastream and contribute coco-shift and flickr-shift. we benchmark these datasets using multi-modal language modeling architectures (tan and bansal, 2019; @xcite which achieve state-of-art performance on multiple vision-language tasks. in particular, we don't use any pre-training, instead train randomly initialized models on streaming data using continual learning algorithms @xcite @xcite and evaluate their resistance to forgetting and compositional generalization. we quantify the performance and forgetfulness of trained models and evaluate on a novel test split to measure compositional generalization, as shown in fig. 1(b) . our proposed viscoll benchmark reveals that the gains observed in image classification tasks from state-of-art continual learning algorithms fail to transfer to viscoll even with increased memory. on the other hand, composition generalization remains challenging even for offline-training. to summarize, our contributions are: (i) we propose viscoll, a task aimed at continual learning of compositional semantics from visually grounded text inputs (ii) we contribute two datasets cocoshift and flickr-shift to study viscoll and benchmark them with multi-modal language models (iii) we show that existing continual learning algorithms fail at learning compositional phrases and provide potential future research direction. ## the viscoll task overview. there are two design considerations for viscoll: compositionality and continual learning. to test compositionality, we choose visually grounded masked language modeling where the model needs to compose atomic words to describe complex and novel visual scenes. to simulate a realistic continual learning setup, we construct a dynamic environment where the training data comes in as a non-stationary data stream without clear "task" boundaries. since the goal is to simulate lan- task 3 task 1 task 2 training data stream training data stream continuously shifting data distribution, task-free splitted data distribution, explicit task boundaries or known task identities compositions closed set of labels (most of) existing continual learning settings viscoll setting k labels prediction space: k fluffy red old ... cat bus boat …. prediction space: figure 3: comparison of traditional continual learning setup for image classification with the setup of viscoll. in most of existing continual learning settings, task identities are obtainable and can be used to define task boundaries. in contrast, models in vis-coll is agnostic to task identities (task-free), and fed/trained by a gradually shifting data distribution. prediction end-task. we employ visually grounded masked language modeling as the prediction task in viscoll. an input instance to this task consists of an image x img , its object bounding boxes x bbox (without object labels), and the caption text x text , where a contiguous text span in x text is masked with mask tokens which the model needs to fill. the masked text span x label always includes a noun and optionally includes verbs or adjectives. we define each noun, verb, and adjective as an atom, and evaluate the model in both "seen" and "novel" composition setting of nouns and verbs/adjectives. for instance, we may test whether the model successfully predicts "red apples" (adjective+noun) when the model has seen examples that involve "red" and "apples" separately (see figure 2 for an illustration). training data stream. unlike traditional offline training setups where a model visits the training examples for multiple passes, we study an online continual learning setup, where the model visits a non-stationary stream of data. we assume the data distribution changes gradually: for example, the model may see more "apples" in the beginning, and see less of them later. unlike prior continual learning benchmarks, we do not assume strict task boundaries, i.e., sudden distribution changes. we illustrate this distinction in figure 3 . formally, at each time step t, the model receives a small mini-batch of stream examples evaluation. in addition to the final performance, we also evaluate the model performance every fixed time interval and compute its forgetting as the performance loss over time. for compositional generalization, a novel composition split is used. details will be covered in the following sections 4 and 6. ## dataset construction we construct our data streams using two popular vision-language datasets: coco-captions @xcite and flickr30k entities @xcite which provide multiple captions for each image in mscoco @xcite and flickr30k @xcite respectively. we call the resulting datasets coco-shift and flickr-shift (see masking tokens. first, we append part-of-speech tags (pos) to each caption using stanza @xcite . for flickr30k entities, we use the annotated noun-phrases as mask tokens. for coco-captions, we identify text spans with a regular expression chunker with the following regular expression. ## methods to benchmark on viscoll and study the challenges it poses on model learning, we establish several continual learning baselines. we use visuallanguage encoder models (sec. 5.1) for masked token predictions. these models are trained from scratch (i.e., randomly initialized) with continual learning algorithms (sec. 5.2) to dissuade catas- ## experiments with the viscoll task formulation in place, we study: (i) performance of continual learning algorithms on viscoll. (ii) effect of the large search space on memory-based continual learning algorithms. (iii) performance on generalizing to novel compositions. (iv) effect of replay memory management strategies. we first describe our implementation details and metrics, and present our results with findings. implementation details. for both vlbert and lxmert, we use a transformer with 6 layers, with 384 hidden units and 6 attention heads each. note that all the parameters are learned from scratch without using pretrained weights. for all continual learning algorithms, we use a memory size of 1k and 10k, corresponding to nearly 0.2% and 2% of data for the two datasets. we report average over 3 runs with the fixed stream and the same set of random seeds. see appendix for more details. ## conclusion we propose viscoll, a novel continual learning setup for visually grounded language acquisition. viscoll presents two main challenges: continual learning and compositionality generalization. to facilitate study on viscoll, we generate continuously shifting data-stream to construct two datasets, namely coco-shift and flickr-shift, and establish evaluation protocols. we benchmark our proposed datasets with extensive analysis using state-of-theart continual learning methods. experiments reveal that continual learning algorithms struggle at composing phrases which have a very large search space, and show very limited generalization to novel compositions. future works include looking into models and continual learning algorithms to better address the challenge.