Visually Grounded Continual Language Learning with Selective Specialization (2023)

Abstract: A desirable trait of an artificial agent acting in the visual world is to continually learn a sequence of language-informed tasks while striking a balance between sufficiently specializing in each task and building a generalized knowledge for transfer. Selective specialization, i.e., a careful selection of model components to specialize in each task, is a strategy to provide control over this trade-off. However, the design of selection strategies requires insights on the role of each model component in learning rather specialized or generalizable representations, which poses a gap in current research. Thus, our aim with this work is to provide an extensive analysis of selection strategies for visually grounded continual language learning. Due to the lack of suitable benchmarks for this purpose, we introduce two novel diagnostic datasets that provide enough control and flexibility for a thorough model analysis. We assess various heuristics for module specialization strategies as well as quantifiable measures for two different types of model architectures. Finally, we design conceptually simple approaches based on our analysis that outperform common continual learning baselines. Our results demonstrate the need for further efforts towards better aligning continual learning algorithms with the learning behaviors of individual model parts.

## introduction grounding language in visual perception is a crucial step towards agents that effectively understand and interact with the physical world @xcite . in a realistic setting, such an agent would require the ability to continuously integrate novel experience and skills with existing knowledge in an open-ended process, a challenge commonly known as continual learning (cl) @xcite @xcite . expert solutions train a copy of the network for each new task, while monolithic networks make unrestricted updates to all parameters. in this paper, we focus on selective specialization to strike a balance between full specialization and generalization. ideally, the underlying neural model of an agent would sufficiently solve each isolated task (i.e., specialization) while harnessing the shared structure and subproblems underlying all tasks to create generalizable representations for knowledge transfer (i.e., generalization). a strategy to provide control over this trade-off is to introduce task-specific parameters to a carefully selected subset of model components (i.e., selective specialization), as can be seen in fig. 1 . understanding which model components (here referred to as modules) are suited for taskspecialization requires insights on their role in solving each task, which prior works on continual vision-language (vl) grounding fail to explain. at the same time, there is a lack of suitable benchmarks that allow for a fine-grained model analysis, as existing cl scenarios for language grounding based on synthetic images are too simplistic concerning the cl problem (e.g., only one distributional shift) @xcite or the vl grounding problem (e.g., single-object, trivial language) @xcite , while those based on real-world images @xcite may encourage models to take shortcuts in the vision-language reasoning process, which leaves the generalizability of statements derived from any model analysis questionable. to this end, we introduce the lifelong language compositions (lilac) benchmark suite that comprises two diagnostic vl datasets that allow for investigating the continual learning behavior of the models with a high degree of control and flexibility while being challenging enough to require object localization, spatial reasoning, concept learning, and language grounding capabilities of the continual learner. based on the two proposed lilac benchmarks, we conduct a fine-grained analysis of two different vision-language model architectures in this work that comprises the following steps: (i) we analyze whether selective specialization generally benefits from dividing the learning process into the two intermittent stages of task adaptation, where only the specialized parameters are being updated, and knowledge consolidation, where only the shared parameters are updated with respect to the taskspecific learned representations. (ii) we derive and evaluate heuristics from prior literature about introducing task-specific parameters with respect to different layer depths and specific vl network modules. (iii) we assess the suitability of parameter importance measures from pruning research as indicators of the efficacy of module selection strategies. we conclude our work by demonstrating the superior performance of module selection strategies found in our analysis, when trained under the adaptation-consolidation (a&c) procedure as described above, over common cl baselines. thus, we summarize the contributions made in our work as follows: ## lilac datasets in what follows, we propose two benchmark datasets that explicitly model the compositional nature of a problem, thus maintaining a high degree of overlap across tasks. this should encourage the models to use as few task-specific parameters as possible and facilitate the exploration of a reasonable selection strategy for specialized modules. during training, the model receives as input three images representing objects in a simulated environment, along with a templated language instruction. examples of the two proposed datasets can be found in fig. 2 and fig. 3 . lilac-2d tasks. this dataset is based on the minigrid (chevalier-boisvert et al., 2018) environments. for each example, three to nine objects are randomly placed in a 7 × 7 grid. each instruction describes a desired interaction with a desig-nated target object and takes the form of "move @xmath0, where we choose from a set of six colors, three object types, and four directions. all three subproblems described by the instruction can be orthogonally combined, yielding 6 × 3 × 4 = 72 distinct instructions. false target images are generated by randomly choosing one of the three subproblems to be wrongly solved, e.g., a blue key is moved down, although a green key was supposed to be moved down. the lilac-2d dataset comprises 500 train, 100 validation, and 100 test samples per instruction, yielding a total of 36,000 train and 7,200 validation and test samples, respectively. for the continual learning stream, we construct @xmath1. lilac-3d tasks. to further narrow the gap to real 3d images while maintaining a high degree of control and flexibility for model analysis, we additionally propose lilac-3d, a dataset with increased spatial complexity and distracting information. it is based on the simulated ravens @xcite benchmark and its extension towards language instructions @xcite . images show a tabletop scenario, where between five and eight blocks and between three and four bowls are randomly placed within the range of a robot arm. instructions describe a pick-and-place operation with a target block and a target bowl and take the form of "put @xmath2, where we choose from a set of two different sizes, six block colors, and six bowl colors, thus yielding a total of 2 × 6 × 6 = 72 distinct instructions. the sets of colors for blocks and bowls are fully disjoint to allow the objects to be clearly distinguished from one another. false target images are designed in a way that either the wrong block and the right bowl or the right block and the wrong bowl are chosen for interaction. the dataset statistics as well as the continual stream design are similar to those of the lilac-2d dataset. ## conclusion striking a balance between specialization and generalization poses a challenge for agents that learn a sequence of language-conditioned tasks grounded in a visual environment. in this work, we consider vision-language models as a composition of modules and propose two datasets that allow us to analyze and compare strategies to selectively specialize modules to continual tasks with respect to different layer depths and module types. we further establish a gradient-based importance measure that quantifies the suitability of modules for specialization. finally, we show that the module specialization strategies found in our analysis outperforms common cl baselines when trained under a conceptually simple adaptation-consolidation learning scheme. with this work, we show the merit of designing cl methods based on a careful analysis of selective specialization. beyond introducing specialization to existing model parts, we plan to leverage our insights for the design of parameter isolation cl methods that introduce additional parameters into a pretrained model for future work.