Non-compositional Expression Generation and its Continual Learning (2024)

Abstract: Non-compositional expressions are an integral part of natural language and their meanings cannot be directly derived from the meanings of their component words. Recent work has shown how their processing remains a challenge for pre-trained language models. Here we consider the fact that prior knowledge of their component words is inadequate to infer their meaning as a whole and that these expressions constitute a long-tailed process in language (based on their occurrence in corpora and their coming into use as an idiomatic expression in a continual manner). Against this backdrop, this paper studies the ability of recent pre-trained language models to generate non-compositional expressions in English and their continual learning. Formulating this as a mask infilling task termed as CLoNE, the study uncovers the combined challenges of non-compositionality and their continual learning. Using a set of three diverse idiomatic expression datasets repurposed for this task, we benchmark different large pre-trained language models and different continual learning methods on the task of non-compositional expression generation. Our experiments on the CLoNE task show that large pre-trained language models are limited in their ability to generate non-compositional expressions and available continual learning methods are inadequate for our proposed CLoNE task which calls for more effective methods for continual learning of non-compositionality. Our datasets and code will be released publicly upon acceptance.

## introduction large language model advancements have provided a new context to study non-compositional expressions (or idiomatic expressions) and the challenges they pose to classical nlp tasks involving them, such as sentiment analysis @xcite , paraphrase generation @xcite , natural language inference @xcite , metaphor detection @xcite , and contextual disambiguation @xcite .furthermore, given the significance of non-compositional expressions in enhancing the naturalness and stylistic richness of everyday language @xcite , their generation by large pretrained language models (trained without explicitly accounting for non-compositionality) remains an important, yet inadequately explored aspect in nlp @xcite . a related, but distinct inquiry pertains to the continual addition of non-compositional expressions into language @xcite . specifically, it raises the question of how large pre-trained language models learn new non-compositional expressions that were not part of their training data without requiring retraining. additionally, since the meaning of non-compositional phrases cannot be inferred from the meanings of their component words @xcite , and learning the meaning of one non-compositional phrase does not aid in understanding another, each noncompositional phrase must be learned individually. this calls for a continual learning (cl) setting to study large language models' ability to generate non-compositional expressions. formulating this as a mask infilling task with idiomatic expressions, the study probes the combined challenges of noncompositionality and their continual learning. continual learning, also known as lifelong learning, is a machine learning paradigm to adaptively learn new knowledge from a continuous data stream over time while still being able to remember and reuse previously learned knowledge @xcite . in efforts to mitigate the issue of "catastrophic forgetting" often seen in neural models, there is a rising interest in creating cl algorithms for natural language processing (nlp) tasks @xcite @xcite that have shown limited success in handling compositional phrases. this focus of this study is the equally significant challenge of processing non-compositional phrases like idioms and metaphors. in a recent study, @xcite investigated non-compositional expression generation by employing cl over examples organized by dynamically changing difficulty levels, potentially reintroducing previously seen idioms at later stages. in contrast, our approach explores their continual learning and generation by randomly selecting expressions from a pool without replacement. this ensures that expressions encountered earlier in training will not reappear later and better represents their continual learning, focusing on their nuanced learning rather than broader difficulty levels. in this novel setup, we study the extent to which popular cl algorithms alleviate the problem of catastrophic forgetting when learning to generate non-compositional expressions. our study is centered around three research questions. r1: how difficult is it for language models trained with an mlm objective to learn to generate multi-word non-compositional expressions (which would still be generated word by word)? r2: to what extent do large pre-trained language models suffer from catastrophic forgetting when learning to generate non-compositional expressions? and r3: to what extent can popular cl algorithms reduce the problem of catastrophic forgetting when learning to generate non-compositional expressions? answers to these questions lead to the main contributions of our work summarized below. 1. we propose clone, a new task for studying noncompositional expression generation in a continual learning setting. 2. we construct three datasets, which will be made publicly available upon acceptance, to simulate non-compositional expression acquisition and to probe a set of pretrained language models that yield comprehensive insights about their capabilities to learn non-compositional expressions, also in a cl setting. 3. comparing cl algorithms with the constructed datasets we find that (a) current large pre-trained language models are limited in their ability to generate non-compositional expressions; (b) current large pre-trained language models are plagued by catastrophic forgetting when learning to generate non-compositional expressions; and (c) sota cl algorithms are not beneficial in alleviating catastrophic forgetting for the clone task. detailed ablation studies and analyses substantiate our claims providing insights about the clone task. the paper is organized as follows. in section 3 we explore the first research question, where we fine-tune a set of pre-trained language models to generate idioms and evaluate their performance. then we explore the second research question in section 4, where the goal is to understand the forgetting problem for pre-trained language models when learning to generate idioms. to this end, we propose the continual learning of noncompositional expressions (clone) task to measure the ability of pre-trained language models to learn to generate them in a continual manner. this is depicted in figure 1 . finally, we explore the third research question in section 5, where using the clone dataset, we explore the extent to which popular cl algorithms alleviate the forgetting problem. ## understanding non-compositional expression generation in pre-trained language models 3.1 task formulation to explore large pre-trained language models' ability to generate non-compositional expressions, we train multiple large pre-trained language models to generate idioms, formulated as a mask infilling task. given a sentence with a masked fragment, the task is to generate an appropriate non-compositional expression to fill the mask. in our setup, we restrict each sentence to contain only one masked fragment and the model is required to fill exactly one non-compositional expression. we create input sentences by selecting a sentence containing a non-compositional expression with a known position and replacing the entire expression with @xmath0. the @xmath1. for example, to test if the model is able to successfully generate "out of the question," we use the sentence, "it is a relatively small basement room, so a large imposing piece was out of the question.", converted to "it is a relatively small basement room, so a large imposing piece @xmath2." to account for instances where the right context of the noncompositional expressions could help with their generation, we formulate our task as a mask infill task instead of a causal generation task. ## learning to generate non-compositional expressions and catastrophic forgetting to explore the catastrophic forgetting problem in large pre-trained language models and their abilities to learn and generate non-compositional expressions in a continual way, we propose our clone task and construct the corresponding datasets for experiments. ## rq3: to what extent can popular cl algorithms alleviate the problem of catastrophic forgetting given the conclusion from section 4 that current large pre-trained language models still suffer from catastrophic forgetting problem when learning to generate non-compositional expressions, we next plan to explore if some traditional and popular cl algorithms could effectively alleviate this forgetting problem in the context of large pre-trained language models and non-compositionality. all the experiments on section 4 are repeated again with the application of different cl algorithms. ## conclusion in this study, we propose clone, the first task focusing on generation of non-compositional expressions and their continual learning, whose challenges are non-compositionality and continual learning. benchmarks and analysis of sota large pre-trained language models and continual learning methods provide exploration towards our proposed three research questions. our experiments show that even the most sota large pre-trained language model struggles with non-compositional expression generation and still suffers from catastrophic forgetting problem. furthermore, even the best-performing cl method still struggles with continual learning of non-compositionality. the gap between the best performance in a cl setting and the upper-bound performance in an offline setting calls for more advanced models for learning non-compositionality and more effective cl algorithms.