Non-compositional Expression Generation Based on Curriculum Learning and Continual Learning (2023)

Abstract: Non-compositional expressions, by virtue of their non-compositionality, are a classic ‘pain in the neck’ for NLP systems. Different from the general language modeling and generation tasks that are primarily compositional, generating non-compositional expressions is more challenging for current neural models, including large pre-trained language models. The main reasons are 1) their non-compositionality, and 2) the limited data resources. Therefore, to make the best use of available data for modeling non-compositionality, we propose a dynamic curriculum learning framework, which learns training examples from easy ones to harder ones thus optimizing the learning step by step but suffers from the forgetting problem. To alleviate the forgetting problem brought by the arrangement of training examples, we also apply a continual learning method into our curriculum learning framework. Our proposed method combined curriculum and continual learning, to gradually improve the model’s performance on the task of non-compositional expression generation. Experiments on idiomatic expression generation and metaphor generation affirm the effectiveness of our proposed curriculum learning framework and the application of continual learning. Our codes are available at https://github.com/zhjjn/CL2Gen.git.

## introduction natural language has a common yet special class of constructions called non-compositional expressions that exhibit semantic non-compositionality, where the meaning of the expression cannot be inferred from that of its constituent words (e.g., metaphors and idioms) @xcite . they are commonly used for specific communicative intents @xcite and are individually rare but collectively frequent, appearing frequently across genres @xcite . most of the idioms indexed by the oxford dictionary have a frequency of less than 1 per million in the corpus of contemporary american english (rafatbakhsh and ahmadi, 2019). they have been classically regarded as a "pain in the neck" to nlp systems @xcite not only because of their non-compositionality, but also because of their contextual semantic ambiguity (used in noncompositional or compositional meaning depending on the context). different nlp tasks related to non-compositional expressions have been studied, including sentiment analysis @xcite , paraphrase generation @xcite , natural language inference @xcite , metaphor detection @xcite ) and idiom usage recognition @xcite . however, the generation of non-compositional expressions remains an important yet under-explored problem. therefore, this paper focuses on the generation of non-compositional expressions. as is shown in ## framework in this section, we briefly introduce our proposed curriculum learning method for non-compositional expression generation. curriculum learning for efficiently leveraging available data resources consists of two main parts: a measure of difficulty of training instances, and an arrangement of the training examples using this measure. accordingly, for non-compositional expression generation, we propose a data arrangement method for dynamically arranging the training examples according to a newly studied difficulty metric. in addition, due to the current large pre-trained language models' insufficiency in processing non-compositional expressions @xcite , non-compositional expressions that are difficulty for lms to understand would have a high perplexity score and the representations between non-compositional expressions and their constituent words would be large. therefore, we use a combination of the representation distance and perplexity score as a measure of examples' difficulty. moreover, in our experiments, we observe that following the curriculum learning principle of arranging the training examples based on their difficulty levels, the problem of forgetting arises due to the gradual shift of distribution in domain difficulty. therefore, to alleviate this forgetting problem, we propose a simple yet effective continual learning method. figure 1 demonstrates the workflow of our proposed curriculum learning framework and its details as follows. ## analysis here we provide some ablation studies based on idiomatic expression generation to analyze the contribution of different modules used in our framework. difficulty measurement. as shown in ## conclusion and future work in this paper, we first utilize curriculum learning to better utilize available data for non-compositional expression generation. we propose a novel curriculum learning framework by utilizing representation distance and perplexity score as a measure of difficulty level and a dynamic scheduling method to better leverage the available training data. furthermore, for the first time we study a continual learning algorithm to alleviate the forgetting problem resulting from curriculum learning. experiments on two non-compositional expression generation tasks of idiomatic expression generation and metaphor generation show that the proposed curriculum learning framework can effectively boost the performance of non-compositional expression generation outperforming previously studied curriculum learning methods. future works should explore other difficulty metrics, more effective scheduling methods and continual learning schemes to further alleviate the forgetting problem, and study them for other text generation problems.