Incorporating Syntactic Knowledge into Pre-trained Language Model using Optimization for Overcoming Catastrophic Forgetting (2023)

Abstract: Syntactic knowledge is invaluable information for many tasks which handle complex or long sentences, but typical pre-trained language models do not contain sufficient syntactic knowledge. Thus it results in failures in downstream tasks that require syntactic knowledge. In this paper, we explore additional training to incorporate syntactic knowledge to a language model. We designed four pre-training tasks that learn different syntactic perspectives. For adding new syntactic knowledge and keeping a good balance between the original and additional knowledge, we addressed the problem of catastrophic forgetting that prevents the model from keeping semantic information when the model learns additional syntactic knowledge. We demonstrated that additional syntactic training produced consistent performance gains while clearly avoiding catastrophic forgetting.

## introduction pre-trained language models are commonly used and improve the performance of a variety of application tasks. it has been shown that those models roughly capture syntactic knowledge @xcite , such as dependency labels, but they lack some syntactic knowledge such as dependency distance and head token required for application tasks @xcite . recent studies have shown that incorporating syntactic knowledge into the models further improves the performance of language understanding @xcite and translation tasks @xcite by adding other modules to the core model in the training and application phases. in contrast, @xcite and @xcite take approaches to embed syntactic knowledge into a model with keeping model structures. in this paper we follow this approach (henseforth we call it additional syntactic training), because the models empowered with syntax knowledge and retaining semantic knowledge can be easily applied to downstream tasks and, the methodology is useful for injecting other knowledge into the language model to create knowledge-enhanced models such as domain-specific ones. figure 1 depicts our approach and techniques to obtain desirable language models. we start with a language model such as bert @xcite which was pre-trained with a large amount of unlabeled text data and self-supervised tasks such as masked language modeling. such a model contains semantic knowledge but does not have explicit syntactic information, thus we enhance the model with labeled data with syntactic information. one major issue of additional syntactic training is that we need to find a good balance between the original and additional knowledge for improving performance of downstream tasks. most of the previous works have selected dependency structure predictions as the task to learn syntactic knowledge. however, there are many other specific syntactic notions, such as phrase structures and coordina-tions from which the additional training may benefit, and it is non-trivial to identify the best ones. another big issue in balancing the model and additional knowledge is catastrophic forgetting @xcite , where additional training causes a significant loss of important knowledge embedded in the original model. to tackle this problem, we exploit multiple optimizers and quantitatively observe the balance of capabilities to predict semantic and syntactic features. to the best of our knowledge, our study is the first to analyze how both the types of additional syntactic training and its optimizations affect downstream tasks. our contributions in this paper are as follows: ## additional syntactic training in additional syntactic training, models mainly learn syntactic relationships between two tokens. there have been proposed tasks such as predicting dependencies @xcite ), predicting parent-child, sibling, or cousin relationships @xcite , and dependency distance. the dependency structure encompasses a variety of word relationships, a number of which are important in application tasks. for example, in noun phrase extraction @xcite and sentiment analysis @xcite , the relationship between main and subordinate clauses and parallel (coordinate) structures are not handled well. to reflect the syntactic knowledge missing and required for downstream tasks in pre-trained models, we need to perform more diverse additional syntactic training tasks. we therefore develop four pre-training tasks to predict various syntactic items. each task predicts specific syntactic information: 1) deprel prediction, 2) phrase detection, 3) main/subordinate classification, and 4) coordination detection. we investigate which task would be effective on applications. ## optimization functions to prevent catastrophic forgetting this section describes two optimization functions that prevent catastrophic forgetting for keeping original semantic knowledge during additional syntactic training. when the model has already been trained for a specific task and is then trained for another task, the performance of the previous task significantly decreases. this is called catastrophic forgetting @xcite . although large-scale pre-trained models are known to be relatively resilient to catastrophic forgetting @xcite , the problem remains when using small models such as bert @xcite . since bert is still often used in the real world task due to computational resources, a variety of methods are developed to prevent catastrophic forgetting even in pre-trained models such as decreasing the learning rate @xcite . among these methods, we used gradient surgery @xcite , an optimization function known in the domain of multi-task learning, and elastic weight consolidation @xcite , an optimization function commonly used in continual learning. ## preliminary experiment: additional syntactic training here, we first show how well pre-trained models with semantic information can learn specific syntactic information by additional syntactic training tasks (see section 2.2 for detail) using various optimization functions while preserving semantic information. train dev test original 12543 2001 2077 after filtering 10271 1471 1485 ## experiment 1: glue we investigate the effects of syntactic pre-trained models to multiple downstream tasks using glue benchmark @xcite . glue has three types of tasks: single sentence tasks (cola, sst-2), similarity and paraphrase tasks (mrpc, qqp and sts-b) and natural language inference tasks (mnli, qnli, rte). we conducted detailed experiments on three tasks for which syntactic information seems to be important in solving the tasks: cola @xcite , rte @xcite , and mrpc @xcite . experiments on other tasks that did not required significant syntactic knowledge are shown in appendix. from experiments of the glue subset, it can be concluded that additional syntactic training is beneficial for tasks involving syntactic knowledge, as opposed to the tasks mentioned in the appendix a, which do not appear to require such knowledge. ## experiment 2: key phrase extraction in addtion to binary classification tasks in glue, we conducted another experiment on the key phrase extraction task @xcite . it is a task to detect domain-specific phrases in sentences, and is considered a good workbench to test the ability to extract syntactically correct sequences as an application close to real-world use cases. ## conclusion this paper investigated the incorporation of syntactic knowledge into a pre-trained bert model by additional training. we designed four additional syntactic training tasks: deprel prediction, phrase detection, main/subordinate classification, and coordination detection. we utilized gs and ewc optimizations for preserving semantic knowledge. the experimental results showed improvements of the models empowered with syntactic knowledge in the downstream tasks, and supported our intuition that each task requires different syntactic perspectives for more accurate predictions.