D omain A dversarial F ine- T uning as an E ffective R egularizer (2020)

Abstract: In Natural Language Processing (NLP), pretrained language models (LMs) that are transferred to downstream tasks have been recently shown to achieve state-of-the-art results. However, standard fine-tuning can degrade the general-domain representations captured during pretraining. To address this issue, we introduce a new regularization technique, AFTER; domain Adversarial Fine-Tuning as an Effective Regularizer. Specifically, we complement the task-specific loss used during fine-tuning with an adversarial objective. This additional loss term is related to an adversarial classifier, that aims to discriminate between in-domain and out-of-domain text representations. Indomain refers to the labeled dataset of the task at hand while out-of-domain refers to unlabeled data from a different domain. Intuitively, the adversarial classifier acts as a regularize which prevents the model from overfitting to the task-specific domain. Empirical results on various natural language understanding tasks show that AFTER leads to improved performance compared to standard fine-tuning.

## introduction current research in nlp focuses on transferring knowledge from a language model (lm), pretrained on large general-domain data, to a target task. the lm representations are transferred to the target task either as additional features of a task-specific model @xcite , or by finetuning @xcite @xcite . standard fine-tuning involves initializing the target model with the pretrained lm and training it with the target data. fine-tuning, however, can lead to catastrophic forgetting @xcite , if the pretrained lm representations are adjusted to such an extent to the target task, that most generic knowledge, captured during pretraining, is in effect for-gotten @xcite . a related problem of fine-tuning is overfitting to the target task, that often occurs when only a small number of training examples is available @xcite . adversarial training is a method to increase robustness and regularize deep neural networks @xcite . it has been used for domain adaptation @xcite to train a model from scratch to produce representations that are invariant to different domains. inspired by this approach, we propose a regularization technique for the fine-tuning process of a pretrained lm, that aims to optimize knowledge transfer to the target task and avoid overfitting. our method, domain adversarial fine-tuning as an effective regularizer (after) extends standard fine-tuning by adding an adversarial objective to the task-specific loss. we leverage out-of-domain unlabeled data (i.e. from a different domain than the target task domain). the transferred lm is fine-tuned so that an adversarial classifier cannot discriminate between text representations from indomain and out-of-domain data. this loss aims to regularize the extent to which the model representations are allowed to adapt to the target task domain. thus, after is able to preserve the general-domain knowledge acquired during the pretraining of the lm, while adapting to the target task. our contributions are: (1) we propose after, an lm fine-tuning method that aims to avoid catastrophing forgetting of general-domain knowledge, acting as a new kind of regularizer. (2) we show that after improves the performance of standard fine-tuning in four natural language understanding tasks from the glue benchmark @xcite , with two different pretrained lms: bert @xcite , and xlnet @xcite . (3) we further conduct an ablation study to provide useful insights regarding the key factors of the proposed approach. various adversarial losses have been used for domain adaptation in several nlp tasks, such as question answering @xcite , machine reading comprehension @xcite and crosslingual named entity recognition @xcite . adversarial approaches have been also used to learn latent representations that are agnostic to different attributes of the input text, such as language @xcite and style @xcite . contrary to previous domain adaptation work, we explore the addition of an adversarial loss term to serve as a regularizer for fine-tuning. other variants of lm fine-tuning include a supplementary supervised training stage in data-rich tasks @xcite or multi-task learning with additional supervised tasks @xcite . however, such methods require additional labeled data. a common way to leverage unlabeled data during fine-tuning is through an additional stage of language modeling. for this stage, the unlabeled data can either come from the task-specific dataset (i.e. the labels are dropped and language modelling is performed on the input data) @xcite , or additional unlabeled in-domain corpora @xcite . this approach adds a computationally expensive step that requires unlabeled data from a specific source. by contrast, our method leverages out-of-domain data with only a small computational overhead and minimal changes to the fine-tuning process. our work is compatible with the semi-supervised learning paradigm @xcite that combines learning from both labeled and unlabeled data. in this setting, unlabeled data from the task domain is leveraged using a consistency loss which enforces invariance of the output given small perturbations of the input @xcite . the adversarial loss term of after can be interpreted as a consistency loss that ensures invariance of representations across domains. recently, adversarial or trust region based approaches @xcite @xcite have been proposed as an extension to the lm fine-tuning process. these methods introduce constraints that prevent aggressive updating of the pretrained parameters or enforce smoothness during fine-tuning. however, these approaches require additional forward and backward computations while our method is more computationally efficient and can be implemented with minimal changes to the fine-tuning procedure. ## proposed approach fig. 1 provides a high-level overview of after. problem definition. we tackle a main task, with a labeled dataset from domain d m . we further exploit an existing unlabeled corpus, auxiliary, that comes from a different domain d aux . we label each sample with the corresponding domain label y d , y @xmath0, and y @xmath1. we note that we do not use any real labels from auxiliary (if there are any). the domain labels are used to train a classifier that discriminates between d m and d aux . model. we initialize our model with pretrained weights from a top-performing language model, such as bert @xcite or xl-net @xcite . the representation of both bert and xlnet for the input sequence is encoded in the [cls] token output embedding. we add a linear layer on top of the sequence representation ([cls] output embedding) for the main task, resulting in a task-specific loss l m ain . we also add another linear layer for the binary domain classifier (figure 1 ), with a corresponding loss l domain , which has the same input. adversarial fine-tuning. the domain discriminator outputs a domain label for each sample of the training set. we seek representations that are both discriminative for the main task and indiscriminative for the domain classifier. hence, we minimize l m ain and at the same time maximize l domain , by fine-tuning the pretrained lm with the joint loss: ## experiments datasets. we experiment with four main datasets from the glue benchmark @xcite . the chosen datasets represent the broad variety of natural language understanding tasks, such as linguistic acceptability (cola) @xcite , sentiment analysis (sst-2) @xcite , paraphrase detection (mrpc) @xcite and textual entailment (rte) @xcite @xcite . the datasets used represent both high (sst-2) and lowresource (rte, cola, mrpc) tasks, as well as single-sentence (cola, sst-2) and sentence-pair (mrpc, rte) tasks. for auxiliary data we select corpora from various domains. for the news domain we use the ag news dataset @xcite and for the reviews domain we use a part of the electronics reviews of @xcite . for the legal domain we use the english part of europarl @xcite and for the medical domain we use papers from pubmed, provided by @xcite . we also use math questions from the dataset of @xcite for the math domain. implementation details. we base our implementation on hugging face's transformers library @xcite in pytorch @xcite . we make our code publicly available foot_0 . we tune the λ hyperparameter of eq. 1 on the validation set for each experiment, finding that most values of λ improve over the baseline. we fine-tune each model for 4 epochs and evaluate the model 5 times per epoch, as suggested by @xcite . we select the best model based on the validation loss. for more implementations details see appendix a.2. ## results bert. we observe that the proposed approach (after) outperforms the first baseline (bert sft) in all four tasks. for most of these tasks, after results in improved performance with every auxiliary dataset, demonstrating the robustness of our approach across domains. specifically, in cola, we observe that finetuning with the adversarial loss substantially outperforms standard fine-tuning. specifically, using an auxiliary dataset from the news domain improves the baseline by 1.8 points. in sst-2, we notice that although standard fine-tuning achieves high accuracy, the use of after still results in slight performance gains (∼ 0.4%). similar to cola, these improvements are consistent across auxiliary datasets and often come with reduced variance, compared to sft. in mrpc, we observe gains of 1.5 points on average in accuracy and 1.0 in f1 over sft. using news data as auxiliary, after outperforms the baseline by 2.1 points in accuracy and 1.5 in f1. in rte, the proposed approach improves upon the baseline from 64.3% to 64.8% in accuracy, using data from the legal domain. however, we also observe deteriorated performance with the use of some auxiliary datasets (e.g. medical, math). we attribute this result to the similarity between the domain of rte (wikipedia) and the domain of the pretraining corpus of bert (wikipedia and books). we test this hypothesis in section 6. xlnet. we observe in specifically, in sst-2 after improves the accuracy of standard fine-tuning (sft) by 0.6% on average and reduces variance, as well. for instance, with the use of auxiliary data from news or math domains, after results in 0.9% improvement in accuracy. in mrpc, the performance boost is also consistent across auxiliary data. in particular, the use of legal data leads in absolute improvement of 1.1% in accuracy and 0.8% in f1. in rte, adversarial fine-tuning outperforms the baseline by 1.4% in accuracy. however, similar to bert, we observe lower performance when using after with some auxiliary data (e.g. news, medical). we attribute this performance degradation to the same reason as bert, the similarity between the pretraining corpus domain and the target task domain (both lms have similar pretraining corpora). ## ablation study we investigate the effect of some key factors of after such as the relation of the target task domain and the domain of the pretraining corpus of the lm, the selection of auxiliary data and the emergence of domain-invariant characteristics. for the experiments of this section we used bert, unless otherwise stated. lm pretraining and task domains. to explore why after fails to improve upon the baseline on rte, we examine if the pretrained representations are already well suited for the task (i.e. no regularization is needed). we calculate the average masked lm (mlm) loss of bert for each main dataset. we observe in ## conclusions and future work we propose after, a domain adversarial method to regularize the fine-tuning process of a pretrained lm. empirical results demonstrate that our method can lead to improved performance over standard fine-tuning. after can be widely applied to any transfer learning setting and model architecture, with minimal changes to the fine-tuning process, without requiring any additional labeled data. we aim to further explore the effect of auxiliary data on the final performance and the use of multiple auxiliary datasets. we also aim to extend the proposed approach as a way to fine-tune a pretrained lm to a different language, in order to produce language-invariant representations.