Multi-Stage Pre-training for Low-Resource Domain Adaptation (2020)

Abstract: Transfer learning techniques are particularly useful for NLP tasks where a sizable amount of high-quality annotated data is difficult to obtain. Current approaches directly adapt a pretrained language model (LM) on in-domain text before fine-tuning to downstream tasks. We show that extending the vocabulary of the LM with domain-specific terms leads to further gains. To a bigger effect, we utilize structure in the unlabeled data to create auxiliary synthetic tasks, which helps the LM transfer to downstream tasks. We apply these approaches incrementally on a pretrained Roberta-large LM and show considerable performance gain on three tasks in the IT domain: Extractive Reading Comprehension, Document Ranking and Duplicate Question Detection.

## introduction pre-trained language models @xcite @xcite have pushed performance in many natural language processing tasks to new heights. the process of model construction has effectively been reduced to extending the pre-trained lm architecture with simpler taskspecific layers, while fine-tuning on labeled target data. in cases where the target task has limited labeled data, prior work has also employed transfer learning by pre-training on a source dataset with abundant labeled data before fine-tuning on the target task dataset @xcite @xcite . however, directly fine-tuning to a task in a new domain may not be optimal when the domain is distant in content and terminology from the pre-training corpora. to address this language mismatch problem, recent work @xcite @xcite has adapted pre-trained lms to specific domains by continuing to train the same lm on target domain text. similar approaches are also used in multilingual adaptation, where the representations learned from multilingual pre-training are further optimized for a particular target language @xcite . however, many specialized domains contain their own specific terms that are not part of the pre-trained lm vocabulary. furthermore, in many such domains, large enough corpora may not be available to support lm training from scratch. to resolve this out-of-vocabulary issue, in this work, we extend the open-domain vocabulary with in-domain terms while adapting the lm, and show that it helps improve performance on downstream tasks. while language modeling can help the model better encode the domain language, it might not be sufficient to gain the domain knowledge necessary for the downstream task. we remark, however, that such unlabeled data in many domains can have implicit structure which can be taken advantage of. for example, in the it domain, technical documents are often created using predefined templates, and support forums have data in the form of questions and accepted answers. in this work, we propose to make use of the structure in such unlabeled domain data to create synthetic data that can provide additional domain knowledge to the model. augmenting training data with generated synthetic examples has been found to be effective in improving performance on low-resource tasks. @xcite , @xcite , @xcite and @xcite develop approaches to generate natural questions that can aid downstream question answering tasks. however, when it is not possible to obtain synthetic data that exactly fits the target task description, we show that creating auxiliary tasks from such unlabeled data can be useful to the downstream task in a transfer learning setting. for preliminary experiments in this short paper, we select the it domain, partly because of the impact such domain adaptation approaches can have in the technical support industry. the main contributions of this paper are as follows: (1) we show that it is beneficial to extend the vocabulary of a pre-trained language model while adapting it to the target domain. (2) we propose to use the inherent structure in unlabeled data to formulate synthetic tasks that can transfer to downstream tasks in a lowresource setting. (3) in our experiments, we show considerable improvements in performance over directly fine-tuning an underlying roberta-large lm @xcite on multiple tasks in the it domain: extractive reading comprehension (rc), document ranking (dr) and duplicate question detection (dqd). 1 ## datasets we use two publicly available it domain datasets. techqa @xcite is an extractive reading comprehension @xcite dataset developed from real user questions in the customer support domain. each question is accompanied by 50 documents, at most one of which has the answer. a companion collection of 801k unlabeled technotes is provided to support lm training. in addition to the primary reading comprehension task (techqa-rc), we also evaluate on a new document ranking task (techqa-dr). given the question, the task is to find the document that contains the answer. askubuntu 2 @xcite is a dataset containing user-marked pairs of similar questions from stack exchange 3 , which was developed for a duplicate question detection task (askubuntu-dqd). a static offline dump of askubuntu, which is organized as a set of forum posts 4 , is also available and can be used for lm training. ## vocabulary extension for lm adaptation texts in specialized fields including technical support in the it domain may contain numerous technical terms which are not found in open domain corpora and are therefore not well captured by the vocabulary of out-of-the-box lms. these terms are often over-segmented into small pieces (sub-word tokens) by the segmenter rules, which are learned from the statistics of open domain language. as an example, the token out-of-vocabulary (oov) rate of the standard roberta vocabulary in the techqa technotes data is 19.8% and the bpe/tok ratio is 1.32. contrast this with the analogous figures for 1m randomly selected wikipedia sentences, where the oov rate is only 8.1% and the bpe/tok ratio is 1.12. while transformer-based pre-trained language models @xcite yield better representations of previously unseen tokens than traditional n-gram models, over-segmentation can still cause degradation in downstream task performance. we address this challenge by augmenting the vocabulary of the pre-trained lm with frequent in-domain words. specifically, the most frequent oov tokens after tokenization are recorded and used to bypass the bpe segmentation stage. this prevents the segmenter from splitting these terms into smaller pieces. new entries in the lm vocabulary and corresponding word embeddings are created for these tokens. in our experiments, the number of such protected tokens is decided using an empirical criterion: we require that 95% of the in-domain data be covered by the extended vocabulary. we add 10k new items to the vocabulary for the technotes corpus and 5k for the askubuntu corpus. the variation in coverage due to different numbers of new vocabulary entries is shown in the appendix. the pre-trained lm is then adapted to the domain-specific corpus via masked lm (mlm) training. the embeddings of the new vocabulary are randomly initialized and then learned during the mlm training. the embeddings of existing vocabulary are also fine-tuned in this phase. ## task-specific synthetic pre-training while in-domain lm pre-training reveals novel linguistic patterns in target domain text, in many domains including technical documents, structure present in unlabeled text can contain useful information closer to actual end tasks. in this section, we propose to utilize such structure in unlabeled data to create auxiliary pre-training tasks and associated synthetic training data, which in turn can help target tasks via transfer learning. techqa. the techqa dataset release contains a companion technotes collection with 801k human written documents with titled sections. we observe that certain sections in these documents (e.g., abstract, error description and question) correspond to a problem description, while others (e.g., cause and resolving the problem) describe the solution 5 . we create an auxiliary reading comprehension (rc) task from these documents. specifically, if a document contains both problem and solution sections, a synthetic example is created where the problem description section is the query, the solution section is the target answer, and the entire document excluding the query section is the context. additionally, ten other documents are sampled from the technotes corpus as negatives to simulate unanswerable examples. this auxiliary task trains an intermediate rc model which predicts the start and end positions of the solution section as the answer given the document and the problem description. while our main goal here is to generate long-answer examples common in techqa, the general idea of utilizing the document structure can be applicable in other scenarios including in scientific domains like bio/medical (g. tsatsaronis, g. balikas, p. @xcite where structured text is relatively common. askubuntu. the askubuntu dataset contains a web dump of forum posts, each containing a question and multiple answers, with one answer possibly labeled by users as "accepted". motivated by @xcite @xcite , we create an auxiliary answer selection task from this structure. each instance in the synthetic data for this task contains a question, its accepted answer as the positive class, and an answer randomly sampled from other question posts 5 here is a sample technote: link as the negative class. an intermediate classification model is learned from these annotations, whose weights are used to initialize the target duplicate question detection (dqd) model. even though this auxiliary task adopts a different question-answer classification objective than the dqd task's objective of question-question classification, our experimental results show that the former still serves a good initialization for the latter. ## conclusion in this work, we show that it is beneficial to extend the vocabulary of the lm while fine-tuning it on the target domain language. we show that extending the pre-training with task-specific synthetic data is an effective domain adaptation strategy. we empirically demonstrate that structure in the unsupervised domain data can be used to formulate auxillary pre-training tasks that can help downstream low-resource tasks like question answering and document ranking. in our preliminary experiments, we empirically show considerable improvements in performance over a standard roberta-large lm on multiple tasks. in future work, we aim to extend our approach to more domains and explore more generalizable approaches for unsupervised domain adaptation.