S urya K iran at P rag T ag 2023 - Benchmarking Domain Adaptation using Masked Language Modeling in Natural Language Processing For Specialized Data (2023)

Abstract: Most transformer models are trained on English language corpus that contain text from forums like Wikipedia and Reddit. While these models are being used in many specialized domains such as scientific peer review, legal, and healthcare, their performance is subpar because they do not contain the information present in data relevant to such specialized domains. To help these models perform as well as possible on specialized domains, one of the approaches is to collect labeled data of that particular domain and fine-tune the transformer model of choice on such data. While a good approach, it suffers from the challenge of collecting a lot of labeled data which requires significant manual effort. Another way is to use unlabeled domain-specific data to pre-train these transformer model and then fine-tune this model on labeled data. We evaluate how transformer models perform when fine-tuned on labeled data after initial pre-training with unlabeled data. We compare their performance with a transformer model fine-tuned on labeled data without initial pre-training with unlabeled data. We perform this comparison on a dataset of Scientific Peer Reviews provided by organizers of PragTag-2023 Shared Task and observe that a transformer model fine-tuned on labeled data after initial pre-training on unlabeled data using Masked Language Modelling outperforms a transformer model fine-tuned only on labeled data without initial pre-training with unlabeled data using Masked Language Modelling.

## introduction transformer based models like @xcite , roberta @xcite , and deberta @xcite have become de-facto models for natural language processing (nlp) tasks outperforming all past techniques by significant margins. however, most of these models are originally 1 https://www.aclweb.org/portal/content/pragtag-sharedtask-argmining-workshop-2023 trained on english corpus such as bookcorpus @xcite , english wikipedia, and open-webtext @xcite . this becomes an issue when dealing with data from specialized domains such as medicine, healthcare, law, scientific peer reviews, etc. because these models are not aware of the specialized vocabulary in the domains due to which their performance is is generally subpar. this can be seen in @xcite where bert performs poorly as compare to a model initialized with bert weights and pre-trained on medical data. training transformer based models on data of specialized domain from the ground up poses significant challenges due to the scarcity of extensive datasets within these domains. so we resort to the practice of refining models originally trained on the english corpus by incorporating data sourced from such domains. traditionally, this refinement process entails acquiring labeled data, structured according to well-defined formats pertinent to a task within the domain of interest. subsequently, the model undergoes fine-tuning using this collected data. this approach is not efficient due to the laborintensive and expensive nature of gathering substantial volume of labeled data. an alternative strategy -when we have a lot of unlabeled data and only a handful of labeled data -is domain adaptation (da). in this paper we benchmark masked language modelling (mlm) @xcite as a da strategy and see how it performs on pragtag-2023 shared task @xcite . although it is one of the strategies used to pre-train bert, it has shown promise as a da technique as can be seen in @xcite , @xcite . ## related work according to v7 labs 2 , domain adaptation (da) is a technique to improve the performance of a model on a target domain containing insufficient annotated data by using the knowledge learned by the model from another related domain with adequate labeled data. source domain is the data distribution on which the model is trained using labeled examples. target domain is the data distribution on which a model pre-trained on a different domain is used to perform a similar task. in this paper, source domain is the data distribution present in english corpus such as bookcorpus, english wikipedia, and openwebtext and target domain is the data distribution present in the data of this shared task. there are primarily four types of da techniques -supervised da, semi-supervised da, weakly supervised da, and unsupervised da. for this paper, we will primarily focus on supervised and unsupervised da. in supervised domain adaptation (sda), target domain data is completely labeled. in unsupervised domain adaptation (uda), any kind of labels for the target domain data are entirely missing. @xcite initialize biobert with weights from bert, which was pre-trained on general domain corpora. then, biobert is pre-trained on biomedical domain corpora. to show the effectiveness of our approach in biomedical text mining, biobert is fine-tuned and evaluated on three popular biomedical text mining tasks -ner, re, and qa. the authors show that pre-training bert on biomedical corpora largely improves its performance on these three tasks. @xcite start from a model that is pretrained on general corpora, keep pretraining it on target domain data using the mlm task. on the final fine-tuning step, the update the model weights using both a classification loss on the labeled source data and masked language modeling loss on the unlabeled target data. @xcite use bert-base model for mlm and finetune it for text classification on the target dataset. they freeze the encoder layer while training only the embedding and final task-specific dense layers. by doing so, they specialise the general domain word representations according to the target tasks and show that the performance of the resultant model is better than only bert-base model. in this paper, we will focus on masked language modelling (mlm) which is a type of pre-training method that was introduced in bert. ## task description in this task, we are given two datasets extracted from @xcite . both of these datasets contain a multi-domain collection of freetext peer reviews labeled with pragmatic labels on the sentence level. in the first dataset, each peer review can belong to medical articles, computer science, and scientific policy research. it has two parts parts -training dataset and test dataset. training dataset is used to train the model and test dataset is used to evaluate the performance of the model trained on the training dataset. going forward, we refer to these two datasets as train dataset and full dataset respectively. the second dataset is a secret test set secret dataset. train, full, and secret dataset contain same domains with secret dataset containing one additional domain not present in train or full datasets. every sentence in these datasets has one of the following pragmatic labels: recap, strength, weakness, todo, other, and structure. our goal in this task is to correctly classify each peer review sentence into one of these categories. in addition to these datasets, we use an auxiliary dataset, f1000raw, extracted from @xcite which is used for pre-training. this is a large unlabeled collection of peer reviews. ## methodology for our experiments, we use deberta-base since it has been shown to perform better than models like bert and roberta. we first pre-train deberta-base on f1000raw using masked language model (mlm) as shown in fig. 1 . we then fine-tune this model using train dataset. we also fine-tune a deberta-base model using only train dataset without the pre-train step. we can see this workflow in fig. 2 . we then pass each review from full and secret datasets, take an average of the logits for all the classes and output the class with the highest logit score as shown in fig. 3 . we then compare the performance of these two models and show that mlm helps improve the performance of the model on this classification task. ## implementation details our solution comprises of two steps - deberta-base (with/without pretraining) train dataset split into 5 folds using groupkfold for every split multi-class classification fine-tuned deberta-base model for every split (with/without pretraining) figure 2: finetuning deberta-base fine-tuned pre-trained deberta-base model split 1 ... split 5 full / secret dataset average logits for every prediction of every sentence predict the category with the highest average logit score as the final category figure 3: inference on fine-tuned deberta-base ## results and discussion we evaluate results on three datasets -1) train dataset, 2) full dataset, and 3) secret dataset. for evaluating performance using train dataset, we use test dataset created in 5.2 of every split and pass it through the model trained using training data from that split. for evaluating performance on full and secret datasets, we pass each review from these datasets through all five models, take an average of the logits for all the classes and output the class with the highest logit score. train dataset gives us an idea about how both of the models compare across different splits and if one model is consistently better than the other model. full dataset contains similar domains as we have in train dataset but doesn't contain target variable. in secret dataset we have a new domain in addition to domains present in full dataset. the scores for every split of train dataset can be found one interesting observation from ## conclusion and future work as we can see in the results, domain adapted deberta-base beat deberta-base without domain adaptation. while this is an encouraging result, how is this performance difference impacted by the scale of models and architecture of the model remains to be studied. we also need to study this problem on datasets from other niche domains as well. in addition to this, we can also study how domain adaptation impacts llms which are orders of magnitude larger than architectures such as bert, roberta, and deberta.