Self-trained Pretrained Language Models for Evidence Detection (2021)

Abstract: Argument role labeling is a fundamental task in Argument Mining research. However, such research often suffers from a lack of large-scale datasets labeled for argument roles such as evidence, which is crucial for neural model training. While large pretrained language models have somewhat alleviated the need for massive manually labeled datasets, how much these models can further benefit from self-training techniques hasn’t been widely explored in the literature in general and in Argument Mining specifically. In this work, we focus on self-trained language models (particularly BERT) for evidence detection. We provide a thorough investigation on how to utilize pseudo labels effectively in the self-training scheme. We also assess whether adding pseudo labels from an out-of-domain source can be beneficial. Experiments on sentence level evidence detection show that self-training can complement pretrained language models to provide performance improvements.

## introduction in the area of argument mining, obtaining highquality manually labeled data is often a complicated and expensive task @xcite . therefore, utilizing unlabeled data can help to achieve further improvements over standard supervised models. recently, pretrained language models have achieved significant improvements in a wide variety of downstream nlp tasks @xcite @xcite . these models utilize large unlabeled datasets to learn meaningful representations that are transferable across several tasks. in this work, we extend the utilization of unlabelled data in argument mining by proposing to use pretrained language models in a self-training manner. we focus on evidence detection which is an essential component in building natural language systems that are capable of arguing and debating. self-training is a semi-supervised technique that employs unlabeled data by using a teacher model, trained on labeled data, to generate pseudo labels out of unlabeled examples @xcite . the pseudo labeled data are then blended with manually labeled data to train a student model of a similar or larger size of the teacher model. another way to use pseudo labels is to blend them with labeled data and train a smaller student model. this method is referred to as knowledge distillation @xcite . in this paper, we seek to answer the following research questions: q1. under which conditions can self-training improve finetuned pretrained language models? to answer this question, we experiment with three different techniques to utilize the automatically generated pseudo labels to assess whether large pretrained models can benefit from self-training or not. (a) bootstrapped self-training: where we select automatically annotated instances of high confidence and add them to the training data during finetuning. (b) pretrain on pseudo labels: where we use the selected samples in pretraining the model before finetuning on the manually labeled set. (c) masked language model pretraining: where we use the selected samples to pretrain the model using a masked language model objective before finetuning the model on the manually labeled set. q2. is there a constraint on the domain of unlabeled data? in our experiments, we employ both in-domain and out-of-domain unlabeled corpora. q3. does increasing the similarity between labeled and unlabeled data improve self-training? we provide a retrieval step to filter unlabeled data by considering the most similar n samples to each example in the training data . our aim is to increase the similarity between labeled and unlabeled data by filtration. our contributions are as follows. (a) we propose a thorough investigation of self-training meth-ods for evidence detection over large pretrained language models. (b) we empirically show that with proper adjustments, self-training can indeed achieve improvements over a pretrained baseline. ## related work argument mining spans various lines of research work. @xcite , @xcite , and @xcite focus on identifying and classifying argument roles in text. another direction is to mine argument units that are relevant to specific claims or topics @xcite @xcite . our work directly extends the work in this area by @xcite . evidence detection, as viewed in this work, aims at classifying relevant sentences to a certain topic. @xcite benefits from large-scale weakly labeled data described in @xcite blended with manually annotated data to train a bilstm with glove embeddings and integrate the topic with an attention mechanism. our work also aims at making use of weakly labeled data. however, instead of using the retrieved data described in @xcite directly, we employ a teacher model trained on the manually annotated data to generate the pseudo labels for the unlabeled set. our work falls under the umbrella of semisupervised learning. in natural language processing, the recent use of unlabeled data has focused on pretraining large language models @xcite @xcite , which has led to remarkable improvements across a wide variety of nlp downstream tasks. in argument mining, @xcite retrieved sentences with seeds as in my opinion (imo) and in my humble opinion (imho) and used them to finetune a language model before finetuning on a claims dataset. @xcite used contextualized embeddings (elmo and bert) to classify and cluster argument components. in their work, they reported an 80% accuracy when finetuning bert over the evidence data described in @xcite and 81% when concatenating the topic with the input evidence. following the same line of work, we first finetune bert over the same evidence dataset and use it as our baseline model. we then extend finetuned bert by utilizing it as a teacher in a self-training manner, which hasn't been explored in the literature before. self-training has proven to be beneficial in a wide variety of tasks @xcite @xcite @xcite . in natural language processing, ruder and plank (2018) evaluated several semi-supervised baselines on sentiment analysis and part of speech tagging. they built their experiment over a bilstm baseline. while their work established solid baselines for semi-supervised learning in neural models, their methods focused on recurrent neural models. on the other hand, our experiments study self-training in the context of large pretrained language models. self-trained pretrained language models haven't been studied extensively in the literature. @xcite studied self-training as another way of leveraging unlabeled data on top of large pretrained language models. they achieved 2.6% improvements in standard classification tasks. @xcite used simple bootstrapped self-training over bert to improve zero-shot and few-shot classification of arabic dialects. we instead use self-training on top of pretrained language models for sentence-level evidence classification. our main incentive is to explore the utility of self-training techniques in argument mining where acquiring manually labeled data is usually hard to get in large quantities. ## approach our primary goal is to investigate different methods of self-training on top of large pretrained language models for evidence extraction. ## results after presenting the baseline pretrained finetuned language model results, we discuss how adding the different self-training approaches sheds light on the three research questions introduced in section 1. evidencebert. we start by replicating the results in @xcite , using the bert implementation from @xcite . for training hyperparameters, we finetune bert for 6 epochs of training. we employ adam (kingma and ba, 2014) optimizer for training and an initial learning rate of 4e -5. we chose our training parameters based on a manual search optimized on 5% of the training data. we achieve an accuracy of 80% on the test set, which is almost the same result reported in @xcite . q1. under which conditions can self-training improve finetuned pretrained language models? results suggest that by adding appropriate n pseudo examples, bootstrapped self-training and masked language model pretraining can improve accuracy over evidencebert. figure 1 shows that both bootstrapped self-training and masked language model pretraining can improve accuracy by 1% to 2% over finetuned bert at optimal n . 3 3 a similar level of improvement was found by @xcite while both masked language model pretraining and bootstrapped self-training can improve performance, the masked language model pretraining is more robust to the selected n pseudo examples (e.g., masked language modeling never degrades baseline performance). on the other hand, figure 1 implies that utilizing pseudo examples in regular pretraining always performs poorly when compared to the evidencebert baseline. q2. is there a constraint on the domain of unlabeled data? comparing figure 1 c and d to figure 1 a and b, respectively, suggests that using an unlabeled corpus from a different distribution like webis-debate-16 largely can achieve similar improvements as using unlabeled wikipedia. this is true for masked language model pretraining and bootstrapped self-training, both when comparing results to evidencebert at optimal n pseudo examples (except for masked language modeling in c versus a) and with respect to robustness over n . q3. does increasing the similarity between labeled and unlabeled data improve self-training? comparing figure 1 b and d to figure 1 a and c, respectively, shows that increasing the similarity between labeled and unlabeled data yields improvements in terms of accuracy at optimal n , particularly when using out of domain unlabeled data as both masked language modeling and bootstrapped self-training improve. robustness also improves. ## conclusion and future work we explored a variety of self-training configurations for evidence detection on top of bert. results show that 1) self-training with bootstrapped self-training and masked language model pretraining (but not with pretrain on pseudo labels) can improve finetuned large pretrained language models such as bert; 2) unlabeled data can be utilized from both in-domain (wikipedia) or out-of-domain (2021) in classification tasks outside of argument mining.