Handling Variance of Pretrained Language Models in Grading Evidence in the Medical Literature (2021)

Abstract: In this paper, we investigate the utility of modern pretrained language models for the evidence grading system in the medical literature based on the ALTA 2021 shared task. We benchmark 1) domain-specific models that are optimized for medical literature and 2) domain-generic models with rich latent discourse representation (i.e. ELECTRA, RoBERTa). Our empirical experiments reveal that these modern pretrained language models suffer from high variance, and the ensemble method can improve the model performance. We found that ELECTRA performs best with an accuracy of 53.6% on the test set, outperforming domain-specific models.1

## background evidence-based medicine (ebm) is an approach by health practitioners to integrate individual clinical expertise and external evidence from medical literatures in making decisions about the care of patients @xcite . in practice, understanding the current best evidence from the literature minimizes the unexpected risk of outdated treatments that can be detrimental to patients. strength of recommendation taxonomy (sort) @xcite is one of the standard scale systems for grading evidence in medical literature and it has been used to assist the ebm approach. sort groups a medical literature into one of three classes: a (consistent and goodquality patient-oriented evidence), b (inconsistent or limited-quality patient-oriented evidence) and c (other evidence, such as consensus guidelines, usual practice and opinion). while obtaining these grades on a wide-scale is expensive and requires in-depth medical expertise, previous works @xcite have attempted to automate the process by modelling the grading system with n-gram language model via svm @xcite and ensemble method @xcite . in this work, we focus on investigating the utility of various modern pretrained language models for modelling the evidence grading system in the medical literature. although transformer @xcite and pretrained language models such as bert @xcite , roberta @xcite have achieved impressive performance across various nlp tasks @xcite and languages @xcite , we hypothesize that such evidence grading task is still challenging because of three reasons. first, in-depth medical expertise and knowledge are not always present in the language models. second, it is very likely that machine learning models suffer from high variance as disagreement in assessing scientific literature is natural, even among the experts. lastly, obtaining high-quality training data for this task is difficult, and the large transformer-based models potentially suffer from overfitting if the available data is limited. to address the aforementioned challenges, we use three main strategies. first, we fine-tune domain-specific pretrained models @xcite that are optimized for medical literature. previous works @xcite @xcite @xcite have shown that such models contain domain-specific knowledge that can boost system performance. second, we argue that discourse is prominent for this task because each of three sort classes might have different document structure. for instance, patient-oriented literature and consensus guidelines potentially are written differently in 00667 a 10796398 11508437 00668 a 9036306 00669 c 7391096 11204962 7790481 6863528 00670 b 9569395 12069675 00671 b 11083602 10875559 15283004 terms of flow and discourse. in this work, rather than employing a complicated discourse parser @xcite @xcite , we rely on modern pretrained language models such as elec-tra @xcite that contains a rich latent discourse representation @xcite . lastly, similar to @xcite , we also perform ensemble learning to tackle the high variance issue of models. ## dataset we conduct our experiments based on the alta 2021 shared task foot_1 which aims to automatically grade evidence in the medical literature. the grading system follows the sort framework @xcite with three classes: a (strong), b (moderate) and c (weak). as shown in figure 1 each line in the training data is a single piece of evidence and consists of an id, a sort grade, and a list of resource/publication id(s) from pubmed. foot_2 each publication id is mapped to an xml file containing bibliographic information (e.g. title, author, affiliation, etc.), abstract, and some meta-data such as type and status of the publication. in ## proposed methods figure 2 describes the best model that we submit to alta 2021 shared task. we use filtered ensemble method over 3 domain-specific pretrained language models: 1) biomed bert @xcite , 2) biomed roberta @xcite and 3) biomed roberta that is further pretrained with the training set for 400 epochs, denoted as task adaptive pretraining (tapt) model; and 3 domaingeneric pretrained language models: 1) roberta @xcite , 2) electra, and 3) elec-tra (large) @xcite . the selection of roberta and electra is based on their rich latent discourse representation as reported by @xcite . given a list of resources or publications @xmath0, r 2 , .., r n } for evidence x, we construct an input sequence as follows. first, each resource r i consists of journal name j i , title t i , and abstract a i . we form an input sequence x as the concatenation of all texts j 1 ⊕ t 1 ⊕ a 1 ⊕ ... ⊕ j n ⊕ t n ⊕ a n . we truncate a resource r i if the tokens are more than 250, and set the maximum length of the input x to be 512. to understand the variance of pretrained language models in this task, we fine-tune each model with 100 different random seeds. for ensemble learning, we first select models with accuracy more than hyper-parameter α (values range between 0 and 1) and apply two types of voting mechanism to aggregate the prediction: 1) simple voting based on majority classes, and 2) filtered voting. for the second approach, if the selected n models have an even class distribution, we set class b as the prediction, otherwise normal majority voting is applied. mathematically, this even prediction is determined based on a threshold β as follows: x100 x100 x100 x100 100 models 100 models 100 models 100 models model filtering ensemble (vote) biomed bert electra (large) 100 models 100 models x100 x100 absolute difference of class a and b occurrence. β is a hyper-parameter with values ranging between 0 and n, and β < 0 means normal majority voting is applied. all parameters (including α and β) are tuned based on the development set. ## discussions and conclusion figure 3 describes label distributions on development and test sets using our best model, electra (large). first, we found that the model tends to predict class b on the development, with a disparity of +23 instances with the gold label b. in contrast, the model only classifies 31 instances as class c, despite being there 50 gold labels c. lastly, our final prediction in the test sets has a ratio of 40:109:34 of class a:b:c, respectively, and the graph in figure 3 describes a similar shape with the development set prediction. in conclusion, we have shown in this experiment that grading evidence in the medical literature is a challenging task, and modern pretrained language models suffer from high-variance issues. interestingly, we found that electra, the domaingeneral models outperform domain-specific models through ensemble methods. we argue that this is because discourse is one of the relevant features for this task. this is in line with @xcite that has shown that the last layer of electra contains the richest latent discourse representation, compared to bert, roberta, albert @xcite , gpt2 @xcite , bart @xcite , and t5 @xcite .