An Ensemble Model for Automatic Grading of Evidence (2021)

Abstract: This paper describes our approach for the automatic grading of evidence task from the Australasian Language Technology Association (ALTA) Shared Task 2021. We developed two classification models with SVM and RoBERTa and applied an ensemble technique to combine the grades from different classifiers. Our results showed that the SVM model achieved comparable results to the RoBERTa model, and the ensemble system outperformed the individual models on this task. Our system achieved the first place among five teams and obtained 3.3% higher accuracy than the second place.

## introduction evidence-based medicine (ebm) requires the making of clinical decisions using the current best external evidence rather than solely relying on clinical experience and pathophysiologic rationale @xcite . to adhere to ebm best practice, practitioners need to identify the best quality evidence associated with a clinical query. to grade the quality of evidence, @xcite proposed the strength of recommendation taxonomy (sort). sort has a three-levels for rating-a (strong), b (moderate), and c (weak), where a-level is based on high-quality studies with consistent results; blevel is based on high-quality studies with inconsistent results or some limitations; c-level is based on the studies with severe limitations. it is a straightforward grading system that allows clinical experts to rate individual studies or bodies of evidence based on quantity, quality, and consistency. to address the challenging problem of automatically grading the quality of evidence, the australasian language technology association (alta) shared task 2021 organized a competition. the participants were required to develop a system to predict the grade of evidence given multiple related medical publications. our team trained several supervised classifiers to address the problem. our approach included traditional supervised classification models such as support vector machines (svm) @xcite , neural network models using pretrained models (roberta) @xcite , and an innovative ensemble system which combines the predictions of multiple classifiers. our results showed that the svm model achieved comparable results to the roberta model, and the ensemble system outperformed the individual models on this task. the ensemble model combines the prediction from multiple classifiers in a unique manner: grades (a, b or c) predicted by each classifier is first converted into a continuous number, and then all the numbers are added for each instance. using the training data, the best separations for the numeric totals are computed. these numeric boundaries are then used to convert continuous scores in the test set to discrete evidence grades. our system achieved the first place among five teams and obtained 3.3% higher accuracy than the second place. ## related work alta shared task 2021 is a re-visit of alta shared task 2011 @xcite . previous studies have developed several svm-based systems for this task. @xcite used a sequential approach to combine multiple individual svm models trained with the features from the titles, body of the abstracts, and publication types. @xcite expanded the feature set proposed by @xcite with the medical subject headings (mesh) terms and developed a stacking-based approach to integrate predictions from multiple svm models. @xcite experimented with a larger set of features and applied multiple machine learning techniques such as classical machine learning mod-els, neural networks, game theory, and consensus methods. in our work, we trained svm models on a feature set similar to @xcite . we also applied a pre-trained transformer-based model named roberta @xcite , which has achieved state-of-the-art results in a wide range of natural language processing (nlp) tasks. ## data description the data for this shared task consisted of a set of evidence grades under the sort criteria and a list of related publications associated with each evidence grade. the publications were obtained from pubmed foot_0 and were provided in the form of xml files which contained the title, the abstract, and some meta-data (eg., publication types, mesh terms). some data statistics are shown in ## results for further error analysis, we plotted the confusion matrix for our best system (ie., the ensemble model), shown in figure 2 . as we can see, the majority of errors can be attributed to the misclassification of the classes a and c. most a-level and c-level evidences were predicted as b. this can be another indicator of overfitting because the majority evidences in the training set were graded as b. ## discussion as illustrated in