Quick, get me a Dr. BERT : Automatic Grading of Evidence using Transfer Learning (2021)

Abstract: We describe our methods for automatically grading the level of clinical evidence in medical papers, as part of the ALTA 2021 shared task. We use a combination of transfer learning and a hand-crafted, feature-based classifier. Our system (ï¿½orangutanV3ï¿½) obtained an accuracy score of 0.4918, which placed third in the leaderboard. From our failure analysis, we find that our classification techniques do not appropriately handle cases when the conclusions of across the medical papers are themselves inconclusive. We believe that this shortcoming can be overcomeï¿½thus improving the classification accuracyï¿½by incorporating document similarity techniques.

## introduction the recent covid-19 pandemic has once again highlighted the importance of evidence-based medicine (ebm) when deciding the course of treatment @xcite as social media and television shows are being flooded by so-called experts, who have voiced unproven treatments for covid-19 such as using hydroxychloroquine @xcite . however, the main challenge with ebm is that it is a manual and tedious process and it is very hard for practitioners to keep up with the rise in medical research @xcite . the challenges with ebm were no less true in 2011, when the australasian language technology association (alta) organised a shared task challenge to automatically grade evidence @xcite . the task was to grade evidence based on an ebm framework which is the strength of recommendation taxonomy (sort) @xcite . alta decided to revisit the 2011 challenge again this year (2021), motivated by the leaps in natural language processing (nlp) techniques that have occurred meanwhile @xcite . we investigate the following research questions with repsect to this challenge: our experimental results suggest that these two approaches do not perform well. our team placed third in the leaderboard with an accuracy score of 0.4918. none of the scores on the leaderboard beat the winning accuracy score in 2011, of 0.6284. this prompted us to perform an in-depth analysis of our approach, and how our work can be improved in the future to increase the overall accuracy of classification. ## related work in the medical literature, there are many different taxonomies that are used in order to rank the grade of a clinical study @xcite . one of the commonly used taxonomies, due to its simplicity, is @xcite . sort has been used in deciding whether to recommend root canal treatments @xcite , sports injury rehabilitation strategies @xcite , and in evaluation of cognitive behavioural treatment @xcite . there are three grades: a (strong), b (moderate) and c (weak). grade a reflects consistent and good-quality, patient-oriented evidence; grade b reflects being based on inconsistent or limited quality patient oriented evidence; lastly, grade c reflects a recommendation based on consensus, usual practice, opinion or disease-oriented evidence. the classification of sort is manually done by medical practitioners, and automating it is still in its infancy. to the best of our knowledge, the only researchers who explored automating sort are @xcite . in their work, the authors used a set of classifiers that utilised different feature sets such as n-grams, publication type and titles and then applied multiple svm classifiers. they obtained an accuracy score of 0.6284. transfer learning has shown vast improvement on a variety of downstream tasks such as summarization, translation, and question and answer interactions @xcite . one popular transfer learning method that is widely adapted is bert @xcite . driven by the success of bert, lee et al. (2020) introduced biobert (a biomedical focused version of bert) for tasks such as biomedical named entity recognition (ner), relation extraction, and summarization in the biomedical literature. recently, oniani and wang (2020) demonstrated that biobert provides an effective method for chatbots answering questions related to covid-19. ## data set the data set 1 provided by the organisers of the alta shared task consists of a collection of pubmed abstracts. there are 677 medical abstracts for training, 178 for development, and 183 for the testing set. the training and development data set come with the evidence id, followed by sort grade, and finally a list of pubmed ids of the abstracts. the test data contains the same, except for the sort grade. we analysed both the development and training data sets to understand the characteristics of the data. ## methodology we employed a two-phase approach to tackle the alta challenge. in the first phase, we used a pretrained biobert model and in the second phase, we used an svm classifier with handcrafted features such as h-index and the journal's impact factor. in this section, we describe our method in detail, along with the steps that we performed. we have made our system's source code publicly available on github. foot_0 ## results alta chose codalabs as the submission platform. the organisers provided us with both the development and test set. in codalabs, participants are allowed to submit an unlimited number of times for the development set but are only limited to three submissions for the test set. the submissions are evaluated by using the accuracy score. we ran our experiments on google cloud platform with 4 vcpus, 16 gib of ram and an nvidia tesla a100. we present our scores on the development set in ## conclusion we presented an approach to automatically grade evidence using a combination of transfer learning and a feature-based classifier. we competed in the alta 2021 competition under the team name "orangutanv3". despite achieving an accuracy score of 0.4918, we did not manage to beat the current state-of-the-art from ten years ago. the primary reason for our low score is attributed to our