Overview of the 2021 ALTA Shared Task: Automatic Grading of Evidence, 10 years later (2021)

Abstract: The 2021 ALTA shared task is the 12th instance of a series of shared tasks organised by ALTA since 2010. Motivated by the advances in machine learning in the last 10 years, this yearï¿½s task is a re-visit of the 2011 ALTA shared task. Set within the framework of Evidence Based Medicine (EBM), the goal is to predict the qual-ity of the clinical evidence present in a set of documents. This yearï¿½s participant results didnot improve over those of participants from 2011.

## introduction evidence based medicine (ebm) urges the medical practitioner to make use of the best available evidence for making decisions about the care of individual patients @xcite . however, medical and biomedical research generates such a volume of publications that it is unrealistic for a medical doctor or researcher to be able to read all relevant publications in order to be up to date on the available medical evidence. for example, pubmed currently contains more than 33 million citations for biomedical literature foot_0 . a more recent collection, cord-19, contains over 500,000 publications on topics related to covid-19, sars-cov-2, and related coronaviruses foot_1 . an important step for determining the best clinical evidence is to grade the quality of the available evidence. to help address this problem, in 2011 the alta shared task launched the task of automatic evidence grading @xcite . the goal of the task was to build a system that predicts the grade of evidence available in a set of medical publications. forward 10 years, in 2021, the task has been re-visited. the 2021 task uses the same training and test data sets as in 2011, and the evaluation framework has been re-created as closely as possible to match the 2011 evaluation framework. we wanted to know whether the recent advances in machine learning over the last 10 years lead to an improvement in the accuracy of the automatic grading of evidence predictors. this paper describes the specific set up of the 2021 alta shared task, and shows the results of the participating systems. back in 2011, no participating systems improved on a majority baseline. in 2021, the results of the participating systems appear to improve over the majority baseline, but the difference is not statistically significant. section 2 gives more details about the automatic grading of evidence task. section 3 presents related work since 2011. section 4 details the evaluation framework. section 5 presents the participating systems and their results, and section 6 concludes this paper. ## evidence grading several taxonomies have been defined to grade the quality of the medical evidence. the strength of recommendation taxonomy (sort) @xcite , used in the 2011 alta shared task, is one such taxonomy. sort uses a 3-point scale defined as follows: a recommendation based on consistent and good quality patient-oriented evidence. b recommendation based on inconsistent or limited quality patient-oriented evidence. c recommendation based on consensus, usual practice, opinion, disease-oriented evidence, and case series for studies of diagnosis, treatment, prevention, or screening. in addition to the above definitions, ebell et al. ( 2004 ) provides details on how to determine each grade, including a flowchart. medical evidence is not necessarily bound to one publication only. there may be several publications related to a particular disease, treatment or diagnosis, and each of them may be of different quality. further, it may indeed happen that each of the separate publications produces consistent results, but the evidence of the set of publications is inconsistent; when that happens the evidence grade cannot be of type a, as per the definitions above. ## related work the 2011 alta shared task overview paper @xcite presents a short survey of related work prior to 2011. as we see in this section, there has been limited research since then. none of the participants to the 2011 alta shared task @xcite ) outperformed a majority baseline ("predict b", with an accuracy of 0.4863), and the participating systems did not publish the system descriptions. a more sophisticated approach developed by the organisers of the 2011 shared task did manage to beat the baseline, reaching an accuracy of 0.6284. their approach was based on cascaded support vector machine (svm) classifiers which were trained to separate class a and c from the default b with high precision. these svm classifiers used combinations of the following features: ngrams of the abstract and title (with general medical semantic types replacing specific medical terms), and publication types (combining the publication types provided in the original abstracts with types generated by applying ad-hoc rules). the work was subsequently extended and published with more detail by @xcite . @xcite reported an improved accuracy of 0.7377 on the same dataset by using a two-level stacking approach. in the first level, multiple svm classifiers are trained using separate feature sets. then, their output is fed to a second svm classifier. their feature sets included publication types, mesh terms, title, abstract text, abstract method section, and abstract conclusion section. all of these features were as provided by the abstracts, except for the method and conclusion section, which were determined heuristically when not provided by the abstracts. @xcite reported an accuracy of 0.7541, again on the same dataset, after applying a wide range of different variants of stacked classifiers. 00001 b 10553790 15265350 00002 c 12804123 16026213 14627885 00003 b 15213586 00004 a 15329425 9058342 11279767 figure 1 : sample training data. each row indicates one evidence that needs to be graded. the first number is the evidence id. this is followed by the evidence grade, and the list of pubmed ids for the relevant documents. ## evaluation framework the data for the 2021 shared task includes a training set and a development set that were available to the participants. the final ranking was made on a separate test set and was available to the participants (without the target labels) for a limited time near the end of the shared task. the training, development, and test sets were the same as for the 2011 shared task, after shuffling the rows and changing the row ids. the corpus from which this data has been obtained has been described by @xcite . figure 1 illustrates a fragment of the training data. together with the data formatted as the samples of figure 1 , the participants were provided with the contents of the relevant abstracts as separate files. the evaluation framework was implemented as a codalab competition foot_2 . the facilities available at codalab made it possible to specify our own evaluation script, and also gave us flexibility to design multiple phases and include a leaderboard and discussion forum. additional information about the 2021 alta shared task was made available in the alta website foot_3 . the codalab competition was structured into two phases. in a first, development phase, all teams had access to the training and development sets and they could make an arbitrary number of submissions daily, for a maximum of 100 submissions in total. during the development phase, participant teams could submit the results of running their system accuracy 95% ci majority baseline 0.4863 0.4150-0.5583 @xcite 0.6284 0.5564-0.6951 @xcite 0.7377 0.6696-0.7961 @xcite 0.7541 0.6869-0.8108 ## participating systems as in past alta shared tasks, submissions were made by teams in two categories: a student category, and an open category. in teams of the student category, all members must be university students and none of the team members could have a phd. teams that did not qualify for the student category could participate in the open category. a total of 16 teams registered in the student category, and 5 teams registered in the open category. of these, only 5 teams, all from the student category, submitted runs in the test phase for final ranking. of the 5 teams submitting in the final phase, 3 published a system description which is available in the 2021 alta proceedings. team sarkerlab @xcite experimented with the use of svm and roberta. team heatwave @xcite applied an ensemble method with transformer variants including biomed, roberta, and electra. finally, team orangutanv3 @xcite applied a cascaded approach that used biobert and svm classifiers. whereas team heatwave's classifiers attempted to generate the final evidence grade of the collection of abstracts related to a question, the other two teams attempted to classify individual abstracts and the final result was obtained by combining the outputs of the individual classifications. ## conclusions the participating systems appeared to obtain a score slightly better than the majority baseline but the difference was not statistically significant. these results underperformed those reported by the organisers of the 2011 shared task paper and subsequent work. the participating systems attempted to use some of the latest developments on machine learning algorithms and architectures. the reason of their relatively lower performance may be due to the choice of features. possibly, better results could have been obtained by incorporating information such as the publication type, or by focusing on specific parts of the abstracts such as the methods or conclusions sections, as related work has shown to be most influential for this task.