Overview of the 2022 ALTA Shared task: PIBOSO sentence classification, 10 years later (2022)

Abstract: The 2022 ALTA shared task has been running annually since 2010. This year, the shared task is a re-visit of the 2012 ALTA shared task. The purpose of this task is to classify sentences of medical publications using the PIBOSO taxonomy. This is a multi-label classification task which can help medical researchers and practitioners conduct Evidence Based Medicine (EBM). In this paper we present the task, the evaluation criteria, and the results of the systems participating in the shared task.

## introduction within the practice of evidence based medicine (ebm), the medical practitioner integrates individual clinical expertise with the best external evidence at point of care @xcite . finding the best available evidence, however, is increasingly difficult given the large amount of medical publications. for example, at the time of writing, pubmed contains more than 34 million citations for biomedical literature 1 . from 2020 to present, cord-19, a resource of medical publications about covid-19, sars-cov-2, and related coronaviruses, has increased from an initial set of 28,000 papers @xcite to over 1,000,000 2 . to assist with the task of finding the best available evidence, best ebm practice suggests users to formulate queries that focus on specific aspects of the clinical information sought. piboso @xcite is a pre-defined set of such aspects of clinical information, and systems participating in the 2012 alta shared task classified sentences from medical publications into piboso labels @xcite . in 2022, 10 years later, alta has re-visited the task, to find out whether recent advances in machine learning would allow to improve the quality of such classifiers. this paper presents the results of systems participating in the 2022 alta shared task. section 2 describes the piboso taxonomy. section 3 briefly mentions related work between the 2012 and the 2022 alta shared tasks. section 4 describes the evaluation framework. section 5 presents two simple baselines that were made available to the participating teams. section 6 presents the results and briefly describes the methods of participating systems. finally, section 7 concludes this paper. ## piboso ebm guidelines recommend the use of structured queries that focus on specific aspects of clinical information @xcite . one of the most widely used systems is pico, which defines 4 types of information: population, for example the number and type of participants in a study; intervention, such as the treatment applied to the population; comparison (if appropriate), for example alternative interventions or placebo; and outcome of an intervention. different variants and extensions of pico have been proposed. the alta 2012 and 2022 shared tasks use piboso @xcite . this schema removes the comparison tag and adds three new tags: background, study design, and other. the piboso tags, as defined by @xcite , are: different parts of a medical publication may focus on different piboso elements. in practice, each sentence of a pubmed abstract will normally focus on one piboso element, but sometimes a sentence may focus on several (see ## related work: from 2012 to 2022 the data used in this 2022 shared task is based on the data from the 2012 task @xcite , which is derived from the original nicta-piboso dataset by @xcite . sentence classification systems participating in alta 2012 used approaches based on conditional random field (crf), support vector machines (svm), stacked logistic regression, maximum entropy, and random forests. the results of the participating systems are summarised in the following additional research has used the nicta-piboso dataset for sentence classification. @xcite used statistical relational learning. @xcite used crf and a discriminative set of features. jin and szolovits (2020) used lstm plus adversarial training and unsupervised pre-training over large corpora. all of these systems report f1 as the evaluation metric, which is different from the metric used in the alta 2012 and alta 2022 datasets (section 4). even though the f1 and auc metrics may lead to similar rankings of systems, as observed in the alta 2012 shared task @xcite , systems fine-tuned for auc might not lead to optimal f1 scores. most notably, systems fine-tuned for auc do not need to set a classification threshold, and an evaluation using f1 will give very different results depending on the choice of classification threshold. foot_2 ## evaluation framework we have been unable to retrieve the labelled test data of the 2012 alta shared task. as a consequence, the data for the 2022 shared task is based on the training data from the 2012 shared task, after shuffling the original data and re-numbering the sample ids. the resulting data has been split into three sets for training, validation, and test. the documents used in the datasets are abstracts of medical publications published in pubmed. each abstract contains multiple sentences, and consequently a single pubmed abstract corresponds to several samples in the dataset. to minimise data leakage between the different partitions, the partitions were made based on the abstracts so that all sentences of the same abstract would be in the same partition. besides preventing data leakage, this partitioning also allows the participating systems to use the context of the other sentences from an abstract during the classification task. the evaluation framework was implemented as a codalab competition foot_3 which consisted of three phases. in the development phase, the training and validation data were available but the labels of the validation data were not available. participant teams were able to make up to 100 submissions to test their systems against the validation data. this phase was not used for the final ranking of the participating systems and ended on the 4th of october 2022. given an input sentence, the output of each participating system must produce, for every piboso label, a number between 0 and 1 that represents the confidence or probability that the label is assigned to the sentence. the evaluation metric is the micro-average of the area under the receiver operating characteristic (roc) curve. the roc curve plots the true positive rate against the false positive rate at various threshold settings for binary classification. we use the micro-average so that labels with more samples are given more importance. the advantage of using this metric instead of metrics such as f1 is that it incorporates the probability scores returned by the system, such that two systems with identical classification predictions but different probability scores will be ranked differently. ## baselines we have provided two simple baselines against which the participating systems can compare. the code for these baselines is publicly available foot_6 . we describe these baselines below. ## participating systems and results a total of 3 teams registered in the student category, and 6 teams registered in the open category. of these, only 5 teams submitted runs in the co-dalab test phase. we can observe that all participating teams outperformed the two baselines. three of the teams have submitted system descriptions and they are available in the proceedings team csecu-dsg @xcite extended deberta with 5-fold cross-training (creating effectively an ensemble approach) and multi-sample dropout. team turknlp @xcite extended scibert by adding a classification layer that incorporated information from the [cls] token and the average of scibert embeddings. ## conclusions participation in the 2022 alta shared task showed the successful use of transformer approaches for this task of multi-label classification of abstract sentences from medical publications using piboso. all participating systems outperformed the baselines. furthermore, the top system outperformed the participating systems of alta 2012 (tables 1 and 4 ). there is a potential caveat in that the test data used in the 2022 alta shared task was different from that of the 2012 alta shared task because of the non-availability of the labels of the original 2012 test data. having said that, given that the test set of the original data was created as a random partition, we would not expect a very large difference in the results.