PICO Corpus: A Publicly Available Corpus to Support Automatic Data Extraction from Biomedical Literature (2022)

Abstract: We present a publicly available corpus with detailed annotations describing the core elements of clinical trials: Participants, Intervention, Control, and Outcomes. The corpus consists of 1011 abstracts of breast cancer randomized controlled trials extracted from the PubMed database. The corpus improves previous corpora by providing detailed annotations for outcomes to identify numeric texts that report the number of participants that experience specific outcomes. The corpus will be helpful for the development of systems for automatic extraction of data from randomized controlled trial literature to support evidence-based medicine. Additionally, we demonstrate the feasibility of the corpus by using two strong baselines for named entity recognition task. Most of the entities achieve F1 scores greater than 0.80 demonstrating the quality of the dataset.

## introduction evidence-based medicine (ebm) is an approach where doctors and health care professionals use the best available research evidence to guide them in making clinical decision about the care of patients @xcite . meta-analyses are one of the essential tools in ebm because they provide the highest form of medical evidence @xcite . a meta-analysis is a statistical technique that combines results of different research studies to determine the effectiveness of a treatment. despite their importance, meta-analyses are laborintensive and time-consuming as they involve manually reading hundreds of unstructured research articles and extracting data from them @xcite . the number of research articles is increasing rapidly making it difficult/impossible for researchers to keep up. for instance, a recent study showed that more than 50,000 research articles related to covid-19 have been published and more articles are being published every day @xcite . machine learning and natural language processing (nlp) techniques to automate data extraction from biomedical literature and speed up dissemination of biomedical evidence have been widely studied. although automatic (or semi-automatic) approaches for extracting data from research articles have been proposed, they are still not ready for practical use @xcite . this is because data extraction requires high accuracy, which may be difficult for automated systems to achieve. the scarcity of publicly available corpora, which are usually expensive to create, is one barrier to the development of high-performance systems. this paper presents a publicly available foot_0 corpus annotated with the core components of clinical trials, i.e., participants, intervention, control, and outcomes (pico). we annotate in detail numeric texts especially those that identify the number of participants having certain outcomes. the annotation of the numeric texts is important for statistical analysis to determine the overall effect of an intervention. currently, the corpus consists of 1011 research abstracts extracted from the pubmed database. the abstracts are of randomized controlled trials (rcts) related to breast cancer, which is one of the leading causes of deaths in the world foot_1 . we focus on rcts as they are considered the gold standard for clinical research methods. ## related work although there are some corpora with pico elements annotated in abstracts and full-text articles, most of the corpora are not publicly available. @xcite developed a dataset containing 182 full-text articles. they annotated 21 entities including treatment dosage, frequency, funding organization, grant number, and so on. summerscales et al. (2011) created a corpus consisting of 263 abstracts and annotated the treatment groups, out-comes, group sizes, and outcome numbers. their work is close to our study as they attempted to identify outcome numbers and group sizes for the purpose of calculating summary statistics. the annotations are however less extensive and the corpus is not publicly available. since constructing large corpora is expensive, wallace et al. (2016) employed a distant supervision approach to create a large corpora consisting of full-text articles. they also manually annotated 133 articles for evaluation. although distant supervision is a cheap way to construct large datasets, the dataset's quality might be low. most of these previous datasets are not publicly available. @xcite developed the ebm-nlp corpus, which is one of the largest publicly available corpora. their annotation was done by crowd-sourcing through amazon mechanical turk and a small part (200 abstracts) was done by medical professionals. the corpus consists of about 5000 abstracts of rcts mostly related to cardiovascular diseases, cancer, and autism. they however do not annotate numeric texts that identify the number of participants who had certain outcomes. ## baseline experiments we evaluate the corpus using named entity recognition (ner) task. this task is important for automatic information extraction from rct research articles. since deep learning language models have gained a lot of attention in nlp tasks, we adopt bidirectional encoder representations from transformers (bert)-based models. bert-based models have achieved state-of-the-art results in nlp tasks including ner @xcite . these models are usually pre-trained on huge amounts of unlabeled data and can be fine-tuned to specific tasks. they use the encoder structure of the transformer which is an attention mechanism that learns contextual relations between words (or subwords). we chose two pre-trained transformer-based baseline models, biobert @xcite and longformer @xcite . biobert is initialized with general domain corpora and further trained on biomedical domain texts (pubmed abstracts and pubmed central articles). longformer is pre-trained on general domain corpora including books, wikipedia, news, stories. the 1011 abstracts were randomly split into 80% training data and 20% test data. as baseline experiments, we followed the standard bert practice of formulating ner task as a sequential tagging task. since neural networks provide different results when initialized with different seeds, we trained the models with five different seeds and averaged the results. the performance of the models was evaluated using f1 score. we performed an error analysis and identified misclassified entities and boundary detection as the common types of errors. in the case of misclassified entities errors, the models identified the correct boundaries but assigned the wrong entities. for example, iv-bin-abs was misclassified as cvbin-abs and vice-versa. boundary detection errors were common in the outcome and eligibility enti-ties, where the models identified longer or shorter entities than those marked in the gold set. ## conclusion we presented a publicly available corpus with detailed annotation of the pico elements. the corpus contains 1011 abstracts related to breast cancer rcts. the corpus provides detailed annotation for outcomes especially numeric texts to identify the number of participants having certain outcomes. this is important for statistical analysis to determine the effectiveness of a treatment. the corpus will facilitate nlp research on automatic information extraction from biomedical literature and contribute towards evidence-based medicine. since the corpus consists of breast cancer related abstracts, one of the future works is to extend it to include other diseases. the corpus is publicly available at https://github.com/sociocom/ pico-corpus .