S ent2 S pan: Span Detection for PICO Extraction in the Biomedical Text without Span Annotations (2021)

Abstract: The rapid growth in published clinical trials makes it difficult to maintain up-to-date systematic reviews, which require finding all relevant trials. This leads to policy and practice decisions based on out-of-date, incomplete, and biased subsets of available clinical evidence. Extracting and then normalising Population, Intervention, Comparator, and Outcome (PICO) information from clinical trial articles may be an effective way to automatically assign trials to systematic reviews and avoid searching and screening—the two most time-consuming systematic review processes. We propose and test a novel approach to PICO span detection. The major difference between our proposed method and previous approaches comes from detecting spans without needing annotated span data and using only crowdsourced sentence-level annotations. Experiments on two datasets show that PICO span detection results achieve much higher results for recall when compared to fully supervised methods with PICO sentence detection at least as good as human annotations. By removing the reliance on expert annotations for span detection, this work could be used in a human-machine pipeline for turning low-quality, crowdsourced, and sentence-level PICO annotations into structured information that can be used to quickly assign trials to relevant systematic reviews.

## introduction systematic reviews are a critical part of regulatory and clinical decision-making because they are designed to robustly make sense of all available evidence from primary research, especially clinical trials, accounting for study design quality and heterogeneity. searching and screening for reports of clinical trials are time-consuming tasks that require specialised expertise but are a necessary component of systematic reviews. the rapid rate at which papers were published about @xcite highlights the need for tools to improve the efficiency of systematic reviews. a range of methods have been developed to help reduce the amount of human effort required to conduct systematic reviews @xcite , but the methods developed to support the screening task are trained on data from a small number of systematic reviews and are not yet able to fully replace humans (o' @xcite . an alternative is to find ways to map all clinical trials to standardised representations of the populations, interventions, comparators, and outcomes (pico) and aggregate information across studies that answer equivalent clinical questions. pico extraction is a well-studied problem structure and was used as one of the examples in the development of scibert @xcite . many pico extraction methods focus on annotating sentences that include pico information. however, if the goal is to fully automate a process for augmenting a systematic review with new studies as their results become available, even a perfect annotation of just the sentence is not enough. an expert will still need to read the sentence and then extract and normalise the information representing the population, intervention, comparator, and primary outcomes (pico) from those sentences. full automation of the task requires the ability to identify the text spans that represent the pico information. named entity recognition (ner) seeks to identify the types and boundaries of targeted spans in unstructured text data. machine learning ner methods have been based on bilstm-crf @xcite and bert @xcite , and these require human annotated data for training. for other application domains where span annotation is designed for entities such as people, locations, and organisations, crowdsourcing of labels is somewhat easier because a broader range of people can annotate data without specialised training. even when annotated by domain experts, there is still substantial inconsistency across annotators @xcite , though sentence level annotations tend to be more consistent @xcite . there is a gap in both the volume of available training data and demonstrated performance between ner in general domains such as news and biomedical applications compared to pico extraction. there is a clear need for new approaches that can handle the more complicated and challenging token structure of biomedical entities including unusual synonyms and subordinate clauses that might include numerical information (e.g. drug dosage, or test result threshold); and can incorporate domainspecific knowledge in intelligent and useful ways. in this paper, we address these challenges by proposing a novel pico annotation task design. we propose a simplified requirement for annotation where we only need to know whether a sentence includes any pico information. we use this coarser set of annotations to learn and infer pico sentence types and span detections. the pre-trained neural language model @xcite @xcite is firstly used as feature representation learning model for sentence classification to be fine-tuned, and identify whether a given sentence contains pico spans. this makes the proposed approach also capable of inferring pico sentences even without crowdsourced annotations. to get the pico spans, we apply a masked span prediction task to assist the inference process. the fine-tuned language model is then used as the taskspecific knowledge provider for the pico spans. scored spans are then fed into an inference algorithm to produce the final detection results. the contributions of this paper are as follows: ## method in this section, we define the task and then introduce the proposed approach with blue @xcite , a bert @xcite structured neural language model in the biomedical domain, as our backbone model and the inference algorithm. ## experiments in this section, we evaluate our method and compare it with supervised, semi-supervised methods, and crowdsourced annotations on two benchmark datasets. we also investigate the pico sentence prediction results and the effect of candidate span reduction in the proposed method. ## discussion the results of the experiments show that our proposed method could be a useful component of a pipeline for pico detection and extraction, in use cases where costly expert annotation is limited and where the aim is to identify all relevant examples. the results show the approach compares favourably to existing supervised methods and achieves high recall for pico span detection even without using any span annotations as training data. our proposed approach is likely to be useful in a range of other application domains. many nlp tasks across ner @xcite @xcite and semantic role labelling @xcite can be formulated as a span detection task. in cases where it is easier to acquire or estimate low-quality sentencelevel annotations, and resource-intensive to acquire high-quality span-level annotations, our proposed approach may be appropriate. several opportunities exist for future work. for the sake of simplicity, we used the blue model @xcite with the bert-baseuncased structure, rather than exploring models with bert-large-cased structure. we also did not explore the use of other nlp tools such as part-ofspeech tagging or dependency parsing. integration of these methods in the proposed sent2span approach may improve the performance. the sent2span method is designed to be part of a larger pipeline of techniques designed to support systematic review processes. other than pico detection, extraction, and representation, other methods have been proposed for identifying which trials should be included in systematic review updates @xcite . future work in the space could include head-to-head comparisons of approaches for rapid systematic review updating that test for maximising the completeness of evidence identification and minimising human effort. ## conclusion the sent2span method for pico span detection we propose and test in this paper could be used to support new tasks in systematic review processes. the difference between sent2span and previous approaches to pico detection include the use of only low-quality sentence-level annotations as training data and the results demonstrating achieve high recall in span detection, which is an important requirement for systematic review processes.