DISTANT - CTO : A Zero Cost, Distantly Supervised Approach to Improve Low-Resource Entity Extraction Using Clinical Trials Literature (2022)

Abstract: PICO recognition is an information extraction task for identifying participant, intervention, comparator, and outcome information from clinical literature. Manually identifying PICO information is the most time-consuming step for conducting systematic reviews (SR), which is already labor-intensive. A lack of diversified and large, annotated corpora restricts innovation and adoption of automated PICO recognition systems. The largest-available PICO entity/span corpus is manually annotated which is too expensive for a majority of the scientific community. To break through the bottleneck, we propose DISTANT-CTO, a novel distantly supervised PICO entity extraction approach using the clinical trials literature, to generate a massive weakly-labeled dataset with more than a million ‘Intervention’ and ‘Comparator’ entity annotations. We train distant NER (named-entity recognition) models using this weakly-labeled dataset and demonstrate that it outperforms even the sophisticated models trained on the manually annotated dataset with a 2% F1 improvement over the Intervention entity of the PICO benchmark and more than 5% improvement when combined with the manually annotated dataset. We investigate the generalizability of our approach and gain an impressive F1 score on another domain-specific PICO benchmark. The approach is not only zero-cost but is also scalable for a constant stream of PICO entity annotations.

## introduction primary care physicians rely on systematic reviews (srs) for informed decision-making. srs are conducted to objectively answer clinical questions and require going through a rigorous process of manually screening tens of thousands of clinical studies to identify terms describing pico. pico information identification is crucial to appraise the relevance of a clinical study for answering the clinical question at hand. a study is only included for writing srs if it mentions relevant pico information. manual pico information screening for a single sr consumes more than 12 months of two medical experts' time. the process can be automated using information extraction (ie) by directly pointing the human reviewers to the correct pico descriptions. automation will accelerate the overall process of writing srs while reducing the burden on health professionals who are required to manually screen for pico entities. automating pico entity detection has garnered lower interest than other biomedical ner tasks because of the lack of publicly available entity annotated corpora. the largest publicly-available pico entity/span dataset (ebm-pico) contains only 5000 annotated abstracts, some of which were annotated through crowd-sourcing and others by hired medical experts @xcite . crowdsourcing involves hiring non-expert workers that require intensive training that is not commonly affordable. hiring medical experts for annotation is equally often too expensive. in general, extracting pico entities/spans is somewhat tricky because of high disagreement between human annotators on the exact spans constituting the mentions. this leads to human errors in hand-labeled corpora. hand-labeled datasets are static and prohibit quick manual re-labeling in case of human errors or when a downstream task requires new entities. for example, pico entities extend to picos, where s denotes the "study type" of included evidence. distant supervision (ds) is a data-centric approach that allows generating massive weakly annotated datasets without human annotators and has previously been used to create large relation extraction corpora for the general and biomedical domains. to address the challenges above and democratize pico entity recognition, we propose distant-cto, a distantly supervised and scalable approach to obtaining clinical trials annotations. we take an integrative approach combining methods of semi-supervised learning (ssl) and gestalt pattern matching (gpm) to develop a continuously extensible dataset. we successfully demonstrate this approach for the "intervention" and "comparator" entity annotations as proof of concept (poc). we summarize our contributions as follows: ## related work a decade of automatic pico information extraction was limited to sentence-level due to the unavailability of entity-annotated corpora @xcite @xcite @xcite . the release of the ebm-pico corpus paved the way for the community to improve upon the pico entity/span extraction task. @xcite . the corpus is biased towards pharma intervention classes overshadowing non-pharma ones leading to a substandard performance on it in the previous sota fully-supervised pico entity/span recognition models @xcite @xcite ) and weakly supervised model @xcite . small-scale annotation projects cannot capture the range and variation of the pico descriptions spanning the entirety of clinical trials literature. at some point, applications of such static corpora will confront the problem of insufficient and irrelevant annotations. manual annotation projects are neither affordable nor scalable for every lab, limiting innovation. a plethora of ds methods have been previously explored for large-scale relation extraction but not for (named) entity extraction @xcite @xcite . entity extraction in high-impact clinical and biomedical domains largely relies on small expert annotated datasets. commonly, obtaining weak annotations using ds rely on aligning terms (a word or phrase) from ontologies onto the unstructured text @xcite @xcite . ontologies are structured, standardized data sources that do not capture various writing variations from clinical literature. weak annotations obtained using custom-built rules like regular expressions are restricted by either task or worse even by entity type @xcite @xcite . bootstrapping approaches like label propagation (lp) still require an expert annotated dataset to obtain pseudo annotations for previously unlabeled data samples @xcite . it is hence not zero-cost. our work focuses on overcoming the discussed bottlenecks using a data-centric ds approach to generate a large clinical entity annotated corpus and train a downstream ner model to assess if it yields adequate results. unlike the reviewed ds approaches, our approach does not use ontologies or rules or lp but rather uses gpm for flexibly aligning structured text in a clinical trials database to the free-text fields in the same database using an adaptable internal scoring scheme. ## data clinicaltrials.gov (cto hereafter) documents more than 350,000 human clinical studies conducted around the globe. the trial's principal investigator enters and updates information about each study stored in cto. it includes the title and description of the clinical trial, participant's eligibility criteria, participant disease and demographics, interventions evaluated, outcomes, etc. cto allows programmatic access to this vast amount of information in the json (javascript object notation) format. the information is stored as a combination of structured tabular and unstructured free-text (see figure 1 ). the 'officialtitle' and 'brieftitle' tags in the json respectively store the official and shorter version of the study title in an unstructured free-text format. the 'briefsummary' and 'detaileddescription' tags store study summaries. interventions used in the study are stored under the 'interventionname' tag and their synonyms under 'interventionothername' tag each of which could be linked to their broad semantic type (drug, device, behavioral, procedural, biological, dietary supple-ment, diagnostic test, radiation, genetic, combination product, other) mentioned under the 'interven-tiontype' tag. as each intervention name is linked to its semantic type, this becomes a structured information store. the 'interventiondescription' tag describes intervention administration procedures often in a detailed passage. ## approach the approach is schematically illustrated in figure 2 and is described below. ## experiments the experiments were designed to evaluate the performance of the distant ner models trained with the distant-cto set alone vs. distant-cto set in combination with the ebm-pico training set. the ebm-pico training set is naturally composed of both positive and negative annotation sentences, but for the distant-cto, we artificially generated the negative sentences a -. to evaluate the impact of these negative annotation sentences, we perform ablation experiments, training the models only with positive annotation sentences a + . finally, we also evaluate the performance when training using the entity annotations with match score d @xmath0.0 alone vs. entity annotations with d @xmath1.9. a simple scibert-crf model trained using positive annotation sentences a + was used as the baseline. transformer-based models incorporate sequence order and self-attention components, so our baseline served to check the impact of removing costly bilstm and self-attention modules. ## results this section reports empirical results for the candidate generation process, evaluation for the annotation quality of distant-cto approach using the validation sets (see ## conclusions and future work we exploit the freely-available clinicaltrials.org (cto) and distant supervision for developing the largest available weakly annotated database of intervention-comparator entities across 11 subtypes. using these weak annotations combined with the manual annotations, we train an "intervention" ner model that surpasses current approaches by more than 5% in terms of f1 on the ebm-pico gold benchmark and demonstrate strong generalizability on a domain-specific physiotherapy benchmark. when the same ner model was trained with the weakly annotated dataset alone, it surpassed other approaches by 2%. this is a prototypical work, and an automatically obtained dataset with i and c annotations are being extended for the participant (p), outcome (o), and study type (s) entities. the code and data are available on github.