Intra-Template Entity Compatibility based Slot-Filling for Clinical Trial Information Extraction (2022)

Abstract: We present a deep learning based information extraction system that can extract the design and results of a published abstract describing a Randomized Controlled Trial (RCT). In contrast to other approaches, our system does not regard the PICO elements as flat objects or labels but as structured objects. We thus model the task as the one of filling a set of templates and slots; our two-step approach recognizes relevant slot candidates as a first step and assigns them to a corresponding template as second step, relying on a learned pairwise scoring function that models the compatibility of the different slot values. We evaluate the approach on a dataset of 211 manually annotated abstracts for type 2 Diabetes and Glaucoma, showing the positive impact of modelling intra-template entity compatibility. As main benefit, our approach yields a structured object for every RCT abstract that supports the aggregation and summarization of clinical trial results across published studies and can facilitate the task of creating a systematic review or meta-analysis.

## introduction the evidence based medicine (ebm) paradigm @xcite propagates that individual medical decisions are taken on the basis of the best available clinical evidence. the activity of summarizing the existing body of evidence is a core activity to support ebm and its most prominent instrument is the systematic review. creating a systematic review involves a high effort, involving on average 67.3 weeks and involving 5 authors per review on average @xcite . keeping systematic reviews up to date involves an even much higher and continuous effort @xcite . thus, there is increased interest in partially automatizing the creation of systematic reviews @xcite . a significant hindrance for the automation of systematic reviews is that data needs to be extracted by hand from published studies. this problem could be alleviated if publications were machine readable, or could be turned into a structured, machine readable form by information extraction methods @xcite . the methods that so far have been applied to the automatic extraction of information from clinical trial publications follow the pico framework and attempt to extract the population, intervention, comparator and outcomes from a publication. most approaches formalize the task as a tagging or classification problem. some approaches for instance attempt to tag spans in the text and label them with the pico elements (e.g. @xcite ). others classify complete text segments into these classes @xcite . however, the pico elements denote structured objects rather than plain tags or classes. an intervention is described by a drug, frequency of administration, administration route, dose, etc. an outcome is described by a certain increase or decrease of a value from a baseline condition, refers to a certain primary or secondary endpoint, and there are outcomes for each arm of a trial that need to be compared to each other. in spite of being structured objects, most previous work treats these elements as flat and unstructured. treating them as such makes the automatic aggregation and summarization of results challenging if not impossible. towards treating information extraction from clinical publications as a problem of predicting structured elements, we model the task as a template extraction task in which each template consists of a number of slots to be extracted. in towards extracting these templates and thus a structured representation of a clinical trial and its results, we present a novel deep learning architecture. the architecture first labels spans of text as candidate slot fillers of a particular slot in a first step. in a second step, the filler is assigned to an instance of a template. with this two-step architecture, we can transform each clinical trial abstract into a structured representation that supports downstream aggregation of results. as there can be multiple interventions, arms and outcomes in a given study, an important challenge is to predict how many instances of each template occur in a given clinical trial publication. we leave this subpart of the problem for future work and assume that the number of interventions, arms and outcomes is known a priori. this assumption is reasonable as this information is typically contained in existing registries for trials such as https://www.clinicaltrials.gov/ . when assigning slot fillers to templates, it is important to model the dependencies between the different slots as some values might be compatible while others not. we model this compatibility by a trained function that predicts a compatibility score. in summary our contributions are as follows: ## related work the template extraction and slot filling task we address is related to the field of event extraction @xcite where the goal is to extract so called event triggers and the arguments of the events. our templates can be seen as complex events and our slots as arguments thereof. wang et al. (2020) adopt the question answering paradigm to extract events from biomedical texts. they introduce two different types of questions for extracting event triggers and event arguments. however, in their approach the extraction of event arguments also relies on the extraction of event triggers. @xcite introduce a framework for taskindependent template-based information extraction. their approach first identifies text spans representing slot-fillers as in our approach. however, their system relies on the successful identification of anchor spans representing template instances as they cast the assignment of slot-fillers to template instances as a binary classification between anchor spans and other text spans. the slot filling system proposed by @xcite is a neural architecture that can exploit the combination of semantic similarity-based attention and position-based attention. the authors address a relation extraction task and develop a large corpus of annotated relations, tacred @xcite . more recent work has framed the task of relation extraction in the biomedical field as a slot filling task as well @xcite . however, the work is limited to extracting binary relationships @xcite . early work on extracting information from text describing clinical trials has focused on the classification of sentences into sections of papers describing randomized controlled trials (rcts), e.g. methods, @xcite @xcite . such systems tackle a very coarse-grained information extraction task as they do not extract the actual content or results of a published rct, but only extract correspondences between content and the standard sections used to describe a clinical trial in a publication. such a sentence classification task can support the indexation and thus retrieval of information from a published rct, but does not support the use case we consider, i.e. the aggregation of evidence across published trials. beyond the classification of sentences into sections of an article, other authors have considered the classification of sentences into pico elements, that is classifying a sentence in a published clinical trial with respect to whether it describes the population, intervention, comparator or an outcome @xcite @xcite . such approaches are able to extract information at a more detailed granularity, but they still do not support aggregation of evidence across studies as the mere classification of sentences with respect to pico elements does not provide a semantic structure that can be used to describe the key results of a study. the work by @xcite goes one step further in that it tags spans of text in an rct abstract into the pico classes, considering the following classes: patient group, intervention, arm, control arm, measured outcome, etc. @xcite rely on maximum entropy models and use integer linear programming to define constraints on the classified tokens, e.g., such that results can not occur in the methods section. they show that their approach is able to extract evidence tables from rct abstracts. yet, the different spans extracted are only indirectly related to each other in the model of @xcite . this gap is addressed by the approach of nye et al. ( 2020 ), which beyond extracting pico elements (intervention arms, outcome measures, results) also relates the different snippets to each other, yielding a relational structure. inspired by the work of @xcite as well as nye et al. (2020) we go one step further in extracting a complete structured object from an rct abstract comprising of nine main template types with overall 85 slots. to our knowledge, this is thus the most fine-grained representation that so far has been considered by an information extraction system in the clinical domain. ## model as already mentioned in the introduction, our proposed model consists of a two-step architecture. the first component, the entity extraction (ee) module, identifies spans of slot filler candidates (sfcs). we assume that we have a set of template types @xmath0, ..., t |l| which correspond to the template types depicted in the set of all sfcs extracted within an abstract is denoted by e. formally speaking, the entity extraction module implements a function f ee that maps each slot filler candidate into a slot type, i.e. f ee : @xmath3. the second component, the template assignment (ta) module, maps each slot filler to a particular instance of a template. hereby, we can have multiple instances of a given template type. for instance, in the general case a clinical study might describe multiple interventions, multiple endpoints and multiple outcomes. we denote the i-th instance of template t by t we denote the template type to which sfc e j has been assigned to as y e j . take the following sentence as an example: mean 24-h iop with btfc was significantly lower than with latanoprost (18.9 vs 21.2 mmhg; @xmath4.001). the first component would recognize the spans 18.9 and 21.2 and map them both to the slot type resultmeasuredvalue. then the ta module assignes these identifies sfcs to template instances of type outcome, together with other scfs extracted from other sentences. note that both modules fully specify a mapping from entities detected in the clinical trial abstract to fully instantiated templates, where f ee identifies and classifies text spans into slots and f ea identifies the appropriate instance of a template. we describe both modules in more detail subsequently. in particular, as the assignment of text spans to slots and template instances should not be modelled completely independently, we introduce an additional component that computes an overall score for a given template instance that quantifies the compatibility of the assigned text spans to all of the slots of the template instance. these scores can be regarded as factors as used in factor graph models @xcite . in order to reduce the complexity, we model the interaction between different slots in a pairwise fashion, limiting the scope of these factors to two slots. ## experiments we conduct experiments on two public datasets @xcite which contain rct abstracts from the glaucoma and type 2 diabetes mellitus (t2dm) domain, respectively. the corpora of both datasets are annotated at two levels: at the first level, salient entities which describe components of the pico elements are annotated. the second level comprises template-based annotations of complex pico elements and their interactions. ## conclusion we have presented a twp-step neural architecture based on a transformer model that can induce a structured representation from an abstract describing a randomized controlled trial (rct). the architecture performs extraction of candidate slot fillers as a first step by identifying spans of 37 we have shown that our approach can extract candidate slot fillers reliably, yielding micro f-measures of 76.21% and 76.49% on our glaucoma and t2dm dataset, respectively. in terms of extraction of templates, our approach yields micro f-measures of 62.27% and 64.38% averaged over all slots on our glaucoma and t2dm dataset, respectively. the structure of our templates is inspired by the c-tro ontology @xcite and induces the most fine-grained and accurate representation of a published rct that has been considered so far by any information extraction system. in future work we intend to show that our information extraction approach indeed supports the aggregation of results across clinical trials. further, we plan to use the intra-template compatibility scores to infer the number of template instance for template types which could have several instances. this can be regarded as an additional layer on top of our proposed optimization algorithm. in addition, we plan to predict links between template instances. yuhao zhang, victor zhong, danqi chen, gabor angeli, and christopher d. manning. 2017. position-aware attention and supervised data improve slot filling. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 35-45, copenhagen, denmark. association for computational linguistics.