Effective Crowd-Annotation of Participants, Interventions, and Outcomes in the Text of Clinical Trial Reports (2020)

Abstract: The search for Participants, Interventions, and Outcomes (PIO) in clinical trial reports is a critical task in Evidence Based Medicine. For an automatic PIO extraction, high-quality corpora are needed. Obtaining such a corpus from crowdworkers, however, has been shown to be ineffective since (i) workers usually lack domain-specific expertise to conduct the task with sufficient quality, and (ii) the standard approach of annotating entire abstracts of trial reports as one task-instance (i.e. HIT) leads to an uneven distribution in task effort. In this paper, we switch from entire abstract to sentence annotation, referred to as the SenBase approach. We build upon SenBase in SenSupport, where we compensate the lack of domain-specific expertise of crowdworkers by showing for each task-instance similar sentences that are already annotated by experts. Such tailored task-instance examples are retrieved via unsupervised semantic short-text similarity (SSTS) method – and we evaluate nine methods to find an effective solution for SenSupport. We compute the Cohen’s Kappa agreement between crowd-annotations and gold standard annotations and show that (i) both sentence-based approaches outperform a Baseline approach where entire abstracts are annotated; (ii) supporting annotators with tailored task-instance examples is the best performing approach with Kappa agreements of 0.78/0.75/0.69 for P, I, and O respectively.

## introduction evidence based medicine is the practice of decision making based on the best available scientific information. finding such information rapidly is essential, especially in the current pandemic crisis where thousands of medical articles about covid-19 are published weekly @xcite . to make the search process time-efficient, the pico model enables specific search for: participants (e.g. "patients with headache"), interventions ("ibuprofen"), comparisons ("placebo"), and outcomes ("pain reduction") @xcite . to allow a search for structured pico information in trial reports, a prior automatic extraction is necessary. the effectiveness of an automatic pico extraction depends on the quality of manually annotated corpora. as an alternative to scarce and expensive expert annotators, @xcite hired crowdworkers from the mechanical turk platform (mturk) to annotate participants, interventions, and outcomes (pio foot_0 ) in clinical trial reports. the crowdworkers, however, reached low agreements to expert annotations, potentially affected by (i) a lack of domain-specific expertise of the crowdworkers, and (ii) an uneven task length distribution. the lack of domain-specific expertise makes it difficult for crowdworkers to understand the terminology and jargon that prevails in medical literature @xcite . as a result, workers experience medical tasks as cognitively overwhelming, with the side effect of a decreased label quality @xcite ). an uneven task length distribution makes the effort to complete individual task-instances foot_1 unevenly distributed thus enticing workers to "cherry pick" short task-instances or rush longer ones @xcite . in the task design of @xcite , entire abstracts of clinical trial reports were annotated. these abstracts contain on average 268 words with a high standard deviation of 89, resulting in an uneven task length distribution. in this paper, we address these problems by proposing two novel pio task designs: senbase: the uneven task length is addressed by shifting from annotating abstracts to a sentencebased annotation. this makes the effort to complete individual task-instances more evenly distributed: sentences have with an average word length of 25 and a standard deviation of 13 a 85% reduced word length variety compared to abstracts. sensupport: this task design builds upon senbase by additionally compensating the lack of domain-specific expertise of crowdworkers. a common strategy to train crowdworkers for a task is to provide a few examples that illustrate how the task should be performed. in addition to such static examples, we propose to show for each taskinstance similar sentences that are already annotated by experts. such tailored task-instance examples are retrieved from a set of expert annotationsusually available to evaluate the performance of non-expert annotators @xcite -via an unsupervised semantic similarity method. during our search for an effective, unsupervised semantic short-text similarity (ssts) method for sensupport, we observed a lack of comparative evaluations for biomedical tasks. therefore, to address this gap, we perform a comparative evaluation of nine ssts methods, ranging from traditional count-based methods (e.g. tfidf) to recent text embedding methods (e.g. sen-bert @xcite ). the results on two biomedical benchmark corpora show the high effectiveness of the biosent2vec model, which we utilize to retrieve the tailored task-instance examples in sensupport. we evaluate the sentence-based approaches sen-base/sensupport and the abstract-based base-line of @xcite by comparing their collected mturk annotations to gold standard annotations. we find that the highest label quality is obtained with sensupport with cohen's kappa agreements of 0.78/0.75/0.69 for p/i/o. we show further that annotations obtained via the sentence-based approaches lead to substantially higher kappa agreements than annotations from the baseline approach, especially for the labeling of interventions and outcomes. the largest source of disagreement in the baseline approach is caused by crowdworkers overlooking entire text phrases that should be annotated -whereas, in the sentence-based approaches, crowdworkers tend to label text phrases that should not be annotated. ## task designs for pio annotation in this section, we first describe the baseline task design of @xcite , followed by our proposed task designs senbase and sensupport. ## evaluation of similarity methods in this section, we evaluate the effectiveness of nine unsupervised semantic short-text similarity (ssts) methods. each method computes a similarity score sim(t, k) ∈ r between two short-texts t, k. for methods that produce a vector representation of a text, we compute the similarity sim(t, k) between the vectors v t , v @xmath0. word count based. these methods compute the similarity between two texts based on their words in common. we evaluate tfidf-weighted word vectors and the levenshtein distance defined as the number of edits to transform the word sequence of a text into the other. @xcite aggregated word embeddings. a word embedding e @xmath1. to obtain a text embedding v t from individual word embeddings, an aggregation step is necessary. we evaluate three aggregation strategies: the average of the word embeddings (avg), a tfidf-weighted average (wavg) @xcite , and an aggregation via the smooth inverse frequency (sif) @xcite . in sif, the relative word frequency is used to obtain aggregated text representations from which the second principal component is removed. text embeddings. methods from this category infer a contextualized text embedding v t for an input text t. we evaluate: sentence bert (sen-bert) @xcite , which uses a siamese network to create bert-based text representations; the transformer-based universal sentence encoder (use) @xcite ; in-fersent @xcite , which uses word embeddings and a combination of lstms and hierarchical cnns to produce universal sentence representations; and finally, sent2vec @xcite , which computes sentence representations by combining the continuous bag of words model (cbow) and character n-gram embeddings. ## experiments on pio task designs in this section, we describe our experimental evaluation of baseline, senbase, and sensupport. ## conclusion & future work we presented two novel task designs for crowdsourcing pio annotations in clinical trial report abstracts. specifically, we propose to switch from annotating entire abstracts of clinical trial reports to the annotation of sentences (senbase), and to additionally support non-expert annotators with tailored task-instance examples (sensupport). the task-instance examples were retrieved from a set of expert annotations using the biosent2vec model, which was found to be effective by an empirical evaluation of nine unsupervised ssts methods based on two biomedical corpora. we evaluated the sentence-based annotation approaches senbase and sensupport, and the abstract-based approach baseline by comparing crowd-annotations of each approach to a set of gold standard annotations. we found that the highest kappa agreement to the gold standard is reached by annotations of the sensupport approach. therefore, whenever expert annotations can be spared, they should be utilized as tailored task-instance examples. furthermore, we showed that annotations from the sentence-based approaches sen-base/sensupport substantially outperform annotations from the baseline approach, especially for interventions and outcomes. finally, we conducted a pairwise comparison of the token overlap of annotations of either approach with the gold standard and find that crowdworkers using the sentence-based approaches are prone to annotate text phrases that should not be annotated, whereas workers using the abstract-based approach are prone to overlook text phrases that should be annotated. the core limitation of the sensupport approach is the availability of reference samples from which the tailored task-instance examples are retrieved. obtaining reference samples is usually expensive since expert annotators need to be employed. therefore, in future work, we aim to identify the minimum number of reference samples that is needed to still preserve a high annotation quality of crowdworkers. more specifically, we aim to combine the selection of reference samples with an active learning approach. by applying active learning, the informativeness of samples can be computed and used as a deciding factor in the selection of a few reference samples that are effective as task-instance examples. another promising future research direction is the application of task-simplification approaches for different tasks and domains. in this study, we showed that a simple shift from annotating sentences rather than entire abstracts can significantly increase the annotation quality obtained from crowdworkers. similar improvements might be possible in other challenging tasks/domains, such as the legal or patent domain. the code of our experiments on the various unsupervised ssts methods, including the implementation of each method and the computed scores, is available at https:// github.com/markus-zlabinger/ssts . the collected annotations of senbase and sensup-port are available at https://github.com/ markus-zlabinger/pico-annotation .