ABSTRACT Objective To present a series of experiments: (1) to evaluate the impact of pre-annotation on the speed of manual annotation of clinical trial announcements; and (2) to test for potential bias, if pre-annotation is utilized. Methods To build the gold standard, 1400 clinical trial announcements from the clinicaltrials.gov website were randomly selected and double annotated for diagnoses, signs, symptoms, Unified Medical Language System (UMLS) Concept Unique Identifiers, and SNOMED CT codes. We used two dictionary-based methods to preannotate the text. We evaluated the annotation time and potential bias through F-measures and ANOVA tests and implemented Bonferroni correction. Results Time savings ranged from 13.85% to 21.5% per entity. Inter-annotator agreement (IAA) ranged from 93.4% to 95.5%. There was no statistically significant difference for IAA and annotator performance in preannotations. Conclusions On every experiment pair, the annotator with the pre-annotated text needed less time to annotate than the annotator with non-labeled text. The time savings were statistically significant. Moreover, the pre-annotation did not reduce the IAA or annotator performance. Dictionary-based pre-annotation is a feasible and practical method to reduce the cost of annotation of clinical named entity recognition in the eligibility sections of clinical trial announcements without introducing bias in the annotation process. Natural language processing (NLP) projects require manually annotated gold standard corpora to train and test supervised, machine learning-based algorithms or, in the case of rule-based methods, to test the performance of the rules. In light of the high cost of expert manual annotations, NLP researchers need robust methods to speed up the annotation process, without biasing the generated gold standard. In our institution, we are working on an NIH-funded project to automate clinical trial eligibility screening by using NLP algorithms. This effort requires the development of a substantial manually annotated gold standard. As such, this annotation is very time-consuming and costly. In this study, our aim is to present a series of experiments: (1) to evaluate the impact of preannotation on the speed of manual annotation of clinical trial announcements (CTA); and (2) to test for potential bias, if pre-annotation is utilized. We define potential bias as either increasing the discrepancy between annotators measured by interannotator agreement (IAA) or decreasing the agreement (called annotator performance in our study) between the annotations of the annotator with preannotated text and the eventual gold standard. The annotation task included labeling medical named entities in two classes: disease/disorder and sign/ symptom. Unified Medical Language System (UMLS) Concept Unique Identifiers (CUI) and SNOMED-CT codes were also annotated for each entity. The rest of the paper is structured as follows. In the 'Background and significance' section, we present relevant literature. In 'Data and methods', we describe the data, experimental methods, and analytical approaches. In the 'Results' section, we present the results. In the 'Discussion' section, we discuss the findings, limitations, and future research questions. In the final section, we provide our conclusions. BACKGROUND AND SIGNIFICANCE
REF suggest medical entity annotations in clinical trial announcements.
Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements
Journal of the American Medical Informatics Association : JAMIA
Journal of the American Medical Informatics Association : JAMIA
Computer Science Medicine