N euro T rial NER : An Annotated Corpus for Neurological Diseases and Therapies in Clinical Trial Registries (2024)

Abstract: Extracting and aggregating information from clinical trial registries could provide invaluable insights into the drug development landscape and advance the treatment of neurologic diseases. However, achieving this at scale is hampered by the volume of available data and the lack of an annotated corpus to assist in the development of automation tools. Thus, we introduce NeuroTrialNER, a new and fully open corpus for named entity recognition (NER). It comprises 1093 clinical trial summaries sourced from ClinicalTrials.gov, annotated for neurological diseases, therapeutic interventions, and control treatments. We describe our data collection process and the corpus in detail. We demonstrate its utility for NER using large language models and achieve a close-to-human performance. By bridging the gap in data resources, we hope to foster the development of text processing tools that help researchers navigate clinical trials data more easily.

## introduction despite substantial investment, developing new treatments for human diseases is a challenging and often unsuccessful endeavour, especially for neurological conditions @xcite . for example, more than 99% of drugs tested in clinical trials for alzheimer's disease fail @xcite . at the same time it has been estimated that nearly 3.40 billion people, or roughly 40% of the global population, were affected by nervous system conditions in 2021 @xcite . in this context, the synthesis of evidence from clinical trials is critical for researchers developing therapies, offering insights into the effectiveness and safety of interventions @xcite . this process entails systematically evaluating data from clinical studies to form reliable conclusions about healthcare practices. public clinical trial reg-istries, such as clinicaltrials.gov foot_0 , are fundamental to this effort, fostering transparency and accessibility in clinical research @xcite . however, extracting information from these resources is challenging due to large data volume, incomplete and unstructured reporting, variability in terminology, and data quality concerns @xcite . computational methods, in particular natural language processing (nlp), can streamline information extraction with techniques for data structuring, standardization, as well as semantic analysis, ultimately facilitating the synthesis of clinical evidence @xcite . named entity recognition (ner) is one such technique that identifies and categorizes key elements in text, such as drug names, and enables downstream tasks such as relationship extraction and question answering @xcite . yet, there is a scarcity of publicly available annotated corpora for clinical trial registries, hindering nlp's effectiveness in processing trial data. here we bridge this gap by introducing a new gold standard annotated dataset for clinical trial registry data in the domain of neurology/psychiatry. the corpus comprises 1093 clinical trial summaries from clinicaltrials.gov, one of the largest international clinical trial registries @xcite . it has been annotated by two to three annotators for key trial characteristics, i.e., condition (e.g., alzheimer's disease), therapeutic intervention (e.g., aspirin), and control arms (e.g., placebo). we demonstrate the corpus's suitability for the ner task using models based on bert (bidirectional encoder representations from transformers) and gpt (generative pre-trained transformers). additionally, we compare the performance of these models against simple baseline methods and human experts. all resources are available on github foot_1 and the corpus is being integrated into the bigbio library of biomedical nlp datasets 3 @xcite . ## related work the aggregate analysis of clinicaltrials.gov database (aact) 4 was released in 2011 to enhance access to clinical trial registry data @xcite . this database provides disease and intervention information in two forms: (1) directly from data contributors, and (2) through medical subject headings (mesh) terms @xcite extracted using a national library of medicine (nlm) algorithm @xcite . direct contributions vary widely in terms of terminology and data quality, making the aggregation of results challenging. the nlm's rule-based algorithm applies mesh ontology to derive terms, yet this method has limitations, such as missing non-ontological entities and lacking a coherent strategy for classifying and analyzing trials across broad disease categories. furthermore, mesh term annotation often fails to capture disease context and specificity, potentially overlooking critical clinical nuances-for instance, not distinguishing between mild and severe cases of covid or between early and late stages of cancer @xcite . the main focus of existing work in ner for clinical trial data has been on pubmed abstracts. in @xcite , the authors extract pico (population, intervention, control, outcome) elements from pubmed abstracts of clinical trial publications, as well as from trial registry data from the world health organization international clinical trials registry platform (ictrp) 5 . for both pubmed and ictrp, the models were trained on the ebm-nlp dataset @xcite , an annotated corpus of pubmed abstracts describing clinical trials for cardiovascular diseases, cancer, and autism. yet, there is no evaluation provided on how this approach performed for ner from the clinical trial registry data. another widely distributed dataset is the bc5cdr corpus to support the task of recognition of chemicals/diseases and mutual interactions @xcite . it consists of 1500 articles sam-pled from the ctd-pfizer corpus, which covers a large sample of pubmed articles related to different disease classes @xcite . existing annotated corpora of clinical trial registries are primarily focused on the eligibility criteria sections to enhance the trial recruitment process @xcite @xcite . additionally, a dataset specifically for spanish has been released @xcite . to the best of our knowledge, our dataset offers several unique characteristics that distinguish it from existing resources. first, we double-annotate the titles and summary sections of prospectively registered clinical trial entries rather than published abstracts of completed trials. second, our dataset specifically targets neurological diseases, which represent a significant portion of the global disease burden, whereas existing corpora generally focus on a broader range of medical conditions. finally, our resource includes highly detailed annotations on aspects such as disease stages and severity, as well as a variety of intervention categories. these annotations enable more granular analysis, further enhancing its value for medical research. ## conclusion and outlook we have presented neurotrialner, a new, openly available corpus comprising 1093 clinical trial registry abstracts annotated for diseases, interventions, and controls. we further demonstrated that the dataset was effective in training neural ner models and analyzed their performance. specifically, biobert emerged as the top-performing model with results as good as a human rater. with this, our dataset provides a fundament to enhance our understanding of disease and intervention relationships in neurological and psychiatric diseases and improve downstream tasks, such as biomedical literature summarization, ultimately improving the development of new interventions. as future work, we plan on expanding the dataset with other disease types, including annotations for trial outcomes, and applying the ner models to other clinical trial registries or even pubmed abstracts. we are also exploring a more advanced entity normalization technique to better align the entities with a common knowledge base. finally, we aim to conduct a comprehensive analysis of clinical trial research and envision integrating our work into the services provided by the aact database.