CR e SE : Benchmark Data and Automatic Evaluation Framework for Recommending Eligibility Criteria from Clinical Trial Information (2024)

Abstract: Eligibility criteria (EC) refer to a set of conditions an individual must meet to participate in a clinical trial, defining the study population and minimizing potential risks to patients. Previous research in clinical trial design has been primarily focused on searching for similar trials and generating EC within manual instructions, employing similarity-based performance metrics, which may not fully reflect human judgment. In this study, we propose a novel task of recommending EC based on clinical trial information, including trial titles, and introduce an automatic evaluation framework to assess the clinical validity of the EC recommendation model. Our new approach, known as CReSE (Contrastive learning and Rephrasing-based and Clinical Relevance-preserving Sentence Embedding), represents EC through contrastive learning and rephrasing via large language models (LLMs). The CReSE model outperforms existing language models pre-trained on the biomedical domain in EC clustering. Additionally, we have curated a benchmark dataset comprising 3.2M high-quality EC-title pairs extracted from 270K clinical trials available on ClinicalTrials.gov. The EC recommendation models achieve commendable performance metrics, with 49.0% precision@1 and 44.2% MAP@5 on our evaluation framework. We expect that our evaluation framework built on the CReSE model will contribute significantly to the development and assessment of the EC recommendation models in terms of clinical validity.

## introduction eligibility criteria (ec) consist of statements that outline the characteristics participants must pos-sess to be included in a randomized controlled trial (rct) @xcite . ec are typically divided into inclusion and exclusion criteria, covering diverse clinical factors such as age, sex, medical history, disease severity, previous treatments, and other physiologic parameters @xcite . ec are a key design factor of rcts, along with randomization and blinding, which contribute to the production of causal evidence between intervention and outcome @xcite . moreover, ec are an important component of the enrichment strategy and minimize potential risk to study participants @xcite . however, there are concerns that ec are overly restrictive @xcite . while restrictive ec ensure homogeneity in the study population @xcite , they may also limit the generalizability of clinical findings and impede the translation of research results into clinical practice. furthermore, the ec used by previous rcts are often employed as templates for new trials without appropriate modifications @xcite . this practice can perpetuate issues such as the under-representation of specific patient subgroups (e.g., children, the elderly, and individuals with infections like hiv infection) @xcite . to overcome these problems, previous studies attempted to automate ec generation or search for similar trials to aid in clinical trial design @xcite . however, these studies relied on similarity-based performance metrics, which do not account for human judgment and clinical semantic similarity @xcite . furthermore, certain ec, such as age requirements, are widely employed across studies and are less specific to the purposes and designs of clinical trials @xcite . the presence of these common ec may have led to an overestimation of the model's performance. in response, this study aims to recommend ec from clinical trial information, such as titles and summaries, to meet the needs of drug development and clinical evidence generation (figure 1b ). in addition, we propose an automatic evaluation framework to assess the clinical validity of ec recommendation models. to accomplish this, we develop sentence embedding, called crese (contrastive learning and rephrasingbased and clinical relevance-preserving sentence embedding) (figures 1a and 2 ). by employing crese, which capture clinical semantic similarities among ec, we assessed the outcomes of the ec recommendation model and leveraged them to enhance the quality of training data. lastly, we investigate the characteristics that ec recommen-dation models should possess to be useful in clinical trial design for drug development, as discerned through human evaluation. to the best of our knowledge, this study is the first attempt to formulate the ec recommendation task. additionally, in this study, we explored the diverse utility of llms in handling biomedical texts in a clinically plausible manner, including rephrasing ec to develop sentence embedding, and streamlining the ec recommendation model into an end-to-end recommendation system. the main contributions of this paper are as follows: 1 • we introduce a task and benchmark dataset of recommending ec from clinical trial information without any manual instruction. ## related works natural language processing research on ec has taken two main paths. the first approach focuses on converting free-text ec into structured criteria or queries using information extraction or contextfree grammars @xcite @xcite . these studies, known as 'patient-trial matching', ultimately aim to estimate the number of patients who match a proposed trial design based on in-hospital electronic medical records (emrs) before patient enrollment @xcite . however, a challenge in this approach is the lack of consensus on a universal query grammar for ec @xcite @xcite . the second research stream involves studies that generate ec with manual instruction or search for similar trials to aid in clinical trial design @xcite @xcite . the autotrial study, for instance, proposed a hybrid approach that combines discrete and neural prompting in generating ec @xcite . furthermore, the pytrial study aimed to create a unified python package that incorporates diverse ai algorithms for tasks related to clinical trials @xcite . however, to this date, no study has endeavored to recommend ec exclusively from clinical trial information without manual instructions. moreover, previous studies have relied on traditional summarization metrics, such as bleu or rouge, and ec parsers in evaluating their models (fair, 2022). however, these metrics are still insufficient for measuring clinical semantic similarity between ec, and clinical trial parsers have limited performances on complex ec @xcite . ## conclusion in this study, we introduce the task of recommending ec from clinical trial information and develop the crese model, designed to preserve clinical relevance between ec, by employing contrastive learning and using rephrasing via llms as text augmentation. we also demonstrate the importance of varied rephrasing prompts for developing the crese model through the ablation study. additionally, we establish the automatic evaluation framework which assesses the clinical validity of the ec recommendation model based on the crese model. in addition, we define common ec and exclude them from the dataset to prevent an overestimation of the ec recommendation model's performances input type binary classification ec recommendation accuracy precision recall f1 p@1 map@5 p@ecno title only 81.6 80.3 83.8 82.0 37.0 29.5 23.7 title + summary 93.1 92.6 93.7 93.1 47.0 41.2 30.0 title + design factors 92.2 91.8 92.7 92.2 46.0 40.4 31.5 title + summary + design factors 93.1 92.6 93.7 93.1 49.0 44.2 29.6 chatgpt 42.3 78.6 13.9 23.7 na na na gpt-4 75.6 92.9 31.0 46.4 na na na random recommendation na na na na 11.3 [6.0, 19.0] 11.5 [8.3, 15.0] 11.6 [10.1, 13.6] while the primary motivation of this study is to provide an appropriate ec template from limited trial information such as trial titles, we also envision the ec recommendation model as a clinical inference tool for exploring new therapeutic strategies and safety concerns by recommending ec. although this work does not conclusively determine the potential of lms as clinical inference tools, we expect that our automatic evaluation framework based on the crese model could enhance the development and evaluation of ec recommendation models in terms of clinical validity.