A uto T rial: Prompting Language Models for Clinical Trial Design (2023)

Abstract: Clinical trials are critical for drug development. Constructing the appropriate eligibility criteria (i.e., the inclusion/exclusion criteria for patient recruitment) is essential for the trial’s success. Proper design of clinical trial protocols should consider similar precedent trials and their eligibility criteria to ensure sufficient patient coverage. In this paper, we present a method named AutoTrial to aid the design of clinical eligibility criteria using language models. It allows (1) controllable generation under instructions via a hybrid of discrete and neural prompting, (2) scalable knowledge incorporation via in-context learning, and (3) explicit reasoning chains to provide rationales for understanding the outputs. Experiments on over 70K clinical trials verify that AutoTrial generates high-quality criteria texts that are fluent and coherent and with high accuracy in capturing the relevant clinical concepts to the target trial. It is noteworthy that our method, with a much smaller parameter size, gains around 60% winning rate against the GPT-3.5 baselines via human evaluations.

## introduction generative large language models (llms) are drawing attention due to their ability to create coherent and human-like text documents. clinical trial design documents are written at the planning stage of the drug development process, which is crucial for the success of the trial. however, it can be challenging even for experienced professionals: around 57% trial protocols have at least one substantial amendment in eligibility criteria @xcite . the suboptimal trial design may cause insufficient recruitment, severe adverse events, or insignificant efficacy, thus inducing huge financial losses and time waste. each amendment will further cause millions of dollars in loss and months of delays. in this paper, we propose to generate the eligibility criteria for clinical trials in natural language using llms, with the solution focusing on the following aspects. in this paper, we propose to augment clinical trial design using llms, motivated by the evidence that llms can act as implicit knowledge bases @xcite . our method is equipped with instruction tuning for trial protocol design and explicit supervision for producing grounding rationales. this is enabled with the following technical features: our autotrial is the first model that utilizes llms for automating clinical trial design. it represents an important step toward using ai to facilitate clinical trial design. the rest of the paper is organized as follows: in §2, we review related work. in §3, we dive into the proposed method in detail. in §4, we present the experiment results. it is noteworthy that autotrial is proven to generate accurate criteria (with precision 0.91, recall 0.92, f1 0.91, and jaccard score of 0.84 in clinical accuracy evaluation) while almost all baselines get less than 0.5 in these metrics. moreover, our method reaches around 60% winning rate against gpt-3.5 foot_0 in trial design tasks via human evaluation. finally, in §5, we conclude and discuss future work. ## method autotrial utilizes a decoder-based architecture for generating a target criterion based on input trial synopsis and manual instructions. the training process consists of two stages: pretraining and finetuning. it is noteworthy that the model can be extended to new instructions and trial exemplars without retraining. the flowchart is depicted in fig. 1 . we will elaborate on the details of the training and inference procedures of autotrial next. ## experiment we conduct extensive experiments to evaluate autotrial in the following aspects: ## conclusion in summary, this paper presents autotrial that uses language models to aid in the design of clinical trial protocols. our method is able to generate high-quality criteria texts that are fluent, coherent, and clinically accurate, by using a combination of controllable generation, scalable knowledge incorporation, and multi-step reasoning. this can potentially reduce the risk of clinical trial failure by ensuring that trials are properly designed and have sufficient power to evaluate proposed therapies.