MDC at S em E val-2023 Task 7: Fine-tuning Transformers for Textual Entailment Prediction and Evidence Retrieval in Clinical Trials (2023)

Abstract: We present our entry to the Multi-evidence Natural Language Inference for Clinical Trial Datatask at SemEval 2023. We submitted entries forboth the evidence retrieval and textual entailment sub-tasks. For the evidence retrieval task,we fine-tuned the PubMedBERT transformermodel to extract relevant evidence from clinicaltrial data given a hypothesis concerning either asingle clinical trial or pair of clinical trials. Ourbest performing model achieved an F1 scoreof 0.804. For the textual entailment task, inwhich systems had to predict whether a hypothesis about either a single clinical trial or pair ofclinical trials is true or false, we fine-tuned theBioLinkBERT transformer model. We passedour evidence retrieval model’s output into ourtextual entailment model and submitted its output for the evaluation. Our best performingmodel achieved an F1 score of 0.695.

## introduction 1.background a large number of clinical trial reports are freely available online; for example, the national library of medicine's clinical trials repository foot_0 lists over 440,000 clinical trial studies. these represent a valuable resource for researchers and medical practitioners but leveraging this data can be challenging due to the large volume of historical clinical trials and the rate at which new trials are conducted @xcite . efforts to efficiently extract key information from clinical trial reports have been numerous and varied. several studies aimed to streamline the systematic review process with a view to transitioning experimental medicines into the clinic more quickly. @xcite built a system to automatically identify clinical trial interventions and comparators along with their associated endpoints. @xcite created a system combining a statistical text classifier for relevant sentence identification with a rule-based information extractor to automatically extract 21 key pieces of information from clinical trial records. the system included a web interface to allow users to review a modify the extracted information. @xcite addressed the challenge of efficiently extracting and reporting clinical trial research outputs to medical practitioners. their system extracts key information from research abstracts and calculates summary statistics relevant to evidence based medicine. other groups have built resources to facilitate the development of tools to streamline the analysis of clinical trial reports. @xcite annotated character spans corresponding to the population, intervention, comparator, and outcomes (pico) in a set of 5000 clinical trial abstracts and mapped the interventions to a medical vocabulary. @xcite presented a data set of full-text clinical trial articles paired with over 10,000 hypotheses about an intervention's effect with respective to a given outcome and comparator alongside several baseline systems for predicting whether an intervention significantly reduced, significantly increased, or had no effect with respect to an outcome. @xcite extended the data set by 25%, introduced an abstract only version, and provided stronger baseline models. ## method our approach focused on fine-tuning pre-trained bert-based transformer models for both tasks @xcite . bert based models can process sequences of up to 512 tokens. the text associated with each trial typically corresponds to a significantly larger number of tokens. fortunately, in both tasks we only needed to consider one section of the clinical trial record which greatly reduced the amount of text the system must process. even so, as figure 1 shows, if we concatenate the hypothesis with the relevant clinical trial record section for each example, the token count exceeds the bert token limit for ~21% of the examples; this issue is particularly pronounced in the eligibility criteria section where nearly half of the examples exceed the token limit. if we consider only the relevant parts of each relevant trial record section, concatenate them with the hypothesis and tokenize the combined text, ~5% exceed the token limit (figure 2 ). therefore, if we are able to accurately identify the relevant evidence prior to making an entailment prediction, we don't need to worry about the token limit in the vast majority of cases. consequently, we decided to tackle the evidence retrieval task before attempting the textual entailment task in order to filter the amount of text input into the model. all models were trained using the transformers library @xcite with the pytorch @xcite back-end.