S tanford ML ab at S em E val 2023 Task 7: Neural Methods for Clinical Trial Report NLI (2023)

Abstract: We present a system for natural language inference in breast cancer clinical trial reports, as framed by SemEval 2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data. In particular, we propose a suite of techniques for two related inference subtasks: entailment and evidence retrieval. The purpose of the textual entailment identification subtask is to determine the inference relation (either entailment or contradiction) between given statement pairs, while the goal of the evidence retrieval task is to identify a set of sentences that support this inference relation. To this end, we propose fine-tuning Bio+Clinical BERT, a BERT-based model pre-trained on clinical data. Along with presenting our system, we analyze our architectural decisions in the context of our model’s accuracy and conduct an error analysis. Overall, our system ranked 20 / 30 on the entailment subtask.

## introduction clinical trial reports (ctrs) detail the treatments and reactions that patients undergo over the course of a clinical study. the growing corpus of ctrs, coupled with a lack of adequate analytical tools, makes it increasingly difficult for clinical practitioners to provide up-to-date, personalized, evidencebased care @xcite . improvements in the reliability of ctrs have not been accompanied by developments in the ability to analyze and compare them, leading to a substantial disparity between the quality of ctrs and physicians' abilities to capitalize on them. @xcite . while natural language inference (nli) models demonstrate impressive capabilities in inferring logical relationships between texts, applications in the medical domain come with a unique set of challenges @xcite . one issue is that accurately inferring relationships between entities in * co-senior authors. the precise interpretation and extraction of evidence from ctrs could drastically diminish the current disconnect between research findings and clinical practices, as it will be much easier for clinicians to parse new publications and stay updated on the most effective treatment practices (?). in this paper, we experiment with a variety of modern and traditional machine learning approaches to accomplish this task. ultimately, we find that bio+clinical bert and biobert achieve the highest performance, obtaining an f1 score of 0.662 on the evaluation set. ## conclusion in this paper, we evaluate a variety of bert-based and other models for two inference sub-tasks: entailment and evidence retrieval. the models are compared on their accuracies and f1 scores. clinical trial reports from semeval task 7 dataset are used for the task. due to the small training dataset size, we use back translation, synonym replacement, and random perturbation to augment our training data. the top 5 performing models for task 1 are bio+clinical bert, distilbert, biobert, elec-tra small, and bio discharge summary bert. each of these has a learning rate of 10 -4 . these models are used to create a single voting ensemble model. the ensembling model results in an f1 score of 0.662 and outperforms other models. for the bm25 evidence retrieval baseline, our model is able to retrieve the majority of the relevant evidence. we use the voting ensembling model to make our final predictions.