K now C omp at S em E val-2023 Task 7: Fine-tuning Pre-trained Language Models for Clinical Trial Entailment Identification (2023)

Abstract: In this paper, we present our system for the textual entailment identification task as a subtask of the SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data. The entailment identification task aims to determine whether a medical statement affirms a valid entailment given a clinical trial premise or forms a contradiction with it. Since the task is inherently a text classification task, we propose a system that performs binary classification given a statement and its associated clinical trial. Our proposed system leverages a human-defined prompt to aggregate the information contained in the statement, section name, and clinical trials. Pre-trained language models are then finetuned on the prompted input sentences to learn to discriminate the inference relation between the statement and clinical trial. To validate our system, we conduct extensive experiments with a wide variety of pre-trained language models. Our best system is built on DeBERTa-v3-large, which achieves an F1 score of 0.764 and secures the fifth rank in the official leaderboard.Further analysis indicates that leveraging our designed prompt is effective, and our model suffers from a low recall. Our code and pre-trained models are available at [ https://github.com/HKUST-KnowComp/NLI4CT ]( https://github.com/HKUST-KnowComp/NLI4CT ).

## introduction recently, the proliferation of clinical trial reports (ctrs) provides a large-scale reference base of scientific and factual knowledge for medical practitioners to make evidence-based clinical diagnoses @xcite . however, it also posits the challenge that clinical practitioners cannot memorize all current literature in order to provide up-to-date personalized evidence-based medical care @xcite . with the recent advances in natural language processing (nlp) ## problem definition in the textual entailment identification task, each input data contains three components: a medical statement, a section name indicating which section the statement claims about, and one or two ctr records that serve as the evidence to verify the statement. specifically, if the statement only makes claims about one certain trial, then only the corresponding trial will be used as input data. on the other hand, if the statement claims a comparison between a primary trial and a second trial, both ctrs need to be considered and are provided to the model as input information. consequently, the textual entailment identification task aims to determine the inferential relation between the medical statement and the associated section(s) in the claimed ctr(s). there are two possible inferential relations for each statement: entailment and contradiction. models are expected to predict whether each statement affirms an entailment or forms a contradiction given the associated section from the claimed ctr(s). formally, denote the medical statement as t, the section name as s t , the inferential relation as r t , and one ctr as @xmath0, c s2 , c s3 , ..., c s|c| }. if two ctrs are involved, they are denoted as c 1 and c 2 . the models will be given (t, s t , c st ) or (t, s t , c 1 st , c 2 st ) as input and required to predict y t that will be compared against r t . we use the dataset provided by jullien et al. ## system overview in this section, we introduce our proposed system. a general sketch is presented in figure 2 . ## results the full results are shown in we further study the ablation of our proposed prompt. deberta-v3-large finetuned on two different input prompts are compared as baselines. special token stands for replacing the separator tokens [sep] with natural language guidance such as "the evidence from ctr1 is". statement stands for using the statement only for prediction. the results, shown in meanwhile, our best model's precision score is ranked fourth, and the recall score is ranked thirteenth. this indicates that the model is wrongly classifying many entailments as contradictions, which causes a relatively low recall and is considered a major limitation of our system. by further observing the errors made by our system, we find that challenging ctr-statement pairs involving complex or mathematical reasoning @xcite are often wrongly classified. in addition, due to the length of some ctr premises being much longer than the ptlm's input max length, important pieces of evidence may be truncated and failed to provide guidance when making inferences on the statement. in our paper, we ignore this issue due to the fact that such cases infrequently occur through our manual inspection. however, positional encoding techniques such as relative position representation @xcite should be implemented to compensate for such information loss. in addition, we foresee three lines of future works that can be addressed to enhance the system's robustness further: (1) leveraging evidence retrieval @xcite or abstractive summarization @xcite to reduce the noise caused by irrelevant evidence and present the model with the most influential evidence. (2) applying augmentation techniques @xcite @xcite and aligning medical inferences against large eventuality knowledge graphs @xcite @xcite to mine inferential knowledge scalably and train the model more robustly. (3) leveraging gpt3.5 @xcite , chatgpt 4 or other advanced large language models to generate or extract @xcite ) key evidence as they have shown strong capabilities in these tasks @xcite . ## conclusion in this paper, we propose to finetune pre-trained language models to tackle the task of textual entailment identification as a solution to semeval2023 task 7. by using a discrete prompt to aggregate the medical statement and ctr premises, we maximally achieve an f1 score of 0.764 and ranked 4 https://chat.openai.com/chat fifth in the official leaderboard. analysis experiments also prove the effectiveness of our proposed prompt. however, our system suffers from an excessively low recall rate. this indicates that the model misclassifies too many entailments as contradictions. future work can focus on improving the ability of the model to identify entailments and improve its robustness. our work demonstrates that applying a larger pre-trained language model can effectively identify clinical trial entailments, which can be applied to real-world scenarios. all of our code, data, and models are publicly available for future contributions.