CPIC at S em E val-2023 Task 7: GPT 2-Based Model for Multi-evidence Natural Language Inference for Clinical Trial Data (2023)

Abstract: This paper describes our system submitted for SemEval Task 7, Multi-Evidence Natural Language Inference for Clinical Trial Data. The task consists of 2 subtasks. Subtask 1 is to determine the relationships between clinical trial data (CTR) and statements. Subtask 2 is to output a set of supporting facts extracted from the premises with the input of CTR premises and statements. Through experiments, we found that our GPT2-based pre-trained models can obtain good results in Subtask 2. Therefore, we use the GPT2-based pre-trained model to fine-tune Subtask 2. We transform the evidence retrieval task into a binary class task by combining premises and statements as input, and the output is whether the premises and statements match. We obtain a top-5 score in the evaluation phase of Subtask 2.

## introduction in recent years, there has been a significant increase in the number of clinical trial reports (ctr) publications. currently, the number of ctrs related to breast cancer alone exceeds 10,000. with this increasing trend, it is impossible for clinical practitioners to keep all the existing literature up to date in the future in order to provide personalized, evidence-based care @xcite . in this scenario, natural language inference (nli) @xcite provides a great opportunity to support the large-scale interpretation and retrieval of medical evidence. a successful natural language inference system enables clinical practitioners to obtain the latest medical evidence for better personalized evidence-based care @xcite . therefore, the study of semeval task 7 @xcite is particularly meaningful to provide technical ideas for the large-scale interpretation and retrieval of medical evidence. task 7 is based on ctrs of breast cancer and contains two subtasks. subtask 1 is textual entailment, which aims to determine the relationship (implication and contradiction) between ctrs and statements. subtask 2 is evidence retrieval, which outputs a set of supporting evidence extracted from the ctr premise with given ctr and statements to prove the prediction labels in subtask 1. our system focuses on subtask 2 which reasons about the statements supported by the facts in the premises by means of the generative capability of gpt-2 @xcite . firstly, our system concatenates the premises (i.e., facts in ctrs) and the inference(i.e., statements) through the prmopt template to obtain the inference sequence. then the inference sequence is input into gpt-2, and the output label is 0 or 1. the label 1 means that the premise fact and the statement are successfully matched, so the fact can be added to the corresponding group of supporting facts. finally, after the matching relationship between each fact and the statement is determined, we can obtain a set of facts supporting the current statement. besides, when exploring subtask 2 with gpt-2, we found that the parameter size of the model has a relatively large impact on the system. at the same time, the fusion between models with different parameters can have a significant improvement on the system. eventually with these strategies, our system achieved a top-five ranking. the remainder of this paper is organized as follows: section 2 is about the task data, task setup, etc. section 3 shows the overall algorithm and strategy of the system. section 4 contains the experimental setup, dataset segmentation, etc. section 5 presents experimental results and analysis. section 6 is the summary and outlook of the system in this paper. ## conclusion in this paper, we design a system based on gpt2based model to obtain the matching templates of premise and statement. our experiment results demonstrate that gpt2-based model significantly outperforms bert-based model on our constructed data. in addition, the combination of crossvalidation and model fusion can lead to significant improvement. finally, we obtained the top-5 ranking for subtask 2 of semeval2023 task 7. due to the time constraint of the competition, our model did not take into account the logical information between ctrs, but only matched the individual ctr with the statement, which can be further optimized in the future.