HW - TSC at S em E val-2023 Task 7: Exploring the Natural Language Inference Capabilities of C hat GPT and Pre-trained Language Model for Clinical Trial (2023)

Abstract: In this paper, we describe the multi strategy system for SemEval-2022 Task 7, This task aims to determine whether a given statement is supported by one or two Clinical Trial reports, and to identify evidence that supports the statement. This is a task that requires high natural language inference capabilities. In Subtask 1, we compare our strategy based on prompt learning and ChatGPT with a baseline constructed using BERT in zero-shot setting, and validate the effectiveness of our strategy. In Subtask 2, we fine-tune DeBERTaV3 for classification without relying on the results from Subtask 1, and we observe that early stopping can effectively prevent model overfitting, which performs well in Subtask 2. In addition, we did not use any ensemble strategies. Ultimately, we achieved the 10th place in Subtask 1 and the 2nd place in Subtask 2.

## introduction in recent years, the introduction of the pre-trained masked language model (lm) bert @xcite has been a significant milestone in the field of natural language processing (nlp). bert achieves an absolute improvement of 7.7 points on the general language understanding evaluation (glue) benchmark and produces state-of-the-art (sota) results in multiple tasks. subsequently, numerous pre-trained language models (plms) based on the transformer architecture appear, such as xlnet @xcite , which addresses the issue of mask independence, biobert @xcite , a domainspecific model focused on biomedical text , and roberta @xcite . more recently, in 2021, deberta @xcite has emerged as a powerful model that achieves scores surpassing human performance on the superglue leaderboard, thanks to its disentangled attention and enhanced decoding mechanisms. in addition to models that reinforce natural language understanding, there are also models that focus on natural language generation, such as gpt @xcite , gpt-2 @xcite , and gpt-3 @xcite . for example, in 2022, chatgpt and in-structgpt @xcite have gained popularity for their remarkable performance using reinforcement learning and proximal policy optimization. their chain-of-thought (cot) ability with super large models enables logical reasoning in zero-resource scenarios. ## task description the goal of this task @xcite is to evaluate the natural language inference capability of systems on clinical trial reports in the medical domain. in recent years, there has been a significant increase in the publication of clinical trial reports, making it impossible for healthcare practitioners to keep up-to-date with all existing reports to provide personalized care. in this context, natural language inference presents an opportunity to retrieve and interpret medical evidence to support personalized care. this paper presents a task based on a set of breast cancer clinical trial registry (ctr) data, with the aim to classify four paragraphs in each ctr -eligibility criteria, interventions, outcomes, and adverse events -based on ctr annotations, interpretations, and domain expert labels. and the data example as shown in figure 1 . subtask 1 is textual entailment, where a given statement is evaluated for the inferential relationship (entailment or contradiction) with 1-2 specified ctrs. this subtask can be viewed as a binary classification task, where a common approach is to concatenate the statement and all the evidence in the ctrs, assume a pre-defined classification relationship, and predict a label of "0" (entailment) or "1" (contradiction). it should be noted that in some cases where the statement may have inherent inconsistencies, the relationship can be predicted as contradiction without reference to the ctr. letrozole of 12.5 mg/day is given for three successive days just prior to the second mri. ## conclusion in this paper, we propose solutions for two subtasks of semeval2023 task 7. for subtask 1, we explore the feasibility of using chatgpt and prompt learning for logical reasoning. although there are still many areas that can be improved, such as more refined prompts and inputting statements and evidence separately in multi-turn dialogues, chat-gpt's reasoning path is visible and analyzable compared with masked lms. we believe that chat-gpt's logical reasoning ability in zero-resource scenarios has great appeal to nlp researchers. for subtask 2, we explore the feasibility of the deberta-large classification model that does not depend on the predictions of subtask 1. we find that the early stopping strategy can effectively prevent overfitting. in addition, the appropriate segmentation strategy can also enhance the model's performance. in future work, we will try other segmentation strategies, such as adding explanations and descriptions for different evidence types. additionally, a prompt-based approach may be an interesting direction to explore.