FZI - WIM at S em E val-2024 Task 2: Self-Consistent C o T for Complex NLI in Biomedical Domain (2024)

Abstract: This paper describes the inference system of FZI-WIM at the SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials. Our system utilizes the chain of thought (CoT) paradigm to tackle this complex reasoning problem and further improve the CoT performance with self-consistency. Instead of greedy decoding, we sample multiple reasoning chains with the same prompt and make thefinal verification with majority voting. The self-consistent CoT system achieves a baseline F1 score of 0.80 (1st), faithfulness score of 0.90 (3rd), and consistency score of 0.73 (12th). We release the code and data publicly.

## introduction the safe biomedical natural language inference for clinical trials (nli4ct) task aims to investigate the consistency and faithfulness of natural language inference (nli) models in clinical settings @xcite . nli is a typical natural language task requiring natural language reasoning @xcite . fine-tuned bert-based @xcite discriminative models have been widely applied to solve nli problems @xcite . studies show increasing reasoning capabilities of large language models (llms), both in proprietary @xcite and open-source llms @xcite . however, problems of inconsistency and unfaithfulness still occur with llms @xcite . compared to other domains, medical applications have much higher standards regarding safety and trustfulness, so inconsistencies and unfaithfulness limit ai applications in the medical domain. chain of thought (cot) has been proposed to elicit the reasoning capabilities of llms @xcite 1 https://github.com/jens5588/fzi-wim-nli4ct 2022). based on the cot, further concepts like tree of thought @xcite , react @xcite , self-consistency @xcite , and so on have been proposed to improve the performance of cot further. one common characteristic of the frameworks mentioned above is the explicit generation of reasoning chains. since only verification labels are provided in the nli4ct training dataset, we utilize gpt-4 to generate reasoning chains. with the distilled knowledge from gpt-4, we further instruction-tune an open-source llm with low-rank adaption (lora) @xcite for claim verification with cot. our system follows the self-consistency concept by generating multiple cot reasoning chains and verifying with majority voting. we summarize our major findings regarding this task as follows: nli4ct contains one text entailment task, namely infer the relationship between a premise and a statement as either entailment or contradiction @xcite . the premises in the nli4ct dataset are collected from publicly available breast cancer clinical trial reports (ctrs), split into four sections: eligibility criteria, intervention, results, and adverse event @xcite . the statements are sentences making claims about the information in the ctr premise, either about a single ctr or a comparison between 2 ctrs. the numbers of training, validation, and test examples are 1700, 200, and 5500, respectively. in the test set, 500 examples are used as anchors. the other 5000 statements are created with interventions on these first 500 examples. figure 1 shows an example with interventions on the statements. using the same clinical report, two statements are modified, one semantics-preserving (modified statement 1) and the other semantics-altering (modified statement 2), based on the original statement. the purpose of the interventions is to investigate the consistency and faithfulness of the inference. for the text entailment task in the first iteration of nli4ct (semeval-2023 task 7), most systems have fine-tuned discriminative transformerbased models @xcite . kanakarajan and sankarasubbu (2023) instruction-tuned flan-t5 model and achieved the 2nd place for the text entailment task. however, the systems mentioned above only predicted the verification labels without the reasoning process. to achieve a trustworthy ver-ification, our system not only predicts the label but also the reasoning chains. since the training and validation datasets have only provided verification labels. we utilize gpt-4 to verify with reasoning chains for training and validation datasets. we further instruction-tune an open-source llm with distilled reasoning chains from gpt-4. to address the inconsistency problem in reasoning chains, we sample multiple chains and then employ a majority voting approach to determine the final verification outcome. ## system overview figure 2 shows the pipelines of data creation, model training, and inference of our system. in the following, we describe each part. ## experimental setup in this section, we describe our experiment setup. we describe the cot generation with gpt-4, the model training, and the inference setups. ## evaluation in this section, we report the results of our selfconsistent cot system. for comparison, we also report the results of label-only verification and cotgreedy. the comparison of three systems serves as an ablation study to analyze the performance improvement of extra steps, reasoning chain generation and self-consistent verification. ## conclusion in this paper, we have reported our self-consistent cot system for the semeval-2024 task 2: safe biomedical natural language inference for clinical trials. for comparison, we also reported the label-only greedy and cot greedy inference systems. to achieve a trustworthy inference system, we utilized the cot reasoning paradigm, not only predicting the verification labels but also rationales. we tackled cot's inconsistency problem with selfconsistent cot. compared to the greedy cot system, we have improved the inference performance by generating multiple reasoning chains and verifying with majority voting. however, the performance improvement is limited. for future work, a more fine-grained evaluation of the correctness of the reasoning steps in the cot paradigm is promising for solving complex reasoning tasks.