Saama Technologies at S em E val-2024 Task 2: Three-module System for NLI 4 CT Enhanced by LLM -generated Intermediate Labels (2024)

Abstract: Participating in SemEval 2024 Task 2, we built a three-module system to predict entailment labels for NLI4CT, which consists of a sequence of the query generation module, the query answering module, and the aggregation module. We fine-tuned or prompted each module with the intermediate labels we generated with LLMs, and we optimized the combinations of different modules through experiments. Our system is ranked 19th ~ 24th in the SemEval 2024 Task 2 leaderboard in different metrics. We made several interesting observations regarding the correlation between different metrics and the sensitivity of our system on the aggregation module. We performed the error analysis on our system which can potentially help to improve our system further.

## introduction while the surge of large language models (llms) revolutionized many nlp tasks including natural language inference (nli), the threat of hallucination of llms is becoming a critical obstacle to applying these models to real-world problems. this is particularly the case for the field where the entailment of the statements with the records is crucial, such as biomedical applications dealing with clinical trials. semeval 2024 task 2 @xcite exactly tackles this question by providing the challenge for the nli4ct dataset @xcite composed of evidence made out of clinical trials, statements made based on this evidence, and the entailment labels of these statements. nli4ct deals with the language of english only. this paper explains our approach to the semeval 2024 task 2 in detail. we notice that many of the statements in the nli4ct dataset require a series of questions to be answered before verifying the agreement of the statements with the evidence. for that, we built the three-module system which contains the query generation module, the query answering module, and the aggregation module. for the development of each module, we needed intermediate labels about queries and their answers, and we sourced these labels by few-shot and zeroshot prompting on the llms. once we sourced the intermediate labels, we made a series of fewshot prompts with these intermediate labels to use them to prompt on fine-tuned models or the plain pre-trained models. then we selected our bestperforming model by experimenting with different combinations of modules. as the result of our experiments, we built the system ranked 19th∼24th in the semeval 2024 task 2 leaderboard in different metrics. despite relatively lower performance, we made several interesting observations. first, our experiments indicate that semantic-preserving interventions are more difficult to detect than semantic-altering interventions. second, we deduce that semantic-altering intervention from negative entailment to positive entailment is more difficult to capture than the opposite, based on our experiment results. we also performed some error analysis on our system and obtained some interesting insights to improve our system in the future. ## system overview our system is composed of three modules: the query generation (qg) module, the query answering (qa) module, and the aggregation (ag) module (figure 1 ). the query generation module generates a set of queries required to verify if the given statement is true or false. each generated query is passed to the query answering module along with the given evidence and generates an answer for that query. then all of the query-answer pairs are appended together with the given evidence and passed to the aggregation module to produce the entailment label prediction. ## limitations our investigation of fine-tuned models is restricted to relatively small open-sourced models due to the limited resources. the smaller size of our training data from our llm-generated intermediate labels compared to the original train set could impact the performance of modules as well. for prompting for qa and ag modules, the number of examples is limited to 3 due to the restricted context size. ## conclusion we built the three-module system to predict entailment labels for nli4ct. while our system's performance is not superb, we made several interesting observations regarding the correlation between different metrics and the sensitivity of our system on the aggregation module. as per our error analysis, it would be interesting to introduce some rule-based systems or llms trained/prompted to generate code to handle the numerical inputs for the aggregation module. it would be also promising to filter the evidence input by cohorts and trials related to the question for the qa module. it would be also interesting to extend our approach to a greater variety of larger llms or to expand the number of examples using llms with larger context sizes.