D - NLP at S em E val-2024 Task 2: Evaluating Clinical Inference Capabilities of Large Language Models (2024)

Abstract: Large language models (LLMs) have garnered significant attention and widespread usage due to their impressive performance in various tasks. However, they are not without their own set of challenges, including issues such as hallucinations, factual inconsistencies, and limitations in numerical-quantitative reasoning. Evaluating LLMs in miscellaneous reasoning tasks remains an active area of research. Prior to the breakthrough of LLMs, Transformers had already proven successful in the medical domain, effectively employed for various natural language understanding (NLU) tasks. Following this trend, LLMs have also been trained and utilized in the medical domain, raising concerns regarding factual accuracy, adherence tosafety protocols, and inherent limitations. In this paper, we focus on evaluating the natural language inference capabilities of popular open-source and closed-source LLMs using clinical trial reports as the dataset. We present the performance results of each LLM and further analyze their performance on a development set, particularly focusing on challenging instances that involve medical abbreviations and require numerical-quantitative reasoning. Gemini, our leading LLM, achieved a test set F1-score of 0.748, securing the ninth position on the task scoreboard. Our work is the first of its kind, offering a thorough examination of the inference capabilities of LLMs within the medical domain.

## introduction large language models (llms) have brought about a paradigm shift in the field of natural language processing (nlp) @xcite . their exceptional performance across various tasks has led to a surge in real-world applications utilizing llm-based technology. however, a notable drawback of llms is their propensity to generate plausible yet incorrect information, commonly referred to as "hallucinations" @xcite . the remarkable breakthrough of llms has raised questions regarding their "intelligent" capabilities, particularly in reasoning and inference @xcite @xcite . two specific areas that have garnered significant attention in relation to llms' reasoning abilities are numerical-quantitative reasoning and natural language inference. these areas are considered integral to human intelligence, prompting researchers to establish benchmarks and evaluate llm performance in these domains @xcite . llms often exhibit limited performance in solving arithmetic reasoning tasks, frequently producing incorrect answers @xcite . unlike natural language understanding, math problems typically possess a single correct solution, making the accurate generation of solutions more challenging for llms. regarding nli, performance reduction can be observed due to shortcut learning @xcite and hallucinations @xcite . these investigations aim to discern whether llms are mere memorizers of training data or possess genuine logical reasoning abilities. the volume of medical publications, including clinical trial data, has experienced a significant upsurge in recent years. the semeval-2023 task 7, known as multi-evidence natural language inference for clinical trial data (nli4ct), aimed to address the challenge of large-scale interpretability and evidence retrieval from breast cancer clinical trial reports @xcite . this task required multi-hop biomedical and numerical reasoning, which are crucial for developing systems capable of interpreting and retrieving medical evidence on a large scale, thereby facilitating personalized evidence-based care. while the previous iteration of nli4ct resulted in the development of llm-based models @xcite @xcite achieving high performance (e.g., f1-score ≈ 85%), the application of llms in critical domains, such as real-world clinical trials, necessitates further investigation. consequently, the second iteration of nli4ct, semeval-2024 task 2, titled "safe biomedical natural language inference for clinical trials" @xcite is proposed, featuring an enriched dataset that includes a novel contrast set obtained through interventions applied to statements in the nli4ct test set. our work involves the evaluation of various popular open-source and closed-source llms on the development and test sets to explore their reasoning capabilities in the domain of medical nli. we present the results by thoroughly analyzing the performance on the development set, with the best-performing llm ranking ninth on the task leaderboard. we have made the results on the development set available on our github repository foot_0 . another aspect of our work was that we deliberately refrained from investing significant effort into prompting or experimenting with different prompts. additionally, we aimed to showcase the remarkable development of llms, demonstrating their capacity to effectively engage with the task while minimizing dependence on the prompt. ## related work with the emergence of large language models (llms), there has been a growing interest in exploring their capabilities within the clinical domain. recent studies have delved into both the potential of llms and the associated risks when applied in clinical settings. for instance, @xcite conducted experiments utilizing gpt-3.5 on various medical nlp datasets, assessing metrics such as factuality and safety, ultimately highlighting the high level of safety offered by gpt-3.5 foot_1 . @xcite focused on the challenges posed by hallucinations in llms and proposed a benchmark dataset called med-halt (medical domain hallucination test) to evaluate popular llms on this front. regarding the reasoning capabilities of llms, @xcite introduced a diagnostic framework that prioritizes reasoning and employs prompt-based learning. the study specifically fo-cused on clinical reasoning for disease diagnosis, where the llms generate diagnostic rationales to provide insights into patient data and the reasoning path leading to the diagnosis, known as clinical chain-of-thought (clinical cot), using gpt-3.5 and gpt-4 (openai, 2024). notably, none of the previous studies simultaneously examined the performance of both open-source and closed-source llms, particularly with a comprehensive focus on inference. consequently, our work stands as the first of its kind in this regard. ## task and dataset description the clinical trials used to construct the dataset were sourced from clinicaltrials.gov foot_2 , a comprehensive database managed by the u.s. national library of medicine. clinicaltrials.gov contains information on various clinical studies conducted worldwide, both publicly and privately funded. the dataset specifically focuses on clinical trials related to breast cancer and includes a total of 1,000 trials written in english. for the task at hand, each ctr may contain one or two patient groups, known as cohorts or arms, which may receive different treatments or have different baseline characteristics. the dataset consists of a total of 7,400 statements. these statements were divided into a training dataset comprising 1,700 statements, a development dataset containing 200 statements, and a hidden test dataset consisting of 5,500 statements. the statements can be categorized into two types: those that are solely related to a single ctr and others that involve a comparison between two different reports. each statement in the dataset is labeled as either "entailment" or "contradiction". figure 1 shows an example statement from the training set. the task primarily involves binary classification, aiming to predict whether the label corresponds to entailment or contradiction. the evaluation process encompasses three aspects. initially, the macro f1score is computed based on the binary classification results. subsequently, two semantic evaluations are conducted: faithfulness and consistency. faithfulness assesses the system's ability to arrive at accurate predictions for the correct reasons, while consistency measures the system's ability to produce consistent outputs for semantically equivalent problems. the task organizers evaluate faithfulness by providing semantically altered instances, and consistency by providing preserved instances for comparison. ## language model performance evaluation this section aims to provide a detailed analysis of the performance achieved by each individual llm. based on the evaluation of various llms, including closed-source models like gpt-3.5 (chatgpt), claude @xcite , and gemini pro (gemini team, 2023), as well as open-source models like falcon 40b @xcite , mixtral 8x7b @xcite , and llama 2 70b @xcite , the performance of these models was assessed on the dev and test sets. all conversations took place on the poe.com platform, providing users with a seamless chat experience. to transmit both the development set and the test set instances, we utilized an api wrapper code in a python script, which can be accessed in our github repository. as mentioned earlier, we intentionally avoided extensive prompting and instead employed a straightforward, consistent prompt for all instances. each model's chat session commenced with a greeting, followed by a brief introductory sentence regarding the task, and subsequently, all instances were dispatched via the python script. appendix a provides information regarding the prompts. in the next section, we delve into the detailed performance analysis of the language models on the development set, focusing on specific cases of model incorrect gemini pro 36 claude 38 palm 42 falcon 40b 51 gpt-3.5 59 llama 2 70b 65 mixtral 8x7b 69 ## conclusion our detailed analysis of llms' performance on various reasoning tasks in the medical domain reveals that they are not merely passive memorizers. they possess the ability to perform numericalquantitative reasoning, general reasoning, and abbreviation resolution, even in a highly specialized domain with unique vocabulary. notably, falcon 40b, an open-source llm, demonstrated impressive performance, rivaling top closed-source llms. despite their successes, llms are not without limitations. occasional nonsensical predictions highlight the need for caution when using them in high-stakes domains such as medicine. however, the results of our study are highly promising and suggest that with increased training data and computational power, llms have the potential to become invaluable tools in the medical field. the future of llms in medicine holds exciting possibilities. as these models continue to evolve, we anticipate that they will play an increasingly significant role in healthcare, transforming the way we diagnose, treat, and prevent diseases.