CRCL at S em E val-2024 Task 2: Simple prompt optimizations (2024)

Abstract: We present a baseline for the SemEval 2024 task 2 challenge, whose objective is to ascertain the inference relationship between pairs of clinical trial report sections and statements.We apply prompt optimization techniques with LLM Instruct models provided as a Language Model-as-a-Service (LMaaS).We observed, in line with recent findings, that synthetic CoT prompts significantly enhance manually crafted ones.The source code is available at this GitHub repository https://github.com/ClementBM-CLB/semeval-2024

## introduction since the introduction of large pre-trained transformer models such as gpt-3.5, released in early 2022, foundational models have begun to be utilized widely. while bert-like models have proven to be effective in various nlp tasks such as named entity recognition @xcite , scaling up the number of parameters in transformer models not only enhances their capabilities but also endows them with new abilities not seen in smaller models @xcite . these capabilities are particularly evident in natural language inference tasks, where the model must deduce the veracity of two given texts @xcite . llms, gaining popularity for their reasoning capabilities, still face trustworthiness concerns, crucial in the medical domain where decisions affect lives. medical devices must exhibit reliability and undergo rigorous testing before they are brought to market. semeval 2024 @xcite focuses on assessing nli system robustness, coherence, and accuracy, particularly llms prone to shortcut learning, factual discrepancies, and performance degradation from word distribution shifts @xcite . fine-tuning, while effective for task and domain adaptation, demands excessive resources in the case of large language models (llms). in the medical field, data is highly sensitive and protected by privacy regulations. therefore, applying finetuning techniques to such sensitive data would imply that medical centers have readily available onpremise infrastructure @xcite . considering these limitations, we investigate hard prompt optimization techniques such as chain-of-thought prompting @xcite . acknowledging the in-context learning (icl) as an indirect method of fine-tuning, we also explored in-context learning strategies @xcite . among them, we were particularly inspired by medprompt, a promising composite prompting method applied to medical datasets, which achieved a 27% reduction in error rates on medqa @xcite . following the semeval 2024 task 7 @xcite , semeval 2024 task 2 focuses on identifying the inference relationship (entailment vs. contradiction) between clinical trial report (ctr) statement pairs. these statements and the supporting evidence are crafted by individuals with expertise in the clinical domain, including clinical trial organizers and research oncologists. the clinical trials information is sourced from the foot_0 clinicaltrials.gov website (maintained by the nih). we have evaluated three llm prompting methods to address this task. ## evaluation metrics faithfulness measures the extent to which a given system arrives at the correct prediction for the correct reason. this is estimated by measuring the ability of a model to correctly change its predictions when exposed to a semantic altering intervention. given n statements x i in the contrast set (c), their respective original statements y i , and model predictions f () we compute faithfulness using equation 1 . consistency aims to measure the extent to which a given system produces the same outputs for semantically equivalent problems. therefore, consistency is measured as the ability of a system to predict the same label for original statements and contrast statements for semantic preserving interventions. given n statements x i in the contrast set (c), their respective original statements y i , and model predictions f () we compute faithfulness using equation 2. ## conclusion by employing prompt optimization techniques with llm instruct models, we see the significant enhancement zero-shot cot prompts provide compared to manually crafted ones. this highlights the critical role of utilizing advanced techniques in llm prompting to enhance inference tasks, particularly in domains like clinical trials.