CL a C at S em E val-2024 Task 2: Faithful Clinical Trial Inference (2024)

Abstract: This paper presents the methodology used for our participation in SemEval 2024 Task 2 (Jullien et al., 2024) – Safe Biomedical Natural Language Inference for Clinical Trials. The task involved Natural Language Inference (NLI) on clinical trial data, where statements were provided regarding information within Clinical Trial Reports (CTRs). These statements could pertain to a single CTR or compare two CTRs, requiring the identification of the inference relation (entailment vs contradiction) between CTR-statement pairs. Evaluation was based on F1, Faithfulness, and Consistency metrics, with priority given to the latter two by the organizers. Our approach aims to maximize Faithfulness and Consistency, guided by intuitive definitions provided by the organizers, without detailed metric calculations. Experimentally, our approach yielded models achieving maximal Faithfulness (top rank) and average Consistency (mid rank) at the expense of F1 (low rank). Future work will focus on refining our approach to achieve a balance among all three metrics.

## introduction clinical trials serve as the cornerstone for evaluating the efficacy and safety of novel medical interventions, playing a pivotal role in advancing healthcare practices @xcite . clinical trial reports (ctrs) encapsulate crucial information regarding trial methodologies and outcomes, serving as indispensable resources for healthcare professionals in treatment decision-making @xcite . however, the sheer volume of available ctrs, coupled with their rapid proliferation, poses significant challenges for comprehensive literature review and evidence synthesis in clinical practice @xcite . natural language inference (nli) emerges as a promising approach to address this issue @xcite @xcite , facilitating the scalable interpretation and retrieval of medical evidence @xcite @xcite . the semeval 2024 task 2 @xcite on safe biomedical nli for clinical trials extends this paradigm to enable automated inference of relationships between statements and ctrs, thus streamlining evidence extraction and enhancing decision-making processes in healthcare. the 2024 task is a continuation of the one introduced by @xcite , specifically it focuses on track 1, which focuses on nli in the context of clinical trials. in this task, the input consists of pairs of clinical trial reports (ctrs) and corresponding statements, where the statements make claims about the information contained within the ctrs. the objective is to determine the inference relation between each ctr-statement pair, classifying them as either entailing or contradicting each other. for instance, given a statement "drug x is effective in treating condition y" and a ctr outlining a clinical trial testing drug x's efficacy, the task is to determine whether the statement is entailed by the ctr or contradicted by it. the datasets used are similar to those introduced by @xcite , and further details can be found in there work. our system primarily focuses on maximizing faithfulness and consistency in the context of se-meval 2024 task 2 @xcite . to achieve this goal, we adopt a strategy centered around introducing controlled input noise during model training. this approach is based on the hypothesis that a certain level of tolerance towards input perturbations could enhance the faithfulness and consistency of the generated models. specifically, we experiment with randomly masking a percentage (k%) of tokens in both clinical trial reports (ctrs) and the corresponding statements, thereby exposing the model to varying degrees of input uncertainty. through these experiments, we aim to optimize model performance in capturing the relationships between statements and ctrs, ultimately improving the system's effectiveness in clinical trial inference tasks. through our participation in semeval 2024 task 2 @xcite , we observed a notable trade-off between different evaluation metrics. while our approach successfully improved faithfulness and consistency metrics, it came at the expense of f1 scores. this finding underscores the challenge of balancing these evaluation criteria and thus the need for future refinement to achieve a more harmonious optimization across all relevant metrics. specifically, our models achieved top-ranking levels of faithfulness but demonstrated only average performance in consistency metrics, resulting in lower ranks in f1 assessment. see sec. 4 for details. ## system overview in our system, we leverage bart @xcite as the primary model for all experiments due to its robustness and effectiveness in various natural language processing tasks, particularly in natural language inference (nli) @xcite @xcite . to streamline the fine-tuning process and enhance efficiency, we adopt the lora technique proposed by @xcite , which significantly reduces the fine-tuning time without sacrificing performance. additionally, we incorporate the contrastive tension loss function introduced by @xcite . to guide the fine-tuning process. this loss function promotes contrastive learning by separately encoding the clinical trial reports (ctrs) and their associated statements using two copies of bart @xcite during each training instance. by allowing only one copy to update its parameters at a time, the model is encouraged to focus on learning the essential semantic relationships between the ctrs and the statements. moreover, we introduce a novel approach to enhance the robustness of the model by incorporating random token masking during each training instance. specifically, we randomly mask a percentage (k%) of tokens in both the ctrs and their associated statements. this introduces noise in the input data, forcing the model to adapt to varying degrees of input uncertainty and preventing it from relying solely on superficial patterns. the rationale behind this approach is to encourage the model to concentrate on the fundamental semantic content of the input rather than exploiting surface-level correlations. balancing between the three required metrics-faithfulness, consistency, and f1-proved to be the primary challenge in our experimental setup. while optimizing for one metric often led to improvements in its performance, it frequently came at the expense of others. we explored different strategies to strike a balance between these metrics. however, finding an optimal solution that simultaneously maximized all three metrics remained unsolved. our system struggled to maintain high levels of f1 score while simultaneously improving faithfulness and consistency metrics. further exploration of optimization strategies and leveraging ensemble methods may offer potential avenues for achieving a better balance between the metrics. ## experimental setup training details the training, validation, and test data for our experiments were all provided by the organizers of the semeval 2024 task 2, as outlined by @xcite . we trained our model for a total of 40 epochs, employing the adam optimizer (kingma and ba, 2014) with a learning rate of 0.0001, with a batch size of 32. to stabilize and accelerate training, we implemented gradient clipping @xcite with a maximum norm of 1. additionally, we incorporated a linear warmup stage consisting of 40 gradient steps followed by a cosine annealing learning rate schedule @xcite . this strategy enabled gradual adjustment of the learning rate during the initial phase of training, allowing the model to converge more smoothly towards an optimal solution. furthermore, we limited the maximum sequence length to 256 tokens for both ctrs and their corresponding statements, aligning with the model architecture and computational capabilities. consequently, each sequence pair was truncated to a total maximum of 512 tokens. this limitation on sequence length helped in handling memory constraints and optimizing the processing of input sequences during training. exploring larger context sizes could be advantageous for future improvements in the task. evaluation metrics our system's performance is measured through three key metrics: (1) faithfulness, (2) consistency, and (3) f1 score. faithfulness: faithfulness measures the degree to which a given system arrives at the correct prediction for the correct reason. intuitively, this is estimated by assessing the model's ability to correctly change its predictions when subjected to a semantic altering intervention. let n denote the number of statements x i in the contrast set (c), y i represent their respective original statements, and f () denote the model predictions. faithfulness is computed using equation below: consistency: consistency aims to measure the extent to which a given system produces the same outputs for semantically equivalent problems. it assesses the system's ability to predict the same label for original statements and contrast statements for semantic preserving interventions. even if the final prediction is incorrect, the representation of the semantic phenomena should be consistent across the statements. let n denote the number of statements x i in the contrast set (c), y i represent their respective original statements, and f () denote the model predictions. consistency is computed using equation below: f1: the f1 score is a commonly used metric in nlp tasks @xcite , measuring the balance between precision and recall of a model's predictions. it is calculated based on the geometric mean of precision and recall, where precision represents the ratio of true positive predictions to the total number of predicted positive instances, and recall represents the ratio of true positive predictions to the total number of actual positive instances. ## results our experimental results demonstrate a clear tradeoff between the level of token masking (k) and the performance metrics of f1, faithfulness, and consistency. as we increase the masking level, we observe a consistent trend of decreasing f1 scores alongside increasing faithfulness and consistency metrics. for @xmath0, representing no token masking, we observe an f1 score of 0.65 (ranking 22 out of 32), a faithfulness score of 0.51 (ranking 21 out of 28), and a consistency score of 0.54 (ranking 25 out of 30). as we progressively increase k, we observe a gradual change in the metrics, specifically decrease in f1, and increase of the other 2 metrics. at @xmath1, the highest masking level tested, we observe a significant drop in f1 score to 0.06 (ranking 28 out of 31), accompanied by substantial increases in faithfulness (0.95, ranking 1 out of 28) and consistency (0.6, ranking 22 out of 32) metrics. given the substantial decrease in f1 scores beyond a masking level of @xmath2, we did not explore higher masking levels. this decision was made due to the observed trade-off, where increasing token masking beyond a certain threshold led to disproportionately low f1 scores, potentially indicating a loss of model generalization and predictive performance. ## conclusion our experiments underscore the intricate balance between token masking levels and performance metrics in biomedical nli for clinical trials. we observed a discernible trade-off: while increasing token masking improves faithfulness and consistency, it results in diminished f1 scores. this finding highlights the necessity of exploring future approaches that could better optimize the model with multiple evaluation criteria as their objective function. additionally, another potential avenue for improvement would involve examining alternative metrics to provide further insights into the behaviour of the model @xcite @xcite . based on our findings, one avenue of research involves refining token masking strategies to achieve a more optimal balance between f1, faithfulness, and consistency metrics. furthermore, exploring ensemble methods and alternative fine-tuning strategies could provide valuable insights into enhancing the overall performance of the model.