Team INF - UFRGS at S em E val-2023 Task 7: Supervised Contrastive Learning for Pair-level Sentence Classification and Evidence Retrieval (2023)

Abstract: This paper describes the EvidenceSCL system submitted by our team (INF-UFRGS) to SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data (NLI4CT). NLI4CT is divided into two tasks, one for determining the inference relation between a pair of statements in clinical trials and a second for retrieving a set of supporting facts from the premises necessary to justify the label predicted in the first task. Our approach uses pair-level supervised contrastive learning to classify pairs of sentences. We trained EvidenceSCL on two datasets created from NLI4CT and additional data from other NLI datasets. We show that our approach can address both goals of NLI4CT, and although it reached an intermediate position, there is room for improvement in the technique.

## introduction there has been a significant increase in medical publications in recent years, including clinical trial data. the semeval-2023 task 7, called multi-evidence natural language inference for clinical trial data (nli4ct), addresses the problem of large-scale interpretability and evidence retrieval from breast cancer clinical trial reports @xcite . currently, there are about 10k reports of breast cancer, making it difficult for clinical practitioners who want to provide care based on reliable clinical evidence to analyze all these reports carefully. thus, nli4ct is divided into task 1 -textual entailment and task 2 -evidence retrieval (er). the goal of task 1 is to determine the inference relation (contradiction/entailment) between statement pairs (premise and hypothesis), and task 2 is to output the supporting facts from the premise to justify the label in task 1. recent work leveraged natural language inference (nli) data to enhance sentence representations, improving results in downstream tasks. pair-supcon @xcite adds a discrimination head to separate entailment and contradiction instances while learning high-level semantic information from the sentence pairs and negative examples. a linear classification head enables supporting pairwise entailment and contradiction reasoning. similarly, pairscl @xcite adds a cross-attention module to enhance pair-level representation by calculating the token-level co-attention matrix to indicate the relevance of the tokens in the premises and hypotheses. our approach explores the cross-attention module in pairscl for evidence retrieval by separating evidence from non-evidence in the embedding space. evidencescl is built on top of pairscl @xcite , employing a supervised contrastive loss in the evidence retrieval objective and a classification head in the textual entailment objective. we also created two datasets combining mednli @xcite , a small part of multinli @xcite , and the nli4ct dataset. we show that evidencescl addresses the tasks of nli4ct, and although it achieved an intermediate position in the ranking of participating systems, there is room for improvement. ## background natural language inference (nli) is the task of determining if two sentences follow each other. we want to determine if a hypothesis can be inferred from a premise. nli datasets present a set of linguistic phenomena such as negation, modals, quantifiers, pronouns, beliefs, conditionals, tense, and a variety of others. some phenomena are more frequent in specific genres of text. for instance, conversational genres have the highest percentage of sentence pairs with an occurrence of negation, whwords, belief-verbs, and time terms. in contrast, the verbatim genre has more sentence pairs with quantifiers and conversational pivots @xcite . however, clinical texts contain specific lin-guistic phenomena, such as medical terms, abbreviations, or medical concepts written in different forms. incorporating domain knowledge has been shown to improve model accuracy in the nli task for clinical data @xcite . mednli (romanov and shivade, 2018) is an expert annotated dataset for nli in the clinical domain. premises are derived from mimic-iii (v1.3) @xcite (which contains more than 2 million clinical notes written by healthcare professionals in english) and clinicians generated three hypotheses for each. premises and hypotheses express various medical concepts from umls semantic types @xcite , such as finding, disease or syndrome, sign or symptom, pharmacological substance, and others. nli4ct @xcite addresses a specific nli task where multiple premises can be evidence to justify the label assigned to the hypothesis. the nli4ct data comprises a set of sentences (i.e., premises) from breast cancer clinical trials, statements (i.e., hypotheses), and labels annotated by domain experts. premises are collected from four sections of the clinical trial reports: eligibility criteria, intervention, results, and adverse events. the sentence pairs may present linguistic phenomena, such as quantifiers, medical concepts, acronyms, drugs, time frames, and dosage. also, not all premises are relevant to a given hypothesis. thus, retrieving the correct premises supporting the hypothesis is essential. some techniques rely on training sentence embeddings with supervised data to improve accuracy in downstream tasks @xcite . models learned from nli datasets perform better than supervised tasks or models trained with unlabeled data. the most likely assumption is that models trained on nli datasets require a highlevel understanding of the semantic relationship between tokens in pairs of sentences. ## evidencescl system this section details the nli datasets and the model used in the evidencescl system. ## experimental setup we fine-tuned evidencescl starting from the pretrained biomed roberta checkpoints (gururangan et al., 2020) for at most 80 epochs using the adamw optimizer @xcite . we used a batch size of 512 1 and set the temperature parameter τ to 0.1. to combine the two objectives, as suggested by @xcite , we set α to 1 in our experiments. we performed experiments by setting the maximum sequence length to 128 to assess the performance when information loss occurs. to mitigate overfitting, we added an l1 regularization term to the overall loss, where the l1 coefficient was set to 0.1. we also used weight decay as an l2 regularization factor with a weight decay coefficient of 1e -4. we implemented an early stopping mechanism to stop the training when the difference between the training and validation losses became negligible. we used different initial values for the learning rate and the cosine annealing warm restart approach to update it following a cosine curve, resetting it each epoch. ## results we conducted experiments with different hyperparameter settings to evaluate the performance of our model. we used the f1 score, precision, and recall as evaluation metrics. we fine-tuned a linear classifier for the er task to distinguish evidence from non-evidence examples in the two-labeled dataset. in the three-labeled dataset, we labeled non-evidence examples as neutral and used them for evaluation. for the nli task, we evaluated pairs of sentences individually and aggregated their predictions to obtain a final classification for each sample. we chose the majority the evidencescl model performs better on instances with fewer sentence pairs, and its accuracy decreases for instances with many pairs. we ruled out the combination of er and nli goals in the loss functions as the cause of misclassification errors, such as the ones observed in figure 5 , and conjectured that the dataset likely caused them. in our study, we created a dataset by combining samples from mednli, multinli, and nli4ct, although the latter contained sentence pairs labeled as evidence and non-evidence. this design choice may have negatively influenced the model since the same premise can be treated as evidence or non-evidence in different instances, making it challenging for the model to learn when it is one or the other. additionally, the medical domain presents challenges, such as acronyms or technical terms in the hypotheses not mentioned in the premises, which can affect performance. improving the mapping of these features could help the model learn to classify a premise as evidence more accurately. regarding model parameters, we found that using learning rates of 5e-05, 1e-06, and 5e-06 in models trained between 10 and 15 epochs yielded better results. we also observed that accuracy decreased as the number of training epochs increased. we noticed that sometimes the model might suffer from overfitting, which relates to how training and validation stages are performed concerning the nli4ct dataset. we performed new experiments and found that grouping entailment/contradiction instances close to their respective negative examples in the nli4ct dataset during training improved the accuracy. ## conclusion in this paper, we described our submission to semeval-2023 task 7. we used supervised contrastive learning to improve pair-level sentence representations of the biomed roberta model. we fine-tuned a linear classifier on top of it to figure out pieces of evidence and perform textual entailment classification on sentence pairs. evidencescl differs from pairscl by addressing er and nli tasks, learning the underlying semantic relationship between terms of the sentence pairs when separating pairs as evidence or non-evidence, and classifying them as entailment or contradiction. our experiments trained models with different hyper-parameter settings on two-labeled and threelabeled datasets. evidencescl outperformed the baselines, reaching an f1 score of 0.666 and ranked 17th on the leaderboard for task 1 -textual entailment. for task 2 -evidence retrieval, we reached an f1 score of 0.681 and ranked 21st on the leaderboard. we also showed that the model could be further improved by hyperparameter tuning. we are performing study ablations to analyze the gain in accuracy when fine-tuning the encoder using different combinations of the three datasets. also, we want to analyze the impact of the hyperparameter tunning on model results, comparing the models on downstream tasks.