T5-Medical at S em E val-2024 Task 2: Using T5 Medical Embedding for Natural Language Inference on Clinical Trial Data (2024)

Abstract: In this work, we address the challenge of identifying the inference relation between a plain language statement and Clinical Trial Reports (CTRs) by using a T5-large model embedding. The task, hosted at SemEval-2024, involves the use of the NLI4CT dataset. Each instance in the dataset has one or two CTRs, along with an annotation from domain experts, a section marker, a statement, and an entailment/contradiction label. The goal is to determine if a statement entails or contradicts the given information within a trial description. Our submission consists of a T5-large model pre-trained on the medical domain. Then the pre-trained model embedding output provides the embedding representation of the text. Eventually, after a fine-tuning phase, the provided embeddings are used to determine the CTRs’ and the statements’ cosine similarity to perform the classification. On the official test set, our submitted approach is able to reach an F1 score of 0.63, and a faithfulness and consistency score of 0.30 and 0.50 respectively.

## introduction in experimental medicine, clinical trials are essential because they verify the effectiveness and safety of novel treatments @xcite . clinical trial reports (ctrs) are documents that describe the design and outcomes of a clinical trial and are used to direct patient interventions that are specific to them. but with over 400,000 published ctrs and more coming out each year @xcite , it is not feasible to manually conduct thorough reviews of all the pertinent literature while developing new treatment procedures. for these reasons, the requirement for technologies that can automatically extract and classify information is always expanding. with the development of machine and deep learning architectures in recent years, there has been a surge in interest in natural language processing, or nlp. many efforts have gone into creating algorithms that can automatically identify and categorize text information that is accessible on the internet. in the literature, to perform text classification tasks, several strategies have already been proposed. in the last fifteen years, some of the most successful ones have been based on svm @xcite , on convolutional neural network (cnn) @xcite , on graph neural network (gnn) @xcite , on ensemble models @xcite and, recently, on transformers @xcite . for example, to address the ctr proposed task, and to enable a higher degree of accuracy and efficiency in individualized evidence-based treatment, natural language inference (nli) (maccartney, 2009) provides a viable solution for the large-scale interpretation and retrieval of medical evidence @xcite . semeval-2024 task 2 -multi-evidence natural language inference for clinical trial data (nli4ct) @xcite ) -relies on the nli4ct dataset foot_0 . the task is to determine the inference relation between a natural language statement, and a ctr. inference chains in this drop-off range have to be constructed for a significant fraction of the nli4ct dataset instances. furthermore, inference on nli4ct requires quantitative and numerical reasoning. research has demonstrated that transformer-based models rely on flimsy heuristics for predictions instead of consistently applying this kind of reasoning @xcite . to develop our model, we thought of a two-stage architecture. in the first stage, we used a sentence transformer specifically trained on the medical domain. on the generated embeddings, we evaluated a cosine similarity to predict the entailment or con-tradiction relationship between the two sentences analyzed. the remainder of the paper is structured as follows. we give some background information on task 2 hosted at semeval-2024 in section 2. section 3 offers an explanation of the submitted approach. we describe the experimental setup to reproduce our work in section 4. the outcomes of the formal assignment and certain debates are given in section 5. we provide our conclusion and suggestions for further research in section 6. we make all the code publicly available and reusable on github foot_1 . ## background we give some background information on task 2 hosted at semeval-2024 in this section. the task is predicated on a set of ctrs, statements, labels, and explanations related to breast cancer that have been annotated by domain experts. the gathered ctrs are compiled into four components for the textual entailment task: with an average length of 19.5 tokens, the annotated statements are sentences that make a claim regarding the data presented in one of the ctr premise's sections. the remarks could compare two ctrs or make assertions about a single ctr. finding the inference relation (entailment vs. contradiction) between ctr is the problem at hand. the training set provided is identical to the training set used in previous tasks @xcite , however, the organizers have performed a variety of interventions on the test set and development set statements, either preserving or inverting the entailment relations. the technical details adopted figure 1 : a sample from the official webpage. given two trials and a section description, a model has to predict if there is entailment or contradiction with regard to the statement provided. to perform the interventions were not disclosed, to guarantee fair competition and in the interest of encouraging approaches that are robust and not simply designed to tackle these interventions. an example is shown in the figure 1 and is provided in the official task webpage available online foot_2 . even if it has already been proved that the transformers are not necessarily the best option for any text classification task @xcite , depending on the goal some strategies like domain-specific fine-tuning @xcite , or data augmentation @xcite @xcite ) can be beneficial for the considered task. the training and practice test sets were made available by the task organizers prior to the competition's official commencement. the gold labels were supplied for both sets. participants could build and test their models during the first phase, called the practice phase, by uploading their predictions to codalab 4 . the second step, known as the evaluation phase, began with the release of the unlabeled test set. @xcite , or, eventually, as in the case of the task 8, where a distilbert model is employed to detect machine-generated text @xcite . to develop our model, we also take advantage from a transformer architecture, creating a two-stage pipeline. in the first stage, we used a sentence transformer specifically trained on the medical domain. this is a python framework to create cuttingedge sentence, text, and image embeddings. the initial work is described in @xcite . more than 100 languages have sentences and text embeddings that can be computed using this method. sentences with a similar meaning can subsequently be found by comparing these embeddings, for example, using cosine-similarity. semantic search, paraphrase mining, and semantic textual similarity can all benefit from this. the framework offers a huge selection of pre-trained models suited for different tasks and is built on pytorch and transformers. moreover, fine-tuning models is also feasible. the model used as sentence transformer is t5large-medical, and it is available on hugging face foot_4 . the base model is t5 @xcite . specifically, sentences and paragraphs are mapped to a dense vector space of 768 dimensions. pytorch was used to convert the tensorflow model st5large-1 to this one. while the tfhub model and this pytorch model can provide somewhat different embeddings, they yield the same results when applied to the same benchmarks. the model was used to map all the words present in the text to the domain-specific embedding. following the embeddings of the primary section and the statement, the cosine similarity between the two was calculated. in the case of presence of a secondary section, the operation was also carried out between the secondary section and the statement. the cosine similarity between the two embedding vectors is calculated as shown in the equation 1 . in the first case, if the cosine similarity was greater than 0.5, the label of entailment was assigned, vice versa that of contradiction. in the second case, before calculating the cosine similarity, the average between the cosine similarity score between the two sections and the statement was calculated. our code is available online together with the predictions generated and sent in relation to the test set. as noted in the recent study by @xcite , the contribution of preprocessing for text classification tasks is generally not impactful when using transformers. more specifically, the best combination of preprocessing strategies does not provide relevant improvements compared to not performing any preprocessing when using transformers. for these reasons, and to keep our system faster and computationally light, we have not performed any preprocessing on the text. ## experimental setup we implemented our model on google colab foot_5 . the library we used is sentence transformer. the library requires python foot_6 (>= 3.8) and pytorch foot_7 (>=1.11.0). the dataset provided for all the phases are available on the official competition page. on the basis of our preliminary experiments, we found beneficial to set the threshold value for the cosine similarity equal to 0.5. we did perform additional fine-tuning on the t5 embedding. to run the experiment, a t4 gpu from google has been used. after the generation of the predictions, we exported the results on the json format required by the organizers. as already mentioned, all of our code is available on github. ## results for the task the official metric used were f1 (also known as balanced f-score or f-measure), faithfulness and consistency. the f1 score can be described as the harmonic mean of the precision and recall, with a maximum score of 1 and a minimum score of 0. recall and precision both contribute equally to the f1 score in terms of relative importance. equation 2 shows the formula for the f1 score. faithfulness is a measure of the extent to which a given system arrives at the correct prediction for the correct reason. intuitively, this is estimated by measuring the ability of a model to correctly change its predictions when exposed to a semanticaltering intervention. given n statements x i in the f1 faith const t5-large-medical 0.63 0.30 0.50 consistency is a measure of the extent to which a given system produces the same outputs for semantically equivalent problems. therefore, consistency is measured as the ability of a system to predict the same label for original statements and contrast statements for semantic preserving interventions. that is, even if the final prediction is incorrect, the representation of the semantic phenomena is consistent across the statements. given n statements x i in the contrast set (c), their respective original statements y i , and model predictions f() we compute consistency using equation 4 . in ## conclusion this paper presents the application of t5-large model embedding for addressing the task 2 at semeval-2024. for our submission we decided to follow an easy zero-shot learning approach, employing as-is, an in-domain pre-trained transformer. after getting the contextual embedding provided by the sentence transformer, we made use of a cosine similarity to calculate the similarity between sentences and generate the entailment/contradiction labels. the task is challenging, and there is still opportunity for improvement, as can be noted looking at the final ranking. possible alternative approaches include utilizing the zeroshot capabilities of models like gpt, increasing the size of the training set by using further data, or directly integrating ontology-based domain knowledge differently than what has been proposed in our work. to assess the effect of biomedical pretraining on mlms, performance consistency between sections, generalization capacity of models trained on nli4ct, performance comparability between numerical and biomedical cases and further error analysis is required. furthermore, given the interesting results recently provided on a plethora of tasks, also few-shot learning @xcite @xcite or data augmentation strategies @xcite @xcite could be employed to improve the performance. eventually, an optimal threshold learnt from the validation dataset could be also employed in future works, in place of the fixed one that we used in this study. compared to the best performing models, our simple approach exhibits some room for improvements. however, it is worth to notice that the proposed approach required no further pre-training and the computational cost to address the task is manageable with the free online resources offered by google colab.