B f3 R at S em E val-2023 Task 7: a text similarity model for textual entailment and evidence retrieval in clinical trials and animal studies (2023)

Abstract: We describe our participation on the Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) of SemEval’23. The organizers provided a collection of clinical trials as training data and a set of statements, which can be related to either a single trial or to a comparison of two trials. The task consisted of two sub-tasks: (i) textual entailment (Task 1) for predicting whether the statement is supported (Entailment) or not (Contradiction) by the corresponding trial(s); and (ii) evidence retrieval (Task 2) for selecting the evidences (sentences in the trials) that support the decision made for Task 1. We built a model based on a sentence-based BERT similarity model which was pre-trained on ClinicalBERT embeddings. Our best results on the official test sets were f-scores of 0.64 and 0.67 for Tasks 1 and 2, respectively.

## introduction the increase on the number of digital resources in biomedicine requires more adequate text mining methods to allow experts to stay up-to-date with the recent findings @xcite . task 7 of the semeval'23 @xcite , namely, multi-evidence natural language inference for clinical trial data (nli4ct) foot_0 , aimed at automatic processing clinical trials for two particular sub-tasks. in the scope of the shared task, the organizers released a collection of clinical trials, each of them composed of four sections, namely, "intervention", "eligibility", "results", and "adverse events". each of these sections contains one or more sentences. the shared task was based on a set of the socalled "premise-statement-evidence" items, which could be related to one or two clinical trials. if related to only one trial, the item was set as "single", otherwise it is a comparison between two trials, i.e., set as "comparison". each premise-statement-evidence item refers to only one particular section of the trial and contained a statement that made a claim about one or two trials, and which was the main input to be considered in the tasks. we present two examples in the "text entailment" task consisted of automatically classifying the relation between the statement and the trial. two values of labels were possible (cf. the "evidence retrieval" task consisted of finding evidences that support the claim. the evidences can be one or more sentences from the corresponding section of the clinical trial, as stated in the premise-statement-evidence item. in the case of a comparison, the evidences should come from both clinical trials. examples of one evidence for the two tasks are shown in type section statement task 1 task 2 single results "the primary trial does not report the pfs or objective response rate of its patient cohort" entailment "outcome measurement: local control using ipsilateral breast tumor recurrence rates, time frame: 2 years after treatment completion" comparison adverse events "a significant number of the participants in the secondary trial and the primary trial suffered from enterocolitis" contradiction "enterocolitis 1/167 (0.60%), enterocolitis 0/167 (0.00%)" (trial 1) and "febrile neutropenia 8/458 (1.75%)", neutropenia 6/458 (1.31%), etc." (trial 2) ## methods in this section we describe the details of our approach, including training based on a sentence bert model, and the pros-processing for deriving the labels for tasks 1 and 2. sentence-bert. we approached the problem as a text similarity task and relied on sentence-bert (sbert) foot_1 @xcite . it is a modification of bert that relies on siamese and triplet networks, thus being more suitable for text similarity tasks. our classifier utilizes the clin-icalbert embeddings foot_2 @xcite , which is based on electronic health record from the mimic iii database, and softmax as loss function. due to time constraints, we only trained using two values of epochs: 1 and 2. datasets. we present the number of entries that is available for each trial type (single or comparison) and label (entailment or contradiction) in ## results we present the results that we obtained the test phase of the shared task, as well as for our additional experiments with documents about animal studies. ## conclusions we presented our participation in the clinical trial data (nli4ct) of semeval'23. we relied on a sentence-based bert similarity model and utilized a pre-trained domain-specific language model, i.e., a clinicalbert embeddings. our best results for task 1 and 2 were f-scores of 0.64 and 0.67, respectively. being task 1 a classification task with only two labels, either "entailment" or "contradiction", our results cannot be considered very significant. however, task 2 is a much harder task since it consisted of selecting the evidences that supported a certain decision, and we were able to obtain a higher score for this task. our preliminary evaluation on animal studies shows that the trained models could potentially be used for this task as well.