FMI - SU at S em E val-2023 Task 7: Two-level Entailment Classification of Clinical Trials Enhanced by Contextual Data Augmentation (2023)

Abstract: The paper presents an approach for solving SemEval 2023 Task 7 - identifying the inference relation in a clinical trials dataset. The system has two levels for retrieving relevant clinical trial evidence for a statement and then classifying the inference relation based on the relevant sentences. In the first level, the system classifies the evidence-statement pairs as relevant or not using a BERT-based classifier and contextual data augmentation (subtask 2). Using the relevant parts of the clinical trial from the first level, the system uses an additional BERT-based classifier to determine whether the relation is entailment or contradiction (subtask 1). In both levels, the contextual data augmentation is showing a significant improvement in the F1 score on the test set of 3.7% for subtask 2 and 7.6% for subtask 1, achieving final F1 scores of 82.7% for subtask 2 and 64.4% for subtask 1.

## introduction a huge amount of clinical trial reports are generated annually, and the ability to automatically extract and analyze data from these reports can help healthcare professionals stay abreast of the latest trends and findings from these trials. semeval 2023 task 7: multi-evidence natural language inference for clinical trial data consists of a dataset of breast cancer clinical trial reports and related statements in english @xcite . the goal is to determine the trial-statement inference relationship (subtask 1) and to extract supporting sentences from the trial text (subtask 2). the proposed system reverses the task order and redefines subtask 2 as identifying the sentences from the clinical trial text relevant to the statement, similar to question-answering natural language inference (qnli) @xcite . after finding these sentences, they are used as input for classifying the inference relation -entailment or contra-diction of the statement with respect to the premise (subtask 1). we evaluated various transformer models based on the original bert architecture @xcite and pre-trained on biomedical/clinical data and a contextual data augmentation approach that improves the model's ability to discriminate related sentences and classify them correctly. the system showed 82.7% f1 score on the test set and ranked fifth in the leaderboard for subtask 2, 2.6% behind the first-place model. subtask 1 is more challenging and the system showed 64.4% f1 score on the test set there. the system struggles the most when the statement requires performing quantitative reasoning to determine the relationship. the code related to this task is available on github foot_0 . ## background the task uses a dataset of breast cancer clinical trial reports, statements, inference relations, and supporting sentences annotated by domain experts. it is separated into two sub-tasks -textual entailment and evidence retrieval @xcite . each clinical trial record consists of 4 standard sections: eligibility, intervention, results, and adverse events. and each section in the dataset consists of multiple lines of text with information about the section. each statement is a section-specific claim and may be related to one or two clinical trials (primary/secondary). the statement type is single or a comparison depending on whether it refers to one or two clinical trials. each statement has a label specifying the inference relation -entailment or contradiction -that can be inferred from the clinical trial section. the dataset is split into 3 partstrain/dev/test and the labels are evenly distributed between the two classes for each section in the train and dev sets. the majority of statements are of type single and comparisons are fewer -approximately 39% of the train set and 30% of the dev set. we have participated in both subtasks: transformer models have been applied to a wide variety of nlp tasks, including natural language inference. in particular, bert and bert-based models have been successfully applied to multiple natural language inference (nli) datasets and are showing state-of-the-art (sota) results on two of the glue @xcite benchmark datasets qnli corpus @xcite ) -albert @xcite , and wnli corpus -deberta @xcite . bert-based models have a significantly smaller number of parameters than models like t5-11b @xcite and palm 540b @xcite which are showing the best results for multi-genre natural language inference (multinli), recognizing textual entailment (rte) @xcite and com-mitmentbank (de marneffe et al., 2019) datasets foot_1 . therefore, bert-based models can still be useful for nli tasks and are not as computationally expensive. ## system overview in order to determine the inference relationship between premise-sentence pairs of the clinical trial report (ctr), we first identify the parts of a section that contain information related to the statement. ## experimental setup we perform separate experiments for subtask 2 and subtask 1. for both tasks, we use the train dataset to train different models, evaluate them on the dev set and select the best-performing model. since subtask 1 uses the output from subtask 2, we select the best-performing model for subtask 2 and feed the output for evaluating subtask 1 models. we optimized subtask 1 hyperparameters using tune @xcite . ## conclusion the proposed two-level system for retrieving relevant clinical trial evidence and classifying entailment shows competitive results for subtask 2 with 82.7% f1 score but could be improved in handling quantitative reasoning. the described approach for augmenting with contextual data improves the performance for both subtasks. as further work, the