FII SMART at S em E val 2023 Task7: Multi-evidence Natural Language Inference for Clinical Trial Data (2023)

Abstract: The “Multi-evidence Natural Language Inference forClinical Trial Data” task at SemEval 2023competition focuses on extracting essentialinformation on clinical trial data, by posing twosubtasks on textual entailment and evidence retrieval. In the context of SemEval, we present a comparisonbetween a method based on the BioBERT model anda CNN model. The task is based on a collection ofbreast cancer Clinical Trial Reports (CTRs),statements, explanations, and labels annotated bydomain expert annotators. We achieved F1 scores of0.69 for determining the inference relation(entailment vs contradiction) between CTR -statement pairs. The implementation of our system ismade available via Github - https://github.com/volosincu/FII_Smart__Semeval2023 .

## introduction nowadays, investigation of clinical natural language processing tasks play a major role in the advancement of medical research, especially for innovation in evidence-based medicine (ebm) @xcite . as reports of publications of clinical trial reports (ctrs) continue to amass at rapid pace, it has become infeasible for clinical practitioners to remain constantly updated with the literature in order to provide personalized care @xcite . based on recent studies @xcite , the legitimate research question is how we successfully connect the latest evidence to support personalized care? one direction is using textual entailment @xcite , also known as natural language inference (nli) @xcite , adapted for clinical texts. the relevant information from ctrs is structured 1 https://github.com/volosincu/fii_smart__semeval2023 around the patient's problem format (eriksen and frandsen 2018): patient, intervention, comparison and outcome (pico), formulation which makes clinical questions accessible and really helpful for further automated processing cause from human perspective the huge amount of data from randomized trials makes analysis endless and unavailing. often, clinical results can be contradictory, with indeterminate findings due to variations in parameters @xcite and creates an immense number of possible combinations between inclusion criteria, interventions, effects and outcomes, context in which the inference of evidence @xcite was seen as a risky or too challenging area to tackle. this perspective has known a great deal of progress in recent years due to developments of transformer models with fine-tuned variants like biobert (bidirectional encoder representations from transformers for biomedical text mining) and scibert (a bert model trained on scientific text) trained on medical corpora that facilitate rapid prototyping and experiments @xcite . the rest of the paper is organized as follows: section 2 briefly presents studies related to textual entailment in clinical text, section 3 provides information about the system designed to determine the inference relation (entailment vs. contradiction) between ctr -statement pairs, while section 4 describes the experimental setups. section 5 resumes the results of the conducted experiments, with their interpretations, followed by section 6 with the conclusions and further directions. ## background this topic has attracted large attention in recent years, evidenced by increasing number of scientific events (e.g., workshop on curative power of medical data (meda 2017; 2018; 2020) @xcite , workshop on on clinical natural language processing @xcite @xcite @xcite @xcite and competitions. competitions such as semeval-2023 task 7: multi-evidence natural language inference for clinical trial data are challenging, especially due to the data labelling problem, which is directly dependent on the automatic determination of the inference relation (entailment vs. contradiction) between ctr -statement pairs. different strategies are used in studies about clinical trial reports, from classical nlp techniques to transformer-based methods. the objective of this paper is to test if a transformer pretrained model on bio medical data can offer a competitive edge over the more classical approaches of nlp processing. a consequent hypothesis, derived from initial analysis of the dataset, was that employing a traditional solution consisting of a cnn (convolutional neural network) model would lead to lesser or inconsistent results. the dataset provided by organizers (mael et al. 2023) for this competition consists of 999 breast cancer ctrs in english 2 , with statements, explanations and labels annotated by domain expert annotators. each ctr 3 may contain 1-2 patient groups, called cohorts or arms. these groups may receive different treatments or have different baseline characteristics. for task 1, each training instance contains 1-2 ctrs, a statement, a section marker, and an entailment/contradiction label. the annotations collected from reports of randomized control trials (rcts) represent the raw material and the input consumed by the system. the task entries provided by organizers for dev and test datasets are following the ico intervention, comparison and outcomes format @xcite and may refer to the same study or to two different studies.. for ease of reference, a demonstrative and trivial example would be to infer entailment between sentences 2 https://clinicaltrials.gov/ct2/home chosen from a scenario like given a treatment a, a comparator b, and an outcome, the inference of entailment is to be established based on the statement and validity of the premises. there is an argument sustained by one or multiple premises with a "total of 2,400 statements split evenly across the different sections and classes" @xcite . the nli4ct instances from where the prompts have been extracted is a collection of clinical trial reports (ctrs) formatted following the pico standards consisting of: ## experimental setups both solutions have been developed and trained in google collab environment 67 . for the first experiment, using the transformer, in the medical domain we choose the biobert pretrained model as best fit for the task and the dataset. the transformer model input tensor can accept up to 500 tokens, but this raises a resources challenge at memory allocation. in order to be able to train the system in reasonable time (minutes), the input has been pre-processed. the solution was adapted so that the input is split into multiple instances, each with its own statement and premise pair, until we reached the parameter set for maximum sequence size for biobert input. as extra processing, in case the size of premises was greater than max size, the premises were split in two or more entries. this helped us to mitigate the resource problem, leaving some questions open, especially regarding the possibility of improving accuracy by optimizing for larger inputs for models. some pairs argument -premises have lost context and the contradiction or entailment label did not match on all tested instances, but at the same time, having a more concise context might have helped in other regards. this is one area that 6 transformer notebook needs further attention, especially in deciding if and how the data can be better organized. for hyperparameters, the experiments have been run with multiple values from different ranges, empirically obtaining best results with size 16 batch and 15 epochs. the practical sequence length for the biobert input was between 100 and 290 tokens, values at which we could train the model within the limits of ram maximum allocation (89gb). beyond that length the machine would go out of gpu memory and the model would crash. the token sequence is built dynamically at runtime depending on the maximum size we set (ex. 290, 100). for each statement we search the premises in the ctr file and then merge arguments and premises (using [cls] + [sep] bert). in the validation phase, after prediction, we iterate the results and search the entries that have been split and choose the final prediction based on majority voting. ## results for the purposes of benchmarking and familiarizing participants with the task's 7 cnn notebook challenges, the organizers provided a starting kit that uses a simple tf-idf algorithm. this baseline solution has achieved results of 48% f1 score, compared to our solution with the best result 69.9% f1 score for development data.