S em E val-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data (2023)

Abstract: This paper describes the results of SemEval 2023 task 7 – Multi-Evidence Natural Language Inference for Clinical Trial Data (NLI4CT) – consisting of 2 tasks, a Natural Language Inference (NLI) task, and an evidence selection task on clinical trial data. The proposed challenges require multi-hop biomedical and numerical reasoning, which are of significant importance to the development of systems capable of large-scale interpretation and retrieval of medical evidence, to provide personalized evidence-based care. Task 1, the entailment task, received 643 submissions from 40 participants, and Task 2, the evidence selection task, received 364 submissions from 23 participants. The tasks are challenging, with the majority of submitted systems failing to significantly outperform the majority class baseline on the entailment task, and we observe significantly better performance on the evidence selection task than on the entailment task. Increasing the number of model parameters leads to a direct increase in performance, far more significant than the effect of biomedical pre-training. Future works could explore the limitations of large models for generalization and numerical inference, and investigate methods to augment clinical datasets to allow for more rigorous testing and to facilitate fine-tuning. We envisage that the dataset, models, and results of this task will be useful to the biomedical NLI and evidence retrieval communities. The dataset, competition leaderboard, and website are publicly available.

## introduction clinical trials are indispensable for experimental medicine as they test the efficacy and safety 1 https://github.com/ai-systems/nli4ct 2 https://codalab.lisn.upsaclay.fr/ competitions/8937#learn_the_details 3 https://sites.google.com/view/nli4ct/ of novel treatments @xcite . clinical trial reports (ctrs) are documents that detail the methodology and results of a trial, implemented to guide personalized and targeted interventions for patients. however, there are 400,000+ published ctrs, with an increasing number being published every year @xcite , making it impractical to manually carry out comprehensive evaluations of all the relevant literature when designing new treatment protocols @xcite . to address this challenge, natural language inference (nli) @xcite offers a potential solution for the large-scale interpretation and retrieval of medical evidence, to support a higher level of precision and efficiency in personalized evidence-based care @xcite . semeval-2023 task 7 -multi-evidence natural language inference for clinical trial data (nli4ct) -is based on the nli4ct dataset which contains 2 tasks on breast cancer ctrs, shown in figure 1 . firstly to determine the inference rela-tion between a natural language statement, and a ctr. secondly, to retrieve supporting facts from the ctr(s) to justify the predicted relation. the inference task requires multi-hop reasoning, that is the ability to combine information from multiple pieces of text to draw inferences @xcite . previous works have shown that although multi-hop reasoning can be implemented on large-scale scientific tasks, there is a significant drop-off in performance as the number of necessary hops increases @xcite @xcite . a large proportion of the nli4ct dataset instances require the construction of inference chains in this drop-off range. additionally, numerical and quantitive reasoning is required to perform inference on nli4ct, exemplified in figure 1 . studies have shown that transformer-based models are unable to consistently apply this type of reasoning, instead relying on shallow heuristics for predictions @xcite @xcite . in the nli4ct inference task, both the multihop and the numerical reasoning have the added complexity of being applied to ctrs. studies have demonstrated that the word distribution shift from general domain corpora to biomedical corpora, such as ctrs, caused by the increased prevalence of aliases, acronyms, and biomedical terminology represents a significant detriment to model performance @xcite @xcite @xcite @xcite @xcite . this word distribution shift challenge is also present in the evidence selection task. although the evidence selection task is arguably simpler than the inference task its importance cannot be understated. state-of-the-art nli models consistently struggle to attend to relevant pieces of evidence when applied to large texts @xcite . additionally, the ability to filter out irrelevant pieces of text reduces the likelihood of distractors @xcite and reduces the length of the input for inference, improving efficiency. this paper introduces semeval-2023 task 7 -multi-evidence natural language inference for clinical trial data (nli4ct) -for biomedical nli and evidence extraction and presents a detailed analysis of the performance of the participating systems. we report the following conclusions; ## related works there are many existing expert-annotated resources for clinical nli. the trec 2021 clinical track @xcite is a large-scale information retrieval task to match patient descriptions to clinical trials for which they are eligible. evidence inference 2.0 (deyoung et al., 2020) is a question-answering (qa) task and span selection task, where provided with an outcome, an intervention, and a comparator intervention, systems must infer if the intervention resulted in a significant increase, significant decrease, or produced no significant difference in the outcome measurement, compared to the comparator, and identify spans that support this inference. the mednli @xcite dataset is an entailment task to infer the entailment relation between a short piece of text extracted from medical history notes, and an annotated statement. none of the aforementioned tasks encompass the full complexity of nlp over ctrs, that this the capability to reason over all sections ctrs and to simultaneously carry out biomedical and numerical inference. instead choosing to focus on one specific ctr section. additionally, these tasks often have repetitive inference chains, i.e. matching statements for eligibility, or retrieving measurements and comparing them. in contrast, nli4ct instances cover all ctr sections and contain minimal repetition in inference chains, as there is no set template for statements. currently, large language models (llm) achieve the best results for clinical nli @xcite . however, they suffer from a plethora of issues. llms demonstrate poor performance on quantitative reasoning and numerical operations within nli @xcite . additionally, there is a notable drop in performance for llms pretrained on general domain data when applied to biomedical tasks @xcite , partially aggravated by a lack of well-annotated clinical data @xcite . nli4ct is designed to assist in the development and benchmarking of models for clinical nli. task 1 classify the inference relation between a ctr premise and a statement, as either an entailment or a contradiction, as shown in figure 1 . task 2 output a subset of facts from the ctr premise, necessary to justify the class predicted in task 1. ## dataset the premises in nli4ct are obtained from 1000 publicly available english language breast cancer ctrs published on clinicaltrials.gov. this data is maintained by the u.s. national library of medicine and is subject to the hipaa privacy rule. the ctrs are split into 4 sections: a group of domain experts, including clinical trial organizers from a major cancer research center, took part in the annotation task. the annotators were given two ctr premises to generate an entailment statement. this is a short text that makes an objectively true claim about the contents of the premise. annotators could choose to write a statement about one, or both premises. non-trivial statements typically involve summarization, comparison, negation, relation, inclusion, superlatives, aggregation, or rephrasing, and require understanding multiple rows of the premise. then the annotators select a subset of facts from the premise(s) that support the claims in the statement. then a negative rewriting technique @xcite was applied, modifying the previously produced entailment statement to contain objectively false claims while retaining the original sentence structure and length. this technique is used to reduce the likelihood of stylistic or linguistic patterns pertaining to either entailment or contradictory statements. annotators then extract a subset of facts from the premise that contradict the claims in the false statement, the resulting dataset includes 2400 annotated statements with labels, premises, and evidence. the dataset was split 70/20/10 train/test/dev. the two classes and four sections are evenly distributed throughout the dataset and its splits. ## evaluation the same strategy is adopted for the evaluation of the results of both tasks. task 1, the textual entailment task, is a binary classification task, so performance is measured using precision, recall, and macro f1-score, comparing predicted labels against the gold labels. we also frame task 2, the evidence selection task, as a binary classification task, classifying each fact in the premise as either relevant evidence, or irrelevant, we compare the predicted labels against the gold labels and compute the precision, recall, and macro f1-score. ## architectural paradigms we observe 5 different categories of approaches described in the system papers, recorded in generative language models are designed to learn the joint probability distribution of p(x,y), where x is the input text, such as the statements or ctr premises, and y is a probability output by a classification layer or a generated label from a decode-only transformer. conversely, discriminative language models encode the conditional probability p(y|x), designed to encode the decision boundary between different classes. biomedical pre-training refers to the technique of training a model on a large, unlabeled biomedical dataset, such as scientific articles or patient health records. this is used to encode general features and patterns within a domain, before fine-tuning a specific task. semantic rule-based models perform inference based on a set of human-defined asserted facts or axioms. ontologies capture the categories, properties, and relations between the concepts of a particular domain. ontology-based models extract entities from the input text and map them to nodes within the ontology to enrich the inputs with domain knowledge. ## results and discussion during the 21-day evaluation period (january 10 th -31 st , 2023), 40 participants submitted a total of 643 submissions for the entailment task, and 23 participants submitted a total of 364 submissions for the evidence selection task. in total 21 participants submit system papers. submissions for which a system paper was not provided are omitted from the tables and discussion. the majority of systems fail to significantly outperform the majority-class baseline on the entailment task. the top-performing systems achieve significant gain across both tasks. ## statistical artifacts statistical characteristics such as imbalanced sequence lengths, token distributions, or discriminative conditions that are disproportionately associated with a particular class can superficially inflate model performance @xcite . @xcite @clemson nlp observes that systems are able to outperform the random baseline on the entailment task using only the statements, this indicates that systems are able to exclusively rely on the presence of superficial statistical patterns within the collection of statements, without learning the underlying rules of the tasks, reporting an f1 of 0.584. this is significantly below the majority baseline (0.66 f1), and as the entailment task is a binary classification task we conclude that the effects of these artifacts on the submitted results are very minimal. alameldin and williamson (2023) @clemson nlp identifies minor differences in statement lengths across classes, however, in our analysis we did not find a significant difference. additionally, we retrieve the 15 most frequently used tokens in the statements for both classes, although we observed some uneven distributions in the training set, these distributions were not present or even inversely correlated in the test set. therefore, we do not believe either of these characteristics explains the observed results, and claim that nli4ct is robust to statistical biases. ## conclusion this paper presents the systems and results submitted to the semeval-2023 task 7 on the nli4ct dataset. the tasks are challenging, with the majority of submitted systems failing to significantly outperform the majority class baseline on the entailment task. potentially due to the requirement for sophisticated numerical reasoning, an elevated frequency of biomedical expressions, or the relatively small training set. we observe significantly better performance on the evidence selection task than on the entailment task, and we find that there is no consistent correlation between task performances. the impact of biomedical pre-training is significantly less profound than expected, far outweighed by the effects of increased model size. there is a direct correlation between model size and task performance, with mlms achieving the highest results in both tasks. there remains room for improvement on both tasks, potentially by exploiting data augmentation to increase training set size, leveraging the zero-shot capabilities of models such as gpt and t5, or through the direct integration of domain knowledge from ontologies. a further error analysis is necessary to evaluate the impact of biomedical pre-training on mlms, consistency of performance across sections, generalization ability of models trained on nli4ct, and comparison of performance on numerical versus biomedical instances.