Clemson NLP at S em E val-2023 Task 7: Applying G ator T ron to Multi-Evidence Clinical NLI (2023)

Abstract: This paper presents our system descriptions for SemEval 2023-Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data sub-tasks one and two. Provided with a collection of Clinical Trial Reports (CTRs) and corresponding expert-annotated claim statements, sub-task one involves determining an inferential relationship between the statement and CTR premise: contradiction or entailment. Sub-task two involves retrieving evidence from the CTR which is necessary to determine the entailment in sub-task one. For sub-task two we employ a recent transformer-based language model pretrained on biomedical literature, which we domain-adapt on a set of clinical trial reports. For sub-task one, we take an ensemble approach in which we leverage the evidence retrieval model from sub-task two to extract relevant sections, which are then passed to a second model of equivalent architecture to determine entailment. Our system achieves a ranking of seventh on sub-task one with an F1-score of 0.705 and sixth on sub-task two with an F1-score of 0.806. In addition, we find that the high rate of success of language models on this dataset may be partially attributable to the existence of annotation artifacts.

## introduction the proliferation of clinical trial reports (ctrs) has made it challenging for medical practitioners to stay up-to-date on the latest literature, inhibiting their ability provide evidence-based patient care. the computerized clinical decision support system (cdss) has aimed to address this problem by providing an interface, typically within the electronic health record system, with which a clinical practitioner can, for example, access automated patient care recommendations or view diagnostic suggestions @xcite . research suggests that these systems can have a positive impact on both clinical practitioners and patients @xcite ; they depend, however, on the ability to interpret and retrieve information from biomedical literature @xcite . in this context, task 7 provides a chance to augment existing approaches for retrieval and interpretation of unstructured biomedical text. task 7 is split into two sub-tasks which are structured around a dataset of english-language breast cancer ctrs: sub-task 1: textual entailment and sub-task 2: evidence retrieval. sub-task one involves identifying the entailment relation (entailed or contradicted) between a statement-a claim about the ctr(s) annotated by a domain-expert-and a premise in the form of a section from the ctr(s). sub-task 2 involves extracting sections of text from the ctr section which are necessary to either entail or contradict the statement. these tasks prove challenging by requiring modeling of long-term dependencies, comparisons between multiple ctrs, and complex numerical reasoning. in this paper, we outline our approach to the multi-evidence natural language inference for clinical trial data shared task and examine our results. for both sub-tasks, our system uses gatortron, a recently introduced transformerbased clinical language model, as a foundation model due its demonstrated success at medical natural language inference tasks @xcite as well as to explore the impact of increased model parameters on a down-stream clinical task. to overcome the distribution shift between the pretraining domain and the target domain, we domain adapt the model on a collection of clinical trial reports similar to the task dataset @xcite . for task 1-textual entailment-we improve base performance by restricting the ctrs to text necessary to determining entailment, as identified by our model for sub-task 2. for sub-task 2, we framed the problem as a binary classification task by splitting each ctr premise into individual lines. our systems placed seventh on sub-task one with an f1-score of 0.705 and sixth on sub-task two with an f1-score of 0.806. additionally, we explore the existence of annotation artifacts @xcite in the dataset, finding that our system for sub-task 1 performs well above the baseline when predicting on statements only (f1-score 0.584). ## dataset the organizers of task 7 provide a dataset of 1,000 english-language, breast cancer clinical trial reports extracted from https://clinicaltrials. gov/ct2/home along with statements, explanations and labels annotated by clinical domain experts from the cancer research uk manchester and the digital experimental cancer medicine team @xcite . each ctr has been pre-processed to include four sections: eligibility criteria, intervention, results, and adverse events. each of these has been further subdivided into individual lines consistent with the structure of the ctr record. each data sample includes: ## conclusion in this paper we outlined our system for semeval 2023 task 7: multi-evidence natural language inference for clinical trial data. we compare the effect of different training approaches on the performance of a large pretrained language model for the two sub-tasks. we find that while increased model size tends to lead to better performance, increased model size is not enough on its own to achieve strong performance on this dataset. we further demonstrate that continued pretraining on data in the target domain leads to a marked performance increase on the test set. due to the potential confounding effect of annotation artifacts, investigation into adversarial approaches for data augmentation on this dataset may be useful in the future @xcite . furthermore, enhancing the numerical reasoning capabilities of language models @xcite could benefit system performance on this dataset and warrants further research.