RGAT at S em E val-2024 Task 2: Biomedical Natural Language Inference using Graph Attention Network (2024)

Abstract: In this work, we (team RGAT) describe our approaches for the SemEval 2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials (NLI4CT). The objective of this task is multi-evidence natural language inference based on different sections of clinical trial reports. We have explored various approaches, (a) dependency tree of the input query as additional features in a Graph Attention Network (GAT) along with the token and parts-of-speech features, (b) sequence-to-sequence approach using various models and synthetic data and finally, (c) in-context learning using large language models (LLMs) like GPT-4. Amongs these three approaches the best result is obtained from the LLM with 0.76 F1-score (the highest being 0.78), 0.86 in faithfulness and 0.74 in consistence.

## introduction clinical trials are advanced treatments and tests to evaluate new ways of treating life-threatening diseases where interventions include new drugs, cells and other biological products, advanced surgical or radiological procedures and devices. as the trial progresses the observations are documented systematically in a clinical trial report that includes the subject selection criteria ('eligibility'), treatments ('interventions') and results at group level including adverse effects. these reports constitute a rich source of past endeavours to learn from and help in formulating new treatment plans. however, the sheer volume of ct reports 1 makes it impossible to conduct extensive manual evaluation. thus, it is necessary to have an automated pipeline that can enquire a ct report for specific hypothesis and provides high accuracy and reliability at the same time. 1 as of jan 17, 2024, clinicaltrials.gov lists 480,795 ct studies natural language inference or nli @xcite is one of the standard nlp tasks where a hypothesis is qualified as true (entailment) or false (contradiction) or even undetermined (neutral) given a premise. this task is adopted for reasoning over ct reports by @xcite where two new tasks are created based on nli4ct dataset, (1) nli over ct reports and (2) extracting the evidence/mention from ct reports to support the inference label. the semeval 2024 task 2 nli4ct is also based on the same nli4ct dataset (identical for training) with modifications in the test split (more details in the data section). the inferencing is challenging as it requires multi-hop reasoning, i.e., dependency and aggregation are required over different pieces of the document. other than the complexity associated with multihop reasoning, the domain and the associated worddistribution also creates significant challenge due to the presence of aliases, acronyms and biomedical terminologies @xcite @xcite . this results in significant drop in model performance as is evident in the nli results last year @xcite where it was found that majority of the submitted solutions failed to outperform the baseline solution with a significant margin. the challenge is also evident in the overall performance of models on general nli datasets (e.g., stanford nli or snli) where the best model results in 93.1% f1-score @xcite . when it comes to different modeling approaches, many of the top-performing models for the snli dataset are ensemble in nature. while initial individual models are based on rnn, most of the latest ones are based on the transformer architecture and pretrained language models like roberta or t5. similar trend can also be seen in @xcite where the best model is an ensemble and both deberta and flan-t5 made their way to the top. interestingly, llms like gpt3.5 could not make a significant boost in the performance. in our approach, we explored three different modeling paradigms, namely, (1) custom graph attention network (gat) based discriminative model with novel features based on the dependency tree of the input query, (2) generative models based on t5 and flan-t5 but enriched with synthetic data used for both pre-training and fine-tuning, and (3) llm like gpt-4 applied with and without few-shot examples. it is not surprising that the best performance was obtained by gpt-4 stressing on the importance of generic knowledge (that is embedded in these llms) rather than fine-tuning, especially when the dataset is not large enough. the organization of the paper is as follows. in the next section we provide a detailed literature survey on the techniques employed for nli. next, we present the details of the proposed approaches. subsequently, the model predictions and comparisons with other baseline methods are discussed. finally, conclusions are drawn and scope for future works is outlined. ## related work the existing body of work for the general nli is quite rich where they are based on the stanford nli (snli) dataset (550k examples but restricted to a single text genre) @xcite and three other nli datasets present in glue @xcite , namely, mnli, qnli and wnli. the mnli (multi-genre natural language inference corpus) dataset @xcite is a crowd-sourced nli dataset gathered from different sources, e.g., government reports (and covers different genres, e.g., fiction, travel). given a premise-hypothesis pair of sentences, the task is to predict one of the three classes, namely, whether the premise sentence entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). the qnli is modified from stanford question answer dataset @xcite where the task is to determine whether the context sentence contains the answer to the question. similarly, the wnli dataset is created from the winograd schema challenge @xcite where a coreference resolution problem is converted into an entailment problem involving a pronoun and its referent. another large nli dataset is multi-genre nli (mnli) that has 433k examples covering multiple genres and supporting cross-genre evaluation. some of the best performances are obtained by roberta @xcite , xlnet @xcite , multi-task deep neural network (mt-dnn) @xcite and generative pre-training (gpt) approach @xcite . there are few nli datasets in the biomedical domain, namely, mednli @xcite and bionli @xcite ). mednli has 14k example pairs created by clinicians on 4,683 premises with three categories, entailment, contradiction and neutral. bionli, on the other hand, goes beyond sentence-level inference and includes large context as premises that requires handling complex texts as well as domain knowledge. bastan et al. also includes negative examples as adversarial hypothesis using nine strategies which is a speciality of this dataset. there are three biomedical domain specific models that are typically used on these datasets. starting with the available weights of bert (pretrained on general domain corpora), biobert @xcite ) is trained on pubmed abstracts and pmc full-text articles and shown to outperform bert on ner, relation extraction and q&a, all in the biomedical domain. pubmedbert @xcite is a bert model created from scratch (rather than starting with general domain corpora) on large biomedical domain dataset like pubmed and achieved impressive performance for tasks like ner and q&a. biolinkbert @xcite further exploited links between pubmed documents to create a richer context that is used to build a language model (lm). this model has obtained sota performance on biomedical datasets such as blurb @xcite and bioasq @xcite . another model that achieved sota performance on mednli is scifive @xcite which is based on t5 paradigm. there are not many studies on the application of graph neural network for nli. inspired by kim @xcite where external knowledge is infused for nli task, @xcite developed a joint training model where graph attention network (gat) is used to represent the sub-graph associated with entities that are involved in the hypothesis. another closely related gat application is from @xcite applied for fact verification on wikipedia articles. typical applications of gat in the nlp domain are for question answering, semantic parsing, information extraction and named entity recognition @xcite the dataset for multi-evidence nli for clinical trial (nli4ct) is based on a collection of breastcancer ct reports foot_0 containing statements, explanations and labels annotated by domain expert annotators @xcite . each ct report has four sections: (a) eligibility criteria (a set of conditions for patients to be included in the trial cohort), (b) intervention (information regarding the details of treatments administered), (c) results (what is the outcome of these treatments) and (d) adverse events (if anything was observed during the period of the trial). the annotated statements (hypothesis) are claims extracted from one of the four sections (with an average length of 19.5 tokens) and may even compare more than one report. each statement is qualified as either 'contradiction' or 'entailment'. there are 1700 examples in the training set and 200 in the development/validation set with exactly 50:50 split of the two classes. the test set has 5500 examples with unknown label distribution. a typical example looks like the following: ## methodology we have explored three different modeling strategies for the prediction of the inference label. they are (1) custom discriminative model with gat applied to create features from the dependency tree of the hypothesis statement, (2) sequence-to-sequence generative models based on t5 and flan-t5 but enriched with synthetic data used in both pre-training and fine-tuning and (3) llm based solution with and without few-shot examples. ## results in this section, first we describe the performance of the custom discriminative model followed by the performance of the fine-tuned t5 model and finally the results from gpt-4. although we compute precision, recall and f1-score for all our experiments we report only f1-score here. it is to be noted that we did not evaluate our model on the test dataset for all our experiments and submitted test results only for the best validation performance. thus, for most of our experiments we report only the validation f1-score and also mention the test f1-score wherever available. ## conclusion in this work we have explored both discriminative and generative models for nli applied to ct reports. while our custom discriminative model outperforms generative models like t5-base and flan-t5-base the same is not true when evaluated on the test dataset indicating the limitation of the small validation dataset and significant change in data distribution. since the training dataset is small (1700) we also explore enriching the same with synthetic data created by llms like gpt-4 for additional task (e.g., q&a) and the same nli task. however, the addition of these synthetic data substantially degrades the performance rather than improving pointing to a deeper analysis of the role of synthetic data for nli task. the only exception is in the pretraining synthetic data created for small flan-t5 model that boosted the final performance. the best result is obtained by gpt-4 without using few-shot examples and we suspect both the addition of examples and modification of the prompt can further improve the performance.