NCUEE - NLP at S em E val-2023 Task 7: Ensemble Biomedical L ink BERT Transformers in Multi-evidence Natural Language Inference for Clinical Trial Data (2023)

Abstract: This study describes the model design of the NCUEE-NLP system for the SemEval-2023 NLI4CT task that focuses on multi-evidence natural language inference for clinical trial data. We use the LinkBERT transformer in the biomedical domain (denoted as BioLinkBERT) as our main system architecture. First, a set of sentences in clinical trial reports is extracted as evidence for premise-statement inference. This identified evidence is then used to determine the inference relation (i.e., entailment or contradiction). Finally, a soft voting ensemble mechanism is applied to enhance the system performance. For Subtask 1 on textual entailment, our best submission had an F1-score of 0.7091, ranking sixth among all 30 participating teams. For Subtask 2 on evidence retrieval, our best result obtained an F1-score of 0.7940, ranking ninth of 19 submissions.

## introduction natural language inference (nli) seeks to determine whether a given hypothesis is true (i.e., entailment), false (contradiction), or undetermined (neutral) according to a known premise. methods used for nli task range from earlier symbolic and statistical approaches to recent deep-learningbased models @xcite . the bert-bilstm-attention model was proposed for medical text inference @xcite . a knowledge adaptive approach was derived to encode the premise/hypothesis text for improving medical nli @xcite . structured domain knowledge from the unified medical language system (umls) was incorporated into the esim model @xcite to improve nli performance in the medical domain @xcite . clinical domain knowledge of bert models @xcite was explored using lexical retrieval, syntactic retrieval, and classification models for external knowledge integration @xcite . the semeval-2023 task 7 focuses on multievidence natural language inference for clinical trial data (denoted as nli4ct) @xcite . the nli4ct task is mainly based on a collection of clinical trial reports (ctr) for breast cancer, with statements, explanations, and labels annotated by domain expert annotators. it includes two subtasks: 1) textual entailment: determining the inference relation (i.e., entailment or contradiction) between ctr-statement pairs; 2) evidence retrieval: extracting a set of supporting facts in a given ctr premise to justify the label predicted in subtask 1. the mediqa-2019 shared task covers an nli subtask in the medical domain @xcite , in which most systems built on the bert model are pre-trained on a large-scale open domain corpus and its variants focus on domain-specific knowledge such as scibert @xcite , biobert @xcite , and clinicalbert @xcite . given the promising results obtained by most such approaches, we apply bert-like neural networks to the nli task in the clinical domain. this paper describes the ncuee-nlp (national central university, dept. of electrical engineering, natural language processing lab) system for semeval-2023 nli4ct task. we use biomedical linkbert @xcite as our main system architecture to first extract a set of ## the ncuee-nlp system figure 1 shows our ncuee-nlp system architecture for the nli4ct task. our system is composed of two main parts: evidence retrieval for subtask 2 and textual entailment for subtask 1. both subtasks mainly depend on the linkbert transformer @xcite . linkbert is an improved version of bert that captures documents links such as hyperlinks and citation links to include knowledge that spans across multiple documents. in addition to bert pretrained on single documents, linkbert is pretrained by feeding linked documents into the same language model context. we use the biomedical linkbert (denoted as biolinbert) pretrained on pubmed with citation links for both subtasks. for evidence retrieval (subtask 2), a ctr premise is represented as a sequence of described sentences. each sentence and its corresponding statement were grouped to train the biolinkbert as a classification problem. if a sentence-statement pair is true (i.e., evidence), the class is assigned as 1, and 0 otherwise (non-evidence). to classify a sentence-statement pair during the test phase, we use the output probability of the [cls] token as an indicator for classification. the class with the highest probability will be regarded as the inference result. for subtask 1 on textual entailment, the identified evidence in the previous phase will be concatenated along with corresponding statements for label classification. if evidence sentences support the statement annotated with the entailment label, the class is assigned as 1, and 0 otherwise (contradiction). similarly, the class with highest probability of the [cls] token will be outputted as prediction result. in addition, a soft voting mechanism, which involves summing the predicted probabilities for class labels and predicting the class label with the largest sum probability, was used to integrate different fine-tuned versions of the pretrained biolinkbert transformer for better classification performance. ## conclusions this study describes the ncuee-nlp system in the semeval-2023 nli4ct task, including system design, implementation and evaluation. we integrate different fine-tuned versions of biolinkbert model to retrieve evidence in the ctr premise for textual entailment in a given statement. for subtask 1, our best submission obtained an f1-score of 0.7091, ranking sixth among all 30 participating teams. for subtask 2, our best result obtained an f1-score of 0.7940, ranking ninth of 19 submissions.