TH i FLY Research at S em E val-2023 Task 7: A Multi-granularity System for CTR -based Textual Entailment and Evidence Retrieval (2023)

Abstract: The NLI4CT task aims to entail hypotheses based on Clinical Trial Reports (CTRs) and retrieve the corresponding evidence supporting the justification. This task poses a significant challenge, as verifying hypotheses in the NLI4CT task requires the integration of multiple pieces of evidence from one or two CTR(s) and the application of diverse levels of reasoning, including textual and numerical. To address these problems, we present a multi-granularity system for CTR-based textual entailment and evidence retrieval in this paper. Specifically, we construct a Multi-granularity Inference Network (MGNet) that exploits sentence-level and token-level encoding to handle both textual entailment and evidence retrieval tasks. Moreover, we enhance the numerical inference capability of the system by leveraging a T5-based model, SciFive, which is pre-trained on the medical corpus. Model ensembling and a joint inference method are further utilized in the system to increase the stability and consistency of inference. The system achieves f1-scores of 0.856 and 0.853 on textual entailment and evidence retrieval tasks, resulting in the best performance on both subtasks. The experimental results corroborate the effectiveness of our proposed method.

## introduction in recent years, with the fast development of digital health, there has been a surge in the publication of clinical trial reports (ctrs). currently, there are over 10,000 ctrs for breast cancer alone 1 . this proliferation of ctrs has enabled the construction of a natural language inference (nli) system that can aid medical interpretation and evidence retrieval for personalized evidence-based care. the *: equal contribution 1 https://sites.google.com/view/nli4ct/ home nli4ct task @xcite focuses on constructing an explainable multi-evidence nli system based on ctrs for breast cancer. it consists of two sub-tasks: textual entailment (task a) and evidence retrieval (task b). task a requires the system to determine if the ctrs entail or contradict the given hypotheses. task b requires the system to find evidence in the ctrs necessary to justify the prediction in task a. the primary difficulty inherent in nli4ct is that the process of hypothesis verification frequently involves the integration of several pieces of evidence from the premise. in certain cases, justifying a hypothesis requires a comparison of two separate premise ctrs. moreover, unlike other clinical nli benchmarks (e.g., mednli), nli4ct presents a unique feature containing four distinct types of premises -intervention, eligibility, results, and adverse events -that align with the four sections typically found in each ctr. validating hypotheses based on each premise type requires different levels of inference skills (textual, numerical, etc.). fig. 1 presents an example of nli4ct. the first hypothesis involves summating the numbers of patients who experienced diarrhoea in two distinct cohorts to conduct the verification. alternatively, verifying the second hypothesis requires evaluating treatment modality against the criteria outlined in the exclusion criteria. this paper proposes a multi-granularity inference system for the nli4ct task to address the aforementioned challenges. an overview of the proposed system is presented in fig. 2 . the system comprises a multi-granularity inference network that performs joint semantics encoding of the hypothesis and premise, followed by multigranularity inference through sentence-level and token-level encoding of the encoded sequence to accomplish task a and task b. to handle the various levels of inference involved, we utilize the scifive model @xcite to enhance the system's performance on hypotheses requiring numerical inference skills. our system is presented as an ensemble of models to increase the stability and consistency of inference. a joint inference module is further proposed to refine the inference outcomes of hypotheses by detecting semantic consistency between hypothesis pairs sharing the same premise. our system achieves micro f1-scores of 0.856 and 0.853 on task a and task b, respectively. the main contributions of this work are as follows: ## related work biolinkbert @xcite is a pre-trained model that utilizes hyperlinks within documents. the model incorporates linked documents into the input of pre-training and is pre-trained using masked language modeling and document relation prediction tasks. biolinkbert has demonstrated state-of-the-art performance on various medical nlp benchmarks, such as blurb @xcite and bioasq @xcite . scifive scifive @xcite is a t5based model pre-trained on large biomedical corpora, namely pubmed abstract and pubmed central. this model has shown the potential of textgenerating models in medical nlp tasks and has achieved state-of-the-art performance on mednli @xcite and ddi @xcite . ## task formulation the multi-evidence natural language inference for clinical trial data (nli4ct) task comprises two sub-tasks: textual entailment and evidence retrieval. both sub-tasks share a common input comprising a premise p and a hypothesis s. the hypothesis is a single sentence, while the premise comprises several sentences in the ctr(s), i.e., @xmath0, p foot_0 , • • • , p m ]. the textual entailment sub-task can be viewed as a binary classification problem, with the entailment module f ent generating a predicted result ŷ ∈ {0, 1} n for all input hypotheses: f ent (s, p ) = ŷ. in this context, the predicted value ŷ = 1 indicates that the premise entails the hypothesis, while ŷ = 0 denotes that the hypothesis contradicts the premise. on the other hand, the evidence retrieval sub-task can be seen as a binary classification problem targeting each sentence in the premise. the evidence retrieval module f ret produces a predicted result @xmath1, 1} m : f ret (s, p ) = r, where @xmath2, @xmath3. ## models in this section, we introduce models applied in our inference system: multi-granularity inference network (mgnet), scifive @xcite , and the joint inference network. ## system details model setup we initialized the joint semantics encoder in mgnet and the joint inference network with biolinkbert-large-mnli-snli, a language model proposed by @xcite that is pre-trained on the pubmed dataset and further finetuned on the multi-genre natural language inference (mnli) @xcite and stanford natural language inference (snli) datasets @xcite . following the setting of biolinkbert-large, we set the number of layers l and the dimension of hidden states d to 24 and 1024, respectively. we apply adam (kingma and ba, 2015) and adafactor optimizers @xcite for finetuning mgnet and scifive, respectively. to provide a comprehensive description of the inference models used in our system, we first outline the loss functions employed in model training: here, @xmath4, 1} represents the label of task a. here, λ is a hyperparameter that controls the pace of learning on both tasks. (4) contrastive learning loss: we use the contrastive learning loss proposed by @xcite , which is a combination of the ce loss and the supervised contrastive learning (scl) loss: here, l scl is the scl loss that takes the global representation hs 1 , hs 2 , • • • , hs n and the labels y 1 , y 2 , • • • , y n in the batch as input (n is the batch size). since our dataset is relatively small, we apply the scl loss as a regularization technique during training. the inference models applied in the proposed system are listed in table .1. to extend the maximum input length of the original biolinkbert model from 512 to 1024, we append the positional embeddings of biolinkbert with randomly initialized parameters. in task a, we set the number of layers l s for the transformer in the sentence-level encoder as one and the learning rates of the joint semantics encoder and other parts as 2e-5 and 1e-4, respectively. the batch size, number of epochs, and warmup ratio are set to 32, 100, and 0.3, respectively. for scifive, we set the learning rate, batch size, number of epochs, and warmup steps as 3e-5, 32, 100, and 500, respectively. the hyperparameter values λ = 0.01 and γ = 0.5 are chosen, and the temperature of the supervised contrastive learning (scl) loss is set to 0.3. in task b, we set the learning rate, l s , batch size, number of epochs, and warmup ratio as 5e-6, 2, 1, 50, and 0.05, respectively. in the joint inference network, the learning rate, batch size, number of epochs, and warmup ratio are set to 2e-5, 32, 100, and 0.3, respectively. positive samples, i.e., hypothesis pairs with the same label, are created using back translation, while negative samples are obtained by identifying hypothesis pairs with different labels that appear in the datasets. further training details of this network are presented in section c in the appendix, owing to space limitations. soft ensembling we apply a cross-validationbased soft ensembling method to summarize the inference results from different models. specifically, we perform 10-fold validation for each model in table .1 using all data in the train and dev sets. for task a, we save the checkpoint that achieves the best f1-score on the corresponding dev set for each fold, while for task b, we save two of the best checkpoints for each fold. then, we average the predictions of 4 × 10 = 40 models trained by crossvalidation for task a and average the predictions of 3 × 10 × 2 + 3 = 63 models trained by crossvalidation (two checkpoints for each fold) and holdout method (one checkpoint for each model) for soft ensembling. the ensemble predictions for the two tasks are denoted as pa and pb , respectively. system recall precision f1-score original system 0.898 0.811 0.853 -m-512-tf-bi 0.884 0.818 0.850 -m-512-bi-bi 0.882 0.801 0.840 -m-512-bi-max 0.895 0.811 0.852 joint inference as mentioned above, we utilize a joint inference module to rectify the ensembling results pa before performing task a. this module corrects the prediction of hypotheses that share the same premise using the following equation: here, pa,i represents the prediction of the i th hypothesis after ensembling, while c i,j 1 denotes the prediction results of the joint inference network discussed in sec.4.3. the function @xmath5,j 1 > 0.5] employs an indicative function i[•], which is equal to 1 if the condition in brackets holds, and 0 otherwise. we set @xmath6. notably, when @xmath7,2 1 < 0.5), the hypothesis with a higher confidence level in its prediction retains its original prediction, while the hypothesis with a lower confidence level modifies its prediction. therefore, the joint inference module improves the system's prediction consistency. system output we use a threshold-based decision strategy in both tasks to obtain the ultimate predictions. specifically, for task a, hypotheses with pa 2 > η a are predicted as "entailment", while the remaining hypotheses are predicted as "contradic-tion." for task b, the system determines whether the i th sentence in the premise supports the inference by checking if pb @xmath8. in our implementation, we set the threshold values to η @xmath9.57 and η @xmath10.53. ## evaluation 6.1 data, metrics, and baselines we conduct all the evaluations on the nli4ct dataset. the train, dev, and test sets contain 1,700, 200, and 500 hypotheses. the hypotheses can be divided by the number of ctrs involved: ( 1 we compare our system with several baseline models (biolinkbert-large, scifive) that achieve sota performance on medical nli tasks. for the biolinkbert-large model, we feed the concatenated hypothesis and premise sequence into the language model and add an mlp that takes the representation of [cls] as input for classification. note that the performance of baseline models is obtained by ensembling 10 models through crossvalidation to make the results comparable. we did not train baseline models for task b because it is hard to find an existing implementation that can directly be applied to task b. ## conclusion we propose a multi-granularity system for ctrbased textual entailment and evidence retrieval. the system comprises an ensembling of inference networks and a joint inference module to enhance prediction consistency. a multi-granularity inference network (mgnet) is proposed to handle the two subtasks within a unified network. another text-generation-based network scifive is also leveraged to enhance the numerical inference of the system. we use model ensembling to integrate different models' output and enhance the inference stability, while a joint inference method is further proposed to increase the consistency of inference. our system achieves the best performance on both subtasks. the ablation experiments and analysis indicate the effectiveness of different parts of the system. in the future, we will try other large language models and new inference methods (e.g, in-context learning) to inject more medical knowledge into the system and further improve the performance on this task. fig. 4 depicts the distribution of the input sequences (concatenation of the hypothesis with premise) regarding the number of tokens in sequences. most input sequences have less than 500 tokens, while there also exists input sequence longer than 1750 tokens. as for the ctr, each ctr contains a total of four sections: 3