ITTC at S em E val 2023-Task 7: Document Retrieval and Sentence Similarity for Evidence Retrieval in Clinical Trial Data (2023)

Abstract: This paper describes the submissions of the Natural Language Processing (NLP) team from the Australian Research Council Industrial Transformation Training Centre (ITTC) for Cognitive Computing in Medical Technologies to the SemEval 2023 Task 7, i.e., multi-evidence natural language inference for clinical trial data (NLI4CT). More specifically, we were working on subtask 2 whose objective is to identify the relevant parts of the premise from clinical trial report that justify the truth of information in the statement. We approach the evidence retrieval problem as a document retrieval and sentence similarity task. Our results show that the task poses some challenges which involve dealing with complex sentences and implicit evidences.

## introduction clinical trial reports (ctrs) are an important source of evidence for clinical practitioners, which provide information on the safety and efficacy of new treatments and interventions @xcite . however, in recent years, the number of ctrs published has increased significantly, making it difficult for clinicians to stay up-to-date with the latest research @xcite . this challenge is further exacerbated by the growing number of patient cases that require customized treatment plans. as the result, it has become infeasible for practitioners to manually sift through all the current literature to provide personalized evidence-based care to their patients. to overcome this issue, there is a need for technologies that can help clinicians efficiently review and interpret the latest medical literature. one such tool is natural language inference (nli), a task in natural language processing (nlp) to determine the semantic relation between two sentences @xcite . nli-powered systems offer a promising solution by extracting relevant information from ctrs and enabling automated evidence synthesis @xcite . by leveraging nli, we could significantly improve the way we link the most current evidence to offer individualized evidence-based care @xcite . task. semeval 2023 task 7: multi-evidence natural language inference for clinical trial data (nli4ct) consists of two subtasks, i.e., textual entailment and evidence retrieval @xcite . in the first subtask, the system is challenged to decide whether a statement making claims about a single ctr (or comparing 2 ctrs) is true or not. in other words, the objective of subtask 1 is to determine the inference relation (entailment vs. contradiction) between ctr and statement pairs. subtask 2 involves a ctr premise and statement, and requires the system to select relevant facts extracted from the premise to support the statement. these facts are necessary to justify the inference label of subtask 1. dataset. the nli4ct shared task uses a collection of breast cancer ctrs. a team of clinicians generated a dataset of 2,400 statements from separate section, annotated the gold standard label for the pairs of statement and corresponding ctr, and provided evidence explaining the inference. ctrs are divided into four sections: "eligibility criteria" describe the conditions necessary for patients to be included or excluded in the clinical trial; "intervention" provides information on the treatments being studied; "results" show the number of participants and outcome measures; and "adverse events" highlight signs and symptoms observed in patients during the clinical trial. while "eligibility criteria" contain free text, "results" and "adverse events" are originally tabular data that contains numerical values. on the other hand, "intervention" is a k-column table (k is number of interventions); the cells in the table contain natural sentences. each statement in nli4ct dataset is based on specific section of ctr. ## system description we participated in subtask 2, evidence retrieval, for this shared task. we approach the problem in evidence retrieval subtask in two different formulations, that are document retrieval and sentence similarity. suppose, we have a statement s and chunks c 1 , c 2 , ..., c k of ctr (a chunk can be single or several sentences, or table cell). ## results and discussion we submit six runs using bm25 and word mover distance algorithms. the results of the submitted runs compared with a baseline are listed in the baseline achieves perfect recall (by design) and an f1 score of 0.622 and 0.716 for the dev and test sets, respectively. the rest of our submitted runs were not able to outperform these scores due to a substantially lower recall (no greater than 0.65). however, some of our runs have a 0.2 improvement in precision w.r.t. the baseline; while the baseline obtains 0.558 for precision in the test set, our runs utilizing bm25 with stopword removal and wmd score precision 0.755 and 0.763, respectively. some queries do not return any relevant document -penalizing recall -even using semantic sentence similarity. in order to improve recall, we apply the closure strategy. specifically, for statements without any documents retrieved when using statements in the dataset belong to one of two types. the "single" type comprises of only one primary trial, where the assertion made in the statement will solely pertain to this trial. on the other hand, the "comparison" type involves two ctrs: a primary trial and a secondary trial, and the evidence for claims made in the statements have to be retrieved from both trials. by analyzing the evidence retrieval by type of ctr, we find that higher performance is obtained by the baseline and the system variations applying the closure difference, as shown in a detailed look at the evidence retrieval performance breakdown by ctr section used as the statement reveal that there are difference in the distribution of evidence among section types. from the baseline performance reported in by analyzing the experimental result of evidence retrieval, we identify several cases causing low recall in our system. the retriever fails to retrieve domain-specific vocabulary, especially for bm25 algorithm that relies on term-matching. query expansion may address this issue. other instances containing numbers or requiring numerical interpretation @xcite may also not be retrieved. for example, the chunk "age 18 years" from eligibility section in ctr should be the evidence for the statement "only adults can take part in the primary trial", but our current system fails to retrieve it. evidence retrieval for supporting natural language inference tasks also poses other challenges not typically found in popular document retrieval task. first, the system needs to retrieve implicit evidence that is not explicitly mentioned in the query. this case happens when the statement involving comparison. for example, the statement "diarrhoea is the most common adverse event in the primary trial and the secondary trial" does not only require the sentences containing keyword "diarrhoea", but also other sentences containing keyword related to other adverse events. second, the system should deal with negative evidence. for example, the statement "the primary trial does not report the pfs or objective response rate of its patient cohort" with entailment label have to retrieve all sentences from ctr to guarantee there is no evidence to support negation of this statement. so, when there is no evidence exists to support the statement, the system must retrieve all sentences as the evidence. ## conclusion and future work in this paper, we described our system submitted to semeval-2023 task 7: multi-evidence natural language inference for clinical trial data (nli4ct). we tested document retrieval and sentence similarity method for subtask 2: evidence retrieval. despite of having low recall, our system improve the precision over the baseline. bm25 retriever and wmd for sentence similarity cannot handle medical vocabulary and reason with numerical information using term matching only. we find that these approaches particularly struggle to retrieve implicit or negative evidence. future directions of this work include incorporating deep architectures using language model, e.g., colbert @xcite and dense passage retrieval @xcite , for this task. on the other hand, we are interested in exploring whether injecting linguistics knowledge (such as negation and coordinating conjunction) and reasoning capabilities (such as numerical understanding and dealing with comparison and superlative) may help model to retrieve more accurate and complete evidence. we are also willing to examine the connection between two subtasks in this shared task by answering the question whether natural language inference task necessarily need complete evidence in context of clinical trial data.