lasige B io TM at S em E val-2023 Task 7: Improving Natural Language Inference Baseline Systems with Domain Ontologies (2023)

Abstract: Clinical Trials Reports (CTRs) contain highly valuable health information from which Natural Language Inference (NLI) techniques determine if a given hypothesis can be inferred from a given premise. CTRs are abundant with domain terminology with particular terms that are difficult to understand without prior knowledge. Thus, we proposed to use domain ontologies as a source of external knowledge that could help with the inference process in theSemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT). This document describes our participation in subtask 1: Textual Entailment, where Ontologies, NLP techniques, such as tokenization and named-entity recognition, and rule-based approaches are all combined in our approach. We were able to show that inputting annotations from domain ontologies improved the baseline systems.

## introduction natural language inference (nli) determines whether a given hypothesis can be deduced from a given premise @xcite . particularly in the clinical text, where there is much variation in terminologies, nli is more challenging @xcite . the systems have to deal with obstacles that are constantly present in the text, such as homonyms, acronyms, or abbreviations, leading to ambiguity since, without context or background, some can be mapped to diverse expanded forms that are not entirely correct in the context @xcite ). take, for example, a part of a sentence with this text "amyotrophic lateral sclerosis (als) patients..." als here is the acronym for the disease amyotrophic lateral sclerosis, but als is also the gene symbol for the human gene sod1 superoxide dismutase 1. without context, when only presented with the als word, it is not possible to know if it is referring to the disease, the gene, or even another possibility not explored in this example. clinical trials play a fundamental role in discovering new health information (national library of medicine, national center for biotechnology information, 2023). in these studies, the aim is to perform pre-defined interventions in volunteers. for instance, the goal can be for clinicians to study the effects of different drug concentrations on patients. the outputs of these studies are registered in clinical trials reports (ctr). these reports store all the information about the conditions to be selected to participate in the trial, groups of participants, information about dosage and duration, results, and adverse events. ctrs are rich in domains with specific terms that are not easy to grasp without prior knowledge. domain ontologies can provide this external knowledge. ontologies are defined by @xcite as being the "specification of conceptualization" and provide a common vocabulary with represented shared knowledge @xcite . in this scenario, it provides domain-specific semantics to the models that can help make the connection between semantics and information extraction. because biomedical ontologies are typically represented as directed acyclic graphs, with each node representing an entity and the edges representing known relationships between those entities, ancestors can be used to obtain further information about an entity. ancestors will contribute with knowledge that cannot be directly assessed in the text @xcite . incorporating this domain knowledge might be significant to grasp all the subtleties and richness of biomedical writing when using natural language processing (nlp) approaches to get more accurate predictions. this paper presents the participation of our team, lasigebiotm (user dpavot), at the semeval-2023 task 7: multi-evidence natural language inference for clinical trial data (nli4ct) subtask 1 @xcite . this task provides a collection of breast cancer ctrs and statements about them to infer the relation label of entailment or contradiction. our approach combines ontologies, nlp techniques such as tokenization and named-entity recognition, and rule-based approaches. all code and steps to reproduce the results regarding this participation are available online foot_0 . the main goal of our participation was to assess if the introduction of external knowledge provided by domain ontologies would improve baseline systems approaches similar to the one provided in the challenge starter kit. ## related work there are some instances where authors integrated domain ontologies with the biomedical text. @xcite created the bo-lstm by incorporating biomedical ontologies and ancestry information alongside a deep learning long short-term memory model. bo-lstm was developed to extract drug-drug interactions using the chebi ontology, and it demonstrated that integrating ontologies enhanced categorization made by the model. instead of the whole instance, the authors used the shortest dependency path between the target entities. besides the word embeddings, the bo-lstm model incorporates wordnet as an external source of information, a generic english language ontology, and domain-specific ontologies. additionally, each entity was matched to an ontology concept in order to obtain their ancestors. using the previous system as a base, sousa and couto (2020) created the biont, which expands the bo-lstm with a multi-ontology integration (four types of domain-specific ontologies) and uses wordnet hypernyms. it uses gene ontology (go), human phenotype ontology (hpo), human disease ontology (do), and the chemical entities of biological interest ontologies (chebi), which can be combined in order to classify ten distinct types of relations. using three distinct datasets that represented drug-drug interactions, phenotype-gene relations, and chemical-induced disease relations, biont had an improvement of 4.93%, 4.99%, and 2.21% of the f1-score in each dataset, respectively. some studies already incorporate domain knowledge to perform nli tasks. a combination of bil-stm with attention word embeddings with definitions of medical concepts provided by the unified medical language system (umls) was used to perform nli on clinical texts @xcite . another study from @xcite also employs the domain knowledge provided by the umls by incorporating it by knowledge graph embeddings and combining it with the bert-based language model bioelmo. ## methodology this section describes the pipeline used at the semeval-2023 task 7 subtask 1. fig. 1 provides a representation of our system. in the preprocessing phase, annotations and ancestors are obtained, and then in the main pipeline, the target section is combined with information from the shortest dependency path, ancestors, and measurement rules. the output is given by the similarity of the statement and the enriched ctr section text. ## results and discussion our team (user dpavot) reached the overall ranking in the 21 st position, obtaining a 0.661 (18) f1score, 0.511 (23) precision, and 0.936 (5) recall. although this shows some improvement, the obtained scores in a balance binary task are the result of the model mainly predicting the entailment class. the best-performing combination consisted in having ancestors on single ctrs while removing them from the ctrs comparison. ctrs comparison statements were the ones that provided a significant challenge to our pipeline. comparison statements consisted of 30% (60) of the development set. for these occurrences, our approach consisted of joining the section text from each ctr as a unique text. this method resulted in noise since the text's origin was lost, and it was impossible to distinguish which part of the text belonged to which ctr. the approach also resulted in a massive comparison text, with much non-relevant information to compare with the statement. regarding count rules, we explored in which situations it fails. one of the reasons count rules may fail is if the measurement of the number is not included in the list of created measurements. this is also true for timespan measurements ("day", "week", "month", et cetera). spacy is used to verify the entity "of" to which the number refers. however, it's important to note that spacy may not always accurately identify the children or the right entity of the number, which can lead to inaccurate conclusions. one practical example of this occurrence is demonstrated in the following example: considering the ctr text: "documented menopausal status premenopausal (having menstrual periods or @xmath0." our rules, wrongly extract: ## conclusion although the state-of-the-art employs deep learning techniques, these are very "hungry" for data and sometimes do not have the pervasiveness ability when encountering different conditions from those on the training set @xcite . since the provided dataset was small, we explored the combination of rule-based nlp enriched with external information. as previously stated, our goal was to evaluate if inputting annotations from domain ontologies could improve the baseline systems. our results showed that the addition of the entities annotations improved these systems. as for future work, we wish to explore incorporating thresholds regarding non-specific annotations and ancestors to generate more significant annotations.