Improving the Evaluation of NLP Approaches for Scientific Text Annotation with Ontology Embedding-Based Semantic Similarity Metrics (2023)

Abstract: Lexical Simplification is a challenging task that aims to improve the readability of text for nonnative people, people with dyslexia, and any linguistic impairments. It consists of 3 components: 1) Complex Word Identification 2) Substitute Generation 3) Substitute Ranking. Current methods use contextual information as a primary source in all three stages of the simplification pipeline. We argue that while context is an important measure, it alone is not sufficient in the process. In the complex word identification step, contextual information is inadequate, moreover, heavy feature engineering is required to use additional linguistic features. This paper presents a novel architecture for complex word identification that uses a pre-trained transformer model’s information flow through its hidden layers as a feature representation that implicitly encodes all the features required for identification. We portray how database methods and masked language modeling can be complementary to one another in substitute generation and ranking process that is built on the foundational pillars of Simplicity, Grammatical and Semantic correctness, and context preservation. We show that our proposed model generalizes well and outperforms the current state-of-the-art on wellknown datasets.

## introduction as several fields of science entered the dataintensive era, ontologies grew increasingly popular for consistent, machine-readable representation of scientific data @xcite @xcite . ontologies proved to be critical in life sciences and particularly in biology to power large comparative analyses @xcite @xcite . the gene ontology (go) was created in 2003 @xcite as a collaborative effort to facilitate meaningful descriptions of genes in a variety of organisms @xcite . ontologies enable standardization of biology by providing a common vocabulary for describing biological entities and their relationships. this helps to reduce ambiguity and improve communication between researchers. they enable data integration and interoperability @xcite . ontologies can be used to link data from different sources, even if they are stored in different formats. this allows researchers to combine data from multiple experiments and databases to get a more complete picture of biological phenomena. ontologies were built for supporting knowledge discovery and hypothesis generation @xcite . ontologies can be used to reason about biological data and identify new patterns and relationships @xcite . this can help researchers to generate new hypotheses and develop new insights into biological systems. finally, ontologies facilitate the development of new tools and applications to support biological research, such as tools for data annotation, knowledge discovery, and computational modeling @xcite . while ontologies provide the necessary structure and concepts, the real benefits of ontologies can be reaped only when knowledge in scientific literature is represented using these ontologies through annotation @xcite . one of the use cases for ontologies is gene annotation -ontologies are used to annotate genes with information about their function, structure, and expression patterns @xcite . this information can be used to better understand the role of genes in biological processes and diseases. creating annotations using ontologies started as a manual task undertaken by expert level human curators @xcite . manual annotation of literature using ontologies is the process of identifying and tagging relevant biological entities and relationships in text with ontology terms. this is a time-consuming and labor-intensive process, but it is essential for creating high-quality, machinereadable annotations that can be used to support a wide range of biological research tasks. the manual annotation process typically involves the following steps: these go annotations are the drivers for several crucial applications in biology such as gene function discovery, genome annotation, comparative genomics, functional genomics, and systems biology @xcite @xcite . the ongoing generation of go annotations as new literature is published is important to be able to conduct comparisons between species and to utilize rich genomic data for answering complex biological questions. however, manual curation of scientific literature by human curators soon became an infeasible practice because it was tedious, slow, and unscalable to the rapid pace of scientific publishing @xcite . the bioinformatics community turned to text mining and natural language processing as a means to automate the process of ontology-based annotation. ontology-based annotation is the process of reading (by a human or a machine) scientific literature and associating words in the text to appropriate ontology concepts (see figure 1 ) @xcite . the initial foray into automated annotation using natural language processing relied on lexical analysis and standard machine learning approaches @xcite . more recently, deep learning approaches have shown promise with text related applications @xcite @xcite @xcite . the past few years have witnessed an increasing focus on automated annotation of scientific literature and the development of sophisticated nlp approaches @xcite @xcite @xcite . with this, a second and equally important problem came to light -robust and accurate methods for evaluating the success of these nlp methods @xcite . evaluating the performance of nlp systems that are ontology-based is different from traditional nlp systems because of the possibility of partial accuracy. traditional information retrieval systems only consider whether the target information is retrieved (success) or not (failure). in contrast, ontologybased information retrieval systems allow for the possibility of partial success. this means that the system can still be successful even if it does not retrieve the exact target information, as long as it retrieves something that is semantically similar @xcite . ontology-based information retrieval systems are evaluated using three possibilities: the goal of our nlp systems is to maximize accurate retrieval rates and minimize inaccurate retrieval rates. in cases where complete accuracy is not achieved, the method aims to maximize partial accuracy. here is an example to illustrate the difference between traditional information retrieval and ontology-based information retrieval: imagine that you are searching for information about the concept of "dog". a traditional information retrieval system would only consider whether it retrieves information that explicitly mentions the word "dog". in contrast, an ontology-based information retrieval system would also consider retrieving information about other concepts that are semantically similar to "dog", such as "canine" or "mammal". if an ontology-based information retrieval system retrieves information about "canine" when you are searching for information about "dog", this would be considered a partial success. the system has not retrieved the exact target information, but it has retrieved something that is semantically similar. ontology-based nlp systems need evaluation metrics that can account for accurate, inaccurate, and partial success @xcite . the answer to this problem can be found in semantic similarity metrics designed to estimate similarity (exact or partial) between ontology concepts @xcite . several papers published in the recent past describing nlp techniques for automated annotation of literature have used semantic similarity metrics for evaluating accuracy @xcite @xcite . however, these existing semantic similarity measures fall short in the area of robustness since they have been shown to be quite susceptible to noise in the data @xcite . in addition, in the case of ontology-based annotation, there is a great deal of information embedded in the ontology hierarchy and semantics that needs to be leveraged for accurate annotation @xcite . here, we present an improved semantic similarity metric for the evaluation of nlp approaches for automated ontology annotation. our metric is based on graph embeddings created from ontologies (which are directed acyclic graphs). we compared our metric to existing metrics applied on a gold-standard dataset. graph embeddings are a type of machine learning technique that can be used to represent graphs in a low-dimensional vector space. this makes it possible to use machine learning algorithms to learn from and make predictions on graph data @xcite . graph embeddings are particularly useful for nlp tasks, as many nlp problems can be represented as and are applied on graphs @xcite . graph embeddings can be used to improve word representations, improve text classification, and for machine translation. overall, graph embeddings are a powerful tool that can be used to improve the performance of a wide range of nlp tasks. while we use data and background from the domain of biology, the methods presented here have wide scientific appeal and are generalizable. ontologies are used in chemistry, physics, astronomy, geoscience, materials science, medicine, psychology, and social science @xcite @xcite . hence, the need for automated annotation methods and effective semantic similarity metrics for evaluation exist in all these domains. ## gold standard dataset the dataset used here was developed for the phenoscape project funded by the united states national science foundation @xcite . three expert scientists were asked to independently annotate the same text related to evolutionary biology. a gold standard dataset was created by consensus of the three curators. the intuition behind this experimental set-up is that the annotations created by each curator are expected to be quite similar to the gold standard if not identical. this allows us to test different semantic similarity metrics by testing how well they retrieve similarity from known biologically similar annotations @xcite . the dataset is publicly accessible from dahdul et. al. @xcite 3 ontology the ontology used in the dataset described above is @xcite . uberon is a cross-species ontology that represents anatomical structures in animals @xcite . this ontology is used to annotate biological data, such as gene expression data and protein interaction data, to enable data integration and knowledge discovery @xcite . it has been used to develop tools and applications to support biological research and clinical practice @xcite . ## results the curator and gold standard datasets obtained from dahdul et al @xcite . contained 1266 uberon annotations. during the embedding creation step, the sentences were generated using the following hyper parameters: i. @xmath0.5 ii. @xmath1.0 iii. @xmath2. @xmath3. @xmath4 the embeddings were learned using a deep learning model trained with the following hyperparameters: i. @xmath5. @xmath6. @xmath7. @xmath8. @xmath9, we obtained 128-dimensional normalized embedding vectors for each of the 15,539 concepts in the uberon ontology. a rf model was trained on predict the gold standard annotation based on the three curators' annotations and their similarity to the gs. these results clearly show that our similarity measures based on ontology embeddings substantially outperform the commonly used jaccard similarity. among the metrics that use ontology embeddings we found dot product to be the most robust. next, we combined our top performing metric (dot product) with all the other metrics to create "super metrics" to increase robustness. ## conclusions our goal was to develop robust semantic similarity metrics based on ontology embeddings to evaluate the performance of nlp approaches for automated annotation of scientific literature. we tested five embedding-based metrics against a widely used traditional metric and showed that the embeddingbased metrics outperformed the traditional metric. we also found that combining the embeddingbased metrics further improved accuracy. these results suggest that traditional semantic similarity metrics, which are based on comparisons of ontology subsumers, can be improved by using ontology embeddings. the new, more robust, and sensitive similarity metrics will enable an accurate assessment of nlp approaches for ontology annotation.