Mention-Agnostic Information Extraction for Ontological Annotation of Biomedical Articles (2024)

Abstract: Biomedical information extraction is crucial for advancing research, enhancing healthcare, and discovering treatments by efficiently analyzing extensive data. Given the extensive amount of biomedical data available, automated information extraction methods are necessary due to manual extraction’s labor-intensive, expertise-dependent, and costly nature. In this paper, we propose a novel two-stage system for information extraction where we annotate biomedical articles based on a specific ontology (HOIP). The major challenge is annotating relation between biomedical processes often not explicitly mentioned in text articles. Here, we first predict the candidate processes and then determine the relationships between these processes. The experimental results show promising outcomes in mention-agnostic process identification using Large Language Models (LLMs). In relation classification, BERT-based supervised models still outperform LLMs significantly. The end-to-end evaluation results suggest the difficulty of this task and room for improvement in both process identification and relation classification.

## introduction in the biomedical domain, unraveling the mechanisms underlying various diseases contributes significantly to their treatment and prevention. however, information about these mechanisms is often scattered across articles, presenting challenges. the challenges include the lack of clarity, the implicit nature of background knowledge, and the ad hoc use of vocabularies with variations in notation. moreover, inherent biological complexity spans molecules, cells, and organs, with external factors such as viruses influencing infection mechanisms. to address these challenges, organizing knowledge through ontologies is crucial as they provide * equal contribution. a clear framework for consistently structuring entities and their relationships. in the homeostasis imbalance process ontology (hoip), manual annotation has been employed to extract and structure knowledge about processes such as cellular senescence and covid-19 infection mechanisms @xcite . despite these systematic approaches, manual annotation faces significant challenges due to its high cost and time-consuming nature. these challenges highlight the need for more efficient and consistent (semi-)automated annotation approaches to improve the overall quality and usefulness of ontologies. in this paper, we propose an application of natural language processing (nlp) as a promising solution. specifically, assuming automatic annotation of the hoip ontology as our ultimate goal, we propose a two-stage information extraction (ie) system. figure 1 shows an overview of our two-stage system. given an input passage foot_0 , the first stage, process identification, identifies process entities that are described in the passage or can be inferred using the domain knowledge. 2 the entities are represented as unique ids in the ontology. the entities are then passed to the second stage, document-level relation extraction (docre) @xcite @xcite @xcite , to classify entity pairs into a pre-defined set of interrelations. the system output is represented as a set of triples: {(head entity id, relation, tail entity id)}. we develop and evaluate different approaches including supervised models based on bert @xcite and progressive respiratory failure develops in many patients with severe covid-19 soon after the onset of dyspnea and hypoxemia. these patients commonly meet the criteria for the acute respiratory distress syndrome (ards), which is defined as the acute onset of bilateral infiltrates, severe hypoxemia, and lung edema that is not fully explained by cardiac failure or fluid overload. ## hoip dataset our ultimate goal is to update and improve ontologies by (semi-)automatically extracting entities and their interrelations from articles. as a testbed ontology, we choose the homeostasis imbalance process ontology (hoip) @xcite , which focuses on understanding the covid-19 infectious mechanism (courses). 3 to facilitate the development of nlp systems and benchmark the task, we construct and release a new dataset named the hoip dataset based on the hoip ontology. the dataset includes passages extracted from pubmed articles describing biomedical processes in the context of covid-19 infectious courses. each passage is a brief portion of a pubmed article that describes at least two specific processes. the processes are manually annotated as a set of triples, i.e., {(head entity, relation, tail entity)}. figure 1 shows a real example in the dataset. ## methods for process identification in the hoip dataset, a biological process entity is annotated depending on whether it is mentioned (explicitly or implicitly) in the passage, without specifying the corresponding phrase of the entity in the passage. this makes the dataset more closely match the real-world scenario, but also brings challenges to the automatic process identification -directly employing named entity recognition methods that require the correspondence between a explicit mention (entity text and offsets) and an input text for model training is no longer an option. to address this task, we propose approaches to identify biological processes without prior recognition of mentions that can be matched to terminological expressions of entities in the hoip ontology. two distinct approaches are developed: bert-based supervised methods and llm-based in-context learning (icl) methods. ## methods for document-level re given an input passage d, a set of entities for the passage {e 1 , • • • , e k }, and a pre-defined set of relations r, document-level relation extraction (docre) @xcite @xcite aims to predict relations from r ∪ {na} for entity pairs (e i , e j ) (i, @xmath0, k]; i ̸ = j), where e i and e j denote head and tail entities respectively, and the na class indicates that the entity pair has no relation. ## experiments on process identification binary classification. in the supervised task formulation, each passage is associated with all 360 process names, with binary labels assigned based on the presence of them in the annotations. numerous negative samples are constructed for each passage. consequently, we incorporate negative sampling using various ratios of negative to positive samples. the classification task involves finetuning bert-based models -biobert @xcite , scibert @xcite , and pub-medbert @xcite on the training set. for hyper-parameter details see generative experiments. two instruction types are utilized: one prompts the model to list all biological processes in the text, while the other instructs it to generate pairs of processes having a relation. this distinction stems from annotation being conducted at the relation level, where only processes involved in relations are annotated. consequently, other processes may exist in the text but aren't annotated. we employ llama2 13b @xcite and llama3 8b (ai@meta, 2024) on the test set to ensure comparability with the supervised method's results. ## experiments on document-level re we evaluate our systems on the hoip dataset and cdr dataset @xcite . cdr consists of 1,500 abstracts from pubmed, manually annotated with chemical or disease entities and chemical-induce-disease relations between them. since entity ids in cdr are mesh unique ids (e.g., d006493), and the hoip ontology, while highly specialized for annotating processes related to covid-19, does not provide the coverage for general chemical compounds and disease terms as the mesh controlled vocabulary, we used mesh instead of the hoip ontology for cdr. we use precision, recall, and f1 metrics, and report the scores averaged independently over 3 trials with different random seeds. a triple (e i , r, e j ) is considered correct when the head entity id (e i ), the tail entity id (e j ), and the relation label (r) are all predicted correctly. we used greedy decoding in the llm-icl methods, the results of which do not depend on the seed differences. ## case study we performed case study to analyze the system outputs qualitatively. we used the best llama3 (8b) and ma-atlop models for process identification and docre, respectively. the quality evaluation reveals several insights: (1) detailing causal relationships in elucidating disease mechanisms often necessitates background knowledge not explicitly mentioned in articles. this background knowledge is sometimes added to intermediate causal entities by manual annotation. the bert-based supervised models and llms have difficulty in obtaining such background knowledge and understanding the task from limited id72. passage: pyroptosis is a highly inflammatory form of lytic programmed cell death that occurs most frequently upon infection with intracellular pathogens and is likely to form part of the antimicrobial response. pyroptosis can take place in immune cells and is also reported to occur in keratinocytes and some epithelial cells. formation of pores causes cell membrane rupture and release of cytokines, as well as various damageassociated molecular pattern (damp) molecules such as hmgb-1, atp and dna, out of the cell. these molecules recruit more immune cells and further perpetuate the inflammatory cascade in the tissue. true-positive entities: * pore formation in membrane of other organism * pyroptosis * release of damp molecules by cell rupture false-negative entities: * binding of pattern recognition receptor to damps true-positive triples: * (pyroptosis, has part, pore formation in membrane of other organism) * (pyroptosis, has result, release of damp molecules by cell rupture) false-negative triples: * (release of damp molecules by cell rupture, has result, binding of pattern recognition receptor to damps) ## related work gene ontology causal activity modeling (go-cam) defines molecular-level causal relation-ships @xcite ; however, it lacks granularity and context for covid-19 infection. our hoip dataset is based on the hoip ontology @xcite , which organizes knowledge about biomedical processes in the context of covid-19 infectious courses and thus essential for analyzing sars-cov-2 infection and progression. in knowledge acquisition, entities are typically identified by named entity recognition (ner) and entity disambiguation (ed). the task of ner is to identify mentions in the given text that represent one of the pre-defined types (e.g., chemical, disease) @xcite @xcite . the entity mentions are then passed to ed to link them to the knowledge-base concept ids that the mentions refer to best @xcite @xcite . these tasks commonly assume that entities are explicitly described in text. in reality, however, entities are not necessarily explicitly described. in this work, we explore mention-agnostic methods for process identification. the most widely used approach to docre is to model entities by a pre-trained transformer and perform pairwise relation classification. @xcite proposed to model entity dependencies via graphs with nodes of various granularities. zhou et al. (2021) proposed atlop, which models entity-pair contexts for pairwise relation classification. xiao et al. (2022) introduced evidence modeling for improving atlop. zhang et al. (2021) used u-net architecture for modeling entity dependencies. these methods commonly rely on mentions and often insert special mentionboundary markers into text to indicate the mention locations to the transformer. however, li et al. (2023) showed that these methods are too sensitive to the accuracy of mentions and it is unrealistic to expect perfect mentions in the real-world scenario. in contrast, we propose mention-agnostic docre methods and investigate how well the mentionagnostic models can identify relations. ## conclusion to assist ontology-based biological knowledge annotation, this work proposes a new dataset and practicable entity-and relation-level biomedical information extraction methods. we will continue to promote relevant research of semi-automatic annotation and advance practical applications.