How Much do Knowledge Graphs Impact Transformer Models for Extracting Biomedical Events? (2023)

Abstract: Biomedical event extraction can be divided into three main subtasks; (1) biomedical event trigger detection, (2) biomedical argument identification and (3) event construction. This work focuses in the two first subtasks. For the first subtask we analyze a set of transformer language models that are commonly used in the biomedical domain to evaluate and compare their capacity for event trigger detection. We fine-tune the models using seven manually annotated corpora to assess their performance in different biomedical subdomains. SciBERT emerged as the highest performing model, presenting a slight improvement compared to baseline models. Then, for the second subtask we construct a knowledge graph (KG) from the biomedical corpora and integrate its KG embeddings to SciBERT to enrich its semantic information. We demonstrate that adding the KG embeddings to the model improves the argument identification performance by around 20 %, and by around 15 % compared to two baseline models. Our results suggest that fine-tuning a transformer model that is pretrained from scratch with biomedical and general data allows to detect event triggers and identify arguments covering different biomedical subdomains, and therefore improving its generalization. Furthermore, the integration of KG embeddings into the model can significantly improve the performance of biomedical event argument identification, outperforming the results of baseline models.

## introduction biomedical event extraction is a complex information extraction task that identifies key information from large sets of textual data for further applications, such as the study of biomolecular mechanisms or epigenetic changes. a biomedical event is constructed from an event trigger and one or more arguments that orbit around the trigger. event triggers generally refer to nouns or verbs that express an action, circumstance or eventuality, while the arguments refer either to biomedical entities or to other events, called nested events. figure 1 1 shows the example of a sentence containing two biomedical events, '-reg' (which stands for 'negative regulation') and 'locl' (which stands for 'localization'). the event constructed from the trigger word 'excretion' of type 'locl' (the event is given the same type as the trigger) presents as single argument the biomedical entity of type 'd/c' (which stands for 'drug or compound'), playing the role of 'th' (which stands for 'theme'). this role allows answering the question 'what is excreted?'. on the other hand, the event constructed from the trigger word 'reduces' of the type '-reg', presents two arguments. the first argument is a biomedical entity of the type 'drug or compound', playing the role of 'cause'. this role allows answering the question 'what causes the reduction?'. the second argument is the nested event 'locl' described before, playing the role of 'theme', answering the question 'what is reduced?'. event extraction is usually divided into three main subtasks, event trigger detection, argument identification and event construction. event trigger detection identifies and classifies the trigger words into a set of predefined types of event triggers, while argument identification identifies and classifies the roles between the event triggers and their respective arguments @xcite . event construction refers to the unmerging of the arguments that correspond to the same event for its construction @xcite . event trigger detection has a fundamental role in the construction of events. indeed, the triggers are the targets that allow us to know that an event may exist @xcite . this subtask is usually considered as a classification problem, where each word need to be classified into a predefined set of trigger types. difficulty for trigger detection comes from the sensitivity to the domain or subdomain (text can present specialized language), linguistic forms (triggers can be single words, multi-words, discontinuous markers) and ambiguity on the trigger class (a trigger word can be given different trigger classes) @xcite . argument identification can be also considered as a multi-category classification problem, where the directed relation between a trigger and an entity or other event needs to be classified into a predefined set of role types. when these arguments are correctly identified, the event extracted has the potential to provide a reliable means of improving domain knowledge. one of the main complexities in identifying arguments is that they can be part of one or multiple events (one-to-one and one-tomultiple relations), where they play the same or different roles. following @xcite , event trigger detection is the main source of errors in event extraction, where around 31 % of the errors correspond to non-detection of triggers and 28 % to overdetection of triggers. further, the non-detection of arguments represents around 23 % of errors and the over-detection of arguments around 7 %. transformer language models have been widely adopted to try to reduce errors in event extraction due to their positive achievements in performance for solving different types of natural language processing tasks. bert @xcite , which stands for bidirectional encoder representations from transformers, is a language model designed to pretrain bidirectional representations of words, taking into account the semantics by considering both left and right directions of the text. from this pretraining, bert can be fine-tuned by including additional layers on top of the model to solve new specific tasks. furthermore, a number of domain-specific bert variants have been developed by being trained on large corpora with the same context, such as the biomedical domain. however, since the learning of the models is limited to the subdomain in which they were trained, they present limitations in per-formance when using them in different biomedical subdomains to improve the integration of domain knowledge, knowledge graph (kg) models have been implemented along with language models for different information tasks in the biomedical domain @xcite @xcite @xcite . biomedical kgs are a resource of integration of one or more sources of information (often manually curated datasets) into a graph, where biomedical entities can be represented by nodes and the relations between them by edges (nicholson and greene, 2020). kg models integrate nodes and edges into a low-dimensional vector space, known as embeddings, preserving the semantic information of the kg. in this work, we first analyze the performance of five previously trained transformer language models to identify whether they allow the identification of triggers in different biomedical subdomains. then, we enrich the semantic information of the best-performing model with kg embeddings to assess whether integrating these embeddings improves the model's ability to identify biomedical arguments and their roles. for this purpose, bert, biobert, scibert, pubmedbert, and biome-droberta are fine-tuned using two different classifiers, a linear layer and a bidirectional long short term memory (bi-lstm) layer, to detect biomedical event triggers. these bert variants are chosen for comparison since they share the same bert architecture but have previously been pretrained with different data in the biomedical and/or general domain, showing positive results in biomedical information extraction tasks @xcite @xcite . models are learned using seven manually annotated data sets merged together. these corpora were originally developed for the event extraction task in different biomedical subdomains. then, a kg is constructed from the biomedical events contained in the biomedical corpora and its kg embeddings are computed. these embeddings are integrated into the transformer language model to classify the roles between the previously identified triggers and the biomedical entities and/or other triggers, in order to detect the event arguments. ## method this work proposes the approach shown in figure 2 . the annotated data is given as input to the pretrained transformer language models for finetuning and then passed to a classification layer for event trigger detection. at this stage, a named entity recognition (ner) task is performed to identify and classify the triggers in the text. simultaneously, a kg is built from the annotated data and the kg embeddings are calculated. these embeddings are later integrated to the transformer language model and the fine-tuning is done to identify arguments. here, a relation extraction (re) task is performed to identify and classify the roles between the triggers and candidate arguments. ## results and discussion we present a comparison of five different transformer language models for the detection of biomedical trigger events using seven biomedical corpora together. also, we conduct different strategies for the enrichment of a transformer language model using kg embeddings for the identification of biomedical arguments. we finally compare our results using the test set of the cg dataset with two baselines models from the state-of-the-art. ## conclusions we analyze different transformer language models for biomedical event trigger detection and argument identification using seven different corpora and kg integration. by comparing the performance of the models, we found that fine-tuning scibert with a bi-lstm classifier is the best strategy to detect events in different biomedical domains. furthermore, when scibert is enriched with kg embeddings, especially those corresponding to the roles of arguments in events, it significantly improves the identification of biomedical arguments.