Global Locality in Biomedical Relation and Event Extraction (2020)

Abstract: Due to the exponential growth of biomedical literature, event and relation extraction are important tasks in biomedical text mining. Most work only focus on relation extraction, and detect a single entity pair mention on a short span of text, which is not ideal due to long sentences that appear in biomedical contexts. We propose an approach to both relation and event extraction, for simultaneously predicting relationships between all mention pairs in a text. We also perform an empirical study to discuss different network setups for this purpose. The best performing model includes a set of multi-head attentions and convolutions, an adaptation of the transformer architecture, which offers self-attention the ability to strengthen dependencies among related elements, and models the interaction between features extracted by multiple attention heads. Experiment results demonstrate that our approach outperforms the state of the art on a set of benchmark biomedical corpora including BioNLP 2009, 2011, 2013 and BioCreative 2017 shared tasks.

## introduction event and relation extraction has become a key research topic in natural language processing with a variety of practical applications especially in the biomedical domain, where these methods are widely used to extract information from massive document sets, such as scientific literature and patient records. this information contains the interactions between named entities such as proteinprotein, drug-drug, chemical-disease, and more complex events. relations are usually described as typed, sometimes directed, pairwise links between defined named entities @xcite . event extraction differs from relation extraction in the sense that an event has an annotated trigger word (e.g., a verb), and could be an argument of other events to connect more than two entities. event extraction is a more complicated task compared to relation extraction due to the tendency of events to capture the semantics of texts. for clarity, figure 1 shows an example from the ge11 shared task corpus that includes two nested events. recently, deep neural network models obtain state-of-the-art performance for event and relation extraction. two major neural network architectures for this purpose include convolutional neural networks (cnns) @xcite and recurrent neural networks (rnns) @xcite @xcite . while cnns can capture the local features based on the convolution operations and are more suitable for addressing short sentence sequences, rnns are good at learning long-term dependency features, which are considered more suitable for dealing with long sentences. therefore, combining the advantages of both models is the key point for improving biomedical event and relation extraction performance @xcite . however, encoding long sequences to incorporate long-distance context is very expensive in rnns @xcite due to their computational dependence on the length of the sequence. in addition, computations could not be parallelized since each token's representation requires as input the representation of its previous token. in contrast, cnns can be executed entirely in parallel across the sequence, and have shown good performance in event and relation extraction @xcite . however, the amount of context incorporated into a single token's representation is limited by the depth of the network, and very deep networks can be difficult to learn @xcite . to address these problems, self-attention networks @xcite come into play. they have shown promising empirical results in various natural language processing tasks, such as information extraction @xcite , machine translation @xcite and natural language inference @xcite . one of their strengths lies in their high parallelization in computation and flexibility in modeling dependencies regardless of distance by explicitly attending to all the elements. in addition, their performance can be improved by multi-head attention @xcite , which projects the input sequence into multiple subspaces and applies attention to the representation in each subspace. in this paper, we propose a new neural network model that combines multi-head attention mechanisms with a set of convolutions to provide global locality in biomedical event and relation extraction. convolutions capture the local structure of text, while self-attention learns the global interaction between each pair of words. hence, our approach models locality for self-attention while the interactions between features are learned by multi-head attentions. the experiment results over the biomedical benchmark corpora show that providing global locality outperforms the existing state of the art for biomedical event and relation extraction. the proposed architecture is shown in figure 2 . conducting a set of experiments over the corpora of the shared tasks for bionlp 2009, 2011 and 2013, and biocreative 2017, we compare the performance of our model with the best-performing system (tees) @xcite in the shared tasks. the results we achieve via precision, recall, and f-score demonstrate that our model obtains state-of-the-art performance. we also empirically assess three variants of our model and elaborate on the results further in the experiments. the rest of the paper is organized as follows. section 2 summarizes the background. the data, and the proposed approach are explained in sections 3 and 4 respectively. section 5 explains the experiments and discusses the achieved results. finally, section 6 summarizes the findings of the paper and presents future work. ## background biomedical event and relation extraction have been developed thanks to the contribution of corpora generated for community shared tasks @xcite @xcite @xcite . in these tasks, relevant biomedical entities such as genes, proteins and chemicals are given and the information extraction methods aim to identify relations alone or relations and events together within a sentence span. a variety of methods have been evaluated on these tasks, which range from rule based methods to more complex machine learning methods, either supported by shallow or deep learning approaches. some of the deep learning based methods include cnns @xcite @xcite and rnns @xcite @xcite . cnns will identify local context relations while their performance may suffer when entities need to be identified in a broader context. on the other hand, rnns are difficult to parallelize while they do not fully solve the long dependency problem @xcite . moreover, such approaches are proposed for relation extraction, but not to extract nested events. in this work, we intend to improve over existing methods. we combine a set of parallel multi-head attentions with a set of 1d convolutions to provide global locality in biomedical event and relation extraction. our approach models locality for self-attention while the interactions between features are learned by multi-head attentions. we evaluate our model on data from the shared tasks for bionlp 2009, 2011 and 2013, and biocreative 2017. the bionlp event extraction tasks provide the most complex corpora with often large sets of event types and at times relatively small corpus sizes. our proposed approach achieves higher performance on the ge09, ge11, epi11, id11, rel11, ge13, cg13 and pc13 bionlp shared task corpora, compared to the top performing system (tees) @xcite for both relation and event extraction in these tasks. since the annotations for the test sets of the bionlp shared task corpora are not provided, we uploaded our predictions to the task organizers' servers for evaluation. word embedding relative position embedding distance embedding 1 2 3 1 2 3 vector representation multi-head attention multi-head attention multi-head attention multi-head attentions 1d-convolutions max pooling merge layer output layer x 1 x n x 2 . . . multi-head attention the chemprot corpus in the biocreative vi chemical-protein relation extraction task (cp17) also provides a standard comparison with current methods in relation extraction. the chemprot corpus is relatively large compared to its low number of five relation types. our model outperforms the best-performing system (tees) (björne and salakoski, 2018) for relation extraction in this task. ## data we develop and evaluate our approach on a number of event and relation extraction corpora. these corpora originate from three bionlp shared tasks @xcite @xcite and the biocreative vi chemical-protein relation extraction task @xcite . the bionlp corpora cover various domains of molecular biology and provide the most complex event annotations. the biocreative corpora use pairwise relation annotations. for further analysis and experiments, we also used the amia gene-mutation corpus available in @xcite . the training/testing sets contain 2656/385 mentions of mutations, and 2799/280 of genes or proteins, and 1617/130 rela- corpus domain e i s ge09 molecular biology 10 6 11380 ge11 molecular biology 10 6 14958 epi11 epigenetics and ptm:s 16 6 11772 id11 infection diseases 11 7 5118 rel11 entity relations 1 2 11351 ge13 molecular biology 15 6 8369 cg13 cancer genetics 42 9 5938 pc13 pathway curation 24 9 5040 cp17 chemical-protein int. -5 24594 ## model we propose a new biomedical event extraction model that is mainly built upon multi-head attentions to learn the global interactions between each pair of tokens, and convolutions to provide locality. the proposed neural network architecture consists of 4 parallel multi-head attentions followed by a set of 1d convolutions with window sizes 1, 3, 5 and 7. our model attends to the most important tokens in the input features foot_0 , and enhances the feature extraction of dependent elements across multiple heads, irrespective of their distance. moreover, we model locality for multi-head attentions by restricting the attended tokens to local regions via convolutions. the relation and event extraction task is modelled as a graph representation of events and relations @xcite . entities and event triggers are nodes, and relations and event arguments are the edges that connect them. an event is modelled as a trigger node and its set of outgoing edges. relation and event extraction are performed through the following classification tasks: (i) entity and trigger detection, which is a named-entity recognition task where entities and event triggers in a sentence span are detected to generate the graph nodes; (ii) relation and event detection, where relations and event arguments are predicted for all valid pairs of entity and trigger nodes to create the graph edges; (iii) event duplication, where each event is classified as an event or a negative which causes unmerging in the graph foot_1 ; (iv) modifier detection, in which event modality (speculation or negation) is detected. in relation extraction tasks where entities are given, only the second classification task is partially used. the same network architecture is used for all four classification tasks, with the number of predicted labels changing between tasks. ## experiments and results we have conducted a set of experiments to evaluate our proposed approach over the benchmark biomedical corpora. in addition to evaluating our main model (4mha-4cnn), we have evaluated the performance of three variants of our proposed approach: (i) 4mha: 4 parallel multi-head attentions apply self-attention multiple times over the input features; (ii) 1mha: only 1 multi-head attention applies self-attention to the input features; (iii) 4cnn-4mha: multiple self-attentions are applied to the input features via a set of 1d convolutions foot_3 . the 4cnn architecture matches the best performing configuration (4cnn -mixed 5 x ensemble) foot_4 used by tees @xcite , which is composed of four 1d convolutions with window sizes 1, 3, 5 and 7. in our models and tees, we set the number of filters for the convolutions to 64. the number of heads for multi-head attentions is also set to 8. the reported results of tees are achieved by running their out-of-the-box system for different tasks. since training a single model can be prone to overfitting if the validation set is too small @xcite , we use mixed 5 model ensemble, which takes 5-best models (out of 20), ranked with micro-averaged f-score on randomized train/validation set split, and considers their averaged predictions. these ensemble predictions are calculated for each label as the average of all the models' predicted confidence scores. precision, recall, and f-score of the proposed approach and its variants are compared to tees in analyzing the results, we observe that the proposed 4mha-4cnn model has the best f-score in the majority of datasets except for epi11, id11 and cg13, where the proposed mha models (i.e., 1mha and 4mha) have the best f-score and recall. these tasks are related to epigenetics and post-translational modifications (epi11), infection diseases (id11) and cancer genetics (cg13), where events typically require long dependencies in most of the cases. it explains why the mha-alone models are better than when combined with convolutions. the f-scores achieved by 4mha-4cnn and 4mha models on ge09 dataset are also very close. in many cases, when using the configurations in which mha is applied to the input features, both precision and recall are better compared to other configurations. moreover, having four parallel mhas applied to the input features outperforms 1mha and the other potential variants 6 . in terms of precision, the advantage of applying 4cnn versus 4mha to the merged input features depends on the dataset. on pc13, the precision when using 4cnn on the merged input features is much higher compared to other configurations, but the recall is significantly lower. the the presence presence of of activating activating tsh-r tsh-r mutations mutations has has also also been been demonstrated demonstrated in in differentiated differentiated thyroid thyroid carcinomas carcinomas . . the presence of activating tsh-r mutations has also been demonstrated in differentiated thyroid carcinomas . the presence of activating tsh-r mutations has also been demonstrated in differentiated thyroid carcinomas . the presence of activating tsh-r mutations has also been demonstrated in differentiated thyroid carcinomas . (4mha-4cnn) (1mha) the the presence presence of of activating activating tsh-r tsh-r mutations mutations has has also also been been demonstrated demonstrated in in differentiated differentiated thyroid thyroid carcinomas carcinomas . . the presence of activating tsh-r mutations has also been demonstrated in differentiated thyroid carcinomas . the presence of activating tsh-r mutations has also been demonstrated in differentiated thyroid carcinomas . the presence of activating tsh-r mutations has also been demonstrated in differentiated thyroid carcinomas . (4mha) figure 3: visualization of multi-head attention in different architectures overall, our observations support the hypothesis that higher recall/f-score is obtained in configurations in which 4mha is applied first to the merged input features, where cnns are not as convenient as mhas to deal with long dependencies. ## conclusion we have proposed a novel architecture based on multi-head attention and convolutions, which deals with the long dependencies typical of biomedical literature. the results show that this architecture outperforms the state of the art on existing biomedical information extraction corpora. while multihead attention identifies long dependencies in extracting relations and events, convolutions provide the additional benefit of capturing more local relations, which improves the performance of existing approaches. the finding that cnn-before-mha is outperformed by mha-before-cnn is very interesting and could be used as a competitive baseline for future work. our ongoing work includes generalizing our findings to other non-biomedical information extraction tasks. current work is focused on event and relation extraction from a single short/long sentence; we would like to experiment with additional contents to study the behaviour of these models across sentence boundaries @xcite . finally, we intend to extend our approach to deal with speculative contexts by considering more semantic linguistic features, e.g., sense embeddings @xcite on biomedical literature.