Fusing Heterogeneous Factors with Triaffine Mechanism for Nested Named Entity Recognition (2022)

Abstract: Nested entities are observed in many domains due to their compositionality, which cannot be easily recognized by the widely-used sequence labeling framework.A natural solution is to treat the task as a span classification problem. To learn better span representation and increase classification performance, it is crucial to effectively integrate heterogeneous factors including inside tokens, boundaries, labels, and related spans which could be contributing to nested entities recognition. To fuse these heterogeneous factors, we propose a novel triaffine mechanism including triaffine attention and scoring.Triaffine attention uses boundaries and labels as queries and uses inside tokens and related spans as keys and values for span representations.Triaffine scoring interacts with boundaries and span representations for classification. Experiments show that our proposed method outperforms previous span-based methods, achieves the state-of-the-art F_1 scores on nested NER datasets GENIA and KBP2017, and shows comparable results on ACE2004 and ACE2005.

## introduction named entity recognition (ner) is a fundamental natural language processing task that extracts entities from texts. flat ner has been well studied and is usually viewed as a sequence labeling problem @xcite . however, nested entities also widely exist in real-world applications due to their multi-granularity semantic meaning @xcite , which cannot be solved by the sequence labeling framework since tokens have multiple labels @xcite . various paradigms for nested ner have been proposed in recent years. a representative direction is the span-based approach that learns deep representation for every possible span and then classifies it to the corresponding type @xcite 2019; @xcite @xcite @xcite . by leveraging the large-scale pretrained language model, several works show that the simple model structure for span representation and classification can achieve satisfactory results @xcite . however, we still believe that explicit modeling of some relevant features will further benefit the span representation and classification under the complex nested setting. taking figure 1 as an example, we claim that the following factors are critical for recognizing whether a span is an entity. (1) tokens: it is obvious that tokens of the given span contribute to the recognition. (2) boundaries: we emphasize boundaries (or boundary tokens) because they are special tokens with rich semantics. works with simple structure may just produce the span representation based on the concatenation or biaffine transformation of boundary representation @xcite . some other works take boundary detection as additional supervision for better representation learning @xcite . more importantly, a unilateral boundary cannot determine the entity type since it can exist in multiple entities with different labels (e.g., "nf", "b", and "cells") under the nested setting. (3) labels: as mentioned above, tokens could belong to entities with different labels. therefore, we propose that the model should learn label-aware span representation to take into consideration of the different token contributions at the label level. 1 for example, "nf" may contribute more to "protein" type when classifying the span "nf -chi b", as well as "chi b" and "site" contribute more to "dna" type when classifying the span "nf -chi b site". (4) related spans: interactions among spans are important in nested entities @xcite @xcite . the insider and outsider entities may hint at each other's types. for example, entities inside "ebv-transformed b cells" have more possibilities to be cell-related entities. interactions can also help the non-entity span like "transformed b cells" to validate its partialness by looking at outer entity "ebv -transformed b cells". although some of the factors may have been explored in previous works, to the best of our knowledge, it is the first work to fuse all these heterogeneous factors into a unified network. as the traditional additive, multiplicative attention, or biaffine transformation cannot interact with such multiple heterogeneous factors simultaneously, we propose a novel triaffine mechanism as the tensor multiplication with three rank-1 tensors (vectors) and a rank-3 tensor, which makes it possible to jointly consider high-order interactions among multiple factors. specifically, our method follows the pipeline of span representation learning and classification. at the stage of span representation learning, we apply the triaffine attention to aggregate the label-wise span representations by considering boundaries and labels as queries as well as inside tokens as keys and values. then, a similar triaffine attention is applied to produce the label-wise crossspan representations by querying boundaries and labels with related spans. at the stage of span classification, we fuse the span representations and boundaries for label-wise classification with a triaffine score function. in practice, we add an auxiliary object function to classify spans without the cross-span interaction, which benefits learning robust span representation and can be used as a span filter to speed up both training and inference without performance degradation. we conduct experiments on four nested ner datasets: ace2004, ace2005, genia, and kbp2017. our model achieves 88.56, 88.83, 81.23, and 87.27 scores in terms of f 1 , respectively. using the bert encoder, our model outperforms stateof-the-art methods on genia and kbp2017 and 1 label is the perdition object that we cannot touch in representation learning. here, leveraging label information only means we need label-aware representation learning. our contributions are summarized as foot_0 : ## approach figure 2 shows an overview of our method. we will first introduce the triaffine transformations, which lie in the heart of our model to fuse heterogeneous factors. then, we will introduce our model including triaffine attention and triaffine scoring based on the proposed triaffine transformations. ## conclusion in this paper, we propose a span-based method for nested ner. heterogeneous factors including tokens, boundaries, labels, and related spans are introduced to improve span classification with a novel triaffine mechanism. experiments show our method outperforms all span-based methods and achieves state-of-the-art performance on four nested ner datasets. ablation studies show the introduced heterogeneous factors and triaffine mechanism are helpful for nested setting. despite that large-scale pretrained language models have shown consistent improvement over many nlp tasks, we argue that the well-designed features and model structures are still useful for complex tasks like nested ner. furthermore, although we only verify our triaffine mechanism in nested ner, we believe it can also be useful in tasks requiring high order interactions like parsing and semantic role labeling.