H i TRANS : A Hierarchical Transformer Network for Nested Named Entity Recognition (2021)

Abstract: Nested Named Entity Recognition (NNER) has been extensively studied, aiming to identify all nested entities from potential spans (i.e., one or more continuous tokens). However, recent studies for NNER either focus on tedious tagging schemas or utilize complex structures, which fail to learn effective span representations from the input sentence with highly nested entities. Intuitively, explicit span representations will contribute to NNER due to the rich context information they contain. In this study, we propose a Hierarchical Transformer (HiTRANS) network for the NNER task, which decomposes the input sentence into multi-grained spans and enhances the representation learning in a hierarchical manner. Specifically, we first utilize a two-phase module to generate span representations by aggregating context information based on a bottom-up and top-down transformer network. Then a label prediction layer is designed to recognize nested entities hierarchically, which naturally explores semantic dependencies among different spans. Experiments on GENIA, ACE-2004, ACE-2005 and NNE datasets demonstrate that our proposed method achieves much better performance than the state-of-the-art approaches.

## introduction named entity recognition (ner) is an essential task in the research of natural language processing, which aims to detect and classify text spans into corresponding semantic categories, e.g., person (per), organization (org), and location (loc), from a chunk of text. most existing studies focus on flat ner, i.e., without nested entities, by sequence labeling methods @xcite . however, named entities are generally nested with each other in the real world @xcite . for example, in figure 1 , the entity "st. louis" with label "city" is nested in "st. louis cardinals" with label "sportsteam". this poses a major technical challenge to the previous methods and thus a more robust model for nested ner (nner) is urgently desirable. previous literature for nner can be roughly categorized into three types: 1) hypergraphbased models focus on designing a complex hypergraph structure to obtain an expressive tagging schema @xcite @xcite , which are time-consuming when encountering ambiguous schemas; 2) span-based models tend to detect candidate spans from an input sentence first, and then train a classifier to predict entity categories @xcite . however, it is hard to get a complete meaning of the sentence because each text span contains only a part of the semantics, and errors may propagate to the prediction stage if the span boundary is divided incorrectly at the first stage; and 3) layered-based models are proposed to utilize layered structures to deal with nner based on the divide and conquer strategy @xcite @xcite . however, it merely breaks down the complex problem into several smaller subtasks and pays little attention to the hierarchical representation learning for multi-grained named entities. to this end, we propose a novel hierarchical transformer network (hitrans) to recognize more named entities (either nested or not) for a given sentence, where we capture the dependencies of adjacent candidate spans and utilizes an attention mech-anism to enhance the representation of text spans. more specifically, the input of our proposed method combines character-level, word-level, and sentencelevel representations, which are obtained by three embedding networks, respectively. we then propose a two-phase span generation model (sgm) on top of the multi-level representations, which hierarchically aggregates adjacent spans based on transformer mechanism at each layer. the sgm includes a bottom-up and a top-down structure to enhance the representation learning of each candidate span which is finally fed into a label prediction layer to assign an entity class for the span. as a result, nested entities are comprehensively contained in the candidate spans and the representation learning is further enhanced based on both multi-level embedding and the hierarchical transformer mechanisms. experimental results on four datasets demonstrate that hitrans establishes new stateof-the-art performances, which verifies the effectiveness of our proposed framework. the main contributions of this work are as follows: ## related work we briefly review some prior works closely related to ours from three perspectives: hypergraph-based, span-based, and layered-based approaches. hypergraph-based approaches obtain expressive tagging schemas for nner @xcite . however, the hypergraph requires specific modules designed to prevent the spurious structure of hypergraphs. @xcite introduced mention separators to facilitate multi-graph representation. @xcite further improved the result using features extracted from a recurrent neural network. recently, @xcite proposed two competitive neural networks using a linearized scheme. however, more expressive and unambiguous schemas will inevitably cause higher time complexity. span-based methods achieve promising results for nner @xcite @xcite , which explicitly enumerate all possible spans from input sentences, which will be fed into a classifier for category prediction based on multitask learning. @xcite proposed a sequence-to-nuggets architecture to recognize nested entities with semantic central words. @xcite extracted answer spans from a passage through a given question. @xcite proposed a novel bipartite flat-graph network to learn the dependencies of inner spans. but most of these methods generally break input sequences into fragments, leading to inferior semantics. layered-based models are recently proposed, e.g., @xcite constructed a syntactic constituency tree to transform each sentence into a tree, @xcite proposed a transitionbased model by mapping a sentence with nested mentions to a designated forest, @xcite and @xcite dynamically stacked multiple flat ner layers from inside to outside, shibuya and hovy (2020) introduced a decoding method that iteratively recognizes entities in an outside-to-inside way, jue et al. (2020) and @xcite utilized a layered model to recursively identify named entity candidates based on a hierarchical structure, which is suitable for nner. however, few of them emphasize on learning more effective span representations, failing to recognize nested named entities in more complex sentences. the core idea of our work is inspired to enhance representation learning for more complex sentences. we propose to leverage the representation power of transformer based on a hierarchical structure for improving nner. particularly, pretrained word embeddings such as glove @xcite , and pre-trained sentence-level embedding such as bert @xcite and albert @xcite have proven to be effective to ner. in this paper, we will apply both kinds of embeddings besides character embeddings to further improve the performance. there are two phases in span generation model (sgm) network, which iteratively generate span representations for each layer by merging adjacent spans in a bottom-up and a top-down manner, respectively. "p" denotes the padding when employing cnn and the outputs "*t spans" denote the representations of candidate spans at each layer, where is set to 6 in the above example. (c) during hierarchical label prediction (hlp), the same labeling network, e.g., 4, is employed in each layer. (d) output entities. different layers are displayed in different colors. ## our proposed method prior hypergraph-based and span-based methods for nner suffer from ambiguous schemas or errors propagation in complex sentences, thus layered-based models are proposed to decompose the problem into several smaller subtasks. however, for nner, learning effective representations and modeling inter-entity dependencies is still a substantial challenge. in this study, we hypothesis that nested entities in the same context are complementary and the text representation at multi-level could improve nner. given an input sentence s is composed of a sequence of words, i.e., @xmath0, 2 , . . . , | s | }, where |s| denotes the number of words. for the nner task, each word is associated with multiple bio2-format 1 labels @xmath1, y 2 , . . . , y }, where denotes the maximum nesting depth. note that @xmath2, a word is associated with one categorical label, which is regarded as flat ner. therefore, we formulate nner as a multi-layer prediction problem. specifically, the topmost layer is processed as flat ner, and other layers merely using -{ } and labels to recognize complete entities from text spans. for each layer, it modeled as sequence labeling, that is, : e 1 e 2 • • • @xmath3, i-, 0 indicate the beginning, intermediate, and outer position of an entity, respectively, and indicates a categorical label takes from a pre-defined tag set, e.g., person, location, or organization. in the following subsections, we will introduce our proposed hitrans, which consists of three parts: multi-level representation, span generation model, and hierarchical label prediction. figure 2 gives an overview of our framework. ## conclusion this paper presents a novel hitrans framework, which learns effective span representations for label prediction of nested entities in a hierarchical manner. the proposed framework iteratively generates candidate span representations by aggregating adjacent features and further refines them based on a bottom-up and top-down transformer network. moreover, a candidate span is further recognized as a named entity sequentially, leveraging the semantic dependency of adjacent spans.