Hero-Gang Neural Model For Named Entity Recognition (2022)

Abstract: Named entity recognition (NER) is a fundamental and important task in NLP, aiming at identifying named entities (NEs) from free text. Recently, since the multi-head attention mechanism applied in the Transformer model can effectively capture longer contextual information, Transformer-based models have become the mainstream methods and have achieved significant performance in this task. Unfortunately, although these models can capture effective global context information, they are still limited in the local feature and position information extraction, which is critical in NER. In this paper, to address this limitation, we propose a novel Hero-Gang Neural structure (HGN), including the Hero and Gang module, to leverage both global and local information to promote NER. Specifically, the Hero module is composed of a Transformer-based encoder to maintain the advantage of the self-attention mechanism, and the Gang module utilizes a multi-window recurrent module to extract local features and position information under the guidance of the Hero module. Afterward, the proposed multi-window attention effectively combines global information and multiple local features for predicting entity labels. Experimental results on several benchmark datasets demonstrate the effectiveness of our proposed model.

## introduction named entity recognition (ner) is one of the most important and fundamental research topics in natural language processing (nlp), which recognizes named entities (nes), such as person, location, disease from raw text. ner has attracted substantial attention in the past decades owing to its importance in downstream tasks, e.g., knowledge † corresponding author. 1 our code is released at https://github.com/ jinpeng01/hgn . in the early stage, the popular methods for solving ner are some traditional machine learning methods, e.g., hidden markov model (hmm) @xcite and conditional random field (crf) @xcite , which require extracting features manually, making the process inefficient and time-consuming. with the breakthrough of recurrent neural networks (rnn) in nlp, long short-term memory (lstm) @xcite and gated recurrent unit (gru) @xcite have become mainstream methods for this task and have achieved promising results since neural networks can automatically extract features from the sequence and also take each token's position information into consideration @xcite @xcite . nevertheless, rnn fails to perform well with long sequences due to the gradients exploding and vanishing. in recent years, transformer-based models @xcite have become mainstream methods because these models are able to capture long-term dependencies with the help of multi-head attention and thus provide better global context information, especially for long sequences @xcite . however, these transformer-based models usually are insensitive to the local context since the representation of each token is computed by the canonical point-wise dot-product self-attention @xcite . besides, although some studies @xcite @xcite have been proposed to inject position information into transformer, they are still inadequate to help transformer obtain appropriate position information @xcite . in other words, the self-attention mechanism is effective in overcoming the constraints of rnn from the perspective of long-sequence con- thus, to alleviate the shortcomings in rnn and transformers while maintaining their respective strengths, in this paper, we propose a novel hero-gang neural model to leverage both global and local contextual information to improve ner. in doing so, on the one hand, we utilize a transformerbased sequence encoder (i.e., hero module) to extract effective global contextual information with the help of the self-attention mechanism. on the other hand, a multi-window recurrent unit (i.e., gang module) is applied to extract local features from multiple sub-sequences under the guidance of the extracted global information. afterward, we propose to use multi-window attention to elaborately combine global and local contextual features. the performance of our proposed model significantly outperforms the strong baseline models on several ner benchmark datasets (including both general and biomedical domains) and achieves new state-of-the-art results on some datasets. ## method ner is usually performed as a sequence labeling problem. in detail, given a sequence of @xmath0, x 2 , ..., x n with n tokens, we aim to learn a function that maps the input sequence into another one with the corresponding label ŷ = ŷ1 , ŷ2 , ŷ3 , ..., ŷn in the same length. as summarized in figure 1 , the transformer-based models (e.g., bert @xcite , xlnet @xcite ) are regarded as the hero module to model the entire sentence for global sequence information extraction and the gang module is responsible for local and relative position information extraction. afterward, we employ the multi-window attention to elaborately combine these different features (i.e., features extracted from the hero and gang modules), which is then used to predict labels for each token. therefore, the aforementioned process can be formulated as: ## conclusion in this paper, we propose a novel hero-gang neural (hgn) structure to effectively combine global and local features for enhancing ner. in detail, the hero module aims to capture global understanding by a transformer-based encoder, which is then used to guide the gang to extract local features and relative position information through a multiwindow recurrent module. afterward, we utilize the multi-window attention to elaborately combine the global information and local features for enhancing representations that are then used to predict the entity label for each token. empirically, our proposed model achieves new state-of-the-art results on several ner benchmark datasets, including both general and biomedical domains. besides, we compare different structures to construct the gang model and investigate the effect of the number of sliding windows, which further illustrates the effectiveness of our proposed model.