Adaptive Threshold Selective Self-Attention for C hinese NER (2022)

Abstract: Recently, Transformer has achieved great success in Chinese named entity recognition (NER) owing to its good parallelism and ability to model long-range dependencies, which utilizes self-attention to encode context. However, the fully connected way of self-attention may scatter the attention distribution and allow some irrelevant character information to be integrated, leading to entity boundaries being misidentified. In this paper, we propose a data-driven Adaptive Threshold Selective Self-Attention (ATSSA) mechanism that aims to dynamically select the most relevant characters to enhance the Transformer architecture for Chinese NER. In ATSSA, the attention score threshold of each query is automatically generated, and characters with attention score higher than the threshold are selected by the query while others are discarded, so as to address irrelevant attention integration. Experiments on four benchmark Chinese NER datasets show that the proposed ATSSA brings 1.68 average F1 score improvements to the baseline model and achieves state-of-the-art performance.

## introduction named entity recognition (ner) aims to identify named entities in the given text, including persons, locations, organizations, etc. it plays an important role in downstream natural language processing (nlp) tasks such as information retrieval @xcite , relation extraction @xcite and question answering @xcite . compared with english ner, chinese ner is more difficult since there is no natural delimiter between words in chinese sentences. to integrate word information related to each character and avoid error propagation of word segmentation, word-character lattice is first applied to chinese ner in lattice-lstm @xcite . however, the rnn-based model is hard to model long-range dependencies, and has poor computational efficiency. recently, transformer @xcite is widely used in chinese ner, which utilizes self-attention to encode context. since vanilla transformer lacks directionality, tener @xcite combines transformer encoder with direction-aware position encoding to catch directions between characters. flat @xcite further exploits span relative position encoding to convert the lattice structure into a flat structure. although transformer-based models perform well on chinese ner, they have a limitation that the fully connected self-attention drives queries to attend to all characters of the inputs, making irrelevant character-level attention integration, which leads to entity boundaries being misidentified. we argue that, in ner task, entities only depend on the most relevant characters, while others should be ignored. as shown in figure 1 , since the attention of the character "上 (shang)" attends to all characters, irrelevant character-level information of "大 (big)" and "众 (zhong)" is integrated, which may lead the model to misidentify "上海大众 (shanghai volkswagen)" as an organization entity rather than "上 海 (shanghai)" as a geopolitical entity. to avoid irrelevant attention integration, researchers have investigated sparse self-attention. @xcite introduce local and strided patterns to select keys for queries. since the sparsity pattern is fixed, it may lack transferability. as alternatives, sparsemax @xcite and entmax @xcite collect most relevant keys via dedicated forward and backward algorithms, which incurs large computational costs. to improve efficiency, explicit sparse transformer @xcite gathers a fixed number of keys with highest attention score through top-k selection at the cost of flexibility. however, none of above works have been applied to chinese ner. in this paper, we propose an adaptive threshold selective self-attention (atssa) mechanism to avoid irrelevant character-level attention integration for chinese ner. this mechanism is datadriven which assigns a various attention mass to each query in different heads parallelly. specifically, we generate an attention score threshold for each query of input sequence via a matching matrix @xcite automatically. then, keys whose attention score exceeds the threshold are selected, while others are discarded. in the end, the softmax function only performs on those selected query-key pairs, thereby improving attention to relevant keys while avoiding irrelevant characterlevel attention integration. as shown in figure 1 , the character "上(shang)" only attends to relevant characters in "上海(shanghai)", while attention to irrelevant characters in "大众(public)" is discarded. we take the advantage of flat @xcite ) in lexicon fusion and apply atssa to it. our code will be released at https://github.com/ hubiao20/atssa-ner . the contributions of this paper can be summarized as: ## background our proposed atssa is applied to flat @xcite , thus, we briefly introduce the structure of flat in this section. flat treats the characters and matched words of the input sequence as spans, and designs an ingenious relative position encoding of spans for lattice structure. let head[i] and tail[i] denote the head and tail position of span x i , four relative distances are used to indicate the relation between x i and x j : then, the span relative position encoding can be calculated by a non-linear transformation of the four distances: with the span relative position encoding, the dot-product attention can be calculated as: ## adaptive threshold selective self-attention intuitively, keys with high attention score are more relevant than others to the correspondin query and attention should be centralized on these keys. however, in the vanilla transformer, fully connected self-attention distributes attention to the entire context, even those irrelevant keys, which weakens the attention to critical ones. to this end, we propose the adaptive threshold selective self-attention, in which keys with attention score lower than threshold of the corresponding query are discarded before the softmax operation. ## experiments experiments are carried out on chinese ner datasets across different domains. f1-score (f1) is exploited to evaluate the performance of the model. all experiments are conducted on a single nvidia titan rtx gpu. ## conclusion in this paper, we propose an adaptive thresholdselective self-attention mechanism to dynamically select keys for queries in parallel to enhance the architecture of transformer, which avoids the integration of irrelevant character-level information when encoding context. this data-driven mechanism maintains computational efficiency without losing flexibility. based on flat, we apply it to chinese ner and achieve state-of-the-art performance on four benchmark chinese ner datasets. in future work, we will adapt atssa to different kinds of nlp tasks, such as text classification and natural language inference.