FLAT : C hinese NER Using Flat-Lattice Transformer (2020)

Abstract: Recently, the character-word lattice structure has been proved to be effective for Chinese named entity recognition (NER) by incorporating the word information. However, since the lattice structure is complex and dynamic, the lattice-based models are hard to fully utilize the parallel computation of GPUs and usually have a low inference speed. In this paper, we propose FLAT: Flat-LAttice Transformer for Chinese NER, which converts the lattice structure into a flat structure consisting of spans. Each span corresponds to a character or latent word and its position in the original lattice. With the power of Transformer and well-designed position encoding, FLAT can fully leverage the lattice information and has an excellent parallel ability. Experiments on four datasets show FLAT outperforms other lexicon-based models in performance and efficiency.

## introduction named entity recognition (ner) plays an indispensable role in many downstream natural language processing (nlp) tasks @xcite . compared with english ner @xcite @xcite , chinese ner is more difficult since it usually involves word segmentation. recently, the lattice structure has been proved to have a great benefit to utilize the word information and avoid the error propagation of word segmentation @xcite . we can match a sentence with a lexicon to obtain the latent words in it, and then we get a lattice like in figure 1( @xcite . the lattice is a directed acyclic graph, where each node is a character or a latent word. the lattice includes a sequence of characters and potential figure 1 : while lattice lstm indicates lattice structure by dynamically adjusting its structure, flat only needs to leverage the span position encoding. in 1(c), , , denotes tokens, heads and tails, respectively. there are two lines of methods to leverage the lattice. (1) one line is to design a model to be compatible with lattice input, such as lattice lstm @xcite and lr-cnn @xcite . in lattice lstm, an extra word cell is employed to encode the potential words, and attention mechanism is used to fuse variable-number nodes at each position, as in figure 1 @xcite . lr-cnn uses cnn to encode potential words at different window sizes. however, rnn and cnn are hard to model long-distance dependencies @xcite , which may be useful in ner, such as coreference @xcite . due to the dynamic lattice structure, these methods cannot fully utilize the parallel computation of gpu. (2) another line is to convert lattice into graph and use a graph neural network (gnn) to encode it, such as lexicon-based graph network (lgn) @xcite and collaborative graph network (cgn) @xcite . while sequential structure is still important for ner and graph is general counterpart, their gap is not negligible. these methods need to use lstm as the bottom encoder to carry the sequential inductive bias, which makes the model complicated. in this paper, we propose flat: flat lattice transformer for chinese ner. transformer @xcite adopts fully-connected selfattention to model the long-distance dependencies in a sequence. to keep the position information, transformer introduces the position representation for each token in the sequence. inspired by the idea of position representation, we design an ingenious position encoding for the lattice-structure, as shown in figure 1(c ). in detail, we assign two positional indices for a token (character or word): head position and tail position, by which we can reconstruct a lattice from a set of tokens. thus, we can directly use transformer to fully model the lattice input. the self-attention mechanism of transformer enables characters to directly interact with any potential word, including self-matched words. to a character, its self-matched words denote words which include it. for example, in figure 1(a) , self-matched words of "药 (drug)" are "人和药 店(renhe pharmacy)" and "药店 (pharmacy)" @xcite . experimental results show our model outperforms other lexicon-based methods on the performance and inference-speed. our code will be released at https://github.com/leesureman/flatlattice-transformer . ## background in this section, we briefly introduce the transformer architecture. focusing on the ner task, we only discuss the transformer encoder. it is composed of self-attention and feedforward network (ffn) layers. each sublayer is followed by residual connection and layer normalization. ffn is a position-wise multi-layer perceptron with nonlinear transformation. transformer performs selfattention over the sequence by h heads of attention individually and then concatenates the result of h heads. for simplicity, we ignore the head index in the following formula. the result of per head is calculated as: the vanilla transformer also uses absolute position encoding to capture the sequential information. inspired by @xcite , we think commutativity of the vector inner dot will cause the loss of directionality in self-attention. therefore, we consider the relative position of lattice also significant for ner. ## conclusion and future work in this paper, we introduce a flat-lattice transformer to incorporate lexicon information for chinese ner. the core of our model is converting lattice structure into a set of spans and introducing the specific position encoding. experimental results show our model outperforms other lexiconbased models in the performance and efficiency. we leave adjusting our model to different kinds of lattice or graph as our future work.