Porous Lattice Transformer Encoder for C hinese NER (2020)

Abstract: Incorporating lexicons into character-level Chinese NER by lattices is proven effective to exploitrich word boundary information. Previous work has extended RNNs to consume lattice inputsand achieved great success. However, due to the DAG structure and the inherently unidirectionalsequential nature, this method precludes batched computation and sufficient semantic interaction. In this paper, we propose PLTE, an extension of transformer encoder that is tailored for ChineseNER, which models all the characters and matched lexical words in parallel with batch process-ing. PLTE augments self-attention with positional relation representations to incorporate latticestructure. It also introduces a porous mechanism to augment localness modeling and maintainthe strength of capturing the rich long-term dependencies. Experimental results show that PLTEperforms up to 11.4 times faster than state-of-the-art methods while realizing better performance. We also demonstrate that using BERT representations further substantially boosts the performanceand brings out the best in PLTE.

## introduction named entity recognition (ner) is a fundamental task in natural language processing (nlp), which aims to automatically discover named entities and identify their corresponding categories from plain text. nlp tasks such as information retrieval @xcite , relation extraction @xcite and entity linking @xcite require the ner as one of their preprocessing components. recent studies show that english ner models have achieved improved performance by integrating character information into word representations based on sequence labeling. different from english ner, east asian languages including chinese are written without explicit word boundary. one intuitive way to solve this problem is to segment the input sentences into words first, and then to apply word sequence labeling @xcite . however, such methods suffer from error propagation between these two subtasks. to overcome this limitation, efforts have been devoted to incorporating word information by leveraging lexicon features and gazetteers @xcite @xcite . as recent state-of-the-art (sota) lattice-based method, @xcite integrated matched lexical words information into character sequence with a directed acyclic graph (dag) structure using lattice lstm. while obtaining promising results, this model faces two challenges. first, as an extension to the non-parallelizable sequential lstm to a dag structured model, lattice lstm is restricted to preprocess one character at a time, which can make it infeasibly to deploy. second, due to the inherently unidirectional sequential nature, lattice lstm fails to incorporate the word-level semantics into the representation of the characters except for the last character in each word, despite that such information can be crucial for character-level sequence tagging. taking the sentence in figure 1 as an example, lattice lstm decodes the information of the lexical word "南京市(nanjing city)" to "市(city)" but skips restricted by the unidirectional sequential nature, lattice lstm cannot model the semantic interaction between word "南京市(nanjing city)" and its constituent characters "南(south)" and "京(capital)", resulting in the loss of crucial information for tagging. besides, lattice lstm cannot perform batched computation due to the directed acyclic graph input structure. the other two inside characters "南(south)" and "京(capital)", although the semantics and boundary information of "南京市(nanjing city)" can be useful knowledge for predicting the tag of "南(south)" as "b-loc". in this paper, we address these issues by considering a novel porous lattice transformer encoder (plte). inspired by previous research on machine translation @xcite , which integrated lattice-structured inputs into self-attention models, we propose a lattice transformer encoder for chinese ner by introducing lattice-aware self-attention, which borrows the idea from the relative positional embedding @xcite to make self-attention aware of the relative position information in lattice structure. considering that self-attention network calculates attention weights between each pair of tokens in a sequence regardless of their distance, we simply concatenate all the characters and lexical words as input to consume lattices without resorting to the dag structure. in this way, characters coupled with lexical words can be processed in batches. a lexical word representation is allowed to build a direct relation with the included characters by lattice-aware self-attention, thus addressing the second issue. some work @xcite demonstrates that self-attention benefits from locality modeling, especially for the ner task. as we can see from the example in figure 1 , the word "位 于(locates in)" is the immediate and most obvious feature to guide the neighboring character "桥(bridge)" to be identified as "e-loc" instead of "e-per", while "中国(china)" has no contribution to this decision. given this observation, we further introduce a novel porous mechanism to enhance the local dependencies among neighboring tokens. the key insight is to modify the self-attention architecture by replacing the fully-connected topology with a pivot-shared structure. in this particular, every two non-neighboring tokens are connected by a shared pivot node to strengthen the dependency for two neighboring tokens. experimental results on four datasets demonstrate that our model performs up to 11.4 times faster than baselines and achieves better performance. furthermore, we show that our model can be easily integrated into the pre-trained language model such as bert @xcite , and combining them further improves the state of the art. in summary, this paper makes the following contributions: (1) we investigate lattice transformer encoder for chinese ner, which is capable of handling lattices in batch mode and capturing dependencies between characters and matched lexical words. (2) we revise lattice-aware attention distribution via a porous mechanism, which enhances the ability of capturing useful local context. (3) experimental results show that the proposed model is effective and efficient. the source code of this paper can be obtained from https://github.com/strawberryx/plte . our work is in line with ner models based on neural networks and lattice transformer models. huang et al. (2015) proposed a bilstm-crf model for ner and achieved strong performance. santos and guimaraes (2015) used word-and character-level representations based on the charwnn deep neural network. lample et al. (2016) designed a character lstm and word lstm for ner. compared to our work, these word-based methods suffer from segmentation errors. to avoid segmentation errors, most recent ner models are built upon character sequence labeling. peng and dredze (2015) proposesd a joint training objective for three types of neural embeddings to better recognize entity boundary. lu et al. (2016) presented a position-sensitive skip-gram model to learn multi-prototype chinese character embeddings. he and sun (2017a) took the positional character embeddings into account. although these methods achieve promising performance, they ignore word information lying in character sequence. some work exploits rich word boundary and semantic information in character sequence. cao et al. (2018) applied an adversarial transfer learning framework to integrate the task-shared word boundary information into chinese ner. liu et al. (2019) explored four different strategies for word-character lstm. gui et al. (2019a) proposed a cnn-based ner model that incorporates lexicons using a rethinking mechanism. recent state-of-the-art methods exploit lattice-structured models to integrate latent word information into character sequence, which has been proven effective on various nlp tasks @xcite . specifically, @xcite utilized the lattice lstm to leverage explicit word information over character sequence labeling. based on this method, @xcite al. (2019b) and sui et al. (2019) formulated the lattice structure as a graph and leveraged graph neural networks (gnns) to integrate lexical knowledge. however, for the ner task, coupling pre-trained language models such as bert @xcite with gnns and fine-tuning them can be non-trivial. lattice transformer has been exploited in nmt @xcite , as well as speech translation @xcite . compared with existing work, our proposed porous lattice transformer encoder is different in both motivation and structure. we revise the fully-connected attention distribution with a pivot-shared structure via the porous mechanism to enhance the local dependencies among neighboring tokens. 1 to our knowledge, we are the first to design a lattice transformer for chinese ner. ## background in this section, we first briefly review the self-attention mechanism, then move on to current lattice transformer that our plte model is built upon. ## models the overall structure of our model is shown in figure 2 (a), which consists of 3 main components, lattice input layer, porous lattice transformer encoder and bigru-crf decoding. ## experiments we conduct experiments to investigate the effectiveness of our proposed plte method across different domains. standard precision (p), recall (r) and f1-score (f1) are used as evaluation metrics. ## conclusion we presented plte, a porous lattice transformer encoder which incorporates lexicons into character-level chinese ner. plte enables the interaction between the matched lexical words and their constituent characters, and proceeds in batches with the lattice-aware self-attention. it also learns a porous attention distribution to enhance the ability of localness modeling. we evaluate the proposed model on four chinese ner datasets. using word2vec embeddings, our plte outperforms various baselines and performs up to 11.4 times faster than previous lattice-based method. switching to bert representations, plte achieves more significant performance gain than existing methods. there are multiple venues for future work, where one promising direction is to apply our model to the pre-training procedure of chinese transformer language models.