Attention Is All You Need for C hinese Word Segmentation (2020)

Abstract: Taking greedy decoding algorithm as it should be, this work focuses on further strengthening the model itself for Chinese word segmentation (CWS), which results in an even more fast and more accurate CWS model. Our model consists of an attention only stacked encoder and a light enough decoder for the greedy segmentation plus two highway connections for smoother training, in which the encoder is composed of a newly proposed Transformer variant, Gaussian-masked Directional (GD) Transformer, and a biaffine attention scorer. With the effective encoder design, our model only needs to take unigram features for scoring. Our model is evaluated on SIGHAN Bakeoff benchmark datasets. The experimental results show that with the highest segmentation speed, the proposed model achieves new state-of-the-art or comparable performance against strong baselines in terms of strict closed test setting.

## introduction chinese word segmentation (cws) is the task of delimiting word boundaries in a sentence, as a basic and essential task for chinese and many other east asian languages which are written without explicit word delimiters, and thus different from alphabetical languages like english. learning from an annotated corpus with segmentation, the cws task may be generally modeled as a decoder which performs segmentation based on a scoring module in terms of contextual feature based representations. markov models such as @xcite and @xcite depend on the maximum entropy model or maximum entropy markov model both with viterbi decoding. besides, conditional random field (crf) or semi-crf for sequence labeling has been used for both traditional and neural models though with different representations @xcite @xcite . recent neural cws research have been concerned about the following three perspectives @xcite . decoder. as cws is a kind of structure learning task, the decoder module generally determines which type of detailed algorithm should be adopted for segmentation, also it may limit the capability of defining feature. as shown in encoder. practice in various natural language processing tasks has shown that effective representation is essential to the performance improvement. for such a module in neural models, it is more than an encoder now, which is regarded as the most improvement perspective against traditional models. thus for better cws, it is crucial to encode the input character, word or sentence into a distinguishable representation. external resources and pre-trained embedding. using external resource such as pre-trained embeddings or language representation provides traditional models neural models decoding algorithm greedy model -ours greedy markov model (ng @xcite , @xcite mmtnn: @xcite ) @xcite , lstm: @xcite viterbi sequence labeling model crf: @xcite , semi-crf: @xcite , @xcite cnn+crf: @xcite , bilstm+crf: @xcite general graph model @xcite lstm+gcnn: @xcite , lstm+gcnn: @xcite ) @xcite beam search ci-2, ci-1, ci, ci+1, ci+2 -( chen et al., 2015) c0, c1, . . . , ci, ci+1, ci+2 word based @xcite , . . . c in wj-1, wj, wj+1 wj-1, wj, wj+1 @xcite c0, c1, . . . , ci w0, w1, . . . , wj an alternative for performance improvement other than designing better models @xcite . sighan bakeoff therefore defines two types of evaluation settings, closed test limits all the data for learning not to be beyond the given training set, while open test does not take this limitation @xcite . this work will focus on the closed test setting by finding a better model design for further cws. generally speaking, both the major difference between traditional and neural models, and what mostly distinguishes the neural models are about the way to represent input sentences, while the options of decoding algorithms are bounded to how to formalize the cws into a structural learning task. as shown in to make the decoding practical, the beam width b has to be carefully tuned for a tradeoff between accuracy and efficiency: a larger b will make the learning and segmentation extremely slow, while a small b cannot sufficiently guarantee the segmentation performance. however, there has long been a unheeded observation that good enough representations can offer good enough segmentation even though only using a greedy segmentation algorithm. @xcite create a topline evaluation by using only using vocabulary from test set to perform a greedy segmentation (maximum matching), which yields around 99% f-scores on all datasets. for neural models, @xcite verify that if the representations are good enough, beam width 1 can still give state-of-the-art performance compared to their early model with a full beam search decoder in @xcite . therefore, undertaking a fixed greedy segmentation algorithm, this paper only focuses on more effective encoder design for even better representation. our model only consists of attention mechanisms as building blocks plus two highway connections via a virtual hidden layer for smooth training. our model is simply stacked by a variant of transformer encoder @xcite and a biaffine attention scorer @xcite . empowered by the self-attention mechanism, the transformer has been good at capturing long-range dependencies for input sentence. we propose gaussian-masked directional (gd) multihead attention to facilitate the learning of localness, position and directional information for cws, so that we have the proposed gd-transformer. with our further improved encoder, our model uses only simple unigram features to generate representation of sentences for scoring. our model will be strictly evaluated on benchmark datasets from sighan bakeoff shared task in terms of closed test setting, and experimental results show that our model achieves new state-of-the-art. the technical contributions of this paper can be summarized as follows. neural word segmentation has been widely used to minimize the efforts in feature engineering. @xcite first introduce the neural model into cws with sliding-window based sequence labeling. @xcite use lstm to enhance the learning of long distance information. however, introducing neural models themselves does not really introduce substantial performance improvement in terms of strict closed test of sighan bakeoff according to @xcite . most researchers actually seek help from joint learning, extra learning resources including dictionaries, pre-trained embedding, deeper information extracted from training set and so on. (1) for joint learning, @xcite explore a joint model that performs segmentation, pos-tagging and chunking simultaneously. @xcite present a joint model to enhance the segmentation of chinese microtext by performing cws and informal word detection simultaneously. (2) for extra resources or clues, @xcite propose to incorporate unlabeled and partially-labeled data. only a few researches are known for concentrating on strengthening the model itself. to accommodate more rich features through a more broadly structural modeling @xcite propose a neural framework that eliminates context windows and utilize complete segmentation history. @xcite propose a character-based convolutional neural model to capture n-gram features automatically and an effective approach to incorporate word embeddings. @xcite further improve the model in @xcite and show that a greedy segmenter can perform fast and accurately in terms of only presenting effective representations. this work follows this line of research by offering even strengthened model design from simple idea, including the least building block type for encoder (attention only), the least feature type for scoring (unigram only) and the least computational complexity for decoding (greedy segmentation). the original transformer encoder consists of a stack of n identical layers and each layer has one multi-head self-attention layer and one positionwise fully connected feed-forward layer @xcite . one residual connection is around two sub-layers and followed by layer normalization. several variants are proposed to enhance ability of capturing the localness relationship. @xcite propose an effcient way to incorporate relative and absolute position representation. @xcite cast localness modeling as a learnable gaussian bias to enhance the ability of capturing useful local context. @xcite propose a transformer with gaussian-weighted self-attention to improved speech-enhancement performance. @xcite propose using syntax to guide the text modeling based on selfattention network sponsored transformer-based encoder. transformer based pre-trained language backward encoder embedding positional encoding input sentence forward encoder central encoder sum sum forward encoder central encoder backward encoder r c r f r b v f v b sum sum biaffine attention scorer biaffine attention scorer decoding layer decoding layer segmentation highway-i sum sum sum gd-transformer hired r fʼ r cʼ r bʼ r fʼ r cʼ r bʼ v fʼ v bʼ highway-o models have become a standard performance enhancement means for various nlp tasks @xcite . ## models our model for cws task is composed of an encoder to represent the input and a decoder based on the encoder to perform actual segmentation. figure 1 is the architecture of our model. the model feeds sentence into encoder. embedding captures the vector e of the input character sequences of c. the encoder maps vector sequences of e to two sequences of vector which are v b and v f as the representation of sentences. with v b and v f , the biaffine scorer scores each segmentation gaps which makes our decoder is as simple as one layer, using a threshold to directly and greedily predict every word boundaries of the input. ## conclusion for chinese word segmentation, upholding the belief that a better representation is all we need and thus taking a greedy decoder for fast segmentation as the basis, we only focus on the encoder design and propose an attention mechanism only based cws model. our model uses the proposed gd-transformer encoder to take sequence input and biaffine attention scorer to directly predict the word boundaries. to improve the ability of capturing the localness and directional information, gaussianmasked directional multi-head attention in the gd-transformer replaces the standard self-attention in the original transformer. with powerful enough encoding ability, our model only needs unigram features for scoring instead of various n-gram features in previous work. our model is evaluated on standard benchmark sighan bakeoff datasets, which shows not only our model performs segmentation faster than any previous models but also gives new higher or comparable segmentation performance against previous state-of-the-art models.