Recently , neural network models for natural language processing tasks have been increasingly focused on for their ability of alleviating the burden of manual feature engineering . However , the previous neural models can not extract the complicated feature compositions as the traditional methods with discrete features . In this paper , we propose a gated recursive neural network ( GRNN ) for Chinese word segmentation , which contains reset and update gates to incorporate the complicated combinations of the context characters . Since GRNN is relative deep , we also use a supervised layer-wise training method to avoid the problem of gradient diffusion . Experiments on the benchmark datasets show that our model outperforms the previous neural network models as well as the state-of-the-art methods . Unlike English and other western languages , Chinese do not delimit words by white-space . Therefore , word segmentation is a preliminary and important pre-process for Chinese language processing . Most previous systems address this problem by treating this task as a sequence labeling problem and have achieved great success . Due to the nature of supervised learning , the performance of these models is greatly affected by the design of features . These features are explicitly represented by the different combinations of context characters , which are based on linguistic intuition and statistical information . However , the number of features could be so large that the result models are too large to use in practice and prone to overfit on training corpus . Figure 1 : Illustration of our model for Chinese word segmentation . The solid nodes indicate the active neurons , while the hollow ones indicate the suppressed neurons . Specifically , the links denote the information flow , where the solid edges denote the acceptation of the combinations while the dashed edges means rejection of that . As shown in the right figure , we receive a score vector for tagging target character " 地 " by incorporating all the combination information . Recently , neural network models have been increasingly focused on for their ability to minimize the effort in feature engineering . Collobert et al . ( 2011 ) developed a general neural network architecture for sequence labeling tasks . Following this work , many methods ( Zheng et al . , 2013 ; Pei et al . , 2014 ; Qi et al . , 2014 ) applied the neural network to Chinese word segmentation and achieved a performance that approaches the state-of-the-art methods . However , these neural models just concatenate the embeddings of the context characters , and feed them into neural network . Since the concatenation operation is relatively simple , it is difficult to model the complicated features as the traditional discrete feature based models . Although the complicated interactions of inputs can be modeled by the deep neural network , the previous neural model shows that the deep model can not outperform the one with a single non-linear model . Therefore , the neural model only captures the interactions by the simple transition matrix and the single non-linear transformation . These dense features extracted via these simple interactions are not nearly as good as the substantial discrete features in the traditional methods . In this paper , we propose a gated recursive neural network ( GRNN ) to model the complicated combinations of characters , and apply it to Chinese word segmentation task . Inspired by the success of gated recurrent neural network ( Chung et al . , 2014 ) , we introduce two kinds of gates to control the combinations in recursive structure . We also use the layer-wise training method to avoid the problem of gradient diffusion , and the dropout strategy to avoid the overfitting problem . Figure 1 gives an illustration of how our approach models the complicated combinations of the context characters . Given a sentence " 雨 ( Rainy ) 天 ( Day ) 地面 ( Ground ) 积水 ( Accumulated water ) " , the target character is " 地 " . This sentence is very complicated because each consecutive two characters can be combined as a word . To predict the label of the target character " 地 " under the given context , GRNN detects the combinations recursively from the bottom layer to the top . Then , we receive a score vector of tags by incorporating all the combination information in network . The contributions of this paper can be summarized as follows : • We propose a novel GRNN architecture to model the complicated combinations of the context characters . GRNN can select and preserve the useful combinations via reset and update gates . These combinations play a similar role in the feature engineering of the traditional methods with discrete features . • We evaluate the performance of Chinese word segmentation on PKU , MSRA and CTB6 benchmark datasets which are commonly used for evaluation of Chinese word segmentation . Experiment results show that our model outperforms other neural network models , and achieves state-of-the-art performance . In this paper , we propose a gated recursive neural network ( GRNN ) to explicitly model the combinations of the characters for Chinese word segmentation task . Each neuron in GRNN can be regarded as a different combination of the input characters . Thus , the whole GRNN has an ability to simulate the design of the sophisticated features in traditional methods . Experiments show that our proposed model outperforms the state-of-the-art methods on three popular benchmark datasets . Despite Chinese word segmentation being a specific case , our model can be easily generalized and applied to other sequence labeling tasks . In future work , we would like to investigate our proposed GRNN on other sequence labeling tasks .

Challenge: Existing neural network-based models with embedding concatenation cannot process complex discrete features, and larger models have problems such as overfitting, and underperforming simple linear systems.
Approach: They propose a gated recursive neural network with two novel gating mechanisms trained with layer-wise training method and dropout to mitigate gradient diffusion and overfitting.
Outcome: The proposed gated recursive neural network outperforms traditional linear models with feature engineering and also neural network-based models on the task of Chinese word segmentation.