Modularized Syntactic Neural Networks for Sentence Classification (2020)

Abstract: This paper focuses on tree-based modeling for the sentence classification task. In existing works, aggregating on a syntax tree usually considers local information of sub-trees. In contrast, in addition to the local information, our proposed Modularized Syntactic Neural Network (MSNN) utilizes the syntax category labels and takes advantage of the global context while modeling sub-trees. In MSNN, each node of a syntax tree is modeled by a label-related syntax module. Each syntax module aggregates the outputs of lower-level modules, and finally, the root module provides the sentence representation. We design a tree-parallel mini-batch strategy for efficient training and predicting. Experimental results on four benchmark datasets show that our MSNN significantly outperforms previous state-of-the-art tree-based methods on the sentence classification task.

## introduction text classification is an important and fundamental problem in natural language processing (nlp). with the increasing spread of the internet, there are numerous applications of classification of short texts with only one sentence, for example, classifying questions according to what product or which part of the product architecture the question regards, sentiment analysis of customer reviews or tweets, and fast category detection based on news titles. different from document classification, in which there are more topic words and features of writing styles, a single sentence contains limited information. thus, understanding the meaning of a sentence is vital. although sequential models like long short term memory (lstm) @xcite and gated recurrent units (gru) @xcite have been widely used and provide excellent performances, it is hard for them to capture the syntactic information, which is essential for understanding sentences @xcite . to utilize the syntactic information, some works proposed models taking parse trees or dependency trees as inputs @xcite @xcite @xcite . previous researchers have empirically verified that these methods help to model sentences @xcite . however, to improve the model efficiency and simplify the implementation, these methods binarize trees @xcite so that they can be traversed by recursive neural networks (rvnn) or as a sequence by rnn. although some models, such as tree-lstm @xcite @xcite , theoretically support original parse trees, child nodes are simply summed, but the relationships among them are not modeled. the authors only conduct experiments on binary trees. binary trees weaken the syntactic information and conceal the relationships among nodes at the same level or different levels. latent tree is another way of modeling sentences, but it does not take full advantage of the syntactic information @xcite @xcite . current graph-based models specifically focus on the dependency tree @xcite . besides, these models do not consider the context information in the bottom-up aggregation. syntax category labels are also not fully utilized. however, they both can affect the meanings of words and phrases, which should be considered. in this work, a novel modularized syntactic neural network (msnn) is proposed to model syntax trees of sentences. each node in the tree is transformed into a syntax module in msnn. the num-ber of distinct syntax modules is the same as the number of distinct syntax category labels. a category label is the syntactic category of a subtree's root, e.g. "np", "vp", etc. the modules are used to build networks according to tree structures, and there is a one-to-one correspondence between tree structures and network structures. the parameters of modules corresponding to the same category labels are shared. note that there is no limitation to binary trees, and our implementation of treeparallel mini-batches based on the original parse trees provides excellent efficiency. each syntax module aggregates outputs of lower-level modules and outputs a representation vector of the sub-tree. syntax category labels and global context information are encoded to guide the propagation and better infer the meaning of the sentence. the root module finally outputs the sentence representation, which is further used for classification. we test msnn on four benchmark datasets, and the results show that it outperforms previous state-of-the-art methods. the main contributions of this work are listed as follows: ## modularized syntactic neural networks an example of modularized syntactic neural networks (msnn) is shown in figure 1 . ## conclusion in this work, a novel model msnn is proposed to model syntax trees for classification. it uses global context information and syntax category labels to help improve the modeling of sub-trees and thus better sentence representations. a tree-parallel mini-batch strategy is further designed for efficient running and support for non-binary trees. our future work will include conducting experiments on dependency trees and more nlp tasks.