Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP (2020)

Abstract: Syntax has been shown useful for various NLP tasks, while existing work mostly encodes singleton syntactic tree using one hierarchical neural network. In this paper, we investigate a simple and effective method, Knowledge Distillation, to integrate heterogeneous structure knowledge into a unified sequential LSTM encoder. Experimental results on four typical syntax-dependent tasks show that our method outperforms tree encoders by effectively integrating rich heterogeneous structure syntax, meanwhile reducing error propagation, and also outperforms ensemble methods, in terms of both the efficiency and accuracy.

## introduction integrating syntactic information into neural networks has received increasing attention in natural language processing (nlp), which has been used for a wide range of end tasks, such as sentiment analysis (sa) @xcite @xcite , neural machine translation (nmt) @xcite @xcite , language modeling @xcite @xcite , semantic role labeling (srl) @xcite @xcite , natural language inference (nli) @xcite ) and text classification @xcite . despite the usefulness of structure knowledge, most existing models use only a single syntactic tree, such as a constituency or a dependency tree. constituent and dependency representation for syntactic structure share underlying linguistic and computational characteristics, while differ also in various aspects. for example, the former focuses (3) (4) (1) figure 1 : an example illustrating the mutual benefit of constituency and dependency tree structures. (1) refers to the constituency tree structure, (2) indicates the semantic role labels, (3) refers to the example sentence, (4) represents the dependency tree structure. on revealing the continuity of phrases, while the latter is more effective in representing the dependencies among elements. by integrating the two representations from heterogeneous trees, the mutual benefit has been explored for joint parsing tasks @xcite @xcite @xcite . intuitively, complementary advantages from heterogeneous trees can facilitate a range of nlp tasks, especially syntax-dependent ones such as srl and nli. taking the sentence of figure 1 as example, where an example is shown from srl foot_0 task. in this case, the dependency links can locate the relations between arguments and predicates more efficiently, while the constituency structure can aggregate the phrasal spans for arguments, and guide the global path to the predicate. integrating the features of two structures can better guide the model to focus on the most suitable phrasal granularity (as circled by the dotted box), and also ensure the route consistency between the semantic objective pairs. in this paper, we investigate the knowledge distillation (kd) method, which has been shown to be effective for knowledge ensembling @xcite @xcite , for heterogeneous structure integration. specifically, we employ a sequential lstm as the student for distilling heterogeneous syntactic structures from various teacher tree encoders, such as gcn @xcite and @xcite . we consider output distillation, syntactic feature injection and semantic learning. in addition, we introduce an alternative structure injection strategy to enhance the ability of heterogeneous syntactic representations within the shared sequential model. the distilled structure-aware student model can make inference using sequential word inputs alone, reducing the error accumulation from external parsing tree annotations. we conduct extensive experiments on a wide range of syntax-dependent tasks, including semantic role labeling, relation classification, natural language inference and sentiment classification. results show that the distilled student outperforms tree encoders, verifying the advantage of integrating heterogeneous structures. the proposed method also outperforms existing ensemble methods and strong baseline systems, demonstrating its high effectiveness on structure information integration. ## method as shown in figure 2 , the overall architecture consists of a sequential lstm @xcite ) student, and several tree teachers for dependency and constituency structures. ## conclusion we investigated knowledge distillation on heterogeneous tree structures integration for facilitating nlp tasks, distilling syntactic knowledge into a sequential input encoder, in both output and feature level distillations. results on four representative syntax-dependent tasks showed that the distilled student outperformed all standalone tree models, as well as the commonly used ensemble methods, indicating the effectiveness of the proposed method. further analysis demonstrated that our method enjoys high robustness and efficiency.