Syntax-guided Contrastive Learning for Pre-trained Language Model (2022)

Abstract: Syntactic information has been proved to be useful for transformer-based pre-trained language models. Previous studies often rely on additional syntax-guided attention components to enhance the transformer, which require more parameters and additional syntactic parsing in downstream tasks. This increase in complexity severely limits the application of syntax-enhanced language model in a wide range of scenarios. In order to inject syntactic knowledge effectively and efficiently into pre-trained language models, we propose a novel syntax-guided contrastive learning method which does not change the transformer architecture. Based on constituency and dependency structures of syntax trees, we design phrase-guided and tree-guided contrastive objectives, and optimize them in the pre-training stage, so as to help the pre-trained language model to capture rich syntactic knowledge in its representations. Experimental results show that our contrastive method achieves consistent improvements in a variety of tasks, including grammatical error detection, entity tasks, structural probing and GLUE. Detailed analysis further verifies that the improvements come from the utilization of syntactic information, and the learned attention weights are more explainable in terms of linguistics.

## introduction pre-trained transformer-based neural language models (lms), such as bert @xcite and roberta @xcite , have achieved remarkable results in a variety of nlp tasks @xcite . however, many studies have found that these lms do not encode enough syntactic knowledge in their learned representations @xcite @xcite . as it is widely acknowledged that structural information is very important for nlp @xcite @xcite , there is an increasing interest in improving pre-trained lms by using syntactic information. most of these works enhance pre-trained lms by adding syntax-driven attention components to the transformer @xcite @xcite . they use the added components to produce a syntax-aware representation, and inject this additional representation into the original one from the vanilla transformer, so as to get a final syntax-enhanced representation. although these works did bring improvements, the additional syntax-aware layers obviously increase application inconvenience and computation complexity, as they need to parse the input text during testing and require more neural parameters. moreover, the performance of such explicit method depends on the parsing quality of test data @xcite . there are also some efforts on incorporating syntaxrelated objectives into the pre-training stage, such as syntax head prediction @xcite and dependency distance prediction @xcite . however, these predictive pre-training tasks often fail to improve performance alone and need to work together with the additional attention components @xcite . overall, it is still an open challenge to effectively and efficiently incorporate syntactic information into pre-trained lms. in order to address the above problems, we propose syntax-guided contrastive language model (synclm). based on contrastive learning, syn-clm uses syntactic information to create contrastive positive and negative examples, and uses them to help the pre-trained lm to learn rich syntactic knowledge through contrastive learning method. synclm only adds contrastive objectives in the pre-training stage, ensuring an effective and efficient utilization of syntax. specifically, based on constituent and dependency structures of syntax trees, we propose phraseguided and tree-guided contrastive objectives for pre-training, as shown in figure 1 . the constituent structure represents the grouping of words into phrases within an input according to constituency grammar @xcite . inspired by recent studies @xcite which prove that lm's attention heads exhibit syntactic structure akin to constituency grammar, in order to better recognize phrases from attentions, we propose the phrase-guided contrastive objective to enhance attention learning by maximizing the similarity of attention distributions between words in the same phrase. the dependency structure further encodes the binary head-dependent relations between words, and the root node aggregates the semantic information of the whole structure from all its descendant words. to make the root node attend to its descendant nodes, we propose the treeguided contrastive objective to enhance word representations by maximizing the similarity between the representation obtained from all tokens and that obtained from syntactically related tokens. the two contrastive objectives are jointly optimized during pre-training, so as to inject syntactic knowledge into pre-trained lms. in summary, our contributions are as following: our code foot_0 will be released. ## related work we first review studies on analyzing the linguistic knowledge learned by pre-trained lms, and then we will introduce recent researches on incorporating linguistic knowledge into pre-trained lms. ## methodology in this section, we first describe the two new contrastive learning objectives in our synclm. then we introduce our pre-training framework and implementation details. ## experiments first, we verify the effectiveness of synclm on several syntax-aware tasks, including grammatical error detection task (section 4.1) and entity tasks (section 4.2), which are often used for testing syntax pre-training models. then, we test the effectiveness of synclm on more general tasks by using glue benchmark (section 4.3). at last, detailed analysis is conducted to show the impact of incorporating syntactic knowledge (section 4.4). please note that ↑ in our reported results means statistically significant improvement over the baseline with p-@xmath0.05. besides, for fair comparison, we report continue training results ## conclusion to the best of our knowledge, this is the first work of leveraging contrasting learning to inject syntax knowledge into pre-trained lms. motivated by the properties of constituent and dependency structures of syntax, we design phrase-guided and tree-guided learning objectives to guide the learning of attention distributions and hidden representations in the transformer. through extensive experiments, we show that synclm consistently improves a wide range of tasks, from ged, entity tasks to gleu, which confirms the advantage of our syntax-guided contrastive learning. detailed analysis also shows that synclm does incorporate rich syntax knowledge and learn explainable attention weights.