Syntax-guided Localized Self-attention by Constituency Syntactic Distance (2022)

Abstract: Recent works have revealed that Transformers are implicitly learning the syntactic information in its lower layers from data, albeit is highly dependent on the quality and scale of the training data. However, learning syntactic information from data is not necessary if we can leverage an external syntactic parser, which provides better parsing quality with well-defined syntactic structures. This could potentially improve Transformer’s performance and sample efficiency. In this work, we propose a syntax-guided localized self-attention for Transformer that allows directly incorporating grammar structures from an external constituency parser. It prohibits the attention mechanism to overweight the grammatically distant tokens over close ones. Experimental results show that our model could consistently improve translation performance on a variety of machine translation datasets, ranging from small to large dataset sizes, and with different source languages.

## introduction although transformer doesn't have any inductive bias on syntactic structures, some studies have shown that it tends to learn syntactic information from data in its lower layers @xcite @xcite . given the pervasiveness of syntactic parsers that provides high quality parsing results with well-defined syntactic structures, transformers may not need to re-invent this wheel if grammar structures could be directly incorporated into it. prior to transformer @xcite , earlier works have demonstrated that syntactic information could be helpful for various nlp tasks. for example, @xcite introduced dependency structure into word embeddings, and @xcite uses tree-lstms to process the grammar trees for machine translation. more recently, dependency grammar has been successfully integrated into transformer in various forms. @xcite improves semantic role labelling by restricting tokens to only attend to its dependency parent. @xcite modifies bert @xcite for named entity recognition as well as glue tasks @xcite by adding an additional attention layer that allows every token to only attend to its ancestral tokens in the dependency parse tree. @xcite improves machine translation by constructing the attention weights from dependency tree, while @xcite masks out distant nodes in the dependency tree from attention. while the dependency grammar demonstrates the relation between nodes, the constituency grammar focuses more on how a sentence is formed in a merging way block by block. constituency grammar contains more information about the global structure of a sentence in a hierarchical way, which we think will greatly improve global attention mechanism like self-attention in transformers. since constituency grammar doesn't directly reflect grammatical relations between words and introduces new constituent nodes, integrating it into transformer becomes less obvious. @xcite explores different ways of utilizing constituency syntax information in the transformer model, including positional embedding, output sequence, etc. @xcite uses dual encoders to encode both source text and template yielded by constituency grammar, at a cost of introducing a large amount of parameters. in this work, we propose a syntax-guided localized self-attention that effectively incorporates constituency grammar into transformers, without introducing additional parameters. we first serial- (c) the attention mask reflecting the syntactic local ranges of each word for example, rather than attending to the whole sequence, "across" encourages attention towards swim, the and river while suppresses the others. ## method we are going to present a form of local selfattention that dynamically controls each word's attention range according to its syntactic role in the sentence (see fig. 1(c )). attention heads that incorporate this localized self-attention could significantly outweight the grammatically close tokens over distant ones, thus incorporate the syntax information as prior knowledge. ## conclusion in this work, we introduce constituency grammar into transformer by using syntactic distance to guide the self-attention mechanism. we implement the complete pipeline of fusion by converting constituency tree to syntactic distance and finally generating syntactic local pattern. experiments show that our method could consistently improve the performance of transformer on a variety of machine translation tasks. we also excavate that the syntactic-guided attention achieve the best effectiveness in the bottom layer. future work could expand to other nlu tasks and larger-scale pretrained models.