Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models (2024)

Abstract: Syntactic Transformer language models aim to achieve better generalization through simultaneously modeling syntax trees and sentences. While prior work has been focusing on adding constituency-based structures to Transformers, we introduce Dependency Transformer Grammars (DTGs), a new class of Transformer language model with explicit dependency-based inductive bias. DTGs simulate dependency transition systems with constrained attention patterns by modifying attention masks, incorporate the stack information through relative positional encoding, and augment dependency arc representation with a combination of token embeddings and operation embeddings. When trained on a dataset of sentences annotated with dependency trees, DTGs achieve better generalization while maintaining comparable perplexity with Transformer language model baselines. DTGs also outperform recent constituency-based models, showing that dependency can better guide Transformer language models. Our code is released at https://github.com/zhaoyd1/Dep_Transformer_Grammars.

## introduction transformer language models have shown strong performance on language modeling tasks and a broad spectrum of downstream tasks @xcite @xcite . despite the great power of the transformer architecture @xcite , it lacks the inductive biases of syntactic structures, which has been hypothesized to improve generalization @xcite . a straightforward way to incorporate such biases into transformers is explicit modeling of syntactic structures. inspired by earlier work of generative parsing as language modeling that integrates syntactic structures into rnns @xcite ; choe and char-* corresponding author niak, 2016), recent studies have focused on adapting this method to transformer architectures @xcite @xcite . the models proposed by these studies are categorized as syntactic language models because they jointly model the distribution of surface strings and their corresponding syntactic trees. experiments show that these models achieve competitive perplexity in language modeling and gain better syntactic generalization, supporting the above hypothesis on the benefits of introducing inductive bias of syntactic structures. however, the structural supervision that has been used in all these models is based on constituency trees and it is unclear of the performance of dependency-based transformer syntactic language models. different from constituency structures, which model recursive syntactic compositions, dependency structures focus more on the relationship between tokens, which is similar to the self-attention mechanism in transformer, hinting at potential synergy between the two. in this paper, we propose dependency transformer grammars (dtgs), dependency-based syntactic language models that learn joint distributions of sentences and dependency trees. dtgs introduce an inductive bias of dependency structures to transformers by (i) modeling transition sequences of transition-based dependency parsers instead of sentences, (ii) simulating the stack operations in transition-based dependency parsers through modification of attention masks, (iii) incorporating the stack information of transition-based systems through relative positional encoding of stack depth, and (iv) representing head-dependent relations through a combination of head token embeddings and transition operation embeddings. following a line of previous work in generative dependency parsing @xcite @xcite , the generative formulation of our model is based on the arc-standard system @xcite , which builds a dependency tree in a bottom-up manner. we also explore models using other dependency transition systems for comparison. our experiments show that dtgs achieve comparable perplexity in language modeling and improved syntactic generalization on both the blimp benchmark @xcite and the sg test suites @xcite ) over transformer language model baselines. furthermore, dtgs outperform constituency-based syntactic language models in both language modeling and syntactic generalization. in summary, our contributions are as follows. ## preliminaries: transition-based dependency parsing given a sentence, transition-based dependency parsing predicts a sequence of predefined transitions between states that incrementally build a dependency parse tree. a state contains a stack σ with token i on the top, a buffer β with j at its leftmost side, and a set a of dependency arcs, denoted as (σ|i, j|β, a). in this work, we focus on unlabeled projective dependency parsing for the simplicity of its transition systems. there are several different transition systems for projective dependency parsing, as shown in the above dependency parsing transition systems can be changed into a generative form, such that they generate sentences along with their associated dependency trees. the main change to the transition systems is that tokens need to be generated instead of being shifted from the buffer. specifically, in arc-standard we substitute shift with a token generation transition gen, while retaining the other transitions @xcite @xcite . other systems require additional efforts to obtain a generative form because they contain the usage of the buffer head in leftarc and/or rightarc before shifting it to the stack. simply replacing shift with gen cannot ensure the existence of the two tokens involved in a newly generated arc. therefore, we need to insert a gen' transition, foot_0 which generates a new token but puts it in the buffer, before any leftarc/rightarc transition that involves an ungenerated token. the shift transitions are omitted because any generated token will be shifted to the stack once a new token is generated. we can use an oracle to extract a transition sequence from a dependency parse tree: an arcinducing transition is generated whenever possible, and a pop transition (in arc-eager) is generated when it is impossible to generate other transitions, i.e., the transition preference order is leftarc/@xmath0. an example is shown in figure 1 . we write α(x, y) = (α 0 , α 1 , ..., α t -1 ) as the transition sequence of length t of sentence x and parse tree y, where each α t belongs to one of the three types mentioned above. dtg is a transformer decoder that models the distribution of α(x, y) in the manner of causal language modeling, that is, p(α(x, y)) = ∏ i p(α i |α <i ). it differs from a standard transformer in several aspects in order to incorporate the dependency inductive bias, including attention masks, positional encoding, augmented representation of arcs, and con-strained generation, which we discuss in the following subsections. ## experiments we compare dtgs with dtg-eager, dtg-swift, two transformer-xl baselines, and constituencybased syntactic transformer language models. the two transformer-xl baselines follow those of sartran et al. ( 2022) ## related work augmenting language models with syntactic bias has been studied for a long time. one line of work adds constituency-based syntactic structures to language models through jointly modeling the distribution of sentences and structures @xcite @xcite @xcite . the rnng model @xcite is a representative work of syntactic language models, using recursive networks to build representations of phrases. more recent work of syntactic language models is based on transformers @xcite @xcite . @xcite and @xcite constrain the attention with syntactic bias, while pushdown layers @xcite enforce structural constraints via gradient based learning. the above work is all based on constituency structures, and there has been some work considering dependency trees with simple neural networks @xcite @xcite . most of them, however, focus more on generative dependency parsing while scratching the surface of a language modeling setting. a more general work is @xcite , which both introduces constituency and dependency graphs to augment transformer language modeling, but it requires given gold trees for generation. following the work of generative dependency parsing and the constrained attention patterns used in @xcite and other work @xcite @xcite @xcite , we propose dtg, a novel class of dependency-based syntactic language models. it is the first syntactic language model that designs a dependency-based constrained attention mechanism for transformers. another line of work augments models by learnable structures. some studies integrate stackstructured memory into models, where updating patterns are learned from data rather than being dictated by predefined syntactic inductive bias @xcite @xcite . besides, some studies propose to learn structural attention patterns @xcite @xcite . for example, @xcite assumes that the attention scores are subject to linear-chain or tree conditional random fields (crfs; @xcite . these kinds of augmentation lead to better generalization but usually cost longer running time than naive counterparts. some other studies focus on examining the syntactic knowledge acquired by standard attention after pretraining @xcite @xcite . these studies have identified that certain attention heads align their attention patterns with syntactic structures, thereby providing substantial evidence for the benefits of introducing syntactic inductive bias. in addition, some work re-invents attention using dependency structures and crfs @xcite , motivating more linguistically principled studies. ## conclusion we propose dtgs, a new type of syntactic language models that add explicit dependency bias into transformers. dtgs simulate dependency transition systems with constrained attention pat-terns and incorporate stack information through relative positional encoding. experiments show that dtgs surpass transformer language model baselines and other constituency-based syntactic language models on syntactic generalization while maintaining competitive perplexity. this implies that the presence of dependency information does improve the performance of transformer language models.