Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision (2024)

Abstract: Syntactic Language Models (SLMs) can be trained efficiently to reach relatively high performance; however, they have trouble with inference efficiency due to the explicit generation of syntactic structures. In this paper, we propose a new method dubbed tree-planting: instead of explicitly generating syntactic structures, we “plant” trees into attention weights of unidirectional Transformer LMs to implicitly reflect syntactic structures of natural language. Specifically, unidirectional Transformer LMs trained with tree-planting will be called Tree-Planted Transformers (TPT), which inherit the training efficiency from SLMs without changing the inference efficiency of their underlying Transformer LMs. Targeted syntactic evaluations on the SyntaxGym benchmark demonstrated that TPTs, despite the lack of explicit generation of syntactic structures, significantly outperformed not only vanilla Transformer LMs but also various SLMs that generate hundreds of syntactic structures in parallel. This result suggests that TPTs can learn human-like syntactic knowledge as data-efficiently as SLMs while maintaining the modeling space of Transformer LMs unchanged.

## introduction recent years have witnessed remarkable success in large language models (llms) based on transformer lms @xcite . however, despite their success, transformer lms have some drawback in training efficiency-especially when compared with humans. for example, gpt-3 @xcite ) is trained on around 2, 000× larger data than a 12-year-old human would have experienced @xcite , indicating that transformer lms lack sufficient inductive bias for language acquisition. on another strand, previous work has revealed that syntactic language models (slms), defined as a generative models of a token sequence and its syntactic structures, can achieve high syntactic performance under data-constrained settings @xcite @xcite @xcite . for example, @xcite showed that some slms can achieve comparable syntactic performance to an llm-like model foot_0 that is trained with medium-around 250× larger-data, suggesting that syntactic supervision is essential for lms to achieve high training efficiency. however, despite their training efficiency, slms have trouble with inference efficiency-they require hundreds of syntactic structures generated via beam search @xcite or an external parser to precisely approximate marginal distribution over a token sequence, which naturally incurs the costs hundreds of times higher than their underlying sequential models. in this paper, we propose a new method dubbed tree-planting: instead of explicitly generating syn-tactic structures, we "plant" trees into attention weights of unidirectional transformer lms to implicitly reflect syntactic structures of natural language. 2 specifically, unidirectional transformer lms trained with tree-planting will be called tree-planted transformers (tpt), which inherit the training efficiency from slms without changing the inference efficiency of their underlying transformer lms (figure 1 ). previous studies have also explored the syntactic supervision of attention weights, mainly targeting bidirectional transformer encoders @xcite @xcite @xcite ; inter alia) ( §2). these encoder-oriented approaches assume the entire sentence as input and typically aim to reflect the syntactic relationship between input words in a bottom-up manner. in contrast, tree-planting is uniquely designed for unidirectional transformer lms that can be used for text generation-and they have recently increased prevalence because of their compatibility with instruct tuning @xcite or reinforcement learning from human feedback @xcite . specifically, tree-planting considers syntactic structures involving the next word to be generated, by focusing on the syntactic distance @xcite @xcite between the next word and the previous words in the context ( §3). targeted syntactic evaluations on the syntax-gym benchmark @xcite demonstrated that tpts, despite the lack of explicit generation of syntactic structures, significantly outperformed not only vanilla transformer lms but also various slms that generate hundreds of syntactic structures in parallel. this result suggests that tpts can learn human-like syntactic knowledge as data-efficiently as slms while maintaining the modeling space of transformer lms unchanged. furthermore, closer inspection of syntactic phenomena implied that tree-planting shows high compatibility with dependency structures ( §4). additionally, we analyzed the two hyperparameters introduced by tree-planting: (i) the number of heads where the tree-planting loss is applied, and (ii) the balance between the next-word prediction loss and the tree-planting loss. our results demonstrated that the highest accuracy was generally achieved when a single head was adopted as a tree-planted head while the balancing parameter emerged as a crucial hyperparameter; excessively high and low weights on the tree-planting loss both led to ineffective outcomes ( §5). ## proposed method: tree-planting in this paper, we propose a new method dubbed tree-planting: we "plant" trees into attention weights of unidirectional transformer lms to re- birds in a cage sing birds sing in a cage cage a in tree planted transformer: 統語的大規模言語モデルの構築に向けて 吉田遼 染谷大河 大関洋平 東京大学 {yoshiryo0617,taiga98-0809,oseki}@g.ecc.u-tokyo.ac.jp 概要 birds in a cage in 3 a 4 3 cage 4 3 2 sing 3 4 5 5 謝辞 本研究は、jst さきがけ jpmjpr21c2 の支援を受 けたものです。 参考文献 measure distances on a syntactic structure implicitly supervise attention weights figure 2: overview of the proposed method: tree-planting ## experiment to investigate whether tpts can learn human-like syntactic knowledge as data-efficiently as slms while maintaining the modeling space of transformer lms unchanged, we conduct training on a small treebank and targeted syntactic evaluations on a syntactic knowledge benchmark. ## analysis in this section, we report the effects of (i) the number of tree-planted heads and (ii) the weight of a tree-planting loss, using tpt[dep.]. ## conclusion in this paper, we propose a new method dubbed tree-planting: instead of explicitly generating syntactic structures, we "plant" trees into attention weights of unidirectional transformer lms to implicitly reflect syntactic structures of natural language. specifically, unidirectional transformer lms trained with tree-planting will be called tree-planted transformers (tpt), which inherit the training efficiency from slms without changing the inference efficiency of their underlying transformer lms. targeted syntactic evaluations on the syntaxgym benchmark demonstrated that tpts, despite the lack of explicit generation of syntactic structures, significantly outperformed not only vanilla transformer lms but also various slms that generate hundreds of syntactic structures in parallel. this result suggests that tpts can learn human-like syntactic knowledge as data-efficiently as slms while maintaining the modeling space of transformer lms unchanged.