Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale (2024)

Abstract: A syntactic language model (SLM) incrementally generates a sentence with its syntactic tree in a left-to-right manner.We present Generative Pretrained Structured Transformers (GPST), an unsupervised SLM at scale capable of being pre-trained from scratch on raw texts with high parallelism. GPST circumvents the limitations of previous SLMs such as relying on gold trees and sequential training. It consists of two components, a usual SLM supervised by a uni-directional language modeling loss, and an additional composition model, which induces syntactic parse trees and computes constituent representations, supervised by a bi-directional language modeling loss. We propose a representation surrogate to enable joint parallel training of the two models in a hard-EM fashion.We pre-train GPST on OpenWebText, a corpus with billion tokens, and demonstrate the superiority of GPST over GPT-2 with a comparable size in numerous tasks covering both language understanding and language generation. Meanwhile, GPST also significantly outperforms existing unsupervised SLMs on left-to-right grammar induction, while holding a substantial acceleration on training.

## introduction pre-training a transformer architecture @xcite as a large language model has dominated the field of natural language processing (nlp) @xcite @xcite @xcite . while transformer language models have exhibited remarkable performance over various downstream nlp tasks @xcite , the recursive compositions behind language are represented in an implicit and entangled form. in contrast, human language understanding exhibits explicit composition decisions, as exemplified by the garden path sentence @xcite "time flies like an arrow; fruit flies like a banana", where distinct syntactic configurations yield vastly divergent meanings foot_0 . in addition, human infants acquire such compositional capability without supervision @xcite . these phenomena motivate us to explore an unsupervised approach to learning explicit compositions in language modeling. a typical approach to achieve language modeling with explicit composition is to model the joint distribution of words and a syntactic tree within the framework of syntactic language models (slms) @xcite . though there has been a long line of research on slms, they are rarely exploited as the backbone in state-of-the-art language modeling due to poor scalability. recent transformer-based slms @xcite require annotated parse trees or supervised parsers as structural supervision, leading to limited training data scales and domain adaption issues @xcite . on the other hand, in unsupervised slms @xcite , non-terminal constituents are composed from their sub-constituents sequentially in a left-to-right manner, resulting in data dependencies that impede training parallelism. in this work, we aim to pre-train an slm at scale on raw texts. to this end, we propose generative pretrained structured transformer (gpst), an unsupervised slm with the transformer architecture as a backbone. a common practice in existing unsupervised slms is to learn structures by a uni-directional language modeling loss (lm loss). however, we empirically find such an asymmetric loss with only right-to-left feedback results in branching biases in the induced parse trees. based on the insight, we propose two components in gpst, a composition model performing structural learning supervised by a bi-directional lm loss, and a generative model for uni-directional syntactic language modeling. specifically, we train the gpst in a fashion similar to hard-em @xcite : in e-step, the composition model runs a pruned deep inside-outside encoder to induce a parse tree and compute inside and outside representations of constituents simultaneously within logarithmic steps @xcite ; while in m-step, we update all parameters of gpst by minimizing both the bi-directional (reconstructing the sentence from outside representations) and uni-directional lm loss given the induced tree. the key in the mstep lies in using the inside representations of constituents computed by the composition model as a surrogate of inputs for the generative model, which enjoys two advantages. first, the representations of all constituents pre-computed in the e-step can be simultaneously fed into the generative model, which breaks the data dependencies and facilitates training parallelism. second, with these representations participating in generation, the uni-directional lm loss in the m-step could be back-propagated to not only the generative model but the composition model used in the e-step as well. in experiments, we pre-train gpsts with sizes comparable to those of gpt-2 small and gpt-2 medium on openwebtext @xcite (∼ 9 billion tokens), and evaluate the models on various tasks including language understanding, language generation, and grammar induction. gpst demonstrates an approximately 60-fold training acceleration and over 15% absolute increase in left-to-right grammar induction in comparison with existing unsupervised slms. meanwhile, gpst also shows advantages over gpt-2 across almost all language understanding/generation benchmarks. gpst provides constituent-level interfaces that are not inherently possessed by the conventional transformerbased language models, and thus exhibits great potential to enhance interpretability @xcite , support multi-modality @xcite , and improve dense retrieval in the future. our contributions are three-fold: • we propose an slm consisting of a composition model in addition to a generative model, which can be trained without gold trees via a novel approach akin to hard-em. composition models. a composition model transforms text encoding into a combinatorial optimization problem, learns and searches for the optimal structure, and encodes the text in a bottom-up manner along a binary tree via a composition function recursively. @xcite proposes a cky-like @xcite @xcite encoder, in which high-level constituents are soft-weighted over composed representations of its sub-constituents. @xcite proposes a deep inside-outside encoder @xcite , enabling the model to learn underlying structures via an auto-encoding objective. recently, a series of studies @xcite @xcite have been conducted to reduce the neural inside encoder complexity from cubic to linear. our slm is built on top of state-of-the-art composition modeling techniques, in which we achieve unsupervised learning and enhance training parallelism by taking advantage of the pruned inside-outside encoder @xcite . ## methodology given a sentence @xmath0, x 2 , . . . , x n ] with x i from a vocabulary v (1 ⩽ i ⩽ n), our goal is to train an slm without gold trees that can simultaneously generate x and its syntactic structure. we first introduce the generative architecture of gpst, and then elaborate on how to perform training and inference with the model. ## experiments to fairly compare gpst and gpt-2, we pre-train both models from scratch on the same corpus with the same setups and comparable parameter sizes. evaluation is conducted on various language understanding/generation tasks. besides, we also evaluate gpst on grammar induction to verify to what extent the induced parse trees are consistent with human annotation. pre-training corpus. we pre-train models on wikitext-103 @xcite and openweb-text @xcite , where the two datasets contain 116 million tokens and 9 billion tokens, respectively. the context window size in pre-training is set to 1024. when a context involves more than one complete sentence, parse trees are induced for each sentence separately. hyper-parameters. following gpt-2 @xcite , we use 768/1024-dimensional embeddings, a vocabulary size of 30522, 3072/4096dimensional hidden layer representations, and 12/16 attention heads for the generative models of gpst small and gpst medium , respectively. to align with transformer layer counts in gpt-2, we configure gpst small with 3 type layers and 9 token layers, and gpst medium with 3 type layers and 21 token layers, respectively. we set the input dimension of the composition model to 256/512, and the number of transformer layers used in the composition function and decomposition function to 4 and 1, corresponding to the small and medium setups. to compare with gpt-2 under the same parameter sizes, we provide gpt-2 small 13 and gpt-2 medium 25 as additional baselines, representing models with 13 and 25 layers, respectively. the token embeddings are down-scaled before being fed into the composition model, and the constituent representations are up-scaled before being fed into gpst. all models are trained on 8 a100 gpus with a learning rate of 5e-5/1e-4, 8 × 32 × 1024 tokens per step, 5 billion and 15 billion total training tokens for wikitext-103 and openwebtext, respectively. ## conclusion in this paper, we propose an unsupervised approach to train gpst at scale efficiently. a key insight of our work is to guide the left-to-right structural learning with symmetric supervision such as an auto-encoding loss, which can receive feedback from both sides. a key technical contribution is that we propose the representation surrogate which enables joint training of all components in parallel. besides, the composition model of gpst can be regarded as an enhancement to the conventional embedding layer, which provides context-invariant embeddings of various granularities beyond token embeddings. our experiment results show the superiority of gpst on language understanding, generation, and left-to-right grammar induction, which demonstrate the potential of gpst as a foundational architecture for large language models.