Leveraging Grammar Induction for Language Understanding and Generation (2024)

Abstract: Grammar induction has made significant progress in recent years. However, it is not clear how the application of induced grammar could enhance practical performance in downstream tasks. In this work, we introduce an unsupervised grammar induction method for language understanding and generation. We construct a grammar parser to induce constituency structures and dependency relations, which is simultaneously trained on downstream tasks without additional syntax annotations. The induced grammar features are subsequently incorporated into Transformer as a syntactic mask to guide self-attention. We evaluate and apply our method to multiple machine translation tasks and natural language understanding tasks. Our method demonstrates superior performance compared to the original Transformer and other models enhanced with external parsers. Experimental results indicate that our method is effective in both from-scratch and pre-trained scenarios. Additionally, our research highlights the contribution of explicitly modeling the grammatical structure of texts to neural network models.

## introduction neural network models, like transformer @xcite , roberta @xcite , and gpts @xcite , have gained widespread adoption in various natural language processing tasks. these models can generate desired answers on different tasks and show strong language understanding ability on multiple datasets. however, they give up explicit parsing of the specific syntactic structure of the text data and cannot effectively establish structured and interpretable language understanding models @xcite @xcite . this limitation has emerged as one of the bottlenecks for neural network models to understand natural language deeply. to this end, researchers attempt to take advantage of traditional syntax parsers @xcite to identify grammatical components within textual data and utilize them in subsequent processing steps. these parsers are built upon established linguistic frameworks and regulations. there has been growing interest in investigating the impact of syntax on neural network models and improving them through the lens of grammar induction @xcite @xcite . while these techniques have exhibited enhanced efficacy, they still grapple with two primary challenges. firstly, these approaches @xcite @xcite usually depend on external parsers to obtain additional specialized annotations, which are expensive and not timeefficient. their performance is greatly influenced by the choice of external parsers and may not be universally applicable across all languages. the other problem is that these approaches only focus on from-scratch scenarios @xcite or pre-trained scenarios @xcite @xcite ). they cannot share consistent improvements. this discrepancy between the two model types engenders a predicament where these methodologies are tailored exclusively to one of the scenarios. in this paper, we introduce a novel method to induce grammar information for language understanding and generation, obviating the necessity for supplementary syntactic annotations. the selfinduced grammar features are integrated into the transformer and simultaneously learned during the training of downstream tasks. additionally, we devise bpe (byte pair encoding) embeddings and trade-off loss functions to facilitate grammar induction. experiments demonstrate the compati-figure 1 : the pipeline for the construction of our syntactic mask. word embeddings and bpe embeddings are utilized to induce the intermediate grammar features s, which are subsequently used to derive the syntactic distance τ and height h. the two vectors are leveraged to estimate the dependency distribution for the sentence, and generate the syntactic mask p d . the mask is then employed to guide the self-attention mechanism within the encoders. these parsing modules are integrated into the transformer model and trained together in downstream tasks. ## preliminary we use syntactic distance @xcite and height @xcite to generate the dependency distribution among tokens. they are two feature vectors used to describe the constituency structure and dependency relations of a sequence of words. ## method alongside the data pipeline in the original transformer, we construct a parser to derive grammar features and estimate a syntactic mask to guide the attention mechanism. the "parsing" foot_1 pipeline is delineated in figure 1 . grammar features s are induced with convolution layers and a self-attention module working on word embeddings. specifically, convolution layers can capture localized details within segments, while the self-attention module can furnish global information of the entire sequence. bpe embeddings are introduced so that grammar status can be shared among congenetic subwords. utilizing grammar features s, syntactic distance τ , and height h can be derived through convolution layers. they are leveraged by the distribution generator to construct a syntactic mask p d that provides the dependency distribution among tokens in the encoder layers of transformer. ## machine translation we use transformer @xcite as our baseline and conduct experiments on six machine translation tasks of three datasets: iwslt14-de/en, nc11-de/en, and aspec-zh/ja. models are trained from scratch and evaluated on each task. ## language understanding we deploy our grammar induction method on roberta @xcite at the stage of finetuning, and evaluate its performance in the glue benchmark @xcite . ## related work there are generally two ways for syntactic knowledge-based language comprehension. explict syntax enhancement one way is to incorporate syntactic information explicitly by using a high-quality external parser. pascal @xcite builds up a syntaxguided localized attention mask where each token's attention range is a gaussian distribution centered by its dependency head. distance transformer @xcite utilizes the relative magnitude of syntactic distance from the constituency tree to build up a syntactic local range. st-nmt @xcite slices the constituency tree at some depth and gets a list of constituents as labels. a new transformer encoder is learned to predict this structure, and its output is integrated with the original encoder output. seprem @xcite also focuses on pretraining architecture and defines a prior attention weight distribution by normalizing the inverse of the token's distance in the dependency structure. however, all of these methods require sophisticated external parsers to generate dependency or constituency syntax trees. implicit syntax enhancement the other way is to implicitly induce the latent grammar structure by learning from data distribution. prpn @xcite enhances rnn neural language modeling by simultaneously learning a cnn-based parsing network, which could induce latent constituency structure represented by syntactic distance. recent works have focused more on attention mechanisms. structformer @xcite introduces a parser layer that can induce dependency and constituency syntax simultaneously and integrates induced dependency structure into self-attention in a differentiable way. lpsi @xcite induces latent phrase structure in an unsupervised fashion and integrates them into the multi-head attention mechanism. syncattn @xcite enables neural openie to induce the latent syntactic structure and adopt multi-view learning to capture multiple relationships from constituency and dependency graphs. nevertheless, very little work pays attention to leveraging implicit induction in downstream tasks. syntax enhancement of pre-trained models in recent years, people also studied to introduce grammar induction into pre-trained models like bert @xcite and roberta @xcite instead of training from scratch. pos-bert @xcite leverages pos tag information apart from the syntactic structure. sla @xcite prohibits two tokens from attending to each other once their distance in the dependency tree exceeds some boundary, and it improves the fine-tuning performance on several glue tasks. syntax-roberta @xcite develops a framework that can be easily plugged into an arbitrary pre-trained checkpoint, which automatically underlines the most relevant syntactic knowledge for each downstream task. synclm @xcite proposes a syntax-guided contrastive learning method where phrase-guided and tree-guided contrastive objectives based on constituency and dependency structures are optimized in the pretraining stage to help the language model capture rich syntactic knowledge in its representations. however, these methods resort to external parsers to obtain syntactic information, which could not work well with pre-trained models across downstream tasks. ## conclusion in this paper, we study the utilization of grammar induction for language understanding and generation. we force the model to be trained with the self-induced grammar without external parsers or annotations, which is a more general approach. moreover, our method is efficient in both the fromscratch and pre-trained scenarios. through assess-ments across various machine translation and language comprehension tasks, we validate the efficacy of our grammar induction technique, showcasing substantial and consistent enhancements. these results underscore the tangible benefits that can be achieved through the incorporation of grammar induction techniques. our research offers promising insights into the practical application of induced grammar in downstream tasks.