Dependency-based Mixture Language Models (2022)

Abstract: Various models have been proposed to incorporate knowledge of syntactic structures into neural language models. However, previous works have relied heavily on elaborate components for a specific language model, usually recurrent neural network (RNN), which makes themselves unwieldy in practice to fit into other neural language models, such as Transformer and GPT-2. In this paper, we introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective to learn the probability distribution of future dependent tokens given context. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention. Extensive experiments and human evaluations show that our method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks.

## introduction syntactic structures serve as the principle of how words are correctly combined to form sentences. it is widely acknowledged that learning syntactic structures should improve neural text generation @xcite @xcite . even though current neural language models, such as transformer @xcite and gpt-2 @xcite have achieved outstanding performance without explicitly modeling latent syntactic structures, these models still fail to learn the long-range syntactic dependencies @xcite . to leverage explicit syntactic knowledge in natural language generation (nlg), many methods have been proposed (wu et al., 2017; shen et al., 2018; zhang et al., 2019; kim et al., 2019; du 1 our code is available at https: //github.com/fadedcosine/ dependency-guided-neural-text-generation et al., 2020). we conclude from previous works that knowledge of syntactic structures can bring four advantages to neural language models: despite these advantages, it is not trivial to incorporate knowledge of syntactic structures into neural language models effectively and efficiently. several practical problems arise: to address these problems while keeping the advantages, we explore incorporating knowledge of syntactic structures in a different manner. in this work, we propose a novel dependency modeling objective to train neural language models to directly predict the current token's future dependent tokens given the history. we define the future dependent to-kens of a specific token in a sentence as its children and parent in the dependency parse tree that will appear in the rest of the sentence. further, we propose dependency-based mixture language models (dmlm) that, at each timestep, mixes the previous dependency modeling probability distributions with self-attention to get the next-token probability. as shown in our core idea can be illustrated in figure 1 and figure 2 : when predicting the next-token "indicate" after reading "red figures on the screen", common language models are easy to predict an incorrect word, such as "indicates", since the prediction of these models relies heavily on the recent word, "screen" in this case. however, our propose dmlm will directly look back into the long-range context, and select the next-token from all the future dependent tokens predicted by previous tokens. according to the underlying dependency structure, dmlm pays different weights to different tokens' future dependent tokens. thus, the model is more likely to predict "indicate" since dmlm tends to think of the next-token as a future dependent token of "figures" rather than "screen". we conduct experiments with different neural language models including lstm @xcite , transformer @xcite , and gpt-2 @xcite across different tasks in conditional text generation, unconditional text generation, and language modeling. through extensive experiments we demonstrate that dmlm consistently improves the generation quality according to both human evaluations and automatic metrics. compared to other neural language models that incorporate syntactic knowledge, root root red figures on the screen indicate falling stocks . ## methodology our goal is to propose a simple yet effective method that can improve neural text generation by learning from the underlying syntactic structure, and can fit into any auto-regressive generation model without using additional elaborate components. we first introduce a novel dependency modeling objective to force the model to directly predict the future dependent tokens of the current token. based on the dependency modeling, we then present the proposed dmlm. ## experiments despite previous works mainly focusing on language modeling, it has always been a thorny issue whether better language models lead to better performance in downstream tasks. therefore, we showcase the performance of our proposed dmlm in three different tasks: conditional text generation (section 3.1), unconditional text generation (section 3.2), and language modeling (section 3.3). to verify the effectiveness and architecturally generalizability of our method, we conduct the generation tasks with three dominant neural language models, including lstm, transformer and gpt-2. we prefix the base model name with "dm-" to denote the corresponding dependency-based mixture language model. specifically, we adopt awd-lstm @xcite as our base lstm, and further compare our dm-lstm with prpn @xcite and on-lstm @xcite which also incorporate knowledge of syntactic structures, and are built on lstm. in the same task, we use exactly the same hyper-parameters and setups for the pairs of base models and corresponding dm-models. other details of the experimental setup for each task can be seen in appendix a. for all the tasks, we use a state-of-the-art parser, hpsg parser foot_0 @xcite to get the dependency parse tree for each sentence in the datasets. we discuss the impact of the dependency parser in appendix b. ## related works many previous studies have shown that leveraging the knowledge of syntactic structures can improve nlg @xcite @xcite . @xcite incorporated syntactic dependencies into the rnn formulation, but they limited the scope to the scoring of complete sentences, not to next word prediction. some other efforts have been done to integrate dependency structure into neural machine translation (nmt) from both the source and target side. however, all these methods, mainly based on rnn @xcite , incorporate knowl-story context: ## conclusion in this paper, we introduce dependency-based mixture language models, which can incorporate knowledge of dependency structures into arbitrary auto-regressive generation models without any changes to the original architectures. both automatic and human evaluation results in extensive experiments across different tasks and different architectures demonstrate the effectiveness and generalizability of our method. in the future, we will explore to incorporate the dependency labels into our method, and combine our dmlm with more neural language models. second, we would like to integrate other linguistic knowledge, such as constituency structures and semantic information, into neural language models in our manner. hpsg parser maintains high-quality on outof-domain text, as shown in its paper @xcite . most importantly, even on the out-of-domain datasets, i.e., rocstories and emnlp2017 wmt news, our work can still obtain a significant improvement, as shown in section 3.1 and section 3.2.