Explicit Syntactic Guidance for Neural Text Generation (2023)

Abstract: Most existing text generation models follow the sequence-to-sequence paradigm. Generative Grammar suggests that humans generate natural language texts by learning language grammar. We propose a syntax-guided generation schema, which generates the sequence guided by a constituency parse tree in a top-down direction. The decoding process can be decomposed into two parts: (1) predicting the infilling texts for each constituent in the lexicalized syntax context given the source sentence; (2) mapping and expanding each constituent to construct the next-level syntax context. Accordingly, we propose a structural beam search method to find possible syntax structures hierarchically. Experiments on paraphrase generation and machine translation show that the proposed method outperforms autoregressive baselines, while also demonstrating effectiveness in terms of interpretability, controllability, and diversity.

## introduction natural language generation (nlg), such as paraphrase generation @xcite ), text summarization @xcite , machine translation @xcite , and language models @xcite , have shown remarkable progress in the past few years. most of the highest-performing nlg models train the model based on source-target correspondence and conduct autoregressive inference, which achieves competitive empirical performances yet deviates from a range of desirable attributes of human language generation, e.g., lack of interpretability @xcite @xcite . it has been shown that humans generate language by learning and manipulating language grammar @xcite , which generative grammar @xcite …………………………………………………... he @xmath0. figure 1 : syntax-guided generation: searching the hypotheses hierarchically throughout the syntax tree in a top-down direction, starting from the root node "<@xmath1. the green blocks denote the possible syntax structures at different tree depths, the blue one denotes the external modification, whereas the gray ones denote the finalized hypotheses, marking the end of search paths. considers as a finite rule set that combines words to form grammatical sentences, thereby avoiding enumeration of surface sequences, which can significantly increase data sparsity and reducing learning efficiency @xcite . in this process, syntax plays a crucial role, imposing constraints on how to construct sentences. syntax knowledge has been found implicitly contained by deep neural models @xcite and also useful for nlg tasks @xcite @xcite . however, relatively little recent work has considered explict syntax in nlg @xcite . inspired by the above psycholinguistic observation, we propose a syntax-guided generation scheme, which generates text by following a welldefined grammar. as shown in figure 1 , instead of sequential generation, the model generates the sentence in a hierarchically top-down manner guided by the constituency parse tree, starting with the root @xmath2. syntactic categories such as noun @xmath3, and the model simultaneously considers multiple syntax structures at each tree depth, hierarchically exploring the syntax tree for reasonable hypotheses. intuitively, such a generation paradigm has the following advantages compared with autoregressive generation. first, akin to the language learning process of human beings, grammar learning breaks down non-enumerable surface sequences into finite pieces, acting as a training curriculum. second, it provides an effective and interpretable pathway to probe into the generation process. consequently, generation errors can be traced back to specific constituent expansion at the respective tree depth. third, one can manipulate the generation process by exerting versatile control at arbitrary depths, e.g., modifying the translation of a verb phrase and constraining the paraphrase style with syntax templates. forth, diverse sequences can be generated by exploring various syntax structures hierarchically throughout the syntax tree. we implement the above process on transformer @xcite . as shown in figure 1 , the generation process proceeds under the guidance of syntactic grammar. starting from the root node "<@xmath4, the model recursively generates the infilling texts (e.g., "he" and "@xmath5.g, "<@xmath6.".), and infills each one accordingly to construct the next-level lexicalized syntax context (e.g., "he @xmath7."). the generation proceeds until there is no remaining constituent. the infilling texts are predicted by a transformerbased model, which is trained by maximizing the likelihood of infilling texts for each constituent in the syntax context based on the source input. to explore more syntactically diverse and reasonable hypotheses during inference, we propose structural beam search, which searches promising syntax structures over the entire syntax tree in a top-down manner, as shown in figure 1 . to isolate the effect of syntax and avoid the influence of other transformation factors, we conduct experiments on two sequence-to-sequence (seq2seq) tasks with semantic equivalence between the source and target sequences: paraphrase generation and machine translation. empirical results demonstrate that our method can generate sequences with higher quality than the seq2seq baselines. quantitative analysis demonstrates that the generation process can be interpreted effectively. in addition, our method demonstrates the ca-pability of executing control from both syntax templates and fine-grained manual modifications. finally, we show the diversity advantage through both automatic evaluation and human evaluation. we release the code on https://github.com/ yafuly/syntacticgen . syntax as extra input. a line of work incorporates syntax knowledge as extra input to boost task performance. in paraphrase generation, iyyer et al. 2018), chen et al. (2019), kumar et al. (2020 and @xcite additionally encode a constituency tree to produce controllable paraphrases. for machine translation, researchers utilize syntactic information to boost the neural machine translation system using syntactic encoders @xcite @xcite @xcite , position encoding @xcite , attention mechanism @xcite , and auxiliary training objectives @xcite . syntax for generation guidance. different from the above work, we focus on guiding generation explicitly following syntactic grammar. typically, @xcite and @xcite learn the mapping from sequences to linearized constituency trees to improve machine translation. @xcite proposes a hybrid decoder with rnng @xcite to jointly learn parse actions and word predictions. @xcite and @xcite design a syntactic tree decoder based on lstm @xcite , with an extra rule decoder. @xcite introduce a syntax-guided soft target template as extra prompts in transformer. different from their work, our method leverages transformer strengths and breaks down the sequence-to-sequence generation process into a hierarchically top-down generation guided by the syntax tree. ## experiment setup datasets for paraphrase generation, we experiment on paranmt-small @xcite , which contains 500k sentence-paraphrase pairs for training, 500 for validation, and 800 for testing. both validation and test sets are provided with humanannotated sentence exemplars from which syntax information can be extracted for controlling paraphrase generation. for machine translation, we use nist chinese-english (zh-en), wmt'16 romanian-english (ro-en), wmt'14 english-german (de-en), and wmt'14 english-german (en-de). for wmt datasets, we follow the official split for validation and testing. for nist zh-en, we use mt06 as the validation set and choose mt02, mt03, mt04, mt05, and mt08 as the test sets. for all datasets, we use berkeley parser @xcite to obtain constituency parse trees and use the most frequent constituents (e.g., <np>, <vp>, <@xmath8. model settings for transformer baselines, we adopt the transformer_base configuration which consists of a 6-layer encoder and decoder. for our model, we keep the 6-layer source context encoder, and set the number of layers for both the syntax context encoder and the decoder as 3, resulting in a similar model size with transformer_base. the accumulation weight α is as 0.8 for structural beam search based on validation experiments. for machine translation, we adopt sequence-level distillation @xcite for both our model and the corresponding baseline transformer. more details are shown in appendix b. evaluation we use the bleu score @xcite to evaluate machine translation performance. for paraphrase generation, we also adopt rouge @xcite bleu(hypothesis, source), which evaluates the generation fidelity with novelty to the source sentence considered * . following bandel et al. (2022), we consider two referencefree metrics: (1) lexical diversity score, i.e., d lex , which is the normalized character-level minima edit distance between the bag-of-words; and (2) syntax diversity score, i.e., d syn , which is the normalized tree edit distance. both scores measure generated paraphrases with the source sequences unless specified. 5 results paraphrase we compare our method with the baselines and previous work on syntax-control paraphrase generation. another two baselines are also * r is set as 0.7. listed, i.e., copy the source input and use the reference as the output. the results are shown in we extend our method to the pre-trained language model (plm) setting and present the result in machine translation as shown in ## analysis we first discuss the influence of grammar quality, then we understand the potential advantages of our method from three perspectives, i.e., interpretability, controllability, and diversity. ## conclusion we proposed a syntax-guided generation paradigm, which leverages the strengths of transformer and generates sequences by hierarchically expanding constituents in the lexicalized syntax contexts throughout the syntax tree. the neural decoder was trained by maximizing the likelihood of the infilling texts for each constituent in the syntax contexts given the source sequence. moreover, we proposed the structural beam search to better explore the hypothesis space. empirical results demonstrated the advantage of generation quality over the seq2seq baseline, and also the effectiveness in terms of interpretability, controllability, and diversity. our method can be seen as a step towards explicit modelling of psycholinguistic structures during neural text generation , helping the model to have a degree of control over what it intends to generate, which can potentially address salient issues of current neural nlg, such as hallucination @xcite and ethical issues @xcite @xcite , if semantics, pragmatics, and other factors are also integrated.