NEUROSTRUCTURAL DECODING : Neural Text Generation with Structural Constraints (2023)

Abstract: Text generation often involves producing coherent and grammatically correct texts that also satisfy a given set of semantic constraints. While most approaches for conditional text generation have primarily focused on lexical constraints, they often struggle to effectively incorporate syntactic constraints, which provide a richer language for approximating semantic constraints. We address this gap by introducing NeuroStructural Decoding, a new decoding algorithm that incorporates syntactic constraints to further improve the quality of the generated text. We build NeuroStructural Decoding on the NeuroLogic Decoding (Lu etal. 2021) algorithm, which enables language generation models to produce fluent text while satisfying complex lexical constraints. Our algorithm is powerful and scalable. It tracks lexico-syntactic constraints (e.g., we need to observe dog as subject and ball as object)during decoding by parsing the partial generations at each step. To this end, we adapt a dependency parser to generate parses for incomplete sentences. Our approach is evaluated on three different language generation tasks, and the results show improved performance in both lexical and syntactic metrics compared to previous methods. The results suggest this is a promising solution for integrating fine-grained controllable generation into the conventional beam search decoding.

## introduction controllable text generation uses decoding algorithms that support lexico-semantic constraints to control what is included in the output. checking for the presence or absence of certain words has been shown to improve applications such as recipe generation @xcite , sentiment analysis @xcite , and predicting implicit consequences of beliefs @xcite . ## related work constrained generation methods aim to produce outputs that satisfy specific constraints. there is a vast body of work spanning methods that operate at inference-time @xcite @xcite @xcite , or trainingtime @xcite @xcite . the inference-time methods, which include the work proposed here, can be divided into two broad categories based on the types of constraints considered. the first category focuses on simple lexical constraints, which specify the presence or absence of certain words or phrases in the generated text. the second focuses on satisfying more sophisticated non-lexical constraints and enforcing rules on the overall organization of the generated text. these two categories are discussed in detail below. @xcite introduced constrained beam search by using a finite-state machine to track constraints. their method forces the inclusion of selected tag words in the output and fixed, pretrained word embeddings to facilitate vocabulary expansion to previously unseen tag words. @xcite developed grid beam search as an extension of traditional beam search that incorporates pre-specified lexical constraints and groups together hypotheses based on the number of constraints satisfied. this method is biased towards fulfilling constraints greedily, which can lead to sub-optimal solutions in many cases. @xcite improved the previous method with a dynamic beam allocation strategy, which can only consider a small number of hypotheses at each step. later, @xcite developed the constrained generative model which allows multiple keywords in the generated sentences. it involves inserting all positive constraint keywords in random order and then randomly sampling edits to improve the fluency of the sentence by replacing, inserting, or deleting words. ## neurostructural decoding our goal is to support structural constraints that require certain relationships to hold between the lexical items in the generated sentence. in this work, we target structural constraints based on syntactic dependencies due to their ubiquity. our method can potentially accommodate other constraints such as semantic roles or other domain-specific relations. we expand on the neurologic decoding @xcite ) framework, which only handles lexical constraints. the fundamental idea is to assign a score to each hypothesis in the beam based on how many lexical constraints have been satisfied. the algorithm uses pruning, grouping, and selection strategies to efficiently search for the highest scoring hypotheses that satisfy as many of the constraints as possible. our work modifies this framework to support structural syntactic constraints and their logical combinations. ## evaluation we demonstrate the utility of neurostruc-tural decodingon a variety of benchmark tasks related to the controllable text generation: com-mongen in section 4.1, sume in section 4.2, and machine translation in section 4.3. to prepare the training data for partial sentences, we convert constituency parse trees of partial sentences to dependency parse trees using stanford corenlp library @xcite . to train the dependency parser for partial sentences, we use stanza @xcite model. we adapt it to parse incomplete sentences using the three-stage training described in section 3.4. ## conclusion we introduced a novel constrained generation decoding algorithm, called neurostructural decoding, which infuses structural constraints driven by syntactic dependencies in the generated output. we evaluated our method on three generation tasks: constrained commonsense generation, summarizing biomedical mechanisms, and mt. we showed that our method generates better texts according to the various task-specific metrics when compared against vanilla decoding and constrained decoding that relies solely on lexical constraints.