N euro L ogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints (2021)

Abstract: Conditional text generation often requires lexical constraints, i.e., which words should or shouldn’t be included in the output text. While the dominant recipe for conditional text generation has been large-scale pretrained language models that are finetuned on the task-specific training data, such models do not learn to follow the underlying constraints reliably, even when supervised with large amounts of task-specific examples. We propose NeuroLogic Decoding, a simple yet effective algorithm that enables neural language models – supervised or not – to generate fluent text while satisfying complex lexical constraints. Our approach is powerful yet efficient. It handles any set of lexical constraints that is expressible under predicate logic, while its asymptotic runtime is equivalent to conventional beam search. Empirical results on four benchmarks show that NeuroLogic Decoding outperforms previous approaches, including algorithms that handle a subset of our constraints. Moreover, we find that unsupervised models with NeuroLogic Decoding often outperform supervised models with conventional decoding, even when the latter is based on considerably larger networks. Our results suggest the limit of large-scale neural networks for fine-grained controllable generation and the promise of inference-time algorithms.

## introduction text generation applications often need to incorporate semantic constraints, i.e., what words should and shouldn't appear in the output generation. consider the task of generating a recipe from a set of ingredients @xcite , such as 'garlic,' 'steak', and 'soy sauce' (figure 1 ). a generated recipe should cover all of those ingredients, without hallucinating new ones (such as 'pork' or 'beans'). this restriction, like others in figure 1 for other the physician told the baker that she had cancer. der arzt sagte dem bäckerin, dass er krebs habe. recipe generation @xcite input output dish name 2 tsp butter, 1 beef steak, 1/4 tsp soy sauce, 1 tsp parsley, 1/8 tsp salt , 1/2 tsp garlic garlic butter steak ## method in this section, we first rigorously define predicate logic constraint, and then present in detail the neu-rologic decoding algorithm. ## related work neurologic distinguishes itself from past works in constrained decoding in 3 fundamental ways. ## experiments i: constrained commonsense generation commongen (lin et al., 2020) is a benchmark dataset designed as a test of generative commonsense reasoning. given a set of common concepts (e.g., dog, frisbee, catch, throw); the task is to generate a coherent sentence describing an everyday scenario using these concepts (e.g., "a man throws a frisbee and his dog catches it"). problem formulation the input is an unordered set of n concepts @xmath0, a 2 , . . . , a n }, where each concept a i is a common object (noun) or action (verb). the expected output is a simple, grammatical sentence @xmath1. to apply neurologic decoding, we impose that each a i must appear in output y under some morphological inflection. ## experiments ii: recipe generation we next study cooking recipe generation, a paragraph-level generation task. given a dish name and a list of ingredients, the task is to generate cooking instructions for the given recipe. ## experiments iii: data-grounded dialogue response generation in dialogue response generation for hotel and restaurant information systems @xcite , we generate a natural language response given a query type (e.g., informing or querying) and a list of facts to convey (e.g., a hotel's name and address). problem formulation the input is a query type, an unordered set of facts @xmath2, ..., f |f | }, where each f i contains attribute and value (i.e. @xmath3, @xmath4. the expected output is a dialogue responses @xmath5. the constraint here is that all given facts f i must be included in responses y in proper natural language form f n i . we use a very simple template to turn f i to natural language form f n i . (i.e. the natural language form for @xmath6. formally, dataset, approach and baseline we use the hotel and restaurant dialogue system corpus and the same train-dev-test split from @xcite . there are 8 query types and 12 attribute types. the standard paradigm for dialogue generation is to consider it as a conditional sentence generation task and finetune a seq2seq model. while this pipeline works effectively with existing data, once we have user queries with new query types or new attribute types, the seq2seq model would not be able to generate plausible responses. the situation can happen frequently with a dialogue generation system in application. thus, we are interested in zero-shot dialogue generation. we give a hand-crafted initial prompt to a pre-trained lm based on the query type and apply neurologic decoding to force given facts to include in generation. the pre-trained lm we use here is gpt-2 @xcite . the baseline we compare against is seq2seq finetuned lms with vanilla beam search, including gpt-2 @xcite , bart @xcite and t5 @xcite . we also compare with previous sota @xcite on dialogue response generation. result ## experiment iv: reducing gender bias in machine translation problem formulation we adopt the task setup and dataset of @xcite . the input x is an english sentence describing a scenario with human entities @xmath7, . . . , n |n | } who are identified by roles. the desired output is a translation y which uses the correct gender inflections in the target language (here, german or french). we obtain indicators of people's gender identity through coreference resolution, linking each entity with their gendered pronoun. 3 we then constrain the correctly-gendered human entities to appear in output y. for a human entity n i , let n f i denote its female inflection in the target language, and n m i denotes its male inflection. let f denotes the set of human entities associated with female characters, and m denotes the set of entities associated with male. formally, the constraint is ∀@xmath8, d(n f i , y) ∧ ¬d(n m i , y) ∧ ∀@xmath9, d(n m i , y) ∧ ¬d(n f i , y) dataset we use @xcite 's dataset, which is built over the english-only coreference gender-bias studies: winogender @xcite and wino-bias @xcite . result our results are shown in ## conclusion we propose neurologic decoding, an efficient and general method for generating with arbitrary predicate logic constraints. we demonstrate its intuitive application to 4 different tasks as an extension to existing models, showing broad and consistent improvement to decoding quality.