Unlocking Anticipatory Text Generation: A Constrained Approach for Large Language Models Decoding (2024)

Abstract: Large Language Models (LLMs) have demonstrated a powerful ability for text generation. However, achieving optimal results with a given prompt or instruction can be challenging, especially for billion-sized models. Additionally, undesired behaviors such as toxicity or hallucinations can manifest. While much larger models (e.g., ChatGPT) may demonstrate strength in mitigating these issues, there is still no guarantee of complete prevention. In this work, we propose formalizing text generation as a future-constrained generation problem to minimize undesirable behaviors and enforce faithfulness to instructions. The estimation of future constraint satisfaction, accomplished using LLMs, guides the text generation process. Our extensive experiments demonstrate the effectiveness of the proposed approach across three distinct text generation tasks: keyword-constrained generation (Lin et al., 2020), toxicity reduction (Gehman et al., 2020), and factual correctness in question-answering (Gao et al., 2023).

## introduction large language models (llms) exhibit impressive textual understanding and reasoning capabilities as evidenced by various studies @xcite @xcite . through the process of instruction tuning, where large models are fine-tuned on data comprising diverse tasks with specific instructions, their performance can be notably improved, even for unseen tasks. however, despite their strong abilities in text understanding and generation, undesirable behaviors such as toxicity @xcite and hallucination @xcite still persist. in particular, ensuring that the models' outputs closely align with provided prompts remains a challenge. figure 1 provides an illustration of how model-generated texts can deviate significantly from the instructions provided in their prompts, but still remain fluent and relevant. traditional sampling methods like nucleus sampling @xcite , top-k sampling, and temperature sampling, as well as search-based methods like greedy or beam search, typically do not take future costs into account. @xcite introduced various heuristics to approximate future lexical constraints. we focus on general language constraint situations @xcite three different language constraints for text generation tasks and using the estimation of future satisfaction score to guide generation. specifically, in order to mitigate undesirable behaviors and ensure faithfulness to instructions, we propose a novel approach for text generation (section 2), by formalizing it as a problem constrained by future language generation. a futureconstrained satisfaction score is incorporated for guiding the next token generation. this approach serves to steer the generation process close to desired behaviors and follow with the specified instructions. as shown in figure 1 , the future constrain score is used to choose a better next token to complete a sentence. a future-constrained satisfaction score is the distance for current generation to satisfy the constraint goal. however, the estimation of this score can be np-complete @xcite . recent investigations by @xcite ; @xcite ; @xcite have showcased the promising potential of utilizing large language models for evaluation on various natural language processing tasks. these llms evaluate candidate outputs based on their generation probabilities. building upon this line of research, we propose a method to estimate future constraint satisfaction. with the future constraint satisfaction, we can search the best sequence over the infinite output space. in order to speed up the process, we present figure 1 : an illustration of the proposed approach utilizing future constraint satisfaction to guide generation. in this example, although "summer" is a more likely next token, generating it will lead to a lower score in the future constraint, which includes the keyword "snow". our method incorporates future constraint satisfaction, making "winter" a more preferable choice. a beam-based algorithm meticulously crafted to recursively generate sequences from left to right, remarkably enhancing the efficiency and efficacy of the generation process. the experimental results (section 3) exhibit desired behaviour improvements in three different tasks: keyword-constrained generation, toxicity reduction, and factual correctness in question answering. we also conduct speed and human evaluation (section 4) of our approach. the decoding time slowdown linear with the number of candidates at each step foot_1 . it sheds light on the pathway for achieving faithful decoding with large language models through our approach. ## method we start by revisiting the generic generation process of an autoregressive language model. given a prompt, represented as a sequence of tokens x, a language model generates an output sequence y step-by-step, proceeding from left to right: here p(y t | @xmath0, x) represents the distribution of the next token at position t given the prompt/prefix x, and the partial output @xmath1. all sequential tokens are generated iteratively based on this conditional probability distribution. there are several popular deterministic decoding methods such as greedy decoding and beam search, as well as non-deterministic sampling methods like temperature sampling, nucleus sampling (holtzman et al., 2020), and top-k sampling. in this context, our focus primarily revolves around deterministic decoding techniques. in this work, we are exploring a distinct formulation to ensure that the generated output y exhibits specific desired behaviors (e.g., reduced toxicity or inclusion of certain keywords). the conditional sequence probability can be derived as follows: the term r(@xmath2,c(x)) denotes the future con-straint satisfaction score, given an output prefix y and a constraint c(x). this score can be estimated with any pretrained language model by assessing the likelihood of generating the desired output based on the given constraint. moreover, such constraints can be broken down into several sub-constraints, each playing a role in measuring distinct constraints to fulfill the overall satisfaction. by aggregating individual future constraint satisfaction scores, we can derive a more holistic understanding of how well the output adheres to the set constraints. ## experiments we investigate the performance of the proposed method on three different tasks: keywordconstrained generation, toxicity reduction, and factual correctness in question-answering. ## analysis speed we test the wall-clock running time of greedy decoding, our method, and the standard beam search. we follow the same configuration. the result is shown in ## related work previously, there are several work like ctrl @xcite , pplm @xcite , gedi @xcite , fudge @xcite on controllable generation. they use additional code or attributes for controllable generation. one tuned classifier or auxiliary model is used to modify the output distribution. the type of control is limit (a label or a category of the sequence). in this work, the constraints are verbalized in natural language. any natural language constraint can be suitable for our method. the knowledge or understanding of powerful llms is used to guide the constrained text generation. another related approach in constrained generation involves refinement with llms after each completion @xcite . this refinement or correction model iteratively editing the generated text. multiple generations are often required, particularly for long-form question-answering tasks, such as eli5 @xcite . another direction in constrained decoding @xcite a lp a c a -7 b -i n s tr u c t f a lc o n -7 b l la m a -2 -1 3 b f a lc o n -7 b -i n s tr u c t commongen eli5 greedy 1.0s 10.2s beam search 1.5s 22.1s our 4.8s 63.2s learning (rl). the generator model parameters need to be updated in this approach. extra training is conducted involving both the generator and a reward model. our work is inspired by a* algoirhtm @xcite , a search algorithm that seeks the highest-scoring path by utilizing heuristic estimations of future scores toward the goal. recently, @xcite ; @xcite develop several heuristics to estimate look-ahead scores. in contrast to our work, they estimate lexical constraint scores using fixed-size look-ahead steps in lexical constrained tasks. in the work of fudge @xcite , an auxiliary binary classifier is trained with random input sequence truncation. recently, @xcite learned a token-level discriminator for knowledge-grounded dialogue and abstractive summarization. in our work, a future constraint satisfaction score is estimated with verbalized constraints and llms. ## future work and conclusion in this work, we delved into decoding methods for llms to mitigate undesired behaviors. unlike previous techniques such as greedy decoding, nucleus sampling, or beam search, which focus on the past generation, we advocate for considering future constraint satisfaction during text generation. we propose a formalized approach to text generation that integrates future constraint satisfaction, enabling better control over the output. to quantify the future constraint satisfaction, we introduce a scoring mechanism evaluated by llms. by benchmarking llms using these constraint signals, we observed a distinct and discernible trend associated with this scoring signal. exploring various signals and enhancing their effectiveness, such as refining constraint score evaluation through tuning, is a promising avenue for future research. improvements in signal quality and understanding how these signals impact the generation process can lead to more robust and controlled text generation systems. incorporating more symbolic components into the estimation could be beneficial. this approach would allow for the inclusion of detailed reasoning paths as integral elements of the estimation. it can be with more interpretation and reliability. this part can be a promising direction for future work. limitation of correctness evaluation. this work primarily prioritizes the correctness of constraint satisfaction. however, in question answering, the generated output of a question may include correct claims alongside hallucinated information. each piece of information in a generation is not guaranteed to be factually supported by a reliable source of knowledge. future work can explore methods to enable llms to generate not only correct answers but also minimize the inclusion of hallucinated information. in the question answering task, we utilized 4 a100 gpus. the beam size was set to @xmath4.