Waste Not, Want Not; Recycled G umbel Noise Improves Consistency in Natural Language Generation (2025)

Abstract: Consistency in the output of language models is critical for their reliability and practical utility. Due to their training objective, language models learn to model the full space of possible continuations, leading to outputs that can vary significantly in style, content, and tone, even for similar inputs. To address this, we propose a novel decoding algorithm that enhances response consistency across different prompts with no degradation in response quality. By incorporating a latent variable into the next-token sampling process based on the Gumbel reparametrisation trick, our method outperforms standard sampling by up to 10% across semantic and stylistic consistency benchmarks. Additionally, our approach integrates seamlessly with existing sampling methods with negligible computational overhead, providing a practical solution for improving the reliability of language model outputs.

## introduction in recent years, state-of-the-art language models (lms) have demonstrated remarkable performance across a wide range of benchmarks, often rivaling human capabilities in tasks such as translation, summarization, and question-answering @xcite . however, these advancements have not always translated into practical usefulness for real-world applications, where reliability and consistency are crucial @xcite . one of the primary challenges is the inconsistency of these models' responses, which can vary significantly in style, factual accuracy, and tone @xcite . this inconsistency, a byproduct of the probabilistic nature of language model training, can lead to a range of issues, including reduced trust in outputs, exposure to more * definition: if you use resources wisely and avoid waste, you'll never suffer from a shortage diverse failure modes, and less reliable behaviour @xcite . although traditional methods (e.g the use of random seeds) can be applied to introduce determinism in natural language generation, ensuring identical responses for identical inputs, they do not help ensure similar responses when inputs are similar. in practice, due to the richness of language, input queries can often be reworded in many ways while retaining their meaning. to achieve greater consistency, it is desirable for the model to generate similar responses across all these variations @xcite . in this paper, we investigate whether next-token sampling procedures can be modified to enhance consistency across different prompts. our main contributions include: in particular, we highlight that our combined sampler outperforms standard sampling across all benchmark suites and models tested, by up to 10% in some cases. ## related works decoding approaches language model decoding strategies can be broadly classified into two categories: optimization and sampling-based approaches @xcite . optimizationbased approaches, such as greedy decoding and beam search @xcite , frame text generation as an optimization problem, searching for sequences that maximize a specific metric such as probability, whereas sampling-based approach incorporate stochasticity into the next-token selection process. optimization-based approaches are typically perceived as yielding less engaging but more accurate responses and so are often favoured for closedended tasks expecting a fixed answer @xcite . however, recent work has put into question the greater accuracy of their responses @xcite . in contrast, sampling-based approaches are usually preferred for open-ended tasks, as they typically yield more engaging answers @xcite . our proposed method falls within this category. many existing methods in the literature, such as nucleus sampling and mirostat @xcite @xcite , aim to improve text generation quality by directly modifying the probability distribution from which tokens are sampled. we consider these methods, which directly alter the nexttoken distribution, as complementary to our approach, which maintains the next-token distribution and instead modifies the joint distribution over responses. our approach is methodologically most closely related to methods @xcite which also adjust the joint distribution of sampled responses. however, while these methods aim to maximize response diversity-an advantage when ensembling multiple responses as done in self-consistency voting @xcite )-our approach is distinct in its focus on minimizing response diversity to achieve more consistent outputs. self-consistency language models lack robustness to prompt variations @xcite and give contradictory responses in such cases, motivating the need for enhanced self-consistency. self-consistency in language models has been studied from many different angles, but usually with a focus on factual rather than stylistic consistency. prior work has proposed a number of fine-tuning approaches for increasing self-consistency, including fine-tuning approaches for increasing the ability of language models to respond consistently to paraphrases of questions @xcite , and approaches for correcting model contradictions using a factor graph over beliefs @xcite . our approach is methodologically orthogonal to previous approaches for enhancing selfconsistency. previous work has relied on finetuning which not only is more cumbersome to implement but also modifies the raw next-token probabilities, potentially affecting responses in unforeseen ways or contributing to catastrophic forgetting. since our approach only modifies the joint distribution over responses without modifying the next-token probability distribution, it does not suffer from the same issues, and comes with principled guarantees around maintaining the model's original response style and quality. additionally, it enhances all aspects of self-consistency, not just factual consistency of responses. ## problem statement let x be a language model prompt composed of a sequence of tokens drawn from a vocabulary of size n v , and let π θ be a language model trained on the task of next-token-prediction. for the remainder of the paper we denote a forward pass through the language model by h @xmath0, y 1:t-1 ) where h @xmath1, with ∆ nv-1 denoting the (n v -1)-dimensional probability simplex. the sequence y 1:t represents the full response obtained by auto-regressively applying the language model with the next token at each step sampled from the categorical distribution parameterized by the model, y t ∼ cat(h t ). in what follows, we use a subscript to represent position in a sequence, and a superscript to represent the token index. so for example, h i t represents the probability of sampling token i at position t. suppose that u is a different prompt that is semantically similar to x for which we generate a response @xmath2:m . motivated by the inconsistency of lm responses, our goal is to modify the lm sampling procedure in a way that increases the similarity between responses y and figure 1: motivating toy example highlighting the aim of our approach. even when language models yield similar probability distributions over responses (top), responses sampled independently (bottom left) can be inconsistent or contradictory due to the inherent stochasticity of sampling. by generating responses in a correlated manner (bottom right) it is possible to alleviate inconsistencies across responses while still respecting the marginal probabilities of each response. in this paper we propose, gumbel consistent sampling, an approach for increasing response consistency through drawing correlated responses, by conditioning all responses on a shared latent variable, that is robust to differences between probability distributions over responses. v according to some yet-to-be-specified notion of similarity. furthermore, we focus on sampling approaches that modify the joint probability of responses p(y, v ) without affecting the marginal probability of individual responses, p(y ) and p(v ), to guarantee that quality of the original responses is maintained. ## approach our proposed sampling approach, motivated in figure 1 , modifies the joint probability distribution over responses by introducing a latent variable g to the sampling process. conditioning the generation of distinct responses on a common realisation of this latent variable introduces a statistical dependency between them. generating responses with greater similarity can then be straightforwardly done by conditioning the generation of all responses on a common realisation of the latent variable, that is to say to sample y ∼ p(y |x, g) and v ∼ p(v |u, g). to ensure the efficacy of the approach, we design the latent variable in such a way that conditioning responses on a common value of the latent variable makes responses as similar as possible. to ensure the preservation of the probability distribution parameterized by the language model, we sample the latent variable from a probability distribution g ∼ p(g) such that marginalising over the latent variable recovers the original distribution over responses, e g [p(y |g)] = p(y ). to construct a latent variable with the above properties, we employ the reparametrization trick for categorical distributions. introduced for normal distributions in @xcite and extended to categorical distributions in @xcite @xcite , the reparametrization trick is a procedure that refactors the sampling from a distribution into a deterministic function of the parameters and a draw from some independent noise with a fixed distribution. for a categorical distribution with parameters p 1 , ..., p nv , this can be cast as first drawing random noise @xmath3, ..., g nv ) where each g i ∼ g(0, 1) is independently drawn from the gumbel distribution @xcite and selecting a category k according to @xmath4. theorem 4.1. suppose we have two different categorical distributions parametrized by p 1 , ..., p nv and q 1 , ..., q nv . define a joint distribution over pairs of categories (y, v ) by defining theorem 4.1 (proved in appendix a) shows that interpreting the gumbel noise as a latent variable and conditioning sampling events on the same realisation of this latent variable increases the probability of selecting the same category with both distributions compared to sampling from each categorical distribution independently, with identical sampling outcomes in the limit where p and q become identical. since generating a response using a lm consists of successive draws from categorical distributions, the above idea can be applied to language modelling in order to increase the token overlap across distinct responses. indeed, we can generate ahead of time a sequence of independent gumbel latent vectors, g 1:t , one for each position in the sequence up to the maximum sequence length, and sample each token using the gumbel latent vector assigned to that position in the sequence when generating a response. that is to say, drawing y t ∼ p(y t |h y,t , g t ) and v t ∼ p(v t |h v,t , g t ), where here we denote by h y,t and h v,t , the next-token probabilities obtained by running the language model on the context-up-to-now (i.e. h y,@xmath5, y 1:t-1 ), h v,@xmath6, v 1:t-1 ) ). we refer to the above approach as gumbel consistency sampling, gcs. this sequential gumbel sampling approach increases similarity of responses by increasing the rate at which identical tokens are generated at fixed positions in the sequence p(y @xmath7, v @xmath8, v @xmath9. we expect that two similar responses are likely to contain some of the same tokens, but likely in different positions, so it would be advantageous for our final sampling approach to reflect this. introducing such an inter-position correlation in sampling outcomes across sequences is made challenging by the requirement of conditional independence between sampling steps. indeed, to respect the lm's probability distribution, it is necessary for sequential sampling steps to be independent of each other, i.e. for p(y t+1 |x, y 1:t ) = p(y t+1 |h t+1 ) = cat(y t+1 ; h t+1 ) which prevents the direct reuse of gumbel samples across sequence positions. theorem 4.2. consider a sequence of tokens y 1:t generated auto-regressively according to the following update rule, where k := arg max j (g j t + log h j t ), q(•) is the quantile function for the g(0, 1) distribution and π θ (•) is a language model: with this update procedure, the probability distribution over a given token conditioned on preceding tokens is in theorem 4.2 we introduce a procedure for recycling a gumbel vector after applying the gumbel reparameterization trick. we prove in appendix b.1 that this procedure is functionally equivalent to independently sampling each token from the true model probability distribution. this means that repeated application of the gumbel reparametrization trick with this recycling procedure yields sequences that are indistinguishable from those obtained by independent sampling at each step from the model's categorical distribution. this property enables the generation of highly correlated responses while preserving adherence to the model's probability distribution. by sampling a single gumbel vector and reusing it across all generated responses, each response remains faithful to the model's predicted probabilities while also being inherently correlated due to the shared gumbel noise. moreover, because the gumbel noise remains highly similar before and after recycling, responses exhibit strong interposition correlations across different sequence positions. we explicitly present the overall procedure in algorithm 1 as well as an illustrative python implementation in appendix i. we refer to this generation approach as gumbel consistency sampling with recycling, (gcswr). at each sequence position, the algorithm resamples a new algorithm 1 gumbel consistency sampling with recycling (gcswr) input: context x, sequence length t , language model parameters θ output: generated token sequence the standard procedure for autoregressive token sampling, which is equivalent to independent sampling of a new gumbel latent vector for every sequence position and every sequence, acts as a baseline for subsequent experiments, and is denoted as independent sampling, (is). ## ensembling semantically similar responses a complementary approach to enhance consistency between responses given semantically similar prompts is to reduce the impact of semantically irrelevant prompt attributes on the next-token probability distributions, which can be achieved by increasing the similarity between the sampling distributions. in our experiments, we explore sampling tokens from an ensembled probability distribution over semantically equivalent prompts as a means of minimising impact of semantically irrelevant prompt variations on responses. specifically, we generate semantically equivalent variations of the user prompt by asking a separate lm (gpt-4o mini) to rephrase the prompt. we then run the target lm separately on all of the prompts, producing a set {p i } of next-token probability distributions. we then sample from an ensembled distribution, ensembled using the following formula: this formula corresponds to selecting the categorical distribution that minimizes the average forward-kl divergence over all next-token probability distributions (see appendix d). we found that direct averaging (which can equivalently be shown to minimize the reverse-kl distribution) tended to generate worse-quality responses due to at times sampling tokens that were only highprobability for a subset of question rewordings. note that, contrary to our proposed gumbel sampling approach, ensembling comes at a cost of additional inference-time compute and also modifies the language model probability distributions. we highlight that ensembling can be applied in conjunction with any of the three samplers discussed in section 4, and we investigate the performance of each sampler with and without ensembling in our experiments. ## experiments in our experiments, we empirically demonstrate the utility and limitations of gcs and gcswr. we begin by quantifying the utility of the procedure for enhancing semantic similarity of responses, and highlight a number of stylistic dimensions of text along which gumbel sampling improves consistency. details for reproducing experiments are shown in appendix e. ## conclusion we have introduced gumbel consistency sampling, a straightforward and computationally inexpensive sampling approach for increasing consistency amongst model responses. the method requires no additional fine-tuning, additional language model calls or apriori knowledge of what prompts will be used, and guarantees responses indistinguishable to those obtained using standard sampling at the level of individual responses. the approach enhances consistency by sampling responses in a correlated manner through the introduction of a latent variable, in a way that increases the token overlap across responses. in our experiments, we find that this approach is not only able to enhance semantic similarity between responses but also stylistic similarity. these results showcase how gumbel consistency sampling offers a principled quick and easy way of enhancing language model consistency. future work could extend the gumbel consistency sampling to imposing local rather than global correlation to responses. currently, all responses are globally coupled due to dependence on the same global latent variable, which makes localised adjustments to model behaviour impossible. however, the framework could easily enable for latent variables to be varied locally depending on question specifics, which would enable finer-grain control of model behaviour and could increase the overall response diversity. another, promising direction for extending the work could be to treat the gumbel noise as a learnable taskspecific parameter. such an approach may be especially useful for building stronger model safeguards while preserving general utility.