Self-Regulated Sample Diversity in Large Language Models (2024)

Abstract: Sample diversity depends on the task; within mathematics, precision and determinism are paramount, while storytelling thrives on creativity and surprise. This paper presents a simple self-regulating approach where we adjust sample diversity inference parameters dynamically based on the input prompt—in contrast to existing methods that require expensive and inflexible setups, or maintain static values during inference. Capturing a broad spectrum of sample diversities can be formulated as a straightforward self-supervised inference task, which we find significantly improves the quality of responses generically without model retraining or fine-tuning. In particular, our method demonstrates significant improvement in all supercategories of the MMLU multitask benchmark (GPT-3.5: +4.4\% , GPT-4: +1.5\% ), which captures a large variety of difficult tasks covering STEM, the humanities and social sciences.

## introduction large language models (llms) and the broader class of foundation models, such as gpt-3 @xcite and @xcite , learn a distribution over large datasets that can be sampled with guidance prompts. these models have shown remarkable capabilities across tasks without specialised training @xcite , where innovative prompting strategies can even outperform special-purpose tuning, improve reasoning @xcite , and potentially remove the need for expert-curated content @xcite . however, these models employ stochastic sampling from the probabilities predicted by the model to generate responses @xcite , which is arguably both their weakness and strength-to quote karpathy "an llm is 100% dreaming and has the hallucination problem. a search engine is 0% dreaming and has the creativity problem." this presents an inevitable tradeoff @xcite . in this paper, we continue the trend of innovative prompting strategies @xcite , and ask whether models can selfregulate their sample diversity given this trade-off. intuitively, it is an easy problem to assess whether a task should be approached logically or creatively. notably, the "unreliable tail" is to blame for degenerate responses, leading to sampling approaches that control the shape of the distribution, suppressing this unreliable distribution tail @xcite . most popularly, "top-p" (nucleus sampling), "top-k" @xcite and "temperature τ " parameters select likely points from the distribution, where τ skews the softmax weights. increasing τ > 1 gives more uniform (random) probabilities and τ < 1 sharpens the distribution, increasing the likelihood of predictable (non-diverse) samples. the "frequency" and "presence" parameters also penalise repeated tokens or promote tokens that have not yet occurred in the text accordingly, implicitly altering the diversity of completions. approaches to managing sample diversity in language models, such as large-scale transformers, often rely on fixing these parameter values @xcite or employ learned context @xcite and fine-tuning @xcite . however, the current adaptive methods are often expensive and inflexible, requiring bespoke solutions for specific contexts or auxiliary training that is not suited for foundation models. in contrast, we introduce a simple prompting strategy that dynamically adjusts diversity parameters based on the input task context, without requiring retraining, auxiliary networks or fine-tuning. the primary contributions of this approach therefore lie in its simplicity, adaptivity, and ease-ofuse-where it is directly applicable to foundation models and complements other strategies. in particular, we find that our method demonstrates marked improvement across the mmlu benchmark @xcite evaluated for gpt-3.5 (+4.4%) and gpt-4 (+1.5%) models. the cat sat on the estimate params for the prompt ''the cat sat on the.... mat, and purred loudly. extract params assess diversity for task 2 related work sample diversity and prompting strategies are active research fields @xcite . here, we categorise related literature according to the way the model distribution is sampled, including static, learned, and task-dependent approaches, and also we review the wider societal impact of sample diversity and amplification effect of model biases. static sampling a significant portion of prior work focuses on static sampling methods (holtzman et al., 2020), predominantly with fixed diversity parameter settings such as for temperature and top-k sampling @xcite and top-p nucleus sampling @xcite . while clearly effective, these methods lack the flexibility to adapt to varying task requirements; it is difficult to find the balance between excessively repetitive answers (such as repeated tokens in mathematics) or excessive randomness in the model outputs. learned heuristic and conditioned models more recent studies have explored learned heuristic approaches for sampling diversity, such as by adjusting sampling according to the model @xcite . similarly, generation can be learned in a conditioned way @xcite that controls style, content and task-specific behaviour @xcite ; however, these methods can be expensive with more limited adaptivity and applicability with large foundation models. context-dependent sampling researchers have recognized the need for context-specific adjustments to the model sampling parameters; prompt engineers have developed cheat sheets (openai developer forum contributors, 2023) and api sampling guidance (chatgpt openai api plugin contributors, 2023) over a variety of tasks. as expected, the creative writing tasks have been empirically observed to benefit from higher sampling temperatures than coding tasks. discovering the best prompts for tasks is a challenging problem; @xcite optimized to discover the compelling instruction of "take a deep breath and work on this problem step-by-step" that scores highly. diversity can be controlled in more specific contexts with bespoke solutions @xcite . within the task of source code generation, @xcite employs an adaptive temperature sampling heuristic based on the location of tokens within a code block. while effective, these strategies lack the adaptability that our work introduces. diversity within other modelling approaches and data modalities other modelling approaches besides autoregressive next token prediction involve trade-offs in terms of mode coverage, modelling quality and sampling costs @xcite . for example, sampling low temperatures from models trained on the ffhq image dataset yields batches of 20-30 year old males with plain white backgrounds and short brown hair, as shown in figure 6 in @xcite . prompt guidance enables greater modelling fidelity, where model hyperparameters significantly impact creative outputs @xcite . ## methodology given a llm f θ with alphabet tokens σ = {possible characters} trained on strings σ @xmath0, s 2 , . . . , s k : s @xmath1, we wish to self-regulate the sample diversity of f θ based on the context of the prompt. we hereon use "sample diversity" as an umbrella term covering the likelihood and randomness of the model outputs, as well as other factors such as their repetition in the text. the sample diversity is adjustable at inference via a set parameters @xmath2, w 2 , . . . , w n ] (in our experiments temperature τ , top-p, 'frequency' penalty, and 'presence' penalty are used). however, these are best tuned according to the task, which is an ill-defined problem subjective to the current world state, i.e., societal biases, which may have changed since the llm f θ was trained. therefore we wish to specify w at inference. to achieve this, we introduce a guidance prompt @xmath3, g 2 , . . . , g k (such as "based on the following prompt, choose the temperature. . . ", which is concatenated with the task @xmath4, x 2 , . . . , x m (such as "solve this equation. . . ", or "write a poem. . . "), thus guiding the specification of w based on x. more formally, we first generate a string s of parameter values in consideration of the task: in other words, the model output is converted to a real vector w via ψ. then, we continue the prompt (and solve the task) using the updated diversity parameters w ′ , giving notably, the subsequently generated text is not biased by the guidance prompt, although the diversity parameters remain constant until the model sampling is completed. the proposed approach is formulated in the pseudo code algorithm 1: ## experiments our experiments were conducted on the massive multitask language understanding (mmlu) dataset, a benchmark comprising 57 tasks across diverse domains and grouped into 4 supercategories: humanity, stem, social sciences, and other. the multitask tests encompass a total of 14,079 multiple choice questions, with each subject containing at least 100 test examples @xcite . this diversity in content and structure provides a comprehensive platform for assessing the effectiveness of our proposed method over many areas. ## conclusion and future work in conclusion, we found that adjusting sampling parameters contextually based on the prompt significantly improves various tasks in different fields. this follows the trend of advances obtained solely from the remarkable power of prompting in foundation models, and indicates another piece of early evidence that sufficiently large models can demonstrate emerging capabilities of self-evaluation and self-regulation, possibly indicating to a future trajectory of prompt-driven alignment and improvement. it would be worthwhile exploring this space further in the future, examining how prompting strategies can be used to drive performance, alignment and bias mitigation-not only during model inference, but also within model design and training phases within a continual learning cycle.