Palette of Language Models: A Solver for Controlled Text Generation (2025)

Abstract: Recent advancements in large language models have revolutionized text generation with their remarkable capabilities. These models can produce controlled texts that closely adhere to specific requirements when prompted appropriately. However, designing an optimal prompt to control multiple attributes simultaneously can be challenging. A common approach is to linearly combine single-attribute models, but this strategy often overlooks attribute overlaps and can lead to conflicts. Therefore, we propose a novel combination strategy inspired by the Law of Total Probability and Conditional Mutual Information Minimization on generative language models. This method has been adapted for single-attribute control scenario and is termed the Palette of Language Models due to its theoretical linkage between attribute strength and generation style, akin to blending colors on an artist’s palette. Moreover, positive correlation and attribute enhancement are advanced as theoretical properties to guide a rational combination strategy design. We conduct experiments on both single control and multiple control settings, and achieve surpassing results.

## introduction the purpose of controlled text generation is to modify the output of the language models with a pregiven attribute, so that the final output conforms to the attribute @xcite @xcite . it is common to utilize bayes rules @xcite to modify the language model p(x) to form with the conditional variable p(x|a), and by virtue of the corresponding discriminative model (generally a classification model), implement the constraints on the generated results. this approach will face two problems: on the one hand, the discriminative model usually needs to be fine-tuned according to the generation * corresponding authors task scenario to better assist controlled text generation, because the accuracy of the discriminative model plays a key role in the generation effect. nevertheless, it is time-consuming to collect the corresponding classification data. on the other hand, the classification effect of the discriminative model often depends on certain words or phrases related to attributes, therefore, in the process of predicting the next token, the discriminative model will guide the language model to bias these words, which makes the generated results lack diversity. in recent years, with the rapid development of large language models @xcite @xcite , models represented by @xcite have powerful text generation capabilities. many generation tasks can be converted to prompt engineering to use large language models to get better solutions. similarly, attributes can be designed as prompts, so that with the powerful generation ability of the large language model, the final generated text will also conform to such attributes. when the number of attributes that need to be met is large, it is difficult to design a suitable prompt to cover these attributes. also, due to the inherent ambiguity and sensitivity of prompts, we can't guarantee that the generated results will be perfectly reproduced according to the attributes mentioned in prompts. model arithmetic @xcite proposes a framework to ensemble multiple attributes, in which through simple arithmetic operations, such as linear composition, multiple attributerelated language models or discriminative models are combined to obtain a multi-attribute controlled text generation strategy. this framework treats the language models associated with different attributes as independent. however, in reality, each language model may have multiple attributes, and the attributes between language models may overlap which can not be effectively modeled with simple arithmetic operations. for example, the main attribute of language model a is "formal response", and that of language model b is "child's tone" (which suggests an attribute of "informal response" as well). with linear combination, the attribute of "formal response" is faded, making the final generation strategy incomplete. deriving from the law of total probability and conditional mutual information minimization, we propose palette of language models, which alleviates the latent attribute overlapping problem when attributes assembled. the improved combination strategy has following contributions: ## related work some works for controlling language models are training or aligning language models conditioned on static or iterative updated control codes that make desired features of generated text more explicit @xcite @xcite . these methods often depend on a large amount of training data, which will result in a lot of annotation costs and resource consumption. in order to solve the above problems, some researchers have proposed methods of fine-tuning @xcite , prompt tuning @xcite or prefix tuning @xcite @xcite large models to generate controlled text. however, due to the use of gpt series models, it will cause infeasible and cannot solve the toxicity and bias problems of the model @xcite . although timo schick et al. @xcite found that pre-trained models can largely recognize their bad biases and the toxicity of the content they produce, by giving textual descriptions of bad behavior when decoding, the probability of generating problematic text can be reduced. but this method is greedy in nature when supporting or opposing a decision must always use a specific word, taking into account only the context in which it has been generated. another category of method uses gradients of single or combination of attribute discriminators based on energy to update generating language model's hidden states to guide decoding process @xcite @xcite @xcite , which do not need labeled training data. despite the use of langevin dynamics, samples the sequence of token embeddings instead of logits and gibbs sampling methods. the main issue of this category is that multiple sampling iterations are required to converge, which slow down the decoding progress. bolt @xcite , instead, preserves token dependencies and simultaneously optimizes via autoregressive decoding to constrain by adding biases. but bolt requires stricter constraints, for example, keyword control requires more than three keywords. bolt also requires careful tuning of the different hyperparameters constituting the energy function-a problem prevalent in energy-based controlled power generation approaches. some methods are inspired by bayes rules that use attribute probability from discriminators to steer language generation towards desired attributes. the attribute discriminators can be discriminative or generative. fudge @xcite learns a binary discriminator for whether attribute will become true in future, and the output probabilities of this discriminator are multiplied with generator's original probabilities to get the desired probabilities. gedi @xcite uses class-conditional language models as generative discriminators, which results in faster computation speed due to the parallel computation of all candidate tokens. compared with these methods, preadd @xcite considers the language generator with a prefix-prepended prompt as the role of attribute discriminator, which does not require an external model and corresponding additional training data. recent work proposes controlling text generation via language model arithmetic @xcite , which enables to combine multiple language models of different attributes into a formula-based composite. the key distinguishing feature of our method is that we explicitly model the attributes overlapping between language models, so that they are not faded during combination. ## proposed method in this paper, we focus on n (where @xmath0. analogously, single attribute control can be treated as the combination with the basic model. we attach each attribute to a generative language model for ease of the formula derivation. ## experiments we evaluate the improved attribute combination strategy in both single and multiple control scenarios to testify the attribute enhancement. specifically, sections 4.1 and 4.2 are for the single setting, and the section 4.4 is for multi-attribute control which also demonstrates the overlapping alleviation. we conduct experiment for positive correlation and complementary event verification in sections 4. ## conclusions considering underlying overlapping between attributes, we deduce palette of language models, an improved linear combination strategy for multiattribute control, with the law of total probability and conditional mutual information minimization. different from previous linear combination methods, the derived formula owns a dynamic coefficient to each model which can enhance the attribute expression. additionally, two pivotal properties are proposed that serve as guiding principles for designing a rational attribute combination strategy. we conduct comprehensive experiments on both single and multiple attributes control settings which further underscore the effectiveness of our method.