Controlled Text Generation for Black-box Language Models via Score-based Progressive Editor (2024)

Abstract: Controlled text generation, aiming to ensure that language models produce text containing only the desired domain or corpus attributes, is immensely crucial in the practical application of language models. Existing methods, however, are inapplicable to black-box models or suffer a significant trade-off between control and fluency in text generation. This paper introduces the Score-based Progressive Editor (ScoPE), a novel approach designed to overcome these issues. ScoPE modifies the context at the token level during the generation process of a backbone language model. This modification guides the subsequent text to naturally include the target attributes. To facilitate this process, ScoPE employs a training objective that maximizes a target score, comprehensively considering both control and fluency. Experimental results on diverse controlled generation tasks demonstrate that ScoPE can effectively regulate the attributes of the generated text while effectively utilizing the capability of the backbone large language models.

## introduction modern language models can now generate text as fluently as humans in response to any given sequence or instruction @xcite @xcite @xcite @xcite . however, the generated text may carry potential risks, such as harmful expressions or inappropriate content @xcite @xcite . if the previous context includes non-preferred attributes, these attributes can potentially appear in the generated text. therefore, controlled text generation, which aims to generate * corresponding author text constrained to target domain attributes regardless of the given context, is crucial for addressing the current issues in language models @xcite @xcite @xcite . recently, many large language models, especially those exceeding hundreds of billion parameters, are presented as de facto black-box models with limited access to model parameters @xcite @xcite . consequently, most existing approaches to controlled text generation that require access to the model's parameters are either inapplicable or have limitations in these black-box scenarios. while access to the language model parameters is unavailable, some previous works assume access to the output token distribution and have studied controlled generation by manipulating this distribution without tuning the parameters @xcite @xcite . however, this approach significantly diminishes the fluency of the generated text, as it manipulates the output distribution based on the previous context. therefore, there is a requirement for a novel approach to effectively leverage the generation performance of black-box large language models. in this paper, we propose score-based progressive editor (scope) to address controlled generation for black-box language models and tackle the trade-off between control and fluency. scope modifies the intermediate output text token blocks during the generation process of a backbone language model, ensuring that the edited text tokens contain the target attributes. this approach effectively guides the subsequent generation to naturally include the target attributes. since it does not access the model parameters, including the output distribution, scope can be adapted to any black-box model. by modifying the previously generated context tokens while maintaining fluency, scope ensures that the text generated user scope backbone black-box lm generation finished? y n feed edited text (𝐱, ො 𝐲) into the next 𝒃 generation steps (next block generation) block-wise edit 𝐲 to ො 𝐲 with enhanced target score controlled text generation for black-box lm with scope initial input text 𝐱 intermediate output text block 𝐲 edited text block ො 𝐲 generated text with target attributes i want to generate a text like the camera review this text is about using these commands in the following problems: this text is about using these lenses on the following cameras: this text is about using these lenses on the following cameras: gimbal x1: no need to buy an expensive gimbal. train scope afterward includes the target corpus's attributes without compromising fluency. to effectively incorporate target attributes during the editing process, we introduce a score aligned with the attributes in the target corpus, which is computed by the score model based on a pre-trained masked language model. to further assist scope's text modification, we additionally consider a repetition score and a task-specific score. given the overall target score, scope is trained with the objective of maximizing the score on text samples from a training set composed of sampling from the language models. in our experiment for the diverse controlled generation tasks, using various corpora constructed from the amazon customer reviews dataset, we comprehensively evaluate scope in terms of control and fluency. we discover that scope, using various black-box language models as its backbone, effectively regulates target attributes while preventing a decline in text fluency through experiments including gpt-4 evaluation. additionally, our research confirms the compatibility of scope with instruction prompting for control purposes. in tasks requiring sentiment control, scope offers an effective solution to the tradeoff between fluency and control, a challenge often encountered in existing baseline models. furthermore, through exploring various settings of scope, we affirm its proficiency in managing multi-attribute control. our work presents a distinctive contribution by facilitating fluent controlled text generation utilizing black-box language models, thereby demon-strating their effectiveness and versatility for a wide range of controlled text generation tasks within the current context. ## related work the elements addressed by previous approaches for controlled text generation can be primarily divided into three categories: input context, weights of language models, and decoding strategy. when handling input context, the objective is to effectively incorporate target attribute information into the input of the language model @xcite @xcite . in the case of weights of language models, the approach involves fine-tuning the weights of the model, either partially or entirely, with data from the target domain @xcite @xcite . lastly, the approach to decoding strategy entails using adaptive modules, such as discriminators, that are tailored to the target domain @xcite @xcite @xcite , or employing distributional approaches @xcite @xcite to perform weighted decoding of the language model. the practical applicability of fine-tuning approaches under black-box conditions is limited due to their dependence on model parameters. while methods of the decoding strategy can be adapted to loose black-box conditions when output distribution is approachable, they still suffer from the decreasing fluency of generated texts. the most prominent novelty of our work, compared to these methods, is that it secures controllability for entirely black-box llms, such as the chatgpt api, where both model parameters and output logits are inaccessible. additionally, whereas previous works have encountered a decrease in the fluency of generated text when securing controllability, our approach addresses this issue by ensuring that the text generated after tokenwise editing naturally continues from the edited previous text. there exist studies that perform controlled generation through iterative sampling, including energybased models, aiming to maximize scores computed from various domain-specific modules, starting from text sampled from the initial distribution @xcite . however, due to numerous iterative sampling steps, they result in significantly slower generation speeds compared to standard language model generation, reducing its practicality. about this issue, our work achieves a significant improvement in generation speed by utilizing the trained editor that only requires a few editing steps. recent studies introduced the method for adapting gray-box language models which only permit access to the generative distribution of the model @xcite . additionally, @xcite proposed a lightweight module that controls black-box language models for specific tasks. our approach effectively adapts the black-box model from a different perspective of this concurrent work. ## experimental setup we conduct a controlled generation evaluation using the amazon customer reviews dataset foot_0 . the dataset includes multiple attributes related to the review style, the categories corresponding to the type of reviewed product, and the sentiments aligned with the product rating. consequently, this dataset is well-suited for the creation of diverse corpora that contain specific attributes. we classify the dataset corpus based on category and sentiment. for categories, we construct four distinct corpora: camera, videogame, grocery, and music. in terms of sentiment, samples rated with 5 stars are compiled into a positive corpus, while those rated with 1 star formed a negative corpus. details on the dataset statistics are discussed in appendix b. using these corpora, we evaluate controlled generation tasks for both category and sentiment attributes. in each task, we test the controllability and fluency of generated continuations when fixing one target attribute among the attributes relevant to the task, starting from an input text with arbitrary attributes. experimental details for our experiments are mentioned in appendix c. ## conclusion we present scope which guides the generation process of a backbone language model to improve target domain scores, enabling fluent controlled text generation. scope effectively addresses the challenges associated with the black-box scenario and the trade-off between controllability and fluency. furthermore, it demonstrates its pragmatic utility when incorporated within various large language model apis, manifesting its tangible applicability in real-world contexts. we believe scope, combined with improved functions, can be effectively utilized for the adaptation of large language models to specific domains or tasks in future works.