G aussian Process Optimization for Adaptable Multi-Objective Text Generation using Linearly-Weighted Language Models (2024)

Abstract: In multi-objective text generation, we aim to optimize over multiple weighted aspects (e.g., toxicity, semantic preservation, fluency) of the generated text. However, multi-objective weighting schemes may change dynamically in practice according to deployment requirements, evolving business needs, personalization requirements on edge devices, or the availability of new language models and/or objective requirements. Ideally, we need an efficient method to adapt to the dynamic requirements of the overall objective. To address these requirements, we propose a linear combination of objective-specific language models to efficiently adapt the decoding process and optimize for the desired objective without the significant computational overhead of retraining one or more language models. We show empirically that we can leverage Gaussian Process black box optimization to adapt the language model decoder weights to outperform other fixed weighting schemes and standard baselines of the task in only a few iterations of decoding. Overall this approach enables highly efficient adaptation of controllable language models via multi-objective weighting schemes that may evolve dynamically in practical deployment situations.

## introduction multi-objective text generation involves compromises between different objectives. in practice, the importance of each objective may dynamically change due to business needs, personalization, or addition of new objectives due to time-evolving deployment requirements. retraining or fine-tuning the language model (lm) may be impractical for each adaptation of the multi-objective target since it imposes significant computational costs. to address this inefficiency, we propose a multi-objective framework that leverages language model decoders * equal contributions pretrained for each objective and a dynamic weighting of each decoder to adapt to the objective without retraining their corresponding models. more specifically, we propose a method to dynamically adapt the weighting of objective-specific lms at the decoding stage to optimize the desired overall text generation objective. we define the overall problem as one of black box function optimization, where the function inputs are n language model decoders and weights (i.e., w 1 , . . . , w n ) and the output is the chosen objective value. we specifically use gaussian process optimization since it is a popular and efficient tool for black box optimization @xcite . empirically, we evaluate on a range of text detoxificaton tasks that serve as a natural and important testbed for multi-objective language model optimization. we demonstrate that our gaussian process bayesian optimization approach can efficiently and quickly adapt the language model decoder weights to outperform other fixed weighting schemes and standard baselines of the task in only a few iterations of decoding. ## multi-objective text decoding problem definition. for multi-objective text generation, we assume that we have different pretrained and fixed language models representing distinct objectives. for example, we might fine-tune a base language model for non-toxic text generation and separately fine-tune the same model for fluent text generation to provide one decoder for each objective. our goal is to devise an efficient weighting strategy that combines the next-token prediction scores from all language models, without fine-tuning them, to optimize the overall objective. it is challenging to manually determine a set of weights that effectively combines these language models. to tackle this challenge, we frame the problem as a black box function optimization as shown in fig. 1 . the figure shows that our inputs consist of n language models, each associated with a weight (denoted as w 1 to w n ), and the output corresponds to the selected objective value. to optimize the black box function, we leverage bayesian optimization with gaussian processes. we describe our solution in detail below. methodology. suppose s in is the input text and s out is the generated text that we want to evaluate. for that, assume that we have n objective functions, i.e., @xmath0.), ..., o n (.)}, that reflect different properties of text such as non-toxicity or fluency, and n language models that correspond to the foregoing objectives, i.e., @xmath1.), ..., m n (.)}. that is, m i (.) is a language trained to maximize the objective o i (.), for any @xmath2. to represent our preferences over the objectives o, we use a set of thresholds, i.e., @xmath3, ..., t n }. overall objective: we want to generate sequences that satisfy our preferences t over the objectives o as follows: decoding: to satisfy o pref (.), we need to combine the models in m using a set of weights, i.e., @xmath4, ..., w n ], in the decoding process as presented in fig. 1 . the combined language model, denoted by m(.|m, w), chooses the next token s out j by a linear combination of next token probabilities of models in m: finally, we use black-box optimization to learn the optimal weights, i.e., w * : to obtain w * , we use bayesian optimization with gaussian processes. we review bayesian optimization with gaussian processes in appx. b. ## experiments recall that, we use the text detoxification task for our proposed method for multi-objective text generation. the detoxification task is commonly evaluated by three objectives of non-toxicity, semantic preservation, and fluency @xcite @xcite . we discuss our experimental setup below and provide all code to reproduce results on github. foot_0 ## conclusion we introduced black box optimization for fast multi-objective adaptation of language models (lms) by leveraging gaussian process bayesian optimization to efficiently adapt the weights of objective-specific decoders. our experimental results showed that our gp approach was able to quickly adapt to changes in nonlinear, nondifferentiable multi-objective targets in only a few decoding iterations as evidenced by its strong performance compared to a variety of baselines.