Multi-Objective Linguistic Control of Large Language Models (2024)

Abstract: Large language models (LLMs), despite their breakthroughs on many challenging benchmark tasks, prefer to generate verbose responses and lack the controllability of output complexity, which is usually preferred by human users in practice. In this paper, we study how to precisely control multiple linguistic complexities of LLM output by finetuning using off-the-shelf data. To this end, we propose multi-control tuning (MCTune), which includes multiple linguistic complexity values of ground-truth responses as controls in the input for instruction tuning. We finetune LLaMA2-7B on Alpaca-GPT4 and WizardLM datasets. Evaluations on widely used benchmarks demonstrate that our method does not only improve LLMs’ multi-complexity controllability substantially but also retains or even enhances the quality of the responses as a side benefit.

## introduction large language models have achieved remarkable success in generating free-from texts for different downstream tasks or human instructions @xcite @xcite . however, existing llms still lack precise control over the linguistic complexity of their outputs @xcite @xcite , e.g., the total number of nouns, the variation of verbs, etc. linguistic controllability is crucial to creating personalized outputs since those complexity indices directly reflect human reading complexity in multiple aspects. for example, a short yes/no answer is required by some users while a detailed explanation is preferred by others. moreover, recent studies @xcite @xcite have discovered a spurious correlation between the quality reward used in llm alignment and the output length (i.e., a specific linguistic complexity). consequently, llms favor generating verbose responses due to the length bias, which may increase unnecessary reading complexity. it is still an open problem to mitigate the bias without hurting the output quality. while existing llm finetuning techniques such as instruction-tuning @xcite @xcite and reinforcement learning from human feedback (rlhf) @xcite have been demonstrated to be effective in aligning the output with human intent or preference, they only focus on maximizing a single objective. instead, achieving the controllability of multiple complexity indices requires a non-trivial multi-objective optimization that has not been thoroughly studied on llms. rather than solely maximizing or minimizing the complexities, it aims to reach different target complexity values on the pareto frontier. this requires llms to adjust the trade-off among objectives and capture their potential correlations or constraints in the text generation process. in addition, due to the huge space of possible combinations of complexity indices, it could be expensive to collect training data for multi-objective control. in this paper, we take the first step towards multiobjective control of the linguistic complexity of llm outputs. instead of collecting new data, our strategy allows the reuse of existing instructiontuning data. in particular, we annotate the groundtruth responses in a dataset by their complexity metrics evaluated using tools developed in computational linguistics. the multiple complexity indices and their values are appended as tags to the input so finetuning an llm on the linguistic-label augmented data helps build a strong connection between the input tags and the linguistic complexity of the output, hence enforcing the llm to adhere to the complexity requirements during sequential decoding. surprisingly, we observe that randomly sampling a small subset of tags for each training example suffice to obtain controllability over all the complexities of test examples, thereby reducing the required amount of training data. we examine our approach by finetuning llama2-7b @xcite using a linguistic-complexity labeled alpaca-gpt4 dataset (peng et al., 2023) (i.e., the prompts are from the original alpaca dataset while the responses are composed by gpt-4) and a wizardlm dataset @xcite . we do not only observe an expected substantial improvement in linguistic controllability but also a side benefit of enhanced response quality, indicating that finetuning under multiple linguistic constraints can improve the llm for general purposes. compared to unconstrained rlhf and ift methods, which suffer from the length bias @xcite , our approach does not introduce the bias but can improve both the controllability and quality simultaneously. ## finetuning llms for linguistic controllability in this section, we delineate our approach to multicontrol tuning. section 3.1 outlines the linguistic features of interest and describes how we extract them from a given text segment. in section 3.2, we explain how the extracted features are incorporated into the multi-control tuning process. ## evaluation of linguistic controllability during evaluation on a reserved test dataset d test , we aim to measure how the model performs as the linguistic control vector f changes. to conduct such an evaluation, we develop a sampling strategy to sample different control vectors for an instruction x i . specifically, we need a sampling strategy that: (1) is specific to an instruction x i , i.e., the sampled control vectors should not stray too far from the reasonable range for a specific x i . for example, an instruction to "generate a short story" should not have a large value for t_word; (2) ensures the sampled control vectors are always valid, meaning that no linguistic controls conflict with each other (e.g., t_uword should always be less than or equal to t_word), and no control is out-ofbound (e.g., fkre should always be less than or equal to 121.22). to achieve the first goal, we utilize y i 's linguistic feature vector as a reference point and sample new control vectors f ′ from the gaussian distribution centered at f (y i ), i.e., f ′ ∼ n (f (y i ), iσ 2 ). however, since each feature f i (y) has a different range, a small σ for some features may be large for others. to avoid this inconsistency, we standardize all features to unit variance before sampling. more formally, the new control vector f ′ is computed by while sampling a valid control vector can be a challenging problem, verifying its validity is straightforward. this can be done using a simple rule-based method, which we outline in appendix b. we utilize this observation to achieve the second goal by performing rejection-based sampling, i.e., keep resampling a new control vector f ′ until we find a valid one. we will show in section 5.7.3 that σ can serve as a hyperparameter to control the evaluation difficulty. lastly, for each example in d test , we randomly sample a number n, a set of n feature indices c, and k new control vectors f ′ c for controllability and generation quality evaluation. ## discussion and conclusions in this paper, we advance the precise control of linguistic complexities in llms through multiobjective control tuning. our method is straightforward and can be seamlessly integrated with existing instruction-tuning datasets without the need to gather new ones. our training objective concurrently optimizes the llm's ability to follow instructions and its controllability. through our experiments, we find that incorporating this dual-focus strategy significantly improves the llm's generative quality, surpassing the results of instruction fine-tuning alone. this finding suggests that controllability and instruction-following ability may have a complementary effect on each other. additionally, we observe that while state-of-the-art llms achieve impressive natural language performance, they are not easy to control. this emphasizes the need for studying methods that improve controllability in llms.