Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLM s (2024)

Abstract: Controlling the attribute intensity of text generation is crucial across scenarios (e.g., writing conciseness, chatting emotion, and explanation clarity). The remarkable capabilities of large language models (LLMs) have revolutionized text generation, prompting us to explore such smooth control of LLM generation. Specifically, we propose metrics to assess the range, calibration, and consistency of the generated text’s attribute intensity in response to varying control values, as well as its relevance to the intended context. To quantify the attribute intensity and context relevance, we leverage an Elo rating system and GPT4, respectively, both renowned for their robust alignment with human judgment. We look into two viable training-free methods for achieving smooth control of LLMs: (1) Prompting with semantic shifters, and (2) Modifying internal model representations. The evaluations of these two methods are conducted on 5 different attributes with various models.

## introduction controllable text generation (ctg) for meeting certain constraints imposed by the target applications and users is an important topic in natural language generation. for example, it is often required to control sentiment @xcite or politeness @xcite in the task of dialogue response generation. controllable text generation becomes even more crucial as the modern natural language generation system is becoming increasingly tailored to individual preferences. for example, a dialogue response generator may need to compose its answer to a question in a completely different way based on the backgrounds of the user @xcite black holes are like space vacuums, sucking in everything, even light! they form when big stars die. black holes are regions in space with incredibly strong gravity that swallows everything, including light. they are created when massive stars collapse. black holes are cosmic objects formed from the collapse of massive stars. they have intense gravity, creating an event horizon where nothing can escape. studying black holes helps us understand the universe's most extreme conditions and test theories like general relativity. ## problem formulation in this section, we formally define smooth control of the llm-generated text's intensity of a certain attribute, and introduce the benchmark data we construct for the evaluation of this task. ## evaluating smooth control we start with the introduction of our automatic rating system and then introduce the metrics we design to measure the smooth control. ## experiments setup in this section, we apply our proposed evaluation framework, along with the constructed benchmark dataset, to assess the smooth control capability of various modern llms through two viable trainingfree methods: (1) prompting with semantic shifters, and (2) editing the internal model representations. we first introduce the experiment settings and then present the results and analyses. ## conclusions and future work this work studies smoothly controllable text generation for large language models. we created an evaluation system on five different attributes to evaluate smooth control methods on different intensity levels for three metrics: error, variation of the generated sentence's intensities and the relevance to the generation questions. the system is implemented based on elo ratings, automatically evaluating using llms, and designed to be efficient in evaluation. we evaluate two representative methods, prompting and representation engineering. we find that (1) model sizes may negatively affect the smooth performance. (2) prompting is almost as good as, if relatively better than repe.