RSA -Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework (2024)

Abstract: Despite significant advancements in natural language generation, controlling language models to produce texts with desired attributes remains a formidable challenge. In this work, we introduce RSA-Control, a training-free controllable text generation framework grounded in pragmatics. RSA-Control directs the generation process by recursively reasoning between imaginary speakers and listeners, enhancing the likelihood that target attributes are correctly interpreted by listeners amidst distractors. Additionally, we introduce a self-adjustable rationality parameter, which allows for automatic adjustment of control strength based on context. Our experiments, conducted with two task types and two types of language models, demonstrate that RSA-Control achieves strong attribute control while maintaining language fluency and content consistency. Our code is available at https://github.com/Ewanwong/RSA-Control.

## introduction controllable text generation (ctg) focuses on producing natural language texts with specified attributes, such as sentiment and readability. this capability is vital for developing functional and reliable natural language generation (nlg) systems. for instance, dialogue systems must be regulated to consistently generate responses that are low in toxicity and bias @xcite @xcite . similarly, summarization systems are expected to be able to create customized summaries for different users by adjusting readability @xcite . many existing studies in ctg rely on fine-tuning pre-trained language models (plms) on attributespecific datasets @xcite . however, due to the increasing scale of plms, fine-tuning them has become resource-intensive. decoding-based methods that navigate the plm decoding process using guide modules @xcite @xcite have achieved strong attribute control and reduced the need to fine-tune plms, but still require additional datasets and computational resources for training the guide modules. besides, introducing external components could potentially hurt coherence during decoding @xcite . as large-scale plms become more adept at understanding human instructions @xcite , prompt-based methods have emerged as a lightweight way to adapt plms to new domains @xcite . previous research has explored direct prompting @xcite and using auxiliary prompts @xcite @xcite for ctg. nonetheless, due to the black-box nature of plms, precise control via prompt-based methods is still challenging and often leads to unexpected outputs @xcite . in this work, we introduce rsa-control, a novel ctg method that bridges decoding-based and prompt-based paradigms through the computational pragmatic framework of rational speech acts (rsa) @xcite . the rsa framework elucidates the effective and efficient human communication through a mutual reasoning process: speakers adjust their utterances by reasoning about listeners' perceptions, while listeners, in turn, infer the speakers' intentions. inspired by rsa's success in modeling conversational behaviors, our approach explicitly models the interactions between speaker and listener modules, enabling a pragmatic speaker to generate utterances that ensure the accurate perception of desired attributes by the listeners. as illustrated in figure 1 , rsa-control constructs a guide module (pragmatic listener l 1 ) using plms with auxiliary control prompts (literal speaker s 0 ) to achieve controllable decoding of the pragmatic speaker s 1 . by replacing fine-tuned discriminator mod-figure 1 : illustration of rsa-control for generating readable summaries. since s 0 assigns higher/lower probability to "sick" than "bedridden" when conditioned on readable/formal prompts, l 1 can infer that "sick" is more readable than "bedridden". s 1 then selects next tokens that are both readable and consistent with article content. specifically, it first decodes with basic rationality α 0 , and the outputs are fed back into plm and l 1 to compute a self-adjusted rationality parameter αn . the real decoding process is then performed with αn . we apply rsa-control to different ctg task types and plms to showcase its efficacy. in section 4 and section 5, we reduce toxicity and stereotypical bias in open-ended generation with gpt2, a foundation model lacking instruction-following abilities. in section 6, we control llama-2-7bchat, an instruction-tuned model, for readabilitycontrolled summarization. unlike open-ended generation which has no content constraints, the summarization task involves an input-output process where plms receive detailed documents and produce summaries that capture salient information from the input content. therefore, we categorize it as an input-output task. experimental results across both types of tasks and plms show that our approach successfully generates texts that satisfy desired attributes while maintaining language fluency and content adherence. ## toxicity reduction plms are at risk of learning toxic and offensive content from their training data @xcite , hence it is crucial to mitigate these risks before deploying them. we apply rsa-control to gpt2 (radford et al., 2019), a family of foundation models with sizes ranging from 117m to 1.5b parameters, aiming to steer them towards producing safer outputs. we conduct our toxicity reduction experiments on the realtoxicityprompts (rtp) dataset @xcite . the rtp dataset comprises 100k prompts from web data, some of which lead to toxic continuations. the examined plms perform open-ended generation conditioned on rtp prompts without content constraints, and the toxicity of each continuation is measured by the perspective api foot_0 . specifically, perspective api predicts a score between 0 and 1 for six attributes: toxicity, severe toxicity, sexually explicit, threat, profanity, and identity attack, indicating the probability that the continuation exhibits each attribute. we use the challenging subset of rtp which contains 1199 strongly toxic prompts. baselines for the evaluation of rsa-control, we include baselines of various types: dapt (gururangan et al., 2020): a fine-tuning method which further trains gpt2 on non-toxic datasets; gedi @xcite and dexperts @xcite : two decoding-based methods that leverage fine-tuned external modules; self-detoxify @xcite and self-debias @xcite : two prompt-based methods that utilize auxiliary prompts. the first three methods require additional datasets and training, while the last two as well as our method are training-free. we also report the results of a vanilla model and a vanilla model prompted by the target prompt. more details about baseline models are provided in appendix c. ## bias mitigation plms are also prone to inheriting stereotypical biases against certain social groups during pretraining @xcite . to address this issue, we apply rsa-control to mitigate these biases and assess its effectiveness using the crows-pairs benchmark @xcite . crows-pairs features pairs of minimally different sentences, with one being stereotypical and the other being anti-stereotypical (e.g., "my mom spent all day cooking for thanksgiving" vs. "my dad spent all day cooking for thanksgiving"). the proportion of instances where a plm assigns higher probabilities to stereotypical sentences is reported, and a score closer to 50 indicates less bias. nine types of social biases are covered by crows-pairs: race/color, gender, socioeconomic status/occupation, nationality, religion, age, sexual orientation, physical appearance, and disability. templates 2a and 2b from ## readability-controlled summarization we then apply rsa-control to enhance readability control in instruction-tuned plms for news summarization, an input-output task. generating summaries with desired readability levels ensures the extracted information is accessible to readers with varying literacy proficiency @xcite @xcite . while most studies rely on additional model training to steer summarization @xcite @xcite , large-scale plms have shown the capability of generating summaries in desired styles following natural language instructions (pu and demberg, 2023; @xcite . thus, we adopt llama-2-7b-chat @xcite hereafter referred to as llama-2) for readabilitycontrolled summarization, aiming to improve its control results beyond direct prompting. unlike gpt2, llama-2 is instruction-tuned (ziegler et al., 2019), making it more capable of following human instructions. for this experiment, we use the cnn/dailymail (cnn/dm) @xcite test set which consists of 11490 news articles. we adapt llama-2 for default summarization by prepending an instruction to each news article (3a in baselines for comparison, we apply off-the-shelf style transfer models 2 to make the prompt outputs more informal/formal (prompt+style transfer). we also choose two baselines which require additional model training: dynamic word unit prediction from cao and @xcite and controllable readability from @xcite . both models are fine-tuned on cnn/dm and employ additional readability signals as supervision. nucleus sampling with @xmath0.9 is used for all models. ## conclusion this work introduces rsa-control, a pragmaticsgrounded lightweight controllable text generation approach which leverages mutual reasoning between speaker and listener modules. with a novel self-adjustable rationality parameter, rsa-control can automatically adjust control strength based on context. empirical results across two types of tasks, open-ended generation and inputoutput tasks, show that our method can effectively guide both foundation models and instructiontuned plms toward desired attributes during generation, while maintaining language fluency and content adherence.