Can LLM s Speak For Diverse People? Tuning LLM s via Debate to Generate Controllable Controversial Statements (2024)

Abstract: Making LLMs speak for different, especially minority groups of people, and generate statements supporting their diverse or even controversial perspectives is critical to creating an inclusive environment. However, existing LLMs lack sufficient controllability to the stance of their generated content, which often contains inconsistent, neutral, or biased statements. In this paper, we improve the controllability of LLMs in generating statements supporting an argument the user defined in the prompt. We find that multi-round debates between two LLMs with opposite stances generate higher-quality and more salient statements for each, which are important training data to improve the controllability of LLMs. Motivated by this, we develop a novel debate & tuning (“DEBATUNE”) pipeline finetuning LLMs to generate the statements obtained via debate. To examine DEBATUNE, we curate the largest dataset of debate topics so far, which covers 710 controversial topics and corresponding arguments for each topic. Evaluations by the GPT-4 judge with a novel controversy controllability metric show that LLMs’ capability of generating diverse perspectives is significantly improved by DEBATUNE. Moreover, such controllability can be generalized to unseen topics, generating high-quality statements supporting controversial arguments.

## introduction despite the remarkable advancement of current llms @xcite @xcite , and efforts to align llms with human preferences and values @xcite @xcite . a fact has long neglected that different people might have distinct, diverse, or even contradicted viewpoints on the same topic. though recent studies @xcite @xcite have acknowledged the inherent diversity of human values, they still attempt figure 1 : the pipeline of debatune. in the debate phase (top), the agents are prompted to debate upon the given topic with an argument. after several rounds of debate, an agent (positive in the example) concludes the debate based on all the previous debate records. the conclusion is a more salient, detailed, and higherquality statement for the agent. it will be used to train an llm in the training phase (bottom) to improve the controllability of generating statements for the given stance (positive in the example). to reach a consensus among various human perspectives, calibrating llm responses to align with an averaged, broadly acceptable viewpoint, potentially endorsed by the majority. however, these methods, while seeking a "safe" middle ground, inadvertently overlook the richness and complexity of diverse opinions that are fundamental to the fabric of our society. what's worse, exclusively aligning llms with the thoughts of the majority is unfair to minorities, who also have the right or need more help to express their viewpoints via llms. an example is showcased in figure 2 , which contains the failure case from vicuna 7b v1.5 (chi-figure 2 : comparing existing llms and debatune-trained llms. (a,c) given the controversial topic, "organ donation should be mandatory", and a user-defined stance (left: positive; right: negative), vicuna 7b v1.5 cannot always generate consistent statements supporting the stance and lacks controllability. it exhibits a bias towards the positive stance and ignores the user's negative stance and religious concerns in (c), which may lead to an offensive statement. (b,d) on the contrary, debatune-trained model generates higher-quality and strong statements that strictly adhere to the user stance (positive or negative). @xcite . specifically, for the topic "organ donation should be mandatory", towards which the users may be positive or negative with some potential initial thoughts, thus expect to obtain controllable responses on either side of the topic. however, existing llms lack the controllability on controversial topics, and thus are not able to strictly adhere to the given stance but wander to a safe middle ground. examples from vicuna-7b v1.5 are shown in (a)(c), where it successfully provides statements for the positive side but fails on the negative side. even if the user explicitly acquires for negative stance, it still tries to convince the user that organ donation is good, regardless of the user's religious concern, which might be offensive. in a world teeming with varied beliefs, cultures, and ideologies, the ability to represent and respect this diversity is not just a technical aspiration but a societal necessity. the current trend of seeking a singular, harmonized response in llms, therefore, poses a significant limitation, which restricts the potential breadth of llms' responses especially on controversial topics. in the desired situation, llms should obtain better controllability, whichever side the user queries, they are expected to generate corresponding responses that adhere to the users' request like figure 2 (b) and (d ). how could llms help people with diverse views express their opinions better to create a more in-clusive environment? how to improve the controllability of an llm in generating different or even contradictory viewpoints and thereby remove the potential bias of the pretrained llm? to solve these problems, we propose to utilize the debate mechanism to enhance llm responses for each side with more salient viewpoints on controversial topics. unlike existing work utilizing debate @xcite to improve specific instructions by converging the debating agents into a consensus, we simulate the debating process as it originally is, without the necessity to force them into a consensus but generating and defending their stance and arguments as they want for controversial debate topics. in our proposed pipeline, "debatune", as shown in figure 1 , two agents are engaging in structured debates, representing positive and negative sides facilitating more nuanced and in-depth understanding and generation of arguments, significantly improving the response quality of llms in handling polarized discussions. then the generated debate-augmented stances and arguments will be utilized to finetune the llms. since both the positive and negative stances and corresponding arguments of each topic are altogether fed into the llm, it is enforced to perceive a supreme variety of viewpoints for every topic, thus increasing its diversity and controllability on the controversy. moreover, due to the lack of a debating topic dataset with a reasonable amount of topics, we collect 710 controversial debate topics and manually modify them for a clear distinction between positives and negatives. another remaining issue is the evaluation metric for our specific purpose. while judging the quality of llms' responses by gpt4 is widely accepted common practice @xcite @xcite @xcite , it only evaluates the quality of one specific response given the topic, stance, and argument, but neglects the extent that llm's response is consistent with the user query. thus we further propose an evaluation method utilizing gpt4 to evaluate llm's controllability on controversial topics. extensive experiments show that our method largely improves llm's ability to generate responses for controversial topics. the contributions of this paper can be summarized as: ## debate dataset instruction tuning requires plenty of training data. however, although there are several debate-related datasets like debatesum @xcite , change my view @xcite or social-chem101 @xcite in the community, most of them either lack a direct topic or the topics are so biased that it is not suitable to support both sides of them foot_0 , e.g., "it is wrong to destroy someone else's property". there are 710 controversial debate topics. each topic t allows a positive stance p t and a negative stance n t , where p t agrees with the topic and n t is against it. we use gpt-3.5-turbo-1106 to generate 5 one-sentence arguments supporting each stance, e.g., a pt,i is the i-th argument for the positive stance on topic t. given an argument a pt,i , a controllable llm is expected to generate a supporting statement s pt,i with detailed explanations and evidence. to build a dataset with adequate debating topics, we first collect topics from existing datasets @xcite @xcite : we first filter and modify them into approximately 200 topics. then we further collect approximately a thousand controversial debate topics across the areas of society, ethics, environment, technology, education, politics, economics, and health. we manually filter out the similar ones and modify the rest to a format that both positive and negative stances can be directly defined for each topic. finally, we achieve a dataset with 710 topics. we split the dataset into a training set of 630 topics and a held-out test set of 80 topics. to our knowledge, this is the largest open-sourced debate dataset so far. our dataset structures are shown in figure 3 . each topic t, e.g., "should there be more regulation on ai ethics and accountability?", allows two controversial stances, i.e., positive stance p t that agrees with the point of view in t and negative stance n t that disagrees with it. we utilize gpt-3.5turbo-1106 to generate 5 diverse seed arguments for each stance on every topic. each argument is a brief sentence, e.g., "ensuring ai systems prioritize human well-being and safety." the i-th positive or negative argument of topic t is denoted by a pt,i or a nt,i . in our paper, both the stance and its argument(s) will be included in an input instruction as the control to an llm. given a pt,i , a controllable llm p θ (•) (with pa-rameters θ) is expected to generate a response y pt,i ∼ p θ (y|t, p t , a pt,i ) that contain detailed explanations, logical reasoning, and evidence adhering to and supporting the input stance. we develop a debate pipeline to generate 710 × 2 × 5 = 7100 ground truth statements, each associated with an argument-i of a stance on a topic-t. for example, statement s pt,i will be used as the ground truth of llm response y pt,i . more specifically, the argument is a brief onesentence summarization of an opinion. it is given in the input to an llm and provides specific guidance to the generation of statements, which are expected to support the argument. the statement is a detailed expression and expansion of an argument, which includes detailed explanations, logical reasoning, and evidence adhering to the input stance of the topic. it is a supporting statement of the argument. ## debatune as shown in figure 1 , debatune has two main phases, i.e., debate and training. in the debate phase, we aim to achieve high-quality, salient, and diverse statements (triggered by different arguments) for each stance on every controversial topic via multi-round debates. in the training phase, debatune finetunes an llm to fit each statement giving its corresponding argument, stance, and topic as controls in the input instruction, hence improving the controllability of the llm. ## evaluation metrics we evaluate llm's ability to generate statements for controversial topics on two orthogonal aspects on our hold-out test set: the response quality and controversy controllability. the response quality measures whether the llm can generate helpful, relevant, accurate, and detailed statements for an instruction, which aligns with the common requirements for llm's responses. however, the response quality fails to measure the extent to which llm's response is stuck to the desired stances. as illustrated previously, existing llm tends to generate average viewpoints endorsed by the majority and neglect the voice of the minority, which might be graded highly by existing judging methods. thus we propose another aspect, noted as controversy controllability, which directly measures whether llm's response is strictly stuck to the desired stances. ## conclusion and future work our study raises the long-neglect issue of generating controllable responses towards controversial 5 https://research.ibm.com/haifa/dept/vst/ debating_data.shtml topics, not only describing and exemplifying but also building an evaluation pipeline for the assessment and a novel method to alleviate this problem. more specifically, our debatune, is a novel pipeline that enhances model controllability over diverse perspectives on controversial topics. we have curated the largest dataset of debate topics to date and introduced a new metric for measuring controllability. our evaluations reveal that llms can be effectively fine-tuned to represent a broader spectrum of opinions, paving the way for more inclusive ai-generated discourse. in our study, it is shown that controversial arguments and statements are beneficial for llms in further generating diverse and high-quality responses supporting different controversial topics. however, our current method mainly focuses on generating desired statements by utilizing llms, though effective, a better strategy would be directly collecting high-quality human-written responses. one potential source would be directly collecting statements directly from the debating websites. another potential source would be reforming the existing high-quality debatesum dataset @xcite . as mentioned in the previous section, debatesum has diverse and high-quality statements while lacking corresponding topics and stances, thus further human annotation is required. we believe that the further collecting and combining of human-written high-quality data can further improve llm's controllability to generate responses for diverse controversial topics and thus help minorities to express their own opinions.