Improving Argument Effectiveness Across Ideologies using Instruction-tuned Large Language Models (2024)

Abstract: Different political ideologies (e.g., liberal and conservative Americans) hold different worldviews, which leads to opposing stances on different issues (e.g., gun control) and, thereby, fostering societal polarization. Arguments are a means of bringing the perspectives of people with different ideologies closer together, depending on how well they reach their audience. In this paper, we study how to computationally turn ineffective arguments into effective arguments for people with certain ideologies by using instruction-tuned large language models (LLMs), looking closely at style features. For development and evaluation, we collect ineffective arguments per ideology from debate.org, and we generate about 30k, which we rewrite using three LLM methods tailored to our task: zero-shot prompting, few-shot prompting, and LLM steering. Our experiments provide evidence that LLMs naturally improve argument effectiveness for liberals. Our LLM-based and human evaluation show a clear preference towards the rewritten arguments. Code and link to the data are available here: https://github.com/roxanneelbaff/emnlp2024-iesta.

## introduction political polarization has significantly increased over decades across countries @xcite @xcite . in the us, for example, it has grown dramatically not only because of differences of opinion in politics but also because of the country's cultural system @xcite . according to @xcite , modern american politics revolves around conflicting worldviews. conservatives and liberals see the world differently and often struggle to understand each other's perspectives. such a divide leads to contentious debates among the political ideologies, either promoting agreement or pushing them apart. while some studies suggest that cross-party online interactions can think that it should be a law because u are taking away an unborns life that's not fair. if u get pregnant then have it and give it up instead of throwing away a life. to me that's not right for an unborn baby to not be born @xcite i believe that enacting a law is necessary as it involves the termination of a potential life, which is unjust. if an individual becomes pregnant, they should carry the pregnancy to term and consider adoption as an alternative to discarding a life. in my perspective, it is morally incorrect for an unborn child to be denied the opportunity to be born. ineffective argument a for a liberal, written by a non-liberal rewrite ã of a for a liberal improve effectiveness maintain content ## related work the study in this paper closely relates to research on argument quality and audience-aware text generation. technologically, it builds on text style transfer and large language models. we discuss related work for each in the following. wachsmuth et al. (2017) present a taxonomy of argument quality with three main dimensions: cogency (whether an argument is reasoned well) @xcite , effectiveness (whether it persuades the target audience) @xcite , and reasonableness (whether it contributes to an issue's resolution) @xcite . we focus on the second, where we consider an argument effective if it changes a reader's stance, in line with our prior work @xcite . ## task and data this section introduces the proposed task and the data created for our experiments and evaluation. ## approach this section presents the methods that we adapt to the task of effectiveness style transfer (figure 2, generation), as well as our general evaluation methodology (figure 2 , evaluation). ## automatic evaluation we report on our experiments by describing how we generated a from a. then, we report on the automatic evaluation metrics and results. the experiments are implemented using langchain @xcite and huggingface @xcite , and the code is executed on google colab with an a100 gpu and 40gb of ram. ## llm-based and human evaluation we conduct a zero-shot llm-based and humanbased evaluation to assess the effectiveness, clarity, consistency, and preference between a (original ar-gument), a chatgpt , and a llama2 for arguments generated respectively from chatgpt-based and llama2based models. for liberals, we chose zero-shot chatgpt (complete-prompt) and one-shot llama-2-7b-chat (style-prompt) since the effectiveness significantly improved ( § 5.3). for conservatives, in turn, we chose two models randomly due to the overall degradation in effectiveness ( we randomly selected 100 arguments (50 per ideology), a, with their generated effective counterparts, a. each triplet contained the original argument and the two rewrites, where each annotator reported an effectiveness score (1: fully ineffective -5: fully effective), a clarity score (1-5), consistency with the original argument (1-5), and a preferred argument among the three (for more details, check appendix d for llm-based evaluation, and appendix e for human-based evaluation). ## conclusion this paper has proposed the new task of effectiveness style transfer, that is, to rewrite an argument such that it better persuades individuals of some target ideology. this task may help bring people closer together, which is particularly important in today's deeply polarized society. our work denotes a fusion between a socially crucial task and a technological trend. we have presented new ideologyoriented datasets for the given task, and we have evaluated instruction-tuned large language models (llms) on the task. we have made 30k arguments accessible for detailed analysis, showing the potential of llms in improving argument effectiveness. further exploration of the textual features defining an effective argument from the perspective of llms is needed, for instance delving into the reasoning behind the llm scores.