Belief-based Generation of Argumentative Claims (2021)

Abstract: In this work, we argue that augmenting argument generation technology with the ability to encode beliefs is of twofold. First, it gives more control on the generated arguments leading to better reach for audience. Second, it is one way of modeling the human process of synthesizing arguments. Therefore, we propose the task of belief-based claim generation, and study the research question of how to model and encode a user’s beliefs into a generated argumentative text. To this end, we model users’ beliefs via their stances on big issues, and extend state of the art text generation models with extra input reflecting user’s beliefs. Through an automatic evaluation we show empirical evidence of the applicability to encode beliefs into argumentative text. In our manual evaluation, we highlight that the low effectiveness of our approach stems from the noise produced by the automatic collection of bag-of-words, which was mitigated by removing this noise. The finding of this paper lays the ground work to further investigate the role of beliefs in generating better reaching arguments.

## introduction according to @xcite , debaters engaging in argumentative discourse, aimed to resolve a disagreement, design their next argumentative move considering the topical potential, the audience demand, and appropriate presentational devices. @xcite stress based on the moral foundation theory @xcite how phrasing arguments to fit the audience's morals leads to a better agreement. for example, in a debate on former us president donald trump, potential topics could have been immigration, health care plans, tax plans, etc. however, knowledge about the audience being middle-class workers would have restricted the selection to trump's tax plans. appropriate usage of presentational devices may have then put a con argument as follows: example "donald trump was a bad president. he did nothing but hurt the poor and middle class. his tax plan benefited only rich people who could afford it." there is a recent growth of interest in argument generation as a subfield of computational argumentation. several tasks have been proposed, including claim negation @xcite , counterargument generation @xcite , and conclusion generation @xcite . while some research considers argumentative strategies when delivering arguments @xcite , no one has worked on adapting arguments to user beliefs yet. our goal is to bridge this gap. in this work, we propose to extend argument generation technologies with the ability to encode beliefs. this not only reflects the process by which humans reason, but it also allows controlling the output, to better reach the audience. in particular, we introduce the task of belief-based claim generation: given a controversial topic and a representation of a user's beliefs, generate a claim relevant to the topic and matches the beliefs. to approach this task, we first model user beliefs by their stances (pro or con) on a set of controversial topics, and then extend two state-of-the-art text generation approaches by conditioning their output on a specific set of beliefs. one approach builds on @xcite , equipping a sequence-to-sequence (seq2seq) model with a context vector representing the given stances. the other approach controls the output of a pre-trained argumentative language model (lm) using the algorithm of @xcite to ensure resembling the user's beliefs. we study the given task empirically on the debate.org dataset of @xcite . the dataset contains users' arguments on various controversial topics and their stances towards the most popular topics on the website, called the big issues. for our purposes, we use these big issues as the controversial topics, and we model beliefs by the user's stances towards them. in our automatic evaluation, we compare both models against their unconditioned correspondents (i.e., the same models without knowledge about a user). we assess the generated claims in terms of the similarity to the ground truth and the likelihood of carrying textual features that reflect users' stances on big issues. our results suggest that using users' beliefs significantly increases the effectiveness of the seq2seq and lm in most cases. moreover, a stance classifier trained on claims generated by the conditioned lm achieves the best-averaged accuracy across all big issues. in a subsequent manual evaluation, we find that claims generated by the conditioned lm are more informative regarding the topic. in terms of predicting stance from generated claims, we analyze the limitations of our approach in detail, which lie in the belief encoding step. by avoiding these limitations, we find that the generated claims enable the annotators to predict correctly a stance on a given big issue in 45% of the cases (26% incorrectly). these results demonstrate the applicability of encoding a user's beliefs into argumentative texts, enabling future research on the effect of belief-based argumentative claims on audiences. the contribution of this work is threefold 1 : ## related work early research on argument generation aimed to create argumentative texts starting from a symbolic representation @xcite @xcite . conceptually, those approaches all had a similar architecture consisting of three main phases: text planning, sentence planning, and realization @xcite . 1 code can be found under: http://www.github.com/ webis-de/eacl21-belief-based-claim-generation while they included a user model to a certain extent and aimed to generate convincing arguments, they were still performed on a limited scale. with the tremendous advances of nlp and machine learning since then, research has begun to address different tasks in the realm of argument generation, showing promising results. @xcite proposed a neural network-based framework for generating counter-arguments. both @xcite and @xcite addressed the task of claim negation, using a rule-based and a neural approach respectively. also, @xcite proposed an approach to argument generation based on sentence retrieval, in which, given a topic, a set of paragraphs covering different aspects is generated. however, these approaches are agnostic to the target audience. @xcite modified the political bias of (often claim-like) news headlines using style transfer, accounting for general political sides (left and right) at least. moreover, @xcite modeled rhetorical strategies in argument synthesis conceptually, but its computational realization @xcite considers the audience implicitly only, using a language model approach to select and arrange argumentative discourse units that are phrased in an argument. in the field of conversational ai, researchers have utilized machine translation techniques to tackle the task of dialog generation @xcite . @xcite worked on augmenting sequence-to-sequence models by learning persona vectors from the given data. in a similar fashion, one of our approaches extends such a model by a context vector representing a user's belief. here, however, we deal with argumentative text. progress in the field of text generation has been made due to the availability of large pre-trained language models @xcite . while these models excel in generating coherent texts, ensuring a generated text possesses a certain property is not straightforward. some research tackled this limitation, offering ways to better control the output @xcite . one of the most flexible of such approaches is by @xcite , which does not require fine-tuning for each controlling theme. their algorithm conditions the output of a language model to contain certain properties defined by a discriminative classifier or a bag-of-words. one of our approaches makes use of this algorithm to condi-tion the output of an argumentative language model on a bag-of-words that represents a user's beliefs. a recent relevant work by @xcite deals with the generation of aspect-controlled arguments. similar to us, the authors utilize a pre-trained language model to generate arguments on a specific topic, with a controlled stance and aspect. their focus is on topical aspects of arguments, though, and their approach based on @xcite is limited to a predefined set of topics and aspects. given a controversial topic and a representation of the audience's beliefs, generate a claim that is both relevant to the topic and matches the beliefs. we focus this task on generating claims rather than full arguments to keep it simple and because claims denote the main units from which arguments are built. as shown by @xcite , better agreement is achieved when arguments are framed with respect to audience's beliefs. therefore, we argue that studying the mentioned task will enable argumentation technology, knowing its audience, to generate more convincing arguments, bridging the gap between disagreeing parties. ## approach to study our research question, we propose and compare two approaches that build on top of known techniques for text generation. both approaches rely on modeling users' beliefs via their stances on big issues. the first is an extension of the seq2seq model @xcite , where the user's stances are encoded as a context vector, while the second conditions the output of a pre-trained argumentative language model via a bag-of-words, constructed based on stances on big issues. ## automatic evaluation in this section, we evaluate whether utilizing user's beliefs as input, modeled as stances on big issues, leads to claims that better match the ground-truth claims and reveal the input stances on big issues. ## manual evaluation to obtain more insights into belief-based claim generation, we let users manually evaluate the output of the given approaches. upon inspecting a sample of generated claims by our approaches, we noticed that the lm-conditioned produces more fluent and informative texts. accordingly, we focused on the lm-conditioned and its baseline in the evaluation, where we conducted two user studies. the goal of the first was to assess the quality of the bigissue bag-of-words collected automatically, while the second targeted the output of the lm-model, its baseline, and a variant that utilizes a manually refined bag-of-words. ## conclusion in this paper, we have proposed to equip argument generation technology with the ability to encode beliefs for two reasons: first, it reflects the human process of synthesizing arguments, and second, it gives more control on the generated arguments leading to a better reach of the audience. for this purpose, we have presented the task of belief-based claim generation. concretely, we studied the research questions of how to model a user's beliefs as well as how to encode them when generating an argumentative text. we have modeled users' beliefs via their stances on big issues, and used them as an extra input in our approaches. our automatic evaluation has provided evidence of the applicability of encoding beliefs into argumentative texts. in manual studies, we found that limitations in the effectiveness of our approach stem from noise produced by the automatic collection of a bag-of-words. the findings of this paper lay the ground to investigate the role of beliefs in generating arguments that reach their audience. we point out that ethical issues arise, when tuning arguments to affect specific people, such as attempts to manipulate them. while the task and settings considered here are rather too fundamental to already make these issues critical, future work should pay attention to them. our goal is to develop systems that bring people together.