Aspect-Controlled Neural Argument Generation (2021)

Abstract: We rely on arguments in our daily lives to deliver our opinions and base them on evidence, making them more convincing in turn. However, finding and formulating arguments can be challenging. In this work, we present the Arg-CTRL - a language model for argument generation that can be controlled to generate sentence-level arguments for a given topic, stance, and aspect. We define argument aspect detection as a necessary method to allow this fine-granular control and crowdsource a dataset with 5,032 arguments annotated with aspects. Our evaluation shows that the Arg-CTRL is able to generate high-quality, aspect-specific arguments, applicable to automatic counter-argument generation. We publish the model weights and all datasets and code to train the Arg-CTRL.

## introduction language models @xcite allow to generate text through learned distributions of a language and have been applied to a variety of areas like machine translation @xcite , summarization @xcite , or dialogue systems @xcite . a rather new field for these models is the task of producing text with argumentative content @xcite . we believe this technology can support humans in the challenging task of finding and formulating arguments. a politician might use this to prepare for a debate with a political opponent or for a press conference. it may be used to support students in writing argumentative essays or to enrich one-sided discussions with counter-arguments. in contrast to retrieval methods, generation allows to combine and stylistically adapt text (e.g. arguments) based on a given input (usually the beginning of a sentence). current argument generation models, however, produce lengthy texts and allow the user little control over the aspect the argument should address @xcite . we show that argument generation can be enhanced by allowing for a fine-grained control and limiting the argument to a single but concise sentence. controllable language models like the ctrl @xcite allow to condition the model at training time to certain control codes. at inference, these can be used to direct the model's output with regard to content or style. we build upon this architecture to control argument generation based solely on a given topic, stance, and argument aspect. for instance, to enforce focus on the aspect of cancer for the topic of nuclear energy, we input a control code "nuclear energy con cancer" that creates a contra argument discussing this aspect, for instance: "studies show that people living next to nuclear power plants have a higher risk of developing cancer.". to obtain control codes from training data, we pre-define a set of topics to retrieve documents for and rely on an existing stance detection model to classify whether a sentence argues in favor (pro) or against (con) the given topic @xcite . regarding argument aspect detection, however, past work has two drawbacks: it either uses simple rule-based extraction of verb-and noun-phrases @xcite or the definition of aspects is based on target-concepts located within the same sentence @xcite . aspects as we require and define them are not bound to any part-of-speech tag and (1) hold the core reason upon which the conclusion/evidence is built and (2) encode the stance towards a general but not necessarily explicitly mentioned topic the argument discusses. for instance: topic: nuclear energy argument: running nuclear reactors is costly as it involves long-time disposal of radioactive waste. the evidence of this argument is based upon the two underlined aspects. while these aspects encode nuclear reactors produce radioactive waste ... ## related work argument aspect detection early work by @xcite focuses mainly on japanese and restricts aspects to noun-and verb-phrases, extracted via hand-crafted rules. @xcite extract noun-phrases and aggregate them into concepts to analyze the microstructure of claims. @xcite introduce facets as low level issues, used to support or attack an argumentation. in that, facets are conceptually similar to aspects, but not explicitly phrased and instead seen as abstract concepts that define clusters of semantically similar text-spans of summaries. @xcite define commonplace arguments that are valid in several situations for specified actions (e.g. "ban") and topics (e.g. "smoking"). these actions are similar to aspects, but limited in number and manually defined. @xcite detect, amongst others, concepts and aspects in arguments with models trained on expert annotations. however, in their definition, aspects have to point to a target concept mentioned in the argument. in our definition, aspects refer to a general topic which is not necessarily part of the sentence and our annotation scheme is applicable by non-experts. the concept of framing dimensions @xcite is close to argument aspects. in the field of argument mining, @xcite recently applied frames to label argument clusters. yet, their method does not allow to detect frames. other works present methods to automatically label sentences of news articles and online discussions with frames @xcite . these methods are, however, limited to a small set of predefined frames that represent high-level concepts. contrarily, we operate on a fine-grained span-level to detect aspects that are explicitly mentioned in arguments. argument generation early approaches rely on rules from argumentation theory and user preference models @xcite . in a more recent work, @xcite construct rules to find arguments in a large data source, which are then filtered and ordered with a neural network based ranker. @xcite use a clustering and regression approach to assemble discourse units (major claims, pro and con statements) to argumentative texts. however, most of these approaches rely on hand-crafted features and do not generalize well. moreover, they all require permanent access to large data sources and are not able to generate new arguments. recently, research on generating arguments with language models gained more attention. @xcite use a sequence to sequence model @xcite to generate argumentative text by attending to the input and keyphrases automatically extracted for the input from, for example, wikipedia. other work focuses on generating argumentative dialogue @xcite and counterarguments @xcite based on a given input sentence, or on generating summaries from a set of arguments @xcite . contrarily, we train a language model that does not require a sentence-level input for generation and allows for direct control over the topic, stance, and aspect of the produced argument. @xcite design a language model that attends to topic information to generate responses for chatbots. @xcite train two models that control the sentiment and topic of the output of pre-trained language models at inference. @xcite fine-tune gpt-2 on existing, labeled datasets to generate claims for given topics. however, the latter works do not explore generation for such a fine-grained and explicit control as proposed in this work. we show that argument generation requires the concept of argument aspects to shape the produced argument's perspective and to allow for diverse arguments for a topic of interest. ## argument aspect detection argument aspect detection is necessary for our argument generation pipeline, as it allows for a finegrained control over the generation process. we create a new dataset, as existing approaches either rely on coarse-grained frames or cannot be applied by non-expert annonators in a scalable manner. ## data collection pipeline this section describes the data collection and preprocessing for the argument generation pipeline. we aim to train a model that is able to transfer argumentative information concisely within a single sentence. we define such an argument as the combination of a topic and a sentence holding evidence with a specific stance towards this topic @xcite . consequently, the following preprocessing steps ultimately target retrieval and classification of sentences. to evaluate different data sources, we use a dump from common-crawl ## model training and analysis in the following, we describe the architecture and the training process of the arg-ctrl and analyze its performance. ## generation in absence of aspects to show the necessity of having prior knowledge of aspects for our controlled argument generation approach, we create training data without prior knowledge of aspects, train a new generation model on it, and compare it to our previous models with prior knowledge of aspects. equally to the original arg-ctrl cc 's procedure, we gather 100,000 sentences for each stance of a topic from the cc data. as we assume to have no knowledge about the aspects of the arguments, we randomly sample arguments from the cc source documents. we create training documents with numbers of arguments varying between 15 and 1,500 to mimic the data generation process of the original models and fine-tune a new generation model on them. after training, we generate the same number of arguments as for the other two models by using our default control code of @xcite . while the new model was only conditioned on topics and stances at training time, we make sure that all aspects used for generation appear in at least one argument of the model's training data. we compare all models by verifying whether or not the aspect used for generation (including synonyms and their stems and lemmas) can be found in the generated arguments. for the original models conditioned on aspects, this is true in 79% of generated sentence: we do n't need more gun control laws when we already have enough restrictions on who can buy guns in this country . training sentence: we have some of the strongest gun laws in the country , but guns do n't respect boundaries any more than criminals do . cosine similarity / edit distance / rel. overlap: 95.59 / 88 / 8% generated sentence: the radioactivity of the spent fuel is a concern , as it can be used to make weapons and has been linked to cancer in humans . training sentence: however , it does produce radioactive waste , which must be disposed of carefully as it can cause health problems and can be used to make nuclear weapons cosine similarity / edit distance / rel. overlap: 92.40 / 99 / 17% the cases for arg-ctrl reddit and in 74% of the cases for arg-ctrl cc . for the model that was not conditioned on aspects, however, it is only true in 8% of the cases. it clearly shows the necessity to condition the model on aspects explicitly, implying the need for argument aspect detection, as the model is unable to learn generating aspect-related arguments otherwise. moreover, without prior detection of aspects, we have no means for proper aggregation over aspects. we notice that for the model without prior knowledge of aspects, 79% of all aspects in the training data appear in only one argument. for these aspects, the model will likely not pick up a strong enough signal to learn them. ## evaluation we evaluate the quality (intrinsic evaluation) of the arg-ctrl and its performance on an exemplary task (extrinsic evaluation). as a basis, we use the 7,991 arguments generated in section 5. ## conclusion we apply the concept of controlled neural text generation to the domain of argument generation. our arg-ctrl is conditioned on topics, stances, and aspects and can reliably create arguments using these control codes. we show that arguments generated with our approach are genuine and of high argumentative and grammatical quality in general. moreover, we show that our approach can be used to generate counter-arguments in a transparent and interpretable way. we fine-tune the arg-ctrl on two different data sources and find that using mixed data from common-crawl results in a higher quality of generated arguments than using user discussions from reddit-comments. further, we define argument aspect detection for controlled argument generation and introduce a novel annotation scheme to crowdsource argument aspect annotations, resulting in a high-quality dataset. we publish the model weights, data, and all code necessary to train the arg-ctrl.