Argue with Me Tersely: Towards Sentence-Level Counter-Argument Generation (2023)

Abstract: Counter-argument generation—a captivating area in computational linguistics—seeks to craft statements that offer opposing views. While most research has ventured into paragraph-level generation, sentence-level counter-argument generation beckons with its unique constraints and brevity-focused challenges. Furthermore, the diverse nature of counter-arguments poses challenges for evaluating model performance solely based on n-gram-based metrics. In this paper, we present the ArgTersely benchmark for sentence-level counter-argument generation, drawing from a manually annotated dataset from the ChangeMyView debate forum. We also propose Arg-LlaMA for generating high-quality counter-argument. For better evaluation, we trained a BERT-based evaluator Arg-Judge with human preference data. We conducted comparative experiments involving various baselines such as LlaMA, Alpaca, GPT-3, and others. The results show the competitiveness of our proposed framework and evaluator in counter-argument generation tasks. Code and data are available at https://github.com/amazingljy1206/ArgTersely.

## introduction counter-argument generation task aims to automatically generate a statement that has a different stance from the original argument @xcite . existing works describe it as a paragraph-level generation task @xcite @xcite . however, sentence-level counter-argument generation can be quite different. the main challenge of sentence-level generation is to condense the counter-argument into a concise topic: can firear m restr ictions reduce gun cr ime? the government should impose str icter regulations on firear m owner ship to reduce gun-related cr imes. str icter firear m regulations may limit the rights of law-abiding citizens without effectively addressing the root causes of gunrelated cr imes, such as mental health issues and socioeconomic disparities. ## arg-llama figure 2 shows the framework we proposed, arg-llama. it is mainly composed of two parts: 1) a language model (lm) with instruct-tuning, for generating counter-argument, and 2) a filter, for se-lecting high-quality counter-argument. we employ lora and instruct-tuning methods to obtain an lm. additionally, we leverage human preference data to train the filter. during inference, we use cot instructions as inputs of the lm. after obtaining a series of outputs from the lm, the filter will select the best counterargument as output. the generation pipeline is detailed in section 3.3. ## validation of arg-judge metric in order to explore the capability of our proposed arg-judge to reflect the actual ranking level and its consistency with human evaluation, we designed two corresponding tasks. ## case study we illustrate the advantage of our model through a case study in counter-argument generation datasets @xcite @xcite for counter-argument generation mainly establish the rebuttal relationship in the conversation using automatic methods such as citation or reply detection. @xcite topic: how should the struggles of white upper middle-class individuals be addressed in the usa? original argument: the struggles faced by white upper middle-class individuals in the usa raises questions about the prospects and viability of investing in the white working-class, let alone providing support for communities of color in inner cities. ours: however, the struggles faced by white upper middle-class individuals in the usa should not be used as an excuse for not investing in the white working-class. gpt-3: the white working-class is not a monolithic group. the white working-class is not a homogeneous group. the white working-class is not a static group. alpaca-lora: the struggles faced by white upper middle-class individuals in the usa raises questions about the prospects and viability of investing in the white working-class and providing support for communities of color in inner cities. llama: let alone providing support for communities of color in inner cities. bolton et al. (2020) introduced kialo, a dataset for sentence-level argument stance classification and counter-argument generation. argtersely distinguishes itself as the first human-annotated dataset of its kind with ranking data reflecting human preferences. early work (hua and wang, 2018; hua et al., 2019) focus on how to introduce external knowledge into the system; alshomary et al. (2021) developed a system to identify weak points in arguments; schiller et al. (2021) developed a controlled argument generation system, which is able to generate arguments based on given information; alshomary and wachsmuth (2023) completed it through multitask and multi-step reasoning. our work primarily introduces benchmark and evaluation metrics for sentence-level argument generation. methodologically, we just establish a usable baseline using llm without introducing too much external knowledge. self-instruct a series of recent works (zhou et al., 2022; ye et al., 2022; singh et al., 2022; honovich et al., 2023) generate instructions of a task given a few examples. singh et al. (2022) use llm and reranking algorithm to generate humaninterpretable instruction, which matches or even im-proves upon human-written instruction. honovich et al. (2023) introduce the instruction induction challenge task and discover the ability to generate instructions emerge when a language model is large enough. wang et al. (2023) provide an almost annotation-free method for aligning pre-trained language models with instructions. the overall process is an iterative bootstrapping algorithm, which starts off with a limited seed set of manually written instructions that are used to guide the overall generation. we fine-tuned the arg-llama model using the self-instruct approach, where we included seed instruction for a variety of related tasks of the counter-argument generation. adaptation as the size of the language model increases, the cost of fine-tuning also increases. a series of works (shin et al., 2020; li and liang, 2021; houlsby et al., 2019) have studied various methods like prompt-tuning and adapter-tuning to alleviate this problem. however, it's difficult to directly optimize the prompt, and introducing the adapter layer will cause a delay in reasoning. considering that, hu et al. (2021) proposed low-rank adaptation (lora), which greatly reduces the number of trainable parameters for downstream tasks. additionally, there have been advancements in extending low-rank adaptation. dettmers et al. (2023) proposed qlora that fine-tunes large models on limited memory gpus through 4-bit quantization and low-rank adapters. chen et al. (2023) introduced longlora that leverages sparse local attention and achieves context extension with minimal computation. since our work does not involve long contexts in generation and does not prioritize optimization techniques, we just utilize lora to fine-tune our model for efficiency. ## conclusions in this paper, we introduce a benchmark argtersely for sentence-level counter-argument generation. specifically, we present a human-annotated dataset and develop a language model based on argumentation instructions. we further construct a framework arg-llama, which leverages the language model. additionally, we propose two model-based metrics, chatgpt eval and arg-judge, as complements to n-gram-based metrics. experiments show that our framework competes well with mainstream models, and our metrics are effective and highly consistent with human evaluations.