From Argument Search to Argumentative Dialogue: A Topic-independent Approach to Argument Acquisition for Dialogue Systems (2021)

Abstract: Despite the remarkable progress in the field of computational argumentation, dialogue systems concerned with argumentative tasks often rely on structured knowledge about arguments and their relations. Since the manual acquisition of these argument structures is highly time-consuming, the corresponding systems are inflexible regarding the topics they can discuss. To address this issue, we propose a combination of argumentative dialogue systems with argument search technology that enables a system to discuss any topic on which the search engine is able to find suitable arguments. Our approach utilizes supervised learning-based relation classification to map the retrieved arguments into a general tree structure for use in dialogue systems. We evaluate the approach with a state of the art search engine and a recently introduced dialogue model in an extensive user study with respect to the dialogue coherence. The results vary between the investigated topics (and hence depend on the quality of the underlying data) but are in some instances surprisingly close to the results achieved with a manually annotated argument structure.

## introduction argumentation is an interesting, yet challenging domain for dialogue systems. existing systems address a multitude of tasks, including full scale debates against a human debater @xcite , persuasion @xcite and customer support @xcite . due to the complex nature of this type of interaction, many systems rely on topic-specific argument structures that encode arguments and their relations to ensure a consistent and challenging interaction @xcite . despite the advantages on the formal side, this dependency limits the range of topics that can be discussed by a system as the required structures are often either annotated by hand @xcite or acquired in time-consuming data collections @xcite . this limitation renders the corresponding systems inflexible, especially in comparison to recent data-driven approaches in domains like question answering @xcite . to address this issue, we propose a combination of argument search technology @xcite with dialogue systems of the discussed kind. our approach maps the list of pro and con arguments retrieved with an argument search engine for a given topic into a general tree structure that encodes bipolar relations (support and attack) between the individual arguments (see figure 1 ). in doing so, our approach combines the strong points of both data-driven and formal models for argumentation and enables a corresponding system to discuss literally any topic on which the search engine can find suitable arguments. throughout this work, we use the argument search engine argu-mentext @xcite to retrieve pro and con arguments for a given topic from a large web crawl. in addition, we train and compare two classifiers to detect relations between pairs of the retrieved arguments which subsequently enables the aforementioned mapping into an argument structure. the approach is evaluated with a formal model for persuasive dialogues that enables the generation of artificial discussions between two virtual agents. the resulting dialogues are then assessed in an extensive user survey with respect to their coherence and compared to the results achieved with an annotated structure. although the annotated structure yields (as expected) an advantage over the automatically generated ones, the results are in some instances fairly close to each other. besides, we observe varying results for the investigated topics, indicating a dependency of the approach on the available data. in summary, our contributions are: the remainder of this paper is as follows: section 2 includes related work from the field of argumentative dialogue systems. the background on argument search and details about the utilized search engine are covered in section 3, followed by a discussion of the proposed mapping in section 4. subsequently, we discuss the formal model for persuasion alongside the generation of artificial dialogues in section 5 and the evaluation setup as well as the results in section 6. the corresponding findings are discussed in section 7. ## related work this section provides an overview of related work with a focus on argument retrieval for argumentative dialogue systems. the arguably most prominent system of this kind is the ibm project debater @xcite , which is able to debate different topics with a human interlocutor. although the system uses state of the art argument mining approaches to retrieve its arguments, it is tailored to the domain of debates and the utilized retrieval engine is currently not available to the public which hinders an application in other systems. approaches that rely on argument structures or graphs were investigated in different scenarios. the system by @xcite engages in a persuasive dialogue with human users. it utilizes a weighted bipolar argumentation framework with arguments collected in human discussions on the investigated topics. @xcite conducted a crowdsourcing experiment to collect an argument graph for their desired domain and topic, whereas the argument structures employed in @xcite were generated using human annotators @xcite . similarly, the systems of @xcite and @xcite also rely on annotated structures. in addition, human-generated argument graphs were considered by @xcite , who selected arguments from multiple online sources manually for use in their system. although the underlying formal frameworks of all these systems allow for complex dialogues, the topics that can be addressed are limited by the time-consuming generation of the argument structures. we propose an approach to generate structures of this kind automatically and independently of a specific topic. in addition, data-based approaches were also investigated. the chatbot introduced by @xcite utilizes semantic similarity measures to retrieve arguments from an argument corpus to generate a response. a similar approach was compared to a generative model by @xcite that was trained on a corpus of debate posts on various topics. although especially the generative approach is focused on providing topic flexibility, aspects like user adaptation or strategy optimization as addressed in some of the previously discussed works are not (yet) considered in these systems. our approach bridges the gap between formal and data-driven argumentation through a combination of argument search with formal models. ## argument search argument search has recently evolved as an application from the field of argument mining @xcite . argument search engines provide users with a (ranked) list of arguments related to a given search query, in some instances also including their stance/polarity towards the topic. ## from arguments to structures in the following, the mapping of the retrieved arguments into an argument structure is discussed. although some structures utilized by the systems discussed in section 2 differ to a certain extent, they all require information about the relations between the individual arguments. we hence pursue a modular pipeline approach that first determines possible relations between the arguments and subsequently maps them into a specific structure. in case the required structure cannot be inferred from the herein discussed one, the second module can be adapted accordingly. this section builds on the work in @xcite . the code of the complete pipeline is publicly available foot_2 . ## argumentative dialogue to evaluate the complete pipeline in a dialogue setup, we generate artificial discussions between two virtual agents. the dialogues are created utilizing a recently introduced dialogue game for argumentation @xcite ) that extends the one introduced in @xcite . it is chosen because it ensures a formally coherent selection of utterances, which means that all incoherent responses in the resulting dialogues can be clearly attributed to the retrieval pipeline. in addition, it offers the flexibility to go back to a previous utterance and respond with an alternative to the earlier response. this enables the agents to explore different branches of the tree structures and ensures a challenging setup for the evaluation. ## evaluation this section discusses the evaluation of the artificial dialogues. we first introduce the study setup and discuss the results subsequently. ## discussion in the following, we discuss our findings from the previous section and the perspective of applications for the proposed method. as already mentioned, the results vary between the investigated topics for all evaluation categories. the difference between the individual topics can be attributed to the different sources the arguments are retrieved from and the resulting performance difference of the pipeline components. the effect of the clustering on the other hand is not so clear as both structures for a topic are based on the same pool of arguments. however, as the position of the arguments in the although the above-discussed data dependency renders generalizations difficult, this coherence difference between the topic marriage and the others indicates that the overall pipeline performance is closer to the one with annotated structures than suggested by the direct comparison. as for the written feedback, multiple annotators reported confusing formulations of the argument as the major difficulty of the task. since this is a direct consequence of the heterogeneous sources the arguments are retrieved from, it is hard to address in the pipeline. therefore, approaches to automatically summarize or reformulate arguments @xcite could be beneficial to improve the performance. regarding applications, it can be seen that the proposed approach is quite flexible: although a specific multi-agent setup was chosen for evaluation, the proposed pipeline itself has no dependency on this particular setting or the corresponding domain of persuasive dialogues. therefore, it can be directly applied in other domains and scenarios as well if the respective dialogue system operates on structures of the retrieved kind. this includes for example systems in the opinion building domain @xcite or systems that combine argumentation with other types of dialogue like question answering @xcite . in addition, the proposed pipeline can be combined with methods that build on the investigated representation of arguments. in particular, the probabilistic rule-based strategy that was used in the evaluation setup can be extended or replaced with more sophisticated ones in compliance with the desired application. examples in this regard are strategies optimized via reinforcement learning @xcite as well as argument selection based on semantics @xcite or user concerns @xcite . in light of the evaluation results, the main task for future work with respect to applications is hence the improvement of the pipeline performance to fully meet the quality requirements of the individual systems. however, as the proposed approach relies on argument search engines, it directly benefits from future developments in this area. moreover, the addition of weights to arguments in the structure could further broaden the range of possible applications. the corresponding weights can for example be derived from the confidence scores of the pipeline components or through automatic approaches to assess argument quality @xcite . ## conclusion we have addressed the automatic generation of argument structures from argument search results for their use in dialogue systems. to this end, a pipeline was introduced that estimates relations between the retrieved arguments and maps them into a general tree structure. we explored two different configurations, namely with and without a prior clustering of the retrieved arguments and utilized a supervised learning-based relation classification to identify related argument pairs. for evaluation purpose, we generated 30 artificial dialogues over 7 different topics and assessed them in a crowdsourcing setup with respect to their coherence. the results indicate that the proposed pipeline depends on the quality of the available data but yields promising results for the majority of the investigated topics and at least one of the two investigated configurations (with and without clustering). in comparison to an annotated structure, we observed a similar performance for individual categories but also the expected room for improvement regarding the overall coherence. in summary, the proposed approach can be seen as a first step towards fully automatized argument acquisition for argumentative dialogue systems. since it is based on argument search engines, it benefits directly from future improvements and developments in this area. future work will investigate automatic evaluation approaches that allow for an estimation of the pipeline performance given a specific topic. in addition, automatically generated structures will be applied in a dialogue system for an evaluation in direct interaction with human users.