Node Placement in Argument Maps: Modeling Unidirectional Relations in High & Low-Resource Scenarios (2023)

Abstract: Argument maps structure discourse into nodes in a tree with each node being an argument that supports or opposes its parent argument. This format is more comprehensible and less redundant compared to an unstructured one. Exploring those maps and maintaining their structure by placing new arguments under suitable parents is more challenging for users with huge maps that are typical in online discussions. To support those users, we introduce the task of node placement: suggesting candidate nodes as parents for a new contribution. We establish an upper-bound of human performance, and conduct experiments with models of various sizes and training strategies. We experiment with a selection of maps from Kialo, drawn from a heterogeneous set of domains. Based on an annotation study, we highlight the ambiguity of the task that makes it challenging for both humans and models. We examine the unidirectional relation between tree nodes and show that encoding a node into different embeddings for each of the parent and child cases improves performance. We further show the few-shot effectiveness of our approach.

## introduction online discussions can have huge numbers of contributors and contributions, making the discussion hard to follow for new users. getting an overview of a discussion and finding points of interest for a new user might be hard in such an unstructured format which is also prone to redundancy. argument maps, in their simplest form, structure arguments into a tree with each node being a pro or contra argument for its parent node (also an argument, see figure 1 ). relying on the structure of the map, users can dive deeper into specific aspects of an argument and collectively add more arguments to support or oppose it: this improves the overall quality of the discourse and at the same time, triggers root n2 n1 n5 n8 n3 n7 n6 n4 n10 n9 new node pro con argument child: "new node" parent: "n6" child: "n9" child: "n10" parent: "n3" child: "n3" x child: "n6" ## related work node placement in argument maps the task is related to two widely explored tasks in argument mining: argument retrieval and modeling argument relationships. argument retrieval can be viewed as a more general form of node placement, in which a system should provide relevant arguments given a controversial question or topic @xcite @xcite or a suitable counter-argument given an input argument @xcite . with respect to general argument retrieval, our task tackles a finergrained problem: finding suitable positions in the argument tree. regarding the modeling of the relationship between arguments @xcite , e.g. support/attack, only few works consider it in the context of a full, structured debate. to automatically construct argument graphs, @xcite use structured debates to classify relationship between argumentative units. @xcite model the relationship between arguments as a polarity prediction task using the tree-structure, and exploiting the ancestors of a node to classify support/attack relationships. retrieval & ranking: cross-encoders like bert @xcite can be used to score pairs of sentences, but this does not scale well for large number of candidates in retrieval & ranking tasks. siamese networks @xcite (also called bi-encoders) have long been used to create embeddings for efficiently tackling those tasks using contrastive learning. sentence-transformers @xcite employs this by using bert or other transformer models and utilizing labeled data while consert @xcite and simcse @xcite also utilize unlabeled data. the original bert can be used to encode each sample (of a pair) into a vector for more efficiency, but the resulting embeddings have high similarity in general which bert-flow @xcite and whiteningbert @xcite tackle using normalization. we utilize bi-encoders and pay attention to the high similarity issue with a thorough analysis of the effect our approach has on the embedding space. templates & low-resource: templates were recently heavily used in prompts to tap into the knowledge encoded in large plms and to make use of their few-shot capabilities by using a taskinformation template @xcite . they were also used to fine-tune plms in a few-shot setup @xcite @xcite while others attempted to do away with them @xcite @xcite , but they were mainly employed to directly solve nlp tasks and not to learn embeddings. prompts were used for the latter more recently @xcite with contrastive learning. we simply utilize templates with no language modeling training or inference and show they are beneficial with contrastive learning to learn embeddings in high and low-resource. ## data we use argument maps from kialo, an online platform on which people engage in discussions on specific topics or statements. in a discussion about a controversial thesis topic, the thesis acts as a root node under which further and increasingly more specific arguments for or against this point of view can be added. an example of how the original data looks like is shown in figure 2 (taken from kialo foot_2 ). it shows how the tree of arguments evolves for the root node or thesis "video game storytelling should portray gender equality." users can navigate through the tree to find aspects of the discussion that they are interested into or to find a good node to attach their new input to. each new argument can in turn be attacked or supported with a variety of different arguments. thus every debate in kialo represents a unidirected tree, where each edge represents a support or attack relation (henceforth, pro and con). we rely on data from @xcite and use a total of 1,378 maps covering a wide variety of topics: politics, technology, ethics, etc. (overview in appendix figure a .2). the majority contain up to 200 nodes, but a quarter of the data are large- ## annotation study to have a better understanding of the task and data, generalize a baseline of human performance, and, estimate the difficulty and cost humans encounter with such a task, we conduct an annotation study. design: we employed 3 annotators with a background in nlp and social sciences (details in section a.2. the annotators were presented a specific contribution to a discussion foot_3 -the child -and 10 candidate parents selected from the discussion's argument map to which the child could attach. the annotators were tasked to classify each of the candidates with one of the following labels: best parent (count 1), suitable parent (max. 4), or less suitable parent. the annotation guidelines and an example are provided in appendix figures @xcite . in order to control for an appropriate variety of candidates that a user might encounter, the candidates consisted of the actual parent, 6 candidates closely related in the tree to the child (with a maximum path distance of 3), and 3 randomly selected candidates from the full tree. in total, the annotated dataset consists of 200 child instances. the instances selected were evenly split between pro relations with its parent and con. the nodes were sampled from small and large-scale maps (90 to 2500 nodes). the topics of the maps are environment, economy, gender, politics and immigration. to better understand the annotators' approach to the task, we asked them to provide their confidence scores for each annotation, as well as short-answer motivation for a subset of 100 annotations. annotation results: we measure the annotator agreement using weighted kappa (κ w ) @xcite as we would like to account for the seriousness of the disagreements, i.e. disagreeing about best and suitable should be penalized less than best and less suitable. the annotators have a fair to moderate agreement of 0.387. while we can conclude that the participants generally agreed, κ w in the lower range of agreement is an indication of the difficult and subjective nature of this task, despite the clear guidelines and training. to calculate the performance, we convert the human annotated labels for each child into a score and thus obtain a ranked list of all candidates foot_4 . our analysis of the motivation behind the annotators' choice shows it can be divided into 3 categories, in decreasing frequency: (1) process of elimination, described often as "most obvious", "best fit", or "makes most sense"; (2) linguistic overlap, reported as similar use of terminology or structure; and (3) logical connection, in which participants found a direct child/parent relationship, such as an elaboration or offering examples. note that the task given to the annotators is rather simple in comparison to a real-world application where dozens, if not hundreds, of options across the full argument map would need to be considered. that said, the three annotators averaged a total of 31 hours to complete the 200 annotations. clearly, this cost can be greatly reduced for users or moderators of the argument maps with a filtered shortlist of candidates provided by a model trained on our proposed task: node placement. ## main modeling experiments the task can be formulated as a ranking problem where a score is predicted for each candidate node and used to rank all candidate nodes. we use a biencoder to scale to the huge number of nodes that each map might contain (up to 6k nodes, c.f. section 3). using a cross-encoder that scores each pair of nodes is not feasible to apply on all nodes, but could still be used to refine the ranking of the top-n candidates based on the scores from the biencoder. this re-ranking step is out-of-scope given the noisy data available and the ambiguous nature of the task, as seen in the annotation study, which makes judging the final ranking not feasible. unidirectional relation between nodes common sentence or document embedding methods assume a bi-directional relation. for example, in the case of semantic similarity: if sentence1 x1 is similar to sentence2 x2 then sentence2 is also similar to sentence1 so: this is clearly not the case when representing parent/child relation so: this makes encoding the tree nodes into an embedding space challenging since the training should make the child nodes (c1,...,cn) closer to their parent node (p), but when p is considered as a child node with the aim of predicting its parent, it should still be closer to its parent (g) than to (c1,..,cn). the latter should be in this case regarded as negative training examples. the recursive structure of the tree might increase the effect of this issue since all nodes are eventually connected to the root. ## few-shot experiments our approach could be used for similar tasks, for which obtaining the scale of data that we used here is not feasible. motivated by this, we investigate the data efficiency of our approach, analyze the results, and where to better invest resources. ## conclusion & contributions we propose and evaluate a solution to support in creating argument maps, contributing: 1) at the methodological level, we define the new task of node placement in argument maps, and conduct an annotation study to establish the human performance on the task gaining insights about factors that affect the choice of suitable parents for a node. including prompt-based fine-tuning might be also of interest to investigate in combination with contrastive training although language modeling training would require more computational resources.