Hi- A r G : Exploring the Integration of Hierarchical Argumentation Graphs in Language Pretraining (2023)

Abstract: The knowledge graph is a structure to store and represent knowledge, and recent studies have discussed its capability to assist language models for various applications. Some variations of knowledge graphs aim to record arguments and their relations for computational argumentation tasks. However, many must simplify semantic types to fit specific schemas, thus losing flexibility and expression ability. In this paper, we propose the **Hi**erarchical **Ar**gumentation **G**raph (Hi-ArG), a new structure to organize arguments. We also introduce two approaches to exploit Hi-ArG, including a text-graph multi-modal model GreaseArG and a new pre-training framework augmented with graph information. Experiments on two argumentation tasks have shown that after further pre-training and fine-tuning, GreaseArG supersedes same-scale language models on these tasks, while incorporating graph information during further pre-training can also improve the performance of vanilla language models. Code for this paper is available at <https://github.com/ljcleo/Hi-ArG>.

## introduction debating is a fundamental formal process to find solutions and gain consensus among groups of people with various opinions. it is widely accepted in politics @xcite , education @xcite and online discussions @xcite . as more and more formal and informal debates are launched and recorded on the internet, organizing arguments within them has become crucially valuable for automated debate preparation and even participation @xcite . recently, language models have shown dominating advantages in various argumentation tasks @xcite . however, such tasks often rely on logical relations we should ban guns. guns are killing people. we shouldn't ban guns. guns can prevent killing if everyone has one. attack support support figure 1 : an example of hi-arg extracted from the conversation between alice and bob. top0 to top3 represent top nodes corresponding to different arguments. g0 to g3 represent intra-argument sub-graphs whose detailed structures are omitted. rec. and -rec. represent recommend and not recommend respectively. despite the effectiveness of akg, we argue that semantics within arguments can be better modeled. in this paper, we propose the hierarchical argumentation graph (hi-arg), a graph structure to organize arguments. this new structure can retain more semantics within arguments at the lower (intra-argument) level and record relations between arguments at the upper (inter-argument) level. we claim that hi-arg provides more explicit information from arguments and debates, which benefits language models in argumentation tasks. for instance, consider the following conversation about whether guns should be banned: alice and bob have opposite opinions, yet their evidence focuses on the topic's same aspect (killing). in hi-arg (see figure 1 ), all their claims and premises can live together and connect through inter-arg (supporting/attacking) and intra-arg (semantic relations between concepts like guns and killing) links. to validate the power of hi-arg, we propose two approaches to exploit information from the graph. first, we introduce greasearg, a text-graph multi-modal model based on greaselm @xcite to process texts with sub-graphs from hi-arg simultaneously. second, we design a new pre-training method assisted by hi-arg, which is available for both greasearg and vanilla language models. to examine the effectiveness of these approaches, we conduct experiments on two downstream argumentation tasks namely key point matching (kpm) and claim extraction with stance classification (cesc). in general, our contributions are: ## related work computational argumentation is an active field in nlp focusing on argumentation-related tasks, such as argument mining @xcite @xcite , stance detection @xcite , argumentation quality assessment @xcite , argument generation @xcite @xcite and automated debating @xcite . a majority of these tasks are boosted with language models recently @xcite @xcite , and studies have discussed various ways to inject external argumentation structure or knowledge to such models @xcite . meanwhile, several works have discussed proper methods to retrieve arguments from data sources @xcite @xcite , yet a majority of them keep the text form unchanged, resulting in loose connections between arguments. one of the methods to manage textual information is to represent it as knowledge graphs; however, popular knowledge graphs mainly record concept relations @xcite @xcite . to cope with this issue, @xcite proposed a knowledge graph that stores causal connections between concepts. @xcite further developed this idea and introduced the argumentation knowledge graph (akg), which can be seen as the first approach to organize arguments structurally, and has been employed in a few downstream tasks @xcite . a flaw of the akg is that relations between concepts are highly simplified, losing rich semantic information. the representation of semantic relations has been long studied as the task of semantic role labeling @xcite @xcite . while traditional srl keeps the linear form of sentences, @xcite proposed abstract meaning representation (amr), a graph-based structure to express semantic relations. being similar to knowledge graphs, the idea becomes natural to record arguments in amr graphs and organize them, then utilize them for downstream tasks in a similar way as typical knowledge graphs, especially accompanied with language models @xcite @xcite . ## exploiting hi-arg knowledge graphs can participate in various stages of standard nlp pipelines, such as information retrieval and model training. in this section, we introduce two methods of exploiting hi-arg on the model side. first, we present a model structure similar to greaselm @xcite , referred to as greasearg, that simultaneously digests text and hi-arg data. we also propose further pre-training tasks specially designed for greasearg. second, we develop a novel pre-training framework that modifies the method to construct training samples with another self-supervised pre-training task. ## experiments to examine our approaches to exploit hi-arg, we apply them to selected argumentation tasks and compare the results with previous work. all models are further pre-trained on argumentation corpus with generated hi-arg and then fine-tuned on specific downstream tasks. we report the main results at the end of this section. ## conclusion in this paper, we propose a new graph structure for argumentation knowledge, hi-arg. we design an automated construction pipeline to generate the lower intra-arg graph. to exploit information from hi-arg, we introduce a text-graph multi-modal model, greasearg, and a novel pre-training framework. on two different argumentation tasks (kpm and cesc), these approaches create models superseding current models under the same scale.