Topic Ontologies for Arguments (2023)

Abstract: Many computational argumentation tasks, such as stance classification, are topic-dependent: The effectiveness of approaches to these tasks depends largely on whether they are trained with arguments on the same topics as those on which they are tested. The key question is: What are these training topics? To answer this question, we take the first step of mapping the argumentation landscape with The Argument Ontology (TAO). TAO draws on three authoritative sources for argument topics: the World Economic Forum, Wikipedia’s list of controversial topics, and Debatepedia. By comparing the topics in our ontology with those in 59 argument corpora, we perform the first comprehensive assessment of their topic coverage. While TAO already covers most of the corpus topics, the corpus topics barely cover all the topics in TAO. This points to a new goal for corpus construction to achieve a broad topic coverage and thus better generalizability of computational argumentation approaches.

## introduction the term "topic" refers to the subject matter of a text. a text may be about one or more topics and the relationship between topics and texts is called "aboutness" @xcite . topics play a central role in argumentation because they determine argumentation strategies and rhetorical devices by setting the appropriate and expected universe of discourse. this view is supported by pragmadialectics @xcite : "the basic aspects of strategic maneuvering [. . . ] are making an expedient selection from the 'topical potential' available at a certain discussion." although debaters often use commonplace arguments across topics @xcite , they must be relevant: a black market argument, for example, can be equally well applied to topics such as banning drugs or banning firearms. as recently shown, for example, by @xcite , training computational models to extract, analyze, or generate arguments with a broad topic coverage improves their generalizability. a set of topics can be organized as a graph, sometimes called a "topic space". information theorists and library scientists map hierarchical subject relationships into ontologies in this way @xcite . for this purpose, topics are labeled with a subject heading, a phrase from a controlled vocabulary that describes a topic in a concise and discriminating manner. while library ontologies are not focused on argumentation, others deal specifically with argumentative topic spaces. we have identified and tapped three authoritative sources of ontological knowledge covering global issues, controversies, and popular debates: the world economic forum's "strategic intelligence" site, wikipedia's list of controversial topics, and debatepedia's debate classification system (section 4). they form the basis for the argument ontology (tao). 1 we compile a comprehensive survey of 59 argument corpora ( section 3) and investigate their topic coverage with respect to the three authoritative ontologies (section 5). the coverage of corpora with topic labels is manually assessed by matching each label with the topics of the ontologies. from this, the ontology topics covered by a corpus and the distribution of corpus arguments in the ontologies are calculated. our analyses show that the existing corpora focus on only a subset of the known topics. for corpora without topic labels, we categorize their argumentative texts by measuring their semantic relatedness to ontology topics. given the large number of ontology topics (748 for wikipedia), this is a challenging classification for which we achieve a remarkable f 1 of 0.59. (section 6). 2 altogether, we lay the foundation for the study and systematic exploration of controversial topics within computational argumentation analysis. the authoritative sources identified already cover their respective areas quite comprehensively. future work will need to extend our approach to other subject areas, such as business, domestic, historical, and scientific argument spaces. ## related work our review of related work focuses on the role of the variable "topic" in computational argumentation. moreover, we briefly review topic ontologies and hierarchical topic classification. ## survey of argument corpora to study arguments and computational argumentation tasks, researchers compile corpora with argumentative texts. to the best of our knowledge, the most elaborate discussion of topic selection is given in @xcite , who chose six topics (homeschooling, public versus private schools, redshirting, prayers in schools, single sex education, mainstreaming) to focus on different education-related aspects. the broadest selection of topics is reported by the researchers of ibm debater, 4 who obtain arguments from wikipedia. however, samples of the topics have been used in their papers without mentioning which ones. the only other work mentioning their source of topics stems from @xcite , who randomly select 8 topics from two lists of controversial topics that originate from an online library and the debate portal procon.org, respectively. peldszus and stede (2015) predefine a set of topics and give writers the freedom to choose which one to write about, but nothing is said about where the set of predefined topics originate from. @xcite and hasan and ng (2014) explicitly select one and 4 https://www.research.ibm.com/haifa/dept/vst/debating_data.shtml four topics, respectively. for all other corpora with topic labels, their authors do not argue on choosing topics, nor selection or sampling criteria. neither do the authors of corpora without topic labels. altogether, it appears that the best practices in argumentation do not as of yet consider topic sampling as a prerequisite task to ensure coverage of a certain domain of interest, diversity, or reproducibility. based on our review, we presume three basic topic selection directives are in use today: the great reset shaping the economic recovery revitalizing global cooperation taxation global governance politics and economics religion science, biology, and health ## bootstrapping the argument ontology topic ontologies provide for a knowledge organization principle, and, especially if widely accepted, also a standard. they are typically modeled as directed acyclic graphs, where nodes correspond to topics and edges indicate "is part of" relations. topics that are part of other topics are called their subtopics. a topic ontology is often displayed in levels, starting with the topics that are not subtopics of others, continuing recursively with each lower level of subtopics. figure 1 shows an excerpt of a two-level topic ontology for arguments. the identification of the topics to be included in the argument ontology (tao), as well as their relations, requires domain expertise. building an all-encompassing ontology thus requires experts from every top-level domain where argumentation of scientific interest is expected. in the following, we suggest and outline three authoritative sources of expert topic ontologies, which comprise a wide selection of important argumentative topics. we use them to bootstrap a first version of tao. ## topic coverage to assess the topic coverage of an argument corpus given the three ontologies, we map their topic labels (if provided) to matching ontology topics. ## unit categorization the previous analysis assesses argument corpora which contain topic labels. about a third of the argument corpora do not. as a heuristic step to assessing their topic coverage, we map the ontology topics for a unit ( to retrieve topics for a corpus unit, we implement and evaluate the following approaches: semantic interpretation (si) and si with text embeddings (text2vec-si). the semantic interpretation approach computes the semantic similarity of a unit and a topic as follows: it uses the cosine similarity of the tf-idf vectors for the unit and the concatenated topic's documents. this corresponds to the semantic interpretation step that is at the core of the well-known esa model @xcite . text2vec-si calculates the similarity of topics and corpus units using bert embeddings @xcite . following common practice, we take the dimension-wise average of the word embeddings for all tokens in the text. 8 we tried other embeddings and approaches that performed similarly. the results of these approaches can be found in the appendix. as a baseline, we implement a direct match approach, which assigns a unit an ontology topic if the topic's text appears in the unit text (ignoring case). for evaluation, we collect 34,638 pooled query relevance judgments (0.53 inter-annotator agreement as per krippendorff's α) on 104 randomly selected argument units as queries from 26 corpora. the annotation process is detailed in the appendix. based on the similarity scores of the approaches, we derive boolean labels that indicate whether a unit is or is not about one of the ontologies' topics using two policies. the threshold policy labels a unit as about a topic if their similarity is above a threshold θ. the top-k policy labels a unit as about a topic if the topic is among the top-k topics with the highest similarity to the unit. we report the parameter of the policy that achieved the highest f 1score on the pooled judgments for each approach. ## conclusion the computational argumentation community risks topic bias in its approaches if the representativeness of topics in future corpora is not ensured. achieving topic coverage is complicated by the fact that the landscape of controversial topics has not yet been well explored, and that there are no widely accepted ontologies for argument topics. in this paper, we venture into this future by mapping the landscape of argument topics and making it accessible for corpus construction and experimental design. we have identified three authoritative sources of ontological knowledge related to argument topics that provide an initial foundation for the argument ontology (tao). for each source ontology, we evaluate the topic coverage of 39 argument corpora labeled with topics by matching the labels with the topics of the ontologies. to evaluate the topic coverage of corpora without topic labels, we develop an approach to identify the ontology topics of an argumentative text and achieve an f 1 of 0.59. our analyses show that the topic coverage of existing argument corpora is both limited to a subset of the topics of the ontologies and skewed. most topics that require expertise, such as mental health, philosophy, or international security, are treated only peripherally in argumentation corpora. therefore, existing argumentation technologies are more suited to teaching people how to construct arguments in general than to helping them make decisions about such and similarly complex topics. for the development of robust argumentation technologies, corpora need to be carefully drawn from a specific domain to allow for reliable experiments and the development of generalizable classifiers. future work for further development of tao consists mainly of further surveying the argument topic landscape and unifying the various available ontologies. in addition to "is part of" relationships between topics, other relationship types can also be considered to build an argument topic knowledge base. however, our first version of tao and our analyses can already help in selecting arguments for future corpus construction and model training.