Semi-Supervised Cleansing of Web Argument Corpora (2020)

Abstract: Debate portals and similar web platforms constitute one of the main text sources in computational argumentation research and its applications. While the corpora built upon these sources are rich of argumentatively relevant content and structure, they also include text that is irrelevant, or even detrimental, to their purpose. In this paper, we present a precision-oriented approach to detecting such irrelevant text in a semi-supervised way. Given a few seed examples, the approach automatically learns basic lexical patterns of relevance and irrelevance and then incrementally bootstraps new patterns from sentences matching the patterns. In the existing args.me corpus with 400k argumentative texts, our approach detects almost 87k irrelevant sentences, at a precision of 0.97 according to manual evaluation. With low effort, the approach can be adapted to other web argument corpora, providing a generic way to improve corpus quality.

## introduction computational argumentation research lays the ground for applications that support opinion formation, including argument search engines @xcite , collective deliberation @xcite , and debating technologies @xcite . such applications rely on large pools of up-to-date arguments, which can hardly be found anywere but on the web. one of the most used web argument sources are debate portals where people jointly collect arguments or debate each other on defined issues. debate portals, and similar web platforms, are rich of argumentatively relevant content and structure, including arguments as well as facts, background information, and similar. this enables researchers to crawl large-scale argument corpora in a distantly-supervised manner @xcite . however, the texts found on debate portals also comprise debate-specific language and boilerplate text that is likely to be irrelevant, if not even detrimental, to the mentioned applications. in the text in figure 1 , for instance, the author defines the debated issue (sentence #2), states a thesis (#3-5), and presents two arguments (#6-8, #9-13) -all of which can be considered argumentatively relevant. in contrast, sentences #1, #14, and #15 add nothing of importance, merely making meta-comments and expressing gratitude. in other cases, irrelevant text includes salutations, insults, purely rhetorical moves, and spam. as detailed in section 2, finding such text differs from finding non-argumentative text segments, since the latter may still be relevant as context for the argumentative segments, as in the case of sentence #2 in figure 1 . many existing approaches relying on debate portals do not clean the crawled arguments from irrelevant text. until now, for example, the argument search engine args.me @xcite has just returned the full shown text as one pro argument for the query "gay marriage". this at least harms user experience, and it might even corrupt the support of opinion formation in some cases. in this paper, we study how to find irrelevant text in web arguments such as those from debate portals automatically, in order to clean respective corpora on this basis. in particular, we develop a semi-supervised learning approach that aims to detect as many irrelevant sentences as possible with very high precision, i.e., hardly any relevant sentence should be classified as irrelevant (section 3). given a seed set of sentences, the approach learns basic lexical n-gram patterns that frequently match text in either relevant or irrelevant irrelevant #1 irrelevant #14 irrelevant #15 https://www.debate.org/debates/gay-marriage/75/ argumentative debate post pro "gay marriage" relevant #2 relevant #3-5 relevant #6-8 relevant #9-13 (non-argumentat.) (argumentative) (argumentative) (argumentative) i would like to thank brainmaster for accepting this debate. gay marriage is basically the marriage between two individuals of the same gender, i trust my opponent will have no problem with this definition. i will be arguing for gay marriage, and that it should be legal. i will be arguing that everything that does not physically harm other individuals should be legalized, gay marriage is one of these things. i will also be arguing that by banning the gay marriage we have gone against human rights. c1: gay marriage does not physically harm other individuals in any way shape or form therefore, it should be legal. a marriage is a union between two individuals that love eachother, and it basically only effects these two individuals. if it is banned then it is hurting people, and if it is legalized then it isn't hurting anyone. c2: banning gay marriage is against human rights. every person is born with the equal human rights which are life, liberty, and the pursuit of happiness. yet, banning gay marriage goes against to of these fundamental rights. how can someone pursue happiness when they can't marry the one they love? how can someone have liberty when they are not allowed to marry the one they love. ## related work initially, research on tasks such as argument mining has largely been carried out on small, well-curated collections of texts, including wikipedia articles @xcite , student essays @xcite , pure arguments @xcite , and presidential debates @xcite . major real-world applications of computational argumentation, however, need to scale up to web contexts to fulfill their purpose. this includes search engines that oppose pro and con arguments on controversial issues @xcite , technologies that debate humans @xcite , and more. to obtain web arguments, many works have relied on crawled debate portals and similar web platforms, often in a distant-supervision manner where argumentative structure and similar annotations are directly derived from available meta-information @xcite . corpora have been built in such a way based on several debate portals, including 4forums.com @xcite , idebate.org @xcite , createdebate.com @xcite , debate.org @xcite , and reddit.com/r/changemyview @xcite . naturally, less curation of the acquired web texts comes at the cost of more noise, which in turn calls for a cleansing of the resulting corpus. cleansing processes are described in several publications on argument corpora, mostly only referring to the acquired annotations though @xcite @xcite . in contrast, the paper at hand targets the cleansing of the corpus texts themselves. only few works describe respective cleansing steps in detail. among these, @xcite deleted special symbols and debate-specific phrases such as "this house" from crawled arguments, and @xcite removed quotations of previous posts in debate posts. @xcite discarded certain types of noisy instances completely for the argument search engine args.me, but the texts in the original associated corpus @xcite still contain much irrelevant text, as our experiments will reveal. applying our approach has led to an improved version of that corpus. in this paper, we introduce a semi-supervised learning approach for corpus cleansing. in general, we follow the bootstrapping idea of successful pattern mining methods, such as dipre @xcite , snowball @xcite , and espresso @xcite . while these methods aim at semantically relevant information, we distinguish pragmatically relevant from irrelevant text within an author's argumentative discourse. we are not aware of any other approach in this direction. it is noteworthy in this regard that the cleansing task at hand differs notably from the unit segmentation of argumentative texts @xcite . while all argumentative units match the notion of relevance considered here (defined in section 3), also non-argumentative units may be seen as relevant, if they give facts, definitions, or other background information serving as context for the argumentative units. as such, our notion of relevance relates to the local relevance with respect to some conclusion rather than the global relevance of an argumentative statement in the discussion of an issue @xcite . ## approach this section presents our semi-supervised learning approach to detecting irrelevant text in web arguments as well as to clean a respective corpus on this basis. the approach aims to find as many irrelevant text units as possible at an estimated precision beyond a threshold τ (in section 5, we use τ = 0.95). to this end, it learns linguistic patterns that occur often in irrelevant units and rarely in relevant units (or vice versa). later, we consider each sentence as one unit, but other granularities would work in principle, too. figure 2 gives an overview of the three main stages of the approach, each of which will be detailed below: it is important to see that the relevance patterns are eventually not used for the actual cleansing. they serve to distinguish relevant from irrelevant units only, thereby aiding the identification high-precision irrelevance patterns. while we have designed our approach for web arguments in particular, notice that the outlined processed is largely generic and could easily be transferred to other cleansing tasks where relevant and irrelevant units can be distinguished. what makes our approach specific to web arguments is what we mean by argumentative relevance and irrelevance. ## data the presented approach targets argumentative language of varying quality, as often observed in web-based corpora. below, we assess its impact on the args.me corpus @xcite , which is to our knowledge the largest available argument corpus to this date, about 7.3 gb in file size. the corpus represents the database underlying the argument search engine args.me @xcite arguments that were mined from four debate portals using distant supervision: debate.org, debatewise.org, idebate.org, and debatepedia.org. each argument consists of a mostly very short conclusion as well as a mostly notably longer premise, the latter containing the actual argumentative text. in total, the corpus spans around seven million sentences. we see each sentence as one unit in our approach. many texts in the args.me corpus include sentences that are irrelevant to the actual argument, such as the example in figure 1 . needless to say, no ground-truth information on irrelevance is given, though. for a rough estimation of the proportion of irrelevant sentences, we conducted a pilot study where the two authors of this paper independently decided about the relevance of a set of sentences, following the definition in section 3. in particular, we considered a corpus sample used previously by @xcite , which contains the top five pro and the top five con arguments each for the top 10 queries. from the 1294 sentences in the 100 sample arguments, one of us classified 147 (11.3%) to be irrelevant, the other one 139 (10.7%). in terms of cohen's κ, we had a substantial inter-annotator agreement of 0.75. in total, 175 sentences (13.5%) were seen as irrelevant by either of us, 111 (8.5%) by both. since we believe that, in doubt, a sentence should be deemed relevant, we take 8.5% as our estimation. in the whole corpus, we thus expect around 600,000 sentences to be irrelevant. the 111 sentences come from only 39 of the 100 arguments. assuming this number is representative, about 150k arguments in the corpus should contain irrelevant sentences. in the following experiments, these numbers will give us a rough idea of the recall of our approach. there, we use a random 10% sample of all corpus arguments for the seed pattern selection, and the whole corpus for all subsequent steps. ## evaluation we now report on the step-by-step application of our approach from section 3 to the corpus from section 4 and on the manual evaluation of the obtained results. the goal was to assess the impact of the approach on the quality of web-based argument corpora. we hypothesized that the approach is able to detect a large number of irrelevant sentences with a precision as high as its internal precision threshold τ . 4 type n-grams seed patterns relevance 1-grams government (94198), states (85388), state (68123), law (59609), society (58695), money (54314), death (52327), universe (49412) 2-grams big bang (9370), minimum wage (8650), human rights (8592), god exists (7959), health care (7537), years ago (7165), global warming (6399), high school (6235), opponent claims (6160), believe god (6151), human beings (6136), video games (6051), god exist (5810), existence god (5636), jesus christ (5592), supreme court (5513), new york (4890), human life (4838), old testament (4640), years old (4632), god created (4621), god would (4495), self defense (4089), merriam webster (2015) ## conclusion web-based argument corpora play an important role in computational argumentation research and its applications. not all text in such corpora is relevant to the arguments, though. in this paper, we have presented an approach that detects irrelevant text units in argumentative texts with low supervision. the approach iteratively bootstraps linguistic patterns of irrelevance and relevance from units matching known patterns. on the 387k arguments in the args.me corpus, the approach detected 87k irrelevant sentences at a precision of 0.97, from which at least 53k can be removed without notably reducing the arguments' coherence. these results demonstrate the potential of our approach to improve corpus quality. naturally, the approach has limitations. on one hand, the results revealed that, under the employed configuration, a large proportion of detected sentences came from the seed patterns. to obtain good seed patterns, manual effort is needed. on the other hand, the recall of our approach seems not so high, as far as we can estimate from the data inspected. while not all irrelevant units can be captured by the simple patterns we considered, another reason may lie in the restriction that only new candidate patterns are found which occur in sentences matching previous patterns. particularly patterns that show up only in short units may thus be overlooked, if they are not covered by the seed patterns already. improvements might, e.g., consider units adjacent to irrelevant units, but this may come at the cost of reduced precision. in this regard, notice that the impact our approach to some extent depends on the availability of a reliable unit boundary detector (say, a sentence splitter), which is not a trivial requirement for noisy web data. finally, an arising question may be how complex it is to apply the approach to other than the data processed here. following our proposed process to obtain frequent candidate seed patterns automatically, the main manual effort boils down to finding reliable seed patterns among these candidates. in our case, this took no more than a few hours, which seems negligible given the potential impact on corpus quality. besides, only some initial tuning of the approach parameters to the data at hand may be needed. we are thus confident that the approach can be easily adopted to clean other argument corpora (including transcribed corpora with spoken argumentative language) as well as to many other cleansing tasks where the irrelevance of text units can be defined in a measurable way.