How to disagree well: Investigating the dispute tactics used on W ikipedia (2022)

Abstract: Disagreements are frequently studied from the perspective of either detecting toxicity or analysing argument structure. We propose a framework of dispute tactics which unifies these two perspectives, as well as other dialogue acts which play a role in resolving disputes, such as asking questions and providing clarification. This framework includes a preferential ordering among rebuttal-type tactics, ranging from ad hominem attacks to refuting the central argument. Using this framework, we annotate 213 disagreements (3,865 utterances) from Wikipedia Talk pages. This allows us to investigate research questions around the tactics used in disagreements; for instance, we provide empirical validation of the approach to disagreement recommended by Wikipedia. We develop models for multilabel prediction of dispute tactics in an utterance, achieving the best performance with a transformer-based label powerset model. Adding an auxiliary task to incorporate the ordering of rebuttal tactics further yields a statistically significant increase. Finally, we show that these annotations can be used to provide useful additional signals to improve performance on the task of predicting escalation.

## introduction disagreements are pervasive in online communication. while usually perceived as a negative phenomenon, research in psychology has shown that debate and disagreement can promote beliefs that are better supported by evidence @xcite . @xcite argue that, in the ideal case, a range of arguments are considered for both sides, increasing the participants' ability to find good arguments in defence of a view and to critique reasons against it, and thereby forcing one to interact with reasoning and evidence which may otherwise be discarded due to confirmation bias. prior work in nlp has investigated detecting negative artefacts of online disagreements such as personal attacks or hate speech, e.g. @xcite and @xcite . research in argumentation mining instead looks at identifying argument structures @xcite and inferring the quality of arguments @xcite , often focusing on classical theories of argumentation (e.g. aristotelian) which do not include less desirable aspects such as personal attacks. however, real world disagreements often contain both well-structured arguments and attacks, in addition to other dialogue acts, such as asking for and providing clarification. in this work, we propose a framework of dispute tactics consisting of rebuttal and coordination strategies, denoting the role of a particular utterance in the context of a disagreement discussion. we build on the disagreement hierarchy proposed by @xcite , which includes a preferential ordering between different rebuttal tactics. we introduce wikitactics foot_0 : a set of 213 disputes (comprising 3,865 utterances) on wikipedia talk pages, manually annotated with the dispute tactics employed in the process of resolving a disagreement between editors. these multiturn, multiparty conversations are sourced from the wikidisputes dataset @xcite which is annotated according to whether the dispute was resolved without the need for a moderator. an example of such a conversation is shown in fig 1 . using this framework and data, we investigate a number of research questions related to disagreements. firstly, we find that a lower mean rebuttal level in a disagreement is correlated with less constructive dispute resolutions, providing empirical validation of the ordering proposed by @xcite and recommended by wikipedia to its editors. individual users are found to utilise a range of rebuttal levels more often than adhering to only the the community put wp:engvar in place exactly because there is no rational way to resolve a style dispute like this. the notion is that if english style x is established in article, don't change it without prior consensus. without that @xcite , articles would be beset by endless edit wars over style issues that would become a time sink across the encyclopedia. hi, i am aware of wp:engvar and would like to point out to you the policy says that one should "use the variety found in the first post-stub revision that introduced an identifiable variety". in the case of this article, that is "a herb", which was introduced in the original article. i will leave the current wording for a few weeks to see if anyone else decides to weigh in, and intend to then change the page to align with policy. fenugreek: a herb / an herb it is impossible to get local consensus on this kind of thing, which is why engvar exists. leave it alone, or waste the community's time with an rfc but stop wasting your time and mine making useless arguments here. i don't care if it says "an" or "a" -what is not acceptable is messing with it. i admit that when i made those edits, i didn't realise it was actually a engvar issue but rather just a mistake, hence my zeal in making the changes. to emphasise: the policy exists to unamibiguously resolve these debates and for this article, it should be "a herb". i see no real arguments for the contrary, and for what it's worth, my having made policy-incorrect edits (in good faith), doesn't diminish the fact that policy is clear on this one. i have warned you to walk away from being a style warrior and wasting everyone's time. you will do as you will. no one further has weighed in on this and so i am making the change in accordance with policy, as i have done on each of the herb-related pages. each of these articles is now in accordance with wp:engvar. please do not edit it without an rfc or dr. we are now within the spirit and letter of policy on each of these pages and i hope we can draw a line under this ridiculous matter. contextualisation coordination rebuttal dh6: refutation dh4: repeated argument dh3: policing the discussion dh1: ad hominem attack coordinating edits suggesting a compromise dh3: policing the discussion dh4: repeated argument conceding / recanting figure 1: an example from the wikitactics dataset. different speakers are indicated by colour. wp:engvar refers to the wikipedia english variants policy. rfc and dr refer to wikipedia dispute resolution procedures. we further develop models for predicting the dispute tactics used in an utterance as a multilabel classification task, experimenting with both the binary relevance and label powerset approaches, as well as models with and without conversation context. our best model is a transformer-based model which uses a label powerset approach and context. a statistically significant improvement is gained by taking into account the preferential ordering of the rebuttal tactics defined in our scheme using multi-task training. finally, we illustrate that these annotations can be leveraged to improve performance on the task of predicting whether a dispute will be resolved without escalating to a moderator. ## online disagreements wikipedia talk pages are a popular source for nlp studies of goal-oriented discussions (e.g. @xcite . the talk pages are linked to specific articles and are used to coordinate edits and in some cases to resolve disputes. wikidisputes @xcite is a dataset of talk page discussions tagged as "disputes" by editors. the dataset provides an "escalation" label for each dispute: under the wikipedia dispute resolution policy, disputes which cannot be resolved by editors themselves are escalated to mediation, which is considered to be a proxy for constructiveness. wikipedia recommends the hierarchy of disagreement formulated by @xcite as a guide for constructive dispute resolution @xcite ). graham's hierarchy posits that there are seven levels of disagreement, ranging from namecalling (at the bottom) to refuting the central point. descriptions for these levels are provided in fig 3 in app a. graham's hierarchy has been used as a framework for research in various fields, including healthcare (cope, 2015), education @xcite and sociology @xcite . within nlp, @xcite have used this taxonomy in combination with lda topic models to distantly annotate and analyse the rationality of online discussions. despite its popularity, this hierarchy has not been verified empirically. individual levels of this hierarchy have been considered in prior work. for instance, name-calling and ad hominem attacks (levels 0 and 1) can be considered subsets of personal attacks @xcite . counterarguments (level 4) and refutations (level 5) are described in terms of arguments and evidence, a topic that has received extensive attention in the field of argumentation mining, which seeks to categorise different elements of an argument as claim or premise (see, e.g. @xcite ). contradiction (level 3), described as stating the opposing case, is related to the task of stance detection @xcite . graham's hierarchy combines these different concepts and proposes that there is an preferential ordering among them which correlates with more favourable disagreement outcomes. walker et al. (2012) also focus on online disagreements and introduced the internet arguments corpus (iac). this corpus contains utteranceresponse pairs annotated for the degree of agreement as well as whether the response is emotional versus factual, respectful versus insulting, asking questions vs asserting, and negotiating versus attack. some of these markers can be mapped to graham's hierarchy; however, @xcite consider pairs of utterances whereas our aim is to understand disagreements as a multiturn conversation. notably, their taxonomy illustrates that different disagreement markers can be used in combination: an utterance may be insulting and attacking while still being factual. it also captures the fact that within a disagreement, all discussion is not necessarily aimed at countering a proposition by an opponent; negotiating compromises and asking questions plays an important part in how disputes are resolved. another taxonomy that considers concepts related to graham's hierarchy is that of @xcite , on the topic of counterspeech. for instance, "presentation of facts to correct misstatements or misperceptions" is similar to refutation, the top level of the rebuttal hierarchy. "denouncing hateful speech" is similar to contradiction, and "hostile denouncing" is similar to name-calling and ad hominem attacks. this taxonomy does not provide the explicit arrangement in levels as @xcite does. @xcite further describes disagreements more abstractly, whereas counterspeech can be thought of as a specific form of disagreement. recent work in argumentation has explored the notion of argument quality. for example, @xcite investigate the effect of personality factors in an audience on the convincingness of an argument. @xcite introduce a corpus of paired arguments on controversial topics, and propose a decision tree for motivating why an argument is more convincing. this includes aspects such as providing an explanation with facts versus attacking the opponent, which aligns with some of the rebuttal levels described above. however, the aforementioned work on argumentation quality does not consider the conversation context but rather looks at monologic arguments. ## analysing dispute tactics in this section, we investigate four research questions regarding theories of disagreement: ## experimental setting truncated lp as mentioned in sec 5, the lp method is a popular modelling choice for multilabel classification. in our case, a large number of labels are being considered relative to the number of samples, thus we do not use the full set of all label combinations, but instead consider only the 20 most commonly applied label sets in the training set. this method provides a coverage of 85% of samples in the dataset. for samples whose labelsets are not in this category, we find the largest subset of their labels which does fall in the top 20 labelsets. if none of their labels qualify, which is the case for 175 utterances, we ignore these samples during training, but keep them in the test set. to allow for comparison between model types, we cast the predictions made by the lp model back to the multilabel setting for evaluation. incorporating conversation context while labels are assigned at the utterance level, some of the classes we aim to predict require knowledge of the preceding conversation context; for instance, "repeated argument" and "refutation" both occur in relation to another utterance. we experiment with both the context-agnostic method (to provide a baseline) and with models that incorporate the preceding context. predicting ordinality the abovementioned models do not incorporate knowledge of the preferential ordering of the rebuttal tactics. to include this signal, we add an auxiliary task which predicts the direction of the rebuttal level of the current utterance relative to the previous level, if a rebuttal tactic is used. using fig 2 as an example, for utterance 2 the model would need to predict both level 4 (main task) and a downward transition (auxiliary task) relative to utterance 1. further labels are added for upwards and same level transitions, as well as a separate class for coordination strategies which have no ordering. for the first utterance, level 3 is used as the reference level. if an utterance has multiple rebuttal labels, as indicated by the vertical lines in the figure, the maximum value was used as the reference point. ## results our experimental results are shown in as expected, the lstm and bert models tend to outperform the bow models. adding conversation context improves performance in all models on the jaccard score. the best performing model on the jaccard metric is the bert model with context. the truncated lp method achieves better results for the majority of models and metrics compared to the binary relevance formulation, despite truncating the label powerset and ignoring 15% of the training data. this can be attributed to the fact that there are certain classes with stronger co-occurrence relations (e.g. personal attacks with higher level arguments) which benefits the lp method. to gain a better understanding of model performance, we calculate the proportion of the test set with at least one label correctly predicted, yield-ing 0.395 on the best model (an increase of 0.11 over the emr). the label most frequently correctly predicted is coordinating edits (111 of 137 cases), which is also the most common label in the training set. the next most correctly predicted label, proportionally, is contextualisation (75%, or 24 of 32 cases), despite not being a commonly used label. this is likely due to the additional positional information available to the model, since this label is often applied to the first utterance in a conversation. on the other hand, refutation and refuting the central point are never correctly predicted (out of 44 cases), with counterargument often mistakenly predicted instead. this is likely because of similarities between the classes and the latter being the second most common label in the training set. we build on the best performing model to evaluate the effect of including the ordinality prediction auxiliary task. using this model, we obtain a statistically significant improvement (@xmath0.03, using the permutation test on the jaccard score) over the best model that is not aware of the ordering of the rebuttal tactics, providing further evidence of the usefulness of the rebuttal hierarchy. we also train models using the median and minimum rebuttal levels of each utterance, and find that while these do provide an increase in the jaccard score, the difference is not significant (@xmath1.32 and @xmath2.13). ## predicting escalation based on the findings of sec 4, we believe that the dispute tactic annotations provided in the wiki-tactics dataset can provide useful additional learning signals for the task of predicting escalation, as formulated by @xcite . we therefore do multitask training with escalation as the main task and tactics as the auxilliary task, such that the features that are predictive of dispute tactics are incorporated in the escalation predictions. de @xcite found that a han network achieved the best results on this task; thus, we reproduce their experiment and achieve a pr-auc score of 0.40. to add the dispute tactic classification as an auxiliary task, we modify the contextaware lstm model, which uses a han model to encode the preceding context to an utterance (as described in sec 6). to predict escalation for a given conversation, this han model is used to obtain a conversation embedding. two further fully connected layers are added to allow for feature specialisation for the escalation prediction task. using this model, a pr-auc score of 0.487 is obtained, indicating that knowledge of these dispute tactics is useful for tasks beyond classifying the tactics employed. ## conclusion in this work, we introduced a framework and dataset for dispute tactics, consisting of rebuttal and coordination strategies. we used these annotations to analyse how different tactics are used in disagreements and by different individuals, providing insights on the context in which personal attacks occur and the extent to which users alter their own tactics to mirror the level of rebuttal used in a conversation. we further develop multilabel models for classifying the dispute tactics used in an utterance, experimenting with both binary relevance and label powerset methods. we further illustrate that knowledge of these tactics increases accuracy on the task of predicting escalation, indicating that the framework and dataset can be of use to researchers working on other aspects of disagreements online.