I Beg to Differ: A study of constructive disagreement in online conversations (2021)

Abstract: Disagreements are pervasive in human communication. In this paper we investigate what makes disagreement constructive. To this end, we construct WikiDisputes, a corpus of 7425 Wikipedia Talk page conversations that contain content disputes, and define the task of predicting whether disagreements will be escalated to mediation by a moderator. We evaluate feature-based models with linguistic markers from previous work, and demonstrate that their performance is improved by using features that capture changes in linguistic markers throughout the conversations, as opposed to averaged values. We develop a variety of neural models and show that taking into account the structure of the conversation improves predictive accuracy, exceeding that of feature-based models. We assess our best neural model in terms of both predictive accuracy and uncertainty by evaluating its behaviour when it is only exposed to the beginning of the conversation, finding that model accuracy improves and uncertainty reduces as models are exposed to more information.

## introduction disagreements online are a familiar occurrence for any internet user. while they are often perceived as a negative phenomenon, disagreements can be useful; as is illustrated in figure 1 , disagreements can lead to an improved understanding of a topic by introducing and evaluating different perspectives. research on online disagreements has focused on mostly negative aspects such as trolling @xcite , hate speech @xcite , harassment @xcite and personal attacks @xcite . recent works by @xcite and @xcite study conversations on wikipedia talk pages that start out as civil but derail into personal attacks, using linguis-figure 1 : an example of a dispute on wikipedia, showing the dispute tag on the article, talk page posts, and summaries of edits occurring during the discussion. tic markers and deep learning approaches, respectively. an alternative approach is to study good faith disagreement through debates on online platforms such as changemyview @xcite , debate.org @xcite and kialo @xcite , or formal oxford-style debates @xcite . while this has benefits, such as readily available annotation of stances and indication of winning and losing sides, it does not mirror the way people naturally converse. for example, in formal debates, temporal and structural constraints are imposed. moreover, in oxford-style debates, participants are not motivated to come to a consensus but rather to persuade an audience of a predefined stance @xcite . finally, the task of detecting disagreement in conversations has been studied by a number of authors, e.g. @xcite and @xcite , without attempting to analyze it further. we are interested instead in constructive disagreement in an uncoerced setting. to this end, we present wikidisputes foot_0 ; a corpus of 7 425 disagreements (totalling 99 907 utterances) mined from wikipedia talk pages. to construct it, we map dispute tags in the platform's edit history (shown in figure 1 ) to conversations in wikiconv @xcite to locate conversations that relate to content disputes. observing that conversations are often conducted in the talk pages and the edit summaries simultaneously, as illustrated in figure 1 , we augment the wikiconv conversations with edit summaries that occur concurrently. to investigate the factors that make a disagreement constructive, we define the task of predicting whether a dispute is eventually considered resolved by its participants, or it was escalated by them to mediation by a moderator, and therefore considered unconstructive. we evaluate both feature-based and neural models for this task. our findings indicate that balancing hedging and certainty plays an important role in constructive disagreements. we find that incorporating edit summaries (which have been ignored in previous work on wikipedia discussions) improves model performance on predicting escalation. we further find that including information about the conversation structure aids model performance. we observe this in two ways. firstly, we include gradient features, which capture changes in linguistic markers throughout the conversations as opposed to averaged values. secondly, we experiment with adding sequential and hierarchical conversation structure to our neural models, finding that a hierarchical attention network @xcite provides the best performance on our task. we further evaluate this model in terms of its predictive accuracy when exposed to the beginnings of the conversation as opposed to completed conversations, as well as its uncertainty (more details in section 6.4). our results indicate that model performance is reduced from a pr-auc of 0.29 to 0.19 when exposed to only the first half of the conversation. however, this reduced model still outperforms toxicity and sentiment models predicting on full conversations. model uncertainty decreases roughly linearly as the model is exposed to more information. we conduct a qualitative analysis of the points in the conversation where the model changes its prediction on constructiveness, finding that some markers from our feature-based models also seem to have been picked up by the neural model. ## dispute resolution on wikipedia wikipedia relies heavily on collaboration and discussion by editors to maintain content standards, using the platform's talk pages @xcite . wikipedia contributors add "dispute" templates to articles @xcite referring to a content accuracy dispute or to a violation of the platform's neutral point of view (npov) policy @xcite . adding such a template to the article creates a dispute tag on the article as shown in figure 1 . variations of these tags allow contributors to flag specific sections or phrases which relate to the issue in question, or to add details of the dispute. all edits to wikipedia, including the aforementioned dispute tags, are retained in the site's edit history. revision metadata contains a description of the edit (hereafter referred to as the edit summary). as observed in figure 1 , when such edits occur while a conversation is underway, editors use the "edit summaries" to clarify their intentions in the context of the conversation. the wikipedia dispute resolution policy stipulates that if contributors are unable to reach a consensus through talk page discussion, they can request mediation by a community volunteer. a prerequisite for this escalation step is proof of recent, extensive discussion on a talk page foot_1 . ## wikidisputes our dataset consists of three facets: disagreements on talk pages, edit summaries, and escalation labels. wikipedia informs users that all contributions on article or talk pages are published publicly, and provides channels for users to permanently remove personal information that they have shared accidentally or otherwise @xcite , thus we are able to release this data to the community. in the remainder of this section, we detail the process for obtaining the conversations, the edit summaries and the labels. ## modelling constructive disagreement 4.1 feature-based models we develop our feature-based models using feature sets suggested in previous research on tasks related to conversational analysis. these include: for each feature, we calculate the average value throughout the conversation, as well as the gradient of a straight line fit of the feature value throughout the conversation. our intuition for including this variation is that the outcome of a disagreement is likely to be influenced by a change in the language usage through the course of a conversation, which is not reflected by the mean. for instance, a disagreement might start out very politely and end impolitely, or the other way around, and would have the same mean. logistic regression is used to infer linear relationships between the linguistic features and disagreement outcomes. ## experimental setup an initial data analysis indicated that escalated disagreements have a median length of 16 utterances, compared to 11 for their non-escalated counterparts. while this may be an informative feature for classification, we are primarily interested in the effects of language on the conversation outcome, and therefore choose to control for this effect. we use matching, a technique developed for causal inference in observational studies @xcite , also employed by @xcite in the context of conversational modelling. we pair each escalated disagreement with a non-escalated disagreement of the same length in utterances, bearing in mind that the dataset is heavily imbalanced, with 7 425 non-escalated disagreements compared to 201 escalated disagreements. to retain an imbalance in the classes while conducting matching, we match every escalated disagreement with up to ten nonescalated disagreements (the actual number depending on availability) by randomly sampling without replacement from the non-escalated disagreements of the same length. characteristics of the dataset after performing matching are shown in we split the dataset for training as indicated in ## results results are shown in ## conclusion in a discussion of online disagreements by @xcite (which wikipedia references in its guidelines for resolving disputes), the author remarks that an increase in disagreements through widespread online interaction has the potential to create a surge in anger among internet users. on the other hand, as illustrated in our data, disagreements can sometimes be constructive. the success of wikipedia shows the benefits of integrating multiple perspectives. as hahn (2020) states, "arguing things through is at the center of our attempts to come to accurate beliefs about the world @xcite decide on a best course of action." for this reason, it is important to understand why disagreement can sometimes be constructive, and in other cases leads to conversational failure. in this work, we investigated constructive disagreements from an nlp perspective. we have proposed a dataset of disagreements online and defined the task of predicting escalation as a proxy label for cases where disagreements were unconstructive. we analysed features that are associated with either class, and drew parallels with existing work. using neural network models, we investigated the effect of modeling conversation structure and found that adding utterance hierarchy lead to an increase in performance. finally, we validated our neural models by evaluating their performance and uncertainty when exposed to partial conversations. our insights on constructive disagreements are not limited to wikipedia and would be transferable to disagreements on other platforms. additionally, our finding that edit summaries are an informative part of talk page conversations should be useful for researchers who work on wikipedia talk pages more generally.