W iki T alk E dit: A Dataset for modeling Editors’ behaviors on W ikipedia (2021)

Abstract: This study introduces and analyzes WikiTalkEdit, a dataset of conversations and edit histories from Wikipedia, for research in online cooperation and conversation modeling. The dataset comprises dialog triplets from the Wikipedia Talk pages, and editing actions on the corresponding articles being discussed. We show how the data supports the classic understanding of style matching, where positive emotion and the use of first-person pronouns predict a positive emotional change in a Wikipedia contributor. However, they do not predict editorial behavior. On the other hand, feedback invoking evidentiality and criticism, and references to Wikipedia’s community norms, is more likely to persuade the contributor to perform edits but is less likely to lead to a positive emotion. We developed baseline classifiers trained on pre-trained RoBERTa features that can predict editorial change with an F1 score of .54, as compared to an F1 score of .66 for predicting emotional change. A diagnostic analysis of persisting errors is also provided. We conclude with possible applications and recommendations for future work. The dataset is publicly available for the research community at https://github.com/kj2013/WikiTalkEdit/ .

## introduction dialogue is a language game of influence, action, and reaction that progresses in a turn-taking manner. persuasion occurs through dialogue when a listener favorably evaluates the authority, claims, and evidentiality through the cues and arguments made by the speaker @xcite @xcite . discussions on wikipedia talk pages can be useful for determining strategies that lead to an improvement of the article discussed, and for examining if they also lead to an amicable dialogic ex-change. previous work @xcite has explored the role of editors and the types of edits made on wikipedia, but have not related them to the ongoing conversation on the wikipedia talk pages. we introduce the wikitalkedit dataset, a novel dataset for research in online collaboration. the dataset is a subset of the wikipedia talk corpus available as of may 2018 foot_0 . it contains 12,882 dialogue triples with labels about editors' subsequent editorial (editing) behavior, and 19,632 triplets with labels corresponding to editors' emotion as manifested in their replies. ## related work conversational quality is largely the focus of a body of work modeling the formal @xcite , polite @xcite and toxic @xcite features of comments on wikipedia, reddit, and other online public forums. the labels in such a task are often subjective as they depend mostly on annotated or crowdsourced labels. on the other hand, gauging the impact of a conversation in terms of a reader's subsequent behavior is a rather different problem. a few studies have modeled the language of arguments to predict their upvotes @xcite @xcite . the best result reported by @xcite was an f 1 score of .35 for the task of predicting which of two arguments was better, using svms and bi-directional lstms. the study by @xcite reported an accuracy of 60% for predicting which argument was most likely to change the original poster's (op's) point of view. @xcite report an auc of .67 on predicting the success on ∼5700 requests. studies predicting users' stance @xcite have done better, but do not usually factor in the feedback from a turn-taking partner, during a dialogic exchange. furthermore, to the best of our knowledge, we did not find an equivalent study to measure the actual subsequent behavior of a conversation partner after a dialogic exchange on social media platforms, forums, or wikipedia. in recent years, computational linguistics has developed computational models of dialogic text that predict the emotional responses associated with any utterance. the findings suggest that interacting speakers generally reinforce each others' point of view @xcite , use emotions to signal agreement, and mirror each other's textual cues @xcite . on the other hand, predicting behavioral responses is potentially a more challenging task for text modeling and prediction, and it is also less explored in the literature. the existing research on online turn-taking behavior has focused on modeling emotional reactions, with little interest in predicting actual behavioral change. this research is discussed in more detail in the supplementary materials 3 . for now, we contextualize the contributions of this dataset by demonstrating how it is applicable to address the following gaps in the scholarship: frontiersmen camping fellowship op. the title of this article should be "frontiersmen camping fellowship". • what are the linguistic features of editorial feedback which predict emotional change in the person that initiates the discussion (henceforth, op, original poster)? • what are the linguistic features of editorial feedback which predict subsequent editorial behavior by the op? first, we report the predictive performance on predicting emotional and editorial behavior change from the linguistic features of the comments, using regression baselines and state-of-the-art deep learning models. performance is evaluated as an f 1 score of predicted labels against the ground truth labels as implemented in scikitlearn. then, we compare the linguistic features associated with emotional change with those associated with subsequent edits. finally, we offer a diagnostic analysis of the prediction errors observed. ## the wikitalkedit dataset in this dataset, we describe how we collected our data from the wikipedia talk dataset and formulated a task around emotional and behavioral actions of an article's editors, who are taking turns in a conversation. ## dataset analysis in the following sections, we analyze what types of linguistic feedback from the editor is effective at creating a positive emotional change or an editorial action by the op. in preliminary data explorations, we found no correlation between the editor's politeness or status, and emotional or editorial change. we observed that the editor's comments that are associated with positive comments (@xmath0, mean jaccard coefficient, @xmath1.16) are significantly shorter and have less overlap (content interplay) with the op's comment than those associated with negative comments (@xmath2, mean @xmath3.18). there was no substantial difference for editorial changes. ## predicting the response to editor's feedback we examine the different linguistic features and discourse markers in predicting emotional and editorial change in the wikitalkedit dataset. our independent variables comprise the linguistic features of the editor's feedback, and the dependent variables are the op's change in emotional and editorial behavior after receiving the feedback. we used logistic regression and many deep learning implementations from the pytorch-pretrainedbert package to predict both the emotional and editorial change of the user. ## results we now examine the test-set performance of these models trained on a subset of the wikitalkedit dataset. the dataset for emotion analysis comprises the 15% of overall dataset where editorial feedback yielded a substantial positive or negative change in the emotion vector (i.e., the emotional change was above or below one standard deviation from the mean). similarly, the dataset for editorial actions (edits performed) comprises the 10% of the conversations that started within 24 hours since an op edited the page. a pairwise correlation found no relationship between emotional and editorial change (ρ= .01, p>.1). the dataset statistics are provided in ## linguistic insights based on the results in ## insights from topics we conducted a benjamini hochberg (bh)corrected pearson correlation of the topic features of comments by the editor. we visualize it as a language confusion matrix introduced in recent work @xcite to compare the topics predictive of emotional vs. editorial change of the op. the word clouds in figure 2 show the correlation of lda topics with emotional change on the x-axis, and the correlation with editorial change on the y-axis. the grey bands depict zones where the topics do not have a statistically significant correlation with either emotional or editorial change. we have distinguished the themes related to content (e.g., medicine, religion, and ethnicity) by coloring them in red. the topics in black are related to wikipedia's content guidelines (i.e., mentions of npov, sources, cite, information). 7 these themes involve the neutrality (neutral point of view, npov), general importance (notability), and verifiability (sources, evidence) of information. finally, the blue topics are meta-commentary centered around the elements in a wikipedia article (mentions of edit, page, title, section). our analysis of the wikitalkedit dataset suggests that mentions of wikipedia's guidelines are associated with a positive editorial change, but a sentiment change -false positives liwc-based classifier xlnet page talk triplet page talk triplet kiev op: i am so confused! what is the policy regarding names of towns?[...] editor: danny, there's no real policy. if there is a name which is widely used for one very famous city (e.g. paris, rome, athens), then the article with that name should be about that city. [...] op': actually, i have a problem with this. yes, paris, rome, and athens immediately bring to mind great european cities, but why should we play favorites?[...] jeff kennet op: ==massive pov/cruft should be @xmath4. this article is filled with pov and cruft. i did not think they would require community consensus but it is what it is. also, why no mention of the psycho who |ssaulted him (fergal downey)?? editor: when i looked at your edits i was just overwhelmed by the scale of them. obviously you've [...] what do you want to remove? what do you want to add? [...] op': i guess sometimes i am too bold. moors op: several days have gone by and [user] has still failed to give me a logical reason for his reversion. i changed the article back to my previous edit and made some [...] editor: in other words, you have no consensus for the changes. i'm not surprised hasn't replied, since you've got further and further from any actual comment on the content of the article. i'm not even sure [...] op': don't revert due solely to no consensus any of wikipedia's basic editing policies? this is the second time i have been reverted without a specific reason why.[...] religion in swaziland op: ==improving the @xmath5. editor: can you please explain why you changed cia statistics? op': ==reverted @xmath6, i therefore reverted it to my last edit on july 15. next, we consider the meta-commentary about page structure (page, title, move and review, section, add). we observe that these are associated with positive emotional changes (.06 < @xmath7.10, @xmath8.05), possibly because they offer concrete and minor suggestions. those meta-commentary topics which directly request an edit or a review inspire editorial change (.03< @xmath9.06, @xmath10.05). finally, topics related to the source, i.e., about wikipedia's guidelines, generate a more nuanced reaction. topics related to evidentiality (source, news, evidence) and notability (notable, articles, deletion) are the strongest predictors of negative emotion (-.18<@xmath11. finally, we could encode the latent space with information about the type of editorial feedback @xcite , which would be helpful in predicting how the op responds. ## conclusion and future applications the wikitalkedit dataset offers insights that have important implications for understanding online disagreements and better supporting the wikipedia community @xcite . we recommend the use of the wikitalkedit dataset to model the dynamics of consensus among multiple contributors. scholars can also use the wikitalkedit dataset to address issues of quality, retention, and loyalty in online communities. for instance, the insights could shed light on how new ops can be retained as sustaining wikipedia contributors @xcite . our exploratory analyses suggest that disagreements on wikipedia arise over "errors": doubts that a given entry leaves no room for improvements. but errors serve a good faith purpose on wikipedia by perpetuating participation and shared collective action @xcite . the dataset would also be useful to understand how references are debated and interpreted as objective pieces of evidence @xcite .