Leveraging W ikipedia article evolution for promotional tone detection (2022)

Abstract: Detecting biased language is useful for a variety of applications, such as identifying hyperpartisan news sources or flagging one-sided rhetoric. In this work we introduce WikiEvolve, a dataset for document-level promotional tone detection. Unlike previously proposed datasets, WikiEvolve contains seven versions of the same article from Wikipedia, from different points in its revision history; one with promotional tone, and six without it. This allows for obtaining more precise training signal for learning models from promotional tone detection. We adapt the previously proposed gradient reversal layer framework to encode two article versions simultaneously and thus leverage this additional training signal. In our experiments, our proposed adaptation of gradient reversal improves the accuracy of four different architectures on both in-domain and out-of-domain evaluation.

## introduction maintaining a neutral point of view is a desideratum in many communication channels, e.g. news articles, scientific writing, and encyclopaedias. biased writing detection can help reduce the distribution of content which contains unfair representations of a topic. for this reason, datasets and methods have been developed to automate it. a number of studies have approached biased writing detection in the context of news media (e.g. @xcite al. (2019); chen et al. (2020); färber et al. (2020) ), primarily considering political stance and partisanship. however, biased writing also arises in other settings. in wikipedia, the online encyclopaedia, it manifests itself in the form of promotional tone which violates the cornerstone "neutral point of view" policy of the platform. the latter allows users to flag articles with such policy violations by adding tags to the article mark-up, which are retained in its edit history. leveraging ## promotional tone on wikipedia a number of studies have utilised wikipedia to develop labelled datasets for content-related issues, including promotional tone detection. wikipedia has several favourable characteristics which enable this form of data collection. firstly, articles evolve over time through different versions. secondly, the chronological revision history of each article is preserved and open-sourced foot_1 , meaning that the evolution of an article can be retrieved. finally, the platform's decentralised quality control system allows users to tag articles that violate the platform's content policies, to warn readers of such issues and to attract the attention of editors to fix them. these tags are removed from the article once the problem is resolved, but they are preserved in the article's edit history. a more details on wikipedia's policy violation tags see @xcite . in this context, a revision of an article which contains a tag is considered a positive instance of that specific policy violation. different methods have been proposed for sampling negatives. a popular approach is to find revisions of the same or other articles which do not contain the tag. however, this approach can introduce noise, as the absence of a policy violation tag from an article does not guarantee that the problem is not present. this characteristic of template-based wikipedia datasets has been noted in previous work, e.g. @xcite ; @xcite ; @xcite . another option is to look to other articles which are known to represent well-written content. for instance, @xcite and @xcite select negatives from wikipedia's list of featured and good articles. however, these articles are of a higher quality generally and therefore have other distinguishing characteristics, which may be misleading if the goal is to detect policy-violating content. additionally, sampling negatives from different articles may introduce a topical bias. ## mining promotional articles our data extraction methodology consists of (i) finding articles tagged by a wikipedia editor as having a promotional tone problem at some point in their edit history, (ii) selecting the revision where such a tag was added as a positive sample, and (iii) sampling negatives from revisions which did not contain the template. finding promotional tone tags to identify tags of interest, we refer to the wikipedia category "articles with a promotional tone" @xcite and identify the quality tags which most frequently occur in this articles of this category. these are "advert", "autobiography", "fanpov", "peacock" and "weasel". each of these tags describes a different type of promotional tone issue, for which the definitions are contained in appendix a. we then use regular expressions to collect all revisions which contain variations of these tags in the wikitext data lake @xcite . finding tag addition events once incidences of promotional tone tags have been identified, we use the wikihistory data lake @xcite ) to find the full edit histories of these articles. for each article, we then identify the point in its edit history where a tag was added, and consider this version of the article as the positive sample. we exclude cases where the tag addition edit was reverted foot_3 by another editor. the article text at this timestamp is retrieved from the wikitext data lake. ## data validation as discussed in sec. 2, wikipedia tag-based datasets are known to contain a certain level of noise. to counteract this, we have implemented three measures: ensuring that the negatives are from the same stage of article development, sampling from different points in the same article's edit history, and sampling negatives before and after the positive. however, there is still a risk of including false negatives, i.e. articles not tagged as containing promotional tone even though they do. an example of such a case from our dataset is shown in fig. 2 . despite containing non-neutral phrases such as "hit show", "made quite an impression", and "prove @xcite herself to be intelligent", the neg_pre (first) sample is not tagged as containing biased language. it does however contain some information that reflects negatively on the subject ("for the wrong reasons"). this is removed in the positive (middle) sample, and more overtly biased descriptions are added ("quick wit, educational background, amazing looks", "bubbly personality and easy on the eye appearrance"). in the neg_post negative sample, the problematic phrasing is removed. we perform manual validation of our dataset to estimate how frequently false negatives are included. we perform two types of validation: pairwise and independent prediction. for the former, the task is to rank two samples (i.e. revisions of the same article) as to which is more promotional. 40 articles, consisting of 20 positive-negative pairs in random order, are evaluated by two of the authors. version 1 17/08/2006 neg_pre version 2 18/10/2006 pos version 3 27/10/2007 neg_post cher tenbush cheryl "cher" beth tenbush made her television debut on ashton kutcher's hit show beauty and the geek. known later as "the smart beauty" cher made quite an impression on her housemates and audience. not only did cher prove herself to be intelligent, she also proved herself to be stubborn and direct, earning her much criticism. cher made her intentions to win beauty and the geek clear from the beginning, causing people to believe she was on the reality show "for the wrong reasons." she soon redeemed herself, causing many to believe she made the biggest transformation out of all the beauties. cher tenbush (born december 2, 1981) is an asian-american reality tv star, actress, and model from fremont, california usa. on the cw hit reality series, beauty and the geek, cheryl (cher) tenbush can be considered an unlikely character among her competitors. cher's quick wit, educational back-ground, and amazing looks accorded her the winning spot along with partner, josh herman, whom she nurtured from geek to stud. some would say her most prized possession from the show is not the monetary prize or the great memories, but is her found love with geek turned stud co-star, wes wilson. [...] she had no idea that she would last to be the final couple on the show, but as tv guide e! proclaimed "a beauty with brains." cher tenbush is a reality tv star, actress, and model from fremont, california, united states. tenbush was on the second installment of the cw reality series, beauty and the geek, in which she was partnered with josh herman. [...] tv guide and e! proclaimed her "a beauty with brains." tenbush dated wes wilson, while on beauty and the geek. the orderings of the annotators agreed with the assigned labels for respectively 16/20 and 14/20 pairs, with a cohen's kappa score of 0.79 indicating substantial agreement. this suggests that the collected data contains a trustworthy signal for comparing the extent to which two texts are promotional. since the task we are mainly interested in is text classification, rather than ranking, we also perform an evaluation on individual samples, annotating 30 samples of each type (positive, neg_pre and neg_post). the concurrence with the mined labels of the neg_pre and neg_post annotations are shown in our evaluation indicates that the negative samples from before the tag was added contain more noise, compared to those sampled after it was removed. this can be attributed to the active removal of the tag by an editor in the version after the tag was added (neg_post), which indicates that the problem is resolved, while the lack of a policy violation tag in earlier versions (neg_pre) does not guarantee lack of promotional tone. however, ignoring the neg_pre samples altogether would expose the temporal bias mentioned in sec. 1: if negatives are always sampled chronologically after positives and from a more developed version of the article, spurious correlations may be inferred. based on these insights, we have chosen to include the automatically mined neg_pre samples in training, but to create a separate set of manually validated neg_pre samples for evaluation. thus, we randomly selected 100 neg_pre samples from the original test set and verified whether they represent a neutral writing style. 42 of the 100 samples were confirmed as true negatives. we balance these negatives with their corresponding positive samples, and refer to this dataset as validnegpre. ## gradient reversal training for promotional tone detection gradient reversal training @xcite jointly optimises two classifiers which rely on a shared underlying encoder model: (i) a label predictor for the main task, which predicts class labels and is used during both training and test time, and (ii) a domain classifier, which predicts either the source or the target domain during training as the auxiliary task. the parameters of the encoder model are optimised to minimise the loss of the main task classifier while maximising the loss of the domain classifier. this is achieved through a gradient reversal layer, which leaves the input unchanged during forward propagation and reverses the gradient by multiplying it by a negative scalar during the backpropagation. this approach is motivated by theory on domain adaptation, which suggests that a good representation for cross-domain transfer is one for which an algorithm cannot learn to identify the domain of origin of the input observation @xcite . primary branch our adaptation of this framework, shown in fig. 3 , differs from @xcite in that it considers two text inputs concurrently (x and x ), as opposed to one. f represents a neural network encoder with parameters θ 1 . f encodes the two texts independently, to produce z and z : the network then splits into two branches. the primary (bottom) branch consists of a neural network model g, with parameters θ 3 , which produces promotional tone predictions ŷt 1 and ŷt 2 for the two samples: the auxiliary branch concatenates the two input encodings as [z, z ], and then the similarity classifier h, a neural network parameterised by θ 2 , provides a prediction ŷsim of whether the two samples originate from the same wikipedia article: our intention with this task is to encourage the encoder f to learn features that are topic agnostic. this should allow for better generalisation across datasets, as well as to avoid learning spurious correlations due to topical biases in the data. the encoder and classifier models are trained simultaneously. given a set of training samples @xmath0, ..., x n , y 1 , ..., y n ], we construct m pairs with indices @xmath1, j) : i, @xmath2, ..., n ]. the process for generating these pairs is described in sec. 6. then, the loss is given by: during testing, only the feature encoder and the main task branch are retained to perform the tone classification task: recall that the model encodes and predicts on the two input samples independently during training. we can therefore obtain predictions for individual test samples (rather than pairs), as is the more general case in other models and datasets for this task. the two classifier models, g and h, are both mlp models. the feature encoder (indicated as f in fig. 3 ) is responsible for producing an embedding of an article to be used in both the main and auxiliary task. we evaluate four options: glove embeddings @xcite for every word in the article are averaged, followed by an mlp model, • hierarchical attention network (han) @xcite : word embeddings are processed using an lstm layer followed by an attention mechanism to build up sentence embeddings. sentence embeddings are similarly combined to form an article embedding. we compare the grl training approach with the standard method of training the classifier with each feature extractor model. this is equivalent to training with the inference model in equation 5 ; the auxiliary branch is removed and one sample is processed at a time. implementation details are provided in app. b. ## results the results from our evaluation on the fulltest are in ## conclusion in this work, we have proposed an alternative data collection method and dataset for promotional tone detection, which leverages the evolution of articles on the platform. to utilise the additional structure in our dataset, we extended the gradient reversal framework to train models which are more effective at detecting promotional tone. this was shown both on our own test set and on a test set from a different domain. we further provided insights on the effects of two negative sampling strategies on wikipedia. these findings should be useful for researchers who use wikipedia-based data more broadly, in addition to those who work on biased language detection.