N w QM : A neural quality assessment framework for W ikipedia (2020)

Abstract: Millions of people irrespective of socioeconomic and demographic backgrounds, depend on Wikipedia articles everyday for keeping themselves informed regarding popular as well as obscure topics. Articles have been categorized by editors into several quality classes, which indicate their reliability as encyclopedic content. This manual designation is an onerous task because it necessitates profound knowledge about encyclopedic language, as well navigating circuitous set of wiki guidelines. In this paper we propose Neural wikipedia Quality Monitor (NwQM), a novel deep learning model which accumulates signals from several key information sources such as article text, meta data and images to obtain improved Wikipedia article representation. We present comparison of our approach against a plethora of available solutions and show 8% improvement over state-of-the-art approaches with detailed ablation studies.

## introduction wikipedia is one of the most prominent sources of free information in the world today. since reliable and advertisement free material is mostly behind pay-walled sources, a huge volume of global population is directed toward this extraordinary crowd sourced platform for information ranging from history, politics, pop-culture to even scientific topics @xcite . although wikipedia has grown significantly in terms of volume and veracity over the last decade, the quality of articles is not uniform @xcite . the quality of wikipedia articles is monitored through a rating system where each article is assigned one of several class indicators. some of the major article categories are fa, ga, b, c, start and stub. most complete * *authors contributed equally and dependable content is annotated by an fa (aka featured article) tag while lowest quality content is annotated with a stub tag. the intention behind this elaborate scheme is to notify editors regarding current state of the article and extent of effort needed for escalating to encyclopedic standards foot_0 . there exist several guidelines which direct editors in annotating articles into respective classes. some of the traits of a fa article are engaging and comprehensive prose with neutral point of view and verifiable claims. it must also rigorously follow the style manual, i.e. the page structure. further the content should be stable, i.e. devoid of edit warring. understanding compliance with these guidelines often require detailed knowledge about language usage as well as domain knowledge about wikipedia page layout and style principles. often it is nontrivial to discern qualifying differences between articles which merits their ratings without inculcating personal biases. consider the wikipage of two prominent us presidents abraham lincoln (ga) and john f. @xcite . both pages are indistinguishable in-terms of coverage and engagement, however on closer assessment it is apparent that president kennedy's page contains unattributed opinion such as the statement this crisis brought the world closer to nuclear war than at any point before or after ... in the cuban missile crisis section. it also has vague quantifiers such as some questioned, some crtics, somewhat successful etc. similarly, if we look at wikipages of historical figures @xcite and cleopatra (fa) it is difficult to discern their quality just from the content. a deep dive into the individual talk pages reveal that the former page has unresolved content issues and disputes which justifies the given rating. hence it is a difficult task to manually judge language specific nuances present in the main page text, topic level disputes manifesting in the talk pages as well as section layout and image positioning before deciding a correct rating. this is also apparent from the quality statistics foot_1 which shows that only 0.09%, 0.5% of the ∼ 6m english wikipedia articles have a fa and a ga tag respectively. current approaches for automatic quality assessment by wikimedia foundation foot_2 use handcrafted features from main article text for classifying quality class @xcite . analogous approaches exist (dang and ignat, 2016) which attempt to automatically generate features from main text using deep learning models such as doc2vec @xcite . other approaches use deep sequence models such as bil-stm @xcite as well as combining representations obtained with additional modalities like image @xcite . the principal focus of existing works have been concentrated on main article text. however one of the key sources of metadata about an article, i.e., the corresponding talk page has been ignored. talk pages contain crucial information concerning stability of a page. they also hold evidence whether discussion threads encompassing topics are decisive. besides, representation of main page text using sequence models cannot capture high level semantic signals such as whether the introduction section is a summary, whether the coverage of topics is polarized or the information is redundant or the wording is convoluted. in this paper we propose neural wikipedia quality monitor (nwqm) which integrates information from multiple sources, i.e., main page text, metadata and html rendering resulting in improved quality assessment. we use bidirectional contextual representation @xcite for encoding article text. however, contrary to document representation using bert @xcite , which is not adequate for large text documents, we first segment articles organically based on sections. we fine tune on each section text individually followed by a summarization layer which preserves the sequential nature of sections. similar representation of atomic units and summarisation is applied on talk pages. we also obtain images from the raw markup using imagekit foot_3 . these images are further embedded in a vector space using inception v3 @xcite . inception v3 is pre-trained on imagenet foot_4 and we fine tune on our dataset to cater to our task. our experiments show that combining information sources from these sources leads to improved result eclipsing current state of the art @xcite ) by 8%. our main contributions are enumerated below. ## related work automatic article assessment is one of the key research agendas of the wikimedia foundation foot_7 . one of the preliminary approaches @xcite seeking to solve this problem extracted structural features such as presence of infobox, references, level 2 headings etc. as indicators of the article quality. other approaches explored distributional representation as well as sequence models @xcite @xcite . @xcite attempted to solve this problem by formulating features capturing dynamic nature of the articles. a complementary direction of exploration has been put forward by @xcite where correlation between article quality and structural properties of co-editor network and editorarticle network has been exploited. this task can also be solved by exploring the rich literature of document classification. one of the characteristics of wikipedia articles are that these are long documents hence sequence models can suffer from (atkinson, 2018) catastrophic forgetting. @xcite proposed to solve this by leveraging a hierarchical organization of documents. further improvements have been demonstrated by employing bidirectional transformers @xcite ., class article count fa 3589 ga 5900 b 5900 c 5900 start 5900 stub 5900 total 33089 ## dataset wikimedia foundation stores all data for its multilingual wikiprojects in the form of wikidumps foot_8 . we downloaded first 100 english wikipedia dumps which are 7z archived xml files. the combined uncompressed size of these files is ∼ 8t b and it contains a sample of ∼ 6m english wikipedia text from the first version to last update as of june 2019. because of limitation of space we did not go though the entire english wikipedia archive. each uncompressed file is approximately of 80gb size and have random samples of approximately ∼ 5k wikipedia pages. we parsed each file using mediawiki xml parser foot_9 and in one linear scan we tried to locate if the main page text and talk page text is in the same dump xml file. more specifically if we encounter the main wikipedia article cleopatra, we remember it in a dictionary and seek to locate talk:cleopatra in future scans or vice versa. however it is entirely possible that talk:cleopatra is not present at all in the currently encountered xml file and may be present somewhere else; in such case we ignore that article. if we can locate both main and talk pages of the same article we save it for reference. we include in our corpus maximum number of articles for ga, b, c, start and stub while maintaining equality. for fa articles, we included entire extracted corpus, because such articles are scarce. although this process is naïve, yet we manage to extract moderately balanced number of datapoints for each class. we would further like to note that few other public datasets exits for this task. the first version is made available by the wikimedia foundation 11 which has 30k datapoints with approximately 5k pages in each of the 6 classes. @xcite has pointed out that this dataset contains many noisy datapoints, e.g., empty pages labeled as fa class. other datasets are made available by @xcite . our investigation shows that none of the former datasets contain meta pages which prompts us to extract this novel data ourselves. also some of the datasets have uniform article length across classes, which is often not the case in reality. for example dodo and grey-necked wood rail are both fa articles with very different article length. considering uniform article length may lead to overfitting. we used stratified random sample of 80% data for training and 10% each for validation and testing. the overall distribution of the articles with respective classes in our dataset is enumerated in ## proposed solution in this section we present a detailed description of our multimodal approach. we start this section by explaining the various notations that we use in the subsequent sections of the paper. we then proceed to first describe representation mechanisms of explicit signals about article quality, obtained from main text, i.e., presence of bias, claim verifiability, coherent wording, citations etc. we further elaborate on the mechanisms employed, to capture implicit quality indicators such as article stability, collaborative nature of editors obtained from article talk text as well as visual renderings of documents to capture how well the article follows the style manual. further, we describe the different ways in which we combine the information to predict the quality of the page. we present the overall architecture of our model in figure 1 . ## experiment in this section we evaluate nwqm against several existing approaches. baselines. we provide a brief outline of the competing methods in the following. ## conclusion in this paper we proposed a novel multimodal deep learning based model nwqm for quality assessment of english wikipedia articles. our model combines signals from article text, meta pages and image rendering to construct an improved document representation. we evaluate it against several existing approaches and obtain at most 8% improvement compared to the state-of-the-art method. for a 6 class classification this leap in accuracy is notable. we also perform extensive investigation of the different components of our model to understand their individual utility. we perform in-depth qualitative analysis of the obtained predictions and contrast them with the closest baseline. to the best of our knowledge this is the first work which combines several aspects of information available for wikipedia articles and, in particular, the talk page dynamics toward quality assessment. we also showcase the utility of fine tuned bidirectional transformers toward document classification especially when combined with niche platform specific signals. we believe our work opens up the necessity of further investigation pertaining to careful information fusion techniques for downstream tasks.