WIKIBIAS : Detecting Multi-Span Subjective Biases in Language (2021)

Abstract: Biases continue to be prevalent in modern text and media, especially subjective bias – a special type of bias that introduces improper attitudes or presents a statement with the presupposition of truth. To tackle the problem of detecting and further mitigating subjective bias, we introduce a manually annotated parallel corpus WIKIBIAS with more than 4,000 sentence pairs from Wikipedia edits. This corpus contains annotations towards both sentence-level bias types and token-level biased segments. We present systematic analyses of our dataset and results achieved by a set of state-of-the-art baselines in terms of three tasks: bias classification, tagging biased segments, and neutralizing biased text. We find that current models still struggle with detecting multi-span biases despite their reasonable performances, suggesting that our dataset can serve as a useful research benchmark. We also demonstrate that models trained on our dataset can generalize well to multiple domains such as news and political speeches.

## introduction people often rely on reference work like encyclopedias and textbooks to gather information, as such sources are designed to present facts fairly and objectively. yet, bias is still pervasive in these sources. for instance, the sentence "this album is arranged by many talented arrangers." is considered biased as the word talented inappropriately reflects the writer's positive opinion. as a result, methods that can automatically detect and reduce bias are in great demand, which could save human efforts and keep the quality of the reference work. in this work, we study how to detect and further mitigate biases in language. specifically, we focus on a particular type of bias, "subjective bias", in which the language is skewed towards an obvious 1 our code and data are publicly available at https:// github.com/cs329yangzhong/wikibias . however, current corpora on subjective bias detection or mitigation tasks suffer from a set of issues. first, noises from automatically collected datasets @xcite are not neglectable. a pilot study conducted by @xcite on their wikipedia neutrality corpus (wnc) demonstrated that over 5% of the revisions are not related to bias mitigation and thus wrongly labeled on the sentence level. meanwhile, existing manually annotated corpora for subjectivity often suffer from the small dataset size in @xcite or limited annotation quality: annotator agreement from @xcite falls source sentence: pre-edit (biased language) target sentence: post-edit (neutral language) it should be noted that a the nuclear-free zone act does not the nuclear-free zone act does not make building land-based make building land-based nuclear power plants illegal, and nuclear power plants illegal, and there is some business there is considerable b support for nuclear power in order support for investigating nuclear power, which could help c to c meet kyoto emissions targets. anti-americanism is a claimed a phenomenon of subvert b anti-americanism is a global a phenomenon of discriminaethnic discrimination c and overt irrational d hostility e toward f the united states. however the term post-fascist has been used to describe however , the term 'post-fascist' has been used to describe their belief a , owing to apparent b intellectual roots in the beliefs of recent national anarchists a , owing to their b neo-fascist third positionism d . at 0.124 measured by krippendorff's alpha. moreover, multiple edits are often needed when editing a subjectively biased framing into a neutral one. for instance, over 30% of wikipedia revisions for npov justification contain two or more edits in the source side and a diverse set of modification strategies are involved. existing work @xcite only focused on single word detection, presupposing a single word as the source of bias, and failing to utilize rich signals and resources of subjectively biased words or phrases as introduced in @xcite . to address these problems, we introduce a highquality manually annotated parallel corpus wikib-ias. it includes over 4,000 biased and neutralized sentence pairs, which cover both 1,525 single word and 2,068 multiple-word span annotations (building upon 53.5k non-identical word alignments with fine-grained bias types on the source sides. samples of our corpus are shown in building on wikibias, we conduct a set of comprehensive analyses to better model subjectivity bias in text via three sub-tasks: bias classification, tagging biased segments and neutralizing biased text. we found that current state-of-the-art models still struggle with detecting multi-span biases despite their reasonable performances, suggesting that our dataset can serve as a useful benchmark. we also demonstrate that models trained on our dataset can generalize well to multiple domains such as news and political speeches. ## construction of the wikibias corpus we create the new wikibias corpus by first extracting wikipedia revisions where editors provide neutral point of view (npov) foot_0 justifications @xcite @xcite to construct automatically labeled data (wikibias-auto); then manually annotating sentences with fine-grained bias types at the span-level to create clean ground truth (wikibias-manual). this is in contrast to the prior work on subjectivity that annotated only on the sentence-level @xcite @xcite . in particular, we design a two-stage human annotation methodology to handle sentences with both single-and multi-edits. we describe the details below. ## modeling subjective bias subjective biases shall be modeled differently for various applications. for instance, automatic bots of online media platforms may choose to flag and filter out biased sentences directly, for which classifying whether a sentence is biased is essential. when human editors work on an article, they might need some hints on potentially biased text snippets, as well as alternatives, where tagging biased segments or even generating a neutralized version becomes important. to this end, we propose three different tasks on top of wikibias. ## generalization to out-of-domain data to demonstrate the out-of-domain generalizability of our tagging model, we perform inferences on three out-of-domain datasets: (1) ideological books corpus (ibc) @xcite which consists of partisan books and magazine article; (2) news headlines of partisan news articles identified as biased according to mediabiasfactcheck.com; and (3) political speeches of the first and third 2020 presidency election debates between donald trump and joe biden. all three sets of corpora can be separated into two groups based on their partisan identifications (liberal/democratic vs. conservative/republican). examples of extracted spans are shown in qualitative results. we find that: (1) our tagging model can extract meaningful multi-word phrases, as well as subtle metaphor phenomena. for instance, "out of thin air" in the last row of the extracted phrases from the speeches domain cover the signature words of the speaker without in-domain knowledge. "have a plan" is prevalent in 2020's presidency debates and signature words "tremendous" and "very powerfully" of donald trump have also been captured. human evaluation. we sampled 50 sentences per corpus for human annotations. for each sentence, 3 qualified turkers were asked to pick the biased spans without length constraints. we con-sider a span receiving more than one annotator vote the gold label. the second column in detection of subjective bias. the study of detection of subjectivity can be dated back to 1990s, when pioneers start noticing the subjectivity genre on document level classification @xcite . later, works like (bruce and @xcite bring people's attention to the subjectivity on sentence level. there is a long line of research focusing on sentence classification utilizing methods based on linguistic features or handcrafted rules @xcite @xcite @xcite , then neural models @xcite @xcite . work of @xcite and @xcite on detecting biased language over singleword edit is closely related to our work, but we study the biased language on a broader scale to cover multi-word spans. ## conclusion in this work, we contribute the first manually annotated parallel corpus of over 4,000 sentence pairs for the task of subjective bias detection. this corpus covers multiple-word span annotations with fine-grained bias type on the source side and sentence level bias type. we perform the first systematic study for the detection of multi-span biased language. experiments results on three tasks: classification, tagging, and generation demonstrated the usefulness of our corpus with state-of-the-art baselines. we also conclude a set of challenges that current models struggled with. in the future, we plan to generalize our models to more domains for bias detection, mitigation, and neutralization.