Differential Bias: On the Perceptibility of Stance Imbalance in Argumentation (2022)

Abstract: Most research on natural language processing treats bias as an absolute concept: Based on a (probably complex) algorithmic analysis, a sentence, an article, or a text is classified as biased or not. Given the fact that for humans the question of whether a text is biased can be difficult to answer or is answered contradictory, we ask whether an “absolute bias classification” is a promising goal at all. We see the problem not in the complexity of interpreting language phenomena but in the diversity of sociocultural backgrounds of the readers, which cannot be handled uniformly: To decide whether a text has crossed the proverbial line between non-biased and biased is subjective. By asking “Is text X more [less, equally] biased than text Y?” we propose to analyze a simpler problem, which, by its construction, is rather independent of standpoints, views, or sociocultural aspects. In such a model, bias becomes a preference relation that induces a partial ordering from least biased to most biased texts without requiring a decision on where to draw the line. A prerequisite for this kind of bias model is the ability of humans to perceive relative bias differences in the first place. In our research, we selected a specific type of bias in argumentation, the stance bias, and designed a crowdsourcing study showing that differences in stance bias are perceptible when (light) support is provided through training or visual aid.

## introduction bias is a multifaceted phenomenon that can be wittingly or unwittingly introduced into language. @xcite traces the term's history with argumentation and concludes that bias implies onesidedness. @xcite distinguishes two forms of biased argumentation, namely the exclusion of arguments of the pro or con position ("stance bias") and the exclusion of arguments discussing a certain aspect (frame) that is relevant to an issue. figure 1 illustrates the spectrum of stance one-sided balanced potential partisan decision boundaries pro con pro con "more one-sided" figure 1 : stance bias induces a partial ordering of argumentations, the extremes being purely one-sided (left) or perfectly balanced (right). where supporters of one side or the other draw the line between biased and nonbiased depends on their degree of conviction. ## related work after a brief review of research on bias in natural language processing (nlp), we survey bias and one-sidedness in argumentation, and the application of pairwise judgments in corpus construction. ## measuring differential bias perception to measure the perceptibility of differential stance bias in argumentation in a controlled environment, we develop a basic experiment as shown in figure 2. for two argumentations, x and y , on the same topic, the central question to participants is: "which of the two argumentations is more biased?" with a corpus of topic-labeled arguments at hand, the construction of pairs of argumentations for a given topic is straightforward. first, the number of pro arguments for x and y is selected, each ranging from zero to four arguments. a corresponding number of pro arguments are randomly selected and randomly assigned to the available slots. then, the remaining slots are randomly filled with random con arguments on the same topic. the two argumentations are shown side by side to be read one after the other and then compared. no 1:1 correspondence is to be assumed between arguments at the i-th position, @xmath0, . . . , 4, and therefore arguments are not aligned across argumentations. alternative experimental setups were explored as part of a pilot study to find the simplest ("atomic") setup to investigate our experiment variables of interest. in the following, an overview these variables is given (@xmath1, @xmath2, @xmath3, @xmath4: figure 3 : two tasks organize the experiments in decreasing and increasing order of difficulty, respectively. d perceptibility of differential stance bias i 1 differential stance bias: 3 levels i 2 difficulty: low, medium, high i 3 participant training: autodidact vs. trained i 4 participant expertise: stance labeling accuracy i 5 participant confidence: 3-level self-assessment c 1 argumentation length: 4 arguments each c 2 argument length: 2-3 sentences each c 3 argument frames: all arguments from 1 frame c 4 order effects i: random order of arguments c 5 order effects ii: no textual coherence c 6 opinion diversity: 9 participants per topic u 1 stance perceptibility: e.g., explicit vs. implicit u 2 other language biases: e.g., at the lexical level key to measuring d, the perceptibility of differential stance bias, is a model for independent variable i 1 : let x denote an argumentation on a given topic that argues pro or con a proposition, where @xmath5, so that x + ∩ x -= ∅. the stance balance δ between pro and con arguments in x is measured as the absolute size difference between x + and x -: the closer δ is to 0, the more balanced the two sides are in x. given a second argumentation y , |x| = |y |, the differential stance bias can then be quantified as the absolute difference between the stance balance scores of x and y : the closer ∆ is to 0, the less strong the differential bias is between x and y . given the designated argumentation length of |x| = |y | = 4, the image of both δ and ∆ is {0, 2, 4}. consider the "tidal energy" example in figure 2 : argumentation x is an extreme case with only con arguments, argumentation y is the balanced case with two pro (first, fourth) and two con arguments (second, third). this yields δ(x) = 4 and δ(y ) = 0 and thus ∆(x, y ) = 4, the maximum possible differential bias in this setting. it follows that x is more biased than y . suppose y consists of only pro arguments instead (δ(y ) = 4) : although x and y argue exclusively on either the pro or the con side, both argumentations are equally imbalanced, so that ∆(x, y ) = 0. the intermediate case of ∆(x, y ) = 2 results when y includes three pro and a single con argument or vice versa. regarding independent variable i 2 (see figure 3 ), high difficulty means no indication is given about argument stance or stance balance before asking which of the two argumentations is more biased. medium difficulty means we first ask which argumentation contains more pro arguments and which contains more con arguments before asking to select the more biased one-a subtle hint at stance balance. low difficulty means the stance of each argument has to be labeled first as pro, neutral, con, or unknown, coloring each argument according to the chosen label. in this way, stance is strongly emphasized when answering the question which argument is more biased. experiments are presented in either decreasing (task 1) or increasing (task 2) order of difficulty (i 3 ; see figure 3 ). task 2 is also preceded by an explanation of the importance of stance balance in relation to bias, as well as an admission test for argument stance labeling that must be successfully completed. note the different learning experiences participants have as they move through the two tasks. two groups of participants are distinguished (i 4 ): those who succeed in stance label-ing and those who get it wrong more than once. also, participants are asked to self-assess their confidence (i 5 ). the variables c 1 -c 6 are tightly controlled by setting a specific value or by introducing diversity through randomization. text lengths (c 1 and c 2 ) are kept as short as possible, and each argument has a similar length to the others, so that confounding by length differences is avoided.c 3 specifically avoids mixing up stance bias with van laar's second type of one-sidedness in argumentation with respect to topic frames. c 5 is due to the requirement to generate argumentative texts rather than reusing existing ones, although the texts are generally not extremely incoherent either. variables beyond our control include the difficulty of distinguishing pro and con stances in the argument corpus (some arguments are more subtly pro or con than others; u 1 ), and (social) language biases that may be due to word choice (u 2 ). nevertheless, we manually checked for hate speech and removed corresponding arguments, as well as analyzed for biased terminology using dictionaries. ## crowdsourcing study our crowdsourcing study implements the assessment of the perceptibility of differential stance bias on amazon mechanical turk (mturk). we give an overview of the task design and its instantiation using an argument corpus. ## results and analysis this section reports workers' perception accuracy for differential stance bias with respect to the independent variables discussed in section 3. ## conclusions and future work how one-sided can an argumentation be without this one-sidedness going unnoticed? in political debates, parties often accuse each other of onesidedness. an advanced debating skill hence is to subtly shift the balance of an argumentation so that it goes unnoticed, especially by potential voters. but political debate is perhaps only one of the most extreme cases in which skewed argumentation can be found, as @xcite point out in their in-depth analysis of letters to the editor regarding the topic of political asylum. we believe that a perfectly balanced argument is the exception rather than the rule because, after all, the goal is to convince the audience of one's own opinion. for this reason, audiences must try to be alert to (subtle) manipulation opportunities, and here formal training to detect differences in stance bias seems a promising future direction. the results of our study indicate that differences in stance bias are perceptible, and even more so when (as in our crowdsourcing setting) workers are trained with a visual aid in the form of green/red signals that emphasize the stance of the arguments. in an online setting where arguments are read at leisure, a tool that highlights the distribution of stances could help readers make more informed decisions, not to mention formal training in the context of a debate. our study further demonstrates, to our knowledge for the first time, the viability of assessing bias via relative judgments rather than absolute ones. in this regard, we focus on stance bias for its ease of operationalization. it is likely that many other types of bias can be similarly addressed in future work, although not all of them can be operationalized as easily as in our study. in any case, the obvious ben-efits of reducing the subjective bias of annotators certainly justify further investigation and the development of a theory in this direction. this should, of course, include research on the interactions of biases, since not all biases can be easily controlled. for instance, some workers based their judgments not only on differences in stance, but also pointed to other language properties that, in their opinion, made one argumentation seem more one-sided than another. although we took great care to exclude examples of language bias, subtle biases may still have been present in our sample. a straightforward and fascinating application of our differential bias model is to extend it to the study of frame bias.