Quantifying Bias from Decoding Techniques in Natural Language Generation (2022)

Abstract: Natural language generation (NLG) models can propagate social bias towards particular demography. Though several studies investigated bias from data and model, NLG task distinctively uses stochastic decoder that can positively or negatively impact the bias-sensitive tokens initially predicted by the model. To address this gap in research, we present an extensive analysis of bias from decoding techniques for open-domain language generation considering the entire decoding space. We analyze to what extent bias metrics like toxicity and sentiment are impacted by the individual components of decoder algorithms. To this extent, we also analyze the trade-off between bias scores and human-annotated generation quality throughout the decoder space. Together, these methods reveal the imperative of testing inference time bias and provide evidence on the usefulness of inspecting the entire decoding spectrum.

## introduction natural language generation (nlg) techniques provide the backbone for many downstream artificial intelligence applications, such as chat-bots, virtual assistance, machine translation, automatic storytelling, text summarization, and writing assistants. with the advancement of deep learning, nlg tasks are commonly powered by auto-regressive language models like gpt-2 @xcite , @xcite , t5 @xcite , or gpt-neo @xcite . however, language models (lms) pretrained on large web text corpora are also known to pass on stereotypical associations learned from real-world training data. such disproportionate generations that produce representational or allocational harms towards a particular group is called "bias" in the context of ai fairness @xcite . although a moderate amount of studies has been conducted on quantifying bias for natural language understanding (nlu) @xcite @xcite @xcite @xcite , exploring the same for nlg is a nascent, yet active area of research. indeed, bias can be introduced at various phases of the model's development and deployment pipeline, such as data, modeling, decoding, evaluation. much of the work on analyzing bias in nlg focuses on benchmarking biases pertaining to models or training data @xcite @xcite @xcite @xcite @xcite . yet, up to now work on examining biases from decoder techniques is relatively scarce. however, nlg models distinctively use search, random sampling, entropy (softmax penalty) that changes the distribution of model predicted tokens at each autoregressive time-step. redistributing the predicted token and inducing randomness during inference can positively or negatively impact the bias-sensitive tokens initially predicted by the model. bias-sensitive tokens are words with negative connotations towards specific demographics as explained by @xcite . this redistribution of the predicted bias sensitive tokens solely due the randomness induced by the decoding algorithm is called as inference time bias. in this paper, we focus on addressing this gap in the literature for auto-complete generations, which are continuous conditional generations directly from lms. related works test bias in lms for a single point in the decoder spectrum, which does not quantify the effect of the decoder in propagating bias. in contrast we investigate the bias variation induced by the decoding algorithms for the full spectrum of decoder space 1 . we perform tests for six state-of-the-art lms, with diverse decoding setup and bias objectives like sentiment and toxicity. to the best of our knowledge, this is the first comprehensive analysis in this regard. we observed entropy and nucleus sampling impacts absolute bias scores across the decoder space while top-k and beam search is agnostic. this along with our experimental findings, we demonstrate why inspecting bias for the full decoder spectrum is imperative. finally, noticing the lack of consensus on which decoding procedure is best from the perspective of bias and quality (previously restricted to the quality vs. diversity @xcite , we also study the trade-off between quality and bias throughout the decoding space using human evaluation. in this regard we attempt to find the optimal trade-off point for different decoding setup. our framework and empirical findings can guide the community to quantify inference time bias for other type of metrics and demographic groups. we share the code associated with this work at github. foot_1 ## related work in the domain of continuous auto-complete generation, bias analysis mostly focuses on probing the models with curated prompts containing the demographic information and then quantifying the generation with some metric. @xcite and @xcite both used this setup. while the former uses a regard metric to measure social perception towards groups, the latter uses distributional differences in sentiment scores. @xcite curated prompts to test biased towards named entities given a name. groenwold et al. (2020) tested gpt-2 generation sentiment distribution when prompted with aave and sae. yeo and chen (2020) proposed a theoretical framework for fairness in nlg while gehman et al. (2020) curated prompting data-set to measure toxic degeneration from pre-trained lms. sheng et al. (2020) also showed that adversarial triggers @xcite can be used to further induce bias in pretrained lms. @xcite extricated the beginnings of wikipedia articles containing demographic mention to collect the bold dataset and used state-of-the-art metric to evaluate bias in generated text. other works anchors around proposing novel metrics to quantify bias towards a primary attribute or secondary dimension @xcite @xcite . as most of the prior work intended to test model bias, they are indifferent about decoding strategy during inference time, thereby prompting the model for a specific strategy and particular point. closely related to our work was a study done by @xcite that compared change in regard score and gendered word co-occurrence for gpt, gpt-2, xlnet generations with decoders but for a single point in the decoder spectrum (which does not quantify the impact of particular decoding strategy). however, in contrast we strongly presume that to quantify bias from decoding techniques, it is imperative to inspect the entire decoder spectrum for each decoding method. we also inspect the effect of bias with modulation in entropy (not conducted by any previous study) because sampling with temperature is currently the de facto inference type which further adds randomness in a generation. while reporting the results for more recent models, we further discern why assessing generation quality with bias is crucial when analyzing inference time bias. ## nlg decoding given a sequence of tokens as context, the task of auto-complete generation is to generate text that forms a legible continuation from the given context. formally, when prompted with a sequence of m tokens x 1 ...x m the model computes p (x 1:m+n ) = m+n @xmath0...x i-1 ) to generate the next n completions x m+1 ...x m+n autoregressively using a particular decoding strategy. one popular decoder is top-k sampling @xcite @xcite . given a distribution p (x|x 1:i-1 ), top-k vocabulary v (k) ⊂ v is defined as a set of size k that maximizes xϵv (k) p (x|x 1:i-1 ). at each time-step the next token is randomly sampled from top-k. @xcite introduced nucleus sampling that exploits the shape of the probability distribution to select the set of tokens to be sampled from. formally, given a distribution p (x|x 1:i-1 ), top-p vocabulary v (p) ⊂ v is defined as the smallest set such that xϵv (p) p (x|x 1:i-1 ) ≥ p. at each time-step random sampling is done from the highest probability tokens whose cumulative probability mass exceeds the pre-chosen threshold @xmath1, 1). typically, temperature-controlled sampling techniques are used where before sampling, temperature @xmath2, 1) is use to control the shape of the distribution (controlling entropy) @xcite @xcite . like for greedy distribution: @xmath3, for flat distribution: @xmath4. formally, before sampling given a temperature @xmath5, the probability that the model would predict the i th token is given by (softmax re-estimation): in this context, we take temperature t as the set containing all the temperature points to be inspected between [0, 1) and sampling parameter s as the set containing all the sampling controllable parameter points to be inspected. we define decoder space d st for a sampling technique as: ## method and metrics we document our evaluation methods as suggested by @xcite , predominantly stressing the details regarding bias measures and metrics. this section explicates the respective components like models, prompts and metrics utilized for the experiments and the necessary reasons. ## experiment and evaluation we use 10 prompts (sec 4.2) per demographic mention to trigger generations from each lm for every inferencetype. in section 5.1 we analyze the effect in bias rating of the lm generations when we sweep through d st for a specific decoder type. from here on, by d st we imply d p t or d kt . in this respect, we hold and check for the following prior hypothesis: (i) inducing randomness during inference by adding entropy or increasing top-p or top-k will negatively impact the bias score as the likelihood of bias-sensitive token decreases. (ii) model size and demographics can have an auxiliary effect on the change in bias score because the training data is the main contributor to bias @xcite and the models tend to amplify such training data bias @xcite @xcite . in section 5.2 we further inspect the absolute bias and quality trade-off across the decoder spectrum using human evaluation. for generations from gpt-3, we used openai's api and huggingface foot_7 library for other models. the gpt-3 api only supports nucleus sampling. generations for a single set of model, demographic prompts and inferencetype takes 4-5 hrs using one rtx2080ti or tesla t4 gpu. ## conclusion this paper proposes a framework for credibly evaluating language generation bias resulting from decoding algorithms. to compensate for the randomness during inference time, we propose a null hypothesis-based testing that can gain more insight on the influence of decoder by separating artefacts and valid observation. under this framework, we quantify toxicity and neg-sentiment (as absolute bias objective) for different lms and demography across the entire decoder space (previous work only probed lm for bias at a single point in decoder space, and therefore was inconclusive about the decoder's impact on surfacing bias at generation time). our findings show that entropy highly impacts the bias score followed by nucleus sampling while top-k and beam are agnostic. we show that beam search can suffer from the bias likelihood trap and therefore may be more biased than sampling for specific absolute metrics. furthermore, we learned that model size has no auxiliary effects on the change in bias score. we also highlight findings (cf. sec 5.1.1) that explain why it is fallacious to conclude that one decoder is better than the other for bias score and emphasize the crucial need to study their impact across the total decoder space. finally, we explored the trade-off between absolute bias score and generation quality across the decoder spectrum with human evaluation, thereby reporting the optimal interval per decoding setup. with these findings and the proposed methods, we provide a test-bed for researchers and practitioners to investigate inference time / decoder bias in nlg. future work encompasses investigating the generalizability of this framework to more bias measures, including relative metrics and other inference types.