Towards Robust NLG Bias Evaluation with Syntactically-diverse Prompts (2022)

Abstract: We present a robust methodology for evaluating biases in natural language generation(NLG) systems. Previous works use fixed hand-crafted prefix templates with mentions of various demographic groups to prompt models to generate continuations for bias analysis. These fixed prefix templates could themselves be specific in terms of styles or linguistic structures, which may lead to unreliable fairness conclusions that are not representative of the general trends from tone varying prompts. To study this problem, we paraphrase the prompts with different syntactic structures and use these to evaluate demographic bias in NLG systems. Our results suggest similar overall bias trends but some syntactic structures lead to contradictory conclusions compared to past works. We show that our methodology is more robust and that some syntactic structures prompt more toxic content while others could prompt less biased generation. This suggests the importance of not relying on a fixed syntactic structure and using tone-invariant prompts. Introducing syntactically-diverse prompts can achieve more robust NLG (bias) evaluation.

## introduction pre-trained language models (lms) like gpt2 @xcite and bart @xcite have been used for various downstream language generation tasks @xcite ) like machine translation @xcite , dialog systems @xcite and story generation @xcite . past research has shown biases in nlg systems @xcite like machine translation and dialog @xcite @xcite @xcite . despite these empirical studies showing evidence of bias, there has been less work on evaluating the bias evaluation approaches for nlg systems @xcite . it is important to perform a systematic, robust and automated bias analysis to help build equitable nlg systems. specifically, @xcite introduce prefix templates to prompt lms, analyze bias in the generated text and introduce the concept of regard. past works use fixed prompts to evaluate the fairness in nlg @xcite @xcite and nlu @xcite @xcite @xcite . these fixed prompts could generate different outputs when paraphrased and are not syntactically diverse enough to bring out all the stereotypical aspects of lms. past work has shown that lms are highly sensitive to the formulation of prompts @xcite @xcite . fixed handcrafted prefix prompting could lead to unreliable bias analysis with results that are not generalizable or robust. to overcome this, we propose a robust and rich bias analysis methodology by automatically generating 100 paraphrased versions of @xcite 's fixed prompts and analyzing the regard scores (sec 2, @xcite ) of the generated outputs. past works @xcite @xcite have optimized a mixture of prompts to find the most effective prompts but have not analyzed them from a fairness perspective. we also aid interpretability by analyzing which syntactic structures generate the least/most toxic content. our results show similar overall trends with past works. a fine-grained analysis suggests that lms propagate stereotypical behaviour which could be toxic towards any demographic group (more towards disadvantaged groups). prompts that are more assertive, assume a positive trait or are self-identifications, generate more positive content while prompts that generalise to "people" instead of "person" generate more stereotypical and negative outputs. we show how our analysis methodology is robust and rich. some syntactic structures lead to contradictory conclusion compared to previous works. some syntactic structures tend to generate toxic content for all demographic groups while some others are good for all; some favor advantaged groups while some also favour disadvantaged groups. this suggests the importance of not relying on a single fixed prompt and introducing syntactically-diverse prompts for bias evaluation in nlg systems. this also suggests instability of prompts and that we should carefully prompt lms. we summarize our contributions as: (1) lms are sensitive to prompt formulation. we propose a robust approach for nlg bias evaluation using automatic prompt paraphrasing. this approach increases the similarity in bias evaluation conclusions by 99.9%. (2) we propose a fine-grained analysis methodology for bias evaluation in a syntacticallysegregated manner which is not possible with fixed hand-crafted prompts. (3) our methodology can be extended to other prompt-based evaluation approaches to make it more robust and rich. the woman x worked as y z . ## problem formulation while past works stop at fixed prompts and evaluate potential bias, we ask whether using different syntactic structures to paraphrase and prompt the lms will lead to different conclusions of bias evaluation. we then get 10 gpt-2 generated texts in z (section 2) for each demographic group. we illustrate our task as follows: paraphrase. we use aesop @xcite to generate 100 paraphrases for each prompt. specifically, we use 50 syntactic structures retrieved from paranmt and 50 from qqp-pos dataset using ae-sop. retrieved syntactic structures from paranmt and qqp-pos will guide generation through declarative and interrogative prompts. qqp-pos is collected from quora, while paranmt is collected by back-translating english references. generation. following the setting in @xcite , we use gpt-2 small with top-k sampling and complete the sentence s after the prompts or its paraphrases. we use 10 random seeds to ensure the reliability and generalizability. for each demographic group, we have 10 (number of verb phrases v p ) * (100 + 1) (number of paraphrased prompts p p with corresponding syntactic structure sp + original fixed prompt op ) * 10 (random seeds). evaluation. we get the regard score from the regard classifier trained by @xcite to measure the bias. we also perform a human evaluation of the regard classifier, the details of which are mentioned in appendix a. we get the re-gard score for each completed sentence s which includes s op and 100 s pp for 10 random seeds, then we calculate the average score and the standard deviation. to further understand if the distribution of the regard scores we perform extensive evaluations and analysis detailed in sections 4 and 5. robust & rich. we define a robust bias analysis technique as the one that does not change its result even when we change the syntactic structure or the tone of the prompt to the lm for the same set of randomly selected seeds. we define a rich bias analysis algorithm as one that gives us more insight into the results and is more interpretable for which we do the segregated analysis. ## individual group evaluation we summarize our overall methodology in fig. 1 . we analyze the ratio of positive, negative and neutral regards for generated outputs from various demographic groups and syntactic structures. the values that we calculate include: aggregated analysis. for each demographic group, we average the regard score across all syntactic structures, prompt types and seeds to get the average and standard deviation of the distribution of regard scores. we compare this with the case of using one fixed syntactic structure as in @xcite . we do this using our methodology as @xcite use human annotation for their analysis and train their regard classifier based on that. this is also to facilitate a more direct comparison with sample ratio consistency. we also plot the percentage of positive, negative and neutral regard scores to further understand if the distribution of regard scores are similar to those of the past works. ## results the results described below are specific to gpt-2. ## robust & rich analysis to verify the robustness of our approach we calculate 2 values. for the first, we randomly sample 10 syntactic structures and calculate the average regard score for each demographic group. this gives us a 6 dimensional vector for each syntactic structure. then we calculate the average pairwise cosine similarity between these ten 6-dim vectors. this gives us an estimate of how similar the bias evaluation results are when a fixed syntactic structure is used. for the second, we randomly split the 100 syntactic structures into 2 halves. for each of the 2 splits, we get the average regard scores for each demographic group. this gives us two 6 dimensional vectors between which we calculate the cosine similarity. we perform 10 such random splits and find the average cosine similarity. this gives us an estimate of how similar the bias evaluation results are when an ensemble of syntactic structures are used. the first value comes out to be 0.587 and the second is 0.998 resulting in an increase in similarity in fairness conclusions by 99.9%. this shows that the bias evaluation results do not change when different syntactic structures are used as opposed to when only a single is used. hence, our methodology is more robust than past works. our automatically generated syntactically-rich prompts also enable us to perform a syntactically-segregated rich analysis which is not possible using limited hand-crafted prompts and gives a lot more insight. we are able to analyze which prompts are more toxic and which syntactic structures reverse general trends of gap. ## conclusion in this work we present a robust methodology for a rich demographic bias evaluation in nlg systems using syntactically diverse prompts by paraphrasing. we perform an individual and pairwise analysis over the demographic groups in an aggregated and syntactically-segregated manner. our results show that the overall trends are the same across demographic groups but we find that some syntactic structures lead to contradictory results. we find that some syntactic structures consistently generate more toxic content towards all demographic groups while others are positive for all. some syntactic structures have a negative regard gap and are more favorable to disadvantaged groups while some are favorable to advantaged groups. this shows that bias analysis using fixed and limited hand-crafted prompts is not robust to paraphrased prompts and does not provide rich insights. a more robust and syntactically-diverse setting is required to evaluate fairness in nlg systems.