Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting (2023)

Abstract: A crucial challenge for generative large language models (LLMs) is diversity: when a user’s prompt is under-specified, models may follow implicit assumptions while generating a response, which may result in homogenization of the responses, as well as certain demographic groups being under-represented or even erased from the generated responses. In this paper, we formalize the problem diversity of representation in LLM generations. We present evaluation datasets and propose metrics to measure diversity in generated responses along people and culture axes. We find that LLMs understand the notion of diversity, and that they can reason and critique their own responses for that goal. This finding motivated a new prompting technique called collective-critique and self-voting (CCSV) to self-improve people diversity of LLMs by tapping into its diversity reasoning capabilities, without relying on handcrafted examples or prompt tuning. Extensive empirical experiments with both human and automated evaluations show that our proposed approach is effective at improving people and culture diversity, and outperforms all baseline methods by a large margin.

## introduction large language models (llms) such as @xcite and palm @xcite have demonstrated impressive capabilities on a variety of tasks, and there is a growing trend for llms to serve as foundation blocks for ai systems. however, these models are known to display unintended behavior such as generating biased or toxic text @xcite @xcite or perpetuating stereotypes @xcite . while these previous works on fairness and bias in llms look through the lens of biases in "how" various sensitive groups are represented (e.g., stereotypes), we focus on the relatively understudied class of fairness and inclusion concern in llms caused due to lack of diversity of representation of various demographic groups in the model responses. consider for example the user prompt "can you recommend a few ceos to follow?" in fig. 1 . a 540b-parameter flan-palm @xcite ) language model's baseline response to the user prompt is highly homogeneous with only white male ceos. such a homogenization @xcite poses concerns for using llms in downstream applications from a responsibility perspective, much like the diversity and inclusion concerns in recommendation @xcite , ranking @xcite and image search @xcite . we aim to both quantify and improve diversity of representation in a language model's response. to this end, we present two evaluation datasets and propose metrics for measuring people-diversity in the generated output of llms. we find that the baseline flan-palm model has very low diversity scores close to 0.0 with ∼ 99% of responses belonging to the same gender on average and ∼ 98% of responses belonging to the same ethnicity. for our mitigation design, we seek inspiration from the recent line of work @xcite @xcite @xcite , which shows that in-context reasoning, self-critique and revision are powerful paradigms that can be used to improve model responses on a variety of tasks. we build on this and propose a new technique called collective-critique and self-voting (ccsv). summarizing our contributions: ## background & related work measuring fairness and bias in llms. prior work on studying and measuring biases in llm generation largely focuses on potential negative representations in the form of perpetuating stereotypes @xcite , generating toxic content @xcite @xcite or misrepresentation @xcite . in the text-to-image generative model space, there are works on measuring biases due to lack of diverse representation in image search @xcite and text-to-image generative models. @xcite . we are not aware of any prior works on improving diversity of representation in open-ended llm generations. there are also several benchmarks proposed to measure models' biases on a wide range of downstream tasks like stereotype via question answering (qa) @xcite , gender bias via co-reference resolution task @xcite , stereotypes @xcite , and toxicity @xcite . however, these benchmarks do not extend to evaluation on openended response generation tasks, and they do not cover this new class of llm harms that occur due to lack of diversity of representation of demographic groups in the model's generated responses. our work fills this gap by proposing an evaluation dataset for measuring the people and cultural diversity in llm generated responses. in-context prompting and reasoning. recently, llms have demonstrated remarkable success across a range of reasoning tasks, merely via fewshot prompting with exemplars and instructions, without requiring any additional training data or modeling changes. chain-of-thought (cot) @xcite shows that simply adding a few chainof-thought demonstration as few-shot exemplars in prompting improves models ability to perform multi-step reasoning on arithmetic tasks. in a follow up work, self-consistency @xcite showed that model performance on arithmetic reasoning tasks can be further improved by first sampling multiple responses from the model to invoke multiple reasoning paths, and then aggregating them by taking their majority vote. as the selfconsistency was designed for arithmetic tasks it expects the final answer to be from a finite answer set, and does not extend to open-ended text generation. the self-voting step in our proposed ccsv method is conceptually similar to the self-consistency idea, and can be seen as an extension of self-consistency idea to open-ended text generation. @xcite show that llms are zero-shot reasoners and can perform multi-step reasoning via the prompt "let's think step by step." our work can be seen as an extension of incontext prompting methods to the problem of diversity and inclusion. however, merely taking techniques designed for mathematical reasoning and applying them at face value to responsible ai problems is unlikely to work @xcite , or might even be detrimental as observed in @xcite . @xcite investigated effectiveness of natural language instructions to mitigate stereotypes and find that merely instruction prompting is insufficient to improve model behavior. this finding is inline with our results in this paper, where we observe that the model responds surprisingly little to diversity instructions. @xcite investigated applying zero-shot cot @xcite to a variety of stereotype benchmarks and observe that cot prompting increases the stereotype biases. we observed similar results on our diversity evaluations on cot in this paper. from a method perspective, our work is closest to constitutional ai (cai) @xcite , which also proposes a self-critique and revision approach, but in the context of ai safety. while their approach was not designed for diversity, we extend their method and compare it as baseline. we further show how our proposed idea of collective-critiques and self-voting can be applied to cai method by simply replacing their decoding strategy to achieve substantially improvements in the diversity. ## evaluation design we now offer and describe multiple complementary designs to evaluate diversity, including new datasets, entropy to measure the full distribution of representation, max-gap to measure the extremes of over-/under-representation, and human-rater evaluation to capture human perception of diversity. ## experiments methods. we compare the following in-context prompting based interventions. zero-shot methods. first, we take a test query from the evaluation dataset, and frame it as a dialogue between a "user" and an "ai model" and prompt the model to respond to the formatted query. we call this the 0-shot standard prompting, and use this as our baseline in all experiments. recent works have shown that llms are zero-shot instruction followers (if). we adapt this idea to the diversity task, and experiment with a variant wherein we add the instruction prompt "instruction: write ai model's response to the user question such that it has diversity," referred to as 0-shot if. our 0shot cot experiment is a variant of zero-shot cot @xcite , which adds "let's think step by step" at the end of the 0-shot prompt. few-shot methods. we also experiment with various standard 5-shot prompting methods with user query and ai model response pairs. additionally, we compare chain-of-thought (cot) prompting @xcite , a variant of few-shot prompting wherein an expert written step-by-step reasoning "thought" is added before the example response. we also experiment with an in-context variant of the constitutional ai (cai) approach @xcite . as the cai approach was not designed for diversity, we extend it to the diversity task by hand-crafting few-shot exemplars for cai, and extending the "critique request" and "revision request" prompts to cover "diversity". similarly, for standard 5-shot prompting and 5-shot cot approaches we hand-craft few-shot and cot reasoning examples, respectively. a full list of the few-shot prompts is presented in tbl. 6, 7, 8 and 9 in appendix. see fig. 6 and 7 for a visualization. ours: finally, we compare with two variants of our proposed method: (i) 0-shot ccsv is the proposed method described in sec.3.1 and (ii) 5-shot ccsv is a few-shot variant of our proposed approach, wherein we use the same few-shot exemplars designed for cai and simply apply our proposed collective-critique and self-voting steps on top by replacing the greedy decoding. this allows us to evaluate the efficasy of the proposed collective-critique and self-voting building blocks independent of the underlying prompts used. base llm and inference setup. we use the instruction-tuned palm 540 billion params model (flan-palm 540b) @xcite as our base llm to run all our experiments on. to turn the llm into a conversational agent, we instantiate the llm with the preamble "you are an ai model. please respond to the user's questions fluently and comprehensively.". for fairness of comparison, inferences for all methods and baselines are performed using top-k decoding at temperature 0.7 and 1024 decode steps. for the proposed approach, which relies on multiple decodes, we sample 5 decodes from the model. implementation. all the methods and baselines used in this paper are implemented via in-context prompting of the model flan-palm at inference time. the supplementary sec. a.3 reports the exact in-context prompt text used for each of the baseline methods, including the few-shot exemplars used (see sec. a.4). ## conclusion we formalize the problem of diversity of representation in llms, and propose metrics and methods to quantify and improve people diversity in llms. we show that by tapping into models reasoning abilities, our proposed in-context prompting technique called collective-critique and self-voting (ccsv) improves people and culture diversity by a large margin over the baseline. strikingly, our zero-shot approach outperforms all few-shot baselines and is able to improve diversity without requiring any additional data, hand-crafted examples or prompt tuning, while demonstrating stronger robustness and generalization properties. we believe the key idea of collectively using insights from multiple decodes is valuable and can have wide-applicability beyond just diversity in improving in-context reasoning methods in general. future work is needed to explore the applicability of the proposed to other downstream tasks beyond diversity.