Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Models (2024)

Abstract: Recently, work in NLP has shifted to few-shot (in-context) learning, with large language models (LLMs) performing well across a range of tasks. However, while fairness evaluations have become a standard for supervised methods, little is known about the fairness of LLMs as prediction systems. Further, common standard methods for fairness involve access to model weights or are applied during finetuning, which are not applicable in few-shot learning. Do LLMs exhibit prediction biases when used for standard NLP tasks?In this work, we analyze the effect of shots, which directly affect the performance of models, on the fairness of LLMs as NLP classification systems. We consider how different shot selection strategies, both existing and new demographically sensitive methods, affect model fairness across three standard fairness datasets. We find that overall the performance of LLMs is not indicative of their fairness, and there is not a single method that fits all scenarios. In light of these facts, we discuss how future work can include LLM fairness in evaluations.

## introduction historically, evaluation of machine learning systems concerned only overall performance: how well did a trained system do on a held-out test set. more recently, practitioners have realized that dataset-level scores can mask uneven performance across different sets of data points @xcite . this can be especially problematic when performance varies significantly between demographic groups, such as systems that do relatively worse on underrepresented and historically oppressed demographic groups (e.g., @xcite . these systems are often called unfair or biased. fairness has implications for the quality of the user experience and system robustness, and can measure user experience in a manner not reflected by overall metrics. additionally, fairness may have legal ramifications when ai regulations intersect with laws against discrimination (e.g., @xcite . to address these disparities, researchers have developed methods for fairness that may be applied to training objectives, alignment after training, and evaluation metrics @xcite . a new approach to prediction relies on large language models (llms), in which an instance is accompanied by a prompt and an llm relies on in-context learning to make a prediction @xcite . this type of learning, which requires no fine-tuning or other gradient updates, uses just a few examples at inference time as a "prompt" to guide inference on a final instance. because incontext learning relies only on a few text examples during inference, the content of these examples can be very important for the quality of the emitted output @xcite . while prior work has shown that llms perform surprisingly well on various prediction tasks, models are measured once again on overall performance alone, not fairness, despite an understanding of the variable nature of llm behavior @xcite . to date, little to no work has measured the fairness of llms as prediction systems, despite numerous studies showing inherent biases in the generations of llms @xcite . furthermore, traditional methods for addressing unfair models, whether pre-, in-, or posttraining, are not applicable to llms as the data they're trained on is often proprietary, pre-training them is expensive, and many leading models are closed source. relying on the importance of the content of examples in few-shot learning, we analyze the fairness of llms as prediction systems considering how different demonstration selection methods affect the resulting social fairness of the model in classification tasks. experiments with 7 popular models ( ## related work in-context learning. large language models are effective in a large number of classification and generative tasks @xcite @xcite . while finetuning a pretrained model is a popular paradigm @xcite , finetuning large models can be cost-prohibitive because of the compute required to do so. furthermore, finetuning requires additional task-specific labeled data, which can also be prohibitively expensive to collect. brown et al. (2020) evaluated in-context learning, or few-shot learning, for llms, a learning paradigm in which the model is given a few examples, or demonstrations, of a task and is then asked to complete the final example. in-context learning has shown impressive results in a variety of tasks, including question answering, translation, and natural language inference @xcite . work on in-context learning has focused on writing better prompts @xcite @xcite , choosing better demonstrations @xcite , and training with an in-context learning objective @xcite . there have also been explorations of the sensitivities of in-context learning, such as the format of the prompts @xcite or the order of the demonstrations @xcite . however, prior work has not studied the effect of demonstration choice on social fairness, only on overall performance @xcite . other work, like @xcite has evaluated the label fairness, i.e. performance differences across different labels or classes in a multi-class prediction setting, of llms in in-context learning by creating a system that chooses prompts to create a "fair" demonstration. similar to our work, they focused on shot or demonstration choice and found that shot selection matters for performance. thus, given the minimal amount of data used for in-context learning, we suspect that the choice of demonstrations has an effect on the social fairness of the model's output. social fairness with large language models. work that identifies and measures the biases of language models have classified these harms in two general categories: allocation and representation harm @xcite . representational harms happen when harmful concepts or relations are associated with demographic groups by a model; in language models these are often measured via token embeddings and model parameters with fill-in the blank, or complete the sentence templates (e.g., @xcite . most bias studies in nlp have focused on representational harms: many studies have demonstrated how generations from llms exhibit bias towards specific groups, or generate text that can be considered offensive, harmful or toxic @xcite @xcite models by finding disparate performances from impersonating different roles, e.g. better performance when impersonating men than women. perhaps the most related work is @xcite , who investigates fairness re-programming techniques for models that cannot be re-trained or finetuned, e.g. in-context learning llms. they append token perturbations to the prompt, fairness triggers, that are learned from a helper model and show that they can decrease performance differences across demographic groups. we, instead, focus on investigating the role of choice of demonstrations or shots in the performance differences of llms on in-context learning settings. ## data we consider three text classification datasets that include demographic information to evaluate the fairness of language models with regard to demographics: bias in bios @xcite , and hat-explain @xcite . bias in bios (demographics: gender) is a collection of english documents from commoncrawl that contain biographies. the task is to predict the occupation from the biography, (mit license.) @xcite found gender bias present in models for this task. following @xcite , we measure gender bias by comparing the relative performance of models across biographies written about men and women. we select professions (labels) that had more than 1000 examples of biographies for each gender in the test set. 1 this yields the following 8 labels: attorney, dentist, journalist, photographer, physician, professor, psychologist, and teacher. we randomly selected 500 for each gender from each profession to create a test set of 8,000 biographies. we then created a training set of 183,638 biographies by selecting all the biographies from the original train split with the professions listed above. twitter sentiment (demographics: race) is a collection of english tweets where the task is to predict binary sentiment in a tweet. tweets have also been annotated with a binary attribute corresponding to online text dialects: african-american english (aae) or standard american english (sae), which has been previously correlated with partsof-speech tagging performance difference in prior work @xcite . we use these text di-1 i.e. professions with at least 1000 men and 1000 women alects as proxies for race and measure racial bias by comparing the relative performance of sentiment classification across the dialects, similar to @xcite . to construct the dataset we follow @xcite (apache licence, v2.0.) we then select 40k and 2k random tweets from each combination of dialect and sentiment for train and test, creating a train set with 160k examples and test set of 8k. hatexplain (demographics: race) is a collection of posts from gab and twitter annotated with toxicity and hate speech labels, as well as demographic labels for the target group of the hate speech. while prior work has shown that there are performance differences for detecting hate speech for different target groups based on gender, religion, and race, we experiment only on race as it was the demographic characteristic with the reported highest disparities @xcite . we remove indigenous and indian examples from our race demographics as they do not appear in all data splits. to construct the dataset, we followed a similar procedure to @xcite : we first reduced the space from multiclass to binary classification by combining the "offensive" and "hatespeech" labels to a singular "toxic" label while keeping the "normal" class the same. because of hatexplain has multiple annotators per example for the labels and demographics, we take the majority label and the majority demographic. if there is not a majority in either, we discard the example. ## methods we measure the effect of different demonstration selection methods on prediction fairness of llms. we hypothesize that, similar to how the choice of demonstrations has been shown to have an effect on performance, different methods of demonstration selection will affect social fairness of the model. this section describes the models evaluated, prompts, demonstration selection methods, and definitions of performance and fairness. overall, we conduct experiments in 36 setups (3 tasks, 12 models), using 6 demonstration selection strategies. ## results & analysis for demographic fairness, we observe that the most fair models are often below random performance. since the ultimate goal of fairness is to maximize the utility of the models across all demographic groups (rather than none), we do not take into account fairness results from models that perform below a random classifier, these are shaded on ## conclusion significant work has gone into evaluating different demonstration selection strategies in the performance of llms as classification systems. this paper represents one of the first studies to consider the fairness of these systems. our study considers 7 widely used family of models ( where do these results leave us? first, fairness must be evaluated alongside task performance when developing prompts, selection strategies, and models. we cannot assume any relationship between fairness and performance. second, we need to better understand why llms are unfair in their predictions. while significant work has examined fairness in supervised training objectives @xcite , and other work demonstrates bias in llm generations (chang and bergen, 2023), we need work that intersects these two. third, how can we determine when a llm is being unfair? work examining confidence in @xcite can help automatically determine the accuracy of the system. can we develop similar metrics for fairness? this would be especially helpful in cases where we do not have demographically labeled data. finally, there is now a large focus on fine-tuning llms (e.g. rlhf @xcite , @xcite ). the goal of these methods has been better instruction following and improved accuracy on prediction tasks, but our results suggest they do not always make models fairer. how can we include fairness objectives in this training process?