Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning (2024)

Abstract: Recent studies highlight the effectiveness of using in-context learning (ICL) to steer large language models (LLMs) in processing tabular data, a challenging task given the structured nature of such data. Despite advancements in performance, the fairness implications of these methods are less understood. This study investigates how varying demonstrations within ICL prompts influence the fairness outcomes of LLMs. Our findings reveal that deliberately including minority group samples in prompts significantly boosts fairness without sacrificing predictive accuracy. Further experiments demonstrate that the proportion of minority to majority samples in demonstrations affects the trade-off between fairness and prediction accuracy. Based on these insights, we introduce a mitigation technique that employs clustering and evolutionary strategies to curate a diverse and representative sample set from the training data. This approach aims to enhance both predictive performance and fairness in ICL applications. Experimental results validate that our proposed method dramatically improves fairness across various metrics, showing its efficacy in real-world scenarios.

## introduction large language models (llms), such as gpt-4 (openai, 2023), claude-3 (anthropicai, 2023), and llama-2 @xcite , have achieved state-of-the-art performance in many natural language processing tasks. these llms can adapt to different tasks by adding in-context prompts, without needing to retrain on the entire new dataset. this optimization technique for llms is called in-context learning (icl), which leverages specific input prompts to guide llms to generate more accurate outputs. recent research suggests that incorporating specially selected demonstrations into these prompts can significantly enhance llm performance @xcite . due to prompt length limitations, traditional llms have faced challenges in processing demonstrations from tabular datasets, which have a large number of features. however, with recent llms relaxing input length constraints, new avenues for applications in tabular datasets are opening up. @xcite has confirmed the predictive capabilities of llms on datasets from uci repository. considering the usages of tabular data in high-stakes domains @xcite , ensuring fairness alongside prediction performance is crucial for increasing trust in llms. @xcite has highlighted biases in llm predictions with tabular datasets, but there are limited further investigations on how the fairness of llms performance varies with different icl demonstrations. to bridge this gap, we aim to answer the following research question: how do different demonstration strategies impact the fairness performance of llms on tabular classification tasks? and is there a demonstration strategy better than other strategies? to better understand the impact of incontext learning on fairness, our proposed demonstration strategy considers the distribution of both demographic groups and target labels. a dataset can be divided into subgroups by demographic features, labelling the smallest as the minority (underrepresented) and the larger one as the majority. the fairness investigation compares differences between majority and minority groups. our investigation includes evaluating five advanced llms, i.e., text-davinci-003, gpt-3.5-turbo, gpt-4-turbo foot_0 , claude-3 haiku, and claude-3 sonnet foot_1 , across two fairness-focused tabular datasets: credit and adult. we found that prioritizing underrepresented samples and conscientiously including minority demographic groups and target labels during few-shot learning can significantly improve the fairness performance in llms output. despite the experimental observations, we are still wondering: why does prioritizing minority group demonstrations benefit the fairness performance of llms in tabular-based classification tasks? to further clarify this phenomenon, we perturb prediction labels and sensitive features in selected demonstrations and compare how the prediction outcomes of llms would be altered. through these perturbation experiments, we found that increasing the proportion of underrepresented labels enhances fairness, but can lead to a decline in prediction performance, and vice versa. up until now, the above findings and explanations have been based on random demonstration selection. we hypothesize that: we can deliberately select demonstrations to further improve fairness performance. motivated by the filter-then-search (lens) @xcite for textual classification, we adopt a similar process for extracting tabular demonstrations: first refine the training data set into a candidate pool and then sample and evaluate these candidates to identify the most supportive demonstrations. to this end, we introduced the fairness via clustering-genetic (fcg) algorithm to effectively extract representative samples, to further enhance the fairness of llm. unlike lens, which relies on progressive iterations on llms for candidate selection, our fcg method utilizes clustering. clustering does not require access to llms and maintains diversity among the selected shots, effectively addressing concerns related to the time required for iterations and the cost of llm access. additionally, previous studies often assume the same selection probabilities for candidates across evaluation iterations, requiring enormous iterations to ensure that each sample is equally considered. inspired by genetic evolution concepts, we adopt dynamic probabilities which give priority to representative samples with higher selection chances. sample representativeness is measured by the llm performance score, whose score is updated for each iteration. in this way, fcg can narrow the final sample set more efficiently, and drastically reduce the number of iterations needed. we implement experiments to evaluate the proposed fcg demonstration selection method. the results confirm that fcg algorithm improves llms performance in almost all strategies, with prioritizing the minority group still yielding the best results. to conclude, the main contributions in this paper are as follows: ## experiment setup our primary goal is to investigate how different few-shot demonstration choices influence the fairness performance of llms under the in-context learning (icl) setting. detailed related work on this area is discussed in appendix 6. in this section, we introduce the overall experimental setups. notations. given a dataset @xmath0, y, z) n @xmath1, the binary classification labels @xmath2:= {0, 1}, and sensitive feature @xmath3:= {0, 1}. we set @xmath4. d is split into training dataset d tr , validation dataset d dev and testing dataset d test . for each data point @xmath5:= {x, y, z}, a classifier f predicts the label f (x) based on the input features x. given a subset d ′ ∈ d tr , the proportion of samples where @xmath6. specifically, r @xmath7, whereas r @xmath8. similarly, the proportion of samples for which models and datasets. we use five llms as f : text-davinci-003 (davinci), gpt-3.5-turbo, gpt-4-turbo, claude-3 haiku, and claude-3 sonnet. the temperature in the model parameter is set to zero to ensure consistent responses. we select two tabular-based fairness datasets: default of credit card clients dataset @xcite ) and adult income (adult, @xcite ). the credit dataset covers information on credit card clients in taiwan, including demographics, bills, payment history, etc. its target is to predict whether there will be an overdue payment next month. ## why prioritizing minority group demonstrations benefit fairness? the above analysis points out a strong correlation between prioritizing minority group demonstrations with improved fairness performance of llms. however, it is not yet clear how and why this phenomenon occurs. thereby our next step is to clarify which part of the demonstrations most influenced the performance of llms. specifically, we perturb the prediction label y and the sensitive feature z in selected demonstrations and compare how the prediction outcomes of llms would be altered. the following experiment is performed on the adult dataset with 'income' as feature y and 'gender' as feature z. we set the random seed to 55 ## mitigation algorithm for fair demonstration selection the above results confirm that the application of diverse demonstrations, particularly those from the minority, can drastically influence the fairness of llms. experiments on different sets of selected shots under the same strategy also reveal a similar trend, albeit with different absolute performance values. this leads to the question: how to extract representative demonstrations that yield better performance among these sets? enumerating and evaluating the outcomes of llms across all possible sets is impractical due to the sheer number of combinations. thus, in this section, we propose a fairness via clusteringgenetic (fcg) algorithm to efficiently select influential demonstrations, leading llms to a better performance without having to explore all possible combinations. the core idea of fcg includes three aspects: (1) use clustering to shrink the selection sets while maintaining diverse samples. (2) define a score that considers both prediction and fairness performance, applying a genetic approach to iteratively select and score these samples within the sets. ## conclusions in this paper, we investigate how the choice of demonstrations impacts the fairness of llms on tabular data classification tasks when using incontext learning. through experiments, we found that prioritizing underrepresented groups and including minority examples in the few-shot demonstrations can significantly enhance fairness performance, without sacrificing predictive accuracy. further analysis revealed that increasing the proportion of underrepresented labels improves fairness metrics like demographic parity and equal odds. to efficiently retrieve effective demonstrations, we proposed the fcg algorithm that uses clustering and genetic evolution to select a diverse and representative set of examples from the training data. across multiple strategies and datasets, experimental results indicate that fcg was able to improve fairness compared to random sampling.