Let The Jury Decide: Fair Demonstration Selection for In-Context Learning through Incremental Greedy Evaluation (2025)

Abstract: Large Language Models (LLMs) are powerful in-context learners, achieving strong performance with just a few high-quality demonstrations. However, fairness concerns arise in many in-context classification tasks, especially when predictions involve sensitive attributes. To address this, we propose JUDGE—a simple yet effective framework for selecting fair and representative demonstrations that improve group fairness in In-Context Learning. JUDGE constructs the demonstration set iteratively using a greedy approach, guided by a small, carefully selected jury set. Our method remains robust across varying LLM architectures and datasets, ensuring consistent fairness improvements. We evaluate JUDGE on four datasets using four LLMs, comparing it against seven baselines. Results show that JUDGE consistently improves fairness metrics without compromising accuracy.

## introduction a key capability of large language models (llms) is in-context learning (icl) -the ability to learn from examples provided within a prompt, without requiring parameter updates @xcite . while research has advanced our understanding of icl and techniques to enhance its effectiveness, a critical open question remains: how should we select fair and representative demonstration examples? this question becomes particularly critical in high-stakes domains where predictions directly impact human lives. consider a parole board using an llm to assess recidivism risk. the model's predictions are shaped by the examples it is shown-if those examples reflect historical biases or overlook key rehabilitation factors, the system may produce plausible-looking predictions that perpetuate or amplify existing disparities. in sensitive domains like criminal justice, healthcare, and hiring, the selection of demonstra-tions directly influences both predictive reliability and equitable decision-making. existing demonstration selection strategies, with a few exceptions, largely focus on optimizing performance metrics such as accuracy @xcite . while these methods are effective for improving icl performance, they often fail to account for fairness concerns. parallel research has explored bias and fairness in llm outputs @xcite and their trustworthiness @xcite , but a key gap remains: how can we improve group fairness directly at the demonstration selection stage in in-context learning? unlike prior work that dynamically selects demonstrations per test query @xcite , we explore an alternative: constructing a single demonstration set for an entire classification task. in our work, we investigate several key questions. do different llms exhibit consistent fairness behavior across datasets? we find significant variations in fairness outcomes across different llms, highlighting the need for adaptive approaches rather than one-size-fits-all solutions. do existing demonstration selection methods generalize across llm architectures and datasets? our results reveal that it is challenging for some prior methods to maintain stable fairness improvements across different models due to inherent variability in llm responses. can we design an effective, fairness-aware demonstration selection approach? we propose a simple yet highly effective method, judge (jury-based demonstration selection via greedy evaluation) that leverages each llm's own predictions on a carefully curated set of jury examples to guide demonstration selection. this paper makes several contributions. first, we provide a comprehensive analysis of existing approaches for fairness-aware demonstration selection across multiple datasets and architecures. second, we present judge 1 , a consistent and efficient framework for extracting fair representative examples from large datasets for icl. third, we validate our approach through extensive empirical evaluation, showing significant fairness improvements without compromising accuracy across multiple fairness benchmarks. finally, our systematic analysis demonstrates that the greedy construction approach is crucial for balancing fairness and accuracy, outperforming alternatives such as top-k selection and pooling-based methods. as llms continue to be deployed in increasingly sensitive domains, our work provides a practical framework for ensuring fairer outcomes while maintaining the efficiency that makes icl attractive. ## preliminaries: protected groups, attributes and group fairness protected groups are demographic subpopulations that should not face disparate treatment in model decisions. let g denote the set of protected groups, each defined by a protected attribute such as race, gender, or age. for any instance x in dataset d, its protected group membership is given by g(x). the population is partitioned into distinct protected groups @xmath0, g 2 , . . . , g l }, with each instance belonging to exactly one. for binary attributes (e.g., gender), this simplifies to @xmath1, g 2 }. group fairness aims to ensure that a model's behavior remains consistent across protected groups by enforcing that certain statistical measures are approximately equal across all protected groups, rather than focusing on individual-level fairness. we consider three popular metrics: demographic parity difference (∆dp), which measures the absolute difference in positive rates between protected groups @xcite ; equalized odds difference (∆eo), which measures the absolute difference in true positive and false positive rates between groups @xcite ; and mutual information (mi) @xcite , which quantifies the mutual information between protected attributes and selection decisions (details in appendix a). ## complexity analysis the complexity is dominated by llm inference. in judge, each demonstration in d reduced is evaluated with every jury member to compute demographic parity and accuracy. since this is repeated k times to build a k-sized set, the overall complexity is o(k • |d reduced | • |j |). unlike our method, exhaustive search evaluates all possible subsets of size k from n demonstrations, i.e., a complexity of o(n k ) which is infeasible for large n and k. a detailed complexity comparison with other baselines can be found in appendix c. ## related work demonstration selection in icl the problem of selecting demonstrations for icl has received significant attention. @xcite showed that finding demonstrations which are semantically similar to the test data often shows promising results. @xcite addressed this challenge by establishing a select-then-rank framework where they first limit the search space of demonstrations and rank the remaining examples through heuristics. @xcite highlighted that both data and model factors contribute to variability in performance. meanwhile, @xcite showed that predictive performance can be improved by selecting examples that minimize predictive bias. to address efficiency concerns, @xcite proposed a two-stage determinantal point process (dpp) method to select a fixed, representative subset of demonstrations, improving efficiency while maintaining performance. ## conclusion we propose judge, a greedy framework for fair demonstration selection in icl, guided by a jury set. across four datasets and four llm architectures, our method consistently improves fairness while maintaining accuracy, outperforming existing approaches. we further highlight the high variability of different methods across different datasets and language models, and establish the importance of considering demonstrations as a cohesive set rather than as individual examples to ensure fairness. as llms expand into critical applications, judge offers a practical and robust solution for ensuring fairness in icl.