Beyond Generation: Leveraging LLM Creativity to Overcome Label Bias in Classification (2025)

Abstract: Large Language Models (LLMs) exhibit impressive capabilities in In-Context Learning (ICL) but are prone to label bias—an undesirable tendency to favor certain answers. Existing calibration methods mitigate bias by leveraging in-domain data, yet such data is often unavailable in real-world scenarios. To address this limitation, we propose SDC (Synthetic Data Calibration), a simple-yet-effective approach that generates synthetic in-domain data from a few in-context demonstrations and utilizes it for calibration. By approximating the benefits of real in-domain data, SDC effectively reduces label bias without requiring access to actual domain-specific inputs. Experimental evaluations on 279 classification and multiple-choice tasks from the Super-NaturalInstructions benchmark. The results show that SDC significantly reduces label bias, achieving an average Bias Score reduction of 57.5%, and outperforming all competitive baselines. Moreover, when combined with Leave-One-Out Calibration (LOOC), further improves performance, underscoring its effectiveness and generalizability in enhancing the reliability of LLMs.

## introduction large language models (llms) demonstrate impressive capabilities in handling unseen tasks by conditioning on examples of input-output pairs, known as in-context learning (icl) demonstrations. however, recent research reveals that llms' predictions exhibit label bias @xcite @xcite , an undesirable tendency to favor certain answers. this phenomenon is influenced by the label distribution in the demonstrations @xcite , or by the order of them @xcite . such a bias undermines the reliability of llm predictions and limits their practical applications, particularly in fields demanding high reliability, i.e finance. to address label bias, several calibration-based methods have been proposed, each using progressively more information from the target task's input. contextual calibration (cc) @xcite uses little to no domain-relevant input, instead feeding tokens like n/a to estimate and correct for the model's prior predictions. domain-context calibration (dcc) @xcite refines this idea by sampling random texts directly from the in-domain input, thereby capturing more domainspecific signals in the calibration process. more recently, leave-one-out calibration (looc) @xcite removes each demonstration in turn to compute a more precise bias estimation, effectively harnessing the original demonstration inputs themselves. although each of these methods reduces label bias, they also reveal that additional, task-related text (domain, random samples, or full demonstrations) can significantly improve calibration quality. motivated by these trends, we first performed a preliminary investigation into how real in-domain inputs help estimate a better prior for calibration. as expected, when in-domain data is available, it yields remarkably accurate estimates of the model's tendency to favor certain labels. however, real indomain inputs are often unavailable in real-world icl scenarios, where the model faces entirely unseen tasks with only a handful of example demonstrations. leveraging the strong generative capabilities of llms, we propose using the model itself to create synthetic in-domain data. in this work, we develop sdc-synthetic data calibration. sdc leverages llms to generate synthetic in-domain data from a few in-context demonstrations. this synthetic data is then used to calibrate model predictions, following the same approach as in our preliminary experiments. by doing so, sdc effectively mitigates label bias without requiring real in-domain data. we compared the proposed method with 2 preliminaries label bias in-context learning (icl) enables llms to solve unseen tasks by prompting them with several demonstrations. let @xmath0, y 1 ), (x 2 , y 2 ), . . . , (x |c| , y |c| )} denotes the demonstrations, where x * and y * are the input and output, respectively. the model is then expected to predict the answer y for the input x by feeding the concatenation of c and x into the model, formally: @xmath1, c), where p(•) denotes the probability predicted by the model, y is the set of all possible output answers. however, icl has been shown to exhibit label bias, where the model displays an unexpected preference for certain answers. this bias can be influenced by the order of examples in c or the token frequency of answers encountered during the llm's pretraining phase. in this work, we follow @xcite to measure label bias and performance using three metrics: bias score, relative standard deviation of class-wise accuracy (rsd), and micro-f1. the first two capture how strongly the model favors certain classes, whereas micro-f1 evaluates its overall classification performance. formal definitions and detailed explanations of these metrics are available in appendix a.1. progressive use of task input in previous studies. several calibration-based methods have been proposed to estimate and correct the model's prior preference over possible labels, each one exploiting progressively more domain-relevant input: contextual calibration (cc) @xcite uses minimal domain information to estimate the prior, simply replacing the real input with a placeholder token (n/a). formally, p cc (y i ) = p y i |[n/a], c . @xcite ) samples text from in-domain data rather than using n/a, thus incorporating more task-related content. this process is described by p dc (y ## sdc: synthetic data calibration building on our findings that domain-relevant input greatly improves calibration (section 2), we now address the more realistic setting in which real in-domain data is unavailable. we propose sdc, a method that leverages the strong generative capability of llms to create synthetic in-domain input from just a few demonstration examples. the key intuition behind sdc is that llms, when prompted with demonstrations, can generate diverse synthetic data that capture essential patterns of the target domain. by calibrating predictions with this synthetic data, we approximate the benefits of real in-domain data without its availability. in sdc, the llm is prompted with in:x 1 ,out:y 1 ...in:x |c| ,out:y |c| ,in: to generate synthetic data. foot_0 by sampling outputs from the model, we can collect a set of unlabeled synthetic in-domain data, x @xmath2, . . . , x s |x s | }. we then follow eq. 1 to estimate the model's prediction prior. ## conclusion this work introduces sdc (synthetic data calibration) to mitigate label bias in llms without requiring real in-domain data. by leveraging llms to generate synthetic calibration data, sdc significantly reduces label bias, achieving a 57.5% bias score reduction across 279 tasks. moreover, combining sdc with looc further enhances performance, demonstrating its effectiveness and adaptability. these results highlight sdc 's potential in improving llm reliability across diverse tasks.