LLM s Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLM s (2025)

Abstract: We present the first systematic evaluation examining format bias in performance of large language models (LLMs). Our approach distinguishes between two categories of an evaluation metric under format constraints to reliably and accurately assess performance: one measures performance when format constraints are adhered to, while the other evaluates performance regardless of constraint adherence. We then define a metric for measuring the format bias of LLMs and establish effective strategies to reduce it. Subsequently, we present our empirical format bias evaluation spanning four commonly used categories—multiple-choice question-answer, wrapping, list, and mapping—covering 15 widely-used formats. Our evaluation on eight generation tasks uncovers significant format bias across state-of-the-art LLMs. We further discover that improving the format-instruction following capabilities of LLMs across formats potentially reduces format bias. Based on our evaluation findings, we study prompting and fine-tuning with synthesized format data techniques to mitigate format bias. Our methods successfully reduce the variance in ChatGPT’s performance among wrapping formats from 235.33 to 0.71 (%^2)

## related works large language models (llms) have shown remarkable proficiency in formatting outputs to meet human expectations. such formats include markdown for lists and pointers @xcite ), 1 our codes and data will be made publicly available at link. ## general experimental setups benchmarks. for mcq bias evaluation ( §5.1), we select two datasets: mmlu @xcite and bbh @xcite . for mmlu, we randomly choose 27 subcategories. for bbh, we select the sports_understanding category following @xcite . for wrapping bias assessment ( §5.2), in addition to mcq benchmarks, the following datasets are experimented: gsm8k (cobbe et al., 2021) for reasoning, fairytaleqa @xcite for narrative comprehension, and hotpotqa (yang et al., 2018) for multi-hop reasoning. for list bias investigation ( §5.3), we use scidocsrr @xcite , a scientific document ranking task as the order list generation task, and semeval 2017 @xcite , the keyphrase extraction task as the unordered list generation. for mapping bias examination ( §5.4), we utilize a document-level information extraction task named scirex @xcite ) by synthesizing three extraction difficulty levels: easy (extracting from 1 sentence for 1 category), medium (3 sentences, 2 categories), and hard (5 sentences, 4 categories). for all benchmarks except mcq, we sample 200 points for evaluation @xcite . ## format evaluation experiments overall, we find that: (1) models show substantial format-following bias across formats for all benchmarks; ## mitigating performance format bias: actionable recommendations we propose methods as actionable recommendations to mitigate format bias in the performance of llms. generally, three primary streams of techniques have been widely studied and applied to tackle lm biases: (1) prompting @xcite ; (2) calibrating @xcite ; and (3) fine-tuning @xcite . while calibration techniques can only be used for white-box models, prompting and fine-tuning can be applied for both black-box (via api) and white-box ones. therefore, we explore prompting and fine-tuning techniques to reduce format bias. we target mitigating the format bias of chatgpt in fig. 6 , the strongest model that we benchmarked, on mmlu. we aim to reduce the wrapping bias ( §5.2) due to resource limits, but our methods can be generalized to any model and format. we also verify (and confirm) our mitigation strategies on gemma-2b-it, a medium-size model in appx. for "placeholder," human investigation reveals that multiple placeholder instructions cause chatgpt to be confused about where the placeholder is, making it frequently misunderstand and fail to follow this format instruction. fine-tuning with additional format data can eliminate format bias. we hypothesize that completely solving the format token bias problem of llms necessitates finetuning them on format data so that they are familiar with tokens in format instructions evenly. we propose a simple data synthesis strategy for finetuning llms: we sample a small set of training data for all evaluated formats, with ratios inversely proportional to their systematic evaluation scores ( §3.1). we chose syse scores over the estt ruee because they reflect the current model performance. practically, based on chatgpt's zero-shot systematic performance on mmlu colored in blue in appx.-tab. 6, we approximate the formats' performance ratios as "1, 1, 1 2 , 1 2 , 1 3 , 1, 1 3 " from left-to-right, resulting in training data ratios of formats of "1, 1, 2, 2, 3, 1, 3". we then preprocess the mmlu auxiliary training data according to these ratios, scaled by 500 (6500 samples total), and train chatgpt on this dataset. the finetuned results are plotted in fig. 6 . firstly, after finetuning, the average fi score across all formats is nearly perfect at 99.58% (ii). secondly, the biasf o score is significantly reduced from 235.33% 2 to 0.71% 2 (ii). these (i) and (ii) indicate finetuning largely eliminates format bias.