When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLM s (2025)

Abstract: Large Language Models (LLMs) are highly sensitive to subtle, non-semantic variations in prompt phrasing and formatting. In this work, we present the first systematic evaluation of 4 methods for improving prompt robustness within a unified experimental framework. We benchmark these techniques on 8 models from Llama, Qwen and Gemma families across 52 tasks from Natural Instructions dataset. Our evaluation covers robustness methods from both fine-tuned and in-context learning paradigms, and tests their generalization against multiple types of distribution shifts. Finally, we extend our analysis to GPT-4.1 and DeepSeek V3 to assess frontier models’ current robustness to format perturbations. Our findings offer actionable insights into the relative effectiveness of these robustness methods, enabling practitioners to make informed decisions when aiming for stable and reliable LLM performance in real-world applications. Code: tt https://github.com/AIRI-Institute/when-punctuation-matters .

## introduction large language models (llms) today excel across a wide range of tasks in both in-context learning (icl) and supervised fine-tuning (sft) paradigms @xcite @xcite @xcite @xcite @xcite . however, a critical yet often overlooked challenge is the high sensitivity of llms to prompt formatting. many large-scale task-rich benchmarks rely on a single instruction format to evaluate all language models on a wide-range of tasks, implicitly assuming that performance is independent of prompt format @xcite . recent work shows that even semantically neutral variations in prompt structure can lead to substantial changes in model predictions, often exceeding the variability introduced by model architecture or inference method @xcite . prompt format (e.g. spacing, capitalization, punctuation) can dramatically influence performance, leading to inconsistent or unreliable outputs @xcite . this phenomenon, known as prompt sensitivity, can be mitigated using specialized robustness methods. a number of robustness techniques have been proposed to address this issue, including template ensembling @xcite , sensitivity-aware decoding @xcite , batch calibration @xcite , and consistency learning @xcite . however, these methods have primarily been evaluated in isolation, making it difficult for practitioners to assess their relative strengths or determine which method is best suited to a given scenario. our work addresses this gap through a compre-hensive, systematic evaluation of prompt robustness techniques under a unified experimental framework. specifically, we benchmark four widely cited robustness methods against standard few-shot prompting and fine-tuning with prompt format augmentation as baselines. we conduct experiments using a carefully curated version of gsm8k dataset @xcite and a representative subset of 52 tasks from the well-known natural instructions dataset, covering domains such as mathematics, logic, and text comprehension. as backbone models, we evaluate three modern llm families -gemma @xcite , llama @xcite , and qwen @xcite -with sizes from 1.5b to 9b parameters. we additionally include closedsource models to study format sensitivity at scale. within our framework, we answer the following research questions: rq1: how do existing robustness methods compare in effectiveness across various models? rq2: how distribution shifts affect the effectiveness of sft-based and icl-based methods? rq3: how do sampling strategies such as greedy decoding, temperature sampling, top-p and min-p sampling affect robustness? rq4: how sensitive are frontier models to prompt format perturbations, and what methods can be applied in black-box setting to improve their robustness? to the best of our knowledge, this is the first study to offer a side-by-side comparison of multiple prompt robustness methods under a unified, largescale evaluation protocol spanning diverse prompt formats, model families, learning paradigms, distribution shifts and inference strategies. by bridging previously disconnected lines of work, our findings provide actionable insights for both practitioners and researchers interested in building more stable and reliable llm-based systems. we also release our code to encourage systematic evaluation in the field of prompt sensitivity mitigation. ## related work recent work has highlighted the sensitivity of language models to subtle prompt variations, but current research remains fragmented @xcite . adversarial-focused studies @xcite expose vulnerabilities to malicious or perturbed prompts, emphasizing safety but targeting directed threat models rather than benign formatting inconsistencies. other works propose robustness-enhancing methods such as consistency learning @xcite , batch calibration @xcite , and template ensembles @xcite , which improve stability either during training or inference. however, these approaches are evaluated in isolation, making it difficult to assess their relative effectiveness. complementary studies @xcite @xcite analyze prompt components and formatting artifacts, showing that even innocuous design choices (e.g., whitespace, punctuation) can introduce large performance shifts. this further underscores the need for unified, standardized evaluation protocols. in summary, while prior research has addressed different aspects of prompt sensitivity, there is a lack of systematic, comparative evaluation across tasks, models, and learning paradigms. our work fills this gap by benchmarking four robustness methods under a unified framework across 52 diverse tasks, multiple llm families and distribution shift scenarios, resulting in actionable takeaways for practitioners. ## experimental setup to answer our research questions, we use a subset of natural instructions with a parametrized set of formats (section 3.1) and implement the methods from section 3.2. we evaluate performance and robustness using the metrics defined in section 3.3. ## results our experiments address the four research questions outlined in section 1: methods comparison in default setting (rq1, section 4.1), effect of distribution shifts on fine-tuning and icl-based methods (rq2, section 4.2), effect of inference strategy (rq3, section 4.3) and evaluations of frontier models (rq4, section 4.4). ## conclusion to the best of our knowledge, we have conducted the first comprehensive comparison of existing prompt sensitivity mitigation methods across multiple model families, sizes and distribution shifts. we provide actionable insights for practitioners. for example, excessive fragility of calibrationbased methods to class imbalance underscores the consequences of implicit assumptions, and hints that a reliable estimate of prior is important. meanwhile, ineffectiveness of light supervised finetuning with augmentations at improving robustness suggests that more research is needed to develop a strong baseline in this paradigm. finally, our experiments on frontier models confirm that scale is positively correlated with robustness. still, on some tasks even large models exhibit 8-10 accuracy point differences solely due to format changes. version of template ensembles with majority voting helps to mitigate this instability. we also release our code to facilitate research aimed to address the problem of format sensitivity.