ULTRABENCH : Benchmarking LLM s under Extreme Fine-grained Text Generation (2025)

Abstract: Fine-grained control is essential for precise and customizable text generation, yet existing benchmarks evaluate models on only a few attributes, typically fewer than five. We introduce UltraBench, a new benchmark for extremely fine-grained controllable generation (EFCG), which evaluates large language models (LLMs) under dense, multi-attribute constraints. Each sample includes approximately 70 attributes, combining LLM-extracted soft constraints (e.g., style and tone) with programmatically enforced hard constraints (e.g., word count). Using UltraBench, we conduct a comprehensive evaluation of state-of-the-art LLMs and prompting strategies. Models such as GPT-4.1 and Qwen3-8B perform well on individual constraints, achieving instruction-level accuracy above 70%, but consistently fail to satisfy all constraints simultaneously. To understand this limitation, we analyze model behavior across different dimensions. First, we observe a clear position bias: models tend to prioritize constraints presented later in the prompt while neglecting those that appear earlier. Second, we find that structural and formatting-related constraints are significantly more difficult to satisfy than content- or style-based ones, suggesting that current models struggle to coordinate global structure with token-level control. Finally, our error analysis reveals distinct failure modes: GPT-4.1 often attempts to follow constraints but falls short in precision, whereas LLaMA frequently omits constraints, particularly in multi-turn settings. These findings highlight fundamental limitations in EFCG and underscore the need for new methods that support dense, instruction-sensitive generation.

## introduction large language models (llms) have demonstrated impressive instruction-following capabil-ities @xcite , enabling them to interpret natural language commands and perform complex tasks such as multiturn dialogues @xcite , question answering @xcite , and structured writing @xcite . these capabilities have powered the rapid development of controllable text generation (ctg) in various domains @xcite . however, real-world applications including personalized content generation, product recommendation, and creative writing, often require simultaneously satisfying dozens of fine-grained constraints. these constraints span content elements, stylistic features, structural formats, and pragmatic goals, far exceeding the complexity handled in existing benchmarks, which typically focus on 3-5 attributes or incrementally added constraints in multiturn setups @xcite @xcite . as illustrated in figure 1 , model performance declines as the number of constraints increases, even for state-of-the-art llms. while these models may achieve reasonable accuracy on individual constraints, their ability to satisfy all constraints simultaneously deteriorates rapidly un-der denser constraint settings. this highlights the growing challenge of extreme fine-grained controllability as constraint scales. to address this gap, we propose ultrabench, a benchmark for extremely fine-grained controllable generation (efcg). ultrabench synthesizes training data with over 70 mixed hard and soft constraints per sample from realistic high-quality corpus, leveraging large language models for soft attribute extraction and programmatic rules for hard constraint enforcement. the key idea behind ul-trabench is to construct a large and diverse set of constraint types, some of which can be instantiated into multiple attributes; by adapting naturally occurring texts to accommodate more constraints and enriching them with 3-5 soft constraints extracted via gpt-4.1 (openai, 2025), we build highquality, densely labeled examples that reflect realistic fine-grained control demands. to ensure data quality, we incorporate a multi-stage quality control pipeline involving both automated validation and human verification (see §2.6). ultrabench offers a rigorous testbed for evaluating llms' ability to satisfy dense, diverse, and interactive control conditions. through extensive experiments on ultrabench, we conduct a comprehensive evaluation of several state-of-the-art llms. our findings are as follows: ## ultrabench construction in this section, we first formally define the efcg task in §2.1, and then introduce the construction process of ultrabench. we categorize the constraints into two types, hard constraints ( §2.2) and soft constraints ( §2.3), and describe how each type is instantiated in the benchmark. in §2.4, we discuss the key capabilities required for successful performance on this benchmark. we then present the constraint distribution statistics in §2.5, and describe the quality control procedures used to ensure data integrity in §2.6. ## case study we show an example in figure 8 . in this case, the model demonstrated several notable successes. it followed the required structure with a proper markdown title, highlighted section, placeholders, and sufficient length, while also incorporating most of the mandated keywords (such as airplane, steward, flagged, and molto). it further satisfied the ner constraints by including both a person (chef marco) and gpe entities (italy, curinga), and it maintained the intended casual and lighthearted blog style throughout. at the same time, the model exhibited important failures. it omitted the product entity, missed three lexical requirements related to keyword frequency and a fixed sentence, and violated strict character-level quotas (for g, i, and d). placeholders were present but not aligned with product constraints, and attempts to repeat certain keywords such as glutine created tradeoffs with the characterlevel rules. overall, this case illustrates how the model handles high-level stylistic and structural demands effectively, but struggles with symbolic constraints and multi-attribute alignment. it underscores the need for constraint-aware decoding strategies and more nuanced reward formulations in controllable generation. evaluation of ctg evaluating llm's adherence to constraints is challenging and typically involves automatic and programmatic assessments using various metrics @xcite @xcite . zhou et al. (2023b) centers on assessing 25 verifiable instructions. jiang et al. (2023) progressively integrates finegrained constraints to develop multi-level instructions, thereby enhancing complexity across six distinct types. wen et al. (2024) constructs a novel benchmark by synthesizing and refining data from the aforementioned benchmarks, with an emphasis on the combinatorial types of constraints. zhang et al. (2024) proposes a comprehensive constraintfollowing benchmark over 50 nlp tasks. however, none of them investigate the effects of extreme fine-grained attributes. 6 conclusions and future work we present ultrabench, a benchmark designed to stress-test llms under extremely fine-grained controllability. through comprehensive evaluation, we uncover systematic limitations in current models' ability to handle 70+ compositional constraints, such as position bias and structural failures. future work may investigate training-time interventions such as compositional instruction tuning and constraint-aware decoding. incorporating richer supervision signals, including constraint-level rewards or curriculum-based training, could improve model adherence to complex constraint sets. 15446 our work is sponsored in part by nsf career award 2239440, nsf proto-okn award 2333790, sponsored research projects from companies like cisco and ebay, as well as generous gifts from google, adobe, and teradata. any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the u.s. government. the u.s. government is authorized to reproduce and distribute reprints for government purposes not withstanding any copyright annotation hereon.