LMUNIT : Fine-grained Evaluation with Natural Language Unit Tests (2025)

Abstract: As language models become integral to critical workflows, assessing their behavior remains a fundamental challenge – human evaluation is costly and noisy, while automated metrics provide only coarse, difficult-to-interpret signals. We introduce natural language unit tests , a paradigm that decomposes response quality into explicit, testable criteria, along with a unified scoring model, LMUnit, which combines multi-objective training across preferences, direct ratings, and natural language rationales. Through controlled human studies, we show this paradigm significantly improves inter-annotator agreement and enables more effective LLM development workflows. LMUnit achieves state-of-the-art performance on evaluation benchmarks including FLASK, BigGenBench, and RewardBench 2, while maintaining competitive results on the original RewardBench. These results validate both our proposed paradigm and scoring model, suggesting a promising path forward for language model evaluation and development. Our code has been released at github.com/ContextualAI/LMUnit with an MIT license.

## introduction the evaluation of generative language models remains one of the most fundamental challenges in natural language processing @xcite @xcite -it determines how we measure progress and shapes the field's trajectory. as these models transition from research prototypes to production systems, users increasingly rely on them for critical workflows @xcite , creating an urgent need for evaluation methods that identify response strengths/weaknesses, ensure reliability, and prevent costly regressions. yet current approaches fall short: human evaluation * co-first author. is expensive and struggles to discern subtle differences among top models @xcite @xcite , while automated metrics compress response quality into coarse scores @xcite that rely on implicitly learned, often biased criteria @xcite @xcite . as models become more deeply integrated into essential workflows, it is imperative that our evaluation methodologies evolve in tandem, empowering llm practitioners to reliably detect subtle failures, meaningfully distinguish among top-performing systems, and generate actionable insights that drive improvements. we focus on measuring response quality -one of the most critical challenges in evaluating language models. defining "response quality" is inherently complex, depending on multiple factors including factual accuracy, logical coherence, and alignment with user objectives, which vary by domain, application, style, and context @xcite @xcite . existing approaches struggle with this complexity: building an effective scoring model for unit tests presents a significant challenge: it must accurately evaluate a wide range of criteria -ranging from broad notions of quality to detailed rubrics that capture intricate, context-specific requirements. existing approaches each address part of the problem: prompted llm judges can be instructed to consider certain criteria @xcite , but their accuracy is limited by generic instruction-following abilities and the inability to learn directly from preference data @xcite ; preference models, while closely aligned with human judgments, lack promptability and struggle to handle more granular, human-defined criteria @xcite . to address these challenges, we propose lmu-nit, a unified modeling approach that optimizes large language models as preference models while supporting flexible, user-defined evaluation criteria. by combining diverse training signals with natural language rationales, lmunit achieves strong results across preference modeling, direct scoring, and fine-grained unit test evaluations, laying a robust foundation for more adaptive and transparent evaluation methodologies. these rationales are optional at inference time but enabling them allows further interpretability of scores. to demonstrate our paradigm's effectiveness in enabling human stakeholder intervention, we assess its real-world impact through human studies. in a controlled annotation study, expert raters achieved higher inter-annotator agreement when evaluating outputs against explicit unit tests compared to standard preference annotations. additionally, in a case study with llm developers, lmunit's transparent, test-driven evaluations enabled identification of more errors than conventional llm judges, demonstrating the value of our proposed paradigm our key contributions include: (1) proposing the paradigm of natural language unit tests, and validating it at scale, (2) developing lmunit as a unified scoring model that achieves state-of-the-art performance, (3) showing the benefits and challenges of effective unit test creation and weighting strategies, (4) demostrating the importance of rationales when incorporating them as part of the training data. (5) validating our approach through human studies that demonstrate improved inter-annotator agreement and more effective llm development workflows. ## lmunit methodology to enable reliable scoring of natural language unit tests, we develop lmunit, a unified modeling approach that combines multi-objective training with natural language rationale generation. the key challenge lies in effectively integrating diverse training signals while maintaining both high accuracy and interpretable outputs. here, we detail our approach to addressing this challenge through careful problem formulation, synthetic data generation, and our training methodology. ## experiments we conducted extensive experiments to evaluate lmunit and the natural language unit test paradigm. first, we evaluated the performance of lmunit on several evaluation benchmarks, comparing to llms as judges, reward models, and trained evaluation models. next, we perform ablations to understand the impact of different methodologies, including loss functions and data mixture choices. also, we examined improving rationales through post-training and analyzed the impact of decomposition through several unit test strategies. finally, as shown in appendix a.1, we also conducted two human subject studies to validate the advantages of the lmunit paradigms over lm judges ## discussion our experiments and analyses reveal several key insights about the effectiveness of our unit test-based evaluation framework and highlight important directions for future work: lmunit shows benefits of unified training: our empirical results validate the benefits of a unified scoring approach through three key findings: combining multiple training objectives improves performance across all evaluation settings ( unit tests enable rich human-in-the-loop evaluation: language model evaluation frameworks should enable precise human steering while reducing noise and manual effort. our results show this paradigm achieves both goals: structured criteria dramatically improve evaluation consistency and inter-annotator agreement (figure 4 ), while offering multiple meaningful intervention points. humans can write or refine test criteria, optimize test weights ( rationale post-training improves task performance: a fundamental challenge in language models is developing genuine reasoning capabilities rather than simply learning to imitate human-like explanations. while training models to generate rationales through supervised learning can produce plausible-sounding explanations, this doesn't necessarily improve their underlying capabilities. our work demonstrates two key insights about moving beyond imitation: first, training with rationales improves model performance even when not generating them at inference time ( query-level unit test creation remains challenging: while our work advanced scoring and evaluation methodology, generating effec-tive query-specific unit tests proved difficult. global-level unit tests with learned weights significantly outperform query-level unit tests ( ## conclusion this paper introduces natural language unit tests, a paradigm for language model evaluation that enables precise assessment through explicit, testable criteria. to implement this paradigm effectively, we develop lmunit, a unified scoring model that combines multi-objective training across preferences, direct ratings, and natural language rationales to achieve state-of-the-art performance on major evaluation benchmarks. our results validate both the broader paradigm of decomposed evaluation and our novel scoring methodology. looking ahead, this work opens several promising research directions: deeper integration of human feedback loops, enhanced scoring models with improved reasoning capabilities, and end-to-end training of unit test generation and scoring.