PROMPTEVALS : A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines (2025)

Abstract: Large language models (LLMs) are increasingly deployed in specialized production data processing pipelines across diverse domains—such as finance, marketing, and e-commerce. However, when running them in production across many inputs, they often fail to follow instructions or meet developer expectations. To improve reliability in these applications, creating assertions or guardrails for LLM outputs to run alongside the pipelines is essential. Yet, determining the right set of assertions that capture developer requirements for a task is challenging. In this paper, we introduce PROMPTEVALS, a dataset of 2087 LLM pipeline prompts with 12623 corresponding assertion criteria, sourced from developers using our open-source LLM pipeline tools. This dataset is larger than previous collections. Using a hold-out test split of PROMPTEVALS as a benchmark, we evaluated closed- and open-source models in generating relevant assertions. Notably, our fine-tuned Mistral and Llama 3 models outperform GPT-4o by 20.93% on average, offering both reduced latency and improved performance. We believe our dataset can spur further research in LLM reliability, alignment, and prompt engineering.

## introduction large language models (llms) have become increasingly popular for various data processing tasks. an open-source tool for building llm pipelines, developed by some of the authors, now has over 3 million weekly downloads. its community has created thousands of specialized prompts for diverse fields like medicine, finance, and sports, leveraging llms' impressive zero-shot and fewshot performance @xcite @xcite . a common desire for developers using llms is to meet specific constraints on outputs, such as adhering to a particular structure or qualitative criteria @xcite . one approach to address this need is to collect large amounts of human preference data @xcite @xcite , and improve models through alignment techniques like supervised finetuning and reinforcement learning from human feedback @xcite @xcite . however, these methods have a high barrier to entry, requiring dataset collection, model fine-tuning, and serving infrastructure, which is more complex than simply manipulating prompts for llm calls. more importantly, fine-tuning isn't supervised at the constraint levelmeaning that even fine-tuned llms often fail to consistently follow instructions that correspond to constraints in detailed prompts @xcite @xcite . an alternative solution involves implementing developer-specified assertions on llm outputs @xcite @xcite . this approach typically follows two steps: first, defining binary evaluation criteria to represent the desired constraints; second, implementing these criteria as assertions to evaluate llm outputs and resample these outputs when assertions fail. however, developing effective assertion criteria is challenging-primarily due to the complexity of defining and conceptualizing these criteria, rather than their technical implementation @xcite . the complexity of coming up with assertions arises due to multiple factors: criteria can differ significantly between developers due to specific data, use cases, and end-user requirements @xcite ; criteria must account for both user preferences and llm-specific failure modes @xcite ; and developers may need to incorporate qualitative or subjective criteria that require llms themselves to perform the evaluation @xcite . to improve custom and task-specific alignment for llm pipelines, we need an approach that can examine developers' prompts and identify assertion criteria. these assertions can then be bolted onto the end of the llm pipeline, allowing for automatic retrying of the pipeline if assertions fail. developing such an approach requires substantial, diverse data on real-world llm applications and their associated constraints. fortunately, our opensource tool provides unique access to a diverse set of use-cases with associated prompts. in this paper, we present promptevals, a dataset created using our unique collection of realworld llm prompts and use-cases. this dataset consists of 2087 human-contributed prompt templates for custom tasks and 12623 comprehensive assertion criteria. promptevals has a median prompt template size of 191 tokens and is more than five times larger than previous collections @xcite . our dataset and corresponding benchmark (20% of the dataset) are hosted on hug-gingface 1 . using this benchmark, we evaluate gpt-4o and two open-source models, llama 3-8b and mistral-7b, on generating task-specific assertion criteria, and find that gpt-4o performs reasonably well out of the box, but its cost and latency to run for every prompt edit or pipeline update can be prohibitive in production environments-especially as prompts become increasingly complex and specialized. to address this, for our prompt engineering tools, we fine-tuned open-source models on our dataset (using mistral-7b and llama 3-8b architectures @xcite ), and these models exceeded gpt-4o's f1 performance in identifying desirable assertions by 20.93% on average. these fine-tuned models are made available to the community foot_1 , offering a faster, more cost-effective solution for generating high-quality assertions. ## related work this section reviews recent developments in prompt engineering, llm evaluation methods, and assertions for llm outputs. ## promptevals dataset this section describes the promptevals dataset, its construction process, and its characteristics. we begin by discussing the relevant background, then detail the dataset's composition and the process for generating ground-truth assertion criteria for each prompt template in our dataset. ## promptevals benchmark we split promptevals into three categories: 60% of the tasks (1252 prompts) for our training set, 20% (418 prompts) for our validation set, and 20% (419 prompts) for our test set. the prompte-vals benchmark evaluates an llm's effectiveness at generating accurate assertion criteria given a prompt template, using four key metrics defined below. the benchmark can be run by following the instructions in our github repository foot_2 . ## benchmarking llms for assertion generation in this section, we present our methodology for evaluating llms with the promptevals benchmark. we assess the performance of baseline and fine-tuned models. ## conclusion this study introduces promptevals, a new benchmark comprising over 2,000 humancontributed prompt templates and 12,000 assertion criteria. promptevals is more than five times larger than previous prompt collections @xcite . this diverse dataset represents a significant contribution to the field of llm pipeline development, offering a robust tool for evaluating and improving task-specific output constraints. using promptevals, we benchmarked several models, including gpt-4o, and additionally fine-tuned open-source models for assertion generation. our experiments demonstrate promptevals' utility in assessing and comparing performance across different approaches to generating relevant assertions. by making promptevals and our fine-tuned models publicly available, our goal is to encourage the development of more reliable and task-specific llm applications across various domains.