TRUEB ench: Can LLM Response Meet Real-world Constraints as Productivity Assistant? (2025)

Abstract: Large language models (LLMs) are increasingly integral as productivity assistants, but existing benchmarks fall short in rigorously evaluating their real-world instruction-following capabilities. Current benchmarks often (i) lack sufficient multilinguality, (ii) fail to capture the implicit constraints inherent in user requests, and (iii) overlook the complexities of multi-turn dialogue. To address these critical gaps and provide a more realistic assessment, we introduce TRUEBench (Trustworthy Real-world Usage Evaluation Benchmark), a novel benchmark specifically designed for LLM-based productivity assistants. TRUEBench distinguishes itself by featuring input prompts across 12 languages, incorporating intra-instance multilingual instructions, employing rigorous evaluation criteria to capture both explicit and implicit constraints, and including complex multi-turn dialogue scenarios with both accumulating constraints and context switches. Furthermore, to ensure reliability in evaluation, we refined constraints using an LLM validator. Extensive experiments demonstrate that TRUEBench presents significantly greater challenges than existing benchmarks; for instance, a strong model like OpenAI o1 achieved only a 69.07% overall pass rate. TRUEBench offers a demanding and realistic assessment of LLMs in practical productivity settings, highlighting their capabilities and limitations.

## introduction the remarkable advancement of large language models (llms) has led to their pervasive integration into daily tasks, highlighting their use as assistants to enhance human productivity (referred to as productivity assistants). productivity assistants are employed for diverse tasks, such as summarizing meeting discussions or deriving insights 1 the leaderboard and sample data are available at https://huggingface.co/spaces/samsungresearch/ truebench figure 1 : an example from followbench @xcite . relying on original (explicit) constraints might fail to capture semantically incorrect translations. from data. to evaluate the performance of llms in real-world applications, there is a growing focus on developing benchmarks aimed at assessing capabilities on instruction-following with various constraints @xcite @xcite @xcite . a benchmark for evaluating productivity assistants should involve data creation oriented toward productivity and reflect the characteristics of user inputs received by productivity assistants in real-world settings. we analyzed patterns of in-house productivity assistant service to derive three key attributes necessary for evaluating the effectiveness of llms as productivity assistants. (i) multilinguality: real-world productivity tasks frequently involve interactions across diverse languages. we observed users frequently involved in cross-border communication or generating/understanding content in foreign languages. (ii) implicit ## truebench in this section, we detail the truebench, explaining its evaluation protocol and the pipeline for its dataset creation. details of the human annotation process can be found in appendix f. ## main results we conduct a comprehensive evaluation of 31 llms on truebench foot_2 . detailed results across various task categories and languages within truebench are provided in tables 5 and 6 , respectively. ## conclusion in this study, we introduce truebench, a benchmark designed to evaluate llms as productivity assistants, addressing the limitations of existing benchmarks. truebench incorporates multilingual instances with implicit and instancelevel constraints, and our evaluation method leverages llms to ensure reliability. our experiments demonstrate that our checklist-based evaluation aligns better with human judgment than other llm evaluation methods. the pass rates below avg. 70% highlights the inherent complexity of real-world productivity tasks and emphasize the necessity of truebench. we believe this work establishes a crucial foundation for assessing the capabilities of future productivity assistants.