A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models (2024)

Abstract: Large language models (LLMs) are essential tools that users employ across various scenarios, so evaluating their performance and guiding users in selecting the suitable service is important. Although many benchmarks exist, they mainly focus on specific predefined model abilities, such as world knowledge, reasoning, etc. Based on these ability scores, it is hard for users to determine which LLM best suits their particular needs. To address these issues, we propose to evaluate LLMs from a user-centric perspective and design this benchmark to measure their efficacy in satisfying user needs under distinct intents. Firstly, we collect 1,846 real-world use cases from a user study with 712 participants from 23 countries. This first-hand data helps us understand actual user intents and needs in LLM interactions, forming the User Reported Scenarios (URS) dataset, which is categorized with six types of user intents. Secondly, based on this authentic dataset, we benchmark 10 LLM services with GPT-4-as-Judge. Thirdly, we show that benchmark scores align well with human preference in both real-world experience and pair-wise annotations, achieving Pearson correlations of 0.95 and 0.94, respectively. This alignment confirms that the URS dataset and our evaluation method establish an effective user-centric benchmark. The dataset, code, and process data are publicly available at https://github.com/Alice1998/URS.

## introduction large language models (llms) are rapidly developing and gradually changing the way people interact with computing systems and permeating diverse facets of work and daily life @xcite . although llms show powerful capacities for completing various tasks @xcite , it is essential to understand how they satisfy users' actual intents and needs @xcite . according to related studies, many attempts have figure 1 : existing benchmarks are mainly model ability-focused and categorized by tasks @xcite . we benchmark llms on user reported scenarios (urs), which are user-centric, intent-driven, multi-cultural, and multi-llm usage cases involved. the first group of work @xcite @xcite considers llms as task-completion models and mainly employs close-domain assessments to measure their various task-solving abilities, such as world knowledge, mathematics ability, coding skill, reasoning, planning, agent task, etc. @xcite . however, the existing attempts focused on predefined tasks, without considering the roles that humans would like llms to play in real-world scenarios. for example, while a task was evaluated, it was unclear if the task was important to humans and frequently required by users in real applications. the second group evaluates whether llms' responses align with human preferences @xcite . benchmark cases language source category alpacaeval 805 english existing datasets and synthetic data 5 sources mt-bench-101 1,388 english synthetic data 13 tasks tencentllmeval 3,000 english, chinese human designed and written tasks 7 areas mt bench 80 english human written questions 8 tasks alignbench 683 chinese selected questions in chatglm logs 8 tasks wildbench 1,024 english selected questions in chatgpt logs 12 tasks urs (ours) 1,846 english, chinese self-reported logs with 15 llms 6 intents a single resource. they do not reflect the true distribution of intended utilization of llms by real users and their effectiveness for these intents. besides, the categorization in current benchmarks is mainly focused on single-step tasks @xcite . however, end users might be more interested in knowing the ability of llms to satisfy their intended needs, rather than llms' performance on predefined individual tasks. in particular, a user intent might involve multiple model abilities with a series of tasks. for example, when a user is asking for advice about travel plans, this might involve model abilities of planning, personalization, and calling apis. consequently, the categorization of model-side abilities complicates the assessment of llm performance in user-side scenarios, making it challenging for users to select the suitable service tailored to their specific needs. this paper addresses the above issues in llm evaluations by developing a user-centric benchmark to examine whether llms' behaviors satisfy user needs in real-world scenarios. as shown in figure 1 , our work is different with the existing ability-focused benchmarks and highlighted with the three unique characteristics: user-centric. we benchmark llms from the user perspective in both dataset construction and evaluation designs. our constructed user reported scenarios (urs) benchmark encompasses 1,846 authentic interactions @xcite in english and 832 in chinese) across 15 diverse llm services, contributed by 712 users from 23 countries, each vetted through third-party manual quality checks. this dataset, reflecting the real-world usage of llms, is publicly available for research with user consent. based on this first-hand data, we design evaluations to measure llms' efficacy in satisfying real user needs. the benchmark results align well with hu-man preference in both real-world user experience across intents and pairwise comparison between llms. this user-centric approach promotes deeper collaboration between llms and humans as their capabilities continue to improve. intent-driven. different from the task-specific evaluations, our benchmark is categorized by user self-selected intents, including factual question answering, professional problem solving, text assisting, asking for advice, seeking creativity, and leisure. such categorization provides the potential utilization of llms by real users and is validated through the user study. based on the divided intents, the users who lack specific knowledge about the model capabilities can simplify their choice of proper service. for researchers and developers, this categorization helps to provide a more precise and targeted evaluation of how well llms meet user needs in diverse scenarios as one simple user intent may demand a blend of model abilities to address a series of tasks @xcite . multi-cultural. our data is contributed by users from 23 countries in asia, europe, north america, oceania, south america, and africa. their cases cover multiple cultural backgrounds, such as nations' traditional festivals, local points of interest, and pop culture across the globe. this brings diversity in llm evaluations, which is not included in simple translations of english-dominant content @xcite . in this study, our contributions are threefold: ## conclusion by focusing on real-world cases, user intent categorization, and the intent-aware evaluation method, we establish a benchmark of high alignment with human preference, endeavoring to advance the usercentric evaluation of llms. we hope the insights gained from this research will not only help improve the performance of llms but also foster a deeper understanding of how these services can be effectively integrated into people's lives to enhance productivity, creativity, and overall welfare.