Understand User Opinions of Large Language Models via LLM -Powered In-the-Moment User Experience Interviews (2025)

Abstract: Which large language model (LLM) is better? Every evaluation tells a story, but what do users really think about current LLMs? This paper presents CLUE, an LLM-powered interviewer that conducts in-the-moment user experience interviews, right after users interact with LLMs, and automatically gathers insights about user opinions from massive interview logs. We conduct a study with thousands of users to understand user opinions on mainstream LLMs, recruiting users to first chat with a target LLM and then be interviewed by CLUE. Our experiments demonstrate that CLUE captures interesting user opinions, e.g., the bipolar views on the displayed reasoning process of DeepSeek-R1 and demands for information freshness and multi-modality. Our code and data are at https://github.com/cxcscmu/LLM-Interviewer.

## introduction foundation and large language models (llms) are redefining the way users interact with the digital world. billions of users now chat with llms regularly through one unified interface and consume information through generative content for their information, entertainment, and task assistance needs @xcite . understanding user opinions about llms is, however, a challenging task. it is difficult to design one evaluation to reflect the rich llm capabilities. the user experiences powered through generative content are also hard to characterize by metrics like accuracy and bleu. the tendency for data contamination during pretraining further complicates the understanding of llm performance @xcite . the community often relies on coarse user preference ratings to understand llms' performances @xcite . * these authors contributed equally to this work. approved by our university institutional review board (irb), we recruited thousands of crowdsource workers to chat with an llm, randomly selected from six llms from the gpt, claude, gemini, llama, and deepseek families. then clue-interviewer interviews participants right after their chat sessions. we obtained 1206 chat-andinterview sessions on diverse topics initiated by crowdsource workers, ranging from travel destinations to programming developments, that we will make publicly available. our human evaluation demonstrates the effectiveness of clue-interviewer in conducting semistructured and in-the-moment ux interviews. we manually annotated 120 interview sessions or 180 interview turns, randomly sampled for each of the six target llms. two annotators annotated 25% of the labeled data and their agreement level was high. human annotations confirmed that clue-interviewer covers 74% of the pre-defined dimensions to evaluate target llms and actively asks follow-up questions to gain deeper insights. the evaluation also confirms that clue-insighter is highly accurate in categorizing interview rounds and evaluating user responses on target dimensions. our user study reflects interesting user opinions of llms. users expressed reserved opinions about llms in their text responses to clue-interviewer, considering all mainstream llms mediocre, but gave a near-perfect rating when asked explicitly. users considered llms decent at understanding their needs but not as effective in meeting them. users also expressed new feature requests in the interviews, for example, visual and multimedia capabilities, fresh knowledge access, personalized responses, and flexible access to reasoning and explanation processes, providing motivations and evidence for future llm development. the main contributions of this paper are: ## related work many benchmarks have been developed to evaluate predictive effectiveness of llms. notable examples include glue @xcite and super-glue @xcite , both including a suite of natural language understanding tasks. the suite of language tasks quickly grows to hundreds, such as in flan @xcite and big-bench (bench authors, 2023) benchmarks. the community also keeps increasing task difficulty, for example, from mmlu @xcite to "humanity's last exam" @xcite , to test the boundary of llm intelligence. these evaluations are effective in reflecting llms' ability to predict the right label, which aligns closely with some real-world applications such as question answering. evaluating generated content is challenging, as two text sequences with high n-gram overlap @xcite or semantic similarities @xcite may not lead to the same user experience @xcite . recent approaches switch to model-based evaluation and employ llm-as-a-judge for predefined dimensions @xcite . model-based evaluation has become a common practice in evaluating generated content, albeit various challenges such as self-biases and inaccuracies @xcite @xcite . the ultimate verdict of an ai model is how it serves its users. for established applications like search engines, understanding model performances based on noisy and coarse user feedback is a longlasting research topic @xcite . there are various efforts to collect user feedback on new llm-powered scenarios. one notable effort is chatbot arena @xcite . it asks users for preferences on side-by-side llm chats and computes arena scores based on that. many view the arena score as a reliable reflection of user preferences on llms @xcite . ux interview is a standard approach to gather in-depth insights about user opinions @xcite . it is widely recognized as an effective tool to guide product developments toward increased adoption, consumer loyalty, and overall product success @xcite . effective ux interviews often require interview experts @xcite and are too expensive to scale up. many explored the potential of aipowered interview bots, for example, to conduct job interviews @xcite and conversational surveys @xcite . recently, @xcite built an llm-powered interview system and demonstrated their effectiveness in evaluating student experiences in ai-assisted classrooms. ## methodology clue leverages llms to conduct in-the-moment ux interviews and understand user opinions. it includes two components: clue-interviewer and clue-insighter. ## user study approved by our university institutional review board (irb), we conduct a large-scale study on user opinions of llms using clue. this section describes the user study methodology and the collected data. ## evaluation of clue before sharing the findings from our user study, this section presents human evaluations on the effectiveness of clue-interviewer and clue-insighter. semi-structured interview. we first manually labeled the evaluation dimensions ( we manually labeled the interview rounds where clue-interviewer asks follow-up questions. the distributions of probing frequency and depth are plotted in figure 3a and 3b. in more than half of interview sessions, clue-interviewer probes users for more detailed feedback, asking on average 1.3 follow-up questions, rather than merely asking predefined questions. in-the-moment interview. this experiment evaluates the ability of clue-interviewer to conduct in-the-moment interviews. we manually labeled explicit references to previous interactions in the interview sessions by the interviewer and the users, as a reflection of the in-the-moment effect. the distributions are plotted in figure 3c and 3d . interestingly, users actively refer to their previous interactions more frequently than the interviewer for more than 1/3 of the time during interviews, reflecting users' in-the-moment status. bird's-eye insights. clue-insighter automatically maps interview questions into target dimensions and converts user responses into categorical ratings. to evaluate this process, we compare these two automatic operations with our human annotations in ## user opinions of llms this section presents the quantitative ratings (sec. 6.1) and qualitative insights (sec. 6.2) about user opinions of llms from our user study. ## conclusion clue is a new methodology to gather user opinions using llm-powered interviews, enabling a deeper understanding of users through large-scale in-the-moment user experience interviews. our study with thousands of users shows that clue collects fine-grained user opinions on current llms, potential improvements, and frequently requested new functionalities. we view clue as a new user understanding tool widely applicable to many products beyond llms. it creates a tight-knit connection between consumers and ai developers, enabling more datadriven decisions based on large-scale, targeted, and authentic user opinions reflected in interviews.