Role of Reasoning in LLM Enjoyment Detection: Evaluation Across Conversational Levels for Human-Robot Interaction (2025)

Abstract: User enjoyment is central to developing conversational AI systems that can recover from failures and maintain interest over time. However, existing approaches often struggle to detect subtle cues that reflect user experience. Large Language Models (LLMs) with reasoning capabilities have outperformed standard models on various other tasks, suggesting potential benefits for enjoyment detection. This study investigates whether models with reasoning capabilities outperform standard models when assessing enjoyment in a human-robot dialogue corpus at both turn and interaction levels. Results indicate that reasoning capabilities have complex, model-dependent effects rather than universal benefits. While performance was nearly identical at the interaction level (0.44 vs 0.43), reasoning models substantially outperformed at the turn level (0.42 vs 0.36). Notably, LLMs correlated better with users’ self-reported enjoyment metrics than human annotators, despite achieving lower accuracy against human consensus ratings. Analysis revealed distinctive error patterns: non-reasoning models showed bias toward positive ratings at the turn level, while both model types exhibited central tendency bias at the interaction level. These findings suggest that reasoning should be applied selectively based on model architecture and assessment context, with assessment granularity significantly influencing relative effectiveness.

## introduction as conversational ai increasingly becomes part of daily life, from customer support bots to personal assistants, it is crucial to make interactions with conversational systems enjoyable, since user enjoyment determines both immediate and future engagement @xcite @xcite . to that end, measuring user experience, such as enjoyment or satisfaction, at both turn and dialogue-level granularity is re-quired for user alignment @xcite . predicting user enjoyment is particularly challenging due to its subjective and variable nature @xcite . traditional models struggle to cope with this variability, but large language models (llms) offer an attractive solution with their ability to recognize subtle conversational cues related to enjoyment more effectively than traditional models relying on handcrafted features @xcite . the practical deployment of conversational ai systems demands automated evaluation metrics that can be run frequently and efficiently, without requiring human annotators @xcite . one of the design choices for using llms to predict enjoyment is whether to have them reason explicitly (e.g., via chain-of-thought @xcite ) and when to attempt to predict enjoyment directly. while explicit reasoning can result in improved performance on complex interactions and enable user-aware interactions @xcite , it can lead to overthinking, increasing inference time, and output length @xcite . this trade-off highlights the value of llms' internal consistency, which in some settings exceeds that of human annotators, particularly in structured evaluation tasks @xcite . this raises the question of when llms must reason to improve the prediction of user enjoyment. despite advances in llm capabilities, we have not found prior work comparing reasoning and nonreasoning models to detect the quality of user experience, particularly at different levels of granularity from overall conversation enjoyment to turn-level interactions. this research bridges that gap by benchmarking four state-of-the-art models: claude-3.7-sonnet (reasoning and non-reasoning), gpt-4.5-preview (reasoning), o1 (reasoning), and gemini-2.0-flash (reasoning and non-reasoning) on the human-robot interaction conversational user enjoyment scale (hri cues) dataset @xcite , expert-annotated with enjoyment ratings both per turn and overall conversation levels, with the following research questions: rq1 do large language models with reasoning capabilities produce more accurate predictions of enjoyment ratings in conversations compared to their non-reasoning counterparts? rq2 how does the effectiveness of reasoning vs. ## related work user enjoyment is a critical factor for conversational ai, affecting both immediate engagement and long-term adoption @xcite @xcite @xcite . unlike task-oriented metrics, enjoyment captures the affective value of interactions, presenting unique measurement challenges due to its subjective nature @xcite . the rising popularity of conversational agents in everyday applications has made enjoyment a key determinant of user retention and overall satisfaction with these systems. studies suggest that enjoyment metrics may be more predictive of long-term user engagement than traditional task completion measures in open-domain dialogue systems @xcite . ## methodology building on prior work that demonstrates llmbased user enjoyment detection in human-robot conversations can outperform human annotators @xcite , and motivated by the strong performance of recent reasoning models on a variety of tasks @xcite , this work evaluates whether such models enhance accuracy in detecting user enjoyment for interactions with robots. ## results this section presents an analysis of different llms in conversation quality assessment tasks, focusing on reasoning capabilities and enjoyment detection. we evaluated both standard (non-reasoning) models and reasoning-enhanced versions, comparing their performance at the turn-by-turn and interaction (full dialogue/conversation) levels. ## discussion our analysis addresses four key research questions about reasoning capabilities in llms for conversation enjoyment prediction. rq1: do reasoning models produce more accurate predictions? reasoning capabilities demonstrate model-dependent effects rather than universal benefits. at the overall enjoyment level, reasoning models showed moderately higher average accuracy (0.49 vs 0.45 for non-reasoning models) with notably lower variance (@xmath0.04 vs 0.10), indicating more consistent performance across different model architectures. however, individual model differences were pronounced: reasoning significantly improved gemini 2.0's turnlevel accuracy (0.30 to 0.41) but slightly decreased claude 3.7's performance. this suggests reasoning interacts with underlying model architectures in complex ways, challenging the assumption that explicit reasoning universally enhances performance. rq2: how does effectiveness differ across granularity levels? reasoning models substantially outperformed at the turn level (0.42 vs 0.36), while non-reasoning models performed slightly better at the interaction level (0.44 vs 0.43). although all models approached but did not exceed the human baseline (0.46), this indicates that reasoning processes offer particular value for granular judgments requiring detailed analysis of specific conversational turns. notably, reasoning models maintained consistent evaluation quality across granularity levels compared to non-reasoning models. while these patterns suggest meaningful differences, overall statistical comparison between reasoning and non-reasoning approaches showed no significant difference (@xmath1.49), indicating the benefits are context-dependent rather than universal. rq3: what distinctive error patterns emerge? at the turn level, non-reasoning models showed a clear bias toward rating 4, while reasoning models distributed predictions more evenly. at the interaction level, both exhibited a central tendency toward rating 3, with reasoning models showing more distributed predictions for extreme cases. both types struggled with extreme ratings (1 and 5), suggesting reasoning processes may introduce nuanced considerations that moderate judgments at the extremes. rq4: how do model ratings compare to human annotations? llm models demonstrated higher internal consistency than human annotators (0.64 vs 0.44 at turn level for non-reasoning models) but generally lower accuracy when compared with annotators. combining human annotators with models consistently decreased overall agreement, with the lowest icc occurring when humans were paired with reasoning models (0.37 at turn level). correlation analysis with users' self-reported metrics revealed satisfaction as the most consistently correlated metric. claude 3.7 (nonreasoning) demonstrated the most comprehensive correlation profile, with significant associations with satisfaction (@xmath2.44, @xmath3.029 turn level, @xmath4.51, @xmath5.009 interaction level), fun (@xmath6.44, @xmath7.030), and perceived strangeness (@xmath8.40, @xmath9.049). for gemini 2.0, reasoning improved correlation with satisfaction at the interaction level (@xmath10.48, @xmath11.014 vs @xmath12.25) while reducing it at the turn level (@xmath13.40, @xmath14.047 vs @xmath15.56, @xmath16.004). notably, o1 (reasoning) showed no significant correlations with user self-reports despite strong accuracy metrics, indicating a disconnect between its evaluation approach and user experience. these findings align with @xcite ) regarding similar performance patterns. our work extends this prior research by systematically comparing reasoning capabilities across multiple gran-ularity levels, revealing that reasoning benefits depend on both model architecture and assessment context. ## conclusion this study shows that reasoning capabilities in llms work differently depending on the model architecture, challenging the common belief that explicit reasoning always improves performance. the model-dependent effects we observed suggest that reasoning processes may strengthen or weaken existing model abilities, rather than providing consistent benefits across all systems. the better performance of reasoning models at turn-level evaluation likely comes from their ability to break down conversation patterns into smaller parts-a process that becomes less helpful when making overall conversation judgments. our finding that models achieve higher internal consistency than humans while showing lower accuracy reveals a basic problem in automated evaluation systems. this pattern suggests that models apply consistent but wrong standards, being precise but inaccurate. the disconnect between o1's strong accuracy metrics and poor correlation with user self-reports shows this problem, indicating that technical performance metrics may not capture what users actually care about in conversation quality. these findings have important effects for conversation system development and evaluation methods. practitioners should choose reasoning-enabled models based on specific use cases rather than assuming they always work better, with reasoning particularly valuable for detailed analysis tasks. the strong correlation between model assessments and user satisfaction scores, despite accuracy problems, suggests that llms may capture how users actually feel in ways that standard measures miss. this work moves the field away from one-size-fits-all approaches toward more thoughtful, context-aware deployment of reasoning capabilities in conversation analysis systems.