I ntr E x: A Dataset for Modeling Engagement in Educational Conversations (2025)

Abstract: Engagement and motivation are crucial for second-language acquisition, yet maintaining learner interest in educational conversations remains a challenge. While prior research has explored what makes educational texts interesting, still little is known about the linguistic features that drive engagement in conversations. To address this gap, we introduce IntrEx, the first large dataset annotated for interestingness and expected interestingness in teacher-student interactions. Built upon the Teacher-Student Chatroom Corpus (TSCC), IntrEx extends prior work by incorporating sequence-level annotations, allowing for the study of engagement beyond isolated turns to capture how interest evolves over extended dialogues. We employ a rigorous annotation process with over 100 second-language learners, using a comparison-based rating approach inspired by reinforcement learning from human feedback (RLHF) to improve agreement. We investigate whether large language models (LLMs) can predict human interestingness judgments. We find that LLMs (7B/8B parameters) fine-tuned on interestingness ratings outperform larger proprietary models like GPT-4o, demonstrating the potential for specialised datasets to model engagement in educational settings. Finally, we analyze how linguistic and cognitive factors, such as concreteness, comprehensibility (readability), and uptake, influence engagement in educational dialogues.

## introduction engagement and motivation are fundamental when learning a second language, influencing both learning outcomes and retention @xcite . however, while existing studies emphasize the relevance of content @xcite @xcite , it is still an open research question how linguistic properties themselves shape 1 https://huggingface.co/collections/xingweit/ intrex-68a8f2c97688157066860ae2 engagement in conversations. this is particularly relevant in educational settings, where structured dialogue plays a central role in knowledge transfer, especially as dialogue-based learning environments continue to expand with the use of ai tutors based mainly on dialogue interactions @xcite @xcite . existing research has identified linguistic features, such as concreteness, that contribute to the interestingness of texts, often relying on wholedocument assessments @xcite @xcite @xcite . yet, in educational conversations, engagement is not static but shaped by thematic continuity, discourse structure, and interaction flow. whole-document assessments offer limited insight into how interest foot_0 develops across multiple exchanges, making it essential to analyze sequence-level discourse features. to investigate this, we build upon the teacher-student chatroom corpus (tscc) @xcite , the largest dataset of chat-based edu-cational interactions between teachers and secondlanguage learners. the tscc provides pedagogical discourse annotations, such as topic opening and explanation. however, while these annotations describe functional aspects of conversation, they do not account for the engagement and interestingness experienced by learners. to bridge this gap, we introduce intrex, a dataset for modelling interestingness and its expectation in educational conversations, resulting from a comprehensive annotation process, involving over 100 second-language learners, to introduce sequence-level annotations of interestingness that, for the first time, capture at scale how interestingness and expectations evolve across extended interactions. we introduce two key dimensions of conversational engagement: interestingness, reflecting the perceived level of engagement, and expected interestingness, capturing the anticipated level of engagement. these two scores indicate not only what learners found interesting, but also what they expected to find interesting @xcite . however, assessing interestingness is inherently subjective and prone to variability among annotators. to improve consistency, we thus devised a comparison-based annotation approach inspired by reinforcement learning from human feedback (rlhf) strategies @xcite . instead of rating conversational turns in isolation, annotators compare original utterances with automatically rewritten 'boring' versions, leading to higher agreement and reduced noise. ## the intrex dataset intrex provides annotations for the conversations in tscc v2. this section describes tscc v2 and the intrex annotation process. fig. 2 shows the overall process. ## experiments in this section, we leverage the new intrex dataset to conduct an experimental assessment of engagement in educational dialogue, focusing on two complementary goals: (i) evaluating the extent to which llms align with human judgments of interestingness, and (ii) identifying the linguistic features that contribute to perceived engagement. ## conclusion we introduced intrex, the first dataset that annotates both interestingness and expected interestingness for teacher-student dialogues, extending tscc with sequence-level labels to track how engagement evolves across a lesson. we collected ratings from more than a hundred l2 learners and designed a comparison-based annotation, achieving substantially higher reliability at the sequencelevel than the turn-level. small, instruction-tuned llms fine-tuned on intrex surpassed gpt-4/4o in predicting human interest ratings, demonstrating the relevance and impact of hiqh-quality labels for the task. future work will be able to leverage intrex for training reward models for generation, and extend the approach to new learner groups and domains.