I nte LLA : Intelligent Language Learning Assistant for Assessing Language Proficiency through Interviews and Roleplays (2024)

Abstract: In this paper, we propose a multimodal dialogue system designed to elicit spontaneous speech samples from second language learners for reliable oral proficiency assessment. The primary challenge in utilizing dialogue systems for language testing lies in obtaining ratable speech samples that demonstrates the user’s full capabilities of interactional skill. To address this, we developed a virtual agent capable of conducting extended interactions, consisting of a 15-minute interview and 10-minute roleplay. The interview component is a system-led dialogue featuring questions that aim to elicit specific language functions from the user. The system dynamically adjusts the topic difficulty based on real-time assessments to provoke linguistic breakdowns as evidence of their upper limit of proficiency. The roleplay component is a mixed-initiative, collaborative conversation aimed at evaluating the user’s interactional competence. Two experiments were conducted to evaluate our system’s reliability in assessing oral proficiency. In experiment 1, we collected a total of 340 interview sessions, 45-72% of which successfully elicited upper linguistic limit by adjusting the topic difficulty levels. In experiment 2, based on the ropleplay dataset of 75 speakers, the interactional speech elicited by our system was found to be as ratable as those by human examiners, indicated by the reliability index of interactional ratings. These results demonstrates that our system can elicit ratable interactional performances comparable to those elicited by human interviewers. Finally, we report on the deployment of our system with over 10,000 university students in a real-world testing scenario.

## introduction language testing plays a critical role in ensuring effective language learning, as it provides valuable feedback on learners' proficiency levels and guides instructional planning @xcite . assessment of oral proficiency is particularly important, as speaking and listening skills are essential for effective communication in a second language. traditional methods of oral proficiency assessment, however, face several challenges, including the subjectivity of human raters and the difficulty of creating standardized, scalable testing environments @xcite . to address these challenges, several studies have explored automated systems for oral assessment. for example, ockey and chukharev-hudilainen (2021) evaluated the potential of spoken dialogue systems (sds) for paired oral discussion tasks, concluding that a standardized assessment may favor sds over human interlocutors due to its systematic behavior. recent advancements in large language models (llms) have further simplified the implementation of such dialogue tasks. however, a significant challenge remains in obtaining "ratable" speech samples that accurately represent the full extent of a learner's language capabilities. assessment of oral proficiency requires not only measuring linguistic competence, such as grammar and vocabulary, but also evaluating interactional behaviours, including turn-taking, topic management, and repair strategies @xcite . additionally, to provide a reliable assessment, it is crucial to observe the upper linguistic limits of the user @xcite . therefore, an effective dialogue system must be capable of engaging users in a manner that naturally reveals these competencies while also being scalable as a testing tool. to this end, we developed the intelligent language learning assistant, intella, a multimodal dialogue system designed to elicit spontaneous speech samples from second language learners through a combination of a 15-minute interview and a 10-minute roleplay session. by dynamically adjusting the topic difficulty based on real-time assessments, the system aims to provoke linguistic breakdowns that serve as evidence of a learner's upper proficiency limits. additionally, the mixedinitiative roleplay component is designed to evaluate the user's interactional competence in a collaborative setting. to ensure the functionality and potential limitations of our system for large-scale real-world implementation, this paper reports two experiments: chapter 4 evaluates how well the system can assess oral proficiency through various experiments designed to test its efficacy; chapter 5 reports on the field testing results of our system used in real-world testing scenarios with university and high school students in japan. we also discuss our first year operation of our system in terms of practicality and social impacts. ## experiments to evaluate the system in terms of its capability of eliciting ratable speech samples, we conducted two experiments. the first experiment was designed to test the system's adaptive speech sampling strategy in system-led interview dialogues in terms of scoring accuracy as well as the frequency of target phenomenon, that is, linguistic breakdown. the second experiment was set up to gauge the quality of mixed-initiative interaction in roleplay tasks. given the multifaceted nature of interactional features, the second experiment aims at holistically evaluating the system using human experts' ratings, comparing the scoring reliability between humaninterlocutors and the current system. ## field testing we report on the system's performance and stakeholders' satisfactions in the real-world scenarios with university and high school students in japan. ## conclusions and future directions in this paper, we presented intella, a multimodal dialogue system designed for the assessment of oral proficiency. intella is designed to elicit ratable speech samples that display the user's full range of interactional skills. to enhance the ratability of speech elicited, the system is required to adaptively change the difficulty levels of questions to collect learners' linguistic breakdowns as the evidence of their upper limit of proficiency. to capture learners' ability to maintain collaborative conversations, the system is expected to enable mixed initiative interaction where learners need to engage with turn-taking management and topic development. to evaluate intella's usefulness in oral proficiency assessment, we conducted two experiments using interview and roleplay conversations. the results from the interview conversations demonstrated that our system consistently elicited ratable samples especially for lower-level learners, and automatically estimated scores based on those samples exhibited a high agreement with experts' ratings. in roleplay conversations, the ratability of the speech samples elicited through intella were comparable to experienced english teachers in terms of scoring reliability by human raters. the current study takes an interdisciplinary approach to integrating research on multimodal dialogue systems into real-world problems in language learning and testing. we demonstrated that intella is suitable for middle-stake assessments, effectively scaling to accommodate a large number of users. the scalability and accuracy position intella as a valuable tool for oral proficiency assessment in varied contexts. although our system showed the potential for reliable standardised assessment of oral proficiency, several challenges should be acknowledged to minimize possible negative consequences of its further real-world implementations @xcite . in the large-scale deployment of intella, quality of service (qos) @xcite ) factors, such as latency derived from the inference model and communication architecture, as well as the resolution of intella's rendering engine, are critical from the perspectives of quality of experience (qoe) @xcite and cost. future work is expected to evaluate how the configuration of the developed large-scale dialogue system impacts user engagement @xcite , considering a dialogue quality management system that can efficiently facilitate system improvements.