Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss (2025)

Abstract: Prior efforts in building computer-assisted pronunciation training (CAPT) systems often treat automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD) as separate fronts: the former aims to provide multiple pronunciation aspect scores across diverse linguistic levels, while the latter focuses instead on pinpointing the precise phonetic pronunciation errors made by non-native language learners. However, it is generally expected that a full-fledged CAPT system should perform both functionalities simultaneously and efficiently. In response to this surging demand, we in this work first propose HMamba, a novel CAPT approach that seamlessly integrates APA and MDD tasks in parallel. In addition, we introduce a novel loss function, decoupled cross-entropy loss (deXent), specifically tailored for MDD to facilitate better-supervised learning for detecting mispronounced phones, thereby enhancing overall performance. A comprehensive set of empirical results on the speechocean762 benchmark dataset demonstrates the effectiveness of our approach on APA. Notably, our proposed approach also yields a considerable improvement in MDD performance over a strong baseline, achieving an F1-score of 63.85%. Our codes are made available at https://github.com/Fuann/hmamba

## introduction in this era of globalization and technologization, computer-assisted pronunciation training (capt) systems have emerged as an appealing alternative to meet the pressing need for second language (l2) learning. in comparison with traditional curriculum learning, capt offers advantages in both time efficiency and cost-effectiveness. more critically, it shifts the conventional pedagogical paradigm from teacher-directed to self-directed learning, thereby providing a stress-free environment for l2 learners @xcite . in addition, capt applications have achieved marked success in various commercial sectors and testing services, such as the apps of duolingo @xcite and the speechrater @xcite developed by educational testing service (ets). typically, a de-facto archetype system for capt encompasses a "reading-aloud" scenario, where a non-native speaker is given a text prompt and instructed to pronounce it correctly. in this context, previous literature broadly divides applications of capt into two categories: automatic pronunciation assessment (apa) and ## conclusion in this paper, we have presented a novel hierarchical selective state space model (dubbed hmamba) for multifaceted capt application. extensive experimental results substantiate the viability and efficacy of the proposed method compared to several top-of-the-line approaches in terms of both the apa and mdd performance. in future work, we envisage mitigating the issue of data imbalance from an optimization perspective. in addition, another key area for future research involves tackling the assessment of open-response scenarios in capt.