An Effective Pronunciation Assessment Approach Leveraging Hierarchical Transformers and Pre-training Strategies (2024)

Abstract: Automatic pronunciation assessment (APA) manages to quantify a second language (L2) learner’s pronunciation proficiency in a target language by providing fine-grained feedback with multiple pronunciation aspect scores at various linguistic levels. Most existing efforts on APA typically parallelize the modeling process, namely predicting multiple aspect scores across various linguistic levels simultaneously. This inevitably makes both the hierarchy of linguistic units and the relatedness among the pronunciation aspects sidelined. Recognizing such a limitation, we in this paper first introduce HierTFR, a hierarchal APA method that jointly models the intrinsic structures of an utterance while considering the relatedness among the pronunciation aspects. We also propose a correlation-aware regularizer to strengthen the connection between the estimated scores and the human annotations. Furthermore, novel pre-training strategies tailored for different linguistic levels are put forward so as to facilitate better model initialization. An extensive set of empirical experiments conducted on the speechocean762 benchmark dataset suggest the feasibility and effectiveness of our approach in relation to several competitive baselines.

## introduction with the rising trend of globalization, more and more people are willing or being demanded to learn foreign languages. this surging need calls for developing computer-assisted pronunciation training (capt) systems, as they can offer tailored and informative feedback for l2 (second-language) figure 1: a running example curated from the speechocean762 dataset @xcite illustrates the evaluation flow of an apa system in the reading-aloud scenario, which offers an l2 learner in-depth pronunciation feedback. we call it bear. we call it bear. ## related work early studies on apa focused primarily on singleaspect assessments, typically through individually constructing scoring modules to predict a holistic ## conclusion in this paper, we have put forward a novel hierarchical modeling method (dubbed hiertfr) for multi-aspect and multigranular apa. to explicitly capture the relatedness between pronunciation aspects, a correlation-aware regularizer loss has been devised. we have further developed model pre-training strategies for our hiertfr model. extensive experimental results confirm the feasibility and effectiveness of the proposed method in relation to several top-of-theline methods. in future work, we plan to examine the proposed hiertfr model on open-response scenarios, where learners speak freely or respond to a given task or question @xcite . in addition, the issues of explainable pronunciation feedback are also left as a future extension.