Automatic Pronunciation Assessment - A Review (2023)

Abstract: Pronunciation assessment and its application in computer-aided pronunciation training (CAPT) have seen impressive progress in recent years. With the rapid growth in language processing and deep learning over the past few years, there is a need for an updated review. In this paper, we review methods employed in pronunciation assessment for both phonemic and prosodic. We categorize the main challenges observed in prominent research trends, and highlight existing limitations, and available resources. This is followed by a discussion of the remaining challenges and possible directions for future work.

## introduction computer-aided pronunciation training (capt) technologies are pivotal in promoting self-directed language learning, offering constant, and tailored feedback for secondary language learners. the rising demand for foreign language learning, with the tide of globalization, fuels the increment in the development of capt systems. this surge has led to extensive research and development efforts in the field @xcite @xcite . capt systems have two main usages: (i) pronunciation assessment, where the system is concerned with the errors in the speech segment; (ii) pronunciation teaching, where the system is concerned with correcting and guiding the learner to fix mistakes in their pronunciation. this paper addresses the former -focusing on pronunciation assessment, which aims to automatically score non-native speech-segment and give meaningful feedback. to build such a robust pronunciation assessment system, the following design aspects should be addressed. modelling mispronunciation detection and diagnosis (mdd), in many cases, are more challenging to model compared to the vanilla automatic speech recognition (asr) system, which converts speech into text regardless of pronunciation mistakes. robust asr should perform well with all variation including dialects and non-native speakers. however, mdd should mark phonetic variations from the learner, which may sometimes be subtle differences @xcite . training resources recent success in deep learning methods emphasized the need for in-domain training data. language learners can be divided into two groups: adult secondary (l2) language learners and children language learners -the former depends on whether to build a system that is native language dependant (l1). at the same time, the latter identifies the need for children's voice, which is a challenging corpus to build @xcite , even the accuracy for asr for children is still behind compared to adult asr @xcite . the scarcity and imbalanced distribution of negative mispronunciation classes pose a significant challenge in training data. evaluation there is no clear definition of right or wrong in pronunciation, instead an entire scale from unintelligible to native-sounding speech @xcite . given that error in pronunciation is difficult to quantify, it can be split into (a) objective evaluations -(i): phonetic or segmental; (ii): prosodic or supra-segmental; and (iii) place or articulation, manner of speech or sub-segmental; (b) subjective evaluations; in many cases measured through listening tasks followed by human judgment, and can be split into three main classes: (i) intelligibility; (ii) comprehensibility and (iii) accentedness (or linguistic native-likeness). see figure 1 for common pronunciation assessment factors. several studies have summarized advances in pronunciation error detection @xcite @xcite @xcite @xcite . @xcite the potentials and limitations of asr for l2 pronunciation assessment, showcasing its practical implementation using an interface developed at cmu. furthermore, the study reports different automatic scoring techniques, emphasizing modalities of interaction, associated algorithms, and the challenges. @xcite presented an overview of pronunciation error detection, encompassing various scoring methodologies and assessing commercial capt systems. @xcite provided a research summary, focusing on phoneme errors and prosodic error detection. more recently, @xcite provided a summary of two automatic scoring approaches (a) asr-based scoring to calculate confidence measures; and (b) acoustic phonetics scoring focusing on comparing or classifying phonetic segments using various acoustic features. with large transformer-based pre-trained models gaining popularity, re-visiting the existing literature and presenting a comprehensive study of the field is timely. we provide an overview of techniques adapted for detecting mispronunciation in (a) segmental space, (b) assessing pronunciation with supra-segmal measures, along with (c) different data generation/augmentation approaches. unlike previous overview studies, we also cover a handful of (d) qualitative studies bringing together the notion of intelligibility, comprehensiveness, and accentedness. we note the resources and evaluation measures available to the speech community and discuss the main challenges observed within prominent research trends, shedding light on existing limitations. additionally, we also explore potential directions for future work. ## nuances of pronunciation pronunciation can be defined as "the way in which a word or letter is said, or said correctly, or the way in which a language is spoken" . foot_0 compared to other language skills, learning pronunciation is difficult. yet, for learners, mastering l2 pronunciation is most crucial for better communication. historically, pronunciation errors (mispronunciations) are characterized by phonetic (segmental) errors and prosodic (supra-segmental) errors @xcite , as represented in figure 1 . this characterization provides some clear distinctions for pronunciation assessment. ## datasets obtaining datasets for pronunciation assessment is often challenging and expensive. most of the available research work focused on private data, leaving only a handful of publicly accessible data to the research community. ## research avenues in this section, we will delve into diverse approaches, old, revised, and current methodologies used for pronunciation modeling of both segmental and supra-segmental features, as illustrated in figure 2 and figure 3 . ## evaluation metrics phoneme error rate (per): is a common metric used in the md evaluation, measuring the accuracy of the predicted phoneme with the humanannotated sequence. however, per might not provide a comprehensive assessment of model performance when mispronunciations are infrequent which is the case for md datasets. hierarchical evaluation structure: the hierarchical evaluation structure developed in @xcite , has also been widely adopted in @xcite ; el kheir et al., 2023a) among others. the hierarchical mispronunciation detection depends on detecting the misalignment over: what is said (annotated verbatim sequence); what is predicted (model output) along with what should have been said (text-dependent reference sequence). based on the aforementioned sequences, the false rejection rate, false acceptance rate, and diagnostic error rate are calculated, using: as a result, we can calculate the false rejection rate (frr) that indicates the number of phones recognized as mispronunciations when the actual pronunciations are correct, false acceptance rate (far) that indicates phones misclassified as correct but are actually mispronounced, and diagnostic error rate (der) using the following equations: precision, recall, and f-measure are also widely used as the performance measures for mispronunciation detection. these metrics are defined as follows: pearson correlation coefficient: pcc is widely used to measure the relation of the predicted score of fluency, stress, and prosody with other suprasegmental and pronunciation constructs with subjective human evaluation for pronunciation assessment. the human scores are typically averaged across all annotators to provide a comprehensive score. ## challenges and future look there are two significant challenges facing advancing the research further: (1) the lack of public resources. multilingual: recent advancements in end-toend asr enabled the development of multi-lingual code-switching systems @xcite @xcite . the great progress in ssl expanded asr capabilities to support from over 100 @xcite , to over 1, 000 @xcite languages. traditional research in pronunciation assessments focused on designing monolingual assessment systems. however, recent advancements in multilingualism allowed for the generalization of findings across different languages. @xcite explored the adaptation of pronunciation assessments from english (a stress-timed language) to malay (a syllable-timed language). meanwhile, @xcite investigated the use of languagespecific embeddings for diverse languages, while optimizing the entire network within a unified framework. children capt: there is a noticeable imbalance in research between children learning pronunciation research papers, for example, reading assessments, compared to adults' l2 language learning. this disparity can be attributed to the scarcity of publicly available corpora and the difficulties in collecting children's speech data. dialectal capt: one implicit assumption in most of the current research in pronunciation assessment is that l2 is a language with a standard orthographic rule. however, cases like dialectal arabic -which is every arab native language, there is no standard orthography. since speaking as a native is the ultimate objective for advanced pronunciation learning, there is a growing demand for this task. ## conclusion this paper serves as a comprehensive resource that summarizes the current research landscape in automatic pronunciation assessment covering both segmental and supra-segmental space. the paper offers insights into the following: