Automatically Detecting Reduced-formed E nglish Pronunciations by Using Deep Learning (2022)

Abstract: Reduced form pronunciations are widely used by native English speakers, especially in casual conversations. Second language (L2) learners have difficulty in processing reduced form pronunciations in listening comprehension and face challenges in production too. Meanwhile, training applications dedicated to reduced forms are still few. To solve this issue, we report on our first effort of using deep learning to evaluate L2 learners’ reduced form pronunciations. Compared with a baseline solution that uses an ASR to determine regular or reduced-formed pronunciations, a classifier that learns representative features via a convolution neural network (CNN) on low-level acoustic features, yields higher detection performance. F-1 metric has been increased from 0.690 to 0.757 on the reduction task. Furthermore, adding word entities to compute attention weights to better adjust the features learned by the CNN model helps increasing F-1 to 0.763.

## introduction the term "reduced forms" refers to the phenomenon of phonological simplification and variation commonly observed in connected speech of native speakers @xcite @xcite . in phonetic reduction, "segments may be shorter, less clearly articulated, or absent compared to canonical or dictionary forms" @xcite . reduced-formed pronunciations appear in daily english communication among native speakers @xcite . on one hand, the existence of reduced form challenges second language (l2) learners in their listening comprehension @xcite . on the other hand, l2 learners often face great challenges on producing reduced forms so that they can sound 0 the work was conducted while lei, jiahong, and yang were working in laix silicon valley ai lab more close to native speakers. english as a second language (esl) teachers have realized the importance of specially training l2 learners on understanding reduced form pronunciations to improve their listening comprehension skills. @xcite is such an effort of designing an app to train students accordingly. compared to the emphasis on reduced forms when training the listening comprehension skills, the effort on training the specific skills on the production of reduced forms is still limited. most computer aided language learning (call) tools focus on training regular form pronunciations and do not provide adequate supports to l2 learners' demands on the production of reduced forms. hence, in this paper, we will report on our initial effort of using a deep learning based classification method to detect l2 learners' reduced form productions. note that the detection is the first required step for creating a training tool that can generate feedback and provide interventions to cultivate l2 learners' specific skills. ## previous research reduced forms have been actively investigated in phonetics. for example, @xcite introduced reduced-formed pronunciation variant phenomenal in phonetics. it pointed out that such variations are quite common in different languages in their casual conversation conditions. @xcite investigated english function words' reduced forms in the switchboard corpus and found that a high percentage of reduced forms appears in the telephone conversations. also, the authors investigated possible reasons causing reduced forms, such as words' frequencies. @xcite examined the role of the perception of reduced forms (e.g., contraction, elision, assimilation) of english words in connected speech comprehension and the phonological skills underpinning reduced forms perception. this study delivers a clear message to esl teachers that the ability of perceiving reduced pronunciation variants is important for l2 listening comprehension skills. there are some emerging technical works on helping l2 learners' perception of reduced forms. for example, @xcite reports on an android app to use authentic native connected speeches as material to teach. reduced forms sometimes are produced as variants to formal forms. based on this fact, methods that can distinguish pronunciation variants in asr can be used to detect the existence of reduced forms. @xcite systematically surveyed the methods for recognizing pronunciation variants. in a widely used approach, extended recognition network (ern) @xcite , extra decoding paths are added to represent pronunciation variants on top of the regular paths built on formal forms' pronunciations. however, reduced forms can sometimes occur without pronunciation variants. hence, solutions working on broader cases are worth investigating. ## data in this paper, we focused on the two types of reduced forms, i.e., reduction and liaison. the first refers to changing pronunciation from its formal form on individual words while the second term refers to co-articulation among adjacent words. in 2019, company-x released a new product for training various specific pronunciation skills, including reduction and liaison, in its main english learning mobile app. this product has already been used by a large number of chinese english learners. from the audio samples collected in this product, we built up our own research data set. we sampled speech files from a large group of english learners from different locations in china. when sampling l2 learners' spoken responses, we used pronunciation scores automatically rated by company-x ai-based pronunciation scoring system to include learners from diverse levels. on 8, 570 practice audio samples for the reduction skill, seven human raters annotated whether learners produce correct reductions or not on the required words. during rating, the annotators considered three aspects, including energy, rhythm (duration and its connection to context words), and pronunciation variations. these raters are high-level non-native english speakers and doing linguistics and phonetics annotation as their full-time jobs. for words (including single words or multi-word with reduced forms, e.g. "got you" to "gotcha") with pronunciation variations, a typical solution is using an asr system with two distinct pronunciation entries in its dictionary to determine real pronunciation on the fly. for example, "are" can be pronounced as its formal form /ar/ or its reduced form /r/. here, we first use a forced alignment step to locate words being considered. then, a recognition network with paths mapping to two pronunciation forms is used to decode the audio portion to find the more possible path. figure 3 shows one concrete example of a decoding network for the word "are". the automatic speech recognition (asr) system was built with the kaldi open-source toolkit. this model is a 9-layer time delayed neural network (tdnn) using mel frequency cepstrum coefficient (mfcc) acoustic features from the current frame plus the previous and following 5 context frames. the asr model was trained on our in-house readaloud corpus containing about 2, 500 hours of native and non-native speech files. the asr system achieved a word error rate (wer) of 9% on learners' speech. ## discussion reduced forms are commonly used by native speakers in their casual conversations. because l2 learners mostly face formal forms in their language learning, perception and production of reduced forms in fact greatly challenges learners' listening comprehension and speaking capabilities. with a goal of building a training application on producing reduced-formed pronunciations, we conducted a research on automatically detecting reduced forms with a high accuracy. following on the work of recognition of pronunciation variants, we firstly utilized an asr decoding method to distinguish formal vs. reduced forms. to cope with reduced forms without obvious pronunciation variations, we then explored using a cnn model to learn distinguishable features from learner speech directly. our experiment results show that cnn method has improved performance over the asr decoding method. using word entities in the cnn model to compute attention weights to adjust the learned features is proven to be useful. overall, on the two reduced form types, i.e., reduction and liaison, our cnn model has f-1 measurement about 0.763 and 0.720 respectively. we envision that there are several research directions in future. firs, so far, we only used cnns to encode mfcc feature sequence, it is worthwhile trying some new encoding method, like transformer. second, human annotation on reduction still have a room to improve. we are hoping to continue increasing rating agreement to provide a even more solid research base. at last, a training module has been added into laix liulishuo app. based on real user data, it is worthwhile tracking whether provided training helps on learners' mastery of reduced forms.