An Automatic Vowel Space Generator for Language Learner Pronunciation Acquisition and Correction (2020)

Abstract: Speech visualisations are known to help language learners to acquire correct pronunciation and promote a better study experience. We present a two-step approach based on two established techniques to display tongue tip movements of an acoustic speech signal on a vowel space plot. First we use Energy Entropy Ratio to extract vowels; and then we apply Linear Predictive Coding root method to estimate Formant 1 and Formant 2. We invited and collected acoustic data from one Modern Standard Arabic (MSA) lecture and four MSA students. Our proof of concept was able to reflect differences between the tongue tip movements in a native MSA speaker to those of a MSA language learner. This paper addresses principle methods for generating features that reflect bio-physiological features of speech and thus, facilitates an approach that can be generally adapted to languages other than MSA.

## introduction second language (l2) learners have difficulties in pronouncing words as well as native speakers @xcite which can create inconveniences in social interactions @xcite . difficulty in providing pronunciation instructions by language teachers add extra challenges on l2 pronunciation training and corrections @xcite . one solution to assist pronunciation acquisition is through the adoption of educational software applications @xcite . a well-designed language educational software can provide straightforward guidance to correct l2 pronunciation through multiple information sources. one instance of auxiliary systems is pronunciation learning aid (pla), which supports language students towards nativelike pronunciation in a target language @xcite . pla achieves this via evaluating students' produced speech to reflect their pronunciation status. another instance of auxiliary systems is visual cues, which serves as a friendly and accessible form of feedback to language students @xcite . through combining language lecturers' teaching with auxiliary systems, our aim is to assist students in both a classroom setting and in their individual practices. we present a prototype system that displays visual feedback on tongue movements to assist language learners to acquire correct pronunciation in the process of l2 studying. we have adopted a human-centred approach for the development of the system using a design-oriented perspective through applying a methodology that draws from design science research (dsr) @xcite and design thinking (dt) @xcite . unlike machine learning methods, which train deep neural networks to predict articulatory movements @xcite , our proposed system uses vowel space plots based on bio-physiological features to help visualise tongue movements. in this present work, we introduce a versatile prototype of our vowel space plot generator to address these challenges for students primarily learning msa. our design aims to allow l2 beginner learners to quickly visualise their status of pronunciation compared to those by their language teachers. we provide a reference vowel space plot adjacent to the students' own plots to reflect clear differences to support self-corrections. the envisioned applicability ranges from in-class activities to provide immediate and personalised suggestion to remote learning where in both cases glossary files are preuploaded by teachers or textbook publishers. ## related work traditional acoustic plots, such as waveforms, spectrograms, and other feature plots are applied to vi-sualise speech signals and can provide sufficient information to phoneticians, expert scientists, and engineers @xcite . however, these methods fall short in providing straightforward suggestions for improving language students' pronunciation or otherwise lack an intuitive and userfriendly graphic user interface @xcite . a study proposed by @xcite adopted the combination of waveform and highlighting syllables to visualise pronunciation in esl studying shows using acoustic plots to support pronunciation acquisition is an implementable method. different from acoustic plots, another thinking of pronunciation visualisation was considered based on people's bio-physiological features. a pioneer study with this idea was introduced by @xcite , in which they discussed the effect of increasing the amount of visible articulatory information, such as non-visible articulatory gestures, on speech comprehension. with the improvement of equipment, ultrasound imaging, magnetic resonance imaging (mri), and elec-tromagnetic articulography (ema) can be alternative approaches to visualise the movement of articulators, and several study cases on pronunciation visualisation were implemented by @xcite , @xcite , and @xcite . however, these approaches are still difficult to be implemented in daily language studying since relevant equipment are often not available for in-class activities and self-learning, and generated images and videos are hard to be understood by ordinary learners. enlightened by imaging the movement of articulators, the idea of talking head, which is using 3d mesh model to display of both the appearance articulators and internal articulators, was introduced. some of the fundamental works of talking head were completed by @xcite , and serrurier and @xcite . with the techniques of articulatory movement prediction, such as gaussian mixture model (gmm) @xcite , hidden markov model (hmm) @xcite , and popular deep learning approach @xcite . although talking head is developing swiftly, the research about performance of talking head for pronunciation training is still insufficient. the place and manner of articulation are well established variables in the study of speech production and perception (e.g. @xcite . early research has already realised the potential of using vowel space plots to achieve pronunciation visualisation, such as the studies by @xcite and @xcite . these studies indicate that for language learners, vowel space plots are easy-to-understand, straightforward, and provide the necessary information for understanding their own tongue placement and movement. therefore, vowel space plots are considered a useful tool for language learners to practice and correct their pronunciation relative to other pronunciation correction tools, such as ultrasound visual feedback or more traditional pedadogical methods like explicit correction and repetition. ## the proposed approach to visualise the tongue movement based on students' pronunciation practice, our proposed system needs to receive students' pronunciation audio signal as its input. after the process of vowel detection, vowel extraction, and formant estimation, the system can automatically generate the corresponding vowel space plot as its output. in this section, we will introduce how engineering and linguistics insights inspired our proposed method, and the details of audio signal processing procedures. ## preliminary evaluation experiment we conducted two experiments to evaluate the performance of our prototype. first, we invited a native arabic speaker who is a modern standard arabic (msa) lecturer at the australian national university (anu) to provide a glossary of msa lexicon and their corresponding utterances. these utterances constituted the gold-standard or target pronunciation for users. then, we invited four msa language students to use our prototype by pronouncing four msa words. for each lexical item pronounced, the articulation was visualised on a vowel space plot so users can compare their pronunciation alongside the native-like, target pronunciation of their lecturer. following this visual comparison, users were prompted to pronounce the same word again. in the experiments, we want to verify the feasibility and accessibility of our prototype. the feasibility of our prototype was determined by whether the interpretation of the comparison plots in the first instance supported improved pronunciation of the same word in subsequent iterations. and the accessibility refers to whether our prototype can provide implementable and correct feedback for learners to visualise their pronunciation. ethical approval (2018/520) was obtained from the human research ethics committee of the australian national university. each study participant provided written informed consent. ## results and discussion we used collected speech signals to test the feasibility and accessibility of our prototype. to test the feasibility, we fed the standard references to our prototype and verify whether the output vowel space plot can reflect the correct tongue motion of the corresponding word. as for accessibility, we used the student test data and generated the vowel space plot, and then found corresponding words from a standard reference and compare these two vowel space plots. an ideal result is the student test data can reflect the student's tongue motion, and the student can find how to improve the pronunciation by compare these two vowel space plots. with the vowel space plots of the same words from student test data and standard reference, we compared the corresponding plots to see if the corresponding plots and if the vowel space plots can provide useful feedback on pronunciation correction. in this paper, we display the msa word "soap" ( , /s . ābūn/) as an example of our results. ## conclusion this paper presented the initial proof of concept that used vowel space plots to enhance language learning in second languages. the idea of our prototype was based on our early stage dsr process and msa language student survey (chao, 2019). our prototype was designed to generate clear visual feedback from speech input, and it was tested to assist the pronunciation of l2 msa beginners. our main contribution is the vowel space plot generator prototype which produces easily understandable visual cues from analysing the biophysiological features of user speech. our prototype is hence user-friendly for improving language learner pronunciation. to gain evidence of our prototype being effective on assisting language learners' pronunciation training, we designed an experiment to test at the vocabulary level the feasibility and accessibility of the prototype and invited language students to provide their audio data for experimental use. also, according to students' feedback, we proposed a series of future developments that are described in the next section. one limitation of our presented work is that there was no re-testing of pronunciation after the students received feedback from the system to check that their pronunciation improved. we plan to deploy re-tests as mentioned in our next stage experiments 7 future work in the future, we aim to build on this current work to verify and quantify the pronunciation improvements gained from each user. this will help us to understand the effectiveness of this current design of the prototype and enable us to select appropriate extensions to enhance l2 learning experiences. we are currently considering to build a correction subsystem for pronunciation practice. in addition to the existing vowel space plots, we theorise that it would be helpful to construct a system that could directly compare our users' speech to a set of externally stored standard references. this should enable the users to correct their pronunciation with higher precision and efficiency. such a design could also potentially provide personalised pronunciation assistance via analysing user-specific pronunciation patterns. future iterations also intend to test a much more varied selection of msa words that capture both short and long vowels in word initial, medial and final positions, as well as the two msa dipthongs /aw/ (e.g. /d . aw/ 'light') and /aj/ (e.g. /bajt/ 'house') and msa consonant. another potential future direction is to animate the tongue motion. @xcite showed that such animations could achieve better results than their static counterparts. we expect the animated version of the vowel space plot to display tongue motions while people speak to help users to better conceptualise pronunciation in real-time.