Discourse on ASR Measurement: Introducing the ARPOCA Assessment Tool (2022)

Abstract: Automatic speech recognition (ASR) has evolved from a pipeline architecture with pronunciation dictionaries, phonetic features and language models to the end-to-end systems performing a direct translation from a raw waveform into a word sequence. With the increase in accuracy and the availability of pre-trained models, the ASR systems are now omnipresent in our daily applications. On the other hand, the models’ interpretability and their computational cost have become more challenging, particularly when dealing with less-common languages or identifying regional variations of speakers. This research proposal will follow a four-stage process: 1) Proving an overview of acoustic features and feature extraction algorithms; 2) Exploring current ASR models, tools, and performance assessment techniques; 3) Aligning features with interpretable phonetic transcripts; and 4) Designing a prototype ARPOCA to increase awareness of regional language variation and improve models feedback by developing a semi-automatic acoustic features extraction using PRAAT in conjunction with phonetic transcription.

## introduction automated speech recognition (asr) is the process of automatically detecting and recognizing the words that have been said in a sample of speech. asr has a wide variety of uses, such as voice assistants, automatic transcription, speech-to-text, and closed-caption generation. many recent asr models have been created using deep learning, with other methods including neural networks, hidden markov models, and gaussian mixture models @xcite . asr models are generally trained on a corpus, which is a collection of audio recordings. corpora are widely available for more common languages, such as english. however, they are either small or nonexistent for less common languages and dialects. this is due to the resources needed to construct a corpus and lack of available speakers. constructing a corpus involves gathering audio recordings from a variety of speakers and is a timeconsuming and costly process. as a result, less common languages remain under-resourced in the asr field. the performance accuracy will also vary with regional language variation and among different groups of users. asr performs especially poorly when given the task of recognizing the speech of nonnative speakers of a language, leading to model biases in common ai-assisted speech technologies @xcite . furthermore, there is a lot of variation in asr systems. in the last decade, the asr technology has evolved from probabilistic frameworks with hand-crafted features and pronunciation dictionaries to deep learning models in which features are extracted and learned in hidden layers @xcite . speech signals also consist of various components, such as acoustic, phonetic, and language-dependent, which jointly provide a representation of word sequences. while some features are interpretable by humans (e.g., place of articulation, vowel formants, pitch), others are the results of transformations and cannot be directly associated with any specific phonetic sound. finally, various evaluation systems are put forth to measure speech model accuracy @xcite . grapheme-based metrics (a written word) are commonly used to compare results, such as word error rate (wer). these measurement systems, however, are not able to diagnose whether phonetic errors resulted from a variation in pronunciation, speech boundary misalignment, noise, or the lack of sufficient data. this research is focused on existing asr evaluation systems and speech signal features used for training. we explore solutions for improving measuring performance metrics. our goal is to 1) develop a semi-automatic phonetic classification be-tween vowels and consonants as these classes are traditionally associated with different salient features (e.g., vowel formants, consonant intensity, aspiration), 2) help asr developers to identify improvement areas by focusing on specific feature engineering tasks, and 3) design an alternative evaluation system to encourage the asr research for less-commonly used languages by incorporating development cost, corpus size, and phonetic transcript as compared to a traditional word error rate evaluation metric. the paper is organized as follows. section 2 presents the overview of asr performance evaluation metrics, current asr models and corpora. section 3 describes the most common types of speech features and tools for their generation. in section 4, we present our proposed evaluation system ar-poca (assessment of asr using phonemes, originality, cost, and accent performance). finally, we provide our preliminary results in section 5, followed by conclusion and future direction. ## speech signal features feature extractions is a pre-processing task which transforms sound files into feature vectors that can be processed and analyzed by a computer. this tasks can be classified into two main groups: segment and suprasegmental prosodic features versus speaker-dependent and speaker independent features @xcite . while most of acoustic phonetics utilize interpretable features (e.g. vowel formant, duration, voice onset time) to describe phonemes (mental representation of sound) and phones (actual sounds), the asr field relies on transformed feature vectors optimized for machine learning tasks (e.g. linear prediction and mel-frequency coefficients). ## arpoca approach in response to the problems previously identified in the field of speech recognition, this proposal aims to develop a more in-depth evaluation system called arpoca. arpoca is an acronym for assessment of asr using phonemes, originality, cost, and accent performance. the main goal is to develop a phoneme recognition system using phoneme classification and transcription, independent from a grapheme representation used in wer. first, we selected open source existing tool praat, a software designed for sound processing @xcite , to extract interpretable feature representations for each phoneme. second, we identified the following salient features for phoneme classification: frequency formants, dispersion (also called standard deviation), center of gravity, and intensity. standard formant ranges for f1, f2, f3 are used to identify vowels. dispersion, center of gravity, and intensity are used to identify consonants. center of gravity measures at what frequency a sound is most concentrated, while dispersion measures how widely the frequencies of a sound are spread. intensity measures the loudness of a sound in decibels. for testing, we obtained a non-transcribed free sample spanish audio corpus @xcite . in our next stage, we will create a manual phonetic transcription of utterances from the corpus, in addition to segmenting and labeling the utterances for usage in praat. we will collect information about expected values of acoustic features used for identifying phonemes and compare our manual phonetic transcription with the output from an available speech recognizer library in python. in addition, we will analyze several existing models to establish a baseline for originality and cost in these models, and use this to create a rating system. furthermore, the phoneme recognition system will incorporate an accent performance analysis. that is, the phoneme recognition system will identify whether a model has a wide pronunciation gap and identify particular areas where a model struggles, which will help close the accent gap. ## preliminary results in the first stage of this proposal, we are exploring features extracted from spectogram and speechwave. fig. 2 displays an example of spanish word 'necesito' (i need). the sound waves help distinguish between sound and silence, amplitude and intensity of sounds, while the spectrogram provides a view of formant frequencies, consonants obstruction and frication. while praat includes scripting, using python in addition makes running the praat script easier to automate, especially for large amounts of audio samples. python code calls a praat script, then performs additional operations on the results of the praat script, such as matching the formants to the correct phoneme class. since consonants cannot be identified using formants only, we use different measurements, including center of gravity, intensity, and standard deviation. currently, using these measurements is only precise enough to differentiate between fricative and non-fricative consonants. more work must be done to refine the expected ranges for consonants to be able to identify individual consonants. originality is another aspect of arpoca. we have determined that a scoring rubric would likely be best to assess originality, since there is little research on this topic. thus, research that is an addition or improvement on an existing model will receive a lower score than more novel research. cost is also an important element of arpoca. preliminary research suggests that a budget of @xmath0500,000 would score the highest, with score decreasing as total cost increases. the reason for this is twofold. firstly, there are many applications that require smaller, less costly models. for instance, such models could be used to assist people with hearing loss by providing real-time transcriptions. secondly, many costly models with large corpora already exist and are prioritized under the prevalent measurement system of wer. therefore, in order to encourage innovation in the field of asr, smaller, less costly models will be encouraged. there are several important outcomes from the preliminary results. in its current state, the phoneme recognizer is unlikely to work with english, due to the presence of a large number of vowels which are not easily distinguishable. the phoneme identifier has been tested using spanish, which is better suited to this purpose due to the smaller number of vowels, which are relatively easy to distinguish. an additional flaw in the phoneme identifier is its difficulty distinguishing between vowels and voiced consonants. ## conclusion and future work the objective of this work is to supplement asr models and developers with an additional tool providing not only a feedback but also more interpretable representation of sound models via phonetic transcription. such feedback could include highlighting phonemes that have been consistently misidentified and/or measuring performance of the model when given audio samples produced by nonnative speakers, which is an area in which asr models typically struggle. this feedback could improve the accuracy of asr models and lessen the accent gap. accuracy of models could also be improved by providing developers more feedback on their models than just using standard performance metrics. for instance, commonly mistaken sounds (phonemes) could be used as a form of feedback to help improve models and augment existing corpora. furthermore, a phonetic approach could help create dictionaries with dialectal variation (regional alternative pronunciation) that can be added to training corpora. finally, language transfer (using the resources from one language to develop resources in another similar language or dialect) could help provide resources for underrepresented spoken languages. arpoca needs more development in order to become more accurate. this could include additional data for improving the cost baseline and grading in addition to more research into expected values of formants, center of gravity, intensity, and dispersion. in its current state of research, ar-poca serves as a proof of concept for the development of a more robust assessment tool for asr models. we envision arpoca being used in settings such as peer reviews and conferences to promote discussion and improvement of asr models. arpoca can aid in supporting different research goals than wer. for instance, a model with a smaller corpus typically costs less to produce and would therefore score better in the cost section of wer. this could encourage the production of models for under-resourced and less widely spoken languages, even if such models do not immediately have a good enough wer score to compete with models for languages such as english. another possible benefit of using arpoca is closing the accent gap. although the accent performance analysis system has not been developed yet, the existing phoneme identification could help developers determine if there are specific groups of formants that a model has misidentified. on the other hand, ar-poca must be carefully revised to ensure that the scoring system is fair and accurate. if there are inaccuracies in arpoca or top scores are unattainable, this could result in a variety of unwanted outcomes, such as giving models the wrong scores or discouraging developers. in addition, while arpoca has been developed with collaboration and discussion in mind, it has the possibility to fuel competition as well, due to its role as a tool for assessment. therefore, arpoca must be used with care and consideration as to whether its use is appropriate for a given situation.