Investigating Phoneme Similarity with Artificially Accented Speech (2023)

Abstract: While the deep learning revolution has led to significant performance improvements in speech recognition, accented speech remains a challenge. Current approaches to this challenge typically do not seek to understand and provide explanations for the variations of accented speech, whether they stem from native regional variation or non-native error patterns. This paper seeks to address non-native speaker variations from both a knowledge-based and a data-driven perspective. We propose to approximate non-native accented-speech pronunciation patterns by the means of two approaches: based on phonetic and phonological knowledge on the one hand and inferred from a text-to-speech system on the other. Artificial speech is then generated with a range of variants which have been captured in confusion matrices representing phoneme similarities. We then show that non-native accent confusions actually propagate to the transcription from the ASR, thus suggesting that the inference of accent specific phoneme confusions is achievable from artificial speech.

## introduction automatic speech recognition (asr) systems, while achieving high levels of performance on usaccented english, still struggle to handle accents for which they have not been trained @xcite . thus, accent robustness is an important challenge for the field of speech recognition, especially since such systems have become widespread and are used worldwide. various approaches have been tried to build accent-robust asr systems. the most straightforward one, building accent-specific models, is limited because of the low availability of data for most accents which are mostly not well sourced. the lack of sourced data for training and testing makes the task of recognising accented speech extremely difficult. this lack of data is mainly due to the wide diversity of accents (native and non-native) leading to the complexity of recording enough examples for each, and the difficulty of accurately labelling and transcribing speech data. some attempts to overcome both lack of data and accent robustness have been proposed. these include multi-task training @xcite @xcite , features adaptation @xcite or adversarial training @xcite . however, these methods do not completely solve the problem of the lack of data, as data would still be needed for testing. instead, generating artificial speech data seems promising, as data augmentation has been proven to be efficient for improving the recognition of accented speech @xcite , and the use of artificial data has been around for some time @xcite . this paper investigates the extent to which artificial speech data can be used to infer accent-related phoneme confusions. we do this by using an offthe-shelf speech synthesis system, in this case microsoft azure tts foot_0 , to synthesise artificially accented speech data and then using the wav2vec 2.0 asr @xcite , to produce a confusion matrix for this data. this matrix is then been examined and compared to other confusion matrices, in order to evaluate its relevance in representing a particular accent. in this paper, we focus on nonnative accents, although the same study could have been applied to native accents. the remainder of the paper is structured as follows. section 2 discusses related work. section 3 describes the process of generating accent related phoneme confusions for artificial accented speech. in sections 4 and 5, we compare the confusions obtained with alternative methods and discuss the extent to which text-to-speech systems can capture accent related phoneme confusions. ## related work recent approaches for automatic speech recognition use end-to-end deep neural networks,(e.g. ctc-based, transformer-based and attention-based models) and have been really successful for this task. commercial options exhibit extremely high performance; however, none of them achieve the same performance on accented speech. attempts to improve end-to-end asr performance on accented speech have had mixed results, and rely mainly on the training process. indeed, the complexity of these architectures makes the understanding of the actual learning process difficult, if not impossible, and leads to an increasing need for explainability. this challenge has been the focus of a number of studies. @xcite highlight the link between linguistic representations of speech and deep learning representation clusters. @xcite look to investigate in more detail the utility of attention layers, which is used in recent asr systems. in the test community, @xcite have proposed a testing framework for asr systems. the work presented in this paper aligns with the goals of these approaches. accents are defined as variation in phoneme realisation due to several factors such as geographical location. in the case of non-native accents, which is the focus of this paper, the differences in pronunciation compared to the native language (l1) come mainly from the differences that exist between the phonetic rules of the native language and those of the target language (l2) @xcite . thus, many pronunciation difficulties are due to phonological transfer -which involves applying l1 rules to l2 pronunciation -are linked to the non-existence of certain l2 characteristics in the l1, and result from discrepancies between the phonetic systems of the two languages. these challenges may include difficulties in producing and perceiving specific segmentals (olsen, 2012) -like phonemes, consonant clusters, vowels -or suprasegmentals @xcite ) -like stress patterns, rhythm and intonation patterns -that are present in the l2 but absent or different in the l1. thus, non-native speakers commonly tend to approximate the pronunciation of phonemes which do not exist in their native language, by known ones they perceive as similar, as showed by @xcite . for instance, pronouncing the english phoneme [d] -corresponding to the grapheme sequence "th" as in "those" -as the french phonemes [z] or [d] is common amongst french people when speaking english @xcite , since [d] is not a phoneme of @xcite . while this is a very simplified version of the concept of accent, which does not include phenomena such as prosodic or phonotactic constraints, we focus in this paper on that definition of an accent, i.e. as the replacement of l2-but-not-l1 phonemes by l1 phonemes. this paradigmatic definition is intended to evolve into a more complete definition to include the syntagmatic and suprasegmental aspects in future work. in order to understand the way in which nonnative speakers switch from a phoneme of the target language (l2) to another phoneme of their native language (l1), we need to characterise phonemes and define what similarity between phonemes means. several phonetic-based feature systems have been proposed to describe the specific phonemes of a language. chomsky and halle (1968) proposed a system to analyse the phonological structure of a language from a generative perspective. they described phonemes through binary features, organised along major features (that distinguish vowels from consonants), place of articulation, manner of articulation and source features (like voicing). since then, multiple phonological feature sets have been proposed and have been used to capture similarities between phoneme classes. this description of phonemes with features allow us to calculate their similarity using distance metrics such as jaccard index (as defined in equation 1, the jaccard index between two sets u and v ), that can easily be used as a similarity measure between phonemes, assuming that they are represented by their binary features. however, while it is a simple similarity to implement as baseline for the work presented in this paper, this measure is not satisfactory in the sense that all features have the same weight and, therefore, it does not take into account the difference in distance between the phonetic realisation of two features. furthermore, it is only an a priori knowledge-based similarity, that does not necessarily follow the real-world realisations of phonemes. while @xcite argued that knowledge-based feature based measures are better at predicting similarity, data driven techniques offer new opportunities to identify confusions and similarities. as an example of a data-driven approach, @xcite built a confusion matrix over the timit @xcite dataset, which contains recordings of 8 major us-english dialects. they created what they call an enhanced confusion matrix, by excluding an acoustic model iteratively, in order to restrict the recognition process and identify what phonemes are recognised in place of those that are missing from the model. this process ends up with a lot more confusions for each phoneme, thus retrieving more similarities. they found that this confusion matrix corresponds better to theoretical expectations. furthermore, phoneme embeddings have been used as the basis of data-driven similarity, in the context of sound analogies @xcite , for determining allophonic relationships @xcite and for capturing distributional properties (o'neill and carson-berndsen, 2019). ## introduction of non-native variations the overall method presented in this paper for synthesising accented speech consists, broadly, of 1) transforming texts into phoneme sequences, 2) applying variations to the phoneme sequence according the target accent, and 3) synthesising speech from the phoneme sequence using a text-to-speech (tts) engine. this workflow is illustrated in figure 1 and is referred in the remainder of the paper as "variation method". the core of this accented speech synthesis lies in the way we choose and apply variations to the phoneme sequence. this is done by 1) selecting the phonemes to vary using a mapping between the phonemes of the different languages -this mapping is called the phonetic compatibility matrix, and 2) varying the selected phonemes by replacing them with their nearest neighbour phonemes in terms of similarity. this mimics the way non-native speakers adjust to the target language pronunciation. these replacements could be regarded as mispronunciations. the construction of the phonetic compatibility matrix is very straightforward. it is built as a boolean matrix, associating the different languages with their phonemes, the values being 1 if the phoneme exists in the target language, and 0 otherwise. when applied, the variation method replaces the incompatible phonemes (i.e. the english phonemes identified in the phonetic compatibility matrix as not existing in the target language) with their nearest neighbour (that is with the higher similarity, or smallest distance to the original phoneme) in the similarity matrix, amongst the phonemes that exist both in english and in the target language. as we saw in the related work, the similarity between phonemes can be defined in several ways. in this paper, we will briefly introduce three different methods we used for building the similarity matrix, with a focus on similarity identification using artificial accented speech data. thus, the next two subsections explore these methods for defining a similarity matrix, which can be separated into two paradigms: knowledge-based and data-driven. ## artificial speech confusions as introduced in the previous section, dd2 method has three stages: 1) generating artificial frenchaccented speech by using an off-the-shelf tts system, 2) generating the recognition confusion matrix using an asr system, and 3) introducing variations in speech, as per the variation method (see figure 1 ), by using the previously obtained confusion matrix as the so-called similarity matrix for choosing the phonemes to vary. the generation of artificial french accented speech is done by providing text inputs (i.e. textual sentences from timit dataset) to the microsoft azure tts, with its two parameters language set to english and voice set to one of the azure french voices: fr-fr-deniseneural or fr-fr-henrineural. this configuration allows the tts to synthesize the english sentences with a french pronunciation, that is reading the sentences as if they were written in french. at the end of this process, we end up with a set of artificially accented speech audios. then, the second step is the generation of the french confusions. for obtaining that matrix, we use wav2vec 2.0 asr with a subsequent grapheme to phoneme mapping and we align the phoneme sequences with the original ones obtained from timit. the confusion matrix created from these alignments is expected to capture the confusions specifically due to the target french accent. lastly, the confusion matrix we just created can be used as the similarity matrix described in sec-tion 3 for getting the replacement phonemes for the phonemes that do not exist in french. as for kb1 and kb2, the variation method first selects the english phonemes that do not exist in french, then selects their replacements in the similarity matrix and finally a phoneme-to-text engine creates the varied speech. this aims to mimic the way french speakers approximate the pronunciation of english. in the next sections, we evaluate the relevance of the similarity matrix described in this section -i.e. based on artificial non-native confusions -in the context of accented speech generation. this evaluation is done by comparing the results obtained by the asr on speech generated using the variation method with the above matrix, referred to as method dd2 in the remainder of the paper, against the other ones described in the paper. ## experiments the experiments aim to evaluate the extent to which it is possible to infer accent-related phonemes confusions from artificially accented speech. for that purpose, we compare the performance of the asr on the data generated as in section 4, that is the speech synthesised from artificial confusions, with respect to the other methods described in section 3, and with respect to speech without variations (artificial and natural native us english speech) as baseline. as a summary, we have the following methods: for comparing the different methods, we use three criteria: word error rate (wer), phoneme error rate (per) and visual inspection of hierarchical similarity clustering in dendogram representations. global metrics, i.e. wer and per, are used to consider the impact that variations have on recognition. the hierarchical view of similarity values of some selected phonemes provides an insight into the impact of specific variations on the recognition. that is, it is possible to see if the variation patterns propagate to the output via the confusions. for the purpose of this paper, we built four similarity matrices, following the methodology described in sections 3 and 4. that is, we built the matrices corresponding to knowledge-based methods kb1 and kb2, as well as the similarity matrices for data driven methods dd1 and dd2. for creating these matrices, we selected 1000 sentences out of the 2366 sentences of timit corpus as a text corpus. the asr system used for conducting these experiments is wav2vec 2.0. the target accent is french, and the reference language is us english. these results confirm the interest of our variation method for challenging asr systems, and they are also encouraging for the identification of nonnative speech learning patterns. indeed, we can expect that the drop in accuracy between knowledgebased variation methods and data-driven variation methods is caused by the addition of new variations patterns. while the knowledge-based approaches only apply phoneme substitutions, many more other phenomena are represented by the datadriven approaches, such as phonotactics, coarticulation or prosodic transfer. the low value of the drop, however, could indicate that phoneme substitutions are the main source of errors for asr systems, but this needs to be investigated further. ## conclusions in this paper, we used artificially accented speech for retrieving non-native similarity patterns. we generated accented speech tts with french voices and were able to use that output for calculating the corresponding confusion matrix. by using this matrix as a representation of similarity for introducing variations in speech, we found that these correspond to actual non-native variations. in the near future, we plan to enhance our knowledge-based methods with other types of variation, in particular phonotactic constraints. in the longer term, there are two motivations for the approach presented in this paper. the first is to investigate and model non-native speech variants as they are captured in deep learning models and the second is to provide a methodology for challenging asr systems to determine how far a variant can be from the expected phoneme and still be recognised correctly.