How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems (2020)

Abstract: In this work, we present a detailed analysis of how accent information is reflected in the internal representation of speech in an end-to-end automatic speech recognition (ASR) system. We use a state-of-the-art end-to-end ASR system, comprising convolutional and recurrent layers, that is trained on a large amount of US-accented English speech and evaluate the model on speech samples from seven different English accents. We examine the effects of accent on the internal representation using three main probing techniques: a) Gradient-based explanation methods, b) Information-theoretic measures, and c) Outputs of accent and phone classifiers. We find different accents exhibiting similar trends irrespective of the probing technique used. We also find that most accent information is encoded within the first recurrent layer, which is suggestive of how one could adapt such an end-to-end model to learn representations that are invariant to accents.

## introduction traditional automatic speech recognition (asr) systems, consisting of independently-trained acoustic, pronunciation and language models, are increasingly being replaced by end-to-end asr systems @xcite . an end-to-end asr system refers to a single model that subsumes all the traditional asr components and directly translates a speech utterance into a sequence of graphemes. such models benefit from jointly training acoustic and language models and eliminating the need for a pronunciation dictionary. while end-to-end asr models have clear merits and are elegant in their formulation, they tend to be opaque in their predictions and difficult to interpret. in order to understand better what is encoded in the layers of an end-to-end asr system, prior work has explored the use of phone probes @xcite to analyze the phonetic content of representations at each layer @xcite . this analysis was restricted to a single accent of english. in this paper, we work with multiple accents of english and propose a number of different tools (other than phone probes) to investigate how accent information is encoded and propagated within an end-to-end asr system. why accented speech? we have witnessed impressive strides in asr performance in the last few years. however, recognizing heavily accented speech still remains a challenge for state-of-the-art asr systems. an end-to-end asr model trained on a standard speech accent significantly underperforms when confronted with a new speech accent. to shed more light on why this happens, a systematic investigation of how such models behave when evaluated on accented speech might be useful. the insights from such an investigation might also come in handy when trying to adapt end-to-end neural architectures to be more accent-agnostic. we tackle the following specific questions of interest in this work: while the analyses of black-box models in computer vision and natural language processing have received a considerable amount of attention, prior work on the analysis of end-to-end asr models are notably few in number. with presenting various analysis techniques that are applicable to speech, we hope this work can serve as a starting point for further studies and spur more analysis-driven investigations into end-to-end asr models. the code used in our work is publicly available. foot_0 ## experimental setup in this section, we first introduce the dataset of accented speech samples used in our experiments, along with details of the phone-level alignments that were necessary for our subsequent analyses. we also provide a detailed description of the specific end-to-end asr model that we use in this work, along with important implementation details. ## gradient-based analysis gradient-based techniques have been widely adopted as an explainability tool in both computer vision and nlp applications. in this section, we adapt some of these techniques to be used with speech and derive insights based on how accents modify gradient behavior. ## information-theoretic analysis in the previous section, we used gradient-based methods to examine how much an input frame (or a set of frames corresponding to a phone or a word) contributes to the output and how these measures change with varying accents. without computing gradients, one could also directly measure how much information about accents is encoded within the representations at each layer. towards this, motivated by @xcite , we compute the mutual information (mi) between random variables e l figure 5 shows how mi varies across different layers for three different values of k. increasing k would naturally result in larger mi values. (the maximum possible value of mi for this task would be log 2 (7).) we observe a dip in mi going from spectral features spec to conv, which is natural considering that unprocessed acoustic features would contain most information about the underlying accent. interestingly, we observe a rise in mi going from conv to rnn 0 signifying that the first layer of rnn-based representations carries the most information about accent (not considering the acoustic features). all subsequent rnn layers yield lower mi values. apart from the mi between representations and accents that capture how much accent information is encoded within the hidden representations, we also compute mi between representations and a discrete random variable signifying phones. the mi computation is analogous to what we did for accents. we will now have a separate mi plot across layers corresponding to each accent. figure 6 shows the mi values across layers for each accent when @xmath0. we see an overall trend of increasing mi from initial to later layers. interestingly, the mi values across accents at rnn 4 exhibit a familiar ordering where us/canadian accents receive the highest mi value while indian and scotland's accents receive the lowest mi value. we also attempt to visualize the learned phone representations by projecting down to 2d. for a specific phone, we use the precomputed alignments to compute averaged layer-wise representations across the frames within each phone alignment. figure 7 shows t-sne based (maaten and hinton, 2008) 2d visualizations of representations for the 10 most frequent phones in our data, {'ah @xcite each subplot corresponds to a layer in the network. the plots for phones from the us-accented samples appear to have slightly more well-formed clusters, compared to the indian-accented samples. these kinds of visualizations of representations are, however, limiting and thus motivates the need for analysis like the mi computation presented earlier. ## discussion this is the first detailed investigation of how accent information is reflected in the internal representations of an end-to-end asr system. in devising analysis techniques for asr, while we do follow the broad approaches in the literature, the details are often different. most notably, the use of emd for attribution analysis is novel, and could be of interest to others working with speech and other temporal data. similarly, the phone focus mea-sures in the information mixing analysis are new. we also highlight that this is the first instance of analysis of asr consisting of multiple analysis techniques. on the one hand, this has uncovered robust trends that manifest in more than one analysis. on the other hand, it also shows how some trends are influenced more by the neural-network architecture more than the data. this provides a platform for future work in speech neural-network analysis, across architectures, data-sets and tasks. in our results, we encountered some unexpected details. for instance, while the rnn 0 layer is seen to reduce the phone focus the most, uniformly across all accents (as shown in figure 3 , it is also seen to segregate accent information the most, recovering accent information "lost" in the convolution layer (as shown in figure 5 ). we also see this trend surfacing in figure 8 where the accent classifier gives the highest accuracy for rnn 0 and the accuracies quickly taper off for subsequent layers. this suggests that the first rnn layer is most discriminative of accents. models that use an adversarial objective to force the representations to be accent invariant (e.g., @xcite ) might benefit from defining the adversarial loss as a function of the representations in the first rnn layer. an interesting line of work exploits the ability to identify accents in order to improve performance. @xcite combine accent detection, accent discriminative acoustic features, acoustic model adaptation using map/mllr and model selection to achieve improvements over accented mandarin speech. @xcite investigate the effect of multiple accents on the performance of an english broadcast news recognition system using a multiple accented english dataset. they report improvements by including data from all accents for an accent-independent acoustic model training. @xcite propose the use of domain adversarial training (dat) with a time delay neu-ral network (tdnn)-based acoustic model. they use native speech as the source domain and accented speech as the target domain, with the goal of generating accent-invariant features which can be used for recognition. @xcite also use an accent classifier in conjunction with a multiaccent tdnn based acoustic model in a multitask learning (mtl) framework. further, @xcite extended the mtl framework to use an end-to-end model based on the ds2 architecture and added a secondary accent classifier that uses representations from intermediate recurrent layers as input. @xcite propose an alternate approach using generative adversarial networks (gans) to disentangle accent-specific and accent-invariant components from the acoustic features. @xcite were the first to examine representations of a dnn-based acoustic model trained to predict phones. they computed selectivity metrics for each phoneme and found better selectivity and more significance in deeper layers. this analysis was, however, restricted to the acoustic model. @xcite were the first to analyze a deep speech 2 model by training phone classifiers that used representations at each layer as its input. these ideas were further extended in @xcite with classifiers used to predict phonemes, graphemes and articulatory features such as place and manner of articulation. @xcite present a comparison of different analysis methods that have been used in prior work for speech and language. the methods include recording activations of pretrained networks on linguistically annotated datasets, using probing classifiers, analyzing attention weights and abx discrimination tasks @xcite . ## summary this work presents a thorough analysis of how accent information manifests within an end-to-end asr system. the insights we gleaned from this investigation provide hints on how we could potentially adapt such end-to-end asr models, using auxiliary losses, to be robust to variations across accents. we will investigate this direction in future work. a a a e a h a o a w a y b c h d d h e h e r e y f g h h ih iy jh k l m n n g o o v o w o y p r s s h s il t th u h u w v w y z z h phones 0.00 0.02 0.04 0.06 0.08 frequency (a) phonetic coverage histogram a a a e a h a o a w a y b c h d d h e h e r e y f g h h ih iy jh k l m n n g o w o y p r s s h t th u h u w v w y z z h phones 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 avg duration(s) (b) phonetic duration histogram