Accented Speech Recognition With Accent-specific Codebooks (2023)

Abstract: Speech accents pose a significant challenge to state-of-the-art automatic speech recognition (ASR) systems. Degradation in performance across underrepresented accents is a severe deterrent to the inclusive adoption of ASR. In this work, we propose a novel accent adaptation approach for end-to-end ASR systems using cross-attention with a trainable set of codebooks. These learnable codebooks capture accent-specific information and are integrated within the ASR encoder layers. The model is trained on accented English speech, while the test data also contained accents which were not seen during training. On the Mozilla Common Voice multi-accented dataset, we show that our proposed approach yields significant performance gains not only on the seen English accents (up to 37% relative improvement in word error rate) but also on the unseen accents (up to 5% relative improvement in WER). Further, we illustrate benefits for a zero-shot transfer setup on the L2Artic dataset. We also compare the performance with other approaches based on accent adversarial training.

## introduction accents in speech typically refer to the distinctive way in which the words are pronounced by diverse speakers. while a speaker's accent may be primarily derived from their native language, speech accents are also influenced by various other factors related to the geographic location, educational background, socio-economic and socio-linguistic factors like race, gender and cultural diversity @xcite . it is therefore infeasible to build automatic speech recognition (asr) systems which comprehensively cover speech accents during training. in such scenarios, novel speech accents continue to have an adverse effect on asr performance @xcite . while humans effectively recognize speech from new and unseen accents (clarke @xcite , asr systems show substantial degradation in performance when dealing with new accents that are unseen during training @xcite . prior works attempting to address accent-related challenges for asr can be categorized into three groups: i) multi-accent training @xcite , ii) accent-aware training using accent embeddings @xcite or adversarial learning @xcite , and iii) accent adaptation using supervised @xcite or unsupervised techniques @xcite . while partial success has been achieved using most of these approaches, the development of robust speech recognition systems that are invariant to accent differences in training and test remains a challenging problem. in this work, we propose a new codebook based technique for accent adaptation of state-of-theart conformer-based end-to-end (e2e) asr models @xcite . for each of the accents observed in the training data, we define a codebook with a predefined number of randomly-initialized vectors. these accent codes are integrated with the self-attended representations in each encoder layer via the cross-attention mechanism, similar to the perceiver framework @xcite . the asr model is trained on multi-accented data with standard end-to-end (e2e) asr objectives. the codes capture accent-specific information as the training progresses. during inference, we propose a beam-search decoding algorithm that searches over a combined set of hypotheses obtained by using each set of accent-specific codes (once for each seen accent) with the trained asr model. on the mozilla common voice (mcv) corpus, we observe significant improvements on both seen and new accents at test-time compared to the baseline and existing supervised accent-adaptation techniques. our main contributions are: ## related work traditional cascaded asr systems @xcite handled accents by either modifying the pronunciation dictionary @xcite or modifying the acoustic model @xcite . more recent work on accented asr has focused on building end-to-end accentrobust asr models. towards this, there are two sets of prior works: accent-agnostic approaches and accent-aware approaches. accent-agnostic asr. such approaches force the model to disregard the accent information present in the speech and focus only on the underlying content. prior work based on this approach uses adversarial training @xcite or similarity losses. using domain adversarial training, with the discriminator being an accent classifier, has shown significant improvements over standard asr models @xcite . pre-training the accent classifier @xcite and clusteringbased accent relabelling @xcite have also led to further performance improvements. the use of generative adversarial networks for this task has also been explored @xcite . rather than being explicitly domain adversarial, other accent agnostic approaches use cosine losses @xcite or contrastive losses @xcite to make the model accent neutral. these losses force the model to output similar representations for inputs with the same underlying transcript. accent-aware asr. accent-aware approaches feed the model additional information about the accent of the input speech. early work in this category focused on using the multi-task learning (mtl) paradigm @xcite @xcite ) that jointly trains accentspecific auxiliary tasks with asr. different types of embeddings like i-vectors @xcite , dialect symbols @xcite , embeddings extracted from tdnn models @xcite or from wav2vec2 models trained as classifier @xcite have also been explored for accented asr. many simple ways of fusing accent information with the input speech have been previously investigated. this fusion can either be a sum @xcite @xcite , a weighted sum @xcite or a concatenation @xcite . few works also explore the possibility of merging both accent-aware and accent-agnostic techniques within the same model @xcite . our work also proposes an accent-aware approach. however, unlike prior work that focuses on prefetched accent information, we learn accent information embedded within codebooks during training. additionally, instead of simply concatenating input speech with accent embeddings, we propose a learned fusion of accent information with speech representations using cross-attention. prior work by @xcite demonstrates fine-grained integration of accent information. however, our proposed framework integrates accent information as part of end-to-end training resulting in robust adaptation. and dec-ctc to jointly predict the output token sequence @xmath0, . . . , y j , . . . , y u }. dec-att is an autoregressive decoder that maximizes the conditional likelihood of producing an output token y j given h and the previous labels y 1 , . . . , y j-1 . in contrast, dec-ctc uses ctc to maximize the likelihood of y given h by marginalizing over all alignments. the encoder is implemented using conformer layers @xcite and the decoder is implemented using transformer layers @xcite . for our proposed technique, we introduce the following three essential modifications to the base architecture: i) constructing codebooks that can encode accent-specific information (section 3.1). ii) enabling fine-grained integration of accent information with a conformer-based asr model using cross-attention (section 3.2). iii) modifying beam search decoding for inference in the absence of accent labels at test-time (section 3.3). ## experiments and results from ## conclusion in this work, we propose a new end-to-end technique for accented asr that uses accent-specific codebooks and cross-attention to achieve significant performance improvements on seen and unseen accents at test time. we experiment with the mozilla common voice corpus and show detailed ablations over our design choices. we also empirically analyze whether our codebooks encode information relevant to accents. the effective use of codebooks for accents opens up future avenues to encode non-semantic cues in speech that affect asr performance, such as types of noise, dialects, emotion styles of speech, etc.