Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers (2023)

Abstract: Transformers have become a key architecture in speech processing, but our understanding of how they build up representations of acoustic and linguistic structure is limited. In this study, we address this gap by investigating how measures of ‘context-mixing’ developed for text models can be adapted and applied to models of spoken language. We identify a linguistic phenomenon that is ideal for such a case study: homophony in French (e.g. livre vs livres), where a speech recognition model has to attend to syntactic cues such as determiners and pronouns in order to disambiguate spoken words with identical pronunciations and transcribe them while respecting grammatical agreement. We perform a series of controlled experiments and probing analyses on Transformer-based speech models. Our findings reveal that representations in encoder-only models effectively incorporate these cues to identify the correct transcription, whereas encoders in encoder-decoder models mainly relegate the task of capturing contextual dependencies to decoder modules.

## introduction in both speech and text processing, variants of the transformer architecture @xcite have become ubiquitous. the key advantage of this neural network topology lies in the modeling of pairwise relations between elements of the input (tokens): the representation of a token at a particular transformer layer is a function of the weighted sum of the transformed representations of all the tokens in the previous layer. this feature of transformers is known as context mixing and understanding how it functions in specific model layers is crucial for tracing the overall information flow. several works (e.g. @xcite @xcite have proposed methods to quantify context mixing and applied them to transformers trained on text data. in the current study we investigate context mixing in models trained on spoken language data. we focus on the task of automatic speech recognition (asr), which consists of transcribing spoken utterances into their written equivalent. in order to probe context mixing for speech models, we exploit a quirk of the french spelling system: modern spoken french does not in general foot_1 mark grammatical number on nouns, adjectives or third-person verbs, only on determiners and pronouns -however written french reflects an earlier period of the language and does mark nouns, adjectives and verbs for number. therefore, for an asr system to produce grammatical transcriptions which feature number agreement, the model needs to rely on very specific cues in the input and/or in the prefix of the generated output. for example, when transcribing the utterance elle a perdu les livres, meaning "she lost the books", when generating the bolded target word, the model needs to decide whether to output livre (singular) or rather livres (plural) and it cannot use the speech signal corresponding to the target word to choose the correct form, as in spoken french both are pronounced /livk/. the correct choice is determined by grammatical agreement with the underlined cue word, here the plural determiner, which is different from the singular form, both in speech (/le/ vs /l@/) and in writing (les vs le). thanks to this phenomenon we can form a strongly supported prior expectation that the representation of the cue token should contribute to the representations of the target. we are therefore able to validate previously suggested context mixing scoring methods on a dataset of french utterances featuring such grammatical homonyms: we find that context-mixing scoring methods, and especially value zeroing, behave plausibly, largely agreeing with prior expectations, as well as with targeted diagnostic probes and faithfulness tests. we furthermore apply context-mixing scoring methods to two types of architectures: encoderdecoder and encoder-only, and discover interesting differences in how they behave. in encoderonly models, spoken cue-word representations contribute to target representations in a straightforward manner. meanwhile, in encoder-decoder architectures it is mostly the decoder which ensures grammatical number agreement, and therefore spoken cue-word representations contribute little to targets, while written cue-words in the generated prefix make a decisive contribution. ## context mixing our goal is to quantify the contribution of a cue word to the representation of a target word across layers of speech transformers. here we first review the formation of contextualized representations in these models, and then describe how we adapt textbased context mixing methods for our purpose. ## experiments we design a number of experiments meant to first validate the context-mixing scoring methods in the context of speech transformers, and further to provide converging evidence on the specific patterns of context-mixing in the two target architectures on our dataset. ## conclusion in this paper, we adapted state-of-the-art context mixing methods from the text domain to analyze sensitivity to syntactic cues and to trace information mixing in transformer-based models of speech recognition. our analyses revealed that in encoderonly models, spoken cue-word representations directly contribute to the formation of target word representations. in contrast, encoder-decoder models largely relegate the task of capturing contextual dependencies to the decoder module. through a series of complementary experiments, we identified two distinct roles performed by the decoder in encoder-decoder speech models when forming contextualized representations: attending to discrete prefix cues using self-attention, and attending to encoded audio from the encoder using crossattention. our study validates the applicability of the adapted context mixing scores that were originally developed for text, and showcases their potential for discovering patterns of information flow in transformer models of language in spoken modality. we hope this convergence in analytical methodologies between speech and text communities can pave the way for developing unified models for natural language processing @xcite . in line with previous research, our results discredit raw attention weights as a reliable source of information for information mixing in speech models.