Multimodal Speech Recognition with Unstructured Audio Masking (2020)

Abstract: Visual context has been shown to be useful for automatic speech recognition (ASR) systems when the speech signal is noisy or corrupted. Previous work, however, has only demonstrated the utility of visual context in an unrealistic setting, where a fixed set of words are systematically masked in the audio. In this paper, we simulate a more realistic masking scenario during model training, called RandWordMask, where the masking can occur for any word segment. Our experiments on the Flickr 8K Audio Captions Corpus show that multimodal ASR can generalize to recover different types of masked words in this unstructured masking setting. Moreover, our analysis shows that our models are capable of attending to the visual signal when the audio signal is corrupted. These results show that multimodal ASR systems can leverage the visual signal in more generalized noisy scenarios.

## introduction jointly modelling linguistic and visual signals is beneficial for several language processing tasks, such as machine translation @xcite , visual question-answering (vqa) @xcite , summarization @xcite and automatic speech recognition (asr) @xcite . however, it is unclear exactly how the visual signals are useful for these tasks. for example, in vqa, it has been observed that models can ignore the visual context and instead rely on linguistic biases in the dataset @xcite ; in machine translation, it has been shown that some models are not affected by incorrect visual signals @xcite ; and in multimodal asr, the visual signals were shown to act as a regularizer instead of useful disambiguating context @xcite . given these uncertainties, there is a need to clarify the circumstances in which visual signals are useful. previous work in multimodal machine translation @xcite and asr @xcite shows that the visual signal is useful when the linguistic signal is degraded by dropping the input. in this setting, multimodal models leverage the visual signals to recover the missing language information. the results in @xcite are a promising start towards verifiably useful multimodality for robust speech recognition. however, the experiments were conducted with structured noise that focused on a predetermined set of groundable entities (i.e., nouns and places). in real world scenarios, however, noise occurs in a more unstructured manner. therefore, it is important that multimodal models can use the visual signal in a wider variety of situations. in this work, we study multimodal asr in more realistic noisy scenarios. we follow the methodology from @xcite but we randomly mask words in an unstructured manner in the audio signal (we refer to this as randwordmask). this is in contrast to the structured masking in @xcite , where the masked audio corresponds to only entities (which we refer to as en-titymask). the example in figure 1 shows that randwordmask can mask any words in the audio signal, whereas entitymask would only mask entities like "girl" and "slide". we apply masking both during training and testing. the main contributions of this work are: the results show that visual signals improve speech recognition in this more difficult, unstructured setting where random words are masked. our models are not only able to recover masked entities, but they also recover words from other syntactic 1 we note that randwordmask is different from robust asr @xcite scenarios, where the whole signal is corrupted with stationary noise. ## methodology in this section, we describe the different asr models and our technique for simulating unstructured audio masking. ## results and analysis in the results in in the unimodal asr and hierattn-df (the best multimodal model in terms of rr) on the different word types. we observe that on the groundable entities i.e., nouns and places, there is a relative improvement of at least 25% compared to the unimodal model. adjectives and colors, which are also groundable in the visual modality, are recovered around 14% better than the unimodal model. the relative rr improvement for verbs is around 7%, whereas adverbs recovery is 4% worse. these results show that visual context can recover words from a variety of categories, even though it is better at recovering entities, and struggles with words that are less groundable in the image. ## conclusions we show that visual signals improve multimodal speech recognition when the audio signal is subject to unstructured masking. randwordmask simulates a wider range of noisy scenarios by masking different types of words in the audio signal during training and evaluation, as opposed to previous work that only masked groundable entities @xcite . future work involves developing new models that attend over visual features extracted from object proposals, which provide better visual signals.