Fine-Grained Grounding for Multimodal Speech Recognition (2020)

Abstract: Multimodal automatic speech recognition systems integrate information from images to improve speech recognition quality, by grounding the speech in the visual context. While visual signals have been shown to be useful for recovering entities that have been masked in the audio, these models should be capable of recovering a broader range of word types. Existing systems rely on global visual features that represent the entire image, but localizing the relevant regions of the image will make it possible to recover a larger set of words, such as adjectives and verbs. In this paper, we propose a model that uses finer-grained visual information from different parts of the image, using automatic object proposals. In experiments on the Flickr8K Audio Captions Corpus, we find that our model improves over approaches that use global visual features, that the proposals enable the model to recover entities and other related words, such as adjectives, and that improvements are due to the model’s ability to localize the correct proposals.

## introduction multimodal language processing is inspired by evidence that conceptual representations in humans are distributed across modality-specific systems @xcite . in recent years, researchers have developed deep learning models that combine visual, linguistic, and auditory modalities for a variety of multimodal tasks, such as automatic image captioning @xcite , visual questionanswering @xcite , and image-speech retrieval @xcite , inter-alia. in multimodal automatic speech recognition (asr), there have been efforts to integrate visual context into acoustic models @xcite and sequence-to-sequence models (palaskar 1 the code is available at https://github.com/ tejas1995/multimodalasr et @xcite @xcite . however, it is not clear if the visual context actually improves asr or if it helps to regularize the model @xcite . @xcite recently showed that global visual context (a single feature vector representing the entire image) is useful when the visually depictable linguistic inputs are masked, i.e., masking the speech that refer to entities. this experimental methodology, inspired by @xcite , creates a systematic gap in the speech signal that can be resolved by leveraging the visual context; for example, when the audio drops during online distance-based learning or video calls with family and friends. we present a model for multimodal asr that learns to integrate visual features from object proposals @xcite , rather than image-level features, which has previously proven to be useful for image captioning and vqa @xcite . object proposals are rectangular image regions that are expected to contain objects. the novelty of our model is that when it encounters masked audio, it grounds @xcite the missing speech to different regions of the image. our model learns separate attention distributions @xcite for each modality and combines them using a hierarchical attention mechanism in the decoder @xcite . this approach to integrating visual context from object proposals allows the model to better learn the relationship between speech and depicted colours, entities, and (to some extent) cardinals. in experiments on the flickr8k audio captions corpus @xcite , we find that our model is much better at recovering masked speech than previous work. we also find that our model is right for the right reasons. in section 4.1, we perform an object localization analysis, finding that 44% -49% of the maximally attended object proposals, and 80% -83% of the top-5 attended proposals, overlap with the ground-truth bounding box annotations. this shows that our model is verifiably leveraging the visual context. the main contributions of this paper are: ## results and analysis in first, we find that the multimodal asr models outperform the unimodal model in terms of recovery rate, and that the difference increases as the masking rate increases from 20% to 60%. the word error rate of the unimodal model is slightly lower than the multimodal models for clean data, but these models perform much better than unimodal with higher speech masking rates. furthermore, the maop model that operates over object proposals substantially outperforms the mag model, which uses a single global visual vector, on both metrics and at all masking levels. we now turn our focus to analysing which types of words are best recovered by our multimodal models. we conduct this analysis across seven categories: five syntactic (nouns, verbs, adjectives, adverbs and cardinals) and two semantic (places and colors). 3 for each category, we create a new test set where we mask all occurrences of words belonging to that category. in we also report the grounding rate of the multimodal models in ## related work inspired by studies of human perception, multimodal processing is spreading into many traditional areas of research, e.g., machine translation @xcite and asr @xcite . it has become an important part of new areas of research such as image captioning @xcite , visual question-answering (vqa; @xcite ), and multimodal summarization @xcite . the representation and integration of visual context in multimodal asr systems is an active area of research. previous approaches incorporate image representations either in the acoustic model @xcite , the language model @xcite , or in end-to-end models @xcite . @xcite and @xcite explore different types of multimodal representations such as imagescene representations and titles of instructional videos respectively. although all these integration methods show improvements over unimodal baselines, it is not clear when such approaches perform better, and which representations are best. it has been argued that traditional multimodal architectures do not necessarily take advantage of image semantics in different tasks. @xcite showed that multimodal asr models trained with shift adaptation @xcite foot_4 use the image as a regularization signal. in a similar direction, elliott (2018) showed that misaligment between image and text representations do not affect multimodal mt models. ramakrishnan et al. (2018) and grand and belinkov (2019) showed that traditional vqa neural architectures ignore the visual context and focus on linguistic biases of the dataset. more related to our work are the studies of @xcite and @xcite , which explore how multimodal models use image information under noisy scenarios. these studies conclude that when certain nouns are dropped from the dominant language modality, multimodal models are capable of properly using the semantics provided by the image. however, unlike this work, their explorations are limited to nouns and not expanded to other types of words. from an image representation perspective, previous works have studied the utility of using local representations, rather than global ones for multimodal language processing tasks. for instance, @xcite show that, by using attention, the model can use different regions of the image while performing image captioning. more recent work shows that bounding boxes @xcite , a discrete variant of attention over images, improve the representation and hence the performance of different tasks such as vqa @xcite , image captioning @xcite and machine translation @xcite . in this work, we apply this methodology to multimodal asr (see section 3.4). ## conclusions in this work, we introduce a new model for multimodal asr that attends overs fine-grained object proposals and is capable of recovering words which are masked in the speech signal. we show that our model recovers masked words because it can accurately identify the relevant object proposal(s), and that this ability allows it to not only recover the object when it has been masked in the speech signal, but also the object's attributes. in future work, we plan to improve our model by masking random speech segments @xcite rather than aligned words. if successful, this methodology would allow us to train and test our multimodal models without the need for word alignments, a current limitation of our framework. we will also experiment with more challenging speech captioning scenarios where speech ambiguities are more likely to occur @xcite .