Multimodal and Multiresolution Speech Recognition with Transformers (2020)

Abstract: This paper presents an audio visual automatic speech recognition (AV-ASR) system using a Transformer-based architecture. We particularly focus on the scene context provided by the visual information, to ground the ASR. We extract representations for audio features in the encoder layers of the transformer and fuse video features using an additional crossmodal multihead attention layer. Additionally, we incorporate a multitask training criterion for multiresolution ASR, where we train the model to generate both character and subword level transcriptions. Experimental results on the How2 dataset, indicate that multiresolution training can speed up convergence by around 50% and relatively improves word error rate (WER) performance by upto 18% over subword prediction models. Further, incorporating visual information improves performance with relative gains upto 3.76% over audio only models. Our results are comparable to state-of-the-art Listen, Attend and Spell-based architectures.

## introduction automatic speech recognition is a fundamental technology used on a daily basis by millions of end-users and businesses. applications include automated phone systems, video captioning and voice assistants providing an intuitive and seemless interface between users and end systems. current asr approaches rely solely on utilizing audio input to produce transcriptions. however, the wide availability of cameras in smartphones and home devices acts as motivation to build av-asr models that rely on and benefit from multimodal input. traditional av-asr systems focus on tracking the user's facial movements and performing lipreading to augment the auditory inputs @xcite @xcite . the applicability of such models in real world environments is limited, due to the need for accurate audio-video alignment and careful camera placement. instead, we focus on using video to contextualize the auditory input and perform multimodal grounding. for example, a basketball court is more likely to include the term "lay-up" whereas an office place is more likely include the term "layoff". this approach can boost asr performance, while the requirements for video input are kept relaxed @xcite . additionally we consider a multiresolution loss that takes into account transcriptions at the character and subword level. we show that this scheme regularizes our model showing significant improvements over subword models. multitask learning on multiple levels has been previously explored in the literature, mainly in the context of ctc @xcite @xcite . a mix of seq2seq and ctc approaches combine word and character level @xcite or utilize explicit phonetic information @xcite . modern asr systems rely on end-to-end, alignment free neural architectures, i.e. ctc @xcite or sequence to sequence models @xcite . the use of attention mechanisms significantly improve results in @xcite and @xcite . recently, the success of transformer architectures for nlp tasks @xcite @xcite has motivated speech researchers to investigate their efficacy in end-to-end asr @xcite . @xcite , apply an end-to-end transformer architecture for mandarin chinese asr @xcite . speech-transformer extends the scaled dot-product attention mechanism to 2d and achieves competitive results for character level recognition @xcite . pham et. al. introduce the idea of stochastically deactivating layers dur-ing training to achieve a very deep model @xcite . a major challenge of the transformer architecture is the quadratic memory complexity as a function of the input sequence length. most architectures employ consecutive feature stacking @xcite or cnn preprocessing @xcite to downsample input feature vectors. @xcite use a vgg-based input network to downsample the input sequence and achieve learnable positional embeddings. multimodal grounding for asr systems has been explored in @xcite , where a pretrained rnn-based asr model is finetuned with visual information through visual adaptive training. @xcite propose a seq2seq model based on rnns for lip-reading that performs cross-modal alignment of face tracking and audio features through an attention mechanism. furthermore, @xcite use a weakly supervised semantic alignment criterion to improve asr results when visual information is present. multimodal extensions of the transformer architecture have also been explored. these extensions mainly fuse visual and language modalities in the fields of multimodal translation and image captioning. most approaches focus on using the scaled dotproduct attention layer for multimodal fusion and cross-modal mapping. @xcite present a transformer model for av-asr targeted for lipreading in the wild tasks. it uses a self attention block to encode the audio and visual dimension independently. a decoder individually attends to the audio and video modalities producing character transcriptions. in comparison our study uses the video features to provide contextual information to our asr. @xcite employ two encoder networks for the textual and visual modalities and propose four methods of using the decoder attention layer for multimodal fusion, with hierarchical fusion yielding the best results. @xcite propose an encoder variant to fuse deep, multi-view image features and use them to produce image captions in the decoder. @xcite use cascaded multimodal attention layers to fuse visual information and dialog history for a multimodal dialogue system. @xcite present multimodal transformers, relying on a deep pairwise cascade of cross-modal attention mechanisms to map between modalities for multimodal sentiment analysis. in relation to the previous studies, the main contributions of this study are a) a fusion mechanism for audio and visual modalities based on the crossmodal scaled-dot product attention, b) an end to end training procedure for multimodal grounding in asr and c) the use of a multiresolution training scheme for character and subword level recognition in a seq2seq setting without relying on explicit phonetic information. we evaluate our system in the 300 hour subset of the how2 database @xcite , achieving relative gains up to 3.76% with the addition of visual information. further we show relative gains of 18% with the multiresolution loss. our results are comparable to state-of-the-art asr performance on this database. ## proposed method our transformer architecture uses two transformer encoders to individually process acoustic and visual information (fig. 1 ). audio frames are fed to the first set of encoder layers. we denote the space of the encoded audio features as the audio space a. similarly, video features are projected to the video space v using the second encoder network. features from audio and visual space are passed through a tied feed forward layer that projects them into a common space before passing them to their individual encoder layers respectively. this tied embedding layer is important for fusion as it helps align the semantic audio and video spaces. we then use a cross-modal attention layer that maps projected video representations to the projected audio space (section 2.1). the outputs of this layer are added to the original audio features using a learnable parameter α to weigh their contributions. the fused features are then fed into the decoder stack followed by dense layers to generate character and subword outputs. for multiresolution predictions (section 2.2), we use a common decoder for both character and subword level predictions, followed by a dense output layer for each prediction. this reduces the model parameters and enhances the regularization effect of multitask learning. ## experimental setup we conduct our experiments on the how2 instructional videos database @xcite . the dataset consists of 300 hours of instructional videos from the youtube platform. these videos depict people showcasing particular skills and have high variation in video/audio quality, camera angles and duration. the transcriptions are mined from the youtube subtitles, which contain a mix of automatically generated and human annotated transcriptions. audio is encoded using 40 mel-filterbank coefficients and 3 pitch features with a frame size of 10 ms, yielding 43-dimensional feature vectors. the final samples are segments of the original videos, obtained using word-level alignment. we follow the video representation of the original paper @xcite , where a 3d resnext-101 architecture, pretrained on action recognition, is used to extract 2048d features @xcite . video features are average pooled over the video frames yielding a single feature vector. for our experiments, we use the train, development and test splits proposed by @xcite , which have sizes 298.2 hours, 3.2 hours and 3.7 hours respectively. our model consists of 6 encoder layers and 4 decoder layers. we use transformer dimension 480, intermediate relu layer size 1920 and 0.2 dropout. all attention layers have 6 attention heads. the model is trained using adam optimizer with learning rate 10 -3 and 8000 warmup steps. we employ label smoothing of 0.1. we weigh the multitask loss with γ = 0.5 which gives the best performance. a coarse search was performed for tuning all hyperparameters over the development set. for character-level prediction, we extract 41 graphemes from the transcripts. for subword-level prediction, we train a sentencepiece tokenizer @xcite over the train set transcriptions using byte-pair encoding and vocabulary size 1200. for decoding we use beam search with beam size 5 and length normalization parameter 0.7. we train models for up to 200 epochs and the model achieving the best loss is selected using early stopping. any tuning of the original architecture is performed on the development split. no language model or ensemble decoding is used in the output. ## results and discussion one of the challenges using scaled dot-product attention is the quadratic increase of layerwise mem-ory complexity as a function of the input sequence length. this issue is particularly prevalent in asr tasks, with large input sequences. we explore three simple approaches to work around this limitation. first, we filter out large input sequences (@xmath0, leading to loss of 100 hours of data. second we, chunk the input samples to smaller sequences, using forced-alignment with a conventional dnn-hmm model to find pauses to split the input and the transcriptions. finally, we stack 4 consecutive input frames into a single feature vector, thus reducing the input length by 4. note that this only reshapes the input data as the dimension of our input is increased by the stacking process foot_0 . results for the downsampling techniques for character and subword level predictions are summarized in next, we evaluate relative performance improvement obtained from utilizing the visual features ( an important question for real-world deployment of multimodal asr systems is their performance when the visual modality is absent. ideally, a robust system satisfactorily performs when the user's camera is off or in low light conditions. we evaluate our av-asr systems in the absence of visual data with the following experiments -a) replace visual feature vectors by zeros b) initialize visual features with gaussian noise with standard deviation 0.2 c) tweak the value α to 0 on inference, gating the visual features completely. ## conclusions this paper explores the applicability of the transformer architecture for multimodal grounding in asr. our proposed framework uses a crossmodal dot-product attention to map visual features to audio feature space. audio and visual features are then combined with a scalar additive fusion and used to predict character as well as subword transcriptions. we employ a novel multitask loss that combines the subword level and character losses. results on the how2 database show that a) multiresolution losses regularizes our model producing significant gains in wer over character level and subword level losses individually b) adding visual information results in relative gains of 3.76% over audio model's results validating our model. due to large memory requirements of the attention mechanism, we apply aggressive preprocessing to shorten the input sequences, which may hurt model performance. in the future, we plan to alleviate this by incorporating ideas from sparse transformer variants @xcite . furthermore, we will experiment with more ellaborate, attention-based fusion mechanisms. finally, we will evaluate the multiresolution loss on larger datasets to analyze it's regularizing effects.