Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent? (2022)

Abstract: The usage of automatic speech recognition (ASR) systems are becoming omnipresent ranging from personal assistant to chatbots, home, and industrial automation systems, etc. Modern robots are also equipped with ASR capabilities for interacting with humans as speech is the most natural interaction modality. However, ASR in robots faces additional challenges as compared to a personal assistant. Being an embodied agent, a robot must recognize the physical entities around it and therefore reliably recognize the speech containing the description of such entities. However, current ASR systems are often unable to do so due to limitations in ASR training, such as generic datasets and open-vocabulary modeling. Also, adverse conditions during inference, such as noise, accented, and far-field speech makes the transcription inaccurate. In this work, we present a method to incorporate a robot’s visual information into an ASR system and improve the recognition of a spoken utterance containing a visible entity. Specifically, we propose a new decoder biasing technique to incorporate the visual context while ensuring the ASR output does not degrade for incorrect context. We achieve a 59% relative reduction in WER from an unmodified ASR system.

## introduction spoken interaction with a robot not only increases its usability and acceptability, it provides a natural mode of interaction even for a novice user. the recent development of deep learning-based end-toend automatic speech recognition (asr) systems has achieved a very high accuracy @xcite as compared to traditional asr systems. as a result, we see a huge surge of speech-based interfaces for many systems including robots. however, the accuracy of any state-of-the-art asr gets significantly impacted based on the dialect of the speaker, distance of the speaker from the microphone, ambient noise, etc., particularly for novel and low-frequency figure 1 depicts a typical scenario where an agent foot_0 uses an asr to translate audio input to text. then, it detects the set of objects in its vicinity using an object detector. finally, it matches the object mentioned in the command and the objects detected in the vicinity (grounding) to narrow down the target object before execution. if the audio translation is erroneous, the grounding can fail, which leads to failure in task execution. for example, in figure 1 , even though the user mentioned "pink pillow", the translation was "bink illow", which results in failure in task grounding. there has been an increasing interest in contextual speech recognition, primarily applied to voice-based assistants @xcite @xcite @xcite . however, incorporating visual context into a speech recognizer is usually modeled as a multi-modal speech recognition problem @xcite , often simplified to lip-reading @xcite . attempts to utilize visual context in robotic agents also follow the same approach @xcite . such models always require a pair of speech and visual input, which fails to tackle cases where the visual context is irrelevant to the speech. in contrast, we consider the visual context as a source of dynamic prior knowledge. thus, we bias the prediction of the speech recognizer to include information from the prior knowledge, provided some relevant information is found. there are two primary approaches to introducing bias in an asr system, namely shallow and deep fusion. shallow fusion based approaches perform rescoring of transcription hypotheses upon detection of biasing words during beam search @xcite . class-based language models have been proposed to utilize the prefix context of biasing words @xcite . @xcite further improved the shallow-fusion biasing model by introducing sub-word biasing and prefix-based activation. @xcite propose 2-pass language model rescoring with sub-word biasing for more effective shallow-fusion. deep-fusion biasing approaches use a pre-set biasing vocabulary to encode biasing phrases into embeddings that are applied using an attentionbased decoder @xcite . this is further improved by using adversarial examples @xcite , complex attention-modeling @xcite , and prefix disambiguation @xcite . these approaches can handle irrelevant and empty contexts but are less scalable when applied to subword units @xcite . furthermore, a static biasing vocabulary is unsuitable for some applications, including the one described in this paper. recent works propose hybrid systems, applying both shallow and deep fusion to achieve state-of-the-art results @xcite . spelling correction models are also included for additional accuracy gains @xcite . in this article, we propose a robust speech interface pipeline for embodied agents, called rosi, that augments existing asr systems (figure 2 ). using an object detector, a set of (natural language) phrases about the objects in the scene are generated. a biasing vocabulary is built using these generated captions on the go or it can be pre-computed whenever a robot moves to a new location. our main contributions are twofold. ## background we adopt a connectionist temporal classification (ctc) @xcite based modeling in the baseline asr model in our experiments. a ctc based asr model outputs a sequence of probability distributions over the target vocabulary @xmath0, . . . , y t } (usually characters), given an input speech signal with length l, @xmath1, . . . , x l }, @xmath2, thus computing, the output sequence with the maximum likelihood is usually approximated using a beam search @xcite . during this beam search decoding, shallow-fusion biasing proposes rescoring an output sequence hypothesis containing one or more biasing words @xcite @xcite . assuming a list of biasing words/phrases is available before producing the transcription, a rescoring function provides a new score for the matching hypothesis that is either interpolated or used to boost the log probability of the output sequence hypothesis @xcite , to counter this, biasing at the sub-word units (grapheme/word-piece) by weight pushing has been proposed in @xcite . biasing at the grapheme level improves recognition accuracy than word level biasing for speech containing bias terms, but the performance degrades severely for general speech where the bias context is irrelevant. subsequently, two improvements are proposed -i) biasing at the word-piece level @xcite @xcite and ii) utilizing known prefixes of the biasing terms to perform contextual biasing @xcite . although word-piece biasing shows less degradation than grapheme level on general speech, it performs worse than word level biasing when using high biasing weights @xcite . also, prefix-context based biasing is ineffective when a biasing term is out of context. moreover, a general problem with the weight-pushing based approach as compared to the sub-word level biasing is that they require a static/class-specific biasing vocabulary to work, usually compiled as a weighted finite state transducer (wfst). also it requires costly operations to be included with the primary wfst-based decoder. however, for frequently changing biasing vocabulary, e.g., changing with the agent's movement, frequent re-compiling and merging of the wfst is inefficient. therefore, we propose an approach to retain the benefits of word-level biasing for general speech, also preventing early pruning of partially matching hypotheses using a modified beam search algorithm. during beam search, our algorithm allocates a fixed portion of the beam width to be influenced by sub-word level look-ahead, which does not affect the intrinsic ranking of the other hypotheses. this property is not guaranteed in weight-pushing @xcite that directly performs subword-level biasing. moreover, we specifically target transcribing robotic instructions that usually include descriptions of everyday objects. thus the words in the biasing vocabulary are often present in standard language models (lm) in contrast to existing biasing models that focus on biasing out-of-vocabulary (oov) terms such as person names. we also utilize this distinction, by incorporating an n-gram language model to contextually scale the biasing score. we describe our shallow-fusion model in section 4.2. ## system overview in this section, we present an overview of the embodied agent that executes natural language instructions, as depicted in figure 3 . given a speech input, the agent also captures an image from its ego-view camera. the dynamic context extraction module extracts the visual context from the captured image before producing the transcription of the speech input. firstly, a dense image captioning model predicts several bounding boxes of interest in the image and generates a natural language description for each of them. given the dense captioned image, a bias target prediction model predicts a list of biasing words to be used in speech recognition. the list of biasing words/phrases is compiled into a prefix tree (trie) that is used by the beam search decoder to prevent the pruning of partially matched hypotheses. the trie is dynamically created with the agent's movement that captures a new image. the acoustic model processes the speech input to produce a sequence of probability distributions over a character vocabulary. we use the wav2vec2 @xcite for acoustic modeling of the speech. this sequence is decoded into the transcription using a modified beam search decoding algorithm. during the beam search, the visual context that is represented using the biasing trie is used to produce a transcription that is likely to contain the word(s) from the visual context. we describe our biasing approach in detail in section 4.2. given the transcribed instruction, the task understanding & planning module performs task type classification, argument extraction, and task planning. we use conditional random field (crf) based model as proposed in our earlier works @xcite . specifically, the transcribed instruction is passed through a task-crf model that labels tokens in the transcription from a set of task types. given the output of task-crf, an argument-crf model labels text spans in the instruction from a set of argument labels. this results in an annotated instruction such as, ## visual context biasing the probability distribution sequence produced by the acoustic model can be sub-optimally decoded in a greedy manner, i.e., performing an argmax computation each time-step and concatenating the characters to produce the final transcription. however, a greedy-decoding strategy is likely to introduce errors in the transcription that can easily avoided by using beam search @xcite . in the following, we propose an approach to further reduce transcription errors by exploiting the embodied nature of the agent. ## conclusion in this article, we have presented a method to utilize contextual information from an embodied agent's visual observation in its speech interface. in particular, we have designed a novel beam search decoding algorithm for efficient biasing of a speech recognition model using prior visual context. our experiments show that our biasing approach improves the performance of a speech recognition model when applied to transcribing spoken instructions given to a robot. we also find that our approach shows less degradation than other approaches when the extracted visual context is irrelevant to the speech. even in adversarial context, the accuracy of our system is well beyond the accuracy of the unmodified speech recognition model.