Multimodal Contextualized Semantic Parsing from Speech (2024)

Abstract: We introduce Semantic Parsing in Contextual Environments (SPICE), a task designed to enhance artificial agents’ contextual awareness by integrating multimodal inputs with prior contexts. SPICE goes beyond traditional semantic parsing by offering a structured, interpretable framework for dynamically updating an agent’s knowledge with new information, mirroring the complexity of human communication. We develop the VG-SPICE dataset, crafted to challenge agents with visual scene graph construction from spoken conversational exchanges, highlighting speech and visual data integration. We also present the Audio-Vision Dialogue Scene Parser (AViD-SP) developed for use on VG-SPICE. These innovations aim to improve multimodal information processing and integration. Both the VG-SPICE dataset and the AViD-SP model are publicly available.

## introduction imagine you are taking a guided tour of an art museum. during the tour as you visit each piece of art, your guide describes not only the artworks themselves but also the history and unique features of the galleries and building itself. through this dialog, you are able to construct a mental map of the museum, whose entities and their relationships with one another are grounded to their real-world counterparts in the museum. we engage in this type of iterative construction of grounded knowledge through dialog every day, such as when teaching a friend how to change the oil in their car or going over a set of x-rays with our dentist. as intelligent agents continue to become more ubiquitous and integrated into our lives, it is increasingly important to develop these same sorts of capabilities in them. toward this goal, this work introduces semantic parsing in contextual environments (spice), a task designed to capture the process of iterative knowledge construction through grounded language. it emphasizes the continuous need to update contextual states based on prior knowledge and new information. spice requires agents to maintain their contextual state within a structured, dense information framework that is scalable and interpretable, facilitating inspection by users or integration with downstream system components. spice accomplishes this by formulating updates as formal semantic parsing, with the formal language defining the allowable solution space of the constructed context. because the spice task is designed to model real-world and embodied applications, such as teaching a mobile robot about an environment or assisting a doctor with medical image annotations, there are crucial differences between spice and traditional text-based semantic parsing. first, spice considers parsing language within a grounded, multimodal context. the language in cases like these may have ambiguities that can only be resolved by taking into account multimodal contextual information, such as from vision. furthermore, spice supports linguistic input that comes in the form of both speech and text. in real-world embodied interactions, language is predominantly spoken, not written. while modern automatic speech recognition (asr) technology is highly accurate, it is still sensitive to environmental noise and reverberation, and representing the input language as both a waveform as well as a noisy asr transcript can improve robustness. while we do not consider it here, the spice framework also supports paralinguistic input such as facial expressions, eye gaze, and hand gestures. we present a novel dataset, vg-spice, derived from the visual genome @xcite , an existing dataset comprised of annotated visual scene graphs representing constituent entities and relational prepositions, enhanced with additional processing and synthetic augmentation to form a foundational representation for spice tasks. vg-spice simulates the conversational construction of visual scene graphs, wherein a knowledge graph representation of the entities and relationships contained within an image must be collected from the visual inputs and audio dialogue. this dataset, along with an initial model trained for vg-spice, sets the baseline for future efforts. figure 1 shows an example of a typical vg-spice sample. the figure shows how potential semantic parses can be extracted from the visual scene and spoken utterance conditioned on what information is already known about the scene. the remainder of this paper is structured as follows: it begins with a detailed analysis of the spice task, introduces the vg-spice dataset, and presents our avid-sp model. it then delves into experimental results, showcasing the model's ability to process and interpret context consistent with the spice framework. finally we outline the implications and directions for future research. the main contributions include: ## related work the spice task intersects with research in dialogue systems and semantic parsing. while previous efforts in these areas have addressed some elements of spice, none have fully encapsulated the comprehensive requirements of the spice task. ## task definition semantic parsing in contextual environments (spice) is defined as follows. consider a model agent, denoted as a, designed to maintain and update a world state across interaction timesteps. let c i represent this world state during the i th turn. for interpretability and downstream use c i is represented as a formal knowledge graph @xcite . this state represents the accumulated context from prior interactions. initially, c i can be set to a default or empty state. during each interaction turn, the agent encounters a set of new inputs, referred to as information inputs f m i , with m indicating the diversity of modalities the agent is processing. the agent's goal is to construct a formal semantic parse, p @xmath0. size visual genome @xcite a(f m i , c i ). this parse is formulated by integrating the prior context c i with the new information inputs f m i . with the aid of an execution function e, this results in an updated context c i+1 = e(p i , c i ). this newly formed context c i+1 should represent all task essential information, both from previous context c i and the most recent interaction round, for future rounds. c i+1 is expected to align with a reference context, denoted as ĉi+1 , which represents the ideal post-interaction state. ## dataset this section introduces vg-spice, a novel dataset for spice tasks, providing a structured benchmark for model training and evaluation. to our knowledge, vg-spice is the first of its kind and is derived from the visual genome dataset @xcite to simulate a "tour guide" providing sequential descriptions of aspects of the environment. in these scenarios, the tour guide describes a visual scene with sequential utterances, each introducing new elements to the scene. these descriptions, combined with a pre-established world state of the scene, mimic the accumulation of world state information through successive interactions. vg-spice utilizes the visual genome's 108k images with human-annotated scene graphs for entity identification via bounding boxes, originally detected using an object identification model. the graphs include named nodes, optional attributes, and directed edges for relational predicates. the dataset is constructed by extracting subgraphs from scene graphs as the initial context, c i , sampled from empty to nearly complete. these are then augmented by reintegrating a portion of the omitted graph to form the updated context, c i+1 . before extracting our samples, the visual genome data underwent preprocessing to enhance dataset quality (section d and summary results shown in for each context pair (c i , c i+1 ), features from c i and modified features for c i+1 are structured into natural language prompts. these prompts are processed by the llama 2 70b llm @xcite to generate plausible sentences that describe the difference between c i and c i+1 . we then synthesize spoken versions of these sentences via the tortoise-tts-v2 @xcite text-tospeech (tts) synthesis system. we configure the tts model to randomly sample speaker characteristics from its pretrained latent space, and use the built-in "high_quality" setup for other generation settings. before tts conversion filtering is performed on the textual utterances to remove common recurrent terms indicative of new information (eg., "there now is a" versus "there is a"). the audio recordings and visual images are the multimodal inputs f m i of vg-spice, emphasizing spoken audio for practicality in real-world applications and necessitating addressing the challenges of semantic parsing from audio such as speaker diversity and noise robustness. the presence of both textual and spoken audio representations for the update utterances allows vg-spice to be utilized for semantic parsing evaluations in either modality. vg-spice includes over 131k spice update samples from 20k unique scenes, with 2.5% allocated to each of the validation and test sets, ensuring distinct scenes across splits. we perform noise augmentation on the input speech using the chime5 dataset @xcite to simulate realistic noise conditions, with performance evaluated at various signal to noise ratios (snr). vg-spice samples and summary statistics are presented in figure 1 and ## avid-sp model to address the challenges of vg-spice, our approach utilizes a range of pretrained models, specifically fine-tuned to enhance spice-focused semantic parsing capabilities. figure 2 illustrates our model architecture, termed audio-vision dialogue scene parser (avid-sp). at the core of our framework lies the pretrained llama 2 7b model @xcite . despite deploying its smallest variant, the extensive pretraining endows our model with robust functional abilities, particularly beneficial for processing the diverse semantic parses inherent to vg-spice. however, llama 2, trained on textual data, lacks inherent support for the multimodal inputs typical in vg-spice. to accommodate diverse inputs, we extend techniques from prior studies @xcite @xcite by projecting embeddings from pretrained modality-specific feature extractors. this approach has been proven to enable text-based llms to process information across various modalities. directly integrating these projected embeddings into the llm's context window, however, introduces significant computational overhead due to their typically extensive context lengths. while previous research often employed pooling methods @xcite to condense embeddings by modality, this strategy incompletely addresses the challenges of merging varied modality embeddings for llm use. for instance, audio embeddings offer finer temporal granularity than textual embeddings, and the reverse is often true for vision embeddings, complicating the adjustment of downsampling factors. moreover, even with optimized downsampling, pooled embeddings must preserve their original sequential order and are restricted to information from solely the pooled segments. many applications could benefit from capabilities to establish downsampled features encompassing both local and global contexts and to rearrange these features to an extent. to surmount these challenges, we introduce a novel grouped modality attention down sam-pler (gmads) module. this module initially projects embeddings from non-textual modalities into a unified, fixed-dimensional space. we form a set of modality groupings, one for each input modality (audio and visual with vg-spice), and a cross-modality grouping derived from concatenating all modality embeddings, each prefixed with a modality-specific token. a series of self-attention layers processes each embedding sequence and downsamples the outputs by a factor of s through mean pooling. these values are then concatenated with the mean-pooled pre-self-attention embeddings along the embedding dimension, akin to a skip connection. a final projection adjusts the outputs to match the dimensionality of the llama 2 7b decoder, and all embedding sequences are concatenated. this process yields an embedding output that is effectively downsampled by a factor of s/2. all weights in the gmads module are shared across the groups, substantially reducing the parameter count. additionally, we employ a self-supervised representation learning objective on the embeddings from the downsampled crossmodality group outputs by upsampling them to their original size and then processing them through a secondary set of self-attention layers. the reconstructed cross-modality embeddings are then segmented by modality, with per-modality projections striving to restore them to their original input size. we apply a contrastive reconstruction loss objective as outlined in eq. 1, using the corresponding ground truth embedding as an anchor and all other embeddings in the batch as contrastive samples. in this equation z i denotes the reconstructed input embedding, k represents the length of each sequence, b denotes the batch size, and τ is a tunable temperature hyperparameter. we also observed that non-textual modality inputs tended to collapse when combined with simpler textual inputs, such as prior context or asr transcripts. to counter this, we include an additional orthogonality loss, designed to encourage maximal dissimilarity among aligned embeddings in each batch sequence. this methodology is similar to previous efforts to promote distinct class embeddings @xcite , but in our case, we treat each embedding as a distinct class sample. however, given the nature of these embedding se-quences, some level of similarity is expected, and entirely dissimilar values (cosine similarity less than zero) are not feasible. thus, we modify eq. 2 to include a slight margin allowing for minimal similarity. below, e i represents a single gmads output embedding (pre-output projection) within a batch of b sequences, each of length k. the gmads module attempts to provide several advantages over the direct use of raw modality embeddings with the llm decoder or mean pooling. firstly, gmads operates at reduced dimensional scales compared to the pretrained llm, which significantly lowers memory requirements, requiring the much larger decoder to process shorter (reduced to only 2/s the size) input sequences. moreover, the modality inputs do not necessitate autoregressive generation alongside these inputs, further conserving cost. secondly, gmads empowers the model to selectively learn its downsampling process, including choices on whether to focus locally or integrate global features, allowing some degree of information restructuring. the incorporation of cross-modality encoding enables parts of the downsampled embeddings to capture essential information across modalities while maintaining individual modality components in the outputs ensuring that some portion of the output embeddings is conditioned on each modality, requiring the attention mechanisms to remain sensitive to all modalities. for feature extraction, we utilize the visual encoder from dinov2 @xcite for visual inputs and the encoder from whisper-large v3 @xcite for audio. we retain only the necessary encoder portions of these pretrained models. in alignment with successful semantic parsing efforts from speech @xcite , we perform asr transcription on the audio, appending these textual embeddings to the prior context embeddings. asr transcriptions are generated using the whisper-medium.en model. to enable scalable fine-tuning, we integrate lora adaptation layers into llama 2 7b and freeze all feature extractors. ## results the performance of the avid-sp model on the vg-spice test set, as shown in under varying snr conditions, both gmads and meanpooling configurations of avid-sp show minimal performance degradation at lower snrs, indicating resilience to reasonable background noise levels. the use of accurate asr transcriptions substantially boosts parsing accuracy, emphasizing the benefits of reliable asr. experiments omitting visual inputs or incorporating incorrectly paired visual inputs exhibit minor performance declines. for the meanpooling based avid-sp a slightly larger, but still quite minor, degradation in metric performance is observed when audio inputs are excluded, with only asr transcriptions being provided. however, a more significant degradation is observed for the gmads variant of avid-sp under these same conditions. this implies that the gmads multimodal adapta for the gmads-based avid-sp, performance metrics on the read audio portion of vg-spice-c align closely with those observed in the tts portion. conversely, the meanpooling variant shows a substantial performance reduction. this discrep-ancy suggests that gmads possesses more robust multimodal processing capabilities, especially in processing out-of-domain real audio distributions. since both model variants use the same asr model without parameter tuning, the observed differences indicate that gmads compensates more effectively for poorer asr performance. ## conclusion in this paper, we introduced semantic parsing in contextual environments (spice), an innovative task designed to enhance artificial agents' contextual understanding by integrating multimodal inputs with prior contexts. through the development of the vg-spice dataset and the audio-vision dialogue scene parser (avid-sp) model, we established a framework for agents to dynamically update their knowledge in response to new information, closely mirroring human communication processes. the vg-spice dataset, crafted to challenge agents with the task of visual scene graph construction from spoken conversational exchanges, represents a significant step forward in the field of semantic parsing by incorporating both speech and visual data integration. meanwhile, the avid-sp model, equipped with the novel grouped multimodal attention down sampler (gmads), provides a strong initial baseline for vg-spice as well as insights into potential methods to improve multimodal information processing and integration. our work highlights the importance of developing systems capable of understanding and interacting within complex, multimodal environments. by focusing on the continuous update of contextual states based on new, and multimodal, information, spice represents a shift towards more natural and effective human-ai communication.