O mni D ialog: A Multimodal Benchmark for Generalization Across Text, Visual, and Audio Modalities (2024)

Abstract: We introduce \textit{OmniDialog} — the first trimodal comprehensive benchmark grounded in a knowledge graph (Wikidata) to evaluate the generalization of Large Multimodal Models (LMMs) across three modalities. Our benchmark consists of more than 4,000 dialogues, each averaging 10 turns, all annotated and cross-validated by human experts. The dialogues in our dataset are designed to prevent shortcut learning by incorporating various formats and misleading or irrelevant multimodal cues. We also evaluate both multimodal and unimodal models to gain insights into how they process modality inputs introduced in the conversation.

## introduction multimodal dialogue systems became a focal point in research, drawing significant attention of both academia and industry. this surge of interest stems from their potential to contribute to more natural and nuanced human-computer interactions by seamlessly integrating text, audio, and visual cues @xcite @xcite . yet, the complexity of these systems has led to challenges in their evaluation. existing benchmarks, in many instances, fall short in capturing the intricacies of the real-world interactions, lacking the necessary depth and diversity to evaluate the true capabilities of multimodal dialogue systems @xcite . in response, we introduce the omnidialog benchmark, a multimodal, multi-turn benchmark designed to evaluate the generalization abilities of large multimodal models (lmms). specifically, our benchmark assesses their capability to support multi-turn conversations, process modality injections at random points within the dialogue, and operate with three modalities simultaneously @xcite . it stands out by grounding on wikidata knowledge graph and encompasses a vast array of more than 4,000 dialogues, each with an average of 10 turns. to ensure the highest quality, our human annotators designed these dialogues from scratch and then cross-validated them to ensure accuracy and consistency. the uniqueness of omnidialog lies in its design: it requires deep understanding of three modalitiestext, visual, and audio. moreover, to ensure that systems truly understand the context rather than exploit shortcuts, we present dialogues in various formats. our contributions are as follows: the primary data for omnidialog is sourced from wikipedia foot_0 and wikidata foot_1 , ensuring both the authenticity and generalization ability of the dialogues. the datasets and evaluation code will be released under an open source licence at https://github.com/ai-forever/omnidialog . ## related work in this section, we provide a brief description of popular multimodal datasets and state-of-the-art multimodal transformer architectures. ## omnidialog in this section, we describe omnidialog -a benchmark for evaluating multimodal dialogue systems in english. our dataset is distinguished by its diversity in dialogue types, human annotation, and strict evaluation metrics. it is grounded to knowledge graphs, which means that the discussed facts, images, and audios in dialogs are taken from wikidata. our benchmark comprised of more than 4,000 dialogues, each averaging 10 turns, with data and facts sourced primarily from wikipedia and wikidata. ## baselines given that the most progress has been achieved for the combination of visual and textual modalities, we have adapted several recently introduced models to process the dialogue input. the baselines include not only the trimodality models but also bimodal (visual-text) and unimodal models (language-only) models. the latter are evaluated using the corresponding part of omnidialog. trimodal. gemini @xcite serves as a baseline model supporting all three modalities. visual-language. several (near) state-of-theart models are assessed on the language-only and visual-language omnidialog parts: series of llava models @xcite , series of in-ternvl 2.0 @xcite chat models with the llm backbones of various size (from 1b to 40b parameters) and strong vision encoder (internvit-6b and its distilled version internvit-300m), and idefics2 8b @xcite vision-language model trained with the interleaved data. these models ignore audio data during the evaluation process and only encode images with proposed visual adapter architectures. language-only. gpt-4-mini is used as a language-only baseline reference for omnidialog. no encoding of visual or audio data is performed during the evaluation and model solely reasons from previosly answered questions. ## discussion we discuss result of omnidialog benchmark evaluation presented in ## conclusion we introduced omnidialog, a diverse and comprehensive benchmark for multimodal dialogue systems, comprising more than 4,000 dialogues based on entities, facts, and media from wikidata knowledge graph. these dialogues, designed to prevent shortcut learning, offer a unique challenge for multimodal systems with various formats and misleading cues. the knowledge graph foundation of omnidialog makes it a valuable resource for comparing multimodal models based on factual information. we also analyze several baselines, both opensource and proprietary, with respect to their generalization across various modalities and types of dialogues introduced with omnidialog. following the teacher-forcing evaluation, we compared models operating with different modalities. we believe that the creation of multimodal benchmarks will motivate the community to develop dialogue assistants further and enhance their evaluation. we also aim to build upon this work by providing various types of assessments using llm oracles, both with and without teacher forcing, which may result in a fairer comparison of the quality of multimodal models.