Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction (2024)

Abstract: In the field of information extraction (IE), tasks across a wide range of modalities and their combinations have been traditionally studied in isolation, leaving a gap in deeply recognizing and analyzing cross-modal information. To address this, this work for the first time introduces the concept of grounded Multimodal Universal Information Extraction (MUIE), providing a unified task framework to analyze any IE tasks over various modalities, along with their fine-grained groundings. To tackle MUIE, we tailor a multimodal large language model (MLLM), Reamo, capable of extracting and grounding information from all modalities, i.e., recognizing everything from all modalities at once. Reamo is updated via varied tuning strategies, equipping it with powerful capabilities for information recognition and fine-grained multimodal grounding. To address the absence of a suitable benchmark for grounded MUIE, we curate a high-quality, diverse, and challenging test set, which encompasses IE tasks across 9 common modality combinations with the corresponding multimodal groundings. The extensive comparison of Reamo with existing MLLMs integrated into pipeline approaches demonstrates its advantages across all evaluation dimensions, establishing a strong benchmark for the follow-up research. Our resources are publicly released at https://haofei.vip/MUIE.

## introduction ie is a pivotal topic @xcite , encompassing subtasks such as named entity recognition (ner; @xcite , relation @xcite , and event extraction (ee; @xcite , which plays a crucial role in constructing domain-specific knowledge bases @xcite and in facilitating deep semantic understanding of data @xcite please extract entity words in the text, and outline them in the image correspondingly: please extract entity words in the text, and outline the audio correspondingly: please extract all possible events in the video, and track the argument mentions in the video: singer mika enjoyed our queen shreya ghoshal's spectacular performances. please extract relations between entities in the text, and outline them in given images and audios: please extract all events in the video, and track the arguments in the video, and outline them in audio: panasonic has reduced its workforce by nearly 10000 in both japan and overseas companies. please extract entity words from the text: text + @xmath0, peer, shreya ghoshal) figure 1: examples of grounded multimodal universal information extraction (muie). firstly, current studies primarily investigate mie tasks within individual modalities (or certain modality combinations) @xcite . with the existence of several modality categories and diverse definitions for different ie tasks, studying each modality separately to construct specialized mie models would inevitably lead to resource wastage and inefficiency. in realworld applications, there is a constant need for building unified systems with "one-for-all" robust generalizability for faster practical deployment. in light of the recent success of textual universal ie (uie; @xcite , mie unification should also be promising. second, the majority of existing studies @xcite exhibit a bias toward text-centric ie outputs, necessitating the decoding of detailed textual ie labels and inherently prioritizing text. while they often treat other modalities as auxiliaries and do not produce outputs for them, this practice does not align with reality, because all modalities can equally carry important information. for example, even infants who have not yet learned to speak can learn about entities from vision. thus, each modality should be treated equally, and detect fine-grained information from all given modalities. last, most current mie @xcite involving multiple modalities (e.g., image&text) tends to extract the modalityaligned part of the information under the assumption that different modalities associate with each other. however, in practical scenarios, the information carried by different modalities can be either shared @xcite , or unrelated @xcite . this suggests that information should be flexibly recognized from any modality sources. in response to these challenges, this paper is dedicated to pioneering a novel task, grounded multimodal universal information extraction (muie). as illustrated in fig. 1 , muie aims to unify the modeling of various ie tasks (e.g., ner, re, ee) with any (or combination) inputs across the most common modalities (e.g., text, audio, image, and video), and produce fine-grained multimodally grounded ie results. to solve muie, we consider taking advantage of the existing generative llms @xcite @xcite with in-context instructions @xcite . we develop a novel multimodal llm, reamo, achieving "recognizing everything from all modalities at once". reamo not only outputs all possible textual ie labels but also identifies corresponding groundings across other modalities: 1) statically, by segmenting visual objects and audio speeches, and 2) dynamically, by tracking textual or vocal events in videos. technically, reamo employs a vicuna @xcite llm as its core semantic reasoner, utilizing im-agebind @xcite as a multimodal encoder to project image, video and audio inputs into llm-understandable signals. at the decoding end, we integrate the seem @xcite for visual grounding&tracking, and the shas @xcite for audio segmentation, where the messages are passing from llm to decoders through structured meta-response effectively. given input multimodal information, reamo is able to output uie label tokens as well as fine-grained groundings recurrently. we then design learning objectives to tune reamo to endow it with robust muie and crossmodal grounding capabilities. first, we repurpose existing textual uie annotation into instruction format, and use it to tune the backbone llm for activating the uie ability. then, we perform both coarse-grained instance-level and fine-grained grounding-aware cross-modal alignment learning, enhancing the reamo's capability in fine-grained multimodal semantic understanding. furthermore, we instruction-tune reamo on specific corpus, to build its working behavior of generating structured meta-response texts. in response to the absence of standard evaluation data for grounded muie, we further introduce an evaluation benchmark, where we annotate a highquality testing set of 3,000 instances covering ner, re, and ee tasks under 9 common modality combinations. the data further advances by annotating both modality-shared/-specific content to simulate aligned and misaligned modality scenarios. extensive zero-shot experiments on these benchmarks demonstrate that reamo shows strong performance over existing mllms with respect to ie tasks and multimodal grounding. overall, we make three key contributions: historically, ie research has treated different tasks as separate studies for a long @xcite @xcite to our knowledge, the work most closely related to this paper is by @xcite , who leverage existing mllms to unify various mie tasks through a two-stage process of span extraction and classification in a multimodal qa format. yet, we identify clear limitations in their approach that fall short of achieving comprehensive mie unification. firstly, while their work only considers text and image modalities, lacking comprehensiveness, we broadly cover the four most common modalities. secondly, beyond addressing these limitations, we also introduce a novel mllm tailored for grounded muie and contribute a new benchmark dataset for muie research. these efforts aim to pioneer the next stage of muie research. ## a benchmark for grounded muie to evaluate the performance of our grounded muie system, we develop a benchmark testing set. ## conclusion this paper introduces a novel task of multimodal information extraction setting: grounded multimodal universal information extraction (muie). first, muie definition unifies all ie tasks across various modalities, including text, audio, images, and video, with fine-grained multimodally grounded targets. to solve muie, we devise reamo, a novel mllm that can extract and ground information from all modalities. reamo is tuned through various strategies to achieve proficiency in recognizing and grounding multimodal information. further we introduce a high-quality, diverse, and challenging benchmark dataset for evaluating muie systems. experimental results demonstrate reamo's stronger performance in extracting and grounding, setting a strong benchmark for the following grounded muie research.