Multimodal Document-level Triple Extraction via Dynamic Graph Enhancement and Relation-Aware Reflection (2025)

Abstract: Multimodal documents, which are among the most prevalent data formats, combine a large amount of textual and visual content. Extracting structured triples knowledge from these documents is a highly valuable task, aimed at helping users efficiently acquire key entities and their relationships. However, existing methods face limitations in simultaneously processing long textual content and multiple associated images for triple extraction. Therefore, we propose a Multimodal Document-level Triple Extraction (MDocTE) framework. Specifically, we introduce a dynamic document graph construction method that extends the model’s scope to the entire document and the external world, while adaptively optimizing the graph structure. Next, we inject the global information and external knowledge learned by the graph neural network into the large language model, generating structured triples after deep interaction. Finally, we design a multimodal relation-aware mechanism and loss function to guide the model in reflecting on the shared information between text and visuals. We release a new triple extraction dataset for multimodal documents and conduct extensive experiments. The results demonstrate that the proposed framework outperforms the state-of-the-art baselines, thus filling the gap in multimodal document extraction. Our data is available at https://github.com/XiangLiphd/Triple-extraction-dataset-for-multimodal-documents.

## introduction triple extraction aims to identify entity pairs and their semantic relationships from unstructured text @xcite @xcite , providing a foundation for downstream tasks like knowledge graph construction, information retrieval, and knowledge enhancement (wang et al., 2024a; ovadia et al., * corresponding author 2024; @xcite . unlike the pipeline approach that separates named entity recognition and relation extraction, joint extraction integrates both tasks, mitigating error propagation and strengthening their intrinsic connections, making it a current research hotspot @xcite @xcite . however, most extraction datasets are sourced directly from vast and unstructured web text @xcite . in many cases, the entities and relationships extracted directly from text are inaccurate @xcite . thus, leveraging image information to correct extracted relationships becomes a promising strategy @xcite @xcite . @xcite proposed the first joint multimodal entity-relation extraction model, achieving fine-grained alignment between textual entities and visual objects. building on this, yuan et al. (2024) further introduced a few-shot multimodal entity-relation extraction model, addressing the challenge of requiring large amounts of labeled data. these methods not only demonstrate that the incorporation of image information enhances the triple extraction model's ability to perceive textual semantic information @xcite , but also provide us with strong support for document-level multimodal triple extraction using large language models (llms) @xcite . however, their datasets typically contain only short text segments paired with a single image, representing an idealized scenario. it is worth mentioning that multimodal documents are among the most common data formats in everyday life, such as academic papers, news reports, social media posts, and more. however, traditional multimodal entity-relation extraction models face numerous challenges when processing multimodal documents. firstly, many existing methods face challenges in capturing the global document structure and multimodal correlations, especially when the document is complex, leading to insufficient interaction of contextual multimodal data @xcite . although current language models can handle long text sequences and visual models can process multiple images, their joint processing capability significantly decreases when extended to documents containing multiple images and long texts. achieving good extraction results requires not only capturing global information from the document but also thoroughly understanding the multimodal correlations @xcite @xcite . our framework is designed based on this concept. secondly, integrating a large amount of textual and visual information at the document level remains a complex task, as models must effectively align entities across modalities while ensuring semantic consistency @xcite . lastly, while image modules can provide additional cues, efficiently leveraging multimodal knowledge to correct and refine extracted relations remains a significant challenge @xcite . to the best of our knowledge, this task is highly challenging, and our proposed framework is the first attempt to address it. to address these challenges, we propose a multimodal document-level triple extraction framework (mdocte) via dynamic graph enhancement and multimodal relation-aware reflection. specifically, it divides the document into multiple fine-grained modules and selectively associates them, primarily including global summaries, chunks, images, entities, visual objects, and external knowledge. it is worth noting that we have added feature interaction nodes between chunks and images, aiming to achieve fine-grained cross-modal information fusion. additionally, in order to improve the extraction effectiveness and efficiency, we optimize the document graph based on the chunks to be extracted. next, the knowledge learned by the graph neural network is deeply interacted with the llms to generate the raw entities and relationships. finally, we design a multimodal relationaware mechanism and loss function to prompt the large model to reflect on the shared information between text and visuals, generating accurate triple answers. the main contributions are summarized as follows: ## related work this section reviews the strengths and limitations of existing triple extraction methods from three perspectives: traditional, multimodal, and documentlevel triple extraction. ## approach building on these foundations, we propose the first triple extraction framework designed specifically for multimodal documents (mdocte). as shown in figure 1 , the proposed framework consists of four main steps: document graph construction, adaptive graph optimization, deep interaction generation, and multimodal relation-aware reflection. the implementation details of each step are presented in the following sections. ## conclusion in this paper, we have integrated a multimodal document extraction dataset and designed a triple ex-3219 traction framework. it not only expands the scope to global document context and external world but also leverages multimodal shared information to prompt the model for relation-aware reflection. experimental results demonstrate that the proposed framework outperforms existing methods and effectively mitigates hallucinations. in the future, we will expand more datasets to validate the effectiveness of the proposed framework.