D ongba MIE : A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictograms (2025)

Abstract: Dongba pictographic is the only pictographic script still in use in the world. Its pictorial ideographic features carry rich cultural and contextual information. However, due to the lack of relevant datasets, research on semantic understanding of Dongba hieroglyphs has progressed slowly. To this end, we constructed DongbaMIE - the first dataset focusing on multimodal information extraction of Dongba pictographs. The dataset consists of images of Dongba hieroglyphic characters and their corresponding semantic annotations in Chinese. It contains 23,530 sentence-level and 2,539 paragraph-level high-quality text-image pairs. The annotations cover four semantic dimensions: object, action, relation and attribute. Systematic evaluation of mainstream multimodal large language models shows that the models are difficult to perform information extraction of Dongba hieroglyphs efficiently under zero-shot and few-shot learning. Although supervised fine-tuning can improve the performance, accurate extraction of complex semantics is still a great challenge at present.

## introduction dongba pictographic script is currently mainly used by the naxi ethnic group in southwest china. as an important part of the cultural heritage of the naxi ethnic group in china @xcite , ancient books written in dongba pictographic script have been officially listed in the international memory of the world register by the united nations educational, scientific and cultural organization (unesco). figure 1 shows an image of dongba pictographic script, which presents content composed of exquisite handwriting. it is recorded on local paper and mainly involves dongba religion, naxi history and lifestyle. however, there are severe challenges in the digital preservation and processing of dongba pictographic script @xcite . first, since there are very few people who understand dongba pictographic script, only a few naxi priests ("dongba") can read this script today. this makes the relevant modern language annotation resources extremely scarce. secondly, dongba pictographic script lacks a standardized encoding system, and its grammatical structure is also significantly different from that of modern languages @xcite . this makes it impossible to process it like regular text. these factors make it difficult to apply traditional natural language processing methods to the processing of dongba pictographic scripts, and further aggravate the severe shortage of modern annotation corpus. therefore, it is particularly important to construct a multimodal semantic understanding dataset based on dongba pictograms, which will provide valuable resources for research and application in this field. recent advances in deep learning have provided new opportunities for endangered language processing @xcite @xcite . however, the understanding of contextual semantic information in dongba pictograms has not been further studied. in addition, dongba pictograms have unique linguistic phenomena. this includes the omission of pictographs in contextual sentences and the presence of polysemous characters with multiple meanings. these characteristics further increase the difficulty of deciphering and preserving these endangered texts through computation. to improve the semantic understanding of dongba hieroglyphs, we introduce a novel multimodal information extraction dataset, dongbamie, which is derived from "the annotated collection of naxi dongba manuscripts" @xcite . dongbamie contains 23,530 sentence-level and 2,539 paragraph-level images of precisely scanned manuscripts. it is also annotated with object, action, relationship and attribute information from its chinese translation text. we evaluated multimodal large language models (mllms) at dong-bamie. this includes zero-shot and few-shot performance of proprietary models such as gpt-4o @xcite , and supervised fine-tuning (sft) performance of open-source models such as llava-next @xcite and qwen2-vl @xcite . experiments show that current mllms struggle with this task. for example, the object extraction f1 of gpt-4o is only 1.60 under zero-shot. despite the performance improvement of sft, mllms show significant shortcomings in extracting complex semantics, especially in extracting relations and attributes. our contributions are as follows: ## methodology this section details our experimental methodology for evaluating mllms on the dongbamie dataset. we conducted zero-shot and one-shot evaluations for leading proprietary mllms, specifically gpt-4o @xcite and gemini-2.0 foot_0 , and supervised fine-tuning for a diverse set of open-source mllms including llava-next @xcite , minicpm-v-2.6 @xcite , qwen2-vl @xcite , and cogvlm2 @xcite . ## conclusion in this study, we introduced dongbamie, the first multimodal information extraction dataset for dongba pictograms. it aims to advance the processing of endangered scripts. by evaluating against state-of-the-art closed-source and opensource mllms, we found the limitations of zeroshot and few-shot methods, while demonstrating the potential of supervised fine-tuning. despite this, current models still struggle to understand the complexity of dongba pictograms. in the future, we will focus on expanding the dataset annotations with more fine-grained annotations. at the same time, we will improve the multimodal representation to further enhance the ability to extract information from dongba pictograms. our dataset will be released soon after the acceptance of the paper. this is intended to facilitate reproducible research in the field of endangered text conservation.