Constructing Multimodal Datasets from Scratch for Rapid Development of a J apanese Visual Language Model (2025)

Abstract: To develop high-performing Visual Language Models (VLMs), it is essential to prepare multimodal resources, such as image-text pairs, interleaved data, and instruction data. While multimodal resources for English are abundant, there is a significant lack of corresponding resources for non-English languages, such as Japanese. To address this problem, we take Japanese as a non-English language and propose Japanese multimodal datasets for rapidly developing a Japanese multimodal model. We collect Japanese image-text pairs and interleaved data from web archives and generate Japanese instruction data using an existing large language model and a VLM. Our experimental results show that a VLM trained on these native datasets outperforms those relying on machine-translated content. The resulting VLM, dataset and code used for training is publicly available.

## introduction we develop a multimodal resource for highperforming visual language models (vlms) in japanese. while english multimodal resources are relatively abundant in existing studies, there is a significant lack of corresponding datasets for non-english languages, such as japanese. one potential solution is translating multimodal resources from english to japanese. however, this approach often produces suboptimal results, as the translation does not account for the contextual relationship between the image and the text. such a translation approach also cannot follow the cultural backgrounds of the image domains as they are collected in english websites. to address this multilingual gap, we propose japanese multimodal datasets for rapidly developing japanese multimodal models. we build two types of datasets: pretraining data and instruction data in japanese. for the pretraining data, we curate a large-scale dataset of japanese image-text pairs and interleaved data by extracting japanese data from web crawls. for the instruction tuning data, we follow the llava @xcite methodology to build japanese instruction data from image captions and object bounding boxes. also, we directly generate japanese instruction data by giving japanese images to an existing vlm via apis, enhancing inference capabilities for images unique to japan. we demonstrate that models trained on our proposed datasets achieve higher accuracy than those relying on machine-translated datasets, and this performance gain is especially noticeable in the instruction data. for this purpose, we apply the vila @xcite architecture to our japanese vlm with english publicly available data and japanese newly collected data. we assume that this approach is adaptable to any languages and not limited to japanese. our contributions significantly enhance the resources available for japanese vlm studies, enabling more effective regional localization and cultural understanding in vlms. resource for visual language models visual language models require a large amount of imagetext pairs for both pretraining and instruction tuning. visual instruction tuning of llava @xcite relies on synthesized data by ope-nai gpt @xcite . this synthesized instruction data are used in the various vlm developments such as instructblip @xcite and vila @xcite . following these successes, we examine the use of the synthesized data in instruction tuning in japanese vlm development. ## conclusion in this paper, we constructed japanese multimodal datasets to develop a high-performance japanese visual language model. we collected images and texts from japanese websites to construct datasets that include data reflecting the cultural background of japan. we also constructed japanese multimodal instruction data using existing llms. by using proposed datasets, we developed llm-jp-3 vila, a japanese visual language model designed to integrate multiple images with natural language understanding. our experiments demonstrate the model's effectiveness across a range of multimodal tasks in japanese. for future work, we plan to extend our model to wide branch of japanese visual datasets.