Developing J apanese CLIP Models Leveraging an Open-weight LLM for Large-scale Dataset Translation (2025)

Abstract: CLIP is a foundational model that bridges images and text, widely adopted as a key component in numerous vision-language models.However, the lack of large-scale open Japanese image-text pairs poses a significant barrier to the development of Japanese vision-language models.In this study, we constructed a Japanese image-text pair dataset with 1.5 billion examples using machine translation with open-weight LLMs and pre-trained Japanese CLIP models on the dataset.The performance of the pre-trained models was evaluated across seven benchmark datasets, achieving competitive average scores compared to models of similar size without the need for extensive data curation. However, the results also revealed relatively low performance on tasks specific to Japanese culture, highlighting the limitations of translation-based approaches in capturing cultural nuances. Our dataset, models, and code are publicly available.

## constructing a japanese image-text pair dataset to construct a japanese image-text pair dataset, we translated the captions of the english subset of relaion-5b foot_1 into japanese using gemma-2-9b-it foot_2 , a high-performance open-weight llm. relaion-5b is a refined version of laion-5b @xcite , with child sexual abuse material (csam) removed. it is a largescale dataset of image-text pairs, where images and their corresponding img-alt text are collected from common crawl and filtered using existing clip models. the dataset is divided into three subsets: english, multilingual, and no-language. to enable the rapid translation of large datasets, we developed text2dataset foot_3 , a translation tool for llms. this tool utilizes vllm @xcite , a fast llm inference library, to efficiently translate large-scale english datasets into japanese. prompt to translate text using llms, it is crucial to provide both the text for translation and a clear instruction prompt @xcite . in this study, we used the following prompt: you are an excellent english-japanese translator. please translate the following sentence into japanese.\n you must output only the translation.\n sentence :{passage}\n translation: the {passage} is replaced with the source text for translation. the llm is then expected to generate the translated text based on this prompt. ## training clip we describe the training settings of llm-jp-clip-vit-b/16 as our default model in this section. we pre-trained clip models using the constructed dataset. in this study, we used vit-b/16 @xcite as the image encoder and roberta base @xcite as the text encoder. the output dimension of each encoder was set to 512, and both were trained from english template japanese template a photo of the {} {}の写真 a sketch of a {} {}のスケッチ a photo of the cool {} かっこいい{}の写真 for optimization, we used adamw with hyperparameters of β 1 = 0.9, β 2 = 0.98, and ϵ = 10 -6 . learning rate scheduling consisted of 2,000 steps of linear warmup followed by cosine decay, with a peak learning rate of 5.0 × 10 -4 and a minimum learning rate of 0.0. we trained the model for 9 epochs, processing a total of 13 billion examples. we employed the contrastive loss function proposed by @xcite . the batch size was set to 8,192, with gradient accumulation over four steps. notably, the accumulated loss differs from the contrastive loss computed directly with a batch size of 32,768. we used openclip (ilharco et al., 2021) as the training framework and trained the model on 16 nvidia h100 80gb gpus, requiring two weeks for training. ## evaluation we evaluated the performance of our models by comparing it with japanese and multilingual baseline clip models on zero-shot image classification and image-text retrieval tasks. 8 https://github.com/llm-jp/llm-jp-tokenizer ## 166 conclusion in this study, we constructed a large-scale japanese image-text dataset using translation with openweight llms and pre-trained japanese clip models on the dataset. the results demonstrated competitive performance in the average score across the benchmark datasets compared to models of similar size. however, the performance on tasks related to japanese culture was relatively low, highlighting the limitations of translation-based approaches in capturing cultural nuances. future work includes building more diverse and high-quality japanese image-text datasets and further improving the performance of japanese clip models.