From Text to Multi-Modal: Advancing Low-Resource-Language Translation through Synthetic Data Generation and Cross-Modal Alignments (2025)

Abstract: In this study, we propose a novel paradigm for multi-modal low resource language dataset generation that eliminates dependency on existing parallel multi-modal datasets. Leveraging advances in large image-generation models, we introduce a systematic pipeline that transforms text-only parallel corpora into rich multi-modal translation datasets. We then validate the generated content through human evaluation. We design and implement a new MMT model framework suitable for our new generated dataset. The model contains a verification mechanism with a large language model to ensure consistency between visual content and textual translations. Experimental results across four African low-resource languages with less than 10k training corpus demonstrate significant improvements over NLLB baselines, with average gains of up to 9.8% in BLEU score and 4.3% in METEOR score. Our method shows particular effectiveness in correctly translating concrete objects and contextual elements, suggesting its potential for improving low-resource machine translation through visual grounding.

## introduction large-scale pre-trained language models lead neural machine translation research into a new stage @xcite , researchers are applying these powerful models to enhance translation for mainstream languages. however, these new models have not yet been widely adopted for translation tasks between low-resource and mainstream languages, especially some low-resource african languages. the few studies about machine translation for hausa @xcite , tigrigna @xcite and kanuri @xcite mainly used lstm or hmm-based models, not powerful transformer-based models. although there are relatively more studies on machine translation for yorùbá, there is still a huge gap in the application of multimodal machine translation models for yorùbá, which deserves further exploration @xcite @xcite . currently, multimodal models have also been widely used in research works on machine translation. but in the research of multimodal lowresource language machine translation, several issues still warrant further discussion. for example, existing multimodal low-resource language parallel translation corpora are scarce, and some of them include only a limited number of lowresource languages. most low-resource languages haven't been included in widely used multimodal low-resource parallel translation corpora. some low-resource multimodal parallel translation corpora, such as multi30k @xcite , are created by researchers through manual annotation, which involves significant time and financial costs. while these datasets are of very high value, most researchers cannot adopt the same approach when creating more low-resource multimodal parallel translation corpora. it's hard to deny that the scarcity of corpora and the high costs of creating them pose significant obstacles to research in multimodal translation for low-resource languages. additionally, the difficulty of adapting pre-trained multimodal models to low-resource languages also hinders the application of works such as cliptrans @xcite to translation tasks involving low-resource languages. furthermore, in the existing low-resource multimodal parallel translation datasets, the semantic consistency between text and images is still an issue worth considering. for instance, in the low-resource multimodal parallel translation dataset wit @xcite created using wikipedia, many texts consist of proper nouns, such as the names of some special buildings. if multimodal models cannot perfectly distinguish each building's unique name from a large number of similar building images, the translation results will easily be erroneous. ## methodology we propose a multimodal translation framework built upon nllb (costa-jussà et al., 2022) that enhances traditional text-based translation with visual context awareness. our approach integrates three specialized components: nllb's powerful transformer-based translation capabilities, vision model blip @xcite for image image caption generation, and open-source large language model llama3 @xcite for crossmodal alignment supervision. this architecture is specifically designed to address the challenges of low-resource language translation by leveraging visual cues while maintaining nllb's broad multilingual support. ## conclusion low-resource languages have been particularly neglected in multimodal translation research precisely because they lack perfectly aligned multimodal datasets. we presented a comprehensive approach to advance low-resource machine translation. our method transforms text-only parallel corpora into visually enriched datasets using llm-guided image generation, ensuring semantic alignment between the generated visual content and original sentences. building on this multimodal data set, we propose a novel translation architecture that takes advantage of visual context and llm assistance to improve translation quality. experimental results in four low-resource african languages demonstrate significant improvements over nllb baselines in bleu scores and meteor scores. beyond these immediate results, our approach offers broader implications for language preservation and accessibility. the framework's adaptability makes it promising for any other low-resource languages, potentially aiding in the digital preservation of endangered languages and facilitating cross-cultural communication. this flexible solution could contribute significantly to reducing language barriers in digital spaces while helping preserve linguistic diversity for future generations.