Cross-Modal Augmentation for Low-Resource Language Understanding and Generation (2025)

Abstract: This paper introduces a multimodal retrieval-augmented generation (RAG) system designed to enhance language understanding and generation for low-resource languages. By integrating textual, visual, and geospatial data, the system leverages cross-lingual adaptation and multimodal augmentation to bridge the gap between high-resource and low-resource languages. Evaluated on the MM-COVID and LORELEI datasets, the system demonstrates superior performance in retrieval (precision: 85%, recall: 82%) and generation (BLEU: 28.4) tasks compared to baselines. Case studies in public health communication and disaster response highlight its practical utility. The results underscore the potential of multimodal AI to democratize access to technology and address global challenges in low-resource settings.

## introduction in recent years, advancements in natural language processing (nlp) have revolutionized how we interact with ai systems, enabling applications like machine translation, summarization, and questionanswering. however, these successes are heavily skewed toward high-resource languages, leaving low-resource languages severely underrepresented. the lack of large-scale textual corpora in low-resource languages poses significant challenges for training robust language models, limiting their ability to understand and generate meaningful content. this disparity not only exacerbates global inequities in access to technology but also hinders efforts to address critical issues such as public health communication, disaster response, and education in multilingual contexts @xcite . to bridge this gap, we propose cross-modal augmentation for low-resource language understanding and generation, a novel framework that leverages multimodal data text, images, geospatial information, and structured data-to enhance language understanding and generation in low-resource settings. by integrating complementary modalities, our approach compensates for the scarcity of textual resources and enriches the semantic context available to language models. for example, visual data can provide additional grounding for concepts that are poorly represented in text, while geospatial data can help localize and contextualize events described in queries @xcite . our work builds on datasets like mm-covid @xcite and lorelei @xcite , which offer rich multimodal information relevant to real-world challenges. mm-covid provides multilingual textual and visual data related to the covid-19 pandemic, enabling us to test the system's ability to generate public health information in low-resource languages. similarly, lorelei offers low-resource language data alongside geospatial and event information, making it ideal for tasks like disaster response and situational awareness. by combining these datasets with retrieval-augmented generation (rag) techniques, we demonstrate how cross-modal augmentation can significantly improve performance in tasks such as translation, summarization, and questionanswering @xcite . the contributions of this paper are threefold: ## experiments to evaluate the effectiveness of our multimodal rag system, we conducted experiments on two key datasets: mm-covid @xcite and lorelei @xcite . these datasets were chosen for their relevance to realworld challenges and their inclusion of multimodal data. below, we describe how these datasets were applied to the task of multimodal rag for lowresource languages, along with the experimental setup. ## results and discussion the results of our experiments demonstrate the effectiveness of our multimodal rag system in enhancing language understanding and generation for low-resource languages. our multimodal rag system outperformed all baselines across both retrieval and generation tasks. in terms of retrieval metrics, the system achieved a precision of 85%, recall of 82%, and f1 score of 83%, surpassing the text-only rag baseline by 10 percentage points. for generation tasks, the system achieved bleu scores of up to 28.4, compared to 20.5 for unimodal models. these improvements highlight the value of integrating multimodal data into the retrieval and generation processes. notably, the system performed particularly well on low-resource languages, where the scarcity of textual data was compensated by visual and geospatial information. the results in these results collectively demonstrate that our multimodal rag system effectively leverages multimodal data to improve both retrieval and generation capabilities. the precision vs. recall plot (figure 1 ) demonstrates several key trends: the bleu and rouge-l plot (figure 2 ) further supports the quantitative findings: • the multimodal rag system achieves the highest bleu score (28.4) and rouge-l score (38.2), as shown by the tallest bars in the plot. human evaluations revealed that the multimodal rag system produced responses that were not only fluent but also contextually relevant and coherent. for example, in the public health case study, the system generated accurate summaries of covid-19 guidelines in swahili, enriched with relevant images. similarly, in the disaster response case study, the system successfully identified flood zones in pashto by combining satellite imagery with textual reports. these qualitative insights underscore the system's ability to leverage multimodal data effectively. ## conclusion in this paper, we presented a multimodal rag system that effectively enhances both understanding and generation capabilities for low-resource languages. by leveraging multimodal data and crosslingual transfer, the system achieved state-of-theart performance on the mm-covid and lorelei datasets, surpassing traditional text-only and unimodal baselines. key findings include significant improvements in retrieval precision, recall, and generation quality, as well as robust performance in real-world applications like disaster response and public health communication. despite challenges such as noisy data and computational overhead, our system demonstrates the transformative potential of multimodal ai in addressing linguistic and resource disparities.