LAMBDA : Large Language Model-Based Data Augmentation for Multi-Modal Machine Translation (2024)

Abstract: Multi-modal machine translation (MMT) can reduce ambiguity and semantic distortion compared with traditional machine translation (MT) by utilizing auxiliary information such as images. However, current MMT methods face two primary challenges. The first is their underperformance compared to MT methods based on pre-trained models. The second is the inadequate exploitation and integration of the image modality within the model, primarily due to a lack of triplet training data. A mainstream approach is to introduce large amounts of parallel and monolingual data to train the text model and the visual model separately. However, incorporating extensive external data can result in data imbalance, which may introduce biases during training. Additionally, the collection and cleaning of such large datasets is labor-intensive. To overcome these challenges, we introduce a novel, low-cost, large language model-based data augmentation method called LAMBDA, which can enrich the original samples and expand the dataset without requiring external images and text. We propose a fine-grained image captioning module with a noise filter to hierarchically and accurately extract unexploited information from images. Additionally, we design two specific prompts to guide the GPT-3.5 model in generating enriched texts and the corresponding translations. The enriched samples contain diverse text and strong connections between text and images, leading to significant improvements for MMT baselines, with the highest being an increase of up to 3.83 BLEU score and 3.61 METEOR score.

## introduction multi-modal machine translation (mmt) integrates additional modalities such as the visual modality (images and videos) to enhance language understanding, which can reduce ambiguity and semantic distortion to improve translation quality (li et al., 2022a; @xcite . however, some studies have found that it is difficult for mmt methods to surpass strong text-only baselines @xcite . a consensus on this problem is that most mmt methods are only trained on a small text-image pair dataset called multi30k @xcite . this leads to two main issues: (i) the number of image-text pairs is small compared to the text-only corpora used for training mt methods. (ii) the description texts are short and lack diversity. each text is simple and only describes a part of the corresponding image, as illustrated in figure 1 , which results in a large portion of visual content being neglected, limiting the opportunity for mmt models to explore the correlation between images and texts thoroughly. a straightforward solution is to develop an extensive multi-modal dataset from scratch, containing diverse samples with detailed annotations. however, this approach incurs substantial costs and requires considerable time for data collection, annotation, and preprocessing. since the data uti-lized for the mmt task is triplet data, where image data, source language data, and target language data must be aligned @xcite . the high costs associated with acquiring such triplet data present a significant challenge. thus, some mmt methods @xcite attempt to leverage widely available doublet data, including image data and source language data (monolingual captions), and source language data and target language data (sentence translation pairs), from different sources to enhance the capabilities of the visual and text models, respectively. however, incorporating extensive external data from different sources may result in data imbalance, leading the model to favor predicting the more prevalent patterns, thus introducing biases during training @xcite . besides, data collection and cleaning significantly increase manual resource demands. another rising direction is using generative models like stable diffusion model @xcite to generate synthetic images based on text descriptions @xcite . nevertheless, synthetic images may exhibit different distribution patterns compared to authentic images @xcite . to address the aforementioned problems and circumvent obstacles encountered by existing mmt methods, we introduce a large language modelsbased data augmentation method (lambda). lambda can automatically enrich the text descriptions and enhance the connections between modalities for the samples in the multi30k dataset without relying on external data. specifically, we divide the image into different fine-grained-level patches, allowing the less prominent regions of the entire image to be transformed into focal points within these patches. then we use the blip model @xcite to generate fine-grained captions from different perspectives to obtain a comprehensive image description. considering the potential noise like incorrect recognition generated by the blip model, we implement the clip model @xcite to assess the relevance between the generated captions and the corresponding image patches, discarding unrelated and incorrect captions. subsequently, we design two specific prompts to guide the gpt-3.5 model @xcite in logically enriching the original description by integrating the different fine-grained-level captions with the original description and generating corresponding, same-style translations using in-context learning @xcite . finally, we combine the enriched descriptions with the orig-inal images and generated translation to create an augmented dataset, which is then trained alongside the original dataset. our main contributions are summarized as follows: ## conclusion in this work, we devise an innovative data augmentation method, lambda, to address faced challenges in the current mmt field. it involves a fine-grained image captioning module, a caption filter, a sentence augmentation component, and two specific prompts to effectively augment the original samples from the multi30k dataset without relying on external image and text data. comprehensive experiments conducted on three benchmark datasets for @xmath0, validating the effectiveness. furthermore, results on the commute dataset confirm that lambda enhances the mmt model's ability to exploit the visual modality. in future work, we will explore effective methods to reduce noise in the generation stage and incorporate a broader range of large llms, including bloom, opt, and gpt-4, to comprehensively compare our method's performance across these diverse models.