Translation-Enhanced Multilingual Text-to-Image Generation (2023)

Abstract: Research on text-to-image generation (TTI) still predominantly focuses on the English language due to the lack of annotated image-caption data in other languages; in the long run, this might widen inequitable access to TTI technology. In this work, we thus investigate multilingual TTI (termed mTTI) and the current potential of neural machine translation (NMT) to bootstrap mTTI systems. We provide two key contributions. 1) Relying on a multilingual multi-modal encoder, we provide a systematic empirical study of standard methods used in cross-lingual NLP when applied to mTTI: Translate Train, Translate Test, and Zero-Shot Transfer. 2) We propose Ensemble Adapter (EnsAd), a novel parameter-efficient approach that learns to weigh and consolidate the multilingual text knowledge within the mTTI framework, mitigating the language gap and thus improving mTTI performance. Our evaluations on standard mTTI datasets COCO-CN, Multi30K Task2, and LAION-5B demonstrate the potential of translation-enhanced mTTI systems and also validate the benefits of the proposed EnsAd which derives consistent gains across all datasets. Further investigations on model variants, ablation studies, and qualitative analyses provide additional insights on the inner workings of the proposed mTTI approaches.

## introduction and motivation text-to-image generation (tti) is an emerging yet rapidly growing area, owing its recent progress to ever-growing deep generative models, largerscale multi-modal datasets, and increasing computational resources. the success of recent tti work is impressive; e.g., it is possible to synthesise not only high-resolution complex scenes (ramesh et al., 2022; rombach et al., 2022), but also sur-realist and 'aesthetics-aware' paintings (gallego, 2022). however, current models are made and deployed almost exclusively for the english language (en). this is primarily due to the lack of annotated imagecaption data in other languages, which might result in inequitable access to tti technology in the long run, especially for low-resource languages (blasi et al., 2022). hiring human annotators to write high-quality image descriptions is time-consuming and expensive; 'gold standard' data, if it exists at all, is thus typically used for evaluation purposes only (lan et al., 2017; aggarwal and kale, 2020). even if we put the crucial concerns of data scarcity aside, training state-of-the-art (sota) tti models from scratch for each language is technically infeasible and impractical: it would consume massive computational resources, exceeding the capabilities of many research labs (ramesh et al., 2021; saharia et al., 2022) and raising concerns of its environmental impact (schwartz et al., 2020). @xcite therefore, in this work, we focus on multilingual tti (mtti) through the optics of nlp's crosslingual transfer learning methods, leaning on the reasonable assumption of having abundant imagetext pairs in english (and/or a pretrained en tti model), but only limited gold-standard data for fine-tuning and evaluation in a target language. @xcite in particular, we investigate the role of crosslingual transfer and (neural) machine translation (mt) in bootstrapping mtti, and we focus on two crucial research questions. (rq1) are standard mt-based cross-lingual transfer methods feasible for mtti, and how do they compare with standard zero-shot cross-lingual transfer methods? (rq2) is it possible to enhance zero-shot cross-lingual transfer relying on (ensembles of) mt-generated output for improved mtti? our experiments and core findings are based on several mtti benchmarks. first, we use the standard and publicly available coco-cn (li et al., 2019) and multi30k (elliott et al., 2016), and we also build a new dataset for finnish as a lower-resource language from laion-5b (schuhmann et al., 2022). regarding rq1, we then conduct a systematic empirical study comparing the standard cross-lingual transfer methods: trans-late train, translate test, and zero-shot transfer. our main results indicate that trans-late train achieves the best performance, followed by zero-shot transfer which outperforms translate test. regarding rq2, we aim to combine mt-based and zero-shot cross-lingual transfer via fast and parameter-efficient fine-tuning. inspired by the speech processing literature where a list of automatic speech recognition (asr) hypotheses can be jointly considered for downstream tasks (ganesan et al., 2021; liu et al., 2021) to alleviate the misrecognition of asr systems, we propose a module within our mtti framework termed ensemble adapter (ensad). it fuses the text encodings of 'non-english' text input and a set of its translations to english. additionally inspired by ponti et al. (2021), the idea is to combine the knowledge from multiple translations to mitigate potential translation errors, and that way boost cross-lingual transfer for mtti. our proposed method derives robust gains across all evaluation datasets. besides offering sota mtti performance, the introduced ensad component also adds only 0.1% dedicated extra parameters (relative to the full mtti model size) per each supported target language. put simply, the use of ensad increases the portability of our mtti framework through quick and parameter-efficient adaptation to new languages. the resources of our work are available at https://www.amazon.sci ence/code-and-datasets/translation-enh anced-multilingual-text-to-image-gener ation . ## methodology in what follows in this section, we first introduce our base mlafite model and three baseline approaches for mtti ( §3.1). next, we propose an ensemble adapter module that can work in synergy with the pretrained mlafite model to improve mtti performance ( §3.2). finally, we describe how we train our ensemble adapter and formulate our loss functions ( §3.3). ## experimental setup in what follows, we outline our experimental setups and choices related to the two core rqs from §1. we also show details concerning our mlafite pretraining, side experiments (most are rq2-related), and evaluation metric. rq1 experiments. on coco-cn, we compare four widely used mt systems: amazon translate foot_11 , a sota commercial mt software, and three sota transformer-based nmt models developed in an academic context including marian (tiedemann and thottingal, 2020; junczys-dowmunt et al., 2018), mbart50 (liu et al., 2020; tang et al., 2021), and m2m100 (fan et al., 2021). we leverage them to generate the 1-best translations for translate train and translate test, and we also compare the bleu scores of the mt systems against the tti performance. note that training a translate train tti model from scratch for each of the mt systems also takes 75 hours; our translate train experiments thus do not extend to other datasets beyond coco-cn due to the high computational cost. given the considerations above along with preliminary evaluations on coco-cn which showed that marian outperforms mbart50 and m2m100, for the other datasets we focus on comparing the marian-based translate test with zero-shot transfer. ## results and discussion the main results are structured around the two central rqs from §1, discussed in §6.1 and §6.2. ## conclusion this work is one of the first investigations of multilingual and cross-lingual text-to-image generation (tti), with a particular focus on investigating the use of machine translation (mt) for the task. we systematically compared standard cross-lingual transfer approaches translate train, trans-late test and zero-shot transfer in the context of tti and also studied the differences over mt systems. we then proposed a novel ensemble adapter (ensad) method that leverages multiple translations to further improve the tti performance, with strong and consistent gains reported across a series of standard tti benchmarks in different languages.