MURAL : Multimodal, Multitask Representations Across Languages (2021)

Abstract: Both image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages. We use both types of pairs in MURAL (MUltimodal, MUltitask Representations Across Languages), a dual encoder that solves two tasks: 1) image-text matching and 2) translation pair matching. By incorporating billions of translation pairs, MURAL extends ALIGN (Jia et al.)–a state-of-the-art dual encoder learned from 1.8 billion noisy image-text pairs. When using the same encoders, MURAL’s performance matches or exceeds ALIGN’s cross-modal retrieval performance on well-resourced languages across several datasets. More importantly, it considerably improves performance on under-resourced languages, showing that text-text learning can overcome a paucity of image-caption examples for these languages. On the Wikipedia Image-Text dataset, for example, MURAL-base improves zero-shot mean recall by 8.1% on average for eight under-resourced languages and by 6.8% on average when fine-tuning. We additionally show that MURAL’s text representations cluster not only with respect to genealogical connections but also based on areal linguistics, such as the Balkan Sprachbund.

## introduction multilingual captions for images provide indirect but valuable associations between languages @xcite . @xcite exploit this to scale multimodal representations to support more languages with a smaller model than prior studies. more recent work learns cross encoder models with multitask training objectives @xcite ; in these, a single multimodal encoder attends to both inputs and exploits deep associations between images and captions. unfortunately, such models do not support efficient retrieval @xcite , and they use object figure 1 : mural learns encoders for both language and images by combining both image-text matching and text-text matching tasks, using scalable dual encoder models trained with contrastive losses. with mural: multimodal, multitask representations across languages (fig. 1 ), we explore dual encoder learning from both image-caption and translation pairs at massive scale: 6 billion translation pairs @xcite and 1.8 billion imagecaption pairs @xcite . we particularly seek to improve performance for under-resourced languages. addressing this was infeasible until now because existing multilingual image-text datasets-multi30k @xcite ), stair @xcite , and xtd (aggarwal and kale, 2020)support only high-resource languages. however, the recent wikipedia image-text (wit) dataset @xcite , which covers 108 languages, addresses this gap. our results, as a whole, demonstrate that align, a state-of-the-art multimodal dual encoder, is improved by adding a bitext ranking objective @xcite zero-shot image-text retrieval performance on wellresourced languages, and it dramatically improves performance on under-resourced languages. for xtd, mural improves recall@10 by 4% on average. on wit zero-shot, mural improves mean recall by 1.7% on average for nine well-resourced languages, and by 8.1% for eight under-resourced ones. after fine-tuning on wit, mural mean recall is 1.8% and 6.8% better than align, on average, for well-resourced and under-resourced languages, respectively. we also show that the resulting dual encoder model can outperform more complex cross-encoder baseline models by a wide margin, thus obtaining stronger performance from models that support scalable retrieval. our largest model, mural-large, improves mean recall for zero-shot retrieval by 47.7% on average for four languages in multi30k over m3p @xcite . it improves mean recall by 5.9% over uc2 @xcite for the fine-tuning setting of multi30k. mural-large also improves over a strong translate-test baseline on wit in a zero-shot setting for wellresourced languages by 13.2% and for underresourced ones by 9.6%. we report results on crisscrossed captions (cxc) @xcite , which additionally provides image-text, text-text, and image-image similarity ratings. mural-large obtains the highest scores to date on cxc @xmath0. our small align model and mural-large model tie for best semantic image similarity, which measures the correlation between model rankings and human rankings over image-image pairs. finally, we show that multilingual representa-tions learned in mural form clusters which are influenced from areal linguistics and contact linguistics, in addition to previously shown genealogical relationships @xcite . ## data for training, we use both publicly available datasets and internal ones that are much larger. we evaluate on many publicly available image captioning datasets. ## models align @xcite is a family of multimodal dual encoders that learn to represent images and text in a shared embedding space. align's encoders are trained from scratch on image-text pairs via an in-batch normalized softmax loss (contrastive learning). this loss encourages the model to encode positive image-text pairs closer to each other while pushing away in-batch negative pairs. align delivers state-of-the-art results for several datasets; however, the alt-text data used to train it is heavily skewed towards well-resourced languages (see fig. 2 ). this imbalance reduces align's ability to represent under-resourced languages; we address that here by using more representative text-text translation pairs mined at scale from the web. ## results we focus on: we number the rows in our results tables to ease reference in our discussion and across tables. the additional text-text task used by mural-base improves zero-shot performance on czech, a relatively lower-resourced language, by a large margin over align-base (4 vs 6), 47.9 → 64.6, while nearly matching or somewhat exceeding performance on higher-resource languages. large, noisy pre-training greatly reduces the need for fine-tuning. m3p sees huge performance gains by fine-tuning foot_3 (1 vs 10), sometimes 3x the zero-shot performance. both align-base and mural-base see large gains, but their zero-shot performance is already near m3p's finetuned performance for highly resourced languages. mural-large's zero-shot (7) actually exceeds m3p's fine-tuned performance (10) and almost matches uc2's fine-tuned performance (11). even with far less data than at+mbt, mural-base trained on cc12m+eobt (5) has much stronger zero-shot performance than m3p (1). with fine-tuning, mural-base (cc12m+eobt) improves on both fine-tuned m3p and uc2 (14 vs 10,11), except for japanese. though mural benefits from four times more image-text pairs than the others (@xmath1, both m3p and uc2 are more complex cross-encoder models that require the translate-train strategy works well compared to using only multilingual image-text pairs (2 vs 4; 12 vs 13) and versus text-text training (2 vs 6; 12 vs 15). given this, using translate-train (2) to increase language diversity in image-text pairs combined with text-text pair training (6) may yield even more gains. as a zero-shot strategy, translate-test also works well . this suggests that smalr's combination of multilingual encoding and translate-test @xcite ) may improve zero-shot performance further with mural (i.e., 3+6+smalr). like others before, we find that training larger models on data of this scale produces remarkable gains: mural-large obtains big improvements even over mural-base. mural-large's results are state-of-the-art for all languages except english (where the larger, english-only align-l2 is best). mural-large does this while-as a dual encoder-also supporting efficient retrieval. this makes a huge difference when retrieving from billions of items rather than the 1k to 5k examples of multi30k's and ms-coco's test sets (for which expensive, exhaustive comparisons can be performed with cross-encoders). see @xcite for extensive discussion and experiments around the computational cost of cross-encoders versus dual encoders for retrieval. wikipedia image text results. we extracted two subsets of wit for evaluation: 1) wellresourced languages and 2) under-resourced languages (more details in appendix a.3). there are no prior results; here, we compare mural with align-base and align-base-en using the translate-test baseline. with wit, the translate-test strategy again proves effective (row 3). it is comparable to both mural-base and align-base in a zeroshot setting-each wins some contests. nevertheless, translate-test fails for the extremely under-resourced waray-waray language because the nmt system lacks support for it. in all, we found that 27 of wit's 108 languages lacked nmt support. thus, we cannot fully rely on translation systems for many under-represented languages; this further bolsters exploration into pivoting on images to overcome data scarcity. furthermore, simple dual-encoder models are fast and simple at test-time, and thus scale better than translate-test. finally, both align-base and mural models benefit from fine-tuning on in-domain multilingual image-text training pairs, foot_4 when available; both obtain very large gains across all languages, and also easily beat the translate-test baseline fine-tuned on @xmath2: semantic simliarity using cxc. wit-en (18, 19, 20 vs 21). xtd. as shown in crisscrossed captions. for cxc image-text retrieval ( the correlation results given in the combined retrieval and correlation lens of cxc indicates there is much more to evaluating multimodal representations than the predominant cross-modal retrieval tasks. ranking a set of items in a manner consistent with human similarity judgments is arguably a harder task than getting a single paired item to be more similar than nearly all others. these two perspectives may reveal useful tensions in finer-grained semantic distinctions. in fact, it is with these correlation measures that we expect cross-encoders to shine compared to the retrievaloriented dual encoders. figure 3 plots the embedding in a 2-dimensional space for two models: 1) labse, a multilingual text-only sentence representation model (feng the fact that mural pivots on images as well as translations thus appears to add an additional view on language relatedness as learned in deep representations, beyond the language family clustering observed in a text-only setting. this suggests potential future work to explore different linguistic phenomena in these representations. it also suggests that it may be worth trying to improve multimodal, multilingual representations for a given lower-resource language by pivoting on a wellresourced language that is linguistically related or which has been in significant contact with itsimilar to previous studies for machine translation @xcite . ## conclusion english provides a strong starting point for learning multilingual representations because it is so widespread and examples of english paired with other languages can be gathered well-beyond that of any other language, currently. we exploit this to train on translation pairs as a means to improve handling of multilingual inputs in cross-modal representations. with simple dual encoder models trained on large-scale datasets via contrastive learning, we obtain consistent, strong retrieval performance across all languages-especially under-resourced ones. our error analysis also indicates that this helps increasing cultural specificity and diversity of the retrieved examples. the nuanced results we obtained for cxc also indicate that further improvements in such models might come from better calibration of the different tasks during learning. we also expect that more aggressive use of the translate-train strategy will straightforwardly yield further gains. embedding visualizations of mural's text representations also illustrates how languages cluster based on multimodal learning. prior work has shown that english is not the ideal pivot language for many under-resourced languages @xcite . our improvements for multilingual and multimodal models suggest further investigations into which well-resourced languages can be better pivots for learning representations for under-resourced languages. in addition to reflecting established language groupings, it also opens up possibilities of discovering new clusters. for instance, the proximity of hungarian and czech (fig 3 ) for mural might be attributed to the geographical proximity of these languages, and warrants further analysis.