Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval (2024)

Abstract: There is a scarcity of multilingual vision-language models that properly account for the perceptual differences that are reflected in image captions across languages and cultures. In this work, through a multimodal, multilingual retrieval case study, we quantify the existing lack of model flexibility. We empirically show performance gaps between training on captions that come from native German perception and captions that have been either machine-translated or human-translated from English into German. To address these gaps, we further propose and evaluate caption augmentation strategies. While we achieve mean recall improvements (+1.3), gaps still remain, indicating an open area of future work for the community.

## introduction vision-language models (vlms) such as clip @xcite are predominantly limited to use in english as a result of the pretraining supervision consisting mostly of english captions. this trend naturally poses an accessibility barrier for non-english speakers. furthermore, cultures around the world differ in their salient concepts @xcite and visual perception @xcite . relying on english supervision in pretraining thus hinders consideration of cross-cultural concepts in object-based tasks such as recognition, detection, and image-text retrieval. example cultural differences present in language with respect to object specificity and importance. for example, past literature @xcite describes differences in how cultures perceive members of an object group (e.g. penguins within the group of birds), indicating that certain groups have stronger associations for specific rather than general object terms. experiments in nisbett and masuda (2013) also illustrate differences between east asians and americans with respect to the perceived importance of background objects figure 1 : example perception differences between native english and german speakers. examples are captions from flickr30k @xcite and multi30k @xcite . note differences in mentioned objects ("sand arena", "parasol") and specificity ("heurigen bench" vs. "table", "horse" vs. "bronco"). german captions here are translated to english. and context as opposed to foreground objects. different cultures notice different objects more; perceptual differences may manifest in objects being included/excluded in a caption, and different objects being relevant in tasks. fig. 1 shows examples of differences in ai datasets for english and german @xcite . there has been some progress in multilingual, multimodal modeling @xcite @xcite and multilingual data creation @xcite @xcite . the models often leverage off-theshelf machine translation techniques to improve multilingual functionality. in this work, we investigate the performance gaps between training models with translations (which reflect english speaker perception) and natively written captions (which reflect non-english speaker perception) for a task in a given language. in line with the observed differences in nisbett and masuda (2013), we reason that translation may not account for specificity differences and may not alter supervision to account for importance differences. we quantify potential differences through an exploration of non-english image-text retrieval. in particular, we finetune and benchmark multilingual clip @xcite on multi30k @xcite using german as the target. we explore multi30k's native german captions (reflecting german speaker perception) and professionally translated captions (from english to german), as well as use of an external machine translation model over flickr30k's english captions @xcite . we find significant performance differences depending on the data used to train the model, i.e. (1) english, (2) german translated from english by a machine translation model, (3) german translated from english by humans, and (4) native german. as (2) and ( 3 ) have gaps vs. (4), we also attempt to improve upon translation. we test three paraphrasing techniques to diversify object descriptions in english before translation, and use the resulting translations as additional finetuning data. first, we experiment with a hypernymization data augmentation technique, where object terms are updated before translation to represent different levels of specificity. second, we use a large language model (llm), llama-3 @xcite , to produce structurally different, but semantically similar paraphrases of english captions before translation. third, we explore llm reasoning to produce targeted paraphrases that capture the perceptual properties captured in a sample set of captions. these techniques outperform the baselines. however, a gap between translation and native perception remains, indicating an open problem. we conclude with analysis in pursuit of this direction. ## key findings in the top block of in the third block, our methods are found to be somewhat effective for bridging the gap between eng2ger-mt and ger. hyper improves the result by 0.3, and para-rnd and para-tgt by 0.7. these models are notably more appropriate for low-resource target languages than eng2ger-mt (trn) since they use no/few reference captions compared to translation finetuning. further combining random and targeted paraphrasing results in the largest gain of 1.3. the result is still 3.7 away from ger. addressing differences in the perception of the visual world and the way captions are written across cultures is thus an open challenge. ## further analysis object mentions in english/german captions. to analyze possible differences in perception, we analyze object mention frequency in flickr30k/multi30k. we specifically translate german captions to english and extract nouns in both (original) english and (translated to english) german captions. the ratio of english and german mentions is about 1.5, i.e. english mentions object nouns 50% more often than german. however, counts vary by object type. for example, english mentions clothing more often (pants-143% more, shirt-112%, hat-60%, jacket-43%), and german mentions furniture more often (table-37% more, bed-20%, bench-15%). these languages also vary in granularity: english captions often say "people", while german ones say "workers", "athletes", etc. analysis of other languages. we conduct initial analysis of the languages and captions in xm3600 @xcite . we group xm3600 languages into european, arabic/farsi, hindi/bengali, indonesian/thai, east asian, and swahili categories. after translating each language to english, we report average mention counts and standard deviations per group for various common objects in paraphrasing. llama picks up on granularity differences. for example, para-tgt changes "man in a red shirt riding his bicycle" to "a bicyclist in a red shirt is riding". further, llama transforms "man on skis" into "skier", "person in blue and red ice climbing" into "ice climber", and "men with children" into "family". the model tends to simplify, irrespective of the reference. for example, "two young people are approached by a flamboyant young woman dressed in a red bikini and a red feathered headress" becomes "two young people are approached by a bikini-clad woman". paraphrasing could thus result in over-simplification. human evaluation. we extend quantification past retrieval by asking two german speakers to gauge the likelihood that captions are made by a german speaker and their naturalness. we provide 50 random captions for each of 3 sets (eng2ger-mt, eng2ger-ht, para-tgt). speakers do not know each set's identity and are tasked with scoring captions as 3=great, 2=good, 1=bad. eng2ger-ht. we take predictions to be ones with clip scores greater than a threshold (the one in range 10:5:50 that maximizes val f1). a prediction is correct only if the object is mentioned in native german. ## conclusion we show notable differences in using native vs. translated german captions to train a retrieval model, and experiment with three strategies to reduce the gaps. we plan to extend investigation to more languages. future work can also involve creation of data augmentation strategies that take inspiration from psychology literature @xcite and solutions for the ambiguity challenges of machine translation, such as by using images @xcite .