Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Concepts (2024)

Abstract: Benchmarks of the multilingual capabilities of text-to-image (T2I) models compare generated images prompted in a test language to an expected image distribution over a concept set. One such benchmark, “Conceptual Coverage Across Languages” (CoCo-CroLa), assesses the tangible noun inventory of T2I models by prompting them to generate pictures from a concept list translated to seven languages and comparing the output image populations. Unfortunately, we find that this benchmark contains translation errors of varying severity in Spanish, Japanese, and Chinese. We provide corrections for these errors and analyze how impactful they are on the utility and validity of CoCo-CroLa as a benchmark. We reassess multiple baseline T2I models with the revisions, compare the outputs elicited under the new translations to those conditioned on the old, and show that a correction’s impactfulness on the image-domain benchmark results can be predicted in the text domain with similarity scores. Our findings will guide the future development of T2I multilinguality metrics by providing analytical tools for practical translation decisions.

## introduction with growth in the popularity of generative textto-image (t2i) models has come interest in assessing their capabilities across many dimensions, including multilingual accessibility. the coco-crola @xcite benchmark attempts to capture how well "concept-level knowledge" within a t2i model is accessible across different input languages. it compares the output image populations of a system under test when prompted to generate images of 193 tangible concepts in 7 test languages to the images generated from a semantically equivalent prompt in a source language. it and similar benchmarks rely on correct translations for validity, lest "possessed" concepts be mistakenly assigned false negatives. concept: bike concept: bike en ja-original ja-corrected en バイク 'motorbike' 自転車 'bicycle' concept: suit en zh-original en zh-corrected 适合 'suitable, fitting' 西装 lit. 'western suit' we find a strict error candidate rate of 4.7% for spanish (es), 8.8% for chinese (zh), and 12.9% for japanese (ja) in the coco-crola v1 (cccl) concept translations through manual analysis by fluent speakers. these error candidates are not filtered by severity. while some candidates are severe translation errors that drive false negatives (figure 1 ), others are marginal annotator disagreements that might not matter ( concept language original corrected reason for correction rock japanese ロック 岩 ロック, rokku, refers principally to "rock music" instead of stones in nature. flame spanish llama flama llama, though a correct translation for "flame," coincides with the animal in english. ground japanese 接地 地面 接地 refers to an electrical ground rather than the surface of the earth. table chinese 表 桌子 表 means a tabular form or a spreadsheet, not a four-legged furniture. milk japanese 乳 牛乳 乳 may mean breast or any kind of milk. 牛乳 means the milk produced by cows. tent spanish tienda ...de acampar tienda alone more often means "store," tienda de acampar specifies (camping) tent. teacher japanese 先生 教師 先生 is a common title to address an educated person, e.g., teacher, doctor, lawyer. father chinese 爸爸 父亲 爸爸 is the colloquial addressing equivalent to 'daddy'. 父亲 is more formal. ## motivation & approach the coco-crola benchmark (cccl) evaluates a t2i model's ability to generate images of an inventory of tangible concepts when prompted in different languages @xcite . given a tangible concept c, written in language ℓ as phrase c ℓ , the i-th image produced by a multilingual t2i model f on the concept c ℓ can be expressed as: the images generated in language ℓ are considered correct if they are faithful to their equivalent counterparts in the source language ℓ s . this is measured by the cccl benchmark by a correctness metric for a single concept c as the crossconsistency score x c (f, c ℓ , c ℓs ): ## results & analyses we generate output images using stablediffusion 1.4, 2.0, 2.1 @xcite and altdiffusion @xcite , for all concepts corrected by our annotators in english, spanish, chinese, and japanese, using both the original concept translations c ℓ from coco-crola v1 @xcite and the corrected translations c ′ ℓ . model details are provided in appendix a.4. figure 2 shows the relationship between ∆sem and ∆x c for all corrected concepts for stablediffusion 1.4, 2.0, 2.1, and altdiffusion 1 . note the pronounced, significant positive slope of the corre-1 error margins are 95% regression-fit confidence intervals. these same high-slope model/language pairs (eg., ja & altdiffusion) were found by @xcite to be "well-possessed" (high average x c across correct concepts) in coco-crola v1. in other words, valid corrections only matter for languages a model already "knows." correct klingon is just as useless as incorrect klingon to a non-klingon model. 574 25 30 35 40 45 50 55 60 mean(x c ) for language, model 0.0 0.2 0.4 0.6 0.8 pcc( xc, sem) @xmath0.011 @xmath1.568 @xmath2.026 @xmath3.987 figure 3 : languages with a high correlation between textual correction significance and image improvement (pcc) are more "well-understood" by the model (x c ), for both realand pseudo-corrections. stablediffusion 1.4 was trained on the primarily-latin script laion-en-2b @xcite , and thus lacks capabilities in non-latin script languages ja and zh. consequently, there is no significant relationship between more semantically divergent corrections with high ∆sem and larger improvements to concept correctness ∆x c for sd 1.4 on those languages. meanwhile, altdiffusion-which conditions output images on the multilingual xlm-roberta encoder @xcite -benefits from all significant corrections in all languages with statistical significance. ## discussion & conclusions our findings motivate important considerations for building future t2i semantic evaluations @xcite @xcite . subjectivity a reliable t2i multilinguality assessment must report true knowledge failuresexamples where a model fails to generate correct images of a concept, when it is correctly prompted to do so. correct translations are required. unfortunately, choosing one "correct translation" is in inherently subjective task. this study tackled this subjectivity by casting a wide net of error candidates, and testing their impact. consequential errors caused false negatives where a concept is erroneously marked as "not possessed" (figure 1 ). cccl's tangible concept constraint and corpusbased approach to finding concepts helps combat subjectivity @xcite . in the tangible sense it's fair to say "orange" is correctly translated in spanish to naranja (the fruit) rather than anaranjado (the adjective). in prompting the t2i model we assume this tangible noun context is induced by using "a picture of an x"-style prompts. while our results show this works, it is a model-specific phenomenon and future work should examine more prompt templates. future work grounded in prototype theory @xcite may enable identification of culturally universal concepts for assessment. need to assess multiple translations one challenge in multilinguality assessments is incoming duplicates, where multiple ways of writing a translation really are equally correct. our homograph errors have examples, such as cigarette in japanese. た ば こ, タ バ コ, and 煙 草 are all translations of cigarette with identical reading, tabako. why should a metric of model-language capabilities only assess one correct translation rather than all? more significant multiple translation problems arise in languages with gendered human-referent terms. for example, in spanish maestro refers to a male teacher, while maestra a female one. should a test of a model's spanish knowledge of "teacher" as a concept test that both translations work equally well? cccl v1 is incapable of assessing these attributes. future benchmarks should contain this flexibility, so multiple incoming translations @xcite can be assessed for the same concept, while also tracing semanticallyencoded secondary attributes such as gender between the source and test language. error severity and error type figure 4 shows the distributions of error types for each language with respect to ∆ sem , our proxy for correction significance or error severity. across all three languages, the sense errors (os and is) are the most severe, while the formality and commonality errors are the least severe (defined in subsection a.3). our original estimated error rate (sum of all candidates per language) is a worst-case bound, the significant-to-evaluation-validity error rate is lower. our impact and significance results show that some of our suggestions (mainly formality and commonality errors) may be more nitpick than correction. some concepts in cccl are inherently erroneous due to intangibility. for example, history, film, and jump are all present in v1 of cccl, picked up for being high-frequency noun concepts across multiple languages in the corpora. there is no sensible prototypical way to generate images "of" those concepts. we removed these for cccl v1.1; future benchmarks should avoid including them.