Multilingual Conceptual Coverage in Text-to-Image Models (2023)

Abstract: We propose “Conceptual Coverage Across Languages” (CoCo-CroLa), a technique for benchmarking the degree to which any generative text-to-image system provides multilingual parity to its training language in terms of tangible nouns. For each model we can assess “conceptual coverage” of a given target language relative to a source language by comparing the population of images generated for a series of tangible nouns in the source language to the population of images generated for each noun under translation in the target language. This technique allows us to estimate how well-suited a model is to a target language as well as identify model-specific weaknesses, spurious correlations, and biases without a-priori assumptions. We demonstrate how it can be used to benchmark T2I models in terms of multilinguality, and how despite its simplicity it is a good proxy for impressive generalization.

## introduction neural text-to-image models convert text prompts into images @xcite using internal representations reflective of the training data population. advancements in conditional language modeling @xcite , variational autoencoders (kingma and welling, 2014), gans @xcite , multimodal representations @xcite , and latent diffusion models @xcite have given rise to sophisticated text-to-image (t2i) systems that exhibit impressive semantic generalization capabilities, with which they generate coherent, visually-appealing images with novel combinations of objects, scenarios, and styles @xcite . their semantic latent spaces @xcite ground words to associated visuals @xcite . characterizing the limits of these systems' capabilities is a challenge. they figure 2 : we hypothesize that a model's ability to generate creative, compositional images depicting tangible concepts (e.g., astronaut, horse, soup, bear) is predicated on its ability to generate simple images of the concepts alone. samples from @xcite . demonstrations of novel t2i model capabilities tend to rely on the subjective impressiveness of their ability to generalize to complex, novel prompts (figure 2 ). unfortunately, the space of creative prompts is in principle infinite. however, we observe that impressive creative prompts are composed of known, tangible concepts. can we directly evaluate a model's knowledge of these tangible concepts as a partial proxy for its capability to generalize to creative novel prompts? perhaps. but finding a diverse set of significant failure cases of basic concept knowledge for these models is challenging-in their training language. we observe that when prompted with simple requests for specific tangible concepts in a constrained style, t2i models can sometimes generate consistent and semantically-correct images in languages for which they received limited training (figure 1 , figure 3 ). we refer to this capacity as concept possession by said model in said language. at scale, we can assess the languageconcept possession for a diverse array of concepts and languages in a model to attempt to describe its overall multilingual generalization capability. we refer to the degree of this capability as the model's multilingual conceptual coverage. in this work we: ## motivation & related work this work is an attempt to produce a scalable, precise technique for characterizing conceptual coverage with minimal assumptions about the concepts or models themselves. in this section we lay out our motivations alongside relevant related work. benchmarks enabling model comparability have been a driving force in the development of pretrained language models (lm) @xcite . for classification and regression tasks, evaluation under fine-tuning @xcite is a straightforward and practical proxy for pretrained lm quality @xcite ) (e.g., for encoder-only transformer networks @xcite ). for these classification models, higher performance on benchmark datasets @xcite @xcite became the epitome of lm advancement. however, other important qualities in models including degree of social biases @xcite and robustness @xcite arising from biases in training data @xcite can only be captured by more sophisticated benchmarks that go beyond simple accuracy @xcite . checklist represented an influential move in this direction by benchmarking model performance through behavioral analysis under perturbed elicitation @xcite . avg cos. similarity avg cos. similarity avg cos. similarity text-to-image models as black boxes dalle 2 sd 1.4 sd 2.0 cog view2 dalle mini distinctiveness: (sd2, en, airplane) coverage (rel. en): (sd2, es, airplane) self-consistency: (sd2, ja, airplane) translation-aligned concept list man,hombre,mann... dog,perro,hund,狗... library,biblioteca,... sea,mar,meer,海,海,‫...ים‬ airplane,avión,飛行機... pizza,pizza,ピザ,‫...פיצה‬ languagespecific prompt sub stable diffusion 2.0 t2i model wrapper prompts ja images en images es images clip image enc. en clip embs es clip embs ja clip embs precomputed clip embs prompt templates en: "a photograph of es: "un foto de {{word ja: "{{word}}の写真" ... a broader range of outputs, use-cases, and capabilities, making evaluation more difficult. for many text-generative tasks such as summarization and creative text generation, the crucial desired quality is subjective, and challenging to evaluate @xcite . however, as these llms operate in a text-only domain, existing supervised tasks could be ported to few-shot or zero-shot evaluations of llm capabilities @xcite . while performance on these benchmarks isn't directly indicative of the impressive generative performance and generalization capabilities, they are a means to measure improvement @xcite . text-to-image models are even more difficult to evaluate than llms. unlike in llms, there are no objective evaluation tasks that can be directly ported as proxy evaluations. for example, while gpt-3 was introduced with impressive zero-shot performance across many classification tasks, the t2i model dall-e 2 was primarily introduced with human opinion scores and cool demo images @xcite . prior efforts in developing t2i evaluations such as drawbench @xcite and dall-eval @xcite fall into the trap of trying to build "everything benchmarks" for which whether the benchmark accurately reflects the practical task being asked of the computer in its real-world context is difficult to assess @xcite . we instead seek to build an atomic benchmark which narrowly and reliably captures a specific characteristic-conceptual knowledge as reflected by a model's ability to reliably generate images of an object. multilingual conceptual coverage is a highvariation t2i model performance setting. (figure 1 ) perhaps more importantly, it has immediate value, as work on improving t2i model multilinguality has has been proposed, but hampered by a lack of evaluation metrics. chen et al. (2022) introduce altclip and alt-diffusion, models produced by performing multilingual contrastive learning on a clip checkpoint for an array of non-english languages including japanese, chinese, and korean. without an objective evaluation benchmark, they can only demonstrate their improvement through human evaluation of impressive but arbitrary examples. coco-crola improves this state of affairs by enabling checklist-like direct comparison of techniques for reducing multilingual conceptual coverage disparities as an objective, capabilities-based benchmark. excitingly, we find that conceptual coverage is upstream of the impressive t2i model creativity that model developers and end-users are fundamentally interested in. this means that not only is coco-crola an objective evaluation of t2i system capabilities, it is also a proxy measure for the deeper semantic generalization capabilities we are interested in enhancing in second languages, as we demonstrate in subsection 5.6. ## definitions & formulations we define a multilingual concept over languages l as a set of words in each language carrying the same meaning and analogous colloquial use. we refer to the equivalent translation of concept c k in language ℓ as c k,ℓ . given a set of concepts c, test language ℓ, a minimal eliciting prompt foot_0 mp ℓ , text-to-image model f , and a desired number of images-per-concept n, we sample n|c||l| images i c k ,ℓ,i , where for every concept word in the language ℓ c @xmath0. given an image feature extractor f , some similarity function sim(•, •), we assess whether f possesses concept c k,ℓ in the test language if using the following metrics from the concept-image set {i c k,ℓ ,i } n @xmath1: distinctiveness. the images are distinct if they tend to not resemble the population of images generated for other concepts in the target language. formally, we compute our (inverse) distinctiveness score dt(f, ℓ, c k ) relative to m images sampled from other concepts in c: self-consistency. the images are self-consistent if they tend to resemble each other as a set. formally, we compute the self-consistency score sc(f, ℓ, c k ) as: correctness. the images are correct if they faithfully depict the object being described. rather than assess this using a classification model (hindering generality depending on the pretrained classifier), we use faithfulness relative to a source language ℓs, cross consistency xc(f, ℓ, c k , ℓs) as a proxy: it is important to note that in the source language (eg. english), xc ≈ sc, as for that language both metrics are essentially comparing the same (concept, language) pair to itself. thus we augment with a second languagegrounded correctness score, utilizing the average text-image similarity score of the english concept text against the set of generated images, for a clip image encoder f and text encoder f t , wc: we compute distinctiveness, self-consistency, and correctness scores across english, spanish, german, chinese (simplified), japanese, hebrew, and indonesian on the models listed in for each (language, concept) pair, we generate 10 images for analysis. we use a clip @xcite checkpoint from huggingface foot_1 as our semantic visual feature extractor f , and cosine similarity as our similarity function (sim(a, b) = a • b/||a||||b||). we collect a translation-aligned concept list c using techniques described in subsection 4.1 and depicted in figure 5 . we release our list generation code, testing code, feature extraction code, and final concept list as coco-crola v1.0 foot_2 . ## conclusion multilingual analysis of text-to-image models is desirable both to improve the multicultural accessibility of t2i systems and to deepen our understanding of their semantic capabilities and weaknesses. analyzing a model's conceptual coverage is a simple and straightforward way to do this. we demonstrated that these concepts are core building blocks for producing impressive images, and that analyzing the concepts is a useful proxy for assessing the creative capabilities of t2i models. our technique, coco-crola is a first step toward further work in this domain. following our recipe, larger benchmarks containing more languages and concepts can be built easily.