VLEU : a Method for Automatic Evaluation for Generalizability of Text-to-Image Models (2024)

Abstract: Progress in Text-to-Image (T2I) models has significantly advanced the generation of images from textual descriptions. Existing metrics, such as CLIP, effectively measure the semantic alignment between single prompts and their corresponding images. However, they fall short in evaluating a model’s ability to generalize across a broad spectrum of textual inputs. To address this gap, we propose the VLEU ( V isual L anguage E valuation U nderstudy) metric. VLEU leverages the power of Large Language Models (LLMs) to sample from the visual text domain, encompassing the entire range of potential inputs for the T2I task, to generate a wide variety of visual text. The images generated by T2I models from these prompts are then assessed for their alignment with the input text using the CLIP model. VLEU quantitatively measures a model’s generalizability by computing the Kullback-Leibler (KL) divergence between the visual text marginal distribution and the conditional distribution over the images generated by the model. This provides a comprehensive metric for comparing the overall generalizability of T2I models, beyond single-prompt evaluations, and offers valuable insights during the finetuning process. Our experimental results demonstrate VLEU’s effectiveness in evaluating the generalizability of various T2I models, positioning it as an essential metric for future research and development in image synthesis from text prompts. Our code and data will be publicly available at https://github.com/mio7690/VLEU .

## introduction the emergence of latent diffusion models (ldms) @xcite marked a significant advancement in generative models, addressing a crucial limitation that was prevalent during the era dominated by generative adversarial networks (gans) @xcite . unlike gans, which were often constrained by limited expressive and computational capabilities and focused on when fine-tuning a t2i model with images of a brown and white dog, as the fine-tuning process advances, prompts for dogs of various colors start to yield outputs that increasingly reflect the characteristics of the dog present in the training dataset. this results in generated images that deviate from the original textual description, indicating a clear case of overfitting and a loss of generalization. when assessing the performance of t2i models, a variety of metrics are employed, including inception score (is) @xcite , fréchet inception distance (fid) @xcite , which primarily gauge the quality and diversity of the gen-erated images. in contrast, metrics like clip score @xcite , dino @xcite , and imagereward @xcite are designed to measure the semantic alignment between the generated images and the input text. despite the effectiveness of these metrics in their respective domains, they do not fully capture a model's generalizability, which refers to its ability to produce accurate and diverse images across a wide range of textual prompts. this aspect of t2i model performance is often evaluated through subjective human judgment, highlighting the need for a standardized measure of generalizability. such issue of evaluating generalizability also extends to the methodologies employed for finetuning t2i models. techniques such as text inversion @xcite , lora @xcite , dream-booth @xcite , and hifi tuner @xcite have been instrumental in adapting pre-trained t2i models to specific subject or style. however, during the evaluation phase of these finetuned models, there is a lack of robust metrics to effectively measure the loss of generalization, which is shown in figure 1 . many studies, including dreambooth among others, have encountered this issue and presented it visually, ultimately relying on subjective human interpretation for assessment. to bridge this gap, we introduce vleu (visual language evaluation understudy) metric. 1 firstly we define the input text prompt for the t2i task as visual text, and also the sets of potential inputs for the t2i task as visual text domain. this delineated definition is essential, as not all text is appropriate as input for t2i models. for example, the chat text from the dialogue system, when used as input for t2i models, is nonsensical. vleu seeks to quantify the generalizability of t2i models by measuring the alignment between the visual text domain and the images generated by the t2i model conditioned on the visual text domain. vleu operates by calculating the kullback-leibler (kl) divergence @xcite between the distribution of the visual text domain and the distribution of the images conditionally generated by the model. this divergence serves as a metric for the alignment between the intended text prompts and the generated images. to facilitate this measurement, large language models (llms) such as chatgpt (openai, 2022), gpt-4 (openai, 2023b), and llama @xcite are utilized to sample from the visual text domain. these descriptions are then paired with images produced by the t2i model, and the clip model @xcite is used to evaluate the semantic congruence between each text-image pair. the principal contributions of this work are as follows: on top of the latent diffusion framework, the adoption of transformer architectures @xcite was another vital innovation. models like imagen @xcite and stable diffusion @xcite incorporated cross-attention layers that align textual prompts with generated image features. this mechanism enabled explicit conditioning on text descriptions and proved crucial for advancing text-to-image capabilities. the breakthrough in scaling up diffusion models to billions of parameters, combined with the effective technique of using transformer architectures for text conditioning, has led to recent t2i models demonstrating impressive capabilities in synthesizing high-fidelity, controllable images from a wide range of textual descriptions. metrics for t2i models: in evaluating t2i models, traditional metrics like inception score (is) @xcite , fréchet inception distance (fid) @xcite focus on image quality. is assesses the clarity and diversity of images using a pre-trained inception network which has fixed classes trained on imagenet @xcite , while fid measures the distance between feature vectors of real and generated images to gauge realism. meanwhile, newer metrics such as dino @xcite , clip similarity @xcite , and imagereward @xcite offer a more nuanced assessment. dino focuses on discerning the similarity between generated and actual images by emphasizing distinctive features. clip similarity metric examines the congruence between images and their g g' visual text domain image domain a black dog a white dog a yellow dog a black dog a white dog a yellow dog textual descriptions. imagereward gauges the aesthetic and creative attributes of generated images, aligning them with human aesthetic preferences. however, all these metrics primarily concentrate on either image quality or semantic alignment, revealing a gap in evaluating a model's generalizability across varied textual prompts. ## vleu how can we quantify the generalizability of a t2i model? this seems like an intractable problem at first glance. our intuition stems from observing the phenomenon of t2i models losing generalizability ## experiments and analysis this section investigates the effectiveness of vleu in evaluating t2i models and analyzes the impact of different components within the evaluation pipeline. specifically, we conducted two experiments: (1) analyze vleu's effectiveness in capturing model generalizability changes during finetuning and across different t2i models, which is presented in section 4.1. (2) assess the impact of key components like visual text sampler and text-image scorer on vleu scores, which is detailed in section 4.2. in line with our objectives, we primarily evaluated four open-source t2i models (sd 1.5, sd 2.0, sd 2.1, sdxl) under various conditions. for finetuning, we utilized the dataset provided by dream-booth @xcite , which comprises several subsets, each containing a series of images related to a specific subject. 2 besides, within all vleu calculations, the value of temperature in equation (1) was set to 0.01 to scale the computed results into a visually convenient range for analysis purposes. ## case studies using vleu in this section, we present two case studies to illustrate the practical application of vleu. racial bias in t2i models. we tested four t2i models to evaluate their vleu scores across african, asian, and caucasian people. for each ethnicity, we sampled 1000 prompts using gpt 3.5 foot_4 . the results, displayed in ## conclusion we introduced vleu, a novel automatic metric to evaluate t2i models' generalizability. vleu quantifies alignment between sampled visual text prompts and generated images using llms and clip. through experiments and case studies, we demonstrated vleu's efficacy in capturing declining generalizability during finetuning, discerning differences across models, and comparing finetuning techniques. vleu provides an automated, standardized metric accounting for a broad space of textual prompts. our results validate vleu as an effective metric for quantifying t2i models' generalizability.