R ef VNLI : Towards Scalable Evaluation of Subject-driven Text-to-image Generation (2025)

Abstract: Subject-driven text-to-image (T2I) generation aims to produce images that align with a given textual description, while preserving the visual identity from a referenced subject image. Despite its broad downstream applicability—ranging from enhanced personalization in image generation to consistent character representation in video rendering—progress in this field is limited by the lack of reliable automatic evaluation. Existing methods either assess only one aspect of the task (i.e., textual alignment or subject preservation), misalign with human judgments, or rely on costly API-based evaluation. To address this gap, we introduce RefVNLI, a cost-effective metric that evaluates both textual alignment and subject preservation in a single run. Trained on a large-scale dataset derived from video-reasoning benchmarks and image perturbations, RefVNLI outperforms or statistically matches existing baselines across multiple benchmarks and subject categories (e.g., Animal , Object ), achieving up to 6.4-point gains in textual alignment and 5.9-point gains in subject preservation.

## introduction in a well-known scene from "the little prince", the narrator attempts to comfort a grieving prince by saying "i'll draw you a fence around your flower". while fairly simple, this offer raises a deeper question: what makes such a drawing adequate? beyond accurately depicting a fence around a flower, the use of 'your' implies that it must portray a specific flower-the prince's own-one with which he shares a history. given the flower's uniqueness and distinct visual traits, the narrator's task proves far more complex than it first appears. subject-driven text-to-image (t2i) generation @xcite @xcite 2023) enables a variety of downstream applications, such as personalized image generation @xcite , character consistency in video generation @xcite , and enhancing vanilla t2i evaluation frameworks for less-known entities via image retrieval @xcite . unlike standard t2i models that are only conditioned on text inputs, this setup takes both a textual prompt and a reference image, enabling more precise subject representation. for example, when creating an image of a fenced-in flower for the little prince, subject-driven models should use a reference image of the prince's flower to ensure the output preserves its unique features (see fig. 1 ). , using examples from the animal, object, and human categories. dreambench++ scores (0-4) are scaled to 0-100 for better readability. refvnli exhibits better robustness to identity-agnostic changes (sp), such as the zoomed-out parrot (top-middle) and the zoomed-out person with different attire (bottom-middle). it is also more sensitive to identity-defining traits, penalizing changed facial features (left-most person) and mismatched object patterns (left and middle balloons). additionally, refvnli excels at detecting text-image mismatches (ta), as seen in its penalization of the top-left image for lacking a waterfall. despite its wide applicability, research in this area has been hindered by a lack of reliable and scalable auto-raters. existing metrics typically correlate poorly with humans, and often focus only on textual alignment between the input prompt and the target image, as in clip-t @xcite and siglip @xcite , or on subject preservation between the input and target images, as in clip-i @xcite and dino-i @xcite , while both aspects are needed for successful subject-driven generation. more correlative metrics, like dreambench++ @xcite and viescore @xcite , depend on costly api calls to models like gpt-4 (openai, 2024), making them less scalable and reproducible. to bridge this gap, we present refvnli, a costeffective fine-tuned auto-rater for subject-driven t2i generation. given a @xmath0, prompt, image tgt >, refvnli predicts two scores-textual alignment and subject preservation-in a single run, as shown in fig. 1 . to train refvnli, we automatically curate a large-scale dataset @xmath1, prompt, image @xmath2, labeled @xmath3, subject @xmath4, 1} 2 . for subject preservation, we identify subjects across video frames, creating positive examples using pairs of frames depicting the same subject, and negative ones by pairing frames of different subjects (yet of the same entity). this approach enables robustness to variations in subject appearance (e.g., rotation, setting, clothing), as well as to the presence of extraneous elements (e.g., dew on the little prince's flower in fig. 1 , top). at the same time, refvnli must also be sensitive to identitydefining traits, such as human facial features or object shapes and colors (e.g., middle image in fig. 1 ). to this end, we modify images by masking and inpainting identity-critical regions, while keeping everything else unchanged. the original subject crops are then paired with the unaltered images as positive pairs and with the modified images as negative pairs, thereby teaching the model to focus on key identity attributes. for textual alignment, we first create positive image-prompt pairs. for that, we use an llm to caption each image in the aforementioned pairs, ensuring focus on the subject by enclosing it within a bounding box. negative pairs are then formed by replacing these captions with those of different scenes. for extra sensitivity to minor mismatches, like a fence drawn next to rather than around the prince's flower (fig. 1 , bottom), we also create negative pairs by altering a single fact in each original (positive) caption. finally, to derive @xmath5, prompt, image @xmath6, we use the cropped subject from one frame as image ref and the entire second frame, alongside its caption, as {prompt, image tgt }, resulting in a total of 1.2 million instances. we evaluate refvnli on multiple humanlabeled test sets for subject-driven generation, including dreambench++ @xcite , ima-genhub @xcite , and kitten @xcite , across categories such as humans, animals, objects, landmarks, and a multi-subject setting. for textual alignment, refvnli consistently matches or outperforms all baselines, with up to 6.4-point gains in landmarks and proficiency at detecting subtle text-image misalignments (e.g., missing waterfall in fig. 2 , top-left). it also leads in subject preservation, with gains of up to 6.3 points on objects ( ## refvnli: automatic metric for subject-driven t2i generation we introduce refvnli, a cost-effective auto-rater specifically tailored for subject-driven t2i generation. this section details the automated pipeline used to construct its training dataset ( §2.1) and the subsequent training process of refvnli ( §2.2). ## experimental settings this section outlines our meta-evaluation protocol and benchmarks ( §3.1), followed by an overview of the baseline models used for comparison ( §3.2). ## results our main results are summarized in tables 1 , 2 , and 3, with qualitative examples in fig. 2 . on dreambench++ ( fig. 2 further showcases refvnli's strengths, like its sensitivity to subtle textual alignment errors, such as a missing waterfall (top-left). for subject preservation, it remains robust to identityagnostic changes, like a zoomed-out parrot or person (top-center and bottom-center) or different clothes (bottom-center), while staying sensitive to key identity traits, e.g., changed facial features (bottom-left) and colors (left and middle balloons). overall, refvnli consistently outperforms or statistically matches all baselines on both criteria, with the only exception of subject preservation in the ood landmarks category, where it still performs competitively. importantly, it offers the best trade-off between textual alignment and subject preservation, surpassing all non-finetuned metrics in unified evaluation across all benchmarks. ## applicability to rare entities to test refvnli on unfamiliar subjects, we use the imagerag benchmark @xcite , which evaluates generated images based on prompts and reference images of uncommon subjects (e.g., scientific animal names, lesser-known dishes). human annotators compared image pairs, selecting the better one based on textual alignment, visual quality (evaluating general depiction of the entity rather than exact reference-adherence), and overall preference. we report per-axis accuracy, defined as the frequency with which a metric ranks the human-preferred image higher, with approximate ties handled by rounding. overall preference is computed as the harmonic mean of textual and visual scores, and significance testing is assessed via bootstrap resampling, akin to §3.1. as shown in ## related work evaluation of visual language models (vlms) spans various settings, including visual reasoning @xcite and visual question-answering @xcite @xcite . for text-to-image (t2i) models, assessments normally focus on image quality @xcite , diversity @xcite , and alignment with the text @xcite @xcite @xcite . assessing subject preservation, which is crucial for subject-driven generation, is typically done using embedding-based metrics like clip @xcite and dino @xcite . other metrics, like viescore @xcite and dream-bench++ @xcite , use gpt-4o (openai, 2024) to measure both criteria. subject-driven t2i models have been gaining much traction, with some methods fine-tuning general models into specialist versions that capture specific subjects and styles (gal et al., 2022; kumari 7 tifa was excluded due to assigning identical scores to 61% of pairs, making accuracy calculations unreliable. @xcite @xcite . others focus on broader applicability using one-shot examples, either through adapter-based methods that integrate encoded reference images into diffusion models @xcite @xcite or via adapter-free techniques that directly use extracted features such as attention maps @xcite @xcite . closely related, image editing complements subject-driven t2i generation in that the generated image's appearance is primarily governed by the input image, with the text only impacting specific aspects, whereas in out setting it is the other way around. the task has evolved from pixel-to-pixel translation for predefined transformations @xcite @xcite to more flexible, text-guided edits @xcite @xcite , with recent diffusion-based methods improving precision via cross-attention manipulation @xcite . beyond images, personalized generation extends to other modalities, including videos and texts. video generation can be conditioned on text @xcite @xcite , reference images @xcite , or other videos @xcite . in text generation, efforts focus on style transfer @xcite , debiasing @xcite , and broader semantic control @xcite @xcite . finally, several studies leveraged intra-frame relationships in videos to learn more human-aligned visual representations. these works aim to improve robustness to identity-agnostic variations (e.g., rotation, lighting), by analyzing consecutive frames sourced from public video datasets @xcite @xcite @xcite or captured by cameras on moving agents @xcite . ## conclusion we present refvnli, a cost-effective and reliable metric for subject-driven t2i evaluation that jointly assesses textual alignment and subject preservation. trained on a large-scale, auto-generated dataset, refvnli is designed to be robust to identityagnostic visual variations (e.g., pose, lighting, background) while remaining sensitive to identityspecific features (e.g., facial features, object shape, and unique details) when evaluating subject preservation. for textual alignment, it leverages subjectspecific prompts with perturbed hard negatives to detect and penalize fine-grained mismatches. across benchmarks, refvnli outperforms or rivals all baselines, including larger gpt-4o-based metrics, particularly on less-common subjects. future work should focus on improving performance across artistic styles as well as identityaltering edits, and supporting multiple reference images. more broadly, refvnli facilitates progress in personalized t2i generation by enabling better checkpoint selection, reinforcement learning, and iterative model refinement.