Holistic Evaluation for Interleaved Text-and-Image Generation (2024)

Abstract: Interleaved text-and-image generation has been an intriguing research direction, where the models are required to generate both images and text pieces in an arbitrary order. Despite the emerging advancements in interleaved generation, the progress in its evaluation still significantly lags behind. Existing evaluation benchmarks do not support arbitrarily interleaved images and text for both inputs and outputs, and they only cover a limited number of domains and use cases. Also, current works predominantly use similarity-based metrics which fall short in assessing the quality in open-ended scenarios. To this end, we introduce InterleavedBench, the first benchmark carefully curated for the evaluation of interleaved text-and-image generation. InterleavedBench features a rich array of tasks to cover diverse real-world use cases. In addition, we present InterleavedEval, a strong reference-free metric powered by GPT-4o to deliver accurate and explainable evaluation. We carefully define five essential evaluation aspects for InterleavedEval, including text quality, perceptual quality, image coherence, text-image coherence, and helpfulness, to ensure a comprehensive and fine-grained assessment. Through extensive experiments and rigorous human evaluation, we show that our benchmark and metric can effectively evaluate the existing models with a strong correlation with human judgments surpassing previous reference-based metrics. We also provide substantial findings and insights to foster future research in interleaved generation and its evaluation.

## introduction multimodal learning has been a rapidly developing research field given the recent advancements in large multimodal models (lmms) @xcite @xcite . while these models can perform diverse tasks such as detailed image description and visual question answering, the outputs are limited to the text-only format, which hinders their broader applications. more recently, there has been a growing focus on enhancing lmms with the capability of interleaved generation, i.e., generating multimodal content that seamlessly integrates both text and one or multiple images @xcite @xcite . this opens new avenues for applications in diverse challenging scenarios, such as creative content generation @xcite , visual storytelling @xcite , and multimodal script generation @xcite . while the lmms for interleaved generation are continuously gaining stronger capabilities, progress in the evaluation of interleaved generation significantly lags behind with several critical challenges remaining. first, most existing works for interleaved generation quantitatively benchmark the models on text-to-image tasks where the output is usually one single image @xcite . however, such evaluation methods would fail to assess model performance in the real-world scenarios of interleaved generation, where the output usually consists of interleaved text and images. second, apart from human evaluation which is costly and time-consuming, existing works still heavily rely on reference-based metrics such as bleu @xcite fid @xcite that measure the similarity between generated samples and gold references. such similarity-based metrics often fail to accurately capture outputs' quality, especially in open-ended tasks such as creative generation and visual storytelling. third, the evaluation of interleaved generation is complex and involves many different aspects, such as perceptual quality, coherence between text and images, and helpfulness of the overall content. one input: chair and sofa and mountains in the background. input: given the task "how to make a toast in an oven" and the first two steps, predict the subsequent steps to complete the task. step 1: put the slices of bread flat on the oven rack. step 2: turn on the broiler of the oven, or set the heat on the toaster oven. output: here're the subsequent steps: step 3: use tongs to flip the bread over half-way through the toasting. step 4: remove the toast from the oven ## interleavedeval in many use cases of interleaved generation, such as "generate a story about snow white using both text and images", comparing the output against a gold reference is unrealistic since the generation can be fairly open-ended. however, existing approaches predominantly use reference-based metrics, e.g., bleu @xcite and fid @xcite , to measure the quality of text and image, respectively. they usually fail to assess the quality accurately. to bridge the gap between existing metrics and the demand in more diverse and realistic scenarios, we present interleavedeval, a strong reference-free metric based on gpt-4o, the current state-of-the-art lmm that supports arbitrarily interleaved inputs. to obtain a holistic and comprehensive evaluation of interleaved generation, we define five fine-grained evaluation aspects, including text quality, perceptual quality, image coherence, text-image coherence and helpfulness, and evaluate the output of each aspect separately. we show the detailed definition for each evaluation aspect in we formulate the evaluation metric inter-leavedeval as follows: we instruct the gpt-4o evaluator to output discrete scores from {0, 1, 2, 3, 4, 5} based on the detailed criteria shown in ## discussions qualitative analysis we conduct a qualitative analysis of benchmarked models in figure 4 and have the following observations: (1) while gill can generate images with reasonable quality, the generated text and images are typically not coherent with the instruction and context. in the example in the first row, the generated text is totally irrelevant to the task, while the image is also inconsistent with input images. (2) emu-2 can often generate text that is relevant to the task, but the quality is not good enough. in the example in the second row, it repeatedly says "soak the fabric in water" but does not contain other useful content. another weakness of emu-2 is its poor conditional image generation capability, where generated images have obvious visual distortions and could be duplicated with input images. (3) on the other hand, the pipeline models can generally better follow the instructions and generate text and images in higher quality. nevertheless, they still occasionally have some drawbacks. for gemini+sdxl, some of the generated images (e.g., the first output image in the second example) still have obvious defects. for gpt-4+dalle3, the style of generated images can be dramatically different from input images, as dalle3 is prone to generate images in cartoon or dramatic styles. (4) maintaining image coherence, i.e., the coherence of style and entities across images, is still very challenging for most models. in the third example, for the pipeline models, the same character has a very different appearance across the images, which makes the content inconsistent. (5) for the instances on the context-free subset, the integrated baselines have significantly worse performance, where they only generate one image with extremely poor quality. we hypothesize the reason is that those models cannot truly understand and follow the instructions. to sum up, our qualitative analysis indicates there is still significant room for improvement in interleaved generation. ## conclusion we introduce interleavedbench, the first benchmark for the evaluation of interleaved textand-image generation. we also propose inter-leavedeval, a strong multi-aspect reference-free evaluation metric based on gpt-4o. with extensive experiments, we first verify that our proposed metric can achieve significantly higher agreement with humans compared with existing metrics. through the lens of interleavedeval, we then observed that while the pipeline models based on proprietary lmms consistently outperform open-source models, interleaved generation is still a challenging task that requires further advancement.