Dataset Creation for Visual Entailment using Generative AI (2025)

Abstract: In this paper we present and validate a new synthetic dataset for training visual entailment models. Existing datasets for visual entailment are small and sparse compared to datasets for textual entailment. Manually creating datasets is labor-intensive. We base our synthetic dataset on the SNLI dataset for textual entailment. We take the premise text from SNLI as input prompts in a generative image model, Stable Diffusion, creating an image to replace each textual premise. We evaluate our dataset both intrinsically and extrinsically. For extrinsic evaluation, we evaluate the validity of the generated images by using them as training data for a visual entailment classifier based on CLIP feature vectors. We find that synthetic training data only leads to a slight drop in quality on SNLI-VE, with an F-score 0.686 compared to 0.703 when trained on real data. We also compare the quality of our generated training data to original training data on another dataset: SICK-VTE. Again, there is only a slight drop in F-score: from 0.400 to 0.384. These results indicate that in settings with data sparsity, synthetic data can be a promising solution for training visual entailment models.

## introduction natural language inference (nli) is a classification problem for pairs of two texts, a premise and a hypothesis. the pair is labeled as entailment (the premise entails the hypothesis), neutral or contradiction (the hypothesis contradicts the premise). in visual entailment (ve) tasks @xcite , the premise is substituted by an image, while the hypothesis is still in text form. in order to create and train effective models for ve, large datasets are needed. while datasets of images combined with hypotheses and labels do exist, they are relatively small and sparse compared to datasets for textual entailment. existing datasets are snli-ve @xcite and @xcite which are both based on nli datasets and which were created by manual labor leveraging amazon mechanical turk workers. in this paper we evaluate the use of generative ai for ve dataset creation which would allow cheaper and easier dataset creation. this is done by first generating a synthetic dataset, of which we then verify the validity. we introduce a synthetic version of the snli-ve dataset called synthetic-nli-ve and show how models trained on this dataset have similar performance when tested on real data compared to models trained on real data. in summary, the contributions of this paper are threefold: (1) we present the new dataset synthetic-nli-ve foot_0 ; (2) we find that the performance of models trained on the generated dataset have similar performance compared to models trained on real data; (3) a cross-data evaluation shows that generalizability of visual entailment models to a different dataset is poor, whether or not the training set was generated or original. ## related work visual entailment and dataset creation the idea of visual entailment was first proposed by @xcite . for this task they introduce the explainable visual entailment (eve) model, based on attention visualization. in the same paper the authors introduce the snli-ve dataset (section 3). @xcite introduced a dataset for visual question answering (qa). they used the microsoft common objects in context (ms coco) dataset @xcite as a starting point: ∼200k images of real-world scenes with 5 captions per image. they added 50k images of abstract scenes for which they also collected 5 captions per image. @xcite created the sick dataset. sick (sentences involving compositional knowl-edge) contains sentence pairs with both relatedness scores and entailment labels. this dataset was created by pairing the flickr8k dataset @xcite and the semeval-2012 sts data @xcite and having amazon mechanical turk workers annotate them with both similarity scores and entailment labels. @xcite created the dutch version of sick using a semi-automatic translation. @xcite introduced the snli dataset on which the aforementioned snli-ve was based, with as motivation that the sick dataset is too small and not balanced enough. for snli they created a balanced dataset of around ∼500k sentence pairs compared to the ∼10k in the sick dataset. there are also efforts made to improve existing datasets. this was already the case with @xcite , who improved and extended the vqa dataset resulting in the vqa-v2 dataset. the dataset was improved by, among other things, reducing bias and extended it by adding more images. this has also been done for the snli-ve dataset by @xcite who created the e-snli-ve-2.0. synthetic data unlike the largely human made datasets that were previously discussed, the clevr dataset @xcite is automatically generated. this dataset contains images of abstract shapes combined with automatically generated questions. the images were created by randomly sampling a scene graph and rendering it using the open-source 3d rendering software blender. @xcite proposed an evaluation framework for assessing synthetic data generated by large language models (llms). this framework includes measures for fidelity, utility and privacy. in this work, we only focus on the fidelity and utility of the generated data. some research suggests that using synthetic datasets for model training could have a negative effect on performance in the future, if generated datasets are used for training computer vision models @xcite . as opposed to synthetic datasets used to train generative models, the images that we generate are used to train classification models. furthermore, these classification models are evaluated on original data, ensuring good real world generalizability. ## data in this work we use two datasets which we briefly describe in this section. snli-ve this was introduced by @xcite , by combining the snli dataset @xcite with the flickr30k dataset @xcite . the flickr30k dataset was created by taking 31,783 photos of everyday activities which were harvested from flickr. each image receives 5 different captions resulting in 158,915 captions in total. figure 3 in the appendix shows an example of an image and its captions. the snli dataset @xcite is a well known dataset specifically created for natural language inference. in short, it was constructed by having amazon mechanical turk workers generate 3 hypotheses per caption, where captions came from the flickr30k dataset. from this, @xcite could therefore create the snli-ve dataset by replacing each premise by the original corresponding image. the dataset contains a total of 31,783 images, 157,567 premises and 565,286 hypotheses. ## methods we generate a synthetic dataset as described in §4.1. we then report on the intrinsic evaluation of image quality by comparing the generated images directly with the original images based on a similarity analysis in §4.2. finally, we perform extrinsic evaluation of synthetic data, comparing it to original data for visual entailment model training in §4.3. ## experiments and results in this section we first report on the results for the intrinsic evaluation ( §5.1), after which we discuss the downstream performance in the visual entailment task ( §5.2), and finally we discuss the results ## conclusion in this paper we introduced a synthetic ve dataset synthetic-nli-ve. the dataset proved to have similar utility compared to the dataset it was based on while being far less costly to create. this also proves the viability of using generative ai to create datasets for the ve task, whereby we pave the way for future research into using synthetic data for ve dataset creation. as future work we propose changing the single set of parameters for the generation model to a variety of different values. secondly, generating more than one image per caption could result in better training data compared to the one image per caption dataset we generated. lastly, evaluating different classification algorithms could further strengthen the findings.