WISMIR 3: A Multi-Modal Dataset to Challenge Text-Image Retrieval Approaches (2024)

Abstract: This paper presents WISMIR3, a multi-modal dataset comprising roughly 300K text-image pairs from Wikipedia. With a sophisticated automatic ETL pipeline, we scraped, filtered, and transformed the data so that WISMIR3 intrinsically differs from other popular text-image datasets like COCO and Flickr30k. We prove this difference by comparing various linguistic statistics between the three datasets computed using the pipeline. The primary purpose of WISMIR3 is to use it as a benchmark to challenge state-of-the-art text-image retrieval approaches, which already reach around 90% Recall@5 scores on the mentioned popular datasets. Therefore, we ran several text-image retrieval experiments on our dataset using current models, which show that the models, in fact, perform significantly worse compared to evaluation results on COCO and Flickr30k. In addition, for each text-image pair, we release features computed by Faster-R-CNN and CLIP models. With this, we want to ease and motivate the use of the dataset for other researchers.

## introduction current multi-modal text-image retrieval approaches already reach over 90% recall@5 on popular evaluation sets @xcite . the reason for this is definitely due to the advances in visio-linguistic approaches implemented by stateof-the-art models like uniter @xcite , teran @xcite , clip @xcite , or beit3 @xcite . however, we argue that this is not solely due to the model's architecture but also because of the simplicity of the widely used training data and its similarity to the evaluation data. although more recent datasets exist, the most popular datasets used to train and evaluate state-of-the-art text-image retrieval methods are still coco @xcite and flickr30k @xcite . both datasets comprise short and simple captions created by crowdsourcing workers for flickr images showing everyday scenes. @xcite showed that recent multi-modal transformer-based approaches trained on these popular datasets cannot generalize well on out-of-domain data with more complexity and variety. in the mentioned work, two preliminary datasets were introduced. however, during detailed data analysis, we found multiple issues in these preliminary datasets, which we address in this work. the main contribution of this work is the release of wismir3 (wikicaps subset for multi-modal text-image retrieval v3) foot_0 , a clean multi-modal dataset, thought of as a benchmark to challenge state-of-the-art text-image retrieval models. wis-mir3 contains more than 300k text-image pairs from wikipedia, scraped, filtered, transformed, and statistically analyzed by a sophisticated automatic etl pipeline tool. further, we provide a detailed overview, discuss and release linguistic statistics of the comprised data, and compare it to coco and flickr30k. additionally, we release pre-computed image features from a popular pre-trained faster-r-cnn @xcite model and image and text embeddings from pre-trained clip models employing vit @xcite as the image encoder. with this, we aim to ease the use of the dataset to train, finetune, or evaluate models on the wismir3 dataset. by evaluating different state-of-the-art text-image retrieval approaches on wismir3 and comparing the results with their performance on coco and flickr30k, we show that these models indeed perform much worse on our dataset. ## related work state-of-the-art approaches for multi-modal textimage retrieval are typically trained on text-image pairs. despite their age, the most popular datasets to train and evaluate models on this task are still coco @xcite and flickr30k @xcite . coco is a well-known dataset for various computer vision tasks like object detection, object segmentation, image captioning, keypoint detection, human pose estimation, and textimage retrieval. besides labels and annotations, the dataset contains about 123k carefully selected images from flickr with five descriptive captions each. flickr30k contains about 30k icon photographs of everyday activities, events, and scenes from flickr, where also five different captions describe each image. both coco and flickr30k are datasets designed by researchers and handcrafted by crowdsourcing workers to describe the images with short, simple, and descriptive captions. less popular but larger datasets like sbu captions @xcite , conceptual captions @xcite , or visual genome @xcite are primarily designed for tasks like image-captioning, visual question answering, or visual entailment. however, since they comprise text-image pairs, the datasets are often part of the training data for text-image retrieval approaches. visual genome contains about 108k images collected from an intersection of ms coco and yfcc-100m @xcite with captions created by crowdsourcing workers. sbu caption contains about 1m photos and their captions from flickr. conceptual captions contains approximately 3.3m text-image pairs scraped from billions of websites and automatically transformed and filtered by a sophisticated pipeline. further, wit @xcite and laion-5b @xcite are huge text-image datasets suitable for pre-training visonlanguage foundation models like clip @xcite , align @xcite , or blip2 @xcite . the wit dataset contains about 37.5m text-image pairs, comprising 11.5m unique images with captions from wikipedia across 108 different languages. the laion-5b dataset contains about 5b non-curated text-image pairs scraped from common crawl dumps. another text-image dataset is wikicaps (schamoni et al., 2018), containing about 3.8m textimage pairs from wikipedia. captions are taken from the associated wikimedia image descriptions, mainly in english. this dataset is the basis of wismir3 and is of particular interest in this work because the data is from random wikipedia articles. therefore, the captions and images cover a wide range of different topics and concepts. ## data collection pipeline a schematic overview of the pipeline used to collect the wismir3 dataset, presented by this work, is shown in figure 1 . in the following, more details about the single steps are described. the input to the pipeline is a csv file released by the wikicaps authors, containing 3.8m wikimedia image file ids and the corresponding english captions. since this file format is unhandy to compute statistics or apply transformations, it is converted into a pandas dataframe, used throughout the whole pipeline. in the first stage, extensive corpus statistics are collected for each caption using a spacy pipeline with the "en_core_web_lg" model. these statistics include, for example, the number of tokens and sentences, pos tags of each token, counts of the universal dependency tags @xcite , the language of each sentence, named entities, and ratios between the number of all tokens and nouns or named entities. the dataframe is then filtered based on these statistics, as described in the following. samples are dropped if further samples were removed if the language of every sentence in the caption was not english. moreover, since the purpose of this dataset is to challenge text-image retrieval approaches, it is essential that most of the words in an image description are also represented in the image. hence, we created a blocklist of non-depictable words like "url", "sarcasm", "confusion" and filtered out every sample that contains one or more of these terms. in the next pipeline stage, the duplicate filtering stage, we remove duplicate captions so that one caption describes at most five different images. this decision was inspired by coco or flickr30k, where it is the other way round, i.e., five different captions describe one image. with the mentioned filtering stages, we reduced the 3.8m wikicaps samples by about 92% to 304317 samples. after downloading the images, we removed 3431 that were too small or had erroneous data format. we applied the following transformations to every image in the final pipeline stage. ## image retrieval experiments this section presents text-image retrieval evaluation results of various recent models on the wis-mir3 dataset and compares them to the models' performances on coco and flickr30k. as listed in further observed is that coco and flickr30k data did not contribute anything meaningful during teran training processes when evaluating the ## conclusion this paper presents wismir3, a clean multi-modal dataset containing roughly 300k text-image pairs. the dataset comprises images with corresponding captions from wikipedia using wikicaps as the source dataset. by implementing a sophisticated automatic etl pipeline tool, we scraped, filtered, and transformed the data so that wismir3 differs from popular datasets like coco and flickr30k. we prove this difference by comparing linguistic statistics between the three datasets also computed using the tool. the purpose of wismir3 is to use it as a hard benchmark to challenge state-of-theart text-image retrieval approaches, which already reach 90% recall@5 scores on the mentioned popular datasets. with the experiments in this paper, we show that the text-image retrieval performance of the current models on wismir3 is much lower than on coco or flickr30k, as anticipated.