Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS - COCO (2021)

Abstract: By supporting multi-modal retrieval training and evaluation, image captioning datasets have spurred remarkable progress on representation learning. Unfortunately, datasets have limited cross-modal associations: images are not paired with other images, captions are only paired with other captions of the same image, there are no negative associations and there are missing positive cross-modal associations. This undermines research into how inter-modality learning impacts intra-modality tasks. We address this gap with Crisscrossed Captions (CxC), an extension of the MS-COCO dataset with human semantic similarity judgments for 267,095 intra- and inter-modality pairs. We report baseline results on CxC for strong existing unimodal and multimodal models. We also evaluate a multitask dual encoder trained on both image-caption and caption-caption pairs that crucially demonstrates CxC’s value for measuring the influence of intra- and inter-modality learning.

## introduction phrases such as blue, chair, and garden path have strong visual components, yet computational word representations are usually created with text-only corpora. encouragingly, some recent work that derives representations using visual contexts shows improvements for both word similarity ranking and image-text retrieval @xcite , and querybased training of image models demonstrates language's power to improve image representations @xcite . learning representations for both vision and language jointly should be even more effective-indeed, much progress has been made on such cross-modal learning using image captioning data @xcite @xcite . however, it is not yet clear whether learning representations in multimodal contexts improves performance within as well as across modalities as there are no datasets ideally suited for this at present. image captioning datasets such as flickr8k @xcite , flickr30k @xcite , multi30k @xcite , microsoft common objects in context (ms-coco) @xcite , and conceptual captions @xcite only capture relationships between images and textual captions created for them. they miss many valid relationships between unassociated images and captions, from captions to other captions, and from images to other images. we address this gap with crisscrossed captions (cxc, exemplified in figure 1 ), a dataset with graded, denser annotations for relationships between and among captions and images in the ms-coco evaluation splits of @xcite (with 25k english captions and 5k images each). cxc extends ms-coco's existing imagecaption pairs with continuous (0-5) semantic similarity ratings for those pairs and new pairs. the rating criteria extend those used for semantic textual similarity @xcite . intramodal pairs are selected for annotation via an indirect sampling scheme biased to gain a broad distribution of similarities. in all, cxc contains human ratings for 267,095 pairs (derived from 1,335,475 independent judgments), a massive extension in scale and detail to the 50k original binary pairings. ms-coco incompletely supports three retrieval tasks: image-text, text-image and text-text. cxc enhances all of these with new positive pairs, and it also supports a new image-image retrieval task. with its graded similarity judgements, cxc also supports correlation measures comparing model and human rankings. retrieval metrics focus on positive pairs, but cxc's correlation scores additionally account for low-scoring items (non matches). supporting these evaluations on a common set of images and captions makes them more valuable for understanding inter-modal learningcompared to disjoint sets of caption-image, captioncaption, and image-image associations. also, multimodal representations such as clip @xcite are useful for downstream tasks such as visual question answering @xcite , vision and language navigation @xcite , referring expressions @xcite and visual commonsense reasoning @xcite , and we hope the additional relationships and evaluations provided by cxc will help develop even better representations for tasks that span these modalities. to establish baselines for cxc, we provide results for existing unimodal models for text (bagof-words, use @xcite ) and images (inceptionv3 @xcite , resnet-152 @xcite , simclrv2 @xcite as well as for two cross-modal retrieval models vse++ @xcite and vsrn @xcite . we furthermore demonstrate cxc's utility by evaluating a dual encoder that combines a bidirectional loss for image-text retrieval with a loss for text-text retrieval. the text encoder is composed of transformer layers over pre-trained bert word representations and the image encoder is a pre-trained efficientnet (b4) @xcite . this model delivers the strongest overall perfor-mance across all four retrieval tasks and correlation with human scores for text-text, image-image and image-text similarity. compared to the same dual encoder trained only with image-text pairs, this model realizes small gains for image-text tasks and large gains for text-text task but with some degradation for image-image tasks. this indicates that the model trades capacity to encode images for better text encoding-an insight that would not be easily assessed without cxc's image-image annotations. our main contributions are the following: ## dataset collection existing resources already support learning joint representations of images and text. however, we need better evaluation resources, so we extend the ms-coco evaluation splits with graded similarity associations within and across modalities. ms-coco has five captions for each image, split by @xcite into 410k training, 25k development, and 25k test captions (82k/5k/5k for images). an ideal extension would rate every pair, but this is infeasible foot_0 and most pairs are dissimilar anyway. to obtain new pairs with high expected similarity, we introduce a biased sampling scheme. the data is collected in two phases. first, we define an indirect sampling scheme that uses modelbased similarities from the co-modality items to caption 1: a tennis player swinging a racket at a ball. ## crisscrossed captions dataset using our selection and annotation methodology, we obtained ratings for 267,095 caption-caption, image-image, and caption-image pairs (1,335,475 total judgments). figure 3 shows rating distributions for each task (validation split). it also shows the distributions of ratings for sts and sis pairs included from other-modality selection and from original ms-coco pairs. the test set distributions are similar. figure 4 gives the distribution of counts of positive examples in each task (validation split), where a @xmath0, sits) and a @xmath1.5 (for sis) is considered positive. these positive examples are used for intermodal and intramodal retrieval evaluation. ## evaluation tasks and metrics cxc supports intermodal retrieval like ms-coco but with denser annotations between image-caption pairs. it also enables intramodal retrieval and semantic similarity correlation evaluations, which were not possible before. @xcite first used ms-coco for image-to-caption and caption-to-image retrieval. we extend the existing associations with positive cxc pairs, and also add new caption-to-caption and image-to-image retrieval tasks using positive sts and sis pairs (a total of four retrieval tasks). to the best of our best knowledge, cxc is the first dataset to support image-to-image retrieval over captioned images. following @xcite , we evaluate using recall@k (r@k), computed as the fraction of times a correct item was found among the top k results, and median rank (med. r) of the closest ground truth result in the list. semantic similarity tasks such as semantic textual similarity @xcite and visual semantic textual similarity(vsts) (de lacalle et al., 2020) require a model to produce a continuous similarity score given two inputs. typically, the models are evaluated based on the pearson's r of their scores with the human judgments over a set of input pairs. this is valid when training data is available to calibrate model scores to the human ratings. with cxc, we do not have such training data, so we instead use spearman's r to assess whether a model ranks pairs similarly to human raters. it would be tempting to simply measure spearman's r over all pairs, but this would be flawed because cxc's dense annotation means that the scores between many pairs are themselves correlated. to mitigate this, we use a sampled bootstrap correlation instead. for each correlation estimate, we sample half of the queries (to increase diversity across samples) and for each selected query, we choose one of the items for which cxc supplies a paired rating. we compute spearman's r between the cxc scores and the model scores for the selected pairs. the final correlation is the average over 1000 of these bootstrap samples. vsts (de lacalle et al., 2020) contains 2677 pairs of ms-coco captions and corresponding images. as noted above, vsts is related dataset for multimodal semantic similarity. we considered mixing cxc and vsts; however, this was infeasible because cxc uses the widely adopted karpathy splits, while items in vsts's training, dev and test splits are spread among the karpathy splits. we could not just make a separate cut of cxc because vsts pairs can cross splits, e.g. an image-caption item in karpathy training and another in karpathy test. given the small size of vsts, we focused our efforts on cxc evaluations. ## evaluated models in order to establish baselines for cxc, we benchmark pretrained models for images and text. note that these are off-the-shelf models that have not been trained on ms-coco. we also evaluate crossmodal retrieval models that are trained on ms-coco. here, we focus on models that support efficient retrieval (e.g. dual encoders). we expect models with extensive cross-modal interactions, such as vilbert @xcite and lxmert @xcite , will show strong performance on cxc tasks, either as standalone models that (inefficiently) score all possible item pairs or as rerankers for outputs of retrieval models. to the best of our knowledge, there is no prior work that explores joint learning or evaluation on intra-and inter-modality retrieval tasks. @xcite @xcite show evidence that inter-modality learning helps improve intra-modality performance, but do not explore multitask learning. @xcite explore multitask learning but only focus on intermodal representation learning for intermodal downstream tasks. to illustrate how cxc allows us to measure how intermodal representation learning can improve both intra-and inter-modal performance, we train a dual encoder model on bidirectional image-text and text-text in-batch retrieval losses. ## conclusion the cxc dataset provides a much more complete set of relationships between and among images and captions than the raw ms-coco image-caption pairs. we demonstrate that a dual encoder that learns from both image-caption pairs and caption-caption pairs (de t2t+i2t ) exhibits strong, balanced performance across four retrieval tasks and three correlation measures. there is much remaining headroom for future models for all these tasks. cxc's annotations themselves validate the strong semantic alignment between images and their original captions-these have an average similarity of 4.85. however, we also find that co-captions (captions for the same image) have an average score of just 3.0. this calls into question the use of such pairs in training and evaluating paraphrase generation models @xcite and reinforces the need for images as context for human evaluation in paraphrasing @xcite .