Image captioning datasets have proven useful for multimodal representation learning, and a common evaluation paradigm based on multimodal retrieval has emerged . Unfortunately, datasets have only limited cross-modal associations: images are not paired with others , captions are only paired with others that describe the same image, there are no negative associations and there are missing positive cross-modal associations. This undermines retrieval evaluation and limits research into how inter-modality learning impacts intra-modality tasks. To address this gap , we create theCrisscrossed Captions (CxC) dataset, extending MS-COCO with new semantic similarity judgments for \textbf{247,315 intra- and inter-modality pairs. We provide baseline model performance results for both retrieval and correlations with human rankings, emphasizing both intra- and inter-modality learning.
By supporting multi-modal retrieval training and evaluation, image captioning datasets have spurred remarkable progress on representation learning . Unfortunately, datasets have limited cross-modal associations: images are not paired with other images , captions are only paired with other captions of the same image, there are no negative associations and there are missing positive cross-modal associations. This undermines research into how inter-modality learning impacts intra-modality tasks. We address this gap with Crisscrossed Captions (CxC) , an extension of the MS-COCO dataset with human semantic similarity judgments for \textbf{267,095 intra- and inter-modality pairs. We report baseline results on CxC for strong existing unimodal and multimodal models. We also evaluate a multitask dual encoder trained on both image-caption and caption-caption pairs that crucially demonstrates CxC's value for measuring the influence of intra- and inter-modality learning.
R
Image
By supporting multi-modal retrieval training and evaluation, image
R
proven useful for multimodal representation learning, and a common evaluation paradigm based on multimodal retrieval has emerged
spurred remarkable progress on representation learning
D
only
R
others
other images
R
others that describe
other captions of
D
retrieval evaluation and limits
R
To
We
D
, we create the
R
Crisscrossed Captions
with Crisscrossed Captions
R
dataset, extending
, an extension of the
R
with new
dataset with human
R
247,315
267,095
R
provide baseline model performance results for both retrieval and correlations with human rankings, emphasizing both
report baseline results on CxC for strong existing unimodal and multimodal models. We also evaluate a multitask dual encoder trained on both image-caption and caption-caption pairs that crucially demonstrates CxC's value for measuring the influence of