Negative Sample is Negative in Its Own Way: Tailoring Negative Sentences for Image-Text Retrieval (2022)

Abstract: Matching model is essential for Image-Text Retrieval framework. Existing research usually train the model with a triplet loss and explore various strategy to retrieve hard negative sentences in the dataset. We argue that current retrieval-based negative sample construction approach is limited in the scale of the dataset thus fail to identify negative sample of high difficulty for every image. We propose our TAiloring neGative Sentences with Discrimination and Correction (TAGS-DC) to generate synthetic sentences automatically as negative samples. TAGS-DC is composed of masking and refilling to generate synthetic negative sentences with higher difficulty. To keep the difficulty during training, we mutually improve the retrieval and generation through parameter sharing. To further utilize fine-grained semantic of mismatch in the negative sentence, we propose two auxiliary tasks, namely word discrimination and word correction to improve the training. In experiments, we verify the effectiveness of our model on MS-COCO and Flickr30K compared with current state-of-the-art models and demonstrates its robustness and faithfulness in the further analysis.

## introduction the task of image-text retrieval takes a query image (sentence) as input and finds out matched sentences (images) from a candidate pool. the key component of the retrieval framework is the similarity computation of an image-sentence pair and it aims to assign higher scores to positive pairs than negative ones. triplet loss is widely applied for training. take image-to-text as example 1 , it constructs two image-sentence pairs using an image and two sentences (one is relevant and the other is not), and the optimization process increases the similarity of the 1 to keep the presentation simple and clear, we use imageto-text as example to represent tasks in both ways throughout the paper. matching score of positive sentence matching score of negative sentence triple with generated negative sentence corresponding to sampled triple with the most difficult negative sentence in dataset triple with generated negative sentence triple with the most difficult negative sentence in dataset triple with the most difficult negative sentence in batch in this line of work, researchers explore various strategies to search mismatched sentences for a query image, from randomly choosing mismatched sentences to using the most similar one. the search backbone ……. ## framework the overall framework of tailoring negative sentences with word discrimination and correction (tags-dc) is shown in figure 2 . for each positive image-text pair (i i , t i ), we first generate negative sentences t - i through scene-graph based masking and refilling t i on the basis of masked language model (mlm) in §2.1. second, we utilizes both retrieved and synthetic negative sentences for the training of image-text matching (irtm and istm) in §2.2, where synthetic negative sentences are exploited in sentence-level. third, we propose to train the synthetic sentence generator in a dynamic way to keep pace with the upgrading of matching model. fourth, in §2.4, we apply word-level tasks of word discrimination (wod) and word correction (woc) on t - i to discover their differences with t i for further training. mlm, irtm, istm, woc and wod share the same backbone m θ and have their own heads, namely, h mlm , h itm , h woc and h wod . the detailed training step is illustrated in algorithm 1 in appendix. ## experiment dataset we evaluate our model on ms-coco @xcite and @xcite . in ms-coco, each image is accompanied with 5 human annotated captions. we split the dataset following @xcite with 113,287 images in the training set and 5,000 images in the validation and test sets, respectively. flickr30k @xcite consists of 31000 images collected from the flickr website, and every image contains 5 text descriptions. we take the same splits as in @xcite , with 1000 images for validation and 1000 images for testing, and the rest for training. 500 ## related work image-text retrieval most works in image-text retrieval focus on better feature extraction and cross-modal interaction. @xcite and @xcite represent the image by semantics gathered from block-based attention. a line of research @xcite @xcite @xcite @xcite @xcite detects features by pre-trained faster r-cnn @xcite . some other methods also focus on enhancing cross-modality relationship modeling, such as the dual attention network @xcite , the stacked cross attention @xcite @xcite , the graph structure attention @xcite , and the multimodal transformer modeling @xcite . uniter @xcite , unicoder @xcite and ernie-vil @xcite follow bert @xcite to pretrain the vision-language transformer model on the large-scale image-text datasets, and finetune in image-text retrieval. negative samples in contrastive learning selection strategies for negative samples have been widely studied in metric learning @xcite @xcite @xcite . @xcite employ distance weighted sampling to select more informative and stable examples. ge (2018) present a novel hierarchical triplet loss capable of automatically collecting informative training samples. in image-text retrieval, early works @xcite @xcite utilize random negative samples for training. vse++ @xcite incorporates difficult negative ones in the multi-modal embedding learning. the method is widely applied in the following works @xcite , and achieves significant performance improvement. uniter @xcite randomly samples a portion of texts (∼512) from the dataset and picks up the hardest ones. aoq @xcite selects these hard-todistinguish cases from the whole dataset through a pre-trained itm model and assigns hierarchical and adaptive penalties for samples with different difficulties. uniter+dg (zhang et al., 2020) samples hard negative sentences according to the structure relevance based on denotation graph @xcite . these methods are retrieval-based and inspire us to find more difficult negative sentences through generation. chuang et al. (2020) propose a method for debiasing, i.e., correcting for the fact that some negative pairs may be false negatives. in our work, we mask keywords (objects, attributes, and relationships) in the positive sentence then refilling, and exclude these sentences of which each token is included in image annotated sentences. this method introduces new keywords and alleviates the generation of false negative samples. @xcite consider applying mixup to produce hard negatives in latent space. in our work, we directly rewrite the positive sentences that is missing in the latent space based method, and this improves the robustness and faithfulness. the most similar work is vse-c @xcite that attacks the vse++ @xcite through replacing the nouns, numerals, and relations according to language priors of human and the wordnet knowledge base. compare with vse-c @xcite , our method has three advantages. (1) our model does not depend on rules. ## conclusion in this paper, we focus on the image-text retrieval task and find that retrieve-based negative sentence construction methods are limited by the dataset scale. to further improve the performance, we propose tailoring negative sentences (tags). it utilizes masking and refilling to produce synthetic negative sentences as negative samples. we also set up the word discrimination and word correction to introduce word-level supervision to better exploit the synthetic negative sentences. our model shows competitive performance in ms-coco and flickr30k compared with current state-of-the-art models. we also demonstrate the behavior of our model is robust and faithful. bowen zhang, hexiang hu, vihan jain, eugene ie, and fei sha. 2020. learning to represent image and text with denotation graph. arxiv preprint arxiv:2010.02949. yi zheng, yuejie zhang, rui feng, tao zhang, and weiguo fan. 2021. stacked multimodal attention network for context-aware video captioning. ieee transactions on circuits and systems for video technology, 32(1):31-42.