Negative Sample is Negative in Its Own Way: Tailoring Negative Sentences for Image-Text Retrieval
Abstract
Matching model is essential for Image-Text Retrieval framework.
Existing research usually train the model with a triplet loss and explore various strategy to retrieve hard negative sentences in the dataset.
We argue that current retrieval-based negative sample construction approach is limited in the scale of the dataset thus fail to identify negative sample of high difficulty for every image.
We propose our
TAiloring neGative Sentences with Discrimination and Correction (TAGS-DC) to generate synthetic sentences automatically as negative samples.
TAGS-DC is composed of masking and refilling to generate synthetic negative sentences with higher difficulty.
To keep the difficulty during training, we mutually improve the retrieval and generation through parameter sharing.
To further utilize fine-grained semantic of mismatch in the negative sentence, we propose two auxiliary tasks, namely word discrimination and word correction to improve the training.
In experiments, we verify the effectiveness of our model on MS-COCO and
Flickr30K compared with current state-of-theart models and demonstrates its robustness and faithfulness in the further analysis.
Introduction
The task of image-text retrieval takes a query image (sentence) as input and finds out matched sentences (images) from a candidate pool.
The key component of the retrieval framework is the similarity computation of an image-sentence pair and it aims to assign higher scores to positive pairs than negative ones.
Triplet loss is widely applied for training.
Take image-to-text as example 1 , it constructs two image-sentence pairs using an image and two sentences (one is relevant and the other is not), and the optimization process increases the similarity of the [Figure] S c o r e o f
N e g a ti
[Figure]
e
[Figure]
Triplewithgeneratednegativesentencecorrespondingto
Sampledtriplewiththemostdifficultnegativesentenceindataset
Triplewithgeneratednegativesentence
Triplewiththemostdifficultnegativesentenceindataset Triplewiththemostdifficultnegativesentenceinbatch
[Figure]
(a) The diagram plots a triplet (image, positive sentence, negative sentence) as a dot is defined by matching score of the positive pair on the X-axis and that of the negative pair on the Y-axis.
The matching scores are also computed by
CLIP(ViT-B/32) (Radford et al., 2021).
[Figure]
(b) Two images with the positive sentence (P), the most difficult negative one (N) retrieved from dataset by CLIP and the generated negative one (G).
The score is the cosine similarity computed by CLIP and larger is better.
The underlined red words are non-correspondence ones to the image.
(b) in Flickr30K (Plummer et al., 2015).
positive pair while decreasing that of the negative one.
Previous research (Xuan et al., 2020) reveals that models trained with harder negative samples, i.e., sentences that are more difficult to be distinguished, can generally achieve better performance.
In this line of work, researchers explore various strategies to search mismatched sentences for a query image, from randomly choosing mismatched sentences to using the most similar one.
The search
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
scope moves from a single training batch (Karpathy and Fei-Fei, 2015; Faghri et al., 2018; Kiros et al., 2014; Socher et al., 2014; Lee et al., 2018;
Li et al., 2019) to the whole dataset (Chen et al., 2020a; Zhang et al., 2020).
Although promising results have been reported by searching for harder negative samples in a larger scope, the effectiveness is limited by the scale of the dataset.
To compare the effectiveness of these strategies, we randomly sample 3 , 000 images in
Flickr30K (Plummer et al., 2015) and plot training triples constructed in Figure 1.
Each dot stands for a triple (image, positive sentence, negative sentence), and X-axis is the matching score of the positive image-sentence pair while Y-axis is that of the negative one.
In general, triples located on the left of the dotted line are more difficult to be distinguished because matching score of the negative pair is higher than the positive one or comparable.
As we can see, triples obtained by searching the most difficult mismatched sample in the batch are largely located on the right of the dotted line, and the matching scores of negative pairs are much smaller with a gap larger than 0.05 on average (in the right of the solid line).
When enlarging the searching scope to the whole dataset, triples move up in positions, and around 40% of negative pairs obtain higher matching scores than positive ones.
However, there are still 18% of images that can only recruit negative samples with a matching score 0.05 lower than its positive counterpart.
This confirms the limitation of retrieve-based negative sample construction strategy.
To have a better understanding, we present two triples in Figure 1 i.e., ( P 1 , N 1 ) and ( P 2 , N 2 ) (denoted as black cross).
It shows that negative sentences N
and N 2 describe scenes with significant differences compared with the query images, therefore, they are easy to be distinguished.
Given that a high percentage of images obtain these low-quality negative sentences in the dataset, we believe it is necessary to collect negative samples beyond retrieval.
Instead of searching for original sentences in the dataset, we explore constructing artificial negative samples by editing positive sentences.
We demonstrate two generated sentences in Figure 1,
G 1 replaces “man" with “woman" on P 1 and Greplaces “area" with “beach" on P 2 .
The generated sentences obtain comparable or even higher matching scores than positive ones.
We further generate artificial sentences for all images to form a new set of triples.
These triples are plotted in Figure 1 as pink dots.
We can see all of them located on the left side of the dotted line, which means they are more difficult to be distinguished.
In this paper, we propose TAiloring neGative
Sentences (TAGS) by rewriting keywords in positive sentences of a query image to construct negative samples automatically.
In specific, we employ the strategy of masking and refilling .
In masking, we construct scene graph for the positive sentence and mask elements in the graph (objects, attributes, and relations).
Refilling replaces the masked original words with mismatched ones to construct the negative sample.
In the training process, we further propose two word-level tasks, word discrimination and word correction , to incorporate fine-grained supervision into consideration.
Word discrimination requires the model to distinguish which words lead to the mismatch, and word correction demands the regeneration of the original words.
Both tasks evaluate the capability of the model to identify minor differences between synthetic sentences and positive ones.
During inference, the output of two tasks can provide fine-grained information through highlighting and revising mismatched words, and these can be regarded as the explanation for the decision made by the model to improve the interpretability.
We evaluate our model on MS-COCO (Lin et al., 2014) and Flickr30K
(Plummer et al., 2015).
Experiment results show the effectiveness of our model.
Our contributions are three-fold: (1) We propose a generation-based method to construct negative samples to improve the training efficiency of image-text retrieval model.
(2) To fully exploit the synthetic negative sentences, we propose two training tasks, word discrimination and word correction, to incorporate the fine-grained supervision to enhance the multi-modal local correspondence modeling.
(3) Our model generates state-of-theart performance on two public datasets MS-COCO and Flickr30K.
Framework
The overall framework of TAiloring neGative Sentences with word Discrimination and Correction (TAGS-DC) is shown in Figure 2.
For each positive image-text pair ( I i , T i ) , we first generate negative sentences T − i through scene-graph based masking and refilling T i on the basis of masked language model (MLM) in §2.1.
Second, we utilizes both retrieved and synthetic negative sentences for the training of image-text matching (IRTM and ISTM) in §2.2, where synthetic negative sentences are exploited in sentence-level.
Third, we propose to train the synthetic sentence generator in a dynamic way to keep pace with the upgrading of matching model.
Fourth, in §2.4, we apply word-level tasks of word discrimination (WoD) and word correction (WoC) on T − i to discover their differences with T i for further training.
MLM, IRTM, ISTM, WoC and WoD share the same backbone M θ and have their own heads, namely, H MLM , H ITM , H WoC and
H WoD .
The detailed training step is illustrated in
Algorithm 1 in appendix.
2. Scene-graph based Sentence Generation and Filtering
In general, negative sentences with more overlapped words with positive sentences tend to obtain higher matching scores with the query image, thus are more difficult to be distinguished.
Therefore, we propose to edit relevant sentences to construct negative samples for a query image.
After the sentence generation, we control the quality by filtering the false negative sentences.
To ensure the editing operates on key semantic units of the sentence, we use a strategy based on scene-graph.
2.1. Scene-graph based Sentence Editing
The module of sentence editing takes a relevant sentence of the query image as input and outputs a synthetic sentence.
It first identifies some key semantic units in the sentence and replaces them with other words.
We employ a masked language model for this process following two steps namely, masking and refilling.
To identify the key semantic of a sentence, we construct the scene graph for a relevant sentence through scene graph parser of SPICE (Anderson et al., 2016) following SGAE
(Yang et al., 2019).
We then collect objects, relations, and attributes as candidates for masking.
To control the semantic offset of the synthetic sentence T ( k ) i , we randomly mask 15% tokens of sentence.
In the step of refilling, we use the output head
H MLM , which is a two-layer feed-forward network (FFN), on top of the backbone M θ for masked language modeling.
Thus, image I i also gets involved in MLM to guide the refilling later.
The detailed computation of L MLM is shown in Eq.
(1), where ◦ is the function composition and NLL is the loss of negative log-likelihood.
[Equation]
Then during refilling process, we put T ( k ) i into
MLM to produce the logit scores, then sample the synthetic sentence T ( k,l ) i following the distribution which originates from the logit with temperature τ as Eq.
[Equation]
We conduct the masking and refilling steps for K and L times to generate candidate synthetic sentences.
2.1. False Negative Sample Filtering
It hurts the training of using sentences that are relevant to the query image as negative samples (Chuang et al., 2020; Huynh et al., 2020).
Therefore we propose a filtering process to remove false negative ones of synthetic sentences.
In vision and language datasets, each image is annotated with multiple descriptive sentences.
For example, there are five in MSCOCO and Flickr30K. For a synthetic sentence, if its replaced tokens are completely included in these annotated sentences, we will treat it as relevant.
Based on this, we filter synthetic sentences which are relevant.
2. Image Text Matching
Given an image
I i and a sentence T j , the retrieval model assigns a matching score s ∈ [0 , 1] of ( I i , T j ) with an output head H ITM , which is a onelayer FFN, as Eq.
[Equation]
Triplet loss (TripL) is widely applied in image text matching.
With a hyper-parameter α , it takes a query image (text) U as an anchor for the matched (positive) image-text pair ( U, V ) against the mismatched (negative) pair ( U, W ) as the following equation.
[Equation]
Matching on Retrieved Cases
During training, for each positive image-text pair ( I i , T i ) , we retrieve a negative image I − i and a sentence T − i , then employ the loss of ITM in Eq.
(5) for training,
[Equation]
Matching on Synthetic Sentences
First, we pick up these relatively better generated negative sentences.
In practice, we compute the matching score between each synthetic negative sentence and I i as
Eq.
(6), and keep a synthetic negative sentence pool
T − i to make each of them as difficult as possible.
[Equation]
T ( k,l ) i | T ( k,l ) i ̸ = T i } ITM ( I i , T − t ) (6) where argmax - m is to pick out m sentences that earn the top- m matching scores.
Second, with synthetic sentences T − i in Eq.
(6), we utilize them and the positive one T i to compute the triplet loss, and get L ISTM in Eq.
[Equation]
2. Dynamic Training Strategy of Negative Sample Generation for Image-Text Matching
The naive choice of MLM is to keep a pre-trained static one: pre-training a MLM in advance and fixing its parameters during the training of ITM.
Recall that L ISTM encourages the ITM model to learn the pattern of synthetic sentences and keep them away from the image, we consider that negative sentences generated by the static MLM would be no longer difficult for the ITM model as the training goes on.
We propose to use the dynamic MLM that shares the M θ with ITM for mutual improvement.
Through the sharing, MLM continuously learns what is more relevant to the positive sentences and produces challenging negative ones for the improvement of ITM.
The stronger ITM helps
MLM to better identify the semantic alignment of image and keywords.
MLM achieves the improvement synchronously with ITM through interaction.
2. Auxiliary Tasks to Incorporate Fine-grained Supervision
L ISTM only provides sentence-level supervision and we argue it does not fully exploit the synthetic negative sentence.
We introduce two auxiliary tasks to utilize the word-level difference and further enhance the model capability in multi-modal local correspondence modeling.
Word Discrimination
The task is to determine whether each word of the synthetic sentence T − t ∈
T − i is matched with I i , and we regard the replaced words of T − t as mismatched ones and others as matched ones.
The target label G t of T − t ∈ T − i is determined following G t,j = 1 if s i,j = s t,j else 0 , where s i,j and s t,j are the j -th token of T i and
T − t .
We set up a new output head H WoD , and the objective of word discrimination is in Eq.
[Equation]
Word Correction
This task is to correct these mismatched words in T − t as Eq.
The task not only requires the model to comprehensively understand the gap between the synthetic negative sentences and the original positive ones, but also word-dependency knowledge and local crossmodal alignment to fill the gap.
H WoC is the output head for word correction, and the objective is
[Table] [Table]
shown in Eq.
[Equation]
2. Overall Training
Details of our training step are shown in Algorithm 1 in appendix.
The overall training loss of our model has five components as Eq.
(10) with hyperparameters λ IRTM , λ MLM , λ ISTM , λ WoD and λ WoC .
[Equation]
During inference, we employ the ITM to determine the matching score of the query image (text) and the candidate text (image) as Eq.
Experiment Dataset
We evaluate our model on MSCOCO (Lin et al., 2014) and Flickr30K (Plummer et al., 2015).
In MS-COCO, each image is accompanied with 5 human annotated captions.
We split the dataset following (Karpathy and Fei-Fei, 2015) with 113,287 images in the training set and 5, images in the validation and test sets, respectively.
Flickr30K (Plummer et al., 2015) consists ofimages collected from the Flickr website, and every image contains 5 text descriptions.
We take the same splits as in (Karpathy and Fei-Fei, 2015), with 1000 images for validation and 1000 images for testing, and the rest for training.
Models for Comparison
We compare our model with some competitive approaches, including
MMCA (Wei et al., 2020), and AOQ (Chen et al., 2020a).
We also compare with methods based on vision language pre-trained models: UNITER+DG (Zhang et al., 2020),
UnicoderVL (Li et al., 2020), LightningDOT (Sun et al., 2021), UNITER (Chen et al., 2020b), CLIP (Radford et al., 2021) and ERNIE-ViL (Yu et al., 2020).
Implementation
We employ the pre-trained
UNITER (Chen et al., 2020b) with base (B) and large (L) settings as our backbone.
Evaluation Metrics
We report recall at K (R@K) and Rsum.
R@K is the fraction of queries for which the correct item is retrieved among the closest K points to the query.
RSum is the sum of
R@1+R@5+R@10 in both image-to-text and textto-image.
3. Overall Performance
The overall result is shown in Table 1.
TAGS is the model trained with generated negative samples, using the dynamic training strategy.
TAGSDC is our model built on top of TAGS, further trained using two auxiliary tasks.
In the base setting, our model achieves the best performance in terms of all metrics except R@1 and R@5 of in text-to-image on Flickr30K. In the large setting, our model also outperforms other models across all metrics except R@5 MS-COCO text-to-image and Flickr30K image-to-image R@10.
Compared with UNITER(L), our model achieves an improvement of 4.0 and 6.4 RSum points in MS-COCO and Flickr30K.
3. Ablation Study
We further demonstrate the effectiveness of different modules, namely, scene-graph based masking (denoted as PM), dynamic sentence generation (denoted as DG), and fine-grained training tasks (denoted as WoD and WoC) in Flickr30K. Original
TAGS is trained with PM and DG.
TAGS-DC is further trained with WoD and WoC.
Scene-graph VS Word based Masking
We replace the scene-graph based masking with wordbased masking (denoted as WM) to form TAGS w/ WM.
Detailed results are shown in Table 2.
WM follows the original sampling method of
UNITER (Chen et al., 2020b) that randomly sample 15% tokens to mask, and PM is introduced in §2.1.
TAGS outperforms TAGS w/ WM in terms of all metrics, and this verifies the effectiveness of
PM.
Dynamic VS Static Generator
We replace DG with a static sentence generator (denoted as SG) to form TAGS w/ SG.
The difference between TAGS and TAGS w/ SG lies in that the former shares the parameters of ITM and MLM while the latter does not.
Both of them are initialized with the pre-trained UNITER-base and share the same hyperparameters.
In detail, we set λ MLM = 0 .
1 and λ ISTM = 0 .
The static generator is fixed as a fine-tuned UNITER+MLM model.
The performance of TAGS w/ SG is not so good as TAGS.
This demonstrates the effectiveness of DG.
WoD and WoC
In Table 2, TAGS-DC outperforms TAGS in both MS-COCO and Flickr30K.
This reveals that word discrimination and correction contribute to the performance of ITM.
Further Analysis 4. Difficulty Distribution of Samples from Dynamic and Static Generator
To see the difficulty of negative samples constructed by various generation strategies, we plot the value distribution of samples.
To evaluate the difficulty, we compute the similarity gap between the positive pair ITM ( I i , T i ) and the negative one
ITM ( I i , T − t ) .
We plot the value of negative pair minus positive one with respect to training steps (X-axis).
In general, higher value means higher difficulty.
The result is shown in Figure 3 where the darker color means more samples.
The overall values of TAGS w/ SG (Figure 3 (a)) are higher than
TAGS w/ DG (Figure 3 (b)).
This implies that the static generator fails to provide negative sentences close to the image for ITM during training while our generator with dynamic generating strategy is effective.
[Figure]
4. Quality Evaluation of Synthetic Sentences
We evaluate the quality of generated synthetic sentences in terms of automatic metrics and human evaluation.
Fluency
We utilize the pre-trained language model GPT-2 (Radford et al., 2019) to compute the perplexity of synthetic negative sentences for the measurement of their fluency.
We use positive sentences in the test set of Flickr30K as original ones and generate negative samples by TAGS and
VSE-C. Furthermore, we look into sentences after correction.
The overall results are shown in Table 3.
Compared with sentences produced by VSE-C, our synthetic sentences have much smaller perplexity.
After correction, the fluency of synthetic sentences can be improved.
Human Evaluation
We perform a human evaluation to see whether all negative sentences generated are true negative.
We randomly samplesentences generated by TAGS and ask two annotators to determine whether the synthetic sentences are mismatched to the corresponding images.
The result shows that 96.5% of synthetic sentences generated are true negative.
[Table]
4. Negative Sentences Discrimination
In this section, we explore to see if the generator can discriminate positive sentences from synthetic ones.
We compare UNITER and TAGS.
For a pair of sentences (one is positive and the other is a synthetic negative one), the generator should assign a higher score to the positive one.
We report the accuracy of discrimination.
We utilize two negative sentence generators TAGS and VSE-C (Shi et al., 2018).
Two versions of TAGS with different seeds are used for cross-validation.
Results are shown in
Table 4.
We have several findings as follows.
TAGS2 is trained with a different seed with TAGS1, but the performance of TAGS1 almost makes no difference in discriminating their generated sentences.
Although the synthetic sentences of VSE-C are constructed with human efforts, TAGS also outperforms UNITER by about 9%.
(3) Three generators [Table] negative sentences with different distributions, but TAGS performs better than UNITER consistently.
These facts validate the robustness of
TAGS.
[Table]
4. Effectiveness of Two Auxiliary Tasks
[Table]
Words with underline mean the regenerated words are different from the original ones.
We show the performance of our model in two auxiliary tasks, namely, word discrimination and correction in the testing set of Flickr30K.
In word discrimination, we use a threshold of 0.5 to split the positive and negative ones in terms of probability.
The accuracy of word discrimination is 66.5%.
In word correction, the accuracy is 87.3%.
With the probability, we can provide additional support information accompanied to the final decision of our model.
Two examples are presented in Figure 4.
TAGS-DC assigns lower scores for synthetic negative sentences than positive ones, but UNITER fails.
(2) Color of “carrying” and “playing playing” are yellow which means that our word discrimination successfully detects these mismatched words.
Our model finds the local alignment in word-level and grammatical errors, then generates “wearing” and “man player” for correction.
In the examples, word discrimination marks the mismatched components and word correction provides reasons for mismatching.
(3) Our model fails to identify two mismatched words, “swimming”, and “ball”.
Considering they are partially related to the image, our model is less effective in determining the relevance of these fuzzy words.
Related Work Image-Text Retrieval
Most works in image-text retrieval focus on better feature extraction and cross-modal interaction.
Nam et al.
(2017) and Ji et al.
(2019) represent the image by semantics gathered from block-based attention.
A line of research (Lee et al., 2018; Li et al., 2019; Wang et al., 2020; Wei et al., 2020; Li et al., 2021) detects features by pre-trained Faster R-CNN (Ren et al., 2015).
Some other methods also focus on enhancing cross-modality relationship modeling, such as the dual attention network (Nam et al., 2017), the stacked cross attention (Lee et al., 2018;
Liu et al., 2019; Hu et al., 2019), the graph structure attention (Liu et al., 2020), and the multimodal transformer modeling (Wei et al., 2020).
UNITER (Chen et al., 2020b), Unicoder (Li et al., 2020) and ERNIE-ViL (Yu et al., 2020) follow
BERT (Devlin et al., 2019) to pre-train the visionlanguage transformer model on the large-scale image-text datasets, and finetune in image-text retrieval.
Negative Samples in Contrastive Learning
Selection strategies for negative samples have been widely studied in metric learning (Schroff et al., 2015;
Oh Song et al., 2016; Harwood et al., 2017;
Suh et al., 2019; Zhang et al., 2020; Chen et al., 2020a).
Wu et al.
(2017) employ distance weighted sampling to select more informative and stable examples.
Ge (2018) present a novel hierarchical triplet loss capable of automatically collecting informative training samples.
In image-text retrieval, early works (Kiros et al., 2014; Karpathy and FeiFei, 2015; Socher et al., 2014) utilize random negative samples for training.
VSE++ (Faghri et al., 2018) incorporates difficult negative ones in the multi-modal embedding learning.
The method is widely applied in the following works (Lee et al., 2018; Wei et al., 2020), and achieves significant performance improvement.
UNITER (Chen et al., 2020b) randomly samples a portion of texts ( ∼ 512) from the dataset and picks up the hardest ones.
AOQ (Chen et al., 2020a) selects these hard-todistinguish cases from the whole dataset through a pre-trained ITM model and assigns hierarchical and adaptive penalties for samples with different difficulties.
UNITER+DG (Zhang et al., 2020) samples hard negative sentences according to the structure relevance based on denotation graph (Plummer et al., 2015).
These methods are retrieval-based and inspire us to find more difficult negative sentences through generation.
Chuang et al.
(2020) propose a method for debiasing, i.e., correcting for the fact that some negative pairs may be false negatives.
In our work, we mask keywords (objects, attributes, and relationships) in the positive sentence then refilling, and exclude these sentences of which each token is included in image annotated sentences.
This method introduces new keywords and alleviates the generation of false negative samples.
Kalantidis et al.
(2020) consider applying mixup to produce hard negatives in latent space.
In our work, we directly rewrite the positive sentences that is missing in the latent space based method, and this improves the robustness and faithfulness.
The most similar work is VSE-C (Shi et al., 2018) that attacks the VSE++ (Faghri et al., 2018) through replacing the nouns, numerals, and relations according to language priors of human and the WordNet knowledge base.
Compare with
VSE-C (Shi et al., 2018), our method has three advantages.
(1) Our model does not depend on rules.
(2) Our model is more flexible and can generate negative sentences with any number, but this is intractable for VSE-C. (
The generated sentences of our model are more fluent than these of VSE-C as the results in Table 4.
Conclusion
In this paper, we focus on the image-text retrieval task and find that retrieve-based negative sentence construction methods are limited by the dataset scale.
To further improve the performance, we propose TAiloring neGative Sentences (TAGS).
It utilizes masking and refilling to produce synthetic negative sentences as negative samples.
We also set up the word discrimination and word correction to introduce word-level supervision to better ex- ploit the synthetic negative sentences.
Our model shows competitive performance in MS-COCO and
Flickr30k compared with current state-of-the-art models.
We also demonstrate the behavior of our model is robust and faithful.
Acknowledgements
This work is partially supported by Natural Science
Foundation of China (No.6217020551), Science and Technology Commission of Shanghai Municipality Grant (No.20dz1200600, 21QA1400600,
GWV-1.1, 21511101000) and Zhejiang Lab (No.2019KD0AD01)
A Appendix A. Implementation Details
We have two settings, base and large.
The base setting of model has 12-layers, 768 hidden size and 12 attention heads and the large one has 24-layers, 1024 hidden size and 16 attention heads.
We utilize grid search to determine the hyperparameters.
In retrieval-based matching, we randomly samples 399 negative sentence (image) from the whole dataset for the query image (sentence), and pick out the top 31 ones from them according to the matching scores.
In the masked language modeling, we utilize the scene graph parser in SPICE to extract the phrases of objects, relationships and attributes from the positive sentence, and take these phrases as a whole to sample and mask.
The mask probability is 0.15.
In the generation enhanced matching, the temperature τ ∈ { 1.0, 1.5 } , and we set K
= L = 20 and | T − i | = 31 / 23 for the base and large settings.
λ ITM , λ MLM , λ ISTM , λ WoD and λ WoC is sampled from { 1 . 0 } , { 5e-2, 1e-1 } , { 1e-4, 5e-4, 1e-3 } , { 5e-4, 1e-3 } and { 5e-4, 1e-3 } , where we set λ WoD = λ WoC .
Our training is composed of two steps, (1) we train with ITM , MLM and ISTM with 5,000 steps as
NSG ; (2) we further train the model with the whole loss function as NSGDC with 1,500 steps.
The learning rate lr is sampled from { 5e-5, 4e-5, 1e-5 } .
We use a linear learning rate scheduler with 10% warmup proportion.
The Adam with β 1 = 0 .
9 and β 2 = 0 . 98 is taken as the optimizer. The dropout is 0.1. Our code is implemented with pytorch.
For base setting in Flickr30K, we utilize 8 V100 for training and the computation time is about 8 hours.
A. Algorithm of TAGS-DC
Input : A positive image-text pair ( I i , T i ) .
Parameter : Backbone M θ , the head of masked language model H MLM , image-text matching H ITM , word discrimination H WoD and word correction
H WoC .
1: # negative sentence generation .
2: Initializing (cid:98) T − i := {} .
3: for k in 1 , . . .
, K do 4:
Randomly masking T i to get the masked one
T ( k ) i .
Computing L MLM in Eq.
(1) with M θ and
H MLM .
6: for l in 1 , . . .
, L do 7:
Refilling T ( k ) i to generate a synthetic sentence T ( k,l ) i following Eq.
8: if T ( k,l ) i satisfies criteria C1 then 9:
Adding T k,l ) i to (cid:98) T − i and computing its matching score with I i .
10: end if 11: end for 12: end for 13:
14: Sampling negative sentence T − i to compute L IRTM Eq.
(5) with M θ H
15: Picking out top- m synthetic sentences by scores T − i .
16: Utilizing T − i
I i to compute L ISTM in
Eq.
(7) with M θ and H ITM .
18: for t in do 19:
Utilizing T − t
and I i to compute L WoD in
Eq.
(8) with M θ
and H WoD .
Utilizing T − t
and I i to compute L WoC in
Eq.
(9) with M θ and H WoC .
21: end for
[Table]