Original: UNITER: L EARNING UN IVERSAL I MAGE -TE XT R EPRESENTATIONS
Revised: UNITER: L EARNING UN IVERSAL I MAGE -TE XT R EPRESENTATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are jointly processed for visual and textual understanding.
Revised: Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are jointly processed for visual and textual understanding.

Original: In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.
Revised: In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.

Original: We design three pre-training tasks: Masked Language Modeling (MLM), ImageText Matching (ITM), and Masked Region Modeling (MRM, with three variants).
Revised: We design three pre-training tasks: Masked Language Modeling (MLM), ImageText Matching (ITM), and Masked Region Modeling (MRM, with three variants).

Original: Different from concurrent work on multimodal pre-training that apply joint random masking to both modalities, we use conditioned masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text).
Revised: Different from concurrent work on multimodal pre-training that apply joint random masking to both modalities, we use conditioned masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text).

Original: Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking.
Revised: Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking.

Original: We also conduct a thorough ablation study to ﬁnd an optimal setting for the combination of pre-training tasks.
Revised: We also conduct a thorough ablation study to ﬁnd an optimal setting for the combination of pre-training tasks.

Original: Extensive experiments show that UNITER achieves new state of the art across six
Revised: Extensive experiments show that UNITER achieves new state of the art across six

Original: V+L tasks (over nine datasets), including Visual Question Answering, Image-Text
Revised: V+L tasks (over nine datasets), including Visual Question Answering, Image-Text

Original: Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning, Visual Entailment, and NLVR 2 .
Revised: Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning, Visual Entailment, and NLVR 2 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Most Vision-and-Language tasks rely on joint multimodel embeddings to bridge the semantic gap between visual and textual clues in images and text, although such representations are usually tailored for speciﬁc tasks.
Revised: Most Vision-and-Language tasks rely on joint multimodel embeddings to bridge the semantic gap between visual and textual clues in images and text, although such representations are usually tailored for speciﬁc tasks.

Original: For example, MCB (Fukui et al., 2017) and BAN (Kim et al., 2018) proposed advanced multimodal fusion methods for Visual Question Answering (VQA) (Antol et al., 2015).
Revised: For example, MCB (Fukui et al., 2017), BAN (Kim et al., 2018), DFAF (Gao et al., 2019) proposed advanced multimodal fusion methods for Visual Question Answering (VQA) (Antol et al., 2015).

Original: SCAN (Lee et al., 2018) and MAttNet (Yu et al., 2018) studied learning latent alignment between words and image regions for Image-Text Retrieval (Wang et al., 2016) and Referring Expression Comprehension (Kazemzadeh et al., 2014) tasks.
Revised: SCAN (Lee et al., 2018) and MAttNet (Yu et al., 2018) studied learning latent alignment between words and image regions for Image-Text Retrieval (Wang et al., 2016) and Referring Expression Comprehension (Kazemzadeh et al., 2014) tasks.

Original: While each of these proposed models has pushed the state of the art on respective benchmarks, their architectures are diverse and the learned representations are highly task-speciﬁc, preventing them from being generalized to other tasks.
Revised: While each of these proposed models has pushed the state of the art on respective benchmarks, their architectures are diverse and the learned representations are highly task-speciﬁc, preventing them from being generalized to other tasks.

Original: This raises a million-dollar question: can we learn a universal image-text representation for all V+L tasks?
Revised: This raises a million-dollar question: can we learn a universal image-text representation for all V+L tasks?

Original: To answer this question, we introduce UN iversal I mage- TE xt R epresentations ( UNITER ), a largescale pre-trained model for multimodal embedding.
Revised: To answer this question, we introduce UN iversal I mage- TE xt R epresentations ( UNITER ), a largescale pre-trained model for multimodal embedding.

Original: We adopt Transformer (Vaswani et al., 2017) as the core of our model, to leverage its elegant self-attention mechanism designed for learning contextualized representations.
Revised: We adopt Transformer (Vaswani et al., 2017) as the core of our model, to leverage its elegant self-attention mechanism designed for learning contextualized representations.

Original: Inspired by BERT (Devlin et al., 2019), which has successfully applied
Revised: Inspired by BERT (Devlin et al., 2019), which has successfully applied

Original: Transformer to NLP tasks through large-scale language modeling, we pre-train UNITER through three pre-training tasks: ( i ) Masked Language Modeling (MLM) conditioned on image ; ( ii ) Masked
Revised: Transformer to NLP tasks through large-scale language modeling, we pre-train UNITER through three pre-training tasks: ( i ) Masked Language Modeling (MLM) conditioned on image ; ( ii ) Masked

Original: Region Modeling (MRM) conditioned on text ; and ( iii ) joint Image-Text Matching (ITM).
Revised: Region Modeling (MRM) conditioned on text ; and ( iii ) joint Image-Text Matching (ITM).

Original: To further investigate the effectiveness of MRM, we propose three MRM variants: ( i ) Masked Region
Revised: To further investigate the effectiveness of MRM, we propose three MRM variants: ( i ) Masked Region

Original: Classiﬁcation (MRC); ( ii ) Masked Region Feature Regression (MRFR); and ( iii ) Masked Region
Revised: Classiﬁcation (MRC); ( ii ) Masked Region Feature Regression (MRFR); and ( iii ) Masked Region

Original: Classiﬁcation with KL-divergence (MRC-kl).
Revised: Classiﬁcation with KL-divergence (MRC-kl).

Original: As shown in Figure 1, UNITER ﬁrst encodes image regions (visual features and bounding box features) and textual words (tokens and positions) into a common embedding space with Image
Revised: As shown in Figure 1, UNITER ﬁrst encodes image regions (visual features and bounding box features) and textual words (tokens and positions) into a common embedding space with Image

Original: Embedder and Text Embedder, then applies a Transformer module to learn generalizable contextualized embeddings for each region and word through aforementioned pre-training tasks. Com-
Revised: Embedder and Text Embedder, then applies a Transformer module to learn generalizable contex-

Original: [Figure]
Revised: [Figure]

Original: man with his dog on a couch
Revised: man with his dog on a couch

Original: Text Embedder +
Revised: Text Embedder +

Original: Text Feature
Revised: Text Feature

Original: LN
Revised: LN

Original: Emb Emb
Revised: Emb Emb

Original: Token Position
Revised: Token Position

Original: [Figure] Embedder +
Revised: Image Embedder +

Revised: LN

Original: [Figure]
Revised: [Figure]

Original: UNITER
Revised: FC

Original: [Figure]
Revised: [Figure]

Revised: CNN Location

Original: his [Figure] UNITER 
Revised: UNITER man with his

Original: [Figure]
Revised: [Figure]

Original: MASK
Revised: UNITER man with his [MASK

Original: [Figure]
Revised: [Figure]

Original: dog
Revised: dog

Original: UNITER [CLS
Revised: UNITER [CLS

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Transformer pared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized representations for image regions and textual words through a single Transformer.
Revised: Transformer tualized embeddings for each region and word through aforementioned pre-training tasks. Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized representations for image regions and textual words through a single Transformer.

Original: Besides, our masked language/region modeling is conditioned on full observation of image/text, different from other concurrent pre-trained models that apply joint random masking to both modalities.
Revised: Besides, our masked language/region modeling is conditioned on full observation of image/text, different from other concurrent pre-trained models that apply joint random masking to both modalities.

Original: We show that the conditional masking strategy can successfully ease the missing-alignment between images and text, and obtain better joint embeddings for downstream tasks.
Revised: We show that the conditional masking strategy can successfully ease the missing-alignment between images and text, and obtain better joint embeddings for downstream tasks.

Original: Detailed ablation study also demonstrates that the combination of MLM+ITM+MRC-kl+MRFR yields the best pre-training performance.
Revised: Detailed ablation study also demonstrates that the combination of MLM+ITM+MRC-kl+MRFR yields the best pre-training performance.

Original: To demonstrate the power of UNITER, we evaluate on six V+L tasks across nine datasets, including: ( i ) VQA; ( ii ) Visual Commonsense Reasoning (VCR) (Zellers et al., 2019); ( iii ) NLVR 2 (Suhr et al., 2019); ( iv ) Visual Entailment (Xie et al., 2019); ( v ) Image-Text Retrieval (including zero-shot setting)
Revised: To demonstrate the power of UNITER, we evaluate on six V+L tasks across nine datasets, including: ( i ) VQA; ( ii ) Visual Commonsense Reasoning (VCR) (Zellers et al., 2019); ( iii ) NLVR 2 (Suhr et al., 2019); ( iv ) Visual Entailment (Xie et al., 2019); ( v ) Image-Text Retrieval (including zero-shot setting)

Original: (Lee et al., 2018); and ( vi ) Referring Expression Comprehension.
Revised: (Lee et al., 2018); and ( vi ) Referring Expression Comprehension.

Original: Our UNITER model is trained on a large-scale V+L dataset composed of four subsets: ( i ) COCO (Lin et al., 2014); ( ii )
Revised: Our UNITER model is trained on a large-scale V+L dataset composed of four subsets: ( i ) COCO (Lin et al., 2014); ( ii )

Original: Visual Genome (VG) (Krishna et al., 2017); ( iii ) Conceptual Captions (CC) (Sharma et al., 2018); and ( iv ) SBU Captions (Ordonez et al., 2011).
Revised: Visual Genome (VG) (Krishna et al., 2017); ( iii ) Conceptual Captions (CC) (Sharma et al., 2018); and ( iv ) SBU Captions (Ordonez et al., 2011).

Original: Experiments show that UNITER achieves new state of the art with signiﬁcant performance boost across all six downstream tasks.
Revised: Experiments show that UNITER achieves new state of the art with signiﬁcant performance boost across all six downstream tasks.

Original: Moreover, training on additional CC and SBU data (containing unseen images/text in downstream tasks) further boosts model performance over training on COCO and VG only.
Revised: Moreover, training on additional CC and SBU data (containing unseen images/text in downstream tasks) further boosts model performance over training on COCO and VG only.

Original: Our contributions can be summarized as follows: ( i ) We introduce UNITER, a powerful UNiversal
Revised: Our contributions can be summarized as follows: ( i ) We introduce UNITER, a powerful UNiversal

Original: Image-TExt Representations for Vision-and-Language tasks.
Revised: Image-TExt Representations for Vision-and-Language tasks.

Original: ( ii ) We achieve new state of the art (SOTA) on multiple V+L benchmarks, outperforming existing SOTA and concurrent multimodal pre-training methods by a large margin.
Revised: ( ii ) We achieve new state of the art (SOTA) on multiple V+L benchmarks, outperforming existing SOTA and concurrent multimodal pre-training methods by a large margin.

Original: ( iii )
Revised: ( iii )

Original: We present extensive experiments and analysis to provide useful insights on the effectiveness of each pre-training task/dataset for multimodal encoder training.
Revised: We present extensive experiments and analysis to provide useful insights on the effectiveness of each pre-training task/dataset for multimodal encoder training.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Self-supervised learning utilizes original data as its own source of supervision, which has been applied to many Computer Vision tasks, such as image colorization (Zhang et al., 2016), solving jigsaw puzzles (Noroozi & Favaro, 2016; Trinh et al., 2019), inpainting (Pathak et al., 2016), rotation prediction (Gidaris et al., 2018), and relative location prediction (Doersch et al., 2015).
Revised: Self-supervised learning utilizes original data as its own source of supervision, which has been applied to many Computer Vision tasks, such as image colorization (Zhang et al., 2016), solving jigsaw puzzles (Noroozi & Favaro, 2016; Trinh et al., 2019), inpainting (Pathak et al., 2016), rotation prediction (Gidaris et al., 2018), and relative location prediction (Doersch et al., 2015).

Original: Recently, pre-trained language models such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019),
Revised: Recently, pre-trained language models such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019),

Original: GPT2 (Radford et al., 2019), and XLNet (Yang et al., 2019) have shown great advances for NLP tasks.
Revised: GPT2 (Radford et al., 2019), and XLNet (Yang et al., 2019) have shown great advances for NLP tasks.

Original: There are two keys to their success: effective pre-training tasks over large language corpus, and the use of Transformer (Vaswani et al., 2017) for learning contextualized text representations.
Revised: There are two keys to their success: effective pre-training tasks over large language corpus, and the use of Transformer (Vaswani et al., 2017) for learning contextualized text representations.

Original: More recently, there has been some concurrent work on self-supervised learning for multimodal tasks, by pre-training on large-scale image/video and text pairs, then ﬁnetuning on downstream tasks.
Revised: More recently, there has been some concurrent work on self-supervised learning for multimodal tasks, by pre-training on large-scale image/video and text pairs, then ﬁnetuning on downstream tasks.

Original: For example, VideoBERT (Sun et al., 2019) applied BERT to learn a bidrectional joint distribution over quantized video frame features and linguistic tokens from video-text pairs.
Revised: For example, VideoBERT (Sun et al., 2019) applied BERT to learn a bidrectional joint distribution over quantized video frame features and linguistic tokens from video-text pairs.

Original: ViLBERT (Lu et al., 2019) and LXMERT (Tan & Bansal, 2019) introduced the two-stream architecture, where two Transformers are applied to images and text independently, which will be fused by a third
Revised: ViLBERT (Lu et al., 2019) and LXMERT (Tan & Bansal, 2019) introduced the two-stream architecture, where two Transformers are applied to images and text independently, which will be fused by a third

Original: Transformer in a later stage.
Revised: Transformer in a later stage.

Original: On the other hand, VisualBERT (Li et al., 2019b), Unicoder-VL (Li et al., 2019a), VL-BERT (Su et al., 2019) and B2T2 (Alberti et al., 2019) proposed the singlestream architecture, where a single Transformer is applied to both image and text.
Revised: On the other hand, VisualBERT (Li et al., 2019b), Unicoder-VL (Li et al., 2019a), VL-BERT (Su et al., 2019) and B2T2 (Alberti et al., 2019) proposed the singlestream architecture, where a single Transformer is applied to both image and text.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: LXMERT model was pre-trained with downstream tasks such as VQA (Antol et al., 2015) and
Revised: LXMERT model was pre-trained with downstream tasks such as VQA (Antol et al., 2015) and

Original: GQA (Hudson & Manning, 2019), while the others were pre-trained on image-text pairs only.
Revised: GQA (Hudson & Manning, 2019), while the others were pre-trained on image-text pairs only.

Original: Our
Revised: Our

Original: UNITER model belongs to the second family.
Revised: UNITER model belongs to the second family.

Original: One key difference between UNITER and the other methods is the masking approach on pre-training tasks.
Revised: One key difference between UNITER and the other methods is the masking approach on pre-training tasks.

Original: Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.
Revised: Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted.

Original: In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.
Revised: In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.

Revised: Another related work is DDAF (Gao et al., 2019), which proposed a novel architecture of intermodality and intra-modality attention modules to learn the latent alignment between two modalities for VQA.

Revised: Compared with Gao et al.

Revised: (2019), UNITER learns a relatively more generic V+L representation via pre-training.

Original:  UN IVERSAL I MAGE -TE XT R EPRESENTATIONS
Revised:  UN IVERSAL I MAGE -TE XT R EPRESENTATIONS

Original: In this section, we ﬁrst introduce the model architecture of UNITER (Section 3.1), then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3).
Revised: In this section, we ﬁrst introduce the model architecture of UNITER (Section 3.1), then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3).

Original: 3. M ODEL O VERVIEW
Revised: 3. M ODEL O VERVIEW

Original: The model architecture of UNITER is illustrated in Figure 1.
Revised: The model architecture of UNITER is illustrated in Figure 1.

Original: Given a pair of image and sentence,
Revised: Given a pair of image and sentence,

Original: UNITER takes the visual regions of the image and textual tokens of the sentence as the input.
Revised: UNITER takes the visual regions of the image and textual tokens of the sentence as the input.

Original: We design an Image Embedder and a Text Embedder to extract their respective embeddings.
Revised: We design an Image Embedder and a Text Embedder to extract their respective embeddings.

Original: These embeddings are then fed into a multi-layer self-attention Transformer to learn a cross-modality contextualized embedding between visual regions and textual tokens.
Revised: These embeddings are then fed into a multi-layer self-attention Transformer to learn a cross-modality contextualized embedding between visual regions and textual tokens.

Original: Note that the self-attention mechanism in Transformer is order-less, thus it is necessary to explicitly encode positions/locations of tokens/regions as additional inputs.
Revised: Note that the self-attention mechanism in Transformer is order-less, thus it is necessary to explicitly encode positions/locations of tokens/regions as additional inputs.

Original: Speciﬁcally, in Image Embedder , we ﬁrst use Faster R-CNN 1 to extract the visual features (pooled
Revised: Speciﬁcally, in Image Embedder , we ﬁrst use Faster R-CNN 1 to extract the visual features (pooled

Original: ROI features) for each region.
Revised: ROI features) for each region.

Original: We also encode the location features for each region via a 7dimensional vector 2 .
Revised: We also encode the location features for each region via a 7dimensional vector 2 .

Original: Both visual and location features are then fed through a fully-connected (FC) layer, to be projected into the same embedding space.
Revised: Both visual and location features are then fed through a fully-connected (FC) layer, to be projected into the same embedding space.

Original: The ﬁnal visual embedding for each region is obtained by summing up the two FC outputs and then passing through a layer normalization (LN) layer.
Revised: The ﬁnal visual embedding for each region is obtained by summing up the two FC outputs and then passing through a layer normalization (LN) layer.

Original: For Text Embedder , we follow BERT (Devlin et al., 2019) and tokenize the input sentence into WordPieces (Wu et al., 2016).
Revised: For Text Embedder , we follow BERT (Devlin et al., 2019) and tokenize the input sentence into WordPieces (Wu et al., 2016).

Original: The ﬁnal representation for each sub-word token 3 is obtained via summing up its word embedding and position embedding, followed by another LN layer 4 .
Revised: The ﬁnal representation for each sub-word token 3 is obtained via summing up its word embedding and position embedding, followed by another LN layer 4 .

Original: We introduce three main tasks to pre-train our model:
Revised: We introduce three main tasks to pre-train our model:

Original: Masked Language Modeling conditioned on image regions (MLM), Masked Region Modeling conditioned on input text (with three variants) (MRM), and Image-Text Matching (ITM).
Revised: Masked Language Modeling conditioned on image regions (MLM), Masked Region Modeling conditioned on input text (with three variants) (MRM), and Image-Text Matching (ITM).

Original: As shown in Figure 1, our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer.
Revised: As shown in Figure 1, our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer.

Original: Speciﬁcally, word masking is realized by replacing the token with a special token [MASK] , and region masking is implemented by replacing the visual feature vector with all zeros.
Revised: Speciﬁcally, word masking is realized by replacing the token with a special token [MASK] , and region masking is implemented by replacing the visual feature vector with all zeros.

Original: Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT.
Revised: Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT.

Original: This prevents potential miss-alignment when a masked region happens to be described by a masked word.
Revised: This prevents potential miss-alignment when a masked region happens to be described by a masked word.

Original: Empirically, we show that with conditional masking, our model is able to learn better embeddings (in Section 4.2).
Revised: Empirically, we show that with conditional masking, our model is able to learn better embeddings (in Section 4.2).

Original: Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM.
Revised: Lastly, we also learn an instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM.

Original: During training, we sample both positive and negative image-sentence pairs and learn their matching scores.
Revised: During training, we sample both positive and negative image-sentence pairs and learn their matching scores.

Original: We also use a special modality embedding to help the distinguish between textual and is also summed modality is omitted Figure
Revised: To pre-train UNITER with the aforementioned different tasks, we randomly sample one pre-training task for each mini-batch and train on only one objective per SGD update.

Original: 3. P RE - TRAINING T ASKS
Revised: 3. P RE - TRAINING T ASKS

Original: Masked Language Modeling (MLM) We denote the image regions as v = { v 1 , ..., v K } , the input words as w = { w 1 , ..., w T } , and the mask indices as m ∈ N M . {+
+}
Revised: Masked Language Modeling (MLM) We denote the image regions as v = { v 1 , ..., v K } , the input words as w = { w 1 , ..., w T } , and the mask indices as m ∈ N M . {+
+}

Original: In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones w m with special token [MASK] 5 .
Revised: In MLM, we randomly mask out the input words with probability of 15%, and replace the masked ones w m with special token [MASK] 6 .

Original: The goal is to predict these masked words based on the observation of their surrounding words w \ m and all image regions v , by minimizing the negative log-likelihood:
Revised: The goal is to predict these masked words based on the observation of their surrounding words w \ m and all image regions v , by minimizing the negative log-likelihood:

Original: [Equation]
Revised: [Equation]

Original: where θ is the trainable parameters.
Revised: where θ is the trainable parameters.

Original: Each pair ( w , v ) is sampled from the whole training set D .
Revised: Each pair ( w , v ) is sampled from the whole training set D .

Original: Image-Text Matching (ITM)
Revised: Image-Text Matching (ITM)

Original: In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities.
Revised: In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities.

Original: We apply an FC layer on top of  theTransformer output  of [CLS] , scoring how well the input image and the  text are matched with each other.
Revised: The inputs to ITM are a sentence and a set of image regions, and the output is a binary label (0 for negative match, and 1 for positive match). We extract the representation of [CLS] token as the joint representation of the input text and image, then fed into a fully connected layer and a sigmoid function to predict a score between 0 and 1.

Original: The scoring function is denoted as s θ ( w , v ) .
Revised: We denote the output score as s θ ( w , v ) .

Revised: The ITM supervision is over the [CLS] token.

Original: During training, we sample a positive or negative pair ( w , v ) from the dataset D at each step.
Revised: During training, we sample a positive or negative pair ( w , v ) from the dataset D at each step.

Original: The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples.
Revised: The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples.

Original: We denote the label as y ∈ { 0 , 1 } , indicating if the sampled pair is a match.
Revised: We denote the label as y ∈ { 0 , 1 } , indicating if the sampled pair is a match.

Original: Then we apply a binary cross-entropy loss for optimization:
Revised: Then we apply a binary cross-entropy loss for optimization:

Original: [Equation]
Revised: [Equation]

Original: Masked Region Modeling (MRM) Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%.
Revised: Masked Region Modeling (MRM) Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%.

Original: The model is trained to reconstruct the masked regions v m given the remaining regions v \ m and all the words w .
Revised: The model is trained to reconstruct the masked regions v m given the remaining regions v \ m and all the words w .

Original: The visual features v m of the masked region are replaced by zeros.
Revised: The visual features v m of the masked region are replaced by zeros.

Original: Unlike textual tokens that are represented as discrete labels, visual features are high-dimensional and continuous, thus cannot be supervised via class likelihood.
Revised: Unlike textual tokens that are represented as discrete labels, visual features are high-dimensional and continuous, thus cannot be supervised via class likelihood.

Original: Instead, we propose three variants for Masked Region Modeling, which share the same objective base:
Revised: Instead, we propose three variants for Masked Region Modeling, which share the same objective base:

Original: [Equation]
Revised: [Equation]

Original: 1) Masked Region Feature Regression (MRFR)
Revised: 1) Masked Region Feature Regression (MRFR)

Original: MRFR learns to regress the Transformer output of each masked region v ( i ) m to its visual features.
Revised: MRFR learns to regress the Transformer output of each masked region v ( i ) m to its visual features.

Original: Speciﬁcally, we apply an FC layer to convert its Transformer output into a vector h θ ( v ( i ) m ) of same dimension as the input ROI pooled feature r ( v ( i ) m ) .
Revised: Speciﬁcally, we apply an FC layer to convert its Transformer output into a vector h θ ( v ( i ) m ) of same dimension as the input ROI pooled feature r ( v ( i ) m ) .

Original: Then we apply L2 regression between the two: f θ ( v m | v \ m , w ) = (cid:80)Mi =1 (cid:107) h θ ( v ( i ) m ) − r
Revised: Then we apply L2 regression between the two: f θ ( v m | v \ m , w ) = (cid:80)

Original: ( v( i ) m ) (cid:107) 22 .
Revised: [Equation] v

Original: 2) Masked Region Classiﬁcation (MRC) MRC learns to predict the object semantic class for each masked region.
Revised: [Equation]

Original: We ﬁrst
Revised: i [Equation]

Original: feed the
Revised: m

Original: Transformer output
Revised: [Equation]

Original: of the masked region v ( i ) m into an FC layer to predict 
Revised: 2) Masked Region Classiﬁcation (MRC) MRC learns to predict the object semantic class for each masked region.

Original:  the scores of K object classes , which further goes through a softmax function to be transformed into a normalized distribution g θ ( v ( i ) m ) ∈ R K .
Revised: We ﬁrst feed the Transformer output of the masked region v ( i ) m into an FC layer to predict the scores of K object classes, which further goes through a softmax function to be transformed into a normalized distribution g θ ( v ( i ) m ) ∈ R K .

Original: Note that there is no ground-truth label, as the object categories are not provided.
Revised: Note that there is no ground-truth label, as the object categories are not provided.

Original: Thus, we use the object detection output from Faster RCNN and take the detected object category (with the highest conﬁdence score) as the label of the masked region, which will be converted into a one-hot vector c ( v ( i ) m ) ∈ R K .
Revised: Thus, we use the object detection output from Faster RCNN, and take the detected object category (with the highest conﬁdence score) as the label of the masked region, which will be converted into a one-hot vector c ( v ( i ) m ) ∈ R K .

Original: The ﬁnal objective minimizes the cross-entropy (CE) loss: f θ ( v m | v 
Revised: The ﬁnal objective minimizes the cross-entropy (CE) loss: f θ ( v m | v \ m , w )

Original: m , w = (cid:80) Mi =1 CE ( c 
Revised: = (cid:80) Mi =1 CE ( c ( v ( i ) m ) ,

Original:  v ( i ) m ) , g θ
Revised: g θ ( v ( i ) m )) .

Original: v
Revised: [Table]

Original: ( i ) m )
Revised: [Table]

Original:  Masked Region Classiﬁcation with KL-Divergence (MRC-kl) MRC takes the most likely object class from the object detection model as the hard label (w.p. 0 or 1).
Revised: 3) Masked Region Classiﬁcation with KL-Divergence (MRC-kl) MRC takes the most likely object class from the object detection model as the hard label (w.p. 0 or 1), which assumes the detected object class is the ground-truth label for the region.

Revised: However, this may not be true as no ground-truth label is provided for a detected region.

Original: We can also use its soft label as  supervision signals, which is the raw output from the detector, i.e., a distribution of object classes
Revised: Thus, in MRC-kl, we avoid this assumption by using the soft label as the supervision signals, which is the raw output from the detector, i.e., a distribution of object classes

Original: ˜ c
Revised: ˜ c

Original: ( v ( i ) m ) .
Revised: ( v ( i ) m ) .

Original: MRC-kl aims to distill such knowledge into UNITER, by minimizing the KL divergence between two distributions: f θ ( v m | v \ m , w ) = (cid:80) Mi
Revised: MRC-kl aims to distill such knowledge into UNITER as Hinton et al. (2015), by minimizing the KL divergence between two distributions: f θ ( v m | v \ m , w ) = (cid:80) Mi

Original: =1 D KL (˜ c ( v ( i ) m ) ||
Revised: =1 D KL (˜ c ( v ( i ) m ) ||

Original: g θ ( v ( i ) m )) .
Revised: g θ ( v ( i ) m )) .

Original: 3. P RE - TRAINING D ATASETS
Revised: 3. P RE - TRAINING D ATASETS

Original: We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014),
Revised: We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin et al., 2014),

Original: Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and
Revised: Visual Genome (VG) (Krishna et al., 2017), Conceptual Captions (CC) (Sharma et al., 2018), and

Original: SBU Captions (Ordonez et al., 2011).
Revised: SBU Captions (Ordonez et al., 2011).

Original: Only image and sentence pairs are used for our pre-training[Table] [Table] purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training.
Revised: Only image and sentence pairs are used for our pre-training purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training.

Original: To study the effects of different datasets on pre-training [Table] we divide the four datasets into two categories.
Revised: To study the effects of different datasets on pre-training, we divide the four datasets into two categories.

Original: The ﬁrst one consists of image captioning data from COCO and dense captioning data from VG.
Revised: The ﬁrst one consists of image captioning data from COCO and dense captioning data from VG.

Original: We call it “In-domain” data, as most V+L tasks are built on top of these two datasets.
Revised: We call it “In-domain” data, as most V+L tasks are built on top of these two datasets.

Original: To obtain a ‘fair’ data split, we merge the raw training and validation splits from COCO [Table] and exclude all validation and test images that appear in downstream tasks.
Revised: To obtain a ‘fair’ data split, we merge the raw training and validation splits from COCO, and exclude all validation and test images that appear in downstream tasks.

Original: We also exclude all co-occurring
Revised: We also exclude all co-occurring

Original: Flickr30K (Plummer et al., 2015) images via URL matching, as both COCO and Flickr30K images were crawled from Flickr and may have overlaps 6 .
Revised: Flickr30K (Plummer et al., 2015) images via URL matching, as both COCO and Flickr30K images were crawled from Flickr and may have overlaps 8 .

Original: The same rule was applied to Visual Genome as well.
Revised: The same rule was applied to Visual Genome as well.

Original: In this way, we obtain 5.6M image-text pairs for training and 131K image-text pairs for our internal validation, which is half the size of the dataset used in LXMERT (Tan & Bansal, 2019), due to the ﬁltering of overlapping images and the use of image-text pairs only.
Revised: In this way, we obtain 5.6M image-text pairs for training and 131K image-text pairs for our internal validation, which is half the size of the dataset used in LXMERT (Tan & Bansal, 2019), due to the ﬁltering of overlapping images and the use of image-text pairs only.

Original: We also use additional
Revised: We also use additional

Original: Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training 7 .
Revised: Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training 9 .

Original: The statistics on the cleaned splits are provided in Table 1.
Revised: The statistics on the cleaned splits are provided in Table 1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate UNITER on six V+L tasks (listed in Table 2), by transferring the pre-trained model to each target task and ﬁnetuning through end-to-end training.
Revised: We evaluate UNITER on six V+L tasks (listed in Table 2), by transferring the pre-trained model to each target task and ﬁnetuning through end-to-end training.

Original: We report experimental results on two model sizes: UNITER-base with 12 layers and UNITER-large with 24 layers 8 . 
Revised: We report experimental results on two model sizes: UNITER-base with 12 layers and UNITER-large with 24 layers 10 . [Table] [Table] [Table]

Original: 4. D OWNSTREAM T ASKS
Revised: D OWNSTREAM T ASKS

Original: In VQA, VCR and NLVR 2 tasks, given an input image (or a pair of images) and a natural language question (or description), the model predicts an answer (or judges the correctness of the description) based on the visual content in the image.
Revised: In VQA, VCR and NLVR 2 tasks, given an input image (or a pair of images) and a natural language question (or description), the model predicts an answer (or judges the correctness of the description) based on the visual content in the image.

Original: For Visual Entailment, we evaluate on the SNLI-VE dataset.
Revised: For Visual Entailment, we evaluate on the SNLI-VE dataset.

Original: The goal is to predict whether a given image semantically entails an input sentence.
Revised: The goal is to predict whether a given image semantically entails an input sentence.

Original: Classiﬁcation accuracy over three classes (“Entailment”, “Neutral” and “Contradiction”) is used to measure model performance.
Revised: Classiﬁcation accuracy over three classes (“Entailment”, “Neutral” and “Contradiction”) is used to measure model performance.

Original: For Image-Text Retrieval, we consider two datasets (COCO and Flickr30K) and evaluate the model in two settings: Image Retrieval (IR) and Text Retrieval (TR).
Revised: For Image-Text Retrieval, we consider two datasets (COCO and Flickr30K) and evaluate the model in two settings: Image Retrieval (IR) and Text Retrieval (TR).

Original: Referring
Revised: Referring

Original: Expression (RE) Comprehension requires the model to select the target from a set of image region proposals given the query description.
Revised: Expression (RE) Comprehension requires the model to select the target from a set of image region proposals given the query description.

Original: Models are evaluated on both ground-truth objects and detected proposals 9 (MAttNet (Yu et al., 2018)).We [Table]
Revised: Models are evaluated on both ground-truth objects and detected proposals 11 (MAttNet (Yu et al., 2018)).

Original: The evaluation splits of RE comprehension using detected proposals are denoted as val d , test d , etc.

Original: [Table] For VQA, VCR, NLVR 2 , Visual Entailment and Image-Text Retrieval, we extract the joint embedding of the input image-text pairs via a multi-layer perceptron (MLP) from the representation of the [CLS] token.
Revised: For VQA, VCR, NLVR 2 , Visual Entailment and Image-Text Retrieval, we extract the joint embedding of the input image-text pairs via a multi-layer perceptron (MLP) from the representation of the [CLS] token.

Original: For RE Comprehension, we use the MLP to compute the region-wise alignment scores.
Revised: For RE Comprehension, we use the MLP to compute the region-wise alignment scores.

Original: These MLP layers are learned during the ﬁnetuning stage.
Revised: These MLP layers are learned during the ﬁnetuning stage.

Original: Speciﬁcally, we formulate VQA,
Revised: Speciﬁcally, we formulate VQA,

Original: VCR, NLVR 2 , Visual Entailment and RE Comprehension as classiﬁcation problems and minimize the cross-entropy loss over the ground-truth answers/responses.
Revised: VCR, NLVR 2 , Visual Entailment and RE Comprehension as classiﬁcation problems and minimize the cross-entropy loss over the ground-truth answers/responses.

Original: For Image-Text Retrieval, we formulate it as a ranking problem.
Revised: For Image-Text Retrieval, we formulate it as a ranking problem.

Original: During ﬁnetuning, we sample three pairs of image and text, one positive pair from the dataset and two negative pairs by randomly replacing its sentence/image with others.
Revised: During ﬁnetuning, we sample three pairs of image and text, one positive pair from the dataset and two negative pairs by randomly replacing its sentence/image with others.

Original: We compute the similarity scores (based on the joint embedding) for both positive and negative pairs, and maximize the margin between them through triplet loss.
Revised: We compute the similarity scores (based on the joint embedding) for both positive and negative pairs, and maximize the margin between them through triplet loss.

Original: 4. E VALUATION ON P RE - TRAINING T ASKS
Revised: 4. E VALUATION ON P RE - TRAINING T ASKS

Original: We analyze the effectiveness of different pre-training settings through ablation studies over VQA,
Revised: We analyze the effectiveness of different pre-training settings through ablation studies over VQA,

Original: NLVR 2 , Flickr30K and RefCOCO+ as representative V+L benchmarks.
Revised: NLVR 2 , Flickr30K and RefCOCO+ as representative V+L benchmarks.

Original: In addition to standard metrics for each benchmark (listed in Table 2) , we also use Meta-Sum (sum of all the scores across all the benchmarks) as a global metric.
Revised: In addition to standard metrics for each benchmark (listed in Table 2) , we also use Meta-Sum (sum of all the scores across all the benchmarks) as a global metric.

Original: Firstly, we establish two baselines: Line 1 (L1) in Table 3 indicates no pre-training is involved, and L2 shows the results from MLM initialized with pre-trained weights from Devlin et al.
Revised: Firstly, we establish two baselines: Line 1 (L1) in Table 3 indicates no pre-training is involved, and L2 shows the results from MLM initialized with pre-trained weights from Devlin et al.

Original: Although MLM trained on text only did not absorb any image information during pre-training, we see a gain of approximately +30 on Meta-Sum over L1.
Revised: Although MLM trained on text only did not absorb any image information during pre-training, we see a gain of approximately +30 on Meta-Sum over L1.

Original: Hence, we use the pre-trained weights in
Revised: Hence, we use the pre-trained weights in

Original: L2 to initialize our model for the following experiments. 
Revised: L2 to initialize our model for the following experiments. [Table]

Original: Secondly, we validate the effectiveness of each pre-training task through a thorough ablation study.
Revised: Secondly [Table] we validate the effectiveness of each pre-training task through a thorough ablation study [Table]

Original: Comparing L2 and L3, MRFR (L3) achieves better results than MLM (L2) only on NLVR 2 .
Revised: Comparing L2 and L [Table] MRFR (L3) achieves better results than MLM (L2) only on NLVR 2 .

Original: On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a signiﬁcant improvement across all the tasks over L1 and L2 baselines.
Revised: On the other hand, when pre-trained on ITM (L4) or MLM (L5) only, we observe a signiﬁcant improvement across all the tasks over L1 and L2 baselines.

Original: When combining different pre-training tasks,
Revised: When combining different pre-training tasks,

Original: MLM + ITM (L6) improves over single ITM (L4) or MLM (L5).
Revised: MLM + ITM (L6) improves over single ITM (L4) or MLM (L5).

Original: When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks.
Revised: When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks.

Original: Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97).
Revised: Among the three variants of MRM (L7-L9), we observe that MRC-kl (L9) achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97).

Original: When combining MRC-kl and MRFR together with MLM and ITM (L10), we ﬁnd that they are complimentary to each other, which leads to the highest Meta-Sum score.
Revised: When combining MRC-kl and MRFR together with MLM and ITM (L10), we ﬁnd that they are complimentary to each other, which leads to the highest Meta-Sum score.

Original: We use this as the optimal pre-training setting for further experiments.
Revised: We use this as the optimal pre-training setting for further experiments.

Original: Additionally, we validate the contributions of conditional masking through a comparison study.
Revised: Additionally, we validate the contributions of conditional masking through a comparison study.

Original: When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that[Table] with conditional masking (399.97).
Revised: When we perform random masking on both modalities simultaneously during pre-training, i.e., w/o conditional masking (L11), we observe a decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97).

Original: This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.
Revised: This indicates that the conditional masking strategy enables the model to learn better joint image-text representations effectively.

Original: Lastly [Table] we study the effects of pre-training datasets.
Revised: Lastly, we study the effects of pre-training datasets.

Original: Our experiments so far have been focused on
Revised: Our experiments so far have been focused on

Original: In-domain data.
Revised: In-domain data.

Original: In this study [Table] we pre-train our model on Out-of-domain data (Conceptual Captions + SBU Captions).
Revised: In this study, we pre-train our model on Out-of-domain data (Conceptual Captions + SBU Captions).

Original: A performance drop (395.45 in L12) from the model trained on In-domain data (COCO + Visual Genome) (399.97 in L10) shows that although Out-of-domain data contain more images, the model still beneﬁts more from being exposed to similar downstream images during pretraining.
Revised: A performance drop (395.45 in L12) from the model trained on In-domain data (COCO + Visual Genome) (399.97 in L10) shows that although Out-of-domain data contain more images, the model still beneﬁts more from being exposed to similar downstream images during pretraining.

Original: We further pre-train our model on both In-domain and Out-of-domain data.
Revised: We further pre-train our model on both In-domain and Out-of-domain data.

Original: With doubled data size, the model continues to improve (402.50 in L13).
Revised: With doubled data size, the model continues to improve (402.50 in L13).

Original: 4. R ESULTS ON D OWNSTREAM T ASKS
Revised: 4. R ESULTS ON D OWNSTREAM T ASKS

Original: Table 4 presents the results of UNITER on all downstream tasks.
Revised: Table 4 presents the results of UNITER on all downstream tasks.

Original: Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting:
Revised: Both our base and large models are pre-trained on In-domain+Out-of-domain datasets, with the optimal pre-training setting:

Original: MLM+ITM+MRC-kl+MRFR.
Revised: MLM+ITM+MRC-kl+MRFR.

Original: The implementation details of each task are provided in Appendix
Revised: The implementation details of each task are provided in Appendix

Original: A.2.
Revised: A.2.

Original: We compare with both task-speciﬁc models and concurrent pre-trained models on each downstream task.
Revised: We compare with both task-speciﬁc models and concurrent pre-trained models on each downstream task.

Original: SOTA task-speciﬁc models include: MCAN (Yu et al., 2019) for VQA, MaxEnt (Suhr et al., 2019) for NLVR 2 , B2T2 (Alberti et al., 2019) for VCR, SCAN (Lee et al., 2018) for Image- [Table] [Table] [Table]
Revised: SOTA task-speciﬁc models include: MCAN (Yu et al., 2019) for VQA, MaxEnt (Suhr et al., 2019) for NLVR 2 , B2T2 (Alberti et al., 2019) for VCR, SCAN (Lee et al., 2018) for ImageText Retrieval, EVE-Image (Xie et al., 2019) for SNLI-VE, and MAttNet for RE Comprehension (RefCOCO, RefCOCO+ and RefCOCOg) 12 .

Original: [Table]

Original: Text Retrieval, EVE-Image (Xie et al., 2019) for SNLI-VE, and MAttNet for RE Comprehension (RefCOCO, RefCOCO+ and RefCOCOg) 10 .

Original: Concurrent pre-trained models include: ViLBERT,
Revised: Concurrent pre-trained models include: ViLBERT,

Original: LXMERT, Unicoder-VL, VisualBERT and VLBERT.
Revised: LXMERT, Unicoder-VL, VisualBERT and VLBERT.

Original: Results show that our UNITER-large model achieves new state of the art across all the benchmarks.
Revised: Results show that our UNITER-large model achieves new state of the art across all the benchmarks.

Original: UNITER-base model also outperforms the others by a large margin across all tasks except VQA.
Revised: UNITER-base model also outperforms the others by a large margin across all tasks except VQA.

Original: Speciﬁcally, our UNITER-base model outperforms SOTA by approximately +2 .
Revised: Speciﬁcally, our UNITER-base model outperforms SOTA by approximately +2 .

Original: 8% for VCR on
Revised: 8% for VCR on

Original: Q → AR, +2 .
Revised: Q → AR, +2 .

Original: 5% for NLVR 2 , +7% for SNLI-VE, +4% on R@1 for Image-Text Retrieval ( +15% for zero-shot setting), and +2% for RE Comprehension.
Revised: 5% for NLVR 2 , +7% for SNLI-VE, +4% on R@1 for Image-Text Retrieval ( +15% for zero-shot setting), and +2% for RE Comprehension.

Original: Note that LXMERT pre-trains with downstream VQA (+VG+GQA) data, which may help adapt the model to VQA task.
Revised: Note that LXMERT pre-trains with downstream VQA (+VG+GQA) data, which may help adapt the model to VQA task.

Original: However, when evaluated on unseen tasks such as NLVR 2 , UNITER-base achieves 3% gain over LXMERT.
Revised: However, when evaluated on unseen tasks such as NLVR 2 , UNITER-base achieves 3% gain over LXMERT.

Original: In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by > 1 . 5% on VQA.
Revised: In addition, among all the models pre-trained on image-text pairs only, our UNITER-base outperforms the others by > 1 . 5% on VQA.

Original: It is also worth mentioning that both VilBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, VilBERT: 221M) 11 .
Revised: It is also worth mentioning that both VilBERT and LXMERT observed two-stream model outperforms single-stream model, while our results show empirically that with our pre-training setting, single-stream model can achieve new state-of-the-art results, with much fewer parameters (UNITER-base: 86M, LXMERT: 183M, VilBERT: 221M) 13 .

Original: For VCR, we propose a two-stage pre-training approach: ( i ) pre-train on standard pre-training datasets; and then ( ii ) pre-train on downstream VCR dataset.
Revised: For VCR, we propose a two-stage pre-training approach: ( i ) pre-train on standard pre-training datasets; and then ( ii ) pre-train on downstream VCR dataset.

Original: Interestingly, while VLBERT and
Revised: Interestingly, while VLBERT and

Original: B2T2 observed that pre-training is not very helpful on VCR, we ﬁnd that the second-stage pretraining can signiﬁcantly boost model performance, while the ﬁrst-stage pre-training still helps but with limited effects (results shown in Table 5).
Revised: B2T2 observed that pre-training is not very helpful on VCR, we ﬁnd that the second-stage pretraining can signiﬁcantly boost model performance, while the ﬁrst-stage pre-training still helps but with limited effects (results shown in Table 5).

Original: This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.
Revised: This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre-training datasets.

Original: Different from other tasks, NLVR 2 takes two images as input.
Revised: Different from other tasks, NLVR 2 takes two images as input.

Original: Thus, directly ﬁnetuning UNITER pre-trained with image-sentence pairs might not lead to optimal performance, as the interactions between paired images are not learned during the pre-training stage.
Revised: Thus, directly ﬁnetuning UNITER pre-trained with image-sentence pairs might not lead to optimal performance, as the interactions between paired images are not learned during the pre-training stage.

Original: Thus, we experimented with three modiﬁed settings on NLVR 2 : ( i ) Triplet : joint embedding of images pairs and query captions; ( ii ) Pair : individual embedding of each image and each query caption; and ( iii ) Pair-biattn : a bidirectional attention is added to the Pair model to learn the interactions between the paired images.
Revised: Thus, we experimented with three modiﬁed settings on NLVR 2 : ( i ) Triplet : joint embedding of images pairs and query captions; ( ii ) Pair : individual embedding of each image and each query caption; and ( iii ) Pair-biattn : a bidirectional attention is added to the Pair model to learn the interactions between the paired images.

Original: Comparison results are presented in Table 6.
Revised: Comparison results are presented in Table 6.

Original: The Pair setting achieves better performance than the
Revised: The Pair setting achieves better performance than the

Original: Triplet setting even without cross-attention between the image pairs.
Revised: Triplet setting even without cross-attention between the image pairs.

Original: We hypothesize that it is due to the fact that our UNITER is pre-trained with image-text pairs.
Revised: We hypothesize that it is due to the fact that our UNITER is pre-trained with image-text pairs.

Original: Thus, it is difﬁcult to ﬁnetune a pairbased pre-trained model on triplet input.
Revised: Thus, it is difﬁcult to ﬁnetune a pairbased pre-trained model on triplet input.

Original: The bidirectional attention mechanism in the Pair-biattn setting, however, compensates the lack of cross-attention between images, hence yielding the best performance with a large margin.
Revised: The bidirectional attention mechanism in the Pair-biattn setting, however, compensates the lack of cross-attention between images, hence yielding the best performance with a large margin.

Original: This show that with minimal surgery on the top layer of UNITER, our pre-trained model can adapt to new tasks that are very different from pre-training tasks. 
Revised: This show that with minimal surgery on the top layer of UNITER, our pre-trained model can adapt to new tasks that are very different from pre-training tasks. [Table] [Table] [Table]

Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present UNITER, a large-scale pre-trained model providing UNiversal ImageTExt Representations for Vision-and-Language tasks.
Revised: In this paper we present UNITER, a large-scale pre-trained model providing UNiversal ImageTExt Representations for Vision-and-Language tasks.

Original: Three main pre-training tasks are proposed and evaluated through extensive ablation studies.
Revised: Three main pre-training tasks are proposed and evaluated through extensive ablation studies.

Original: Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a signiﬁcant margin.
Revised: Trained with both in-domain and out-of-domain datasets, UNITER outperforms state-of-the-art models over multiple V+L tasks by a signiﬁcant margin.

Original: Future work includes studying early interaction between raw image pixels and sentence tokens, as well as developing more effective pre-training tasks.
Revised: Future work includes studying early interaction between raw image pixels and sentence tokens, as well as developing more effective pre-training tasks

Original: A A PPENDIX A. D ATASET C OLLECTION
Revised: A. D ATASET C OLLECTION

Original: As introduced, our full dataset is composed of four existing V+L datasets: COCO, Visual Genome,
Revised: As introduced, our full dataset is composed of four existing V+L datasets: COCO, Visual Genome,

Original: Conceptual Captions, and SBU Captions.
Revised: Conceptual Captions, and SBU Captions.

Original: The dataset collection is not simply combining them, as we need to make sure none of the downstream evaluation images are seen during pre-training.
Revised: The dataset collection is not simply combining them, as we need to make sure none of the downstream evaluation images are seen during pre-training.

Original: Among them, COCO is the most tricky one to clean, as several downstream tasks are built based on it.
Revised: Among them, COCO is the most tricky one to clean, as several downstream tasks are built based on it.

Original: Figure 2 lists the splits from VQA, Image-Text Retrieval, COCO Captioning, RefCOCO/RefCOCO+/RefCOCOg, and the bottom-up top-down (BUTD) detection (Anderson et al., 2018), all from COCO images.
Revised: Figure 2 lists the splits from VQA, Image-Text Retrieval, COCO Captioning, RefCOCO/RefCOCO+/RefCOCOg, and the bottom-up top-down (BUTD) detection (Anderson et al., 2018), all from COCO images.

Original: As observed, the validation and test splits of different tasks are scattered across the raw COCO splits.
Revised: As observed, the validation and test splits of different tasks are scattered across the raw COCO splits.

Original: Therefore, we exclude all those evaluation images that appeared in the downstream tasks.
Revised: Therefore, we exclude all those evaluation images that appeared in the downstream tasks.

Original: In addition, we also exclude all co-occurring Flickr30K images via URL matching, making sure the zero-shot image-text retrieval evaluation on Flickr is fair.
Revised: In addition, we also exclude all co-occurring Flickr30K images via URL matching, making sure the zero-shot image-text retrieval evaluation on Flickr is fair.

Original: The remaining images become the COCO subset within our full dataset, as shown in Figure 2 bottom row.
Revised: The remaining images become the COCO subset within our full dataset, as shown in Figure 2 bottom row.

Original: We apply the same rules to Visual Genome, Conceptual Captions, and SBU Captions.
Revised: We apply the same rules to Visual Genome, Conceptual Captions, and SBU Captions.

Original: [Figure]
Revised: [Figure]

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: Our models are implemented based on PyTorch 12 (Paszke et al., 2017).
Revised: Our models are implemented based on PyTorch 14 (Paszke et al., 2017).

Original: To speed up training, we use Nvidia Apex 13 for mixed precision training.
Revised: To speed up training, we use Nvidia Apex 15 for mixed precision training.

Original: All pre-training experiments are run on Nvidia
Revised: All pre-training experiments are run on Nvidia

Original: V100 GPUs (16GB VRAM; PCIe connection).
Revised: V100 GPUs (16GB VRAM; PCIe connection).

Original: Finetuning experiments are implemented on the same hardware or Titan RTX GPUs (24GB VRAM).
Revised: Finetuning experiments are implemented on the same hardware or Titan RTX GPUs (24GB VRAM).

Original: To further speed up training, we implement dynamic sequence length to reduce padding and batch examples by number of input units (text tokens + image regions).
Revised: To further speed up training, we implement dynamic sequence length to reduce padding and batch examples by number of input units (text tokens + image regions).

Original: For large pre-training experiments, we use Horovod 14 + NCCL 15 for multi-node communications (on TCP connections through ethernet) with up to 4 nodes of 4x V server.
Revised: For large pre-training experiments, we use Horovod 16 + NCCL 17 for multi-node communications (on TCP connections through ethernet) with up to 4 nodes of 4x V server.

Original: Gradient accumulation (Ott et al., 2018) is also applied to reduce multi-GPU communication overheads.
Revised: Gradient accumulation (Ott et al., 2018) is also applied to reduce multi-GPU communication overheads.

Original: Visual Question Answering (VQA)
Revised: Visual Question Answering (VQA)

Original: We follow Yu et al.
Revised: We follow Yu et al.

Original: (2019) to take 3129 most frequent answers as answer candidates, and assign a soft target score to each candidate based on its relevancy to the 10 human responses.
Revised: (2019) to take 3129 most frequent answers as answer candidates, and assign a soft target score to each candidate based on its relevancy to the 10 human responses.

Original: To ﬁnetune on VQA dataset, we use a binary cross-entropy loss to train a multi-label classiﬁer using batch size of 10240 input units over maximum 5 K steps.
Revised: To ﬁnetune on VQA dataset, we use a binary cross-entropy loss to train a multi-label classiﬁer using batch size of 10240 input units over maximum 5 K steps.

Original: We use
Revised: We use

Original: AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate of 3 e − 4 and weight decay of 0 . 01 .
Revised: AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate of 3 e − 4 and weight decay of 0 . 01 .

Original: At inference time, the max-probable answer is selected as the predicted answer.
Revised: At inference time, the max-probable answer is selected as the predicted answer.

Original: For results on test-dev and test-std splits, both training and validation sets are used for training, and additional question-answer pairs from Visual Genome are used for data augmentation as in Yu et al.https org https
Revised: For results on test-dev and test-std splits, both training and validation sets are used for training, and additional question-answer pairs from Visual Genome are used for data augmentation as in Yu et al.

Original: com/

Original: https

Original: horovod horovod https Visual Commonsense Reasoning (VCR) VCR can be decomposed into two multiple-choice subtasks: question-answering task (Q → A) and answer-justiﬁcation task (QA → R).
Revised: Visual Commonsense Reasoning (VCR) VCR can be decomposed into two multiple-choice subtasks: question-answering task (Q → A) and answer-justiﬁcation task (QA → R).

Original: In the holistic setting (Q → AR), a model needs to ﬁrst choose an answer from the answer choices, then select a supporting rationale from rationale choices if the chosen answer is correct.
Revised: In the holistic setting (Q → AR), a model needs to ﬁrst choose an answer from the answer choices, then select a supporting rationale from rationale choices if the chosen answer is correct.

Original: We train our model in two settings simultaneously.
Revised: We train our model in two settings simultaneously.

Original: When testing in the holistic setting, we ﬁrst apply the model to predict an answer, then obtain the rationale from the same model based on the given question and the predicted answer.
Revised: When testing in the holistic setting, we ﬁrst apply the model to predict an answer, then obtain the rationale from the same model based on the given question and the predicted answer.

Original: To ﬁnetune on VCR dataset, we concatenate the question (the qeustion and the ground truth answer) and each answer (rationale) choice from the four possible answer (rationale) candidates.
Revised: To ﬁnetune on VCR dataset, we concatenate the question (the qeustion and the ground truth answer) and each answer (rationale) choice from the four possible answer (rationale) candidates.

Original: The ‘modality embedding’ is extended to help distinguish question, answer and rationale.
Revised: The ‘modality embedding’ is extended to help distinguish question, answer and rationale.

Original: Crossentropy loss is used to train a classiﬁer over two classes ( ‘‘right’’ or ‘‘wrong’’ ) for each question-answer pair (question-answer-rationale triplet) with a batch size of 4096 input units over maximum 5 K steps.
Revised: Crossentropy loss is used to train a classiﬁer over two classes ( ‘‘right’’ or ‘‘wrong’’ ) for each question-answer pair (question-answer-rationale triplet) with a batch size of 4096 input units over maximum 5 K steps.

Original: We use AdamW optimizer with a learning rate of 1 e − 4 and weight decay of 0 . 01 .
Revised: We use AdamW optimizer with a learning rate of 1 e − 4 and weight decay of 0 . 01 .

Original: Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks.
Revised: Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kl as the pre-training tasks.

Original: ITM is discarded because the text in VCR does not explicitly describe the image.
Revised: ITM is discarded because the text in VCR does not explicitly describe the image.

Original: The results of both pretrainings on VCR are reported in Table 5 and discussed in the main text.
Revised: The results of both pretrainings on VCR are reported in Table 5 and discussed in the main text.

Original: In conclusion, for downstream tasks that contain new data which is very different from the pre-training datasets, secondstage pre-training helps further boost the performance.
Revised: In conclusion, for downstream tasks that contain new data which is very different from the pre-training datasets, secondstage pre-training helps further boost the performance.

Original: In our implementation, the second-stage pre-training is implemented with a batch size of 4096 intput units, a learning rate of 3 e − 4 and a weight decay of 0 .
Revised: In our implementation, the second-stage pre-training is implemented with a batch size of 4096 intput units, a learning rate of 3 e − 4 and a weight decay of 0 .

Original: 01 over maximum 60 K steps.
Revised: 01 over maximum 60 K steps.

Original: After secondstage pre-traing, we ﬁnetune our model with a learning rate of 6 e − 5 over maximum 8 K steps.
Revised: After secondstage pre-traing, we ﬁnetune our model with a learning rate of 6 e − 5 over maximum 8 K steps.

Original: Natural Language for Visual Reasoning for Real (NLVR 2 ) NLVR 2 is a new challenging task for visual reasoning.
Revised: Natural Language for Visual Reasoning for Real (NLVR 2 ) NLVR 2 is a new challenging task for visual reasoning.

Original: The goal is to determine whether a natural language statement is true about the given image pair.
Revised: The goal is to determine whether a natural language statement is true about the given image pair.

Original: Here we discuss the three architecture variants of NLVR 2 ﬁnetuning in detail.
Revised: Here we discuss the three architecture variants of NLVR 2 ﬁnetuning in detail.

Original: Since UNITER only handles one image and one text input at pre-training, the ‘modality embedding’ is extended to help distinguish the additional image presented in the NLVR 2 task.
Revised: Since UNITER only handles one image and one text input at pre-training, the ‘modality embedding’ is extended to help distinguish the additional image presented in the NLVR 2 task.

Original: For the Triplet setup, we concatenate the image regions and then feed into the UNITER model.
Revised: For the Triplet setup, we concatenate the image regions and then feed into the UNITER model.

Original: An MLP transform is applied on the [CLS] output for binary classiﬁcation.
Revised: An MLP transform is applied on the [CLS] output for binary classiﬁcation.

Original: For the Pair setup, we treat one input example as two text-image pairs by repeating the text.
Revised: For the Pair setup, we treat one input example as two text-image pairs by repeating the text.

Original: The two [CLS] outputs from UNITER are then depth concatenated as the joint embedding for the example.
Revised: The two [CLS] outputs from UNITER are then depth concatenated as the joint embedding for the example.

Original: Another MLP further transform this embedding for the ﬁnal classiﬁcation.
Revised: Another MLP further transform this embedding for the ﬁnal classiﬁcation.

Original: For the Pair-biattn setup, the input format is the same as the Pair setup.
Revised: For the Pair-biattn setup, the input format is the same as the Pair setup.

Original: As for the joint representation, instead of rely on only two [CLS] outputs, we apply a multi-head attention layer (Vaswani et al., 2017) on one sequence of joint image-text embeddings to attend to the other sequence of embeddings, and vice versa.
Revised: As for the joint representation, instead of rely on only two [CLS] outputs, we apply a multi-head attention layer (Vaswani et al., 2017) on one sequence of joint image-text embeddings to attend to the other sequence of embeddings, and vice versa.

Original: After this ‘bidirectional’ attention interactions, a simple attentional pooling is applied on each output sequences and then a ﬁnal concat+MLP layer transforms the cross-attended joint representation for true/false classiﬁcation.
Revised: After this ‘bidirectional’ attention interactions, a simple attentional pooling is applied on each output sequences and then a ﬁnal concat+MLP layer transforms the cross-attended joint representation for true/false classiﬁcation.

Original: We ﬁnetune UNITER on NLVR 2 for 8K steps with a batch size of 10K input units.
Revised: We ﬁnetune UNITER on NLVR 2 for 8K steps with a batch size of 10K input units.

Original: AdamW optimizer is used with learning rate of 1 e − 4 and weight decay of 0 .
Revised: AdamW optimizer is used with learning rate of 1 e − 4 and weight decay of 0 .

Original: Image-Text Retrieval Two datasets are considered for this task: COCO and Flickr30K. COCO consists of 123 K images, each accompanied with ﬁve human-written captions.
Revised: Image-Text Retrieval Two datasets are considered for this task: COCO and Flickr30K. COCO consists of 123 K images, each accompanied with ﬁve human-written captions.

Original: We follow Karpathy & Fei-Fei (2015) to split the data into 82K/5K/5K training/validation/test images.
Revised: We follow Karpathy & Fei-Fei (2015) to split the data into 82K/5K/5K training/validation/test images.

Original: Additional 30K images from MSCOCO validation set are also included to improve training as in Lee et al.
Revised: Additional 30K images from MSCOCO validation set are also included to improve training as in Lee et al.

Original: Flickr30K dataset contains 31K images collected from the Flickr website, with ﬁve textual descriptions per image.
Revised: Flickr30K dataset contains 31K images collected from the Flickr website, with ﬁve textual descriptions per image.

Original: We follow Karpathy & Fei-Fei (2015) to split the data into 30K/1K/1K training/validation/test splits.
Revised: We follow Karpathy & Fei-Fei (2015) to split the data into 30K/1K/1K training/validation/test splits.

Original: During ﬁnetuning, we sample two negative image-text pairs per positive sample from image and text sides, respectively.
Revised: During ﬁnetuning, we sample two negative image-text pairs per positive sample from image and text sides, respectively.

Original: For COCO, we use batch size of 60 examples, learning rate of 2 e − 5 and ﬁnetune our model for 20 K steps.
Revised: For COCO, we use batch size of 60 examples, learning rate of 2 e − 5 and ﬁnetune our model for 20 K steps.

Original: For Flickr30K, we ﬁnetune our model with a batch size of 120 examples and a learning rate of 5 e − 5 over maximum 16 K steps.
Revised: For Flickr30K, we ﬁnetune our model with a batch size of 120 examples and a learning rate of 5 e − 5 over maximum 16 K steps.

Original: To obtain the ﬁnal results in Table 4, we further sample hard negatives to facilitate the ﬁnetuning.
Revised: To obtain the ﬁnal results in Table 4, we further sample hard negatives to facilitate the ﬁnetuning.

Original: For every N steps, we randomly sample 128 negative images per text input and obtain a sparse scoring matrix for the whole training set.
Revised: For every N steps, we randomly sample 128 negative images per text input and obtain a sparse scoring matrix for the whole training set.

Original: For each image, we choose the top 20 ranked negative sentences as hard negative samples.
Revised: For each image, we choose the top 20 ranked negative sentences as hard negative samples.

Original: Similarly, we get 20 hard negative images for each sentence according to their scores.
Revised: Similarly, we get 20 hard negative images for each sentence according to their scores.

Original: The hard negatives are sent to the model as additional negative samples.
Revised: The hard negatives are sent to the model as additional negative samples.

Revised: [Figure]

Original: In the end, we have two randomly sampled negatives and two hard negative samples per positive sample.
Revised: In the end, we have two randomly sampled negatives and two hard negative samples per positive sample.

Original: N is set to 4000 for COCO and 2500 for Flickr30K.
Revised: N is set to 4000 for COCO and 2500 for Flickr30K.

Original: Visual Entailment (SNLI-VE)
Revised: Visual Entailment (SNLI-VE)

Original: Visual Entailment is a task derived from Flickr30K images and
Revised: Visual Entailment is a task derived from Flickr30K images and

Original: Stanford Natural Language Inference (SNLI) dataset, where the goal is to determine the logical relationship between a natural language statement and an image.
Revised: Stanford Natural Language Inference (SNLI) dataset, where the goal is to determine the logical relationship between a natural language statement and an image.

Original: Similar to BERT for Natural
Revised: Similar to BERT for Natural

Original: Language Inference (NLI), we treat SNLI-VE as a three-way classiﬁcation problem and apply an
Revised: Language Inference (NLI), we treat SNLI-VE as a three-way classiﬁcation problem and apply an

Original: MLP Transform on [CLS] output.
Revised: MLP Transform on [CLS] output.

Original: The UNITER model is ﬁnetuned using cross-entropy loss.
Revised: The UNITER model is ﬁnetuned using cross-entropy loss.

Original: The batch size is set to 10K input units and we use AdamW with learning rate of 8 e − 5 to train for 3K steps.
Revised: The batch size is set to 10K input units and we use AdamW with learning rate of 8 e − 5 to train for 3K steps.

Original:  Comprehension We use three referring expression datasets: RefCOCO,
Revised: Referring Expression Comprehension We use three referring expression datasets: RefCOCO,

Original: RefCOCO+, and RefCOCOg for the evaluation, all collected on COCO images.
Revised: RefCOCO+, and RefCOCOg for the evaluation, all collected on COCO images.

Original: To ﬁnetune
Revised: To ﬁnetune

Original: UNITER on this task, we add a MLP layer on top of the region outputs from Transformer, to compute the alignment score between the query phrase/sentence and each region.
Revised: UNITER on this task, we add a MLP layer on top of the region outputs from Transformer, to compute the alignment score between the query phrase/sentence and each region.

Original: Since only one object is paired with the query phrase/sentence, we apply cross-entropy loss on the normalized alignment scores.
Revised: Since only one object is paired with the query phrase/sentence, we apply cross-entropy loss on the normalized alignment scores.

Original: The ﬁnetuning is efﬁcient - we train the model with a batch size of 64 examples and a learning rate of 5 e − 5 for only 5 epochs, and achieve state-of-the-art performance.
Revised: The ﬁnetuning is efﬁcient - we train the model with a batch size of 64 examples and a learning rate of 5 e − 5 for only 5 epochs, and achieve state-of-the-art performance.

Original: Note all works including ours use off-the-shelf object detectors trained on COCO (and Visual
Revised: Note all works including ours use off-the-shelf object detectors trained on COCO (and Visual

Original: Genome) to extract the visual features.
Revised: Genome) to extract the visual features.

Original: While this does not affect other downstream tasks, it raises an issue for RE comprehension, as the val/test images of RefCOCO, RefCOCO+, and RefCOCOg are a subset of COCO’s training split.
Revised: While this does not affect other downstream tasks, it raises an issue for RE comprehension, as the val/test images of RefCOCO, RefCOCO+, and RefCOCOg are a subset of COCO’s training split.

Original: Strictly, our object detector is not allowed to train with these val/test images.
Revised: Strictly, our object detector is not allowed to train with these val/test images.

Original: However, just for a “fair” comparison with concurrent works, we ignore this issue and use the same features (Anderson et al., 2018) as the others.
Revised: However, just for a “fair” comparison with concurrent works, we ignore this issue and use the same features (Anderson et al., 2018) as the others.

Original: We also update the results of
Revised: We also update the results of

Original: MAttNet using this ”contaminated” features, whose accuracy is 1.5% higher than the original one.
Revised: MAttNet using this ”contaminated” features, whose accuracy is 1.5% higher than the original one.

Original: As aforementioned, the interaction between sentence and image could start from tokens and pixels instead of the extracted features.
Revised: As aforementioned, the interaction between sentence and image could start from tokens and pixels instead of the extracted features.

Original: We leave this study and RE comprehension with strictly correct features to future work. 
Revised: We leave this study and RE comprehension with strictly correct features to future work. A. V ISUALIZATION

Revised: Similar to Kovaleva et al.

Revised: (2019), we observe several patterns in the attention maps of the UNITER model as shown in Fig. 3.

Revised: Note that different from Kovaleva et al.

Revised: (2019), our attention mechanism operates in both inter- and intra-modalitiy fashions.

Revised: For completeness, we brieﬂy discuss each pattern here:

Revised: • Vertical : attention to special tokens [CLS] or [SEP] ; • Diagonal : attention to the token/region itself or preceding/following tokens/regions; • Vertical + Diagonal : mixture of vertical and diagonal; • Block : intra-modality attention,

Revised: i

Revised: e. , textual self-attention and visual self-attention; • Heterogeneous : scattered attention that cannot be categorized and is highly dependent on actual input; • Reversed Block : inter-modality attention, i.e. , text-to-image and image-to-text attention.

Revised: Note that Reversed Block (Fig. 3f) shows cross-modality alignment of tokens and regions.

Revised: In

Revised: Fig. 4, 5, and 6, we visualize several examples of text-to-image attention to demonstrate the local cross-modality alignment between regions and tokens.

Revised: [Figure]

Revised: [Figure]

Revised: A. C ONDITIONAL M ASKING VS . J OINT R ANDOM M ASKING

Revised: We further discuss the advantage of using our proposed conditional masking over joint random masking used in (Tan & Bansal, 2019; Lu et al., 2019).

Revised: Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities.

Revised: Fig.

Revised: 7 shows an example of an image of “man with his dog and cat sitting on a sofa”.

Revised: For our conditional masking, when the region of dog is masked, our model should be able to infer that region is dog based on the context of both the surrounding regions and the full sentence as in Fig. 7(a), and vice versa.

Revised: However, for the

Revised: [Table]

Revised: [Table]

Revised: joint masking implementation, it could happen when both the region of dog and the word of dog are masked as in Fig. 7(b).

Revised: Then the model has to predict blindly, which might lead to miss-alignment.

Revised: To verify this intuition, we show the validation curves during pre-training of MLM and MRC-kl in Fig. 8.

Revised: Each sub-ﬁgure shows a comparison of applying conditional masking and joint random masking during the pre-training of UNITER.

Revised: The MLM accuracy measures how well our UNITER can reconstruct the masked words, and MRC-kl accuracy 18 measures how well our UNITER can classify the masked regions.

Revised: For both cases, as shown in Fig. 8, our conditional masking converges faster and achieves higher ﬁnal accuracy than the joint random masking.

Revised: In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on the ﬁne-tuned downstream tasks.

Revised: A. M ORE R ESULTS ON VCR AND NLVR

Revised: Following the VCR setup in Table.

Revised: 5, we further made an ensemble model using 10 UNITERlarge.

Revised: Table.

Revised: 7 shows the comparison between VLBERT, ViLBERT, and our UNITER on VCR.

Revised: The

Revised: Q → AR accuracy of our ensemble outperforms ViLBERT (Lu et al., 2019) ensemble by large margin of 7.0%.

Revised: Note even our single UNITER-large already outperforms ViLBERT ensemble and

Revised: VLBERT-large by 3.0%.

Revised: Besides, we also compare our UNITER-large with LXMERT (Tan & Bansal, 2019) and VisualBERT (Li et al., 2019b) on a additional testing split of NLVR 2 in Table.

Revised: Our results consistently outperform the previous SOTA on all metrics 19 by a large margin of ∼ 4.0%.