Original: H OW
Revised: H OW

Original: M UCH
Revised: M UCH

Original: C
Revised: C

Original: AN CLIP B ENEFIT V ISION - ANDL ANGUAGE
Revised: AN CLIP B ENEFIT V ISION - ANDL ANGUAGE

Original: T
Revised: T

Original: ASKS
Revised: ASKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world.
Revised: Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world.

Original: However, it has been observed that large-scale pre-training usually can result in better generalization performance, e.g.,
Revised: However, it has been observed that large-scale pre-training usually can result in better generalization performance, e.g.,

Original: CLIP (Contrastive Language-Image
Revised: CLIP (Contrastive Language-Image

Original: Pre-training)
Revised: Pre-training)

Original: , trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks.
Revised: , trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks.

Original: To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging
Revised: To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging

Original: CLIP into task-speciﬁc ﬁne-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks.
Revised: CLIP into task-speciﬁc ﬁne-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks.

Original: We show that CLIP signiﬁcantly outperforms widely-used visual encoders trained with in-domain annotated data, such as
Revised: We show that CLIP signiﬁcantly outperforms widely-used visual encoders trained with in-domain annotated data, such as

Original: BottomUp-TopDown.
Revised: BottomUp-TopDown.

Original: We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering,
Revised: We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering,

Original: Visual Entailment, and V&L Navigation tasks.
Revised: Visual Entailment, and V&L Navigation tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Vision-and-Language (V&L) tasks such as VQA (Antol et al., 2015) test a system’s ability to understand and reason about the semantics of the visual world with the help of natural language.
Revised: Vision-and-Language (V&L) tasks such as VQA (Antol et al., 2015) test a system’s ability to understand and reason about the semantics of the visual world with the help of natural language.

Original: Most
Revised: Most

Original: V&L models rely on a visual encoder to perceive the visual world, which translates the raw pixels into vectors from a representation space.
Revised: V&L models rely on a visual encoder to perceive the visual world, which translates the raw pixels into vectors from a representation space.

Original: Recent work (Anderson et al., 2018a; Jiang et al., 2020; Zhang et al., 2021) observes that the visual representation has become the performance bottleneck of V&L models and stress the importance of learning a powerful visual encoder.
Revised: Recent work (Anderson et al., 2018a; Jiang et al., 2020; Zhang et al., 2021) observes that the visual representation has become the performance bottleneck of V&L models and stress the importance of learning a powerful visual encoder.

Original: These high-performing visual encoders are trained on manually-annotated data with class labels (e.g., ImageNet)
Revised: These high-performing visual encoders are trained on manually-annotated data with class labels (e.g., ImageNet)

Original: (Russakovsky et al., 2015) or bounding boxes (e.g., Visual Genome) (Krishna et al., 2017).
Revised: (Russakovsky et al., 2015) or bounding boxes (e.g., Visual Genome) (Krishna et al., 2017).

Original: However, such detection or image classiﬁcation data is costly to collect, and the visual representation is limited by the predeﬁned class labels.
Revised: However, such detection or image classiﬁcation data is costly to collect, and the visual representation is limited by the predeﬁned class labels.

Original: Thus, there is a need for a visual encoder that is trained on more diverse and large-scale data sources, unbounded by a ﬁxed set of labels, and with generalization ability to unseen objects and concepts.
Revised: Thus, there is a need for a visual encoder that is trained on more diverse and large-scale data sources, unbounded by a ﬁxed set of labels, and with generalization ability to unseen objects and concepts.

Original: Recently, CLIP (Radford et al., 2021) has been proposed to learn visual concepts with language supervision.
Revised: Recently, CLIP (Radford et al., 2021) has been proposed to learn visual concepts with language supervision.

Original: CLIP consists of a visual encoder and a text encoder.
Revised: CLIP consists of a visual encoder and a text encoder.

Original: It is trained on 400M noisy imagetext pairs crawled from the Internet.
Revised: It is trained on 400M noisy imagetext pairs crawled from the Internet.

Original: Thus, the data collection process is scalable and requires little human annotation.
Revised: Thus, the data collection process is scalable and requires little human annotation.

Original: CLIP has shown strong zero-shot capabilities on benchmarks such as ImageNet classiﬁcation.
Revised: CLIP has shown strong zero-shot capabilities on benchmarks such as ImageNet classiﬁcation.

Original: We hypothesize that it also bears great potential for the V&L tasks.
Revised: We hypothesize that it also bears great potential for the V&L tasks.

Original: However, directly applying CLIP as a zero-shot model to V&L tasks proves to be difﬁcult (Section 5 and Kim et al.
Revised: However, directly applying CLIP as a zero-shot model to V&L tasks proves to be difﬁcult (Section 5 and Kim et al.

Original: (2021)), as many V&L tasks require complex multi-modal reasoning.
Revised: (2021)), as many V&L tasks require complex multi-modal reasoning.

Original: Thus, we propose to integrate
Revised: Thus, we propose to integrate

Original: CLIP with existing V&L models by replacing their visual encoder with CLIP’s visual encoder.
Revised: CLIP with existing V&L models by replacing their visual encoder with CLIP’s visual encoder.

Original: To the best of our knowledge, we present the ﬁrst large-scale empirical study on using CLIP as the visual encoder for diverse V&L tasks.
Revised: To the best of our knowledge, we present the ﬁrst large-scale empirical study on using CLIP as the visual encoder for diverse V&L tasks.

Original: We consider two typical scenarios: 1) we use CLIP in direct task-speciﬁc ﬁne-tuning (Section 3); 2) we integrate CLIP with V&L pre-training on image-text pairs
Revised: We consider two typical scenarios: 1) we use CLIP in direct task-speciﬁc ﬁne-tuning (Section 3); 2) we integrate CLIP with V&L pre-training on image-text pairs

Original: [Figure]
Revised: [Figure]

Original: and transfer to downstream tasks (Section 4).
Revised: and transfer to downstream tasks (Section 4).

Original: For clarity, we denote the models used in these two scenarios as CLIP-ViL (without V&L pre-training) and CLIP-ViL p (with V&L pre-training).
Revised: For clarity, we denote the models used in these two scenarios as CLIP-ViL (without V&L pre-training) and CLIP-ViL p (with V&L pre-training).

Original: In direct task-speciﬁc ﬁne-tuning , we consider three widely-adopted tasks: Visual Question Answering (Antol et al., 2015), Image Captioning (Chen et al., 2015), and Vision-and-Language
Revised: In direct task-speciﬁc ﬁne-tuning , we consider three widely-adopted tasks: Visual Question Answering (Antol et al., 2015), Image Captioning (Chen et al., 2015), and Vision-and-Language

Original: Navigation (Anderson et al., 2018b).
Revised: Navigation (Anderson et al., 2018b).

Original: On all three tasks, CLIP-ViL brings sizable improvement over strong baselines, 1.4% accuracy on VQA v2.0, 6.5 CIDEr on COCO Captioning, and 4.0% success rate on Room-to-Room navigation.
Revised: On all three tasks, CLIP-ViL brings sizable improvement over strong baselines, 1.4% accuracy on VQA v2.0, 6.5 CIDEr on COCO Captioning, and 4.0% success rate on Room-to-Room navigation.

Original: In V&L pre-training , we replace the conventionally used region-based representation (Anderson et al., 2018a) with CLIP.
Revised: In V&L pre-training , we replace the conventionally used region-based representation (Anderson et al., 2018a) with CLIP.

Original: CLIP-ViL p performs exceptionally well on three benchmarks, including VQA v2.0,
Revised: CLIP-ViL p performs exceptionally well on three benchmarks, including VQA v2.0,

Original: SNLI-VE (Xie et al., 2019), and GQA (Hudson and Manning, 2019), setting a new state-of-the-art (SotA) on VQA (76.70% on test-std), and SNLI-VE (80.20% on test).
Revised: SNLI-VE (Xie et al., 2019), and GQA (Hudson and Manning, 2019), setting a new state-of-the-art (SotA) on VQA (76.70% on test-std), and SNLI-VE (80.20% on test).

Original: CLIP-ViL p with CLIP-Res outperforms models based on the widely used region-based encoder, BottomUp-TopDown (BUTD)
Revised: CLIP-ViL p with CLIP-Res outperforms models based on the widely used region-based encoder, BottomUp-TopDown (BUTD)

Original: ResNet101 (Anderson et al., 2018a).
Revised: ResNet101 (Anderson et al., 2018a).

Original: Moreover, CLIP-ViL p with CLIP-Res50x4 surpasses VinVLResNeXt152 (Zhang et al., 2021), which is an extreme scale-up attempt of the region-based encoder with human-annotated data.
Revised: Moreover, CLIP-ViL p with CLIP-Res50x4 surpasses VinVLResNeXt152 (Zhang et al., 2021), which is an extreme scale-up attempt of the region-based encoder with human-annotated data.

Original:  B ACKGROUND AND M OTIVATION
Revised:  B ACKGROUND AND M OTIVATION

Original: Vision-and-Language (V&L) models.
Revised: Vision-and-Language (V&L) models.

Original: V&L tasks require a model to understand the visual world and to ground natural language to the visual observations.
Revised: V&L tasks require a model to understand the visual world and to ground natural language to the visual observations.

Original: Prominent tasks include visual question answering (Antol et al., 2015), image captioning (Chen et al., 2015), vision-language navigation (Anderson et al., 2018a), image-text retrieval (Wang et al., 2016) and so on.
Revised: Prominent tasks include visual question answering (Antol et al., 2015), image captioning (Chen et al., 2015), vision-language navigation (Anderson et al., 2018a), image-text retrieval (Wang et al., 2016) and so on.

Original: V&L models designed for these tasks often consist of a visual encoder, a text encoder, and a cross-modal interaction module (Kim et al., 2021).
Revised: V&L models designed for these tasks often consist of a visual encoder, a text encoder, and a cross-modal interaction module (Kim et al., 2021).

Original: We illustrate the three typical training stages in Figure 1: 1) the visual encoder is trained on annotated vision datasets (Russakovsky et al., 2015; Krishna et al., 2017) (denoted as visual encoder pretraining ); 2) (optionally) pre-training on paired image-caption data with a reconstructive objective and an image-text matching objective (denoted as vision-and-language pre-training )
Revised: We illustrate the three typical training stages in Figure 1: 1) the visual encoder is trained on annotated vision datasets (Russakovsky et al., 2015; Krishna et al., 2017) (denoted as visual encoder pretraining ); 2) (optionally) pre-training on paired image-caption data with a reconstructive objective and an image-text matching objective (denoted as vision-and-language pre-training )

Original: (Lu et al., 2019); 3) ﬁne-tuning on task-speciﬁc data (denoted as task-speciﬁc ﬁne-tuning ).
Revised: (Lu et al., 2019); 3) ﬁne-tuning on task-speciﬁc data (denoted as task-speciﬁc ﬁne-tuning ).

Original: Visual encoders in V&L models.
Revised: Visual encoders in V&L models.

Original: Different models employ different visual encoders, we illustrate their architectures and pre-training processes in Figure 2.
Revised: Different models employ different visual encoders, we illustrate their architectures and pre-training processes in Figure 2.

Original: The encoders can be categorized as follows:
Revised: The encoders can be categorized as follows:

Original: 1) region-based models such as BUTD object detector (Anderson et al., 2018a; Kamath et al., 2021);
Revised: 1) region-based models such as BUTD object detector (Anderson et al., 2018a; Kamath et al., 2021);

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: P r opo
Revised: P r opo

Original: [Figure]
Revised: [Figure]

Original: a l s
Revised: a l s

Original: [Figure]
Revised: [Figure]

Original: 2) grid-based models such as Jiang et al.
Revised: 2) grid-based models such as Jiang et al.

Original: (2020) that directly extract grid-like feature maps from the visual backbone (He et al., 2016; Dosovitskiy et al.,
Revised: (2020) that directly extract grid-like feature maps from the visual backbone (He et al., 2016; Dosovitskiy et al.,

Original: The encoder is ﬁrst pre-trained on human-annotated vision datasets.
Revised: The encoder is ﬁrst pre-trained on human-annotated vision datasets.

Original: Region-based encoders are pre-trained with detection data such as Visual Genome (Krishna et al., 2017).
Revised: Region-based encoders are pre-trained with detection data such as Visual Genome (Krishna et al., 2017).

Original: Grid-based encoders are pre-trained with image classiﬁcation data such as ImageNet (Russakovsky et al., 2015) or detection data (Jiang et al., 2020).
Revised: Grid-based encoders are pre-trained with image classiﬁcation data such as ImageNet (Russakovsky et al., 2015) or detection data (Jiang et al., 2020).

Original: However, these manually labeled datasets are expensive to construct and hard to scale up.
Revised: However, these manually labeled datasets are expensive to construct and hard to scale up.

Original: They only provide supervision for a limited number of predetermined visual concepts.
Revised: They only provide supervision for a limited number of predetermined visual concepts.

Original: This motivates us to use CLIP as the visual encoder.
Revised: This motivates us to use CLIP as the visual encoder.

Original: CLIP.
Revised: CLIP.

Original: CLIP (Contrastive Language-Image
Revised: CLIP (Contrastive Language-Image

Original: Pre-training)
Revised: Pre-training)

Original: (Radford et al., 2021) 3 falls into the line of research that learns visual representations from natural language supervision (Desai and Johnson, 2020;
Revised: (Radford et al., 2021) 3 falls into the line of research that learns visual representations from natural language supervision (Desai and Johnson, 2020;

Original: Sariyildiz et al., 2020; Jia et al., 2021).
Revised: Sariyildiz et al., 2020; Jia et al., 2021).

Original: CLIP follows a “shallow-interaction design”, where a visual encoder and a text encoder encode an input image and text independently, and the dot-product between the two encoder’s output is used as the similarity score between the input image and text.
Revised: CLIP follows a “shallow-interaction design”, where a visual encoder and a text encoder encode an input image and text independently, and the dot-product between the two encoder’s output is used as the similarity score between the input image and text.

Original: It is pre-trained with a contrastive loss where the model needs to distinguish aligned pairs from randomly sampled pairs.
Revised: It is pre-trained with a contrastive loss where the model needs to distinguish aligned pairs from randomly sampled pairs.

Original: CLIP leverages an abundantly available source of supervision without human annotation: 400M image-text pairs found across the Internet.
Revised: CLIP leverages an abundantly available source of supervision without human annotation: 400M image-text pairs found across the Internet.

Original: As a result, CLIP achieves SotA performance in a range of image classiﬁcation and image-text retrieval tasks in a zero-shot setting.
Revised: As a result, CLIP achieves SotA performance in a range of image classiﬁcation and image-text retrieval tasks in a zero-shot setting.

Original: 2. M OTIVATION
Revised: 2. M OTIVATION

Original: Despite the strong zero-shot capability of CLIP on vision tasks, CLIP does not exhibit the same level of performance on certain V&L downstream tasks.
Revised: Despite the strong zero-shot capability of CLIP on vision tasks, CLIP does not exhibit the same level of performance on certain V&L downstream tasks.

Original: For instance, if we cast VQA 2.0 (Goyal et al., 2017) into a zero-shot image-to-text retrieval task, we only observe chance performance (Section 5).
Revised: For instance, if we cast VQA 2.0 (Goyal et al., 2017) into a zero-shot image-to-text retrieval task, we only observe chance performance (Section 5).

Original: Thus, we propose to integrate CLIP’s visual encoder with previous V&L models (Figure 1).
Revised: Thus, we propose to integrate CLIP’s visual encoder with previous V&L models (Figure 1).

Original: We consider the following CLIP variants with different visual backbones (He et al., 2016; Dosovitskiy et al., 2020) (CLIP-ResNet denoted as CLIP-Res): CLIP-Res50, CLIP-Res101, CLIP-Res50x4,and
Revised: We consider the following CLIP variants with different visual backbones (He et al., 2016; Dosovitskiy et al., 2020) (CLIP-ResNet denoted as CLIP-Res): CLIP-Res50, CLIP-Res101, CLIP-Res50x4,

Original: CLIP-ViT-B. We next describe our methods in two scenarios: 1) direct task-speciﬁc ﬁne-tuning (Section 3) and 2)
Revised: CLIP-ViT-B/16 and CLIP-ViT-B/32. We next describe our methods in two scenarios: 1) direct task-speciﬁc ﬁne-tuning (Section 3) and 2)

Original: V&L pre-training (Section 4).
Revised: V&L pre-training (Section 4).

Original:  CLIP-V I L
Revised:  CLIP-V I L

Original: In this section, we directly use CLIP as the visual encoder in task-speciﬁc models (referred as CLIPViL) and ﬁne-tune on three representative tasks including Visual Question Answering (Section 3.1),
Revised: In this section, we directly use CLIP as the visual encoder in task-speciﬁc models (referred as CLIPViL) and ﬁne-tune on three representative tasks including Visual Question Answering (Section 3.1),

Original: Image Captioning (Section 3.2), and Vision-Language Navigation (Section 3.3).
Revised: Image Captioning (Section 3.2), and Vision-Language Navigation (Section 3.3).

Original: 3. V ISUAL Q UESTION A NSWERING
Revised: 3. V ISUAL Q UESTION A NSWERING

Original: The task of Visual Question Answering (VQA) (Antol et al., 2015) is to provide the answer given an image and a related question.
Revised: The task of Visual Question Answering (VQA) (Antol et al., 2015) is to provide the answer given an image and a related question.

Original: Various methods have been introduced (Fukui et al., 2016;
Revised: Various methods have been introduced (Fukui et al., 2016;

Original: Yang et al., 2016; Anderson et al., 2018a; Jiang et al., 2018; Gao et al., 2019; Jiang et al., 2020).
Revised: Yang et al., 2016; Anderson et al., 2018a; Jiang et al., 2018; Gao et al., 2019; Jiang et al., 2020).

Original: Here, we select two representative approaches (i
Revised: Here, we select two representative approaches (i

Original: [Table]
Revised: [Table]

Original: e., Pythia (Jiang et al., 2018) and MCAN (Yu et al., 2019)) to study the impact of the CLIP visual encoders in VQA
Revised: e., Pythia (Jiang et al., 2018) and MCAN (Yu et al., 2019)) to study the impact of the CLIP visual encoders in VQA.

Original: [Table]
Revised: outperform all baselines, visual encoders pre-trained with domain

Original: [Table]
Revised: [Table]

Original: Experimental Setup.

Original: We evaluate on VQA v2.0 (Goyal et al., 2017) and follow Jiang et al.
Revised: We evaluate on VQA v2.0 (Goyal et al., 2017) and follow Jiang et al.

Original: (2020) 4 for grid feature extraction.
Revised: (2020) 4 for grid feature extraction.

Original: Details of
Revised: Details of

Original: Pythia and MCAN as well as full implementation details are included in the Appendix.
Revised: Pythia and MCAN as well as full implementation details are included in the Appendix.

Original: Experimental Results.
Revised: Experimental Results.

Original: We report results on the VQA v2.0 Test-dev / Test-std set in Table 1.
Revised: We report results on the VQA v2.0 Test-dev / Test-std set in Table 1.

Original: Compared to the  visual feature extractors pretrained on ImageNet classiﬁcation task, CLIP visual modules demonstrate clear improvement (ﬁrst two blocks of Table 1). CLIP-Res achieves 65.55% with Pythia (4.01% better than
Revised: We compare with the following visual encoders:

Original: ImageNet-Res50) and 71.49% with MCAN (4.26% better than ImageNet-Res50) on Testdev. With larger models (i.e., CLIP-Res101 and CLIP-Res50x4) , the results continue improving and the largest CLIP-ViL CLIP-Res50x outperforms ImageNet-ResNeXt-101 R (+0.16) with Pythia.
Revised: • Standard ImageNet pre-trained visual encoders ( ImageNet-* ); • Visual encoders with SotA performance on ImageNet ( BiT M -* ) (Kolesnikov et al., 2020); • Visual encoders pre-trained with detection data ( VG-* and BUTD-* ) (Anderson et al., 2018a; Jiang et al., 2020).

Original: CLIP-ViL CLIP-Res50x4 also achieves the best performance of 74.01% on
Revised: Compared to all baselines, CLIP visual encoders demonstrate clear improvement.

Original: Test-dev and 74.17% on Test-std with MCAN. We also show results after further detection pretraining on VG (Jiang et al., 2020).
Revised: We especially note that VG-* and BUTD- models are pre-trained on in-domain detection data,

Original: We mark these results as Pythia VG and MCAN VG in the last two blocks of Table 1. With ImageNetRes50 encoder, it helps boost the performance by 2.82% on Pythia VG (61.54% vs. 64.36%) and 2.90% on MCAN VG (67.23% vs. 70.13%).
Revised: Visual Genome, which contain the sames images as VQA data. Thus, they signiﬁcantly outperform baselines without such detection data (ImageNet-* and BiT M -*).

Original: However, the performance drops dramatically for CLIP-Res50 by 5.54% with Pythia VG and 4.08% with MCAN VG .
Revised: However, CLIP-* models without in-domain detection data can outperform VG-* and BUTD-*.

Original: The potential reason is  that CLIPRes50 is trained on different data and with a different method compared to ImageNet counterparts, so following the previous Visual-Genome ﬁne-tuning practice designed for ImageNet models may hurt. We also notice that our best-performing CLIP-ViL CLIP-Res50x4 MCAN (74.01%) still surpasses the best ImageNet-ResNeXt-101 MCAN VG (72.59%) on Test-dev.
Revised: This is the ﬁrst time that a visual encoder without detection data can outperform those with detection data. Detection data are hard to scale up and contain limited object categories, while our results suggest training visual encoders on noisy image-text data as in CLIP is promising and scalable.

Original: 3. I MAGE C APTIONING
Revised: 3. I MAGE C APTIONING

Original: Image captioning aims at generating a natural language description of an image.
Revised: Image captioning aims at generating a natural language description of an image.

Original: Various methods have been delivered for image captioning (Karpathy and Fei-Fei, 2015; Rennie et al., 2017; Anderson et al., 2018a;
Revised: Various methods have been delivered for image captioning (Karpathy and Fei-Fei, 2015; Rennie et al., 2017; Anderson et al., 2018a;

Original: Luo et al., 2018; Luo, 2020).
Revised: Luo et al., 2018; Luo, 2020).

Original: We investigate the effectiveness of the CLIP model for this popular task with (Luo, 2020) method.
Revised: We investigate the effectiveness of the CLIP model for this popular task with (Luo, 2020) method.

Original: Experimental Setup.
Revised: Experimental Setup.

Original: For the model architecture, we experiment with the basic Transformer model adapted from Vaswani et al.
Revised: For the model architecture, we experiment with the basic Transformer model adapted from Vaswani et al.

Original: (2017) in Luo (2020).
Revised: (2017) in Luo (2020).

Original: Grid feature maps are extracted for each image.
Revised: Grid feature maps are extracted for each image.

Original: We evaluate our model on COCO dataset (Chen et al., 2015).
Revised: We evaluate our model on COCO dataset (Chen et al., 2015).

Original: We use the standard automatic evaluation metrics including CIDEr (Anderson et al., 2016), BLEU (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), and SPICE (Anderson et al., 2016).
Revised: We use the standard automatic evaluation metrics including CIDEr (Anderson et al., 2016), BLEU (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), and SPICE (Anderson et al., 2016).

Original: The scores are obtained on Karparthy test split (Karpathy and Fei-Fei, 2015) with beam search of 5 beams.
Revised: The scores are obtained on Karparthy test split (Karpathy and Fei-Fei, 2015) with beam search of 5 beams.

Original: Details are given in Appendix.
Revised: Details are given in Appendix.

Original: [Table]

Original: ‘Pre-Training’ methods are mostly in-domain pre-trained on

Original: [Table]

Original: Experimental Results.
Revised: Experimental Results.

Original: We report Image Captioning results with different models in Table 2.
Revised: We report Image Captioning results with different models in Table 2.

Original: Using the Transformer architecture from (Luo, 2020), we see that CLIP-Res models outperform ImageNet pre-trained alternatives for both ResNet50 (+9.1 / +1.5 in CIDEr / SPICE) and ResNet101 (+9.2 / +1.5 in CIDEr / SPICE).
Revised: Using the Transformer architecture from (Luo, 2020), we see that CLIP-Res models outperform ImageNet pre-trained alternatives for both ResNet50 (+9.1 / +1.5 in CIDEr / SPICE) and ResNet101 (+9.2 / +1.5 in CIDEr / SPICE).

Original: It even surpasses the strong in-domain region-based feature from BUTD.
Revised: It even surpasses the strong in-domain region-based feature from BUTD

Original:  As the model size grows in CLIP-ViL, the results also improve and the largest CLIP-Res50x4 achieves the best performance, although there still remains a gap to the pre-trained models that have interactive image-text pre-training phase like Oscar base and VinVL base .
Revised: [Table] [Table] and grid-based feature from BiT. As the model size grows in CLIP-ViL, the results also improve and the largest CLIP-Res50x4 achieves the best performance, although there still remains a gap to the pre-trained models that have interactive image-text pre-training phase like Oscar base and VinVL base .

Original: Again, CLIP-ViT-B variant leads to dramatically worse performance compared to other visual modules, that we will discuss in Section 5.
Revised: Again, CLIP-ViT variant leads to dramatically worse performance compared to other visual modules, that we will discuss in Section 5.

Original: 3. V ISION - AND -L ANGUAGE N AVIGATION
Revised: 3. V ISION - AND -L ANGUAGE N AVIGATION

Original: Vision-and-language navigation tests the agent’s ability to take action according to human instructions, which recently gains popularity in embodied AI (Anderson et al., 2018b;
Revised: Vision-and-language navigation tests the agent’s ability to take action according to human instructions, which recently gains popularity in embodied AI (Anderson et al., 2018b;

Original: Chen et al., 2019; Jain et al., 2019; Chen et al., 2019;
Revised: Chen et al., 2019; Jain et al., 2019; Chen et al., 2019;

Original: Qi et al., 2020b; Krantz et al., 2020; Nguyen and Daumé III, 2019;
Revised: Qi et al., 2020b; Krantz et al., 2020; Nguyen and Daumé III, 2019;

Original: Ku et al., 2020).
Revised: Ku et al., 2020).

Original: Speciﬁcally, the agent is put at a location in the environment (Chang et al., 2017) and asked to reach a target by following the language instructions.
Revised: Speciﬁcally, the agent is put at a location in the environment (Chang et al., 2017) and asked to reach a target by following the language instructions.

Original: Here, we investigate the impact of the CLIP visual encoder on this new task.
Revised: Here, we investigate the impact of the CLIP visual encoder on this new task.

Original: Model Architecture.
Revised: Model Architecture.

Original: We experiment with the basic attentive neural agent as in Fried et al.
Revised: We experiment with the basic attentive neural agent as in Fried et al.

Original: (2018) (please refer to the original paper for implementation details).
Revised: (2018) (please refer to the original paper for implementation details).

Original: At each time step, the agent attends to the panoramic views and the instruction to make an action.
Revised: At each time step, the agent attends to the panoramic views and the instruction to make an action.

Original: We replace the pre-trained visual encoder from ImageNet pre-trained ResNet to the pre-trained CLIP visual encoders.
Revised: We replace the pre-trained visual encoder from ImageNet pre-trained ResNet to the pre-trained CLIP visual encoders.

Original: Different from the VQA task that uses a feature map to include detailed information, we use a single-vector output for the entire image following previous works (Fried et al., 2018).
Revised: Different from the VQA task that uses a feature map to include detailed information, we use a single-vector output for the entire image following previous works (Fried et al., 2018).

Original: For CLIP-ViT-B models, we take the output of the [CLS] token.
Revised: For CLIP-ViT-B/32 models, we take the output of the [CLS] token.

Original: For CLIP-ResNet models, we take the attentive pooled feature (Radford et al., 2021) of the feature map.
Revised: For CLIP-ResNet models, we take the attentive pooled feature (Radford et al., 2021) of the feature map.

Original: These features are also linearly projected and L2-normalized as in the CLIP model.
Revised: These features are also linearly projected and L2-normalized as in the CLIP model.

Original: Experimental Setup.
Revised: Experimental Setup.

Original: We apply our model to two vision-and-language navigation datasets: Room-toRoom (R2R, Anderson et al.
Revised: We apply our model to two vision-and-language navigation datasets: Room-toRoom (R2R, Anderson et al.

Original: (2018b))
Revised: (2018b))

Original: and Room-across-Room (RxR, Ku et al.
Revised: and Room-across-Room (RxR, Ku et al.

Original: R2R is built on the indoor environments from the MatterPort3D dataset (Chang et al., 2017).
Revised: R2R is built on the indoor environments from the MatterPort3D dataset (Chang et al., 2017).

Original: The environments are split into training, unseen validation, and unseen test.
Revised: The environments are split into training, unseen validation, and unseen test.

Original: RxR extends the R2R dataset to multiple[Table] languages and follows the environment split.
Revised: RxR extends the R2R dataset to multiple languages and follows the environment split.

Original: For R2R dataset
Revised: For R2R dataset, we follow the hyperparameter of the

Original: [Table]
Revised: [Table]

Original: we follow the hyperparameter of the publicly available implementation 5 R2R-EnvDrop (Tan et al., 2019) and replace the input featureswith the CLIP features.
Revised: publicly available implementation 5 R2R-EnvDrop (Tan et al., 2019) and replace the input featureswith the CLIP features.

Original: For RxR dataset, we change the path length and instruction length; details are given in Appendix.
Revised: For RxR dataset, we change the path length and instruction length; details are given in Appendix.

Original: [Table]
Revised: [Table]

Original: Experimental Results.
Revised: Experimental Results.

Original: We show the testunseen results of our best model (CLIPRes50x4) and the comparison to the previous methods.
Revised: We show the testunseen results of our best model (CLIPRes50x4) and the comparison to the previous methods.

Original: On R2R dataset (in Table 3), CLIPViL reaches 8% higher in SR (success rate) and 6% higher in SPL (Success Rate normalized by Path Length) than our baseline, EnvDrop.
Revised: On R2R dataset (in Table 3), CLIPViL reaches 8% higher in SR (success rate) and 6% higher in SPL (Success Rate normalized by Path Length) than our baseline, EnvDrop.

Original: CLIP-ViL outperforms previous nonpre-training agents and shows competitive results to VLN-speciﬁc pre-trained models.
Revised: CLIP-ViL outperforms previous nonpre-training agents and shows competitive results to VLN-speciﬁc pre-trained models.

Original: On
Revised: On

Original: RxR dataset (Table 5), CLIP-ViL achieves the best success rate and nDTW (normalized Dynamic Time Warping) under the mono-lingual setup (Ku et al., 2020) and is 4.3% better then the previous results for nDTW.
Revised: RxR dataset (Table 5), CLIP-ViL achieves the best success rate and nDTW (normalized Dynamic Time Warping) under the mono-lingual setup (Ku et al., 2020) and is 4.3% better then the previous results for nDTW.

Original: In Table 4, we compare different CLIP variants with the previous standard ResNet-152 feature extractors.
Revised: In Table 4, we compare different CLIP variants with the previous standard ResNet-152 feature extractors.

Original: These extractors are pre-trained on ImageNet and use the mean-pooled features as the representation for the image.
Revised: These extractors are pre-trained on ImageNet and use the mean-pooled features as the representation for the image.

Original: CLIP-Res50 shows a clear improvement over the IN alternative (‘ImageNet-Res152’).
Revised: CLIP-Res50 shows a clear improvement over the IN alternative (‘ImageNet-Res152’).

Original: With larger models (i.e., ‘CLIP-Res101’ and ‘CLIP-Res50x4’), the agent performance scales well on both R2R and RxR. Lastly, we ﬁnd that the CLIP ViT model (‘CLIP-ViTB’) has similar results as CLIP-Res50 model.
Revised: With larger models (i.e., ‘CLIP-Res101’ and ‘CLIP-Res50x4’), the agent performance scales well on both R2R and RxR. Lastly, we ﬁnd that the CLIP ViT model (‘CLIP-ViTB/32’) has similar results as CLIP-Res50 model.

Original: ViT also shows a relatively better result when back translation (BT) is applied.
Revised: ViT also shows a relatively better result when back translation (BT) is applied.

Original: The success of ViT model in VLN is possibly due to the use of [CLS] feature instead of the feature map.
Revised: The success of ViT model in VLN is possibly due to the use of [CLS] feature instead of the feature map.

Original:  V ISION - AND -L ANGUAGE P RE - TRAINING
Revised:  V ISION - AND -L ANGUAGE P RE - TRAINING

Original: Recently, V&L pre-training has been proposed as an effective technique to improve the performance on various V&L tasks (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019a;
Revised: Recently, V&L pre-training has been proposed as an effective technique to improve the performance on various V&L tasks (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019a;

Original: Su et al., 2019; Chen et al., 2020;
Revised: Su et al., 2019; Chen et al., 2020;

Original: Zhou et al., 2020; Huang et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b).
Revised: Zhou et al., 2020; Huang et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b).

Original: Before task-speciﬁc ﬁne-tuning, the model is pre-trained on aligned image-text data with a reconstructive objective and an image-text matching objective.
Revised: Before task-speciﬁc ﬁne-tuning, the model is pre-trained on aligned image-text data with a reconstructive objective and an image-text matching objective.

Original: We seek to test the potential of combining CLIP pre-training and V&L pre-training.
Revised: We seek to test the potential of combining CLIP pre-training and V&L pre-training.

Original: We introduce CLiP-ViL p , a vision-and-language model pre-trained on image-text data with CLIP visual encoder as its visual backbone.
Revised: We introduce CLiP-ViL p , a vision-and-language model pre-trained on image-text data with CLIP visual encoder as its visual backbone.

Original: In the following, we introduce the model architecture and pre-training process of CLiP-ViL p in detail.
Revised: In the following, we introduce the model architecture and pre-training process of CLiP-ViL p in detail.

Original: [Table]
Revised: [Table]

Original: Model Architecture.
Revised: Model Architecture.

Original: CLiP-ViL p assumes a text segment T and an image I as input.
Revised: CLiP-ViL p assumes a text segment T and an image I as input.

Original: As in BERT, the text is tokenized into a sequence of subwords { w 1 , w 2 , ..., w k } .
Revised: As in BERT, the text is tokenized into a sequence of subwords { w 1 , w 2 , ..., w k } .

Original: Every subword is embedded as the sum of its token, position, and segment embeddings (Devlin et al., 2019) and thus the text is embedded as a sequence of word embeddings { w 1 , w 2 , ..., w n } .
Revised: Every subword is embedded as the sum of its token, position, and segment embeddings (Devlin et al., 2019) and thus the text is embedded as a sequence of word embeddings { w 1 , w 2 , ..., w n } .

Original: The image is embedded as a set of visual vectors { v 1 , v 2 , ..., v m } from the grid-like feature map.
Revised: The image is embedded as a set of visual vectors { v 1 , v 2 , ..., v m } from the grid-like feature map.

Original: The text and visual input are then concatanated into a sequence, { w 1 , w 2 , ..., w n , v 1 , v 2 , ..., v m } , and processed by a single
Revised: The text and visual input are then concatanated into a sequence, { w 1 , w 2 , ..., w n , v 1 , v 2 , ..., v m } , and processed by a single

Original: Transformer.
Revised: Transformer.

Original: In most region-based models, the visual backbone is frozen as ﬁne-tuning the object detector along with the Transformer remains an open problem (Su et al., 2019).
Revised: In most region-based models, the visual backbone is frozen as ﬁne-tuning the object detector along with the Transformer remains an open problem (Su et al., 2019).

Original: In CLiP-ViL p , the
Revised: In CLiP-ViL p , the

Original: CLIP backbone is trained during both V&L pre-training and task-speciﬁc ﬁne-tuning (see discussion in Section 5).
Revised: CLIP backbone is trained during both V&L pre-training and task-speciﬁc ﬁne-tuning (see discussion in Section 5).

Original: Pre-training on Image-Text Data.
Revised: Pre-training on Image-Text Data.

Original: To learn uniﬁed representations for both vision and language, we follow prior work and pre-train the model on image-text pairs.
Revised: To learn uniﬁed representations for both vision and language, we follow prior work and pre-train the model on image-text pairs.

Original: We consider three pre-training objectives from LXMERT (Tan and Bansal, 2019): 1) grounded masked language modeling, where we randomly mask out 15% of words in the input sentence and train the model to reconstruct the masked words; 2) text-image matching, where the model is provided with a mismatched sentence with a probability of 0.5, and is trained to classify whether the text corresponds to the image; 3) visual question answering, where we train the model to predict the correct answer given a question.
Revised: We consider three pre-training objectives from LXMERT (Tan and Bansal, 2019): 1) grounded masked language modeling, where we randomly mask out 15% of words in the input sentence and train the model to reconstruct the masked words; 2) text-image matching, where the model is provided with a mismatched sentence with a probability of 0.5, and is trained to classify whether the text corresponds to the image; 3) visual question answering, where we train the model to predict the correct answer given a question.

Original: 4. E XPERIMENTS
Revised: 4. E XPERIMENTS

Original: Setup.
Revised: Setup.

Original: We experiment with two variants of CLIP as the visual encoder, CLIP-Res50 and CLIPRes50x4.
Revised: We experiment with two variants of CLIP as the visual encoder, CLIP-Res50 and CLIPRes50x4.

Original: Following LXMERT, we use the same corpora aggregated from MS COCO Captions (Chen et al., 2015), Visual Genome Captions (Krishna et al., 2017), VQA (Antol et al., 2015), GQA (Hudson and Manning, 2019), and VG-QA (Zhu et al., 2016) for pre-training.
Revised: Following LXMERT, we use the same corpora aggregated from MS COCO Captions (Chen et al., 2015), Visual Genome Captions (Krishna et al., 2017), VQA (Antol et al., 2015), GQA (Hudson and Manning, 2019), and VG-QA (Zhu et al., 2016) for pre-training.

Original: We follow the same preprocessing procedure and exclude any test data from the pre-training dataset.
Revised: We follow the same preprocessing procedure and exclude any test data from the pre-training dataset.

Original: This results in 9.18M image-text pairs.
Revised: This results in 9.18M image-text pairs.

Original: For computational efﬁciency, we use a relatively small resolution for images.
Revised: For computational efﬁciency, we use a relatively small resolution for images.

Original: We resize the shorter edges of images to 384 and the longer edges to under 640 with preserved aspect ratios.
Revised: We resize the shorter edges of images to 384 and the longer edges to under 640 with preserved aspect ratios.

Original: During pre-training, as the number of image patches is large, we randomly sample 100 image patches for every image following PixelBERT (Huang et al., 2020).
Revised: During pre-training, as the number of image patches is large, we randomly sample 100 image patches for every image following PixelBERT (Huang et al., 2020).

Original: We pre-train the model for 20 epochs and unfreeze the CLIP backbone during pre-training and ﬁne-tuning.
Revised: We pre-train the model for 20 epochs and unfreeze the CLIP backbone during pre-training and ﬁne-tuning.

Original: For details see the Appendix.
Revised: For details see the Appendix.

Original: Tasks.
Revised: Tasks.

Original: For evaluation, we ﬁne-tune the pre-trained model on three V&L tasks: VQA v2.0 (Goyal et al., 2017), visual entailment SNLI-VE (Xie et al., 2019), and GQA (Hudson and Manning, 2019).
Revised: For evaluation, we ﬁne-tune the pre-trained model on three V&L tasks: VQA v2.0 (Goyal et al., 2017), visual entailment SNLI-VE (Xie et al., 2019), and GQA (Hudson and Manning, 2019).

Original: We provide more details in the Appendix.
Revised: We provide more details in the Appendix.

Original: Results.
Revised: Results.

Original: We report the results in Table 6.
Revised: We report the results in Table 6.

Original: We include previous best pre-trained V&L models and their V&L pre-training data and epochs.
Revised: We include previous best pre-trained V&L models and their V&L pre-training data and epochs.

Original: As our model is based on BERT B ASE , we compare only with models based on BERT B ASE .
Revised: As our model is based on BERT B ASE , we compare only with models based on BERT B ASE .

Original: The models are grouped by their visual encoder type.
Revised: The models are grouped by their visual encoder type.

Original: We ﬁrst note that v2.0 mini-eval , “PE” denotes we follow similar prompt engineering as suggested in CLIP paper.
Revised: We ﬁrst note that v2.0 mini-eval , “PE” denotes we follow similar prompt engineering as suggested in CLIP paper.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: our two models perform competitively on all metrics.
Revised: our two models perform competitively on all metrics.

Original: Especially, CLIP-ViL with CLIP-Res50x establishes a new SotA on VQA and SNLI-VE.
Revised: Especially, CLIP-ViL with CLIP-Res50x establishes a new SotA on VQA and SNLI-VE.

Original: When comparing with the BUTD visual encoder trained on in-domain data (including LXMERT (Tan and Bansal, 2019), UNITER (Chen et al., 2020), and Oscar (Li et al., 2020)), our two models (CLIP-ViL with CLIP-Res50 and CLIP-Res50x4) signiﬁcantly outperform most BUTD-Res based models.
Revised: When comparing with the BUTD visual encoder trained on in-domain data (including LXMERT (Tan and Bansal, 2019), UNITER (Chen et al., 2020), and Oscar (Li et al., 2020)), our two models (CLIP-ViL with CLIP-Res50 and CLIP-Res50x4) signiﬁcantly outperform most BUTD-Res based models.

Original: We especially note that LXMERT is trained on the same pre-training dataset and for the same number of epochs as our model, yet our CLiP-ViL p with CLIP-Res50 outperforms
Revised: We especially note that LXMERT is trained on the same pre-training dataset and for the same number of epochs as our model, yet our CLiP-ViL p with CLIP-Res50 outperforms

Original: LXMERT on VQA by 2.59.
Revised: LXMERT on VQA by 2.59.

Original: VinVL (Li et al., 2020) is an extreme scale-up of the region-based paradigm, which is pre-trained on multiple object detection datasets, including MS COCO (Lin et al., 2014), OpenImages (Kuznetsova et al., 2020), Object365 (Shao et al., 2019), and Visual Genome (Krishna et al., 2017).
Revised: VinVL (Li et al., 2020) is an extreme scale-up of the region-based paradigm, which is pre-trained on multiple object detection datasets, including MS COCO (Lin et al., 2014), OpenImages (Kuznetsova et al., 2020), Object365 (Shao et al., 2019), and Visual Genome (Krishna et al., 2017).

Original: Yet, our model with CLIP-Res50x4 outperforms VinVL on VQA, while requiring signiﬁcantly less steps of
Revised: Yet, our model with CLIP-Res50x4 outperforms VinVL on VQA, while requiring signiﬁcantly less steps of

Original: V&L pre-training.
Revised: V&L pre-training.

Original: On GQA, our model under-performs VinVL.
Revised: On GQA, our model under-performs VinVL.

Original: The potential reason is that GQA is automatically constructed from object bounding box data, which may give region-based models trained on such object data a signiﬁcant advantage.
Revised: The potential reason is that GQA is automatically constructed from object bounding box data, which may give region-based models trained on such object data a signiﬁcant advantage.

Original: Lastly, we compare to Pixel-BERT (Huang et al., 2020), which takes a similar design as our model, but with an ImageNet initialized ResNet.
Revised: Lastly, we compare to Pixel-BERT (Huang et al., 2020), which takes a similar design as our model, but with an ImageNet initialized ResNet.

Original: CLIP initialization clearly holds advantage over ImageNet initialization, as CLIP-Res50 signiﬁcantly outperforms Pixel-BERT with ImageNet-Res50.
Revised: CLIP initialization clearly holds advantage over ImageNet initialization, as CLIP-Res50 signiﬁcantly outperforms Pixel-BERT with ImageNet-Res50.

Original:  A NALYSIS
Revised:  A NALYSIS

Original: In this section, we provide detailed analyses on a few interesting phenomena we observe during our experiments, which may help guide future exploration.
Revised: In this section, we provide detailed analyses on a few interesting phenomena we observe during our experiments, which may help guide future exploration.

Original: Quantitative and qualitative analysis are provided to support our ﬁndings.
Revised: Quantitative and qualitative analysis are provided to support our ﬁndings.

Original: Zero-Shot Performance of CLIP in VQA.
Revised: Zero-Shot Performance of CLIP in VQA.

Original: In the original paper, CLIP is intended as a zero-shot model and shows strong performance on various vision and image retrieval tasks.
Revised: In the original paper, CLIP is intended as a zero-shot model and shows strong performance on various vision and image retrieval tasks.

Original: We are thus curious if CLIP can also perform well as a zero-shot model on V&L tasks that may require complex reasoning.
Revised: We are thus curious if CLIP can also perform well as a zero-shot model on V&L tasks that may require complex reasoning.

Original: To conduct zero-shot image classiﬁcation, CLIP (Radford et al., 2021) uses the names of all classes in the dataset as the set of candidate text and predict the most probable (image, text) pair.
Revised: To conduct zero-shot image classiﬁcation, CLIP (Radford et al., 2021) uses the names of all classes in the dataset as the set of candidate text and predict the most probable (image, text) pair.

Original: We thus experiment with a similar setting on VQA but modify the candidate text to be the concatenation of question and answer pair for each question.
Revised: We thus experiment with a similar setting on VQA but modify the candidate text to be the concatenation of question and answer pair for each question.

Original: Moreover, Radford et al.
Revised: Moreover, Radford et al.

Original: ﬁnd a result improvement from prompt engineering.
Revised: ﬁnd a result improvement from prompt engineering.

Original: We follow this design by constructing “question: [question text] answer: [answer text]” as the prompt template.
Revised: We follow this design by constructing “question: [question text] answer: [answer text]” as the prompt template.

Original: The results on VQA v2.0 mini-eval are shown in Table 7.
Revised: The results on VQA v2.0 mini-eval are shown in Table 7.

Original: All CLIP variants perform at near-chance level in the zero-shot setting while prompt engineering helps only a little.
Revised: All CLIP variants perform at near-chance level in the zero-shot setting while prompt engineering helps only a little.

Original: CLIP models also perform worse when the question becomes harder (“other” vs. “yes/no”).
Revised: CLIP models also perform worse when the question becomes harder (“other” vs. “yes/no”).

Original: All these results suggest the need of a deep interactive model and additional pre-training/ﬁne-tuning.
Revised: All these results suggest the need of a deep interactive model and additional pre-training/ﬁne-tuning.

Original: Unfreezing the Visual Backbone.
Revised: Beneﬁt of V&L Pre-training.

Original: Because of technical difﬁculty in ﬁne-tuning the object detector, most V&L models rely on frozen region-based encoders (Lu et al., 2019).
Revised: In Table 8, we compare the performance of models with or without V&L pre-training.

Original: However, we ﬁndthat unfreezing the visual backbone (Section 4) may bring performance improvement.
Revised: We ﬁnd

Original: Speciﬁcally, we test the backbone ﬁne-tuning performance of two CLIP features (i.e., CLIP-Res50, CLIP-Res50x4) on VQA (test-dev) 7 and compare with the frozen BUTD-Res101 features. Without pre-training, BUTD-Res101 achieves higher performance than CLIP-Res50.
Revised: that V&L pre-training brings signiﬁcant performance improvement for the three models we test. Interestingly, we ﬁnd that CLIP models beneﬁt more from V&L pre-training. Our conjecture is that the additional beneﬁt could come from unfreezing the visual backbone.

Original: However, after V&L pre-training, CLIP-Res50 signiﬁcantly outperforms BUTD-Res101, because CLIP-Res beneﬁts more from pre-training (+9.25) than BUTD-Res101 (+5.72).
Revised: Because of technical difﬁculty in ﬁne-tuning the object detector, most V&L models rely on frozen region-based encoders (Lu et al., 2019).

Original: This suggests that unfreezing the visual backbone during pre-training allows CLIP-Res50 to adapt to the pre-training task.
Revised: But for grid-features such as CLIP, we can easily ﬁne-tune the visual backbone and could potentially aid CLIP to adapt to the pre-training task.

Original: We hope that our ﬁnding inspires future work to further explore unfreezing the visual backbone in V&L models when computational budget allows.
Revised: We hope that our ﬁnding inspires future work to further explore unfreezing the visual backbone in V&L models when computational budget allows.

Original: [Figure]
Revised: [Figure]

Original: Low Detection Performance of CLIP-ViT-B.
Revised: Low Detection Performance of CLIP-ViT-B/32.

Original: As shown in Table 1 and Table 2, CLIP-ViT-B with grid features has a large performance degradation compared with other models.
Revised: As shown in Table 1 and Table 2, CLIP-ViT-B/ with grid features has a large performance degradation compared with other models.

Original: We hypothesize that such decrease is due to the lack of visual localization inside the ViT feature map since different pooling strategies may affect the localization ability (Zhou et al., 2016).
Revised: We hypothesize that such decrease is due to the lack of visual localization inside the ViT feature map since different pooling strategies may affect the localization ability (Zhou et al., 2016).

Original: We thus follow Jiang et al.
Revised: We thus follow Jiang et al.

Original: (2020) to train a detector on Visual Genome over the CLIP-ViT-B grid feature maps to conﬁrm it.
Revised: (2020) to train a detector on Visual Genome over the CLIP-ViT-B/32 grid feature maps to conﬁrm it.

Original: We ﬁnd that the Average Precision (AP)of CLIP-ViT-B is only 0.03, which is much lower than its ImageNet-Res50 alternative (3.14 as we reproduced).
Revised: We ﬁnd that the Average Precision (AP) is only 0.03 for CLIP-ViT-B/32 and 0.17 for CLIP-ViT-B/16, which is much lower than its ImageNet-Res50 alternative (3.14 as we reproduced).

Revised: This ﬁnding of the localization ability of vision transformers is also in recent studies (Wang et al., 2021;

Revised: Raghu et al., 2021; Dai et al., 2021).

Original: Qualitative Comparison of CLIP Variants.
Revised: Qualitative Comparison of CLIP Variants.

Original: As we discussed above, we suspect that CLIP-ViT-B lacks certain localization ability.
Revised: As we discussed above, we suspect that CLIP-ViT-B/ lacks certain localization ability.

Original: To understand this better, we perform Gradient-Based Localization (Grad-CAM)
Revised: To understand this better, we perform Gradient-Based Localization (Grad-CAM)

Original: (Selvaraju et al., 2017) to visualize the salient regions of CLIP models.
Revised: (Selvaraju et al., 2017) to visualize the salient regions of CLIP models.

Original: The qualitative example in Figure 3 clearly shows CLIP-Res50 localizes the sentence “What color is the woman’s shirt on the left?”
Revised: The qualitative example in Figure 3 clearly shows CLIP-ResNet variants localizes the sentence “What color is the woman’s shirt on the left?”

Original: better than CLIP-ViT-B. We provide more qualitative examples in the Appendix.
Revised: better than CLIP-ViT variants without ﬁnetuning. We provide more qualitative examples in the Appendix.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we propose to leverage CLIP as the visual encoder for different V&L models across various tasks.
Revised: In this paper, we propose to leverage CLIP as the visual encoder for different V&L models across various tasks.

Original: We experiment with two approaches: in the ﬁrst, we directly plug CLIP in task-speciﬁc ﬁne-tuning; in the second, we integrate CLIP with V&L pre-training and ﬁne-tune on downstream tasks afterwards.
Revised: We experiment with two approaches: in the ﬁrst, we directly plug CLIP in task-speciﬁc ﬁne-tuning; in the second, we integrate CLIP with V&L pre-training and ﬁne-tune on downstream tasks afterwards.

Original: A variety of substantial experiments on different V&L tasks demonstrates that
Revised: A variety of substantial experiments on different V&L tasks demonstrates that

Original: CLIP-ViL and CLIP-ViL p can achieve competitive or better performance as compared to strong baselines.
Revised: CLIP-ViL and CLIP-ViL p can achieve competitive or better performance as compared to strong baselines.

Original: Analyses from different perspectives explain certain intriguing phenomena and offer new directions for future V&L research.
Revised: Analyses from different perspectives explain certain intriguing phenomena and offer new directions for future V&L research.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We provide the code to reproduce the main results in this paper in the supplementary material, which contains comprehensive instructions to reproduce our results.
Revised: We provide the code to reproduce the main results in this paper in the supplementary material, which contains comprehensive instructions to reproduce our results.

Original: The code and model checkpoints will be made public.
Revised: The code and model checkpoints will be made public.

Original: A A PPENDIX A. V ISUAL Q UESTION A NSWERING
Revised: A A PPENDIX A. V ISUAL Q UESTION A NSWERING

Original: Model Architecture Pythia encodes the question with an attention-based GRU (Chung et al., 2014) network and fuse the information with a multi-modal factorized bilinear pooling network.
Revised: Model Architecture Pythia encodes the question with an attention-based GRU (Chung et al., 2014) network and fuse the information with a multi-modal factorized bilinear pooling network.

Original: MCAN takes a LSTM (Hochreiter and Schmidhuber, 1997) as question encoder and an encoder-decoder based modular co-attention network for fusing multiple representations.
Revised: MCAN takes a LSTM (Hochreiter and Schmidhuber, 1997) as question encoder and an encoder-decoder based modular co-attention network for fusing multiple representations.

Original: Both models employ an output classiﬁer on top of the fused representation to predict the ﬁnal answer.
Revised: Both models employ an output classiﬁer on top of the fused representation to predict the ﬁnal answer.

Original: To integrate CLIP for the VQA models, we extract grid features using CLIP.
Revised: To integrate CLIP for the VQA models, we extract grid features using CLIP.

Original: For CLIP-ViT-B models, we reshape the patch representation from the ﬁnal layer into grid features.
Revised: For CLIP-ViT-B/32 models, we reshape the patch representation from the ﬁnal layer into grid features.

Original: For CLIP-ResNet models, we simply take the grid features from the last layer before the pooling
Revised: For CLIP-ResNet models, we simply take the grid features from the last layer before the pooling.

Original: Implementation Details We follow (Jiang et al., 2020) to resize all input images to have a maximum shorter side of 600 pixels (longest 1000) when keeping the aspect ratio ﬁxed.
Revised: Implementation Details We follow (Jiang et al., 2020) to resize all input images to have a maximum shorter side of 600 pixels (longest 1000) when keeping the aspect ratio ﬁxed.

Original: For training the detector on the VG dataset, we replace the backbone with CLIP visual module using implementation of Faster R-CNN in Detectron2 8 .
Revised: For training the detector on the VG dataset, we replace the backbone with CLIP visual module using implementation of Faster R-CNN in Detectron2 7 .

Original: For training VQA models, we use hyperparameters of the opensource implementation 9 from (Jiang et al., 2020) for the large version of the MCAN and base version of Pythia.
Revised: For training VQA models, we use hyperparameters of the opensource implementation 8 from (Jiang et al., 2020) for the large version of the MCAN and base version of Pythia.

Original: A. I MAGE C APTIONING
Revised: A. I MAGE C APTIONING

Original: Implementation Details For training, we follow the ‘long epoch’ hyperparameter of the publicly available implementation 10 .
Revised: Implementation Details For training, we follow the ‘long epoch’ hyperparameter of the publicly available implementation 9 .

Original: During the self-critical stage, we sample 5 captions for each image as in Luo (2020).
Revised: During the self-critical stage, we sample 5 captions for each image as in Luo (2020).

Original: For training objective, we experiment with the Self-Critical Sequence Training (SCST) in Rennie et al.
Revised: For training objective, we experiment with the Self-Critical Sequence Training (SCST) in Rennie et al.

Original: (2017), where CIDEr (Vedantam et al., 2015) metric is optimized using REINFORCE algorithm
Revised: (2017), where CIDEr (Vedantam et al., 2015) metric is optimized using REINFORCE algorithm

Original: (Williams, 1992).
Revised: (Williams, 1992).

Original: A. V ISION - AND -L ANGUAGE N AVIGATION
Revised: A. V ISION - AND -L ANGUAGE N AVIGATION

Original: Model For the model architecture, we experiment with the basic attentive neural agent as in Fried et al.
Revised: Model For the model architecture, we experiment with the basic attentive neural agent as in Fried et al.

Original: The agent model (i.e., another LSTM) then attends to the visual features and the language representations to predict the actions.
Revised: The agent model (i.e., another LSTM) then attends to the visual features and the language representations to predict the actions.

Original: At each time step t , the agent attends to the panoramic views { v t,i } i and the instruction { w j } to make the action.
Revised: At each time step t , the agent attends to the panoramic views { v t,i } i and the instruction { w j } to make the action.

Original: The panoramic view is processed with a pre-trained visual encoder (e.g., ResNet ) and the instructions are processed by a language LSTM (Hochreiter and Schmidhuber, 1997), denoted LSTM L .
Revised: The panoramic view is processed with a pre-trained visual encoder (e.g., ResNet ) and the instructions are processed by a language LSTM (Hochreiter and Schmidhuber, 1997), denoted LSTM L .

Original: The agent model, LSTM A , then attends to the visual features and the language representations to predict the actions.
Revised: The agent model, LSTM A , then attends to the visual features and the language representations to predict the actions.

Original: [Equation]
Revised: [Equation]

Original: where h t and c t are the hiddens and states of the action LSTM at time step t , respectively.
Revised: where h t and c t are the hiddens and states of the action LSTM at time step t , respectively.

Original: Please refer to Fried et al.
Revised: Please refer to Fried et al.

Original: (2018) for the implementation details.
Revised: (2018) for the implementation details.

Revised: [Table]

Original: Implementation Details We apply our model to two vision-and-language navigation datasets:
Revised: Implementation Details We apply our model to two vision-and-language navigation datasets:

Original: Room-to-Room (R2R, Anderson et al.
Revised: Room-to-Room (R2R, Anderson et al.

Original: (2018b))
Revised: (2018b))

Original: and Room-across-Room (RxR, Ku et al.
Revised: and Room-across-Room (RxR, Ku et al.

Original: R2R is built on the indoor environments from the MatterPort3D dataset (Chang et al., 2017).
Revised: R2R is built on the indoor environments from the MatterPort3D dataset (Chang et al., 2017).

Original: The environments are split into training (61 environments), unseen validation (11 environments), and unseen test (18 environments).
Revised: The environments are split into training (61 environments), unseen validation (11 environments), and unseen test (18 environments).

Original: The agent is trained on the training environments (with 14, navigation instructions) and tested on separate sets of environments (2,349 in the unseen-validation and 4,173 in the unseen-test).
Revised: The agent is trained on the training environments (with 14, navigation instructions) and tested on separate sets of environments (2,349 in the unseen-validation and 4,173 in the unseen-test).

Original: RxR extends the R2R dataset with multiple languages and follow the environment split.
Revised: RxR extends the R2R dataset with multiple languages and follow the environment split.

Original: Besides the multilingual nature, RxR is also more diverse in the navigation paths and richer in the present language.
Revised: Besides the multilingual nature, RxR is also more diverse in the navigation paths and richer in the present language.

Original: For R2R dataset, we follow the hyperparameter (e.g., batch size, learning rate, optimizer) of the publicly available implementation 11 R2R-EnvDrop (Tan et al., 2019) and replace the input features 12 with the CLIP features.
Revised: For R2R dataset, we follow the hyperparameter (e.g., batch size, learning rate, optimizer) of the publicly available implementation 10 R2R-EnvDrop (Tan et al., 2019) and replace the input features 11 with the CLIP features.

Original: To reduce the computational cost, the features are pre-extracted and frozen during the training of the navigational agent.
Revised: To reduce the computational cost, the features are pre-extracted and frozen during the training of the navigational agent.

Original: For RxR dataset, we take the processed multilingual data provided in Li et al.
Revised: For RxR dataset, we take the processed multilingual data provided in Li et al.

Original: (2021a) with Stanza tokenizers (Qi et al., 2020a).
Revised: (2021a) with Stanza tokenizers (Qi et al., 2020a).

Original: Since RxR dataset contains instructions longer than R2R, we change the maximum input length to 160 (from 80 ) and increase the imitation learning ratio from 0 .
Revised: Since RxR dataset contains instructions longer than R2R, we change the maximum input length to 160 (from 80 ) and increase the imitation learning ratio from 0 .

Original: 2 to 0 .
Revised: 2 to 0 .

Original: 4 to stabilize the training.
Revised: 4 to stabilize the training.

Original: Other training hyperparameters of RxR are the same as R2R. The models are trained on one
Revised: Other training hyperparameters of RxR are the same as R2R. The models are trained on one

Original: RTX 2080 Ti GPU.
Revised: RTX 2080 Ti GPU.

Original: It takes 1 days to converge in R2R and about 1.5 days to converge in RxR. We report two signiﬁcant digits for R2R unseen test results following the leaderboard convention.
Revised: It takes 1 days to converge in R2R and about 1.5 days to converge in RxR. We report two signiﬁcant digits for R2R unseen test results following the leaderboard convention.

Original: [Table]

Original: Results Comparison to Grid Features In the main paper, we compare the results regarding the
Revised: Results Comparison to Grid Features In the main paper, we compare the results regarding the

Original: ImageNet-pre-trained ResNet-152.
Revised: ImageNet-pre-trained ResNet-152.

Original: We also report the comparison to grid features Jiang et al.
Revised: We also report the comparison to grid features Jiang et al.

Original: (2020) that is trained with detection dataset.
Revised: (2020) that is trained with detection dataset.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2020) showed that the results with these features are comparable to the original bottom-up attention with a heavy detection module.
Revised: (2020) showed that the results with these features are comparable to the original bottom-up attention with a heavy detection module.

Original: The same as the
Revised: The same as the

Original: VQA task in Section 3.1, we test the performance of these detection-trained grid features on VLN tasks.
Revised: VQA task in Section 3.1, we test the performance of these detection-trained grid features on VLN tasks.

Original: Speciﬁcally, we use the mean pooling of the feature map as the representation of each view following previous works (Anderson et al., 2018b).
Revised: Speciﬁcally, we use the mean pooling of the feature map as the representation of each view following previous works (Anderson et al., 2018b).

Original: As shown in Table 9, under the same ResNet backbone 13 , we ﬁnd that the detection-trained grid features are on par with the classiﬁcation-trained grid features, still showing a gap to the contrastive-trained grid features.
Revised: As shown in Table 9, under the same ResNet backbone 12 , we ﬁnd that the detection-trained grid features are on par with the classiﬁcation-trained grid features, still showing a gap to the contrastive-trained grid features.

Original: We hypothesize that the grid features inject regional knowledge into the dense feature map thus showing good results with grid-based modules (as shown in Section 3.1).
Revised: We hypothesize that the grid features inject regional knowledge into the dense feature map thus showing good results with grid-based modules (as shown in Section 3.1).

Original: However, pooling the feature map into a single feature vector (as in previous VLN works) leads to a loss of this dense information.
Revised: However, pooling the feature map into a single feature vector (as in previous VLN works) leads to a loss of this dense information.

Original: A. D ETAILS OF CLIP-V I L P
Revised: A. D ETAILS OF CLIP-V I L P

Original: Pre-training We pre-train with a batch size of 512.
Revised: Pre-training We pre-train with a batch size of 512.

Original: The Transformer is initialized from BERT B ASE and optimized with an AdamW (Loshchilov and Hutter, 2017) optimizer.
Revised: The Transformer is initialized from BERT B ASE and optimized with an AdamW (Loshchilov and Hutter, 2017) optimizer.

Original: We use a linearly-decaying schedule and a peak learning rate of 1 × 10 − 4 for the model with CLIP-Res50 and 5 × 10 − 5 for the model with CLIP-Res50x4.
Revised: We use a linearly-decaying schedule and a peak learning rate of 1 × 10 − 4 for the model with CLIP-Res50 and 5 × 10 − 5 for the model with CLIP-Res50x4.

Original: The ResNet is initialized from CLIP and we use SGD with a learning rate of 3 × 10 − 3 .
Revised: The ResNet is initialized from CLIP and we use SGD with a learning rate of 3 × 10 − 3 .

Original: We decay the learning rate of SGD at epochs 12, 17 by a factor of 10.
Revised: We decay the learning rate of SGD at epochs 12, 17 by a factor of 10.

Original: Per the suggestion of Tan and Bansal (2019), we only add the visual question answering loss during the later stage of the pre-training (the last 11 epochs) as the model is prone to overﬁt to the visual questionanswering loss.
Revised: Per the suggestion of Tan and Bansal (2019), we only add the visual question answering loss during the later stage of the pre-training (the last 11 epochs) as the model is prone to overﬁt to the visual question

Original: The model is trained on 8 Nvidia A100 GPUs and the pre-training takes arounddays.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Fine-tuning We ﬁne-tune CLIP-ViL p on three tasks: VQA v2.0, SNLI-VE, and GQA.
Revised: Fine-tuning We ﬁne-tune CLIP-ViL p on three tasks: VQA v2.0, SNLI-VE [Figure] and GQA.

Original: We introduce the task speciﬁcs and ﬁne-tuning hyper-parameters in the following.
Revised: We introduce the task speciﬁcs and ﬁne-tuning hyper-parameters in the following.

Original: Every example in VQA consists of an image and a question, where the task is to predict the correct answer.
Revised: Every example in VQA consists of an image and a question, where the task is to predict the correct answer.

Original: We use the Karpathy split for training and validation (Karpathy and Fei-Fei, 2015).
Revised: We use the Karpathy split for training and validation (Karpathy and Fei-Fei, 2015).

Original: We ﬁne-tune the model with the binary cross-entropy loss for 5 epoch with a batch size of 256.
Revised: We ﬁne-tune the model with the binary cross-entropy loss for 5 epoch with a batch size of 256.

Original: The
Revised: The

Original:  Transformer is optimized with AdamW and a peak learning rate of 5 × 10 − 5 .
Revised: [Table] Transformer is optimized with AdamW and a peak learning rate of 5 × 10 − 5 .

Original: The ResNet is optimized with SGD and an initial learning rate of 1 × 10 − 3 .
Revised: The ResNet is optimized with SGD and an initial learning rate of 1 × 10 − 3 .

Original: We decay the learning rate of ResNet by a factor of 10 after epoch 3.
Revised: We decay the learning rate of ResNet by a factor of 10 after epoch 3.

Original: SNLI-VE is a three-way classiﬁcation task, which involves determining the relation between an image and a sentence.
Revised: SNLI-VE is a three-way classiﬁcation task, which involves determining the relation between an image and a sentence.

Original: The three possible relations include entailment, contradiction, and neutral.
Revised: The three possible relations include entailment, contradiction, and neutral.

Original: We ﬁne-tune the model with the negative log-likelihood loss for 2 epoch with a batch size of 256.
Revised: We ﬁne-tune the model with the negative log-likelihood loss for 2 epoch with a batch size of 256.

Original: The Transformer is optimized with AdamW and a peak learning rate of 5 × 10 − 5 .
Revised: The Transformer is optimized with AdamW and a peak learning rate of 5 × 10 − 5 .

Original: The ResNet is optimized with SGD and an initial learning rate of 1 × 10 − 3 .
Revised: The ResNet is optimized with SGD and an initial learning rate of 1 × 10 − 3 .

Original: We decay the learning rate of ResNet by a factor of 10 after epoch 1.
Revised: We decay the learning rate of ResNet by a factor of 10 after epoch 1.

Original: GQA follows the format of VQA but the questions and answers of GQA are automatically generated from ground-truth scene graphs.
Revised: GQA follows the format of VQA but the questions and answers of GQA are automatically generated from ground-truth scene graphs.

Original: We use the same hyper-parameters as in VQA.
Revised: We use the same hyper-parameters as in VQA.

Original: A. M ORE Q UALITATIVE E XAMPLES
Revised: A. M ORE Q UALITATIVE E XAMPLES

Original: Here we present more qualitative examples using (Grad-CAM) (Selvaraju et al., 2017) to visualize the salient regions of CLIP models.
Revised: Here we present more qualitative examples using (Grad-CAM) (Selvaraju et al., 2017) to visualize the salient regions of CLIP models.

Original: Figure 4 and Figure 5 suggest that CLIP-Res50 localizes the sentence better than CLIP-
Revised: Figure 4 and Figure 5 suggest that CLIP-ResNet localizes the sentence better than CLIP-ViT variants.

Original: ViT-B.
Revised: For further analyzing the ﬁne-tuned CLIP text encoder, we extended the experiments in Table 8 (VQA test-dev ) without pre-training while ﬁne-tuning both visual encoder and text encoder on VQA mini-eval . The results suggest the CLIP text encoder consistently perform worse than BERT/RoBERTa counterparts even though CLIP is pre-trained with “in-domain” image-text pairs.

Revised: For directly analyzing the capability of the CLIP text encoder, we add experiments with ﬁne-tuning

Revised: CLIP text encoder on representative NLU tasks (GLUE).

Revised: On the largest two tasks (QQP, MNLI),

Revised: BERT-base (12 layer, 768 width) achieves 87.1 ± 0.2 on QQP and 77.9 ± 0.3 on MNLI.

Revised: CLIP-Res text encoder (12 layer, 512 width) achieves 72.0 ± 0.3 and 51.6 ± 0.4.

Revised: CLIP-ViT-B/32 text encoder achieves similar performance as CLIP-Res50 with the same architecture.

Revised: -Res50x encoder (12 layer, 640 width) achieves 73.8 ± 0.3 and 53.8 ± 0.3.

Revised: We conduct the experiments with 32 batch size, three epochs, three random seeds and search the learning rate in [1e-6, 1e-5, 3e-5, and 5e-5].

Revised: These results may directly reﬂect the inferior text encoder of CLIP which may be caused by the noisy and short text in the paired image-text data (Tan and Bansal, 2020).