Original: OmniVL: One Foundation Model for Image-Language and Video-Language Tasks
Revised: OmniVL: One Foundation Model for Image-Language and Video-Language Tasks

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: This paper presents OmniVL, a new foundation model to support both imagelanguage and video-language tasks using one universal architecture.
Revised: This paper presents OmniVL, a new foundation model to support both imagelanguage and video-language tasks using one universal architecture.

Original: It adopts aunified transformer-based visual encoder for both image and video inputs, and thuscan perform joint image-language and video-language pretraining.
Revised: It adopts aunified transformer-based visual encoder for both image and video inputs, and thuscan perform joint image-language and video-language pretraining.

Original: We demonstrate,for the first time, such a paradigm benefits both image and video tasks, as opposedto the conventional one-directional transfer ( e.g. , use image-language to help videolanguage).
Revised: We demonstrate,for the first time, such a paradigm benefits both image and video tasks, as opposedto the conventional one-directional transfer ( e.g. , use image-language to help videolanguage).

Original: To this end, we propose a decoupled joint pretraining of image-languageand video-language to effectively decompose the vision-language modeling intospatial and temporal dimensions and obtain performance boost on both image andvideo tasks.
Revised: To this end, we propose a decoupled joint pretraining of image-languageand video-language to effectively decompose the vision-language modeling intospatial and temporal dimensions and obtain performance boost on both image andvideo tasks.

Original: Moreover, we introduce a novel unified vision-language contrastive(UniVLC) loss to leverage image-text, video-text, image-label ( e.g. , image classification),
Revised: Moreover, we introduce a novel unified vision-language contrastive(UniVLC) loss to leverage image-text, video-text, image-label ( e.g. , image classification),

Original: video-label
Revised: video-label

Original: ( e.g. , video action recognition) data together, so that bothsupervised and noisily supervised pretraining data are utilized as much as possible.
Revised: ( e.g. , video action recognition) data together, so that bothsupervised and noisily supervised pretraining data are utilized as much as possible.

Original: Without incurring extra task-specific adaptors, OmniVL can simultaneously supportvisual only tasks ( e.g. , image classification, video action recognition), cross-modalalignment tasks ( e.g. , image/video-text retrieval), and multi-modal understandingand generation tasks ( e.g. , image/video question answering, captioning).
Revised: Without incurring extra task-specific adaptors, OmniVL can simultaneously supportvisual only tasks ( e.g. , image classification, video action recognition), cross-modalalignment tasks ( e.g. , image/video-text retrieval), and multi-modal understandingand generation tasks ( e.g. , image/video question answering, captioning).

Original: We evaluate OmniVL on a wide range of downstream tasks and achieve state-of-the-art orcompetitive results with similar model size and data scale.
Revised: We evaluate OmniVL on a wide range of downstream tasks and achieve state-of-the-art orcompetitive results with similar model size and data scale.

Original:  Introduction
Revised:  Introduction

Original: Vision-language pretraining has been demonstrated to be a promising direction for building foundationmodels that can support a broad range of downstream AI tasks.
Revised: Vision-language pretraining has been demonstrated to be a promising direction for building foundationmodels that can support a broad range of downstream AI tasks.

Original: By pretraining on web-scale noisyimage-text data, the pioneering works [48, 26, 65] suggest a unified model can be equipped withunprecedented capabilities ( e g. , zero-shot classification) and achieve outstanding performance onvarious tasks, thus significantly reducing the cost of designing task-specific models.
Revised: By pretraining on web-scale noisyimage-text data, the pioneering works [41, 22, 58] suggest a unified model can be equipped withunprecedented capabilities ( e g. , zero-shot classification) and achieve outstanding performance onvarious tasks, thus significantly reducing the cost of designing task-specific models.

Original: Following thisthread, some works [50, 55, 3, 33, 69] are further proposed to support more tasks.
Revised: Following thisthread, some works [43, 48, 2, 29] are further proposed to support more tasks.

Original: There are alsosome efforts [21, 66] studying video-language pretraining to solve video-related multi-modal tasks.
Revised: There are also someefforts [17, 59] studying video-language pretraining to solve video-related multi-modal tasks.

Original: In this paper, we take a step forward and aim to design an omni-vision-language foundation model
Revised: In this paper, we take a step forward and aim to design an omni-vision-language foundation model

Original: OmniVL, to support both image-language and video-language pretraining and corresponding downstream tasks 1 , including visual only tasks ( e.g. , image classification, video action recognition),cross-modal alignment tasks ( e.g. , image/video-text retrieval), and multi-modal understanding andgeneration tasks ( e.g. , image/video question answering, captioning) simultaneously.
Revised: OmniVL, to support both image-language and video-language pretraining and corresponding downstream tasks 1 , including visual only tasks ( e.g. , image classification, video action recognition),cross-modal alignment tasks ( e.g. , image/video-text retrieval), and multi-modal understanding andgeneration tasks ( e.g. , image/video question answering, captioning) simultaneously.

Original: To the best of ourknowledge, it is the first time to demonstrate that one model can benefit both image and video tasks
Revised: To the best of ourknowledge, it is the first time to demonstrate that one model can benefit both image and video tasks

Original: [Table]
Revised: [Table]

Original:  bidirectionally , as opposed to conventional single directional way, i.e. , using image (/image-language)to help video(/video-language).
Revised: OmniVL (Ours) (cid:33) (cid:33) (cid:33) (cid:33) (cid:33) (cid:33) (cid:33) (cid:33) bidirectionally , as opposed to conventional single directional way, i.e. , using image (/image-language)to help video(/video-language).

Original: To support both image and video inputs, OmniVL adopts a unified transformer-based visual encoderto extract visual representations, where video inputs share most transformer layers with images exceptfor the 3D patch tokenizer and temporal attention blocks [8].
Revised: To support both image and video inputs, OmniVL adopts a unified transformer-based visual encoderto extract visual representations, where video inputs share most transformer layers with images exceptfor the 3D patch tokenizer and temporal attention blocks [6].

Original: Similar to existing vision-languagemodels, OmniVL has another text encoder to extract language representations.
Revised: Similar to existing vision-languagemodels, OmniVL has another text encoder to extract language representations.

Original: To support multipletasks learning within the same architecture, OmniVL follows an encoder-decoder structure withtwo visual-grounded decoders.
Revised: To support multipletasks learning within the same architecture, OmniVL follows an encoder-decoder structure withtwo visual-grounded decoders.

Original: One decoder is designed with bidirectional attention for visual-textsemantic alignment, while the other is equipped with causal attention for text generation.
Revised: One decoder is designed with bidirectional attention for visual-textsemantic alignment, while the other is equipped with causal attention for text generation.

Original: We pretrain
Revised: We pretrain

Original: OmniVL with image-language and video-language data in a decoupled joint way, which is differentfrom existing works [33, 64, 65, 48, 66] that apply image-language only pretraining, video-languageonly pretraining or their joint pretraining from scratch.
Revised: OmniVL with image-language and video-language data in a decoupled joint way, which is differentfrom existing works [29, 57, 58, 41, 59] that apply image-language only pretraining, video-languageonly pretraining or their joint pretraining from scratch.

Original: More specifically, we first pretrain on imagelanguageto focus on spatial representation learning, and then do joint pretraining with video-languagetogether to learn the temporal dynamics incrementally while preserving/polishing the well-learnedspatial representations.
Revised: More specifically, we first pretrain on imagelanguage and then do joint pretraining with video-language together.

Original: We believe this not only makes the learning more efficient from spatial totemporal dimensions, but also enforces the learning complementary to each other.
Revised: It not only makes the learningmore efficient from spatial to temporal dimensions, but also enforces the learning complementary toeach other.

Original: This bidirectionalhelp has not been unraveled in prior works, and is important in pushing one foundation model toboost the performance on both image and video tasks.
Revised: This bidirectional help has not been unraveled in prior works, and is important in pushingone foundation model to boost the performance on both image and video tasks.

Original: Moreover, OmniVL is motivated by the unified contrastive learning [63] used in Florence [65],and extends its scope to cover video-text and video-label ( e.g. , video action recognition) data.
Revised: Moreover, OmniVL is motivated by the unified contrastive learning [56] used in Florence [58],and extends its scope to cover video-text and video-label ( e.g. , video action recognition) data.

Original: The underlying consideration lies in two aspects: 1) As mentioned above, we aim to leverageas much supervised (or noisily supervised) pretraining corpus as possible; 2) As shown in [63],human-annotated visual-label data ( e.g. , ImageNet [16]) can help to derive more discriminativerepresentations and benefit transfer learning tasks ( e.g. , image classification), while webly-crawledvision-language data cover broader visual concepts and benefit cross-modal and multi-modal tasks.
Revised: The underlying consideration lies in two aspects: 1) As mentioned above, we aim to leverageas much supervised (or noisily supervised) pretraining corpus as possible; 2) As shown in [56],human-annotated visual-label data ( e.g. , ImageNet [12]) can help to derive more discriminativerepresentations and benefit transfer learning tasks ( e.g. , image classification), while webly-crawledvision-language data cover broader visual concepts and benefit cross-modal and multi-modal tasks.

Original: This simple extension facilitates us to enjoy both advantages.
Revised: This simple extension facilitates us to enjoy both advantages.

Original: We call our foundation model OmniVL, since it unifies in three dimentions: modality ( i.e. , imagelanguage and video-language pretrainings), functionality ( i.e. , non-generative and generative tasks),and data unification ( i.e. , image-text, video-text, image-label and video-label data) as demonstratedin Table 1.
Revised: We call our foundation model OmniVL, since it unifies in three dimentions: modality ( i.e. , imagelanguage and video-language pretrainings), functionality ( i.e. , non-generative and generative tasks),and data unification ( i.e. , image-text, video-text, image-label and video-label data) as demonstratedin Table 1.

Original: With the similar model size and data scale, OmniVL achieves new state-of-the-art or atleast competitive results on a wide scope of downstream tasks.
Revised: With the similar model size and data scale, OmniVL achieves new state-of-the-art or atleast competitive results on a wide scope of downstream tasks.

Original: For example, when using ViT-Basescale model to pretrain on a moderate data scale ( e.g. , ∼ 14M image-text, ∼ 2.5M video-text), weachieve state-of-the-art performance on image-text retrieval (82.1/64.8 R@1 on COCO for image-totext/text-to-image), image captioning (39.8 BLEU@4 on COCO), text-to-video retrieval (47.8 R@1on MSRVTT), and video question answering (51.9% accuracy on MSVD).
Revised: For example, when using ViT-Basescale model to pretrain on a moderate data scale ( e.g. , ∼ 14M image-text, ∼ 2.5M video-text), weachieve state-of-the-art performance on image-text retrieval (82.1/64.8 R@1 on COCO for image-totext/text-to-image), image captioning (39.8 BLEU@4 on COCO), text-to-video retrieval (47.8 R@1on MSRVTT), and video question answering (51.9% accuracy on MSVD).

Original:  Related Work
Revised:  Related Work

Original: Vision-Only Pretraining.
Revised: Vision-Only Pretraining.

Original: Large-scale pretraining plays a key role in the success of deep neuralnetworks recently.
Revised: Large-scale pretraining plays a key role in the success of deep neuralnetworks in the past years.

Original: In the computer vision field, supervised pretraining [25, 24, 29] on ImageNet [16]is the most classical setting.
Revised: In the computer vision field, supervised pretraining [21, 20, 25] on ImageNet [12] is the most classical setting.

Original: Recently, BiT [29] shows that supervised pretraining on larger-scaledatasets with larger models offers better transfer ability.
Revised: Recently, BiT [25] shows that supervised pretraining onlarger-scale datasets with larger models offers better transfer ability.

Original: In parallel, self-supervised pretraining hasalso been extensively studied in the literature, and dominant methods include contrastive learning [12,36, 23] approaches or BERT-pretraining strategies [18, 7, 56].
Revised: In parallel, self-supervised pretraining has also been extensively studied in the literature, and dominant methods include contrastivelearning [9, 32, 19] approaches or BERT-pretraining strategies [14, 5, 49].

Original: Despite their great success, they focuson unimodal pretraining and fail to support cross-modal or multi-modal tasks.
Revised: Despite their great success,they mainly focus on unimodal pretraining and fail to support cross-modal or multi-modal tasks.

Original: Vision-Language Pretraining.
Revised: Vision-Language Pretraining.

Original: Vision-Language pretraining (VLP) [41, 53, 52, 13, 51, 48, 27] hasattracted surging attention in the vision-language community, which aims to learn generic multimodal representations to solve various tasks, e.g. , image captioning, image-text retrieval, and videoquestion answering.
Revised: Vision-Language pretraining (VLP) [37, 46, 45, 10, 44, 41, 23] hasattracted surging attention in the vision-language community, which aims to learn generic multimodal representations to solve various tasks, e.g. , image captioning, image-text retrieval, and videoquestion answering.

Original: Depending on the modality of the input data and targeted downstream tasks,existing VLP approaches can be roughly divided into two categories: image-language pretrainingmethods [13, 38, 68, 58] which learn a joint distribution over visual and linguistic representationsfrom image-text pairs, and video-language methods [35, 31, 32, 21, 6, 44, 4, 2, 46] which model thesemantic associations between video frames and texts from video-text pairs.
Revised: Depending on the modality of the input data and targeted downstream tasks,existing VLP approaches can be roughly divided into two categories: image-language pretrainingmethods [10, 34, 61, 51] which learn a joint distribution over visual and linguistic representationsfrom image-text pairs, and video-language methods [31, 27, 28, 17, 4] which model the semanticassociations between video frames and texts from video-text pairs.

Original: Among them, somerecent works [6, 21] also explore image-language and video-language joint pretraining to improvevideo-language tasks.
Revised: Among them, some recentworks [4, 17] also explore image-language and video-language joint pretraining to improve videolanguage tasks.

Original: Instead, OmniVL aims to integrate image-language and video-language withinone foundation model.
Revised: Instead, OmniVL aims to integrate image-language and video-language within onefoundation model.

Original: We introduce a decoupled joint pretraining paradigm, which first learns spatialvisual representations with image-language and then conducts joint pretraining.
Revised: We introduce a decoupled joint pretraining paradigm, which first learns spatialvisual representations with image-language and then conducts joint pretraining.

Original: With such a design,we demonstrate for the first time that they can help each other in a bidirectional way.
Revised: With such a design,we demonstrate for the first time that they can help each other in a bidirectional way.

Original: Moreover, as afoundation model, we enable more unification in terms of functionality and pretraining corpus.
Revised: Moreover, as afoundation model, we enable more unification in terms of functionality and pretraining corpus.

Original: Vision Foundation Models.
Revised: Vision Foundation Models.

Original: Automating the understanding of our multi-modal world with machinesrequires the development of foundation models that work across different modalities and domains [10,42].
Revised: Automating the understanding of our multi-modal world with machinesrequires the development of foundation models that work across different modalities and domains [7,38].

Original: CLIP [48] and ALIGN [27] are typically regarded as the pioneering explorations of foundationmodels.
Revised: CLIP [41] and ALIGN [23] are typically regarded as the pioneering explorations of foundationmodels.

Original: By pretraining on web-scale noisy image-text pair data, they excel at cross-modal alignmentand zero-shot classification tasks.
Revised: By pretraining on web-scale noisy image-text pair data, they excel at cross-modal alignmentand zero-shot classification tasks.

Original: Florence [65] further extends the scope of foundation models tocover Space-Time-Modality space and performs better especially on vision-only tasks with unifiedcontrastive learning.
Revised: Florence [58] further extends the scope of foundation models tocover Space-Time-Modality space and performs better especially on vision-only tasks with unifiedcontrastive learning.

Original: Despite their success, all the above approaches do not naturally support multimodal generation tasks ( e.g. , visual question answering and captioning).
Revised: Despite their success, all the above approaches do not naturally support multimodal generation tasks ( e.g. , visual question answering and captioning).

Original: To address this limitation,some recent works like FLAVA [50], BLIP [33] and CoCa [64] design one image-language foundationmodel to support both cross-modal alignment tasks and multi-modal generation tasks.
Revised: To address this limitation,some concurrent works like FLAVA [43], BLIP [29] and CoCa [57] design one image-languagefoundation model to support both cross-modal alignment tasks and multi-modal generation tasks.

Original: While suchimage-language foundation models can be extended to support video-language tasks in the fine-tuningstage, they either need heavy task-specific adaptors or simply treat video as independent frames.
Revised: While such image-language foundation models can be extended to support video-language tasks in thefine-tuning stage, they either need heavy task-specific adaptors or simply treat video as independentframes.

Original: Incontrast, OmniVL is designed to support both image-language and video-language starting from thepretraining stage without any extra adaptors.
Revised: In contrast, OmniVL is designed to support both image-language and video-language startingfrom the pretraining stage without any extra adaptors.

Original:  Methodology
Revised:  Methodology

Revised: Our goal is to develop one foundation model for both image-language and video language, whileunifying functionality and pretraining corpus.

Revised: In this section, we first introduce the system designin Sec 3.1, and then elaborate on the pretraining objectives in Sec 3.2.

Revised: Finally, we describe thepretraining corpus and the newly proposed decoupled joint pretraining paradigm in Sec 3.3.

Original: 3.1 Overall Framework
Revised: 3.1 Overall Framework

Original: The overall framework of OmniVL is illustrated in Figure 1, which follows an encoder-decoderlike structure.
Revised: The overall framework of OmniVL is illustrated in Figure 1, which follows an encoder-decoderlike structure.

Original: OmniVL consists of a unified visual encoder to extract the representations for bothimages and videos, a text encoder to obtain text representations, and two visual-grounded decodersfor semantic alignment and open-ended text generation, respectively.
Revised: OmniVL consists of a unified visual encoder to extract the representations for bothimages and videos, a text encoder to obtain text representations, and two visual-grounded decodersfor semantic alignment and open-ended text generation, respectively.

Original: Below we briefly introduceeach component and leave the detailed structure in the supplementary material.
Revised: Below we briefly introduceeach component and leave the detailed structure in the supplementary material.

Original: Unified Visual Encoder.
Revised: Unified Visual Encoder.

Original: We unify images and videos in a transformer-based visual encoder byconverting both of them into a series of tokens, where the independent 2D/3D convolution-basedpatch tokenizers are used for image/video respectively.
Revised: We unify images and videos in a transformer-based visual encoder byconverting both of them into a series of tokens, where the independent 2D/3D convolution-basedpatch tokenizers are used for image/video respectively.

Original: Accordingly, spatial and temporal positionalencodings are added to the input tokens to incorporate positional information.
Revised: Accordingly, spatial and temporal positionalencodings are added to the input tokens to incorporate positional information.

Original: For the transformerstructure, we follow TimeSformer [8] to employ decoupled spatial-temporal attention, which individually models the static spatial appearance and temporal dynamics in visual data.
Revised: For the transformerstructure, we follow TimeSformer [6] to employ decoupled spatial-temporal attention, which individually models the static spatial appearance and temporal dynamics in visual data.

Original: Specifically, withineach transformer block, we sequentially perform temporal self-attention and spatial self-attention. The temporal self-attention blocks will be automatically skipped for the image inputs. The finalvisual representation v cls is obtained from the [ CLS ] token of the last block. Note that we share themodel weights for image and video inputs except for the temporal self-attention.
Revised: Specifically, within

Original: Pretraining Corpus Unification
Revised: Pretraining Corpus Unification

Original: ModalityUnification
Revised: Modality

Original: [Figure]
Revised: [Figure]

Original: Visual

Original: [Figure]

Original: “A man riding on a motorcycle.”
Revised: “A man riding on a motorcycle.”

Original: / “A fireman holds a fire hose. ”
Revised: / “A fireman holds a fire hose. ”

Original: A photo of airplane .
Revised: A photo of airplane .

Original: A video footage of the skateboarding .
Revised: A video footage of the skateboarding .

Original: [Figure]
Revised: [Figure]

Original: FunctionalityUnification
Revised: Functionality

Original: [Figure]
Revised: [Figure]

Revised: each transformer block, we sequentially perform temporal self-attention and spatial self-attention.

Revised: The temporal self-attention blocks will be automatically skipped for the image inputs.

Revised: The finalvisual representation v cls is obtained from the [ CLS ] token of the last block.

Revised: Note that we share themodel weights for image and video inputs except for the temporal self-attention.

Original: Text Encoder.
Revised: Text Encoder.

Original: We adopt BERT [17] as the Text Encoder, which transforms  input text into a sequenceof token embeddings.
Revised: We adopt BERT [13] as the Text Encoder, which transforms the input text intoa sequence of token embeddings.

Original: The embedding of [ CLS ] token w cls is used as the language representation.
Revised: The embedding of [ CLS ] token w cls is used as the languagerepresentation.

Original: Visual-grounded Alignment Decoder.
Revised: Visual-grounded Alignment Decoder.

Original: Even though the above unimodal encoders can supportcross-modal alignment like CLIP [48], we employ an extra visual-grounded alignment decoder tofurther facilitate the learning and enhance the alignment accuracy like [33, 21].
Revised: Even though the above unimodal encoders can supportcross-modal alignment like CLIP [41], we employ an extra visual-grounded alignment decoder tofurther facilitate the learning and enhance the alignment accuracy like [29, 17].

Original: It takes the text andoutput visual features from the unified visual encoder as input, and fuses the information of bothmodalities with stacked transformer blocks.
Revised: It takes the text andoutput visual features from the unified visual encoder as input, and fuses the information of bothmodalities with stacked transformer blocks.

Original: Each block basically contains a self-attention layer, across-attention layer and a feed-forward layer.
Revised: Each block basically contains a self-attention layer, across-attention layer and a feed-forward layer.

Original: Additionally, a task-specific [ ENC ] token is added tothe input text, the output embedding of which will be used as the fused cross-modal representation.
Revised: Additionally, a task-specific [ ENC ] token is added tothe input text, the output embedding of which will be used as the fused cross-modal representation.

Original: Visual-grounded Generation Decoder.
Revised: Visual-grounded Generation Decoder.

Original: We empower our model to own the multi-modal generationcapability by attaching a visual-grounded text generation decoder.
Revised: We empower our model to own the multi-modal generationcapability by attaching a visual-grounded text generation decoder.

Original: It adopts the similar architectureto the above alignment decoder, but replaces the bidirectional self-attention with causal self-attention.
Revised: It adopts the similar architectureto the above alignment decoder, but replaces the bidirectional self-attention with causal self-attention.

Original: A [ DEC ] token and an [ EOS ] token are added to indicate the task type and signal the end, separately.
Revised: A [ DEC ] token and an [ EOS ] token are added to indicate the task type and signal the end, separately.

Original: 3.2 Pre-training Objectives
Revised: 3.2 Pre-training Objectives

Original: We jointly optimize OmniVL with the following three objectives:
Revised: We jointly optimize OmniVL with the following three objectives:

Original: Unified Vision-Language Contrastive (UniVLC) Loss.
Revised: Unified Vision-Language Contrastive (UniVLC) Loss.

Original: UniCL [63] introduces a novel paradigmfor visual representation learning by unifying the supervised learning from image-label data andcontrastive learning from the natural language supervision.
Revised: UniCL [56] introduces a novel paradigmfor visual representation learning by unifying the supervised learning from image-label data andcontrastive learning from the natural language supervision.

Original: In this paper, we extend its scope to theunified visual domain, which incorporates both image and video data for cross-modal pretraining viaa joint visual-label-text space.
Revised: In this paper, we extend its scope to theunified visual domain, which incorporates both image and video data for cross-modal pretraining viaa joint visual-label-text space.

Original: More specifically, we define manually-annotated image/video-label data and web-crawledimage/video-text data in a triplet format S =
Revised: More specifically, we define manually-annotated image/video-label data and web-crawledimage/video-text data in a triplet format S =

Original: ( x, y, t ) , where x ∈ X is the image/video data,y ∈ Y is the unique label indicating the index of the grouped language description in the wholepretrain dataset, and t ∈ T is its corresponding language description.
Revised: ( x, y, t ) , where x ∈ X is the image/video data,y ∈ Y is the unique label indicating the index of the grouped language description in the wholepretrain dataset, and t ∈ T is its corresponding language description.

Original: For image/video-label data, t isgenerated with the same prompt strategy used in CLIP [48] and ActionCLIP [54]( i.e. , filling the classnames into the prompt templates).
Revised: For image/video-label data, t isgenerated with the same prompt strategy used in CLIP [41] and ActionCLIP [47]( i.e. , filling the classnames into the prompt templates).

Original: Note that in this joint visual-label-text space, visual data frommanually-annotated dataset belonging to the same category shares the common textual description.
Revised: Note that in this joint visual-label-text space, visual data from themanually-annotated dataset which belong to the same category will also share the common textualdescription.

Original: Based on this, given the visual embedding of image/video x i and the language embedding of its textt i in a batch B , we follow CLIP  to apply a linear projection and normalization layer on them to obtainthe latent visual vector v i and text vector w i .
Revised: Based on this, given the visual embedding of image/video x i and the language embedding of its textt i in a batch B , we follow CLIP [41] to apply a linear projection and normalization layer on themto obtain the latent visual vector v i and text vector w i .

Original: To enjoy a large batch size for contrastive learning, wemaintain three memory banks as [23, 34] to store the most recent M visual vectors { v m } Mm =1 andtext vectors { w m } Mm =1 from the momentum encoders, and the corresponding labels { y m } Mm =1 .
Revised: To enjoy a large batch size for contrastivelearning, we maintain three memory banks as [19, 30] to store the most recent M visual vectors{ v m } Mm =1 and text vectors { w m } Mm =1 from the momentum encoders, and the corresponding labels{ y m } Mm =1 .

Original: Then we calculate the vision-to-text and text-to-vision contrastive loss as:
Revised: Then we calculate the vision-to-text and text-to-vision contrastive loss as:

Original: [Equation]
Revised: [Equation]

Original: where k ∈ P ( i ) =
Revised: where k ∈ P ( i ) =

Original: { k | k ∈ M, y k
Revised: { k | k ∈ M, y k

Original: = y i } , and τ is a learnable temperature parameter.
Revised: = y i } , and τ is a learnable temperature parameter.

Original: Finally, theunified vision-language contrastive loss is defined as:
Revised: Finally, theunified vision-language contrastive loss is defined as:

Original: [Equation]
Revised: [Equation]

Original:  where θ ve and θ te denote the parameters of the unified visual encoder and text encoder.
Revised: ) where θ ve and θ te denote the parameters of the unified visual encoder and text encoder.

Original: Vision-Language Matching (VLM) Loss.
Revised: Vision-Language Matching (VLM) Loss.

Original: VLM loss encourages the model to learn aligned visualand text representations.
Revised: VLM loss encourages the model to learn aligned visualand text representations.

Original: Specifically, we randomly replace the text t i for x i with the text t j froma different image/video in the same batch B , and input them to the unified visual encoder andvisual-grounded alignment decoder, respectively.
Revised: Specifically, we randomly replace the text t i for x i with the text t j froma different image/video in the same batch B , and input them to the unified visual encoder andvisual-grounded alignment decoder, respectively.

Original: Then a linear layer is applied to the output ofvisual-grounded alignment decoder to produce a two-category probability p vlm , which measureswhether the input pair is matched.
Revised: Then a linear layer is applied to the output ofvisual-grounded alignment decoder to produce a two-category probability p vlm , which measureswhether the input pair is matched.

Original: Finally, we optimize the parameters of the unified visual encoderθ ve and the parameters of visual-grounded alignment decoder θ ad with VLM loss:
Revised: Finally, we optimize the parameters of the unified visual encoderθ ve and the parameters of visual-grounded alignment decoder θ ad with VLM loss:

Original: [Equation]
Revised: [Equation]

Original: where y vlm = 1 if j ∈ B and y j = y i , otherwise y vlm = 0.
Revised: where y vlm = 1 if j ∈ B and y j = y i , otherwise y vlm = 0.

Original: Language Modeling (LM) Loss.
Revised: Language Modeling (LM) Loss.

Original: Previous works indicate that LM facilitates the model to developbetter text-induced generalization ability [58].
Revised: Previous works indicate that LM facilitates the model to developbetter text-induced generalization ability [51].

Original: Therefore, we optimize the output of visual-groundedgeneration decoder with a cross-entropy loss, which directly maximizes the likelihood of the inputtext sequence in an autoregressive manner:
Revised: Therefore, we optimize the output of visual-groundedgeneration decoder with a cross-entropy loss, which directly maximizes the likelihood of the inputtext sequence in an autoregressive manner:

Original: [Equation]
Revised: [Equation]

Original: where L is the length of the input text, θ ve and θ gd represent the parameters of the unified visualencoder and visual-grounded generation decoder.
Revised: where L is the length of the input text, θ ve and θ gd represent the parameters of the unified visualencoder and visual-grounded generation decoder.

Original: Combining Eqn.
Revised: Combining Eqn.

Original: 2-Eqn. 4, the overall objectives can be summarized as:
Revised: 2-Eqn. 4, the overall objectives can be summarized as:

Original: [Equation]
Revised: [Equation]

Original:  where λ 1 , λ 2 , and λ 3 are weighting hyper-parameters and all set as 1 by default.
Revised: (5) where λ 1 , λ 2 , and λ 3 are weighting hyper-parameters and all set as 1 by default.

Original: 3.3 Pretraining Corpus and Paradigms
Revised: 3.3 Pretraining Corpus and Paradigms

Original: Corpus.
Revised: Corpus.

Original: As mentioned before, our pretraining corpus includes both visual-text data and visual-labeldata, benefiting from the unified visual-label-text space.
Revised: As mentioned before, our pretraining corpus includes both visual-text data and visual-labeldata, benefiting from the unified visual-label-text space.

Original: For the image-text data, we adopt the samepre-training dataset as [34, 33] with 14M images in total by default, including two human-annotateddatasets (COCO [39] and Visual Genome [30]), and three web datasets (CC3M [49], CC12M [11],and SBU captions [45]).
Revised: For the image-text data, we adopt the samepre-training dataset as [30, 29] with 14M images in total by default, including two human-annotateddatasets (COCO [35] and Visual Genome [26]), and three web datasets (CC3M [42], CC12M [8], and SBU captions [39]).

Original: For the video-text data, we use WebVid [6] which contains 2.5M videos fromthe web.
Revised: For the video-text data, we use WebVid [4] which contains 2.5M videos fromthe web.

Original: The visual-label datasets that we adopt includes the image dataset ImageNet-1K [16] andvideo dataset Kinetics-400 [28].
Revised: The visual-label datasets that we adopt includes the image dataset ImageNet-1K [12] andvideo dataset Kinetics-400 [24].

Original: As some baseline image-language methods only use 4M image-textdata by excluding CC12M, we also provide the corresponding results in the experiment part.
Revised: As some baseline image-language methods only use 4M image-textdata by excluding CC12M, we also provide the corresponding results in the experiment part.

Original: Paradigms.
Revised: Paradigms.

Original: Different from some existing methods [6, 21] that conduct joint pretraining on image dataand video data from scratch, we adopt a decoupled joint pretraining paradigm instead.
Revised: Different from some existing methods [4, 17] that conduct joint pretraining on image dataand video data from scratch, we adopt a decoupled joint pretraining paradigm instead.

Original: Specifically, wefirst pretrain our model on image-label-text data, and then perform joint training on both image-labeltext data and video-label-text data.
Revised: Specifically, wefirst pretrain our model on image-label-text data, and then perform joint training on both image-labeltext data and video-label-text data.

Original: In this way, we decouple the multi-modal modeling into spatialand temporal dimensions.
Revised: In this way, we decouple the multi-modal modeling into spatial [Table] [Table] and temporal dimensions.

Original: Such a design has two potential benefits: 1)
Revised: Such a design has two potential benefits: 1)

Original: Considering the expensivecomputational cost of video pretraining, applying the image data to learn the spatial representationfirst is more efficient.
Revised: Considering the expensivecomputational cost of video pretraining, applying the image data to learn the spatial representationfirst is more efficient.

Original: The decoupled pattern makes the multimodal representation learning moreeffective, which is the key to make image-language and video-language benefit each other.[Table] [Table]
Revised: The decoupled pattern makes the multimodal representation learning moreeffective, which is the key to make image-language and video-language benefit each other.

Original: Experiments
Revised: Experiments

Original: Implementation Details.
Revised: Implementation Details.

Original: {+We implement our model with Pytorch and conduct pretraining with 
+}
Revised: {+We implement our model with Pytorch and conduct pretraining with 
+}

Revised: NVIDIA A100 GPUs.

Original: By default, we use the TimeSformer base model and BERT base modelfor visual encoder and text encoder, respectively.
Revised: By default, we use the TimeSformer base model and BERT base model forvisual encoder and text encoder, respectively.

Original: As mentioned in Sec 3.3, our pretraining follows adecoupled paradigm.
Revised: As mentioned in Sec 3.3, our pretraining follows adecoupled paradigm, where we first pretrain on image-language and then perform joint pretrainingwith video-language.

Original: For the image-language pretraining stage, we initialize  spatial attention with
Revised: For the image-language pretraining, we initialize the spatial attention with

Original: ViT-B/16 [19] pretrained on ImageNet-1K [16].
Revised: ViT-B/16 [15] pretrained on ImageNet-1K [12].

Original: We take random image crops of resolution 224 ×224 as inputs and apply RandAugment [15].
Revised: We take random image crops of resolution 224 ×224 as inputs and apply RandAugment [11].

Original: The model is pretrained for 20 epochs using a batch sizeof 2880.
Revised: The model is pretrained for 20 epochs using a batchsize of 2880.

Original: For the joint pretraining, we sparsely sample 8 × 224 × 224 video clips, and train the modelfor 10 epochs with a batch size of 800 for video data and 2880 for image data.
Revised: While for the joint pretraining, we sparsely sample 8 × 224 × 224 video clips, andtrain the model for 10 epochs with a batch size of 800 for video data and 2880 for image data.

Original: Our joint pretrainingalternates batches between the image and video data.
Revised: Ourjoint pretraining alternates batches between the image and video data.

Original: The model is optimized with AdamW [40]using a weight decay of 0.05.
Revised: The model is optimized with AdamW [36] using a weight decay of 0.05.

Original: The learning rate is warmed-up to 3e-4 (image) / 8e-5 (joint) anddecayed linearly with a rate of 0.85.
Revised: The learning rate is warmed-up to 3e-4 (image) / 8e-5(joint) and decayed linearly with a rate of 0.85.

Original: During downstream fine-tuning, we increase the image resolutionto 384 × 384 for both image-text and video-text tasks [34, 33], unless otherwise specified.
Revised: During downstream fine-tuning, we increase the imageresolution to 384 × 384 for both image-text and video-text tasks [30, 29], unless otherwise specified.

Original: Werandomly sample 8 frames per video for retrieval and 16 for QA, following [32].
Revised: We randomly sample 8 frames per video for retrieval and 16 for QA, following [28].

Original: Temporal positionembeddings in the spatial-temporal visual encoder are interpolated to accommodate the inputs ofdifferent lengths.
Revised: Temporalposition embeddings in the spatial-temporal visual encoder are interpolated to accommodate theinputs of different lengths.

Original: Besides, we use task-specific learning rates and training epochs due to various datascales and domains, the details of which will be further illustrated in the supplementary material.
Revised: Besides, we use task-specific learning rates and training epochs due tovarious data scales and domains, the details of which will be further illustrated in the supplementarymaterial.

Original: 4.1 Visual Only Tasks
Revised: 4.1 Visual Only Tasks

Original: We first evaluate the representations of the visual encoder on visual only tasks.
Revised: We first evaluate the representations of the visual encoder on visual only tasks.

Original: Here the classicalimage classification and video action recognition tasks are adopted for benchmarking.
Revised: Here the classicalimage classification and video action recognition tasks are adopted for benchmarking.

Original: Note that, inthe following tables, we use the superscript "*" to denote extra video data is used .
Revised: Note that, inthe following tables, we use the superscript "*" to denote extra video data is used.

Original: Image Classification.
Revised: Image Classification.

Original: Linear probing is a commonly used method to evaluate the representationquality [48, 65, 50].
Revised: Linear probing is a commonly used method to evaluate the representationquality [41, 58, 43].

Original: Following the implementation of CLIP [48], we freeze the visual encoderand fine-tune the newly appended linear layers for linear probing.
Revised: Following the implementation of CLIP [41], we freeze the visual encoderand fine-tune the newly appended linear layers for linear probing.

Original: We evaluate our method on6 image classification datasets, which are not included in our pretraining set.
Revised: We evaluate our method on6 image classification datasets, which are not included in our pretraining set.

Original: We compare with
Revised: We compare with

Original: METER [20], ALBEF [34], BLIP [33], and CLIP [48] in Table 2.
Revised: METER [16], ALBEF [30], BLIP [29], and CLIP [41] in Table 2.

Original: Compared to METER, ALBEFand BLIP, OmniVL offers consistently better results.
Revised: Compared to METER, ALBEFand BLIP, OmniVL offers consistently better results.

Original: Compared to CLIP, even though we use far lesspre-training data, our method still achieves overall comparable results.
Revised: Compared to CLIP, even though we use far lesspre-training data, our method still achieves overall comparable results.

Original: Video Action Recognition.
Revised: Video Action Recognition.

Original: Video action recognition is one of the most representative tasks forvideo understanding [70].
Revised: Video action recognition is one of the most representative tasks forvideo understanding [62].

Original: We first report the linear probing results on UCF101 and HMDB51. 
Revised: We first report the linear probing results on UCF101 and HMDB51. [Table] [Table]

Revised: [Table]

Original: The results are summarized in the first two columns of Table 3.
Revised: The results are summarized in the first two columns of Table 3.

Original: We see that OmniVL achievesexcellent performance, i.e. , 93.2% on UCF101, even without end-to-end training, which beatsbaseline methods by a large margin.
Revised: We see that OmniVL achievesexcellent performance, i.e. , 93.2% on UCF101, even without end-to-end training, which beatsbaseline methods by a large margin.

Original: Furthermore, we conduct fine-tuning experiments on Kinetics400 [28] and Something-Something v2 [22] dataset.
Revised: Furthermore, we conduct fine-tuning experiments on Kinetics400 [24] and Something-Something v2 [18] dataset.

Original: We compare our method with supervisedpretrained TimeSformer [8] and FiT [6] in Table 3 since we share the same model architecture.
Revised: We compare our method with supervisedpretrained TimeSformer [6] and FiT [4] in Table 3 since we share the same model architecture.

Original: Thesame training settings with TimeSformer are adopted for fair comparisons.
Revised: Thesame training settings with TimeSformer are adopted for fair comparisons.

Original: OmniVL outperforms
Revised: OmniVL outperforms

Original: TimeSformer by 1.4% and 5.0% on Kinetics-400 and Something-Something v2 in terms of Top1 accuracy, respectively.
Revised: TimeSformer by 1.4% and 5.0% on Kinetics-400 and Something-Something v2 in terms of Top1 accuracy, respectively.

Original: Compared to the video-language model FiT [6] ( i.e. , performing jointpretraining from scratch), our results are also overall better, highlighting the advantage of our method.[Table] [Table] [Table]
Revised: Compared to the video-language model FiT [4] ( i.e. , performing jointpretraining from scratch), our results are also overall better, highlighting the advantage of our method.

Original: [Table]

Original: [Table]

Original: [Table]

Original: 4.2 Cross-modal Alignment Tasks
Revised: 4.2 Cross-modal Alignment Tasks

Original: [Table]

Original: Using visual and text embeddings generated from the unimodal encoders, OmniVL can easily handlecross-modal alignment tasks, e.g. , image-text retrieval and text-to-video retrieval.
Revised: Using visual and text embeddings generated from the unimodal encoders, OmniVL can easily handlecross-modal alignment tasks, e.g. , image-text retrieval and text-to-video retrieval.

Original: In addition, in orderto balance the inference efficiency and the deep fusion of multi-modal information, we follow [33, 34]to first select Top-K ( K = 128 by default) candidates based on the vision-language similarity scores,which are further re-ranked by calculating their pairwise VLM scores.
Revised: In addition, in orderto balance the inference efficiency and the deep fusion of multi-modal information, we follow [29, 30]to first select Top-K ( K = 128 by default) candidates based on the vision-language similarity scores,which are further re-ranked by calculating their pairwise VLM scores.

Original: The pretrained model isfine-tuned with UniVLC loss and VLM loss.
Revised: The pretrained model isfine-tuned with UniVLC loss and VLM loss.

Original: Image-Text Retrieval.
Revised: Image-Text Retrieval.

Original: We first evaluate OmniVL on COCO [39] and Flickr30K [47] for bothimage-to-text retrieval and text-to-image retrieval.
Revised: We first evaluate OmniVL on COCO [35] and Flickr30K [40] for bothimage-to-text retrieval and text-to-image retrieval.

Original: As shown in Table 4, OmniVL outperforms othermethods by clear margins.
Revised: As shown in Table 4, OmniVL outperforms othermethods by clear margins.

Original: With 14M image-text pairs for pretraining, our model surpasses BLIP by1.8% and 0.8% in terms of average recall@1 on COCO and Flickr30K, respectively, which is evencompetitive compared with Florence that is trained with much larger scale data and model .[Table] [Table]
Revised: With 14M image-text pairs for pretraining, our model surpasses BLIP by1.8% and 0.8% in terms of average recall@1 on COCO and Flickr30K, respectively, which is evencompetitive compared with Florence that is trained with much larger scale data and model .

Original: Text-to-Video Retrieval.
Revised: Text-to-Video Retrieval.

Original: We compare OmniVL with other methods on MSRVTT [61] and
Revised: We compare OmniVL with other methods on MSRVTT [54] and

Original: DiDeMo [5] for text-to-video retrieval under both fine-tuning and zero-shot transfer settings in
Revised: DiDeMo [3] for text-to-video retrieval under both fine-tuning and zero-shot transfer settings in

Original: Table 5.
Revised: Table 5.

Original: Note that directly using the pretrained models, OmniVL achieves 42.0% and 40.6% on
Revised: Note that directly using the pretrained models, OmniVL achieves 42.0% and 40.6% on

Original: MSRVTT and DiDeMo in terms of recall@1, respectively, significantly surpassing existing methods.
Revised: MSRVTT and DiDeMo in terms of recall@1, respectively, significantly surpassing existing methods.

Original: The performance of OmniVL is further improved under the fine-tuning settings.
Revised: The performance of OmniVL is further improved under the fine-tuning settings.

Original: The results suggestthe multi-modal representations learned by our method are very discriminative. 
Revised: The results suggestthe multi-modal representations learned by our method are very discriminative. [Table] [Table] [Table]

Revised: ClipBERT [27]

Original: 4.3 Multi-modal Understanding and Generation Tasks
Revised: [Table] JustAsk [Table] [Table]

Revised: ALPRO

Revised: [Table]

Revised: [Table]

Revised: MERLOT

Revised: [Table]

Revised: [Table]

Revised: VIOLET

Original:  The visual-grounded generation decoder equips OmniVL with the capability for multi-modal understanding and generation so as to reason and describe.
Revised: [Table] [Table] 4.3 Multi-modal Understanding and Generation Tasks The visual-grounded generation decoder equips OmniVL with the capability for multi-modal understanding and generation so as to reason and describe.

Original: In this part, we fine-tune our model with the
Revised: In this part, we fine-tune our model with the

Original: LM loss, and evaluate its performance on captioning and image/video question answering tasks.
Revised: LM loss, and evaluate its performance on captioning and image/video question answering tasks.

Original: Image Captioning.
Revised: Image Captioning.

Original: Image captioning requires the model to generate a textual description for agiven image.
Revised: Image captioning requires the model to generate a textual description for agiven image.

Original: We fine-tune our model on COCO and then evaluate on both COCO [39] NoCaps [1].
Revised: We fine-tune our model on COCO and then evaluate on both COCO [35] NoCaps [1].

Original: Following [58, 33], we adopt a prefix prompt “a picture of” to guide the caption generation.
Revised: Following [51, 29], we adopt a prefix prompt “a picture of” to guide the caption generation.

Original: Theresults are presented in Table 6, from which we can see that OmniVL achieves superior results on bothdatasets, e.g. , 107.5 and 133.9 on NoCaps and COCO in terms of CIDEr.
Revised: Theresults are presented in Table 6, from which we can see that OmniVL achieves superior results onboth datasets, e.g. , 106.8 and 133.9 on NoCaps and COCO in terms of CIDEr.

Original: Although SimVLM [58]adopt much larger pretraining data than ours, OmniVL still achieves comparable or even better resultson some metrics, e.g. , CIDEr, on COCO dataset. For fair comparison with OFA [55], we pre-traintheir model on the 14M image-text data (with only image-text matching and image captioningobjectives, denoted as OFA 14 M ), the results demonstrate that using the same amount of pre-trainingdata, OmniVL performs better than OFA measured by all the metrics. Note that we don’t show theresults of SIMVLM on 14M data since they haven’t released their code.
Revised: We also show somecaptions generated by OmniVL in the supplementary material.

Original: Visual Question Answering.
Revised: Visual Question Answering.

Original: For VQA, the model is expected to predict an answer given an imageand a related question.
Revised: For VQA, the model is expected to predict an answer given an imageand a related question.

Original: To this end, we follow [34, 33] to formulate it as a generation task and focus onopen-ended VQA.
Revised: To this end, we follow [30, 29] to formulate it as a generation task and focus onopen-ended VQA.

Original: It is worth noting that during fine-tuning, we first input the image and the questioninto the unified visual encoder and visual-grounded alignment decoder separately to obtain a fusedmulti-modal representation, and then feed the visual representation and multi-modal representationinto the visual-grounded generation decoder to predict the final answer.
Revised: It is worth noting that during fine-tuning, we first input the image and the questioninto the unified visual encoder and visual-grounded alignment decoder separately to obtain a fusedmulti-modal representation, and then feed the visual representation and multi-modal representationinto the visual-grounded generation decoder to predict the final answer.

Original: We compare with existing
Revised: We compare with existing

Original: SOTA methods in Table 7.
Revised: SOTA methods in Table 7.

Original: We observe consistent performance gain compared with existing methods.
Revised: We observe consistent performance gain compared with existing methods.

Original: Especially, OmniVL even outperforms SimVLM [58] by 0.6% trained with 1.8B image-text pairs.
Revised: Especially, OmniVL even outperforms SimVLM [51] by 0.6% trained with 1.8B image-text pairs.

Original: Video Question Answering.
Revised: Video Question Answering.

Original: Table 8 summarizes the results of video question answering on
Revised: Table 8 summarizes the results of video question answering on

Original: MSRVTT-QA [59] and MSVD-QA [59].
Revised: MSRVTT-QA [52] and MSVD-QA [52].

Original: OmniVL surpasses both QA-specific methods, e.g. , JustAsk [62], and pretraining methods, e.g. , VIOLET [21] on both datasets, which validates the effectiveness of our method for complex multimodal modeling.
Revised: OmniVL surpasses both QA-specific methods, e.g. , JustAsk [55], and pretraining methods, e.g. , VIOLET [17] on both datasets, which validates the effectiveness of our method for complex multimodal modeling.

Original: 4.4 Ablation Study
Revised: 4.4 Ablation Study

Original: Decoupled Joint Pretraining.
Revised: Decoupled Joint Pretraining.

Original: To verify the effect of decoupled joint pretraining, we conductfour ablation experiments with different pretraining strategies: image-only pretraining, video-onlypretraining,  joint pretraining from scratch, and Img2Vid pretraining where we first pretrain OmniVLon image and then on video.
Revised: To verify the effect of decoupled joint pretraining, we conductthree ablation experiments with different pretraining strategies: image-only pretraining, video-onlypretraining, and joint training from scratch that is widely adopted by video-language methods [4, 17].

Original: We list some representative results of each task in Table 9 (seedatadetails and more results in the supplementary material).
Revised: We list some representative results of each task in Table 9 (see more results in the supplementarymaterial).

Original: We can see that video-only pretraining leads to significant performance degradation due to limiteddata scale for pretraining.
Revised: We can see that video-only pretraining leads to significant performance degradation due to limiteddata scale for pretraining.

Original: Comparatively, image-only pretraining is a competitive baseline, whichhowever, is still far behind decoupled joint pretraining on video tasks, e.g. , text-video-retrieval
Revised: Comparatively, image-only pretraining is a competitive baseline, whichhowever, is still far behind decoupled joint pretraining on video tasks, e.g. , text-video-retrieval

Original: [Table]

Original: MSRVTT (ret) and video question answering MSRVTT (QA).
Revised: MSRVTT (ret) and video question answering MSRVTT (QA).

Original: Compared to video-only pretraining,joint pretraining from scratch can significantly improve the performance on video tasks, which hasalso been verified in [6, 21].
Revised: Compared to video-only pretraining, [Table] joint pretraining from scratch can significantly improve the performance on video tasks, which hasalso been verified in [4, 17].

Original: However, it produces limited results on image tasks, which is even worsethan image-only pretraining. Img2Vid is another competitive baseline, whichhowever demonstratesdegraded performance on image tasks compared to image-only pretraining. This indicates the naivecombination of image-language and video-language cannot enjoy their synergy.
Revised: However, it produces limited results on image tasks, which is evenworse than image-only pretraining, which indicates the naive combination of image-language andvideo-language cannot enjoy their synergy.

Original: By contrast, our simple decoupled joint pretraining strategy can achieve better performance on bothimage and video tasks.
Revised: By contrast, our simple decoupled joint pretrainingstrategy can achieve better performance on both image and video tasks.

Original: We hypothesize this is because starting from image-only pretraining canhelp the model focus on spatial representation learning first and provides better initialization, andthus the subsequent joint pretraining would be more concentrated on learning the temporal dynamicsincrementally while preserving/polishing the well-learned spatial representations.
Revised: We hypothesize this isbecause starting from image-only pretraining can help the model focus on spatial representationlearning first and provides better initialization, and thus the subsequent joint pretraining would bemore concentrated on learning the temporal dynamics incrementally while preserving/polishing thewell-learned spatial representations.

Original: UniVLC Loss.
Revised: UniVLC Loss.

Original: We further replace the UniVLC loss with vanilla contrastive loss to study its impacton various downstream tasks.
Revised: We further replace the UniVLC loss with vanilla contrastive loss to study its impacton various downstream tasks.

Original: Note that we exclude the visual-label data from the pretraining corpusunder the “w/o UniVLC” setting.
Revised: Note that we exclude the visual-label data from the pretraining corpusunder the “w/o UniVLC” setting.

Original: The example results shown in Figure 2 illustrate that our methodperforms comparably to vanilla contrast-based model on vision-language tasks, e.g. , image/video-textretrieval, image captioning, and image/video question answering.
Revised: The example results shown in Figure 2 illustrate that our methodperforms comparably to vanilla contrast-based model on vision-language tasks, e.g. , image/video-textretrieval, image captioning, and image/video question answering.

Original: But on visual only tasks, e.g. , linearprobing for image/video classification and video action recognition fine-tuning, the performance gainis much higher, indicating that UniVLC could facilitate model to learn more discriminative visualrepresentations and benefit transfer learning tasks.
Revised: But on visual only tasks, e.g. , linearprobing for image/video classification and video action recognition fine-tuning, the performance gainis much higher, indicating that UniVLC could facilitate model to learn more discriminative visualrepresentations and benefit transfer learning tasks.

Original: [Figure]
Revised: [Figure]

Original:  Conclusionand Discussion of Broader Impact
Revised:  Conclusion

Original: In this paper, we presented OmniVL, a novel vision-language foundation model that unifies imagelanguage and video-language.
Revised: In this paper, we presented OmniVL, a novel vision-language foundation model that unifies imagelanguage and video-language.

Original: It naturally supports visual only tasks, cross-modal alignment tasks,and multi-modal understanding and generation tasks.
Revised: Without the need of extra heavy task adaptation, it naturally supportsvisual only tasks, cross-modal alignment tasks, and multi-modal understanding and generation tasks.

Original: The unified vision-language contrastive lossalso enables OmniVL to utilize image-text, image-label, video-text and video-label data together.
Revised: The unified vision-language contrastive loss also enables OmniVL to utilize image-text, imagelabel, video-text and video-label data together.

Original: Accordingly, a decoupled pretraining paradigm is introduced to decouple vision-language modelinginto spatial and temporal dimensions, which boosts the performance on both image and video tasks.
Revised: Accordingly, a decoupled pretraining paradigm isintroduced to decouple vision-language modeling into spatial and temporal dimensions, which booststhe performance on both image and video tasks.

Original: Although our model has achieved superior results on a wide range of downstream tasks, it still lacksof the commonsense reasoning capability required by some visual-language interaction tasks (e.g.,visual/video question answering).
Revised: Experimental results on a variety of vision andvision-language benchmarks validate the preeminent transferability of our approach.  Broader Impacts

Original: It also needs better architecture design to support the zero-shotcapability for visual question answering and few-shot task customization capability like GPT-3.
Revised: OmniVL pushes the boundary of foundation models by supporting more unification.

Original: Fromthe societal impact perspective, since our model is pretrained on the large-scale web-crawled datawhich may contain some toxic language or bias and it is not easy to explicitly control the modeloutput, much attention should be paid to ensure responsible model deployment.
Revised: Although ourpretrained models have achieved superior results on a wide range of downstream tasks, it requiresa significant amount of computational resources for training. We leave it as future work to explorelightweight models and advanced training strategies. In addition, the potential negative social impactof the foundation models also needs to be further studied.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [No] We will release the code after the paper is accepted
Revised: [Yes]

Original: (b) Did you specify all the training details ( e.g. , data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details ( e.g. , data splits, hyperparameters, how theywere chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report 
Revised: (c) Did you report error bars ( e

Original: error bars ( e.g. , with respect to the random seed after running experiments multiple times)?
Revised: g. , with respect to the random seed after running experiments multiple times)?

Original: [No] Our method is stable
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used ( e g. , type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used ( e.g. , typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets ( e g. , code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets ( e.g. , code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: [Table]

Original: ) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No]
Revised: [Yes]

Original: [Table]

Original: ) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: [Table]

Original: ) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating?
Revised: using/curating?

Original: [No]
Revised: [Yes]

Original: [Table] ) Did you discuss whether the data you are using/curating contains personally identifiable[Table]
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [No]
Revised: [Yes]

Original: [Table]