Original: OmniVL: One Foundation Model for Image-Language and Video-Language Tasks
Revised: OmniVL: One Foundation Model for Image-Language and Video-Language Tasks

Original: Abstract
Revised: Abstract

Original: This paper presents OmniVL, a new foundation model to support both imagelanguage and video-language tasks using one universal architecture.
Revised: This paper presents OmniVL, a new foundation model to support both imagelanguage and video-language tasks using one universal architecture.

Original: It adopts a unified transformer-based visual encoder for both image and video inputs, and thus can perform joint image-language and video-language pretraining.
Revised: It adopts a unified transformer-based visual encoder for both image and video inputs, and thus can perform joint image-language and video-language pretraining.

Original: We demonstrate, for the first time, such a paradigm benefits both image and video tasks, as opposed to the conventional one-directional transfer ( e.g. , use image-language to help videolanguage).
Revised: We demonstrate, for the first time, such a paradigm benefits both image and video tasks, as opposed to the conventional one-directional transfer ( e.g. , use image-language to help videolanguage).

Original: To this end, we propose a decoupled joint pretraining of image-language and video-language to effectively decompose the vision-language modeling into spatial and temporal dimensions and obtain performance boost on both image and video tasks.
Revised: To this end, we propose a decoupled joint pretraining of image-language and video-language to effectively decompose the vision-language modeling into spatial and temporal dimensions and obtain performance boost on both image and video tasks.

Original: Moreover, we introduce a novel unified vision-language contrastive (UniVLC) loss to leverage image-text, video-text, image-label ( e.g. , image classification), video-label ( e.g. , video action recognition) data together, so that both supervised and noisily supervised pretraining data are utilized as much as possible.
Revised: Moreover, we introduce a novel unified vision-language contrastive (UniVLC) loss to leverage image-text, video-text, image-label ( e.g. , image classification), video-label ( e.g. , video action recognition) data together, so that both supervised and noisily supervised pretraining data are utilized as much as possible.

Original: Without incurring extra task-specific adaptors, OmniVL can simultaneously support visual only tasks ( e.g. , image classification, video action recognition), cross-modal alignment tasks ( e.g. , image/video-text retrieval), and multi-modal understanding and generation tasks ( e.g. , image/video question answering, captioning).
Revised: Without incurring extra task-specific adaptors, OmniVL can simultaneously support visual only tasks ( e.g. , image classification, video action recognition), cross-modal alignment tasks ( e.g. , image/video-text retrieval), and multi-modal understanding and generation tasks ( e.g. , image/video question answering, captioning).

Original: We evaluate OmniVL on a wide range of downstream tasks and achieve state-of-the-art or competitive results with similar model size and data scale.
Revised: We evaluate OmniVL on a wide range of downstream tasks and achieve state-of-the-art or competitive results with similar model size and data scale.

Original:  Introduction
Revised:  Introduction

Original: Vision-language pretraining has been demonstrated to be a promising direction for building foundation models that can support a broad range of downstream AI tasks.
Revised: Vision-language pretraining has been demonstrated to be a promising direction for building foundation models that can support a broad range of downstream AI tasks.

Original: By pretraining on web-scale noisy image-text data, the pioneering works [53, 29, 71] suggest a unified model can be equipped with unprecedented capabilities ( e.g. , zero-shot classification) and achieve outstanding performance on various tasks, thus significantly reducing the cost of designing task-specific models.
Revised: By pretraining on web-scale noisy image-text data, the pioneering works [53, 29, 71] suggest a unified model can be equipped with unprecedented capabilities ( e.g. , zero-shot classification) and achieve outstanding performance on various tasks, thus significantly reducing the cost of designing task-specific models.

Original: Following this thread, some works [55, 60, 3, 37, 79] are further proposed to support more tasks.
Revised: Following this thread, some works [55, 60, 3, 37, 79] are further proposed to support more tasks.

Original: There are also some efforts [23, 72] studying video-language pretraining to solve video-related multi-modal tasks.
Revised: There are also some efforts [23, 72] studying video-language pretraining to solve video-related multi-modal tasks.

Original: In this paper, we take a step forward and aim to design an omni-vision-language foundation model
Revised: In this paper, we take a step forward and aim to design an omni-vision-language foundation model

Original: OmniVL, to support both image-language and video-language pretraining and corresponding downstream tasks 1 , including visual only tasks ( e.g. , image classification, video action recognition),
Revised: OmniVL, to support both image-language and video-language pretraining and corresponding downstream tasks 1 , including visual only tasks ( e.g. , image classification, video action recognition),

Original: [Table]
Revised: [Table]

Original: cross-modal alignment tasks ( e.g. , image/video-text retrieval), and multi-modal understanding and generation tasks ( e.g. , image/video question answering, captioning) simultaneously.
Revised: cross-modal alignment tasks ( e.g. , image/video-text retrieval), and multi-modal understanding and generation tasks ( e.g. , image/video question answering, captioning) simultaneously.

Original: To the best of our knowledge, it is the first time to demonstrate that one model can benefit both image and video tasks bidirectionally , as opposed to conventional single directional way, i.e. , using image (/image-language) to help video(/video-language).
Revised: To the best of our knowledge, it is the first time to demonstrate that one model can benefit both image and video tasks bidirectionally , as opposed to conventional single directional way, i.e. , using image (/image-language) to help video(/video-language).

Original: To support both image and video inputs, OmniVL adopts a unified transformer-based visual encoder to extract visual representations, where video inputs share most transformer layers with images except for the 3D patch tokenizer and temporal attention blocks [8].
Revised: To support both image and video inputs, OmniVL adopts a unified transformer-based visual encoder to extract visual representations, where video inputs share most transformer layers with images except for the 3D patch tokenizer and temporal attention blocks [8].

Original: Similar to existing vision-language models, OmniVL has another text encoder to extract language representations.
Revised: Similar to existing vision-language models, OmniVL has another text encoder to extract language representations.

Original: To support multiple tasks learning within the same architecture, OmniVL follows an encoder-decoder structure with two visual-grounded decoders.
Revised: To support multiple tasks learning within the same architecture, OmniVL follows an encoder-decoder structure with two visual-grounded decoders.

Original: One decoder is designed with bidirectional attention for visual-text semantic alignment, while the other is equipped with causal attention for text generation.
Revised: One decoder is designed with bidirectional attention for visual-text semantic alignment, while the other is equipped with causal attention for text generation.

Original: We pretrain
Revised: We pretrain

Original: OmniVL with image-language and video-language data in a decoupled joint way, which is different from existing works [37, 70, 71, 53, 72] that apply image-language only pretraining, video-language only pretraining or their joint pretraining from scratch.
Revised: OmniVL with image-language and video-language data in a decoupled joint way, which is different from existing works [37, 70, 71, 53, 72] that apply image-language only pretraining, video-language only pretraining or their joint pretraining from scratch.

Original: More specifically, we first pretrain on imagelanguage to focus on spatial representation learning, and then do joint pretraining with video-language together to learn the temporal dynamics incrementally while preserving/polishing the well-learned spatial representations.
Revised: More specifically, we first pretrain on imagelanguage to focus on spatial representation learning, and then do joint pretraining with video-language together to learn the temporal dynamics incrementally while preserving/polishing the well-learned spatial representations.

Original: We believe this not only makes the learning more efficient from spatial to temporal dimensions, but also enforces the learning complementary to each other.
Revised: We believe this not only makes the learning more efficient from spatial to temporal dimensions, but also enforces the learning complementary to each other.

Original: This bidirectional help has not been unraveled in prior works, and is important in pushing one foundation model to boost the performance on both image and video tasks.
Revised: This bidirectional help has not been unraveled in prior works, and is important in pushing one foundation model to boost the performance on both image and video tasks.

Original: Moreover, OmniVL is motivated by the unified contrastive learning [69] used in Florence [71], and extends its scope to cover video-text and video-label ( e.g. , video action recognition) data.
Revised: Moreover, OmniVL is motivated by the unified contrastive learning [69] used in Florence [71], and extends its scope to cover video-text and video-label ( e.g. , video action recognition) data.

Original: The underlying consideration lies in two aspects: 1) As mentioned above, we aim to leverage as much supervised (or noisily supervised) pretraining corpus as possible; 2) As shown in [69], human-annotated visual-label data ( e.g. , ImageNet [16]) can help to derive more discriminative representations and benefit transfer learning tasks ( e.g. , image classification), while webly-crawled vision-language data cover broader visual concepts and benefit cross-modal and multi-modal tasks.
Revised: The underlying consideration lies in two aspects: 1) As mentioned above, we aim to leverage as much supervised (or noisily supervised) pretraining corpus as possible; 2) As shown in [69], human-annotated visual-label data ( e.g. , ImageNet [16]) can help to derive more discriminative representations and benefit transfer learning tasks ( e.g. , image classification), while webly-crawled vision-language data cover broader visual concepts and benefit cross-modal and multi-modal tasks.

Original: This simple extension facilitates us to enjoy both advantages.
Revised: This simple extension facilitates us to enjoy both advantages.

Original: We call our foundation model OmniVL, since it unifies in three dimensions: modality ( i.e. , imagelanguage and video-language pretrainings), functionality ( i.e. , non-generative and generative tasks), and data unification ( i.e. , image-text, video-text, image-label and video-label data) as demonstrated in Table 1.
Revised: We call our foundation model OmniVL, since it unifies in three dimensions: modality ( i.e. , imagelanguage and video-language pretrainings), functionality ( i.e. , non-generative and generative tasks), and data unification ( i.e. , image-text, video-text, image-label and video-label data) as demonstrated in Table 1.

Original: With the similar model size and data scale, OmniVL achieves new state-of-the-art or at least competitive results on a wide scope of downstream tasks.
Revised: With the similar model size and data scale, OmniVL achieves new state-of-the-art or at least competitive results on a wide scope of downstream tasks.

Original: For example, when using ViT-Base scale model to pretrain on a moderate data scale ( e.g. , ∼ 14M image-text, ∼ 2.5M video-text), we achieve state-of-the-art performance on image-text retrieval (82.1/64.8 R@1 on COCO for image-totext/text-to-image), image captioning (39.8 BLEU@4 on COCO), text-to-video retrieval (47.8 R@ on MSRVTT), and video question answering (51.9% accuracy on MSVD).
Revised: For example, when using ViT-Base scale model to pretrain on a moderate data scale ( e.g. , ∼ 14M image-text, ∼ 2.5M video-text), we achieve state-of-the-art performance on image-text retrieval (82.1/64.8 R@1 on COCO for image-totext/text-to-image), image captioning (39.8 BLEU@4 on COCO), text-to-video retrieval (47.8 R@ on MSRVTT), and video question answering (51.9% accuracy on MSVD).

Original:  Related Work
Revised:  Related Work

Original: Vision-Only Pretraining.
Revised: Vision-Only Pretraining.

Original: Large-scale pretraining plays a key role in the success of deep neural networks recently.
Revised: Large-scale pretraining plays a key role in the success of deep neural networks recently.

Original: In the computer vision field, supervised pretraining [27, 26, 33, 18] on ImageNet [16] is the most classical setting.
Revised: In the computer vision field, supervised pretraining [27, 26, 33, 18] on ImageNet [16] is the most classical setting.

Original: Recently, BiT [33] shows that supervised pretraining on larger-scale datasets with larger models offers better transfer ability.
Revised: Recently, BiT [33] shows that supervised pretraining on larger-scale datasets with larger models offers better transfer ability.

Original: In parallel, self-supervised pretraining has also been extensively studied in the literature, and dominant methods include contrastive learning [12, 40, 25] approaches or BERT-pretraining strategies [19, 7, 61, 20].
Revised: In parallel, self-supervised pretraining has also been extensively studied in the literature, and dominant methods include contrastive learning [12, 40, 25] approaches or BERT-pretraining strategies [19, 7, 61, 20].

Original: Despite their great success, they focus on unimodal pretraining and fail to support cross-modal or multi-modal tasks.
Revised: Despite their great success, they focus on unimodal pretraining and fail to support cross-modal or multi-modal tasks.

Original: Vision-Language Pretraining.
Revised: Vision-Language Pretraining.

Original: Vision-Language pretraining (VLP) [45, 58, 57, 13, 56, 53, 30, 31] has attracted surging attention in the vision-language community, which aims to learn generic multimodal representations to solve various tasks, e.g. , image captioning, image-text retrieval, and video question answering.
Revised: Vision-Language pretraining (VLP) [45, 58, 57, 13, 56, 53, 30, 31] has attracted surging attention in the vision-language community, which aims to learn generic multimodal representations to solve various tasks, e.g. , image captioning, image-text retrieval, and video question answering.

Original: Depending on the modality of the input data and targeted downstream tasks, existing VLP approaches can be roughly divided into two categories: image-language pretraining methods [13, 42, 75, 63] which learn a joint distribution over visual and linguistic representations from image-text pairs, and video-language methods [39, 35, 36, 23, 6, 49, 4, 2, 51] which model the semantic associations between video frames and texts from video-text pairs.
Revised: Depending on the modality of the input data and targeted downstream tasks, existing VLP approaches can be roughly divided into two categories: image-language pretraining methods [13, 42, 75, 63] which learn a joint distribution over visual and linguistic representations from image-text pairs, and video-language methods [39, 35, 36, 23, 6, 49, 4, 2, 51] which model the semantic associations between video frames and texts from video-text pairs.

Original: Among them, some recent works [6, 23] also explore image-language and video-language joint pretraining to improve video-language tasks.
Revised: Among them, some recent works [6, 23] also explore image-language and video-language joint pretraining to improve video-language tasks.

Original: Instead, OmniVL aims to integrate image-language and video-language within one foundation model.
Revised: Instead, OmniVL aims to integrate image-language and video-language within one foundation model.

Original: Moreover, inspired by the observation that decoupling spatial and temporal learning is better than direct joint spatial-temporal learning [61, 73], we introduce a decoupled joint pretraining paradigm, which first learns spatial visual representations with image-language and then conducts joint pretraining.
Revised: Moreover, inspired by the observation that decoupling spatial and temporal learning is better than direct joint spatial-temporal learning [61, 73], we introduce a decoupled joint pretraining paradigm, which first learns spatial visual representations with image-language and then conducts joint pretraining.

Original: With such a design, we demonstrate for the first time that they can help each other in a bidirectional way.
Revised: With such a design, we demonstrate for the first time that they can help each other in a bidirectional way.

Original: Moreover, as a foundation model, we enable more unification in terms of functionality and pretraining corpus.
Revised: Moreover, as a foundation model, we enable more unification in terms of functionality and pretraining corpus.

Original: Vision Foundation Models.
Revised: Vision Foundation Models.

Original: Automating the understanding of our multi-modal world with machines requires the development of foundation models that work across different modalities and domains [10, 46].
Revised: Automating the understanding of our multi-modal world with machines requires the development of foundation models that work across different modalities and domains [10, 46].

Original: CLIP [53] and ALIGN [30] are typically regarded as the pioneering explorations of foundation models.
Revised: CLIP [53] and ALIGN [30] are typically regarded as the pioneering explorations of foundation models.

Original: By pretraining on web-scale noisy image-text pair data, they excel at cross-modal alignment and zero-shot classification tasks.
Revised: By pretraining on web-scale noisy image-text pair data, they excel at cross-modal alignment and zero-shot classification tasks.

Original: Florence [71] further extends the scope of foundation models to cover Space-Time-Modality space and performs better especially on vision-only tasks with unified contrastive learning.
Revised: Florence [71] further extends the scope of foundation models to cover Space-Time-Modality space and performs better especially on vision-only tasks with unified contrastive learning.

Original: Despite their success, all the above approaches do not naturally support multimodal generation tasks ( e.g. , visual question answering and captioning).
Revised: Despite their success, all the above approaches do not naturally support multimodal generation tasks ( e.g. , visual question answering and captioning).

Original: To address this limitation, some recent works like FLAVA [55], BLIP [37] and CoCa [70] design one image-language foundation model to support both cross-modal alignment tasks and multi-modal generation tasks.
Revised: To address this limitation, some recent works like FLAVA [55], BLIP [37] and CoCa [70] design one image-language foundation model to support both cross-modal alignment tasks and multi-modal generation tasks.

Original: While such image-language foundation models can be extended to support video-language tasks in the fine-tuning stage, they either need heavy task-specific adaptors or simply treat video as independent frames.
Revised: While such image-language foundation models can be extended to support video-language tasks in the fine-tuning stage, they either need heavy task-specific adaptors or simply treat video as independent frames.

Original: In contrast, OmniVL is designed to support both image-language and video-language starting from the pretraining stage without any extra adaptors.
Revised: In contrast, OmniVL is designed to support both image-language and video-language starting from the pretraining stage without any extra adaptors.

Original:  Methodology 3.1 Overall Framework
Revised:  Methodology 3.1 Overall Framework

Original: The overall framework of OmniVL is illustrated in Figure 1, which follows an encoder-decoder like structure.
Revised: The overall framework of OmniVL is illustrated in Figure 1, which follows an encoder-decoder like structure.

Original: OmniVL consists of a unified visual encoder to extract the representations for both images and videos, a text encoder to obtain text representations, and two visual-grounded decoders for semantic alignment and open-ended text generation, respectively.
Revised: OmniVL consists of a unified visual encoder to extract the representations for both images and videos, a text encoder to obtain text representations, and two visual-grounded decoders for semantic alignment and open-ended text generation, respectively.

Original: Below we briefly introduce each component and leave the detailed structure in the supplementary material.
Revised: Below we briefly introduce each component and leave the detailed structure in the supplementary material.

Original: Pretraining Corpus Unification
Revised: Pretraining Corpus Unification

Original: Modality Unification
Revised: Modality Unification

Original: Unified Visual Encoder
Revised: Unified Visual Encoder

Original: Text Encoder
Revised: Text Encoder

Original: Visual-grounded Alignment Decoder
Revised: Visual-grounded Alignment Decoder

Original: Visual-grounded Generation Decoder “A man riding on a motorcycle.”
Revised: Visual-grounded Generation Decoder “A man riding on a motorcycle.”

Original: / “A fireman holds a fire hose. ”
Revised: / “A fireman holds a fire hose. ”

Original: A photo of airplane .
Revised: A photo of airplane .

Original: A video footage of the skateboarding .
Revised: A video footage of the skateboarding .

Original: Q:
Revised: Q:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: :Two
Revised: :Two

Original: Q:
Revised: Q:

Original: [Figure]
Revised: [Figure]

Original: holds
Revised: holds

Original: [Figure]
Revised: [Figure]

Original: A: a man
Revised: A: a man

Original: Multi-modal Understanding/Generation
Revised: Multi-modal Understanding/Generation

Original: Functionality Unification
Revised: Functionality Unification

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Image
Revised: Image

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: -Label
Revised: -Label

Original: [Figure]
Revised: [Figure]

Original: playing basketball
Revised: playing basketball

Original: Visual
Revised: Visual

Original: [Figure]
Revised: [Figure]

Original: (e.g., classification)
Revised: (e.g., classification)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Unified Visual Encoder.
Revised: Unified Visual Encoder.

Original: We unify images and videos in a transformer-based visual encoder by converting both of them into a series of tokens, where the independent 2D/3D convolution-based patch tokenizers are used for image/video respectively.
Revised: We unify images and videos in a transformer-based visual encoder by converting both of them into a series of tokens, where the independent 2D/3D convolution-based patch tokenizers are used for image/video respectively.

Original: Accordingly, spatial and temporal positional encodings are added to the input tokens to incorporate positional information.
Revised: Accordingly, spatial and temporal positional encodings are added to the input tokens to incorporate positional information.

Original: For the transformer structure, we follow TimeSformer [8] to employ decoupled spatial-temporal attention, which individually models the static spatial appearance and temporal dynamics in visual data.
Revised: For the transformer structure, we follow TimeSformer [8] to employ decoupled spatial-temporal attention, which individually models the static spatial appearance and temporal dynamics in visual data.

Original: Specifically, within each transformer block, we sequentially perform temporal self-attention and spatial self-attention.
Revised: Specifically, within each transformer block, we sequentially perform temporal self-attention and spatial self-attention.

Original: The temporal self-attention blocks will be automatically skipped for the image inputs.
Revised: The temporal self-attention blocks will be automatically skipped for the image inputs.

Original: The final visual representation v cls is obtained from the [ CLS ] token of the last block.
Revised: The final visual representation v cls is obtained from the [ CLS ] token of the last block.

Original: Note that we share the model weights for image and video inputs except for the temporal self-attention.
Revised: Note that we share the model weights for image and video inputs except for the temporal self-attention.

Original: Text Encoder.
Revised: Text Encoder.

Original: We adopt BERT [17] as the Text Encoder, which transforms input text into a sequence of token embeddings.
Revised: We adopt BERT [17] as the Text Encoder, which transforms input text into a sequence of token embeddings.

Original: The embedding of [ CLS ] token w cls is used as the language representation.
Revised: The embedding of [ CLS ] token w cls is used as the language representation.

Original: Visual-grounded Alignment Decoder.
Revised: Visual-grounded Alignment Decoder.

Original: Even though the above unimodal encoders can support cross-modal alignment like CLIP [53], we employ an extra visual-grounded alignment decoder to further facilitate the learning and enhance the alignment accuracy like [37, 23].
Revised: Even though the above unimodal encoders can support cross-modal alignment like CLIP [53], we employ an extra visual-grounded alignment decoder to further facilitate the learning and enhance the alignment accuracy like [37, 23].

Original: It takes the text and output visual features from the unified visual encoder as input, and fuses the information of both modalities with stacked transformer blocks.
Revised: It takes the text and output visual features from the unified visual encoder as input, and fuses the information of both modalities with stacked transformer blocks.

Original: Each block basically contains a self-attention layer, a cross-attention layer and a feed-forward layer.
Revised: Each block basically contains a self-attention layer, a cross-attention layer and a feed-forward layer.

Original: Additionally, a task-specific [ ENC ] token is added to the input text, the output embedding of which will be used as the fused cross-modal representation.
Revised: Additionally, a task-specific [ ENC ] token is added to the input text, the output embedding of which will be used as the fused cross-modal representation.

Original: Visual-grounded Generation Decoder.
Revised: Visual-grounded Generation Decoder.

Original: We empower our model to own the multi-modal generation capability by attaching a visual-grounded text generation decoder.
Revised: We empower our model to own the multi-modal generation capability by attaching a visual-grounded text generation decoder.

Original: It adopts the similar architecture to the above alignment decoder, but replaces the bidirectional self-attention with causal self-attention.
Revised: It adopts the similar architecture to the above alignment decoder, but replaces the bidirectional self-attention with causal self-attention.

Original: A [ DEC ] token and an [ EOS ] token are added to indicate the task type and signal the end, separately.
Revised: A [ DEC ] token and an [ EOS ] token are added to indicate the task type and signal the end, separately.

Original: 3.2 Pre-training Objectives
Revised: 3.2 Pre-training Objectives

Original: We jointly optimize OmniVL with the following three objectives:
Revised: We jointly optimize OmniVL with the following three objectives:

Original: Unified Vision-Language Contrastive (UniVLC) Loss.
Revised: Unified Vision-Language Contrastive (UniVLC) Loss.

Original: UniCL [69] introduces a novel paradigm for visual representation learning by unifying the supervised learning from image-label data and contrastive learning from the natural language supervision.
Revised: UniCL [69] introduces a novel paradigm for visual representation learning by unifying the supervised learning from image-label data and contrastive learning from the natural language supervision.

Original: In this paper, we extend its scope to the unified visual domain, which incorporates both image and video data for cross-modal pretraining via a joint visual-label-text space.
Revised: In this paper, we extend its scope to the unified visual domain, which incorporates both image and video data for cross-modal pretraining via a joint visual-label-text space.

Original: More specifically, we define manually-annotated image/video-label data and web-crawled image/video-text data in a triplet format S =
Revised: More specifically, we define manually-annotated image/video-label data and web-crawled image/video-text data in a triplet format S =

Original: ( x, y, t ) , where x ∈ X is the image/video data, y ∈ Y is the unique label indicating the index of the grouped language description in the whole pretrain dataset, and t ∈ T is its corresponding language description.
Revised: ( x, y, t ) , where x ∈ X is the image/video data, y ∈ Y is the unique label indicating the index of the grouped language description in the whole pretrain dataset, and t ∈ T is its corresponding language description.

Original: For image/video-label data, t is generated with the same prompt strategy used in CLIP [53] and ActionCLIP [59]( i.e. , filling the class names into the prompt templates).
Revised: For image/video-label data, t is generated with the same prompt strategy used in CLIP [53] and ActionCLIP [59]( i.e. , filling the class names into the prompt templates).

Original: Note that in this joint visual-label-text space, visual data from manually-annotated dataset belonging to the same category shares the common textual description.
Revised: Note that in this joint visual-label-text space, visual data from manually-annotated dataset belonging to the same category shares the common textual description.

Original: Based on this, given the visual embedding of image/video x i and the language embedding of its text t i in a batch B , we follow CLIP to apply a linear projection and normalization layer on them to obtain the latent visual vector v i and text vector w i .
Revised: Based on this, given the visual embedding of image/video x i and the language embedding of its text t i in a batch B , we follow CLIP to apply a linear projection and normalization layer on them to obtain the latent visual vector v i and text vector w i .

Original: To enjoy a large batch size for contrastive learning, we maintain three memory banks as [25, 38] to store the most recent M visual vectors { v m } Mm =1 and text vectors { w m } Mm =1 from the momentum encoders, and the corresponding labels { y m } Mm =1 .
Revised: To enjoy a large batch size for contrastive learning, we maintain three memory banks as [25, 38] to store the most recent M visual vectors { v m } Mm =1 and text vectors { w m } Mm =1 from the momentum encoders, and the corresponding labels { y m } Mm =1 .

Original: Then we calculate the vision-to-text and text-to-vision contrastive loss as:
Revised: Then we calculate the vision-to-text and text-to-vision contrastive loss as:

Original: [Equation]
Revised: [Equation]

Original: where k ∈ P ( i ) =
Revised: where k ∈ P ( i ) =

Original: { k | k ∈ M, y k
Revised: { k | k ∈ M, y k

Original: = y i } , and τ is a learnable temperature parameter.
Revised: = y i } , and τ is a learnable temperature parameter.

Original: Finally, the unified vision-language contrastive loss is defined as:
Revised: Finally, the unified vision-language contrastive loss is defined as:

Original: [Equation]
Revised: [Equation]

Original: where θ ve and θ te denote the parameters of the unified visual encoder and text encoder.
Revised: where θ ve and θ te denote the parameters of the unified visual encoder and text encoder.

Original: Vision-Language Matching (VLM) Loss.
Revised: Vision-Language Matching (VLM) Loss.

Original: VLM loss encourages the model to learn aligned visual and text representations.
Revised: VLM loss encourages the model to learn aligned visual and text representations.

Original: Specifically, we randomly replace the text t i for x i with the text t j from a different image/video in the same batch B , and input them to the unified visual encoder and visual-grounded alignment decoder, respectively.
Revised: Specifically, we randomly replace the text t i for x i with the text t j from a different image/video in the same batch B , and input them to the unified visual encoder and visual-grounded alignment decoder, respectively.

Original: Then a linear layer is applied to the output of visual-grounded alignment decoder to produce a two-category probability p vlm , which measures whether the input pair is matched.
Revised: Then a linear layer is applied to the output of visual-grounded alignment decoder to produce a two-category probability p vlm , which measures whether the input pair is matched.

Original: Finally, we optimize the parameters of the unified visual encoder θ ve and the parameters of visual-grounded alignment decoder θ ad with VLM loss:
Revised: Finally, we optimize the parameters of the unified visual encoder θ ve and the parameters of visual-grounded alignment decoder θ ad with VLM loss:

Original: [Equation]
Revised: [Equation]

Original: where y vlm = 1 if j ∈ B and y j = y i , otherwise y vlm = 0.
Revised: where y vlm = 1 if j ∈ B and y j = y i , otherwise y vlm = 0.

Original: Language Modeling (LM) Loss.
Revised: Language Modeling (LM) Loss.

Original: Previous works indicate that LM facilitates the model to develop better text-induced generalization ability [63].
Revised: Previous works indicate that LM facilitates the model to develop better text-induced generalization ability [63].

Original: Therefore, we optimize the output of visual-grounded generation decoder with a cross-entropy loss, which directly maximizes the likelihood of the input text sequence in an autoregressive manner:
Revised: Therefore, we optimize the output of visual-grounded generation decoder with a cross-entropy loss, which directly maximizes the likelihood of the input text sequence in an autoregressive manner:

Original: [Equation]
Revised: [Equation]

Original: where L is the length of the input text, θ ve and θ gd represent the parameters of the unified visual encoder and visual-grounded generation decoder.
Revised: where L is the length of the input text, θ ve and θ gd represent the parameters of the unified visual encoder and visual-grounded generation decoder.

Original: Combining Eqn.
Revised: Combining Eqn.

Original: 2-Eqn. 4, the overall objectives can be summarized as:
Revised: 2-Eqn. 4, the overall objectives can be summarized as:

Original: [Equation]
Revised: [Equation]

Original: where λ 1 , λ 2 , and λ 3 are weighting hyper-parameters and all set as 1 by default.
Revised: where λ 1 , λ 2 , and λ 3 are weighting hyper-parameters and all set as 1 by default.

Original: 3 Pretraining Corpus and Paradigms
Revised: 3 Pretraining Corpus and Paradigms

Original: Corpus.
Revised: Corpus.

Original: As mentioned before, our pretraining corpus includes both visual-text data and visual-label data, benefiting from the unified visual-label-text space.
Revised: As mentioned before, our pretraining corpus includes both visual-text data and visual-label data, benefiting from the unified visual-label-text space.

Original: For the image-text data, we adopt the same pre-training dataset as [38, 37] with 14M images in total by default, including two human-annotated datasets (COCO [43] and Visual Genome [34]), and three web datasets (CC3M [54], CC12M [11], and SBU captions [50]).
Revised: For the image-text data, we adopt the same pre-training dataset as [38, 37] with 14M images in total by default, including two human-annotated datasets (COCO [43] and Visual Genome [34]), and three web datasets (CC3M [54], CC12M [11], and SBU captions [50]).

Original: For the video-text data, we use WebVid [6] which contains 2.5M videos from the web.
Revised: For the video-text data, we use WebVid [6] which contains 2.5M videos from the web.

Original: The visual-label datasets that we adopt includes the image dataset ImageNet-1K [16] and video dataset Kinetics-400 [32].
Revised: The visual-label datasets that we adopt includes the image dataset ImageNet-1K [16] and video dataset Kinetics-400 [32].

Original: As some baseline image-language methods only use 4M image-text data by excluding CC12M, we also provide the corresponding results in the experiment part.
Revised: As some baseline image-language methods only use 4M image-text data by excluding CC12M, we also provide the corresponding results in the experiment part.

Original: [Table]
Revised: [Table]

Original: Paradigms.
Revised: Paradigms.

Original: In contrast to some existing methods [6, 23] that conduct joint pretraining on image data and video data from scratch
Revised: In contrast to some existing methods [6, 23] that conduct joint pretraining on image data and video data from scratch

Original: [Table]
Revised: [Table]

Original: we adopt a decoupled joint pretraining paradigm instead.
Revised: we adopt a decoupled joint pretraining paradigm instead.

Original: Specifically, we first pretrain our model on image-label-text data
Revised: Specifically, we first pretrain our model on image-label-text data

Original: [Table]
Revised: [Table]

Original: and then perform joint training on both image-labeltext data and video-label-text data.
Revised: and then perform joint training on both image-labeltext data and video-label-text data.

Original: In this way
Revised: In this way

Original: [Table]
Revised: [Table]

Original: we decouple the multi-modal modeling into spatial and temporal dimensions.
Revised: we decouple the multi-modal modeling into spatial and temporal dimensions.

Original: Such a design has two potential benefits: 1) Considering the expensive computational cost of video pretraining, applying the image data to learn the spatial representation first is more efficient.
Revised: Such a design has two potential benefits: 1) Considering the expensive computational cost of video pretraining, applying the image data to learn the spatial representation first is more efficient.

Original: The decoupled pattern makes the multimodal representation learning more effective, which is the key to make image-language and video-language benefit each other
Revised: The decoupled pattern makes the multimodal representation learning more effective, which is the key to make image-language and video-language benefit each other

Original: [Table]
Revised: [Table]

Original:  Experiments
Revised:  Experiments

Original: Implementation Details.
Revised: Implementation Details.

Original: By default, we use the TimeSformer base model and BERT base model for visual encoder and text encoder, respectively.
Revised: By default, we use the TimeSformer base model and BERT base model for visual encoder and text encoder, respectively.

Original: As mentioned in Sec 3.3, our pretraining follows a decoupled paradigm.
Revised: As mentioned in Sec 3.3, our pretraining follows a decoupled paradigm.

Original: For the image-language pretraining stage, we initialize spatial attention with
Revised: For the image-language pretraining stage, we initialize spatial attention with

Original: ViT-B/16 [21] pretrained on ImageNet-1K [16].
Revised: ViT-B/16 [21] pretrained on ImageNet-1K [16].

Original: We take random image crops of resolution 224 × 224 as inputs and apply RandAugment [15].
Revised: We take random image crops of resolution 224 × 224 as inputs and apply RandAugment [15].

Original: The model is pretrained for 20 epochs using a batch size of 2880.
Revised: The model is pretrained for 20 epochs using a batch size of 2880.

Original: For the joint pretraining, we sparsely sample 8 × 224 × 224 video clips, and train the model for 10 epochs with a batch size of 800 for video data and 2880 for image data.
Revised: For the joint pretraining, we sparsely sample 8 × 224 × 224 video clips, and train the model for 10 epochs with a batch size of 800 for video data and 2880 for image data.

Original: Our joint pretraining alternates batches between the image and video data.
Revised: Our joint pretraining alternates batches between the image and video data.

Original: The model is optimized with AdamW [44] using a weight decay of 0.05.
Revised: The model is optimized with AdamW [44] using a weight decay of 0.05.

Original: The learning rate is warmed-up to 3e-4 (image) /
Revised: The learning rate is warmed-up to 3e-4 (image) /

Original: 8e-5 (joint) and decayed linearly with a rate of 0.85.
Revised: 8e-5 (joint) and decayed linearly with a rate of 0.85.

Original: During downstream fine-tuning, we increase the image resolution to 384 × 384 for both image-text and video-text tasks [38, 37], unless otherwise specified.
Revised: During downstream fine-tuning, we increase the image resolution to 384 × 384 for both image-text and video-text tasks [38, 37], unless otherwise specified.

Original: We randomly sample 8 frames per video for retrieval and 16 for QA, following [36].
Revised: We randomly sample 8 frames per video for retrieval and 16 for QA, following [36].

Original: Temporal position embeddings in the spatial-temporal visual encoder are interpolated to accommodate the inputs of different lengths.
Revised: Temporal position embeddings in the spatial-temporal visual encoder are interpolated to accommodate the inputs of different lengths.

Original: Besides, we use task-specific learning rates and training epochs due to various data scales and domains, the details of which will be further illustrated in the supplementary material.
Revised: Besides, we use task-specific learning rates and training epochs due to various data scales and domains, the details of which will be further illustrated in the supplementary material.

Original: 4.1 Visual Only Tasks
Revised: 4.1 Visual Only Tasks

Original: We first evaluate the representations of the visual encoder on visual only tasks.
Revised: We first evaluate the representations of the visual encoder on visual only tasks.

Original: Here the classical image classification and video action recognition tasks are adopted for benchmarking.
Revised: Here the classical image classification and video action recognition tasks are adopted for benchmarking.

Original: Note that, in the following tables, we use the superscript "*" to denote extra video data is used .
Revised: Note that, in the following tables, we use the superscript "*" to denote extra video data is used .

Original: Image Classification.
Revised: Image Classification.

Original: Linear probing is a commonly used method to evaluate the representation quality [53, 71, 55].
Revised: Linear probing is a commonly used method to evaluate the representation quality [53, 71, 55].

Original: Following the implementation of CLIP [53], we freeze the visual encoder and fine-tune the newly appended linear layers for linear probing.
Revised: Following the implementation of CLIP [53], we freeze the visual encoder and fine-tune the newly appended linear layers for linear probing.

Original: We evaluate our method on 6 image classification datasets, which are not included in our pretraining set.
Revised: We evaluate our method on 6 image classification datasets, which are not included in our pretraining set.

Original: We compare with METER [22],
Revised: We compare with METER [22],

Original: ALBEF [38], BLIP [37], FLAVA [55], and CLIP [53] in Table 2.
Revised: ALBEF [38], BLIP [37], FLAVA [55], and CLIP [53] in Table 2.

Original: Note that for fair comparisons with
Revised: Note that for fair comparisons with

Original: FLAVA [55], we pretrain their model (we adopt the implementation in torchmultimodal 2 ) on the 14M image-text data.
Revised: FLAVA [55], we pretrain their model (we adopt the implementation in torchmultimodal 2 ) on the 14M image-text data.

Original: Compared to METER, ALBEF, BLIP, and FLAVA 14 M , OmniVL offers consistently better results.
Revised: Compared to METER, ALBEF, BLIP, and FLAVA 14 M , OmniVL offers consistently better results.

Original: Compared to CLIP and FLAVA 70 M , even though we use far less pre-training data, our method still achieves overall comparable results.
Revised: Compared to CLIP and FLAVA 70 M , even though we use far less pre-training data, our method still achieves overall comparable results.

Original: Video Action Recognition.
Revised: Video Action Recognition.

Original: Video action recognition is one of the most representative tasks for video understanding [80].
Revised: Video action recognition is one of the most representative tasks for video understanding [80].

Original: We first report the linear probing results on UCF101 and HMDB51.
Revised: We first report the linear probing results on UCF101 and HMDB51.

Original: The results are summarized in the first two columns of Table 3.
Revised: The results are summarized in the first two columns of Table 3.

Original: We see that OmniVL achieves
Revised: We see that OmniVL achieves

Original: [Table] [Table]
Revised: [Table] [Table]

Original: excellent performance, i.e. , 93.2% on UCF101, even without end-to-end training
Revised: excellent performance, i.e. , 93.2% on UCF101, even without end-to-end training

Original: [Table]
Revised: [Table]

Original: which beats baseline methods by a large margin.
Revised: which beats baseline methods by a large margin.

Original: Furthermore
Revised: Furthermore

Original: [Table]
Revised: [Table]

Original: we conduct fine-tuning experiments on Kinetics400 [32] and Something-Something v2 [24] dataset.
Revised: we conduct fine-tuning experiments on Kinetics400 [32] and Something-Something v2 [24] dataset.

Original: We compare our method with supervised pretrained TimeSformer [8] and FiT [6] in Table 3 since we share the same model architecture.
Revised: We compare our method with supervised pretrained TimeSformer [8] and FiT [6] in Table 3 since we share the same model architecture.

Original: The same training settings with TimeSformer are adopted for fair comparisons.
Revised: The same training settings with TimeSformer are adopted for fair comparisons.

Original: OmniVL outperforms
Revised: OmniVL outperforms

Original: TimeSformer by 1.4% and 5.0% on Kinetics-400 and Something-Something v2 in terms of Top1 accuracy, respectively.
Revised: TimeSformer by 1.4% and 5.0% on Kinetics-400 and Something-Something v2 in terms of Top1 accuracy, respectively.

Original: Compared to the video-language model FiT [6] ( i.e. , performing joint pretraining from scratch), our results are also overall better, highlighting the advantage of our method.
Revised: Compared to the video-language model FiT [6] ( i.e. , performing joint pretraining from scratch), our results are also overall better, highlighting the advantage of our method.

Original: 4.2 Cross-modal Alignment Tasks
Revised: 4.2 Cross-modal Alignment Tasks

Original: Using visual and text embeddings generated from the unimodal encoders, OmniVL can easily handle cross-modal alignment tasks, e.g. , image-text retrieval and text-to-video retrieval.
Revised: Using visual and text embeddings generated from the unimodal encoders, OmniVL can easily handle cross-modal alignment tasks, e.g. , image-text retrieval and text-to-video retrieval.

Original: In addition, in order to balance the inference efficiency and the deep fusion of multi-modal information, we follow [37, 38] to first select Top-K ( K = 128 by default) candidates based on the vision-language similarity scores, which are further re-ranked by calculating their pairwise VLM scores.
Revised: In addition, in order to balance the inference efficiency and the deep fusion of multi-modal information, we follow [37, 38] to first select Top-K ( K = 128 by default) candidates based on the vision-language similarity scores, which are further re-ranked by calculating their pairwise VLM scores.

Original: The pretrained model is fine-tuned with UniVLC loss and VLM loss.
Revised: The pretrained model is fine-tuned with UniVLC loss and VLM loss.

Original: Image-Text Retrieval.
Revised: Image-Text Retrieval.

Original: We first evaluate OmniVL on COCO [43] and Flickr30K [52] for both image-to-text retrieval and text-to-image retrieval.
Revised: We first evaluate OmniVL on COCO [43] and Flickr30K [52] for both image-to-text retrieval and text-to-image retrieval.

Original: As shown in Table 4, OmniVL outperforms other methods by clear margins.
Revised: As shown in Table 4, OmniVL outperforms other methods by clear margins.

Original: With 14M image-text pairs for pretraining, our model surpasses BLIP by 1.8% and 0.8% in terms of average recall@1 on COCO and Flickr30K, respectively, which is even competitive compared with Florence that is trained with much larger scale data and model .
Revised: With 14M image-text pairs for pretraining, our model surpasses BLIP by 1.8% and 0.8% in terms of average recall@1 on COCO and Flickr30K, respectively, which is even competitive compared with Florence that is trained with much larger scale data and model .

Original: Text-to-Video Retrieval.
Revised: Text-to-Video Retrieval.

Original: We compare OmniVL with other methods on MSRVTT [67] and
Revised: We compare OmniVL with other methods on MSRVTT [67] and

Original: DiDeMo [5] for text-to-video retrieval under both fine-tuning and zero-shot transfer settings in
Revised: DiDeMo [5] for text-to-video retrieval under both fine-tuning and zero-shot transfer settings in

Original: Table 5.
Revised: Table 5.

Original: Note that directly using the pretrained models, OmniVL achieves 42.0% and 40.6% on
Revised: Note that directly using the pretrained models, OmniVL achieves 42.0% and 40.6% on

Original: MSRVTT and DiDeMo in terms of recall@1, respectively, significantly surpassing existing methods.
Revised: MSRVTT and DiDeMo in terms of recall@1, respectively, significantly surpassing existing methods.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: SPICE, B@4: BLEU@
Revised: SPICE, B@4: BLEU@

Original: [Table]
Revised: [Table]

Original: The performance of OmniVL is further improved under the fine-tuning settings.
Revised: The performance of OmniVL is further improved under the fine-tuning settings.

Original: The results suggest the multi-modal representations learned by our method are very discriminative
Revised: The results suggest the multi-modal representations learned by our method are very discriminative

Original: [Table]
Revised: [Table]

Original: 4.3 Multi-modal Understanding and Generation Tasks
Revised: 4.3 Multi-modal Understanding and Generation Tasks

Original: The visual-grounded generation decoder equips OmniVL with the capability for multi-modal understanding and generation so as to reason and describe.
Revised: The visual-grounded generation decoder equips OmniVL with the capability for multi-modal understanding and generation so as to reason and describe.

Original: In this part, we fine-tune our model with the
Revised: In this part, we fine-tune our model with the

Original: LM loss
Revised: LM loss

Original: [Table]
Revised: [Table]

Original: and evaluate its performance on captioning and image/video question answering tasks.
Revised: and evaluate its performance on captioning and image/video question answering tasks.

Original: Image Captioning.
Revised: Image Captioning.

Original: Image captioning requires the model to generate a textual description for a given image.
Revised: Image captioning requires the model to generate a textual description for a given image.

Original: We fine-tune our model on COCO and then evaluate on both COCO [43] NoCaps [1].
Revised: We fine-tune our model on COCO and then evaluate on both COCO [43] NoCaps [1].

Original: Following [63, 37], we adopt a prefix prompt “a picture of” to guide the caption generation.
Revised: Following [63, 37], we adopt a prefix prompt “a picture of” to guide the caption generation.

Original: The results are presented in Table 6, from which we can see that OmniVL achieves superior results on both datasets, e.g. , 107.5 and 133.9 on NoCaps and COCO in terms of CIDEr.
Revised: The results are presented in Table 6, from which we can see that OmniVL achieves superior results on both datasets, e.g. , 107.5 and 133.9 on NoCaps and COCO in terms of CIDEr.

Original: Although SimVLM [63] adopt much larger pretraining data than ours, OmniVL still achieves comparable or even better results on some metrics, e.g. , CIDEr, on COCO dataset.
Revised: Although SimVLM [63] adopt much larger pretraining data than ours, OmniVL still achieves comparable or even better results on some metrics, e.g. , CIDEr, on COCO dataset.

Original: For fair comparison with OFA [60], we pre-train their model on the 14M image-text data (with only image-text matching and image captioning objectives, denoted as OFA 14 M ), the results demonstrate that using the same amount of pre-training data, OmniVL performs better than OFA measured by all the metrics.
Revised: For fair comparison with OFA [60], we pre-train their model on the 14M image-text data (with only image-text matching and image captioning objectives, denoted as OFA 14 M ), the results demonstrate that using the same amount of pre-training data, OmniVL performs better than OFA measured by all the metrics.

Original: Note that we don’t show the results of SIMVLM on 14M data since they haven’t released their code.
Revised: Note that we don’t show the results of SIMVLM on 14M data since they haven’t released their code.

Original: Video Captioning.
Revised: Video Captioning.

Original: In this section, we evaluate OmniVL on YouCook2 [76] for video captioning.
Revised: In this section, we evaluate OmniVL on YouCook2 [76] for video captioning.

Original: We follow [77] to report the results on the validation sets in Table 7.
Revised: We follow [77] to report the results on the validation sets in Table 7.

Original: We can see that OmniVL outperforms most existing methods in all the metrics.
Revised: We can see that OmniVL outperforms most existing methods in all the metrics.

Original: The models marked in gray, e.g. , UniVL [47], apply an extra pre-trained backbone network S3D [64] to extract video-level features offline.
Revised: The models marked in gray, e.g. , UniVL [47], apply an extra pre-trained backbone network S3D [64] to extract video-level features offline.

Original: Comparatively,
Revised: Comparatively,

Original: OmniVL is fine-tuned in an end-to-end manner.
Revised: OmniVL is fine-tuned in an end-to-end manner.

Original: Visual Question Answering.
Revised: Visual Question Answering.

Original: For VQA, the model is expected to predict an answer given an image and a related question.
Revised: For VQA, the model is expected to predict an answer given an image and a related question.

Original: To this end, we follow [38, 37] to formulate it as a generation task and focus on open-ended VQA.
Revised: To this end, we follow [38, 37] to formulate it as a generation task and focus on open-ended VQA.

Original: It is worth noting that during fine-tuning, we first input the image and the question into the unified visual encoder and visual-grounded alignment decoder separately to obtain a fused
Revised: It is worth noting that during fine-tuning, we first input the image and the question into the unified visual encoder and visual-grounded alignment decoder separately to obtain a fused

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: multi-modal representation, and then feed the visual representation and multi-modal representation into the visual-grounded generation decoder to predict the final answer.
Revised: multi-modal representation, and then feed the visual representation and multi-modal representation into the visual-grounded generation decoder to predict the final answer.

Original: We compare with existing
Revised: We compare with existing

Original: SOTA methods in Table 8.
Revised: SOTA methods in Table 8.

Original: We observe consistent performance gain compared with existing methods.
Revised: We observe consistent performance gain compared with existing methods.

Original: Especially, OmniVL even outperforms SimVLM [63] by 0.6% trained with 1.8B image-text pairs.
Revised: Especially, OmniVL even outperforms SimVLM [63] by 0.6% trained with 1.8B image-text pairs.

Original: Video Question Answering.
Revised: Video Question Answering.

Original: Table 9 summarizes the results of video question answering on
Revised: Table 9 summarizes the results of video question answering on

Original: MSRVTT-QA [65] and MSVD-QA [65].
Revised: MSRVTT-QA [65] and MSVD-QA [65].

Original: OmniVL surpasses both QA-specific methods, e
Revised: OmniVL surpasses both QA-specific methods, e

Original: [Table]
Revised: [Table]

Original: g. , JustAsk [68], and pretraining methods, e.g. , VIOLET [23] on both datasets, which validates the effectiveness of our method for complex multimodal modeling
Revised: g. , JustAsk [68], and pretraining methods, e.g. , VIOLET [23] on both datasets, which validates the effectiveness of our method for complex multimodal modeling

Original: [Table]
Revised: [Table]

Original: 4.4 Ablation Study
Revised: 4.4 Ablation Study

Original: Decoupled Joint Pretraining.
Revised: Decoupled Joint Pretraining.

Original: To verify the effect of decoupled joint pretraining, we conduct four ablation experiments with different pretraining strategies: image-only pretraining, video-only pretraining, joint pretraining from scratch
Revised: To verify the effect of decoupled joint pretraining, we conduct four ablation experiments with different pretraining strategies: image-only pretraining, video-only pretraining, joint pretraining from scratch

Original: [Table]
Revised: [Table]

Original: and Img2Vid pretraining where we first pretrain OmniVL on image and then on video.
Revised: and Img2Vid pretraining where we first pretrain OmniVL on image and then on video.

Original: We list some representative results of each task in Table 10 (see data details and more results in the supplementary material).
Revised: We list some representative results of each task in Table 10 (see data details and more results in the supplementary material).

Original: [Table]
Revised: [Table]

Original: We can see that video-only pretraining leads to significant performance degradation due to limited data scale for pretraining.
Revised: We can see that video-only pretraining leads to significant performance degradation due to limited data scale for pretraining.

Original: Comparatively, image-only pretraining is a competitive baseline, which however, is still far behind decoupled joint pretraining on video tasks, e.g. , text-video-retrieval
Revised: Comparatively, image-only pretraining is a competitive baseline, which however, is still far behind decoupled joint pretraining on video tasks, e.g. , text-video-retrieval

Original: MSRVTT (ret) and video question answering MSRVTT (QA).
Revised: MSRVTT (ret) and video question answering MSRVTT (QA).

Original: Compared to video-only pretraining, joint pretraining from scratch can significantly improve the performance on video tasks, which has also been verified in [6, 23].
Revised: Compared to video-only pretraining, joint pretraining from scratch can significantly improve the performance on video tasks, which has also been verified in [6, 23].

Original: However, it produces limited results on image tasks, which is even worse than image-only pretraining.
Revised: However, it produces limited results on image tasks, which is even worse than image-only pretraining.

Original: Img2Vid is another competitive baseline, which however demonstrates degraded performance on image tasks compared to image-only pretraining.
Revised: Img2Vid is another competitive baseline, which however demonstrates degraded performance on image tasks compared to image-only pretraining.

Original: This indicates the naive combination of image-language and video-language cannot enjoy their synergy.
Revised: This indicates the naive combination of image-language and video-language cannot enjoy their synergy.

Original: By contrast, our simple decoupled joint pretraining strategy can achieve better performance on both image and video tasks.
Revised: By contrast, our simple decoupled joint pretraining strategy can achieve better performance on both image and video tasks.

Original: We hypothesize this is because starting from image-only pretraining can help the model focus on spatial representation learning first and provides better initialization, and thus the subsequent joint pretraining would be more concentrated on learning the temporal dynamics incrementally while preserving/polishing the well-learned spatial representations.
Revised: We hypothesize this is because starting from image-only pretraining can help the model focus on spatial representation learning first and provides better initialization, and thus the subsequent joint pretraining would be more concentrated on learning the temporal dynamics incrementally while preserving/polishing the well-learned spatial representations.

Original: UniVLC Loss.
Revised: UniVLC Loss.

Original: We further replace the UniVLC loss with vanilla contrastive loss to study its impact on various downstream tasks.
Revised: We further replace the UniVLC loss with vanilla contrastive loss to study its impact on various downstream tasks.

Original: Note that we exclude the visual-label data from the pretraining corpus under the “w/o UniVLC” setting.
Revised: Note that we exclude the visual-label data from the pretraining corpus under the “w/o UniVLC” setting.

Original: The example results shown in Figure 2 illustrate that our method performs comparably to vanilla contrast-based model on vision-language tasks, e.g. , image/video-text retrieval, image captioning, and image/video question answering.
Revised: The example results shown in Figure 2 illustrate that our method performs comparably to vanilla contrast-based model on vision-language tasks, e.g. , image/video-text retrieval, image captioning, and image/video question answering.

Original: But on visual only tasks, e.g. , linear probing for image/video classification and video action recognition fine-tuning, the performance gain is much higher, indicating that UniVLC could facilitate model to learn more discriminative visual representations and benefit transfer learning tasks.
Revised: But on visual only tasks, e.g. , linear probing for image/video classification and video action recognition fine-tuning, the performance gain is much higher, indicating that UniVLC could facilitate model to learn more discriminative visual representations and benefit transfer learning tasks.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  Conclusion and Discussion of Broader Impact
Revised:  Conclusion and Discussion of Broader Impact

Original: In this paper, we presented OmniVL, a novel vision-language foundation model that unifies imagelanguage and video-language.
Revised: In this paper, we presented OmniVL, a novel vision-language foundation model that unifies imagelanguage and video-language.

Original: It naturally supports visual only tasks, cross-modal alignment tasks, and multi-modal understanding and generation tasks.
Revised: It naturally supports visual only tasks, cross-modal alignment tasks, and multi-modal understanding and generation tasks.

Original: The unified vision-language contrastive loss also enables OmniVL to utilize image-text, image-label, video-text and video-label data together.
Revised: The unified vision-language contrastive loss also enables OmniVL to utilize image-text, image-label, video-text and video-label data together.

Original: Accordingly, a decoupled pretraining paradigm is introduced to decouple vision-language modeling into spatial and temporal dimensions, which boosts the performance on both image and video tasks.
Revised: Accordingly, a decoupled pretraining paradigm is introduced to decouple vision-language modeling into spatial and temporal dimensions, which boosts the performance on both image and video tasks.

Original: Although our model has achieved superior results on a wide range of downstream tasks, it still lacks of the commonsense reasoning capability required by some visual-language interaction tasks (e.g., visual/video question answering).
Revised: Although our model has achieved superior results on a wide range of downstream tasks, it still lacks of the commonsense reasoning capability required by some visual-language interaction tasks (e.g., visual/video question answering).

Original: It also needs better architecture design to support the zero-shot capability for visual question answering and few-shot task customization capability like GPT-3.
Revised: It also needs better architecture design to support the zero-shot capability for visual question answering and few-shot task customization capability like GPT-3.

Original: From the societal impact perspective, since our model is pretrained on the large-scale web-crawled data which may contain some toxic language or bias, and it is not easy to explicitly control the model output, much attention should be paid to ensure responsible model deployment.
Revised: From the societal impact perspective, since our model is pretrained on the large-scale web-crawled data which may contain some toxic language or bias, and it is not easy to explicitly control the model output, much attention should be paid to ensure responsible model deployment.

Original: Acknowledgement This project was supported by NSFC under Grant No. 62102092.
Revised: Acknowledgement This project was supported by NSFC under Grant No. 62102092.

Original: Y.-G. Jiang was sponsored in part by “Shuguang Program” supported by Shanghai Education Development
Revised: Y.-G. Jiang was sponsored in part by “Shuguang Program” supported by Shanghai Education Development

Original: Foundation and Shanghai Municipal Education Commission (No. 20SG01).
Revised: Foundation and Shanghai Municipal Education Commission (No. 20SG01).

Original: A Specification for the Visual-grounded Alignment / Generation Decoder

Original: As mentioned in the paper, the visual-grounded alignment decoder is applied to enable the deep interaction of multimodal information with cross-attention blocks, while the visual-grounded generation decoder is adopted to generate natural languages conditioned on the visual input.

Original: We further specify their architectures in Figure 3.

Original: [Equation]

Original: [Figure]

Original: Note that both visual-grounded alignment decoder and visual-grounded generation decoder are initialized with the Bert-base model [17], which stacks 12 transformer layers.

Original: B Image / Video Question Answering

Original: Image / video question answering requires the model to answer a question according to a given image / video, which models the complex interaction between visual and linguistic representations.

Original: During finetuning, we rearrange the pre-trained model, as shown in Figure 4.

Original: [Figure]

Original: Our setup is based on the following considerations.

Original: We first input the image / video to unified visual encoder, the output of which will be combined with the text features of the questions through the visual-grounded alignment decoder.

Original: Based on these deeply fused representations, we finally generate the predicted answers with the visual-grounded generation decoder.

Original: C Finetuning Setups

Original: In this section, we describe the settings used when fine-tuning the pretrained models on various downstream tasks.

Original: C.1 Image-Language Tasks

Original: For image-text retrieval and image captioning, we resize the images to 384 × 384, while for visual question answering, we resize the images to 480 × 480, following [37].

Original: We use RandomAugment [15] for data augmentation.

Original: The default settings for finetuning on each dataset are shown in Table 11.

Original: [Table]

Original: C.2 Video-Language Tasks

Original: For all video-language downstream tasks, we resize video frames to 384 × 384.

Original: During fine-tuning, we randomly sample N frames from each video, where N = 8 for text-to-video retrieval, N = 16 for video question answering following [36], and N = 24 for video captioning.

Original: We perform uniform sampling during inference.

Original: Similar with image-language tasks, we also adopt RandomAugment [15] for data augmentation.

Original: The default settings for finetuning on each dataset are shown in Table 12.

Original: D More Comparison Results on Vision-language Tasks for Different Pretraining Paradigms

Original: We demonstrate more comparison results using different pretraining paradigms ( i.e. , image-only, video-only, joint pretraining from scratch, and our decoupled pretraining) on various vision-language downstream tasks in Table 13.

Original: Details of the pretraining data can be found in Table 14.

Original: Moreover, an “img2vid” strategy is also adopted for further comparison, where we start with image-only pretraining and then implement video-only pretraining.

Original: We can see our decoupled joint pretraining paradigm achieves consistently better results on all the downstream tasks.

Original: [Table]

Original: [Table]

Original: [Table]

Original: E Image/Video Captioning Examples

Original: We show some image and video captioning results generated by our method in Figure 5 and Figure 6, respectively.

Original: We can see that the captions generated by OmniVL are both natural and abundant.

Original: Specifically, for the image captioning, when the visual information in the images is relatively simple, the generated captions are relatively general (line 2 and line 3).

Original: While when the contents are rich,

Original: OmniVL can generate more fine-grained descriptions (line 1).

Original: Fo video captioning, OmniVL could accurately describe the actions ( e.g. , “add” and “pour”) and objects ( e.g. , “lemon juice” and “fried chicken”) in videos.

Original: The visualization results demonstrate the superior multimodal generation capability of OmniVL.

Original: a living room filled with furniture and a flat screen tv.

Original: a woman wearing a brown hat and a red shirt.

Original: a group of people standing on top of a lush green.

Original: a man standing next to a red car in a parking lot.

Original: a red and blue motorcycle parked in front of a grassy field.

Original: a light that is shining in the dark.

Original: add chickpeas parsley and lemon juice to the food processor and blend cut the salmon into thin slices pour the sauce on the fried chicken