Original: Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Revised: Zero-Shot Video Question Answering via Frozen Bidirectional Language Models

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Video question answering (VideoQA) is a complex task that requires diverse multi- modal data for training.
Revised: Video question answering (VideoQA) is a complex task that requires diverse multimodal data for training.

Original: Manual annotation of question and answers for videos, however, is tedious and prohibits scalability.
Revised: Manual annotation of questions and answers for videos, however, is tedious and prohibits scalability.

Original: To tackle this problem, recent methodsconsider zero-shot settings with no manual annotation of visual question-answer.
Revised: To tackle this problem, recent methods consider zero-shot settings with no manual annotation of visual question-answer.

Original: In particular, a promising approach adapts frozen autoregressive language models pretrained on Web-scale text-only data to multi-modal inputs.
Revised: In particular, a promising approach adapts frozen autoregressive language models pretrained on Web-scale text-only data to multi-modal inputs.

Original: In contrast, we here build on frozen bidirectional language models (BiLM) and show that such an approach provides a stronger and cheaper alternative for zero-shot VideoQA.
Revised: In contrast, we here build on frozen bidirectional language models (BiLM) and show that such an approach provides a stronger and cheaper alternative for zero-shot VideoQA.

Original: In particular, (i) we combine visual inputs with the frozen BiLM using light trainable modules, (ii) we train such modules using Web-scraped multi-modal data, and finally (iii) we perform zero-shot VideoQA inference through maskedlanguage modeling, where the masked text is the answer to a given question.
Revised: In particular, (i) we combine visual inputs with the frozen BiLM using light trainable modules, (ii) we train such modules using Web-scraped multi-modal data, and finally (iii) we perform zero-shot VideoQA inference through masked language modeling, where the masked text is the answer to a given question.

Original: Our proposed approach, FrozenBiLM , outperforms the state of the art in zero-shot
Revised: Our proposed approach, FrozenBiLM , outperforms the state of the art in zero-shot

Original: VideoQA by a significant margin on a variety of datasets, including LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA, TGIF-FrameQA, How2QAand TVQA.
Revised: VideoQA by a significant margin on a variety of datasets, including LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA, TGIF-FrameQA, How2QA and TVQA.

Original: It also demonstrates competitive performance in the few-shot andfully-supervised setting.
Revised: It also demonstrates competitive performance in the few-shot and fully-supervised setting.

Original: Our code and models will be made publicly available.
Revised: Our code and models are publicly available at [1]. Little [Table]

Revised: toy poodle dog running fast

Original: [Figure]
Revised: [Table]

Revised: beach.

Revised: [Table]

Original: Zero-Shot VideoQA
Revised: Shot VideoQA

Original: [Figure]
Revised: What is the dog doing? FrozenBiLM PretrainedBiLM Test data:

Original: without using any explicit visual question-answer supervision.
Revised: Video Question+ Answer: Running FrozenBiLM

Revised: PretrainedBiLM

Revised: Training data:

Revised: Web-scraped

Revised: Video + Caption

Revised: [Figure]

Original:  Introduction
Revised:  Introduction

Original: Video question answering (VideoQA) is a challenging task that requires fine-grained multi-modalunderstanding.
Revised: Video question answering (VideoQA) is a challenging task that requires fine-grained multi-modal understanding.

Original: State-of-the-art approaches to VideoQA [40, 102, 104] rely on large video datasetsmanually annotated with question-answer pairs.
Revised: State-of-the-art approaches to VideoQA [43, 107, 109] rely on large video datasets manually annotated with question-answer pairs.

Original: Yet, collecting such annotations is time consuming,expensive and therefore not scalable.
Revised: Yet, collecting such annotations is time consuming, expensive and therefore not scalable.

Original: This has motivated the development of zero-shot VideoQAapproaches [96, 97, 105], that use no visual question-answer annotation for training, see Figure 1.
Revised: This has motivated the development of zero-shot VideoQA approaches [101, 102, 110], that use no visual question-answer annotation for training, see Figure 1.

Original: Recently, a promising line of work builds on frozen large autoregressive language models [17, 65,88, 99, 106] for zero-shot visual question answering.
Revised: Recently, a promising line of work builds on frozen large autoregressive language models [19, 68, 91, 96, 104, 111] for zero-shot visual question answering.

Original: This has been motivated by the findings from GPT-3 [6] which exhibited strong zero-shot text-only question answering abilities from largeautoregressive language models.
Revised: This has been motivated by the findings from GPT-3 [8] which exhibits strong zero-shot text-only question answering abilities from large autoregressive language models.

Original: Such models [6, 69, 79, 89] can predict an arbitrarily long sequenceof text, one token at each step from left to right.
Revised: Such models [8, 72, 82, 92] can predict an arbitrarily long sequence of text, one token at each step from left to right.

Original: However, they usually require billion parameters towork well, making them computationally expensive to train, and challenging to deploy in practice.
Revised: However, they usually require billion parameters to work well, making them computationally expensive to train, and challenging to deploy in practice.

Original: In contrast, recent work in natural language [62, 73, 74, 84] demonstrates strong zero-shot performance for lighter bidirectional language models (BiLM).
Revised: In contrast, recent work in natural language [65, 76, 77, 87] demonstrates strong zero-shot performance for lighter bidirectional language models (BiLM).

Original: Such models [15, 23, 32, 39, 58, 72] canpredict a few masked tokens in an input sequence given left and right context in a single forward pass.
Revised: Such models [17, 25, 35, 42, 61, 75] can predict a few masked tokens in an input sequence given left and right context in a single forward pass.

Original: These works cast downstream tasks in cloze form 1 [87], similar to the masked language modeling task(MLM) [15] solved by these models at pretraining.
Revised: These works cast downstream tasks in cloze form 1 [90], similar to the masked language modeling task (MLM) [17] solved by these models at pretraining.

Original: This motivates us to tackle diverse zero-shot multimodal tasks (open-ended VideoQA [94], multiple-choice VideoQA [43] and fill-in-the-blank [63]) byformulating them in cloze form and leveraging the text-only knowledge of pretrained BiLM models.
Revised: This motivates us to tackle diverse zero-shot multimodal tasks (open-ended VideoQA [98], multiple-choice VideoQA [46] and fill-in-the-blank [66]) by formulating them in cloze form and leveraging the text-only knowledge of pretrained BiLM.

Original: To adapt a pretrained BiLM to multi-modal inputs, we combine it with a frozen pretrained visualbackbone and a set of lightweight additional modules including adapters [26].
Revised: To adapt a pretrained BiLM to multi-modal inputs, we combine it with a frozen pretrained visual backbone and a set of lightweight additional modules including adapters [28].

Original: We train these moduleson Web-scraped video-text data using a simple visually-conditioned MLM loss.
Revised: We train these modules on Web-scraped video-text data using a simple visually-conditioned MLM loss.

Original: We preserve theuni-modal knowledge of a BiLM by freezing its weights.
Revised: We preserve the uni-modal knowledge of a BiLM by freezing its weights.

Original: To our knowledge, our approach is the firstto explore the zero-shot visual-linguistic capabilities of frozen non-autoregressive language models.
Revised: To our knowledge, our approach is the first to explore the zero-shot visual-linguistic capabilities of frozen non-autoregressive language models.

Original: We show that our approach largely improves the state of the art on various zero-shot VideoQAbenchmarks.
Revised: We show that our approach largely improves the state of the art on various zero-shot VideoQA benchmarks.

Original: Furthermore, we demonstrate that frozen bidirectional language models perform betterwhile being cheaper to train than frozen autoregressive language models [88].
Revised: Furthermore, we demonstrate that frozen bidirectional language models perform better while being cheaper to train than frozen autoregressive language models [91].

Original: Moreover, our ablationstudies show (i) the ability of our model to effectively perform zero-shot multi-modal reasoningusing both visual cues and speech transcripts, (ii) the importance of adapters combined with frozenpretrained language models, (iii) the impact of multi-modal data scale, (iv) the impact of the languagemodel size and of bidirectional modeling.
Revised: Moreover, our ablation studies show (i) the ability of our model to effectively perform zero-shot multi-modal reasoning using both visual cues and speech transcripts, (ii) the importance of adapters combined with frozen pretrained language models, (iii) the impact of multi-modal data scale, (iv) the impact of the language model size and of bidirectional modeling.

Original: Our approach also performs competitively in the fullysupervised setting.
Revised: Our approach also performs competitively in the fullysupervised setting.

Original: Indeed, we show the benefits of freezing the weights of a BiLM when using
Revised: Indeed, we show the benefits of freezing the weights of a BiLM when using

Original: VideoQA training data, while updating considerably less parameters compared to alternative methods.
Revised: VideoQA training data, while updating considerably less parameters compared to alternative methods.

Original: Finally, we introduce a new few-shot VideoQA task in which we finetune our pretrained model on asmall fraction of the downstream training dataset, and show promising results in this setting.
Revised: Finally, we introduce a new few-shot VideoQA task in which we finetune our pretrained model on a small fraction of the downstream training dataset, and show promising results in this setting.

Original: In summary, our contributions are three-fold:
Revised: In summary, our contributions are three-fold:

Original: (i)
Revised: (i)

Original: We present FrozenBiLM , a framework that handles multi-modal inputs using frozen bidirec- tional language models and enables zero-shot VideoQA through masked language modeling.
Revised: We present FrozenBiLM , a framework that handles multi-modal inputs using frozen bidirectional language models and enables zero-shot VideoQA through masked language modeling.

Original: (ii) We provide an extensive ablation study and demonstrate the superior performance of our framework in the zero-shot setting when compared to previous autoregressive models.
Revised: (ii) We provide an extensive ablation study and demonstrate the superior performance of our framework in the zero-shot setting when compared to previous autoregressive models.

Original: (iii)
Revised: (iii)

Original: Our approach improves the state of the art in zero-shot VideoQA by a significant margin.
Revised: Our approach improves the state of the art in zero-shot VideoQA by a significant margin.

Original: FrozenBiLM also demonstrates competitive performance in the fully- supervised
Revised: FrozenBiLM also demonstrates competitive performance in the fully-supervised setting and shows strong

Original: setting and
Revised: results in

Original: shows strong results in the few-shot VideoQA setting which we introduce.
Revised: the few-shot VideoQA setting which we introduce.

Original: Our codeis provided in the Supplementary Material, and will be made publicly available.
Revised: Our code and trained models are publicly available at [1].

Original:  Related Work
Revised:  Related Work

Original: Zero-shot VideoQA.
Revised: Zero-shot VideoQA.

Original: A vast majority of VideoQA approaches rely on relatively small, manuallyannotated VideoQA datasets [1, 7, 8, 11–13, 18, 21, 22, 27, 30, 31, 33–36, 40, 41, 44, 55, 57, 66,67, 71, 75, 76, 80, 86, 93, 95, 98, 100, 107, 111].
Revised: A vast majority of VideoQA approaches rely on relatively small, manually annotated VideoQA datasets [3, 9, 10, 13–15, 20, 23, 24, 30, 33, 34, 36–39, 43, 44, 47, 58, 60, 69, 70, 74, 78, 79, 83, 89, 97, 100, 103, 105, 112, 116].

Original: Recently, a few work [96, 105] have exploredzero-shot approaches for VideoQA, where models are only trained on automatically mined videoclips with short text descriptions.
Revised: Recently, a few work [101, 110] have explored zero-shot approaches for VideoQA, where models are only trained on automatically mined video clips with short text descriptions.

Original: In contrast to VideoQA annotations, such video-text pairs arereadily-available at scale on the Web [4, 64, 104].
Revised: In contrast to VideoQA annotations, such video-text pairs are readilyavailable at scale on the Web [6, 67, 109].

Original: In detail, Yang et al.
Revised: In particular, Yang et al.

Original: [96] automatically generate
Revised: [101] automatically generate

Original: VideoQA training data using language models [69] pretrained on a manually annotated text-onlyquestion-answer corpus [70].
Revised: VideoQA training data using language models [72] pretrained on a manually annotated text-only question-answer corpus [73].

Original: Reserve [105] uses GPT-3 [6] to rephrase questions into sentencescompleted by a multi-modal model.
Revised: Reserve [110] uses GPT-3 [8] to rephrase questions into sentences completed by a multi-modal model.

Original: In contrast to these prior works [96, 105], our method does notrequire any kind of explicitly annotated language dataset or the use of data generation pipelines forzero-shot VideoQA.
Revised: In contrast to these prior works [101, 110], our method does not require any kind of explicitly annotated language dataset or the use of data generation pipelines for zero-shot VideoQA.

Original: Note that BLIP [50] studies a related setting where a model trained on manuallyannotated image-question-answer triplets is transferred to VideoQA, which is a less challenging task.
Revised: Note that BLIP [53] studies a related setting where a model trained on manually annotated image-question-answer triplets is transferred to VideoQA, which is a less challenging task.

Revised: Also note that VideoCLIP [99] considers a related zero-shot multiple-choice video-to-text retrieval task as VideoQA, but in this setting the model is not provided with natural language questions.

Original: Visual language models.
Revised: Visual language models.

Original: As language models require large amounts of training data to performwell [25], recent works have studied transferring pretrained language models [6, 91] to image-texttasks.
Revised: As language models require large amounts of training data to perform well [27], recent works have studied transferring pretrained language models [8, 94] to image-text tasks.

Original: VisualGPT [9] and VC-GPT [61] showed the benefit of initializing the weights of an image captioning model with a pretrained autoregressive language-only model.
Revised: VisualGPT [11] and VC-GPT [64] showed the benefit of initializing the weights of an image captioning model with a pretrained autoregressive language-only model.

Original: Recent work pushed thisidea further by freezing the weights of a pretrained autoregressive language model for tacklingvision and language tasks [17, 65, 88, 99].
Revised: Recent work pushed this idea further by freezing the weights of a pretrained autoregressive language model for tackling vision and language tasks [2, 19, 68, 91, 96, 104, 111].

Original: Our approach also leverages a frozen pretrained languagemodel.
Revised: Our approach also leverages a frozen pretrained language model.

Original: Similarly as MAGMA [17], we also use adapter layers [26].
Revised: Similar to MAGMA [19], we also use adapter layers [28, 29].

Original: However, we differ from theseapproaches as we propose to instead use lighter bidirectional masked language models , insteadof autoregressive ones, and rely on a masked language modeling objective (MLM) instead of anautoregressive one.
Revised: However, we differ from these approaches as we propose to instead use lighter bidirectional masked language models , instead of autoregressive ones, and rely on a masked language modeling objective (MLM) instead of an autoregressive one.

Original: Moreover our model is specifically designed for videos, for which high-qualityvisual question answering annotation is even more scarce, instead of images.
Revised: Moreover, our model is specifically designed for videos, for which high-quality visual question answering annotation is even more scarce compared to still images [19, 68, 91, 104].

Original: We also explore theuse of the speech modality, and tackle tasks which are challenging for autoregressive languagemodels such as video-conditioned fill-in-the-blank [63].
Revised: We also explore the use of the speech modality, and tackle tasks which are challenging for autoregressive language models such as video-conditioned fill-in-the-blank [66].

Original: Finally we show in Section 4.3 the superiorperformance of frozen bidirectional language models in comparison with autoregressive ones [88].
Revised: Finally we show in Section 4.3 the superior performance of frozen bidirectional language models in comparison with autoregressive ones [91].

Original: Masked Language Modeling in vision and language.
Revised: Masked Language Modeling in vision and language.

Original: The MLM objective was initially introducedin natural language [15, 39, 58] to pretrain bidirectional transformers and learn generic representations.
Revised: The MLM objective was initially introduced in natural language [17, 42, 61] to pretrain bidirectional transformers and learn generic representations.

Original: This approach achieved state-of-the-art results in many language tasks after finetuning on downstreamdatasets.
Revised: This approach achieved state-of-the-art results in many language tasks after finetuning on downstream datasets.

Original: Its success inspired numerous works to adapt it to train multi-modal transformer modelson paired visual-linguistic data [10, 19, 20, 24, 28, 37, 45, 48,53, 51, 56, 49, 47, 59, 60, 77, 78,82, 83, 85, 90, 92, 101, 104, 109, 110].
Revised: Its success inspired numerous works to adapt it to train multi-modal transformer models on paired visual-linguistic data [12, 21, 22, 26, 31, 40, 48, 51, 56, 54, 59, 52, 50, 62, 63, 80, 81, 85, 86, 88, 93, 95, 106, 109, 114, 115].

Original: However, these works typically use it to learn genericvisual-linguistic representations by updating the transformer weights, and then use expensive manualsupervision to train randomly initialized task-specific answer classifiers for VQA [10, 20, 24, 48, 49,53, 56, 59, 77, 78, 82, 85, 92, 101] or VideoQA [19, 45, 47, 90, 104].
Revised: However, these works typically use it to learn generic visual-linguistic representations by updating the transformer weights, and then use expensive manual supervision to train randomly initialized task-specific answer classifiers for VQA [12, 22, 51, 52, 59, 62, 80, 81, 85, 88, 95, 106] or VideoQA [21, 48, 50, 93, 109].

Original: In contrast, we tackle zero-shot
Revised: In contrast, we tackle zero-shot

Original: VideoQA, i.e. without using any manual annotation.
Revised: VideoQA, i.e. without using any manual annotation.

Original: Moreover, we do not update the transformerweights during cross-modal training, but instead exhibit the benefits of freezing these weights aftertext-only pretraining, for both zero-shot and fully-supervised VideoQA (see Sections 4.2 and 4.5).
Revised: Moreover, we do not update the transformer weights during cross-modal training, but instead exhibit the benefits of freezing these weights after text-only pretraining, for both zero-shot and fully-supervised VideoQA (see Sections 4.2 and 4.5).

Original:  Method
Revised:  Method

Original: This section presents our approach to tackle zero-shot video question answering.
Revised: This section presents our approach to tackle zero-shot video question answering.

Original: Here, zero-shotmeans that we do not use any visual question answering annotation and only rely on scalable data fromthe Web.
Revised: Here, zero-shot means that we do not use any visual question answering annotation and only rely on scalable data from the Web.

Original: Our approach starts with two strong pretrained components: (i) a text-only bidirectionalmasked language model (BiLM) pretrained on data from the Internet, which has the capability of zeroshot question answering but is not capable of visual reasoning, and (ii) a vision encoder pretrainedto map images to text descriptions, but which does not have the ability to perform visual questionanswering.
Revised: Our approach starts with two strong pretrained components: (i) a text-only bidirectional masked language model (BiLM) pretrained on data from the Internet, which has the capability of zeroshot question answering but is not capable of visual reasoning, and (ii) a vision encoder pretrained to map images to text descriptions, but which does not have the ability to perform visual question answering.

Original: We aim at connecting these two components while keeping the language componentfrozen to avoid catastrophic forgetting [14], where the large language model would specialize to anew task while forgetting its initial capabilities.
Revised: We aim at connecting these two components while keeping the language component frozen to avoid catastrophic forgetting [16], where the large language model would specialize to a new task while forgetting its initial capabilities.

Original: The end-goal is to design a unified model havingthe best of both worlds: visual understanding capabilities of a powerful visual encoder and questionanswering capabilities of a powerful language model.
Revised: The end-goal is to design a unified model having the best of both worlds: visual understanding capabilities of a powerful visual encoder and question answering capabilities of a powerful language model.

Original: This requires several technical innovations,which are described in the rest of this section.
Revised: This requires several technical innovations, which are described in the rest of this section.

Original: First, we explain in Section 3.1 how we augment afrozen pretrained bidirectional masked language model with new layers to enable joint video andlanguage reasoning, see Figure 2.
Revised: First, we explain in Section 3.1 how we augment a frozen pretrained bidirectional masked language model with new layers to enable joint video and language reasoning, see Figure 2.

Original: Second, we present in Section 3.2 how we train these layers onvideo-text data scraped from the Web [4].
Revised: Second, we present in Section 3.2 how we train these layers on video-text data scraped from the Web [6].

Original: Finally, we describe in Section 3.3 how we enable zero-shotpredictions for several video-language downstream tasks, including open-ended VideoQA, by castingthem in a cloze form, similar to the masked language modeling task solved during training.
Revised: Finally, we describe in Section 3.3 how we enable zero-shot predictions for several video-language downstream tasks, including open-ended VideoQA, by casting them in a cloze form, similar to the masked language modeling task solved during training.

Original: 3.1 Architecture
Revised: 3.1 Architecture

Original: The proposed architecture, illustrated in Figure 2, brings together a powerful frozen pretrainedbidirectional language model with a strong visual encoder.
Revised: The proposed architecture, illustrated in Figure 2, brings together a powerful frozen pretrained bidirectional language model with a strong visual encoder.

Original: The difficulty lies in enabling multi-modalreasoning while keeping the large language model frozen .
Revised: The difficulty lies in enabling multi-modal reasoning while keeping the large language model frozen .

Original: To address this challenge, we unify thesetwo models via a visual-to-text projection module together with small adapter modules insertedwithin the frozen language model.
Revised: To address this challenge, we unify these two models via a visual-to-text projection module together with small adapter modules inserted within the frozen language model.

Original: Next, we describe in more detail the three main components of thearchitecture: (i) the frozen pretrained bidirectional language model, (ii) the pretrained video encoderand (iii) the lightweight modules that seamlessly connect the two components.Frozen Bidirectional Masked Language Model.
Revised: Next, we describe in more detail the three main components of the architecture: (i) the frozen pretrained bidirectional language model, (ii) the pretrained video encoder and (iii) the lightweight modules that seamlessly connect the two components.

Original: Our method starts from a pretrained bidirectionallanguage model based on a Transformer encoder [89]. Theinput text is decomposed into a sequenceof tokens x = { x i } L 1 ∈ [ 1 , V ] L by a tokenizer of a vocabulary size V . The language model, parameterized by θ , makes use of an embedding function g θ which independently transforms each
Revised: The

Original: The …
Revised: [Figure]

Original: Visual Backbone Visual Backbone
Revised: Backbone Visual Backbone

Original: TokenEmbedder bride and the groom [MASK] the [MASK] at the [MASK]
Revised: TokenEmbedder bride and the groom [MASK] the [MASK] at the [MASK]

Original: Visual-TextProjection Visual-TextProjection …
Revised: Visual-TextProjection Visual-TextProjection …

Original: Frame
Revised: Frame

Original: Frame T
Revised: Frame T

Original: Caption
Revised: Caption

Original: Add& Norm
Revised: Add& Norm

Original: Feed-Forward
Revised: Feed-Forward

Original: Self-Attention
Revised: Self-Attention

Original: Adapter
Revised: Adapter

Original: Add& Norm
Revised: Add& Norm

Original: Adapter
Revised: Adapter

Original: Frozen cake wedding
Revised: Frozen cake wedding

Original: MLM Trained cut × N
Revised: MLM Trained cut × N

Original: Transformer Encoder (LanguageModel) withAdapters
Revised: Transformer Encoder (LanguageModel) withAdapters

Original: MLM
Revised: MLM

Original: MLM 𝑡 1 𝑡 2 𝑡 3 𝑡𝑡𝑡𝑡𝑡 8 𝑡 9 𝑡 10 𝑡𝑡′ 1 𝑡′ 2 𝑡′ 3 𝑡′ 4 𝑡′𝑡′𝑡′𝑡′ 8 𝑡′ 9 𝑡′ 10 𝑡′… 𝑥 𝑦 !
Revised: MLM 𝑡 1 𝑡 2 𝑡 3 𝑡𝑡𝑡𝑡𝑡 8 𝑡 9 𝑡 10 𝑡𝑡′ 1 𝑡′ 2 𝑡′ 3 𝑡′ 4 𝑡′𝑡′𝑡′𝑡′ 8 𝑡′ 9 𝑡′ 10 𝑡′

Original: 𝑦 " 𝑢𝑢 𝑇 𝑣𝑣 𝑇 …
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  token into a D -dimensional continuous embedding t = { t i } L 1 :=
Revised: Frozen Bidirectional Masked Language Model. Our method starts from a pretrained bidirectional language model based on a Transformer encoder [92]. The input text is decomposed into a sequence of tokens x = { x i } L 1 ∈ [1 , V ] L by a tokenizer of a vocabulary size V . The language model, parameterized by θ , makes use of an embedding function g θ which independently transforms each token into a D -dimensional continuous embedding t = { t i } L 1 :

Original:  { g θ ( x i ) } L 1 ∈ R L × D , a Transformer encoder f θ which computes interactions between all input tokens and outputs contextualizedrepresentations t ′ =
Revised: = { g θ ( x i ) } L 1 ∈ R L × D , a Transformer encoder f θ which computes interactions between all input tokens and outputs contextualized representations t ′ =

Original: { t ′ i } L 1 , and a masked language modeling (MLM) classifier head m θ whichindependently maps the D -dimensional continuous embedding for each token t ′ i to a vector of logitsparameterizing a categorical distribution over the vocabulary V .
Revised: { t ′ i } L 1 , and a masked language modeling (MLM) classifier head m θ which independently maps the D -dimensional continuous embedding for each token t ′ i to a vector of logits parameterizing a categorical distribution over the vocabulary V .

Original: This distribution is referred to bylog p θ ( x ) :
Revised: This distribution is referred to by log p θ ( x ) :

Original: = { m θ ( t ′ i ) }
Revised: = { m θ ( t ′ i ) }

Original: L 1 ∈ R L × V .
Revised: L 1 ∈ R L × V .

Original: We assume that the language model is pretrained, i.e. θ has beenoptimised with a standard MLM objective [15] on a large dataset of text from the Web.
Revised: We assume that the language model is pretrained, i.e. θ has been optimised with a standard MLM objective [17] on a large dataset of text from the Web.

Original: We show in
Revised: We show in

Original: Section 4.2 that this text-only pretraining has a crucial importance for zero-shot VideoQA.
Revised: Section 4.2 that this text-only pretraining has a crucial importance for zero-shot VideoQA.

Original: Pretrained Video Encoder.
Revised: Pretrained Video Encoder.

Original: The video is represented by a sequence of frames y =
Revised: The video is represented by a sequence of frames y =

Original: { y i } T 1 .
Revised: { y i } T 1 .

Original: Eachframe is forwarded separately through a visual backbone h ϕ , which outputs one feature vector perframe u = { u i } T 1 : = { h ϕ ( y i ) } T 1 ∈ R T × D u .
Revised: Each frame is forwarded separately through a visual backbone h ϕ , which outputs one feature vector per frame u = { u i } T 1 := { h ϕ ( y i ) } T 1 ∈ R T × D u .

Original: In detail, the visual backbone is CLIP ViT-L/14 [16, 68]at resolution 224 × 224 pixels, pretrained to map images to text descriptions with a contrastive [-losson 
 M-] Web-scraped image-text pairs.
Revised: In detail, the visual backbone is CLIP ViT-L/14 [18, 71] at resolution 224 × 224 pixels, pretrained to map images to text descriptions with a contrastive [-losson 
 M-]loss on 400M Web-scraped image-text pairs.

Original: The backbone is kept frozen throughout our experiments.
Revised: The backbone is kept frozen throughout our experiments.

Original: Note that a CLIP-baseline for zero-shot VideoQA results in poor performance, see Section 4.4.
Revised: Note that a CLIP-baseline for zero-shot VideoQA results in poor performance, see Section 4.4.

Original: Connecting the Frozen Language and Frozen Vision components.
Revised: Connecting the Frozen Language and Frozen Vision components.

Original: The video features are incorporated into the language model as a prompt [46, 54, 108] v of length T (Figure 2, left).
Revised: The video features are incorporated into the language model as a prompt [49, 57, 113] v of length T (Figure 2, left).

Original: Thisprompt is obtained by linearly mapping the visual features u to the text token embedding space viaa visual-to-text projection P ∈ R D u × D , i.e. v = { v i } T 1 := { P ( u i ) } T 1 .
Revised: This prompt is obtained by linearly mapping the visual features u to the text token embedding space via a visual-to-text projection P ∈ R D u × D , i.e. v = { v i } T 1 := { P ( u i ) } T 1 .

Original: The prompt is then concatenated with the text embeddings before being forwarded to the transformer encoder that models jointvisual-linguistic interactions.
Revised: The prompt is then concatenated with the text embeddings before being forwarded to the transformer encoder that models joint visual-linguistic interactions.

Original: We show in Section 4.2 that incorporating the input video considerablyimproves zero-shot VideoQA results.
Revised: We show in Section 4.2 that incorporating the input video considerably improves zero-shot VideoQA results.

Original: In addition, to learn powerful multi-modal interactions whilekeeping the transformer encoder weights frozen , we equip the transformer encoder with lightweightadapter modules A [26] (Figure 2, right).
Revised: In addition, to learn powerful multi-modal interactions while keeping the transformer encoder weights frozen , we equip the transformer encoder with lightweight adapter modules A [28] (Figure 2, right).

Original: We use an adapter which transforms the hidden state z witha multi-layer perceptron transformation and a residual connection, i.e. A ( z ) =
Revised: We use an adapter which transforms the hidden state z with a multi-layer perceptron transformation and a residual connection, i.e. A ( z ) =

Original: z
Revised: z

Original: + W up ψ ( W down z )with W down ∈ R D
Revised: + W up ψ ( W down z ) with W down ∈ R D

Original: × D h , W up ∈ R D h × D , D the hidden dimension of the transformer, D h the bottleneck dimension, and ψ a ReLU activation function.
Revised: × D h , W up ∈ R D h × D , D the hidden dimension of the transformer, D h the bottleneck dimension, and ψ a ReLU activation function.

Original: D h is typically set to be smaller than D such thatthe adapters are lightweight.
Revised: D h is typically set to be smaller than D such that the adapters are lightweight.

Original: In detail, we add an adapter module before the layer normalization, aftereach self-attention layer and each feed-forward layer of the transformer encoder.
Revised: In detail, we add an adapter module before the layer normalization, after each self-attention layer and each feed-forward layer of the transformer encoder.

Original: 3.2 Cross-modal training
Revised: 3.2 Cross-modal training

Original: We wish to train the newly added modules introduced in the previous section (shown in orange in
Revised: We wish to train the newly added modules introduced in the previous section (shown in orange in

Original: Figure 2) for the VideoQA task.
Revised: Figure 2) for the VideoQA task.

Original: This is hard because we assume that no explicit manual annotationfor the VideoQA task is available, such annotations being expensive and therefore hard to obtain atscale.
Revised: This is hard because we assume that no explicit manual annotation for the VideoQA task is available, such annotations being expensive and therefore hard to obtain at scale.

Original: Instead we train our architecture using only readily-available video-caption pairs scraped fromthe Web.
Revised: Instead we train our architecture using only readily-available video-caption pairs scraped from the Web.

Original: Such data is easy to obtain [4, 64, 104], ensuring the scalability of our approach.
Revised: Such data is easy to obtain [6, 67, 109], ensuring the scalability of our approach.

Original: During training, we keep the weights of the pretrained BiLM and pretrained visual backbone frozenas previously explained.
Revised: During training, we keep the weights of the pretrained BiLM and pretrained visual backbone frozen as previously explained.

Original: We train from scratch the parameters of (i) the visual-to-text projectionmodule P and (ii) the adapter modules A .
Revised: We train from scratch the parameters of (i) the visual-to-text projection module P and (ii) the adapter modules A .

Original: We show in Section 4.2 the importance of freezing the
Revised: We show in Section 4.2 the importance of freezing the

Original: BiLM weights combined with training the adapter modules.
Revised: BiLM weights combined with training the adapter modules.

Original: Note that all normalization layers [3] ofthe pretrained BiLM are also updated to adjust to the new distribution of the training data.
Revised: Note that all normalization layers [5] of the pretrained BiLM are also updated to adjust to the new distribution of the training data.

Original: We denoteall the trainable parameters of our model by the subscript µ .
Revised: We denote all the trainable parameters of our model by the subscript µ .

Original: In practice, they sum up to about 5% ofthe BiLM parameters, hence the training of our model is computationally efficient.
Revised: In practice, they sum up to about 5% of the BiLM parameters, hence the training of our model is computationally efficient.

Original: We use a visually-conditioned masked language modeling objective (MLM), in which some texttokens { x m } are randomly masked and the model has to predict these tokens based on the surroundingtext tokens and the video input.
Revised: We use a visually-conditioned masked language modeling objective (MLM), in which some text tokens { x m } are randomly masked and the model has to predict these tokens based on the surrounding text tokens and the video input.

Original: Formally, we minimize the following loss:
Revised: Formally, we minimize the following loss:

Original: [Equation]
Revised: [Equation]

Original: where ˜ x is the corrupted text sequence, y is the sequence of video frames, p µ (˜ x, y )
Revised: where ˜ x is the corrupted text sequence, y is the sequence of video frames, p µ (˜ x, y )

Original: x
Revised: x

Original: m m is the probabilityfor the (masked) m -th token in ˜ x to be x m , and M is the number of masks in the sequence ˜ x .
Revised: m m is the probability for the (masked) m -th token in ˜ x to be x m , and M is the number of masks in the sequence ˜ x .

Original: Indetail, we follow [15] and corrupt 15% of text tokens, replacing them 80% of the time with a masktoken, 10% of the time with the same token and 10% of the time with a randomly sampled token.
Revised: In detail, we follow [17] and corrupt 15% of text tokens, replacing them 80% of the time with a mask token, 10% of the time with the same token and 10% of the time with a randomly sampled token.

Original: 3.3 Adapting to downstream tasks
Revised: 3.3 Adapting to downstream tasks

Original: After training, our model is able to fill gaps in the input text given an input video together with leftand right textual context as part of the input text.
Revised: After training, our model is able to fill gaps in the input text given an input video together with left and right textual context as part of the input text.

Original: We wish to apply our model out-of-the-box topredict an answer given a question about a video.
Revised: We wish to apply our model out-of-the-box to predict an answer given a question about a video.

Original: The video can optionally come with textual subtitlesobtained using automatic speech recognition.
Revised: The video can optionally come with textual subtitles obtained using automatic speech recognition.

Original: To avoid using manual supervision, we formulate thedownstream tasks in cloze form [73, 87], i.e. such that the model only has to fill-in a mask token inthe input prompt similarly to the MLM objective optimized during training.
Revised: To avoid using manual supervision, we formulate the downstream tasks in cloze form [76, 90], i.e. such that the model only has to fill-in a mask token in the input prompt similarly to the MLM objective optimized during training.

Original: The adaptation to thedownstream tasks brings several challenges, as described next.
Revised: The adaptation to the downstream tasks brings several challenges, as described next.

Original: First, we describe how we formulatethe input text prompts for several downstream tasks.
Revised: First, we describe how we formulate the input text prompts for several downstream tasks.

Original: Then, we explain how we map the mask tokenfrom the input text prompt to an answer via a frozen answer embedding module.
Revised: Then, we explain how we map the mask token from the input text prompt to an answer via a frozen answer embedding module.

Original: Finally, we presenthow we finetune our architecture in a supervised setting.
Revised: Finally, we present how we finetune our architecture in a supervised setting.

Original: Input prompt engineering.
Revised: Input prompt engineering.

Original: We describe how we design the input text prompts for several downstream video-language tasks.
Revised: We describe how we design the input text prompts for several downstream video-language tasks.

Original: Each downstream task is formulated as a masked language modelingproblem.
Revised: Each downstream task is formulated as a masked language modeling problem.

Original: This allows us to apply FrozenBiLM
Revised: This allows us to apply FrozenBiLM

Original: out-of-the-box.
Revised: out-of-the-box.

Original: A [CLS] token and a [SEP] token arerespectively inserted at the start and the end of each sequence following [15].
Revised: A [CLS] token and a [SEP] token are respectively inserted at the start and the end of each sequence following [17].

Original: Open-ended VideoQA.
Revised: Open-ended VideoQA.

Original: Given a question and a video, the task is to find the correct answer in a largevocabulary A of about 1K answers.
Revised: Given a question and a video, the task is to find the correct answer in a large vocabulary A of about 1K answers.

Original: Answers are concise, i.e. the great majority of answers consist ofone word [29, 94, 96, 103].
Revised: Answers are concise, i.e. the great majority of answers consist of one word [32, 98, 101, 108].

Original: We design the following prompt:“[CLS]
Revised: We design the following prompt:

Original: Question: <Question>? Answer: [MASK]. Subtitles: <Subtitles> [SEP]”
Revised: [Equation]

Original: Multiple-choice VideoQA.
Revised: Multiple-choice VideoQA.

Original: Given a question and a video, the task is to find the correct answerin a small number of candidates C , typically up to 5 choices [43, 51].
Revised: Given a question and a video, the task is to find the correct answer in a small number of candidates C , typically up to 5 choices [46, 54].

Original: We set the vocabulary to
Revised: We set the vocabulary to

Original: A =
Revised: A =

Original: [ Yes , No ] and compute a confidence score for each candidate by using the following prompt:“[CLS] Question: <Question>?
Revised: [ Yes , No ] and compute a confidence score for each candidate by using the following prompt: “ [Equation] Question [Equation]

Original: Is it ’<
Revised: Is it ’

Original: Answer Candidate>’? [MASK].
Revised: [Equation]

Original: Subtitles:<Subtitles>
Revised: [Equation] Subtitles:

Original: [SEP]”
Revised: [Equation]

Original: We choose the best option by selecting the candidate with the highest Yes logit value.
Revised: We choose the best option by selecting the candidate with the highest Yes logit value.

Original: Video-conditioned fill-in-the-blank task.
Revised: Video-conditioned fill- in-the-blank task.

Original: Given a video and a sentence with a blank space, the task isto fill in the blank with the correct word from a vocabulary A of about 1K answers.
Revised: Given a video and a sentence with a blank space, the task is to fill in the blank with the correct word from a vocabulary A of about 1K answers.

Original: We replace theblank in the sentence with a mask token, and design the following prompt:“[CLS < Sentence with a
Revised: We replace the blank in the sentence with a mask token, and design the following prompt:

Original: MASK] token>. Subtitles: <Subtitles> [SEP]” Note that all prompts are prepended with the video prompt (see Section 3.1) before being forwardedto the transformer encoder.
Revised: Note that all prompts are prepended with the video prompt (see Section 3.1) before being forwarded to the transformer encoder.

Original: Answer embedding module.
Revised: Answer embedding module.

Original: For each downstream task, we wish to map the mask token in theinput text prompt to an actual answer prediction in the set of possible answers A , as described above.
Revised: For each downstream task, we wish to map the mask token in the input text prompt to an actual answer prediction in the set of possible answers A , as described above.

Original: For this we use the frozen MLM classifier head m θ .
Revised: For this we use the frozen MLM classifier head m θ .

Original: However, m θ ∈ R V × D covers V different tokenswhere V >> N and N ≈ 1 , 000 is the size of A .
Revised: However, m θ ∈ R V × D covers V different tokens where V >> N and N ≈ 1 , 000 is the size of A .

Original: Therefore, we introduce a task-specific answerclassification head l which linearly maps a contextualized mask representation t ′ i to a vector of logitsparameterizing a categorical distribution over the vocabulary A , i.e. l ∈ R N × D .
Revised: Therefore, we introduce a task-specific answer classification head l which linearly maps a contextualized mask representation t ′ i to a vector of logits parameterizing a categorical distribution over the vocabulary A , i.e. l ∈ R N × D .

Original: We set the weights of this answer module l with the corresponding weights of the pretrained MLM classifier m θ forone-token answers.
Revised: We set the weights of this answer module l with the corresponding weights of the pretrained MLM classifier m θ for one-token answers.

Original: In the case of multi-tokens answers, we average the weights of their differenttokens.
Revised: In the case of multi-token answers, we average the weights of their different tokens.

Original: We, hence, enable zero-shot inference at test time.
Revised: We, hence, enable zero-shot inference at test time.

Revised: We also discuss other alternative strategies to handle multi-token answers in the Supplementary Material.

Original: Fully-supervised training.
Revised: Fully-supervised training.

Original: In order to also evaluate our approach on fully-supervised benchmarks,we explore finetuning  our model on datasets that provide manual annotations for the target task.
Revised: To evaluate our approach on fully-supervised benchmarks, we also explore finetuning of our model on datasets that provide manual annotations for the target task.

Original: Forthis, we train the same parameters as explained in Section 3.2, i.e. the transformer weights and theanswer embedding moduleare frozen .
Revised: To this end, we train the same parameters as explained in Section 3.2, while keeping the transformer weights and the answer embedding module frozen .

Original: For open-ended VideoQA and video-conditioned fill-in-theblank, we use a cross-entropy loss on the task-specific vocabulary A .
Revised: For open-ended VideoQA and video-conditioned fill-in-theblank, we use a cross-entropy loss on the task-specific vocabulary A .

Original: For multiple-choice VideoQA,we use a binary cross-entropy loss applied to each answer candidate.
Revised: For multiple-choice VideoQA, we use a binary cross-entropy loss applied to each answer candidate.

Original: We show in Section 4.5 thebenefit of freezing the language model weights during fully-supervised training.
Revised: We show in Section 4.5 the benefit of freezing the language model weights during fully-supervised training.

Original:  Experiments
Revised:  Experiments

Original: This section demonstrates the benefits of our FrozenBiLM framework and compares our method tothe state of the art.
Revised: This section demonstrates the benefits of our FrozenBiLM framework and compares our method to the state of the art.

Original: We first outline our experimental setup in Section 4.1.
Revised: We first outline our experimental setup in Section 4.1.

Original: We then present ablationstudies in Section 4.2.
Revised: We then present ablation studies in Section 4.2.

Original: Next we compare our bidirectional framework to its autoregressive variant in
Revised: Next we compare our bidirectional framework to its autoregressive variant in

Original: Section 4.3.
Revised: Section 4.3.

Original: The comparison to the state of the art in zero-shot VideoQA and qualitative results arepresented in Section 4.4.
Revised: The comparison to the state of the art in zero-shot VideoQA and qualitative results are presented in Section 4.4.

Original: Finally, we finetune our model on the VideoQA task in Section 4.5, wherewe show few-shot and fully-supervised results.
Revised: Finally, we finetune our model on the VideoQA task in Section 4.5, where we show few-shot and fully-supervised results.

Original: 4.1 Experimental setup
Revised: 4.1 Experimental setup

Original: Frozen bidirectional language model.
Revised: Frozen bidirectional language model.

Original: We use a tokenizer based on SentencePiece [38] with
Revised: We use a tokenizer based on SentencePiece [41] with

Original: V = 128 , 000 , and a bidirectional language model with 900M parameters, DeBERTa-V2-XLarge [23],trained with the MLM objective on a corpus of 160G text data.
Revised: V = 128 , 000 , and a bidirectional language model with 900M parameters, DeBERTa-V2-XLarge [25], trained with the MLM objective on a corpus of 160G text data.

Original: We also show how our approachgeneralizes to other MLM-pretrained bidirectional language models such as BERT [15] in Section 4.2.
Revised: We also show how our approach generalizes to other MLM-pretrained bidirectional language models such as BERT [17] in Section 4.2.

Original: Datasets.
Revised: Datasets.

Original: For training we use the publicly available WebVid10M dataset [4], which consists of10 million of video-text pairs scraped from the Shutterstock website where video captions areobtained from readily-available alt-text descriptions.
Revised: For training we use the publicly available WebVid10M dataset [6], which consists of 10 million of video-text pairs scraped from the Shutterstock website where video captions are obtained from readily-available alt-text descriptions.

Original: We evaluate results on eight downstreamdatasets covering a wide range of textual and video domains ( e.g. GIFs, YouTube videos, TVshows, movies), and multiple VideoQA paradigms: open-ended VideoQA ( iVQA [96], MSRVTTQA [94], MSVD-QA [94], ActivityNet-QA [103] and TGIF-QA FrameQA [29]), multiple-choice
Revised: We evaluate results on eight downstream datasets covering a wide range of textual and video domains ( e.g. GIFs, YouTube videos, TV shows, movies), and multiple VideoQA paradigms: open-ended VideoQA ( iVQA [101], MSRVTTQA [98], MSVD-QA [98], ActivityNet-QA [108] and TGIF-QA FrameQA [32]), multiple-choice

Original: VideoQA ( How2QA [51] and TVQA [43]) and video-conditioned fill-in-the-blank ( LSMDC -Fillin-the-blank [63]).
Revised: VideoQA ( How2QA [54] and TVQA [46]) and video-conditioned fill-in-the-blank ( LSMDC -Fillin-the-blank [66]).

Original: Unless stated otherwise, we report top-1 test accuracy using the original splitsfor training, validation and test.
Revised: Unless stated otherwise, we report top-1 test accuracy using the original splits for training, validation and test.

Original: For How2QA, we report results on the public validation set forcomparison with prior work [75, 96, 102].
Revised: For How2QA, we report results on the public validation set for comparison with prior work [78, 101, 107].

Original: For TVQA, we report results on the validation set for theablation studies and on the hidden test set for the comparison to the state of the art.
Revised: For TVQA, we report results on the validation set for the ablation studies and on the hidden test set for the comparison to the state of the art.

Revised: More details are included in the Supplementary Material.

Original: Implementation Details.
Revised: Implementation Details.

Original: The training for 2 epochs on WebVid10M lasts 20 hours on 8 Tesla [-V
-]
Revised: The training for 2 epochs on WebVid10M lasts 20 hours on 8 Tesla [-V
-]V

Original: GPUs.
Revised: GPUs.

Original: We give further details in the Supplementary Material, together with our code.
Revised: We give further details in the Supplementary Material.

Original: 4.2 Ablation studies
Revised: 4.2 Ablation studies

Original: In this section, we evaluate the zero-shot performance of different variants of our method.
Revised: In this section, we evaluate the zero-shot performance of different variants of our method.

Original: By default,we use the frozen pretrained DeBERTa-V2-XLarge language model and train the visual-to-textprojection layer together with adapters for 2 epochs on WebVid10M.We refer to this default modelas FrozenBiLM .
Revised: By default, we use the frozen pretrained DeBERTa-V2-XLarge language model and train the visual-to-text- [Table] [Table] projection layer together with adapters for 2 epochs on WebVid10M.

Revised: We refer to this default model as FrozenBiLM .

Original: This model uses three input modalities in terms of video, question, and speech.
Revised: This model uses three input modalities in terms of video, question, and speech.

Original: Ablation of the model training.
Revised: Ablation of the model training.

Original: We ablate the effect of initializing parameters of the languagemodel, freezing its weights and training adapters in Table 1.
Revised: We ablate the effect of initializing parameters of the language model, freezing its weights and training adapters in Table 1.

Original: We observe that the language modelpretraining is crucial.
Revised: We observe that the language model pretraining is crucial.

Original: Indeed, a model with randomly initialized language weights (row 1) performspoorly compared to models initialized with language pretrained weights (rows 2 to 4).
Revised: Indeed, a model with randomly initialized language weights (row 1) performs poorly compared to models initialized with language pretrained weights (rows 2 to 4).

Original: Moreover,the model which updates the language model weights (row 2) during cross-modal training performsconsiderably worse compared to variants that freeze them (rows 3 and 4).
Revised: Moreover, the model which updates the language model weights (row 2) during cross-modal training performs considerably worse compared to variants that freeze them (rows 3 and 4).

Original: This shows the benefit offreezing the language model for zero-shot VideoQA.
Revised: This shows the benefit of freezing the language model for zero-shot VideoQA.

Original: We also notice the benefit of the adapter layersby comparing rows 3 and 4, especially for multiple-choice datasets.
Revised: We also notice the benefit of the adapter layers by comparing rows 3 and 4, especially for multiple-choice datasets.

Original: Finally, we note that trainingvariants with the frozen language model is twice faster compared to updating all parameters, as thereis a significantly lower number of parameters to be trained.[Table] [Table]
Revised: Finally, we note that training variants with the frozen language model is twice faster compared to updating all parameters, as there is a significantly lower number of parameters to be trained.

Original: Impact of modalities.
Revised: Impact of modalities.

Original: Table 2 shows the impact of the visual and speech modalities on the zero-shotperformance of our model.
Revised: Table 2 shows the impact of the visual and speech modalities on the zero-shot performance of our model.

Original: First, we evaluate the text-only performance of our model using neithervisual input nor speech input in row 1.
Revised: First, we evaluate the text-only performance of our model using neither visual input nor speech input in row 1.

Original: We can observe that adding speech (row 2) marginallyimproves the results and that the importance of speech highly depends on the dataset.
Revised: We can observe that adding speech (row 2) marginally improves the results and that the importance of speech highly depends on the dataset.

Original: When addingvision (rows 3 and 4), the performance increases significantly, e.g. +13.6% accuracy on iVQA and+22.1% on MSVD-QA between rows 4 and 2.
Revised: When adding vision (rows 3 and 4), the performance increases significantly, e.g. +13.6% accuracy on iVQA and +22.1% on MSVD-QA between rows 4 and 2.

Original: Finally, the model with vision also benefits from thespeech, e.g. +16.5% accuracy on How2QA and +29.5% accuracy on TVQA (compare rows 3 and 4).
Revised: Finally, the model with vision also benefits from the speech, e.g. +16.5% accuracy on How2QA and +29.5% accuracy on TVQA (compare rows 3 and 4).

Original: Note that in practice, speech is missing for many videos, as we obtain the speech directly from the
Revised: Note that in practice, speech is missing for many videos, as we obtain the speech directly from the

Original: YouTube API and many videos are no longer available.
Revised: YouTube API and many videos are no longer available.

Original: Exceptions are How2QA and TVQA forwhich the authors [43, 52] provide speech for all videos.
Revised: Exceptions are How2QA and TVQA for which the authors [46, 55] provide speech for all videos.

Original: Consequently, we have speech data foronly 44.3%,14.2%, 8.2%, 7.1% and 25.3% of test samplesin LSMDC-FiB, iVQA, MSRVTT-QA,
Revised: Consequently, we have speech data for only 44.3%, samples

Original: MSVD-QA and ActivityNet-QA respectively.
Revised: LSMDC

Original: GIFs in TGIF-QA do not contain speech.
Revised: FiB iVQA

Original: [Table]
Revised: [Table]

Original: Size of the cross-modal training dataset.
Revised: Size of the cross-modal training dataset.

Original: Zero-shot resultsof FrozenBiLM after training for a fixed number of iterationson different fractions of WebVid10M are shown in Table 3.
Revised: Zero-shot results of FrozenBiLM after training for a fixed number of iterations on different fractions of WebVid10M are shown in Table 3.

Original: We construct these subsets such that larger subsets includethe smaller ones.
Revised: We construct these subsets such that larger subsets include the smaller ones.

Original: We find that performance increases monotonically with more multi-modal training data.
Revised: We find that performance increases monotonically with more multi-modal training data.

Original: Size of the language model.
Revised: Size of the language model.

Original: In Table 4, we ablate the importance of the language model size for thezero-shot performance.
Revised: In Table 4, we ablate the importance of the language model size for the zero-shot performance.

Original: Note that when comparing different language models, we use no adapters toavoid biases related to the choice of the bottleneck dimension hyperparameter [26].
Revised: Note that when comparing different language models, we use no adapters to avoid biases related to the choice of the bottleneck dimension hyperparameter [28].

Original: We find that usingthe 900M-parameter DeBERTA-V2-XLarge (row 6) outperforms the 300M-parameter BERT-Large(row 5) which also improves over the 100M-parameter BERT-Base (row 4). 
Revised: We find that using the 900M-parameter DeBERTA-V2-XLarge (row 6) outperforms the 300M-parameter BERT-Large (row 5) which also improves over the 100M-parameter BERT-Base (row 4). [Table] [Table]

Original: Importance of the suffix.
Revised: Importance of the suffix.

Original: Our text input prompts include a suffix after the mask token whichconsists in a point and an end-of-sentence token for the variant without speech (or a point followedby the speech subtitles for the variant with speech).
Revised: Our text input prompts include a suffix just to the right of the mask token which consists in a point and an end-of-sentence token for the variant without speech (or a point followed by the speech subtitles for the variant with speech).

Original: We found that removing this suffix leads toa considerable drop of performance ( e.g. the test accuracy on MSVD-QA in  row 3 Table 2 dropsfrom 33.7% to 2.8%).
Revised: We found that removing this suffix leads to a considerable drop of performance ( e.g. the test accuracy on MSVD-QA in the row 3 of Tabledrops from 33.7% to 2.8%).

Revised: Note that we do not observe such a large drop in performance when removing the [CLS] token e.g. the accuracy on MSVD-QA drops only from 33.8% to 33.2%.

Original: This shows that the bidirectional nature of our framework is a key factor forthe performance.
Revised: This shows that the bidirectional nature of our framework is a key factor for the performance.

Original: Intuitively, this suffix forces the model to provide a concise answer.
Revised: Intuitively, this suffix forces the model to provide a concise answer.

Original: Such a hardconstraint cannot be given to unidirectional autoregressive models compared next in Section 4.3.
Revised: Such a hard constraint cannot be given to unidirectional autoregressive models compared next in Section 4.3.

Revised: We further ablate the importance of the prompt design in the Supplementary Material.

Original: 4.3 Comparison with frozen autoregressive models
Revised: 4.3 Comparison with frozen autoregressive models

Original: In this section, we compare our bidirectional framework using language models of various sizesto the larger, autoregressive GPT-based counterparts recently used for zero-shot image questionanswering [88, 99].
Revised: In this section, we compare our bidirectional framework using language models of various sizes to the larger, autoregressive GPT-based counterparts recently used for zero-shot image question answering [91, 104].

Original: For fair comparison, we adapt autoregressive models to video and languageinputs similarly as our bidirectional models.
Revised: For fair comparison, we adapt autoregressive models to video and language inputs similarly as our bidirectional models.

Original: In detail, autoregressive variants train a similar visual-totext projection by using a left-to-right language modeling loss [88].
Revised: In detail, autoregressive variants train a similar visual-totext projection by using a left-to-right language modeling loss [91].

Original: All models in our comparison aretrained on WebVid10M for the same number of epochs.
Revised: All models in our comparison are trained on WebVid10M for the same number of epochs.

Original: At inference, autoregressive variants use thesame template as [88] to which we prepend speech subtitles, greedily decode sequences as [88], anduse the same answer vocabulary as bidirectional models.
Revised: At inference, autoregressive variants use the same template as [91] to which we prepend speech subtitles, greedily decode sequences as [91], and use the same answer vocabulary as bidirectional models.

Original: Autoregressive variants select the top answerthat maximizes the log-likelihood when appended to the question prompt.
Revised: Autoregressive variants select the top answer that maximizes the log-likelihood when appended to the question prompt.

Original: Here also, we use noadapters for all models, such that the architecture of autoregressive models closely follows [88].
Revised: Here also, we use no adapters for all models, such that the architecture of autoregressive models closely follows [91].

Original: Thisis to avoid biases related to the tuning of the bottleneck reduction hyperparameter in the adapters [26]. [Table] [Table] [Table]
Revised: This is to avoid biases related to the tuning of the bottleneck reduction hyperparameter in the adapters [28].

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] [Figure] We compare autoregressive and bidirectional language models in terms of accuracy and efficiencyin Table 4.
Revised: We compare autoregressive and bidirectional language models in terms of accuracy and efficiency in Table 4.

Original: We observe that our bidirectional framework (rows 4-6) achieves significantly betterzero-shot performance-efficiency trade-off compared to its autoregressive counterpart (rows 1-3).
Revised: We observe that our bidirectional framework (rows 4-6) achieves significantly better zero-shot performance-efficiency trade-off compared to its autoregressive counterpart (rows 1-3).

Original: Forinstance, our framework with BERT-Base [15] (row 4) outperforms the autoregressive variant basedon GPT-Neo-1.3B [5] (row 1) which uses 12 times more parameters and 8 times more training time.
Revised: For instance, our framework with BERT-Base [17] (row 4) outperforms the autoregressive variant based on GPT-Neo-1.3B [7] (row 1) which uses 12 times more parameters and 8 times more training time.

Original: Likewise, our framework with DeBERTa-V2-XLarge [23] (row 6) improves over the autoregressivevariant based on GPT-J-6B [91] (row 3) that has 7 times more parameters and requires 5 times moretraining time, showing the efficiency of our bidirectional framework for zero-shot VideoQA.4.4 Comparison to the state of the art for zero-shot VideoQA
Revised: Likewise, our framework with DeBERTa-V2-XLarge [25] (row 6) improves over the autoregressive variant based on GPT-J-6B [94] (row 3) that has 7 times more parameters and requires 5 times more training time, showing the efficiency of our bidirectional framework for zero-shot VideoQA. 4.4 Comparison to the state of the art for zero-shot VideoQA

Original: Quantitative comparison.
Revised: Quantitative comparison.

Original: Table 5 presents results of our method in comparison to the state ofthe art in zero-shot VideoQA settings [96], i.e. when using no manually annotated visual data fortraining.
Revised: Table 5 presents results of our method in comparison to the state of the art in zero-shot VideoQA settings [101], i.e. when using no manually annotated visual data for training.

Original: Our approach outperforms previous methods by a significant margin on all 8 datasets. 
Revised: Our approach outperforms previous methods by a significant margin on all 8 datasets. In [Figure] [Figure]

Revised: HCRN [45] 44M

Revised: [Figure]

Original: Inparticular, FrozenBiLM outperforms Reserve [105], which is trained on one billion YouTube videoclips jointly with vision, language and sound, Just Ask [97], which uses large-scale automaticallygenerated VideoQA data, and a CLIP baseline [68] matching the text concatenating question andanswer to the middle frame of the video.
Revised: [Figure] [Figure] HERO [54] 119M — — — — — — [Figure]

Revised: [Figure]

Revised: ClipBERT [48] 114M — —

Revised: [Figure]

Revised: [Figure]

Revised: — — Just Ask [102] 157M —

Revised: [Table]

Revised: [Table]

Revised: particular, FrozenBiLM outperforms Reserve [110], which is trained on one billion YouTube video clips jointly with vision, language and sound, Just Ask [102], which uses large-scale automatically generated VideoQA data, and a CLIP baseline [71] matching the text concatenating question and answer to the middle frame of the video.

Revised: Note that FrozenBiLM performs competitively even when using no speech input.

Original: Finally, we note that BLIP [50] has a different definitionof zero-shot where a network finetuned on the image-VQA dataset [2] is evaluated directly onopen-ended VideoQA datasets.
Revised: Finally, we note that BLIP [53] has a different definition of zero-shot where a network finetuned on the image-VQA dataset [4] is evaluated directly on VideoQA datasets.

Original: Our Supplementary Material presents results where we outperform BLIP [50] in their settings and also includes an analysis of results by question type.
Revised: Our Supplementary Material presents results where we outperform BLIP [53] in their settings and also includes an analysis of results by question type.

Original: In summary, ourevaluation shows the excellent performance of our model in the challenging zero-shot setup.
Revised: In summary, our evaluation shows the excellent performance of our model in the challenging zero-shot setup.

Original: Qualitative results.
Revised: Qualitative results.

Original: Figure 3 illustrates qualitative results of zero-shot VideoQA for our FrozenBiLMmodel and compares them to Just Ask [97], as well as to variants of our approach that do not freeze thelanguage model ( UnFrozenBiLM ) and use no visual modality (text-only), as evaluated in Section 4.2.
Revised: Figure 3 illustrates qualitative results of zero-shot VideoQA for our FrozenBiLM model and compares them to Just Ask [102], as well as to variants of our approach that do not freeze the language model ( UnFrozenBiLM ) and use no visual modality (text-only), as evaluated in Section 4.2.

Original: We observe that the unfrozen variant can predict answers that lack text-only commonsense reasoning,e.g. in the third example, it is unlikely that a sitting man is swimming.
Revised: We observe that the unfrozen variant can predict answers that lack text-only commonsense reasoning, e.g. in the third example, it is unlikely that a sitting man is swimming.

Original: The text-only variant does havestrong language understanding, but makes visually-unrelated predictions.
Revised: The text-only variant does have strong language understanding, but makes visually-unrelated predictions.

Original: In contrast, consistentlywith our quantitative results, our model FrozenBiLM is able to correctly answer various questions,showing both a strong textual commonsense reasoning and a complex multi-modal understanding.
Revised: In contrast, consistently with our quantitative results, our model FrozenBiLM is able to correctly answer various questions, showing both a strong textual commonsense reasoning and a complex multi-modal understanding.

Original: [Table] [Table]
Revised: We show additional qualitative results in the Supplementary Material.

Original: 4.5 Freezing the BiLM is also beneficial in supervised settings
Revised: 4.5 Freezing the BiLM is also beneficial in supervised settings

Original: Fully-supervised VideoQA.
Revised: Fully- supervised VideoQA

Original: We next present an evaluation in a supervised setup where we finetune
Revised: We next present an evaluation in a supervised setup where we finetune

Original: FrozenBiLM on a downstream VideoQA task.
Revised: FrozenBiLM on a downstream VideoQA task.

Original: We emphasize that we also keep our pretrained languagemodel weights frozen all throughout finetuning.
Revised: We emphasize that we also keep our pretrained language model weights frozen all throughout finetuning.

Original: As shown in Table 6, our approach improves the stateof the art on LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA.
Revised: As shown in Table 6, our approach improves the state of the art on LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA.

Original: Inparticular, FrozenBiLM outperforms strong recent baselines such as All-in-one [90] on 2/3 datasets,
Revised: In particular, FrozenBiLM outperforms strong recent baselines such as All-in-one [93] on 2/3 datasets,

Original: VIOLET [19] on 3/4 datasets and MERLOT [104] on 4/5 datasets.
Revised: VIOLET [21] on 3/4 datasets and MERLOT [109] on 4/5 datasets.

Original: Our approach has significantlyless trainable parameters compared to the state of the art [19, 90, 104] as we freeze the weights of thepretrained language model.
Revised: Our approach has significantly less trainable parameters compared to the state of the art [21, 93, 109] as we freeze the weights of the pretrained language model.

Original: We ablate this major difference in Table 6, and find that our FrozenBiLMwith the frozen language model performs better and trains twice faster compared to UnFrozenBiLMwhere we update the language model during training.
Revised: We ablate this major difference in Table 6, and find that our FrozenBiLM with the frozen language model performs better and trains twice faster compared to UnFrozenBiLM where we update the language model during training.

Original: This shows that freezing the transformerencoder is not only beneficial for zero-shot but also in fully-supervised settings, therefore suggestingthat our FrozenBiLM framework also provides a parameter-efficient solution for VideoQA training.
Revised: This shows that freezing the language model is not only beneficial for zero-shot but also in fully-supervised settings, therefore suggesting that our FrozenBiLM framework also provides a parameter-efficient solution for VideoQA training.

Revised: Finally, we note that FrozenBiLM performs competitively even without speech input, although speech helps significantly for the performance on LSMDC, How2QA and TVQA.

Original: Few-shot VideoQA.
Revised: Few-shot VideoQA.

Original: The low number of trainable parameters when training FrozenBiLM makes itparticularly well-suited in the low data regime.
Revised: The low number of trainable parameters when training FrozenBiLM makes it particularly well-suited in the low data regime.

Original: To verify this, we explore a few-shot VideoQA settingwhere we finetune our pretrained model using varying fractions of VideoQA training data.
Revised: To verify this, we explore a few-shot VideoQA setting where we finetune our pretrained model using varying fractions of VideoQA training data.

Original: From
Revised: From

Original: Table 7 we observe significant improvements over zero-shot when using only 1% of training data.
Revised: Table 7 we observe significant improvements over zero-shot when using only 1% of training data.

Revised: Finally, we show in Supplementary Material that freezing the BiLM highly benefits the few-shot performance, consistently with the results in the zero-shot and fully-supervised settings.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented FrozenBiLM , a framework that extends frozen bidirectional language models tomulti-modal inputs by training additional modules on Web-scraped data, and
Revised: We have presented FrozenBiLM , a framework that extends frozen bidirectional language models to multi-modal inputs by training additional modules on Web-scraped data, and

Original: that tackles zero-shot
Revised: that tackles zero-shot

Original: VideoQA through masked language modeling.
Revised: VideoQA through masked language modeling.

Original: We have provided extensive ablation studies andshown the efficiency of our framework compared to its autoregressive variant.
Revised: We have provided extensive ablation studies and shown the efficiency of our framework compared to its autoregressive variant.

Original: FrozenBiLM improvesthe state-of-the-art zero-shot VideoQA on various datasets, performs competitively in fully-supervisedsettings and exhibits strong performance in the few-shot VideoQA setting we newly introduce.
Revised: FrozenBiLM improves the state-of-the-art zero-shot VideoQA on various datasets, performs competitively in fully-supervised settings and exhibits strong performance in the few-shot VideoQA setting we newly introduce.

Original: Limitations.
Revised: Limitations.

Original: Promising directions not explored in this work include scaling the size of a bidirectionallanguage model to several billion parameters, and additional training on large datasets of YouTubevideos with accompanying speech transcripts and/or audio [105].
Revised: Promising directions not explored in this work include scaling the size of a bidirectional language model to several billion parameters, and additional training on large datasets of YouTube videos with accompanying speech transcripts and/or audio [110].

Original: Also, our model cannot be appliedout-of-the-box to complex multi-modal text generation tasks such as video captioning.
Revised: Also, our model cannot be applied out-of-the-box to complex multi-modal text generation tasks such as video captioning.

Original: Broader Impact.
Revised: Broader Impact.

Original: We have showed the superior compute-efficiency of our bidirectional frameworkcompared to autoregressive models for zero-shot VideoQA, and believe it is a step towards reducingthe environmental impact of such research and its applications [81].
Revised: We have showed the superior compute-efficiency of our bidirectional framework compared to autoregressive models for zero-shot VideoQA, and believe it is a step towards reducing the environmental impact of such research and its applications [84].

Original: In addition, our models mightreflect biases present in videos and captions from Shutterstock used to train ourfrozen model, the textdata used to train the language model or the images and captions used to train the visual backbone.
Revised: In addition, our models might reflect biases present in videos and captions from Shutterstock used to train our model, the text data used to train the language model or the images and captions used to train the visual backbone.

Original: Itis important to keep this in mind when deploying, analysing and building upon these models.Checklist
Revised: It is important to keep this in mind when deploying, analysing and building upon these models.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope? [Yes] See Section 3 for contribution (i), Sections 4.2 and 4.3for contribution (ii), and Sections 4.4 and 4.5 for contribution (iii). (b) Did you describe the limitations of your work? [Yes] See Section 5.
Revised: Acknowledgements.

Original: (c) Did you discuss any potential negative societal impacts of your work? [Yes] SeeSection 5. (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes]
Revised: This work was granted access to the HPC resources of IDRIS under the allocation 2022-AD011011670R2 made by GENCI.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoreticalresults. (b) Did you include complete proofs of all theoretical results?
Revised: The work was funded by a Google gift, the French government under management of Agence Nationale de la Recherche as part of the "Investissements d’avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute), the Louis Vuitton ENS

Original: [N/A] No theoretical results.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Our code,together with instructions needed to download and process the datasets we use, andinstructions needed to reproduce the main experimental results, is provided in a zip filein the Supplementary Material.(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: Chair on Artificial Intelligence, the European Regional Development Fund under project IMPACT (reg.

Original: [Yes] See Section 4.1 and Supplementary Material.(c)

Original: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We report them in the Supplementary Material, as errorbars are in general not reported [96, 102, 104, 105]. (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1 and Supplementary Material.
Revised: no.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1. (b) Did you mention the license of the assets?
Revised: CZ.02.1.01/0.0/0.0/15 003/0000468).

Original: [Yes] See Supplementary Material.(c)

Original: Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]We provide code in the Supplementary Material and plan to release trained models.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A] The datasets we use are publicly available and released for 
Revised: We thank anonymous reviewers for giving interesting feedback.

Original: non-commercial use only, and this is already specified in the license.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A]

Original: The datasets we use are based on websites such as YouTube which strictly remove videos that contain offensive content or do not follow their community guidelines. If you used crowdsourcing or conducted research with human subjects...
Revised: We thank Gaspard Beugnot, Clémence Bouvier and Pierre-Louis Guhur for proofreading.

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A] No crowdsourcing or conducted research with human subjects.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A] No crowdsourcing or conducted researchwith human subjects.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]

Original: No crowdsourcing or conducted research

Original: with human subjects.