Original: Egocentric Video-Language Pretraining
Revised: Egocentric Video-Language Pretraining

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Video-Language Pretraining (VLP), aiming to learn transferable representation toadvance a wide range of video-text downstream tasks, has recently received increasing attention.
Revised: Video-Language Pretraining (VLP), aiming to learn transferable representation toadvance a wide range of video-text downstream tasks, has recently received increasing attention.

Original: Dominant works that achieve strong performance rely on large-scale,3rd-person video-text datasets, such as HowTo100M.
Revised: Dominant works that achieve strong performance rely on large-scale,3rd-person video-text datasets, such as HowTo100M.

Original: In this work, we exploit therecently released Ego4D dataset to pioneer Egocentric VLP along three directions.(i)
Revised: In this work, we exploit therecently released Ego4D dataset to pioneer Egocentric VLP along three directions.(i)

Original: We create EgoClip, a 1st-person video-text pretraining dataset comprising 3.8Mclip-text pairs well-chosen from Ego4D, covering a large variety of human dailyactivities.
Revised: We create EgoClip, a 1st-person video-text pretraining dataset comprising 3.8Mclip-text pairs well-chosen from Ego4D, covering a large variety of human dailyactivities.

Original: (ii)
Revised: (ii)

Original: We propose a novel pretraining objective, dubbed as EgoNCE, whichadapts video-text contrastive learning to egocentric domain by mining egocentricaware positive and negative samples.
Revised: We propose a novel pretraining objective, dubbed as EgoNCE, whichadapts video-text contrastive learning to egocentric domain by mining egocentricaware positive and negative samples.

Original: (iii)
Revised: (iii)

Original: We introduce EgoMCQ, a developmentbenchmark that is close to EgoClip and hence can support effective validation andfast exploration of our design decisions regarding EgoClip and EgoNCE.
Revised: We introduce EgoMCQ, a developmentbenchmark that is close to EgoClip and hence can support effective validation andfast exploration of our design decisions regarding EgoClip and EgoNCE.

Original: Furthermore, we demonstrate strong performance on five egocentric downstream tasksacross three datasets: video-text retrieval on EPIC-KITCHENS-100; action recognition on Charades-Ego; and natural language query, moment query, and objectstate change classification on Ego4D challenge benchmarks.
Revised: Furthermore, we demonstrate strong performance on five egocentric downstream tasksacross three datasets: video-text retrieval on EPIC-KITCHENS-100; action recognition on Charades-Ego; and natural language query, moment query, and objectstate change classification on Ego4D challenge benchmarks.

Original:  Introduction
Revised:  Introduction

Original: With the recent interest boom in computer vision and natural language processing
Revised: With the recent interest boom in computer vision and natural language processing

Original: Video-Language
Revised: Video-Language

Original: Pretraining (VLP) has prevailed, which aims to learn strong and transferable video-language representation for powering a broad spectrum of video-text downstream tasks, such as video-textretrieval [1, 2, 3], video question answering [3, 4, 5, 6], video-captioning [7, 8, 9].
Revised: Pretraining (VLP) has prevailed, which aims to learn strong and transferable video-language representation for powering a broad spectrum of video-text downstream tasks, such as video-textretrieval [1, 2, 3], video question answering [3, 4, 5, 6], video-captioning [7, 8, 9].

Original: The success of
Revised: The success of

Original: VLP mainly stems from the availability of large-scale open-world video-text datasets [10] which contain a large number of videos sourced from the Web (e g., YouTube) and pair videos with associatedtextual information.
Revised: VLP mainly stems from the availability of large-scale open-world video-text datasets [10] which contain a large number of videos sourced from the Web (e g., YouTube) and pair videos with associatedtextual information.

Original: For instance, HowTo100M [10] collects 134 Khours of instructional videos accompanied by noisy narrations yielded from Automatic Speech Recognition (ASR).
Revised: For instance, HowTo100M [10] collects 134 K instructional videos accompaniedby noisy narrations yielded from Automatic Speech Recognition (ASR).

Original: WebVid-

Revised: WebVid-

Original: M [11]scrapes 2 .
Revised: M [11] scrapes2 .

Original: 5 M descriptive videos with well-formed long captions.
Revised: 5 M descriptive videos with well-formed long captions.

Original: Despite reaching an impressive data scale, videos in the existing video-text pretraining datasets areoften of 3rd-person views and might have been edited before posting on the web.
Revised: Despite reaching an impressive data scale, videos in the existing video-text pretraining datasets areoften of 3rd-person views and might have been edited before posting on the web.

Original: Yet, there is anoticeable domain gap between the existing video-text pretraining datasets and 1st-person view videossuch as those videos captured by wearable cameras or smart glasses.
Revised: Yet, there is anoticeable domain gap between the existing video-text pretraining datasets and 1st-person view videossuch as those videos captured by wearable cameras or smart glasses.

Original: Egocentric video has receivedincreasing interest from academia (e.g., activity recognition [12], activity anticipation [13], videosummarization [14]) and industry (various applications in robotics and augmented reality).
Revised: Egocentric video has receivedincreasing interests from academia (e.g., activity recognition [12], activity anticipation [13], videosummarization [14]) and industry (various applications in robotics and augmented reality).

Original: But, dueto such a domain gap, directly transferring the existing VLP models to egocentric downstream taskscannot fully unleash the potential of large-scale pretraining approaches, which we have confirmedin the later experimental section.
Revised: But, dueto such a domain gap, directly transferring the existing VLP models to egocentric downstream taskscannot fully unleash the potential of large-scale pretraining approaches, which we have confirmedin the later experimental section.

Original: To bridge this gap, we are motivated to develop Egocentric
Revised: To bridge this gap, we are motivated to develop Egocentric

Original: VLP models, which can greatly benefit various egocentric video downstream applications.
Revised: VLP models, which can greatly benefit various egocentric video downstream applications.

Original: However, existing egocentric video datasets are of small scale and domain-specific, making Egocentric
Revised: However, existing egocentric video datasets are of small scale and domain-specific, making Egocentric

Original: VLP prohibitive.
Revised: VLP prohibitive.

Original: As illustrated in Tab. 1, the formerly largest egocentric video dataset EPIC-
Revised: As illustrated in Tab. 1, the formerly largest egocentric video dataset EPIC-

Original: Dataset
Revised: Dataset

Original: Ego?
Revised: Ego?

Original: Domain
Revised: Domain

Original: Dur (
Revised: Dur (

Original: [Table]
Revised: [Table]

Original: # Clips # Texts
Revised: # Clips # Texts

Original: Example
Revised: Example

Original: [Table] [Table]
Revised: [Table] [Table]

Original: diverse
Revised: diverse

Original: [Table]
Revised: [Table]

Original: YouCook
Revised: YouCook

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: KITCHENS-100 [15] focuses on kitchens scenarios and its size is far smaller than the 3rd-personpretraining sets
Revised: KITCHENS-100 [15] focuses on kitchens scenarios and its size is far smaller than the 3rd-personpretraining sets

Original: WebVid-2M [11] and HowTo100M [10].
Revised: WebVid-2M [11] and HowTo100M [10].

Original: Fortunately, with the recent introduction ofthe massive-scale egocentric video dataset Ego4D [16], it becomes possible to unlock Egocentric VLP.
Revised: Fortunately, with the recent introduction ofthe massive-scale egocentric video dataset Ego4D [16], it becomes possible to unlock Egocentric VLP.

Original: Ego4D consists of 3 , 670 hours of videos with manually annotated narrations from 74 worldwidelocations, covering a large variety of daily-life scenarios and activities.
Revised: Ego4D consists of 3 , 670 hours of videos with manually annotated narrations from 74 worldwidelocations, covering a large variety of daily-life scenarios and activities.

Original: In this work, roused by the favorable scale and diversity of Ego4D, we make a significant effort topave the way for Egocentric VLP with the following steps:
Revised: In this work, roused by the favorable scale and diversity of Ego4D, we make a significant effort topave the way for Egocentric VLP with the following steps:

Original: (i) To address the aforementioned issue of lacking a suitable large-scale egocentric video-language
Revised: (i) To address the aforementioned issue of lacking a suitable large-scale egocentric video-language

Original: pretraining dataset
Revised: pretraining dataset

Original: , we create a video-text pretraining dataset EgoClip which contains a total of 3 .
Revised: , we create a video-text pretraining dataset EgoClip which contains a total of 3 .

Original: Mclean 1st-person clip-text pairs selected from Ego4D and cover diverse human daily activities.(ii)
Revised: Mclean 1st-person clip-text pairs selected from Ego4D and cover diverse human daily activities.(ii)

Original: To make full use of EgoClip for video-text representation learning, we propose a novel video-textcontrastive objective EgoNCE to address unique challenges in egocentric pretraining datasets.(iii)
Revised: To make full use of EgoClip for video-text representation learning, we propose a novel video-textcontrastive objective EgoNCE to address unique challenges in egocentric pretraining datasets.(iii)

Original: We create a development benchmark i.e. Egocentric Multiple-Choices-Question, dubbed as
Revised: We create a development benchmark i.e. Egocentric Multiple-Choices-Question, dubbed as

Original: EgoMCQ , which contains 39 K questions created from Ego4D and focuses on evaluating video-textalignment.
Revised: EgoMCQ , which contains 39 K questions created from Ego4D and focuses on evaluating video-textalignment.

Original: In contrast to other downstream benchmarks, EgoMCQ has less discrepancy with EgoClip, powering us to accurately validate and quickly iterate our designs of EgoClip and EgoNCE .(iv)
Revised: In contrast to other downstream benchmarks, EgoMCQ has less discrepancy with EgoClip, powering us to accurately validate and quickly iterate our designs of EgoClip and EgoNCE .(iv)

Original: We conduct extensive experiments to demonstrate the superiority of Egocentric VLP by transferring our pretrained representation to five egocentric downstream benchmarks.
Revised: We conduct extensive experiments to demonstrate the superiority of Egocentric VLP by transferring our pretrained representation to five egocentric downstream benchmarks.

Original: We achieve state-of-theart performances: 59 .
Revised: We achieve state-of-theart performances: 59 .

Original: 4% nDCG on video-text retrieval of EPIC-KITCHENS-100 [15], 32 . 1% mAPon action recognition of Charades-Ego [19], and significant boosts over three Ego4D challenges 1 :natural language query, moment query and object state change classification.
Revised: 4% nDCG on video-text retrieval of EPIC-KITCHENS-100 [15], 32 . 1% mAPon action recognition of Charades-Ego [19], and significant boosts over three Ego4D challenges 1 :natural language query, moment query and object state change classification.

Original:  Related Work
Revised:  Related Work

Original: Video-Language Pretraining.
Revised: Video-Language Pretraining.

Original: The introduction of large-scale video-text datasets [10, 11] hasenabled the emergence of VLP approaches to improve the video-text representation for variousvideo-language tasks [17, 23, 18, 7, 24], such as MIL-NCE [25] proposes to match clips with multiplecaptions close in temporal to adapt the video-text misalignment of HowTo100M [10].
Revised: The introduction of large-scale video-text datasets [10, 11] hasenabled the emergence of VLP approaches to improve the video-text representation for variousvideo-language tasks [17, 23, 18, 7, 24], such as MIL-NCE [25] proposes to match clips with multiplecaptions close in temporal to adapt the video-text misalignment of HowTo100M [10].

Original: Dominant
Revised: Dominant

Original: VLP methods can be classified into two groups, namely: joint- and dual-encoders.
Revised: VLP methods can be classified into two groups, namely: joint- and dual-encoders.

Original: The formercombines videos and texts as a single input to the encoder that performs the multimodal fusion.
Revised: The formercombines videos and texts as a single input to the encoder that performs the multimodal fusion.

Original: Forinstance, [26, 27] concatenate videos and texts together before feeding them to a unified transformer.
Revised: Forinstance, [26, 27] concatenate videos and texts together before feeding them to a unified transformer.

Original: Conversely, methods as [11, 25] exploit dual encoders to independently project the video and textinputs into a common space and minimize the distance between the paired representations.
Revised: Conversely, methods as [11, 25] exploit dual encoders to independently project the video and textinputs into a common space and minimize the distance between the paired representations.

Original: Theseapproaches are to be preferred in retrieval settings as they allow for efficient indexing of a singlemodality [2, 28].
Revised: Theseapproaches are to be preferred in retrieval settings as they allow for efficient indexing of a singlemodality [2, 28].

Original: For example, Frozen [11] employs two separate transformers to encode video andtext features and align them by video-text InfoNCE [29].
Revised: For example, Frozen [11] employs two separate transformers to encode video andtext features and align them by video-text InfoNCE [29].

Original: In our work, we adopt the Frozen [11] butextend its InfoNCE to EgoNCE via positive and negative sampling for egocentric-friendly pretraining.
Revised: In our work, we adopt the Frozen [11]but extend the InfoNCE to EgoNCE via positive and negative sampling for egocentric-friendlypretraining.

Original: Egocentric Video Datasets.
Revised: Egocentric Video Datasets.

Original: Egocentric videos, collected by participants using wearable cameras,offer a natural perspective of people’s daily activities and raise a range of challenging researchtopics [14, 20, 12, 13].
Revised: Egocentric videos, collected by participants using wearable cameras,offer a natural perspective of people’s daily activities and raise a range of challenging researchtopics [14, 20, 12, 13].

Original: Several egocentric video datasets have been developed in decades, e.g.,[22, 19, 30].
Revised: Several egocentric video datasets have been developed in decades, e.g.,

Original: However, since the collection of egocentric videos is expensive, previous egocentric

Original: [Figure]
Revised: [Figure]

Original: Video clip

Original: Narration

Original: EgoClip
Revised: EgoClip

Original: EgoMCQ
Revised: EgoMCQ

Original: [Figure]
Revised: [Figure]

Original: Clip

Original: [Figure] E v [Figure] Feedback of EgoClip
Revised: Q: #

Original: [Figure]
Revised: [Figure]

Original: Q: #C C holds cloth [Figure] datasets tend to be small-scale and domain-specific.
Revised: However, since the collection of egocentric videos is expensive, previous egocentricdatasets tend to be small-scale and domain-specific.

Original: These limitations hinder 1st-person view researchand fail to match the progress of 3rd-person counterparts, such as VLP [25, 26, 11].
Revised: These limitations hinder 1st-person view researchand fail to match the progress of 3rd-person counterparts, such as VLP driven by large-scale datasets.

Original: Recently, amassive egocentric video dataset Ego4D [16] has been released, which consists of 3 , 670 hours ofvideos collected by 931 people from 74 worldwide locations in 9 different countries, where mostvideos are accompanied by narrations, audio, 3D meshes, and more.
Revised: Recently, a massive egocentric video dataset Ego4D [16] has been released, which consists of 3 , 670hours of videos collected by 931 people from 74 worldwide locations in 9 different countries, wheremost videos are accompanied by narrations, audio, 3D meshes of the scene, and more.

Original: Furthermore, Ego4D introducesa suite of new challenging benchmarks (e.g., Natural language query and moment query that wetreated as downstream tasks) to fully explore the 1st-person visual experience.
Revised: Furthermore, Ego4D introduces a suite of new challenging benchmarks (e.g., Natural language query and momentquery that we treated as downstream tasks) to fully explore the 1st-person visual experience.

Original: With this step-changedataset and benchmarks, Ego4D would lead to a new research surge on egocentric visual perception.
Revised: Withthis step-change dataset and benchmarks, Ego4D would lead to a new research surge on egocentricvisual perception.

Original:  EgoClip: An Egocentric Video-Language Pretraining Dataset
Revised:  EgoClip: An Egocentric Video-Language Pretraining Dataset

Original: Data curation.
Revised: Data curation.

Original: For our EgoClip dataset, we source data from Ego4D [16], which contains 9 , 645untrimmed videos of varying lengths from 5 sec to 7 hrs.
Revised: For our EgoClip dataset, we source data from Ego4D [16], which contains 9 , 645untrimmed videos of varying lengths from 5 sec to 7 hrs.

Original: From these videos, most are associatedwith dense timestamp-level narrations assigned by two different annotators, describing the camerawearer’s activities and interactions with objects.
Revised: From these videos, most are associatedwith dense timestamp-level narrations assigned by two different annotators, describing the camerawearer’s activities and interactions with objects.

Original: For example, the narration “ #C C puts thescrapper down. ”
Revised: For example, the narration “ #C C puts thescrapper down. ”

Original: corresponds to video content that occurred at 3 .
Revised: corresponds to video content that occurred at 3 .

Original: 70 s , where “ #C ” refers to thecamera-wearer.
Revised: 70 s , where “ #C ” refers to thecamera-wearer.

Original: Notably, narrations in Ego4D are well-aligned with the videos, both temporallyand visually.
Revised: Notably, narrations in Ego4D are well-aligned with the videos, both temporallyand visually.

Original: Prior pretraining datasets are characterized by a much greater level of temporalmisalignment between the video and text (e.g., HowTo100M [10] narrations are scraped from ASR,yielding sentences misaligned or even unrelated to video content).
Revised: Prior pretraining datasets are characterized by a much greater level of temporalmisalignment between the video and text (e.g., HowTo100M [10] narrations are scraped from ASR,yielding sentences misaligned or even unrelated to video content).

Original: We first filter Ego4D videoswith missing narrations ( 7 . 4% of the total video duration) and exclude videos that belong to thevalidation and test sets of Ego4D benchmark challenge [16] (a further 23 .
Revised: We first filter Ego4D videos withmissing narrations ( 7 . 4% of the total video duration) and exclude videos that belong to the validationand test sets of Ego4D benchmark challenge [16] (a further 23 .

Original: 9% of the total videoduration).
Revised: 9% of the total video duration).

Original: Next, we retain textual annotation from both narrators in EgoClip, allowing us to considernarration diversity when pairing video and text for pretraining purposes.
Revised: Next,we retain textual annotation from both narrators in EgoClip, allowing us to consider narration diversitywhen pairing video and text for pretraining purposes.

Original: Finally, we adopt severalcriteria to filter the video and textual narrations, further reducing noise (detailed steps are provided in Supplementary).
Revised: Finally, we adopt several criteria to filter thevideo and textual narrations, further reducing noise (detailed steps are provided in Supplementarymaterial).

Original: Overall, this procedure yields 2 .
Revised: Overall, this procedure yields 2 .

Original: 9 K hours of videos with 3 . 85 million narrationswhich covers 2927 hours of video from 129 different scenarios.
Revised: 9 K hours of videos with 3 . 85 million narrations whichcovers 2927 hours of video from 129 different scenarios.

Original: EgoClip has 21 .
Revised: EgoClip has 21 .

Original: 9 clips per minutewith an average clip length of 1 . 0 seconds and a standard deviation of 0 .
Revised: 9 clips per minute with anaverage clip length of 1 . 0 seconds and a standard deviation of 0 .

Original: 9 seconds (the longest clip isup to 60 s).
Revised: 9 seconds (the longest clip is up to60 s).

Original: Additional analyses are included in the Supplementary.
Revised: Additional analyses are included in the Supplementary material.

Original: Creation of clip-text pairs.
Revised: Creation of clip-text pairs.

Original: Clip-text pairs are the common data format for VLP, but are usuallynot present in untrimmed video datasets with only a weak matching between narrations captionsand videos.
Revised: Clip-text pairs are the common data format for VLP, but are usually notpresent in untrimmed video datasets with only a weak matching between textual captions and videos.

Original: This was first discussed in HowTo100M [10], which pairs subtitles to video clips withcorresponding time intervals to result in noisy pairs.
Revised: This was first discussed in HowTo100M [10], which pairs subtitles to video clips with correspondingtime intervals to result in noisy pairs.

Original: This is not suitable for Ego4D since eachnarration is annotated with a single timestamp rather than an interval.
Revised: This is not suitable for Ego4D since each narration is annotatedwith a single timestamp rather than an interval.

Original: Thus, we design a contextualvariable-length clip pairing strategy .
Revised: Thus, we design a contextual variable-length clippairing strategy .

Original: Formally, narrations per video in Ego4D are organized asa sequence of sentences {T 0 , · · · , T n } with exact timestamps { t 0 , · · · , t n } , indicating an event idescribed by T i happened in the moment t i .
Revised: Formally, narrations per video in Ego4D are organized as a sequence of sentences{T 0 , · · · , T n } with exact timestamps { t 0 , · · · , t n } , indicating an event i described by T i happened in the moment t i .

Original: For a narration T i with timestamp t i , we pair a clip V i with following start and end timepoints:
Revised: For a narration T i with timestamp t i , we pair a clip V i with following start and endtimepoints:

Original: [Equation]
Revised: [Equation]

Original: which represents a window centered around the timestamp t i with temporal duration equal to β i
Revised: which represents a window centered around the timestamp t i with temporal duration equal to β i

Original: /α .β
Revised: /α .β

Original: i is an adjustable parameter equal to the average temporal distance between pairs of consecutivenarrations, i.e., (cid:80) n − 1 j =0 ( t j +1 − t j ) /n .
Revised: i is an adjustable parameter equal to the average temporal distance between pairs of consecutivenarrations, i.e., (cid:80) n − 1 j =0 ( t j +1 − t j ) /n .

Original: We compute β
Revised: We compute β

Original: i on a per video basis.
Revised: i on a per video basis.

Original: Conversely, α is ascale factor computed as the average of all β i across all videos in the EgoClip ( α = 4 . 9 seconds).
Revised: Conversely, α is ascale factor computed as the average of all β i across all videos in the EgoClip ( α = 4 . 9 seconds).

Original: Intuitively, Eq. 1 is derived from three observations: (i)
Revised: Intuitively, Eq. 1 is derived from three observations: (i)

Original: Centering t i helps to involve prior informationabout the event i ; (ii) β i measures the clip duration according to its scenario, such as longer clipswatching television ( 352 . 9 seconds) v.s. shorter clips harvesting crops ( 0 . 9 seconds); (iii) α controlsthe context granularity of clips (e.g., a large α pays more attention to rapid, atomic actions).
Revised: Centering t i helps to involve prior informationabout the event i ; (ii) β i measures the clip duration according to its scenario, such as longer clipswatching television ( 352 . 9 seconds) v.s. shorter clips harvesting crops ( 0 . 9 seconds); (iii) α controlsthe context granularity of clips (e.g., a large α pays more attention to rapid, atomic actions).

Original: Weablate these design choices in our experimental section.
Revised: Weablate these design choices in our experimental section.

Original:  Video-Language Pretraining Model
Revised:  Video-Language Pretraining Model

Original: To efficiently transfer video-language representation to egocentric downstream tasks (e.g., video-textretrieval on EPIC-KITCHENS-100 [22]), We prefer the dual-encoder (discussed in Sec. 2) as our VLPmodel architecture.
Revised: To efficiently transfer video-language representation to egocentric downstream tasks (e.g., video-textretrieval on EPIC-KITCHENS-100 [22]), We prefer the dual-encoder (discussed in Sec. 2) as our VLPmodel architecture.

Original: In particular, we emphasize devising a general pretraining objective EgoNCE toadapt existing VLP model to the egocentric environment (e.g., EgoClip).
Revised: In particular, we emphasize devising a general pretraining objective EgoNCE toadapt existing VLP model to the egocentric environment (e.g., EgoClip).

Original: 4.1 Architecture: Dual-encoder Pipeline
Revised: 4.1 Architecture: Dual-encoder Pipeline

Original: We choose Frozen [11] as our pretraining architecture.
Revised: We choose Frozen [11] as our pretraining architecture.

Original: Frozen [11] design encompasses an elegant andsimple dual encoder strategy (one per modality) which has favorable characteristics (e.g., indexabilityand efficiency [2, 28]).
Revised: Frozen [11] design encompasses an elegant andsimple dual encoder strategy (one per modality) which has favorable characteristics (e.g., indexabilityand efficiency [2, 28]).

Original: Note that this allows us to use our pretrained network in single-modality tasks(e.g.
Revised: Note that this allows us to use our pretrained network in single-modality tasks(e.g.

Original: , video-only tasks).
Revised: , video-only tasks).

Original: In practice, the video encoder is a TimeSformer [31] architecture while thetext encoder builds upon DistillBERT [32].
Revised: In practice, the video encoder is a TimeSformer [31] architecture while thetext encoder builds upon DistillBERT [32].

Original: However, our approach is not limited to the encoder’sdesign (e.g., the video backbone can be replaced by SlowFast [33]).
Revised: However, our approach is not limited to the encoder’sdesign (e.g., the video backbone can be replaced by SlowFast [33]).

Original: In the rest of the paper we adoptthis notation: ( V i , T i ) represent the video-test input to the model, while v i and t i are used to identifythe video and text embedding.
Revised: In the rest of the paper we adoptthis notation: ( V i , T i ) represent the video-test input to the model, while v i and t i are used to identifythe video and text embedding.

Original: 4.2 EgoNCE: An Egocentric-friendly Pretraining Objective
Revised: 4.2 EgoNCE: An Egocentric-friendly Pretraining Objective

Original: A common pretraining objective for dual-encoder VLP is InfoNCE [29], where the matching visualtext pairs in the batch are treated as positives while all other pairwise combinations in the batch areregarded as negatives.
Revised: A common pretraining objective for dual-encoder VLP is InfoNCE [29], where the matching visualtext pairs in the batch are treated as positives while all other pairwise combinations in the batch areregarded as negatives.

Original: Formally, within a batch B = { 1 , · · · , N } , InfoNCE is computed by the sumof video-to-text loss L v2t and text-to-video loss L t2v .
Revised: Formally, within a batch B = { 1 , · · · , N } , InfoNCE is computed by the sumof video-to-text loss L v2t and text-to-video loss L t2v .

Original: For simplicity, we only formulate L v2t whereas
Revised: For simplicity, we only formulate L v2t whereas

Original: L t2v is defined in a symmetry way:
Revised: L t2v is defined in a symmetry way.

Original: [Equation]
Revised: [Equation]

Original: where i -th video embedding v i and j -th text embedding t j are L2 normalized features, and τ is atemperature factor.
Revised: where i -th video embedding v i and j -th text embedding t j are L2 normalized features, and τ is atemperature factor.

Original: However, this simple objective performs not well on large-scale video-text datasets like
Revised: However, this simple objective performs not well on large-scale video-text datasets like

Original: HowTo100M [10] due to the serious misalignment between the two modalities.
Revised: HowTo100M [10] due to the serious misalignment between the two modalities.

Original: Therefore, [34]proposes MIL-NCE which treats temporal nearest captions as positive samples.
Revised: Therefore, [34]proposes MIL-NCE which treats temporal nearest captions as positive samples.

Original: In this work, our 1st-person human daily activity dataset, i.e. EgoClip, presents two unique challengescompared to the existing 3rd-person view video-text datasets: Challenge (i) :
Revised: In this work, our 1st-person human daily activity dataset, i.e. EgoClip, presents two unique challengescompared to the existing 3rd-person view video-text datasets: Challenge (i) :

Original: The same action oftenoccurs in different scenarios (e.g., “unlock the phone” could happen when “lying in bed” or “walkingoutdoors”).
Revised: The same action oftenoccurs in different scenarios (e.g., “unlock the phone” could happen when “lying in bed” or “walkingoutdoors”).

Original: Challenge (ii) :
Revised: Challenge (ii) :

Original: Often, different actions appearing in the same scenario tend to haveindistinguishable visual differences (e.g., when “working in front of the laptop”, “typing on thekeyboard” or “moving the mouse” could have similar feature representations).
Revised: Often, different actions appearing in the same scenario tend to haveindistinguishable visual differences (e.g., when “working in front of the laptop”, “typing on thekeyboard” or “moving the mouse” could have similar feature representations.)

Original: To overcome these two unique challenges, we propose a novel EgoNCE training objective whichtakes into account two simple yet efficient sampling strategies based on the vanilla InfoNCE.
Revised: To overcome these two unique challenges, we propose a novel EgoNCE training objective whichtakes into account two simple yet efficient sampling strategies based on the vanilla InfoNCE.

Original: Action-aware Positive Sampling.
Revised: Action-aware Positive Sampling.

Original: In this work, we make a reasonable assumption that the criticalelements in linking visual actions to textual narrations are verbs and objects mentioned in the
Revised: In this work, we make a reasonable assumption that the criticalelements in linking visual actions to textual narrations are verbs and objects mentioned in the

Original: Evaluation on the text-video retrieval task is unreliable due to duplications #C C closes the refrigerator. 
Revised: Evaluation on the text-video retrieval task is unreliable due to duplications #C C closes the refrigerator. #C C closes

Original: #C C closes the fridge #C C closes the lower part of the fridge #C C closes the refrigerator.
Revised: [Figure] fridge #C C closes [Figure] lower part of the fridge #C C closes the refrigerator.

Original: Retrieval result: Top clips are not GT but shall be considered as correct.
Revised: Retrieval result: Top clips are not GT but shall be considered as correct.

Original: Text query: #C C closes the refrigerator.
Revised: Text query: #C C closes the refrigerator.

Original: Inter-video
Revised: Inter-video

Original: Intra-video
Revised: Intra-video

Original: Text query
Revised: Text query

Original: Select the correct video clip fromcandidates
Revised: Select the correct video clip fromcandidates

Original: Answer with GT#
Revised: Answer with GT

Original: [Figure]
Revised: [Figure]

Original: EgoMCQ
Revised: EgoMCQ

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: (a) (

Original: [Figure]
Revised: [Figure]

Original: ) (c) (
Revised: ) (c) (

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: )(b) (c) (
Revised: ) (c) (

Original: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: camping

Original: [Figure][Figure]
Revised: [Figure]

Original: picks

Original: [Figure]
Revised: [Figure]

Original: siliconesealant
Revised: silicone

Original: [Figure]
Revised: [Figure]

Original: Top1:
Revised: Top1:

Original: Top2:
Revised: Top2:

Original: Top3:
Revised: Top3:

Original: TopN (GT): narrations (e.g. “drinking coffee”, “openingfridge”).
Revised: TopN (GT): narrations (e.g. “drinking coffee”, “opening

Original: Following this assumption, we can devise aclever method to address challenge (i).
Revised: Following this assumption, we can devise aclever method to address challenge (i).

Original: Specifically, for each narration, we identify its nouns andverbs and merge synonym words based on the Ego4D taxonomy dictionary [16], a thesaurus recordsmeaningful nouns/verbs in Ego4D narrations.
Revised: Specifically, for each narration, we identify its nouns andverbs and merge synonym words based on the Ego4D taxonomy dictionary [16], a thesaurus recordsmeaningful nouns/verbs in Ego4D narrations.

Original: Then, batch samples that shared at least one nounand at least one verb are treated as positive samples.
Revised: Then, batch samples that shared at least one nounand at least one verb are treated as positive samples.

Original: At last, for the sample i , we select its positivesamples set within batch B as P i = { j ∈ B
Revised: At last, for the sample i , we select its positivesamples set within batch B as P i = { j ∈ B

Original: | noun ( j ) ∩ noun ( i ) ̸
Revised: | noun ( j ) ∩ noun ( i ) ̸

Original: = ∅ , verb ( j ) ∩ verb ( i ) ̸ = ∅ } .
Revised: = ∅ , verb ( j ) ∩ verb ( i ) ̸ = ∅ } .

Original: Scene-aware Negative Sampling.
Revised: Scene-aware Negative Sampling.

Original: To address challenge (ii), we consider different actions in thesame scenario as hard negative samples.
Revised: To address challenge (ii), we consider different actions in thesame scenario as hard negative samples.

Original: Specifically, for each video clip i , we sample an adjacentclip i ′
Revised: Specifically, for each video clip i , we sample an adjacentclip i ′

Original: ∈ N ( i ) , which is close to i in time within the same video.
Revised: ∈ N ( i ) , which is close to i in time within the same video.

Original: We augment the original batch Bwith such hard negative samples and each sample i in B has its negative counterparts i ′ .
Revised: We augment the original batch Bwith such hard negative samples and each sample i in B has its negative counterparts i ′ .

Original: Hence thebatch is updated as (cid:101) B = { 1 , 2 , · · · N (cid:124) (cid:123)(cid:122) (cid:125) B , 1 ′ , 2 ′ , · · · , N ′ (cid:124) (cid:123)(cid:122) (cid:125) N ( B ) } .
Revised: Hence thebatch is updated as (cid:101) B = { 1 , 2 , · · · N (cid:124) (cid:123)(cid:122) (cid:125) B , 1 ′ , 2 ′ , · · · , N ′ (cid:124) (cid:123)(cid:122) (cid:125) N ( B ) } .

Original: With these two sampling strategies, our new pretraining objective EgoNCE can be formulated as:
Revised: With these two sampling strategies, our new pretraining objective EgoNCE can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: Here the item in purple corresponds to our proposed action-aware positive samples and blue corresponds to our proposed scene-aware negative samples.
Revised: Here the item in purple corresponds to our proposed action-aware positive samples and blue corresponds to our proposed scene-aware negative samples.

Original: EgoNCE provides a general extension toadapt the existing VLP models for video-text pretraining datasets in the egocentric domain.
Revised: EgoNCE provides a general extension toadapt the existing VLP models for video-text pretraining datasets in the egocentric domain.

Original:  EgoMCQ: A Benchmark for Egocentric VLP Development
Revised:  EgoMCQ: A Benchmark for Egocentric VLP Development

Original: The need for a development benchmark.
Revised: The need for a development benchmark.

Original: We find that most egocentric benchmarks are domainspecific and focus on single-modality tasks (see Tab. 1).
Revised: We find that most egocentric benchmarks are domainspecific and focus on single-modality tasks (see Tab. 1).

Original: However, our purpose is to exploit Ego4D’sdiversity to learn rich video-text representations.
Revised: However, our purpose is to exploit Ego4D’sdiversity to learn rich video-text representations.

Original: Hence, to validate our design choices of thepretraining dataset (e.g., EgoClip), and model (e.g., EgoNCE), it is essential to measure performanceon a benchmark highly aligned with the pretraining task.
Revised: Hence, to validate our design choices of thepretraining dataset (e.g.,EgoClip), and model (e.g.,EgoNCE), it is essential to measure performanceon a benchmark highly aligned with the pretraining task.

Original: Therefore, we propose EgoMCQ, a newegocentric benchmark for reliable and fast developments of Egocentric VLP.
Revised: Therefore, we propose EgoMCQ, a newegocentric benchmark for reliable and fast developments of Egocentric VLP.

Original: Data source.
Revised: Data source.

Original: We start from the Ego4D data excluded from constructing the EgoClip, which mainlycovers the validation set of the Ego4D challenge benchmarks.
Revised: We start from the Ego4D data excluded from constructing the EgoClip, which mainlycovers the validation set of the Ego4D challenge benchmarks.

Original: Additionally, to assure that the sceneis not visible during pretraining, we manually remove videos that share multiple views with the videos in EgoClip.
Revised: Additionally, to assure that the sceneis not visible during pretraining, we manually remove videos that share multiple views with thevideos in EgoClip.

Original: To ensure diversity, we randomly select one annotator’s narration for each video.
Revised: To ensure diversity, we randomly select one annotator’s narration for each video.

Original: We follow the same clip pairing strategy as Eq.
Revised: We follow the same clip pairing strategy as Eq.

Original: 1 to be consistent with the data format of EgoClippreventing a domain gap.
Revised: 1 to be consistent with the data format of EgoClippreventing a domain gap.

Original: Benchmarking task design.
Revised: Benchmarking task design.

Original: To determine the task for development, we first consider video-textretrieval since it highly aligns with the VLP pretraining objective.
Revised: To determine the task for development, we first consider video-textretrieval since it highly aligns with the VLP pretraining objective.

Original: However, as depicted in the tophalf of Fig. 2, for an action (e.g., close the refrigerator), there are substantial duplicate/semanticallysimilar captions in Ego4D. This can cause issues in retrieval evaluation [35] causing model trainingto become unreliable.
Revised: However, as depicted in the tophalf of Fig. 2, for an action (e.g., close the refrigerator), there are substantial duplicate/semanticallysimilar captions in Ego4D. This can cause issues in retrieval evaluation [35] causing model trainingto become unreliable.

Original: A straightforward approach to prevent this is deduplication (dedup), but it ischallenging to devise a dedup criterion and perform well in the retrieval settings of a “one-to-wholevalidation set”.
Revised: A straightforward approach to prevent this is deduplication (dedup), but it ischallenging to devise a dedup criterion and perform well in the retrieval settings of a “one-to-onevalidation set”.

Original: Therefore, we select the Multiple-Choice Questions (MCQ) task for developmentsince repetitions are highly unlikely given a small number of answers.
Revised: Therefore, we select the Multiple-Choice Questions (MCQ) task for developmentsince repetitions are highly unlikely given a small number of answers.

Original: Grouping strategies.
Revised: Grouping strategies.

Original: To set up the MCQ task, a naive construction randomly groups five videoclips to form options for a question.
Revised: To set up the MCQ task, a naive construction randomly groups five videoclips to form options for a question.

Original: But we find randomly grouping is not challenging since optionsare highly likely to come from different videos and vary widely in content.
Revised: But we find randomly grouping is not challenging since optionsare highly likely to come from different videos and vary widely in content.

Original: We redefine this basicsetting as “ inter-video ” and ensure that the five clips originate from different videos, aiming todistinguish instances from different scenarios (the left side of Fig. 2).
Revised: We redefine this basicsetting as “ inter-video ” and ensure that the five clips originate from different videos, aiming todistinguish instances from different scenarios (the left side of Fig. 2).

Original: Furthermore, we propose amore challenging setting “ intra-video ” by grouping five continuous clips together (with dedup).
Revised: Furthermore, we proposea more challenging setting “ intra-video ” by grouping five continuous clips together (removingduplicates).

Original: Thissetting is regarded as a specific form of video-text localization focused on fine-grained context clues,such as hand interaction (the right side of Fig. 2).
Revised: This setting is regarded as a specific form of video-text localization focused on finegrained context clues, such as hand interaction (the right side of Fig. 2).

Original: Dedup is performed within five options for eachquestion for reliable assessment(see Supp) and we adopt accuracy as the EgoMCQ metric.
Revised: De-dup is performed withinfive options for each question for reliable assessment and we adopt accuracy as the EgoMCQ metric.

Original: Statistics.
Revised: Statistics.

Original: We finalize 39 K questions covering 198 K narrations with 468 hours of video, wherethe “inter-video” has 24 K questions covering 290 .
Revised: We finalize 39 K questions covering 198 K narrations with 468 hours of video, wherethe “inter-video” has 24 K questions covering 290 .

Original: 3 hours of videos.
Revised: 3 hours of videos.

Original: And the “intra-video” has 15 Kquestions and covers 178 .
Revised: And the “intra-video” has 15 Kquestions and covers 178 .

Original: 3 hours of videos.
Revised: 3 hours of videos.

Original: The average duration among the five options is 34 .
Revised: The average duration among the five options is 34 .

Original: 2seconds (More statistics of  EgoMCQ are shown in Supplementary.).
Revised: 2seconds (More statistics of EgoClip and EgoMCQ are shown in Supplementary.).

Original:  Experiments
Revised:  Experiments

Original: We assess our Egocentric VLP along two directions: (i) We conduct extensive analysis to explorekey components of Egocentric VLP (e.g., EgoClip, EgoNCE, and EgoMCQ); (ii) We transfer ourpretrained model to various downstream tasks to validate the quality of our video-text representation.
Revised: We assess our Egocentric VLP along two directions: (i) We conduct extensive analysis to explorekey components of Egocentric VLP (e.g., EgoClip, EgoMCQ, and EgoNCE); (ii) We transfer ourpretrained model to various downstream tasks to validate the quality of our video-text representation.

Original: 6.1 Benchmarks and Settings
Revised: 6.1 Benchmarks and Settings

Original: We evaluate our VLP model on five egocentric benchmarks, spanning video-text tasks and pure videotasks, across three different datasets.
Revised: We evaluate our VLP model on five egocentric benchmarks, spanning video-text tasks and pure videotasks, across three different datasets.

Original: We briefly describe each task below.
Revised: We briefly describe each task below.

Original: Multi-Instance Retrieval of EPIC-KITCHENS-100.
Revised: Multi-Instance Retrieval of EPIC-KITCHENS-100.

Original: This task is modelled as a text-to-videoretrieval which considers the semantic overlap between different videos narrations, where multiplevideos may correspond to the same narration.
Revised: This task is modelled as a text-to-videoretrieval which considers the semantic overlap between different videos narrations, where multiplevideos may correspond to the same narration.

Original: The training set contains 67 .
Revised: The training set contains 67 .

Original: 2 K clips and validationset contains 9 .
Revised: 2 K clips and validationset contains 9 .

Original: 7 K clips.
Revised: 7 K clips.

Original: The evaluation metrics are mean Average Precision (mAP) and normalised
Revised: The evaluation metrics are mean Average Precision (mAP) and normalised

Original: Discounted Cumulative Gain (nDCG).
Revised: Discounted Cumulative Gain (nDCG).

Original: Natural Language Query of Ego4D Challenges.
Revised: Natural Language Query of Ego4D Challenges.

Original: The Natural Language Query task is modelled asa natural language grounding problem [1, 36, 37].
Revised: The Natural Language Query task is modelled asa natural language grounding problem [1, 36, 37].

Original: Given a language query and a video, the task aimsat localizing the temporal interval within the video, in which the answer is deducible.
Revised: Given a language query and a video, the task aimsat localizing the temporal interval within the video, in which the answer is deducible.

Original: The trainingset contains 11 .
Revised: The trainingset contains 11 .

Original: 3 K queries annotated from 1 K clips for this task, while the validation contains 3 .
Revised: 3 K queries annotated from 1 K clips for this task, while the validation contains 3 .

Original: 9 Kqueries collected from 0 .
Revised: 9 Kqueries collected from 0 .

Original: 3 K clips.
Revised: 3 K clips.

Original: The evaluation metric is Recall@ K for IoU =
Revised: The evaluation metric is Recall@ K for IoU =

Original: θ (R@ K -IoU =
Revised: θ (R@ K -IoU =

Original: θ )
Revised: θ )

Original: [1]where θ is a threshold.
Revised: [1]where θ is a threshold.

Original: We evaluate for K ∈{ 1 , 5 } and θ ∈{ 0 . 3 , 0 . 5 } .
Revised: We evaluate for K ∈{ 1 , 5 } and θ ∈{ 0 . 3 , 0 . 5 } .

Original: Action Recognition of Charades-Ego.
Revised: Action Recognition of Charades-Ego.

Original: This dataset has 64 K instances spanning 1st-person andthird-person views, and covering 157 activity categories for training.
Revised: This dataset has 64 K instances spanning 1st-person andthird-person views, and covering 157 activity categories for training.

Original: We train and evaluate onlyon the 1st-person videos.
Revised: We train and evaluate onlyon the 1st-person videos.

Original: The validation set contains 847 videos for classification and each videobelongs to multiple classes.
Revised: The validation set contains 847 videos for classification and each videobelongs to multiple classes.

Original: The evaluation metric is mAP.
Revised: The evaluation metric is mAP.

Original: Moment Query of Ego4D Challenges.
Revised: Moment Query of Ego4D Challenges.

Original: The Moment Query task is a video-only task modelled as
Revised: The Moment Query task is a video-only task modelled as

Original: Temporal Action Localization [12].
Revised: Temporal Action Localization [12].

Original: Given a particular high-level activity category, the task solutionconsist of retrieving all the possible temporal windows where the activity occurs.
Revised: Given a particular high-level activity category, the task solutionconsist of retrieving all the possible temporal windows where the activity occurs.

Original: The training setcontains 13 .
Revised: The training setcontains 13 .

Original: 6 K instance from 1 .
Revised: 6 K instance from 1 .

Original: 5 K clips, while the validation set contains 4 .
Revised: 5 K clips, while the validation set contains 4 .

Original: 3 K instances from 0 .
Revised: 3 K instances from 0 .

Original: 5 Kclips.
Revised: 5 Kclips.

Original: The evaluation metrics are mAP and R@ K -IoU
Revised: The evaluation metrics are mAP and R@ K -IoU

Original: = θ for K ∈{ 1 , 5 } and θ ∈{ 0 . 3 , 0 . 5 , 0 .
Revised: = θ for K ∈{ 1 , 5 } and θ ∈{ 0 . 3 , 0 . 5 , 0 .

Original: [Table]
Revised: [Table]

Original: Object State Change Classification (OSCC) of Ego4D Challenges.
Revised: Object State Change Classification (OSCC) of Ego4D Challenges.

Original: This OSCC task is modelled asan (N+1)-way classification aiming to identify an object’s state change in a given video.
Revised: This OSCC task is modelled asan (N+1)-way classification aiming to identify an object’s state change in a given video.

Original: The trainingand val.
Revised: The trainingand val.

Original: sets contain 41 K and 28 K clips, respectively.
Revised: sets contain 41 K and 28 K clips, respectively.

Original: The evaluation metric is accuracy.
Revised: The evaluation metric is accuracy.

Original: Implementation details.
Revised: Implementation details.

Original: Our codebase is based on the official Frozen 2 one and retains the samesettings unless specified.
Revised: Our codebase is based on the official Frozen 2 one and retains the samesettings unless specified.

Original: During pretraining, we sample 4 frames for each clip, and use the Adamoptimizer [38] with a learning rate of 3 × 10 − 5 .
Revised: During pretraining, we sample 4 frames for each clip, and use the Adamoptimizer [38] with a learning rate of 3 × 10 − 5 .

Original: To select the best method we pretrain our architecturefor 10 epochs and use the best performing model on the EgoMCQ benchmark.
Revised: To select the best method we pretrain our architecturefor 10 epochs and use the best performing model on the EgoMCQ benchmark.

Original: Pretraining takes twodays on 32 A100 GPUs ( 1 , 536 GPU hrs).
Revised: Pretraining takes twodays on 32 A100 GPUs ( 1 , 536 GPU hrs).

Original: 6.2 Ablation Studies
Revised: 6.2 Ablation Studies

Original: Ablation of the strategy used when creating  EgoClip.
Revised: Ablation of the strategy used when creating the EgoClip.

Original: We validate our proposed strategiesi.e.
Revised: We validate our proposed strategiesi.e.

Original: Eq. 1 here by comparing the following variants: (a) fixed length α , start at timestamp; (b) fixedlength α , center at timestamp; (c) variable clip, start and end by adjacent timestamps; (d) our proposedstrategy, scaled by 2 ; (e) our proposed strategy, scaled by 4 ; (f) our proposed strategy.
Revised: Eq.1 here by comparing the following variants: (a) fixed length α , start at timestamp; (b) fixedlength α , center at timestamp; (c) variable clip, start and end by adjacent timestamps; (d) our proposedstrategy, scaled by 2 ; (e) our proposed strategy, scaled by 4 ; (f) our proposed strategy.

Original: We consider a good pretraining dataset creation strategy should satisfy the following: (1) the VLPmodel trained on EgoClip should be able to well distinguish instances in EgoMCQ with the samedata format; (2) the VLP model pretrained on EgoClip with the specificclip creation strategy shouldperform well in public downstream tasks (e.g., video-text retrieval on [22], zero-shot for efficiency).
Revised: We consider a good pretraining dataset creation strategy should satisfy the following: (1) the VLPmodel trained on EgoClip should be able to well distinguish instances in EgoMCQ with the samedata format; (2) the VLP model pretrained on EgoClip with the specific strategy should perform wellin public downstream tasks (e.g., video-text retrieval on [22], zero-shot for efficiency).

Original: We draw several conclusions from Tab. 2: (i) The performance of EgoMCQ is well aligned withthe zero-shot result on EPIC-KITCHENS-100, especially minor gain on downstream but noticeableon EgoMCQ, which means EgoMCQ provides valid feedback and is suitable as a development set.(ii)
Revised: We draw several conclusions from Tab. 2: (i) The performance of EgoMCQ is well aligned withthe zero-shot result on EPIC-KITCHENS-100, especially minor gain on downstream but noticeableon EgoMCQ, which means EgoMCQ provides valid feedback and is suitable as a development set.(ii)

Original: Under the same clip length α , (b) surpassing (a) proves that centering at timestamp includes priorinformation is helpful.
Revised: Under the same clip length α , (b) surpassing (a) proves that centering at timestamp includes priorinformation is helpful.

Original: (iii) Variable-length clips make a big difference, as shown in (c) and (d).
Revised: (iii) Variable-length clips make a big difference, as shown by comparing(c) and (d).

Original: [Table]
Revised: Notably, with our designed β i , (d) outperforms (b) with a similar average clip length,which validates our idea of “contextual varied clip length”.

Original: varied clip length” on  factor great than 1 is preferred, which helps to focus ona large of instantaneous actions densely labeled by Ego4D [16].
Revised: (iv) Based on (d), (e), and (f), we found aproper scale factor great than 1 is preferred, which helps to focus on a large of instantaneous actionsdensely labeled by ego4d.

Original: These ablation studies demonstrate theeffectiveness of our proposed EgoClip creation strategy and EgoMCQ for development.
Revised: These ablation studies demonstrate the effectiveness of our proposed EgoClip creation strategy and EgoMCQ for development.

Revised: [Table]

Original: Effect of EgoNCE.
Revised: Effect of EgoNCE.

Original: In this section, we evaluate theeffect of the proposed sampling strategies for the
Revised: In this section, we evaluate theeffect of the proposed sampling strategies for the

Original: EgoNCE objective (Eq. 3) on EgoMCQ and compareagainst a vanilla InfoNCE loss (Eq. 2).
Revised: EgoNCE objective (Eq. 3) on EgoMCQ and compareagainst a vanilla InfoNCE loss (Eq. 2).

Original: We ablate several configurations for positive and negative samplingstrategies.
Revised: We ablate several configurations for positive and negative samplingstrategies.

Original: The sampling strategy for positive pairsexploits language cues, while negative pairs rely on temporal, visual cues.
Revised: The sampling strategy for positive pairsexploits language cues, while negative pairs rely ontemporal, visual cues.

Original: Given a text-video pair, weregard other text-video pairs as positive if the textual narrations: (a) share at least one noun, (b) shareat least one verb, (c) share at least a verb-noun pair.
Revised: Given a text-video pair, weregard other text-video pairs as positive if the textualnarrations: (a) share at least one noun, (b) share atleast one verb, (c) share at least a verb-noun pair.

Original: Conversely, we define the following heuristicsfor negative sampling: (d) a random text-video pair from EgoClip, (e) a text-video pair from the samevideo, (f) a text video pair within 1 minute from the given video [Table]
Revised: Conversely, we define the following heuristics fornegative sampling: (d) a random text-video pair from EgoClip, (e) a text-video pair from the same video,(f) a text-video pair within 1 minute from the given video-text pair annotation timestamps.

Original: Tab. 3 shows that only using verbs (a) or nouns (b) for positive selection degrades the accuracyperformance with respect to naive InfoNCE.
Revised: Tab. 3shows that only using verbs (a) or nouns (b) for positive selection degrades the accuracy performancewith respect to naive InfoNCE.

Original: However, we successfully push the performance beyondthe baseline  results when considering both verbs and nouns jointly (c).
Revised: However, we successfully push the performance beyond the baseline [Table] results when considering both verbs and nouns jointly (c).

Original: Moreover, we notice that onlyselecting negatives within the same video leads to better performance.
Revised: Moreover, we notice that only selectingnegatives within the same video leads to better performance.

Original: In particular, we obtain the bestperformance for temporally “hard negatives” (f).
Revised: In particular, we obtain the best performance for temporally “hard negatives” (f).

Original: Finally, we pick the optimal settings from positiveand negative samplings and combine them together for (g) EgoNCE and reach the best results.6.3 Comparisons with State-of-the-arts
Revised: Finally, we pick the optimal settings from positive andnegative samplings and combine them together for (g) EgoNCE and reach the best results. 6.3 Comparisons with State-of-the-arts

Revised: We first evaluate Egocentric VLP on joint text-video tasks including Multi-Instance Retrieval and

Revised: Natural Language Query.

Original: Multi-Instance Retrieval.
Revised: Multi-Instance Retrieval.

Original: In Tab. 4, we report both zero-shot and fine-tuning evaluation results.
Revised: We report the results in Tab. 4, for both zero-shot and fine-tuningevaluations.

Original: Inthe zero-shot setting, pretraining with EgoClip ( 3 . 8 M), despite being smaller in scale, still outperforms CC3M+WebVid-2M ( 5 . 5 M) and HowTo100M ( 136 M), validating the unique benefit of pretrainingon egocentric data.
Revised: In the zero-shot setting, pretraining with EgoClip ( 3 . 8 M), despite being smaller in scale,still outperforms CC3M+WebVid-2M ( 5 . 5 M) and HowTo100M ( 136 M), validating the unique benefitof pretraining on egocentric data.

Original: When fine-tuned with 4 frames (rows 5-9), EgoClip pretraining maintains amargin over the best baseline CC3M+WebVid-2M, further verifying the viewpoint domain gap withinfine-tuning.
Revised: When fine-tuned with 4 frames (rows 5-9), EgoClip pretrainingmaintains a margin over the best baseline CC3M+WebVid-2M, further verifying the viewpointdomain gap within fine-tuning.

Original: Lastly, we increase the sample frames of our finalized model as well as best competitor CC3M+WebVid-2M pretraining to 16 (rows 10-11).
Revised: Lastly, we increase the sample frames of our finalized model as wellas best competitor CC3M+WebVid-2M pretraining to 16 (rows 10-11).

Original: As expected, performance gains accompanythe frame increase.
Revised: As expected, performancegains accompany the frame increase.

Original: We deem that notable benefits come from better temporal modeling for frequentaction interactions in the 1st-person view.
Revised: We deem that notable benefits come from better temporalmodeling for frequent action interactions in the 1st-person view.

Original: Overall, our pretraining model  outperforms the bestbaseline (JPoSE) by 1 . 0 mAP and 5 .
Revised: Overall, our pretraining model [Table] outperforms the best baseline (JPoSE) by 1 . 0 mAP and 5 .

Original: 9% nDCG while requiring fewer frames  and input modalities.
Revised: 9% nDCG while requiring fewer frames [Table] and input modalities.

Original: Action Recognition.
Revised: Action Recognition.

Original: We conduct action recognition on Charades-Ego, where categories are shortphrases like “Holding some clothes”.
Revised: We conduct action recognition on Charades-Ego, where categories are shortphrases like “Holding some clothes”.

Original: Thus this task can be solved as a text-to-video retrieval byleveraging the text representation.
Revised: Thus this task can be solved as a text-to-video retrieval byleveraging the text representation.

Original: We present the result in Tab. 5 under zero-shot and fine-tuningsettings.
Revised: We present the result in Tab. 5 under zero-shot and fine-tuningsettings.

Original: In  zero-shot settings,our model outperforms two supervised baselines [Table] which validate thestronger generalization of jointly  video-text feature.
Revised: In the zero-shot configuration, our model outperforms two supervised baselines, whichvalidate the stronger generalization of jointly trained video-text features.

Original: After fine-tuning (rows 5-9), our model surpassesall VLP counterparts and improves over the state-of-the-art video classifier Ego-Exo by 2 . 0 % withfewer sampled video frames, which shows the superior advantage of joint video-text representations
Revised: After fine-tuning (rows 5-9),our model surpasses all VLP counterparts and improves over the state-of-the-art video classifier Ego-Exo by 2 . 0 % with fewer sampled video frames ( 0 . 5 × ), which shows the superior advantage ofjoint video-text representations

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Natural Language Query.
Revised: Natural Language Query.

Original: We re-
Revised: We re-

Original: [Table]
Revised: [Table]

Original: port validation results on Tab. 6.
Revised: port validation results on Tab. 6.

Original: Weadopt the same baselines as introduced in [16], namely: 2DTAN [24]and VSLNet [44], and substitute the
Revised: Weadopt the same baselines as introduced in [16], namely: 2DTAN [24]and VSLNet [44], and substitute the

Original: SlowFast-BERT features with ourvideo and language representations.
Revised: SlowFast-BERT features with ourvideo and language representations.

Original: [Table]

Original: We observe a large boost in per- [Table] formance offered by our pretrained
Revised: We observe a large boost in performance offered by our pretrained

Original: [Table]
Revised: [Table]

Original: model for all metrics.
Revised: model for all metrics.

Original: Notably, we improve R@1 for IoU=0.3 from 5 .
Revised: Notably, we improve R@1 for IoU=0.3 from 5 .

Original: 45 to 10 .
Revised: 45 to 10 .

Original: 84 , despite ourvideo branch not being pre-trained on Kinetics400.
Revised: 84 , despite ourvideo branch not being pre-trained on Kinetics400.

Original: Besides, we significantly surpass VLP pretrainedon CC3M+WebVid-2M and HowTo100M. We believe this increase is due to the egocentric dataavailability and the video-text interaction learned from large-scale pretraining.
Revised: Besides, we significantly surpass VLP pretrainedon CC3M+WebVid-2M and HowTo100M. We believe this increase is due to the egocentric dataavailability and the video-text interaction learned from large-scale pretraining.

Original: See Supplementaryfor test set results. 
Revised: See Supplementary [Table] material for test set results. [Table]

Original: Moment Query.
Revised: Moment Query.

Original: This task investigates the quality of video-only features.
Revised: This task investigates the qualityof video-only features.

Original: We extract video featuresand provide them as input to the VSGN model [45].
Revised: We extract video featuresand provide them as input to the VSGN model [45].

Original: We report the validation results in Tab.7,
Revised: We report the validation results in Tab.

Original:  We find that our features achieves the best performance over SlowFast features with an increase of4 . 66% in Avg mAP.
Revised: 7, We findthat our features achieves the best performance over SlowFast features with an increase of 4 . 66% in AvgmAP.

Original: Moreover, we maintain better performance with respect to 3rd-person large-scalepretraining datasets.
Revised: Moreover, we maintain better performance withrespect to 3rd-person large-scale pretraining datasets.

Original: This demonstrates that the 1st-person VLP model also learns competitive videorepresentations.
Revised: This demonstrates that the 1st-person VLP modelalso learns competitive video representations.

Original: See the Supplementary  for test set results.
Revised: Seethe Supplementary material for test set results.

Original: [Table]
Revised: [Table]

Original: Object State Change Classification.
Revised: Change Classification.

Original: We report thevalidation results on Tab. 8.
Revised: We report thevalidation results on Tab. 8.

Original: Once again, our modelachieves the best performance of all baselines, 2 .
Revised: Once again, our model achieves the best performance of all baselines,2 .

Original: 4%than CC3M+WebVid-2M counterparts, which indicates our visual representations are able to focus onthe fine-grained clues related to changes.
Revised: 4% than CC3M+WebVid-2M counterparts, which indicates our visual representations are able tofocus on the fine-grained clues related to changes.

Original: Summary of EgoNCE.
Revised: Summary of EgoNCE.

Original: From the above experimentalresults, Frozen pretrained on EgoClip withEgoNCEobjective brings a consistent improvements over the
Revised: From the above experimental results, Frozen pretrained on EgoClip with

Original:  InfoNCE in all downstream tasks, which comprehensively demonstrates the effect of EgoNCE, and thedecision from EgoMCQ.
Revised: EgoNCE objective brings a consistent improvements over the InfoNCE in all downstream tasks,which comprehensively demonstrates the effect of EgoNCE, and the decision from EgoMCQ.

Original:  Conclusion, Limitations, and Societal Impacts.
Revised:  Conclusion, Limitations, and Societal Impacts.

Original: To the best of our knowledge, this work is the pioneering work to unlock Egocentric VLP. (i)
Revised: To the best of our knowledge, we pioneer Egocentric VLP from three aspects: (i)

Original: Wedevise a principled data curation and create EgoClip, an egocentric large-scale text-video pretrainingdataset with 3 .
Revised: We devise aprincipled data curation and create EgoClip, an egocentric large-scale text-video pretraining datasetwith 3 .

Original: 8 M clip-text pairs well-chosen from Ego4D. (ii)
Revised: 8 M clip-text pairs well-chosen from Ego4D. (ii)

Original: We exploit the particular characteristics of egocentric videos and devise EgoNCE with meaningful sampling strategies for effectiveegocentric pretraining.
Revised: We exploit the particular characteristics ofegocentric videos and devise EgoNCE with meaningful sampling strategies for effective egocentricpretraining.

Original: (iii)
Revised: (iii)

Original: We create EgoMCQ, an egocentric video-language benchmark close tothe pretraining set to support efficient exploration and development of EgoClip and EgoNCE.
Revised: We create EgoMCQ, an egocentric video-language benchmark close to the pretraining set to support efficient exploration and development of EgoClip and EgoNCE.

Original: Finally,we further demonstrate the strong representation of our egocentric pretraining on five tasks acrossthree datasets.
Revised: Finally, wefurther demonstrate the strong representation of our egocentric pretraining on five tasks across threedatasets.

Original: We believe our EgoClip, EgoMCQ and EgoNCE would greatly benefit the egocentricvideo community, laying down strong foundation for this new scientific topic of egocentric VLP.
Revised: We believe our EgoClip, EgoMCQ and EgoNCE would greatly benefit the egocentric videocommunity, laying down strong foundation for this new scientific topic of egocentric VLP.

Original: Limitations.
Revised: Limitations.

Original: Our pretraining approach does not take into account the long-term temporal dependencies in long Ego4D videos.
Revised: Our pretraining approach does not take into account the long-term temporal dependencies in long Ego4D videos.

Original: We leave it for future work.
Revised: We leave it for future work.

Original: Societal impact.
Revised: Societal impact.

Original: Egocentric VLP learns real-world perception knowledge that may contribute topractical applications such as AI-assisted technologies, robotics, and augmented reality. However, Ego4D videos collected by participants may contain users privacy and unintended biases, thus shouldbe used cautiously.
Revised: We acknowledge that the intensive electricity consumption related to our researchhas a significant carbon footprint that is negative for the environment. The research communityshould favor more efficient strategies for computation-intensive applications to limit this issue.

Original: We refer the reader to the Ego4D paper about further privacy and societal impacts.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?
Revised: imental results (either in the supplemental material or as a URL)?

Original: [No] We plan to
Revised: [No] We plan to

Original: release the dataset, code, pretrained models upon acceptance.
Revised: release the dataset, code, pretrained models upon acceptance.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] We included the most relevant ones.
Revised: [Yes] We included the most relevant ones.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running ex-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running ex-

Original: periments multiple times)?
Revised: periments multiple times)?

Original: [No] Since the pretraining procedure consumes huge
Revised: [No] Since the pretraining procedure consumes huge

Original: computational resources (1,536 GPU hrs), we follow previous literature on this aspect.(d)
Revised: computational resources (1,536 GPU hrs), we follow previous literature on this aspect.

Original:  Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: of GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] We included the total amount of
Revised: [Yes] We included the total amount of

Original: compute in GPU hours.
Revised: compute in GPU hours.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Ego4D have its own made license in
Revised: [Yes] The MIT License (MIT) for all

Original: this link and we have obtained the official approval agreement.(c)
Revised: datasets and baselines code.

Original:  Did you include any new assets either in the supplemental material or as a URL?
Revised: c) Did you include any new assets either in the supplemental material or as a URL?

Original: No
Revised: [No]

Original: ](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: N /A](e)
Revised: [N/A]

Original:  Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: Such discussion is included in the paperspresenting the datasetes we adopt in this work.
Revised: Such discussion is included in the papers presenting the datasetes we adopt in this work.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A
Revised: [N/A]

Original: b) Did you describe any potential participant risks , with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A
Revised: [N/A]

Original: c)

Original:  Did you include the estimated hourly wage paid to participants and the total 
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: amountspent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A
Revised: [N/A]