Original: Egocentric Video-Language Pretraining
Revised: Egocentric Video-Language Pretraining

Original: Abstract
Revised: Abstract

Original: Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention.
Revised: Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention.

Original: Best performing works rely on large-scale, 3rd-person videotext datasets, such as HowTo100M.
Revised: Best performing works rely on large-scale, 3rd-person videotext datasets, such as HowTo100M.

Original: In this work, we exploit the recently released
Revised: In this work, we exploit the recently released

Original: Ego4D dataset to pioneer Egocentric VLP along three directions.
Revised: Ego4D dataset to pioneer Egocentric VLP along three directions.

Original: (i)
Revised: (i)

Original: We create
Revised: We create

Original: EgoClip, a 1st-person video-text pretraining dataset comprising 3.8M clip-text pairs well-chosen from Ego4D, covering a large variety of human daily activities.
Revised: EgoClip, a 1st-person video-text pretraining dataset comprising 3.8M clip-text pairs well-chosen from Ego4D, covering a large variety of human daily activities.

Original: (ii) We propose a novel pretraining objective, dubbed EgoNCE, which adapts video-text contrastive learning to the egocentric domain by mining egocentricaware positive and negative samples.
Revised: (ii) We propose a novel pretraining objective, dubbed EgoNCE, which adapts video-text contrastive learning to the egocentric domain by mining egocentricaware positive and negative samples.

Original: (iii)
Revised: (iii)

Original: We introduce EgoMCQ, a development benchmark that is close to EgoClip and hence can support effective validation and fast exploration of our design decisions in EgoClip and EgoNCE.
Revised: We introduce EgoMCQ, a development benchmark that is close to EgoClip and hence can support effective validation and fast exploration of our design decisions in EgoClip and EgoNCE.

Original: Furthermore, we demonstrate strong performance on five egocentric downstream tasks across three datasets: video-text retrieval on EPIC-KITCHENS-100; action recognition on Charades-Ego; natural language query, moment query, and object state change classification on Ego4D challenge benchmarks.
Revised: Furthermore, we demonstrate strong performance on five egocentric downstream tasks across three datasets: video-text retrieval on EPIC-KITCHENS-100; action recognition on Charades-Ego; natural language query, moment query, and object state change classification on Ego4D challenge benchmarks.

Original: The dataset and code are available at https://github.com/showlab/EgoVLP .
Revised: The dataset and code are available at https://github.com/showlab/EgoVLP .

Original:  Introduction
Revised:  Introduction

Original: With the recent interest boom in computer vision and natural language processing
Revised: With the recent interest boom in computer vision and natural language processing

Original: Video-Language
Revised: Video-Language

Original: Pretraining (VLP) has prevailed, which aims to learn strong and transferable video-language representation for powering a broad spectrum of video-text downstream tasks, such as video-text retrieval [1, 2, 3], video question answering [4, 5, 6], and video captioning [7, 8, 9].
Revised: Pretraining (VLP) has prevailed, which aims to learn strong and transferable video-language representation for powering a broad spectrum of video-text downstream tasks, such as video-text retrieval [1, 2, 3], video question answering [4, 5, 6], and video captioning [7, 8, 9].

Original: The success of
Revised: The success of

Original: VLP mainly stems from the availability of large-scale open-world video-text datasets [10], which subsume a large number of videos sourced from the Web (e.g., YouTube) and pair videos with associated textual information.
Revised: VLP mainly stems from the availability of large-scale open-world video-text datasets [10], which subsume a large number of videos sourced from the Web (e.g., YouTube) and pair videos with associated textual information.

Original: For instance, HowTo100M [10] collects 134 K hours of instructional videos accompanied by noisy narrations yielded from Automatic Speech Recognition (ASR).
Revised: For instance, HowTo100M [10] collects 134 K hours of instructional videos accompanied by noisy narrations yielded from Automatic Speech Recognition (ASR).

Original: WebVid-

Revised: WebVid-

Original: M [3] scrapes 2 .
Revised: M [3] scrapes 2 .

Original: 5 M descriptive videos with well-formed long captions.
Revised: 5 M descriptive videos with well-formed long captions.

Original: Despite reaching an impressive data scale, videos in those existing video-text pretraining datasets are often of 3rd-person views and may have been edited before posting on the Web.
Revised: Despite reaching an impressive data scale, videos in those existing video-text pretraining datasets are often of 3rd-person views and may have been edited before posting on the Web.

Original: Yet, there is a noticeable domain gap between the existing video-text pretraining datasets and 1st-person view videos such as those videos captured by wearable cameras or smart glasses.
Revised: Yet, there is a noticeable domain gap between the existing video-text pretraining datasets and 1st-person view videos such as those videos captured by wearable cameras or smart glasses.

Original: Egocentric video has received increasing interests from the academia (e.g., activity recognition [11], activity anticipation [12], and video summarization [13]) and industry (various applications in robotics and augmented reality).
Revised: Egocentric video has received increasing interests from the academia (e.g., activity recognition [11], activity anticipation [12], and video summarization [13]) and industry (various applications in robotics and augmented reality).

Original: (cid:66)
Revised: (cid:66)

Original: [Table]
Revised: [Table]

Original: Dataset
Revised: Dataset

Original: Ego?
Revised: Ego?

Original: Domain
Revised: Domain

Original: Dur (hrs) # Clips # Texts
Revised: Dur (hrs) # Clips # Texts

Original: Example
Revised: Example

Original: [Table]
Revised: [Table]

Original: ] (cid:37)
Revised: ] (cid:37)

Original: [Table]
Revised: [Table]

Original: YouCook
Revised: YouCook

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ActivityNet Captions
Revised: ActivityNet Captions

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: However, due to such a domain gap, directly transferring the existing VLP models to egocentric downstream tasks cannot fully unleash the potential of large-scale pretraining approaches, which we have confirmed in the later experimental section.
Revised: However, due to such a domain gap, directly transferring the existing VLP models to egocentric downstream tasks cannot fully unleash the potential of large-scale pretraining approaches, which we have confirmed in the later experimental section.

Original: To bridge this gap, we are motivated to develop
Revised: To bridge this gap, we are motivated to develop

Original: Egocentric VLP models, which can greatly benefit various egocentric video downstream applications.
Revised: Egocentric VLP models, which can greatly benefit various egocentric video downstream applications.

Original: However, existing egocentric video datasets are of small scale and domain-specific, making Egocentric
Revised: However, existing egocentric video datasets are of small scale and domain-specific, making Egocentric

Original: VLP prohibitive.
Revised: VLP prohibitive.

Original: As illustrated in Tab. 1, the formerly largest egocentric video dataset EPICKITCHENS-100 [14] focuses on kitchens scenarios and its size is far smaller than those of the 3rd-person pretraining sets
Revised: As illustrated in Tab. 1, the formerly largest egocentric video dataset EPICKITCHENS-100 [14] focuses on kitchens scenarios and its size is far smaller than those of the 3rd-person pretraining sets

Original: WebVid-2M [3] and HowTo100M [10].
Revised: WebVid-2M [3] and HowTo100M [10].

Original: Fortunately, with the recent introduction of the massive-scale egocentric video dataset Ego4D [15], it becomes possible to unlock Egocentric VLP.
Revised: Fortunately, with the recent introduction of the massive-scale egocentric video dataset Ego4D [15], it becomes possible to unlock Egocentric VLP.

Original: Ego4D consists of 3 , 670 hours of videos with manually annotated narrations from 74 worldwide locations, covering a large variety of daily-life scenarios and activities.
Revised: Ego4D consists of 3 , 670 hours of videos with manually annotated narrations from 74 worldwide locations, covering a large variety of daily-life scenarios and activities.

Original: In this work, roused by the favorable scale and diversity of Ego4D, we make a significant effort to pave the way for Egocentric VLP with the following steps
Revised: In this work, roused by the favorable scale and diversity of Ego4D, we make a significant effort to pave the way for Egocentric VLP with the following steps

Original: : (i) To address the aforementioned issue of lacking a suitable large-scale egocentric video-language pretraining dataset, we create a video-text pretraining dataset EgoClip which contains a total of 3 .
Revised: : (i) To address the aforementioned issue of lacking a suitable large-scale egocentric video-language pretraining dataset, we create a video-text pretraining dataset EgoClip which contains a total of 3 .

Original: 8 M clean 1st-person clip-text pairs selected from Ego4D and covers diverse human daily activities.
Revised: 8 M clean 1st-person clip-text pairs selected from Ego4D and covers diverse human daily activities.

Original: (ii) To make full use of EgoClip for video-text representation learning, we propose a novel video-text contrastive objective EgoNCE to address unique challenges in egocentric pretraining datasets.
Revised: (ii) To make full use of EgoClip for video-text representation learning, we propose a novel video-text contrastive objective EgoNCE to address unique challenges in egocentric pretraining datasets.

Original: (iii)
Revised: (iii)

Original: We create a development benchmark
Revised: We create a development benchmark

Original: i
Revised: i

Original: e., Egocentric Multiple-Choices-Question, dubbed EgoMCQ , which contains 39 K questions created from Ego4D and focuses on evaluating video-text alignment.
Revised: e., Egocentric Multiple-Choices-Question, dubbed EgoMCQ , which contains 39 K questions created from Ego4D and focuses on evaluating video-text alignment.

Original: In contrast to other downstream benchmarks, EgoMCQ has a less discrepancy from EgoClip, powering us to accurately validate and quickly iterate our designs of EgoClip and EgoNCE.
Revised: In contrast to other downstream benchmarks, EgoMCQ has a less discrepancy from EgoClip, powering us to accurately validate and quickly iterate our designs of EgoClip and EgoNCE.

Original: (iv) We conduct extensive experiments to demonstrate the superiority of Egocentric VLP by transferring our pretrained representation to five egocentric downstream benchmarks and achieving state-of-the-art performance: 59 .
Revised: (iv) We conduct extensive experiments to demonstrate the superiority of Egocentric VLP by transferring our pretrained representation to five egocentric downstream benchmarks and achieving state-of-the-art performance: 59 .

Original: 4% nDCG on video-text retrieval of EPIC-KITCHENS-100 [14] 1 , 32 . 1% mAP on action recognition of Charades-Ego [17], and significant boosts over three Ego4D challenges 2 : natural language query, moment query and object state change classification.
Revised: 4% nDCG on video-text retrieval of EPIC-KITCHENS-100 [14] 1 , 32 . 1% mAP on action recognition of Charades-Ego [17], and significant boosts over three Ego4D challenges 2 : natural language query, moment query and object state change classification.

Original:  Related Work
Revised:  Related Work

Original: Video-Language Pretraining.
Revised: Video-Language Pretraining.

Original: The introduction of large-scale video-text datasets [10, 3] has enabled the emergence of VLP approaches to improve the video-text representation for various visionlanguage tasks [21, 22, 4], such as MIL-NCE which [23] proposes to match clips with multiple captions close in temporal to adapt the video-text misalignment of HowTo100M [10].
Revised: The introduction of large-scale video-text datasets [10, 3] has enabled the emergence of VLP approaches to improve the video-text representation for various visionlanguage tasks [21, 22, 4], such as MIL-NCE which [23] proposes to match clips with multiple captions close in temporal to adapt the video-text misalignment of HowTo100M [10].

Original: Dominant
Revised: Dominant

Original: VLP methods can be classified into two groups, namely: joint- and dual-encoders.
Revised: VLP methods can be classified into two groups, namely: joint- and dual-encoders.

Original: The former combines videos and texts as a single input to the encoder that performs the multimodal fusion.
Revised: The former combines videos and texts as a single input to the encoder that performs the multimodal fusion.

Original: For instance, [24, 25] concatenate videos and texts together before feeding them to a unified transformer.
Revised: For instance, [24, 25] concatenate videos and texts together before feeding them to a unified transformer.

Original: Conversely, methods like [3, 26] exploit dual encoders to independently project the video and text inputs into a common space and minimize the distance between the paired representations.
Revised: Conversely, methods like [3, 26] exploit dual encoders to independently project the video and text inputs into a common space and minimize the distance between the paired representations.

Original: These approaches are preferred in retrieval settings as they allow for efficient indexing of a single
Revised: These approaches are preferred in retrieval settings as they allow for efficient indexing of a single

Original: #C C gets food for the dog from pocket
Revised: #C C gets food for the dog from pocket

Original: Video clip
Revised: Video clip

Original: Narration
Revised: Narration

Original: EgoClip
Revised: EgoClip

Original: EgoMCQ
Revised: EgoMCQ

Original: Video Enc
Revised: Video Enc

Original: Text Enc
Revised: Text Enc

Original: Clip
Revised: Clip

Original: Text
Revised: Text

Original: EgoNCE P r e t r a i n
Revised: EgoNCE P r e t r a i n

Original: E v a l u a t e
Revised: E v a l u a t e

Original: Feedback of EgoClip design
Revised: Feedback of EgoClip design

Original: F e e d b a c k o f
Revised: F e e d b a c k o f

Original: M o d e l d e s i g n
Revised: M o d e l d e s i g n

Original: [Figure]
Revised: [Figure]

Original: Q: #C C holds cloth
Revised: Q: #C C holds cloth

Original: [Figure]
Revised: [Figure]

Original: modality [27, 28].
Revised: modality [27, 28].

Original: For example, Frozen [3] employs two separate transformers to encode video and text features and aligns them by video-text InfoNCE [29].
Revised: For example, Frozen [3] employs two separate transformers to encode video and text features and aligns them by video-text InfoNCE [29].

Original: In our work, we adopt the Frozen [3] but extend its InfoNCE to EgoNCE via positive and negative sampling for egocentric-friendly pretraining.
Revised: In our work, we adopt the Frozen [3] but extend its InfoNCE to EgoNCE via positive and negative sampling for egocentric-friendly pretraining.

Original: Egocentric Video Datasets.
Revised: Egocentric Video Datasets.

Original: Egocentric videos, collected by participants using wearable cameras, offer a natural perspective of people’s daily activities and raise a range of challenging research topics [11, 12, 30].
Revised: Egocentric videos, collected by participants using wearable cameras, offer a natural perspective of people’s daily activities and raise a range of challenging research topics [11, 12, 30].

Original: Several egocentric video datasets have been developed in decades, e.g., [20, 17, 31].
Revised: Several egocentric video datasets have been developed in decades, e.g., [20, 17, 31].

Original: However, since the collection of egocentric videos is expensive, previous egocentric datasets tend to be small-scale and domain-specific.
Revised: However, since the collection of egocentric videos is expensive, previous egocentric datasets tend to be small-scale and domain-specific.

Original: These limitations hinder 1st-person view research and fail to match the progress of 3rd-person counterparts, such as VLP [23, 24, 3].
Revised: These limitations hinder 1st-person view research and fail to match the progress of 3rd-person counterparts, such as VLP [23, 24, 3].

Original: Recently, a massive egocentric video dataset Ego4D [15] has been released, which consists of 3 , 670 hours of videos collected by 931 people from 74 worldwide locations in 9 different countries, where most videos are accompanied by narrations, audio, 3D meshes, and more.
Revised: Recently, a massive egocentric video dataset Ego4D [15] has been released, which consists of 3 , 670 hours of videos collected by 931 people from 74 worldwide locations in 9 different countries, where most videos are accompanied by narrations, audio, 3D meshes, and more.

Original: Furthermore, Ego4D introduces a suite of new challenging benchmarks (e.g., Natural language query and moment query) to fully explore the 1st-person visual experience.
Revised: Furthermore, Ego4D introduces a suite of new challenging benchmarks (e.g., Natural language query and moment query) to fully explore the 1st-person visual experience.

Original: With this step-changing dataset and benchmarks, Ego4D would lead to a new research surge on egocentric visual perception.
Revised: With this step-changing dataset and benchmarks, Ego4D would lead to a new research surge on egocentric visual perception.

Original:  EgoClip: An Egocentric Video-Language Pretraining Dataset
Revised:  EgoClip: An Egocentric Video-Language Pretraining Dataset

Original: Data curation.
Revised: Data curation.

Original: For our EgoClip dataset, we source data from Ego4D [15], which contains 9 ,untrimmed videos of varying lengths from 5 sec to 7 hrs.
Revised: For our EgoClip dataset, we source data from Ego4D [15], which contains 9 ,untrimmed videos of varying lengths from 5 sec to 7 hrs.

Original: From these videos, most are associated with dense timestamp-level narrations assigned by two different annotators, describing the camera wearer’s activities and interactions with objects.
Revised: From these videos, most are associated with dense timestamp-level narrations assigned by two different annotators, describing the camera wearer’s activities and interactions with objects.

Original: For example, the narration “ #C C puts the scrapper down. ”
Revised: For example, the narration “ #C C puts the scrapper down. ”

Original: corresponds to video content that occurred at 3 .
Revised: corresponds to video content that occurred at 3 .

Original: 70 s , where “ #C ” refers to the camera-wearer.
Revised: 70 s , where “ #C ” refers to the camera-wearer.

Original: Notably, narrations in Ego4D are well-aligned with the videos, both temporally and visually.
Revised: Notably, narrations in Ego4D are well-aligned with the videos, both temporally and visually.

Original: Prior pretraining datasets are characterized by a much greater level of temporal misalignment between the video and text (e.g., HowTo100M [10] narrations are scraped from ASR, yielding sentences misaligned or even unrelated to video content).
Revised: Prior pretraining datasets are characterized by a much greater level of temporal misalignment between the video and text (e.g., HowTo100M [10] narrations are scraped from ASR, yielding sentences misaligned or even unrelated to video content).

Original: We first filter Ego4D videos with missing narrations ( 7 . 4% of the total video duration) and exclude videos that belong to the validation and test sets of the Ego4D benchmark challenge [15] (a further 23 .
Revised: We first filter Ego4D videos with missing narrations ( 7 . 4% of the total video duration) and exclude videos that belong to the validation and test sets of the Ego4D benchmark challenge [15] (a further 23 .

Original: 9% of the total video duration).
Revised: 9% of the total video duration).

Original: Next, we retain textual annotation from both narrators in EgoClip, allowing us to consider narration diversity when pairing video and text for pretraining purposes.
Revised: Next, we retain textual annotation from both narrators in EgoClip, allowing us to consider narration diversity when pairing video and text for pretraining purposes.

Original: Finally, we adopt several criteria to filter the video and textual narrations, further reducing noise (detailed steps are provided in
Revised: Finally, we adopt several criteria to filter the video and textual narrations, further reducing noise (detailed steps are provided in

Original: Supplementary B.1).
Revised: Supplementary B.1).

Original: Overall, this procedure yields 2 .
Revised: Overall, this procedure yields 2 .

Original: 9 K hours of videos with 3 . 85 million narrations which cover 2927 hours of video from 129 different scenarios.
Revised: 9 K hours of videos with 3 . 85 million narrations which cover 2927 hours of video from 129 different scenarios.

Original: EgoClip has 21 .
Revised: EgoClip has 21 .

Original: 9 clips per minute with an average clip length of 1 . 0 seconds and a standard deviation of 0 .
Revised: 9 clips per minute with an average clip length of 1 . 0 seconds and a standard deviation of 0 .

Original: 9 seconds (the longest clip is up to 60 s).
Revised: 9 seconds (the longest clip is up to 60 s).

Original: Additional analyses are included in the Supplementary B.3.
Revised: Additional analyses are included in the Supplementary B.3.

Original: Creation of clip-text pairs.
Revised: Creation of clip-text pairs.

Original: Clip-text pairs are the common data format for VLP, but are usually not present in untrimmed video datasets with only a weak matching between narrations captions and videos.
Revised: Clip-text pairs are the common data format for VLP, but are usually not present in untrimmed video datasets with only a weak matching between narrations captions and videos.

Original: This was first discussed in HowTo100M [10], which pairs subtitles to video clips with corresponding time intervals to produce noisy pairs.
Revised: This was first discussed in HowTo100M [10], which pairs subtitles to video clips with corresponding time intervals to produce noisy pairs.

Original: This is not suitable for Ego4D since each narration is annotated with a single timestamp rather than an interval.
Revised: This is not suitable for Ego4D since each narration is annotated with a single timestamp rather than an interval.

Original: Thus, we design a contextual variable-length clip pairing strategy .
Revised: Thus, we design a contextual variable-length clip pairing strategy .

Original: Formally, narrations per video in Ego4D are organized as a sequence of sentences {T 0 , · · · , T n } with exact timestamps { t 0 , · · · , t n } , indicating an event i described by T i happened in the moment t i .
Revised: Formally, narrations per video in Ego4D are organized as a sequence of sentences {T 0 , · · · , T n } with exact timestamps { t 0 , · · · , t n } , indicating an event i described by T i happened in the moment t i .

Original: For a narration T i with timestamp t i , we pair a clip V i with following start and end timepoints:
Revised: For a narration T i with timestamp t i , we pair a clip V i with following start and end timepoints:

Original: [Equation]
Revised: [Equation]

Original: which represents a window centered around the timestamp t i with temporal duration equal to β i /α
Revised: which represents a window centered around the timestamp t i with temporal duration equal to β i /α

Original: β i is an adjustable parameter equal to the average temporal distance between pairs of consecutive narrations, i.e., (cid:80) n − 1 j =0 ( t j +1 − t j ) /n .
Revised: β i is an adjustable parameter equal to the average temporal distance between pairs of consecutive narrations, i.e., (cid:80) n − 1 j =0 ( t j +1 − t j ) /n .

Original: We compute β
Revised: We compute β

Original: i on a per video basis.
Revised: i on a per video basis.

Original: Conversely, α is a scale factor computed as the average of all β i across all videos in the EgoClip ( α = 4 . 9 seconds).
Revised: Conversely, α is a scale factor computed as the average of all β i across all videos in the EgoClip ( α = 4 . 9 seconds).

Original: Intuitively, Eq. 1 is derived from three observations: (i)
Revised: Intuitively, Eq. 1 is derived from three observations: (i)

Original: Centering t i helps involve prior information about the event i ; (ii) β i measures the clip duration according to its scenario, such as longer clips watching television ( 352 . 9 seconds) v.s. shorter clips harvesting crops ( 0 . 9 seconds); (iii) α controls the context granularity of clips (e.g., a large α pays more attention to rapid, atomic actions).
Revised: Centering t i helps involve prior information about the event i ; (ii) β i measures the clip duration according to its scenario, such as longer clips watching television ( 352 . 9 seconds) v.s. shorter clips harvesting crops ( 0 . 9 seconds); (iii) α controls the context granularity of clips (e.g., a large α pays more attention to rapid, atomic actions).

Original: We ablate these design choices in our experimental section.
Revised: We ablate these design choices in our experimental section.

Original:  Video-Language Pretraining Model
Revised:  Video-Language Pretraining Model

Original: To efficiently transfer video-language representation to egocentric downstream tasks (e.g., video-text retrieval on EPIC-KITCHENS-100 [20]), We prefer the dual-encoder (discussed in Sec. 2) as our VLP model architecture.
Revised: To efficiently transfer video-language representation to egocentric downstream tasks (e.g., video-text retrieval on EPIC-KITCHENS-100 [20]), We prefer the dual-encoder (discussed in Sec. 2) as our VLP model architecture.

Original: In particular, we emphasize devising a general pretraining objective EgoNCE to adapt the existing VLP model to the egocentric domain (e.g., EgoClip).
Revised: In particular, we emphasize devising a general pretraining objective EgoNCE to adapt the existing VLP model to the egocentric domain (e.g., EgoClip).

Original: 4.1 Architecture: Dual-encoder Pipeline
Revised: 4.1 Architecture: Dual-encoder Pipeline

Original: We choose Frozen [3] as our pretraining architecture.
Revised: We choose Frozen [3] as our pretraining architecture.

Original: Frozen [3] design encompasses an elegant and simple dual encoder strategy (one per modality) which has favorable characteristics (e.g., indexability and efficiency [27, 28]).
Revised: Frozen [3] design encompasses an elegant and simple dual encoder strategy (one per modality) which has favorable characteristics (e.g., indexability and efficiency [27, 28]).

Original: Note that this allows us to use our pretrained network in single-modality tasks (e.g., video-only tasks).
Revised: Note that this allows us to use our pretrained network in single-modality tasks (e.g., video-only tasks).

Original: In practice, the video encoder adopts the TimeSformer [32] architecture, while the text encoder builds upon DistillBERT [33].
Revised: In practice, the video encoder adopts the TimeSformer [32] architecture, while the text encoder builds upon DistillBERT [33].

Original: However, our approach is not limited to the encoder’s design (e.g., the video backbone can be replaced by SlowFast [34] or Video Swin [35]).
Revised: However, our approach is not limited to the encoder’s design (e.g., the video backbone can be replaced by SlowFast [34] or Video Swin [35]).

Original: In the rest of the paper we adopt this notation: ( V i , T i ) represents the video-text input to the model, while v i and t i are used to identify the video and text embeddings.
Revised: In the rest of the paper we adopt this notation: ( V i , T i ) represents the video-text input to the model, while v i and t i are used to identify the video and text embeddings.

Original: 4.2 EgoNCE: An Egocentric-friendly Pretraining Objective
Revised: 4.2 EgoNCE: An Egocentric-friendly Pretraining Objective

Original: A common pretraining objective for the dual-encoder VLP is InfoNCE [29], where the matching visual-text pairs in the batch are treated as positives while all other pairwise combinations in the batch are regarded as negatives.
Revised: A common pretraining objective for the dual-encoder VLP is InfoNCE [29], where the matching visual-text pairs in the batch are treated as positives while all other pairwise combinations in the batch are regarded as negatives.

Original: Formally, within a batch B = { 1 , · · · , N } , InfoNCE is computed by the sum of the video-to-text loss L v2t and text-to-video loss L t2v .
Revised: Formally, within a batch B = { 1 , · · · , N } , InfoNCE is computed by the sum of the video-to-text loss L v2t and text-to-video loss L t2v .

Original: For simplicity, we only formulate
Revised: For simplicity, we only formulate

Original: L v2t , whereas L t2v is defined in a symmetric way:
Revised: L v2t , whereas L t2v is defined in a symmetric way:

Original: [Equation]
Revised: [Equation]

Original: (2) where the i -th video embedding v i and j -th text embedding t j are L 2 normalized features, and τ is a temperature factor.
Revised: (2) where the i -th video embedding v i and j -th text embedding t j are L 2 normalized features, and τ is a temperature factor.

Original: However, this simple objective performs not well on large-scale video-text datasets like
Revised: However, this simple objective performs not well on large-scale video-text datasets like

Original: HowTo100M [10] due to the serious misalignment between the two modalities of data.
Revised: HowTo100M [10] due to the serious misalignment between the two modalities of data.

Original: Therefore, [36] proposes MIL-NCE which treats temporal nearest captions as positive samples.
Revised: Therefore, [36] proposes MIL-NCE which treats temporal nearest captions as positive samples.

Original: In this work, our 1st-person human daily activity dataset, i.e. EgoClip, presents two unique challenges compared to the existing 3rd-person view video-text datasets: Challenge (i) :
Revised: In this work, our 1st-person human daily activity dataset, i.e. EgoClip, presents two unique challenges compared to the existing 3rd-person view video-text datasets: Challenge (i) :

Original: The same action often occurs in different scenarios (e.g., “unlock the phone” could happen when “lying in bed” or “walking outdoors”).
Revised: The same action often occurs in different scenarios (e.g., “unlock the phone” could happen when “lying in bed” or “walking outdoors”).

Original: Challenge (ii) :
Revised: Challenge (ii) :

Original: Often, different actions appearing in the same scenario tend to have indistinguishable visual differences (e.g., when “working in front of the laptop”, “typing on the keyboard” or “moving the mouse” have similar feature representations).
Revised: Often, different actions appearing in the same scenario tend to have indistinguishable visual differences (e.g., when “working in front of the laptop”, “typing on the keyboard” or “moving the mouse” have similar feature representations).

Original: Evaluation on the text-video retrieval task is unreliable due to duplications
Revised: Evaluation on the text-video retrieval task is unreliable due to duplications

Original: [Figure]
Revised: [Figure]

Original: Retrieval result: Top clips are not GT but shall be considered as correct.
Revised: Retrieval result: Top clips are not GT but shall be considered as correct.

Original: Text query: #C C closes the refrigerator.
Revised: Text query: #C C closes the refrigerator.

Original: Inter-video
Revised: Inter-video

Original: Intra-video
Revised: Intra-video

Original: Text query
Revised: Text query

Original: Select the correct video clip fromcandidates
Revised: Select the correct video clip fromcandidates

Original: Answer with GT
Revised: Answer with GT

Original: [Figure]
Revised: [Figure]

Original: EgoMCQ
Revised: EgoMCQ

Original: [Figure]
Revised: [Figure]

Original: To overcome these two unique challenges, we propose a novel EgoNCE training objective which takes into account two simple yet efficient sampling strategies based on the vanilla InfoNCE.
Revised: To overcome these two unique challenges, we propose a novel EgoNCE training objective which takes into account two simple yet efficient sampling strategies based on the vanilla InfoNCE.

Original: Action-aware Positive Sampling.
Revised: Action-aware Positive Sampling.

Original: In this work, we make a reasonable assumption that the critical elements in linking visual actions to textual narrations are verbs and objects mentioned in the narrations (e.g., “drinking coffee” and “opening fridge”).
Revised: In this work, we make a reasonable assumption that the critical elements in linking visual actions to textual narrations are verbs and objects mentioned in the narrations (e.g., “drinking coffee” and “opening fridge”).

Original: Following this assumption, we can devise a clever method to address challenge (i).
Revised: Following this assumption, we can devise a clever method to address challenge (i).

Original: Specifically, for each narration, we identify its nouns and verbs and merge synonym words based on the Ego4D taxonomy dictionary [15], a thesaurus recording meaningful nouns/verbs in Ego4D narrations.
Revised: Specifically, for each narration, we identify its nouns and verbs and merge synonym words based on the Ego4D taxonomy dictionary [15], a thesaurus recording meaningful nouns/verbs in Ego4D narrations.

Original: Then, batch samples that shared at least one noun and at least one verb are treated as positive samples.
Revised: Then, batch samples that shared at least one noun and at least one verb are treated as positive samples.

Original: At last, for the sample i , we define its positive samples set within batch B as P i = { j ∈ B
Revised: At last, for the sample i , we define its positive samples set within batch B as P i = { j ∈ B

Original: | noun ( j ) ∩ noun ( i ) ̸
Revised: | noun ( j ) ∩ noun ( i ) ̸

Original: = ∅ , verb ( j ) ∩ verb ( i ) ̸ = ∅ } .
Revised: = ∅ , verb ( j ) ∩ verb ( i ) ̸ = ∅ } .

Original: Scene-aware Negative Sampling.
Revised: Scene-aware Negative Sampling.

Original: To address challenge (ii), we consider different actions in the same scenario as hard negative samples.
Revised: To address challenge (ii), we consider different actions in the same scenario as hard negative samples.

Original: Specifically, for each video clip i , we sample an adjacent clip i ′
Revised: Specifically, for each video clip i , we sample an adjacent clip i ′

Original: ∈
Revised: ∈

Original: N
Revised: N

Original: ( i ) , which is close to i in time within the same video.
Revised: ( i ) , which is close to i in time within the same video.

Original: We augment the original batch B with such hard negative samples and each sample i in B has its negative counterparts i ′ .
Revised: We augment the original batch B with such hard negative samples and each sample i in B has its negative counterparts i ′ .

Original: Hence the batch is updated as (cid:101) B = { 1 , 2 , · · · N
Revised: Hence the batch is updated as (cid:101) B = { 1 , 2 , · · · N

Original: [Equation]
Revised: [Equation]

Original: With these two sampling strategies, our new pretraining objective EgoNCE can be formulated as:
Revised: With these two sampling strategies, our new pretraining objective EgoNCE can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: Here the item in purple corresponds to our proposed action-aware positive samples and blue corresponds to our proposed scene-aware negative samples.
Revised: Here the item in purple corresponds to our proposed action-aware positive samples and blue corresponds to our proposed scene-aware negative samples.

Original: EgoNCE provides a general extension to adapt the existing VLP models for video-text pretraining datasets in the egocentric domain.
Revised: EgoNCE provides a general extension to adapt the existing VLP models for video-text pretraining datasets in the egocentric domain.

Original:  EgoMCQ: A Benchmark for Egocentric VLP Development
Revised:  EgoMCQ: A Benchmark for Egocentric VLP Development

Original: The need for a development benchmark.
Revised: The need for a development benchmark.

Original: We find that most egocentric benchmarks are domainspecific and focus on single-modality tasks (see Tab. 1).
Revised: We find that most egocentric benchmarks are domainspecific and focus on single-modality tasks (see Tab. 1).

Original: However, our purpose is to exploit Ego4D’s diversity to learn rich video-text representations.
Revised: However, our purpose is to exploit Ego4D’s diversity to learn rich video-text representations.

Original: Hence, to validate our design choices of the pretraining dataset (e.g., EgoClip), and model (e.g., EgoNCE), it is essential to measure performance on a benchmark highly aligned with the pretraining task.
Revised: Hence, to validate our design choices of the pretraining dataset (e.g., EgoClip), and model (e.g., EgoNCE), it is essential to measure performance on a benchmark highly aligned with the pretraining task.

Original: Therefore, we propose EgoMCQ, a new egocentric benchmark for reliable and fast developments of Egocentric VLP.
Revised: Therefore, we propose EgoMCQ, a new egocentric benchmark for reliable and fast developments of Egocentric VLP.

Original: Data source.
Revised: Data source.

Original: We start from the Ego4D data excluded from constructing the EgoClip, which mainly covers the validation set of the Ego4D challenge benchmarks.
Revised: We start from the Ego4D data excluded from constructing the EgoClip, which mainly covers the validation set of the Ego4D challenge benchmarks.

Original: Additionally, to assure that the scene is not visible during pretraining, we manually remove videos that share multiple views with the videos in EgoClip.
Revised: Additionally, to assure that the scene is not visible during pretraining, we manually remove videos that share multiple views with the videos in EgoClip.

Original: To ensure diversity, we randomly select one annotator’s narration for each video.
Revised: To ensure diversity, we randomly select one annotator’s narration for each video.

Original: We follow the same clip pairing strategy as Eq.
Revised: We follow the same clip pairing strategy as Eq.

Original: 1 to be consistent with the data format of EgoClip.
Revised: 1 to be consistent with the data format of EgoClip.

Original: Benchmarking task design.
Revised: Benchmarking task design.

Original: To determine the task for development, we first consider video-text retrieval since it highly aligns with the VLP pretraining objective.
Revised: To determine the task for development, we first consider video-text retrieval since it highly aligns with the VLP pretraining objective.

Original: However, as depicted in the top half of Fig. 2, for an action (e.g., close the refrigerator), there are substantial duplicates or semantically similar captions in Ego4D. This can cause issues in retrieval evaluation [37] making model training unreliable.
Revised: However, as depicted in the top half of Fig. 2, for an action (e.g., close the refrigerator), there are substantial duplicates or semantically similar captions in Ego4D. This can cause issues in retrieval evaluation [37] making model training unreliable.

Original: A straightforward approach to prevent this is deduplication (dedup), but it is challenging to devise a dedup criterion and perform well in the retrieval settings of a “one-to-whole validation set”.
Revised: A straightforward approach to prevent this is deduplication (dedup), but it is challenging to devise a dedup criterion and perform well in the retrieval settings of a “one-to-whole validation set”.

Original: Therefore, we select the Multiple-Choice Questions (MCQ) task for development since repetitions are highly unlikely given a small number of answers.
Revised: Therefore, we select the Multiple-Choice Questions (MCQ) task for development since repetitions are highly unlikely given a small number of answers.

Original: Grouping strategies.
Revised: Grouping strategies.

Original: To set up the MCQ task, a naive construction randomly groups five video clips to form options for a question.
Revised: To set up the MCQ task, a naive construction randomly groups five video clips to form options for a question.

Original: But we find randomly grouping is not challenging since options are highly likely to come from different videos and vary widely in content.
Revised: But we find randomly grouping is not challenging since options are highly likely to come from different videos and vary widely in content.

Original: We redefine this basic setting as “ inter-video ” and ensure that the five clips originate from different videos, aiming to distinguish instances from different scenarios (the left-bottom of Fig. 2).
Revised: We redefine this basic setting as “ inter-video ” and ensure that the five clips originate from different videos, aiming to distinguish instances from different scenarios (the left-bottom of Fig. 2).

Original: Furthermore, we propose a more challenging setting “ intra-video ” by grouping five continuous clips together.
Revised: Furthermore, we propose a more challenging setting “ intra-video ” by grouping five continuous clips together.

Original: This setting is regarded as a specific form of video-text localization focused on fine-grained context clues, such as hand interaction (the right-bottom of Fig. 2).
Revised: This setting is regarded as a specific form of video-text localization focused on fine-grained context clues, such as hand interaction (the right-bottom of Fig. 2).

Original: Dedup is performed within five options for each question for reliable assessment (see Supp.
Revised: Dedup is performed within five options for each question for reliable assessment (see Supp.

Original: C.1)
Revised: C.1)

Original: and we adopt accuracy as the EgoMCQ metric.
Revised: and we adopt accuracy as the EgoMCQ metric.

Original: Statistics.
Revised: Statistics.

Original: We finalize 39 K questions covering 198 K narrations with 468 hours of video, where the “inter-video” has 24 K questions covering 290 .
Revised: We finalize 39 K questions covering 198 K narrations with 468 hours of video, where the “inter-video” has 24 K questions covering 290 .

Original: 3 hours of videos.
Revised: 3 hours of videos.

Original: And the “intra-video” has 15 K questions and covers 178 .
Revised: And the “intra-video” has 15 K questions and covers 178 .

Original: 3 hours of videos.
Revised: 3 hours of videos.

Original: The average duration among the five options is 34 .seconds
Revised: The average duration among the five options is 34 .seconds

Original: (More statistics of EgoMCQ are shown in Supplementary C.3).
Revised: (More statistics of EgoMCQ are shown in Supplementary C.3).

Original:  Experiments
Revised:  Experiments

Original: We assess our Egocentric VLP along two directions: (i) We conduct an extensive analysis to explore key components of Egocentric VLP (e.g., EgoClip, EgoNCE, and EgoMCQ); (ii) we transfer our pretrained model to various downstream tasks to validate the quality of our video-text representation.
Revised: We assess our Egocentric VLP along two directions: (i) We conduct an extensive analysis to explore key components of Egocentric VLP (e.g., EgoClip, EgoNCE, and EgoMCQ); (ii) we transfer our pretrained model to various downstream tasks to validate the quality of our video-text representation.

Original: 6.1 Benchmarks and Settings
Revised: 6.1 Benchmarks and Settings

Original: We evaluate our VLP model on five egocentric benchmarks, spanning video-text tasks and pure video tasks, across three different datasets.
Revised: We evaluate our VLP model on five egocentric benchmarks, spanning video-text tasks and pure video tasks, across three different datasets.

Original: We briefly describe each task below.
Revised: We briefly describe each task below.

Original: Multi-Instance Retrieval of EPIC-KITCHENS-100.
Revised: Multi-Instance Retrieval of EPIC-KITCHENS-100.

Original: This task is modelled as a video-text retrieval which considers the semantic overlap between different videos narrations, where multiple videos may correspond to the same narration.
Revised: This task is modelled as a video-text retrieval which considers the semantic overlap between different videos narrations, where multiple videos may correspond to the same narration.

Original: The training set contains 67 .
Revised: The training set contains 67 .

Original: 2 K clips and validation set contains 9 .
Revised: 2 K clips and validation set contains 9 .

Original: 7 K clips.
Revised: 7 K clips.

Original: The evaluation metrics are mean Average Precision (mAP) and the normalized Discounted
Revised: The evaluation metrics are mean Average Precision (mAP) and the normalized Discounted

Original: Cumulative Gain (nDCG).
Revised: Cumulative Gain (nDCG).

Original: Natural Language Query of Ego4D Challenges.
Revised: Natural Language Query of Ego4D Challenges.

Original: The Natural Language Query task is modelled as a natural language grounding problem [38, 39, 40].
Revised: The Natural Language Query task is modelled as a natural language grounding problem [38, 39, 40].

Original: Given a language query and a video, the task aims at localizing the temporal interval within the video, in which the answer is deducible.
Revised: Given a language query and a video, the task aims at localizing the temporal interval within the video, in which the answer is deducible.

Original: The training set contains 11 .
Revised: The training set contains 11 .

Original: 3 K queries annotated from 1 K clips for this task, while the validation contains 3 .
Revised: 3 K queries annotated from 1 K clips for this task, while the validation contains 3 .

Original: 9 K queries collected from 0 .
Revised: 9 K queries collected from 0 .

Original: 3 K clips.
Revised: 3 K clips.

Original: The evaluation metric is Recall@ K for IoU =
Revised: The evaluation metric is Recall@ K for IoU =

Original: θ (R@ K -IoU =
Revised: θ (R@ K -IoU =

Original: θ ) [38] where θ is a threshold.
Revised: θ ) [38] where θ is a threshold.

Original: We evaluate for K ∈{ 1 , 5 } and θ ∈{ 0 . 3 , 0 . 5 } .
Revised: We evaluate for K ∈{ 1 , 5 } and θ ∈{ 0 . 3 , 0 . 5 } .

Original: Action Recognition of Charades-Ego.
Revised: Action Recognition of Charades-Ego.

Original: This dataset has 64 K instances, spanning 1st-person and 3rd-person views and covering 157 activity categories for training.
Revised: This dataset has 64 K instances, spanning 1st-person and 3rd-person views and covering 157 activity categories for training.

Original: We train and evaluate only on the 1st-person videos.
Revised: We train and evaluate only on the 1st-person videos.

Original: The validation set contains 847 videos for classification and each video belongs to multiple classes.
Revised: The validation set contains 847 videos for classification and each video belongs to multiple classes.

Original: The evaluation metric is
Revised: The evaluation metric is

Original: Moment Query of Ego4D Challenges.
Revised: Moment Query of Ego4D Challenges.

Original: The Moment Query task is a video-only task modelled as
Revised: The Moment Query task is a video-only task modelled as

Original: Temporal Action Localization [11].
Revised: Temporal Action Localization [11].

Original: Given a particular high-level activity category, the task solution consists of retrieving all the possible temporal windows where the activity occurs.
Revised: Given a particular high-level activity category, the task solution consists of retrieving all the possible temporal windows where the activity occurs.

Original: The training set
Revised: The training set

Original: [Table]
Revised: [Table]

Original: In all experiments, and contains 13 .
Revised: In all experiments, and contains 13 .

Original: 6 K instances from 1 .
Revised: 6 K instances from 1 .

Original: 5 K clips, while the validation set contains 4 .
Revised: 5 K clips, while the validation set contains 4 .

Original: 3 K instances from 0 .
Revised: 3 K instances from 0 .

Original: clips.
Revised: clips.

Original: The evaluation metrics are mAP and R@ K -IoU
Revised: The evaluation metrics are mAP and R@ K -IoU

Original: = θ for K ∈{ 1 , 5 } and θ ∈{ 0 . 3 , 0 . 5 , 0 .
Revised: = θ for K ∈{ 1 , 5 } and θ ∈{ 0 . 3 , 0 . 5 , 0 .

Original: Object State Change Classification (OSCC) of Ego4D Challenges.
Revised: Object State Change Classification (OSCC) of Ego4D Challenges.

Original: This OSCC task is modelled as an (N+1)-way classification aiming to identify an object’s state change in a given video.
Revised: This OSCC task is modelled as an (N+1)-way classification aiming to identify an object’s state change in a given video.

Original: The training and val.
Revised: The training and val.

Original: sets contain 41 K and 28 K clips, respectively.
Revised: sets contain 41 K and 28 K clips, respectively.

Original: The evaluation metric is accuracy.
Revised: The evaluation metric is accuracy.

Original: Implementation Details.
Revised: Implementation Details.

Original: Our codebase is based on the official Frozen 3 one and retains the same settings unless specified.
Revised: Our codebase is based on the official Frozen 3 one and retains the same settings unless specified.

Original: During pretraining, we sample 4 frames for each clip, and use the Adam optimizer [41] with a learning rate of 3 × 10 − 5 .
Revised: During pretraining, we sample 4 frames for each clip, and use the Adam optimizer [41] with a learning rate of 3 × 10 − 5 .

Original: To select the best method we pretrain our architecture for 10 epochs and use the best performing model on the EgoMCQ benchmark.
Revised: To select the best method we pretrain our architecture for 10 epochs and use the best performing model on the EgoMCQ benchmark.

Original: Pretraining takes two days on 32 A100 GPUs ( 1 , 536 GPU hrs).
Revised: Pretraining takes two days on 32 A100 GPUs ( 1 , 536 GPU hrs).

Original: 6.2 Ablation Studies
Revised: 6.2 Ablation Studies

Original: Ablation of the strategy used when creating EgoClip.
Revised: Ablation of the strategy used when creating EgoClip.

Original: We validate our proposed strategies, i.e.,
Revised: We validate our proposed strategies, i.e.,

Original: Eq.1 in Tab. 2, by comparing the following variants: (a) fixed length α , start at timestamp; (b) fixed length α , center at timestamp; (c) variable clip, start and end by adjacent timestamps; (d) our proposed strategy, scaled by 2 ; (e) our proposed strategy, scaled by 4 ; (f) our proposed strategy.
Revised: Eq.1 in Tab. 2, by comparing the following variants: (a) fixed length α , start at timestamp; (b) fixed length α , center at timestamp; (c) variable clip, start and end by adjacent timestamps; (d) our proposed strategy, scaled by 2 ; (e) our proposed strategy, scaled by 4 ; (f) our proposed strategy.

Original: We consider that a good pretraining dataset creation strategy should satisfy: (1) the VLP model trained on EgoClip should be able to well distinguish instances in EgoMCQ with the same data format; (2) the VLP model pretrained on EgoClip with the specific clip creation strategy should perform well on public downstream tasks (e.g., video-text retrieval on [20] and zero-shot for efficiency).
Revised: We consider that a good pretraining dataset creation strategy should satisfy: (1) the VLP model trained on EgoClip should be able to well distinguish instances in EgoMCQ with the same data format; (2) the VLP model pretrained on EgoClip with the specific clip creation strategy should perform well on public downstream tasks (e.g., video-text retrieval on [20] and zero-shot for efficiency).

Original: We draw several conclusions from Tab. 2: (i) The performance of EgoMCQ is well aligned with the zero-shot result on EPIC-KITCHENS-100, especially minor gain on downstream but noticeable on EgoMCQ, which means EgoMCQ provides valid feedback and is suitable as a development set.
Revised: We draw several conclusions from Tab. 2: (i) The performance of EgoMCQ is well aligned with the zero-shot result on EPIC-KITCHENS-100, especially minor gain on downstream but noticeable on EgoMCQ, which means EgoMCQ provides valid feedback and is suitable as a development set.

Original: (ii) Under the same clip length α , (b) surpassing (a) proves that centering at timestamp includes prior information is helpful.
Revised: (ii) Under the same clip length α , (b) surpassing (a) proves that centering at timestamp includes prior information is helpful.

Original: (iii) Variable-length clips make a big difference, as shown in (c) and (d).
Revised: (iii) Variable-length clips make a big difference, as shown in (c) and (d).

Original: [Table]
Revised: [Table]

Original: We evaluate accuracy performance on our development benchmark EgoMCQ.
Revised: We evaluate accuracy performance on our development benchmark EgoMCQ.

Original: Notably, with our designed β i , (d) outperforms (b) with a similar average clip length, which validates our key idea of “contextual varied clip length”.
Revised: Notably, with our designed β i , (d) outperforms (b) with a similar average clip length, which validates our key idea of “contextual varied clip length”.

Original: (iv) Based on (d), (e), and (f), we found a proper scale factor greater than 1 is preferred, which helps focus on a large of instantaneous actions densely labeled by
Revised: (iv) Based on (d), (e), and (f), we found a proper scale factor greater than 1 is preferred, which helps focus on a large of instantaneous actions densely labeled by

Original: Ego4D [15].
Revised: Ego4D [15].

Original: These ablation studies demonstrate the effectiveness of our proposed EgoClip creation strategy and EgoMCQ for development.
Revised: These ablation studies demonstrate the effectiveness of our proposed EgoClip creation strategy and EgoMCQ for development.

Original: Effect of EgoNCE.
Revised: Effect of EgoNCE.

Original: In this section, we evaluate the effect of the proposed sampling strategies for the
Revised: In this section, we evaluate the effect of the proposed sampling strategies for the

Original: EgoNCE objective (Eq. 3) on EgoMCQ and compare against a vanilla InfoNCE loss (Eq. 2).
Revised: EgoNCE objective (Eq. 3) on EgoMCQ and compare against a vanilla InfoNCE loss (Eq. 2).

Original: We ablate several configurations for positive and negative sampling strategies.
Revised: We ablate several configurations for positive and negative sampling strategies.

Original: The sampling strategy for positive pairs exploits language cues, while negative pairs rely on temporal, visual cues.
Revised: The sampling strategy for positive pairs exploits language cues, while negative pairs rely on temporal, visual cues.

Original: Given a text-video pair, we regard other text-video pairs as positive if the textual narrations: (a) share at
Revised: Given a text-video pair, we regard other text-video pairs as positive if the textual narrations: (a) share at

Original: [Table]
Revised: [Table]

Original: least noun, (
Revised: least noun, (

Original: ) share least
Revised: ) share least

Original: , and (c share at least noun pair.
Revised: , and (c share at least noun pair.

Original: Conversely, we define the following heuristics for negative sampling: (d) a random text-video pair from EgoClip, (e) a text-video pair from the same video, and (f) a text-video pair within 1 minute from the given video-text pair annotation timestamp.
Revised: Conversely, we define the following heuristics for negative sampling: (d) a random text-video pair from EgoClip, (e) a text-video pair from the same video, and (f) a text-video pair within 1 minute from the given video-text pair annotation timestamp.

Original: Tab.
Revised: Tab.

Original: 3 shows that using solely verbs (a) or nouns (b) for positive selection degrades the accuracy performance with respect to naive InfoNCE.
Revised: 3 shows that using solely verbs (a) or nouns (b) for positive selection degrades the accuracy performance with respect to naive InfoNCE.

Original: However, we successfully push the performance beyond the baseline results when considering both verbs and nouns jointly (c).
Revised: However, we successfully push the performance beyond the baseline results when considering both verbs and nouns jointly (c).

Original: Moreover, we notice that merely selecting negatives within the same video leads to better performance.
Revised: Moreover, we notice that merely selecting negatives within the same video leads to better performance.

Original: In particular, we obtain the best performance for temporally “hard negatives” (f).
Revised: In particular, we obtain the best performance for temporally “hard negatives” (f).

Original: Finally, we pick the optimal settings from positive and sides and combine them together for and
Revised: Finally, we pick the optimal settings from positive and sides and combine them together for and

Original: 6.3 Comparisons with State
Revised: 6.3 Comparisons with State

Original: of
Revised: of

Original: the-arts
Revised: the-arts

Original: Multi-Instance Retrieval.
Revised: Multi-Instance Retrieval.

Original: In Tab. 4, we report both zero-shot and fine-tuning evaluation results.
Revised: In Tab. 4, we report both zero-shot and fine-tuning evaluation results.

Original: In the zero-shot setting, pretraining with EgoClip ( 3 . 8 M), despite being smaller in scale, still outperforms
Revised: In the zero-shot setting, pretraining with EgoClip ( 3 . 8 M), despite being smaller in scale, still outperforms

Original: CC3M+WebVid-2M ( 5 . 5 M) and HowTo100M ( 136 M), validating the unique benefit of pretraining on egocentric data.
Revised: CC3M+WebVid-2M ( 5 . 5 M) and HowTo100M ( 136 M), validating the unique benefit of pretraining on egocentric data.

Original: When fine-tuned with 4 frames (rows 5-9), EgoClip pretraining maintains a margin over the best baseline CC3M+WebVid-2M, further verifying the viewpoint domain gap within fine-tuning.
Revised: When fine-tuned with 4 frames (rows 5-9), EgoClip pretraining maintains a margin over the best baseline CC3M+WebVid-2M, further verifying the viewpoint domain gap within fine-tuning.

Original: Lastly, we increase the sample frames of our finalized model as well as the best competitor CC3M+WebVid-2M pretraining to 16 (rows 10-11).
Revised: Lastly, we increase the sample frames of our finalized model as well as the best competitor CC3M+WebVid-2M pretraining to 16 (rows 10-11).

Original: As expected, performance gains accompany the frame increase.
Revised: As expected, performance gains accompany the frame increase.

Original: We deem that notable benefits come from better temporal modeling for frequent action in the 1st-person view.
Revised: We deem that notable benefits come from better temporal modeling for frequent action in the 1st-person view.

Original: Overall, our pretraining model outperforms the best baseline (JPoSE) by 1 . 0 mAP and 5 .
Revised: Overall, our pretraining model outperforms the best baseline (JPoSE) by 1 . 0 mAP and 5 .

Original: 9% nDCG while requiring fewer frames and input modalities.
Revised: 9% nDCG while requiring fewer frames and input modalities.

Original: Natural Language Query.
Revised: Natural Language Query.

Original: We report validation results on Tab. 5.
Revised: We report validation results on Tab. 5.

Original: We adopt the same baselines as introduced in [15], namely: 2DTAN [44] and VSLNet [45], and substitute the SlowFast-BERT features with our video and language representations.
Revised: We adopt the same baselines as introduced in [15], namely: 2DTAN [44] and VSLNet [45], and substitute the SlowFast-BERT features with our video and language representations.

Original: We observe a large boost in performance offered by our pretrained model on all metrics.
Revised: We observe a large boost in performance offered by our pretrained model on all metrics.

Original: Notably, we improve R@ 1 for IoU= 0 .
Revised: Notably, we improve R@ 1 for IoU= 0 .

Original: 3 from 5 .to 10 . 84 , despite our video branch not being pre-trained on Kinetics400.
Revised: 3 from 5 .to 10 . 84 , despite our video branch not being pre-trained on Kinetics400.

Original: Besides, we significantly surpass VLP pretrained on CC3M+WebVid-2M and HowTo100M. We believe that this increase is due
Revised: Besides, we significantly surpass VLP pretrained on CC3M+WebVid-2M and HowTo100M. We believe that this increase is due

Original: [Table] [Table]
Revised: [Table] [Table]

Original: to the egocentric data availability and the video-text interaction learned from large-scale pretraining.
Revised: to the egocentric data availability and the video-text interaction learned from large-scale pretraining.

Original: Please see
Revised: Please see

Original: [Table]
Revised: [Table]

Original: Action Recognition.
Revised: Action Recognition.

Original: We conduct action recognition on Charades-Ego, where categories are short phrases like “Holding some clothes”.
Revised: We conduct action recognition on Charades-Ego, where categories are short phrases like “Holding some clothes”.

Original: Thus this task can be solved as a video-text retrieval by leveraging the text representation.
Revised: Thus this task can be solved as a video-text retrieval by leveraging the text representation.

Original: We present the result in Tab. 6 under zero-shot and fine-tuning settings.
Revised: We present the result in Tab. 6 under zero-shot and fine-tuning settings.

Original: In zero-shot settings, our model outperforms two supervised baselines, which validates the stronger generalization of jointly learning video-text features.
Revised: In zero-shot settings, our model outperforms two supervised baselines, which validates the stronger generalization of jointly learning video-text features.

Original: After fine-tuning (rows 5-9), our model surpasses all VLP counterparts and improves over the state-of-the-art classifier Ego-Exo by 2 . 0 % with fewer sampled frames, which shows the superior advantage of joint video-text representations
Revised: After fine-tuning (rows 5-9), our model surpasses all VLP counterparts and improves over the state-of-the-art classifier Ego-Exo by 2 . 0 % with fewer sampled frames, which shows the superior advantage of joint video-text representations

Original: [Table]
Revised: [Table]

Original: Moment Query.
Revised: Moment Query.

Original: This task investigates the quality of video-only features.
Revised: This task investigates the quality of video-only features.

Original: We extract video features and provide them as input to the VSGN model [49].
Revised: We extract video features and provide them as input to the VSGN model [49].

Original: We report the validation results in Tab. 7,
Revised: We report the validation results in Tab. 7,

Original: We find that our features achieves the best performance over SlowFast features with an increase of 4 . 66% in Avg mAP.
Revised: We find that our features achieves the best performance over SlowFast features with an increase of 4 . 66% in Avg mAP.

Original: Moreover, we maintain better performance with respect to 3rd-person large-scale pretraining datasets.
Revised: Moreover, we maintain better performance with respect to 3rd-person large-scale pretraining datasets.

Original: This demonstrates that the 1st-person VLP model also learns competitive video representations.
Revised: This demonstrates that the 1st-person VLP model also learns competitive video representations.

Original: Please see the Supplementary E.5 for the test set results
Revised: Please see the Supplementary E.6 for the test set results

Original: [Table]
Revised: [Table]

Original: Object State Change Classification.
Revised: Object State Change Classification.

Original: We report the validation results on Tab. 8.
Revised: We report the validation results on Tab. 8.

Original: Once again, our model achieves the best performance of all baselines, 2 . 4% than CC3M+WebVid-2M counterparts, which indicates our visual representations are able to focus on the fine-grained clues related to state changes [Table]
Revised: Once again, our model achieves the best performance of all baselines, 2 . 4% than CC3M+WebVid-2M counterparts, which indicates our visual representations are able to focus on the fine-grained clues related to state changes.

Original: Summary of EgoNCE.
Revised: Summary of EgoNCE.

Original: From the above experimental results
Revised: From the above experimental results

Original: [Table]
Revised: [Table]

Original: Frozen pretrained on EgoClip with the
Revised: Frozen pretrained on EgoClip with the

Original: EgoNCE objective brings a consistent improvement over the InfoNCE on all downstream tasks, which comprehensively demonstrates the effect of EgoNCE, as well as the decision from EgoMCQ.
Revised: EgoNCE objective brings a consistent improvement over the InfoNCE on all downstream tasks, which comprehensively demonstrates the effect of EgoNCE, as well as the decision from EgoMCQ.

Original:  Conclusion, Limitations, and Societal Impacts.
Revised:  Conclusion, Limitations, and Societal Impacts.

Original: To the best of our knowledge, this work is the pioneering work to unlock Egocentric VLP.
Revised: To the best of our knowledge, this work is the pioneering work to unlock Egocentric VLP.

Original: (i)
Revised: (i)

Original: We devise a principled data curation and create EgoClip, an egocentric large-scale text-video pretraining dataset with 3 .
Revised: We devise a principled data curation and create EgoClip, an egocentric large-scale text-video pretraining dataset with 3 .

Original: 8 M clip-text pairs well-chosen from Ego4D. (ii)
Revised: 8 M clip-text pairs well-chosen from Ego4D. (ii)

Original: We exploit the particular characteristics of egocentric videos and devise EgoNCE with meaningful sampling strategies for effective egocentric pretraining.
Revised: We exploit the particular characteristics of egocentric videos and devise EgoNCE with meaningful sampling strategies for effective egocentric pretraining.

Original: (iii)
Revised: (iii)

Original: We create EgoMCQ, an egocentric video-language benchmark close to the pretraining set to support efficient exploration and development of EgoClip and EgoNCE.
Revised: We create EgoMCQ, an egocentric video-language benchmark close to the pretraining set to support efficient exploration and development of EgoClip and EgoNCE.

Original: Finally, we further demonstrate the strong representation of our egocentric pretraining on five tasks across three datasets.
Revised: Finally, we further demonstrate the strong representation of our egocentric pretraining on five tasks across three datasets.

Original: We believe that our EgoClip, EgoMCQ and EgoNCE would greatly benefit the egocentric video community, laying a good foundation for the new research trend of egocentric VLP.
Revised: We believe that our EgoClip, EgoMCQ and EgoNCE would greatly benefit the egocentric video community, laying a good foundation for the new research trend of egocentric VLP.

Original: Limitations.
Revised: Limitations.

Original: Our pretraining approach does not take into account the long-term temporal dependencies in long Ego4D videos.
Revised: Our pretraining approach does not take into account the long-term temporal dependencies in long Ego4D videos.

Original: We leave this for future work.
Revised: We leave this for future work.

Original: Societal impact.
Revised: Societal impact.

Original: Egocentric VLP learns real-world perception knowledge that may contribute to practical applications such as augmented reality and robotics.
Revised: Egocentric VLP learns real-world perception knowledge that may contribute to practical applications such as augmented reality and robotics.

Original: However, Ego4D videos collected by participants may contain users’ privacy and unintended biases, so should be used cautiously.
Revised: However, Ego4D videos collected by participants may contain users’ privacy and unintended biases, so should be used cautiously.

Original: We refer the readers to the Ego4D paper about further privacy and societal impacts.
Revised: We refer the readers to the Ego4D paper about further privacy and societal impacts.

Original:  Acknowledgements
Revised:  Acknowledgements

Original: This project is supported by the National Research Foundation, Singapore under its NRFF Award
Revised: This project is supported by the National Research Foundation, Singapore under its NRFF Award

Original: NRF-NRFF13-2021-0008, and Mike Zheng Shou’s Start-Up Grant from NUS.
Revised: NRF-NRFF13-2021-0008, and Mike Zheng Shou’s Start-Up Grant from NUS.

Original: The computational work for this article was partially performed on resources of the National Supercomputing Centre,
Revised: The computational work for this article was partially performed on resources of the National Supercomputing Centre,

Original: Singapore.
Revised: Singapore.

Original: Michael Wray and Dima Damen are supported by EPSRC UMPIRE (EP/T004991/1).
Revised: Michael Wray and Dima Damen are supported by EPSRC UMPIRE (EP/T004991/1).

Original: Mattia Soldan and Bernard Ghanem are supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding, as well as, the SDAIA-KAUST Center of Excellence in Data Science and Artificial
Revised: Mattia Soldan and Bernard Ghanem are supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding, as well as, the SDAIA-KAUST Center of Excellence in Data Science and Artificial

Original: Intelligence (SDAIA-KAUST AI).
Revised: Intelligence (SDAIA-KAUST AI).

Original: Thanks to Tencent Data Platform for the support of computing resources.
Revised: Thanks to Tencent Data Platform for the support of computing resources.

Original: Our work is built upon the Ego4D dataset, and we greatly appreciate the contributions and efforts of the Ego4D community.
Revised: Our work is built upon the Ego4D dataset, and we greatly appreciate the contributions and efforts of the Ego4D community.

Original: Appendix
Revised: Appendix

Original: We present the following items in the supplemental material:
Revised: We present the following items in the supplemental material:

Original: A .
Revised: A .

Original: Differentiating Egocentric VLP and Ego4D in Sec.
Revised: Differentiating Egocentric VLP and Ego4D in Sec.

Original: A. B .
Revised: A. B .

Original: Construction details and statistics of EgoClip pretraining dataset in Sec.
Revised: Construction details and statistics of EgoClip pretraining dataset in Sec.

Original: B. C .
Revised: B. C .

Original: Construction details and statistics of EgoMCQ benchmark in Sec.
Revised: Construction details and statistics of EgoMCQ benchmark in Sec.

Original: C. D .
Revised: C. D .

Original: Technical details of our VLP model in Sec.
Revised: Technical details of our VLP model in Sec.

Original: D. E .
Revised: D. E .

Original: Additional experimental details and results in Sec.
Revised: Additional experimental details and results in Sec.

Original: E.
Revised: E.

Original: A Differentiating Egocentric VLP and Ego4D
Revised: A Differentiating Egocentric VLP and Ego4D

Original: In our work, we study the video-language pretraining in a specific yet significant domain - the 1st-person view, which is motivated by the release of the Ego4D dataset.
Revised: In our work, we study the video-language pretraining in a specific yet significant domain - the 1st-person view, which is motivated by the release of the Ego4D dataset.

Original: However, there is a long way to pave from the Ego4D dataset to Egocentric VLP, which consists of the pretraining dataset, development set, model designs, and transferability evaluation.
Revised: However, there is a long way to pave from the Ego4D dataset to Egocentric VLP, which consists of the pretraining dataset, development set, model designs, and transferability evaluation.

Original: Since they are not as fully explored as their third-person counterparts, thus we pioneer them by ourselves and conduct a systematic study toward the egocentric video-language pretraining - the contribution of our work.
Revised: Since they are not as fully explored as their third-person counterparts, thus we pioneer them by ourselves and conduct a systematic study toward the egocentric video-language pretraining - the contribution of our work.

Original: A.1 Pretraining dataset
Revised: A.1 Pretraining dataset

Original: Despite the merits of Ego4D, it has not been proposed for video-language pretraining, and cannot be directly used as its untrimmed videos, no direct video-text pairs, and noisy data.
Revised: Despite the merits of Ego4D, it has not been proposed for video-language pretraining, and cannot be directly used as its untrimmed videos, no direct video-text pairs, and noisy data.

Original: We thus see our clear distinction and contribution in proposing a successful approach to curate a pretraining dataset, our proposed EgoClip.
Revised: We thus see our clear distinction and contribution in proposing a successful approach to curate a pretraining dataset, our proposed EgoClip.

Original: Notably, It is also non-trivial to figure out what is the best way of curating Ego4D to create a pretraining dataset EgoClip, e.g., our pairing approach outperforms the naive strategy with a large margin in the development set, which requires substantial design and experimental validations.
Revised: Notably, It is also non-trivial to figure out what is the best way of curating Ego4D to create a pretraining dataset EgoClip, e.g., our pairing approach outperforms the naive strategy with a large margin in the development set, which requires substantial design and experimental validations.

Original: We add a Tab. 9, as an extension of Tab. 1, to clearly show their difference.
Revised: We add a Tab. 9, as an extension of Tab. 1, to clearly show their difference.

Original: [Table]
Revised: [Table]

Original: Clips
Revised: Clips

Original: [Table]
Revised: [Table]

Original: (cid:33) diverse 3 .
Revised: (cid:33) diverse 3 .

Original: 6K5 .
Revised: 6K5 .

Original: M
Revised: M

Original: [Table]
Revised: [Table]

Original: (well-
Revised: (well-

Original: [Table]
Revised: [Table]

Original: from Ego4D)
Revised: from Ego4D)

Original: (cid:33) diverse 2 .
Revised: (cid:33) diverse 2 .

Original: 9 K 3 .
Revised: 9 K 3 .

Original: 8M 3 .
Revised: 8M 3 .

Original: 8 M
Revised: 8 M

Original: A.2 Development set
Revised: A.2 Development set

Original: In the 1st-person domain, there is lacking a satisfactory benchmark that good aligns with pretraining data diversity and focuses on video-text alignment.
Revised: In the 1st-person domain, there is lacking a satisfactory benchmark that good aligns with pretraining data diversity and focuses on video-text alignment.

Original: Therefore, we propose a new development set i.e. EgoMCQ to power rapid design of video-text pretraining i.e. its pretraining dataset and model pretraining objective.
Revised: Therefore, we propose a new development set i.e. EgoMCQ to power rapid design of video-text pretraining i.e. its pretraining dataset and model pretraining objective.

Original: A.3 Model designs
Revised: A.3 Model designs

Original: We select Frozen as the baseline because its elegant and scalable dual-encoder architecture is representative in state-of-the-art VLP methods.
Revised: We select Frozen as the baseline because its elegant and scalable dual-encoder architecture is representative in state-of-the-art VLP methods.

Original: Besides, corresponding to MIL-NCE [23] built on top of the 3rd-person domain’s HowTo100M [10], we aim to explore a general pretraining objective i.e., EgoNCE to learn rich video-text representations in 1st-person domains.
Revised: Besides, corresponding to MIL-NCE [23] built on top of the 3rd-person domain’s HowTo100M [10], we aim to explore a general pretraining objective i.e., EgoNCE to learn rich video-text representations in 1st-person domains.

Original: A.4 Transferability evaluation
Revised: A.4 Transferability evaluation

Original: Extensive experiments and promising results demonstrate the effectiveness and necessity of Egocentric VLP, which will greatly benefit the egocentric community.
Revised: Extensive experiments and promising results demonstrate the effectiveness and necessity of Egocentric VLP, which will greatly benefit the egocentric community.

Original: Note that Ego4D has not been used previously for any downstream tasks on other datasets.
Revised: Note that Ego4D has not been used previously for any downstream tasks on other datasets.

Original: This is also where our work makes significant value.
Revised: This is also where our work makes significant value.

Original: B Construction details and statistics of EgoClip pretraining dataset B.1 Data filtering
Revised: B Construction details and statistics of EgoClip pretraining dataset B.1 Data filtering

Original: After we source video-text data for EgoClip, we adopt the following criteria to further reduce noise:
Revised: After we source video-text data for EgoClip, we adopt the following criteria to further reduce noise:

Original: (i)
Revised: (i)

Original: We select double-sized stereo videos ( 1 . 3% videos dur) and keep half per video for a normal size.
Revised: We select double-sized stereo videos ( 1 . 3% videos dur) and keep half per video for a normal size.

Original: (ii)
Revised: (ii)

Original: We discard videos with an aspect ratio greater than 2 ( 0 .
Revised: We discard videos with an aspect ratio greater than 2 ( 0 .

Original: 4% videos dur).
Revised: 4% videos dur).

Original: (iii) We filter narrations with unsure tags ( 4 . 0% texts)
Revised: (iii) We filter narrations with unsure tags ( 4 . 0% texts)

Original: e
Revised: e

Original: g. “ #C C washes #unsure in sink ”.
Revised: g. “ #C C washes #unsure in sink ”.

Original: (iv) We remove narrations less than 3 words ( 0 . 9% texts), since such narrations generally cannot be deduced from the video, e.g., “ #C C speaks ”, “ #C C looks ”.
Revised: (iv) We remove narrations less than 3 words ( 0 . 9% texts), since such narrations generally cannot be deduced from the video, e.g., “ #C C speaks ”, “ #C C looks ”.

Original: B.2 Data compression
Revised: B.2 Data compression

Original: The Ego4D videos are untrimmed, which tend to be very long (average 24 mins and max to 7 hrs) and have large resolution (e.g., 1920 × 1080 , 1440 × 1080 ), so it is impossible to adopt untrimmed videos as model input due to heavy data loading.
Revised: The Ego4D videos are untrimmed, which tend to be very long (average 24 mins and max to 7 hrs) and have large resolution (e.g., 1920 × 1080 , 1440 × 1080 ), so it is impossible to adopt untrimmed videos as model input due to heavy data loading.

Original: Therefore we propose to compress them:
Revised: Therefore we propose to compress them:

Original: (i)
Revised: (i)

Original: We first resize all videos with short size 256 .
Revised: We first resize all videos with short size 256 .

Original: (ii) Chunk each resized video into several segments, which are up to 10 min in length.
Revised: (ii) Chunk each resized video into several segments, which are up to 10 min in length.

Original: During pretraining, given the start and end time points of a clip, we only load the segment that this clip belongs to, rather than the whole video.
Revised: During pretraining, given the start and end time points of a clip, we only load the segment that this clip belongs to, rather than the whole video.

Original: To this end, we are able to perform efficient end-to-end pretraining with raw RGB videos as model input.
Revised: To this end, we are able to perform efficient end-to-end pretraining with raw RGB videos as model input.

Original: One epoch of pretraining 3 .
Revised: One epoch of pretraining 3 .

Original: 8 M video-text pairs costs 6 hrs on 32 V100 GPUs
Revised: 8 M video-text pairs costs 6 hrs on 32 V100 GPUs

Original: 192 GPU hrs).
Revised: 192 GPU hrs).

Original: B.3 Data analysis
Revised: B.3 Data analysis

Original: Geographic diversity.
Revised: Geographic diversity.

Original: We present the distribution of EgoClip clips source in Fig 3, which covers worldwide 13 institutions from 9 different countries [15], including: Europe (UK, Italy); Asia (India,
Revised: We present the distribution of EgoClip clips source in Fig. 3, which covers worldwide 13 institutions from 9 different countries [15], including: Europe (UK, Italy); Asia (India,

Original: Japan, Singapore, Kingdom of Saudi Arabia); America (USA, Colombia); Africa (Rwanda).
Revised: Japan, Singapore, Kingdom of Saudi Arabia); America (USA, Colombia); Africa (Rwanda).

Original: Therefore, our created pretraining dataset inherited the good geographic as well as participants diversities of Ego4D (More details can be found in “Supp.
Revised: Therefore, our created pretraining dataset inherited the good geographic as well as participants diversities of Ego4D (More details can be found in “Supp.

Original: C. Demographics” in Ego4D paper [15]).
Revised: C. Demographics” in Ego4D paper [15]).

Original: b r i s t o l c m u k a u s t
Revised: b r i s t o l c m u k a u s t

Original: u t o k y
Revised: u t o k y

Original: o f
Revised: o f

Original: r l _ t r a c k _ 1 _ p u b li c m i nn e s o t a un i c t iii t h nu s c m u _ a f r i c a un i a n d e s g
Revised: r l _ t r a c k _ 1 _ p u b li c m i nn e s o t a un i c t iii t h nu s c m u _ a f r i c a un i a n d e s g

Original: e o r
Revised: e o r

Original: g
Revised: g

Original: i a t e c h i n d i a n a
Revised: i a t e c h i n d i a n a

Original: Universities     # C li p - t e x t p a i r s 1e
Revised: Universities     # C li p - t e x t p a i r s 1e

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Scenario diversity.
Revised: Scenario diversity.

Original: We have statistics the scenario distribution of EgoClip in Fig 4, which covershuman daily scenarios e.g., household (cooking, cleaning), outdoor (shopping, hiking), workplace (at desk, on a laptop), leisure (playing board games), etc.
Revised: We have statistics the scenario distribution of EgoClip in Fig. 4, which covershuman daily scenarios e.g., household (cooking, cleaning), outdoor (shopping, hiking), workplace (at desk, on a laptop), leisure (playing board games), etc.

Original: Notably, this distribution is long tailed, where the largest scenario “Crafting/knitting/sewing/drawing/painting” includes 622 K (11 .
Revised: Notably, this distribution is long tailed, where the largest scenario “Crafting/knitting/sewing/drawing/painting” includes 622 K (11 .

Original: 1%) and the smallest scenario “Hair and Makeup stylist” contains 35 instances.
Revised: 1%) and the smallest scenario “Hair and Makeup stylist” contains 35 instances.

Original: Clip analysis.
Revised: Clip analysis.

Original: We present the statistics on the created clips in EgoClip.
Revised: We present the statistics on the created clips in EgoClip.

Original: Fig 6 (a) shows the distribution of clip frequency over the 2 .
Revised: Fig. 6 (a) shows the distribution of clip frequency over the 2 .

Original: 9 K pretraining set videos (For each video, we calculate two frequencies from two annotators respectively).
Revised: 9 K pretraining set videos (For each video, we calculate two frequencies from two annotators respectively).

Original: The varying clip frequencies are mainly dependent on manual narrations that are annotated based on the video scenarios and activities.
Revised: The varying clip frequencies are mainly dependent on manual narrations that are annotated based on the video scenarios and activities.

Original: There have average 13 . 4 clips per minute of video, maximize to 175 .
Revised: There have average 13 . 4 clips per minute of video, maximize to 175 .

Original: 8 narrations / minute and minimize to 0 .
Revised: 8 narrations / minute and minimize to 0 .

Original: 06 narrations / minute.
Revised: 06 narrations / minute.

Original: Our clip creation strategy Eq.
Revised: Our clip creation strategy Eq.

Original: (1) takes this characteristic into account by estimating clip length based on the frequency of the video that the clip belongs.
Revised: (1) takes this characteristic into account by estimating clip length based on the frequency of the video that the clip belongs.

Original: Fig 6 (b) displays the distribution of clip duration.
Revised: Fig. 6 (b) displays the distribution of clip duration.

Original: The average duration is 0 .
Revised: The average duration is 0 .

Original: 98 seconds with a standard deviation of 0 .
Revised: 98 seconds with a standard deviation of 0 .

Original: 95 seconds, and 69 .
Revised: 95 seconds, and 69 .

Original: 5% of clips are less than 1 . 0 seconds in length, due to the massive atomic instantaneous actions densely labeled by Ego4D. Besides, the clip might be max to 65 .
Revised: 5% of clips are less than 1 . 0 seconds in length, due to the massive atomic instantaneous actions densely labeled by Ego4D. Besides, the clip might be max to 65 .

Original: 36 seconds, which corresponding to the scenario that “a people walking in a forest”.
Revised: 36 seconds, which corresponding to the scenario that “a people walking in a forest”.

Original: [Figure]
Revised: [Figure]

Original: Narration analysis.
Revised: Narration analysis.

Original: In Fig 6 (c), we present the distribution of narration words length.
Revised: In Fig. 6 (c), we present the distribution of narration words length.

Original: The average words length of EgoClip narration is 9 . 39 .
Revised: The average words length of EgoClip narration is 9 . 39 .

Original: Notably, the EgoClip narrations cover 116 verbs and 555 nouns, where we merge the semantically synonyms words, e.g., the nouns of “handkerchief”,“napkin”,“serviette”,“tissue”,“wipe” both belong to “napkin”.
Revised: Notably, the EgoClip narrations cover 116 verbs and 555 nouns, where we merge the semantically synonyms words, e.g., the nouns of “handkerchief”,“napkin”,“serviette”,“tissue”,“wipe” both belong to “napkin”.

Original: Each narration of EgoClip have 1 . 84 nouns and 0 .
Revised: Each narration of EgoClip have 1 . 84 nouns and 0 .

Original: 87 verbs on average.
Revised: 87 verbs on average.

Original: p u t t a k e i n s p e c t h o l d m o v e w a l k t u r n c u t a d j u s t t o u c h r e m o v e o p e n o p e r a t e p u ll w i p e t h r o w c l o s e p a i n t t a l k p o u r p l a y a pp l y a rr a n g e h
Revised: p u t t a k e i n s p e c t h o l d m o v e w a l k t u r n c u t a d j u s t t o u c h r e m o v e o p e n o p e r a t e p u ll w i p e t h r o w c l o s e p a i n t t a l k p o u r p l a y a pp l y a rr a n g e h

Original: i t c l e a n r o ll c
Revised: i t c l e a n r o ll c

Original: o n s u m e w a s h s t a n d p r e ss d i p s c oo p m i x f o l d
Revised: o n s u m e w a s h s t a n d p r e ss d i p s c oo p m i x f o l d

Original: d
Revised: d

Original: r
Revised: r

Original: i v e p
Revised: i v e p

Original: u s h d r a w g
Revised: u s h d r a w g

Original: i v e s e w k n i t r e a d s i t s h a k e a tt a c h c li m b w r
Revised: i v e s e w k n i t r e a d s i t s h a k e a tt a c h c li m b w r

Original: i t e s t r e t
Revised: i t e s t r e t

Original: c h f r y s c r a p e r e p a i r
Revised: c h f r y s c r a p e r e p a i r

Original: [Figure]
Revised: [Figure]

Original: 024# C li p - t e x t p a i r s1e
Revised: # C li p - t e x t p a i r s

Original: [Figure]
Revised: [Figure]

Original: n a p k i n p e r s o n t a b l e h a n d f l oo r w oo d c o n t a i n e r c
Revised: n a p k i n p e r s o n t a b l e h a n d f l oo r w oo d c o n t a i n e r c

Original: l o t h p a p e r p l a y i n g s t
Revised: l o t h p a p e r p l a y i n g s t

Original: r
Revised: r

Original: i n g b a g b oo k p a i n t b r u s h
Revised: i n g b a g b oo k p a i n t b r u s h

Original: p h o n
Revised: p h o n

Original: e b r u s h li
Revised: e b r u s h li

Original: d d o u g h w a ll b o w l p l a t e b r i c k c e m e n t m e t a l s a n dp a p e r w a t e r p a n t s k n i f e t r a y s p oo n s t i c k s i n k s h
Revised: d d o u g h w a ll b o w l p l a t e b r i c k c e m e n t m e t a l s a n dp a p e r w a t e r p a n t s k n i f e t r a y s p oo n s t i c k s i n k s h

Original: i r t p l a n t p u zz l e t r o w e l s o il p
Revised: i r t p l a n t p u zz l e t r o w e l s o il p

Original: o t c u p w i r e c a b i n e t c
Revised: o t c u p w i r e c a b i n e t c

Original: l
Revised: l

Original: a y c o m p u t e r c o un t e r t
Revised: a y c o m p u t e r c o un t e r t

Original: o p f a u c e t m o l d a r t w o r k p e n b a
Revised: o p f a u c e t m o l d a r t w o r k p e n b a

Original: ll f oo d
Revised: ll f oo d

Original: [Figure]
Revised: [Figure]

Original: We further display the distribution of the top 50 most frequently verbs and nouns of EgoClip in Fig 7.
Revised: We further display the distribution of the top 50 most frequently verbs and nouns of EgoClip in Fig.

Original: The most common nouns is “napkin”, which appeared in 1 . 

Revised: The most common nouns is “napkin”, which appeared in 1 .

Original: M (27 . 06%) clips.
Revised: M (27 . 06%) clips.

Original: Visualizations.
Revised: Visualizations.

Original: In Fig 8, we visualize some clip-text pairs created by our strategy.
Revised: In Fig. 8, we visualize some clip-text pairs created by our strategy.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: Question 2: #C C hits the wooden plank with the hammer in his right hand.
Revised: Question 2: #C C hits the wooden plank with the hammer in his right hand.

Original: Question 1: #C C adjusts wood
Revised: Question 1: #C C adjusts wood

Original: [Table]
Revised: [Table]

Original: Question 3: #C C bends a cable using a cable stripper
Revised: Question 3: #C C bends a cable using a cable stripper

Original: Question 4: #C C sprays the disinfectant on the sink
Revised: Question 4: #C C sprays the disinfectant on the sink

Original: [Table]
Revised: [Table]

Original: Question 2: #C C pulls the sisal fiber
Revised: Question 2: #C C pulls the sisal fiber

Original: Question 1: #C C takes off the pancake
Revised: Question 1: #C C takes off the pancake

Original: [Figure]
Revised: [Figure]

Original: Question 3: #C C washes the chopping board.
Revised: Question 3: #C C washes the chopping board.

Original: Question 4: #C C picks out a case from the drawer with his left hand.
Revised: Question 4: #C C picks out a case from the drawer with his left hand.

Original: [Figure]
Revised: [Figure]

Original: C Construction details and statistics of EgoMCQ benchmark C.1 Data deduplication
Revised: C Construction details and statistics of EgoMCQ benchmark C.1 Data deduplication

Original: To ensure repetitions do not appear in five options, we devise a deduplication strategy.
Revised: To ensure repetitions do not appear in five options, we devise a deduplication strategy.

Original: Initially, we use Sentence-BERT to extract sentence-level embeddings of narrations and set a manual threshold to remove repetitions.
Revised: Initially, we use Sentence-BERT to extract sentence-level embeddings of narrations and set a manual threshold to remove repetitions.

Original: But in this way, it is hard to control the fine-grained diversity between narrations, e.g., two narrations “ #C C closes the refrigerator with his left hand. ”
Revised: But in this way, it is hard to control the fine-grained diversity between narrations, e.g., two narrations “ #C C closes the refrigerator with his left hand. ”

Original: and “ #C
Revised: and “ #C

Original: C opens the refrigerator with his left hand. ”
Revised: C opens the refrigerator with his left hand. ”

Original: only differ in one word.
Revised: only differ in one word.

Original: These two sentences have a high score in sentence-level similarity, but are entirely different in semantic meanings.
Revised: These two sentences have a high score in sentence-level similarity, but are entirely different in semantic meanings.

Original: We hope to keep them and let the model distinguish them, especially in our intra-video setting.
Revised: We hope to keep them and let the model distinguish them, especially in our intra-video setting.

Original: Therefore, we propose to extract the first verb and the first noun of each narration and use them to define a tag for each narration.
Revised: Therefore, we propose to extract the first verb and the first noun of each narration and use them to define a tag for each narration.

Original: The narrations shared with the same verb and the noun will be assigned the same tag.
Revised: The narrations shared with the same verb and the noun will be assigned the same tag.

Original: We also consider the words synonyms (based on Ego4D taxonomy dictionary [15]).
Revised: We also consider the words synonyms (based on Ego4D taxonomy dictionary [15]).

Original: For instance, “ #C C take the phone ” and “ #C C pick the cellphone ” are semantically same in verb and noun thus will be assigned the same tag.
Revised: For instance, “ #C C take the phone ” and “ #C C pick the cellphone ” are semantically same in verb and noun thus will be assigned the same tag.

Original: Then the narrations shared with the same tag are treated as repetitions, we only keep one of them and sample a new one until the tags of the five options are different.
Revised: Then the narrations shared with the same tag are treated as repetitions, we only keep one of them and sample a new one until the tags of the five options are different.

Original: C.2 Multiple-views removing
Revised: C.2 Multiple-views removing

Original: We first select videos from NUS/Minnesota/Georgia Tech/Indiana sources, which contribute to the multi-camera video data.
Revised: We first select videos from NUS/Minnesota/Georgia Tech/Indiana sources, which contribute to the multi-camera video data.

Original: Then, based on the metadata of the video (i.e. times when videos were collected), we observed that videos collected in the same timeframe tend to be multi-views of the same recording, so we manually group these videos into the same split to ensure the same scene does not appear in another split.
Revised: Then, based on the metadata of the video (i.e. times when videos were collected), we observed that videos collected in the same timeframe tend to be multi-views of the same recording, so we manually group these videos into the same split to ensure the same scene does not appear in another split.

Original: C.3 Data analysis
Revised: C.3 Data analysis

Original: We finalize 39 K questions covering 198 K narrations with 468 hours of video, where the “inter-video” has 24 K questions covering 290 .
Revised: We finalize 39 K questions covering 198 K narrations with 468 hours of video, where the “inter-video” has 24 K questions covering 290 .

Original: 3 hours of videos.
Revised: 3 hours of videos.

Original: And the “intra-video” has 15 K questions and covers 178 .
Revised: And the “intra-video” has 15 K questions and covers 178 .

Original: 3 hours of videos.
Revised: 3 hours of videos.

Original: The average duration among the five options is 34 .
Revised: The average duration among the five options is 34 .

Original: 2 seconds.
Revised: 2 seconds.

Original: Geographic diversity.
Revised: Geographic diversity.

Original: We present the geographic diversity of EgoMCQ in Fig 10, which coversinstitutions and is align with the geographic diversity of EgoClip.
Revised: We present the geographic diversity of EgoMCQ in Fig. 10, which coversinstitutions and is align with the geographic diversity of EgoClip.

Original: k a u s t c m u nu s un i a n d e s b
Revised: k a u s t c m u nu s un i a n d e s b

Original: r
Revised: r

Original: i s t o l iii t
Revised: i s t o l iii t

Original: h f r l _ t r a c k _ 1 _ p u b li c un i c t g e o r
Revised: h f r l _ t r a c k _ 1 _ p u b li c un i c t g e o r

Original: g
Revised: g

Original: i a t e c h m i nn e s o t a u t
Revised: i a t e c h m i nn e s o t a u t

Original: o k y
Revised: o k y

Original: o c m u _ a f r
Revised: o c m u _ a f r

Original: i c a i n d i a n a
Revised: i c a i n d i a n a

Original: Universities      # C li p - t e x t p a i r s 1e
Revised: Universities      # C li p - t e x t p a i r s 1e

Original: Scenario diversity.
Revised: Scenario diversity.

Original: In Fig 5, we present the scenario distribution of EgoMCQ, which coversscenario.
Revised: In Fig. 5, we present the scenario distribution of EgoMCQ, which coversscenario.

Original: The largest scenario “Cooking” includes 49 K (15 . 3%) clips and the smallest scenario “Bus” contains 6 instances.
Revised: The largest scenario “Cooking” includes 49 K (15 . 3%) clips and the smallest scenario “Bus” contains 6 instances.

Original: EgoMCQ covers 71% of scenarios in EgoClipand has other 3 scenarios not appear in EgoClip.
Revised: EgoMCQ covers 71% of scenarios in EgoClipand has other 3 scenarios not appear in EgoClip.

Original: EgoMCQ is close to EgoClip both in terms of geography and scene diversity, making it a good development set for EgoClip pretraining.
Revised: EgoMCQ is close to EgoClip both in terms of geography and scene diversity, making it a good development set for EgoClip pretraining.

Original: Verbs and Nouns.
Revised: Verbs and Nouns.

Original: EgoMCQ covers 198 K narrations and each narration contains 3 . 15 nouns and 0 .
Revised: EgoMCQ covers 198 K narrations and each narration contains 3 . 15 nouns and 0 .

Original: 97 verbs in average.
Revised: 97 verbs in average.

Original: In Fig 11, we display the top 50 most frequently verb and nouns of EgoMCQ.
Revised: In Fig. 11, we display the top 50 most frequently verb and nouns of EgoMCQ.

Original: The mostly common noun is “hand”, covering 86 K (36 . 2%) instances and the mostly frequently verb is “pick”, which covers 28 K (12 . 0%) instances.
Revised: The mostly common noun is “hand”, covering 86 K (36 . 2%) instances and the mostly frequently verb is “pick”, which covers 28 K (12 . 0%) instances.

Original: [Table]
Revised: [Table]

Original: # C li p - t e x t p a i r s 1e
Revised: # C li p - t e x t p a i r s 1e

Original: [Figure]
Revised: [Figure]

Original: Visualization.
Revised: Visualization.

Original: In Fig 9, we display examples of both the intra and inter settings of EgoMCQ.
Revised: In Fig. 9, we display examples of both the intra and inter settings of EgoMCQ.

Original: D Technical details of our VLP model
Revised: D Technical details of our VLP model

Original: In this section, we present more technical details of our VLP model, mainly architecture and pretraining objective.
Revised: In this section, we present more technical details of our VLP model, mainly architecture and pretraining objective.

Original: D.1 Architecture: Frozen-in-time [3]
Revised: D.1 Architecture: Frozen-in-time [3]

Original: Video Encoder.
Revised: Video Encoder.

Original: The video encoder is built upon Timesformer [32], a convolution-free video backbone that divides space-time attention in an efficient manner.
Revised: The video encoder is built upon Timesformer [32], a convolution-free video backbone that divides space-time attention in an efficient manner.

Original: Take a RGB video clip V i ∈ R T × 3 × H × W with T frames and resolution H × W as input, the clip is first divided into M × N patches
Revised: Take a RGB video clip V i ∈ R T × 3 × H × W with T frames and resolution H × W as input, the clip is first divided into M × N patches

Original: p ∈ R M × N × 3 × P × P with size of P × P , where N = HW/P 2 .
Revised: p ∈ R M × N × 3 × P × P with size of P × P , where N = HW/P 2 .

Original: Next, patches p are linearly embed as a token sequence z ∈ R MN
Revised: Next, patches p are linearly embed as a token sequence z ∈ R MN

Original: × D with D dimension.
Revised: × D with D dimension.

Original: Then, the learned temporal embeddings
Revised: Then, the learned temporal embeddings

Original: E s ∈ R N × D and spatial positional embeddings E t ∈ R N × D are added to each input token.
Revised: E s ∈ R N × D and spatial positional embeddings E t ∈ R N × D are added to each input token.

Original: Besides, a learnable CLS token is concatenated at the beginning of the token sequence.
Revised: Besides, a learnable CLS token is concatenated at the beginning of the token sequence.

Original: Finally, these token sequences are fed into Timesformer and output the CLS token of the last block, which is further projected to a d dimension embedding by a linear layer to form the final clip representation v i ∈ R d .
Revised: Finally, these token sequences are fed into Timesformer and output the CLS token of the last block, which is further projected to a d dimension embedding by a linear layer to form the final clip representation v i ∈ R d .

Original: Text Encoder.
Revised: Text Encoder.

Original: The text encoder is built upon DistillBERT [33], which has 40% less parameters than
Revised: The text encoder is built upon DistillBERT [33], which has 40% less parameters than

Original: BERT while also preserves over 95% performance, thus is efficient.
Revised: BERT while also preserves over 95% performance, thus is efficient.

Original: Taking a sentence T i as input, first tokenize it as a sequence of tokens and feed it into DistillBERT.
Revised: Taking a sentence T i as input, first tokenize it as a sequence of tokens and feed it into DistillBERT.

Original: Similar to the video encoder, the
Revised: Similar to the video encoder, the

Original: CLS token of DistillBERT’s output is projected as t i ∈ R d for the final text representation.
Revised: CLS token of DistillBERT’s output is projected as t i ∈ R d for the final text representation.

Original: D.2 Pretraining objective: EgoNCE
Revised: D.2 Pretraining objective: EgoNCE

Original: To supplement the Eq. (2) and Eq. (3), we first formulate the complete form of InfoNCE
Revised: To supplement the Eq. 2 and Eq. 3, we first formulate the complete form of InfoNCE:

Original: [Equation]
Revised: [Equation]

Original: and our EgoNCE extends the above as Eq.5 via two sampling strategies:
Revised: and our EgoNCE extends the above as Eq. 5 via two sampling strategies:

Original: [Equation]
Revised: [Equation]

Original: For positive sampling (the numerator term), we pre-extract the nouns and verbs for each narration T i before pretraining and define two word vectors w ni ∈ { 0 , 1 } K 1 and w vi ∈ { 0 , 1 } K 2 to encode the appearing nouns and verbs in sentence, where K 1 and K 2 denote the number of nouns and verbs in
Revised: For positive sampling (the numerator term), we pre-extract the nouns and verbs for each narration T i before pretraining and define two word vectors w ni ∈ { 0 , 1 } K 1 and w vi ∈ { 0 , 1 } K 2 to encode the appearing nouns and verbs in sentence, where K 1 and K 2 denote the number of nouns and verbs in

Original: EgoClip (Refer to Sec B narration analysis).
Revised: EgoClip (Refer to Sec B narration analysis).

Original: During pretraining, for another instance j within batch, we calculate the s ij = ( w ni ) T w nj · ( w vi ) T w vj , if s ij > 0 , we regard instance j is one of the positive sample j ∈ P i of instance i .
Revised: During pretraining, for another instance j within batch, we calculate the s ij = ( w ni ) T w nj · ( w vi ) T w vj , if s ij > 0 , we regard instance j is one of the positive sample j ∈ P i of instance i .

Original: Notably, the positive sampling space P would cover (cid:101) B when working with the negative sampling strategy.
Revised: Notably, the positive sampling space P would cover (cid:101) B when working with the negative sampling strategy.

Original: For negative sampling (the denominator term), each time we sample an instance i , we sample an instance i ′
Revised: For negative sampling (the denominator term), each time we sample an instance i , we sample an instance i ′

Original: ∈ V i in the same video and close in time (less than 1 min) to generate the negative sample i ′
Revised: ∈ V i in the same video and close in time (less than 1 min) to generate the negative sample i ′

Original: ∈ N ( i ) of instance i .
Revised: ∈ N ( i ) of instance i .

Original: Notably, in this way, the actual instance within the batch | (cid:101) B| = 2 N will be double the batch size |B| =
Revised: Notably, in this way, the actual instance within the batch | (cid:101) B| = 2 N will be double the batch size |B| =

Original: N .
Revised: N .

Original: In practice, we have to halve the batch size due to GPU memory limitations.
Revised: In practice, we have to halve the batch size due to GPU memory limitations.

Original: Under halving the batch size, random sampling doesn’t help in our method, which can be concluded by comparing baseline InfoNCE and variants (d) in Tab.
Revised: Under halving the batch size, random sampling doesn’t help in our method, which can be concluded by comparing baseline InfoNCE and variants (d) in Tab.

Original: 3 of the main body, where the batch size of the latter is half of the former.
Revised: 3 of the main body, where the batch size of the latter is half of the former.

Original: Despite this, our proposed sampling strategy (f) can successfully improve the pretraining effect beyond baseline.
Revised: Despite this, our proposed sampling strategy (f) can successfully improve the pretraining effect beyond baseline.

Original: In contrast to the conventional negative sampling from the same video [37, 52], we specifically design our temporally adjacent negative sampling strategy to focus on the frequent appearance changes in egocentric videos, which has not been explored in previous approaches.
Revised: In contrast to the conventional negative sampling from the same video [37, 52], we specifically design our temporally adjacent negative sampling strategy to focus on the frequent appearance changes in egocentric videos, which has not been explored in previous approaches.

Original: E Additional experimental details and results E.1 Implementation details
Revised: E Additional experimental details and results E.1 Implementation details

Original: Following the settings of official Frozen [3], the video encoder is initialized with ViT [53] weights trained on ImageNet-21K with sequence dimension D = 768 .
Revised: Following the settings of official Frozen [3], the video encoder is initialized with ViT [53] weights trained on ImageNet-21K with sequence dimension D = 768 .

Original: The text encoder is based on huggingface’s distilbert-base-uncased .
Revised: The text encoder is based on huggingface’s distilbert-base-uncased .

Original: The dimension of common feature space is set as 256 , and the temperature parameter is set to 0 .
Revised: The dimension of common feature space is set as 256 , and the temperature parameter is set to 0 .

Original: During pretraining, each video is resized to 224 × 224 as input with sample frames number 4 and batch size 512 .
Revised: During pretraining, each video is resized to 224 × 224 as input with sample frames number 4 and batch size 512 .

Original: We use the Adam optimizer with a learning rate of 3 × 10 − 5 with a total epoch of 10 .
Revised: We use the Adam optimizer with a learning rate of 3 × 10 − 5 with a total epoch of 10 .

Original: When transferring to downstream tasks, we select the checkpoints with the best score on EgoMCQ benchmark i.e. average accuracy of inter-video and intra-video settings by default.
Revised: When transferring to downstream tasks, we select the checkpoints with the best score on EgoMCQ benchmark i.e. average accuracy of inter-video and intra-video settings by default.

Original: E.2 Downstream settings
Revised: E.2 Downstream settings

Original: We present the setting details of the downstream tasks we evaluated.
Revised: We present the setting details of the downstream tasks we evaluated.

Original: For a fair comparison, for VLPs variants pretrained on different datasets, we use the same settings on downstream tasks, such as the fine-tuning objective.
Revised: For a fair comparison, for VLPs variants pretrained on different datasets, we use the same settings on downstream tasks, such as the fine-tuning objective.

Original: EPIC-KITCHENS-100 text-video retrieval.
Revised: EPIC-KITCHENS-100 Multi-Instance Retrieval.

Original: In this task, after we finalize video-text pretraining, we continue to fine-tune the VLP model and keep most settings of pretraining (e.g., input resolution, learning rate).
Revised: In this task, after we finalize video-text pretraining, we continue to fine-tune the VLP model and keep most settings of pretraining (e.g., input resolution, learning rate).

Original: Notably, we set the training epoch as 100 and replace the training objective as Multiinstance Maxmargin loss in Eq.6, which is same as the baseline method JPoSE [43].
Revised: Notably, we set the training epoch as 100 and replace the training objective as Multi-instance Maxmargin loss in Eq. 6, which is same as the baseline method JPoSE [43].

Original: The reason for this is that in this task a narration may be jointly associated with multiple clips, so multi-instance learning mechanism can better handle such a situation.
Revised: The reason for this is that in this task a narration may be jointly associated with multiple clips, so multi-instance learning mechanism can better handle such a situation.

Original: And this dataset also provides the action label to calculate the correlation c ij between two clip-text pairs ( i, j ) , which supports the implementation of Multi-instance Maxmargin loss.
Revised: And this dataset also provides the action label to calculate the correlation c ij between two clip-text pairs ( i, j ) , which supports the implementation of Multi-instance Maxmargin loss.

Original: [Equation]
Revised: [Equation]

Original: where Ω = { ( i, j, k ) | j ∈ i + , k ∈
Revised: where Ω = { ( i, j, k ) | j ∈ i + , k ∈

Original: i − } is a triple, which indicates a positive instance j and a negative instance k for i .
Revised: i − } is a triple, which indicates a positive instance j and a negative instance k for i .

Original: In our setting, we define the positive set as i + =
Revised: In our setting, we define the positive set as i + =

Original: { j | c ij > 0 .
Revised: { j | c ij > 0 .

Original: 1 } and the negative as the remains sample within batch.
Revised: 1 } and the negative as the remains sample within batch.

Original: The γ is a margin factor and we set it as 0 . 2 .
Revised: The γ is a margin factor and we set it as 0 . 2 .

Original: Charades-Ego action recognition.
Revised: Charades-Ego Action Recognition.

Original: In this task, the textual categories are short phrases like “Holding some clothes”.
Revised: In this task, the textual categories are short phrases like “Holding some clothes”.

Original: Thus, we regard this task as a kind of video-text retrieval by leveraging the text representation and using the InfoNCE as fine-tuning objective.
Revised: Thus, we regard this task as a kind of video-text retrieval by leveraging the text representation and using the InfoNCE as fine-tuning objective.

Original: We set the epoch number as 10 and keep other parameters unchanged.
Revised: We set the epoch number asand keep other parameters unchanged.

Original:  Natural Language Query This task is a kind of video-text localization and is hard to perform end-to-end training (since a clip might long to 1200 seconds).
Revised: Ego4D Natural Language Query This task is a kind of video-text localization and is hard to perform end-to-end training (since a clip might long to 1200 seconds).

Original: The baseline method [45] takesdim SlowFast features ( 1 . 87 fps, with Kinetics 400 pretrained) and 768 dim BERT features as input.
Revised: The baseline method [45] takesdim SlowFast features ( 1 . 87 fps, with Kinetics 400 pretrained) and 768 dim BERT features as input.

Original: Therefore, we propose to replace the baseline input features as features of pretrained VLP video and text encoders to evaluate the pretraining effectiveness.
Revised: Therefore, we propose to replace the baseline input features as features of pretrained VLP video and text encoders to evaluate the pretraining effectiveness.

Original: We extract the features with the same fps 1 .and sampling frame number 4 .
Revised: We extract the features with the same fps 1 .and sampling frame number 4 .

Original: In fine-tuning stage, we keep the default setting of [45].
Revised: In fine-tuning stage, we keep the default setting of [45].

Original:  Moment Query This task is a video-only task: temporal action localization.
Revised: Ego4D Moment Query This task is a video-only task: temporal action localization.

Original: Similar toNatural
Revised: Similar to

Original:  Language Query task, we replace the input Slowfast features of baseline VSGN [49] with VLP video features for evaluation.
Revised: Natural Language Query task, we replace the input Slowfast features of baseline VSGN [49] with VLP video features for evaluation.

Original: The extraction details are the same as Natural Language Query.
Revised: The extraction details are the same as Natural Language Query.

Original:  Object State Change Classification This is an action classification task, we sample each clip with 16 frames as input and use the cross-entropy as fine-tuning objective.
Revised: Ego4D Object State Change Classification This is an action classification task, we sample each clip with 16 frames as input and use the cross-entropy as fine-tuning objective.

Original: The epoch is set as 10 .
Revised: The epoch is set as 10 .

Original: E.3 VLP Evaluation on EgoMCQ
Revised: E.3 VLP Evaluation on EgoMCQ

Original: In Tab. 10, we display EgoMCQ evaluation result of Frozen pretrained on different video-text datasets.
Revised: In Tab. 10, we display EgoMCQ evaluation result of Frozen pretrained on different video-text datasets.

Original: [Table]
Revised: [Table]

Original: Random 
Revised: Random 20 .20 .

Original: [Table] EPIC-KITCHENS
Revised: EPIC-KITCHENS28 .22 .

Original: [Table] [Table] HowTo100M [Table]
Revised: HowTo100M 31 .21 .

Original: [Table]

Original: CC3M+WebVid-

Original: [Table]
Revised: [Table]

Original: As shown, pretraining with EPIC-KITCHENS-100 dataset (1st-person view, 67 . 2 K pairs) reach comparable performance with HowTo100M pretraining (3rd-person view, 136 M noisy pairs), which demonstrates the major domain gaps.
Revised: As shown, pretraining with EPIC-KITCHENS-100 dataset (1st-person view, 67 . 2 K pairs) reach comparable performance with HowTo100M pretraining (3rd-person view, 136 M noisy pairs), which demonstrates the major domain gaps.

Original: Besides, Frozen with CC3M+WebVid-2M pretraining reach significant improvement on the intra-video setting, but minor in inter-video.
Revised: Besides, Frozen with CC3M+WebVid-2M pretraining reach significant improvement on the intra-video setting, but minor in inter-video.

Original: We speculate this due to
Revised: We speculate this due to

Original: CC3M+WebVid-2M dataset covering a wide range of appearance information but still less exploration in the fine-grained action e.g. human-object interaction.
Revised: CC3M+WebVid-2M dataset covering a wide range of appearance information but still less exploration in the fine-grained action e.g. human-object interaction.

Original: E.4 Training Curves of EPIC-KITCHENS-100 text-video retrieval
Revised: E.4 Training Curves of EPIC-KITCHENS-100 video-text retrieval

Original: [Figure]
Revised: [Figure]

Original: In Fig
Revised: In Fig.

Original: [Table]
Revised: [Table]

Original: we display training curves of EPIC-KITCHENS-100 text-video retrieval under different video-text pretraining, which also includes a baseline without video-text pre-training.
Revised: we display training curves of EPIC-KITCHENS-100 video-text retrieval under different video-text pretraining, which also includes a baseline without video-text pre-training.

Original: We can found that: Variants with video-text pretraining have a faster rise in performance.
Revised: We can found that: Variants with video-text pretraining have a faster rise in performance.

Original: Except for HowTo100M, which is similar to variant without video-text pretraining.
Revised: Except for HowTo100M, which is similar to variant without video-text pretraining.

Original: Especially with EgoClip for egocentric pretraining, the VLP model achieves nearly convergent performance with only a small number of epochs (less than 20 ).
Revised: Especially with EgoClip for egocentric pretraining, the VLP model achieves nearly convergent performance with only a small number of epochs (less than 20 ).

Original: With EgoNCE as pretraining objective, this positive effect is further enhanced.
Revised: With EgoNCE as pretraining objective, this positive effect is further enhanced.

Original: E.5 Results on test set of Moment Query
Revised: E.5 Results on test set of Natural Language Query

Revised: In Tab.

Revised: [Table]

Revised: we found the similar conclusions in test set of Natural Language Query, pretraining with

Revised: EgoClip and EgoNCE reach the optimum performance

Original:  We further display the test set results of Moment Query in Tab. 11, pretraining with EgoClip and
Revised: [Table] E.6 Results on test set of Moment Query We further display the test set results of Moment Query in Tab. 12, pretraining with EgoClip and

Original: EgoNCE reach the best performance, 3 . 78% on R @1 and 4 .
Revised: EgoNCE reach the best performance, 3 . 78% on R @1 and 4 .

Original: 65% on Avg mAP over the baseline.
Revised: 65% on Avg mAP over the baseline.

Original: Methods
Revised: Methods

Original: [Table]

Original: Pre

Original: [Table]
Revised: [Table]

Original: Features
Revised: Features

Original: [Table]
Revised: [Table]

Original: E.6 Results on test set of Natural Language Query

Original: In Tab.

Original: [Table]

Original: we found the similar conclusions in test set of Natural Language Query, pretraining with

Original: EgoClip and EgoNCE reach the optimum performance.

Original: Methods

Original: [Table]

Original: E.7 Visualization
Revised: E.7 Visualization

Original: To intuitively understand the effect of egocentric pre-training, in Fig 13, we compare the EPICKITCHENS-100 text-video retrieval results between our pre-training (EgoClip w/ EgoNCE) and
Revised: To intuitively understand the effect of egocentric pre-training, in Fig. 13, we compare the EPICKITCHENS-100 video-text retrieval results between our pre-training (EgoClip w/ EgoNCE) and

Original: CC3M+WebVid-2M pre
Revised: CC3M+WebVid-2M pre

Original: -training, both fine-tuning with 16 frames.
Revised: -training, both fine-tuning with 16 frames.

Original: The numbers after each narration represent the correlation scores between the query and the retrieval result, with 1 being the best.
Revised: The numbers after each narration represent the correlation scores between the query and the retrieval result, with 1 being the best.

Original: [Figure]
Revised: [Figure]