Original: Searching for Better Spatio-temporal Alignment in Few-Shot Action Recognition
Revised: Searching for Better Spatio-temporal Alignment in Few-Shot Action Recognition

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Spatio-Temporal feature matching and alignment are essential for few-shot actionrecognition as they determine the coherence and effectiveness of the temporalpatterns.
Revised: Spatio-Temporal feature matching and alignment are essential for few-shot actionrecognition as they determine the coherence and effectiveness of the temporalpatterns.

Original: Nevertheless, this process could be not reliable, especially when dealingwith complex video scenarios.
Revised: Nevertheless, this process could be not reliable, especially when dealingwith complex video scenarios.

Original: In this paper, we propose to improve the performanceof matching and alignment from the end-to-end design of models.
Revised: In this paper, we propose to improve the performanceof matching and alignment from the end-to-end design of models.

Original: Our solutioncomes at two-folds.
Revised: Our solutioncomes at two-folds.

Original: First, we encourage to enhance the extracted Spatio-Temporalrepresentations from few-shot videos in the perspective of architectures.
Revised: First, we encourage to enhance the extracted Spatio-Temporalrepresentations from few-shot videos in the perspective of architectures.

Original: With thisaim, we propose a specialized transformer search method for videos, thus the spatialand temporal attention can be well-organized and optimized for stronger featurerepresentations.
Revised: With thisaim, we propose a specialized transformer search method for videos, thus the spatialand temporal attention can be well-organized and optimized for stronger featurerepresentations.

Original: Second, we also design an efficient non-parametric temporalprototype alignment strategy to better handle the high variability of motion.
Revised: Second, we also design an efficient non-parametric spatio-temporalprototype alignment strategy to better handle the high variability of motion.

Original: In particular, a query-specific class prototype will be generated for each query sampleand category, which can better match query sequences against all support sequences.
Revised: Inparticular, a query-specific class prototype will be generated for each query sampleand category, which can better match query sequences against all support sequences.

Original: By doing so, our method SST enjoys significant superiority over the benchmark
Revised: By doing so, our method SST enjoys significant superiority over the benchmark

Original: UCF101 and HMDB51 datasets.
Revised: UCF101 and HMDB51 datasets.

Original: For example, with no pretraining, our methodachieves 17.1% Top-1 accuracy improvement than the baseline TRX on UCF1015-way 1-shot setting but with only 3x fewer FLOPs.
Revised: For example, with no pretraining, our methodachieves 17.1% Top-1 accuracy improvement than the baseline TRX on UCF1015-way 1-shot setting but with only 3x fewer FLOPs.

Original:  Introduction
Revised:  Introduction

Original: One of the most important tasks in video understanding is to understand human actions, which isone of the representative tasks for video understanding [45].
Revised: One of the most important tasks in video understanding is to understand human actions, which isone of the representative tasks for video understanding [39].

Original: In recent years, video understandingand action recognition have been witnessing remarkable progress with the emergence of high-qualitylarge-scale video datasets [5, 15, 9].
Revised: In recent years, video understandingand action recognition have been witnessing remarkable progress with the emergence of high-qualitylarge-scale video datasets [5, 14, 9].

Original: However, this success relies quite heavily on a large amountof manually labeled samples.
Revised: However, this success relies quite heavily on a large amountof manually labeled samples.

Original: The dataset annotation process is laborious, cumbersome and timeconsuming and restricts practical algorithm applications.
Revised: The dataset annotation process is laborious, cumbersome and timeconsuming and restricts practical algorithm applications.

Original: Therefore, how to classify unseen actionclasses with extremely few annotated samples gives rise to the investigation of few-shot (FS) actionrecognition [43, 4].
Revised: Therefore, how to classify unseen actionclasses with extremely few annotated samples gives rise to the investigation of few-shot (FS) actionrecognition [38, 4].

Original: Compared with the image task, introducing temporal dimension makes the video task more complicated.
Revised: Compared with the image task, introducing temporal dimension makes the video task more complicated.

Original: For example, actions in videos are often performed at various speeds and occur at differenttime points.
Revised: For example, actions in videos are often performed at various speeds and occur at differenttime points.

Original: Further, action recognition usually needs to integrate multiple distinctive sub-actioninformation to generate corresponding temporal representations, which are used for subsequentspatio-temporal feature matching.
Revised: Further, action recognition usually needs to integrate multiple distinctive sub-actioninformation to generate corresponding temporal representations, which are used for subsequentspatio-temporal feature matching.

Original: To address the few-shot action classification problem, mostexisting studies [43, 4, 3, 35] use metric-based meta-learning framework to implement similaritycomparison between query (test) and support (reference) videos.
Revised: To address the few-shot action classification problem, mostexisting studies [38, 4, 3, 31] use metric-based meta-learning framework to implement similaritycomparison between query (test) and support (reference) videos.

Original: It first maps the input video to anembedding space via representation learning and then achieves the distance metric for comparingvideo similarities in an episodic task.
Revised: It first maps the input video to anembedding space via representation learning and then achieves the distance metric for comparingvideo similarities in an episodic task.

Original: These approaches follow the episodic training to meta-learna backbone network [43] or temporal relational module [4, 3, 35].
Revised: These approaches follow the episodic training to meta-learna backbone network [38] or temporal relational module [4, 3, 31].

Original: However, we argue that these methods underestimate the importance of spatio-temporal representation, which is crucial for thebasic concept of few-shot action classification.
Revised: However, we argue that these methods underestimate the importance of spatio-temporal representation, which is crucial for thebasic concept of few-shot action classification.

Original: And, there are some concerns about existing temporalrelational strategies [35] on spatial features: (a) insufficient sparse sampling frames may limit thecapacity for long-term temporal models, and (b) increasing the number of sampling frames will leadto the complexity of combination and matching strategy.
Revised: And, there are some concerns about existing temporalrelational strategies [31] on spatial features: (a) insufficient sparse sampling frames may limit thecapacity for long-term temporal models, and (b) increasing the number of sampling frames will leadto the complexity of combination and matching strategy.

Original: Inspired by these works, we present a novel approach to explore the prime architectures as to sequenceorder of temporal attention and spatial attention blocks for the few-shot action recognition.
Revised: Inspired by these works, we present a novel approach to explore the prime architectures as to sequenceorder of temporal attention and spatial attention blocks for the few-shot action recognition.

Original: To boostthe search, we leverage a pre-defined transformer space that considers both temporal attention andspatial attention, and propose a space shrinking strategy with the analysis of collected training lossesduring the search to explore the emphasis of spatial and temporal based information in different stagesfor the superior architectures in terms of accuracy and budgets.
Revised: To boostthe search, we leverage a pre-defined transformer space that considers both temporal attention andspatial attention, and propose a space shrinking strategy with the analysis of collected training lossesduring the search to explore the emphasis of spatial and temporal based information in different stagesfor the superior architectures in terms of accuracy and budgets.

Original: Finally, we propose a nonparametrictemporal prototype alignment method to encourage each query video to match all reference videos inthe support set for long-term temporal models.
Revised: Finally, we propose a nonparametricspatio-temporal alignment method to encourage each query video to match all reference videos in thesupport set for long-term temporal models.

Original: The main contribution of this paper is summarized asfollows:
Revised: The main contribution of this paper is summarized as follows:

Original: • To explore the possibility of the optimal video understanding architecture for few-shot action recognition, we incorporate spatial and temporal attention elements in our transformer space.
Revised: • To explore the possibility of the optimal video understanding architecture for few-shot actionrecognition, we incorporate spatial and temporal attention elements in our transformer space.

Original: It allows the model to spontaneously choose and focus on different types of information at
Revised: It allows the model to spontaneously choose and focus on different types of information at

Original: various stages, which is crucial for obtaining better spatio-temporal representation.
Revised: various stages, which is crucial for obtaining better spatio-temporal representation.

Original: • We introduce a transformer space shrinking strategy to adaptively evolve Transformer space
Revised: • We introduce a transformer space shrinking strategy to adaptively evolve Transformer space

Original: to speed up the supernet training and structure search.
Revised: to speed up the supernet training and structure search.

Original: Based on this strategy, the time and
Revised: Based on this strategy, the time and

Original: cost for video understanding architecture search are drastically decreased.
Revised: cost for video understanding architecture search are drastically decreased.

Original: • We propose a more efficient and general temporal prototype alignment method, whichcan
Revised: • We propose a more efficient and general spatio-temporal prototype alignment method, which

Original:  be conveniently adapted to arbitrary length video matching.
Revised: can be conveniently adapted to arbitrary length video matching.

Original: Our method breaks thelimits of
Revised: Our method breaks the

Original:  frames of standard relational modules for few-shot action recognition and achievesbetter
Revised: limits of frames of standard relational modules for few-shot action recognition and achieves

Original:  capacity for long-term temporal models.
Revised: better capacity for long-term temporal models.

Original:  Related Work
Revised:  Related Work

Original: Few-shot Image Classification .
Revised: Few-shot Image Classification .

Original: Few-shot learning [12, 26, 27] aims to learn novel classes withonly a small number of labeled samples.
Revised: Few-shot learning [12, 23, 24] aims to learn novel classes withonly a small number of labeled samples.

Original: One of the fundamental tasks in FSL is few-shot imageclassification [29, 18], which is preferable to using a combination of representation learning and ametric-based classifier.
Revised: One of the fundamental tasks in FSL is few-shot imageclassification [26, 17], which is preferable to using a combination of representation learning and ametric-based classifier.

Original: The terminology “ n -way k -shot” means that models are used to classify ndistinct classes with only k reference samples per class.
Revised: The terminology “ n -way k -shot” means that models are used to classify ndistinct classes with only k reference samples per class.

Original: This process is also called “episode” [38].
Revised: This process is also called “episode” [34].

Original: Recent works often average the representation of a support class to calculate a more robust classprototype [18, 11].
Revised: Recent works often average the representation of a support class to calculate a more robust classprototype [17, 11].

Original: CrossTransformers [10] employs spatial alignment between query- and supportset images to improve generalization for unseen classes.
Revised: CrossTransformers [10] employs spatial alignment between query- and supportset images to improve generalization for unseen classes.

Original: Matching-based methods [14, 31, 34, 39]tend to learn a deep distance metric or similarity functions to compare examples [7, 8, 20, 36].
Revised: Matching-based methods [13, 28, 30, 35]tend to learn a deep distance metric or similarity functions to compare examples [7, 8, 19, 32].

Original: Few-shot Video Action Recognition .
Revised: Few-shot Video Action Recognition .

Original: Recent attempts [19, 45, 24, 41, 37, 47, 5] focus moreon temporal alignment [46], which aims to align two video sequences in temporal dimension.
Revised: Recent attempts [18, 39, 21, 37, 33, 41, 5] focus more ontemporal alignment [40], which aims to align two video sequences in temporal dimension.

Original: Notably, [4] explicitly learns distance measure and representation independent of non-linear temporalvariations in video sequences.
Revised: Notably,[4] explicitly learns distance measure and representation independent of non-linear temporal variationsin video sequences.

Original: [44] proposes a prototype-centered contrastive learning loss anda hybrid attentive learning mechanism to minimize the negative impacts of outliers and promoteclass separation in few-shot action recognition task.

Original: [23] designs temporal transform module tohandle action misalignment, which consists of two parts: a localization network and a temporalaffine transformation.

Original: [25] formulates the distance between video sequences as transportation cost.
Revised: [22] formulates the distance between video sequences as transportation cost.

Original: Temporal Relational CrossTransformer (TRX) [28] operates over a different number of frames forhigher-ordered temporal model and query-specific class prototypes construction.
Revised: Temporal Relational CrossTransformer (TRX) [25] operates over a different number of frames forhigher-ordered temporal model and query-specific class prototypes construction.

Original: However, thismethod combines a small number of frames that may have limited capacity for modeling long-termdependencies.
Revised: However, thismethod combines a small number of frames that may have limited capacity for modeling long-termdependencies.

Original: In addition, as the number of input frames grows, the number of combinations increasessharply, which makes the processing of temporal relation module more complicated.
Revised: In addition, as the number of input frames grows, the number of combinations increasessharply, which makes the processing of temporal relation module more complicated.

Original: Neural Architecture Search (NAS) .
Revised: Neural Architecture Search (NAS) .

Original: NAS aims to automatically learn/search for better neural network architectures.
Revised: NAS aims to automatically learn/search for better neural network architectures.

Original: Earlier, MetaQNN [1] is proposed to search high-performing CNN architecturesvia reinforcement learning.
Revised: Earlier, MetaQNN [1] is proposed to search high-performing CNN architecturesvia reinforcement learning.

Original: [30] proves that evolutionary algorithms are capable of constructinglarge, accurate networks in a huge search space.
Revised: [27] proves that evolutionary algorithms are capable of constructing large,accurate networks in a huge search space.

Original: Single Path [17] proposes to train each architecturealternately with a two-stage search schema.
Revised: Single Path [16] proposes to train each architecture alternately with a two-stage search schema.

Original: AutoFormer 
Revised: AutoFormer [6] first adopted the one-shot NAS frameworkfor the ViT based architecture search.

Original: QueryVideo c o n v 3 × c o n v 3 × c o n v 3 × c o n v 3 ×
Revised: QueryVideo c o n v 3 × c o n v 3 × c o n v 3 × c o n v 3 ×

Original: TimeAttn.
Revised: TimeAttn.

Original: H
Revised: H

Original: Time Attn.
Revised: Time Attn.

Original: H
Revised: H

Original: Time Attn.
Revised: Time Attn.

Original: H
Revised: H

Original: Space Attn.
Revised: Space Attn.

Original: H
Revised: H

Original: SpaceAttn.
Revised: SpaceAttn.

Original: H
Revised: H

Original: SpaceAttn.
Revised: SpaceAttn.

Original: H
Revised: H

Original: S D B
Revised: S D B

Original: TimeAttn.
Revised: TimeAttn.

Original: H
Revised: H

Original: Time Attn.
Revised: Time Attn.

Original: H
Revised: H

Original: Time Attn.
Revised: Time Attn.

Original: H
Revised: H

Original: Space Attn.
Revised: Space Attn.

Original: H
Revised: H

Original: SpaceAttn.
Revised: SpaceAttn.

Original: H
Revised: H

Original: SpaceAttn.
Revised: SpaceAttn.

Original: H
Revised: H

Original: TimeAttn.
Revised: TimeAttn.

Original: H
Revised: H

Original: Time Attn.
Revised: Time Attn.

Original: H
Revised: H

Original: Time Attn.
Revised: Time Attn.

Original: H
Revised: H

Original: Space Attn.
Revised: Space Attn.

Original: H
Revised: H

Original: SpaceAttn.
Revised: SpaceAttn.

Original: H
Revised: H

Original: SpaceAttn.
Revised: SpaceAttn.

Original: H
Revised: H

Original: S D B
Revised: S D B

Original: N o r m
Revised: N o r m

Original: SupportVideo
Revised: SupportVideo

Original: Spatio-TemporalAlignment 𝐿𝐿 𝑐𝑐𝑐𝑐 𝑓𝑓 ( 𝑥𝑥 𝑠𝑠 ) 𝑓𝑓 ( 𝑥𝑥 𝑞𝑞 )
Revised: Spatio-TemporalAlignment 𝐿𝐿 𝑐𝑐𝑐𝑐 𝑓𝑓 ( 𝑥𝑥 𝑠𝑠 ) 𝑓𝑓 ( 𝑥𝑥 𝑞𝑞 )

Original: [ 𝑇𝑇 , 𝐶𝐶 , 𝑊𝑊 , 𝐻𝐻 ] [ 𝑇𝑇 , 𝐶𝐶, 𝑊𝑊 / 16, 𝐻𝐻 / 16] [ 𝑇𝑇 , 𝐶𝐶, 𝑊𝑊 / , 𝐻𝐻 / ] [ 𝑇𝑇 , 𝐶𝐶, 𝑊𝑊 / , 𝐻𝐻 / ]
Revised: [ 𝑇𝑇 , 𝐶𝐶 , 𝑊𝑊 , 𝐻𝐻 ] [ 𝑇𝑇 , 𝐶𝐶, 𝑊𝑊 / 16, 𝐻𝐻 / 16] [ 𝑇𝑇 , 𝐶𝐶, 𝑊𝑊 / , 𝐻𝐻 / ] [ 𝑇𝑇 , 𝐶𝐶, 𝑊𝑊 / , 𝐻𝐻 / ]

Original: Stage 1 ( 𝑁𝑁
Revised: Stage 1 ( 𝑁𝑁

Original: Layers)
Revised: Layers)

Original: Stage 2 ( 𝑁𝑁
Revised: Stage 2 ( 𝑁𝑁

Original: Layers)
Revised: Layers)

Original: Stage 3 ( 𝑁𝑁
Revised: Stage 3 ( 𝑁𝑁

Original: Layers)
Revised: Layers)

Original:  Method3.1 Problem Formulation
Revised:  Method3.1 Problem Formulation

Original: The goal of few-shot video action classification is to classify an unlabelled query video into several new classes with only a few labeled video samples, referred to as the ‘support set’.
Revised: The goal of few-shot video action classification is to classify an unlabelled query video into several new classes with only a few labeled video samples, referred to as the ‘support set’.

Original: Thedataset is separated into a training set D
Revised: Thedataset is separated into a training set D

Original: tr = { ( x i , y i ) | y i ∈ C train } , a validation set D val ={ ( x i , y i )
Revised: tr = { ( x i , y i ) | y i ∈ C train } , a validation set D val ={ ( x i , y i )

Original: | y i
Revised: | y i

Original: ∈ C validation } , and a test set D test = { ( x i , y i )
Revised: ∈ C validation } , and a test set D test = { ( x i , y i )

Original: | y i ∈ C test } , where the validation setis split from the training set with 1 / 10 samples, x i is the i -th video sample with class label
Revised: | y i ∈ C test } , where the validation setis split from the training set with 1 / 10 samples, x i is the i -th video sample with class label

Original: y i .
Revised: y i .

Original: Different from the general supervised learning, few-shot training and test categories are disjoint,i.e. C train ∩ C test = ∅ .
Revised: Different from the general supervised learning, few-shot training and test categories are disjoint,i.e. C train ∩ C test = ∅ .

Original: To make training more faithful to the testing method, we follow episodictraining [28][40] to train the model.
Revised: To make training more faithful to the testing method, we follow episodictraining [25][36] to train the model.

Original: In each episodic task, training samples are randomly sampledfrom the training set in a n -way k -shot setting.
Revised: In each episodic task, training samples are randomly sampledfrom the training set in a n -way k -shot setting.

Original: Concretely, a support set S and a query set Q aresampled from D tr as S , Q = { ( x i , y i )
Revised: Concretely, a support set S and a query set Q aresampled from D tr as S , Q = { ( x i , y i )

Original: | y i
Revised: | y i

Original: ∈ C support } , where C support ⊆
Revised: ∈ C support } , where C support ⊆

Original: C train is the n selectedclasses.
Revised: C train is the n selectedclasses.

Original: Compared with few-shot image classification, the introduction of the time dimension invideo classification demands the model to be able to learn temporal patterns with limited sample size.
Revised: Compared with few-shot image classification, the introduction of the time dimension invideo classification demands the model to be able to learn temporal patterns with limited sample size.

Revised: Next, we describe our proposed method for few-shot video action recognition.

Original: 3.2 NAS for Few-shot Video Action Recognition
Revised: 3.2 NAS for Few-shot Video Action Recognition

Original: For a supernet N with weight W and Transformer space A , each subnetwork (subnet) a inheritsits weight w a from W .
Revised: For a supernet N with weight W and Transformer space A , each subnetwork (subnet) a inheritsits weight w a from W .

Original: The optimization and search problem in our supernet training stage can beformulated as:
Revised: The optimization and search problem in our supernet training stage can beformulated as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where W ∗A is a set of trained weights for the whole Transformer space A , L ce and ACC val denotethe cross-entropy loss and the accuracy for each subnet a in the episodic task, respectively.
Revised: where W ∗A is a set of trained weights for the whole Transformer space A , L ce and ACC val denotethe cross-entropy loss and the accuracy for each subnet a in the episodic task, respectively.

Original: We build our overall Transformer space A with 4 .
Revised: We build our overall Transformer space A with 4 .

Original: 74 × 10 18 subnets as in Table 1.
Revised: 74 × 10 18 subnets as in Table 1.

Original: Since the videounderstanding model may need to adjust the layout of temporal and spatial attention in different layersfor better temporal model ability, we design independent Space “SAB” and Time Attention Blocks“TAB” for our Transformer space.
Revised: Since the videounderstanding model may need to adjust the layout of temporal and spatial attention in differentlayers for better temporal model ability, we design independent Space “SAB” and Time Attention Blocks “TAB” for our Transformer space.

Original: Although the video features are transformed by TAB and SAB, theoutput feature dimension is unchanged.

Original: To avoid themodel being overly complex, We incorporatethe dimension downsample block, named “Spatial Downsampling Block (SDB)” in Table 1 as in [16].
Revised: Besides, we incorporate the dimension downsample block,named “Spatial Downsample” in Table 1 as in [15].

Original: The core operation of this block is to use the down sampling in spatial self-attention, which maps aninput tensor of size ( C, W, H ) to an output tensor of size ( C ′ , W/ 2 , H/ 2) with C ′ > C .

Original: Due to thechange in scale, we can easily control the complexity of the model in different stages.

Original: We presentmore details about our Transformer space in section 3.5.
Revised: We present more details about our Transformerspace in section 3.5.

Original: 3.3 Transformer Space Shrinking
Revised: 3.3 Transformer space Shrinking

Original: In the previous subsection, we design a large Transformer space covering a variety of operationsto explore the optimal network structure.
Revised: In the previous subsection, we design a large Transformer space covering a variety of operationsto explore the optimal network structure.

Original: However, this also causes some problems: 1) too large
Revised: However, this also causes some problems: 1) too large

Original: Transformer space increases the difficulty of training and optimization of supernet, 2) redundantoperations in Transformer space increase the difficulty of structure search.
Revised: Transformer space increases the difficulty of training and optimization of supernet, 2) redundantoperations in Transformer space increase the difficulty of structure search.

Original: How to design a varietyof Transformer spaces for different parts is another challenging task.
Revised: How to design a varietyof Transformer spaces for different parts is another challenging task.

Original: To this end, we propose awhile τ ≤ do if τ ≥ T w then the if τ % P = 0 then
Revised: To this end, we propose a

Original: Transformer space evolving strategy.
Revised: Transformer space evolving strategy.

Original: Towards a given Transformer space, the unreasonable spaceparts are automatically shrunk, and the whole space gradually converges to a more compact state.
Revised: Towards a given Transformer space, the unreasonable spaceparts are automatically shrunk, and the whole space gradually converges to a more compact state.

Revised: while τ ≤ T do if τ ≥ T w then if τ % P then

Original: To evaluate each operation in Transformer space, we define score S ( i, j ) as the evaluation metricfor j -th operation in i -th layer.
Revised: To evaluate each operation in Transformer space, we define score S ( i, j ) as the evaluation metricfor j -th operation in i -th layer.

Original: Formally, any subnet a ∈ A can be expressed as the result oflayer by layer operation stacking.
Revised: Formally, any subnet a ∈ A can be expressed as the result oflayer by layer operation stacking.

Original: The subnet a can be represented as a = (cid:83) i (cid:80) j 1 ij o i,j , where1 ij ∈ { 0 , 1 } denotes indicator function, which indicates whether o i,j is selected in a .
Revised: The subnet a can be represented as a = (cid:83) i (cid:80) j 1 ij o i,j , where1 ij ∈ { 0 , 1 } denotes indicator function, which indicates whether o i,j is selected in a .

Original: In each layer ofsupernet, subnet selects only one operation. Thus, for the indicator function 1 ij ∈ { 0 , 1 } in i − thlayer, the sum of the indicator functions  is 1: (cid:80) j 1 ij = 1 .
Revised: The sum of theindication functions of each layer is 1.

Original: The training loss for subnet a can also beexpressed as
Revised: The training loss for subnet a can also be expressed as

Original: [Equation]
Revised: [Equation]

Original: To represent the bi-directional relationship between operation
Revised: To represent the bi-directional relationship between operation

Original: o i,j and subnet a , we denote the subnetwhich includes o i,j by a o i,j .
Revised: o i,j and subnet a , we denote the subnetwhich includes o i,j by a o i,j .

Original: For a subnet a with o i,j , it is with huge variance to directly measurethe contribution of o i,j to the performance of a .
Revised: For a subnet a with o i,j , it is with huge variance to directly measurethe contribution of o i,j to the performance of a .

Original: However, the same operation will appear in manysubnets, and we can transverse the search space and leverage the performance expectation of all thesesubnets a o i,j to evaluate the score of o i,j .
Revised: However, the same operation will appear in manysubnets, and we can transverse the search space and leverage the performance expectation of all thesesubnets a o i,j to evaluate the score of o i,j .

Original: Thus, we can obtain the relationship between score andtraining loss:
Revised: Thus, we can obtain the relationship between score andtraining loss:

Original: [Equation]
Revised: [Equation]

Original: Considering that different operations have distinct effects on the computing budgets of subnets, itis often desirable to limit the computing budgets of subnets in practical scenarios.
Revised: Considering that different operations have distinct effects on the computing budgets of subnets, itis often desirable to limit the computing budgets of subnets in practical scenarios.

Original: However, withthe sufficient and same amount of data, those models with big computing budgets may have moreadvantages in persuing good results.
Revised: However, withthe sufficient and same amount of data, those models with big computing budgets may have moreadvantages in persuing good results.

Original: If left unconstrained, the operation scores can be as
Revised: If left unconstrained, the operation scores can be as

Original: [Equation]
Revised: [Equation]

Original: where B ( a o i,j ) denotes the computing budget of subnet a o i,j .
Revised: where B ( a o i,j ) denotes the computing budget of subnet a o i,j .

Original: To address this problem, we propose amodified approach considering the impact on the computing budget of subnets.
Revised: To address this problem, we propose amodified approach considering the impact on the computing budget of subnets.

Original: We update Eq.
Revised: We update Eq.

Original: (4) as
Revised: (4) as

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where δ ( · ) is a correction term related to the statistical expectation for subnet a o i,j ,
Revised: where δ ( · ) is a correction term related to the statistical expectation for subnet a o i,j ,

Original: B
Revised: B

Original: o i,j denotes thestatistical expectation for subnet
Revised: o i,j denotes thestatistical expectation for subnet

Original: a o i,j .
Revised: a o i,j .

Original: We hope to correct the impact of different operations on thesubnet FLOPs by δ ( · ) , so as to accurately evaluate the relative performance of different opertions by
Revised: We hope to correct the impact of different operations on thesubnet FLOPs by δ ( · ) , so as to accurately evaluate the relative performance of different opertions by

Original: S ( i, j ) .
Revised: S ( i, j ) .

Original: Then, the correction term can be defined as
Revised: Then, the correction term can be defined as

Original: [Equation]
Revised: [Equation]

Original: Those high-scoring (low performance) operations that deviate from the average score by more than acertain threshold will be dropped.
Revised: Those high-scoring (low performance) operations that deviate from the average score by more than acertain threshold will be dropped.

Original: Specifically, we make statistics for all the operations after each P epochs during supernet training.
Revised: Specifically, we make statistics for all the operations after each P epochs during supernet training.

Original: We discard low-performing operations that deviate from the average score by more than a certain
Revised: We discard low-performing operations that deviate from the average score by more than a certainthreshold but not more than K percent of the overall operation.

Revised: In this way, Transformer spaceshrinking can be achieved in supernet training.

Revised: As a result, the following search process can becarried out more efficiently on this shrunk space.

Original: Linear
Revised: Linear

Original: Linear Linear
Revised: Linear Linear

Original: MatMul
Revised: MatMul

Original: Scale & Softmax
Revised: Scale & Softmax

Original: Time Attention MatMul
Revised: Time Attention MatMul

Original: Linear
Revised: Linear

Original: MLP [ 𝑇𝑇𝑤𝑤𝑤 , 𝑐𝑐 ] [ 𝑤𝑤𝑤 , 𝑇𝑇 , 𝑐𝑐 ] [ 𝑤𝑤𝑤 , 𝑇𝑇 , 𝑇𝑇 ] [ 𝑤𝑤𝑤 , 𝑇𝑇 , 𝑐𝑐 ] [ 𝑇𝑇𝑤𝑤𝑤 , 𝑐𝑐 ]
Revised: MLP [ 𝑇𝑇𝑤𝑤𝑤 , 𝑐𝑐 ] [ 𝑤𝑤𝑤 , 𝑇𝑇 , 𝑐𝑐 ] [ 𝑤𝑤𝑤 , 𝑇𝑇 , 𝑇𝑇 ] [ 𝑤𝑤𝑤 , 𝑇𝑇 , 𝑐𝑐 ] [ 𝑇𝑇𝑤𝑤𝑤 , 𝑐𝑐 ]

Original: Linear
Revised: Linear

Original: Linear Linear
Revised: Linear Linear

Original: MatMul
Revised: MatMul

Original: Scale & Softmax
Revised: Scale & Softmax

Original: Space Attention MatMul
Revised: Space Attention MatMul

Original: Linear
Revised: Linear

Original: MLP [ 𝑇𝑇𝑤𝑤𝑤 , 𝑐𝑐 ] [ 𝑇𝑇 , 𝑤𝑤𝑤 , 𝑐𝑐 ] [ 𝑇𝑇 , 𝑤𝑤𝑤 , 𝑤𝑤𝑤 ] [ 𝑇𝑇 , 𝑤𝑤𝑤 , 𝑐𝑐 ] [ 𝑇𝑇𝑤𝑤𝑤 , 𝑐𝑐 ]
Revised: MLP [ 𝑇𝑇𝑤𝑤𝑤 , 𝑐𝑐 ] [ 𝑇𝑇 , 𝑤𝑤𝑤 , 𝑐𝑐 ] [ 𝑇𝑇 , 𝑤𝑤𝑤 , 𝑤𝑤𝑤 ] [ 𝑇𝑇 , 𝑤𝑤𝑤 , 𝑐𝑐 ] [ 𝑇𝑇𝑤𝑤𝑤 , 𝑐𝑐 ]

Original: Heads
Revised: Heads

Original: [Figure]
Revised: [Figure]

Revised: 𝑄𝑄𝑄𝑄𝑆𝑆𝑄𝑄𝑄𝑄𝑣𝑣𝐴𝐴𝑣𝑣𝑆𝑆𝐴𝐴𝑥𝑥 𝑞𝑞 𝑓𝑓 ( 𝑥𝑥 𝑞𝑞 ) 𝑓𝑓 ( 𝑥𝑥 𝑠𝑠 ) 𝑓𝑓 ( 𝑥𝑥 𝑞𝑞 ) ̃𝑓𝑓 ( 𝑥𝑥 𝑠𝑠 ) 𝐴𝐴𝐵𝐵𝐴𝐴𝑙𝑙𝐴𝐴 𝐴𝐴𝐴𝐴𝑇𝑇𝑆𝑆

Original: [Figure]
Revised: [Figure]

Original: threshold but not more than K percent of the overall operation. In this way, Transformer spaceshrinking can be achieved in supernet training.
Revised: ) 𝑆𝑆𝑄𝑄𝑆𝑆𝑆𝑆𝐴𝐴𝑄𝑄𝐴𝐴𝑣𝑣𝐴𝐴𝑣𝑣𝑆𝑆𝐴𝐴𝑥𝑥

Original: As a result, the following search process can becarried out more efficiently on this shrunk space.
Revised: [Figure]

Original: 3.4 Temporal Prototype Alignment
Revised: 3.4 Spatio-temporal Prototype Alignment

Original: Given a query video x q and a support video x s and their embedded features f ( x q ) , f ( x s ) ∈ R T ×C o ,a commonly used similarity measurement method is
Revised: Given a query video x q and a support video x s and their embedded features f ( x q ) , f ( x s ) ∈ R T ×C o ,a commonly used similarity measurement method is

Original: [Equation]
Revised: [Equation]

Original: (8) where d ( x q , x s ) is the similarity distance between video x q and x s , T and C o are the temporal lengthand feature dimension, respectively.
Revised: where d ( x q , x s ) is the similarity distance between video x q and x s , T and C o are the temporal lengthand feature dimension, respectively.

Original: The feature spacing between two videos is the mean value of thedifference in temporal and channel dimensions.
Revised: The feature spacing between two videos is the mean value of thedifference in temporal and channel dimensions.

Original: However, this direct measurement has disadvantagesin few-shot video action recognition: it is complicated to match changeable actions performed atvarious speeds directly and occurred at different time points in extremely few videos.
Revised: However, this direct measurement has disadvantagesin few-shot video action recognition: it is complicated to match changeable actions performed atvarious speeds directly and occurred at different time points in extremely few videos.

Original: To this end, wepropose a nonparametric temporal feature prototype alignment method to better deal with featurematching in few-shot video action recognition.
Revised: To this end,we propose a nonparametric spatio-temporal feature prototype alignment method to better deal withfeature matching in few-shot video action recognition.

Original: Specifically,
Revised: Specifically,

Original: [Equation]
Revised: [Equation]

Original: where M ( x q , x s ) denotes a frame-level attention map with a dimension of T × T .
Revised: where M ( x q , x s ) denotes a frame-level attention map with a dimension of T × T .

Original: This provides thepossibility for us to calculate the query-specific prototype w.r.t the query x q , such as
Revised: This provides thepossibility for us to calculate the query-specific prototype w.r.t the query x q , such as

Original: [Equation]
Revised: [Equation]

Original: ) where ˜ f ( x s ) is a temporal aligned support sample representation, as shown in Figure 3.
Revised: where ˜ f ( x s ) is a temporal aligned support sample representation, as shown in Figure 3.

Original: The updatedfeature distance between the aligned support feature and query feature can be calculated as
Revised: The updatedfeature distance between the aligned support feature and query feature can be calculated as

Original: [Equation]
Revised: [Equation]

Original: Then, this measurement method can be easily applied to the support set S c of class c .
Revised: Then, this measurement method can be easily applied to the support set S c of class c .

Original: i.e.
Revised: i.e.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where x p denotes the support example in support set S c .
Revised: where x p denotes the support example in support set S c .

Original: During training or inference, the query videox q is assigned the class of the closest query-specific prototype.
Revised: During training or inference, the query videox q is assigned the class of the closest query-specific prototype.

Original: This is an efficient nonparametrictemporal prototype alignment method that can easily be applied to two spatio-temporal featuresimilarity measurements.
Revised: This is an efficient nonparametricspatio-temporal alignment method that can easily be applied to two spatio-temporal feature similaritymeasurements.

Original: Generating an aligned query-specific prototype for every support video canobtain more reliable prediction results in the few-shot video action classification.
Revised: Generating an aligned query-specific prototype for every support video can obtainmore reliable prediction results in the few-shot video action classification.

Original: 3.5 Transformer Space Design
Revised: 3.5 Transformer Space Design

Original: To find the optimal combination layout for SAB and TAB, we construct a transformer space for the
Revised: To find the optimal combination layout for SAB and TAB, we construct a transformer space for the

Original: FS video action classification.
Revised: FS video action classification.

Original: Let T × C × W × H be the dimension of the input frames of our [Table] supernet.
Revised: Let T × C × W × H be the dimension of the input frames of oursupernet.

Original: We extract multi-scale features hierarchy in every stage with a scaling step of 2 from theinput frames.
Revised: We extract multi-scale features hierarchy in every stage with a scaling step of 2 from theinput frames.

Original: These intermediate embeddings have a spatial stride of 16 , 32 , 64 w.r.t the input. 
Revised: These intermediate embeddings have a spatial stride of 16 , 32 , 64 w.r.t the input. We

Original: Wedo not reduce the time dimension to generate a feature representation with the same length as theinput sequence.
Revised: [Table] do not reduce the time dimension to generate a feature representation with the same length as theinput sequence.

Original: For each layer in our SST model, we search over the operation type and the numberof heads in SAB and TAB.
Revised: For each layer in our SST model, we search over the operation type and the numberof heads in SAB and TAB.

Original: The model can not only choose what kind of attention to do in differentlayers, but also obtain the most appropriate number of self-attention heads.
Revised: The model can not only choose what kind of attention to do in differentlayers, but also obtain the most appropriate number of self-attention heads.

Original: Diagrams of our TAB and SAB are illustrated in Figure 2.
Revised: Diagrams of our TAB and SAB are illustrated in Figure 2.

Original: The input dimensions are both Twh × c ,where the w , h , c denote the width, height, and channel of intermediate embeddings.
Revised: The input dimensions are both Twh × c ,where the w , h , c denote the width, height, and channel of intermediate embeddings.

Original: In the timeattention block, a temporal self-attention mask with dimension wh × T × T is computed to capturetemporal dependencies across frames.
Revised: In the timeattention block, a temporal self-attention mask with dimension wh × T × T is computed to capturetemporal dependencies across frames.

Original: Similarly, spatial attention with dimension T × wh × wh canbe obtained in the space attention block.
Revised: Similarly, spatial attention with dimension T × wh × wh canbe obtained in the space attention block.

Original: Thus, these two modules have different functions in theprocess of video understanding.
Revised: Thus, these two modules have different functions in theprocess of video understanding.

Original: To facilitate the architecture search, these two modules share thesame input and output dimensions.
Revised: To facilitate the architecture search, these two modules share thesame input and output dimensions.

Original: Table 1 provides a basic overview of the Transformer space.
Revised: Table 1 provides a basic overview of the Transformer space.

Original:  Experiments
Revised:  Experiments

Original: 4.1 Datasets and experimental setups
Revised: 4.1 Datasets and experimental setups

Original: Datasets.
Revised: Datasets.

Original: Our proposed method is evaluated on two popular benchmarks: UCF101 [32] and
Revised: Our proposed method is evaluated on two popular benchmarks: UCF101 [29] and

Original: HMDB51 [22], using split from [28].
Revised: HMDB51 [20], using split from [25].

Original: UCF101 [32] is a human action videos dataset from Youtube,which contains 13320 video clips and 101 action categories.
Revised: UCF101 [29] is a human action videos dataset from Youtube,which contains 13320 video clips and 101 action categories.

Original: We used the split with 70/10/21 actioncategories in training/validation/testing.
Revised: We used the split with 70/10/21 actioncategories in training/validation/testing.

Original: HMDB

Revised: HMDB

Original: [22] contains 6849 video clips divided into 51classes.
Revised: [20] contains 6849 video clips divided into 51classes.

Original: A split with 31/10/10 classes is used for HMDB51 [22].
Revised: A split with 31/10/10 classes is used for HMDB51 [20].

Original: The standard 5-way 1-shot and5-way 5-shot is applied to evaluate the performance of our method.
Revised: The standard 5-way 1-shot and5-way 5-shot is applied to evaluate the performance of our method.

Original: Implementation details.
Revised: Implementation details.

Original: For each n -way k -shot episodic task, we randomly sample n class witheach class containing k examples as the support set.
Revised: For each n -way k -shot episodic task, we randomly sample n class witheach class containing k examples as the support set.

Original: And we randomly sample one sample for eachclass in n classes as the query set.
Revised: And we randomly sample one sample for eachclass in n classes as the query set.

Original: Thus each episodic task contains n × ( k +1) samples.
Revised: Thus each episodic task contains n × ( k +1) samples.

Original: For videopreprocessing, the sparse sampling strategy is used to fetch T frames for each video.
Revised: For videopreprocessing, the sparse sampling strategy is used to fetch T frames for each video.

Original: During training,we resize sampled frames to 256 × 256 and then randomly crop a 224 × 224 region as input.
Revised: During training,we resize sampled frames to 256 × 256 and then randomly crop a 224 × 224 region as input.

Original: Duringtesting, random crop is replaced by center crop.
Revised: Duringtesting, random crop is replaced by center crop.

Original: For supernet, we construct the base Transformerspace according to Table 1.
Revised: For supernet, we construct the base Transformer space according to Table 1.

Original: The stage and block setting (the numbers of MHSA and FC layers) isinspired by [2, 16].
Revised: The stage and blocksetting (the numbers of MHSA and FC layers) is inspired by [2, 15].

Original: Follow LeViT [16], the depth of each stage is set to the same.
Revised: Follow LeViT [15], the depthof each stage is set to the same.

Original: And the headnumber h in the SAB and TAB is chosen within the set { 6 , 12 , 16 } to encourage each block to selectits optimal head numbers freely.
Revised: And the head number h in the SAB and TAB is chosen within theset { 6 , 12 , 16 } to encourage each block to select its optimal head numbers freely.

Original: During training, the AdamW optimization method is used to trainthe supernet from scratch.
Revised: During training,the AdamW optimization method is used to train the supernet from scratch.

Original: All experiments are implemented with 8 Nvidia 1080Ti GPUs.
Revised: All experiments areimplemented with 8 Nvidia 1080Ti GPUs.

Original: Furthermore, it should be mentioned that existing FS action classification all tend to rely on thepre-trained weights (e.g., pre-trained on ImageNet) to initialize the model.
Revised: Furthermore, it should be mentioned that existing FS action classification all tend to rely on thepre-trained weights (e.g., pre-trained on ImageNet) to initialize the model.

Original: However, this pre-trainingmay violate the basic assumption of few-shot learning that the query classes cannot be seen duringmeta-training [46].
Revised: However, this pre-trainingmay violate the basic assumption of few-shot learning that the query classes cannot be seen duringmeta-training [40].

Original: We find that ImageNet contains very high-related classes to those query classesduring meta-testing, such as the “Guitar” class in ImageNet vs. “PlayingGuitar” class in UCF101.
Revised: We find that ImageNet contains very high-related classes to those query classesduring meta-testing, such as the “Guitar” class in ImageNet vs. “PlayingGuitar” class in UCF101.

Original: This makes ImageNet pre-training unreasonable and problematic.
Revised: This makes ImageNet pre-training unreasonable and problematic.

Original: Thus, a large number of modeloptimization in this paper is based on randomly initialized weights to ensure the fairness of thecomparison results.
Revised: Thus, all model optimization inthis paper is based on randomly initialized weights to ensure the fairness of the comparison results.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 2 Comparison with state-of-the-arts
Revised: 4.2 Comparison with state-of-the-arts

Original: We compare our method with existing few-shot action classification methods such as ARN [40],
Revised: We compare our method with existing few-shot action classification methods such as ARN [36],

Original: TRX [28] and existing video understanding methods such as TimeSformer [2].
Revised: TRX [25] and existing video understanding methods such as TimeSformer [2].

Original: Table 2 shows thecomparison of our method to state-of-the-art in 5-way 1-shot and 5-way 5-shot problems on HMDB51.
Revised: Table 2 shows the

Original: As in Table 2, our searched models outperforms ARN [40], TRX [28] and TimeSformer [2] by a largemargin on the two protocols.
Revised: As in Table 2, our searched models outperforms ARN [36], TRX [25] and TimeSformer [2] by alarge margin on the two protocols.

Original: Meanwhile, SST models have fewer parameters and FLOPs comparedwith previous methods.
Revised: Meanwhile, SST models have fewer parameters and FLOPscompared with previous methods.

Revised: It is well known that processing more dense frames often meansgreater computing budgets.

Original: Even if our SST processes dense 12 frames, it also has a good advantage over FLOPs.
Revised: However, even if our SST processes dense 12 frames, it also has a goodadvantage over FLOPs.

Original: From these comparison results, we can also see that the results of 5-shot setting are generallyhigher than that of 1-shot.
Revised: From these comparison results, we can also see that the results of 5-shotsetting are generally higher than that of 1-shot.

Original: This is because more available video examples can help the model matchthe action pattern more accurately in the inference process.
Revised: This is because more available video examples canhelp the model match the action pattern more accurately in the inference process.

Original: The results on UCF101 dataset are listed in Table 3.
Revised: The results on UCF101 dataset are listed in Table 3.

Original: On UCF101 dataset, our method also outperformsthe previous methods by a large margin.
Revised: On UCF101 dataset, our method also outperformsthe previous methods by a large margin.

Original: In particular, when the number of input frames is 8, SSTimproves TRX by 2.7% in 5-shot case and by surprisingly 17.1% in 1-shot case.
Revised: In particular, when the number of input frames is 8, SSTimproves TRX by 2.7% in 5-shot case and by surprisingly 17.1% in 1-shot case.

Original: Note that theperformance of SST with 4 input frames can exceed that of TRX with 8 frames, which demonstratesthe superiority of our method in dealing with extremely few videos.
Revised: Note that theperformance of SST with 4 input frames can exceed that of TRX with 8 frames, which demonstratesthe superiority of our method in dealing with extremely few videos.

Original: 4.3 Ablation Study
Revised: 4.3 Ablation Study

Original: We systematically analyze the impact of different modules in our method.
Revised: Here, we systematically analyze the impact of different modules in our method.

Original: Specifically, we first showthe effectiveness of each component under the 8 input frames and 5-way 5-shot setting in Table 6.
Revised: Specifically, we firstshow the effectiveness of each component under the 8 input frames and 5-way 5-shot setting in Table 6.

Original: Wefollow the TimeSformer[2] and LeVit [16] to construct a baseline model, in which the intermediatemodules are set manually in a way similar to TimeSformer[2].
Revised: We follow the TimeSformer[2] and LeVit [15] to construct a baseline model, in which the intermediatemodules are set manually in a way similar to TimeSformer[2].

Original: These results show that the NAScan improve the baseline method with an improvement of 2.07%, which indicates the searchedmodel has a stronger learning ability for spatio-temporal features.
Revised: These results show that the NAScan improve the baseline method with an improvement of 2.07%, which indicates the searchedmodel has a stronger learning ability of spatio-temporal features.

Original: The proposed Transformer Space
Revised: The proposed Transformer Space

Original: Shrinking and Temporal Alignment contribute to an improvement of 1.37% and 2.56%, respectively.
Revised: Shrinking and Temporal Alignment contribute to an improvement of 1.37% and 2.56%, respectively.

Original: All components collaborate and complement each other, reaching 60.42% finally.
Revised: All components collaborate and complement each other, reaching 60.42% finally.

Original: In what follows,we conduct several ablation experiments to evaluate the effectiveness of each module on HMDB51carefully.
Revised: In what follows,we conduct several ablation experiments to evaluate the effectiveness of each module on HMDB51carefully.

Original: 4.3.1 The impact of NAS
Revised: 4.3.1 The impact of NAS

Original: To study the impact of NAS, we conduct several experiments on hand-engineered and searchedarchitectures on HMDB51.
Revised: To study the impact of NAS, we conduct several experiments on hand-engineered and searchedarchitectures.

Original: In “Plain model”, we follow the TimeSformer[2] and LeVit [16] toconstruct a plain model, which has a structure similar to our Transformer space, but the intermediatemodules are set manually.
Revised: In “Plain model”, we follow the TimeSformer[2] and LeVit [15] to construct a plainmodel, which has a structure similar to our Transformer space, but the intermediate modules are setmanually.

Original: According to Figure 1, we manually set the alternating TAB and SABmodules in each stage to build the final model.
Revised: According to Figure 1, we manually set the alternating TAB and SAB modules in each stageto build the final model.

Original: Then, the same hyperparameters are used to retrainthese hand-engineered and searched models.
Revised: Then, the same hyperparameters are used to retrain these hand-engineeredand searched models.

Original: As shown in Table 4, the searched architectures surpassthe non-searched version by a considerable gap while using fewer parameters and FLOPs.
Revised: As shown in Table 4, the searched architectures surpass the non-searchedversion by a considerable gap while using fewer parameters and FLOPs.

Original: Theseresults demonstrate the effectiveness of exploiting Transformer architecture in our method.
Revised: These results demonstratethe effectiveness of exploiting Transformer architecture in our method.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original:  Transformer 
Revised: NAS Transformer Space Shrinking

Original: [Table]
Revised: [Table]

Original: 4.3.2 The impact of Transformer Space Shrinking
Revised: 4.3.2 The impact of Transformer Space Shrinking

Original: To better study the impact of Transformer space shrinking
Revised: To better study the impact of Transformer space shrinking

Original: [Table]
Revised: [Table]

Original: we conduct two comparative experimentson HMDB51.
Revised: we conduct two comparative experimentson HMDB51.

Original: We train a supernet on 1-shot task with 4 input frames.
Revised: We train a supernet on 1-shot task with 4 input frames.

Original: The average metrics ofrandomly sampled subnetworks are regarded as the metrics of supernet.
Revised: The average metrics ofrandomly sampled subnetworks are regarded as the metrics of supernet.

Original: The top-1 accuracy and losson test sets are reported in Figure 4 and Figure 5, respectively.
Revised: The top-1 accuracy and losson test sets are reported in Figure 4 and Figure 5, respectively.

Original: From figures, we can observe that theintroduction of the Transformer space shrinking strategy not only speeds up the convergence of thesupernet, but also greatly improves the performance.
Revised: From figures, we can observe that theintroduction of the Transformer space shrinking strategy not only speeds up the convergence of thesupernet, but also greatly improves the performance.

Original: Theoretically, the search space is wider, searchtime is longer, and training speed is slower, the stronger the need for space shrinking.
Revised: Theoretically, the search space is wider, searchtime is longer, and training speed is slower, the stronger the need for space shrinking.

Original: 4.3.3 The impact of Temporal Alignment (TA)
Revised: 4.3.3 The impact of Temporal Alignment (TA)

Original: Here we show the effectiveness of the proposed  Temporal Alignment strategy.
Revised: Here we show the effectiveness of the proposed Spatio-temporal Temporal Alignment strategy.

Original: For “w/o TA”, wedirectly measure the distance between temporal features with the dimension of T × C o in the methodof Eq.
Revised: For“w/o Temporal Alignment”, we directly measure the distance between temporal features with thedimension of T × C o in the method of Eq.

Original: For our complete method, support video features are firstly aligned to generated queryspecific prototype and then compared with query features.
Revised: For our complete method, support video features arefirst aligned to generated query-specific prototype and then compared with query features.

Original: From Table 5, we find the TA strategyimproves the baseline on various settings.

Original: For example, when using 12 frames as input, our methodoutperforms the non-aligned version by 5.8% on 5-way 1-shot setting. In addition, it should be noted that our proposed alignment approach differs from TRX [28] in severalfolds. First, TRX [28] constructs exhaustive pairs/triplets of sampled spatial features  to generate thevideo representations. However, our searched model naturally generates frame-level spatio-temporalrepresentation, which already contains rich temporal information, so the laborious combinationoperation is omitted.
Revised: For ourcomplete method, support video features are first aligned to generate a query-specific prototype andthen compared with query features.

Original: Second, proposed prototype alignment is a nonparametric and concise solutionsuitable for few-shot video understanding, which can generate high-quality frame-level attentionmap without relying on complex subnetworks. TRX [28] relies on a CrossTransformer subnetworkto perform alignment operations. Third, proposed method greatly alleviates the difficulty of longvideo modeling, and makes it possible to align and match two complex actions. The pairs/tripletsbased methods have limited video understanding ability for complex actions (some complex humanactions cannot be represented by only 2 or 3 sparse sampled frames).
Revised: From Table 5, we find the spatio-temporal alignment improvesthe baseline on various settings, which indicates the alignment strategy helps the FS video actionsimilarity measurement.

Original: A simple example is that [Table] [Table] pairs/triplets-based approaches cannot distinguish whether a person hits the desk 3 times or 4 times.
Revised: For example, when using 12 frames as input, our method outperforms thenon-aligned version by 5.8% on 5-way 1-shot setting.

Original: Because the maximum number of sampling frames is only [Table] and it is difficult for the model tounderstand this repetitive action beyond triplet.
Revised: Furthermore, the proposed alignment operationis very efficient and does not introduce too much computational cost.

Original: 4.3.4 The impact of Pre-training
Revised: 4.3.4 The impact of Pre-training

Original: To prove the effectiveness of our method using pretrained weight [Table] we  pretrain model on UCF101dataset and then finetune in a few-shot learning way on HMDB51 dataset.
Revised: Moreover, to prove the effectiveness of our method using pretrained weight, we also pretrain modelon UCF101 dataset and then finetune in a few-shot learning way on HMDB51 dataset.

Original: The results are listedin Table 7.
Revised: The resultsare listed in Table 7.

Original: All the methods are finetuned on 5-way 5-shot tasks with 8 input frames.
Revised: All the methods are finetuned on 5-way 5-shot tasks with 8 input frames.

Original: As shown intable, the performance of three methods can be greatly improved after pretraining on UCF101 dataset.
Revised: As shown in table, the performance of three methods can be greatly improved after pretraining on UCF101 dataset.

Original: Among them, our method still performs better than the previous two methods. 
Revised: Among them, our method still performs better than the previous two methods. [Figure] [Figure] [Figure]

Revised: HMDB

Original: In addition, we alsoprovide the results of TRX using Imagenet pretrained weight.
Revised: ImageNet [Figure] UCF101-pretrain Non-pretrain

Revised: TimeSformer [2] - 48.

Original:  It can be seen from the results that theperformance of TRX can also be greatly improved after pretraining with large-scale ImageNet dataset.
Revised: TRX [25] 75. [Table] In addition, we also provide the results of TRX using Imagenet pretrained weight. It can be seenfrom the results that the performance of TRX can also be greatly improved after pretraining withlarge-scale ImageNet dataset.

Original: However, as stated in section 4.1, there is a possibility that relevant information may be leaked tothe unseen query class when few-shot learning methods use pretrained weights.
Revised: However, as stated in section 4.1, there is a possibility that relevantinformation may be leaked to the unseen query class when few-shot learning methods use pretrainedweights.

Original: By pretrained witha larger dataset, there may be more severe class information leakage.
Revised: By pretrained with a larger dataset, there may be more severe class information leakage.

Original: In this work, whether using UCF pretrained or randomly initialized weights, our method significantly outperforms the previousmethods.
Revised: Inthis work, whether using UCF pretrained or randomly initialized weights, our method significantlyoutperforms the previous methods.

Original: The experiment in this subsection demonstrates that our proposed method is still better thanthe previous methods after pretraining to ensure that the model has better feature extraction ability.
Revised: The experiment in this subsection demonstrates that our proposedmethod is still better than the previous methods after pretraining to ensure that the model has betterfeature extraction ability.

Original: We also report the performance of our model after pretraining on large-scale Kinetics-400 [5] dataset.
Revised: In addition to ablation experiments, we visualize some searched architecture and statistics underdifferent settings.

Original: Similarly, all the methods are pre-trained with 8 input frames.
Revised: In Figure 6, two searched structures are shown in detail, in which “5s-4f” denotesthe searched model under 4 input frames and 5-shot setting, “TAB-H12” denotes a TAB block with12 self-attention heads, and so on for the other abbreviations. And Figure 7 shows some statisticalresults, in which the horizontal axis represents the three stages of four models, and the vertical axisrepresents the number of spatial and temporal blocks (positive to the SAB, negative to the TAB). It can be seen from the results that although there are some differences in the layout of TAB and

Original: Through pretraining on Kineticsdataset, all models are compared under a fair pretraining condition.
Revised: SAB modules in the first two stages, the proportion of these two modules is relatively balanced.

Original: After that, the few-shot datasets
Revised: Surprisingly, we notice that the layout of the third stage is relatively large, which is quite differentfrom the design in current hand-crafted models. Obviously, our method can search for the appropriateand specific architectures for different input dimensions and different tasks. For example, for thesame 5-shot task, the models select 7 TAB modules in the third stage when 4 frames are fed in.

Original: UCF101 and HMDB are used for fine-tuning and testing, and the results are shown in Table 8.
Revised: Butwhen 8 frames are used as input, the models only select 3 TAB modules in the third stage.

Original: Fromthese results, we can see that our proposed model still has the best performance after pretraining onlarge-scale dataset.
Revised: We arguethat the demands for the temporal modeling ability of the model are related to the time dimensionsampling rate of the input data.

Original: With only 1/3 FLOPs, our method can surpass TRX by 2.5% and 10.8% on the UCF101 and HMDB51 dataset, respectively.
Revised: When the model can obtain only fewer input frames, more temporalattention blocks will be placed in the third stage to improve the action modeling capability.

Original:  Conclusion
Revised:  Conclusion

Original: This paper introduced a FS action recognition method for a neural architecture search method with a Transformer space shrinking strategy and temporal prototype alignment.
Revised: This paper introduced a FS action recognition method for a neural architecture search method witha Transformer space shrinking strategy and spatio-temporal prototype alignment.

Original: The discovered architecturescan learn steadier frame-level representations from FS video samples.
Revised: The discoveredarchitectures can learn steadier frame-level representations from FS video samples.

Original: The spatio-temporal featuresare then utilized in prototype alignment operation for learning similarity between query and supportactions.
Revised: The spatiotemporal features are then utilized in prototype alignment operation for learning similarity betweenquery and support actions.

Original: The query-specific class prototypes for each query sequence are generated to deal withvarious actions at different speeds and temporal offsets.
Revised: The query-specific class prototypes for each query sequence are generatedto deal with various actions at different speeds and temporal offsets.

Original: An extensive set of ablations shows howoptimal video Transformer architectures are searched, the benefits of Transformer space shrinking,and the importance of the temporal prototype alignment.
Revised: An extensive set of ablationsshows how optimal video Transformer architectures are searched, the benefits of Transformer spaceshrinking, and the importance of the spatio-temporal prototype alignment.

Original: In future studies, we will delve into therelationship between pretraining and few-shot learning.
Revised: In future studies, we willdelve into the relationship between pretraining and few-shot learning.

Original: Avoiding the possible leakage of categoryinformation while using pretrained weights is still a matter of research.
Revised: Avoiding the possible leakageof category information while using pretrained weights is still a matter of research.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .•

Original: • Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data areproprietary.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes](c)
Revised: [Yes](c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes](b) Did you include complete proofs of all theoretical results?
Revised: [Yes](b) Did you include complete proofs of all theoretical results?

Original: [Yes]3.
Revised: [Yes]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A](b)

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes](c)
Revised: [Yes](c)

Original: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No]
Revised: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A](b)

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]