Rethinking Resolution in the Context of Efficient Video Recognition
Anonymous Author(s)
AffiliationAddress
Abstract
In this paper, we empirically study how to make the most of low-resolution framesfor efficient video recognition.
Existing methods mainly focus on developingcompact networks or alleviating temporal redundancy of video inputs to increaseefficiency, whereas compressing frame resolution has rarely been considered apromising solution.
A major concern with using low-resolution frames is its poorrecognition accuracy.
We thus start by analyzing the underlying causes of performance degradation on low-resolution frames.
Our key finding is that, degradationin performance is not necessarily a result of degradation in input quality, but rathermismatch between network architecture and input scale.
Motivated by the successof knowledge distillation (KD), we propose to bridge the gap between networkand input size via cross-resolution KD (ResKD).
Our work shows that ResKD is asimple but effective method to boost recognition accuracy on low-resolution frames.
Without bells and whistles, ResKD considerably surpasses all competitive methodsin terms of efficiency and accuracy on four large-scale benchmark datasets, i.e.,
ActivityNet, FCVID, Mini-Kinetics, Something-Something V2.
In addition, weextensively demonstrate its effectiveness over state-of-the-art architectures, i.e.,3D-CNNs and Video Transformers, and scalability towards super low-resolutionframes.
The results suggest ResKD can serve as a general inference accelerationmethod for state-of-the-art video recognition.
Introduction
In recent years, video recognition has gained popularity due to the phenomenal growth of socialmedia platforms and the deluge of online videos.
With the introduction of various temporal fusionmodules [25, 27, 30, 31], 3D-CNNs [41, 4, 9], and video transformers [34, 2, 35, 29], remarkableprogress has been made towards increasing recognition accuracy.
Nevertheless, the impressiveperformance comes at the cost of high computational budgets and latency.
State-of-the-art modelsusually require thousands of GFLOPs of computation to recognize a ten-second video [29], whichlimits their deployment to resource-restricted devices or latency-constrained applications.
To overcome this limitation, extensive studies [42, 8, 23, 32, 46, 28] have been conducted to increase inference speed on videos.
A large group of works [50, 24, 28] concentrates on developing efficient frame sampling policy networks to select the most salient frames for recognition in an adaptive manner.
Essentially, the idea is to alleviate the inherent temporal redundancy of videos by compressing temporal resolution of the input.
However, another crucial factor affecting efficiency, namely spatial resolution of the input, is largely under-explored.
Recent work [45] points out that video contents are spatially redundant as well, providing thetheoretical feasibility of compressing spatial resolution to achieve higher efficiency.
Nevertheless, inpractice, the performance of video recognition models usually degrades drastically on low-resolutioninputs.
For this reason, frame down-sampling is rarely considered a practical way to increaseefficiency.
Even in AR-Net [32], low resolutions are only used when processing less important frames.
Whereas in this paper, we challenge this common belief by showing that low-resolution frames arefar from being fully exploited.
In particular, our empirical study reveals the following facts:
(i) Low-resolution frames are not necessarily low-quality frames.
An intuitive explanation for theperformance degradation is the poor quality of input, given the fact that considerable information islost in the down-sampling process, e.g., the high-frequency signals.
However, through controlledexperiments, we find that information loss only has mild impact on accuracy.
This observation
indicates that substantial redundancy actually exists in high-resolution (e.g., 224 × 224 ) frames, e.g.,task-irrelevant information, while low resolutions mitigate such redundancy and potentially provide
higher efficiency.
(ii) Mismatch between network and input scale leads to sub-optimal performance at lowresolutions.
Instead of information loss, the experimental results suggest that scale variation is
the main cause of performance degradation.
In an investigation into how scale variation influencesperformance, we notice that network architecture plays an important role.
Specifically, we find
current models initially designed for high-resolution input can be poor learners at low resolutions.
Our observations align with the conclusions drawn in recent study of network design [40], which
posit network architecture should coordinate with resolution to
make
the most of
input
(iii) Performance gap between resolutions can be effectively closed via cross-resolution knowl-
edge distillation (ResKD).
Driven by simplicity and generality considerations,
we ask
whether
ResKD could serve as a general solution to alleviate network and input mismatch and unlock the
potential of low resolution frames for efficient video recognition.
The idea of ResKD is quite straight-
forward: a teacher taking high-resolution frames as input is leveraged to guide
the learning of thestudent on low-resolution frames.
We benchmark ResKD in terms of effectiveness, generality, andscalability on Kinetics-400 and four popular efficient video recognition datasets, i.e., ActivityNet,
FCVID, Mini-Kinetics, and Something Something V2.
Extensive experiments show that
ResKDoutperforms all competitive efficient video recognition methods by large margins.
Especially, ResKDuses 4x fewer FLOPs than recently proposed AdaFocus V2 [46], while achieving 1.4% higher top-1accuracy on Mini-Kinetics.
In addition, we demonstrate the effectiveness of ResKD is agnostic tonetwork architectures.
For instance, applying ResKD on Video Swin [29] brings 1000+ savings in
GFLOPs without sacrificing accuracy.
Related Work
Video recognition.
The ability to relate frames at different temporal locations is of key importancefor video understanding.
This motivates recent research on equipping deep neural networks withtemporal modeling capability, which has spawned a cascade of 2D-CNNs, 3D-CNNs, and transformers for video recognition.
In modeling temporal information with 2D-CNNs, a common practice isto extract frame-wise features and aggregate them along the temporal dimension, with the help ofrecurrent neural networks [6, 26] or carefully designed temporal fusion modules [25, 27, 30, 31].
Onthe other hand, 3D-CNNs, such as C3D [41], I3D [4], and X3D [9], directly extract spatial-temporalrepresentations from videos by densely applying 3D convolutions on sampled clips.
As for videotransformers, based on ViT, several variants [34, 2, 35, 29] are proposed for video recognition byextending attention mechanism to spatial-temporal patches.
Although aforementioned models demonstrate impressive recognition accuracy, high computationalcost and latency (especially for 3D-CNNs and transformers) restrict their applicability in real-lifesettings.
Efficiency thus comes into the picture.
Current works mainly tackle the problem throughdesigning compact networks [43, 51, 53, 42, 8, 23] or dynamically allocating computation to the most informative part of a video [50, 24, 32, 46, 39, 28].
But our approach is orthogonal to thesemethods, in the sense that it is agnostic to network design and robust to video contents.
Reducing video redundancy.
Videos are inherently redundant medium in carrying information,e.g., there exists considerable still scenes and task-irrelevant clips in untrimmed videos.
Based onthis observation, various methods have been proposed to mitigate temporal redundancy in videoinput, such as adaptive frame selection [24, 47, 50, 11, 28], early exiting without watching thefull video [7, 48, 13], reusing feature from previous frames [33], processing less important frameswith lower resolution/precision [32, 39].
Besides, video redundancy is not limited to the temporaldimension.
Recently, AdaFocus [45] points out significant spatial redundancy exists in the formof uninformative regions in each frame, which can be removed by adaptive patch selection.
Ourmethod shares a similar idea as AdaFocus on reducing spatial redundancy, but studies the problemfrom the perspective of frame resolution.
Moreover, compared with popular adaptive methods[50, 11, 32, 39, 33, 13, 45, 28, 46], our method is easier to implement as it does not contain additionalmodules or require complex training strategies.
Knowledge distillation.
Knowledge transfer pioneered by [1, 17] to distill structure knowledgefrom teacher network to student network.
For the action recognition task, depth [12], temporalinformation [38] and multiple teacher ensemble [47] are explored as structure knowledge to distill.
High-resolution teacher is first explored in [36] to guide the recognition of extreme low-resolutionvideo.
However, the reason behind the inferior performance of low-resolution video recognition isunclear.
Besides, the naive logit knowledge distillation lacks the consideration of spatial information.
We analyse the underlying causes of performance degradation on low-resolution frames and align thespatial knowledge in the cross-resolution distillation scenario.
Empirical Studies on Low-resolution Video Recognition
Compressing spatial resolution of videos has rarely been considered a promising solution to increasingthe efficiency of video recognition.
This is because performance of current models usually degradesdrastically on low-resolution frames, leading to a poor trade-off between efficiency and accuracy.
Inthis section, we are interested in investigating the underlying causes of such performance degradation,in the hope that our findings might shed light on how to make the most of low-resolution framesfor efficient video recognition.
We initiate the exploration from two perspectives: quality of lowresolution frames, and mismatch between network structures and input scales.
Quality of low-resolution frames.
An intuitive explanation to the underperformance is the poorquality of low-resolution frames.
Compared with original high-resolution frames, considerableinformation, which might be crucial for action recognition, is lost in the process of spatial downsampling.
This raises the first question: Do low-resolution frames contain sufficient information foraccurate video recognition?
To answer this question, we quantitatively evaluate the impacts of information loss on recognitionaccuracy.
Specifically, we first obtain frames at different resolutions, 112 × 112 , 96 × 96 , 72 × 72 ,56 × 56 , by resizing original 224 × 224 frames to simulate information loss in the down-samplingprocess.
Then we up-sample these low-resolution frames back to 224 × 224 to exclude the influencebrought by scale variations.
We train and evaluate video recognition models using these up-sampledframes.
In addition, another group of models which is directly trained and tested on low-resolutionframes is included for comparison.
The experimental results are shown in Figure 1 (a).
In contrast tothe common belief, we find that information loss is not a main cause of performance degradation.
Even for super low resolutions, e.g., 56 × 56 , the preserved information can well support highlyefficient video recognition with decent performance.
This study suggests that high-resolution framesmay not be necessary for efficient video recognition, in the sense that they contain highly redundantor task-irrelevant information. [Figure]
Mismatch between network and input scale.
In Figure 1 (a), we notice that input scale is a criticalfactor affecting performance.
But in what way do scale variations influence performance is not clear.
Inspired by prior work [40] which highlights the importance of coordinating and balancing networkarchitecture with input scales, we hypothesize that networks designed with high-resolution (e.g.,224 × 224 ) inputs in mind may not be good learners on low-resolution frames.
That is, mismatchbetween network and input scale degrades the performance.
To verify this hypothesis, we adjust the strides of convolution layers at each stages in ResNet-50from (1, 2, 2, 2) to (1, 1, 2, 2), and compare the performance of original ResNet-50 (R50-H) withmodified ResNet-50 (R50-S) at different resolutions.
As shown in Figure 1 (b), with this simpleadjustment, R50-S significantly outperforms R50-H at low resolutions but underperforms R50-H athigh resolutions, which is consistent with our speculation.
Based on this observation, a promising solution to address such performance degradation is to designtailored network architectures for low-resolution video recognition.
However, the potential numberof resolutions and network variations render this solution costly and inflexible in practice.
Here,we aim at a general solution for this problem with knowledge distillation.
We are interested ininvestigating whether knowledge distillation is sufficient to bridge the performance gap throughtransferring knowledge or feature representations from a teacher network with high-resolution inputsto a student network with low-resolution inputs.
Cross-Resolution Knowledge Distillation
In this section, we give an overview of Cross-Resolution Knowledge Distillation (ResKD) andcomprehensively study its properties in terms of: (i) Effectiveness: How does ResKD compare toother efficient video recognition methods?
(ii) Generality: Does ResKD work well for SOTA modelswith dense sampling?
(iii) Scalability: Is ResKD scalable with varying resolutions?
(iv) Benefits:
How does ResKD help low-resolution video recognition?
4.1 The ResKD Framework
As shown in Figure 3, the ResKD framework exhibits no difference from traditional knowledgedistillation [17] except that the teacher and the student take input of different resolutions duringtraining.
The idea is quite straightforward: We utilize a teacher good at high-resolution videorecognition to guide the learning of student on low-resolution frames.
To do so, we adopt commonly [Figure] used feature knowledge distillation to transfer knowledge across resolutions.
The overall loss functioncan be written as
L = L cls ( y s , y ) + α · L kd , where L kd =
MSE ( f t , Φ( f s ))
L cls is the standard cross-entropy loss between the predictions y s of the student and ground-truthlabels y .
While f t and f s denote features generated by the teacher and student, respectively.
Φ is abilinear interpolation operation to align the spatial dimensions of f s with f t .
MSE () stands for thestandard mean squared error.
4.2 Benchmark Setups
Datasets.
We benchmark ResKD on five commonly used action recognition datasets.
In particular,
ActivityNet-v1.3 [3] and FCVID [18] are used for evaluation on untrimmed videos: (1) ActivityNetv1.3 contains 10,024 training videos and 4,926 validation videos from 200 action classes, with anaverage duration of 117 seconds.
(2) FCVID includes 45,611 training videos and 45,612 validationvideos labeled into 239 classes, with an average length of 167 seconds.
As for trimmed videoevaluation, we use: (3) Kinetics-400 [19] is a large-scale scene-related dataset covering 400 humanaction categories, with at least 400 video clips for each class.
(4) Mini-Kinetics is a subset of
Kinetics-400 introduced by [32, 33].
It includes 121,215 videos for training and 9,867 videos fortesting, coming from 200 action classes.
(5) Something-Something V2 [14] is a temporal-relateddataset which contains 168,913 training videos and 24,777 validation videos over 174 classes.
Evaluation Protocols.
Our method is evaluated in terms of accuracy and efficiency.
For accuracy measurement, we adopt mean average precision (mAP) for multi-label classification on
ActivityNet-v1.3 and FCVID, and top-1 accuracy for multi-class classification on remaining datasets.
For efficiency measurement, we use giga floating point operations (GFLOPs) which is hardwareindependent to represent computational cost.
Since the number of frames used per video varies acrossdifferent methods and datasets, for fair comparison, We report GFLOPs per video in the followingexperiments.
Besides, we measure the practical efficiency with throughput on a single Tesla V
SXM2 GPU.
Implementation Details.
Unless otherwise specified, we uniformly sample 8 frames from eachtrimmed video and 16 frames from each untrimmed video, respectively.
For data pre-processing,following [27, 45, 28], we first apply random scaling to all sampled frames, then augment them with
× 224 random cropping and random flipping in the training stage.
For the input to student, wefurther down-sample the resolution of video frames to 112 × 112 .
During inference, we resize theshort side of all frames to 128 while keeping the aspect ratio, then center-crop them to 112 × 112 .
By
[Table]
default, we adopt ResNet-152 [15] as the teacher network, ResNet-50 [15] as the student network, and112 × 112 as the student input resolution.
If not mentioned, we perform feature KD on the last-layeroutputs of the teacher and the student.
We use the codebase provided by [5] for implementation.
4.3 How does ResKD compare to other efficient video recognition methods?
We start by comparing the efficiency of ResKD with state-of-the-art efficient video recognitionmethods.
As presented in Table 1, ResKD substantially surpasses all alternative methods by achievingsuperior accuracy with 1 . 5 × - 5 .
5 × computation savings on ActivityNet and FCVID.
In particular,
ResKD outperforms AR-Net which also tries to increase efficiency with low resolution frames, by6.2% mAP on ActivityNet, while using half of the GFLOPs.
It is worth noting that ResKD demonstrates even higher efficiency on trimmed videos.
On MiniKinetics, ResKD requires only 40% of computation of the recently proposed method OCSampler,while excelling OCSampler by 1.7% top-1 accuracy.
Remarkable success on short-range videosdemonstrates the robustness of ResKD against temporal redundancy variance in videos, which givesthe probability of combing ResKD with existing temporal-redundancy-removing methods, e.g.,
OCSampler, to achieve higher efficiency.
Additionally, we benchmark ResKD against state-of-the-art efficient CNNs proposed for videorecognition.
TSM-ResNet50 is adopted as the backbone of teacher and student network on Something
Something V2.
As shown in Table 2, without using cumbersome teacher, ResKD-TSM uses 73% lesscomputation (8.7 v.s. 32.7 GFLOPs) but achieves 1.4% higher top-1 accuracy than baseline TSM.
The results verify the effectiveness of ResKD and further confirm the great potential of low-resolutionframes in efficient video recognition.
4.4 Does ResKD work well for SOTA models with dense sampling?
Although numerous efficient video recognition methods have been recently proposed, most of themare limited to efficient video recognition settings, i.e., using light-weight 2D-CNNs as backbonesand sparsely sampled frames as input.
We believe it will make more sense if ResKD could serveas a general acceleration method for state-of-the-art video recognition.
Therefore, we take one stepforward to evaluate the effectiveness of ResKD on 3D-CNNs and transformers, with densely sampledframes.
[Table] SlowOnly [10] and Video Swin [29] are chosen as the representatives of 3D-CNN and transformer forevaluation, respectively.
Following practice in [10, 29], we densely sample 8 × 10 × 3 frames for teston SlowOnly and 32 × 4 × 3 frames for test on Video Swin.
The results are reported on Kinetics-400in Table 3.
One can observe that ResKD largely closes the performance gap between high-resolutionand low-resolution input on both architectures.
Especially, significant savings in computation (up to1218 less GFLOPs per video) is achieved with little sacrifice in accuracy.
[Table]
4.5 Is ResKD scalable with varying resolutions?
In this subsection, we access the scalability of ResKD in terms of accuracy and efficiency by changingthe input resolutions.
Scalability in terms of accuracy.
We change input resolution within {56x56, 72x72, 96x96,112x112, 144x144}, and plot the corresponding mAP v.s. GFLOPs trade-off curves on ActivityNet in
Figure 3.
In comparison with state-of-the-art methods, ResKD consistently achieves better trade-off atvarious computational costs.
Besides, different from other methods whose accuracy drops drasticallywith low computational budgets, ResKD maintains decent performance even at very low resolutions.
Scalability in terms of practical efficiency.
As pointed out in [28], the design of many efficientmethods cannot fully leverage the parallel computing power of current devices.
Consequently, fewer
GFLOPs do not necessarily imply faster inference speed.
To verify that the practical efficiencyof ResKD is also scalable with input resolution, we measure the inference speed of ResKD withdifferent input resolutions under the same hardware setup.
It can be seen from Table 4 that reductionin GFLOPs brings continuous and stable boost in inference speed.
4.6 How does ResKD help low-resolution video recognition?
In the following, we analyse the benefits of ResKD from a qualitative perspective, with a specialfocus on how does it alleviate the mismatch between network architecture and input scale.
[Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Figure]
Shrinking effective receptive field.
We showed that mismatch between network architecture and input size is a maincause of performance degradation on low-resolution recognition.
We also empirically observed that decreasing the downsampling ratio of network or upscaling the intermediate featuremaps can largely alleviate thisproblem.
Essentially, these operations shrink the amplified receptive fields of the network onlow-resolution inputs.
We therefore hypothesize ResKD has similar effects on the student network.
To verify this hypothesis,we compare the effective receptive fields (ERF) of the studentnetwork before and after ResKD.
In Figure 4 (a), one can observe that the ERF contracts significantlyafter ResKD, resembling the original ERF on high-resolution input.
Improving localization.
Another interesting finding is that the model after ResKD demonstratedstronger localization capability on low-resolution inputs.
Following practice in [22], we visualize theactivation-based attention of the student.
From the demos in Figure 4 (b), it can be seen that modelwith ResKD can better focus its attention on objects or actions of interest, while the attention ofmodel without ResKD spreads across larger areas including uninformative backgrounds.
Based onthese observations, we conjecture that ResKD helps low-resolution video recognition by contractingthe ERF to match object scales in low-resolution inputs.
[Table] [Table]
Ablation Study
Design choices of ResKD
We investigate how does the specific design of KD impact the performance of ResKD.
Depending on the type of supervision signals from the teacher, we categorize KDinto (i) Clip-level KD which uses classification score of the whole clip as supervision (ii) Frame-level
KD which uses frame-wise classification score as supervision (iii) Pixel-level KD which is supervisedon features before global average pooling.
Detailed information of these design choices are presentedin Table 5.
Notably, traditional KD,
i
[Table]
e., Clip-level KD, only brings mild gains in cross-resolution
KD settings.
But adding temporal dimension supervision, i.e., Frame-level KD, significantly booststhe effectiveness of KD by more than 3% in mAP.
A possible explanation to this result is that highfrequency supervision provides more hints because frames are noisy in nature.
In addition, the resultsof Pixel-level KD indicates that supervision in spatial dimensions is also crucial for cross-resolutionknowledge transfer.
Model distillation v.s. Resolution distillation To fully exert the potential of knowledge distillation,we employ a teacher model with higher capacity than the student.
One may argue that performancegains brought by ResKD is mainly attributed to knowledge transfer from high-capacity teachernetwork rather than high-resolution input.
To show that knowledge from high-resolution frames is themain factor contributing to the efficiency of ResKD, we conduct an ablation on two settings, namelymodel distillation and resolution distillation.
In model distillation, we use high-capacity network(ResNet-152) as teacher but feed both teacher and student with low-resolution ( 112 × 112 ) framesduring training.
In resolution distillation, teacher network is the same as student (ResNet-50) buttakes high-resolution ( 224 × 224 ) frames as input.
Ablation results in Table 6 verify the effectivenessof resolution distillation.
Conclusion
In this paper, we systematically study how to make the most of low-resolution frames in the contextof efficient video recognition.
We identify that information in high-resolution frames is actuallyredundant, which can be alleviated by simple spatial down-sampling.
However, mismatch betweennetwork architecture and input scale prevents us from fully exploiting the potential of low-resolutionframes.
To address this issue, we leverage cross-resolution knowledge distillation (ResKD) to guidethe learning of student on low-resolution frames with a teacher expert in high-resolution recognition.
Extensive experiments demonstrate that ResKD can serve as an effective, general, and scalableinference acceleration method for state-of-the-art video recognition.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See Section Abstract and Introduction(b)
Did you describe the limitations of your work?
[Yes] Provided in Supplementary Materials
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work
is only for academic research purpose.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] Our code is proprietary, but we will release the code once the paper is accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Subsection Benchmark Setups and Supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Our model is trained on Ubuntu
system, 8 V100 GPUs with 32G.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See the Reference
Part
(b) Did you mention the license of the assets?
[No] The data and models used in our work are publicly released.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] The data and models used in our work are publicly released.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] The data used in our work does not contain
personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
Our work does not involve human subjects.
And the following
items are the same.
(b) Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
By default, ResKD applies feature KD to transfer knowledge across resolutions.
In case of theteacher and the student may have different number of output channels, we provide an alternative KDmethod, namely pixel-level logit KD to replace feature KD. As illustrated in Figure 5, pixel-levellogit KD skips the common pooling operation preceding the classification head to obtain pixel-levelclassification scores, which are then used for knowledge distillation.
Such simple adjustments enableit to combine the advantages of feature KD and logit KD. On the one hand, it preserves spatialinformation for ResKD, which is shown to be beneficial in Section 5 of the main paper. On theother hand, it aligns the channel of features between the teacher and the student, without introducingadditional learnable modules to transform the features [37, 21]. The effectiveness of pixel-level logit
KD is verified with experiments on Video Swin in Section 4.4 of the main paper. A.2 Implementation Details Training Hyper-parameters for ResNet In our implementation, we first pre-train the teachernetwork using an SGD optimizer with cosine learning rate annealing and a Nesterov momentum of0.9 for 50 epochs [15, 27, 32, 28, 45]. The size of the mini-batch is set to 64, while the weight decay
[Figure]
is set to 1e-4.
The training settings of the student are basically the same as the teacher, except thatwe train the student for 100 epochs on FCVID, Mini-Kinetics, Something Something V2, and 200epochs on ActivityNet.
Both the teacher (ResNet-152 or ResNet-50) and the student (ResNet-50) areinitialized with ImageNet pre-trained parameters from the official models provided by PyTorch.
Theweight of knowledge distillation loss is set to α = 100 .
Notably, during knowledge distillation, weunfreeze the batch normalization layer in the teacher network as suggested in [16].
In ablation studies and experiments in Section 3, we use the same set of aforementioned hyperparameters except that we only train the student for 50 epochs.
Training Hyper-parameters for SlowOnly and Video Swin We follow the configurations in [5]to train SlowOnly-teacher and SlowOnly-student.
The weight of knowledge distillation loss is setto α = 100 .
For experiments on Video Swin, we use pre-trained Swin_B provided in the officialimplementation for Video Swin Transformers as the teacher.
{+We train the student Swin_S initializedwith ImageNet pre-trained parameters for 60 epochs, using default configurations in [29]. Pixel-levellogit KD is applied for ResKD between Swin_B and Swin_S, where the weight of KD loss is set toα = 
 and the temperature is set to T = 4 .+}
A.3 Limitations of ResKD
As a spatial-redundancy-removing method, ResKD lacks particular treatment for temporal redundancyof input, making it incapable of fully addressing redundancy issues such as temporal redundancy onuntrimmed videos.
Sparse sampling strategy [44] is employed in most of our experiments to alleviatethis issue.
While another potential solution is to combine ResKD with other temporal-redundancyremoving methods.
For example, one may leverage selected frames from adaptive frame selectionmethods as input to ResKD, so that ResKD can further compress resolution of frames to achievehigher efficiency.
In fact, how to combine ResKD with existing efficient video recognition methodscan be an interesting topic, e.g., could ResKD be applied to the skim network of current adaptivemethods so that the skim network could provide higher-quality features to the policy network inidentifying important frames?
We leave this for future exploration.