Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning
Abstract
Recent incremental learning for action recognition usually stores representative videos to mitigate catastrophic forgetting.
However, only a few bulky videos can be stored due to the limited memory.
To address this problem, we propose FrameMaker , a memory-efficient video class-incremental learning approach that learns to produce a condensed frame for each selected video.
Specifically,
FrameMaker is mainly composed of two crucial components: Frame Condensing and Instance-Specific Prompt .
The former is to reduce the memory cost by preserving only one condensed frame instead of the whole video, while the latter aims to compensate the lost spatio-temporal details in the Frame Condensing stage.
By this means, FrameMaker enables a remarkable reduction in memory but keep enough information that can be applied to following incremental tasks.
Experimental results on multiple challenging benchmarks, i.e ., HMDB51, UCF101 and SomethingSomething V2, demonstrate that FrameMaker can achieve better performance to recent advanced methods while consuming only 20% memory.
Additionally, under the same memory consumption conditions, FrameMaker significantly outperforms existing state-of-the-arts by a convincing margin.
Introduction
Training video action recognition models with all classes in a single fine-tuning stage have been widely studied [6, 13, 30, 34, 37, 51, 52, 62] in recent years.
However, in many realistic scenarios, limited by privacy matters or technologies, the different classes can only be presented in sequence, where the previously trained classes are either unavailable or partially available in limited memory.
Naively training models will overfit currently available data and cause performance deterioration on previously seen classes, which is named catastrophic forgetting [39].
Class-incremental learning is a machine learning paradigm to fight this challenge when fine-tuning a single model in a sequence of independent classes.
A branch of class-incremental learning methods [5, 7, 9, 12, 21, 33, 45, 60] has achieved remarkable performances in the image domain by re-training a portion of past examples.
Meanwhile, some existing video incremental learning methods [41, 54] have demonstrated that storing more past [Figure] [Figure] examples can improve the ability to mitigate forgetting.
Nevertheless, videos are informationredundant and require a high memory load to be saved, hence it is impractical to store a good deal of training videos for each class.
Therefore, herding strategy [45] is adopted to select a small set of representative videos for the exemplar memory [41, 54].
vCLIMB [54] tries to down-sample frames for each video to reduce the memory consumption, and imposes a temporal-consistency regularization for better performance.
Despite the remarkable performance, these methods still demand to conserve multiple ( i.e. , 8-16) frames for each representative video, leading to non-negligible memory overhead, which limits their further real-world applications.
To remedy the above weaknesses, we present FrameMaker, a memory-efficient video classincremental learning approach.
Concretely, the proposed FrameMaker contains two components,
Frame Condensing and Instance-Specific Prompt.
In Frame Condensing, FrameMaker assigns a learnable weight to each frame, and then these weights are adjusted to make the weighted sum frame i.e ., the condensed frame, share the same embedding features with the original video clip.
Obviously, the collapsed temporal dimension and mixed spatial cues by Frame Condensing may degrade the accuracy of action recognition.
To compensate the missing spatio-temporal details, inspired by TaskSpecific Prompt [36], we further propose Instance-Specific Prompt, which learns a set of parameters for each condensed frame to perform fine-grained pixel-wise modifications.
In this way, the memory required to store a single video can be remarkably compressed.
An intuitive comparison is given in
Figure 1(a).
On the premise of comparable performance with TCD [41], FrameMaker significantly reduces memory overhead by 80%.
Moreover, the comparison curve displayed in Figure 1(b) further indicates that FrameMaker achieves a consistent performance improvement over recent TCD with the same memory consumption.
In summary, we make the following contributions: 1) We present a memory-efficient video classincremental learning method FrameMaker, which significantly reduces the memory consumption of example videos by integrating multiple frames into one condensed frame; 2)
For the first time, we show a novel perspective of Prompt, i.e., Instance-Specific Prompt, and demonstrate that it can embed effective spatio-temporal interactions into learned condensed frames for video class-incremental learning; 3) Experimental results reveal that FrameMaker consistently outperforms recent state-of-theart methods with the same memory overhead on three challenging benchmarks (HMDB51, UCF101, and Something-Something V2).
Related Works 2.1 Class-Incremental Learning
The class-incremental learning has been well-studied in image domain [1, 26, 32, 63], existing approaches could be grouped into knowledge distillation-based methods [20], memory-based methods [40, 45, 47], and architecture-based methods.
In video domain , class-incremental learning for action recognition is still an underexplored area.
The existing solutions are designed for better temporal constraints, such as decomposing the spatio-temporal features [65], exploiting time-channel importance maps [41], and applying temporal consistency regularization [54].
Notably, both [41] and
Embedding Feature
Framesample ⋮ ⨂ ⨂ ⨂ ⊕ ⊕
V !"
𝐼 !
#" 𝐼 !
$" 𝐼 !%"
Frame Condensing
Instance-Specific Prompt 𝐼 !"
𝑃 !"
𝐿 &’( 𝐿 )( 𝐿 )& 𝑓( ;Θ " ) 𝑓( ;Θ " ) 𝑓( ;Θ " ) ⨁ : Pixel-wise Addition ⨂ : Multiplication
GroundTruth 𝑤 !"
# 𝑤 !
$# 𝑤 !
# 𝐿 &’&
Condensed Frame
Prompt (𝑃 !"
+𝐼 !" )
[Figure]
Frames h h Classification Score [54] have shown that more stored examples in memory can effectively encourage the performance. However, FrameMaker attempts to minimize the stored frames for each video by a pretty maneuver, which can provide decent performance with slight storage requirements.
2.2 Prompt-Based Learning
Prompting is an emerging transfer learning technique in natural language processing [36], which adapts the language model to downstream tasks by a fixed function.
Since the design of prompting functions is challenging, the learnable prompts [29, 31] are proposed to learn task-specific knowledge parameters for the input.
Recently, in the visual domain, some works have also emerged to adapt the pre-trained models to new tasks by adding prompt tokens [24] or pixels [3] in the data space.
Besides,
DualPrompt [58] and L2P [57] get rid of the rehearsal buffer in continual learning by learning a set of task-specific parameters.
In this paper, we draw inspiration from prompting, and present an
Instance-Specific Prompt to replenish the collapsed spatio-temporal cues for condensed frames.
2.3 Action Recognition
The approaches designed for action recognition task can be grouped into three categories: 2D CNN, 3D CNN and Transformer-based methods.
2D CNN based methods is more efficient [19, 49], while 3D CNN [6, 10, 15, 16, 43, 51–53, 56, 62] achieves better accuracy in action recognition at the cost of more computation cost. With the rapid development of vision transformer [22, 35, 38], researchers also apply it into the video recognition [2, 4, 13, 37], which is proven to be remarkable effective.
In this work, we follow the settings in [41, 65], and employ the Temporal Shift Module (TSM) [34] based on 2D CNN for efficient experiments.
The video summarization and efficient action recognition tasks have provided some methods for keyframe selection and generation, which are related to this work.
The video summarization [55, 64, 67] task selects the most representative frames for efficient human understanding, which has a different paradigm with action recognition task, hence we will mainly discuss efficient The efficient video classification task aims to learn a strategy to improve classification accuracy with fewer discriminative frames as input.
The approaches designed for this task can be grouped into two categories: rule-based and model-based methods.
The former [27, 66] select frames via some pre-defined rules such as motion distribution [66] or channel sampling [25], and the later design a submodel to select or generate key frames [14, 17, 23, 44, 59, 61].
In this paper, we propose model-free frame condensing method for video class-incremental learning, which only learns a few condensing weights and prompting parameters for each instance.
Method
This section presents the general framework of FrameMaker as shown in Figure 2, which mainly consists of two crucial components: Frame Condensing and Instance-Specific Prompt.
Firstly, Frame
Condensing is designed to integrate multiple frames into one frame for efficient storage.
Then we introduce Instance-Specific Prompt to compensate the collapsed temporal information and the mixed spatial knowledge.
Finally, the specific class-incremental training details for subsequent tasks using
Condensed Frames are given.
3.1 Problem formulation
The purpose of video class-incremental learning is to train a model F (; Θ)
=
h ( f (; θ ); ξ ) parameterized by Θ as the tasks {T 1 , T 2 , · · · , T K } arrive in a sequence, where f (; θ ) is the feature extractor and h (; ξ ) is the classification head.
Each task T K has its specific dataset D k
= { ( d ki , y ki ) ,
y ki ∈ L k } whose labels belong to a predefined label set L k and have not appeared in previous tasks.
The memory-based methods [41, 45, 54] have been shown to be effective in preventing catastrophic forgetting.
Specifically, after the incremental training step k , a memory bank M k will be established for the dataset D k in this task to store the representative exemplars or features.
Exemplars stored in previous datasets M 1: k = M 1 (cid:83) M 2 (cid:83) · · · (cid:83) M k will be used in subsequent incremental tasks to mitigate forgetting.
At the incremental step k , the model F (; Θ k ) is trained from F (; Θ k − 1 ) with
D ′ k
= D k (cid:83) M 1:( k − 1) and evaluated on all seen classes.
3.2 FrameMaker
Frame Condensing.
We follow the standard protocol of video class-incremental methods [41, 54], which are based on memory-replay strategy and knowledge distillation.
Differently, we devotes to reducing the number of stored frames for representative videos to achieve memory-efficient video class-incremental learning.
Formally, we first select a subset of video instances V k by herding strategy [45] from D k after the incremental step k .
Given a video V ki ∈ V k , we then sample T frames { I ki 1 , I ki 1 , · · · , I kiT } uniformly and integrate them to condense a more representative one.
To this end, we define learnable weights W ki = { w ki 1 , w ki 2 , · · · , w kiT } for the i -th video instance V ki in exemplar set V k .
And the condensed frame can be calculated as follows:
[Equation]
where C , H and W are the size of channel, height and width of input frames, respectively.
Next, the condensed frame I ki is excepted to empower the same or much similar expressive ability as the original video clip.
Hence, the embedding features of the condensed frame extracted from current model should be consistent with the features of the original video clip:
[Equation]
where f (; θ k ) is the feature extractor of current model F (; Θ k ) .
This consistency regularization can foster the condensing weights to exploit memory-efficient expressions of video clips.
To further improve the adaptability of the condensed frames to the current model, we employ cross entropy loss to supervise the classification confidence from the condensed frame:
[Equation]
The full objective function for condensing weights W ki is given by:
[Equation]
With the methods described above, we can learn an effective representation, i.e. , the condensed frame for each video based on current model.
Instance-Specific Prompt.
However, condensing a video into one frame would inevitably lose the the temporal dynamics and complete spatial information of the original video to a certain extent.
Therefore, an intriguing perspective of prompt, termed as Instance-Specific Prompt, is proposed to replenish the missing spatio-temporal cues for the condensed frame.
Specifically, we first construct a learnable prompt P ki ∈ R C × H × W for each exemplar video V ki , which shares the same spatial resolution as its original clip.
The prompt P ki is then pixel-wise summed with the condensed frame I ki , and learns to pull the embedding feature of the summed frame ( I ki + P ki ) and the video clip V ki together, which is similar to Eq. 2:
[Equation]
Notably, the Eq. 2 for condensing weights is difficult to learn the satisfactory spatio-temporal features due to the collapse of the temporal dimension.
Nevertheless, the Eq. 5 can enrich the representations of condensing frames by introducing more flexible learnable parameters in the input space.
Besides, the Cross-Entropy loss is also utilized to enhance the semantic perception:
[Equation]
Theoretically, the condensing weights W ki and the prompting parameters P ki can be jointly updated by L p =
L pf + L pce .
Interestingly, we observed in our practice that the flexible prompt P ki leads to the condensing weights W ki being under optimized.
Hence, we give the final training objective function for frame condensing as:
[Equation]
where additional L c f and L c ce are added to achieve stronger constraint on W k i for its effective training. α , β , γ and η are the balance weights for each term, and they are empirically set to 1.0 unless otherwise specified.
After the optimization of condensing weights and prompting parameters, the prompt is added directly to the condensed frame and stored together.
And the learned prompts will be frozen when the corresponding task is completed.
Therefore, there is no extra memory cost for prompts.
3.3 Training
It is worth noting that FrameMaker aims to condense the stored frames for representative videos, and other class-incremental learning steps still follow the standard pipeline.
Specifically, when training the incremental step k , we use the dataset D ′ k
= D k (cid:83) M k − 1 to update the model from F (; Θ k − 1 ) , where D k is the dataset of the task k which consists of videos belonging to L k , and M k − 1 is memory bank which contains condensed frames generated by FrameMaker.
The training samples from D k and M k − 1 are alternately fed into the current model according to the proportion of their sample number.
To further prevent the catastrophic forgetting, we also employ the knowledge distillation method proposed in PODNet [12] to transfer the knowledge from the previous model F (; Θ k − 1 ) to current model F (; Θ k ) .
The overall objective function for task k reads:
[Equation]
where L dce and L mce are the Cross Entropy losses for new task data D k and condensed frame exemplars
M k − 1 , respectively.
And L mdist is the knowledge distillation loss [12] for condensed frame exemplars.
Experiments 4.1 Experimental Setup
Datasets and evaluation metrics.
The proposed FrameMaker is evaluated on three standard action recognition datasets, UCF101 [50], HMDB51 [28] and Something-Something V2 [18].
HMDB dataset consists of 6.8K videos belonging to 51 classes from YouTube or other websites.
UCF dataset contains 13.3K videos from 101 classes.
Something-Something V2 is a crowd-sourced dataset that includes 220K videos from 174 classes.
For UCF101, the model is trained on 51 classes first, and the remaining 50 classes are divided into 5, 10 and 25 tasks.
For HMDB51, we train the base model using videos from 26 classes, and the remaining 25 classes are separated into 5 or 25 groups.
For Something-Something V2, we first train 84 classes in the initial stage, and generate the groups of 10 and 5 classes.
To evaluate the performance of class-incremental learning, we infer the test videos from all seen categories after each task, and finally, report the average accuracy of all tasks. Following [41], two [Table] [Table]
[Table]
different metrics are reported, i [Table] e., CNN and NME.
CNN refers to training a fully-connected layer for extracted features, which is a standard classification protocol.
NME is proposed by iCaRL [45] [Table] which assigns the labels for test data by comparing the feature embeddings with the mean-of-exemplars.
Implementation details.
TSM [34] is employed as our backbone, and we follow the data preprocessing procedure of TSM.
For UCF101, we train a ResNet-
TSM
for 50 epochs with a batch size 256 from an initial learning rate 0.04.
For HMDB51 and Something-Something V2, we train a ResNet-50 TSM for 50 epochs with a batch size of 128 from an initial learning rate of 1e-3 and 0.04, respectively.
All used networks are first pre-trained on ImageNet [8] for initialization.
These settings are consistent with TCD [41].
We train all models on eight NVIDIA V100 GPUs and use
PyTorch [42] for all our experiments.
4.2 Comparison with State-of-the-art Results
This section presents the qualitative comparison of our proposed FrameMaker with the existing class incremental learning approaches under multiple challenging settings on three datasets: UCF101 [50],
HMDB51 [28] and Something-Something V2 [18].
For a fair comparison, we use the same exemplar memory size per class, model structure and pre-training initialization as the existing methods [41].
Table 1 summarizes the results on UCF101 and HMDB51, which shows that FrameMaker outperforms other methods consistently under different configurations in terms of both CNN and NME scores.
The average accuracy of our FrameMaker surpasses TCD by around 3.0% on UCF101 and 2.0% on
HMDB51 in terms of CNN, respectively.
These results demonstrate that one condensed frame can also be equipped with an effective spatio-temporal representation.
FrameMaker is compared with recent advanced methods on Something-Something V2 in Table 2.
FrameMaker sets new state-of-the-art performance under multiple challenging settings.
Although
FrameMaker only uses one condensed frame for future replay, it still achieves decent performance on the large-scale motion-sensitive dataset.
We speculate that these gains mainly come from two aspects:
The proposed Instance-Specific Prompt effectively rescues the lost spatio-temporal details, which [Table] [Table] [Table]
[Table] [Table] is also discussed in Section 4.3. (2) Memory-efficient FrameMaker allows us to store more exemplar videos with less memory consumption, which greatly alleviates catastrophic forgetting.
4.3 Ablation Study
In this section, we present ablation studies to analyze the properties and the effectiveness of
FrameMaker.
If not specified, the ablation studies are performed on UCF101 with 10 steps and
HMDB51 with 5 steps.
For comparison with the case of storing all frames, we select 5 exemplar videos for each class.
Frame Condensing and Instance-Specific Prompt.
To show the effectiveness of Frame Condensing and Instance-Specific Prompt, we compare them with the following alternative methods to produce stored frames for future replay: (1) All Frames, which simply saves the whole video.
(2) One Random
Frame, which randomly selects one frame for each exemplar video.
(3) One Average Frame, which averages the input frames for each exemplar video.
(4) One Condensed Frame, which is generated by our Frame Condensing procedure.
The results are reported in Table 3.
From the table, in terms of CNN, we observe that Instance-Specific Prompt yields an improvement of 2.07% and 0.22% with randomly sampled frames, 0.69% and 0.75% with averaged frames on UCF101 and HMDB51, respectively, which implies that prompts can always replenish the ignored spatio-temporal cues in video frames. Further, Frame Condensing with Instance-Specific Prompt leads to at least on par or better performance than preserving all frames while using just one condensed frame under all settings on both datasets.
Loss terms.
To preserve the best spatio-temporal information in the condensed frames, the distillation loss L *f and the cross entropy loss L *ce are both applied for Frame Condensing and Instance-Specific
Prompt.
Table 4 presents the results with Frame Condensing from several different combinations of loss terms.
L cce provides more explicit semantic supervision, making the accuracy to step further.
Similar experiments for Instance-Specific Prompt are shown in Table 5.
One noticeable thing is that once introducing prompt, we only need to calculate the final L pf and L pce to realize the joint update of condensing weights W k i and prompt P k i .
However, we observe that the learned weights, in this case, tend to be average, which indicates that W ki is under-optimized.
We speculate that this is due to the strong plasticity of prompt, which leads to the insufficient optimization of W ki .
To this end, we further employ L cf and L cce for W ki as the guidance.
The improvement in Table 5 and the visualizations in Figure 3 show that W ki receives effective learning. [Table]
The number of frames used for Frame Condensing.
As discussed in Section 3.2, the T frames are uniformly sampled from a representative video for Frame Condensing.
In Table [Table] we show the effect of T in video class-incremental learning.
We observe that the performance increases as T increases for both CNN and NME.
However, the performance is saturated when using 16 frames, which is in line with the conclusion of existing methods [41, 54], i.e., storing more frames in a video does not necessarily deliver performance improvement.
This phenomenon reveals that FrameMaker can empower one Condensed Frame to absorb plenty of spatio-temporal features from original clips.
Different backbones.
To evaluate the applicability of proposed FrameMaker, we replace the backbone with 3D CNN-based method R3D50 [16] and video transformer ViT[11], as shown in Table 7.
In these experiments, we simply replicate the condensed frames multiple times along the the temporal dimension to meet the 3D models and Transformers.
From the results, we can find that our FrameMaker can consistently achieve comparable performance on the three different backbones with only 12% memory cost.
These results demonstrate the better generalization of FrameMaker.
Alternative prompting strategies.
We conduct experiments to explore the types and positions of the prompts in Table 8.
(i)
We compare our Instance-Specific (I.-Spec.)
Prompt (ISP) with the
Task- (T.-Spec.) and Class-Specific (C.-Spec.)
prompts and observe that our ISP can achieve the best performance when operating on both features and frames.
Since the actions in videos have great intra-class variance, simply sharing task and class-specific prompts is insufficient to reserve the important spatia-temporal cues for each instance; (ii) Positions of prompt : For the feature settings, we add the prompt on the features of the 4 stages of the ResNet.
The feature-based prompt is worse than our frame-based prompt, which may be caused by the mismatch between the running model and fixed prompts during incremental procedure.
Moreover, additional features require additional storage space because the corresponding prompt cannot be directly added on the changed feature.
Memory budget comparison.
We now compare the memory budget of FramMaker with existing approaches.
To make a direct comparison, we use the definition of the working memory size in terms of stored frames, following [54].
The results are summarized in Table 9.
We can see that our FrameMaker only costs 1.2Mb memory, which surpasses TCD [41] using 6.0Mb by 0.21% and 1.63% in terms of CNN and NME.
FrameMaker saves 80% of memory space with the comparable performance.
Meanwhile, we further increase the stored videos but keep the same memory with existing methods.
FrameMaker can still effectively promote performance and shows superior abilities to fight catastrophic forgetting.
Our results are about 3% ahead of TCD in both indicators.
Besides, we also attempt to select the same number of videos as the existing methods.
However, the accuracy of the trained linear classifier, i.e., CNN, is weaker with a small number of video instances.
Interestingly,
NME evaluated by the exemplar class centre yielded fairly or even better results, implying that the model suffers only slight forgetting.
We hypothesize that this is caused by the poor feature diversity that a few condensed frames are provided.
Therefore, the classifier tends to overfit fixed sample features, resulting in poor generalization. [Figure] [Figure] [Figure]
d) Per-category Accuracy on HMDB
A cc u r ac y ov
e r s ee n c l a ss e s A cc u r ac y ov e r s ee n c l a ss e s Number of classes
Number of classes
A cc u r ac y ov
e r d i ff e r e n t c
l a ss e s
A cc u r ac y ov
e r d i ff e r e n t c
l a ss e s
Visualization of condensing weights and instance-specific prompts.
To intuitively understand our
FrameMaker, we provide visualization of condensing weights, Instance-Specific Prompts, condensed frames and learned prompts in Figure 3.
To save space, we uniformly select the most typical 4 frames from T frames for display.
Based on this, we have the following fascinating observations: (1) For frames with strong scene bias, the learned weights are more uniform, while those with motion bias are not.
For example, the action classification for (a) and (b) in Figure 3 does not depend on a specific frame.
Hence, the condensing weight for each frame is roughly the same.
In contrast, stair climbing and hugging in (c) and (d) only occur in some specific frames, and those keyframes are highlighted by condensing weights.
This demonstrates that it is reasonable to condense multiple frames into one frame since our proposed Frame Condensing can assist the learning of which frame is meaningful for the action recognition task.
(2) It is difficult for humans to absorb useful knowledge from the learned prompts shown in Figure 3, and even they seem to be the same.
It should be emphasized that the prompts are different.
We try to share the same prompt for all videos to verify this.
The experiment is conducted on UCF101 with 10 steps, and the performances decrease by 0.83% and 0.78% in terms of
CNN and NME, respectively.
This fully validates that our Instance-Specific Prompt is to replenish the specific information for different instances.
Per-category and per-task analysis.
We depict the task-wise and category-wise accuracy with different forms of stored frames in Figure 4, i.e ., " All Frames ", " Frame Condensing without InstanceSpecific Prompt " and " Frame Condensing with Instance-Specific Prompt ".
As expected, integrating multiple frames into one condensed frame does undermine performance in almost all cases unless
Instance-Specific Prompt is introduced.
As shown in Figure 4(a) and (b), the gains on incremental tasks yielded by Instance-Specific Prompt mainly fall in the later tasks, which implies that the spatio-temporal knowledge absorbed by prompt can effectively alleviate forgetting.
Meanwhile, from
Figure 4(c) and (d), we also observe that Frame Condensing can lead to the performance degradation of actions with large temporal and spatial variations or short duration, such as " CliffDiving ", " Ride horse " and " Hug ", etc.
Therefore, Frame Condensing with only a few parameters is difficult to capture these complex spatio-temporal interactions.
However, Instance-Specific Prompt helps alleviate this issue since it recovers the lost spatio-temporal details in Frame Condensing for each instance.
Discussions
Limitations.
Compared with the previous video incremental learning approaches, FrameMaker provides a simple framework that significantly reduces the amount of memory required for each exemplar video.
Nevertheless, FrameMaker may fail when the memory budget is highly constrained, which needs to be further explored in future work.
Societal Impacts.
Although FrameMaker only stores condensed frames in memory buffer for experience replay, which still retains some information about the original video.
Therefore, FrameMaker may be used in applications with privacy concerns [48].
Conclusion.
This paper proposes FrameMaker, a memory-efficient approach for video classincremental learning.
It explores how to learn an effective condensed frame with less memory for each video that can be applied to the future incremental task.
FrameMaker mainly consists of two key components, i.e., Frame Condensing and Instance-Specific Prompt .
The former learns a group of condensed weights for each video to integrate multiple frames into one condensed frame, while the latter learns to retrieve the collapsed spatio-temporal structure for the condensed frame.
In this way, FrameMaker offers better results with only 20% memory consumption compared to recent advanced methods and sets a new state-of-the-art class-incremental learning performance on multiple challenging datasets.
Memory-based class-incremental learning is one of the effective methods to fight forgetting.
The efficient utilization of limited space is worth exploring in future work. Acknowledgments.
This work is supported by the National Key R&D Program of China under Grant
No. 2018AAA0101501, the National Natural Science Foundation of China undergrantand 62272380, Fundamental Research Funds for the Central Universities no.2019kfyXKJC024, the
Science and Technology Program of Xi’an, China under Grant 21RGZN0017, and by Alibaba Group through Alibaba Innovative Research Program.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
This is supplementary to Section 4.2 "Comparison with State-of-the-art Results". Limited by space, we only summarize the results of our approach using the same exemplar memory size per class as others and the experiment results of "Memory budget comparison" on UCF101 with 10 steps in the main body. Table 10 supplements the average accuracy of FrameMaker with different memory budgets under all benchmarks of UCF101 and HMDB51 datasets, which is compared with the best performance of TCD [41]
[Table] [Table] [Table] As can be seen from Table 10, under other settings, the results of our method are similar to that on
UCF101 with 10 steps.
When using about 20% of the storage space, our method can achieve better performance than TCD, and the accuracy is further improved when the number of saved exemplars is increased, which also proves the robustness of our approach.
B Additional Analysis of the Balance Weights
This is supplementary to Section 4.3 "Ablation Study".
We further discuss the sensitivity to the balance weight for each term in L fc on UCF101 with 10 steps.
We first discuss the sensitivity of α and β , the balance weights for L cf and L cce which are used to achieve stronger constraint on W ki for its effective training.
Table 11 shows the performance of frame condensing (without instance-specific prompt) under various combinations of α and β .
We find the performance under the combination of { α = 1 , β = 1 } consistently exceeds others.
Although a slightly higher performance is obtained on NME under { α = 2 , β = 1 } , it is difficult to achieve the same gain on the two different metrics continuous adjustment.
[Table]
Then we further analyze sensitivity of γ and η , the balance weights for L pf and L pce , when setting α and β as the optimal value, 1 .
Table 12 shows the performance of FrameMaker (frame condensing with instance-specific prompt) under various combinations of γ and η .
We find the performance under the combination of { γ = 1 , η = 1 } always exceeds others.
Some groups of weights make one of the test metrics higher but only { γ = 1 , η = 1 } obtain the best balance of them.
Therefore, we set all the balance weights as 1 finally in our experiments.
[Table]
C Implementation Details
This section provides some additional details about the experiments in the main body.
Training Details.
This is supplementary to Section 3.3 "Training".
We further supplement the elaborate training process and knowledge distillation loss in this section.
Figure 5 shows the overall framework of training process.
[Figure]
In the training process of the incremental step k , we use the dataset D ′ k
= D k (cid:83) M k − 1 to update the model from F (; Θ k − 1 ) , where D k is the dataset of the task k which consists of videos belonging to current classes, and M k − 1 is memory bank which contains condensed frames generated by
FrameMaker after the step k − 1 .
The model classifies the samples from the two datasets respectively and use exemplars in old classes for knowledge distillation.
For the classification task, we adopt cross entropy loss to supervise.
L dce and L mce are the cross entropy losses for video example V ki in D k and condensed frame exemplar
I k − 1 i in M k − 1 .
L dce
[Equation]
{+where y ki and y k − 
+}
i is the ground truth label of V ki
and I k − 1 i , respectively.
In order to save the information of old classes better, the knowledge distillation function of PODNet [12] is employed on the old classes data M k − 1 and it consists of two components, the spatial distillation function
L
spatial
and the final embedding distillation function
L flat .
[Equation]
where f l (; Θ k ) is the output of intermediate convolution layer l of model F (; Θ k ) .
C, H, W are the size of channel, height and width of the output feature of layer l .
[Equation]
where f (; Θ k ) is the feature extractor of model F (; Θ k ) .
The final distillation loss of the condensed frame exemplars is given by:
[Equation]
To simplify, the hyperparameters used in PODNet [12] final distillation loss is ignored.
The total objective function for task k is given by:
[Equation]
which is the same as Eq.
8 in the manuscript.
Dataset Details.
Following TCD [41], we use three action recognition datasets, UCF101 [50],
HMDB51 [28] and Something-Something V2 [18].
The HMDB51 dataset is a large collection of realistic videos from various sources which can be found in https://serrelab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/. UCF101 dataset is an extension of UCF50 [46] and consists of 13,320 video clips, which can be found in https://www.crcv.ucf.edu/data/UCF101.php.
Something-SomethingV2 dataset is a large-scale motionsensitive dataset that shows humans performing pre-defined basic actions with everyday objects, which can be found in https://developer.qualcomm.com/software/ai-datasets/something-something.
For UCF101, we have three settings, 51 + 10 × 5 stages, 51 + 5 × 10 stages and 51 + 1 × 25 stages.
For HMDB51, the number of initial task classes is 26, and the remaining 25 classes are separated into 5 or 25 groups.
For Something-Something V2, the benchmarks are 84 + 10 × 9 stages and 84 + 5 × 18 stages.
The random seed we use is also the same as TCD [41].
Model and Hyperparameter Setup.
TSM [34] is employed as our backbone, and we follow the data preprocessing procedure and the basic training setting of TSM.
We train a ResNet-34 TSM for UCF101 and a ResNet-50 TSM for HMDB51 and Something-Something V2.
Each incremental training procedure takes 50 epochs.
We use SGD optimizer with an initial learning rate of 0.04 for
UCF101 and Something-Something V2, and 0.002 for HMDB51, which is reduced by half afterepochs.
For incremental steps, we set the learning rate for all datasets as 0.001 and make the same reduction.
The learning rate is set as 0.01 and 0.001 for condensing weights and the instance-specific prompt, respectively, and the total iteration for their optimization is set to be 8k.
Hardware Information.
We train all models on eight NVIDIA V100 GPUs and use PyTorch [42] for all our experiments.
D Visualizations
Visual comparison of features.
Figure 6 plots visual comparisons of the extracted features with different frames as model inputs.
We can summarize that the average frame shows the largest feature difference compared to the features extracted from the original clips.
Our proposed Frame
Condensing and Instance-Specific Prompt can effectively reduce the feature differences with the original clips and thus effectively improve the feature quality of condensed frames.
[Figure]
[Figure]
Visualizations of condensed frames.
Figure 7 plots the condensed frames, the sum of the condensed frames and their prompts, and the activation region generated by the GradCAM technology.
From the results, we observe the learned prompts have very small magnitudes (aver- age value is 0.03), hence no significant difference is observed actually.
However, from the GradCAM attention maps, we can observe the condensed frames without prompts may focus on background regions, while the learned prompts tend to increase the attention of motion regions or reduce the attention of the backgrounds.
We think the reason behind this is that the collapsed temporal dynamics may easily lead the model pay more attention to the background to some extent, while the prompt can guide the model to refocus on the motion region and weaken the interference from backgrounds.
E Training Prompt without Condensed Frame
We attempt to train Instance-Specific Prompt with out condensed frames, and the learned prompting parameters are visualized in Figure 8(c) and (d).
We observe that the learned prompts have no intuitive semantics.
To further understand this phenomenon, we plot the per-task accuracy and the loss curve of the prompts in Figure 8(a) and (b), respectively.
We can summarize two points: (i) the learned prompts without condensed frames cannot fight against catastrophic forgetting, which indicates that the semantic information of the learned prompts is weak; (ii) the loss curve of the prompts without condensed frames is much higher than that with condensed frames, which implies
[Figure]
[Figure]
that it is difficult to train a single prompts from scratch.
Thus, it is crucial that the condensed frames provide a good initialization for the prompts to avoid the local minima of prompts.