Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Recent incremental learning for action recognition usually stores representativevideos to mitigate catastrophic forgetting.
However, only a few bulky videoscan be stored due to the limited memory.
To address this problem, we propose FrameMaker , a memory-efficient video class-incremental learning approachthat learns to produce a condensed frame for each selected video.
Specifically,
FrameMaker is mainly composed of two crucial components: Frame Condensing and Instance-Specific Prompt .
The former is to reduce the memory cost bypreserving only one condensed frame instead of the whole video, while the latter aims to compensate the lost spatio-temporal details in the Frame Condensingstage.
By this means, FrameMaker enables a remarkable reduction in memorybut keep enough information that can be applied to following incremental tasks.
Experimental results on multiple challenging benchmarks, i.e ., HMDB51, UCF101and Something-Something V2, demonstrate that FrameMaker can achieve betterperformance to recent advanced methods while consuming only 20% memory.
Additionally, under the same memory consumption conditions, FrameMaker significantly outperforms existing state-of-the-arts by a convincing margin.
The sourcecode and models will be released when this manuscript is public.
Introduction
Training video action recognition models with all classes in a single fine-tuning stage have beenwidely studied [6, 13, 29, 33, 35, 47, 48, 58] in recent years.
However, in many realistic scenarios,limited by privacy matters or technologies, the different classes can only be presented in sequence,where the previously trained classes are either unavailable or partially available in limited memory.
Naively training models will overfit currently available data and cause performance deterioration onpreviously seen classes, which is named catastrophic forgetting [36].
Class-incremental learning is amachine learning paradigm to fight this challenge when fine-tuning a single model in a sequence ofindependent classes.
A branch of class-incremental learning methods [5, 7, 9, 12, 21, 32, 42, 56] has achieved remarkableperformances in the image domain by re-training a portion of past examples.
Meanwhile, someexisting video incremental learning methods [38, 50] have demonstrated that storing more pastexamples can improve the ability to mitigate forgetting.
Nevertheless, videos are informationredundant and require a high memory load to be saved, hence it is impractical to store a good dealof training videos for each class.
Therefore, herding strategy [42] is adopted to select a small set ofrepresentative videos for the exemplar memory [38, 50].
vCLIMB [50] tries to down-sample framesfor each video to reduce the memory consumption, and imposes a temporal-consistency regularizationfor better performance.
Despite the remarkable performance, these methods still demand to conserve
[Figure]
[Figure]
multiple ( i.e. , 8-16) frames for each representative video, leading to non-negligible memory overhead,which limits their further real-world applications.
To remedy the above weaknesses, we present FrameMaker, a memory-efficient video classincremental learning approach.
Concretely, the proposed FrameMaker contains two components, Frame Condensing and Instance-Specific Prompt.
In Frame Condensing, FrameMaker assigns alearnable weight to each frame, and then these weights are adjusted to make the weighted sum framei.e . , the condensed frame, share the same embedding features with the original video clip.
Obviously,the collapsed temporal dimension and mixed spatial cues by Frame Condensing may degrade theaccuracy of action recognition.
To compensate the missing spatio-temporal details, inspired by TaskSpecific Prompt [34], we further propose Instance-Specific Prompt, which learns a set of parametersfor each condensed frame to perform fine-grained pixel-wise modifications.
In this way, the memoryrequired to store a single video can be remarkably compressed.
An intuitive comparison is given in Figure 1(a).
On the premise of comparable performance with TCD [38], FrameMaker significantlyreduces memory overhead by 80%.
Moreover, the comparison curve displayed in Figure 1(b) furtherindicates that FrameMaker achieves a consistent performance improvement over recent TCD with thesame memory consumption.
In summary, we make the following contributions: 1) We present a memory-efficient video classincremental learning method FrameMaker, which significantly reduces the memory consumption ofexample videos by integrating multiple frames into one condensed frame; 2)
For the first time, weshow a novel perspective of Prompt, i.e., Instance-Specific Prompt, and demonstrate that it can embedeffective spatio-temporal interactions into learned condensed frames for video class-incrementallearning; 3) Experimental results reveal that FrameMaker consistently outperforms recent state-of-theart methods with the same memory overhead on three challenging benchmarks (HMDB51, UCF101,and Something-Something V2).
Related Works2.1 Class-Incremental Learning
The class-incremental learning has been well-studied in image domain [1, 25, 31, 59], existingapproaches could be grouped into knowledge distillation-based methods [20], memory-based methods [37, 42, 43], and architecture-based methods.
In video domain , class-incremental learningfor action recognition is still an underexplored area.
The existing solutions are designed for bettertemporal constraints, such as decomposing the spatio-temporal features [61], exploiting time-channelimportance maps [38], and applying temporal consistency regularization [50].
Notably, both [38] and[50] have shown that more stored examples in memory can effectively encourage the performance.
However, FrameMaker attempts to minimize the stored frames for each video by a pretty maneuver,which can provide decent performance with slight storage requirements.
2.2 Prompt-Based Learning
Prompting is an emerging transfer learning technique in natural language processing [34], whichadapts the language model to downstream tasks by a fixed function.
Since the design of prompting functions is challenging, the learnable prompts [28, 30] are proposed to learn task-specific knowledgeparameters for the input.
Recently, in the visual domain, some works have also emerged to adapt thepre-trained models to new tasks by adding prompt tokens [23] or pixels [3] in the data space.
Besides, DualPrompt [54] and L2P [53] get rid of the rehearsal buffer in continual learning by learning aset of task-specific parameters.
In this paper, we draw inspiration from prompting, and present an Instance-Specific Prompt to replenish the collapsed spatio-temporal cues for condensed frames.
2.3 Action Recognition
The approaches designed for action recognition task can be grouped into three categories: 2D CNN,3D CNN and Transformer-based methods.
2D CNN based methods is more efficient [19, 45], while3D CNN [6, 10, 15, 16, 40, 47–49, 52, 58] and Transformer-based [2, 4, 13, 29, 35] can achieveremarkable accuracy in action recognition at the cost of more computation cost.
In this work, wefollow the settings in [38, 61], and employ the Temporal Shift Module (TSM) [33] based on 2D CNN for efficient experiments.
The video summarization and efficient action recognition tasks haveprovided some methods for key-frame selection and generation, which are related to this work.
Thevideo summarization [51, 60, 63] task selects the most representative frames for efficient humanunderstanding, which has a different paradigm with action recognition task, hence we will mainlydiscuss efficient The efficient video classification task aims to learn a strategy to improve classificationaccuracy with fewer discriminative frames as input.
The approaches designed for this task can begrouped into two categories: rule-based and model-based methods. The former [26, 62] select framesvia some pre-defined rules such as motion distribution [62] or channel sampling [24], and the laterdesign a submodel to select or generate key frames [14, 17, 22, 41, 55, 57]. In this paper, we proposemodel-free frame condensing method for video class-incremental learning, which only learns a fewcondensing weights and prompting parameters for each instance.  Method
This section presents the general framework of FrameMaker as shown in Figure 2, which mainlyconsists of two crucial components: Frame Condensing and Instance-Specific Prompt.
Firstly, Frame
Condensing is designed to integrate multiple frames into one frame for efficient storage.
Then weintroduce Instance-Specific Prompt to compensate the collapsed temporal information and the mixedspatial knowledge.
Finally, the specific class-incremental training details for subsequent tasks using
Condensed Frames are given.
3.1 Problem formulation
The purpose of video class-incremental learning is to train a model F (; Θ)
=
h ( f (; θ ); ξ ) parameterized by Θ as the tasks {T 1 , T 2 , · · · , T K } arrive in a sequence, where f (; θ ) is the feature extractorand h (; ξ ) is the classification head.
Each task T K has its specific dataset D k
= { ( d ki , y ki ) ,
y ki ∈ L k }whose labels belong to a predefined label set L k and have not appeared in previous tasks.
Thememory-based methods [38, 42, 50] have been shown to be effective in preventing catastrophicforgetting.
Specifically, after the incremental training step k , a memory bank M k will be establishedfor the dataset D k in this task to store the representative exemplars or features.
Exemplars stored inprevious datasets M 1: k = M 1 (cid:83) M 2 (cid:83) · · · (cid:83) M k will be used in subsequent incremental tasks tomitigate forgetting.
At the incremental step k , the model F (; Θ k ) is trained from F (; Θ k − 1 ) with
D ′ k
= D k (cid:83) M 1:( k − 1) and evaluated on all seen classes. 3.2 FrameMaker Frame Condensing.
We follow the standard protocol of video class-incremental methods [38, 50],which are based on memory-replay strategy and knowledge distillation.
Differently, we devotes toreducing the number of stored frames for representative videos to achieve memory-efficient videoclass-incremental learning.
Formally, we first select a subset of video instances V k by herdingstrategy [42] from D k after the incremental step k . Given a video V ki ∈ V k , we then sample Tframes { I ki 1 , I ki 1 , · · · , I kiT } uniformly and integrate them to condense a more representative one. Tothis end, we define learnable weights W ki = { w ki 1 , w ki 2 , · · · , w kiT } for the i -th video instance V ki in Embedding Feature
Framesample ⋮ ⨂ ⨂ ⨂ ⊕ ⊕
V !"
𝐼 !
#" 𝐼 !
$" 𝐼 !%"
Frame Condensing
Instance-Specific Prompt 𝐼 !"
𝑃 !"
𝐿 &’( 𝐿 )( 𝐿 )& 𝑓( ;Θ " ) 𝑓( ;Θ " ) 𝑓( ;Θ " ) ⨁ : Pixel-wise Addition ⨂ : Multiplication
GroundTruth 𝑤 !"
# 𝑤 !
$# 𝑤 !
# 𝐿 &’&
Condensed Frame
Prompt (𝑃 !"
+𝐼 !" )
Video Frames h h
Classification Score exemplar set V k .
And the condensed frame can be calculated as follows:
[Equation]
where C , H and W are the size of channel, height and width of input frames, respectively.
Next,the condensed frame I ki is excepted to empower the same or much similar expressive ability as theoriginal video clip.
Hence, the embedding features of the condensed frame extracted from currentmodel should be consistent with the features of the original video clip:
[Equation]
(2) where f (; θ k ) is the feature extractor of current model F (; Θ k ) .
This consistency regularization canfoster the condensing weights to exploit memory-efficient expressions of video clips.
To furtherimprove the adaptability of the condensed frames to the current model, we employ cross entropy lossto supervise the classification confidence from the condensed frame:
[Equation]
The full objective function for condensing weights W ki is given by:
[Equation]
With the methods described above, we can learn an effective representation, i.e. , the condensed framefor each video based on current model.
Instance-Specific Prompt.
However, condensing a video into one frame would inevitably lose thethe temporal dynamics and complete spatial information of the original video to a certain extent.
Therefore, an intriguing perspective of prompt, termed as Instance-Specific Prompt, is proposed toreplenish the missing spatio-temporal cues for the condensed frame.
Specifically, we first construct a learnable prompt P ki ∈ R C × H × W for each exemplar video V ki ,which shares the same spatial resolution as its original clip.
The prompt P ki is then pixel-wisesummed with the condensed frame I ki , and learns to pull the embedding feature of the summed frame( I ki + P ki ) and the video clip V ki together, which is similar to Eq. 2:
[Equation]
Notably, the Eq. 2 for condensing weights is difficult to learn the satisfactory spatio-temporal featuresdue to the collapse of the temporal dimension.
Nevertheless, the Eq. 5 can enrich the representationsof condensing frames by introducing more flexible learnable parameters in the input space.
Besides,the Cross-Entropy loss is also utilized to enhance the semantic perception:
[Equation]
[Equation]
Theoretically, the condensing weights W ki and the prompting parameters P ki can be jointly updatedby L p =
L pf + L pce .
Interestingly, we observed in our practice that the flexible prompt P ki leads to thecondensing weights W ki being under optimized.
Hence, we give the final training objective functionfor frame condensing as:
[Equation]
where additional L cf and L cce are added to achieve stronger constraint on W ki for its effective training.α , β , γ and η are the balance weights for each term, and they are empirically set to 1.0 unlessotherwise specified.
After the optimization of condensing weights and prompting parameters, the prompt is added directlyto the condensed frame and stored together.
And the learned prompts will be frozen when thecorresponding task is completed.
Therefore, there is no extra memory cost for prompts.
3.3 Training
It is worth noting that FrameMaker aims to condense the stored frames for representative videos, andother class-incremental learning steps still follow the standard pipeline.
Specifically, when trainingthe incremental step k , we use the dataset D ′ k
= D k (cid:83) M k − 1 to update the model from F (; Θ k − 1 ) ,where D k is the dataset of the task k which consists of videos belonging to L k , and M k − 1 is memorybank which contains condensed frames generated by FrameMaker.
The training samples from D kand M k − 1 are alternately fed into the current model according to the proportion of their samplenumber.
To further prevent the catastrophic forgetting, we also employ the knowledge distillationmethod proposed in PODNet [12] to transfer the knowledge from the previous model F (; Θ k − 1 ) tocurrent model F (; Θ k ) .
The overall objective function for task k reads:
[Equation]
where L dce and L mce are the Cross Entropy losses for new task data D k and condensed frame exemplars
M k − 1 , respectively.
And L mdist is the knowledge distillation loss [12] for condensed frame exemplars.
Experiments
4.1 Experimental Setup
Datasets and evaluation metrics.
The proposed FrameMaker is evaluated on three standard actionrecognition datasets, UCF101 [46], HMDB51 [27] and Something-Something V2 [18].
HMDB51dataset consists of 6.8K videos belonging to 51 classes from YouTube or other websites.
UCF101dataset contains 13.3K videos from 101 classes.
Something-Something V2 is a crowd-sourced datasetthat includes 220K videos from 174 classes.
For UCF101, the model is trained on 51 classes first, and the remaining 50 classes are divided into5, 10 and 25 tasks.
For HMDB51, we train the base model using videos from 26 classes, and theremaining 25 classes are separated into 5 or 25 groups.
For Something-Something V2, we first train84 classes in the initial stage, and generate the groups of 10 and 5 classes.
To evaluate the performance of class-incremental learning, we infer the test videos from all seencategories after each task, and finally, report the average accuracy of all tasks.
Following [38], twodifferent metrics are reported, i.e., CNN and NME.
CNN refers to training a fully-connected layer forextracted features, which is a standard classification protocol.
NME is proposed by iCaRL [42], whichassigns the labels for test data by comparing the feature embeddings with the mean-of-exemplars.
Implementation details.
TSM [33] is employed as our backbone, and we follow the data preprocessing procedure of TSM.
For UCF101, we train a ResNet-
TSM
for 50 epochs with a batchsize 256 from an initial learning rate 0.04.
For HMDB51 and Something-Something V2, we traina ResNet-50 TSM for 50 epochs with a batch size of 128 from an initial learning rate of 1e-3 and0.04, respectively.
All used networks are first pre-trained on ImageNet [8] for initialization.
Thesesettings are consistent with TCD [38].
We train all models on eight NVIDIA V100 GPUs and use
PyTorch [39] for all our experiments. [Table] [Table] [Table]
4.2 Comparison with State-of-the-art Results
This section presents the qualitative comparison of our proposed FrameMaker with the existing classincremental learning approaches under multiple challenging settings on three datasets: UCF101 [46],
HMDB51 [27] and Something-Something V2 [18].
For a fair comparison, we use the same exemplarmemory size per class, model structure and pre-training initialization as the existing methods [38].
Table 1 summarizes the results on UCF101 and HMDB51, which shows that FrameMaker outperformsother methods consistently under different configurations in terms of both CNN and NME scores.
The average accuracy of our FrameMaker surpasses TCD by around 3.0% on UCF101 and 2.0% on
HMDB51 in terms of CNN, respectively.
These results demonstrate that one condensed frame canalso be equipped with effective spatio-temporal representation.
FrameMaker is compared with recent advanced methods on Something-Something V2 in Table 2.
FrameMaker sets new state-of-the-art performance under multiple challenging settings.
Although
FrameMaker only uses one condensed frame for future replay, it still achieves decent performance onthe large-scale motion-sensitive dataset.
We speculate that these gains mainly come from two aspects:
The proposed Instance-Specific Prompt effectively rescues the lost spatio-temporal details, whichis also discussed in Section 4.3.
(2) Memory-efficient FrameMaker allows us to store more exemplarvideos with less memory consumption, which greatly alleviates catastrophic forgetting.
4.3 Ablation Study
In this section, we present ablation studies to analyze the properties and the effectiveness of
FrameMaker.
If not specified, the ablation studies are performed on UCF101 with 10 steps and
HMDB51 with 5 steps.
For comparison with the case of storing all frames, we select 5 exemplarvideos for each class.
Frame Condensing and Instance-Specific Prompt.
To show the effectiveness of Frame Condensingand Instance-Specific Prompt, we compare them with the following alternative methods to producestored frames for future replay: (1) All Frames, which simply saves the whole video. (2) One
Random Frame, which randomly selects one frame for each exemplar video.
(3) One Average Frame,which averages the input frames for each exemplar video.
(4) One Condensed Frame, which isgenerated by our Frame Condensing procedure.
The results are reported in Table 3, we observe that Frame Condensing with Instance-Specific Prompt leads to at least on par or better performance than [Table] [Table]
[Table] [Table]
[Table]
preserving all frames while using just one condensed frame under all settings on both datasets. Inparticular, Instance-Specific Prompt yields an improvement of 2.07%, and 0.22% with randomlysampled frames on UCF101 and HMDB51, respectively, which implies that prompts can alwaysreplenish the ignored sptiao-temporal cues in video frames.
Loss terms.
To preserve the best spatio-temporal information in the condensed frames, the distillationloss L *f and the cross entropy loss L *ce are both applied for Frame Condensing and Instance-Specific
Prompt.
Table 4 presents the results with Frame Condensing from several different combinationsof loss terms.
L cce provides more explicit semantic supervision, making the accuracy to step further.
Similar experiments for Instance-Specific Prompt are shown in Table 5.
One noticeable thing is thatonce introducing prompt, we only need to calculate the final L pf and L pce to realize the joint updateof condensing weights W ki and prompt P ki .
However, we observe that the learned weights, in thiscase, tend to be average, which indicates that W ki is under-optimized.
We speculate that this isdue to the strong plasticity of prompt, which leads to the insufficient optimization of W ki .
To thisend, we further employ L cf and L cce for W ki as the guidance.
The improvement in Table 5 and thevisualizations in Figure 3 show that W ki receives effective learning.
The number of frames used for Frame Condensing.
As discussed in Section 3.2, the T frames areuniformly sampled from a representative video for Frame Condensing.
In Table 6, we show the effectof T in video class-incremental learning.
We observe that the performance increases as T increasesfor both CNN and NME.
However, the performance is saturated when using 16 frames, which isin line with the conclusion of existing methods [38, 50], i.e., storing more frames in a video doesnot necessarily deliver performance improvement.
This phenomenon reveals that FrameMaker canempower one Condensed Frame to absorb plenty of spatio-temporal features from original clips.
Different backbones. To evaluate the applicability of proposed FrameMaker, we replace the TSMwith R3D50 [16] and ViT[11], as shown in Table 7. In these experiments, we simply replicatethe condensed frames multiple times along the the temporal dimension to meet the 3D modelsand Transformers.
From the results, we can find that our FrameMaker can consistently achievecomparable performance on the three different backbones with only 12% memory cost. These resultsdemonstrate the better generalization of FrameMaker. Alternative prompting strategies.
We conduct experiments to explore the types and positions ofthe prompts in Table 8.
(i) We compare our Instance-Specific (I.-Spec.) Prompt (ISP) with the Task- (T.-Spec.) and Class-Specific (C.-Spec.)
prompts and observe that our ISP can achieve thebest performance when operating on both features and frames. Since the actions in videos havegreat intra-class variance, simply sharing task and class-specific prompts is insufficient to reserve theimportant spatia-temporal cues for each instance; (ii) Positions of prompt : For the feature settings,we add the prompt on the features of the 4 stages of the ResNet. The feature-based prompt is worsethan our frame-based prompt, which may be caused by the mismatch between the running model and
Memory Per Class
[Table]
Classifier
[Table]
iCaRL
[Table]
[Table]
[Table]
UCIR [21] 61.
[Table]
[Table]
[Table]
[Table]
[Table]
PODNet [12] 63.
[Table] [Table]
[Table] [Table]
[Table]
TCD [38] 64.
[Table]
Memory Per Class 1 F × 1 V=0.15Mb 1 F × 2 V=
[Table]
3Mb 1 F × 5 V
[Table]
75Mb FrameMaker 49.
[Table] [Table]
Memory Per Class 1 F × 8 V=
[Table]
2Mb 1 F
[Table]
16 V
[Table]
4Mb 1 F × 40 V
[Table] [Table]
0Mb FrameMaker 73.
[Table] [Table] [Table]
[Figure]
during incremental procedure.
Moreover, additional features require additional storagespace because the corresponding prompt cannot be directly added on the changed feature.
Memory budget comparison.
We now compare the memory budget of FramMaker with existingapproaches.
To make a direct comparison, we use the definition of the working memory size interms of stored frames, following [50].
The results are summarized in Table 9. We can see thatour FrameMaker only costs 1.2Mb memory, which surpasses TCD [38] using 6.0Mb by 0.21% and1.63% in terms of CNN and NME. FrameMaker saves 80% of memory space with the comparableperformance. Meanwhile, we further increase the stored videos but keep the same memory withexisting methods.
FrameMaker can still effectively promote performance and shows superior abilitiesto fight catastrophic forgetting.
Our results are about 3% ahead of TCD in both indicators.
Besides,we also attempt to select the same number of videos as the existing methods.
However, the accuracy ofthe trained linear classifier, i.e., CNN, is weaker with a small number of video instances.
Interestingly, NME evaluated by the exemplar class centre yielded fairly or even better results, implying that themodel suffers only slight forgetting.
We hypothesize that this is caused by the poor feature diversitythat a few condensed frames are provided.
Therefore, the classifier tends to overfit fixed samplefeatures, resulting in poor generalization.
Visualization of condensing weights and instance-specific prompts.
To intuitively understand our
FrameMaker, we provide visualization of condensing weights, Instance-Specific Prompts, condensedframes and learned prompts in Figure 3.
To save space, we uniformly select the most typical 4 framesfrom T frames for display.
Based on this, we have the following fascinating observations: (1) Forframes with strong scene bias, the learned weights are more uniform, while those with motion biasare not.
For example, the action classification for (a) and (b) in Figure 3 does not depend on a specificframe.
Hence, the condensing weight for each frame is roughly the same.
In contrast, stair climbingand hugging in (c) and (d) only occur in some specific frames, and those keyframes are highlightedby condensing weights.
This demonstrates that it is reasonable to condense multiple frames into oneframe since our proposed Frame Condensing can assist the learning of which frame is meaningful forthe action recognition task.
(2) It is difficult for humans to absorb useful knowledge from the learnedprompts shown in Figure 3, and even they seem to be the same.
It should be emphasized that theprompts are different.
We try to share the same prompt for all videos to verify this.
The experiment is
[Figure]
ll
[Figure]
ti
[Figure]
P
[Figure]
H a nd s t a nd
D r
i bb l e
C a t c h
[Figure]
hoo t bo w R i d e ho r s
[Figure]
P ou r
W a l k
H
[Figure]
conducted on UCF101 with 10 steps, and the performances decrease by 0.83% and 0.78% in terms of
CNN and NME, respectively.
This fully validates that our Instance-Specific Prompt is to replenishthe specific information for different instances.
Per-category and per-task analysis.
We depict the task-wise and category-wise accuracy withdifferent forms of stored frames in Figure 4, i.e ., " All Frames ", " Frame Condensing without InstanceSpecific Prompt " and " Frame Condensing with Instance-Specific Prompt ".
As expected, integratingmultiple frames into one condensed frame does undermine performance in almost all cases unless
Instance-Specific Prompt is introduced.
As shown in Figure 4(a) and (b), the gains on incrementaltasks yielded by Instance-Specific Prompt mainly fall in the later tasks, which implies that thespatio-temporal knowledge absorbed by prompt can effectively alleviate forgetting.
Meanwhile, from
Figure 4(c) and (d), we also observe that Frame Condensing can lead to the performance degradationof actions with large temporal and spatial variations or short duration, such as " CliffDiving ", " Ridehorse " and " Hug ", etc.
Therefore, Frame Condensing with only a few parameters is difficult to capturethese complex spatio-temporal interactions.
However, Instance-Specific Prompt helps alleviate thisissue since it recovers the lost spatio-temporal details in Frame Condensing for each instance.
Discussions
Limitations.
Compared with the previous video incremental learning approaches, FrameMakerprovides a simple framework that significantly reduces the amount of memory required for eachexemplar video.
Nevertheless, FrameMaker may fail when the memory budget is highly constrained,which needs to be further explored in future work.
Societal Impacts.
Although FrameMaker only stores condensed frames in memory buffer for experience replay, which still retains some information about the original video.
Therefore, FrameMakermay be used in applications with privacy concerns [44].
Conclusion.
This paper proposes FrameMaker, a memory-efficient approach for video classincremental learning.
It explores how to learn an effective condensed frame with less memoryfor each video that can be applied to the future incremental task.
FrameMaker mainly consists oftwo key components, i.e., Frame Condensing and Instance-Specific Prompt .
The former learns agroup of condensed weights for each video to integrate multiple frames into one condensed frame,while the latter learns to retrieve the collapsed spatio-temporal structure for the condensed frame.
Inthis way, FrameMaker offers better results with only 20% memory consumption compared to recentadvanced methods and sets a new state-of-the-art class-incremental learning performance on multiplechallenging datasets.
Memory-based class-incremental learning is one of the effective methods tofight forgetting.
The efficient utilization of limited space is worth exploring in future work.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]