BMU-MoCo: Bidirectional Momentum Update for Continual Video-Language Modeling
Abstract
Video-language models suffer from forgetting old/learned knowledge when trained with streaming data.
In this work, we thus propose a continual video-language modeling (CVLM) setting, where models are supposed to be sequentially trained on five widely-used video-text datasets with different data distributions.
Although most of existing continual learning methods have achieved great success by exploiting extra information ( e.g. , memory data of past tasks) or dynamically extended networks, they cause enormous resource consumption when transferred to our CVLM setting.
To overcome the challenges ( i.e. , catastrophic forgetting and heavy resource consumption) in CVLM, we propose a novel cross-modal MoCo-based model with bidirectional momentum update (BMU), termed BMU-MoCo.
Concretely, our
BMU-MoCo has two core designs: (1) Different from the conventional MoCo, we apply the momentum update to not only momentum encoders but also encoders ( i.e. , bidirectional) at each training step, which enables the model to review the learned knowledge retained in the momentum encoders.
(2) To further enhance our BMU-MoCo by utilizing earlier knowledge, we additionally maintain a pair of global momentum encoders (only initialized at the very beginning) with the same
BMU strategy.
Extensive results show that our BMU-MoCo remarkably outperforms recent competitors w.r.t.
video-text retrieval performance and forgetting rate, even without using any extra data or dynamic networks.
Introduction
Existing video-language modeling (VLM) methods have achieved promising performance for videotext retrieval [59, 39, 61, 30, 24, 53, 28, 4] with non-streaming data.
However, in real-world application scenarios, VLM models need to evolve with streaming data ( e.g. , collected from the
Internet [39, 42]) to accommodate more tasks.
Under this setting, since it costs too much resource to retrain the model with both old and new data for each task, a common practice is to fine-tune VLM models with only the newly-arrived data.
Note that such model fine-tuning leads to severe performance degradation on previous tasks.
This is a well-documented phenomenon called catastrophic forgetting [18, 38] under the conventional continual learning setting [48, 45, 34, 16, 23, 60].
Therefore, in this work, we propose a continual video-language modeling (CVLM) setting to better simulate the realistic scenario.
Under our CVLM setting, models are supposed to be sequentially trained on five widely-used video-text datasets: VATEX [54], ActivityNet [25], MSR-VTT [55],
DiDeMo [20], and MSVD [10].
An evaluation protocol is also established for CVLM, which contains three metrics to respectively measure the text-to-video retrieval performance (Recall@1, shortened as [Figure]
[Figure]
current
[Figure]
final model
[Figure]
R@1), forgetting rate (FR), and harmonic mean (HM) performance (see more details of the evaluation protocol in Sec. 4.1).
Moreover, we implement a basic cross-modal MoCo [19] model (Base-MoCo) as our baseline method since it has shown superior performance on video-language modeling [33, 35].
As illustrated in Figure 1(a), we can observe that in spite of achieving great R@1 results with current models (evaluated right after trained on each task), the performance of the final Base-MoCo model (trained across all five tasks) drops significantly (i.e., catastrophic forgetting).
To tackle the catastrophic forgetting problem, most recent continual learning works attempt to preserve the learned knowledge from a variety of perspectives: (1) Maintaining a memory buffer to save and exploit data from previous tasks [46, 5, 34, 3, 8, 45]; (2) Generating pseudo data of learned tasks [51, 27, 43, 57]; (3) Extending the network architecture dynamically as each new task arrives [48, 16, 1, 7, 31].
However, when these methods are transferred to our CVLM setting, the resource consumption is enlarged rapidly as the number of tasks grows, due to the characteristic of video data.
In addition, another branch of continual learning works focus on imposing a regularization constraint with quadratic penalty [23, 60, 14, 49] or knowledge distillation [32, 2, 21, 44, 22], which leads to an unwanted trade-off on the performance of old and new tasks with limited neural resources [40].
Therefore, it is a long-standing and arduous challenge to train a video-language modeling network under the CVLM setting with both effectiveness and efficiency taken into consideration.
To overcome this challenge under the CVLM setting, we devise BMU-MoCo, a cross-modal MoCobased model with a novel bidirectional momentum update (BMU) strategy.
As shown in Figure 1(b), our BMU-MoCo needs neither extra memory data nor dynamically extended neural networks, which is quite different from previous continual learning methods.
Concretely, similar to cross-modal MoCo applied in [33, 35], our proposed BMU-MoCo has a video encoder ( i.e. , ViT-Base [13]) and a text encoder ( i.e. , BERT-Base [12]), followed by the momentum video/text encoders.
Different from the original MoCo [19] and its cross-modal versions [15, 33, 35] that utilize momentum update for only momentum encoders to maintain a large consistent queue, our BMU strategy imposes momentum update on both momentum encoders and (video/text) encoders.
As a result, at each training step, the encoders of our BMU-MoCo learn the new knowledge by end-to-end update with back-propagation whilst reviewing the old knowledge directly from the parameters of momentum encoders by momentum update.
In our opinion, our BMU-MoCo outperforms existing methods for two main reasons: (1) Momentum encoders are initialized by current encoders at the beginning of each new task and then progress slowly, which helps our model preserve adequate old knowledge without sacrificing the performance on new tasks; (2) Since there is no category information under the CVLM setting, learning from memory data or distilling with a batch of new data only absorbs part of previous knowledge while our BMU-MoCo learns holistic old knowledge directly from the parameters of momentum encoders.
To further enhance our proposed BMU-MoCo, we also maintain a pair of global (cross-task) momentum encoders with the same BMU strategy, which are only initialized at the very beginning of the whole training process and thus preserve earlier knowledge than the normal local (task-specific) momentum encoders.
The main contributions of this paper are four-fold: (1) We propose a new continual video-language modeling (CVLM) setting, where models are supposed to be sequentially trained on five widely-used video-text datasets.
(2) To effectively and efficiently overcome the catastrophic forgetting problem under the CVLM setting, we devise BMU-MoCo, a cross-modal MoCo-based model with a novel bidirectional momentum update (BMU) strategy.
For the first time, our BMU strategy can review holistic old knowledge directly from the parameters of momentum encoders while learning on new tasks, without extra memory or dynamic network across all tasks.
(3) To bring further improvements to our proposed BMU-MoCo, a pair of global momentum encoders are maintained by the same BMU strategy to preserve and review earlier knowledge under the CVLM setting.
(4) Extensive experimental results demonstrate that our BMU-MoCo outperforms recent continual learning methods by large margins w.r.t.
both text-to-video retrieval performance and forgetting rate, even without any extra memory data or dynamically extended networks.
Related Work
Video-Language Modeling.
Most existing methods for video-language modeling follow two paradigms: (1) Single-stream methods [36, 61, 52, 29, 28, 56] typically include a multi-modal transformer to achieve fine-grained cross-modal interaction between the video and language modalities.
Although achieving great performance, they suffer from the huge time complexity caused by the pairwise inputs during inference, which makes them unsuitable for practical applications.
(2) Two-stream methods [17, 41, 4, 33, 35] learn video and text representations independently, and align them after encoding.
To ensure the inference efficiency, both the baseline method ( i.e. , Base-MoCo) and
our
BMU-MoCo for CVLM are set to be two-stream methods.
Importantly, different from Base-MoCo, our BMU-MoCo has a novel BMU strategy to address the catastrophic forgetting problem and two extra global momentum encoders to further boost the model performance.
Continual Learning.
Conventional continual learning methods mainly focus on image classification tasks.
They can be roughly categorized into three groups: (1) Rehearsal-based methods apply extra memory to store sampled data [45, 34, 46, 9, 5, 3, 8, 6] or generate pseudo data [51, 27, 43, 57] from previous tasks.
The memory size and the training complexity tend to be enlarged significantly as the number of tasks grows.
(2) Expansion-based methods either add extra extended networks for new tasks [48, 16, 1, 7, 31] or select partial model parameters to update for different tasks [58, 47, 50].
They need more computational resources especially for a long sequence of training tasks (e.g., under our CVLM setting).
(3) Regularization-based methods modify the model parameters with quadratic loss penalty [23, 60, 14, 49] or knowledge distillation constraints [32, 2, 21, 44, 22].
Although succeeded in image classification tasks, they still face a large challenge in balancing the model performance between old and new tasks when applied to our CVLM setting.
Although our
BMU-MoCo can be classified as a regularization-based method, it has a vital difference from existing regularization-based methods: benefiting from the bidirectional momentum updating process,
our
BMU-MoCo can directly utilize the holistic previous knowledge from the parameters of momentum encoders for model training ( i.e. , updating the encoders), and simultaneously update the momentum encoders at each training step to accommodate new tasks.
Methodology 3.1 Preliminary
We propose a new continual video-language modeling (CVLM) setting, where models are supposed to be sequentially trained on n video-text datasets D =
[ D 1 , D 2 , · · · , D n ] .
For each task t , it contains a dataset D t = { V i , T i } N t − 1 i =0 with N t video-text pairs, where V i denotes a video with S i frames and T i represents an English text description.
The target of CVLM is to learn a video encoder f
θ V and a text encoder f
θ T , which can respectively project the input video and its related text description into a joint embedding space with nearest metric distance.
Different from the classical VLM setting which only considers the model performance on the current dataset D t , our CVLM setting requires the models to prevent the catastrophic forgetting on the previously-used datasets [ D 1 , D 2 , · · · , D t − 1 ] ( t > 1) while also performing well on the current dataset D t .
Note that our proposed BMU-MoCo for CVLM is a memory-free method which only utilizes the current dataset D t for each task t (without the need of reloading the other datasets).
Therefore, without particular statement, we only consider task t with D t in the following subsections for formulation simplicity.
3.2 Network Architecture
Video Encoder.
We follow the most recent video-language modeling works [28, 56, 33, 35] to learn video representation by fusing the image embeddings of sampled frames per video.
Concretely, given each video V i with S i frames, we randomly sample s frames ( s < S i ) and embed them with an image encoder f img ( i.e. , ViT-Base [13]) to obtain the frame embeddings: [Equation] i
[Equation]
where x ji denotes the j -th sampled frames of video V i and F i,
jimg denotes its image embedding encoded by f img .
Then we project F i,jimg by a Linear layer f proj :
[Equation]
where F i,jproj ∈ R d denotes the projected d -dimensional image embedding of F i,jimg .
Following COTS [35] and HiT [33], we obtain the final video embedding of V i by adopting a fusing layer f avg to aggregate the image embeddings { F i,jimg } :
[Equation]
where f avg denotes an Average Pooling layer and
F iV ∈ R d is the video embedding of V i .
In summary, our video encoder f
θ V encodes the video inputs by adopting f img , f proj and f avg in Eqs.
Text Encoder.
For the language modality, we adopt BERT-Base [12] as our backbone to encode each input text T i .
In detail, we first tokenize T i into a sequence of text tokens [ l 1 i , l 2 i , · · · , l r i i ] , where r i denotes the length of T i .
Then we obtain the text token embeddings through the backbone f bert :
[Equation]
where F ibert denotes the token embeddings of T i obtained by f bert .
We then project them by a Linear layer ˆ f proj into the d -dimensional space as:
[Equation]
where F ibert [ j ] denotes the j -th element of F ibert , and ˆ F iproj [ j ] ∈ R d represents the projected text embedding of token j in T i (which has the same dimension d as video embedding F iV ).
To obtain the final text embedding of T i , we apply an Average Pooling layer
f avg :
[Equation]
where F iT ∈ R d denotes the text embedding of T i .
In summary, our text encoder f
θ T encodes the text inputs by adopting f bert , ˆ f proj , and f avg in Eqs.
3.3 BMU-MoCo
Cross-Modal MoCo.
Similar to the original single-modal MoCo [19], recent state-of-the-art videolanguage modeling works COTS [35] and HiT [33] construct a cross-modal MoCo architecture to maintain video/text momentum encoders by the same momentum update mechanism, which creates consistent queues for cross-modal contrastive learning objectives.
As illustrated in Figure 2, our
BMU-MoCo follows this paradigm and further transfers it to our CVLM setting.
Concretely, for a mini-batch of N B video-text pairs B =
{ V i , T i } N B i =1 , we first obtain the query embeddings q Vi , q Ti of
V i , T i by video encoder f
θ V and text encoder f
θ T :
[Equation]
Then we maintain two momentum encoders f θ V,m , f θ T,m (termed local momentum video/text en- coders in Figure 2) for both video and text modalities, whose parameters θ V,m , θ T,m are initialized by θ V , θ T at the beginning of each task t .
During the training process, θ V,m , θ T,m are continuously updated by θ V , θ T with the momentum update strategy:
[Equation]
queue queue 𝑉 1𝑘𝑒𝑦 𝑉 2𝑘𝑒𝑦 𝑉 3𝑘𝑒𝑦 ··· 𝑇 𝑞𝑢𝑒𝑟𝑦 𝑇 1𝑘𝑒𝑦 𝑇 2𝑘𝑒𝑦 𝑇 3𝑘𝑒𝑦 ··· 𝑉 𝑞𝑢𝑒𝑟𝑦 local momentumvideo encoder text encoder local momentumtext encoder similarity video encoder 𝑘 1𝑇 𝑘 2𝑇 𝑘 3𝑇 ··· , ෨𝑘 1𝑇 ෨𝑘 2𝑇 ෨𝑘 3𝑇 ··· 𝑞 𝑉 𝑞 𝑇 ෨𝑘 1𝑉 ෨𝑘 2𝑉 ෨𝑘 3𝑉 ··· , 𝑘 1𝑉 𝑘 2𝑉 𝑘 3𝑉 ··· 𝐿 𝑇2𝑉 similarity 𝐿 𝑉2𝑇 gradient gradient text encoder video encoder
Task t
Task t𝜃
𝑉 𝜃 𝑇 local momentumtext encoder local momentumvideo encoder global momentumtext encoder global momentumtext encoder global momentumvideo encoder global momentumvideo encoder 𝜃 𝑉,𝑚 𝜃 𝑇,𝑚
[Figure]
where m is the coefficient of momentum update.
To form the contrastive learning loss of cross-modal
MoCo, we need two consistent queues to preserve the negative video/text samples.
In detail, the key embeddings k Vi , k Ti of V
i , T i are firstly acquired by momentum video and text encoders:
[Equation]
We then respectively push k Vi and k Ti into the negative video queue Q V and the negative text queue
Q T (after computing loss), where Q V =
{ k V 1 , k V 2 , k V 3 , · · · k VN Q } and Q T =
{ k T 1 , k T 2 , k T 3 , · · · , k TN Q } ( N Q is the queue size).
The contrastive losses of cross-modal MoCo (Base-MoCo) are:
[Equation]
where τ is the temperature.
Note that the queue size N Q is decoupled from the batch size N B .
Therefore, it can take a large value for better representation of the data distribution.
Bidirectional Momentum Update.
Although achieving great success with non-streaming data ( e.g. , a single video-text dataset), the original cross-modal MoCo has difficulty in coping with the catastrophic forgetting problem under our CVLM setting.
To overcome this difficulty, we propose a novel bidirectional momentum update (BMU) strategy for cross-modal MoCo to review the old knowledge retained in momentum encoders at each training step.
Concretely, for video/text encoders f θ V , f θ T , in addition to the end-to-end update by back-propagation, we further update their parameters θ V , θ T using the parameters θ V,m , θ T,m of momentum encoders f θ V,m , f θ T,m by momentum update:
[Equation]
where ˆ m is a momentum coefficient, and θ V,m , θ T,m are simultaneously updated by Eq.
Together,
Eq.
(8) and Eq.
(12) compose our BMU strategy.
Note that the advantages of BMU lie in two aspects:
(1) At the beginning of each new task t , θ V,m and θ T, m
are respectively initialized by θ V and θ T , which makes the knowledge of task t − 1 be preserved.
During the training process, θ V,m and θ T,m are constantly and slowly updated by the momentum update strategy, which enables our model to review the old knowledge but without sacrificing the performance on new tasks.
Global Momentum Encoders.
To further enhance our BMU-MoCo, we propose to maintain a pair of global momentum encoders which can preserve earlier knowledge.
As shown in Figure 2, they are only initialized at the very beginning of the whole training process under our CVLM setting, and their parameters are transmitted across tasks.
Formally, let f ˜ θ V,m and f ˜ θ T,m denote the global momentum video and text encoders, respectively.
Their parameters ˜ θ V,m and ˜ θ T,m are updated by the BMU strategy along with the parameters θ V and θ T of encoders:
[Equation]
Note that Eq.
(12) and Eq.
(13) are implemented subsequently.
For each video-text input { V i , T i } , we obtain a new group of key embeddings ˜ k
Vi , ˜ k
Ti with the global momentum encoders f ˜ θ [Equation]
[Equation]
We push ˜ k
Vi and ˜ k
Ti respectively into two negative queues ˜ Q V and ˜ Q T , where ˜ Q V =
{ ˜ k V 1 , ˜ k
V 2 , ˜ k V 3 , · · · , ˜ k VN Q } , ˜ Q T = { ˜ k T 1 , ˜ k T 2 , ˜ k T 3 , · · · , ˜ k TN Q } .
Note that each query embedding ( e.g. , q Ti ) has two corresponding positive embeddings ( k Vi , ˜ k
Vi ) and two corresponding negative queues ( Q V , ˜ Q V ).
The cross-modal contrastive losses are defined as:
[Equation]
where τ is the temperature.
Now we have the final loss of BMU-MoCo for our CVLM setting:
[Equation]
The full (pseudocode) algorithm of our BMU-MoCo is presented in the supplementary material.
Experiments 4.1 Experimental Setup
Datasets.
Our CVLM setting is defined over a sequence of five video-text datasets: (1) VATEX [54] is a large-scale open-domain dataset, which has 25,991 videos with 250K text descriptions for training, 3,000 videos for validation and 6,000 videos for testing.
(2) ActivityNet [25] is an action domain dataset, which consists of 20K YouTube videos with 100K text descriptions.
We follow the standard setting in [4, 28] to use 10K videos for training and 4.9K for test (the val1 split), where all texts of each video are concatenated into one query paragraph.
(3) MSR-VTT [55] contains 10K videos, with 20 text descriptions per video.
We follow the 1k-A split in recent works [28, 4, 56, 33] with 9K training videos and 1K test videos.
(4) DiDeMo
[20] consists of 10K Flickr videos with 40K text annotations.
Following [4, 28], we train and evaluate our model on paragraph-to-video retrieval (the same setting for ActivityNet).
(5) MSVD [10] has 1,200 videos with 48K texts for training,videos for validation and 670 ones for testing.
Overall, there are around 50K videos with 500K text descriptions in all five datasets ( i.e. , each dataset has 100K video-text pairs in average).
Evaluation Metrics.
Similar to the standard video-language modeling setting, we evaluate the text-to-video retrieval performance of a model on Recall@1 (shortened as R@1).
R@1 refers to the percentage of text queries that correctly retrieve the ground-truth candidate at top-1.
For our CVLM setting, we further define two evaluation metrics: forgetting rate (FR) and harmonic mean (HM).
Formally, let M i denote the model after trained on task i and A it ( t ≤ i ) denotes the R@1 result of
M i on task t .
The overall R@1 ( 1 n (cid:80) n t =1 A n t ) is the average R@1 results of the final model M n on all tasks.
Based on these notations, we then define the FR and HM as follows:
(1) Forgetting rate (FR) of M i on task t ( t ≤ i ) is the performance decrease between M i and M t : FR
=
A tt − A it , where a lower FR indicates the model forgets less knowledge.
Note that there is no catastrophic forgetting (FR = 0 ) when t = i .
The Overall FR is obtained by just summing the FR results of the final model M n across all tasks: Overall FR = (cid:80) nt[-=
-]
=1 ( A tt − A nt ) .
[Table]
[Table]
(2) Harmonic mean (HM) calculates the harmonic mean of the overall R@1 (current) and the overall
R@1 (final), where the overall R@1 (current) denotes the average of the R@1 values obtained by each current model M i ( i = 1 , 2 , · · · , n ) on each current task i , and the overall R@1 (final) denotes the average R@1 for the final model M n on all tasks.
Formally, we have: HM
[Table]
Note that HM can alleviate the trade-off problem between overall R@1 (current) and overall R@ (final), which is otherwise an inherent limitation of FR.
Specifically, when a model has a lower overall R@1 (current) and a lower overall R@1 (final), it could also have a better/lower FR (which is unsatisfactory) but still lead to a worse/lower HM (see Figure 4(c)).
Implementation Details.
Recent works on vision + language demonstrate that video-language models benefit from image-text pre-training [28, 4], which can accelerate the model convergence and is more suitable for the real-world application scenarios.
We thus apply ViT-Base [13]/BERTBase [12] as our image/text encoder, and follow the recent state-of-the-art MoCo-based model COTS [35] to pre-train our model with 5.3M image-text pairs.
We then sequentially train all the models (BMU-MoCo and all competitors) on five video-text datasets/tasks.
For each task, we train a model for 10 epochs and choose the best trained one w.r.t. the validation R@1 results.
For those competitors using a memory buffer during training ( e.g. , ER-ring [9]), we set the memory size to 10% of the average data size 100K ( i.e. , 10K video-text pairs).
Note that the percentage 10% is larger than the buffer size of most recent rehearsal-based continual learning methods [9, 5, 37].
More implementation details are given as follows: (1) In the training phase, all sampled frames of each video are resized to 384 × 384 and augmented by gray-scaling and color-jitter. For the first epoch of each task under our CVLM setting, we set the learning rate to 5e-5 and decay it to 5e-6 afterwards.
(3) We select the two momentum coefficients m = 0 .
99 , ˆ m = 0 .
99 , and the temperature τ = 0 .
We set the batch size N B to 48 and the queue size N Q to 1,440.
The total training time on five tasks is around 20 hours with 8 Tesla V100 GPUs for each model.
4.2 Main Results
Table 1 summarizes the comparative results in terms of text-to-video retrieval (R@1), forgetting rate and harmonic mean (HM) obtained by the final model M n (per method) on five datasets.
We re-implement five recent continual learning methods (fused with cross-modal MoCo) under our
CVLM setting, including rehearsal-based methods (ER-ring [9], DER [5]), regularization-based methods (LwF [32]) and their combinations (Co2L [6], LUMP [37]).
We can observe that: (1)
Our BMU-MoCo outperforms recent methods by large margins without using any extra memory or dynamically extended networks.
Concretely, our method achieves the best R@1 and FR results on all tasks, and outperforms the second best by 2 .
93% for overall R@1, 11 .
85% for overall FR and 1 .
78% for overall HM.
(2) Without applying global momentum encoders, our BMU-MoCo (local) also beats all competitors, directly showing the effectiveness of our BMU strategy.
The improvements over
Base-MoCO obtained by utilizing knowledge distillation ( e.g. , LwF [32]) or extra memory data ( e.g. ,
Co2L [6]) are limited due to the lack of category information under our CVLM setting.
35 40 45 50 55    5 R @ 1 ( % )
Task index 15 20 25 30 35   5 R @ 1 ( % )
Task index 25 27 29 31 33  5 R @ 1 ( % ) Task index 27 29 31 33 35 5 R @ 1 ( % )
Task index VATEX
ActivityNet
MSR-VTT
DiDeMo 36 38 40 42 44 46 5 R @ 1 ( % ) Task index Base-MoCo ER Co2L BMU-MoCo MSVD
[Table] [Figure]
[Figure] [Table] index R @
[Table] [Table]
index
VATEX
ActivityNet
Base-MoCo
ER-ring
Co2L
BMU-MoCo
[Table]
Figure 3 shows more detailed comparative results for text-to-video retrieval (R@1) obtained by each model M i (per method) on the first two datasets (VATEX [54] and ActivityNet [25]).
We compare our BMU-MoCo with three representative competitors, including Base-MoCo [19], ER-ring [9], and
Co2L [6].
Concretely, the left sub-figure presents the results of M 1 ∼ M 5 (per method) on task(VATEX), i.e. , A i 1 (1 ≤ i ≤ 5) .
The right sub-figure presents the results of M 2 ∼ M 5 (per method) on task 2 (ActivityNet), i.e. , A i 2 (2 ≤ i ≤ 5) .
It can be observed that: (1) For task 1 (VATEX), the performance of our BMU-MoCo drops the most slowly after it is trained on the following tasks (task 2 to task 5).
(2) For task 2 (ActivityNet), our BMU-MoCo also leads to the slowest performance drop after trained on the following tasks (task 3 to task 5).
Overall, our BMU-MoCo indeed significantly alleviates the performance decrease problem during the whole training process.
4.3 Ablation Study
We first conduct ablation study to demonstrate the contributions of the BMU strategy, the local momentum encoders and the global momentum encoders applied in our BMU-MoCo.
The ablative results are shown in Table 2.
It can be clearly seen that: (1) With our BMU strategy, our model achieves remarkable improvements (4th row vs. 1st row).
(2) Simultaneously applying local and global encoders is better than using only one of them (3rd row vs. 1st/2nd row), which indicates that the knowledge preserved in local and global momentum encoders are quite different and thus complementary to each other.
(3) Our BMU strategy helps our model to excavate old knowledge preserved in different momentum encoders (6th row vs. 3rd row) and achieve the best performance (6th row vs. 4th/5th row), which further validates the effectiveness of our BMU.
Considering the core role of BMU, we thus analyze the impact of the momentum coefficient ˆ m utilized in our BMU-MoCo.
According to COTS [35], the other momentum coefficient m of our model is fixed at 0.99 (only the value of ˆ m is changed).
Figure 4 shows the results for overall R@ (current), overall R@1 (final), and overall HM, respectively.
We find that the value of ˆ m cannot be too big or too small.
Concretely, when ˆ m is too big ( e.g. , 0.999 and 1), the knowledge preserved in momentum encoders cannot be well-reviewed by our model.
When ˆ m is too small ( e.g. , 0.9), the end-to-end update by back-propagation is influenced too much, which leads to bad results for
O v e r a ll
R @ 1 ( c u rr e n t ) 35.39 35.
O v e r a ll
R @ 1 (f i n a l ) 36.82 37.59 36.
O v e r a ll
H M
[Figure]
O v e r a ll F R memory size (%)
ER-ring
BMU-MoCo 30.
[Figure]
overall R@1 (current) and overall HM. It is worth mentioning that the model with smaller ˆ m (0.9) has lower/better FR (14.95) since it sacrifices the model performance on overall R@1 (current).
This phenomenon demonstrates the necessity of utilizing the overall HM to measure the overall (trade-off) model performance.
Therefore, we set the momentum coefficient ˆ m to 0.99 in all our experiments, which helps our model to review old knowledge while learning well on new tasks.
4.4 Further Evaluation
To demonstrate both the efficiency and effectiveness of our BMU-MoCo under our CVLM setting, we compare our model with a representative rehearsal-based method ER-ring [9] by different memory sizes in Figure 5.
Note that our BMU-MoCo has two global momentum encoders that need 0.5GB more storage space than the original cross-modal MoCo (used by all competitors including ER-ring).
As shown in Figure 5, when the memory size of ER-ring becomes 0.05%, it equals to the size of extra storage space used by our BMU-MoCo (but our model performs significantly better).
In real-world application scenarios, the memory size of rehearsal-based methods like ER-ring enlarges rapidly as the number of tasks grows, while the fixed extra space size (0.5GB) of our BMU-MoCo is negligible.
More importantly, our BMU-MoCo (with a fixed 0.5GB sapce size) even outperforms ER-ring using 10% memory (about 200GB under our CVLM setting) by large margins on both overall R@1 (final) and overall FR.
This directly indicates the efficiency and effectiveness of our BMU-MoCo.
Note that the key factor of our BMU-MoCo is to learn holistic knowledge from momentum encoders by momentum update.
This momentum update strategy has two characteristics: (1) encoders are updated on each iteration by momentum update and (2) momentum encoders are simultaneously updated. We thus consider two alternative strategies to further validate the effectiveness of our BMU-MoCo in Table 3: (1) BMU-MoCo-ensemble updates the encoders by adding the parameters of the momentum encoders at the end of each task ( i.e. , ensemble), which is different from the iteration-by-iteration strategy of our BMU-MoCo. It can be observed that BMU-MoCo-ensemble learns better than the baseline method (Base-MoCo) but still has a large performance gap compared with our BMU-MoCo.
(2) BMU-MoCo-fixed fixes the momentum encoders during each task (only re-initialized by encoders at the beginning of each task) and utilizes it to update the encoders in
[Table]
[Table]
the same iteration-by-iteration way of our BMU-MoCo.
It can be observed that BMU-MoCo-fixed focuses on reviewing the old knowledge but sacrifices the model performance on current tasks, which leads to worse overall R@1 (final) and HM.
Therefore, considering all evaluation metrics (R@1, FR, and HM), the update strategy (for encoders) of our BMU-MoCo is better than its alternatives.
Finally, to show the transferability of our BMU-MoCo under other continual learning settings, we apply it to the unsupervised continual learning (UCL) setting which is originally proposed in
LUMP [37].
This UCL setting includes three benchmark datasets for training and evaluation: (1)
Split CIFAR-10 [26] consists of five tasks with 2 random classes per task; (2) Split CIFAR-100 [26] consists of 20 tasks with 5 random classes per task; (3) Split Tiny-ImageNet consists of 20 tasks with 5 random classes (out of 100 classes from ImageNet [11]) per task.
The comparative results of UCL are shown in Table 4.
We can observe that our BMU-MoCo outperforms all the competitors including
PNN [48], SI [60], DER [5], and LUMP [37], showing the effectiveness of our BMU-MoCo under the UCL setting.
This also demonstrates the general applicability of our BMU strategy.
Conclusion
In this paper, we propose a new continual video-language modeling (CVLM) setting, where models are supposed to be sequentially trained on five widely-used video-text datasets.
To overcome the catastrophic forgetting and heavy resource consumption challenges, we propose a novel framework
BMU-MoCo, which is a cross-modal MoCo-based model with bidirectional momentum update (BMU).
We maintain both local and global momentum encoders with our BMU strategy to review broader old knowledge while learning on new tasks.
Extensive experimental results show that
our
BMU-MoCo outperforms recent competitors by large margins, even without using extra memory data or dynamically extended networks.
The limitation of our work lies in that we have only evaluated
BMU-MoCo under the CVLM setting, and thus we need to transfer it to other continual learning settings ( e.g. , continual image-text pre-training) for comprehensive study. Acknowledgments and Disclosure of Funding
This work was supported in part by National Natural Science Foundation of China (61976220 and 61832017), Beijing Outstanding Young Scientist Program (BJJWZYJH012019100020098), and the
Research Seed Funds of School of Interdisciplinary Studies, Renmin University of China.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Please see Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Please see Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Please see Section 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cited the creators of all five datasets in Section 4.
(b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]