BMU-MoCo: Bidirectional Momentum Update for Continual Video-Language Modeling
Anonymous Author(s)
AffiliationAddress email
Abstract
Video-language models suffer from forgetting old/learned knowledge when trainedwith streaming data.
In this work, we thus propose a continual video-language modeling (CVLM) setting, where models are supposed to be sequentially trained on fivewidely-used video-text datasets with different data distributions.
Although most ofexisting continual learning methods have achieved great success by exploiting extrainformation ( e.g. , memory data of past tasks) or dynamically extended networks,they cause enormous resource consumption when transferred to our CVLM setting.
To overcome the challenges ( i.e. , catastrophic forgetting and heavy resource consumption) in CVLM, we propose a novel cross-modal MoCo-based model withbidirectional momentum update (BMU), termed BMU-MoCo.
Concretely, our
BMU-MoCo has two core designs: (1) Different from the conventional MoCo, weapply the momentum update to not only momentum encoders but also encoders( i.e. , bidirectional) at each training step, which enables the model to review thelearned knowledge retained in the momentum encoders.
(2) To further enhanceour BMU-MoCo by utilizing earlier knowledge, we additionally maintain a pair ofglobal momentum encoders (only initialized at the very beginning) with the same
BMU strategy.
Extensive results show that our BMU-MoCo remarkably outperforms recent competitors w.r.t.
video-text retrieval performance and forgetting rate,even without using any extra data or dynamic networks.
Introduction
Existing video-language modeling (VLM) methods have achieved promising performance for videotext retrieval [56, 36, 58, 27, 22, 50, 25, 4] with non-streaming data.
However, in real-worldapplication scenarios, VLM models need to evolve with streaming data ( e g. , collected from the
Internet [36, 39]) to accommodate more tasks.
Under this setting, since it costs too much resource toretrain the model with both old and new data for each task, a common practice is to fine-tune VLMmodels with only the newly-arrived data.
Note that such model fine-tuning leads to severe performancedegradation on previous tasks.
This is a well-documented phenomenon called catastrophic forgetting[16, 35] under the conventional continual learning setting [45, 42, 31, 14, 21, 57].
Therefore, in this work, we propose a continual video-language modeling (CVLM) setting to bettersimulate the realistic scenario.
Under our CVLM setting, models are supposed to be sequentiallytrained on five widely-used video-text datasets: VATEX [51], ActivityNet [23], MSR-VTT [52],
DiDeMo [18], and MSVD [10].
An evaluation protocol is also established for CVLM, which containsthree metrics to respectively measure the text-to-video retrieval performance (Recall@1, shortened
[Figure] [Figure]
final model
[Figure]
as R@1), forgetting rate (FR), and harmonic mean (HM) performance.
Moreover, we implementa basic cross-modal MoCo [17] model (Base-MoCo) as our baseline method since it has shownsuperiority on video-language modeling [30, 32].
As illustrated in Figure 1(a), we observe that inspite of achieving great R@1 results with current models (evaluated right after trained on each task),the performance of the final Base-MoCo model (trained across all five tasks) drops significantly.
To tackle the catastrophic forgetting problem, most recent continual learning works attempt topreserve the learned knowledge from a variety of perspectives: (1) Maintaining a memory buffer tosave and exploit data from previous tasks [43, 5, 31, 3, 8, 42]; (2) Generating pseudo data of learnedtasks [48, 24, 40, 54];
(3) Extending the network architecture dynamically as each new task arrives[45, 14, 1, 7, 28].
However, when these methods are transferred to our CVLM setting, the resourceconsumption is enlarged rapidly as the number of tasks grows, due to the characteristic of video data.
In addition, another branch of continual learning works focus on imposing a regularization constraintwith quadratic penalty [21, 57, 13, 46] or knowledge distillation [29, 2, 19, 41, 20], which leads toan unwanted trade-off on the performance of old and new tasks with limited neural resources [37].
Therefore, it is a long-standing and arduous challenge to train a video-language modeling networkunder the CVLM setting with both effectiveness and efficiency taken into consideration.
To overcome this challenge, we devise BMU-MoCo, a cross-modal MoCo-based model with anovel bidirectional momentum update (BMU) strategy.
As shown in Figure 1(b), our BMU-MoConeeds neither extra memory data nor dynamically extended neural networks.
Concretely, similar tocross-modal MoCo applied in [30, 32], our BMU-MoCo has a video encoder ( i.e. , ViT-Base [12])and a text encoder ( i.e. , BERT-Base [11]), followed by the momentum video/text encoders.
Differentfrom the original MoCo [17] and its cross-modal versions [30, 32] that utilize momentum update foronly momentum encoders to maintain a consistent queue, our BMU strategy imposes momentumupdate on both momentum encoders and encoders.
As a result, at each training step, the encoders ofour BMU-MoCo learn new knowledge by end-to-end update with back-propagation whilst reviewingold knowledge directly from the parameters of momentum encoders by momentum update.
In ouropinion, our BMU-MoCo outperforms existing methods for two main reasons: (1) Momentumencoders are initialized by current encoders at the beginning of each new task and then progressslowly, which helps our model preserve adequate old knowledge without sacrificing the performanceon new tasks; (2) Since there is no category information under the CVLM setting, learning frommemory data or distilling with a batch of new data only absorbs part of previous knowledge whileour BMU-MoCo learns holistic knowledge directly from the parameters of momentum encoders.
Tofurther enhance our BMU-MoCo, we also maintain a pair of global (cross-task) momentum encoders with the same BMU strategy, which are only initialized at the very beginning and thus preserve earlier knowledge than the normal local (task-specific) momentum encoders
Our main contributions are four-fold: (1) We propose a new continual video-language modeling(CVLM) setting, where models are supposed to be sequentially trained on five widely-used video-textdatasets.
(2) To effectively and efficiently overcome the catastrophic forgetting problem under the
CVLM setting, we devise BMU-MoCo, a cross-modal MoCo-based model with a novel bidirectionalmomentum update (BMU) strategy.
It can review holistic old knowledge directly from the parametersof momentum encoders while learning on new tasks.
(3) To further boost our BMU-MoCo, a pair ofglobal momentum encoders are maintained by the same BMU strategy to preserve and review earlierknowledge.
(4) Extensive results demonstrate that our BMU-MoCo outperforms recent continuallearning methods by large margins w.r.t.
both text-to-video retrieval performance and forgetting rate,even without any extra memory data or dynamically extended networks.
Related Work
Video-Language Modeling.
Most existing methods for video-language modeling follow twoparadigms: (1) Single-stream methods [33, 58, 49, 26, 25, 53] typically include a multi-modal transformer to achieve fine-grained cross-modal interaction between the video and language modalities.
Although achieving great performance, they suffer from the huge time complexity caused by the pairwise inputs during inference, which makes them unsuitable for practical applications.
(2) Two-streammethods [15, 38, 4, 30, 32] learn video and text representations independently, and align them afterencoding.
To ensure the inference efficiency, both the baseline method ( i.e. , Base-MoCo) and
our
BMU-MoCo for CVLM are set to be two-stream methods.
Importantly, different from Base-MoCo,our BMU-MoCo has a novel BMU strategy to address the catastrophic forgetting problem and twoextra global momentum encoders to further boost the model performance.
Continual Learning.
Conventional continual learning methods mainly focus on image classificationtasks.
They can be roughly categorized into three groups: (1) Rehearsal-based methods apply extramemory to store sampled data [42, 31, 43, 9, 5, 3, 8, 6] or generate pseudo data [48, 24, 40, 54]from previous tasks.
The memory size and the training complexity tend to be enlarged significantlyas the number of tasks grows.
(2) Expansion-based methods either add extra extended networksfor new tasks [45, 14, 1, 7, 28] or select partial model parameters to update for different tasks[55, 44, 47].
They need more computational resources especially for a long sequence of training tasks(e.g.
, under our CVLM setting).
(3) Regularization-based methods modify the model parameterswith quadratic loss penalty [21, 57, 13, 46] or knowledge distillation constraints [29, 2, 19, 41, 20].
Although succeeded in image classification tasks, they still face a large challenge in balancing themodel performance between old and new tasks when applied to our CVLM setting.
Although our
BMU-MoCo can be classified as a regularization-based method, it has a vital difference from existingregularization-based methods: benefiting from the bidirectional momentum updating process,
our
BMU-MoCo can directly utilize the holistic previous knowledge from the parameters of momentumencoders for model training ( i.e. , updating the encoders), and simultaneously update the momentumencoders at each training step to accommodate new tasks.
Methodology
3.1 Preliminary
We propose a new continual video-language modeling (CVLM) setting, where models are supposedto be sequentially trained on n video-text datasets D =
[ D 1 , D 2 , · · · , D n ] .
For each task t , it containsa dataset D t = { V i , T i } N t − 1 i = with N t video-text pairs, where V i denotes a video with S i framesand T i represents an English text.
The target of CVLM is to learn a video encoder f
θ V and atext encoder f
θ T , which can respectively project the input video and its related text into a jointembedding space with nearest metric distance.
Different from the classical VLM setting which onlyconsiders the model performance on the current dataset D t , our CVLM setting requires the modelsto prevent the catastrophic forgetting on previously-used datasets [ D 1 , D 2 , · · · , D t − 1 ] ( t > 1) whilealso performing well on the current dataset D t .
Note that our proposed BMU-MoCo for CVLM is amemory-free method which only utilizes the current dataset D t for each task t .
Therefore, withoutparticular statement, we only consider task t with D t in the following subsections for simplicity.
Video Encoder.
We follow the most recent video-language modeling works [25, 53, 30, 32] tolearn video representation by fusing the image embeddings of sampled frames per video.
Concretely,given each video V i with S i frames, we randomly sample s frames ( s < S i ) and embed them with animage encoder f img ( i.e. , ViT-Base [12]) to obtain the frame embeddings:
[Equation]
) where x ji denotes the j -th sampled frames of video V i and F i,
jimg denotes its image embedding encodedby f img .
Then we project F i,jimg by a Linear layer f proj :
[Equation]
where F i,jproj ∈ R d denotes the projected d -dimensional image embedding of F i,jimg .
Following COTS[32] and HiT [30], we obtain the final video embedding of V i by adopting a fusing layer f avg toaggregate the image embeddings { F i,jimg } :
[Equation]
where f avg denotes an Average Pooling layer and
F iV ∈ R d is the video embedding of V i .
In summary,our video encoder f
θ V encodes the video inputs by adopting f img , f proj and f avg in Eqs.
Text Encoder.
For the language modality, we adopt BERT-Base [11] as our backbone to encodeeach input text T i .
In detail, we first tokenize T i into a sequence of text tokens [ l 1 i , l 2 i , · · · , l r i i ] , wherer i denotes the length of T i .
Then we obtain the text token embeddings through the backbone f bert :
[Equation]
where F ibert denotes the token embeddings of T i obtained by f bert .
We then project them by a Linearlayer ˆ f proj into the d -dimensional space as:
[Equation]
where F ibert [ j ] denotes the j -th element of F ibert , and ˆ F iproj [ j ] ∈ R d represents the projected textembedding of token j in T i (which has the same dimension d as video embedding F iV ).
To obtain thefinal text embedding of T i , we apply an Average Pooling layer
f avg :
[Equation]
where F iT ∈ R d denotes the text embedding of T i .
In summary, our text encoder f
θ T encodes the textinputs by adopting f bert , ˆ f proj , and f avg in Eqs.
3.3 BMU-MoCo
Cross-Modal MoCo.
Similar to the original single-modal MoCo [17], recent state-of-the-art videolanguage modeling works COTS [32] and HiT [30] construct a cross-modal MoCo architecture tomaintain video/text momentum encoders by the same momentum update mechanism, which createsconsistent queues for cross-modal contrastive learning objectives.
As illustrated in Figure 2, our
BMU-MoCo follows this paradigm and further transfers it to our CVLM setting.
Concretely, for amini-batch of N B video-text pairs B =
{ V i , T i } N B i =1 , we first obtain the query embeddings q Vi , q Ti of
V i , T i by video encoder f
θ V and text encoder f
θ T :
[Equation]
Then we maintain two momentum encoders f θ V,m , f θ T,m (termed local momentum video/text encoders in Figure 2) for both video and text modalities, whose parameters θ V,m , θ T,m are initializedby θ V , θ T at the beginning of each task t .
During the training process, θ V,m , θ T,m are continuouslyupdated by θ V , θ T with the momentum update strategy:
[Equation]
[Figure]
𝑉 1𝑘𝑒𝑦 𝑉 2𝑘𝑒𝑦 𝑉 3𝑘𝑒𝑦 ··· 𝑇 𝑞𝑢𝑒𝑟𝑦 𝑇 1𝑘𝑒𝑦 𝑇 2𝑘𝑒𝑦 𝑇 3𝑘𝑒𝑦 ··· 𝑉 𝑞𝑢𝑒𝑟𝑦 local momentumvideo encoder text encoder local momentumtext encoder similarity video encoder 𝑘 1𝑇 𝑘 2𝑇 𝑘 3𝑇 ··· , ෨𝑘 1𝑇 ෨𝑘 2𝑇 ෨𝑘 3𝑇 ··· 𝑞 𝑉 𝑞 𝑇 ෨𝑘 1𝑉 ෨𝑘 2𝑉 ෨𝑘 3𝑉 ··· , 𝑘 1𝑉 𝑘 2𝑉 𝑘 3𝑉 ··· 𝐿 𝑇2𝑉 similarity 𝐿 𝑉2𝑇 gradient gradient text encoder video encoder
[Figure]
t
Task t𝜃
𝑉 𝜃 𝑇 local momentumtext encoder local momentumvideo encoder global momentumtext encoder global momentumtext encoder global momentumvideo encoder global momentumvideo encoder 𝜃 𝑉,𝑚 𝜃 𝑇,𝑚
[Figure]
where m is the coefficient of momentum update.
To form the contrastive learning loss of cross-modal
MoCo, we need two consistent queues to preserve the negative video/text samples.
In detail, the keyembeddings k
Vi , k Ti of V i , T i are firstly acquired by momentum video and text encoders:
[Equation]
We then respectively push k Vi and k Ti into the negative video queue Q V and the negative text queue
Q T (after computing loss), where Q V =
{ k V 1 , k V 2 , k V 3 , · · · k VN Q } and Q T =
{ k T 1 , k T 2 , k T 3 , · · · , k TN Q }( N Q is the queue size).
The contrastive losses of cross-modal MoCo (Base-MoCo) are:
[Equation]
where τ is the temperature.
Note that the queue size N Q is decoupled from the batch size N B .
Therefore, it can take a large value for better representation of the data distribution.
Bidirectional Momentum Update.
Although achieving great success with non-streaming data( e.g. , a single video-text dataset), the original cross-modal MoCo has difficulty in coping with thecatastrophic forgetting problem under our CVLM setting.
To overcome this difficulty, we proposea novel bidirectional momentum update (BMU) strategy for cross-modal MoCo to review the oldknowledge retained in momentum encoders at each training step.
Concretely, for video/text encodersf θ V , f θ T , in addition to the end-to-end update by back-propagation, we further update their parametersθ V , θ T using the parameters θ V,m , θ T,m of momentum encoders f θ V,m , f θ T,m by momentum update:
[Equation]
where ˆ m is a momentum coefficient, and θ V,m , θ T,m are simultaneously updated by Eq.
Together,
Eq.
(8) and Eq.
(12) compose our BMU strategy.
Note that the advantages of BMU lie in two aspects:
(1) At the beginning of each new task
t , θ V,m and θ T,m are respectively initialized by θ V and θ T ,which makes the knowledge of task t − 1 be preserved.
During the training process, θ V,m andθ T,m are constantly and slowly updated by the momentum update strategy, which enables our modelto review the old knowledge but without sacrificing the performance on new tasks.
Global Momentum Encoders.
To further enhance our BMU-MoCo, we propose to maintain apair of global momentum encoders which can preserve earlier knowledge.
As shown in Figure 2,they are only initialized at the very beginning of the whole training process under our CVLM setting, and their parameters are transmitted across tasks.
Formally, let f ˜ θ V,m and f ˜ θ T,m denote the globalmomentum video and text encoders, respectively.
Their parameters ˜ θ V,m and ˜ θ T,m are updated bythe BMU strategy along with the parameters θ V and θ T of encoders:
[Equation]
[Equation]
Note that Eq.
(12) and Eq.
(13) are implemented subsequently.
For each video-text input { V i , T i } ,we obtain a new group of key embeddings ˜ k
Vi , ˜ k
Ti with the global momentum encoders f ˜ θ V,m , f ˜ θ T,m :
[Equation]
We push ˜ k
Vi and ˜ k
Ti respectively into two negative queues ˜ Q V and ˜ Q T , where ˜ Q V =
{ ˜ k V 1 , ˜ k
V 2 , ˜ k V 3 , · · · , ˜ k VN Q } , ˜ Q T = { ˜ k T 1 , ˜ k T 2 , ˜ k T 3 , · · · , ˜ k TN Q } .
Note that each query embedding ( e.g. ,q Ti ) has two corresponding positive embeddings ( k Vi , ˜ k
Vi ) and two corresponding negative queues( Q V , ˜ Q V ).
The cross-modal contrastive losses are defined as:
[Equation]
where τ is the temperature.
Now we have the final loss of BMU-MoCo for our CVLM setting:
[Equation]
The full (pseudocode) algorithm of our BMU-MoCo is presented in the supplementary material.
Experiments
4.1 Experimental Setup
Datasets.
Our CVLM setting is defined over a sequence of five video-text datasets: (1) VATEX [51]is a large-scale open-domain dataset, which has 25,991 videos with 250K text descriptions for training,3,000 videos for validation and 6,000 videos for testing.
(2) ActivityNet [23] is an action domaindataset, which consists of 20K YouTube videos with 100K text descriptions.
We follow the standardsetting in [4, 25] to use 10K videos for training and 4.9K for test (the val1 split), where all texts ofeach video are concatenated into one query paragraph.
(3) MSR-VTT [52] contains 10K videos,with 20 text descriptions per video.
We follow the 1k-A split in recent works [25, 4, 53, 30] with9K training videos and 1K test videos.
(4) DiDeMo
[18] consists of 10K Flickr videos with 40Ktext annotations.
Following [4, 25], we train and evaluate our model on paragraph-to-video retrieval(the same setting for ActivityNet).
(5) MSVD [10] has 1,200 videos with 48K texts for training, 100videos for validation and 670 ones for testing.
Overall, there are around 50K videos with 500K textdescriptions in all five datasets ( i.e. , each dataset has 100K video-text pairs in average).
Evaluation Metrics.
Similar to the standard video-language modeling setting, we evaluate thetext-to-video retrieval performance of a model on Recall@1 (shortened as R@1).
R@1 refers to thepercentage of text queries that correctly retrieve the ground-truth candidate at top-1.
For our CVLMsetting, we further define two evaluation metrics: forgetting rate (FR) and harmonic mean (HM).
Formally, let M i denotes the model after trained on task
i and A it ( t ≤ i ) denotes the R@1 result of
M i on task t .
The overall R@1 ( 1 n (cid:80) n t =1 A n t ) is the average R@1 results of the final model M n onall tasks.
Based on these notations, we then define the FR and HM as follows:
(1) Forgetting rate (FR) of M i on task t ( t ≤ i ) is the performance decrease between M i and M t :FR
=
A tt − A it , where a lower FR indicates the model forgets less knowledge.
Note that there is no
catastrophic forgetting (FR = 0 ) when t = i .
The Overall FR is obtained by just summing the FR
results of the final model M n across all tasks: Overall FR = (cid:80) nt =
( A tt − A nt ) .
[Table]
(2) Harmonic mean (HM) calculates the harmonic mean of the overall R@1 (current) and the overall
R@1 (final), where the overall R@1 (current) denotes the average of the R@1 values obtained by eachcurrent model M i ( i = 1 , 2 , · · · , n ) on each current task i , and the overall R@1 (final) denotes theaverage R@1 for the final model M n on all tasks.
Formally, we have: HM
[Table]
Note that HM can alleviate the trade-off problem between overall R@1 (current) and overall R@1(final), which is otherwise an inherent limitation of FR.
Specifically, when a model has a loweroverall R@1 (current) and a lower overall R@1 (final), it could also have a better/lower FR (which isunsatisfactory) but still lead to a worse/lower HM (see Figure 4(c)).
Implementation Details.
Recent works show that video-language models benefit from image-textpre-training [25, 4], which can accelerate the model convergence and is more suitable for the practicalscenarios.
We thus apply ViT-Base [12]/BERT-Base [11] as our image/text encoder and followrecent state-of-the-art MoCo-based model COTS [32] to pre-train our model with 5.3M image-textpairs.
We then sequentially train all the models (BMU-MoCo and all competitors) on five video-textdatasets/tasks.
For each task, we train a model for 10 epochs and choose the best trained one w.r.t.the validation R@1 results.
For those competitors using a memory buffer during training ( e.g. ,
ER-ring [9]), we set the memory size to 10% of the average data size 100K ( i.e. , 10K video-textpairs).
Note that the percentage 10% is larger than the buffer size of most recent rehearsal-basedcontinual learning methods [9, 5, 34].
More details are given as follows: (1) In the training phase, allsampled frames of each video are resized to 384 × 384 and augmented by gray-scaling and color-jitter.(2) For the first epoch of each task under our CVLM setting, we set the learning rate to 5e-5 anddecay it to 5e-6 afterwards.
(3) We select the two momentum coefficients m = 0 .
99 , ˆ m = 0 .
99 , andthe temperature τ = 0 .
We set the batch size N B to 48 and the queue size N Q to 1,440.
Thetotal training time on five tasks is around 20 hours with 8 Tesla V100 GPUs for each model.
4.2 Main Results
Table 1 summarizes the comparative results in terms of text-to-video retrieval (R@1), forgettingrate and harmonic mean (HM) obtained by the final model M n (per method) on five datasets.
Were-implement five recent continual learning methods (fused with cross-modal MoCo) under our
CVLM setting, including rehearsal-based methods (ER-ring [9], DER [5]), regularization-basedmethods (LwF [29]) and their combinations (Co2L [6], LUMP [34]).
We can observe that: (1)
Our BMU-MoCo outperforms recent methods by large margins without using any extra memory ordynamically extended networks.
Concretely, our method achieves the best R@1 and FR results on alltasks, and outperforms the second best by 2 .
93% for overall R@1, 11 .
85% for overall FR and 1 .
78%for overall HM.
(2) Without applying global momentum encoders, our BMU-MoCo (local) also beatsall competitors, directly showing the effectiveness of our BMU strategy.
The improvements over
Base-MoCO obtained by utilizing knowledge distillation ( e.g. , LwF [29]) or extra memory data ( e.g. ,
Co2L [6]) are limited due to the lack of category information under our CVLM setting.
35 40 45 50 55    5 R @ 1 ( % )
Task index 15 20 25 30 35   5 R @ 1 ( % )
Task index 25 27 29 31 33  5 R @ 1 ( % ) Task index 27 29 31 33 35 5 R @ 1 ( % )
Task index VATEX
ActivityNet
MSR-VTT
DiDeMo 36 38 40 42 44 46 5 R @ 1 ( % ) Task index Base-MoCo ER Co2L BMU-MoCo MSVD
[Figure]
[Table]
[Figure]
[Table] [Table]
Task index
VATEX
ActivityNet
Base-MoCo
ER-ring
Co2L
BMU-MoCo
[Table]
Figure 3 shows more detailed comparative results for text-to-video retrieval (R@1) obtained by eachmodel M i (per method) on the first two datasets (VATEX [51] and ActivityNet [23]).
We compareour BMU-MoCo with three representative competitors, including Base-MoCo [17], ER-ring [9], and
Co2L [6].
Concretely, the left sub-figure presents the results of M 1 ∼ M 5 (per method) on task 1(VATEX), i.e. , A i 1 (1 ≤ i ≤ 5) .
The right sub-figure presents the results of M 2 ∼ M 5 (per method)on task 2 (ActivityNet), i.e. , A i 2 (2 ≤ i ≤ 5) .
It can be observed that: (1) For task 1 (VATEX), theperformance of our BMU-MoCo drops the most slowly after it is trained on the following tasks (task2 to task 5).
(2) For task 2 (ActivityNet), our BMU-MoCo also leads to the slowest performance dropafter trained on the following tasks (task 3 to task 5).
Overall, our BMU-MoCo indeed significantlyalleviates the performance decrease problem during the whole training process.
4.3 Ablation Study
We first analyze the contributions of the BMU strategy, the local momentum encoders and the globalmomentum encoders applied in our BMU-MoCo.
The ablative results are shown in Table 2.
It can beclearly seen that: (1) With our BMU strategy, our model achieves remarkable improvements (4th rowvs.
1st row).
(2) Simultaneously applying local and global encoders is better than using only one ofthem (3rd row vs. 1st/2nd row), which indicates that the knowledge preserved in local and globalmomentum encoders are different.
(3) Our BMU strategy helps our model to excavate knowledgepreserved in different momentum encoders (6th row vs. 3rd row) and achieve the best performance(6th row vs. 4th/5th row), which further validates the effectiveness of our BMU.
Considering the core role of BMU, we thus analyze the impact of the momentum coefficient ˆ mutilized in our BMU-MoCo.
According to COTS [32], the other momentum coefficient m of ourmodel is fixed at 0.99 (only the value of ˆ m is changed).
Figure 4 shows the results for overall R@1(current), overall R@1 (final), and overall HM, respectively.
We find that the value of ˆ m cannot betoo big or too small.
Concretely, when ˆ m is too big ( e.g. , 0.999 and 1), the knowledge preservedin momentum encoders cannot be well-reviewed by our model.
When ˆ m is too small ( e.g. , 0.9),the end-to-end update by back-propagation is influenced too much, which leads to bad results foroverall R@1 (current) and overall HM.
It is worth mentioning that the model with smaller ˆ m (0.9)
O v e r a ll
R @ 1 ( c u rr e n t ) 35.39 35.
O v e r a ll
R @ 1 (f i n a l ) 36.82 37.59 36.
O v e r a ll
H M
[Figure]
BMU
[Figure] [Figure]
O v e r a ll F R memory size (%)
ER-ring
BMU-MoCo 30.
[Figure]
has lower/better FR (14.95) since it sacrifices the model performance on overall R@1 (current).
Thisphenomenon demonstrates the necessity of utilizing the overall HM to measure the overall (trade-off)model performance.
Therefore, we set the momentum coefficient ˆ m to 0.99 in all our experiments,which helps our model to review old knowledge while learning well on new tasks.
4.4 Further Evaluation
To demonstrate both the efficiency and effectiveness of our BMU-MoCo under our CVLM setting, wecompare our model with a representative rehearsal-based method ER-ring [9] by different memorysize in Figure 5.
Note that our BMU-MoCo has two global momentum encoders that need 0.5GBmore storage space than the original cross-modal MoCo (used by all competitors including ER-ring).
As shown in Figure 5, when the memory size of ER-ring becomes 0.05%, it equals to the size of extrastorage space used by our BMU-MoCo (but our model performs significantly better).
In real-worldapplication scenarios, the memory size of rehearsal-based methods like ER-ring enlarges rapidly asthe number of tasks grows, while the fixed extra space size (0.5GB) of our BMU-MoCo is negligible.
More importantly, our BMU-MoCo (with a fixed 0.5GB sapce size) even outperforms ER-ring using10% memory (about 200GB under our CVLM setting) by large margins for both overall R@1 (final)and overall FR.
This directly indicates the efficiency and effectiveness of our BMU-MoCo.
Conclusion
In this paper, we propose a new continual video-language modeling (CVLM) setting, where modelsare supposed to be sequentially trained on five widely-used video-text datasets.
To overcome thecatastrophic forgetting and heavy resource consumption challenges, we propose a novel framework
BMU-MoCo, which is a cross-modal MoCo-based model with bidirectional momentum update(BMU).
We maintain both local and global momentum encoders with our BMU strategy to reviewbroader old knowledge while learning on new tasks.
Extensive experimental results show that
our
BMU-MoCo outperforms recent competitors by large margins, even without using extra memory dataor dynamically extended networks.
The limitation of our work lies in that we have only evaluated
BMU-MoCo under the CVLM setting, and thus we need to transfer it to other continual learningsettings (
e.g. , continual image-text pre-training) for comprehensive study.
Checklist
For all authors...(
a
) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](
b
) Did you describe the limitations of your work?
[Yes] Please see Section 5.
(c)
Did
you
discuss
any potential negative societal impacts of your work
[No]
(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem
[Yes
If you are including theoretical results...
(a) Did
you state the full set of assumptions of all theoretical results
[N/A]
(b) Did
you include complete proofs of all theoretical results
[N/A]
If
you ran experiments
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Please see Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please see Section 4.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited the creators
of all five datasets in Section 4.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]