Retaining Knowledge for Learning with Dynamic Deﬁnition
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Machine learning models are often deployed in settings where they must be constantly updated in response to the changes in class deﬁnitions while retaining highaccuracy on previous learned deﬁnitions.
A classical use case is the task of frauddetection, where new fraud scheme comes one after another.
While such an updatecan be accomplished by re-training on the complete data, the process is inefﬁcient and prevents real-time and on-device learning.
On the other hand, efﬁcientmethods that incrementally learn from new data often result in the forgetting ofpreviously-learned knowledge.
We formally deﬁne this problem as Learning with
Dynamic Deﬁnition (LDD) and demonstrate that popular models, such as the Vision Transformer and Roberta, exhibit substantial forgetting of past deﬁnitions.
Wepresent a ﬁrst practical and provable solution to LDD.
Our proposal is a hash-basedmemory model RIDDLE that solves evolving deﬁnitions by associating differentinstances only to relevant parameters.
We prove that our model is a universalfunction approximater and theoretically bounds the knowledge lost during theupdate process.
On practical tasks with evolving class deﬁnition in vision and
NLP, our model outperforms baselines by up to 30% on the original dataset whileproviding competitive accuracy on the update dataset.
Introduction
Motivation: A common machine learning workﬂow is to ﬁrst develop a task objective basedon historical experience, then deploy a model trained to optimize the objective over a dataset.
However, machine learning models rarely operate in such a static environment.
Instead, models mustbe dynamically updated to accommodate changes in class deﬁnitions without forgetting previousknowledge.
Consider the case of fraud detection, where models must adapt to emerging scam patternswhile also providing protection against previously-known patterns.
Also, machine learning modelsare used to identify harmful microbes given gene sequences.
And it is important to perform well onnew variants or newly discovered microorganisms.
A similar situation arises when models are used toﬁlter inappropriate content such as hate speech and violence.
New phrases or previously-innocuouslanguage may become inappropriate due to world events or attempts by users to circumvent theﬁlter.
While the supervised task remains the same (fraud or safe, harmful or not, inappropriate orappropriate, etc.), class deﬁnition evolves over time and likely differ signiﬁcantly from the point ofthe ﬁrst training.
We refer to this setting as Learning under Dynamic Deﬁnition ( LDD).
It should benoted that this notion is very different for several other notions of distribution shift in the literaturewhich we review in the next section.
A standard solution is to retrain the model on all available data (original dataset and data for the new deﬁnitions).
This methodology is inefﬁcient due to data storage and extra computation, and evenprohibitive in large data or streaming scenarios.
Furthermore, it prevents real-time and on-devicelearning.
An ideal and efﬁcient solution is to update the trained model incrementally only with data for the new deﬁnitions (e.g. using online gradient descent).
However, this is a challenging process due to the tendency of machine learning models, in particular neural networks, to forget previous knowledge.
This situation, known as collapse/catastrophic forgetting [25], causes performance on theold deﬁnitions to decay.
Our Proposal: Catastrophic forgetting occurs because gradient updates from training examples in theupdate process are able to change parameters that are tuned for optimal performance on the originaldataset [29].
Inspired by recent connections between random partitions and distribution models [20],we propose RIDDLE ( Retaining Information in a Dynamically Deﬁned Learning Environment), anovel architecture with meaningful parameter groupings.
RIDDLE works by partitioning the inputspace and assigning a set of parameters to each partition.
Gradient updates are forced to avoidparameters that are critical to prior knowledge.
RIDDLE can be plugged into advanced networkarchitectures, just like a fully connect layer, to ease the memory loss while utilize the learned features,as shown in Figure 2.
We summarize our technical contributions below.
• We introduce the RIDDLE (in Section 3), consisting of a set of parameters indexed usinglocality-sensitive hash functions.
RIDDLE is computationally efﬁcient and requires less memory than popular machine learning models with similar accuracy.
• We prove that RIDDLE is universal function approximator and can therefore represent
any function( Section 4).
We prove a formal connection between parameter usage anddistribution similarity in Theorem 3.2.
• In Section 5, we apply RIDDLE in the dynamic deﬁnition setting for four real-world tasks.
Our method outperforms baselines by up to 30% on the original dataset while achieving
competitive accuracy on the new dataset.
Related Work
Continual Learning:
Continual learning refers to learning and remembering multiple tasks froma stream of information.
Current approaches can be split into three main categories (1) memoryreplay to complement neural networks [42, 28], (2) constraints on network updates [36, 27, 49],and (3) dynamic architectural adjustments to accommodate new information, such as additionallayers [46, 58].
Recently, few-shot learning methods are also applied into continue learning to solvesequential learning with limited data [9, 55, 50].
Continual learning methods encounter similarchallenges to the ones presented by learning under dynamic distribution, most notably catastrophicforgetting.
However, continual learning attempts to solve a sequence of related tasks, often expressedas task embeddings, over data that does not necessarily reﬂect changes in class deﬁnitions.
Train-Test Distribution Shift: Transfer learning and test-time adaptation attempt to remedy misalignments between the training and test distribution.
Although such methods may seem relevant inlearning with dynamic deﬁnition, they have dramatically different goals.
Transfer learning attempts toadapt a model from a data-rich source domain to a data-scarce target domain, without regard for theperformance on the source domain [62, 53, 56].
While several works on train-test distribution shiftfocus on robustness, it is from the perspective of out-of-distribution detection [24, 37] or uncertaintyestimation [41, 31, 54].
Our objective is to provide models that perform well on both the originaldataset and new dataset, where class deﬁnitions are different, rather than simply detect or adapt to thedistribution change.
Online Learning:
Online learning regards optimization as a process where a model is trained byinteracting with a stream of data.
Online learning algorithms attempt to minimize the regret , ordifference between the loss of the predictions by the incrementally-trained model and the loss of theglobally optimal model [45, 43].
Online learning methods attain minimal regret against adaptiveadversaries and could be expected to perform well in the dynamic distribution setting.
However, thebest performing methods for non-convex online learning require retraining over all previous data atevery time step [52], making them practically unsuitable for the objectives we consider.
Locality-Sensitive Hashing in Machine Learning: Locality-sensitive hash (LSH) functions are acentral component of our proposal.
An LSH family is a set of functions that maps similar inputs to thesame hash value (with high probability).
LSH is an incredibly powerful and well-studied framework,with the following formal deﬁnition [26, 8, 21, 5].
Deﬁnition 2.1 ( ( S 0 , cS 0 , p 1 , p 2 ) -sensitive hash family).
A family H is called ( S 0 , cS 0 , p 1 , p 2 ) -sensitive with respect to a similarity function sim ( · , · ) if for any two points x, y 2 R D and h chosenuniformly from H satisﬁes:
• If sim ( x, y )  S 0 then Pr [ h ( x )
= h ( y )]
p 1•
If sim ( x, y )  cS 0 then Pr [ h ( x ) = h ( y )]  p
The probability Pr [ h ( x ) =
h ( y )] is known as the collision probability of x and y , where we usethe term “collision” to mean that two points map to the same hash value.
For the purpose of ourarguments, we use a stronger notion of LSH, in which the collision probability is a monotonicincreasing function of the similarity between x and y .
That is Pr [ h ( x ) =
h ( y )] / f (sim( x, y )) .
Under the above conditions, the collision probability Pr [ h ( x )
= h ( y )] forms a positive semi-deﬁniteradial kernel [20].
A number of useful kernels can be obtained via LSH, including MinHash for
Jaccard kernel [13], signed random projections for the angular kernel [16], and p-stable LSH for akernel in Euclidean space [21].
Recent works have shown that LSH can be used for efﬁcient statistical estimation [51, 15, 39, 20, 33,34, 35, 59], compressive machine learning [32],nearest neighbor search [6, 8, 7] and to accelerateneural network training and inference [18, 17, 30, 38, 60].
A particularly relevant result is the RACEsketch [20], which solve kernel density estimation using the number of examples that fall into therandom partitions induced by an LSH function.
RIDDLE for Learning with Dynamic Deﬁnition
Learning with Dynamic Deﬁnition: We formally deﬁne the problem as follows.
Deﬁnition 3.1.
Given a model f ( x ; ✓ ) and a loss ` ( f ( x ; ✓ ) , y ) , consider an original dataset D o = { ( x oi , y oi ) } mi =1 drawn i.i.d.
from a distribution P o and an update dataset D u = { ( x ui , y ui ) } ni =0 drawn i.i.d from distribution P u .
Let ✓ o be the parameters that optimizes L o ( ✓ ) =
P
D
o ` ( f ( x ; ✓ ) , y ) .
The dynamic deﬁnition learning problem is to ﬁnd a solution minimizing L o + u ( ✓ ) when we are only given access to ✓ o and D u .
[Equation]
o -0.2 0.3 2.8 0.
Mean0.
1h(x)
In this paper, we consider the model performance in response to the changes from P o to P u , speciﬁcally byallowing x to be drawn from different input spaces.
Notethat Deﬁnition 3.1 generalizes in a straightforward wayto sequences of updating dataset by considering D
o tobe the union of all prior data.
The difﬁculty of the problem may be expressed in terms of the Kullback-Leiblerdivergence D KL ( P o k P u ) .
Intuitively, small changes indistribution are easier to handle because ✓ need not changesubstantially to perform well on D u .
3.1 Architecture
RIDDLE is a 2D parameter array S that contains Lrows and R columns S 2 R L ⇥ R .
The model is indexed by a set of L randomly generated LSH functionsh l ! { 1 , . . .
R } 2 Z for l = 1 , . . .
L .
The prediction 124 evaluated at a query x is deﬁned as follows:
[Equation]
Algorithmically, the process consists of three steps: (1) hash code computation, (2) partition look-ups,and (3) output averaging.
Speciﬁcally, the model output for a query x is an average over the Lparameters retrieved from S using the hash codes h ( x ) .
Figure 3 illustrates a toy example.
For a classiﬁcation problem with C classes, the 2D array is repeated C times, resulting in a 3Dtensor of parameters S 2 R C ⇥
L ⇥ R .
The same LSH functions are used for all C repetitions, and theclassiﬁcation prediction for query x is deﬁned as follows:
[Equation]
RIDDLE can be used as a stand alone machine learning model on simple datasets. RIDDLE can alsobe combined with existing neural network. Speciﬁcally, RIDDLE could take advantage the learnedrepresentation from neural network, see Figure 2.
RIDDLE can be implemented in standard deeplearning frameworks and trained using SGD (Algorithm 1).
It should be noted that the parametersof LSH functions h ( x ) are not trainable.
Only the values in S 2 R L ⇥ R will be updated by SGD.
The inference process consists only of inexpensive hash evaluations and memory look-ups.
Weuse LSH functions based on sparse projections [3], which compute h ( x ) via an inner product withparameter vectors randomly drawn from {  1 , 0 , 1 } with probability { 1 / 6 , 2 / 3 , 1 / 6 } .
With such ahash function, computation can be reduced to addition, subtraction and memory lookup.
Input: Training Dataset D , | D | = m , number of rows L , number of cells R , learning rate ⌘ , error function E ( · ) , number of epochs e , batch size b , random seed s .
Output: Trained Model S , Counters C
Initialize: Zero initialize S 2 R L ⇥ R ; Zero initialize counters C 2 R L ⇥ R ;Random generated L independent LSH functions h 1 , . .
, h L using random seed s ; for i
= 1 !
e do Shufﬂe dataset D for j = 0 ! m/
b  1 do Take b samples { x jb , x jb +1 , . . .
, x ( j +1) b  1 } with labels { y jb , y jb +1 , . . .
, y (
j +1) b  1 }
Compute gradient g = 1 m r S
P i E ( f S,h ( x i ) , y i )
Apply update S !
S  ⌘ g end for end for for every x i 2 D do Increase counters C
[ l, h l ( x i )]
+
Before we determine the class of functions that are representable using our model, we ﬁrst demonstratethat this architecture addresses the problem from Deﬁnition 3.1.
We consider a RIDDLE modelf ( x ; S, h ) following the notation from Section 3.1.
That is, we are given a model with parameters✓ =
( S, h ) trained on D o .
We wish to adapt the model on D u using gradient descent.
Parameter Separation:
The RIDDLE partitions parameters in such a way that inputs from differentdistributions are likely to use different parameters.
The parameter update may be written as:
S t [Equation] 1 =
[Equation] t [ l, h l ( x )] + ⌘ g [ l ] for l = 1 . . .
L, where g = r `
[Equation]
From this expression, we can see that L parameters are involved in the computation of f ( x ; S, h )- speciﬁcally, the parameters selected by the hash functions { h 1 ( x ) ...
h L ( x ) } .
Based on the LSHproperty (Deﬁnition 2.1), we expect unrelated inputs to have different hash values and therefore usedifferent parameters.
When P u is signiﬁcantly different from P o , x u 2 D u and x o 2 D u are likelyto activate different partition and thus use different parameter sets.
We formalize this intuition below by showing that the difference between f ( x ; S, h ) and its initialization is bounded by an estimate of the distribution P o .
Theorem 3.2.
Given a dataset D , train a RIDDLE model f ( x ; S, h ) by running Algorithm 1 for e epochs with learning rate ⌘ and initialization S 0 .
Suppose the gradient norms are bounded by G .
Then
[Equation]
where KDE( x, D ) is the kernel density of x over D using the kernel of
Theorem 3.2 allows us to prove the following guarantee:
Theorem 3.3.
Using the notation from Deﬁnition 3.1, let S o be the sketch that optimizes the loss
L o ( S ) over D o and S o + u be the sketch obtained by training over D u for e epochs with learning rate ⌘ and initialization S o .
If the loss is L -Lipschitz with G -bounded gradients, then:
[Equation]
The loss difference in Theorem 3.3 quantiﬁes the amount of information about D o that is forgottenby the model when trained on D u .
The excess loss on example x 2 D o is bounded by the kerneldensity estimate KDE( x, D u ) .
When D o and D u are drawn from different distributions, whichis the characteristic of learning under dynamic distribution, the density becomes small (i.e. as thedivergence D KL ( P o k P u ) !
1 , KDE( x, D u ) ! 0 ).
Parameter Importance:
The RIDDLE allows us to determine the importance level of each parameter for encoding information from the original distribution with simple counting.
We may do this by counting the number of inputs from D o that fall into each partition, using a set of counters C 2 R L ⇥ R .
Parameters with a large counter c can be regarded as important for the original distribution, as they participate in prediction for a large fraction of examples from D o .
Our intuition is that parameters corresponding to partitions with large count values are critical for performance on D o and should not be adjusted.
We use this idea to improve performance via partition-dependent learning rates.
Speciﬁcally, we allow the learning rate ⌘ ( c ) depend on the partition count c .
We ﬁnd that the following update rule provides good empirical results:
[Equation]
RIDDLE is a Universal Function Approximator
In this section, we analyze the class of functions that can be represented using the RIDDLE.
Usingtechniques from the sketching literature, we prove a probabilistic guarantee.
Intuition: To see why it is reasonable for our model to be a universal function approximator, observethat the function learned by RIDDLE is a piecewise-constant spline function.
Splines are an incrediblypowerful representation that can approximate any function, given sufﬁcient ﬂexibility in choosingthe partitions (or knots) of the spline [48, 47].
For example, many neural networks can be written ascompositions of piecewise-linear splines over a constrained set of partitions[12, 11, 10, 57].
In ourcase, the partition boundaries are determined by the intersection of LSH functions.
With sufﬁcientindependent overlapping partitions (large L ), our model attains a high-quality representation.
Proof Sketch: We prove that the expected output of our model is a universal function approximator,where the expectation is taken over the choice of LSH functions.
We begin with a proof that anycontinuous and bounded function can be approximated arbitrarily well by a weighted kernel sum overan N point dataset.
We then develop a process that constructs a RIDDLE S 2 R L ⇥ R whose expectedoutput is this kernel sum.
Using Chernoff bounds, we show that the error between the model and thekernel sum becomes arbitrarily small as L !
(with high probability).
Taken together, we have aprobabilistic guarantee for the error between the RIDDLE and any continuous and bounded function.
Simpliﬁed optimization in practice: Constructing a RIDDLE over a carefully chosen N pointdataset can approximate any continuous and bounded function.
However, learning both the dataand data weights with N being arbitrarily large poses difﬁculty in optimization.
We simplify theoptimization by observing that the output of the learning process is not a dataset but a model.
Therefore, we consider the easier task of learning the model parameters values directly in practiceas shown in Algorithm 1.
This is a slight relaxation of the problem, however, we argue that witha row-sum constraint and a reasonable uniqueness assumption about the hash partitions, the twoprocesses are equivalent in practice.
Thus, the RIDDLE is a universal function approximator.
4.1 Weighted LSH Kernel Sums are Universal Function Approximators
[Equation]
The crucial property is that LSH kernels are universal .
Informally, a kernel is universal if a weightedkernel sum over a carefully-chosen point set can approximate any function with arbitrary accuracy [40].
Some restrictions apply; the size of the point set is allowed to be arbitrarily large and the accuracy iscomputed over any compact subset of R d rather than the entire space.
Deﬁnition 4.1.
Universal Kernel [40]: Let S ( X ) denote the space of bounded and continuousfunctions on a compact domain X .
Then a kernel K ( x, y ) is universal if it is continuous and inducesa reproducing kernel Hilbert space that is dense in S ( X ) .
Applied to our problem, a universal kernel is one which can approximate any well-behaved functionover compact subsets of R d using a linear combination of kernels.
The theory of universal kernelsallows for a convergence-style proof for weighted LSH kernel sums.
We begin by showing that thep -stable LSH functions are universal kernels.
We defer proofs to the appendix.
Lemma 4.2.
The L2 LSH kernel [20] induced by the p-stable LSH function [21] is shift-invariantand universal.
Theorem 4.3.
Given a continuous and bounded function g ( q ) and any ✏ > 0 , there exists a set of coefﬁcients { ↵ n } , set of points { x n } and an integer N such that
[Equation]
where K ( x n , q ) is the L2 LSH kernel and X is any compact subset of R d .
4.2 RIDDLE for Weighted LSH Kernel Sums
The next step of our proof is to construct a RIDDLE capable of approximating f N ( q ) from Theorem 4.3.
We consider a RIDDLE model with S 2 R L ⇥ R constructed from a set of points { x n } withcoefﬁcient { ↵ n } in the following way.
For every data point x 2 { x n } , we calculate the indices of x 208 using a collection of L2 LSH functions h l ( x ) for l = 1 , . . .
, L .
Then, we increment the value of S atlocation ( l, h l ( x ) with the weight ↵ .
To query the model, we calculate the index of the query q usingthe same LSH functions and retrieve the values S [ l, h l ( q )] for l = 1 , . . .
, L , which we aggregateand return.
It should be noted that recent algorithms from the sketching literature propose similarconstructions [20, 19], but with the weights constrained to ↵ n 2 { +1 ,  1 } .
One can prove S yields a sharp unbiased estimator for the weighted kernel sum (Proofs in appendix).
Theorem 4.4 (RIDDLE Estimator) .
Given a dataset D of weighted samples { ( ↵ x i , x i ) } , let h ( x ) 215 be an LSH function drawn from an LSH family with collision probability
K ( · , · ) .
Let S be a 2d 216 parameter array constructed using h ( x ) .
For any query q ,
[Equation]
The key insight is that each row of S is an unbiased estimator for weighted sums of LSH kernels.
By ﬁnding the central tendency of the L rows, we can obtain a sharp concentration around f N ( q ) .
Although we often use the average to do the aggregation in practice, we analyze the median-of-meansestimator [4, 15], as it allows us to prove an exponential concentration of the estimate around theweighted KDE.
Note that while this simpliﬁes the proof, it does not affect the convergence to f N ( q )as L ! 1 .
We combine the variance bound from Theorem 4.4 with the median-of-means guaranteeto obtain a relationship between the RIDDLE parameters S , weighted kernel values, and estimationerror.
Theorem 4.5 (Weighted-KDE Estimation Error) .
Let Z ( q ) be the median-of-means estimate constructed using the L unbiased estimators.
Then with probability 1   ,
[Figure]
[Equation]
where f K ( q ) and ˜ f K ( q ) are the weighted KDE with kernels K ( x, q ) and p
K ( x, q ) , respectively.
The universal approximation property of our model follows by observing that the error ✏ in Theorem 4.5 goes to zero as N ! 1 and ✏ ! 0 in Theorem 4.3 as L ! 1 .
Experiments
In this section, we evaluate the RIDDLE on practical dynamic distribution learning tasks.
Speciﬁcally,we investigate whether the model can adapt to new distributions and whether it can retain goodperformance on the original.
We also provide an extensive study on the expressiveness and efﬁciency.
5.1 RIDDLE for Learning under Dynamic Distribution
We consider a set of datasets that reﬂect dynamic distributions.
We organize each dataset into two parts:the original dataset D o and the update dataset D u .
D o
and D u have signiﬁcant semantic differencesand do not share any overlapping training data.
We perform grid search over the hyperparameterspace and train all methods to full convergence on the original dataset D o .
Then, for updating themodels, we train models on the update dataset D u , where the initialization consists of the parametersobtained on D o .
We are interested in the performance on the original test set (drawn from distribution
P o ) as well as the updated test set (drawn from P u ) after the updating process.
For tasks where it is reasonable (CIFAR10, ImageNet and News), we exploit recent advances in largescale pretrained models.
Pretrained models are usually trained on a diverse corpus of unlabeled dataand are likely to provide good featurization across distributions.
[44] also observe that pretrainedmodels are more resistant to forgetting.
We replace the ﬁnal layer with RIDDLE.
MNIST Binary: We randomly partitioned digits into two groups ( 4 , 1 , 7 , 5 , 3 and 9 , 0 , 8 , 6 , 2 ) andtrain the model to predict the group assignment of an image.
For the original distribution, werandomly choose 3 digits from the ﬁrst group and 3 digits from the second group.
The update datasetconsists of the remaining digits.
CIFAR10: The task is to predict whether the object in the image is capable of carrying things (e.g.a car can carry things, while a banana cannot).
The original dataset contains images of 6 out of 10objects that can and cannot carry things, and the update dataset contains the remaining 4 objects.
ImageNet: The task is to predict whether a person is allergic to the object in the image.
We assignimages to the allergen and non-allergen classes based on the national allergen directory [1].
Theoriginal dataset contains a subset of allergens and non-allergens, while the update dataset containsthe remaining objects.
This task was built to mimic the the seasonal addition and sensitization of newallergens.
[Table] News: [2] The task is to predict whether a person would be interested in a new article given a captionand short description.
We organize this dataset to reﬂect changes in a user’s interests.
The newsdataset can be divided into 34 topics.
We randomly ﬂag half of these topics as interesting.
Theoriginal dataset contains news from some topics, while the update dataset contains the rest.
5.2 Results
Table A.1 summarizes the accuracy of all models after training on the original dataset and after theupdate.
After training on the original dataset, we observe that all models achieve roughly the sameaccuracy on original test set but much worse accuracy on update test set.
This behavior is expectedsince the update dataset is signiﬁcantly different from the original dataset.
We observe differences after training on the update dataset.
Our baseline models experience signiﬁcantperformance degradation on the original test set.
The RIDDLE outperforms all baselines by up to 30%in terms of original test set accuracy.
Figure 3 shows the accuracy on the original test set during theupdate process.
Unsurprisingly, the baseline performance decreases as we see more training samples [Table] [Table] from update test set.
Models such as Resnet18 and ViT display catastrophic forgetting behavior,eventually performing no better than random.
The RIDDLE model performance, on the other hand,remains nearly constant and lies above all baselines throughout the update process, especially onimage datasets.
On the updated test set, the RIDDLE achieves competitive accuracy (Table A.1),indicating that it is learning the properties of both distributions.
Note that ViT outperforms Resnet18on both CIFAR10 and ImageNet, which conﬁrms our intuition that pretrained models are inherentlymore robust to distribution changes.
By switching the ﬁnal classiﬁcation layer to the RIDDLE, weincrease the accuracy by a large margin.
The experiment results strongly support our hypothesisthat the RIDDLE retains the ability to adapt to new distributions while being less vulnerable thanestablished methods to knowledge loss.
5.3 Study of Expressiveness and Efﬁciency
These experiments are intended to determine whether our model is competitive among popularmachine learning models.
Our goal is to show that RIDDLE provides a good model representationfor a variety of tasks.
We curated a diverse set of datasets from UCI and Kaggle.
These datasetscover a wide range of tasks and domains and are widely used as benchmarks in other works [61, 23].
All experiments are conducted on a machine equipped with 96 24-core/2-thread/2-socket processors(Intel Xeon(R) Gold 5220R 2.20GHz) and 8 Nvidia V100 32GB.
In Table A.1, we observe that RIDDLE outperforms popular linear methods and is competitivecompared to deep learning methods.
NN1 and NN2 denote two different neural network architectures(Details in Appendix).
Both the architecture and evaluation metric is included in Table A.2.
Wealso investigate the efﬁciency and memory requirement.
This is important, as it shows that RIDDLEdoes not outperform other methods due to an increased parameter size.
Table 3 shows that RIDDLEattains the same accuracy as baseline methods with a smaller memory footprint, fewer FLOPs and afaster inference speed.
Speciﬁcally, we use up to 3.7X less memory, use 17X less FLOPs, and achieve3.9x faster inference speed.
The memory improvements likely arise from the meaningful parametergroupings, while the computational improvements are because it only uses addition, subtraction andarray lookup operations.
This experiment conﬁrms that even though the RIDDLE allocates differentparameters to different distributions, we do not require more parameters than existing methods toachieve robustness or accuracy.
One limitation now is that memory grows linear with number ofclass, which can be address with smart hashing mechanism.
Conclusion
In this work, we propose an expressive model RIDDLE for learning under dynamic distribution.
Oursetup concerns common scenarios where model has to be updated given distribution changes.
Themodel is structured to be invulnerable to forgetting because gradients updates are naturally restrictedto only input relevant parameters.
This work poses minimal negative social impact.
Our methodcould be used to increase model robustness by updating model on interested distribution.
Moreover,our method could save computation cost since updating doesn’t require training over all data.