Retaining Knowledge for Learning with Dynamic Definition
Abstract
Machine learning models are often deployed in settings where they must be constantly updated in response to the changes in class definitions while retaining high accuracy on previously learned definitions.
A classical use case is fraud detection, where new fraud schemes come one after another.
While such an update can be accomplished by re-training on the complete data, the process is inefficient and prevents real-time and on-device learning.
On the other hand, efficient methods that incrementally learn from new data often result in the forgetting of previouslylearned knowledge.
We define this problem as Learning with Dynamic Definition (LDD) and demonstrate that popular models, such as the Vision Transformer and Roberta, exhibit substantial forgetting of past definitions.
We present a first practical and provable solution to LDD.
Our proposal is a hash-based sparsity model RIDDLE that solves evolving definitions by associating samples only to relevant parameters.
We prove that our model is a universal function approximator and theoretically bounds the knowledge lost during the update process.
On practical tasks with evolving class definition in vision and natural language processing, RIDDLE outperforms baselines by up to 30% on the original dataset while providing competitive accuracy on the update dataset.
Introduction
Motivation: A common machine learning pipeline is to first define a task objective based on historical experience, then deploy a model trained to optimize the objective over collected data.
However, machine learning models rarely operate in such a static environment.
Instead, models must be dynamically updated to accommodate changes in class definitions without forgetting previous knowledge.
Consider the case of fraud detection, where models must adapt to emerging scam patterns while also providing protection against previously-known patterns.
Machine learning models are also used to identify harmful microbes given gene sequences, where it is important to perform well on new variants or newly discovered microorganisms.
A similar situation arises when models are used to filter inappropriate content such as hate speech and violence.
New phrases or previously-innocuous language may become inappropriate due to world events or attempts by users to circumvent the filter.
While the classification task remains the same (fraud or safe, harmful or not, inappropriate or appropriate, etc.), class definitions evolve over time and likely differ significantly from the point of the initial training.
We refer to this setting as Learning under Dynamic Definition (LDD).
It should be noted that this notion is very different for several other notions of distribution shift in the literature, which we review in the next section. o -0.2 0.3 2.8 0. Mean0. 1h(x)
A standard solution is to retrain the model on all available data (original dataset and data for the new definitions).
This methodology is inefficient due to data storage and extra computation, and even prohibitive when dealing with large models and streaming data.
Furthermore, it prevents real-time and on-device learning.
An ideal and efficient solution is to update the trained model incrementally only with data for the new definitions (e.g. using online gradient descent).
However, this is a challenging process due to the tendency of machine learning models, in particular neural networks, to forget previous knowledge.
This situation, known as collapse/catastrophic forgetting [25], causes performance decay on the old definitions.
Our Proposal: Catastrophic forgetting occurs because gradient updates from training examples in the update process are able to change parameters that are tuned for optimal performance on the original dataset [30].
Inspired by recent connections between random partitions and distribution models [19], we propose RIDDLE (Retaining Information in a Dynamically Defined Learning Environment), a novel architecture with meaningful parameter groupings.
RIDDLE works by partitioning the input space and assigning a set of parameters to each partition.
Gradient updates are restricted only to locally relevant parameters, identified by cheap hashing computations.
RIDDLE can be viewed as a Mixure-of-Expert-Layer where the gating function is Locality Sensitive Function (LSH) and each expert is just one number.
At the same time,
RIDDLE can be viewed as a kernel method where the prediction is an efficient weighted kernel density estimation. RIDDLE can be plugged into diffirent neural network architectures, just like fully connect layers, to ease the memory loss while utilizing the learned features, as shown in Figure 2.
We summarize our technical contributions below.
• We introduce the RIDDLE (in Section 3), consisting of a set of parameters indexed by locality-sensitive hash functions.
RIDDLE is computationally efficient and requires less memory than popular machine learning models with similar accuracy.
• We prove that RIDDLE is universal function approximator and can therefore represent any function( Section 4).
We prove a formal connection between parameter usage and distribution similarity in Theorem 3.2.
• In Section 5, we apply RIDDLE in the dynamic definition setting for four real-world tasks.
Our method outperforms baselines by up to 30% on the original dataset while achieving competitive accuracy on the new dataset.
Related Work
Continual Learning:
Continual learning refers to learning and remembering multiple tasks from a stream of information.
Current approaches can be split into three main categories (1) memory replay to complement neural networks [45, 29], (2) constraints on network updates [38, 28, 52], and (3) dynamic architectural adjustments to accommodate new information, such as additional layers [49, 61].
Recently, few-shot learning methods are also applied into continue learning to solve sequential learning with limited data [8, 58, 53].
Continual learning methods encounter similar challenges to the ones presented by learning under dynamic distribution, most notably catastrophic forgetting.
However, continual learning attempts to solve a sequence of related tasks, often expressed as task embeddings, over data that does not necessarily reflect changes in class definitions.
Train-Test Distribution Shift: Transfer learning and test-time adaptation attempt to remedy misalignments between the training and test distribution.
Although such methods may seem relevant in learning with dynamic definition, they have dramatically different goals.
Transfer learning attempts to adapt a model from a data-rich source domain to a data-scarce target domain, without regard for the performance on the source domain [65, 56, 59].
While several works on train-test distribution shift focus on robustness, it is from the perspective of out-of-distribution detection [24, 39] or uncertainty estimation [44, 33, 57].
Our objective is to provide models that perform well on both the original dataset and new dataset, where class definitions are different, rather than simply detect or adapt to the distribution change.
Online Learning:
Online learning regards optimization as a process where a model is trained by interacting with a stream of data.
Online learning algorithms attempt to minimize the regret , or difference between the loss of the predictions by the incrementally-trained model and the loss of the globally optimal model [48, 46].
Online learning methods attain minimal regret against adaptive adversaries and could be expected to perform well in the dynamic distribution setting.
However, the best performing methods for non-convex online learning require retraining over all previous data at every time step [55], making them practically unsuitable for the tasks we consider.
Locality-Sensitive Hashing in Machine Learning: Locality-sensitive hash (LSH) functions are a central component of our proposal.
An LSH family is a set of functions that maps similar inputs to the same hash value (with high probability).
LSH is an incredibly powerful and well-studied framework, with the following formal definition [27, 7, 20, 4].
Definition 2.1 ( ( S 0 , cS 0 , p 1 , p 2 ) -sensitive hash family).
A family H is called ( S 0 , cS 0 , p 1 , p 2 ) - sensitive with respect to a similarity function sim ( · , · ) if for any two points x, y ∈ R D and h chosen uniformly from H satisfies:
• If sim ( x, y ) ≥ S 0 then Pr [ h ( x )
= h ( y )] ≥ p• If sim ( x, y ) ≤ cS 0 then Pr [ h ( x )
= h ( y )] ≤ p
The probability Pr [ h ( x ) =
h ( y )] is known as the collision probability of x and y , where we use the term “collision” to mean that two points map to the same hash value.
For the purpose of our arguments, we use a stronger notion of LSH, in which the collision probability is a monotonic increasing function of the similarity between x and y .
That is Pr [ h ( x ) =
h ( y )] ∝ f (sim( x, y )) .
Under the above conditions, the collision probability Pr [ h ( x )
= h ( y )] forms a positive semi-definite radial kernel [19].
A number of useful kernels can be obtained via LSH, including MinHash for the
Jaccard kernel [12], signed random projections for the angular kernel [15], and p-stable LSH for a kernel in Euclidean space [20].
Recent works have shown that LSH can be used for efficient statistical estimation [54, 14, 41, 19, 35, 36, 37, 62], compressive machine learning [34],nearest neighbor search [5, 7, 6] and to accelerate neural network training and inference [17, 16, 32, 40, 63].
A particularly relevant result is the RACE sketch [19], which solves kernel density estimation using the number of examples that fall into the random partitions induced by an LSH function.
RIDDLE for Learning with Dynamic Definition
Learning with Dynamic Definition: We formally define the problem as follows.
Definition 3.1.
Given a model f ( x ; θ ) and a loss ℓ ( f ( x ; θ ) , y ) , consider an original dataset D o = { ( x oi , y oi ) } mi =1 drawn i.i.d.
from a distribution P o and an update dataset D u = { ( x ui , y ui ) } ni =0 drawn i.i.d from distribution P u .
Let θ o be the parameters that optimize L o ( θ )
= (cid:80) D o ℓ ( f ( x ; θ ) , y ) .
The dynamic definition learning problem is to find a solution minimizing L o + u ( θ ) when we are only given access to θ o and D u .
[Equation]
In this paper, we consider the model performance in response to the changes from P o to P u , specifically by allowing x to be drawn from different input spaces.
Note that Definition 3.1 generalizes in a straightforward way to sequences of dataset updates by considering D o to be the union of all prior data.
The difficulty of the problem may be expressed in terms of the Kullback-Leibler divergence D KL ( P o ∥ P u ) .
Intuitively, small changes in distribution are easier to handle because θ need not change substantially to perform well on D u .
3.1 Architecture
RIDDLE is a 2D parameter array S that contains L rows and R columns S ∈ R L × R .
The model is indexed by a set of L randomly generated LSH functions h l → { 1 , . . .
R } ∈ Z for l = 1 , . . .
L .
The prediction evaluated at a query x is defined as following:
[Equation]
Algorithmically, the process consists of three steps: (1) hash code computation, (2) partition look-ups, and (3) output averaging.
Specifically, the model output for a query x is an average over the L parameters retrieved from S using the hash codes h ( x ) .
Figure 1 illustrates a toy example.
For a classification problem with C classes, the 2D array is repeated C times, resulting in a 3D tensor of parameters S ∈ R C ×
L × R .
The same LSH functions are used for all C repetitions, and the classification prediction for query x is defined as follows:
[Equation]
RIDDLE can be used as a stand alone machine learning model on simple datasets but can also be combined with existing neural networks.
Specifically, RIDDLE can use the learned representation from neural network (Figure 2).
RIDDLE can also be implemented in standard deep learning frameworks and trained using SGD (Algorithm 1).
It should be noted that the parameters of LSH functions h ( x ) are not trainable.
Only the values in S ∈ R L × R will be updated by SGD.
The inference process consists only of inexpensive hash evaluations and memory look-ups.
We use LSH functions based on sparse projections [2], which compute h ( x ) via an inner product with parameter vectors randomly drawn from {− 1 , 0 , 1 } with probability { 1 / 6 , 2 / 3 , 1 / 6 } .
With such a hash function, computation can be reduced to addition, subtraction and memory lookup. 3.2 Intuition
RIDDLE may seem conceptually different from neural networks at first gasp, however, RIDDLE can be viewed as a combination of Mixture-of-Expert-Layers [23] with Kernel Methods [26].
To understand RIDDLE as a mixture of experts, observe that the gating function for every row in RIDDLE is a randomly initialized hashing function
h i , and the expert is one number stored at
S [ i, h i ( x )] .
RIDDLE can be viewed as an ensemble of L such Mixture-of-Expert-Layers.
On the other side, RIDDLE is also a kernel density estimation process where the LSH collision probability is the kernel function.
The number stored at S [ i, h i ( x )] is an efficient estimation of the weighted kernel values between the test sample and training samples.
Before we determine the class of functions that are representable using our model, we first demonstrate that this architecture addresses the problem from Definition 3.1.
We consider a RIDDLE model f ( x ; S, h ) following the notation from Section 3.1.
That is, we are given a model with parameters θ =
( S, h ) trained on D o .
We wish to adapt the model on D u using gradient descent.
Parameter Separation:
The RIDDLE partitions parameters in such a way that inputs from different distributions are likely to use different parameters.
The parameter update may be written as:
S t +1 =
S t [ l, h l ( x )] + ηg [ l ] for l = 1 . . .
L, where g = ∇ ℓ (cid:32) L
L (cid:88) l = S [ l, h l ( x )] , y (cid:33) From this expression, we can see that L parameters are involved in the computation of f ( x ; S, h ) - specifically, the parameters selected by the hash functions { h 1 ( x ) ...
h L ( x ) } .
Based on the LSH property (Definition 2.1), we expect unrelated inputs to have different hash values and therefore use different parameters.
When P u is significantly different from P o , x u ∈ D u and x o ∈ D u are likely to activate different partition and thus use different parameter sets.
Input: Training Dataset D , | D | = m , number of rows L , number of cells R , learning rate η , error function E ( · ) , number of epochs e , batch size b , random seed s .
Output: Trained Model S , Counters C
Initialize: Zero initialize S ∈ R L × R ; Zero initialize counters C ∈ R L × R ; Randomly generated L independent LSH functions h 1 , . . {+. , h L using random seed s ; for i = 
+}
→ e do
Shuffle dataset D for j = 0 → m/b − 1 do
Take b samples { x jb , x jb +1 , . . .
, x ( j +1) b − 1 } with labels { y jb , y jb +1 , . . .
, y ( j +1)
b − 1 }
Compute gradient g = 1 m ∇ S (cid:80) i
E
( f S,h ( x i ) , y i )
Apply update S → S − ηg end for end for for every x i ∈
D do
Increase counters C [ l, h l ( x i )]+ = 1 for l = 1 . . .
L end for
We formalize this intuition below by showing that the difference between f ( x ; S, h ) and its initialization is bounded by an estimate of the distribution P o .
Theorem 3.2.
Given a dataset D , train a RIDDLE model f ( x ; S, h ) by running Algorithm 1 for e epochs with learning rate η and initialization S 0 .
Suppose the gradient norms are bounded by G .
Then
[Equation]
where KDE( x, D ) is the kernel density of x over D using the kernel of the LSH function h .
Theorem 3.2 allows us to prove the following guarantee:
Theorem 3.3.
Using the notation from Definition 3.1, let S o be the sketch that optimizes the loss
L o ( S ) over D o and S o + u be the sketch obtained by training over D u for e epochs with learning rate η and initialization S o .
If the loss is L -Lipschitz with G -bounded gradients, then:
[Equation]
The loss difference in Theorem 3.3 quantifies the amount of information about D o that is forgotten by the model when trained on D u .
The excess loss on example x ∈ D o is bounded by the kernel density estimate KDE( x, D u ) .
When D o and D u are drawn from different distributions, which is the characteristic of learning under dynamic distribution, the density becomes small (i.e. as the divergence D KL ( P o ∥ P u ) → ∞ , KDE( x, D u ) → 0 ).
Parameter Importance:
The RIDDLE allows us to determine the importance level of each parameter for encoding information from the original distribution with simple counting.
We may do this by counting the number of inputs from D o that fall into each partition, using a set of counters C ∈ R L × R .
Parameters with a large counter c can be regarded as important for the original distribution, as they participate in prediction for a large fraction of examples from D o .
Our intuition is that parameters corresponding to partitions with large count values are critical for performance on D o and should not be adjusted.
We use this idea to improve performance via partition-dependent learning rates.
Specifically, we allow the learning rate η ( c ) depend on the partition count c .
We find that the following update rule provides good empirical results:
[Equation]
RIDDLE is a Universal Function Approximator
In this section, we analyze the class of functions that can be represented using the RIDDLE.
Using techniques from the sketching literature, we prove a probabilistic guarantee.
Intuition: To see why it is reasonable for our model to be a universal function approximator, observe that the function learned by RIDDLE is a piecewise-constant spline function.
Splines are an incredibly powerful representation that can approximate any function, given sufficient flexibility in choosing the partitions (or knots) of the spline [51, 50].
For example, many neural networks can be written as compositions of piecewise-linear splines over a constrained set of partitions[11, 10, 9, 60].
In our case, the partition boundaries are determined by the intersection of LSH functions.
With sufficient independent overlapping partitions (large L ), our model attains a high-quality representation.
Proof Sketch: We prove that the expected output of our model is a universal function approximator, where the expectation is taken over the choice of LSH functions.
We begin with a proof that any continuous and bounded function can be approximated arbitrarily well by a weighted kernel sum over an N point dataset.
We then develop a process that constructs a RIDDLE S ∈ R L × R whose expected output is this kernel sum.
Using Chernoff bounds, we show that the error between the model and the kernel sum becomes arbitrarily small as L [-! 
-]→ ∞ (with high probability).
Taken together, we have a probabilistic guarantee for the error between the RIDDLE and any continuous and bounded function.
Simplified optimization in practice: Constructing a RIDDLE over a carefully chosen N point dataset can approximate any continuous and bounded function.
However, learning both the data and data weights with N being arbitrarily large poses difficulty in optimization.
We simplify the optimization by observing that the output of the learning process is not a dataset but a model.
Therefore, we consider the easier task of learning the model parameters values directly in practice as shown in Algorithm 1.
This is a slight relaxation of the problem, however, we argue that with a row-sum constraint and a reasonable uniqueness assumption about the hash partitions, the two processes are equivalent in practice.
Thus, the RIDDLE is a universal function approximator.
4.1 Weighted LSH Kernel Sums are Universal Function Approximators
We are interested in the representation capabilities of weighted kernel sums over an N -point dataset D = { x 1 , ... x N } . We consider the specific case where the kernel is the collision probability of an
LSH function.
Such kernels are referred to in the literature as LSH kernels [19].
[Equation]
The crucial property is that LSH kernels are universal .
Informally, a kernel is universal if a weighted kernel sum over a carefully-chosen point set can approximate any function with arbitrary accuracy [42].
Some restrictions apply; the size of the point set is allowed to be arbitrarily large and the accuracy is computed over any compact subset of R d rather than the entire space.
Definition 4.1.
Universal Kernel [42]: Let S ( X ) denote the space of bounded and continuous functions on a compact domain X .
Then a kernel K ( x, y ) is universal if it is continuous and induces a reproducing kernel Hilbert space that is dense in S ( X ) .
Applied to our problem, a universal kernel is one which can approximate any well-behaved function over compact subsets of R d using a linear combination of kernels.
The theory of universal kernels allows for a convergence-style proof for weighted LSH kernel sums.
We begin by showing that the p -stable LSH functions are universal kernels.
We defer proofs to the appendix.
Lemma 4.2.
The L2 LSH kernel [19] induced by the p-stable LSH function [20] is shift-invariant and universal.
Theorem 4.3.
Given a continuous and bounded function g ( q ) and any ϵ > 0 , there exists a set of coefficients { α n } , set of points { x n } and an integer N such that
[Equation]
where K ( x n , q ) is the L2 LSH kernel and X is any compact subset of R d .
4.2 RIDDLE for Weighted LSH Kernel Sums
The next step of our proof is to construct a RIDDLE capable of approximating f N ( q ) from Theorem 4.3.
We consider a RIDDLE model with S ∈ R L × R constructed from a set of points { x n } with coefficient { α n } in the following way.
For every data point x ∈ { x n } , we calculate the indices of x using a collection of L2 LSH functions h l ( x ) for l = 1 , . . .
, L .
Then, we increment the value of S at location ( l, h l ( x ) with the weight α .
To query the model, we calculate the index of the query q using the same LSH functions and retrieve the values S [ l, h l ( q )] for l = 1 , . . .
, L , which we aggregate and return.
It should be noted that recent algorithms from the sketching literature propose similar constructions [19, 18], but with the weights constrained to α n ∈ { +1 , − 1 } .
One can prove S yields a sharp unbiased estimator for the weighted kernel sum (Proofs in appendix).
Theorem 4.4 (RIDDLE Estimator) .
Given a dataset D of weighted samples { ( α x i , x i ) } , let h ( x ) be an LSH function drawn from an LSH family with collision probability K ( · , · ) .
Let S be a 2d parameter array constructed using h ( x ) .
For any query q ,
[Equation]
The key insight is that each row of S is an unbiased estimator for weighted sums of LSH kernels.
By finding the central tendency of the L rows, we can obtain a sharp concentration around f N ( q ) .
Although we often use the average to do the aggregation in practice, we analyze the median-of-means estimator [3, 14], as it allows us to prove an exponential concentration of the estimate around the weighted KDE.
Note that while this simplifies the proof, it does not affect the convergence to f N ( q ) as L → ∞ .
We combine the variance bound from Theorem 4.4 with the median-of-means guarantee to obtain a relationship between the RIDDLE parameters S , weighted kernel values, and estimation error.
Theorem 4.5 (Weighted-KDE Estimation Error) .
Let Z ( q ) be the median-of-means estimate constructed using the L unbiased estimators.
Then with probability 1 − δ ,
[Equation]
where f K ( q ) and ˜ f K ( q ) are the weighted KDE with kernels K ( x, q ) and (cid:112) K ( x, q ) , respectively.
The universal approximation property of our model follows by observing that the error ϵ in Theorem 4.5 goes to zero as N → ∞ and ϵ → 0 in Theorem 4.3 as L → ∞ .
Experiments
In this section, we first evaluate RIDDLE on practical dynamic distribution learning task in Section 5.1.
Further, since we prove RIDDLE as a universal function approximator, we provide an extensive study on the expressiveness and efficiency of RIDDLE in Section 5.2. Code is available at https://github.com/lzcemma/RIDDLE.
5.1 RIDDLE for Learning under Dynamic Definition
Here, we investigate whether RIDDLE can adapt to new distributions while retaining good performance on the original distributions.
We consider four datasets that reflect dynamic distributions.
Every dataset is organized into two parts: the original dataset D o and the update dataset D u .
D o
and D u have significant semantic differences and do not share any overlapping training data.
MNIST Binary: We randomly partitioned digits into two groups (e.g. 4 , 1 , 7 , 5 , 3 and 9 , 0 , 8 , 6 , 2 ) and train the model to predict the group assignment of an image.
For the original distribution, we randomly choose 3 digits from the first group and 3 digits from the second group such as D o only consists images of 4, 1, 7, 9, 0, 8.
The update dataset consists of images of the remaining digits.
[Figure] CIFAR10: The task is to predict whether the object in the image is capable of carrying things (e.g car and horse can carry things, while a frog cannot).
The original dataset D o contains images ofout of 10 objects that can cannot carry things, and the dataset D u contains the remaining 4 objects.
ImageNet: The task is to predict whether a person is allergic to the object in the image.
We select the image classes overlapping with national allergen directctory [1], and assign images to the allergen and non-allergen classes based on the common allergen group in national allergen directory [1].
The original dataset D o contains a subset of allergens and non-allergens, while the update dataset D u contains the remaining objects.
This task was built to mimic the the seasonal addition and sensitization of new allergens.
News: The original task for the News dataset [43] is to predict the news topics given a caption and short description. We re-task the dataset to predict whether a person would be interested in a new article by assigning some topics as interesting and the rest as not-interesing.
The original dataset D o contains news from a subset of interesting and non-interesting topics, while the update dataset D u contains the rest.
We organize this dataset to reflect changes in a user’s interests. 5.1.2 Baselines
We compare RIDDLE with two updating strategies on different architectures.
Update: We train a neural network to full convergence on the original dataset D o . Then, for updating, we train models only on the update dataset D u , where the initialization is the parameters obtained on
D o . This approach does not require data storage nor training computation over the original dataset. Elastic Weight Consolidation(EWC): [31] was proposed to mitigate catastrophic forgetting by selectively slowing down learning on the weights important for previous tasks using Fisher information. 5.1.3 Experiment Details
For tasks where it is reasonable (CIFAR10, ImageNet and News), we exploit recent advances in large scale pretrained models and replace the final layer with RIDDLE.
Pretrained models are usually trained on a diverse corpus of unlabeled data and are likely to provide good featurization across distributions.
[47] also observe that pretrained models are more resistant to forgetting.
5.1.4 Main Results
Table A.1 summarizes the test accuracy of all models after training on the original dataset and after the update.
After training on the original dataset, we observe that all models achieve roughly the Trained on original dataset Updated on update dataset Dataset
Method Original
Update
Original [Table] MNIST Binary Update-MLP
[Table] [Table] [Table] [Table]
[Table] RIDDLE [Table] CIFAR
Update [Table] [Table] [Table]
[Table]
RIDDLE
[Table]
ImageNet
Update [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
RIDDLE
[Table]
News
Update
[Table]
RoBERTa
[Table]
RoBERTa
[Table]
RIDDLE
[Table]
same accuracy on original test set.
Accuracy on update test dataset is much worse, which is expected since update distribution is significantly different from the
We are interested in the accuracy on original dataset after updating the model on update dataset.
We observe a 20% to 30 % accuracy drop on the original accuracy with naive updating.
Models such as Resnet18 and ViT display catastrophic forgetting behavior, eventually performing no better than random.
EWC is effective in prevent forgetting to some extend, reducing the degradation to around 10% to 20%.
The RIDDLE significantly outperforms both baselines combined with all architectures across datasets up to in terms of original test set accuracy. Indeed, the RIDDLE keeps the accuracy degradation below 7 %. For example, the accuracy drop is only 0.9% on Mnist-Binary.
On the updated test set, the RIDDLE achieves competitive accuracy (Table A.1), indicating that it is learning the properties of both distributions.
Note that ViT outperforms Resnet18 on both CIFAR10 and ImageNet, which confirms our intuition that pretrained models are inherently more robust to distribution changes.
By switching the final classification layer to the RIDDLE, we increase the accuracy by a large margin.
The experiment results strongly support our hypothesis that the RIDDLE retains the ability to adapt to new distributions while being less vulnerable than established methods to knowledge loss. 5.1.5 Ablation on Update Rule
There are two main drivers of performance when training RIDDLE over dynamically evolving definitions: the hash function and the parameter-dependently learning rate.
Without a learning rate that depends on parameter importance, the test accuracy on the original
MNIST test set after updating is 94.9%, which is 26 % higher than Update-MLP, 4 % higher than
EWC-MLP. Using the dynamic learning rate, RIDDLE achieves 98%. There could be smarter or simpler update functions based on the parameter count, which we leave for future work. [Table]
[Table] [Table] 5.2 Study on Expressiveness and Efficiency In this section, we investigate whether RIDDLE is competitive among popular machine learning models in terms of expressiveness and efficiency.
Our goal is to show that RIDDLE provides a good model representation for a variety of tasks.
We curated a diverse set of datasets from UCI and Kaggle.
These datasets cover a wide range of tasks and domains and are widely used as benchmarks in other works [64, 22].
All experiments are conducted on a machine with 96 24-core/2-thread/2-socket processors (Intel Xeon(R) Gold 5220R 2.20GHz) and 8 Nvidia V100 32GB.
In Table 5.2, we observe that RIDDLE outperforms popular linear methods and is competitive compared to deep learning methods.
NN1 and NN2 denote two different neural network architectures(Details in Appendix).
Both the architecture and evaluation metric is included in Table A.2.
We also investigate the efficiency and memory requirement.
This is important, as it shows that RIDDLE does not outperform other methods due to an increased parameter size.
Table 5.2 shows that RIDDLE attains the same accuracy as baseline methods with a smaller memory footprint, fewer FLOPs and a faster inference speed.
Specifically, we use up to 3.7X less memory, use 17X less FLOPs, and achieve 3.9x faster inference speed.
The memory improvements likely arise from the meaningful parameter groupings, while the computational improvements are because it only uses addition, subtraction and array lookup operations.
This experiment confirms that even though the RIDDLE allocates different parameters to different distributions, we do not require more parameters than existing methods to achieve robustness or accuracy.
One limitation now is that memory grows linear with number of class, which can be address with smart hashing mechanism.
Conclusion
In this work, we propose an expressive and efficient model RIDDLE for learning under dynamic distribution.
Our setup concerns common scenarios where model has to be updated given distribution changes.
The model is structured to be invulnerable to forgetting because gradients updates are naturally restricted to only input relevant parameters. Further, RIDDLE can be used for efficient weighted kernel density estimation. Acknowledgments
This work was supported by National Science Foundation SHF-2211815, BIGDATA-1838177, ONR DURIP Grant, and grants from Adobe, Intel, Total, and VMware. A Appendix A.1 Additional Experiment Result In this section, we provide additional experiment results.
Counter matrix C on D o versus C on D u could be a good visualization over distribution shift.
As a toy example, we plot C o and C u on Mnist-binary with R = 32 .
We randomly sample 3 rows as it is easier to visualize.
For each comparison, top row is from C o , bottom is from C u .
We can observe the density difference. [Table] We investigate the choice of number of rows L and number of cells R . L and R are the two parameters controlling the tradeoff between efficiency and expressiveness.
It is clear that increase memory budget leads to better accuracy.
At the same R , increase the L leads to a higher accuracy.
At the same L , increase R also lead to a higher accuracy.
Increase L increases the repetition, leading to more robust model.
Increase R increase the hashing precision.
[Table] A.2 Experiment Setup We use Adam Optimizer with a learning rate 0.0001 for all baselines. We use early stopping on all methods.
We do not use any data augmentation or feature engineering, except normalizing images.
We do not use regularization.
Following two tables provides architecture details
[Table]
[Table]
A.3 Theory
In this section, we provide complete proofs for our theoretical results.
A.3.1 Proof of Theorem 4.
Theorem A.1.
Given a dataset D of weighted samples { ( α x i , x i ) } , let h ( x ) be an LSH function drawn from an LSH family with collision probability K ( · , · ) .
Let S be a row of the sketch constructed using h ( x ) .
For any query q ,
[Equation]
Proof.
Let 1 i denote the indicator function 1 h ( x i )
=
h ( q ) .
That is, 1 i = 1 when
data x
i from the dataset collides with the query q .
To simplify the presentation, let Z = S [ h ( q )] .
Expectation: The value in the sketch can be written as
[Equation]
By the linearity of the expectation
[Equation]
We know that E ( 1 i ) is the collision probability of h ( x ) , thus,
[Equation]
Variance:
The variance is bounded by the second moment.
The second moment of this estimator can be written as
[Equation]
By the Cauchy-Schwarz inequality,
[Equation]
Then,
E
[Equation]
Z 2 ) ≤
[Equation]
i
[Equation]
 
[Equation]
i
[Equation]


[Equation]
Thus, var( Z ) ≤
[Equation]

[Equation]
A.3.2 Proof of Lemma A.
We use a result from [3], where we suppose for convenience that g evenly divides N .
Lemma A.2.
Let Z 1 , ...
Z R be L i.i.d.
random variables with mean E [ Z ] = µ and variance ≤ σ 2 .
Divide the L variables into g groups so that each group contains m =
L/g elements, and take the empirical average within each group.
The median-of-means estimate ˆ µ is the median of the g group means.
If g = 8 log(1 /δ ) and m =
L/g , then the
[Equation]
Proof.
This proof is given in [3] as the proof of Theorem 2.1 (which is a slightly more general version of the statement above).
A.3.3 Proof of Theorem 4.
Theorem A.3.
Let Z ( q ) be the median-of-means estimate constructed using the L unbiased estimators of the sketch with L rows.
Then with probability 1 − δ ,
[Equation]
where f K ( q ) and ˜ f K ( q ) are the weighted KDE with kernels K ( x, q ) and (cid:112) K ( x, q ) , respectively.
Proof.
From Theorem 4.4, we know that
[Equation]
Substituting this variance bound into Lemma A.2 proves the theorem.
A.3.4 Proof of Lemma 4.
Lemma A.4.
The L2 LSH kernel from [19] is shift-invariant and universal.
Proof.
To see that the kernel is shift-invariant, observe that K ( x, y )
=
K (dist( x, y )) and that
the
Euclidean distance ∥ x − y ∥ 2 is a function of the difference x − y .
Thus, K ( x, y ) =
K ( x − y ) .
Since the kernel is shift-invariant, it is sufficient to show that the support of the Fourier transform of the kernel is the entire real line [13].
Observe from [21] that the kernel may be written as
[Equation]
where c = dist( x, y ) and r is a parameter.
The Fourier transform of this quantity is
[Equation]
The integrand satisfies the requirements of Fubini’s Theorem (absolutely integrable), so we may exchange the order of integration.
[Equation]
Observe that the Fourier transform of the inner integrand has full support because the exponential function is nonzero everywhere.
Now observe that the outer integral is the limit of a sum of Fourier transforms, each of which has the real line as its support.
Because
[Equation]
− t/r ) > 0 over the integration region, the coefficients of this sum are positive.
Therefore, the Fourier transform of the kernel has the real line as its support.
A.3.5 Proof of Theorem 4.
Theorem A.5.
Given a continuous and bounded function g ( q ) and ϵ > 0 , there exists a set of coefficients { α n } , set of points { x n } and an integer N such that
[Equation]
where K ( x n , q ) is the L2 LSH kernel and X is any compact subset of R d .
Proof.
This follows directly from the definition of a universal kernel.
A.3.6 Proof of Theorem 3.
Theorem A.6.
Given a dataset D , construct a representer sketch model f
( S,h ) ( x ) by running
Algorithm 1 for e epochs with learning rate η .
Suppose the gradient norms are bounded by G .
Then
[Equation]
where KDE( x ) is the kernel density of x over D using the kernel of the LSH function h .
Proof.
Given a sketch S equipped with a set of hash functions h , the value of the model output for a query x is:
[Equation]
We will examine the values of the array cells S [ l, h l ( x )] when trained using Algorithm 1 on a dataset
D of m points (i.e. | D | = m ).
Under Algorithm 1, the sketch is updated once by each point in the dataset during every epoch.
If we update the sketch at time t with the feature-target pair ( z, y z ) , the update takes the form:
[Equation]
where E ( f ( S,h ) ( z ) , y z ) is the error function used to train the model.
Observe that
[Equation]
In other words, the only parameters in S that contribute to f ( S,h ) ( z ) are those at the array locations { (1 , h 1 ( z )) , ...
( L, h L ( z )) } .
Since the other parameters do not participate in the computation of f ( S,h ) ( z ) , each example ( z, y z ) ∈ D updates only those cells in S that correspond to the values of { h 1 ( z ) , ...
h L ( z ) } .
Let U ∈
Z L × R be the sparse matrix formed by placing a ‘ 1 ’ at the locations { (1 , h 1 ( z )) , ...
( L, h L ( z )) } and ‘ 0 ’ otherwise.
Since the gradient norms are bounded by G , we can bound the size of the update to the sketch.
[Equation]
If we recursively apply this inequality to all m updates in the epoch, we obtain:
[Equation]
Note that the inequality remains true if we swap the positions of S t and S t +1 .
{+Also note that with zero initialization, S 0 = 
+}
L × R .
Therefore, we have:
[Equation]
If we obtain the sketch S after training for e epochs, we pick up an additional multiplicative factor e .
[Equation]
This means that we can bound the absolute value of the sketch output f ( S,h ) ( x ) :
[Equation]
Taking the expectation of both sides of the inequality and applying the linearity of the expectation operator, we have:
[Equation]
[Equation]
[Equation]
x
[Equation]
Note that we may re-write U t [ l, h l ( x )] as the indicator 1 { h l ( z t )=
h l ( x ) } .
That is, the indicator is ‘ 1 ’ if the example z t from gradient descent iteration t collides with x .
Since each epoch processes all elements in the dataset exactly one time, we may re-write the sum over U t [ l, h l ( x )] to be over the dataset.
[Equation]
The expected value of the indicator 1 { h l ( z )=
h l ( x ) } is the collision probability
Pr[ h l ( z ) =
h l ( x )] of the LSH function h l .
Under the assumptions stated in the theorem, this collision probability is equal to the kernel K ( z, x ) , leading to the following inequality:
[Equation]
To prove the theorem, recall that for any random variable Z , | E [ Z ] | ≤ E [ | Z | ] .
This yields the desired statement.
[Equation]
A.3.7 Proof of Theorem 3.
We want to bound the difference between the loss induced by S o and the loss induced by S o + u on
D o .
In other words, we wish to bound:
[Equation]
Note that because S o is the argmin, this quantity is always positive.
Assume that the loss ℓ is at least locally L -Lipschitz in the first argument.
That is, we suppose that the following bound holds in a ball surrounding ˆ y :
[Equation]
Then, applying this to the loss from earlier:
[Equation]
Using Theorem 3.2, we may bound the left hand side of this expression in expectation:
[Equation]
Here, we used the elementary fact that | E [ Z ] | ≤ E [ | Z | ] for any random variable Z .