L IFELONG L EARNING WITH O UTPUT K ERNELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Lifelong learning poses considerable challenges in terms of effectiveness (minimizing prediction errors for all tasks) and overall computational tractability for real-time performance.
This paper addresses continuous lifelong multitask learning by jointly re-estimating the inter-task relations ( output kernel) and the per-task model parameters at each round, assuming data arrives in a streaming fashion.
We propose a novel algorithm called Online Output Kernel Learning Algorithm (OOKLA) for lifelong learning setting.
To avoid the memory explosion, we propose a robust budget-limited versions of the proposed algorithm that efﬁciently utilize the relationship between the tasks to bound the total number of representative examples in the support set.
In addition, we propose a two-stage budgeted scheme for efﬁciently tackling the task-speciﬁc budget constraints in lifelong learning.
Our empirical results over three datasets indicate superior AUC performance for OOKLA and its budget-limited cousins over strong baselines.
I NTRODUCTION
The power of joint learning in multiple tasks arises from the transfer of relevant knowledge across said tasks, especially from information-rich tasks to information-poor ones.
Instead of learning individual models, multi-task methods leverage the relationships between tasks to jointly build a better model for each task.
Most existing work in multi-task learning focuses on how to take advantage of these task relationships, either to share data directly Crammer & Mansour (2012) or to learn model parameters via cross-task regularization techniques Argyriou et al.
; Xue et al.
Zhang & Yeung (2014); Kshirsagar et al. (2015), or to embed tasks into a common spaceMoon & Carbonell (2017).
In a broad sense, there are two settings to learn these task relationships 1) batch learning, in which an entire training set is available to the learner 2) online learning, in which the learner sees the data in a sequential fashion.
In recent years, online multi-task learning has attracted extensive research attention Abernethy et al.
; Dekel et al.
; Lugosi et al.
; Cavallanti et al.
; Saha et al.
Following the online setting, particularly from Saha et al.
; Cavallanti et al. (2010), at each round t , the learner receives an example from a task, along with the task identiﬁer and predicts the output label for the example.
Subsequently, the learner receives the true label and updates the model(s) as necessary.
This process is repeated as we receive additional data from a given dataset.
Our approach follows an error-driven update rule in which the model for a given task is updated only when the prediction for that task is in error.
The goal of an online learner is to minimize errors compared to the full hindsight learner.
The key challenge in online learning with a large number of tasks is to adaptively learn the model parameters and the task relationships, which potentially change over time.
Without manageability-efﬁcient updates at each round, learning the task relationship matrix automatically may impose a severe computational burden.
In other words, we need to make predictions and update the models in an efﬁcient real time manner.
We propose an online learning framework that efﬁciently learns multiple related tasks by estimating the task relationship matrix from the data, along with the model parameters for each task.
We learn the model for each task by sharing data from related tasks directly.
Our model provides a natural way to specify the trade-off between learning the hypothesis from each task’s own (possibly quite limited) data and shared data from multiple related tasks.
We propose an iterative algorithm to learn the task parameters and the task-relationship matrix alternatively.
We ﬁrst describe our proposed approach under a batch setting and then extend it to the online learning paradigm.
In addition, we provide a theoretical analysis for our online algorithm and show that it can achieve a sub-linear regret compared to the best linear predictor in hindsight.
We evaluate our model with several state-of-the-art online learning algorithms for multiple tasks.
There are many useful application areas for online multi-task learning, including optimizing ﬁnancial trading, email prioritization, personalized news, and spam ﬁltering.
Consider the latter, where some spam is universal to all users (e.g. ﬁnancial scams), some messages might be useful to certain afﬁnity groups, but spam to most others (e.g. announcements of meditation classes or other special interest activities), and some may depend on evolving user interests.
In spam ﬁltering each user is a "task," and shared interests and dis-interests formulate the inter-task relationship matrix.
If we can learn the matrix as well as improving models from speciﬁc spam/not-spam decisions, we can perform mass customization of spam ﬁltering, borrowing from spam/not-spam feedback from users with similar preferences.
The primary contribution of this paper is precisely the joint learning of inter-task relationships and its use in estimating per-task model parameters in an online setting.
1. R ELATED W ORK
Most existing work in online learning of multiple task focuses on how to take advantage of task relationships.
To achieve this, Lugosi et.
al Lugosi et al.
(2009) imposed a hard constraint on the K simultaneous actions taken by the learner in the expert setting, Agarwal et.
al Agarwal et al.
(2008) used matrix regularization, and Dekel et.
al Dekel et al.
(2007) proposed a global loss function, as an absolute norm, to tie together the loss values of the individual tasks.
Different from existing online multitask learning models, our paper proposes an intuitive and efﬁcient way to learn the task relationship matrix automatically from the data, and to explicitly take into account the learned relationships during model updates.
Cavallanti et al.
(2010) assumes that task relationships are available a priori .
However such task-relation prior knowledge is either unavailable or infeasible to obtain for many applications especially when the number of tasks K is large (Weinberger et al.
and/or when the manual annotation of task relationships is expensive (Kshirsagar et al.
Saha et al.
(2011) formulated the learning of task relationship matrix as a Bregman-divergence minimization problem w.r.t.
positive deﬁnite matrices.
The model suffers from high computational complexity as semi-deﬁnite programming is required when updating the task relationship matrix at each online round.
We show that with a different formulation, we can obtain a similar but much cheaper updating rule for learning the inter-task weights.
Recent works in output kernel learning learn task covariance matrix in RKHS space, inferred directly from the data (Dinuzzo et al.
; Sindhwani et al.
; Jawanpuria et al.
The task covariance matrix is called the output kernel deﬁned on the tasks, similar to the scalar kernel on the inputs.
Most recently, Jawanpuria et al.
(2015) showed that for a class of regularization functions, we can efﬁciently learn this output kernel.
Unfortunately most of the proposed methods for learning output kernels require access to the entire data for the learning algorithm, a luxury unavailable in the lifelong learning setting.
Unlike in online multitask learning, most lifelong learning approaches use a single model for all the tasks or reuse the models from the previous tasks to build a model for the new task Alquier et al.
; Pentina & Urner (2016);
Pentina & Ben-David (2015); Balcan et al.
; Rusu et al.
; Hinton et al.
; Rebufﬁ et al.
These approaches either increases the computation time on iterations where we encounter a novel task or reduces the prediction power of the model learned from the previous tasks due to catastrophic forgetting.
To the best of our knowledge, relationship between the tasks has not been explicitly learned in the lifelong learning setting due to the difﬁculty in learning a positive semi-deﬁnite task relationship matrix in large-scale applications.
This paper provides an efﬁcient way to learn the task relationship matrix in the lifelong learning setting.
L IFELONG L EARNING OF M ULTIPLE T ASKS
Let (( x t , i t ) , y t ) be the example received by the learner from the task i t (at the time step t ) where we assume that the x t ∈ X and y t is its corresponding true label.
The task i t can be a new task or seen by the learner in the previous iterations.
We denote by [ N ] the consecutive integers ranging from 1 to N .
In this paper, we do not assume that the number of tasks is known to the learner ahead of time, an important constraint in lifelong learning problems.
Let K be the number of tasks seen so far until the current iteration t .
For brevity, we consider a binary classiﬁcation problem for each task y t ∈ {− 1 , +1 } , but the methods generalize to multi-class and are also applicable to regression tasks.
We assume that the learner made a mistake if y t (cid:54) =
ˆ y t where ˆ y t is the predicted label.
Our approach follows a mistake-driven update rule in which the model for a given task is updated only on rounds where the learner made a mistake.
Let K : X × X
→ R (kernel on input space) and Ω : N × N → R ( output kernel) be symmetric, positive semi-deﬁnite (p.s.d) multitask kernel functions and denote H as their corresponding RKHS of functions with the norm (cid:107) · (cid:107) H on multitask examples (Smola & Schölkopf (1998)
; Dinuzzo et al.
; Jawanpuria et al.
Using the above notation, we can deﬁne a kernel representation of an example based on a set of representative examples collected on the previous iterations ( prototypes ).
Formally, given an example x ∈ X , its kernel representation can be written using this set:
[Equation]
Initialize: S = ∅ , Ω
For t = 1 , 2 , · · ·
Receive new example ( x t , i t )
Let f i t = (cid:80) ( x s ,i s ) ∈S
y s Ω
i s i t φ ( x s )
Predict ˆ y t = sign ( f i t ( x t ))
Receive label y t
If y t (cid:54) =
ˆ y t
S ← S ∪ ( x t , i t )
Update Ω i t k ∀ k ∈
[ K ] Ω i t k ← Ω
i t k + 1 λ (cid:88) ( x s ,i s ) ∈S i s =
k y t K ( x t , x s )
y s (2) (or) Ω i t k ← Ω i t k exp (cid:26)λ (cid:88) ( x s ,i s ) ∈S i s =
k y t K ( x t , x s )
y s (cid:27) (3) end end
S is the set of stored examples for which the learner made a mistake in the past.
The set S is called the support set .
The online classiﬁcation function is then deﬁned as the weighted sum of the kernel combination of the examples in the support set.
To account for the examples from the different tasks, we consider both the kernel on the input spaces K and the output kernel Ω in our classiﬁcation function.
[Equation]
We set α s = y s .
The predicted label for a new example is computed from the linear combination of the labels of the examples from the support set S weighted by their input similarity K and the task similarity Ω to the new example.
Using the kernel trick , one can write:
[Equation]
Note that, in the above representation, we need learn both the support set S and the output kernel Ω from the data.
As explained in the previous section, for a successful lifelong learning with kernels, we need to address two key challenges: (1) learn the relationships between the tasks (output kernel) efﬁciently from the data arriving in an online fashion and (2) bound the size of the support set S to avoid memory explosion.
We address these two challenges in the following sections.
2. O NLINE O UTPUT K ERNEL L EARNING
Our objective function for the lifelong learning problem is given as follows:
[Equation]
where (cid:96) ( · ) is some loss function such as hinge loss or logistic loss, R ( · ) is the regularization on the task relationship matrix Ω and λ is the regularization parameter.
Note that f in the above equation depends on Ω .
In order to reduce the time taken for each time-step, we require an efﬁcient update to the task relationship matrix Ω .
Following the work of Jawanpuria et al.
(2015) in the batch setting, we consider a subset of regularization functions R for which we can efﬁciently learn the task covariance matrix.
Consider the dual function of the above equation, at time-step t (see Barzilai & Crammer (2015); Jawanpuria et al. (2015)):
[Equation]
When we consider entry-wise l p norm between Ω and Ω ( t − 1) from the previous iteration as our regularization i.e.,
R (Ω , Ω ( t − 1) )
= (cid:80) k
| Ω i t k − (Ω ( t − 1) )
i t k | p for p ≥ 1 1 , we get the update function in Equation 2.
Similarly, if we consider generalized KL-divergence between Ω and Ω ( t − 1) i.e., R (Ω , Ω ( t − 1) )
= (cid:80) k
Ω
i t k log Ω itk (Ω ( t − 1) )
itk − Ω i t k
+ (Ω ( t − 1) )
i t k , we get the update function in Equation 3.
Unlike in the previous works, we update only the row (and the corresponding column) of the task relationship matrix Ω speciﬁc to the task i t , which signiﬁcantly reduces the time taken per example.
We can see that the update equations are simple and quite intuitive.
For a given new example ( x t , i t ) at round t , the algorithm updates Ω i t k
(for some k ∈ [ K ] ) by computing the similarity between the new example and the examples in the support set S that belongs to the task k .
If the two examples have similar (different) labels and high similarity
K ( x t , x s ) , then the Ω i t k is increased to reﬂect the positive (negative) correlation and vice versa.
The value closer to 0 implies no signiﬁcant relationship between the tasks.
The update to the Ω i t k is normalized by the regularization parameter λ for scaling.
It is worth noting that our update equations does not violate the p.s.d constraints on Ω in Equation 5.
If Ω from the previous iteration is a p.s.d matrix and the update is a p.s.d matrix (as it is computed using the Gram matrix of the example from the previous iteration), the sum and Hadamard product of two p.s.d matrices satisfy the p.s.d constraint (using Schur Product Theorem).
Algorithm 1 outlines the key steps in our proposed method.
We write f (( x t , i t ))
as f i t ( x t ) for notational convenience.
At each time-step t , the learner receives an example x t ∈ X and predicts the output label y t using ˆ y t = sign ( f i t ( x t )) .
We update both the support set S and the output kernel Ω i t · when the learner makes a mistake.
Algorithm
Initialize: S = ∅ , Ω
For t = 1 , 2 , · · ·
Receive new example ( x t , i t )
Let f i t = (cid:80) ( x s ,i s ) ∈S
y s Ω
i s i t φ ( x s )
Predict ˆ y t = sign ( f i t ( x t ))
Receive label y t
If y t (cid:54) =
ˆ y t
If |S| < B
S ← S ∪ ( x t , i t )
Else
Find an example to remove arg
[Equation]
S ← S ∪ ( x t , i t ) \ ( x r , i r ) end
Update Ω i t k ∀ k ∈
[ K ] as in Algorithm 1.
end end
Initialize: S = ∅ , T k = ∅ , k ∈
[ K ] , Ω
For t = 1 , 2 , · · ·
Receive new example ( x t , i t )
Let f i t = (cid:80) ( x s ,i s ) ∈T it
y s Ω
i s i t φ ( x s )
Predict ˆ y t = sign ( f i t ( x t ))
Receive label y t
If y t (cid:54) =
ˆ y t
S ← S ∪ { t }\{ r } /* r = ∅ if |S| < B , else choose r using Equation*/
For k ∈ [ K ]
T k ← T k ∪ { l }\{ r } /* remove r from T k ; choose l from S − T k */ end
Update Ω i t k ∀ k ∈
[ K ] as in Algorithm 1.
end end
2. B UDGETED L EARNING
In Algorithm 1, we can see that both the classiﬁcation function f and the update equations for Ω uses the support set S .
When the target function changes over time, the support set S grows unboundedly.
This leads to serious computational and runtime issues especially in the lifelong learning setting.
The most common solution to this problem is to impose a bound on the number of examples in the support set S .
There are several budget maintenance strategies proposed recently (Cavallanti et al.
; Dekel et al.
; Orabona et al.
Unfortunately these schemes cannot be directly used in our setting due to the output kernels in our learning formulation.
Cavallanti & Cesa-Bianchi (2012) proposed the multitask variant of these schemes but they are impractical for the lifelong learning setting.
We follow a simple support set removal schemes based on Crammer et al.
In single-task setting, when the number of examples in the support set S exceeds the limit (say B ), a simple removal scheme chooses an example x r with the 1 We set p = 2 in this paper.
highest conﬁdence from S .
The conﬁdence of an example x r is measured using y r f ( x r ) after removing x r from the support set S .
[Equation]
We can extend the above approach to the multitask and lifelong learning settings.
Since the support set S is shared by all the tasks, we choose an example x r with high conﬁdence to remove from each task function f k , weighted by the relationship between the tasks.
The objective function to choose the example is shown in Equation 6.
We show in the experiment section, this simple approach is efﬁcient and performs signiﬁcantly better than the state-of-the-art budget maintenance strategies.
Algorithm 2 shows pseudocode of the proposed budgeted learning algorithm.
2.2. T WO -S TAGE B UDGETED L EARNING
In lifelong learning setting, the number of tasks is typically large.
The support set S has hundreds of examples from all the tasks.
Each task does not use all the examples from the support set S .
For example, in movie recommendations task, recommendation for each user (task) can be characterized by just a few movies (subset of examples) in the support set
S .
Motivated by this observation, we propose a two-stage budgeted learning algorithm for the lifelong learning setting.
Algorithm 3 shows pseudocode of the proposed two-stage budgeted learning algorithm.
In addition to the support set S , we maintain task-speciﬁc support set T k .
We choose the budget for each task (say L ) where L <<< B .
Similar to the removal strategies for S , we remove an example from T k when |T k | >
L
and replace with an example from the set
S − T k .
One may consider a different budget size for each task L k based on the complexity of the task.
In addition, the proposed two-stage budgeted learning algorithm provides an alternative approach to using state-of-the-art budget maintenance strategies.
For example, it is easier to use the Projectron algorithm (Orabona et al.
on T k , rather than on S .
We will explore this line of research in our future work.
E XPERIMENTS
We evaluate the performance of our algorithm under batch and online settings.
All reported results in this section are averaged over 10 random runs or permutations of the training data.
Unless otherwise speciﬁed, all model parameters are chosen via 5-fold cross validation.
3. B ENCHMARK D ATASETS
We use three datasets for our experiments.
Details are given below:
Newsgroups Dataset 2 consists of 20 tasks generated from two subject groups: comp and talk.politics .
We paired two newsgroups, one from each subject (e.g.,comp.graphics vs talk.politics.guns), for each task.
In order to account for positive/negative correlation between the tasks, we randomly choose one of the newsgroups as positive (+) or negative ( − ) class.
Each post in a newsgroup is represented by a vocabulary of approximately 60 K unique features.
Spam Dataset
We use the dataset obtained from ECML PAKDD 2006 Discovery challenge for the spam detection task.
We used the task B challenge dataset which consists of labeled training data from the inboxes of 15 users.
We consider each user as a single task and the goal is to build a personalized spam ﬁlter for each user.
Each task is a binary classiﬁcation problem: spam (+) or non-spam ( − ) and each example consists of approximately 150 K features representing term frequency of the word occurrences.
Since some spam is universal to all users (e.g. ﬁnancial scams), some messages might be useful to certain afﬁnity groups, but spam to most others.
Such adaptive behavior of user’s interests and dis-interests can be modeled efﬁciently by utilizing the data from other users to learn per-user model parameters.
Sentiment Dataset 4 We evaluated our algorithm on product reviews from amazon.
The dataset contains product reviews from 25 domains.
We consider each domain as a binary classiﬁcation task.
Reviews with rating > 3 were labeled positive (+) , those with rating < 3 were labeled negative ( − ) , reviews with rating = 3 are discarded as the sentiments were ambiguous and hard to predict.
Similar to the previous datasets, each example consists of approximately 350 K features representing term frequency of the word occurrences.
We choose 2000 examples ( 100 posts per task) for 20 Newsgroups , 1500 emails for spam ( 100 emails per user inbox) and 2500 reviews for sentiment ( 100 reviews per domain) for our experiments.
Note that we intentionally kept the size
[Table]
of the training data small to drive the need for learning from other tasks, which diminishes as the training sets per task become large.
Since these datasets have a class-imbalance issue (with few (+) examples as compared to ( − ) examples), we use average Area Under the ROC Curve ( AUC ) as the performance measure.
3. R ESULTS
To evaluate the performance of our proposed algorithm ( OOKLA ), we use the three datasets ( Newsgroups , Spam and Sentiment ) for evaluation and compare our proposed methods to 5 baselines.
We implemented Perceptron and
Passive-Aggressive algorithm ( PA ) Crammer et al.
(2006) for online multitask learning.
Both Perceptron and PA learns independent model for each task.
These two baselines do not exploit the task-relationship or the data from other tasks during model update.
Next, we implemented two online multitask learning related to our approach: FOML – initializes Ω with ﬁxed weights Cavallanti et al.
(2010), Online Multitask Relationship Learning ( OMTRL )
Saha et al.
(2011) – learns a task covariance matrix along with task parameters.
Since OMTRL requires expensive call to SVD routines, we update the task-relationship matrix every 50 iterations.
In addition, we compare our proposed methods against the performance of Online Smooth Multitask Learning ( OSMTL ) which learn a probabilistic distribution over all tasks, and adaptively reﬁning such distribution over time Murugesan et al.
We implement two versions of our proposed algorithm with different updates for the task-relationship matrix: OOKLA-sum (OOKLA with sum update) OOKLA-exp (OOKLA with exponential update) as shown in Algorithm 1.
Table 1 summarizes the performance of all the above algorithms on the three datasets.
In addition to the AUC scores, we report the average total number of support vectors ( nSV ) and the CPU time taken for learning from one instance ( Time ).
From the table, it is evident that both OOKLA- sum and OOKLA- exp outperform all the baselines in terms of both
AUC and nSV .
This is expected for the two default baselines ( Perceptron and PA ).
The update rule for FOML is similar to ours but using ﬁxed weights.
The results justify our claim that learning the task-relationship matrix adaptively leads to improved performance.
As expected, both OOKLA and OSMTL consume less or comparable CPU time than the baselines which take into account for learning inter-task relationships.
Unlike in the OMTRL algorithm that recomputes the task covariance matrix every ( 50 ) iteration using expensive SVD routines, the task-relationship matrix in our proposed methods (and OSMTL ) are updated independently for each task.
We implement the OSMTL with exponential update for our experiments as it have shown to perform better than the other baselines.
One of the major drawbacks of
OSMTL is that it learn only the positive correlations between the tasks.
The performance of OSMTL worsens when the tasks are negatively correlated.
As we can see from the table, our proposed methods outperform OSMTL signiﬁcantly in the 20Newsgroup dataset.
Table 2 compares the proposed methods with different budget schemes and budget sizes in terms of test set AUC scores and the runtime.
We use OOKLA-sum for this experiment.
We set the value of B to { 50 , 100 , 150 } for all the datasets.
We compare our proposed budgeted learning algorithm (Algorithm 2) with the following state-of-the-art algorithm for online budget learning: (1) Random Budgeted Perceptron (RBP)
Cavallanti et al.
(2007)- randomly chooses an
[Table]
.5s 0.
[Table]
02) 2804.5s
Budget -OKL 0.
(0.02) 6.3s 0.
(0.03) 10.2s 0.
(0.03) 13.4s 0.
(0.03) 18.8s
[Table]
[Table]
03) 22.6s
[Table]
[Table]
[Table]
7s
[Table]
[Table] [Table]
8s
[Table] [Table]
01) 1674.4s
[Table] [Table]
) 1935.2s
Budget -OOKLA 0.
[Table] [Table]
[Table] [Table]
[Table]
3s example to remove (2) (Self-tuned)
Forgetron Dekel et al.
(2008)- forgets the oldest example (3) Projectron ++ Orabona et al.
(2008)- projects the new example on the others (4) Budgeted Online Kernel learning (Budget-OKL)- removes an example using equation 7 (5) Budgeted Online Output Kernel Learning (Budget-OOKLA)- removes an example using equation 6.
Following Cavallanti & Cesa-Bianchi (2012), we implement the multitask version of RBP, Forgetron and
Projectron for our experiments.
Table 2 shows both the test set AUC scores (ﬁrst line) and time taken for learning from one instance (including the removal step).
It is evident from the table, our proposed budgeted learning algorithm for online multitask learning outperforms the other state-of-the-art budget schemes on most settings.
Our proposed algorithm uses the relationship between the tasks efﬁciently to choose the next example for removal.
[Figure]
Figure 1 shows the average rate of mistakes on newsgroup dataset for different values of L .
We ﬁx the value of B = 50 .
As we can see from the ﬁgure, even for the small values of L (say 15 ), the mistake rate is close to the model which uses all the examples from the support set S .
On average, we achieved 36% improvement in running time compared to the budget maintenance scheme in Algorithm 2.
C ONCLUSIONS
We proposed a novel lifelong learning algorithm using output kernels.
The proposed method efﬁciently learns both the model and the inter-task relationships at each iteration.
Our update rules for learning the task relationship matrix, at each iteration, were motivated by the recent work in output kernel learning.
In order to handle the memory explosion from the unbounded support set in lifelong learning setting, we proposed a novel budget maintenance scheme that utilize the task relationship matrix to remove the least-useful (high conﬁdence) example from the support set.
In addition, we proposed two-stage budget learning scheme based on the intuition that each task only requires a subset of the representative examples in the support set for efﬁcient learning.
It provides a competitive and efﬁcient approach to handle large number of tasks in many real-life applications.
Effectiveness of our algorithm is empirically veriﬁed over several benchmark datasets, outperforming several competitive baselines both in the unconstrained case and the budget-limited case, where selective forgetting was required.
Future research includes other forms of budget limits (e g. time or reprocessing capacity), more complex inter-task relationships (e.g. hierarchical), and additional factors such as task reliability (e.g. prefer transferring from a well-trained related task, vs from a relatively data-sparse one, whose model is less reliable)