LUSTERED T ASK -A WARE M ETA -L EARNING BY L EARNING FROM L EARNING P ATHS
Anonymous authors
Paper under double-blind review A BSTRACT
To enable effective learning of new tasks with only few samples, meta-learning acquires common knowledge from the existing tasks with a globally shared metalearner.
To further address the problem of task heterogeneity, recent developments balance between customization and generalization by incorporating task clustering to generate the task-aware modulation to be applied on the global meta-learner.
However, these methods learn task representation mostly from the features of input data, while the task-speciﬁc optimization process with respect to the baselearner model is often neglected.
In this work, we propose a C lustered T askAware M eta-
L earning (CTML) framework with task representation learned from both features and learning path.
We ﬁrst conduct rehearsed task learning from the common initialization, and collect a set of geometric quantities that adequately describes this learning path.
By inputting this set of values into a meta path learner, we automatically abstract path representation optimized for the downstream clustering and modulation.
To further save the computational cost incurred by the additional rehearsed learning, we devise a shortcut tunnel to directly map between the path and feature cluster assignments.
Extensive experiments on two real-world application domains: few-shot image classiﬁcation and cold-start recommendation demonstrate the superiority of CTML compared to state-of-the-art baselines.
I NTRODUCTION
The astonishing performance of deep learning relies on large amounts of data, which are not always available.
Humans, on the other hand, are able to learn new tasks much more quickly, leveraging prior experience to relate knowledge among tasks.
Inspired by this property of human intelligence, meta-learning (also known as learning to learn) (Thrun & Pratt, 1998) acquires transferable commonalities from existing tasks for more efﬁcient learning of new tasks.
Although recent advances in meta-learning have demonstrated success in ﬁelds like few-shot image classiﬁcation and cold-start recommendation (Vinyals et al., 2016;
Snell et al., 2017; Ravi & Larochelle, 2016; Munkhdalai &
Yu, 2017), most of them typically assume that all the tasks are drawn from a single distribution and face the challenge of handling tasks that come from different underlying distributions, a problem known as task heterogeneity (Devos & Dandi, 2020; Yao et al., 2020;
Suo et al., 2020).
To overcome this challenge, many recently developed works leverage task-speciﬁc information to customize the global meta-learner (Vuorio et al., 2019; Oreshkin et al., 2018;
Requeima et al., 2019;
Finn et al., 2018; Yoon et al., 2018; Li et al., 2019; Rusu et al., 2018).
To further consider generalization among related tasks, methods that perform various types of task clustering are proposed (Yao et al., 2019; 2020;
Suo et al., 2020; Lu et al., 2020; Dong et al., 2020; Lin et al., 2021).
Despite their effectiveness in improving over the globally shared meta-learning algorithms, they learn task representations only based on the input data distribution in (original or projected) feature space, while the interaction between data and the base-learner is often neglected.
The amount of information contained in the task-speciﬁc data about a network that is responsible for performing the task can be seen as a good representation of the task itself.
This can be manifested as the gradients of the network parameters with respect to the task-speciﬁc loss, or Fisher Information Matrix (FIM) which indicates the importance of different network parameters in solving the task.
Achille et al.
(2019) introduce a task embedding method based on FIM to assist the meta task of selecting a pre-trained feature extractor.
To alleviate the conﬂict issue of the global initialization methods, Baik et al.
(2020) utilize task gradients at the initialization as task representation to produce attenuation.
However, these methods represent task based on gradients at only a single point in parameter space (e.g., at the initialization), while the potential of exploiting a wider range on the task manifold remains unexplored .
For the parameter initialization approach, the key of success lies in that the task adaptation process is accounted for when training the meta-learner.
This task-speciﬁc learning may involve multiple gradient descent steps, and thus constitute a learning trajectory on the loss surface (Flennerhag et al., 2018; Zenke et al., 2017).
To better characterize the task optimization behaviors, it is more beneﬁcial to look at the complete learning trajectory as opposed to only the ﬁrst gradient step at the initialization, as it is likely that tasks with similar gradients at ﬁrst will have their learning paths diverged as the update proceeds.
With that in mind, we are motivated to leverage the entire learning path for better task representation, which will then be used to condition the global initialization.
In particular, we propose a C lustered T ask-aware M eta-
L earning (CTML) framework, building upon the well-known global initialization method Model-Agnostic Meta-Learning (MAML)
(Finn et al., 2017).
To address the problem of task heterogeneity with a good balance between customization and generalization, we modulate the common initialization based on task representation learned from both its local task-speciﬁc information and global clustering results.
In addition to using the input features to represent task, we further leverage the task learning path with respect to the baselearner to characterize task from the perspective of optimization.
To facilitate clustering, we devise a
GRU-based meta path learner which automatically abstracts path representations from the step-wise geometric quantities.
We realize that it may be too costly to rehearse the entire learning process for task representation.
Hence, we further propose a shortcut tunnel to bypass the rehearsed task learning during meta-testing and predict path cluster assignment directly from the feature cluster assignment.
We carefully study the effectiveness of CTML in two real-world application domains: few-shot image classiﬁcation and cold-start recommendation, and show that our method is able to outperform the baselines with comparable inference time.
R ELATED W ORK
To address the problem of task heterogeneity, recent developments often tailor the shared metalearner to different tasks using task-speciﬁc information.
Finn et al.
(2018) and Yoon et al.
(2018) model the uncertainty exists in task distribution with probabilistic framework.
Oreshkin et al.
Li et al.
(2019), Requeima et al. (2019) and Bateni et al.
(2020) condition the base network on taskspeciﬁc data by designing a meta adaptation network.
To enable more robust training, Rusu et al.
(2018) propose to learn a lower-dimensional latent space speciﬁc to each task to generate the base network parameters.
Relevant to our work, Vuorio et al.
(2019) build upon MAML and modulate the global initialization based on the task representation learned from an encoder network.
However, customizing the common knowledge to individual task without considering the relations among similar tasks may lead to poor generalization.
In regard to this, Dong et al.
(2020) and Lin et al.
(2021) apply K-means clustering on users (treated as tasks) based on their proﬁle information to address the cold-start problem in recommender systems.
Yao et al.
(2019) employ a hierarchical structure to model the nested relationships inherent in domains with clear taxonomy, such as image classiﬁcation.
Yao et al.
(2020) further develop an automatic relational graph method by constructing a meta-knowledge graph to preserve and propagate the global structural information.
Suo et al.
(2020) take advantages of an external knowledge base to facilitate task clustering.
Despite their effectiveness in generalizing across tasks, they rely solely on input features for task representation, while the interaction between task data and the base-learner (e.g., gradients) is often neglected.
The idea of using gradients for task representation is commonly adopted in continual learning (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018).
In the context of task-aware metalearning, Achille et al.
(2019) propose to use the FIM of a “probe” network to represent task, and
Baik et al.
(2020) leverage gradients of the globally initialized parameters to generate task-speciﬁc attenuation.
However, these methods utilize gradients only at a single point in parameter space, whereas meta-learning often incorporates the entire task learning trajectory while learning the metalearner.
Garcia et al.
(2021) perform task clustering at each adaptation step and aggregate the gradients for more stable task adaptation.
Flennerhag et al.
(2018) highlight the beneﬁts of leveraging the learning paths for deriving the common knowledge, but the proposed algorithm only relies on one geometric quantity – the length of the path, which can be limited in characterizing the learning paths, and the initialization is not tailored to speciﬁc task.
Our work serves to address the above-mentioned limitations, whereby higher-order behaviours of the learning path are also taken into account to learn a better task representation for modulating the global initialization.
P RELIMINARIES AND P ROBLEM S TATEMENT
In task-heterogeneous setting, tasks {T 1 , T 2 , ..., T N } are sampled from a mixture of task distributions { p 1 ( T ) , p 2 ( T ) , ... } , where the number of underlying distributions is usually unknown.
The goal of meta-learning is to learn the sharable knowledge by training the meta-learner on a set of metatraining tasks T tr = {T i } N tr i =1 , and test it on a set of meta-testing tasks T te =
{T i } Ni = N tr +1 .
For each task T i ∈ T tr ∪ T te , its samples are further divided into a training set (also termed support set)
D tr T i = { ( x i,j , y i,j ) } n tr T i j =1 and a test set (also termed query set)
D te T
i = { ( x i,j , y i,j ) } n T i j = n tr T
i +1 , which together form an “episode” (Vinyals et al., 2016).
This episodic scheme allows us to train tasks to learn fast during meta-training, and test the learning performance of new tasks in the same way during meta-testing.
For few-shot learning, the size of the training set n tr T i is usually small.
Our work builds upon Model-Agnostic Meta-Learning (MAML)
(Finn et al., 2017).
It implements the meta-learner as an initialization of parameters θ 0 ∈ R D of the base-learner f θ responsible for the prediction task.
During meta-training, the global initialization θ 0 is ﬁrst adapted to each metatraining task T i ∈ T tr by learning on the respective training set D tr T i , which yields the task-speciﬁc parameters θ T i .
After that, loss is computed on the test set D te T i based on θ T i and backward propa- gated to update θ 0 .
Taking one step adaptation as an example, the meta-optimization is as follows:
[Equation]
where α is the adaptation rate, L ( f θ , D ) can be mean square error loss for regression problem (i.e., |D| " ( x ,y ) ∈ D ( y − f θ ( x )) 2 ), or cross-entropy loss for classiﬁcation problem (i.e., − |D| [Equation]
x , [Equation]
[Equation]
D y log f θ ( x ) ).
During meta-testing, the learned initialization θ ∗ 0 is adapted to each meta-testing task T i ∈ T te using D tr T i , and the learning performance is evaluated on D te T i .
However, with a globally shared initialization, MAML is not capable of handling task heterogeneity.
Though task-specic methods have been developed to tailor the global initialization, they lack explicit modeling of the global clustering structure.
Hence, a framework considering both task-speciﬁc information and global structure is desired.
M ETHODOLOGY
Grounded on MAML, our CTML framework modulates the common initialization based on task representation learned from two different perspectives enhanced with the clustering information.
Speciﬁcally, given the training set of a speciﬁc task, we ﬁrst conduct rehearsed learning of the task from the common initialization and compute a set of step-wise quantities along the learning path.
This set of values will be inputted into the meta path learner to generate a task-speciﬁc path embedding.
On the other hand, input features of the given task will also be extracted to form the feature embedding.
Both feature and path embeddings will then undergo soft K-means clustering in their respective latent space.
Finally, the two embeddings enriched with cluster information will be aggregated to produce the task-aware modulation to be applied on the global initialization.
From this modulated intialization, standard MAML follows, which conducts the actual task learning on the training set and perform meta-update across tasks using the test sets.
To further improve the inference efﬁciency during meta-testing, a shortcut tunnel is used to bypass the rehearsed learning process and generate the path cluster assignment directly from the feature cluster assignment.
An overview of CTML framework is shown in Figure 1.
In the following sections, we ﬁrst elaborate on task representation learning based on learning path and features respectively, and then introduce the task-aware modulation and the shortcut tunnel.
4. T ASK R EPRESENTATION BASED ON L EARNING P ATH 4.1. P ATH C ONSTRUCTION
To obtain representation of task T i based on learning path, we ﬁrst conduct a τ -step rehearsed learning from the global initialization θ 0 on training set D tr T i .
Applying the same gradient descent update as in MAML, we obtain the updated parameters at each step t ∈ { 1 , 2 , ..., τ } by: ˜ θ t
T i = ˜
θ t − 1 T i − α ▽ θ L ( f ˜ θ t − 1 T i , D tr T i ) , where ˜ θ
0 T i = θ 0 .
The overhead ˜ · is used to differentiate between the parameters updated from the rehearsed learning and from the actual learning.
Joining the discrete points { (˜ θ t T i ,
˜ L t T i ) }
τ t =0 constitutes the rehearsed learning path of T
i (we write
L ( f ˜ θ t T i , D tr T i ) as ˜ L t T i for brevity) on the unique task manifold M T i ∈
R D +1 , characterized by the base-learner function and the task-speciﬁc data distribution.
We collect the coordinates of point (˜ θ t T i ,
˜ L t T i ) at each update step t ∈ { 0 , 1 , 2 , ..., τ } to indicate the exact learning trajectory.
To account for higher-order behaviours of the learning path, we further incorporate the gradients ▽ θ ˜
L t T i and the Fisher Information Matrix (FIM) ˜
F t T i at each step, which specify the direction and curvature respectively.
Note that by deﬁnition, the FIM is equivalent to the Hessian matrix in expectation for cross-entropy loss: ˜ F t T i = E x ,y ∼ T i
[ ▽ θ L ( f ˜ θ t T i , x , y ) ▽ θ L ( f ˜ θ t T i , x , y ) ⊤ ] = E x ,y ∼ T i [ ▽ 2 θ L ( f ˜ θ t T i , x , y )] .
It provides a measure of the amount of information that task T i contains about the parameters θ (Gianfelici & Battistelli, 2009).
Following Achille et al.
(2019), we assume negligible correlation among different parameters and consider only the diagonal entries of ˜ F t T i , denoted by ˜ F t T i .
As a result, we obtain a set of step-wise quantities { (˜ θ t T i ,
˜
L t T i , ▽ θ ˜
L t T i , ˜ F t T i ) } τ t =0 that adequately describes the rehearsed learning path.
As opposed to looking at only a single point in the parameter space (e.g., use gradients at the initialization to represent task (Baik et al., 2020)), it is beniﬁcial to take into account the entire learning path traversed, which gives a more complete picture of the optimization process.
Take for instance, two tasks T a and T b may have gradients forming a small angle ψ 0 a,b at the initialization, i.e., cos( ψ 0 a,b ) > 0 .
However, it is likely that their learning paths will diverge as the gradient update proceeds, i.e., the accumulated angle
[Equation]
[-( ψ ta,b ) < 
-]
(see dotted paths of T 2 and T 3 in Figure 2).
Conversely, the tasks may have gradients pointing at different directions at ﬁrst, but eventually converge towards the same direction as the learning proceeds (see dotted paths of T 1 and T 2 ).
Considering a single step only can be restrictive for task representation from the optimization perspective.
Looking further down the paths allows better characterization of the learning behaviors and even the ﬂexibility to determine the “important” steps (as shown in experiments later), producing more informative task representations.
Furthermore, leveraging path representations serves to provide a stronger inductive bias to condition the initialization in favorable direction, as the rehearsed learning path can be considered as encoding the quality of the initialization with respect to the task (i.e., tasks with longer and more zigzag paths imply that the initialization is not a good one for them).
4.1. P ATH R EPRESENTATION
Having collected the step-wise quantities { (˜ θ t T
i , ˜
L t T i , ▽ θ ˜
L t T i , ˜ F t T i ) } τ t =0 along the rehearsed learn- ing path, the problem now is how to evaluate “similarity” among different task learning paths for clustering.
Previous works with single-step gradients usually compute dot product or cosine similarity between gradients of two tasks (Riemer et al., 2018; Katoch et al., 2019; Yu et al., 2020;
Baik et al., 2020).
With multiple steps, a straightforward adaptation will be to simply sum up the similarity scores at all steps.
However, this human-deﬁned rule may not be the best way of measuring path similarity.
Instead, we propose to employ a meta path learner to automatically learn a path embedding from the step-wise quantities, and then measure similarity based on these vector representations.
In other words, the meta path learner induces a latent space on which the distance metric best characterizes what is considered as “similar” (or “disimilar”) among task learning paths.
Before we delve into the design of the path learner, we ﬁrst elaborate how we construct the functional input from the step-wise quantities.
Recall that at each step t , we have ˜ θ t T i , ▽ θ ˜
L t T i , ˜ F t T i ∈
R D and ˜
L t T i ∈
R .
To obtain a regularly shaped input, we duplicate ˜ L t T i to form a D -dimensional vector and stack it together with the other 3 components to form a matrix P t T
i ∈
R 4 × D .
Further stacking the τ + 1 steps gives the overall 3-D matrix P T
i ∈ R ( τ +1)
× 4 × D .
To avoid high model complexity, we apply the meta path learner coordinate-wise on the base-learner parameters.
That is, the same path learner is used to process the matrix P d T i ∈ R ( τ +1)
× 4 independently for all d ∈ { 1 , 2 , ..., D } .
For the path learner design, we propose to leverage the Gated Recurrent Units (GRUs) to model the sequential dependencies among steps 1 .
Speciﬁcally, the hidden state h d,ti at step t is obtained by inputting the t -th sliced vector P dt, : ∈ R 4 of P d T i and the hidden state h d,t − 1 i at step t − 1 into the
GRU cell:
[Equation]
where [ · , · ] denotes concatenation, σ ( · ) is the sigmoid activation, r d,ti , z d,ti ∈ (0 , 1) are gates that control how much of past and present information to be retained, and W r , W z , W ˜ h are learnable weights shared across all steps.
For step t = 0 , we use a zero-initialized input hidden state.
The path representation h di at the d -th dimension is obtained from the ﬁnal step hidden state h d, τ i via a linear transformation: h di = W o · h d, τ i + b o .
Concatenating the path representations at all D dimensions and passing it through a fully-connected layer gives the ﬁnal path embedding e path [Equation]
i
[Equation]
Inspired by (Suo et al., 2020;
Dong et al., 2020; Lin et al., 2021), we handle task heterogeneity without jeopardizing generalization among similar tasks by employing a simple yet effective soft
K-means clustering on the path embeddings 2 .
Speciﬁcally, we maintain k path learnable cluster centroids { a pathj | ∀ j ∈ [1 , k path ] } ∈ R k path ×
d e for path embeddings.
The cluster-enhanced path embedding ˜ e path T i ∈
R d e is simply the weighted sum of the cluster centroids:
[Equation]
where q pathij =
[Equation]
[Equation]
|
[Equation]
1 is the probability of assigning e path
T i to cluster j , computed using the Student’s t -distribution as a kernel, following Xie et al.
4. T ASK R EPRESENTATION BASED ON F EATURES
Task representation based on learning path can be interpreted as encoding the conditional distribution p ( y | x ) (as the learning path is informed by the labels), whereas task is best described by the joint distribution p ( x , y )
=
p ( y | x ) p ( x ) .
Therefore, we further incorporate the marginal distribution of input features by learning another representation solely based on features, as what have been done in most of the existing task-aware meta-learning methods (Vuorio et al., 2019; Yao et al., 2019; 2020; Lin et al., 2021).
Another signiﬁcance of including feature-based representation is that, we can create a mapping between the path and feature representations to bypass the rehearsed learning during the meta-testing phase (details will be elaborated in Section 4.4).
Generally, the design of the feature extractor may vary for different application domains.
Let E ( · ) denote an arbitrary feature extractor, the feature embedding e feat T i ∈
R
d e for task T i is obtained by aggregating the extracted features of all samples in the training set: e feat T i
[Equation]
Similar to the path embeddings, we also employ a soft K-means clustering to promote generalization among related feature embeddings.
Speciﬁcally, given k feat cluster centroids { a featj | ∀ j ∈ [1 , k feat ] } ∈ R [Equation] feat
[Equation]
e , the cluster-enhanced feature embedding ˜ e feat T i ∈
R d e is obtained by:
[Equation]
4) where q featij =
(1+ || e
[Equation]
−
[Equation]
j
[Equation]
(1+ || e feat [Equation] − a featj ′ || 2 ) − 1 is the probability of assigning e feat
T i to cluster j .
4. T ASK -A WARE M ODULATION
We aggregate the path and feature embeddings via a learnable weight vector λ to generate the ﬁnal task representation.
This task representation will then be used to produce a modulation to tailor the global initialization θ 0 to speciﬁc task.
The modulated initialization θ 0 i for task T i is obtained by:
[Equation]
where W , b , λ are learnable parameters, σ ( · ) is the sigmoid function, ⊕ and ⊙ denote element-wise addition and element-wise multiplication respectively.
From this modulated initialization, task T i will undergo regular task adaptation learning for τ steps.
After that, the meta-learner φ will be updated by optimizing loss across all the test sets, the same procedure as in MAML.
The meta-optimization objective is:
[Equation]
(6) where φ includes the global initialization θ 0 , the learnable parameters used to generate the clusterenhanced path and feature embeddings, and also the ﬁnal modulation.
4. I MPROVING M ETA -T ESTING E FFICIENCY VIA S HORTCUT T UNNEL
The need to conduct rehearsed learning before the actual learning of each task inevitably leads to twice the inference time compared to the original MAML.
Though it is not possible to bypass the rehearsed learning during meta-training as we need to optimize the meta path learner and the downstream cluster centroids, it is possible to improve the inference efﬁciency during meta-testing with the well-trained cluster centroids.
Note that the cluster-enhanced path embedding ˜ e path T i used to generate modulation is obtained based on the path cluster centroids and the soft assignment (see equation 3).
If we can estimate the path cluster assignment without actually going through the rehearsed learning process to obtain e path T i , we will be able to reduce the inference time by half.
Inspired by this, we devise a shortcut tunnel to predict the path cluster assignment directly from the feature cluster assignment of the same task.
The assumption behind is that there exists a one-toone mapping between the feature cluster assignments and the path cluster assignments, which can be linear or non-linear.
For better expressivity, we employ a two-layer fully connected network to approximate the mapping and use it to reconstruct the path cluster assignment of T i from its feature cluster assignment: [Equation] where q feati =
[ q feati 1 , ..., q featik feat ] ⊤ ∈ R k feat and ˆ q pathi =
[ˆ q pathi, ..., ˆ q pathik path ]
⊤ ∈ R k path .
The mapping handles cases where k path ∕
= k feat and allows for different softness of cluster assignments.
To align the reconstructed and the actual assignment distributions, we use the Jensen-Shannon (JS) divergence (a symmetrized version of Kullback-Leibler (KL) divergence) as the reconstruction loss:
[Equation]
[Equation]
where p i = 12 (ˆ q pathi + q pathi ) , and KL ( q || p ) =
" j q j log q j p j is the KL divergence.
During metatraining, the reconstruction loss is optimized together with the loss in equation 6, resulting in the following overall objective:
[Equation]
where ζ controls the weight of L r ( D tr T i ) , and φ now further includes the shortcut tunnel parameters.
To apply the shortcut tunnel during meta-testing, we simply replace q pathi with ˆ q pathi in equationto obtain the cluster-enhanced path embedding ˜ e path , i.e., ˜ e path T i = "
[Equation]
ˆ q pathij a pathj .
The overall meta-training and meta-testing procedures of CTML are summarized in Algorithm 1 and 2.
Required: Meta-training tasks T tr
} N tr i =1 ;
Meta feat do
B do
T to
˜ feat T
Extract e
T i rehearsed and cluster to e
T i
Aggregate ˜ e feat T i and ˜ e
T i to generate θ
i
Evaluate L ( θ T i
D te T i ) r
( D tr T i )
Meta-update φ ← φ
Required
Meta do approximation shortcut to
˜ e path T
i else
i
Aggregate
E XPERIMENTS
To test the effectiveness of the proposed CTML framework, we conduct experiments 3 on two realworld application domains: few-shot image classiﬁcation and cold-start recommendation
. 5. F EW -S HOT I MAGE C LASSIFICATION
Datasets and Settings In few-shot image classiﬁcation, each task is deﬁned as assigning images to N classes after training with K samples (i.e., N -way K -shot)
(Vinyals et al., 2016).
To simulate task heterogeneity, we construct the Mixture-Of-Datasets consisting of 6 widely used benchmark datasets: CUB-200-2011 (Bird) (Wah et al., 2011), FGVC-Aircraft (Aircraft) (Maji et al., 2013),
FGVCx-Fungi (Fungi) (Schroeder & Cui, 2018), VGG Flower (Flower) (Nilsback & Zisserman, 2008), Describable Textures (Texture) (Cimpoi et al., 2014), and GTSRB Trafﬁc Signs (Trafﬁc Sign) (Stallkamp et al., 2012).
Following Yao et al.
(2019), we create each task by sampling N classes from one of the 6 sub-datasets.
This step ensures that tasks are drawn from different underlying distributions.
Apart from the Mixture-Of-Datasets, we also conduct experiments on the MiniImagenet (Ravi & Larochelle, 2016), which can be considered as a task-homogeneous setting as tasks are sampled from a single dataset (results on MiniImagenet are discussed in Appendix D.2).
More details on datasets and pre-processing can be found in Appendix A.1.
Baselines and Our Method We compare the performance of our method against 5 baselines: (1) global initialization method: MAML (Finn et al., 2017); (2) task-speciﬁc initialization methods:
MMAML (feature-based customization) (Vuorio et al., 2019) and L2F (gradient-based customization)
(Baik et al., 2020); (3) cluster-enhanced initialization methods: HSML (feature-based hierarchical clustering)
(Yao et al., 2019) and ARML (feature-based relational graph) (Yao et al., 2020).
For our proposed CTML, we report both the original meta-testing performance (CTML) and the one applying shortcut approximation (CTML(approx.)).
We further implement 2 variants to compare with the baselines: CTML-feat which only depends on feature representation and CTML-path which only depends on path representation.
Following Finn et al.
, we adopt both the feature extractor E ( · ) and the base-learner f θ as a four-layer 3 × 3 convolutions with 32 ﬁlters for all the compared methods (experiments on deeper network for base-learner can be found in Appendix D.1).
Further details on hyper-parameters settings can be found in Appendix B.1.
Baseline Comparison Table 1a presents the 5-way 1-shot & 5-shot classiﬁcation performance on Mixture-Of-Datasets (disentangled performance on individual sub-datasets are included in Appendix D.3).
First of all, we see that all the task-aware methods outperform the globally initialized MAML in task-heterogeneous setting.
L2F with gradient-based conditioning on the initialization performs better than the feature-based MMAML.
After considering generalization across similar tasks with some clustering techniques, HSML and ARML further improve over MMAML.
Under our proposed CTML framework, CTML-feat with K-means clustering on features exhibits comparable performance as HSML and ARML.
For CTML-path, the improvements over baselines are more
[Table]
[Figure]
signiﬁcant, especially for the 5-shot scenario where the learning paths are more reliable with more training data.
The ﬁnal CTML combining the advantages of the two representations achieves further improvements and outperforms the strongest baseline by a signiﬁcant margin.
Nevertheless, we can see that the meta-testing time of CTML is about twice that of the baselines due to the additional rehearsed learning process.
After applying the shortcut approximation, we are able to cut down the inference time by half with just a small compromise on performance.
Ablation Study We further conduct ablation study to test the efﬁcacy of various CTML designs, including the step-wise input components, the path learner design, and the effects of clustering.
From the results in Table 1b, we see that the lower-order quantities seem more important.
Regarding the path learner design, our proposed GRU-based model achieves the best result due to its ability to capture the sequential dependencies among the steps.
Attention model is also a competitive alternative, as the positional encodings and the causal mask serve to constrain the sequential order (detailed descriptions of the different path learner designs can be found in Appendix C).
As for clustering, we see that it indeed helps to generalize better as the global structure is explicitly considered.
Understanding the Path Learning Mechanism To understand what has actually been learned by the GRU path learner, we randomly sample 6 tasks (5-way 1-shot) from Aircraft, Flower and Trafﬁc
Sign sub-datasets (Figure 3a) and visualize their path learning results.
In Figure 3b, we plot the 5step rehearsed learning paths (upper) and the corresponding path embeddings generated by the GRU (lower) on the ﬁrst two PCA components.
We can see that the learning paths of tasks from the same sub-dataset generally move along the same direction, and the corresponding path embeddings are able to encode the path geometric information and positioned closely for similar tasks in the latent space.
In Figure 3c, we further visulize the z gates and r gates of the GRU at each step t , which control how much to retain for the current step input and previous step memory respectively.
We can see that the GRU path learner is able to identify the “important” steps (darker-blue blocks) and absorbs a larger portion of the inputs at these steps into the ﬁnal path embedding.
More examples on path visualization can be found in Appendix D.
[Figure]
5. C OLD -S TART R ECOMMENDATION
Datasets and Settings In the ﬁeld of recommender systems, meta-learning has been applied to solve the cold-start problem, which refers to making recommendations related to new items or new users unseen before (Vartak et al., 2017; Volkovs et al., 2017; Lee et al., 2019).
Here, we focus on the user cold-start problem where recommending for each user is treated as a task.
This setting is task-heterogeneous in nature as users may belong to different preference groups.
We conduct experiments on 3 public datasets: MovieLens-1M (Harper & Konstan, 2015), Yelp (Dataset, 2019), and Amazon-CDs (McAuley et al., 2015), consisting of user’s ratings on movies, business services and CD products respectively.
We split each dataset into meta-train/val/test by ratio 7/1/2 according to the timestamps of ratings.
For each user, we use the ﬁrst 10 samples as training set, and the rest as test set.
Further details about datasets and pre-processing can be found in Appendix A.2.
Baselines and Our Method We choose 4 baselines that speciﬁcally tackle the user cold-start problem in recommender systems developed based on MAML.
MeLU (Lee et al., 2019) is the ﬁrst to adopt MAML in recommender systems with locally adapted decision layers; MetaHIN (Lu et al., 2020) further incorporates Heterogeneous Information Networks (HIN) for data augmentation and multi-faceted adaptations; MAMO (Dong et al., 2020) involves a global memory module to group users based on proﬁle information and personalizes the initial bias; PAML (Yu et al., 2021) personalizes the adaptation learning rate to allow for better ﬁtting of the minor users.
Following the baselines, we implement the feature extractor E ( · ) as several embedding lookup matrices for different features, and the base-learner f θ as a general recommender system model consists of embedding matrices followed by multiple fully-connected layers (Lee et al., 2019).
Detailed hyper-parameters settings can be found in Appendix B.2.
Baseline Comparison We evaluate the rating prediction performance in terms of mean absolute error (MAE) on 3 datasets, reported in Table 2 (more detailed results can be found in Appendix D.4).
From the results, we can see that PAML and MAMO with personalized adaptation rate and initialization achieve better performance than MELU.
Our CTML surpasses all the baselines, including
MetaHIN which leverages HIN for data augmentation.
Furthermore, we notice that for MovieLens,
CTML-feat is slightly better than CTML-path, while for the other two datasets, CTML-path signiﬁcantly outperforms CTML-feat.
Given that the MovieLens dataset contains more complete user proﬁle, this result implies the potential of representing users based on learning paths when the side information is seriously lacking or inaccurate, which is often the case in real-world applications.
Number of Clusters For recommendation problem, there is no ground-truth grouping of users.
In
Figure 4a, we investigate the effect of varying the number of clusters k path and k feat for MovieLens dataset.
We see that the best performance occurs at smaller k feat values and larger k path values, which implies greater complexity for clustering the learning paths.
Our design of the shortcut tunnel allows ﬂexibility to set different k path and k feat .
In Figure 4b, we visualize the shortcut approximation of path assignment from feature assignment for a random user of MovieLens.
We can see that the shortcut tunnel is rather reliable as the reconstructed path assignment highly resembles the actual path assignment.
More examples on shortcut approximation are provided in Appendix D.6.
C ONCLUSION
In this work, we introduce a CTML framework which leverages both features and learning paths for task representation.
We employ a GRU-based meta path learner to process the step-wise geometric quantities, and introduce a shortcut tunnel to bypass the rehearsed learning during meta-testing.
Experiments on two real-world application domains demonstrate the effectiveness of our framework.
R EPRODUCIBILITY S TATEMENT
Our code can be accessed at: https://github.com/didiya0825/ctml_code
For data pre-processing and hyper-parameters settings, please see Appendix A and B.
E THICS S TATEMENT
Since the proposed method is a general meta-learning framework we do not foresee any potential ethic issue to the best our knowledge
A D ATASETS AND P RE -P ROCESSING D ETAILS
If not specially mentioned, datasets are downloaded from the respective ofﬁcial websites, where we have tried our best to seek permission for usage if applicable.
The open-source datasets are licensed under CC0: Public Domain.
For recommendation datasets, all records are anonymized and no personal identiﬁable information is involved.
A. I MAGE D ATASETS
In our experiments, Mixture-Of-Datasets consist of six widely used benchmark datasets.
For the CUB-200-2011 (Bird), FGVC-Aircraft (Aircraft), FGVCx-Fungi (Fungi), and DTD (Texture) datasets, we directly acquire the pre-processed version released by the authors of HSML 4 (Yao et al., 2019).
Pre-processing details of these four sub-datasets can be found in the original paper of HSML.
We have obtained the authors’ consent to use this asset for our research.
For the VGG Flower (Flower) 5 and GTSRB (Trafﬁc Sign) 6 datasets, we download the raw datasets from the respective ofﬁcial websites.
We follow the pre-processing protocols of Finn et al.
(2017) to resize all the images to 84 × 84 × 3 .
We also try to maintain the same meta-train/validation/test ratio as Yao et al.
for class splits.
For VGG Flower, there are in total 102 ﬂower classes, and each class consists of 40 to 258 images.
For GTSRB, we use the ofﬁcial training data which containstrafﬁc sign classes, and each class has 30 images.
For MiniImagenet, we acquire the dataset as well as the meta-train/validation/test splits from Ravi & Larochelle (2016).
Table 3 summarizes the meta-train/validation/test splits for each of the subdatasets as well as MiniImagenet.
[Table]
A. R ECOMMENDATION D ATASETS A.2. M OVIE L ENS -1M
MovieLens-1M 7 dataset (Harper & Konstan, 2015) contains 1,000,209 anonymous ratings on around 3,900 movies made by 6,040 users.
We acquire the additional side information released by the authors of MeLU 8 (Lee et al., 2019), which contains user proﬁle information of gender, age group, occupation, zipcode, and movie proﬁle information of rate, genre, director and actor.
The dataset is split into meta-train/validation/test set according to the timestamps of ratings by ratio 7/1/2.
As a result, the start time is 2000-04-26, the meta-train cut-off time is 2000-11-22, the metavalidation cut-off time is 2000-12-02, and the end time is 2003-03-01.
Users with less thansamples in each split are removed.
In the end, the average sample size for each user is 163.
The statistics of the pre-processed MovieLens-1M dataset and the number of meta-train/validation/test users are recorded in Table 4 and 5.
Note that it is possible for meta-train users to appear in metavalidation and meta-test set, and these users are termed warm users .
Conversely, meta-validation or meta-test users that have never appeared in the meta-train set are termed cold users .
A.2. Y ELP
Yelp 9 dataset (Dataset, 2019) contains 8,635,403 reviews for 160,585 business services made by 2,189,457 users.
We extract the star ratings from the reviews for our experiments.
For each business service, we collect the city it resides and the categories it belongs to from the business metadata.
However, for the user, the personal information is lacking, and we only have access to the user ID.
The dataset is split into meta-train/validation/test set according to the timestamps of ratings by ratio 7/1/2.
As a result, the start time is 2004-10-12, the meta-train cut-off time is 2018-02-07, the metavalidation cut-off time is 2018-09-04, and the end time is 2019-12-13.
Users with less thansamples in each split are removed.
In the end, the average sample size for each user is 89.
The statistics of the pre-processed Yelp dataset and the number of meta-train/validation/test users are recorded in Table 4 and 5.
A.2. A MAZON -CD S
Amazon-CDs 10 dataset (McAuley et al., 2015) contains 3,749,004 ratings for 544,442 CD and Vinyl products made by 1,578,597 users.
For each CD, we collect its brand and category, while for the user, the personal proﬁle information is lacking and we only have access to the user ID.
The dataset is split into meta-train/validation/test set according to the timestamps of ratings by ratio 7/1/2.
As a result, the start time is 1997-09-13, the meta-train cut-off time is 2012-01-13, the metavalidation cut-off time is 2013-03-05, and the end time is 2014-07-23.
Users with less thansamples in each split are removed.
In the end, the average sample size for each user is 82.
The statistics of the pre-processed Amazon-CDs dataset and the number of meta-train/validation/test users are recorded in Table 4 and 5.
[Table] [Table]
com
[Table] [Table]
B H YPER -P ARAMETERS S ETTINGS B. F EW -S HOT I MAGE C LASSIFICATION
For fair comparison, we adopt the same feature extractor and base-learner for all the compared methods.
Following Finn et al.
, the feature extractor is a four-layer 3 × 3 convolutions with 32 ﬁlters, followed by a one-layer fully-connected layer that maps the extracted features into the task embedding dimension d e .
The base-learner is also a four-layer 3 × 3 convolutions withﬁlters, followed by a one-layer fully-connected layer that projects the output to prediction scores for 5 classes.
Each convolution layer is accompanied with batch normalization, ReLU nonlinearity, and 2 × 2 max-pooling.
For all the compared methods, we follow the hyper-parameters settings for meta-training in MAML (Finn et al., 2017), which sets the adaptation learning rate α as 0.01, the meta-update learning rate β as 0.001, the number of adaptation steps τ as 5, the meta batch size |B| as 4, the size of test set |D te T i | as 15, and the number of meta-update iterations as 60,000.
For all the task-aware modulation methods (i.e., MMAML (Vuorio et al., 2019), L2F (Baik et al., 2020), HSML (Yao et al., 2019),
ARML (Yao et al., 2020)), we set the dimension of task embedding d e as 128 (following Yao et al.
Additionally, for HSML (Yao et al., 2019), we set the number of clusters in the ﬁrst and second layer of hierarchical tree as 4 and 2 respectively.
For ARML (Yao et al., 2020), we set the number of vertices in the meta-knowledge graph as 4.
For our proposed CTML, we set the number of clusters for both feature and path representations as 6, i.e., k path =
k feat = 6 .
B. C OLD -S TART R ECOMMENDATION
Different from the image dataset, recommendation dataset typically consists of user-item interactions and proﬁle features for both sides.
The feature extractor is implemented as several embedding lookup tables, each corresponds to one categorical feature.
We set the embedding dimension asfor all features.
The base-learner is implemented as a general deep learning-based recommender system model, composed of embedding matrices followed by a two-layer fully connected layer that models the interaction between different features.
The embedding dimension for base-learner is set to 8, and the hidden sizes for the fully-connected layers is set to [32, 16].
For all the compared methods, we follow the hyper-parameters settings for meta-training in MeLU (Lee et al., 2019), which sets the adaptation learning rate α as 5e-3, the meta-update learning rate β as 5e-5, the number of adaptation steps τ as 5.
For other hyper-parameters that are not explicitly speciﬁed by Lee et al.
(2019), we set the meta batch size |B| as 1, the size of training set |D
tr T i | for each user as 10 (the remaining are used as test set D te T i for that user, whose size varies for different users), and the number of meta-training epochs as 20.
For MAMO (Dong et al., 2020) which also employs K-means clustering, we apply the same hyper-parameters tuning procedure as our CTML.
That is, we tune both k path and k feat in { 4 , 8 , 12 , 16 , 20 } , and tune the dimension of task embedding d e in { 64 , 128 , 256 , 512 , 1024 } .
Other hyper-parameters settings speciﬁc to the baseline methods are directly adopted from the source codes released by the authors.
C O THER P ATH L EARNER D ESIGNS
In addition to the GRU-based meta path learner design described in equation 2, we also experiment on 3 other network designs to test their effectiveness in modeling the step-wise geometric quantities.
Recall that the meta path learner is applied coordinate-wise on the base-learner parameters.
That is, the same meta path learner is used to process the sequence of step-wise tuples P d T i ∈ R ( τ +1) ×independently for all parameters at different dimensions d ∈ { 1 , 2 , ..., D } .
The ﬁrst design linearly transforms the vectorized P d T i with learnable weight and bias:
[Equation]
The second one employs multiple fully-connected layers with nonlinear activation σ ( · ) :
[Equation]
The third design applies multiple self-attention layers on the sequence with scaled dot-product similarity following Vaswani et al.
To inform about the sequential order, we apply sinusoid positional encodings at the input level, and also apply causal mask to prevent interaction with future steps (i.e., set the upper triangular entries of the similarity matrix to −∞ before inputting into softmax).
The ﬁnal representation h di is given by the ( τ + 1) -th row (i.e., the last row) of the ﬁnal-layer output:
Attention :
[Equation]
where Q l = σ ( U l − 1 · W lQ + b lQ
[Equation]
Concatenating the path representations at all D dimensions and passing it through a fully-connected layer gives the ﬁnal path embedding e path T i ∈
R d e for task T i :
[Equation]
D A DDITIONAL E XPERIMENTAL R ESULTS D. P ERFORMANCE ON R ES N ET -
We have conducted additional experiments to test the effectiveness of our method using a deeper network ResNet-12 as the backbone (i.e., the base-learner).
We follow the implementation of Sun et al.
(2019) which adopts (64-128-256-512) for the number of ﬁlters at the four blocks.
It was shown recently that, with a deeper backbone, simple training paradigm like pre-training and ﬁne-tuning (Tian et al., 2020; Chen et al., 2018) can achieve very satisfactory performance.
We therefore further include the following 3 baselines for comparison on ResNet-12: • Finetune: This baseline simply pretrains a global feature extractor on the entire training dataset and then ﬁnetunes the classiﬁer for individual tasks. • Finetune-distance Chen et al., 2018):
This method modiﬁes Finetune by computing cosine distance between the input feature vector and the class vectors for the classiﬁer.
• Finetune-distill (Tian et al., 2020):
This method modiﬁes Finetune by employing a sequential self-distillation technique to pretrain the feature extractor.
Table 6 below presents the 5-way 1-shot performance on the Mixture-Of-Datasets with 2 different backbones: Conv-4 and ResNet-12.
We can see that the gaps between Finetune methods and
MAML are narrower for deeper backbone, which can be attributed to that the deeper models have larger capacity to accommodate the transferable knowledge obtained from the pre-training (in line with the ﬁndings of Chen et al.
However, under this task-heterogeneous setting, the taskconditioned methods still yield better performance than MAML and the Finetune baselines for
ResNet-12, demonstrating the effectiveness of task-conditioning even for deeper backbone.
And our method with path representation yields the best result.
[Table]
D. P ERFORMANCE ON M INI I MAGENET
Table 7 presents the results on MiniImagenet benchmark (Ravi & Larochelle, 2016)
[Table]
where tasks are constructed from a single dataset (i [Table] e., there is only one underlying distribution) and hence can be considered as a task-homogeneous setting. Since in this case, task-aware methods do not have advantages over the global initialization methods, we see that all the baselines have comparable performance as the original MAML.
Nevertheless, the results show that though the task-aware modulation is rendered less meaningful in task-homogeneous setting, CTML will yield performance at least comparable to the global initialization method in this scenario [Table]
[Table]
D. P ERFORMANCE ON S UB -D ATASETS
Table 8 presents the few-shot classiﬁcation performance of CTML and the baselines on each subdataset.
We see that CTML consistently outperforms all the baselines for all the sub-datasets.
Other trends are similar to that of the overall results in Table 1a.
[Table]
P ERFORMANCE ON R ECOMMENDATION D ATASETS
Table 9 presents the rating prediction performance on 3 datasets evaluated in two metrics: mean absolute error (MAE) and normalized discounted cumulative gain at top 20 (NDCG@20).
The meta-testing users are further divided into warm and cold users, where warm users are those that have appeared in the meta-training set [Table] and cold users are those that have not 11 .
From the results, we see that CTML consistently outperforms the baselines.
[Table]
D. A DDITIONAL L EARNING P ATH V ISUALIZATION D.5. F EW -S HOT I MAGE C LASSIFICATION
Here we provide more examples for visualizing the path learning results for the few-shot image classiﬁcation experiments.
Figure 5, 6 and 7 each shows the path learning results of 6 tasks randomly sampled from 3 of the 6 sub-datasets.
We see that in this application scenario, the overall direction of the learning path plays an important role in determining the task representation.
Looking at the GRU gates at each step, it seems that the ﬁrst 2 to 3 steps are often the most important/informative steps, after which the learning begins to converge and the subsequent steps no longer provide useful information about the learning direction.
[Figure] [Figure]
[Figure]
D.5. C OLD -S TART R ECOMMENDATION
We also provide examples for visualizing the learning paths for the cold-start recommendation experiments.
Figure 8 and 9 each shows the path learning results of 6 users randomly selected from the MovieLens-1M dataset, and Figure 10 and 11 each shows the results of 6 users sampled from the Amazon-CDs dataset.
In this application scenario we notice that apart from the overall direction of the path, the length of the path also plays an important role in determining the task representation.
For instance, in Figure 11, the blue, orange, green and brown paths (upper plot) are clustered closely in the representation space (lower plot) due to similar lengths, although they are pointing at very different directions.
Similar observation can also be found in Figure 8.
Although blue and orange paths are pointing at the same directions, their embeddings are positioned far away from each other due to the big difference in path lengths.
D. A DDITIONAL S HORTCUT A PPROXIMATION V ISUALIZATION
In this section, we give more examples to verify the effectiveness of the shortcut approximation.
Figure 12 shows the approximation results of 2 users sampled from the MovieLens-1M dataset, and
Figure 13 shows the results of 3 tasks sampled from 3 different image sub-datasets.
We can see that the shortcut tunnel is able to learn and accommodate different mappings, provide guarantee on the performance of the approximated model.
[Figure] [Figure]