A UTOMATED R ELATIONAL M ETA - LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In order to efﬁciently learn with small amount of data on new tasks, meta-learning transfers knowledge learned from previous tasks to the new ones.
However, a critical challenge in meta-learning is the task heterogeneity which cannot be well handled by traditional globally shared meta-learning methods.
In addition, current task-speciﬁc meta-learning methods may either suffer from hand-crafted structure design or lack the capability to capture complex relations between tasks.
In this paper, motivated by the way of knowledge organization in knowledge bases, we propose an automated relational meta-learning (ARML) framework that automatically extracts the cross-task relations and constructs the meta-knowledge graph.
When a new task arrives, it can quickly ﬁnd the most relevant structure and tailor the learned structure knowledge to the meta-learner.
As a result, the proposed framework not only addresses the challenge of task heterogeneity by a learned meta-knowledge graph, but also increases the model interpretability.
We conduct extensive experiments on 2D toy regression and few-shot image classiﬁcation and the results demonstrate the superiority of ARML over state-of-the-art baselines.
I NTRODUCTION
Learning quickly is the key characteristic of human intelligence, which remains a daunting problem in machine intelligence.
The mechanism of meta-learning is widely used to generalize and transfer prior knowledge learned from previous tasks to improve the effectiveness of learning on new tasks, which has beneﬁted various applications, ranging from computer vision (Kang et al., 2019; Liu et al., 2019) to natural language processing (Gu et al., 2018; Lin et al., 2019).
Most of existing meta-learning algorithms learn a globally shared meta-learner (e.g., parameter initialization (Finn et al., 2017; 2018), meta-optimizer (Ravi & Larochelle, 2016), metric space (Snell et al., 2017; Garcia & Bruna, 2017)).
However, globally shared meta-learners fail to handle tasks lying in different distributions, which is known as task heterogeneity (Vuorio et al., 2019; Yao et al., 2019).
Task heterogeneity has been regarded as one of the most challenging issues in meta-learning, and thus it is desirable to design meta-learning models that effectively optimize each of the heterogeneous tasks.
The key challenge to deal with task heterogeneity is how to customize globally shared meta-learner by using task-speciﬁc information?
Recently, a handful of works try to solve the problem by learning a task-speciﬁc representation for tailoring the transferred knowledge to each task (Oreshkin et al., 2018; Vuorio et al., 2019; Lee & Choi, 2018).
However, the expressiveness of these methods is limited due to the impaired knowledge generalization between highly related tasks.
Recently, learning the underlying structure across tasks provides a more effective way for balancing the customization and generalization.
Representatively, Yao et al.
propose a hierarchically structured meta-learning method to customize the globally shared knowledge to each cluster (Yao et al., 2019).
Nonetheless, the hierarchical clustering structure completely relies on the handcrafted design which needs to be tuned carefully and may lack the capability to capture complex relationships.
Hence, we are motivated to propose a framework to automatically extract underlying relational structures from historical tasks and leverage those relational structures to facilitate knowledge customization on a new task.
This inspiration comes from the way of structuring knowledge in knowledge bases (i.e., knowledge graphs).
In knowledge bases, the underlying relational structures across text entities are automatically constructed and applied to a new query to improve the searching efﬁciency.
In the meta-learning problem, similarly, we aim at automatically establishing the metaknowledge graph between prior knowledge learned from previous tasks.
When a new task arrives, it queries the meta-knowledge graph and quickly attends to the most relevant entities (vertices), and then takes advantage of the relational knowledge structures between them to boost the learning effectiveness with the limited training data.
The proposed meta-learning framework is named as A utomated R elational M eta-
L earning (ARML).
Speciﬁcally, the ARML automatically builds the meta-knowledge graph from meta-training tasks to memorize and organize learned knowledge from historical tasks, where each vertex represents one type of meta-knowledge (e.g., the common contour between birds and aircrafts).
To learn the meta-knowledge graph at meta-training time, for each task, we construct a prototype-based relational graph for each class, where each vertex represents one prototype.
The prototype-based relational graph not only captures the underlying relationship behind samples, but alleviates the potential effects of abnormal samples.
The meta-knowledge graph is then learned by summarizing the information from the corresponding prototype-based relational graphs of meta-training tasks.
After constructing the meta-knowledge graph, when a new task comes in, the prototype-based relational graph of the new task taps into the meta-knowledge graph for acquiring the most relevant knowledge, which further enhances the task representation and facilitates its training process.
Our major contributions of the proposed ARML are three-fold: (1) it automatically constructs the meta-knowledge graph to facilitate learning a new task; (2) it empirically outperforms the state-ofthe-art meta-learning algorithms; (3) the meta-knowledge graph well captures the relationship among tasks and improves the interpretability of meta-learning algorithms.
R ELATED W ORK
Meta-learning designs models to learn new tasks or adapt to new environments quickly with a few training examples.
There are mainly three research lines of meta-learning: (1) black-box amortized methods design black-box meta-learners to infer the model parameters (Ravi & Larochelle, 2016;
Andrychowicz et al., 2016; Mishra et al., 2018; Gordon et al., 2019); (2) gradient-based methods aim to learn an optimized initialization of model parameters, which can be adapted to new tasks by a few steps of gradient descent (Finn et al., 2017; 2018; Lee & Choi, 2018; Yoon et al., 2018; Grant et al., 2018); (3) non-parametric methods combine parametric meta-learners and non-parametric learners to learn an appropriate distance metric for few-shot classiﬁcation (Snell et al., 2017; Vinyals et al., 2016;
Yang et al., 2018; Oreshkin et al., 2018; Yoon et al., 2019; Garcia & Bruna, 2017).
Our work is built upon the gradient-based meta-learning methods.
In the line of gradient-based meta-learning, most algorithms learn a globally shared meta-learners from previous tasks (Finn et al., 2017; Li et al., 2017; Flennerhag et al., 2019), to improve the effectiveness of learning process on new tasks.
However, these algorithms typically lack the ability to handle heterogeneous tasks (i.e., tasks sample from sufﬁcient different distributions).
To tackle this challenge, recent works tailor the globally shared initialization to different tasks by leveraging task-speciﬁc information (Lee &
Choi, 2018; Vuorio et al., 2019) and using probabilistic models (Yoon et al., 2018; Finn et al., 2018).
Recently, HSML customizes the globally shared initialization with a manually designed hierarchical clustering structure to balance the generalization and customization (Yao et al., 2019).
However, the hand-crafted designed hierarchical structure may not accurately reﬂect the real structure and the clustering structure constricts the complexity of relationship.
Compared with these methods, ARML leverages the most relevant structure from the automatically constructed meta-knowledge graph.
Thus,
ARML not only discovers more accurate underlying structures to improve the effectiveness of metalearning algorithms, but also the meta-knowledge graph further enhances the model interpretability.
P RELIMINARIES
Few-shot Learning Considering a task T i , the goal of few-shot learning is to learn a model with a dataset D i
=
{D tri , D tsi } , where the labeled training set D tri = { x trj , y trj |∀ j ∈ [1 , N tr ] } only has a few samples and D tsi represents the corresponding test set.
A learning model (a.k.a., base model) f with parameters θ are used to evaluate the effectiveness on D tsi by minimizing the expected empirical loss on D tri , i.e., L ( D tr T i , θ ) , and obtain the optimal parameters θ i .
For the regression problem, the loss function is deﬁned based on the mean square error (i.e., (cid:80) ( x j , y j ) ∈D tri (cid:107) f θ ( x j ) − y j (cid:107) 22 ) and for the classiﬁcation problem, the loss function uses the cross entropy loss (i.e., − (cid:80) ( x j , y j ) ∈D tri log p ( y j | x j , f θ ) ).
Usually, optimizing and learning parameter θ for the task T i with a few labeled training samples is difﬁcult.
To address this limitation, meta-learning provides us a new perspective to improve the performance by leveraging knowledge from multiple tasks.
Meta-learning and Model-agnostic Meta-learning In meta-learning, a sequence of tasks {T 1 , ..., T I } are sampled from a task-level probability distribution p ( T ) , where each one is a few-shot learning task.
To facilitate the adaption for incoming tasks, the meta-learning algorithm aims to ﬁnd a well-generalized meta-learner on I training tasks at meta-learning phase.
At meta-testing phase, the optimal meta-learner is applied to adapt the new tasks T t .
In this way, meta-learning algorithms are capable of adapting to new tasks efﬁciently even with a shortage of training data for a new task.
Model-agnostic meta-learning (MAML)
(Finn et al., 2017), one of the representative algorithms in gradient-based meta-learning, regards the meta-learner as the initialization of parameter θ , i.e., θ 0 , and learns a well-generalized initialization θ ∗ 0 during the meta-training process.
The optimization problem is formulated as (one gradient step as exemplary):
[Equation]
At the meta-testing phase, to obtain the adaptive parameter θ t for each new task T t , we ﬁnetune the initialization of parameter θ ∗ 0 by performing gradient updates a few steps, i.e., f θ t = f
[Equation]
L ( f θ , D trt ) .
M ETHODOLOGY
In this section, we introduce the details of the proposed ARML.
To better explain how it works, we show its framework in Figure 1.
The goal of ARML is to facilitate the learning process of new tasks by leveraging transferable knowledge learned from historical tasks.
To achieve this goal, we introduce a meta-knowledge graph, which is automatically constructed at the meta-training time, to organize and memorize historical learned knowledge.
Given a task, which is built as a prototypebased relational structure, it taps into the meta-knowledge graph to acquire relevant knowledge for enhancing its own representation.
The enhanced prototype representations further aggregate and incorporate with meta-learner for fast and effective adaptions by utilizing a modulating function.
In the following subsections, we elaborate three key components: prototype-based sample structuring, automated meta-knowledge graph construction and utilization, and task-speciﬁc knowledge fusion and adaptation, respectively.
[Figure]
[Equation]
4. P ROTOTYPE - BASED S AMPLE S TRUCTURING
Given a task which involves either classiﬁcations or regressions regarding a set of samples, we ﬁrst investigate the relationships among these samples.
Such relationship is represented by a graph, called prototype-based relational graph in this work, where the vertices in the graph denote the prototypes of different classes while the edges and the corresponding edge weights are created based on the similarities between prototypes.
Constructing the relational graph based on prototypes instead of raw samples allows us to alleviate the issue raised by abnormal samples.
As the abnormal samples, which locate far away from normal samples, could pose signiﬁcant concerns especially when only a limited number of samples are available for training.
Speciﬁcally, for classiﬁcation problem, the prototype, denoted by c ki ∈ R d , is deﬁned as:
[Equation]
where N trk denotes the number of samples in class k .
E is an embedding function, which projects x j into a hidden space where samples from the same class are located closer to each other while samples from different classes stay apart.
For regression problem, it is not straightforward to construct the prototypes explicitly based on class information.
Therefore, we cluster samples by learning an assignment matrix P i ∈
R K × N tr .
Speciﬁcally, we formulate the process as:
[Equation]
where P i [ k ] represents the k -th row of P i .
Thus, training samples are clustered to K clusters, which serve as the representation of prototypes.
After calculating all prototype representations { c ki |∀ k ∈ [1 , K ] } , which serve as the vertices in the the prototype-based relational graph R i , we further deﬁne the edges and the corresponding edge weights.
The edge weight A R i ( c ji , c mi ) between two prototypes c ji and c mi is gauged by the the similarity between them.
Formally:
[Equation]
where W r and b r represents learnable parameters, γ r is a scalar and σ is the Sigmoid function, which normalizes the weight between 0 and 1 .
For simplicity, we denote the prototype-based relational graph as R i =
( C R i , A R i ) , where C R i = { c ji |∀ j ∈ [1 , K ] } ∈ R K × d represent a set of vertices, with each one corresponds to the prototype from a class, while A R i = {| A R i ( c ji , c mi ) |∀ j, m ∈ [1 , K ] } ∈ R K × K gives the adjacency matrix, which indicates the proximity between prototypes.
4. A UTOMATED M ETA - KNOWLEDGE G RAPH C ONSTRUCTION AND U TILIZATION
In this section, we ﬁrst discuss how to organize and distill knowledge from historical learning process and then expound how to leverage such knowledge to beneﬁt the training of new tasks.
To organize and distill knowledge from historical learning process, we construct and maintain a meta-knowledge graph.
The vertices represent different types of meta-knowledge (e.g., the common contour between aircrafts and birds) and the edges are automatically constructed to reﬂect the relationship between meta-knowledge.
When serving a new task, we refer to the meta-knowledge, which allows us to efﬁciently and automatically identify relational knowledge from previous tasks.
In this way, the training of a new task can beneﬁt from related training experience and get optimized much faster than otherwise possible.
In this paper, the meta-knowledge graph is automatically constructed at the meta-training phase.
The details of the construction are elaborated as follows:
Assuming the representation of an vertex g is given by h g ∈ R d , we deﬁne the meta-knowledge graph as G =
( H G , A G ) , where H G = { h j |∀ j ∈ [1 , G ] } ∈ R G × d and A G =
{ A G ( h j , h m ) |∀ j, m ∈ [1 , G ] } ∈ R G × G denote the vertex feature matrix and vertex adjacency matrix, respectively.
To better explain the construction of the meta-knowledge graph, we ﬁrst discuss the vertex representation H G .
During meta-training, tasks arrive one after another in a sequence and their corresponding vertices representations are expected to be updated dynamically in a timely manner.
Therefore, the vertex representation of meta-knowledge graph are deﬁned to get parameterized and learned at the training time.
Moreover, to encourage the diversity of meta-knowledge encoded in the meta-knowledge graph, the vertex representations are randomly initialized.
Analogous to the deﬁnition of weight in the prototype-based relational graph R i in equation 4, the weight between a pair of vertices j and m is constructed as:
[Equation]
where W o and b o represent learnable parameters and γ o is a scalar.
To enhance the learning of new tasks with involvement of historical knowledge, we query the prototype-based relational graph in the meta-knowledge graph to obtain the relevant knowledge in history.
The ideal query mechanism is expected to optimize both graph representations simultaneously at the meta-training time, with the training of one graph facilitating the training of the other.
In light of this, we construct a super-graph S i by connecting the prototype-based relational graph R i with the meta-knowledge graph G for each task T i .
The union of the vertices in R i and G contributes to the vertices in the super-graph.
The edges in R i and G are also reserved in the super-graph.
We connect
R i with G by creating links between the prototype-based relational graph with the meta-knowledge graph.
The link between prototype c ji in prototype-based relational graph and vertex h m in metaknowledge graph is weighted by the similarity between them.
More precisely, for each prototype c ji , the link weight A S ( c ji , h m ) is calculated by applying softmax over Euclidean distances between c ji and { h m |∀ m ∈
[1 , G ] } as follows:
[Equation]
where γ s is a scaling factor.
We denote the intra-adjacent matrix as A S =
{ A S ( c ji , h m ) |∀ j ∈ [1 , K ] , m ∈ [1 , G ] } ∈ R K × G .
Thus, for task T i , the adjacent matrix and feature matrix of super-graph
S i
=
( A i , H i ) is deﬁned as A i =
( A R i , A S ; A T S ,
A G ) ∈ R ( K + G ) × ( K + G ) and H i =
( C R i ; H G )
∈ R ( K + G ) × d , respectively.
After constructing the super-graph S i , we are able to propagate the most relevant knowledge from meta-knowledge graph G to the prototype-based relational graph R i by introducing a Graph Neural
Networks (GNN).
In this work, following the “message-passing” framework (Gilmer et al., 2017), the GNN is formulated as:
[Equation]
where MP( · ) is the message passing function and has several possible implementations (Hamilton et al., 2017; Kipf & Welling, 2017; Veliˇckovi´c et al., 2018), H ( l ) i is the vertex embedding after l layers of GNN and W ( l ) is a learnable weight matrix of layer l .
The input H (0) i
= H i .
After stacking
L GNN layers, we get the information-propagated feature representation for the prototype-based relational graph R i as the top- K rows of H ( L ) i , which is denoted as ˆ C R
i
=
{ ˆ c ji | j ∈ [1 , K ] } .
4. T ASK - SPECIFIC K NOWLEDGE F USION AND A DAPTATION
After propagating information form meta-knowledge graph to prototype-based relational graph, in this section, we discuss how to learn a well-generalized meta-learner for fast and effective adaptions to new tasks with limited training data.
To tackle the challenge of task heterogeneity, in this paper, we incorporate task-speciﬁc information to customize the globally shared meta-learner (e.g., initialization here) by leveraging a modulating function, which has been proven to be effective to provide customized initialization in previous studies (Wang et al., 2019; Vuorio et al., 2019).
The modulating function relies on well-discriminated task representations, while it is difﬁcult to learn all representations by merely utilizing the loss signal derived from the test set D tsi .
To encourage such stability, we introduce two reconstructions by utilizing two auto-encoders.
There are two collections of parameters, i.e, C R i and ˆ C R
i , which contribute the most to the creation of the task-speciﬁc meta-learner.
C R i express the raw prototype information without tapping into the meta-knowledge graph, while ˆ C R i give the prototype representations after absorbing the relevant knowledge from the meta-knowledge graph.
Therefore, the two reconstructions are built on C R i and ˆ C R i .
To reconstruct
C R i , an aggregator AG q ( · ) (e.g., recurrent network, fully connected layers) is involved to encode C R i into a dense representation, which is further fed into a decoder AG qdec ( · ) to achieve reconstructions.
Then, the corresponded task representation q i of C R i is summarized by applying a mean pooling operator over prototypes on the encoded dense representation.
Formally,
[Equation]
Similarly, we reconstruct ˆ C R i and get the corresponded task representation t i as follows:
[Equation]
The reconstruction errors in Equations 8 and 9 pose an extra constraint to enhance the training stability, leading to improvement of task representation learning.
Require: p ( T ) : distribution over tasks; K : Number of vertices in meta-knowledge graph; α : stepsize for gradient descent of each task (i.e., inner loop stepsize); β : stepsize for meta-optimization (i.e., outer loop stepsize); µ 1 , µ 2 : balancing factors in loss function 1: Randomly initialize all learnable parameters Φ 2: while not done do 3:
Sample a batch of tasks {T i | i ∈ [1 , I ] } from p ( T ) 4: for all T i do 5:
Sample training set D tri and testing set D tsi 6:
Construct the prototype-based relational graph R i by computing prototype in equationand weight in equation7:
Compute the similarity between each prototype and meta-knowledge vertex in equationand construct the super-graph S i 8:
Apply GNN on super-graph S i and get the information-propagated representation ˆ C R i 9:
Aggregate C R i in equation
and ˆ C R i in equation 9 to get the representations
q i , t i and reconstruction loss L q , L t 10:
Compute the task-speciﬁc initialization θ
i in equation 10 and update parameters θ i
= θ
i − α ∇ θ L ( f θ , D tri ) 11: end for 12:
Update Φ ← Φ − β ∇ Φ (cid:80) Ii =
L
( f θ i , D tsi ) + µ i L t + µ 2 L q 13: end while
After getting the task representation q i and t i , the modulating function is then used to tailor the task-speciﬁc information to the globally shared initialization θ 0 , which is formulated as:
[Equation]
) where W g and b g is learnable parameters of a fully connected layer.
Note that we adopt the Sigmoid gating as exemplary and more discussion about different modulating functions can be found in ablation studies of Section 5.
For each task T i , we perform the gradient descent process from θ
i and reach its optimal parameter θ i .
Combining the reconstruction loss L t and L q with the meta-learning loss deﬁned in equation 1, the overall objective function of ARML is:
[Equation]
where µ 1 and µ 2 are introduced to balance the importance of these three items.
Φ represents all learnable parameters.
The algorithm of meta-training process of ARML is shown in Alg.
The details of the meta-testing process of ARML are available in Appendix A.
E XPERIMENTS
In this section, we conduct extensive experiments to demonstrate the effectiveness of the ARML on 2D regression and few-shot classiﬁcation with the goal of answering the following questions: (1) Can
ARML outperform other meta-learning methods?
; (2) Can our proposed components improve the learning performance?
; (3) Can ARML framework improve the model interpretability by discovering reasonable meta-knowledge graph?
5. E XPERIMENTAL S ETTINGS
Methods for Comparison We compare our proposed ARML with two types of baselines: (1)
Gradient-based meta-learning methods : both globally shared methods (MAML (Finn et al., 2017),
Meta-SGD (Li et al., 2017)) and task-speciﬁc methods (MT-Net (Lee & Choi, 2018), MUMOMAML (Vuorio et al., 2019), HSML (Yao et al., 2019), BMAML (Yoon et al., 2018)) are considered for comparison.
(2) Other meta-learning methods (non-parametric and black box amortized methods) : we select globally shared methods VERSA (Gordon et al., 2019), Prototypical Network (ProtoNet) (Snell et al., 2017),
TapNet (Yoon et al., 2019) and task-speciﬁc method TADAM (Oreshkin et al., 2018) as baselines.
Following the traditional settings, non-parametric baselines are only used in few-shot classiﬁcation.
Detailed implementations of baselines are discussed in Appendix B.3.
Hyperparameter Settings For the aggregated function in autoencoder structure ( AG t , AG tdec AG q ,
AG qdec ), we use the GRU as the encoder and decoder in this structure.
We adopt one layer GCN (Kipf &
Welling, 2017) with tanh activation as the implementation of GNN in equation 7.
For the modulation network, we test sigmoid, tanh and Film modulation, and ﬁnd that sigmoid modulation achieves best performance.
Thus, in the future experiment, we set the sigmoid modulation as modulating function.
More detailed discussion about experiment settings are presented in Appendix B.
5. 2D R EGRESSION
Dataset Description.
In 2D regression problem, we adopt the similar regression problem settings as (Finn et al., 2018;
Vuorio et al., 2019; Yao et al., 2019; Rusu et al., 2019), which includes several families of functions.
In this paper, to model more complex relational structures, we design a 2D regression problem rather than traditional 1D regression.
Input x ∼ U [0 . 0 , 5 . 0] and y ∼ U [0 . 0 , 5 . 0] are sampled randomly and random Gaussian noisy with standard deviation 0.3 is added to the output.
Furthermore, six underlying functions are selected, including (1) Sinusoids:
z ( x, y ) =
a s sin ( w s x + b s ) , where a s ∼ U [0 . 1 , 5 . 0] , b s ∼ U
[0 , 2 π ]
w s ∼ U [0 . 8 , 1 . 2] ; (2) Line: z ( x, y )
= a l x + b l , where a l ∼ U [ − 3 . 0 , 3 . 0] , b l ∼ U [ − 3 . 0 , 3 . 0] ; (3) Quadratic: z ( x, y )
=
a q x 2 + b q x + c q , where a q ∼
U [ − 0 . 2 , 0 . 2] , b q ∼ U [ − 2 . 0 , 2 . 0] , c q ∼ U [ − 3 . 0 , 3 . 0] ; (4) Cubic: z ( x, y )
= a c x 3 + b c x 2 + c c x + d c , where a c ∼ U [ − 0 .
1 , 0 . 1] , b c ∼ U [ − 0 .
2] , c c ∼ U [ − 2 . 0 , 2 . 0] ,
d c ∼ U [ − 3 . 0 , 3 . 0] ; (5)
Quadratic
Surface: z ( x, y )
= a qs x 2 + b qs y 2 , where a qs ∼ U [ − 1 . 0 , 1 . 0] , b qs ∼ U [ − 1 . 0 , 1 . 0] ; (6) Ripple: z ( x, y ) =
sin ( − a r ( x 2 + y 2 ))
+ b r , where a r ∼ U [ − 0 .
2 , 0 . 2] , b r ∼ U [ − 3 . 0 , 3 . 0] .
Note that, function 1-4 are located in the subspace of y = 1 .
Follow (Finn et al., 2017), we use two fully connected layers with 40 neurons as the base model.
The number of vertices of meta-knowledge graph is set as 6.
Results and Analysis.
In Figure 2, we summarize the interpretation of meta-knowledge graph (see top ﬁgure, and more cases are provided in Figure 8 of Appendix H.2) and the the qualitative results (see bottom table) of 10-shot 2D regression.
In the bottom table, we can observe that ARML achieves the best performance as compared to competitive gradient-based meta-learning methods, i.e., globally shared models and task-speciﬁc models.
This ﬁnding demonstrates that the meta-knowledge graph is necessary to model and capture task-speciﬁc information.
The superior performance can also be interpreted in the top ﬁgure.
In the left, we show the heatmap between prototypes and metaknowledge vertices (darker color means higher similarity).
We can see that sinusoids and line activate
V1 and V4, which may represent curve and line, respectively.
V1 and V4 also contribute to quadratic and quadratic surface, which also show the similarity between these two families of functions.
V3 is activated in P0 of all functions and the quadratic surface and ripple further activate V1 in P0, which may show the different between 2D functions and 3D functions (sinusoid, line, quadratic and cubic lie in the subspace).
Speciﬁcally, in the right ﬁgure, we illustrate the meta-knowledge graph, where we set a threshold to ﬁlter the link with low similarity score and show the rest.
We can see that V3 is the most popular vertice and connected with V1, V5 (represent curve) and V4 (represent line).
V1 is further connected with V5, demonstrating the similarity of curve representation.
[Figure]
5. F EW - SHOT C LASSIFICATION
Dataset Description and Settings
In the few-shot classiﬁcation problem, we ﬁrst use the benchmark proposed in (Yao et al., 2019), where four ﬁne-grained image classiﬁcation datasets are included (i.e., CUB-200-2011 (Bird), Describable Textures Dataset (Texture), FGVC of Aircraft (Aircraft), and FGVCx-Fungi (Fungi)).
For each few-shot classiﬁcation task, it samples classes from one of four datasets.
In this paper, we call this dataset as Plain-Multi and each ﬁne-grained dataset as subdataset.
Then, to demonstrate the effectiveness of our proposed model for handling more complex underlying structures, in this paper, we increase the difﬁculty of few-shot classiﬁcation problem by introducing two image ﬁlters: blur ﬁlter and pencil ﬁlter.
Similar as (Jerfel et al., 2019), for each image in PlainMulti, one artistic ﬁlters are applied to simulate a changing distribution of few-shot classiﬁcation tasks.
After applying the ﬁlters, the total number of subdatasets is 12 and each tasks is sampled from one of them.
This data is named as Art-Multi .
More detailed descriptions of the effect of different ﬁlters is discussed in Appendix C.
Following the traditional meta-learning settings, all datasets are divided into meta-training, metavalidation and meta-testing classes.
The traditional N-way K-shot settings are used to split training and test set for each task.
We adopt the standard four-block convolutional layers as the base learner (Finn et al., 2017; Snell et al., 2017) for ARML and all baselines for fair comparison.
The number of vertices of meta-knowledge graph for Plain-Multi and Art-Multi datasets are set as 4 and 8, respectively.
Additionally, for the miniImagenet and tieredImagenet, similar as (Finn et al., 2018), which tasks are constructed from a single domain and do not have heterogeneity, we compare our proposed ARML with other gradient-based meta-learning models and present the results in Appendix D.
5.3. P ERFORMANCE V ALIDATION
Overall Qualitative Analyses Experimental results for Plain-Multi and Art-Multi are shown in
Table 1 and Table 2, respectively.
For each dataset, the performance accuracy with 95% conﬁdence interval is reported.
Due to the space limitation, in Art-Multi dataset, we only show the average value of each ﬁlter here.
The full results are shown in Table 10 of Appendix E.
In these two tables, ﬁrst, we can observe that task-speciﬁc gradient-based models (MT-Net, MUMOMAML, HSML, BMAML) signiﬁcantly outperforms globally shared models (MAML, Meta-SGD).
Second, compared ARML with other task-speciﬁc gradient-based meta-learning methods, the better performance conﬁrms that
ARML can model and extract task-speciﬁc information more accurately by leveraging the constructed meta-knowledge graph.
Especially, the performance gap between the ARML and HSML veriﬁes the beneﬁts of relational structure compared with isolated clustering structure.
Finally, as a gradient-based meta-learning algorithm, ARML can also outperform methods of other research lines (i.e., ProtoNet,
TADAM, TapNet and VERSA).
Model Ablation Study
In this section, we perform the ablation study of the proposed ARML to demonstrate the effectiveness of each component.
The results of ablation study on 5-way, 5-shot scenario for Art-Multi dataset are presented in Table 3.
In Appendix F, we show the full results for Art-Multi in Table 6 and the ablation study of Plain-Multi in Table 7.
Speciﬁcally, to show the effectiveness of prototype-based relational graph, in ablation
I, we apply the mean pooling to aggregate each sample and then feed it to interact with meta-knowledge graph.
In ablation II, we use all samples to construct the sample-level relational graph without constructing prototype.
In ablation III, we remove the links between prototypes.
Compared with ablation I, II and III, the better performance of ARML shows that structuring samples can (1) better handling the underlying relations (2) alleviating the effect of potential anomalies by structuring samples as prototypes.
In ablation IV, we remove the meta-knowledge graph and use the prototype-based relational graph with aggregator AG q as the task representation.
The better performance of ARML demonstrates the effectiveness of meta-knowledge graph for capturing the relational structure and facilitating the classiﬁcation performance.
We further remove the reconstruction loss in ablation VI and replace the encoder/decoder structure as MLP.
The results demonstrate that the autoencoder structure beneﬁts the process of task representation learning and selected encoder and decoder.
In ablation VIII and IX, we change the modulate function to Film (Perez et al., 2018) and tanh, respectively.
We can see that ARML is not very sensitive to the modulating activation, and sigmoid function is slightly better in most cases.
In ablation X, we share the gate value within each ﬁlter in
Convolutional layers.
Compared with VIII, the better performance of ARML indicates the beneﬁt of customized gate for each parameter.
[Table] [Table]
5.3. A NALYSIS OF C ONSTRUCTED M ETA - KNOWLEDGE G RAPH
In this section
[Table]
we conduct extensive analysis for the constructed meta-knowledge graph
[Table]
which is regarded as the key component in ARML.
Due to the space limit, we present the results on
Art-Multi datasets here and the analysis of Plain-Multi with similar observations
are discussed in
Appendix G.1.
We further analyze the effect of different number of vertices in meta-knowledge graph in Appendix G.2 and conduct a comparison with HSML about the learned structure in Appendix H.
[Table]
[Table]
To analyze the learned meta-knowledge graph
[Table]
for each subdataset
[Table]
we randomly select one task as exemplary (see Figure 9 of Appendix H.2 for more cases).
For each task, in the left part of Figurewe show the similarity heatmap between prototypes and vertices in meta-knowledge graph
[Table]
where deeper color means higher similarity.
V0-V8 and P1-P5 denotes the different vertices and prototypes, respectively.
The meta-knowledge graph is also illustrated in the right part.
Similar as the graph in 2D regression, we set a threshold to ﬁlter links with low similarity and illustrate the rest of them.
First, we can see that the V1 is mainly activated by bird and aircraft (including all ﬁlters), which may reﬂect the shape similarity between bird and aircraft.
Second, V2, V3, V4 are ﬁrstly activated by texture and they form a loop in the meta-knowledge graph.
Especially, V2 also beneﬁts images with blur and pencil ﬁlters.
Thus, V2 may represent the main texture and facilitate the training process on other subdatasets.
The meta-knowledge graph also shows the importance of V2 since it is connected with almost all other vertices.
Third, when we use blur ﬁlter, in most cases (bird blur, texture blur, fungi blur), V7 is activated.
Thus, V7 may show the similarity of images with blur ﬁlter.
In addition, the connection between V7 and V2 and V3 show that classify blur images may depend on the texture information.
Fourth, V6 (activated by aircraft mostly) connects with V2 and
V3, justifying the importance of texture information to classify the aircrafts.
[Figure]
C ONCLUSION
In this paper, to improve the effectiveness of meta-learning for handling heterogeneous task, we propose a new framework called ARML, which automatically extract relation across tasks and construct a meta-knowledge graph.
When a new task comes in, it can quickly ﬁnd the most relevant relations through the meta-knowledge graph and use this knowledge to facilitate its training process.
The experiments demonstrate the effectiveness of our proposed algorithm.
A A LGORITHM IN M ETA - TESTING P ROCESS
Training data D a new task T
B H YPERPARAMETERS S ETTINGS B. 2D R EGRESSION
In 2D regression problem, we set the inner-loop stepsize (i.e., α ) and outer-loop stepsize (i.e., β ) as 0.001 and 0.001, respectively.
The embedding function E is set as one layer with 40 neurons.
The autoencoder aggregator is constructed by the gated recurrent structures.
We set the meta-batch size as 25 and the inner loop gradient steps as 5.
B. F EW - SHOT I MAGE C LASSIFICATION
In few-shot image classiﬁcation, for both Plain-Multi and Art-Multi datasets, we set the corresponding inner stepsize (i.e., α ) as 0.001 and the outer stepsize (i.e., β ) as 0 . 01 .
For the embedding function E , we employ two convolutional layers with 3 × 3 ﬁlters.
The channel size of these two convolutional layers are 32.
After convolutional layers, we use two fully connected layers with 384 and 64 neurons for each layer.
Similar as the hyperparameter settings in 2D regression, the autoencoder aggregator is constructed by the gated recurrent structures, i.e., AG t , AG tdec AG q , AG qdec are all GRUs.
The meta-batch size is set as 4.
For the inner loop, we use 5 gradient steps.
B. D ETAILED B ASELINE S ETTINGS
For the gradient-based baselines (i.e., MAML, MetaSGD, MT-Net, BMAML.
MUMOMAML,
HSML), we use the same inner loop stepsize and outer loop stepsize rate as our ARML.
As for non-parametric based meta-learning algorithms, both TADAM and Prototypical network, we use the same meta-training and meta-testing process as gradient-based models.
Additionally, TADAM uses the same embedding function E as ARML for fair comparison (i.e., similar expressive ability).
C A DDITIONAL D ISCUSSION OF D ATASETS
In this dataset, we use pencil and blur ﬁlers to change the task distribution.
To investigate the effect of pencil and blur ﬁlters, we provide one example in Figure 4.
We can observe that different ﬁlters result in different data distributions.
All used ﬁlter are provided by OpenCV 1 .
D R ESULTS ON M INI I MAGENET AND TIERED I MAGENET
For miniimagenet and tieredImagenet, since it do not have the characteristic of task heterogeneity, we show the results in Table 4 and Table 5.
In this table, we compare our model with other gradient-based meta-learning models (the top baselines are globally shared models and the bottom baselines are task-speciﬁc models).
Similar as (Finn et al., 2018), we also apply the standard 4-block convolutional
[Figure]
layers for each baseline.
For MT-Net on MiniImagenet, we use the reported results in (Yao et al., 2019), which control the model with the same expressive power.
The results indicate that our proposed ARML outperforms the original MAML and achieves comparable performance with taskspeciﬁc models (e
[Table]
g., MT-Net, PLATIPUS, HSML).
Most task-speciﬁc models achieve the similar performance on the standard benchmark due to the homogeneity between tasks.
[Table] [Table]
E A DDITIONAL R ESULTS OF F EW - SHOT I MAGE C LASSIFICATION
We provide the full results table of Art-Multi Dataset in Table 10.
In this table, we can see our proposed ARML outperforms almost all baselines in every sub-datasets.
F F URTHER I NVESTIGATION OF A BLATION S TUDY
In this section
[Table]
we ﬁrst show the full evaluation results of model ablation study on Art-Multi dataset in Table 6.
Note that, for the tanh activation (ablation model VIII), the performance is similar as applying the sigmoid activation.
On some subdatasets, the results are even better.
We choose the sigmoid activation for ARML because it achieves overall better performance than the tanh activation on more subdatasets.
Then, for Plain-Multi dataset, we show the results in Table 7.
The conclusion of ablation study in Plain-Multi dataset is similar as the conclusion drawn from the results on Art-Multi dataset.
The improvement on these two datasets veriﬁes the necessity of the joint framework in
ARML
[Table] [Table]
II
[Table]
[Table]
G A DDITIONAL A NALYSIS OF M ETA - KNOWLEDGE G RAPH G. Q UALITATIVE A NALYSIS ON P LAIN -M ULTI D ATASET
Then, we analyze the meta-knowledge graph on Plain-Multi dataset by visualizing the learned metaknowledge graph on Plain-Multi dataset (as shown in Figure 5).
In this ﬁgure, we can see that different subdatasets activate different vertices.
Speciﬁcally, V2, which is mainly activated by texture, plays a signiﬁcantly important role in aircraft and fungi.
Thus, V2 connects with V3 and V1 in the meta-knowledge graph, which are mainly activated by fungi and aircraft, respectively.
In addition,
V0 is also activated by aircraft because of the similar contour between aircraft and bird.
Furthermore, in meta-knowledge graph, V0 connects with V3, which shows the similarity of environment between bird images and fungi images.
[Figure]
G. P ERFORMANCE V . S . V ERTICE N UMBERS
We ﬁrst investigate the impact of vertice numbers in meta-knowledge graph.
The results of Art-Multi (5-way, 5-shot) are shown in Table 8 (full table in Table 9).
From the results, we can notice that the performance saturates as the number of vertices around 8.
One potential reason is that 8 vertices are enough to capture the potential relations.
If we have a larger datasets with more complex relations, more vertices may be needed.
In addition, if the meta-knowledge graph do not have enough vertices, the worse performance suggests that the graph may not capture enough relations across tasks.
[Table]
[Table]
H A DDITIONAL Q UALITATIVE A NALYSIS H. D ISCUSSION BETWEEN ARML AND HSML
In this part
[Table]
we provide the case study to visualize the task structure of HSML and ARML.
HSML is one of representative task-speciﬁc meta-learning methods, which adapts transferable knowledge by introducing a task-speciﬁc representation.
It proposes a tree structure to learn the relations between tasks.
However, the structure requires massive labor efforts to explore the optimal structure.
By contrast, ARML automatically learn the relation across tasks by introducing the knowledge graph.
In addition, ARML fully exploring there types of relations simultaneously, i.e., the prototype-prototype, prototype-knowledge and knowledge-knowledge relations.
To compare these two models, we show the case studies of HSML and ARML in Figure 6 and
Figure 7.
For tasks sampled from bird, bird blur, aircraft and aircraft blur are selected for this comparison.
Following case study settings in the original paper (Yao et al., 2019), for each task, we show the soft-assignment probability to each cluster and the learned hierarchical structure.
For
ARML, like 3, we show the learned meta-knowledge and the similarity heatmap between prototypes and meta-knowledge vertices.
In this ﬁgures we can observe that ARML constructs relations in a more ﬂexible way by introducing the graph structure.
More speciﬁcally, while HSML activate relevant node in a ﬁxed two-layer hierarchical way, ARML provides more possibilities to leverage previous learned tasks by leveraging prototypes and the learned meta-knowledge graph.
H. A DDITIONAL C ASES OF M ETA - KNOWLEDGE G RAPH A NALYSIS
We provide additional case study in this section.
In Figure 8, we show the cases of 2D regression and the additional cases of Art-Multi are illustrated in Figure 9.
We can see the additional cases also support our observations and interpretations.
[Figure]
N
N N N
N
[Figure]
N
N N
N N
[Figure]
N3 N
[Figure] [Figure]
[Figure] [Figure]
T a b l e 1 0 :
F u ll r e s u lt s on
A r tM u lti d a t a s e t .
I n t h i s t a b l e ,
B , T ,
A , F r e p r e s e n t b i r d , t e x t u r e , a i r c r a f t , f ung i , r e s p ec ti v e l y .
P l a i n m ea n s o r i g i n a l i m a g e .
S e tti ng s
A l go r it h m s
B P l a i n
B B l u r
B P e n c il
T P l a i n
T
[Table]
l u r
T P e n c il
A P
[Table]
a i n
A
[Table]
l u r
A P
[Table]
n c il
F P l a i n
F
[Table]
r
F P
[Table]
n c il
[Table]
T
ADA
M 54 .% 52 .% 48 .% 32 .% 29 .% 30 .% 50 .% 47 .%
M A
M L 55 .% 52 .% 48 .% 30 .% 28 .% 28 .% 45 .% 42 .% 34 .% 39 .% 38 .% 35 .%
M e t a
S GD 55 .% 53 .% 48 .% 29 .% 28 .% 28 .% 51 .% 47 .% 35 .% 41 .% 40 .% 36 .%
B M
A M
L 56 .% 52 .% 47 .% 31 .% 29 .% 29 .% 46 .% 42 .% 36 .% 40 .% 39 .% 35 .%
M T - N e t 56 .% 54 .% 50 .% 32 .% 29 .% 29 .% 43 .%
M U
M O
M A
M L 57 .%
H S M
L 58 .% 53 .% 51 .% 32 .% 30 .% 30 .% 49 .% 45 .% 40 .% 42 .% 41 .% 37 .%
A R
M L 59 .% 54 .% 52 .% 32 .% 30 .% 31 .% 51 .% 47 .% 41 .% 44 .% 42 .%
38 .% 5 - w a y 5 - s ho t
V E R
S A 66 .% 65 .% 60 .% 38 .% 35 .% 33 .% 64 .% 62 .% 53 .% 51 .%
P r o t o
N e t 70 .%
[Table]
T a p N e t 68 .% 68 .% 62 .% 43 .% 37 .% 38 .% 65 .% 64 .% 54 .%
[Table]
T
ADA
M 70 .% 69 .% 65 .% 44 .% 41 .% 40 .%
M A
M L 71 .% 68 .% 63 .% 42 .
[Table]