Sharing Knowledge for Meta-learning with Feature Descriptions
Abstract
Language is an important tool for humans to share knowledge.
We propose a meta-learning method that shares knowledge across supervised learning tasks using feature descriptions written in natural language, which have not been used in the existing meta-learning methods.
The proposed method improves the predictive performance on unseen tasks with a limited number of labeled data by metalearning from various tasks.
With the feature descriptions, we can ﬁnd relationships across tasks even when their feature spaces are different.
The feature descriptions are encoded using a language model pretrained with a large corpus, which enables us to incorporate human knowledge stored in the corpus into meta-learning.
In our experiments, we demonstrate that the proposed method achieves better predictive performance than the existing meta-learning methods using a wide variety of real-world datasets provided by the statistical ofﬁce of the EU and Japan.
Introduction
Humans can learn new tasks using descriptions written in natural language, which support us in improving the performance on the new tasks especially when we have little experience since the descriptions help understand how experience on familiar tasks can be used for the new tasks.
For example, Table 1 shows a dataset from Eurostat, which is the statistical ofﬁce of the European
Union.
Some people can guess the production value at basic price in Denmark on soft wheat and spelt in 2017 given Table 1 using knowledge about countries, cereals, and agriculture.
On the other hand, supervised learning methods using data shown in Table 1 cannot guess the value since some features, i e., ‘Denmark’ and ‘soft wheat and spelt’, do not appear in the training data.
Meta-learning methods have been proposed for sharing knowledge across supervised learning tasks to improve the performance on unseen tasks with a small amount of training data.
However, the existing metalearning methods do not use feature descriptions written in natural language although many real-world datasets contain feature descriptions as shown in Table 1.
In this paper, we propose a meta-learning method that learns from datasets with feature descriptions to improve the generalization performance on unseen tasks with small labeled data.
With feature descriptions, we can share knowledge between different tasks with similar descriptions even when their feature spaces are different.
We transform the feature descriptions to sentence embeddings by a sentence encoder, e.g., Bidirectional Encoder Representations from Transformers (BERT) [3], pretrained with a large-scale corpus, by which we can exploit a huge amount of human knowledge accumulated in the corpus for meta-learning.
The sentence embeddings reﬂect the meaning of the descriptions but they are not necessarily suitable for supervised learning tasks.
We translate them by a feature encoder network to improve the generalization performance.
We embed instances in a task-speciﬁc space by an instance encoder network using their feature values and descriptions translated into sentence embeddings.
By adapting a Gaussian process (GP) [17] to 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
[Table]
small labeled data in the space, we obtain a task-speciﬁc prediction model.
The neural networks for feature encoding, instance encoding, and prediction are shared among all tasks, which enables us to learn and share knowledge that is useful in different tasks.
The parameters of the neural networks are trained by minimizing the expected test error of the task-adapted prediction model using meta-training datasets based on a bilevel optimization framework.
The meta-training datasets consist of labeled data from a wide variety of tasks with feature descriptions.
The main contributions of this paper are as follows: 1) We newly present a meta-learning problem where feature descriptions are given for each task.
We propose a simple yet effective neural networkbased meta-learning method that improves the generalization performance with small labeled data using feature descriptions by learning from various datasets.
We empirically demonstrate that the proposed method outperforms the existing methods using a wide variety of real-world datasets provided by the statistical ofﬁce of the EU and Japan.
Related work
Many meta-learning methods have been proposed [19, 1, 4, 26, 20].
However, they usually assume that only pairs of feature and label values are given for each task, and do not use task information.
Several existing meta-learning methods use task information, such as class descriptions and task descriptions.
Class descriptions, e.g., image category name in natural language, have been used for few-shot [30] and zero-shot learning [5, 21].
Task descriptions, e.g., the brightness of images in the task, have been used for active meta-learning [9], few-shot learning [13], zero-shot domain adaptation [31], and natural language processing systems [28].
However, they cannot use feature descriptions.
Feature descriptions are helpful for understanding relationships between tasks especially when feature spaces are different across tasks.
Most existing meta-learning methods assume feature spaces are the same for all tasks although tasks with different feature spaces can contain useful knowledge to be shared.
Heterogeneous meta-learning can handle tasks with different feature spaces [7].
However, it cannot use the information about features.
Text information has been used in some applications with supervised learning settings, such as cross-lingual text classiﬁcation [14], recommendation [27], and image classiﬁcation [6].
However, they are inapplicable to meta-learning.
We use text information for sharing knowledge to improve performance on unseen tasks.
Proposed method 3.1 Problem formulation
In the meta-training phase, we are given D labeled meta-training datasets with feature descriptions,
D = {{ ( x dn , y dn ) } N d n =1 , W d } Dd =1 , where x dn = ( x dn 1 , . . .
, x dnJ d ) is the feature vector of the n th instance, y dn ∈ R is its label, W d = {{ w djc } C dj c
=1 } J d j =1 , is the set of feature descriptions, N d is the number of instances, x dnj ∈ { 1 , . .
, C dj } is the categorical value of the j th feature, C dj is the number of categories, J d is the number of features, and w djc is the feature description of the c th category of the j th feature.
For example, w d 11 =
‘Production value at basic price’, w d 12 = ‘Subsidies on products’, and w d 21 = ‘Wheat and spelt’ are feature descriptions of the dataset in Table 1.
The feature description sets W d , the number of features J d , and the number of categories C dj are different across tasks.
In the meta-test phase, we are given a support set, which consists of a small number of labeled examples D S =
{ ( x S n , y S n ) } N S n =1 with description W in an unseen task, which are different from the meta-training datasets.
Our aim is to improve the test label prediction performance in test tasks.
Although we explain the proposed method for categorical features and numerical labels for simplicity, it can also use numerical features and categorical labels as described in Section 3.4.
3.2 Model
With our model, instances with feature descriptions are embedded for each task as described in
Section 3.2.1, and their labels are predicted in the task-speciﬁc embedding space as described in
Section 3.2.2.
3.2.1 Instance embedding with feature descriptions
We embed each instance x
= ( x 1 , . .
, x J ) using feature descriptions W = {{ w jc } C j c =1 } Jj =1 as shown in Figure 1, where x j ∈ { 1 , . . .
, C j } is the j th feature value, J is the number of features, C j is the number of categories of the j th feature, and w jc is the feature description of the c th categorical value at the j th feature.
Here, we omit the task and instance indices.
First, each feature description w jc is transformed into sentence embedding v jc ∈ R M v by a pretrained sentence encoder, e.g., BERT,
[Equation]
where f SE is the sentence encoder.
With BERT pretrained using a huge number of text documents, we can incorporate knowledge in the documents.
Second, feature embedding u j ∈ R M u of the j th feature value is obtained using the sentence embedding and feature value,
[Equation]
where f FE is a trainable feature encoder, such as a feed-forward neural network and transformer [25].
By the feature encoder, we translate sentence embeddings, which are calculated based on the pretraining documents written for humans, to feature embeddings that are suitable for supervised learning tasks.
Third, instance embedding z ∈ R M z is obtained by averaging the feature embeddings,
[Equation]
where f IE is a trainable instance encoder based on neural networks, and f is an encoder to output instance embedding z given feature vector x and feature description W .
Eq.
(3) is a permutation invariant neural network [32] since the summation is invariant even when the elements are permuted, and our model can handle instances with any number of features.
Neural networks, f FE and f IE , are shared across all tasks, by which we can store knowledge meta-learned in various tasks in the neural networks, and use it for unseen tasks.
Although size J of input feature vector x can be different across tasks, size M z of instance embedding vector z is the same for all tasks.
3.2.2 Task-speciﬁc prediction
The proposed model predicts label y of query instance x using support set D S
= { ( x S n , y S n ) } N S n =1 and feature description W of the task, where x S n is the feature vector of the n th support instance, y S n is its label and N S is the number of support instances.
Here, we omit the task index.
Figure 2 illustrates our model.
First, instance embeddings are obtained by f in Eq.
(3) for the query and support instances, z = f
( x , W ) , and z S n = f
( x S n , W ) , n = 1 , . .
. , N S .
Second, the query label is predicted by Gaussian process regression with kernel function k and with neural network-based mean function g using the instance embeddings.
The predicted label adapted to the support set is given in a closed form,
[Equation]
where k =
[ k ( z , z S1 ) , . .
k ( z , z S N S )]
∈ R N S is the kernel vector between the query and support instances, K ∈ R N S × N S is the kernel matrix between the support instances, K nn (cid:48) =
k ( z S n , z S n (cid:48) )
∈ R , y S =
[ y S1 , . . .
, y S N S ] ∈ R N S is the label vector of the support instances, and g S =
[ g ( z S1 ) , . . .
, g ( z S N S )]
∈ R N S is the mean vector evaluated on the support instances.
Θ is parameters in our model, which consists of neural network parameters in f FE , f IE , and g , and kernel parameters in k .
Neural network g and kernel parameters in k are shared across all tasks.
GP regression is a nonparametric nonlinear regression method successfully used in a wide variety of applications, and it can output prediction adapted to the support set ﬂexibly.
Since the task-adapted GP prediction is given in a closed form as shown in Eq.
(4), we can efﬁciently perform backpropagation through the adaptation, and it has been used for meta-learning [16, 8].
The prediction in Eq.
(4) can be seen as ﬁtting GP with a zero mean function to residual y S − g S of prediction by neural network g .
Neural network g makes predictions based on the feature descriptions without using the support set.
Even when the feature descriptions are not helpful for the prediction, the proposed model makes reasonable predictions by adapting them to the support set by GP.
When labels can be predicted well by neural network g with the feature descriptions, since the residual becomes zero, the adaptation is skipped, by which we can avoid the risk of overﬁtting to the small support set. ∈ D
3.3 Meta-training
We train model parameters Θ by minimizing the expected test mean squared error using an episodic training framework [18],
[Equation]
where E D S , D Q , W represents the expectation over support set D S , query set D Q
= { ( x Q n , y Q n ) } N Q n =1 , and feature descriptions W .
By using the query set, which is different from the support set used for adapting GP, we can evaluate the test error.
Minimizing our meta-learning loss in Eq.
(5) is a bilevel optimization problem.
In the inner optimization, a task-speciﬁc prediction model is adapted by minimizing the loss on the support set based on GP as described in Section 3.2.2.
In the outer optimization, neural network parameters shared among all tasks are estimated by minimizing the error on the query set when the task-adapted prediction model obtained in the inner optimization is used.
Algorithm 1 shows the meta-training procedure of the proposed model.
The expectation in Eq.
(5) is approximated by the Monte Carlo method, where a task, support and query instances are randomly sampled from the given meta-training datasets.
The computational complexity of calculating the test error for a set of support and query instances in Line 6 is O ( N Q + N S3 ) since we need the inverse of the kernel matrix whose size is N S × N S .
In few-shot settings, which is our focus, the number of support instances N S is small.
3.4 Variants 3.4.1 Feature type description
In addition to feature descriptions (e.g., “Production value at basic price” and “Wheat and spelt” in Table 1), feature type descriptions (e.g., “Agricultural indicator” and “List of products” in Table 1), are available in some datasets.
Let w j be the feature type description of the j th feature.
In this case, the feature embedding is obtained by taking the sentence embeddings of the feature description and feature type description as input of the feature encoder,
[Equation]
instead of Eq.
This enables us to incorporate information of feature type descriptions for meta-learning.
Descriptions might be missing for a part of features in some datasets.
Let x be features with descriptions, and x be those without descriptions in x .
Instance embedding z is obtained using features with descriptions by encoder f in Eq.
(3), z = f ( x ) , by which we can incorporate observed descriptions in the embedding.
The task-speciﬁc GP is adapted based on the concatenation of the instance embedding and onehot vector of features without descriptions.
In particular, the kernel function is calculated by, k
([ z , onehot( x )] , [ z Sn , onehot( x Sn )]) , where onehot represents the onehot encoder, and [ · , · ] represents the concatenation of vectors.
The prediction is performed by Eq.
(4) using the kernel function with the concatenation, by which we can use the information on both of the features with and without descriptions for prediction, where the mean function takes instance embedding z as input.
When features are numerical values, we obtain the feature embedding by multiplying numerical value x j and the sentence embedding of its feature type description w j as follows, [Equation]
In classiﬁcation tasks, we can use the GP-based adaptation in Eq.
(4) by representing labels by binary vectors as shown in [16], or we can use the Gaussian mixture model-based adaptation as in prototypical networks [20].
For the loss function in Eq.
(5), we can use the cross-entropy loss instead of the mean squared error.
Experiments 4.1 Data
For evaluating the proposed method, we used the following two data: e-Stat and Eurostat.
These data contained statistical tables in various ﬁelds, including economy, health, education, sport, and crime.
The e-Stat data were obtained from the ofﬁcial statistics of Japan using API 1 .
The Eurostat data were obtained from the statistical ofﬁce of the European Union using API 2 .
A statistical table contained numerical values of statistics, which were used for labels, and categorical descriptions in natural language, which were used for features, as shown in Table 1.
We used a statistical table as a dataset in a task.
When a statistical table comprised multiple units of measurement, it was decomposed into multiple datasets based on the unit of measurement.
We used datasets where the number of instances was between 50 and 1,000.
The labels were normalized with zero mean and one standard deviation for each dataset.
The statistics of the two data are shown in Table 2.
The descriptions were in Japanese in e-Stat, and in English in Eurostat.
For e-Stat, we used the Japanese pretrained BERT 3 .
For Eurostat, we used the English pretrained BERT [3, 29] 4 .
The BERT vector size was 768.
For each of the e-Stat and Eurostat data, we sampled 700 datasets for meta-training, 100 for meta-validation, and 200 for meta-test.
For each task, the number of support instances was 5, 10, or 20, and the number of query instances was 20.
We evaluated the test mean squared error averaged over ten experiments with different splits of datasets.
Note that we did not use commonly-use meta-learning datasets, such as Omniglot [11], Mini-imagenet [26], and datasets in Torchmeta [2] and Meta-Dataset [23], since they do not contain feature descriptions.
4.2 Comparing methods
We compared the proposed method (Ours) with the following methods: ridge regression (Ridge),
Gaussian process regression (GP), heterogeneous meta-learning [7] (HML), neural networks with
BERT sentence embeddings (NN), meta deep kernel learning [16] with shared categorical onehot vector features (MDK+C), meta deep kernel learning with word frequency vectors (MDK+W), meta deep kernel learning with BERT sentence embeddings (MDK+B), the proposed method without mean functions (Ours-M) and the proposed method with feature type descriptions (Ours+T).
Tablesummarizes the comparing methods.
Ridge and GP methods were trained using support instances for each task.
The hyperparameters, i.e., the regularization weight in Ridge and the kernel parameters in GP, were optimized using the meta-training datasets.
We used RBF kernels for GP.
The methods except for Ridge and GP had [Table]
[Table]
neural networks shared among all tasks, and they were meta-trained using the meta-training datasets.
HML was a meta-learning method for tasks with different feature spaces, which did not use feature information.
Note that since other existing meta-learning methods cannot handle tasks with different feature spaces, we did not compare with them.
In Ridge, GP, and HML, categorical feature values were transformed into onehot vectors for each task.
NN was a neural network-based meta-learning method, where feature descriptions were used for its input.
With NN, feature descriptions were embedded by BERT, translated by a feature encoder network, and taken to deep sets [32] as input for predicting labels.
NN corresponds to the proposed method without GP adaptation to the support set for each task.
In MDK+C, MDK+W and MDK+B, a task-speciﬁc GP was adapted using the support set, where RBF kernels were used with neural networks for input transformation.
MDK+C used onehot vectors of categorical feature values that were shared in all tasks, MDK+C corresponds to the proposed method that uses feature descriptions for only identifying feature correspondences across different tasks.
MDK+W used word frequency vectors for representing feature descriptions, where the vectors were embedded in a 768-dimensional space by principal component analysis.
MDK+W corresponds to the proposed method with word frequency vectors without pretrained BERT.
MDK+B used BERT sentence embeddings, but they were not translated by a feature encoder for each description in meta-learning.
MDK+B corresponds to the proposed method without feature encoders.
Since the feature type descriptions were unavailable by Eurostat API, we did not evaluate Ours+T on Eurostat data.
4.3 Settings
In the proposed model, we used a three-layered feed-forward neural network with 128 hidden and output units for the sentence, feature, and instance encoders, f SE , f FE , and f IE , and a three-layered feed-forward neural network with 128 hidden units and a single output unit for mean function g .
For the activation function, we used rectiﬁed linear unit ReLU( x )
= max(0 , x ) .
For GP, we used RBF [Table]
kernels, k ( z , z (cid:48) )
=
α exp (cid:0) − γ 2 (cid:107)
z − z (cid:48) (cid:107) 2 (cid:1) + βδ ( z , z (cid:48) ) , where α , β , and γ were kernel parameters to be meta-trained.
We optimized our models using Adam [10] with learning rate 10 − 3 , batch dataset size 32 , and dropout rate 0 .
The meta-validation datasets were used for early stopping, for which the maximum number of meta-training epochs was 5,000.
In the comparing methods, we used the same neural network architectures and training procedures as the proposed method.
We implemented the proposed method with PyTorch [15].
4.4 Results
Table 4 shows the test mean squared error with different numbers of support instances per task.
The proposed method achieved the best performance in all cases.
The performance of the Ridge and GP methods was low since they were trained using only support instances in the meta-test datasets.
The HML method did not perform well since it needs to meta-learn relationships among tasks using support instances without feature descriptions.
This result indicates the effectiveness of the feature descriptions for meta-learning.
The NN method predicts labels using only feature descriptions, where the prediction is not adapted to the labeled support instances.
Since the feature descriptions in natural language are often not quantitative, it is difﬁcult for the NN method to predict labels accurately.
On the other hand, the proposed method achieved better performance than the NN method by minimizing the prediction error on the support instances.
The MDK+C method uses feature descriptions for corresponding features among tasks, but does not use sentence embeddings. The MDK+C method performed worse than the proposed method, which implies sentence embeddings are important for meta-learning since they can quantify the relationships among features even when their descriptions are not the same.
The MDK+W method underperformed the proposed method, which implies that incorporating knowledge in large-scale corpus stored by pretrained BERT is effective for meta-learning.
The error by the MDK+B method was higher than the proposed method, which indicates the effectiveness of the translation using the feature encoder for meta-learning.
The proposed method without mean functions (Ours-M) performed worse than the NN method on e-Stat data with ﬁve and ten support instances.
It is because feature descriptions are relatively important for task-speciﬁc prediction when the number of support instances is small.
With the proposed method
[Figure] [Figure] [Table] (Ours), the residual error of the neural network-based mean function is adapted by Gaussian processes using support instances for each task, which enables us to achieve better performance for all the cases.
With the proposed method [Figure] the use of feature type descriptions (Ours+T) improved the performance when the number of support instances was ﬁve.
The feature type descriptions can be helpful when support instances are small.
Figure 3 shows the test mean squared error with different numbers of meta-training datasets.
As the number of the meta-training datasets increased, the performance of the proposed method increased.
The proposed method achieved the best performance in all cases except for 100 meta-training datasets in Eurostat data.
It is effective for the proposed method to use more meta-training datasets.
Figureshows the test mean squared error with different rates of observed feature descriptions by the proposed method.
The error decreased as the observed rate increased especially in e-Stat data.
Table 5 shows the test mean squared error by the proposed method using word2vec [12] for sentence embeddings instead of BERT. We used Japanese word2vec 5 for e-Stat, and English word2vec 6 for Eurostat, and the sentence embedding was obtained by the average of the word embeddings in the description. The error with word2vec was higher than that with BERT (Ours in Table 4).
This result indicates that the meta-learning performance depends on the performance of the sentence encoder.
Table 6 shows the test mean squared error on tasks where more than half of the feature descriptions do not appear in the meta-training tasks.
The proposed method also achieved lower errors in tasks that contain feature descriptions that do not appear in the meta-training data than the other methods.
[Table]
[Figure] [Table] Figure 5 shows the visualization of instances. With the onehot vectors of categorical feature values (a), it is difﬁcult to represent the instances well.
With the BERT sentence embeddings (b), instances with similar labels formed clusters, but the clusters were separated.
With the proposed method (c), clusters with similar labels were located more closely than those with the BERT sentence embeddings by the feature and instance encoders, which helps label prediction in the instance embedding space.
With translation by the feature encoder, similar BERT sentence embeddings can be encoded to different feature embeddings for improving the meta-learning performance.
For example, in Eurostat data, BERT embeddings of ‘Accommodation (10 or more employees and self-employed persons)’ and ‘Information and communication (10 or more employees and self-employed persons)’ were similar (cosine=0.943) since the words in the parenthesis were the same, while their translated feature embeddings by the proposed method were different (cosine=0.003).
BERT embeddings of ‘Outwards’ and ‘Inwards’ were similar (cosin=0.951), while their feature embeddings were different (cosine=0.025), since they were related but antonyms.
BERT embeddings of ‘Warmond - Hoofddorp’ and ‘Amersfoort - Hilversum’ were different (cosine=0.280), while their feature embeddings were similar (cosine=0.901) since they are towns in the Netherland and related.
Table 7 shows the average computation time in seconds for meta-training on computers with 2.60GHz CPUs.
The Ridge and GP methods were efﬁcient since they did not have neural networks to be meta-trained. The computational time of the proposed method was shorter than that of the HML method.
Conclusion
We proposed a meta-learning method that effectively uses feature descriptions, and conﬁrmed that the proposed method achieves better performance than the existing methods.
Although our results are encouraging as an effective approach for sharing knowledge between various tasks in meta-learning by incorporating human knowledge, there are some limitations to overcome.
First, we need to empirically evaluate the variants of the proposed method, e.g., for classiﬁcation tasks and numerical feature values.
Second, we would like to use other types of neural networks for components in our model, such as attention networks [25], since we used feed-forward neural networks in our experiments.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See the supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results
N
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] The code is proprietary.
The datasets and experimental procedures are described in Section 4.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
The proposed method requires feature descriptions for meta-training datasets.
When there are not enough meta-training datasets, the performance can be comparable with methods that are trained using support instances for each task. When meta-training and meta-test tasks are signiﬁcantly different, there is a risk of degrading the performance. The evaluation of the proposed method with classiﬁcation tasks and numerical feature values are not performed since the data used in our experiments were regression tasks with categorical feature values. B Potential negative societal impacts
Since the proposed method uses various datasets for meta-training, there is a potential risk that users might include biased datasets without careful thought, which might result in biased predictions. We encourage research to automatically detect biased datasets. Although the proposed method improves performance, the prediction might not always be correct. We encourage research to estimate the uncertainty of the prediction by meta-learning from datasets with feature descriptions.