F EW -S HOT R EGRESSION VIA L EARNED B ASIS F UNCTIONS
A BSTRACT
The recent rise in popularity of few-shot learning algorithms has enabled models to quickly adapt to new tasks based on only a few training samples.
Previous few-shot learning works have mainly focused on classiﬁcation and reinforcement learning.
In this paper, we propose a few-shot meta learning system that focuses exclusively on regression tasks.
Our model is based on the idea that the degree of freedom of the unknown function can be signiﬁcantly reduced if it is represented as a linear combination of a set of appropriate basis functions .
This enables a few labelled samples to approximate the function.
We design a Feature Extractor network to encode basis functions for a task distribution, and a Weights Generator to generate the weight vector for a novel task.
We show that our model outperforms the current state of the art meta-learning methods in various regression tasks.
I NTRODUCTION
Regression deals with the problem of learning a model relating a set of inputs to a set of outputs.
The learned model can be thought as function y = F ( x ) that will give a prediction
y ∈ R
d y given input x ∈ R d x where d y and d x are dimensions of the output and input respectively.
Typically, a regression model is trained on a large number of data points to be able to provide accurate predictions of new inputs.
Recently, there have been a surge in popularity on few-shot learning methods (Vinyals et al., 2016; Koch et al., 2015; Gidaris & Komodakis, 2018).
Few-shot learning methods require only a few examples from each task to be able to quickly adapt and perform well on a new task.
The fewshot learning model in essence is learning to learn i.e. the model learns to quickly adapt itself to new tasks rather than just learning to give the correct prediction for a particular input sample.
In this work, we propose a few shot learning model that targets few-shot regression tasks.
We evaluate our model on the sinusoidal regression tasks and compare our model’s performance to several meta-learning algorithms.
We further introduce two more regression tasks, namely the 1D heat equation task modeled by partial differential equations and the 2D Gaussian distribution task.
R ELATED W ORK
Regression problems has long been a topic of study in the machine learning and signal processing community (Myers & Myers, 1990; Specht, 1991).
Though similar to classiﬁcation, regression estimates one or multiple scalar values and is usually thought of as a single task problem.
A single model is trained to only perform regression on a only one task.
Our model instead reformulates the regression problem as a few-shot learning problem, allowing for our model to be able to perform regressions of tasks sampled from the same task distribution.
The problem of meta-learning has similarly long been a topic of interest in the general machine learning community (Thrun & Pratt, 1998; Schmidhuber, 1987; Naik & Mammone, 1992).
Meta learning has been applied to few-shot learning problem, which is concerned with models that can learn from prior experiences to adapt to new tasks.
Lake et al.
ﬁrst proposed the one-shot classiﬁcation problem in 2011 and introduced the Omniglot data set as a few-shot classiﬁcation data set similar to how the MNIST data set (LeCun, 1998) is for traditional classiﬁcation.
Since then, there has been a surge of few-shot learning methods (Vinyals et al., 2016; Finn et al., 2017;
Gidaris & Komodakis, 2018; Rusu et al., 2018), but most of them focus on few shot classiﬁcation and reinforcement learning domains.
F EW - SHOT REGRESSION VIA LEARNED BASIS FUNCTIONS
We ﬁrst discuss our idea.
We aim at developing a model that can rapidly adapt to regress a novel function based on only a few samples from this function.
Speciﬁcally, we would like to model the unknown function y
= F ( x ) given only D train = { ( x k , y k )
| k
= 1 ...K } .
With small K , e.g.
K = 10 , this is an intractable task, as F ( x ) can take any form.
We follow the common setup: we assume that each function we would like to regress is a task T i drawn from an unknown distribution p ( T ) .
To simplify discussion, we assume scalar input and scalar output.
Our idea is to learn sparse or compressible representation of the unknown function F ( x ) , so that a few samples { ( x k , y k ) | k
= 1 ...K } can provide adequate information to estimate F ( x ) .
Speciﬁcally, we model the unknown function F ( x ) as a linear combination of a set of basis functions { φ i ( x ) } :
[Equation]
Many basis functions have been developed to expand F ( x ) .
For example, the Maclaurin series expansion (Taylor series expansion at x = 0 ) uses { φ i ( x ) } = { 1 , x, x 2 , x 3 , ... } :
[Equation]
If F ( x ) is a polynomial, (2) can be a compressible representation, i.e. only a few non-zero/signiﬁcant w i .
However, if F ( x ) is a sinusoid, it would require many terms to represent F ( x ) adequately, e.g.:
[Equation]
In (3), M is large and M (cid:29) K .
Given only K samples { ( x k , y k )
| k
= 1 ...K } , it is not adequate to determine { w i } and model the unknown function.
On the other hand, if we use the Fourier basis instead, i.e., { φ i ( x ) } = { 1 , sin( x ) , sin(2 x ) , ..., cos( x ) , cos(2 x ) , ... } , clearly, we can obtain a sparse representation: we can represent a sinusoid with only a few terms.
Under Fourier basis, there are only a few non-zero linear weights w i , and K samples are sufﬁcient to estimate w i and estimate the function.
Essentially, with an appropriate { φ i ( x ) } , the degree of freedom of F ( x ) can be signiﬁcantly reduced when it is modeled using (1), so that K samples can well estimate F ( x ) .
Our approach is to use the set of training tasks drawn from p ( T ) to learn { φ i ( x ) } that result in sparse/compressible representation for any task drawn from p ( T ) .
The set of { φ i ( x ) } is encoded in the Feature Extractor that takes in x and outputs Φ( x ) =
[ φ 1 ( x ) , φ 2 ( x ) , ..., φ M ( x )] T .
In our framework, Φ( x ) is the same for any task drawn from p ( T ) , as it encodes the set of { φ i ( x ) } that can sparsely represent any task from p ( T ) .
We further learn a Weights Generator to map the K training samples of a novel task to a constant vector w =
[ w 1 , w 2 , ..., w M ] T .
The unknown function is modeled as w T Φ( x ) .
3. M ODEL A RCHITECTURE
We hereby introduce our few-shot regression model in detail.
Given a regression task T with D train =
{ ( x k , y k )
| k
= 1 ...K } , the model is tasked to predict the entire regression function across a value range.
The training samples, x ∈ R d x ﬁrst passed though the Feature Extractor which is represented as a function, F ( x | θ F ) with trainable parameters θ F .
The Feature Extractor outputs a high dimensional feature representation, f ∈ R d f , where d f is the dimension of the feature representation, for each training point of the task T .
Note that d f is the number of basis functions encoded in the Feature
Extractor.
The feature representation x f , together with the labels y ∈ R d y and task labels t ∈ R d t generated from the Task Label Generator are then passed through the Weights Generator .
The Weights
Generator, represented as a function G ( f , y , t | θ G ) , with trainable parameters θ G , outputs a weights vector, w k for each training sample of a regression task.
The ﬁnal weights vector, w (cid:48) for task T is then obtained by taking a mean of the k weight vectors.
The Weights Generator itself consists a series self-attention blocks with scaled dot product attention introduced by Vaswani et al.
Each of the self-attention modules allows the weights generator to ”look” at the embedding of the
Weights Generator’s input to let generator ”choose” the parts of the embedding which is most useful in generating the optimal weights for each of the training sample.
The model is then able to make predictions on set of points D pred = { x n | n = 1 ...N } for task T , by taking a dot product between task-speciﬁc weights vector, w and feature representation of the prediction set.
[Equation]
Note that for all of our regression experiments, y has a dimension of 1.
However, our model is able to predict regression tasks with higher dimensional y by outputting d y weight vectors from the
Weights Generator, the predictions can be obtained by doing a dot product at each dimension of y between the individual weight vectors and f .
An overview of our model can be found in Figure 1.
3. T HE T ASK L ABEL G ENERATOR
Outside of the label information y at the sample level, few-shot regression tasks, unlike other fewshot learning tasks also possess additional label information at the task level.
These task level labels are parameters that describe a regression function that we can leverage to improve the performance of a few-shot regression model.
For example, a sinusoidal function has parameters labels such as the amplitude, phase and frequency.
We dub these task-level labels as task labels, t and we use it as an additional input to the Weights Generator.
Though we assume that the model to have access to task labels during the training phase, it is unrealistic to assume that such information will be available or reliable as well during testing.
Therefore, we introduce the Task Label Generator as an additional component to our model.
We represent it as a function T ( x f , y | θ T ) with trainable parameters θ T .
It takes in feature representation x f and labels y of the regression task T and attempts to output the correct task labels, t g for T .
Similar to the
[Table] [Table] Weights Generator, we also employ the use of self-attention blocks within the Task Label Generator to enable the Task Label Generator to ”look” at parts of the input that are most useful to generating the correct task labels.
R ESULTS AND E VALUATION
We evaluate our model on three few-shot regression tasks.
The ﬁrst task is the sinusoidal regression task which has been used in many recent few-shot learning papers (Finn et al., 2017; Rusu et al., 2018; Li et al., 2017).
We also introduce two more regression tasks, namely the 1D heat equation task (Cannon, 1984) and the higher dimensional 2D Gaussian distribution task.
For the sinusoidal task we compare our model to Meta-SGD proposed by Li et al.
We also compare our results on sinusoidal task to that of Yoon et al.
(2018) where they add a noise component and only train with limited number of tasks for added complexity.
We follow their set up of training of model with only 1000 tasks but we used a pre-trained Task Label Generator instead.
For 1D Heat Equation and 2D Gaussian, we compare our models performance against MAML (Finn et al.
and include results of our model with and without using the Task Label Generator.
We calculate mean squared error across 1000 test tasks for all regression tasks and present our results in Table 1.
Our model manages to outperform both Meta-SGD and BMAML on both variants of the sinusoidal task.
Our model also manage to achieve a superior performance on 1D Heat Equation and 2D Gaussian.
Our results show that even without the use of task labels, our model setup is already able to perform well in the two regression tasks.
4. A BLATION S TUDY
Furthermore, we conduct a ablation study to study the effects of adding the Task Label Generator.
We compare three variants of our model an evaluate them on the advanced sinusoidal task .For
the ﬁrst variant when we use the true task labels in both training and testing.
In the second variant, we do not use task labels at all.
Finally we compare the two variants to the base model.
We show the results of this study in Table 2.
C ONCLUSION
We propose a few-shot meta learning system that focuses exclusively on regression tasks.
Our model is based on the idea of linear representation of basis functions.
We design a Feature extractor network to encode the basis functions for the entire task distribution.
We design a Weight generator network to generate the weights from the K training samples of a novel task drawn from the same task distribution.
We show that our model has competitive performance in in various few short regression tasks.
A CKNOWLEDGMENTS
This work was supported by both ST Electronics and the National Research Foundation (NRF),
Prime Minister’s Ofﬁce, Singapore under Corporate Laboratory @
University Scheme (Programme Title: STEE Infosec - SUTD Corporate Laboratory). The authors would also like to thank the anonymous reviewers for their valuable comments A T ECHNICAL D ETAILS A. T HE F EATURE E XTRACTOR
We represent the feature extractor as a a 2 layer 40-dimensional fully connected network which transforms each 1-dimensional input sample into a 40-dimensional feature, Each hidden layer in the feature extractor also followed by a ReLU non-linearity activation function (Nair & Hinton, 2010).
A. T HE W EIGHTS G ENERATOR
The Weights Generator takes in a 41 + t n dimensional input where t n is the dimension of the t ask label of each task.
The input is passed through a fully connected layer of dimension 64 to produce the embedding for each of the task’s training samples.
The extracted embedding is then passed through a series of 3 attention blocks.
Each block is comprised of a dot-product self-attention operation followed by two fully connected layers of dimensions 128 and 64 respectively and ﬁnally a layer normalization operation
(Ba et al., 2016) at the end.
The ﬁrst fully connected layer after the selfattention module in each block is also followed by a ReLU non-linearity activation function (Nair &
Hinton, 2010).
Each block also has a residual connection (He et al., 2016) from the embedding to the output of the second fully connected layer.
The attention function we used is the scaled dot-product attention introduced by Vaswani et al.
[Equation]
where Q , K and V represent the Query, Key and Value of the Attention module respectively.
d k represents the dimension of K and is used as a scaling factor.
In our case as we using self-attention,
Q , K and V are all represented by the task embedding.
The output of the attention blocks are then passed though another fully connected layer of dimension 40 to ensure the same dimensionality as the sample features.
The ﬁnal weight vector, w is obtained by taking the mean of the output.
A. T ASK L ABEL G ENERATOR
The Task Label Generator takes in a 41 dimensional input has a similar architecture to the Weights
Generator.
It consists of a 64 dimensional fully connected layer followed by 3 attention blocks similar to that in the Weights Generator.
The Task Label Generator outputs a t n dimensional task label for each training sample and the ﬁnal task label, t is obtained by taking the mean of all the individual task labels.
The generated task label is then passed through a sigmoid activation layer.
A. T RAINING P ROCEDURE
We train our model in two stages.
In the ﬁrst stage, only the Feature Extractor and Weights Generator are updated.
In this stage we use the true task labels from the regression tasks as input to the Weights
Generator.
The loss function used is mean squared error between the labels of the validation set labels and the prediction.
[Equation]
Where j is number of samples in the validation set.
As the Feature Extractor is tasked to only map a low dimensional samples x to high dimensional features f , through training the Feature Extractor eventually learns to map samples to features that are most suited for the task domain.
The Feature
Extractor can therefore be thought as implicitly modeling and capturing the task distribution p ( T ) .
For any given regression task, the set of features f is ﬁxed.
The reason that the model is able to predict different y (cid:48) value from the same f is purely due to the weights vector w .
As the Weights
Generator has access to both sample label and task label information, it is trained to output weights vectors that capture a speciﬁc task instance T i within the task distribution p ( T ) .
In the second stage, we train the entire model including the Task Label Generator.
In this stage the predicted task labels from the Task Label Generator are used as input to the Weights Generator instead of the true task labels.
We also use a identical mean-squared loss function for the this stage of training but instead updating all three sets of parameters θ F , θ G and θ T .
B D ATASET AND T RAINING D ETAILS
Here we provide details on all of our regression tasks and some additional training details of our model.
For all the regression tasks we normalize the task labels to a [0 , 1] range using it for training.
We also provide visual results of our model’s performance on all three regression tasks in the following section
B. S INUSOIDAL R EGRESSION
For the sinusoidal regression task, we follow the experimental set up in Yoon et al.
The target function is deﬁned as y ( x ) =
Asin ( ωx + b ) + (cid:15) , where amplitude A , phase b , frequency ω and noise (cid:15) are the parameters of the function.
We sample the each parameters uniformly from range A ∈ [0 . 1 , 5 . 0] , b ∈ [0 , 2 π ] and ω ∈ [0 . 5 , 2 . 0] .
For noise (cid:15) , we sample it from distribution
N ∼ (0 , 0 .
A 2 ) .
We use the amplitude, phase and frequency values to form a 3-dimensional task labels for the sinusoidal task
B. 1D H EAT E QUATION
For the heat equation task, we deﬁne it as such: Consider a 1-dimensional rod of length L with both of its ends connected to heat sinks, i.e. the temperature of the ends will always be ﬁxed at 0 K unless a heat source is applied at the end.
a constant point heat source is then applied to a random point s on the rod such the the heat point source will always have a temperature of 1.0 K .
We would like the model the temperature u ( x, t ) at each point of the rod a certain time t after applying the heat source until the temperature achieves equilibrium throughout the rod.
The temperature at each point x after time t is given by the heat equation:
[Equation]
In our experiments, we set L to be 5 and randomly sample 10 points of position range [0 , 5] on the heat equation curve for a 10-shot regression task.
We use both the x and t to form a 2 dimensional task label for the heat equation task.
B. 2D G AUSSIAN
For the 2D Gaussian tasks, the model is trained to predict the probability distribution function of a two-dimensional Gaussian distribution.
We train our model from Gaussian distribution task with mean ranging from ( − 2 , − 2) to (2 , 2) and standard deviation of range [0 . 1 , 2] .
We ﬁx the standard deviation to be of the same value in both directions.
The model is once again trained on 10 randomly sampled input points for a 10-shot regression task.
We plot the predictions of points in the range of [ µ − 3 , µ + 3 ] where µ is mean of the Gaussian task..
Similar to the heat equation tasks, we use the mean and standard deviation values to form a 3 dimensional task label.
B. A DDITIONAL T RAINING D ETAILS
For all of our experiments except the one comparing against BMAML, we train both our model on regression tasks of batch size 32 for 50000 steps for both training stages.
For experiment comparing against BMAML for the sinusoidal task, we train our model on tasks of batch size 10 forsteps using a pre-trained Task Label Generator trained on 50000 training steps.
We also limit the data set in this experiment to just 1000 regression tasks for fair comparison.
We use the Adam
Optimizer (Kingma & Ba, 2014) as the optimization method to preform stochastic gradient decent on our model.
We implement all our models using the Tensorﬂow (Abadi et al., 2016) library and train them on a Nvidia GTX 1080 Ti GPU.
C A DDITIONAL A BLATION S TUDY
In this section, we further illustrate our justiﬁcation made for our model by conducting an additional ablation study.
We intend to show that that learned features f indeed correspond to a set of basis functions {
φ i ( x ) } that correspond to the function F ( x ) .
If that is indeed the case, each of the learned basis function φ i ( x ) should correspond to certain characteristics of the regression function, and removing one basis function from the ﬁnal prediction should give noticeable difference in the models ﬁnal prediction.
Thus, we conduct an ablation study where we remove one of the dimensions in f and visualize the change in the ﬁnal prediction.
We conduct this study on sinusoidal regression task show the results in Figure 2.
The results shows that removing certain dimensions from the feature vector does give signiﬁcantly different results and does correspond to certain characteristics of the regression function.
Namely for the ﬁrst case, the removed feature correspond the accurate magnitude and position of the regression curve whereas for the second case, the removed feature correspond to the ”shape” of the regression curve at range [-2.0 ,2.4].
[Figure]
D V ISUAL R ESULTS
In this section we show some example visual results of our model’s results on the three regression tasks.
In Figure 3, we show some examples of our models results on the sinusoidal task.
In Figure 4, we show examples of our model results on the 1D heat equation task and compare it against the results from MAML (Finn et al., 2017).
Finally in Figure 5, we show examples our model results on the 2D Gaussian task.
MAML (Finn et al., 2017)
[Figure]
Model
[Figure]