M ULTI -T ASK N EURAL P ROCESSES
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural processes have recently emerged as a class of powerful neural latent variable models that combine the strengths of neural networks and stochastic processes.
As they can encode contextual data in the network’s function space, they offer a new way to model task relatedness in multi-task learning.
To study its potential, we develop multi-task neural processes , a new variant of neural processes for multi-task learning.
In particular, we propose to explore transferable knowledge from related tasks in the function space to provide inductive bias for improving each individual task.
To do so, we derive the function priors in a hierarchical Bayesian inference framework, which enables each task to incorporate the shared knowledge provided by related tasks into its context of the prediction function.
Our multi-task neural processes methodologically expand the scope of vanilla neural processes and provide a new way of exploring task relatedness in function spaces for multi-task learning.
The proposed multi-task neural processes are capable of learning multiple tasks with limited labeled data and in the presence of domain shift.
We perform extensive experimental evaluations on several benchmarks for the multi-task regression and classiﬁcation tasks.
The results demonstrate the effectiveness of multi-task neural processes in transferring useful knowledge among tasks for multi-task learning and superior performance in multi-task classiﬁcation and brain image segmentation 1 .
I NTRODUCTION
As deep neural networks are black-box function approximations, it is difﬁcult to introduce prior domain or expert knowledge into a prediction function (Jakkala, 2021).
In contrast, Gaussian processes (Rasmussen, 2003) explicitly deﬁne distributions over functions and perform inference over these functions given some training examples.
This enables reliable and ﬂexible decisionmaking.
However, Gaussian processes can suffer from high computational complexity due to the manipulation of kernel matrices.
Therefore, there has been continuous interest in bringing together neural networks and Gaussian processes (Damianou & Lawrence, 2013; Wilson et al., 2016; Garnelo et al., 2018a; Jakkala, 2021) into so-called neural processes.
Neural processes (Garnelo et al., 2018b) combine the computational efﬁciency of neural networks with the uncertainty quantiﬁcation of stochastic processes.
They are a class of neural latent variables model, which deploy a deep neural network to encode context observations into a latent stochastic variable to model prediction functions.
Neural processes provide an elegant formalism to efﬁciently and effectively incorporate multiple datasets into learning distributions over functions.
This formalism is also promising in multi-task learning to improve individual tasks by transferring useful contextual knowledge among related tasks.
Their capability of estimating uncertainty over predictions also makes them well-suited for multi-task learning with limited data, where each task has only a few training samples.
However, neural processes rely on the implicit assumption that the context and target sets are from the same distribution and can be aggregated by a simple average pooling operation (Kim et al., 2019; Volpp et al., 2020).
This makes it non-trivial to directly apply neural processes to modeling multiple heterogeneous tasks from different domains, where the context data of different tasks are from distinctive distributions (Long et al., 2017).
[Equation]
[Figure]
[Equation]
In this paper, we develop multi-task neural processes (MTNPs), a methodological extension of neural processes for multi-task learning, which ﬁlls the theoretical gap of neural processes for multi-task learning.
Particularly, we propose to explore task relatedness in the function space by specifying the function priors in a hierarchical Bayesian inference framework.
The shared knowledge from related tasks is incorporated into the context of each individual task, which serves as the inductive bias for making predictions in this task.
The hierarchical architecture allows us to design expressive datadependent priors.
This enables the model to capture the complex task relationships in multitask learning.
By leveraging hierarchical modeling, multi-task neural processes are capable of exploring shared knowledge among related tasks in a principled way by specifying the function prior.
We validate the effectiveness of the proposed multi-task neural processes by extensive experiments in both multi-task classiﬁcation and regression.
The results demonstrate that multi-task neural processes can effectively capture task relatedness in the function space and consistently improve the performance of each individual task, especially in the limited data regime.
P RELIMINARIES : N EURAL P ROCESSES
In this section, we brieﬂy review vanilla neural processes (Garnelo et al., 2018b) based on which we derive our multi-task neural processes.
Let a data set be given composed of training samples and corresponding labels.
In order to better reﬂect the desired model behaviour at test time (Garnelo et al., 2018b), the training data is split into the context set D =
( X , Y ) and a target set D ∗ =
( X ∗ , Y ∗ ) , where X = { x 1 , · · · , x n } is a subset of the training data, Y = { y 1 , · · · , y n } the corresponding set of labels and n the size of the context set.
To ensure both sets have the same data distribution, the context set is split from the target set.
Now, given the context set ( X , Y ) , we would like to estimate a function that can make predict the labels Y ∗ for target samples X ∗ .
In general, we deﬁne a stochastic process by a random function f : X → Y .
Given the context set, we deﬁne the joint distribution over the function values { f ( x 1 ) , · · · , f ( x n ) } , which in Gaussian processes is a multivariate Gaussian distribution parameterized by a kernel function.
The rationale of neural processes is to extract knowledge from the context set to specify the prior over the prediction function.
Instead of a kernel function, neural processes adopt a deep neural network to deﬁne the prior distribution.
Speciﬁcally, the model introduces a latent variable z to account for uncertainty in the predictions of Y ∗ .
The observed context set is encoded into the latent variable which is conditioned on the context set, i.e. it follows the prior distribution p ( z | X , Y ) .
The latent variable z is a high-dimensional random vector parameterising the stochastic process by f ( X ∗ )
=
g ( X ∗ , z ) .
The function g ( · , · ) is an extra ﬁxed and learnable decoder function, which is also implemented by a neural network.
Thus, the neural process model can be formulated as follows:
[Equation]
The graphical model for neural processes is shown in Figure 1 (a).
The neural process model is optimized using amortized variational inference.
Let q ( z | X ∗ , Y ∗ ) be a variational posterior of the latent variable z .
The evidence lower-bound (ELBO) for neural processes is given as follows:
[Equation]
In neural processes, the function space deﬁned by deep neural networks allows the model to extract deep features while retaining a probabilistic interpretation (Jakkala, 2021).
In multi-task learn- ing, usually different tasks can be from different domains and have their speciﬁc data distributions (Lawrence & Platt, 2004; Long et al., 2017).
Due to the complex data structure of multi-task learning, it is not straightforward to explore task relatedness in such function spaces.
In this paper, we aim to extend the methodology of the neural process to the scenarios of multi-task learning to learn the shared knowledge among tasks for improving individual tasks.
M ULTI -T ASK N EURAL P ROCESSES
The common setup of multi-task learning is that there are multiple related tasks for which we would like to improve the learning of each individual one by sharing information across the different tasks (Williams et al., 2007).
Multi-task learning has been studied under different settings (Requeima et al., 2019;
Williams et al., 2007; Lawrence & Platt, 2004; Yu et al., 2005; Long et al., 2017).
In this paper, we tackle the multi-input multi-output setting, where each task has a different distribution while different tasks share the same target space (Lawrence & Platt, 2004; Yu et al., 2005;
Long et al., 2017; Zhang et al., 2020).
It aims to improve the overall performance of all multiple tasks simultaneously different from the sequential multi-task leaning (Requeima et al., 2019;
Garnelo et al., 2018a).
This is a challenging scenario due to the domain shift between tasks, which makes it sub-optimal to directly apply neural processes by incorporating the data from other related tasks into the context of each individual task.
3. H IERARCHICAL C ONTEXT MODELING
Multi-task learning considers the estimation of random functions f l , l = 1 , ..., L for each of the L related tasks.
Each task l has its own training data, which is split into a context set D l =
( X l , Y l ) and a target set D ∗ l =
( X ∗ l , Y ∗ l ) .
X ∗ l ∈ R n ∗ l × d and X l ∈ R n l × d are inputs while Y ∗ l ∈ R n ∗ l × C and
Y l ∈ R n l × C are outputs.
d and C are the sizes of the input space and output space, respectively.
n ∗ l and n l are the sizes of respectively the target and context set for the l -th task.
Thus, we obtain the task-speciﬁc latent variables f l = f l (
X l ) ∈ R n l × C .
We use { D l } Ll =1 to denote all context sets in the dataset, which for brevity we represent as { D l } and likewise will do for other sets.
Formulated this way the goal of multi-task learning is to predict { Y ∗ l } for given { X ∗ l } simultaneously with the assistance information { D l } from all tasks.
To this end, we construct a joint prediction distribution with respect to the latent random functions { f l } as:
[Equation]
Here, to enable shared knowledge to be transferred among tasks, we introduce a global variable M which works as a container to collect the useful information from { D l } of all tasks.
In contrast to neural processes for single tasks, the global variable M provides the contextual information from all tasks for each individual task.
The concrete formation of M depends on the learning scenarios.
For regression tasks, M ∈ R L ∗ d and each row corresponds to one task, which is the average of all feature vectors from a task.
For classiﬁcation tasks, M ∈ R L ∗ C ∗ d where each vector M l,c is the average of all features of each category from the corresponding task.
Similar to Gaussian processes, we assume that the function value is Y ∗ l = f l ( X ∗ l ) + (cid:15) , where (cid:15) ∼ N (0 , σ 2 ) is the observation noise.
For regression tasks, we can deﬁne the predictive likelihood on the target set as p ( Y ∗ l | X ∗ l , f l ) =
N ( Y ∗ l | f l ( X ∗ l ) , σ 2 ) .
For classiﬁcation tasks, we use log p ( Y ∗ l | X ∗ l , f l )
=
(cid:80) n ∗ l i =
y ∗ l,i log( f l ( x ∗ l,i )) as the log-likelihood function, where x ∗ l,i is the i -th target sample from the l -th task.
In order to combine Gaussian processes and neural networks in the context of multi-task learning, we deﬁne p ( f l | M ) in (3) as a deep neural network in place of a Gaussian distribution parameterized by a kernel function.
To be more speciﬁc, we assume that f l is parameterized by a random variable ψ l by deﬁning f l ( X ) =
X ψ (cid:62) l .
We specify a data dependent prior by conditioning ψ l on the global variable M .
In this way, we incorporate the transferable knowledge into the learning of the prediction function of the current tasks.
Thus, the predictive distribution for the l -th task over its target set can be formulated as follows:
[Equation]
In effect, ψ l denotes the parameters of classiﬁer for classiﬁcation tasks or regressors for regression tasks.
Particularly, the introduced latent variable ψ l ∈ R C ×
d denotes the task speciﬁc classiﬁer, where d is the dimension of the input feature and C is the number of classes in the dataset.
As done in (Requeima et al., 2019), we generate each column of ψ l independently from the context samples of the corresponding class.
In our case, each column of ψ l encodes the context information of its class from all tasks p
[Equation]
Directly aggregating M as done in neural processes for single tasks is not applicable for multi-task learning due to the distribution shift between tasks.
The data from related tasks should be processed and adapted to the current task as the contextual information.
To this end, we introduce a higherlevel latent variable α l to extract the shared knowledge from M , which is conditioned on the data
D l of each task.
This results in a hierarchical Bayesian modeling of functions in the neural process:
[Equation]
where α l is the latent variable to control the access to shared knowledge for each task, which is used to explore the relevant knowledge to the task l .
p θ 1 ( ψ l | α l , M ) and p θ 2 ( α l | D l ) are prior distributions of the latent variable ψ l and α l , respectively, which are parameterized with neural networks.
To be speciﬁc, we deﬁne p θ 1 ( ψ l | α l , M ) =
N ( ψ l | µ ( m l ) , Σ( m l )) .
Here m l contains the relevant knowledge to the task l , which is adapted from the global variable M by a deterministic function m
l = h
( α l , M ) , where h ( · ) is a learnable function implemented with a neural network.
By substituting (5) into (4), we obtain the model of multi-task neural processes as follows:
[Equation]
The designed hierarchical context modeling provides a principled way to explore task relatedness in the function space, which allows task-speciﬁc function variables to leverage the shared knowledge from related tasks.
We provide theoretical proof in Appendix to show that the proposed multi-task neural processes are a valid stochastic processes, which completes the theory of multi-task neural processes.
The graphical model of the multi-task neural processes is shown in Figure 1 (b).
3. V ARIATIONAL H IERARCHICAL I NFERENCE
The previous section developed the model of multi-tasks neural processes with a hierarchical context model.
We now describe how to optimize the model and obtain the predictions by leveraging a variational Bayesian inference framework.
To that end, based on a conditional independence assumption, we introduce the variational joint posterior distribution factorized as follows:
[Equation]
where q ϕ 1 ( ψ l | D ∗ l ) and q ϕ 2 ( α l | D ∗ l ) are variational posteriors of the latent variables ψ l and α l for the task l , respectively.
Both variational posteriors are parameterized as Gaussian distributions.
ϕand ϕ 2 are amortized inference networks to generate variational posteriors
We make use of the amortized variational inference technique (Kingma & Welling, 2013) to learn such distributions over latent variables.
Learning By incorporating the variational posteriors into (7), we derive the ELBO for the multitask neural processes as follows:
[Equation]
The detailed derivation is provided in Appendix A. By adopting the Monte Carlo sampling, we obtain the empirical objective for the proposed multi-task neural processes:
[Equation]
where ψ ( j ) l ∼ q ϕ 1 ( ψ l | D ∗ l ) and α ( i ) l ∼ q ϕ 2 ( α l | D ∗ l ) .
N f and N a are the number of Monte Carlo samples for the variational posteriors of ψ and α , respectively.
λ
f and λ a are the hyperparameters to help stably train the KL-divergence terms, which are set following the annealing scheme of (Bowman et al., 2015).
In practice, we apply the local reparameterization trick (Kingma et al., 2015) to reduce the variance of stochastic gradients.
Prediction Having the learned model, we can make prediction on the target set.
Given a test sample x l from the l -th task, we can produce the predictive distribution which involves the prior distributions p θ 1 ( ψ l | α l , M ) and p θ 2 ( α l | D l ) .
The predictive distribution with the integration for the introduced latent variables is formulated as:
[Equation]
Here we also need to apply the Monte Carlo estimation over (10) and obtain the predictions as follows:
[Equation]
where ψ ( j ) l ∼ p θ 1 ( ψ l | α ( i ) l , M ) and α ( i ) l ∼ p θ 2 ( α l | D l ) .
In particularly, the prior distribution of the latent variable ψ l is generated from the output of the learned function h ( α ( i ) l , M ) .
R ELATED W ORK
Neural network based models have achieved impressive results on various applications (LeCun et al., 2015).
However, due to the large number of parameters these neural models need large-scale data with annotation.
It is challenging to train a deep neural model that generalizes well with numberlimited labeled data.
To reduce such labeling consumption, recent works (Long et al., 2017; Liu et al., 2016) follow a multi-task learning strategy to fully leverage information from relevant tasks as inductive bias (Caruana, 1997) to improve each task’s performance.
The main challenge of multitask learning is that each task is provided with a limited amount of labeled data, which is insufﬁcient to build reliable classiﬁers without overﬁtting (Long et al., 2017).
Multi-task learning (MTL) aims to learn several tasks simultaneously and improve their overall performance.
The crux of MTL is how to explore task relatedness from different tasks, which could be particularly signiﬁcant when limited data for each task is available (Long et al., 2017; Zhang et al., 2020).
Recently, the task relatedness is learned in many different aspects of the model, e.g., loss functions (Liu et al., 2020;
Qian et al., 2020; Kendall et al., 2018), parameter space (Long et al., 2017; Bakker & Heskes, 2003), or representation space (Misra et al., 2016).
In this paper, we focus on the data-insufﬁcient problem for multi-task learning.
Different from other branches of MTL models (Huang et al., 2021; Fu et al., 2021; Phillips et al., 2021) which leverage several channels of supervision information simultaneously included by the same input, our data setting allows each task to have its own individual data.
This is much more challenging due to the distribution shift between inputs of different tasks.
In previous works (Lawrence & Platt, 2004; Yu et al., 2005; Youseﬁ et al., 2019), multi-task learning beneﬁts from Gaussian Processes by generally incorporating the shared information in the Gaussian
Processes prior to synergize several random functions of different tasks simultaneously.
The multitask Gaussian processes (Yu et al., 2005) propose a hierarchical prior which enables task speciﬁc random functions to share some common structure in the hyper prior.
As an alternative to using
[Figure] [Figure]
samples
[Figure]
Gaussian processes for multi-task learning, nowadays deep neural networks have become popular as parameterized functions, which construct an information sharing architecture between tasks.
Some methods (Misra et al., 2016; Liu et al., 2019) learns expressive combinations of features from different tasks by deep neural networks.
Some approaches (Gao et al., 2020; Sun et al., 2019) ﬂexibly adjusts the deep architecture for each individual tasks.
Yu et al.
(2020) mitigates gradient interference by performing the proposed gradient surgery.
However, these deep multi-task learning methods relies on large amounts of training data and therefore tends to overﬁt with limited data.
Recently, neural processes (Garnelo et al., 2018b) and the related works (Garnelo et al., 2018a;
Kim et al., 2019; Wang & Van Hoof, 2020; Requeima et al., 2019; Gordon et al., 2019) combine
Gaussian Processes and neural networks, which are not only computationally efﬁcient but also retain a probabilistic interpretation of the model.
(Garnelo et al., 2018b; Kim et al., 2019; Wang &
Van Hoof, 2020) introduce the latent representation variables to model the function randomness.
(Requeima et al., 2019; Gordon et al., 2019) introduce a latent parameter variable to directly model the predictive distribution, which acts as the parameters of the neural network.
Hierarchical modeling in the Bayesian framework has been successful to design the form of the prior (Daum ´ e III, 2009;
Zhao et al., 2017; Klushyn et al., 2019; Wang & Van Hoof, 2020) and posterior distributions (Ranganath et al., 2016; Krueger et al., 2017; Zhen et al., 2020)
based on many observations.
It allows the latent variable to follow a complicated distribution and forms a highly ﬂexible approximation (Krueger et al., 2017).
E XPERIMENTS AND R ESULTS 5. 1-D M ULTI -T ASK F UNCTION R EGRESSION
Our method fully utilizes the transferable knowledge provided by related tasks to improve each task’s performance.
To show this, we test the proposed multi-task neural processes on the 1-D multi-task function regression.
Setup.
We deﬁne several tasks with different data distributions: inputs of each task are sampled from the separated intervals without overlapping, such as [ − 2 π, π ) , [ − π, 0) , [0 , π ) , and [ π, 2 π ) .
Each x -value is drawn uniformly at random in its belonging intervals.
We assume that all tasks share the same ground truth function to ensure that there is transferable knowledge between them.
The ground truth function is characterized as a sum of sine or cosine functions (Wang & Van Hoof, 2020), such as y = sin( x )+sin(2 x ) − cos(0 .
5 x )+ (cid:15) , where (cid:15) is the noise drawn from N (0 , 0 . 0003 2 ) .
At each training step, the hyperparameters of the ground truth function are ﬁxed as set in (Kim et al., 2019; Wang & Van Hoof, 2020).
Results.
As shown in Figure 2, we visualize prediction results of different tasks for the comparison of neural processes and multi-task neural processes.
The predictions of the proposed multi-task neural processes (the second row) more resemble the ground truth functions than that of neural processes (the ﬁrst row), especially in the boundary of different tasks.
Moreover, we ﬁnd that when training with fewer context samples, the improvement is more signiﬁcant.
This demonstrates that our method enhances the generalization of each task by fully utilizing the transferable knowledge from related tasks.
5. M ULTI - TASK C LASSIFICATION
Datasets.
We evaluate the performance of our multi-task neural processes on real-world multitask classiﬁcation, where different tasks are deﬁned as image classiﬁcation problems in different domains.
The tasks are related to each other since they share the same label space.
Office-Home (Venkateswara et al., 2017) contains images from four domains/tasks: Artistic (A), Clipart (C),
Product (P) and Real-world
(R).
Each task contains images from 65 categories collected under ofﬁce and home settings.
There are about 15 , 500 images in total.
Office-Caltech (Gong et al., 2012) contains the ten categories shared between Ofﬁce-31 (Saenko et al., 2010) and Caltech-256 (Grifﬁn et al., 2007).
One task uses data from Caltech-256 (C), and the other three tasks use data from
Ofﬁce-31, whose images were collected from three distinct domains/tasks, namely Amazon (A),
Webcam (W) and DSLR (D).
There are 8 ∼ 151 samples per category per task, and 2 , 533 images in total.
ImageCLEF (Long et al., 2017), the benchmark for the ImageCLEF domain adaptation challenge, contains 12 common categories shared by four public datasets/tasks:
Caltech-256 (C),
ImageNet ILSVRC 2012 (I), Pascal VOC 2012 (P), and Bing (B).
There are 2 , 400 images in total.
Setup.
We adopt the standard evaluation protocols (Zhang & Yang, 2021) for multi-task classiﬁcation datasets.
We randomly select 5% , 10% and 20% labeled data for training, which correspond to about 3, 6 and 12 samples per category per task, respectively.
In this case, each task has a limited amount of training data, which is insufﬁcient for building the reliable classiﬁer without overﬁtting.
For all three benchmarks, we extract the input features by the pretrained VGGnet as (Long et al., 2017).
The architectures of inference networks used in our model are provided in Appendix C. All the results with error bars are obtained based on a 95 % conﬁdence interval from ﬁve runs.
Compared methods.
To show the effectiveness of the proposed multi-tasks neural processes, we conduct a thorough comparison implementing multiple different baseline models.
Single task learning (STL) is implemented by task-speciﬁc feature extractors and classiﬁers without knowledge sharing among tasks.
Basic multi-task learning (BMTL) shares feature extractors and adds task speciﬁc classiﬁers.
We also deﬁne variational extensions of the single task learning (VSTL) and basic multitask learning (VBMTL), which cast models as variational Bayesian problems and treat classiﬁers as latent variables (Shen et al., 2021).
We implement neural processes (NPs) and its variant, neural processes with all task context for comparison.
NPs with all task context is a straightforward extension of NPs for MTL, which treats context sets from all tasks equally without hierarchical context modeling.
For a fair comparison, all the above-mentioned methods share the same architecture of the feature extractor.
[Table]
Results.
We provide comprehensive comparisons on Office-Home in Table 1, which is a more challenging multi-task classiﬁcation dataset with 65 categories.
The results show that our MTNPs outperform other counterpart methods.
NPs with all task context performs better than NPs by a large margin when 5% labeled data is available, showing the beneﬁt of exploring shared knowledge from related tasks with limited data.
Noticeably compared to NPs with all task context, the proposed multi-task neural processes beneﬁt from hierarchical context modeling and show even better performance.
[Table] [Table]
[Table] [Table]
ImageCLEF datasets are shown in Tables 2, 3 and 4, respectively.
The average accuracy of all tasks is used for overall performance measurement.
The best results are marked in bold.
Our MTNPs achieve competitive and even better performance on such multi-task classiﬁcation datasets with different train-test splits.
Compared with Bayesian baselines, including VSTL, VBMTL, NPs and (Bakker & Heskes, 2003), our MTNPs directly infer the parameters of prediction functions rather than the input representation, which is able to model a broader range of functional distribution.
Moreover, in function space the hierarchical context modeling can better explore the task relatedness, which enable the models to capture the relevant knowledge even in presence of distribution shift among tasks.
Experimental results on all three benchmarks with 20% labeled data are provided in Appendix D.
5. M ULTI -T ASK R EGRESSION
Setup.
In order to show the effectiveness of MTNPs for multi-task regression, we conduct experiments on the Rotated MNIST dataset (LeCun et al., 1998).
We adopt this dataset to study multi-task regression, where each task is an angle estimation problem for each digit and different tasks corresponding to different digits are related because they share the same rotation angle space.
Each image is rotated by 0 ◦ through 90 ◦ in intervals of 10 ◦ , where the rotation angle is the regression target.
We randomly choose 0 .
1% samples per task per angle as the training set.
Results.
Since we would like to improve the overall performance of all regression tasks, we use the average of normalized mean squared errors of all tasks as the measurement.
As shown in Table 5, our MTNPs outperform other counterpart methods by yielding an overall lower mean error.
[Figure]
5. B RAIN I MAGE S EGMENTATION
In this section, we demonstrate that multi-task neural processes are also able to explore spatial context information to improve image segmentation.
To this end, we adopt a brain image dataset (Buda et al., 2019) with lower-grade gliomas collected from 110 patients.
The number of images varies among patients from 20 to 88 .
The goal is to segment the tumor in each brain image.
Setup.
To apply our multi-task neural processes, we reformulate the segmentation task as a pixelwise regression problem, where each pixel corresponds to a regression task to predict the probability of this pixel belonging to the tumor.
In doing so, the spatial correlation and dependency among pixels are effectively modeled by capturing the task relatedness.
To be speciﬁc, we consider the prediction of each pixel to be a regression task.
For the task l , we deﬁne Ω l as a local region centered at the spatial position l , which provides the local context information.
In this case, the region centered at the pixel provides the local context information.
Each task incorporates the shared knowledge provided by related tasks into its context of the prediction function.
This offers an effective way to model the long-range interdependence of pixels in one image.
For implementation, we use the
U-Net architecture (Ronneberger et al., 2015) as the backbone and add our model as the ﬁnal layer.
Results.
We compare our method and U-Net on the brain segmentation dataset.
The results show that the proposed multi-task neural processes surpass the baseline U-Net by 0.5% in terms of dice similarity coefﬁcients (DSC) for the overall validation set.
Figure 3 shows segmentation results of the proposed multi-task neural processes (bottom row) and the U-Net (upper row), where the green outline corresponds to the ground truth and the red to the segmentation output.
Our multi-task neural processes predict contours closer to the ground truth.
This demonstrates the advantages of exploring context information by multi-task neural processes for segmentation.
C ONCLUSION
In this paper, we develop multi-task neural processes, a new variant of neural processes for multitask learning.
We propose to explore the task relatedness in the function space by specifying the function priors in a hierarchical Bayesian inference framework.
The shared knowledge from related tasks is incorporated into the context of each individual task, which serves as the inductive bias for making predictions of this task.
The hierarchical architecture allows us to design expressive datadependent prior, enabling the model to explore the complex task relationships in multi-task learning.
By leveraging the hierarchical modeling, multi-tasks neural processes are capable of capturing the shared knowledge from other tasks in a principled way by specifying the function prior.
We evaluate multi-task neural processes on multi-task regression and classiﬁcation datasets.
Results demonstrate the effectiveness of multi-task neural processes in transferring useful knowledge among tasks for multi-task learning.
E THICS S TATEMENT
Since our method is a general contribution to the multi-task learning ﬁeld, it has the potential to beneﬁt any societal problem where multiple tasks are performed simultaneously, e.g., medical diagnosis.
Accordingly, the method would also potentially face some negative social impacts accompanying with these applications, e.g., lack of fairness with the model trained by incomplete data due to the privacy of patients in medical imaging tasks.
R EPRODUCIBILITY S TATEMENT
To make our method reproducible, we provide the details of the method in Section 3.
We also give the derivations of the evidence lower bound proposed in Section 3 in Appendix A.
The datasets used in this paper can be obtained by referencing Section 5.
There are also the experimental settings and implementation details for each dataset in Section 5.
We also provide an anonymous link for the code of our method, which is in the abstract of this paper.
A D ERIVATION OF THE ELBO FOR MULTI - TASK NEURAL PROCESSES
We provide a derivation of ELBO of the proposed multi-task processed with hierarchical context modeling.
The likelihood of multi-task learning is as follows:
[Equation]
Based on conditional independence assumption
[Equation]
we introduce the variational joint posterior distribution factorized as (7).
By incorporating the variational posteriors in the log likelihood
[Equation]
we can obtain the ELBO as follows:
[Equation]
B P ROOF OF E XCHANGABILITY AND C ONSISTENCY
We further provide theoretical proof to show that the proposed multi-task neural processes are valid stochastic processes, which completes the theory of multi-task neural processes.
As the statement in Garnelo et al.
(2018b): the conditions, including (ﬁnite) exchangeability and consistency, are sufﬁcient to deﬁne a stochastic process.
In our multi-input multi-output setting, we observe L tasks, { X ∗ l } where X ∗ l = { x ∗ l,i } n ∗ l i =1 and { Y ∗ l } where Y ∗ l = { y ∗ l,i } n ∗ l i =1 .
x ∗ l,i denotes the i -th target samples from task l and y ∗ l,i is its corresponding target or label.
Here are the two propositions to state the exchangeability and consistency of the proposed multi-task neural processes.
We model the functional posterior distribution of the stochastic process by approximating the joint predictive distribution over each target set p ( { Y ∗ l }|{ X ∗ l } , { D l } ) , which is conditioned on all context samples { D l } .
Proposition 1 (Exchangability) For ﬁnite n = (cid:80)
L l =1 n ∗ l , if π = { π l } L l =1 is a permutation of { 1 , ..., n } where π l is a permutation of the corresponding order set { 1 , ..., n ∗ l } , then:
[Equation]
where π ( { Y ∗ l } ) :
= ( π 1 ( Y ∗ 1 ) , ..., π L
( Y ∗ L ))
=
( y ∗ π (1) , ..., y ∗ π ( n ) ) and π ( { X ∗ l } ) :
= ( π 1 ( X ∗ 1 ) , ..., π L ( X ∗ L )) =
( x ∗ π (1) , ..., x ∗ π ( n ) ) .
Proof.
[Equation]
Proposition 2 (Consistency)
Given m = (cid:80) Ll m l =1 , if 1 ≤ m ≤ n or for each task 1 ≤ m l ≤ n ∗ l , then:
[Equation]
(( { Y ∗ l } ) 1: m | ( { X ∗ l } ) 1: m , { D l } ) , (16) where ( { Y ∗ l } ) 1: m =
(( Y ∗ 1 ) 1: m 1 , ..., ( Y ∗ L ) 1: m L ) =
( y ∗ 1 , ..., y ∗ m ) and ( { X ∗ l } ) 1: m =
(( X ∗ 1 ) 1: m 1 , ..., ( X ∗ L ) 1: m L ) =
( x ∗ 1 , ..., x ∗ m ) .
Proof.
[Equation]
C M ORE E XPERIMENTAL D ETAILS
Details of iteration numbers and batch sizes for different benchmarks are provided in Table 6.
In each batch, the number of training samples from each task and category is identical.
We train all models and parameters by the Adam optimizer Kingma & Ba (2014) using an NVIDIA Tesla V
GPU.
The learning rate is initially set as 1 e − 4 and decreases with a factor of 0 .
5 every 3 ,iterations.
The network architectures of the proposed multi-task neural processes for multi-task classiﬁcation are provided as follows
[Table]
[Table] [Table]
[Table]
The architecture of the inference network ϕ 1 is provided in Table 7.
The architecture of the inference network ϕ 2 is provided in Table 8.
We note that the inference network θ 1 and θ 2 share the same architectures with ϕ 1 and ϕ 2 , respectively.
The architecture of the neural network h is provided in
Table 9.
The network is needed because it provides a data-driven way for the model to incorporate the task-speciﬁc latent variable α l and the global variable M , which are usually deﬁned in different feature spaces.
During inference, we apply the reparameterization trick to generate the samples for the latent variables (Kingma & Welling, 2013).
D M ORE E XPERIMENTAL R ESULTS D. M ULTI - TASK REGRESSION AND CLASSIFICATION WITH THE 20% SPLIT
Further, we provide experiments results on the three multi-task classiﬁcation datasets with 20% training samples in Table 10, 11 and 12.
The proposed multi-task neural processes consistently achieve the best performance on all three benchmarks.
[Table]
[Table] [Table]
D. M ULTI - TASK REGRESSION WITH LESS DATA
To show the advantages of our model
[Table]
we compare them on the setting of less data with the 0 .
05% split.
In this case, there are only 20 samples per task during training.
As shown in the Table 13, the improvement of our method becomes larger.
[Table]
D. S TABILITY OF OUR METHOD
The computational advantage of our method can be illustrated by the training loss as function of iteration on Ofﬁce-Home .
As shown in Fig.
4, our MTNPs converges more stable than NPs under 5% , 10% and 20% train-test splits.
Moreover, we have added a new experiment to investigate the training stability by introducing the noise to the input data
[Table]
We apply the the fast gradient sign method (Goodfellow et al., 2014) to generate the noise.
The results are given in Table 14, where η denotes the noise level.
We observe that our MTNPs show better stability than NPs at different noise levels.
[Table]
D. S ENSITIVITY OF THE H YPERPARAMETER N f AND N a
In practice, we set N f and N a to be 10 and 5, which offer a good balance between performance and efﬁciency.
We determine them by grid search as shown in Table 15 and
[Table]
[Table] [Table]