M ULTI -T ASK N EURAL P ROCESSES
A BSTRACT
Neural Processes (NPs) consider a task as a function realized from a stochastic process and ﬂexibly adapt to unseen tasks through inference on functions.
However, naive NPs can model data from only a single stochastic process and are designed to infer each task independently.
Since many real-world data represent a set of correlated tasks from multiple sources ( e.g., multiple attributes and multi-sensor data), it is beneﬁcial to infer them jointly and exploit the underlying correlation to improve the predictive performance.
To this end, we propose Multi-Task Neural Processes (MTNPs), an extension of NPs designed to jointly infer tasks realized from multiple stochastic processes.
We build MTNPs in a hierarchical way such that inter-task correlation is considered by conditioning all per-task latent variables on a single global latent variable.
In addition, we further design our MTNPs so that they can address multi-task settings with incomplete data ( i.e., not all tasks share the same set of input points), which has high practical demands in various applications.
Experiments demonstrate that MTNPs can successfully model multiple tasks jointly by discovering and exploiting their correlations in various real-world data such as time series of weather attributes and pixel-aligned visual modalities.
I NTRODUCTION
Neural Processes (NPs) (Garnelo et al., 2018b) are a class of meta-learning methods that model a distribution of functions ( i.e. a stochastic process).
By considering a task as a function realized from the underlying stochastic process, they can ﬂexibly adapt to various unseen tasks through inference on functions.
The adaptation requires only one forward step of a trained neural network without any costly retraining or ﬁne-tuning, and has linear complexity to the data size.
NPs can also quantify their prediction uncertainty, which is essential in risk-sensitive applications (Gal &
Ghahramani, 2016).
Thanks to such appealing properties, there have been increasing attempts to improve NPs in various domains, such as image regression (Kim et al., 2019; Gordon et al., 2020), image classiﬁcation (Requeima et al., 2019; Wang & Van Hoof, 2020), time series regression (Qin et al., 2019; Norcliffe et al., 2021), and spatio-temporal regression (Singh et al., 2019).
In this paper, we explore extending NPs to a multi-task setting where correlated tasks are realized simultaneously from multiple stochastic processes.
Many real-world data represent multiple correlated functions, such as different attributes or modalities.
For instance, medical data (Johnson et al., 2016;
Harutyunyan et al., 2019) or climate data (Wang et al., 2016) contain various correlated attributes on a patient or a region that need to be inferred simultaneously.
Similarly, in multi-task vision data (Lin et al., 2014; Zhou et al., 2017; Zamir et al., 2018), multiple labels of different visual modalities are associated with an image.
In such scenarios, it is beneﬁcial to exploit functional correlation by modeling the functions jointly rather than independently, in terms of performance and efﬁciency (Caruana, 1997).
Unfortunately, naive NPs lack mechanisms to jointly handle a set of multiple functions and cannot capture their correlations either.
This motivates us to extend NPs to model multiple tasks jointly by exploiting the inter-task correlation.
In addition to extending NPs to multi-task settings, we note that handling multi-task data often faces a practical challenge where observations can be incomplete ( i.e. not all the functions share the common sample locations).
For example, when we collect multi-modal signals from different sensors, the sensors may have asynchronous sampling rates, in which case we can observe signals from only an arbitrary subset of sensors at a time.
To fully utilize such incomplete observations, the model should be able to associate functions observed in different inputs such that it can improve the predictive performance of all functions using their correlation.
A multivariate extension of Gaussian Processes (GPs) (Álvarez et al., 2012) can handle incomplete observations to infer multiple functions jointly.
However, naive GPs suffer from cubic complexity to the data size and needs approximations to reduce the complexity.
Also, their behaviour depend heavily on the kernel choice (Kim et al., 2019).
To address these challenges, we introduce Multi-Task Neural Processes (MTNPs), a new family of stochastic processes that jointly models multiple tasks given possibly incomplete data.
We ﬁrst design a combined space of multiple functions, which allows not only joint inference on the functions but also handling incomplete data.
Then we deﬁne a Latent Variable Model (LVM) of MTNP that theoretically induces a stochastic process over the combined function space.
To exploit the inter-task correlation, we introduce a hierarchical LVM consists of (1) a global latent variable that captures knowledge about all tasks and (2) task-speciﬁc latent variables that additionally capture knowledge speciﬁc to each task conditioned on the global latent variable.
Inducing each task conditioned on the global latent, the hierarchical LVM allows MTNP to effectively learn and exploit functional correlation in multi-task inference.
MTNP also inherits advantages of NP, such as ﬂexible adaptation, scalable inference, and uncertainty-aware prediction.
Experiments in synthetic and real-world datasets show that MTNPs effectively utilize incomplete observations from multiple tasks and outperform several NP variants in terms of accuracy, uncertainty estimation, and prediction coherency.
P RELIMINARY 2. B ACKGROUND : N EURAL P ROCESSES
We consider a task f t : X → Y t as a realization of a stochastic process over a function space ( Y t ) X that generates a data D t =
( X D , Y tD ) = { ( x i , y t i ) } i ∈I ( D t ) , where I ( D t ) denotes a set of data index.
Neural Processes (NPs) use a conditional latent variable model to learn the stochastic process.
Given a set of observations C t =
( X C , Y tC )
= { ( x i , y ti ) } i ∈I ( C t ) , NP infers the target task f t through a latent variable z and models the data D t by a factorized conditional distribution p ( Y tD | X D , z ) : p [Equation] , z [Equation]
z |
[Equation]
We refer to the set of observations C t as a context data and the modeling data D t as a target data.
NP models the generative model p ( Y tD | X D , z ) and the conditional prior p ( z | C t ) by two neural networks, a decoder p θ and an encoder q φ , respectively.
Since the direct optimization of Eq.1 is intractable, the networks are trained by maximizing the following variational lower-bound.
[Equation]
Note that the decoder network q φ is also used as a variational posterior q φ ( z | D ) .
The parameter sharing between model prior and variational posterior gives us an intuitive interpretation of the loss function: the KL term acts as a regularizer for the encoder q φ such that the summary of the context is close to the summary of the target.
This reﬂects the assumption that the context and target are generated by the same underlying data-generating process and aids effective test-time adaptation.
After training, NP infers the target function according to the latent variable model (Eq.1).
2. E XTENDING TO M ULTIPLE T ARGET F UNCTIONS
Now we extend the setting to multi-task learning problems where multiple tasks f 1 , · · · , f T are realized from T stochastic processes simultaneously, each of which has its own function space ( Y t ) X , ∀ t ∈ T = { 1 , 2 , · · · , T } .
Let D =
( X D , Y 1: T D ) =
(cid:83) t ∈T D t be a multi-task target data, where each D t corresponds to the data of task f t .
Then the learning objective for the set of T realized tasks is to model the conditional probability p ( Y 1: T D | X D , C ) given the multi-task context
C =
( X C , Y 1: T C )
= (cid:83) t ∈T C t , where each C t is a set of observations of task f t .
The sets C and D can be arbitrarily chosen, but we assume C ⊂ D for simplicity.
However, assuming the complete context C for all tasks is often challenged by many practical issues, such as asynchronous sampling across multiple sensors or missing labels in multi-attribute data.
To address such challenges, we relax the assumptions on context C and let I ( C t ) be different across t ∈ T .
In this case, an input point x i can be associated with a partial set of output values { y ti } t ∈T i , T i (cid:40) T , which is referred incomplete observation.
Next, we present two ways to use NPs to model the multi-task data and discuss their limitations.
Single-Task Neural Processes (STNPs)
A straightforward application of NPs to the multi-task setting is assuming independence across tasks and deﬁne independent NPs over the function spaces ( Y 1 ) X , · · · , ( Y T ) X .
We refer to this approach as Single-Task Neural Processes (STNPs).
Speciﬁcally, a STNP has T independent latent variables v 1 , · · · , v T , where each v t implicitly represents a task f t .
[Equation]
[Figure]
Thanks to the independence assumption, STNPs can handle incomplete context by conditioning on each task-speciﬁc data C t independently.
However, this approach can only model the marginal distributions for each task, ignoring complex inter-task correlation within the joint distribution of the tasks.
Note that this is especially impractical for multi-task settings under the incomplete data since each task f t can be learned only from C t , ignoring rich contexts available in other data C t (cid:48) , ∀ t (cid:48) (cid:54) =
t .
Joint-Task Neural Process (JTNP)
An alternative approach is to combine output spaces to a product space Y 1:
T = (cid:81) t ∈T Y t and deﬁne a single NP over the function space ( Y 1: T ) X .
We refer to this approach as Joint-Task Neural Processes (JTNPs).
In this case, a single latent variable z governs all T tasks jointly.
[Equation]
[Equation]
JTNPs are amenable to incorporate correlation across tasks through the shared variable z .
However, by deﬁnition, they require complete context and target for both training and inference.
This is because any incomplete set of output values { y ti } t ∈T i for an input point
x i such that T i (cid:54)
= T is not a valid element of the product space Y 1: T .
In addition, it relies solely on a single latent variable to explain all tasks, ignoring per-task stochastic factors in each function f t .
In what follows, we propose an alternative formulation for jointly handling multiple tasks on incomplete data, which (1) enables a probabilistic inference on the incomplete data and (2) is more amenable for learning both task-speciﬁc and task-agnostic functional representations.
M ULTI -T ASK N EURAL P ROCESSES
In this section, we describe Multi-Task Neural Processes (MTNPs), a family of stochastic processes to model multiple functions jointly and handle incomplete data.
We ﬁrst formulate MTNPs using a hierarchical LVM.
Then we propose the training objective and a neural network model.
3. F ORMULATION
Our objective is to extend NPs to jointly infer multiple tasks from incomplete context.
Discussions in Section 2.2 suggest that direct modeling of a distribution over functions of form f : X → (cid:81) t ∈T Y t is achievable via JTNP (Eq. 4), yet it requires complete data in both training and inference.
To circumvent this problem, we reformulate the functional form by h : X × T →
(cid:83) t ∈T Y t .
Note that this functional form allows us to model the same set of functions as JTNP by f ( x i ) =
( h ( x i , 1) , · · · , h ( x i , T )) .
However, by using the union form we can exploit incomplete data since any partial set of output values { y ti } t ∈T i now becomes a set of valid output values at different input points ( x i , t ) , t ∈ T i .
For notational convenience, we denote x ti =
( x i , t ) and assume input points in the context C and the target D are embedded by the task indices, i.e., C =
( X 1: TC , Y 1: T C )
= (cid:83) t ∈T C t where C t =
( X tC , Y tC )
= { ( x ti , y ti ) } i ∈I ( C t ) and the same for D .
Next, we present a latent variable model that induces a stochastic process over functions of form h .
To make use of both task-agnostic and task-speciﬁc knowledge, we deﬁne a hierarchical latent variable model (Figure 1(c)).
In this model, the global latent variable z captures shared stochastic factors across tasks using the whole context C , while per-task stochastic factors are captured by the task-speciﬁc latent variable v t using C t and z .
It induces the predictive distribution on the target by:
[Equation]
where v 1: T := ( v 1 , · · · , v T ) .
Similar to Eq.
1, we assume the conditional independence on p ( Y tD | X tD , v t ) .
Note that this hierarchical model can capture and leverage the inter-task correlation by sharing the same z across v 1: T .
Also, it is amenable to fully utilize the incomplete data: since the global variable z is inferred from the entire context data C
= (cid:83) t
∈T C t and is conditioned to infer task-speciﬁc latent variable v t , each function f t induced by v t exploits the observations available for not only itself C t , but also for other tasks C t (cid:48) , ∀ t (cid:48) (cid:54) =
t .
Next, we show that Eq.
5 induces a stochastic process over the functions of form h : X × T
→
(cid:83) t ∈T Y t .
Proposition 1.
Consider the following generative process on data D and context C , which is a generalized form of Eq. 5.
[Equation]
Then under some mild assumptions, there exists a stochastic process over functions of form h :
X × T →
(cid:83) t ∈T Y t , where the data D is generated.
Proof.
We leave the proof in Appendix A.2.
We refer to the resulting stochastic processes as Multi-Task Neural Processes (MTNPs).
In the perspective of stochastic process, Eq. 5 allows us to learn functional posterior not only on each task via v t , but also across the tasks via z .
Then optimizing Eq.
5 can be interpreted as learning to learn each task captured by v t together with the functional correlation captured by z .
3. L EARNING AND INFERENCE
We use an encoder network q φ and a decoder network p θ to approximate the conditional prior and generative model in Eq. 5, respectively.
Since the direct optimization of Eq.
5 is intracable, we train the networks via the following variational lower bound, where we use the same network q φ for both conditional prior and variational posterior as in NP:
[Equation]
We leave the derivation in Appendix A.3.
The above objective reﬂects several desirable behaviors for our model.
Similar to NP, the KL divergences encourage that both latent variables z and v t inferred from the context data are consistent with those inferred from the entire target data.
On the other hand, we observe that minimizing the KL divergence on task-speciﬁc variables forces the global latent z to be informative across all tasks, such that it can induce the task-speciﬁc factors v t from the limited context C t .
This makes the model encode correlated information across tasks in z and use it for inferring each task with v t , which is critically important for joint inference with incomplete context data.
After training, MTNP infers the target functions according to the latent variable model (Eq. 5).
3. N EURAL N ETWORK MODEL FOR MTNP
This section presents an implementation of MTNPs composed of an encoder q φ and a decoder p θ (Eq. 7).
While our MTNP formulation is not restricted to a speciﬁc architecture, we adopt ANP (Kim et al., 2019) as our backbone, which implements the encoder by attention layers (Vaswani et al., 2017) and the decoder by a MLP.
Figure 2 illustrates the overall architecture.
In the following, we denote a stacked multi-head attention block (Parmar et al., 2018) by
Attn ( Q, K, V ) and a MLP by ψ ( x ) .
Also, we denote e t by a learnable task embedding for t ∈ T which is used to condition on the task index t .
Latent Encoder
The latent encoder samples global and per-task latent variables by aggregating the context C .
For each context example ( x ti , y ti ) ∈ C t , we ﬁrst project it to a hidden representation s ti =
ψ s ( x i , y ti ) + e t .
Then we aggregate them to a task-speciﬁc representation s t via self-attention followed by a pooling operation, which is further aggregated to a global representation s .
[Equation]
Note that the ﬁrst attention is applied along the example axis (per-task) to encode information of each task, while the second one is applied along the task axis (across-task) to aggregate the information across the tasks.
Then, we get the global and task-speciﬁc latent variables via ancestral sampling.
[Equation]
s
Ts Tn
Cross-Attn d Td Tn ...
u T* s T
Per-task inference
Value
Key
Query xx n x * ...
Self-Attn& Pool ...
Taskx 1 y 1 1 ex n y Tn e T ...
Per-task decodingx *
MLP y 1 * vr 1 * es Ts Tn
Cross-Attn d Td Tn ...
u T* s T
Per-task inference T
Value
Key
Query xx n x * ...
Self-Attn& Pool ...
Task T x 1 y T 1 e T x n y Tn e T v T z ...
Self-Attn& Pool vs T s...
Across-task inference ...
Self-Attn ...
u T* u 1 * r T* r 1 * ~ ~ ~ x *
MLP y T* v T r T* e T
Per-task decoding T ~ ...
Concat
Deterministic path Latent path ~
Sampling xx n x * ...
Context
Target y ty tn y t
Deterministic Encoder To further improve the expressiveness of model, we extend the deterministic encoder of Kim et al.
(2019) that produces local representation speciﬁc to both target example and task via attention mechanism.
As in the latent encoder, we ﬁrst project each context example ( x ti , y ti )
∈ C t to a hidden representation d ti = ψ d ( x i , y ti )
+ e t that serves as value embedding in cross-attention.
Also, we use context and target input x t i as key and query embeddings for the crossattention, respectively.
Then we apply cross-attention along the example axis (per-task) followed by self-attention along the task axis (across-task).
[Equation]
Decoder Finally, the decoder produces predictive distributions for the target output y ti ∈ Y tD for each target input x ti .
We ﬁrst project the input to w ti = ψ w ( x i ) + e t , then concatenate it with the corresponding latent variable v t and determinstic representation r ti .
The output distribution is computed by MLPs, whose output depends on the type of the task.
[Equation]
R ELATED W ORK
Stochastic Processes for Multi-Task Learning There exist several stochastic processes related to
MTNPs that consider learning multiple tasks.
Multi-Output Gaussian Processes (MOGPs) (Álvarez et al., 2012) extend Gaussian Processes (GPs) to infer multiple tasks together, and also handle incomplete data.
However, MOGPs are usually trained on a single set of tasks, thus require a lot of observations to produce accurate predictions.
Recently there have been some attempts to combine meta-learning and GPs (Fortuin et al., 2019; Titsias et al., 2020), while they do not consider multitask settings.
Sequential Neural Processes (SNPs) (Singh et al., 2019; Yoon et al., 2020) treat a sequence of tasks using a sequence of NPs.
While SNPs are designed to model temporal dynamics underlying a sequence of homogeneous tasks, MTNPs can capture arbitrary correlation across a set of heterogeneous tasks.
Finally, Conditional Neural Adaptive Processes (CNAPs) (Requeima et al., 2019; Bateni et al., 2020) consider a general classiﬁcation model for different sets of classes.
However, like NPs, they infer each task independently and do not explicitly consider inter-task correlation during inference.
Also, CNAPs are designed speciﬁcally to classiﬁcation tasks, while MTNPs are generally applicable to various tasks including classiﬁcation and regression.
Hierarchical Models in Neural Process Family Since the pioneering work by Garnelo et al.
(2018b), several variants of NP introduced the concept of hierarchical modeling.
Attentive Neural
Processes (ANPs) (Kim et al., 2019) incorporate attention mechanism to a deterministic variable, which is additional context information for each target example to improve the expressive power of the model and prevent the underﬁtting issues of vanilla NPs.
Similarly, Wang & Van Hoof (2020) introduce local latent variables to incorporate example-speciﬁc stochasticity, which extends the graphical model of NPs to the hierarchical one.
Our MTNP formulation also involves a hierarchical latent variable model but has a different structure orthogonal to the prior works.
MTNPs use a global latent variable to jointly model multiple task functions while using per-task latent variables to capture task-speciﬁc stochasticity.
Although extending the model to contain example-level local latent variables is possible, we adopt the deterministic local representation as in ANPs for simplicity.
E XPERIMENTS
We evaluate MTNP on three datasets, including both synthetic and real-world tasks.
In all experiments, we construct incomplete context data by selecting a complete subset C ⊂ D of size m = |I
( C ) | from the target, then randomly drop the output points independently according to the missing rate γ ∈ [0 , 1] ( γ = 0 means complete data).
We repeat the procedure with ﬁve different random seeds and report the mean values of each evaluation metric.
Baselines In each experiment, we compare MTNP with two NP variants, STNP and JTNP.
We adopt ANP (Kim et al., 2019) as a backbone architecture for STNP and JTNP, which is a strong NP baseline.
Since JTNP cannot handle incomplete data, we build a stronger baseline by the combination of STNP and JTNP (S+JTNP), where missing labels are imputed by STNP and then used to jointly infer the tasks by JTNP.
In 1D regression tasks, we additionally compare two Multi-Output Gaussian Processes baselines, CSM (Ulrich et al., 2015) and MOSM (Parra & Tobar, 2017), and two metalearning baselines, MAML (Finn et al., 2017) and Reptile (Nichol et al., 2018), where we slightly modify the meta-learning baselines to learn multiple tasks jointly from incomplete data.
At training time, we set γ = 0 .
5 for all models but keeping γ = 0 for JTNP.
At test time, we evaluate the models in various missing rates γ ∈ { 0 , 0 . 25 , 0 .
We provide architectural and training details in
Appendix B. We also provide ablation studies on architectural designs such as self-attentions and pooling, parameter sharing and task embedding, latent and deterministic encoders in Appendix H.
5. 1D C URVE R EGRESSION ON S YNTHETIC D ATA
Dataset and Metric We begin with 1D synthetic regression tasks where the target functions are correlated by shared parameters ( e.g., scale, bias, phase) but have different shapes.
Inspired by Guo et al.
(2020b), we ﬁrst randomly sample global parameters a, b, c, w ∈ R shared across the tasks, then generate four correlated tasks using different activation functions as follows.
[Equation]
To simulate task-speciﬁc stochasticity, we perturb the parameters ( a, b, c, w ) with small i.i.d.
Gaussian noises per task.
In this setting, the model has to learn per-task functional characteristics imposed by different activation functions and per-task noises, as well as how to share the underlying parameters unseen during training among the tasks.
For evaluation, we generate training and testing sets via non-overlapping splits of the parameters, then measure mean squared error (MSE) normalized by the scale parameter a to aggregate results on functions with different scales.
See Appendix C for details.
Results Table 1 shows the quantitative results with γ = 0 .
More comprehensive results with different missing rates and standard deviations for the metrics are provided in Appendix D. As it shows, MTNP outperforms all baselines in all tasks and context sizes.
This can be attributed to the ability of MTNP to (1) exploit all available context examples to infer inter-task general knowledge ( i.e. a, b, c, w ) and (2) translate it back to functional representations for each task.
In contrast, STNP fails to predict multiple tasks accurately due to the independent task assumption.
Although JTNP is designed to discover and utilize inter-task correlations, its performances do not show dramatic improvement over STNP since its observations are largely based on noisy imputations from STNP.
We also observe that GP baselines (MOSM, CSM) perform even worse than STNP when the context size is small, despite their inherent ability to joint inference on incomplete data.
We conjecture that it is because GPs lack a meta-training mechanism that allows NPs (and MTNPs) to quickly learn the tasks using a few examples.
Gradient-based meta-learning baselines (MAML, Reptile) are also comparable to STNP and JTNP but perform worse than MTNP.
This could be due to the lack of global inference on function space, which leads them to overﬁt the context points.
As an illustrating example, we also plot predicted distributions from the models in a highly incomplete scenario ( m = 10 and γ = 0 . 5 ) in Figure 3 (a).
We observe that STNP generally suffers from inaccurate predictions due to limited context, while MTNP successfully exploits incomplete observations from different tasks to improve the predictive performance.
The qualitative results for all baselines are provided in Appendix D.
We also perform an ablation study on the latent variable model to justify the effectiveness of our hierarchical formulation.
We consider two variants of MTNP that consist of the global latent variable only (MTNP-G) and the task-speciﬁc latent variables only (MTNP-T).
Then we evaluate the models in three synthetic datasets generated with different levels of inter-task correlation.
Speciﬁcally, we construct partially correlated tasks as described before, totally correlated tasks by removing the task-speciﬁc noises, and independent tasks by sampling the parameters a, b, c, w independently for
[Table]
[Figure]
each task.
Figure 3 (b) shows the result in m = 10 and γ = 0 .
When the tasks are correlated (ﬁrst and second rows of the ﬁgure), we can see introducing global latent improves the overall performance, which is further improved by the hierarchical formulation.
When the tasks are independent (third row of the ﬁgure), sharing all knowledge through a single global latent degrades the performance (MTNP-G).
On the other hand, MTNP and MTNP-T do not suffer from such a negative transfer since each of the independent tasks can be addressed by per-task latent variables separately.
The overall results demonstrate that incorporating both global and task-speciﬁc information is the most effective and robust against various levels of inter-task correlation.
5. 1D T IME -S ERIES R EGRESSION ON W EATHER D ATA
Dataset and Metric To demonstrate our method in a practical, real-world domain, we perform an experiment on weather data.
Weather attributes are physically correlated with each other and the observations are often incomplete due to different sensor conﬁgurations or coverage per station.
Also, the observed attributes are highly stochastic, making MTNP’s stochastic process formulation ﬁts it well.
We use a dataset gathered by Dark Sky API 1 , consisting of 12 daily weather attributes collected at 266 cities for 258 days.
We choose six attributes, namely low and high temperatures (TempMin,
TempMax), humidity (Humidity), precipitation probability (Precip), cloud cover (Cloud) and dew point (Dew), which forms six correlated tasks.
We normalize each attribute to be standard Gaussian and the time to be in [0 , 1] .
We divide the data into 200 training, 30 valid, and 33 test sets of time series, where each set corresponds to a unique city.
We evaluate the prediction performance by MSE.
Since the data is noisy, we also report negative log-likelihood as a metric of uncertainty estimation.
Results Table 2 summarizes quantitative results.
More comprehensive results with different missing rates, context sizes, and standard deviations for the metrics are provided in Appendix E. MTNP outperforms all baselines in both accuracy and uncertainty estimation, which demonstrates that it generalizes well to real-world stochastic data.
More interestingly, Figure 4 illustrates how MTNP
[Table]
variance transfer its knowledge from one task (Cloud) to another (Precip) given the incomplete observations.
When the observation is sparse (Figure 4(a)), the model produces an inaccurate prediction with high uncertainty for unobserved input domains.
However, when the additional observations are available for the other attribute (Figure 4(b),(c)), MTNP successfully transfers the knowledge to improve the prediction.
It shows that MTNP can effectively learn to exploit the incomplete observation by transferring knowledge across tasks.
5. 2D I MAGE C OMPLETION ON F ACE D ATA
Dataset and Metric We further demonstrate our approach to more challenging 2D structured function regression tasks.
Following Garnelo et al.
(2018a)
, we interpret an RGB image as a function that maps a 2D pixel location x i ∈ [0 , 1] 2 to its RGB values y i ∈ [0 , 1] 3 , and extend its concept to pixel-aligned 2D spatial data for the multi-task setting.
Speciﬁcally, we consider four pixel-aligned visual modalities with a resolution of 32 × 32 on celebrity faces as a set of tasks, namely RGB image (RGB) (Liu et al., 2015), semantic segmentation map (Segment) (Lee et al., 2020), Sobel edge (Edge) (Kanopoulos et al., 1988), and Projected Normalized Coordinate Code (PNCC)
(Zhu et al., 2016).
We then construct training and testing sets with non-overlapping splits of face images.
To evaluate the Segment task, we report mean Intersection-over-Union (mIoU).
For the other tasks, we report MSE.
We also measure prediction coherency across tasks to evaluate the task correlation captured by models.
To measure the coherency between the predictions, we generate pseudo-labels by translating the RGB prediction into the other three modalities using image-to-image translation methods (Kanopoulos et al., 1988; Guo et al., 2020a; Chen et al., 2018), then measure errors (MSE or 1 - mIoU) between the pseudo-labels and predictions.
Additional details are provided in Appendix F.
Results Table 3 summarizes the quantitative comparison results.
More comprehensive results with different missing rates are provided in Appendix G. Overall, we observe similar results with the 1D regression experiments where MTNP generates more accurate predictions over STNP and S+JTNP by effectively exploiting the incomplete data.
We also observe that the MTNP produces more coherent predictions over the baselines, which shows that it indeed learns to exploit the correlation across tasks effectively.
To further validate the results, we present qualitative comparison results in Figure 5.
We observe that STNP and S+JTNP produce inaccurate (red boxes) or incoherent (green box) outputs when the number of contexts is extremely small.
On the other hand, MTNP (1) consistently regresses coherent functions regardless of the number of observable contexts, and (2) its predictions are more accurate than the baselines given the same number of contexts (green box).
Finally, we investigate the discovery and exploitation of task correlations achieved by MTNP.
We ﬁrst partition tasks into source and target tasks.
Then, we measure relative performance improvement on the target tasks before and after the model observes data from source tasks.
We summarize the results in Table 4, where we average performance gains coming from all possible combinations of source tasks for each target task.
By observing which task is the most beneﬁcial to each of the other tasks, we observe that there are two groups of highly correlated tasks (RGB-Edge) and (Segment-PNCC).
These
[Table]
m =
( e.g. m = 10 ), STNP and S+JTNP produce inaccurate outputs ( e.g. mis-predicting hairs and poses as in the green box) or incoherent outputs ( e.g. different head poses as in the red box).
results demonstrate that MTNP successfully captured dependence among tasks considering that (1)
RGB and Edge are composed of two correlated low-level signals ( e.g. color intensity and its gradients)
[Table]
(2) while both Segment and PNCC contain high-level semantic information on facial landmarks.
Note that discovering inter-task correlations is one of the actively studied topics in the machine learning literature, where the efforts often come at the cost of extra computations and resources due to hard-coded (Zamir et al., 2018; Standley et al., 2020) or hand-crafted (Pal & Balasubramanian, 2019) algorithms.
C ONCLUSION
We propose Multi-Task Neural Processes (MTNPs), a new family of stochastic processes designed to infer multiple functions jointly from incomplete data, along with a hierarchical latent variable model.
Through extensive experiments, we demonstrate that the proposed MTNPs can leverage incomplete data to solve multiple heterogenous tasks by learning to discover and exploit task-agnostic and task-speciﬁc knowledge.
Scaling up our method to large-scale datasets will be a promising research direction.
To this end, our method can be improved in several aspects by (1) generalizing to unseen task space T and (2) allowing empty context data for some tasks such that we can generalize MTNPs in more diverse real-world scenarios such as zero-shot inference and semi-supervised learning.
Ethics Statement Recently, detecting and removing data bias have become essential problems towards producing fair machine learning models.
We believe that our work can contribute to detect unintentional data bias present in multi-attribute data.
MTNP can be seen as a universal correlation learner who learns arbitrary correlation across tasks purely data-driven way.
Therefore, given potentially biased multi-attribute data (e g., multiple personal attributes), MTNP may detect any biased relationship by learning the correlation between them.
For example, we may perform the task-to-task transfer analysis on a trained MTNP as discussed in Section 5.2 and Section 5.3, then see which task (or attribute) has a high correlation with another task (or attribute).
Reproducibiltiy Statement
In this work, we present two major theoretical results (Propositionand Eq. 7), a neural network model (Section 3.3), and experiments on three datasets (Section 5).
We give a complete proof of Proposition 1 in Appendix A.2 and the ELBO derivation for Eq.
7 in
Appendix A.3.
We provide architectural details and training hyper-parameters of the models used in the experiments in Appendix B. Finally, details on the experimental settings and datasets are provided in Appendix C and Appendix F
A PPENDIX A T HEORETICAL J USTIFICATIONS
In this section, we give a proof of Proposition 1 with a brief introduction of the Kolmogorov Extension
Theorem (Itô et al., 1984), and derive training objectives of STNP, JTNP, and MTNP.
A. S TOCHASTIC P ROCESSES AND THE K OLMOGOROV E XTENSION T HEOREM
A stochastic process F : X × Ω → Y is a collection of random variables { Y x : Ω → Y} x ∈X which is indexed by an index set X .
Also, all the random variables are deﬁned on a single probability space (Ω , F , P ) and a value space Y .
This can be interpreted as a distribution over a function space Y X , such that sampling a function f corresponds to f ( · ) =
F ( · , ω ) , ω ∈ Ω .
Another interpretation of F is a random function, since F ( x, · ) =
Y x is a random variable.
Suppose we have observed input and output sequences X
= ( x 1 , x 2 , · · · , x n ) and Y =
( y x 1 , y x 2 , · · · , y x n ) of a function f : X → Y .
With a slight abuse of a notation, let p ( Y | X ) =
ρ X ( Y ) be the marginal distribution of Y on a product space (cid:81) n i =1 Y , where each i -th space of the product is the output space of Y x i , 1 ≤ i ≤ n .
Then by the Kolmogorov Extension Theorem, the data ( X, Y ) induces a stochastic process F such that ∃ ω ∈ Ω s.t.
y x i =
F ( x i , ω ) for all i
= 1 , 2 , · · · , n , if the distribution p ( Y | X ) satisﬁes two conditions: consistency and exchangability.
(Consistency)
For any m such that 1 ≤ m < n ,
[Equation]
where X i 1 : i 2 =
( x i 1 , x i 1 +1 , · · · , x i 2 ) and Y i 1 : i 2 =
( y i 1 , y i 1 +1 , · · · , y i 2 ) for all i 1 ≤ i 2 .
(Exchangability)
For any permutation π on X 1: n (a permutation π on set S is a bijection π : S → S ),
[Equation]
where X 1: n = { x 1 , · · · , x n } , π ◦ X =
( π ( x 1 ) , π ( x 2 ) , · · · , π ( x n )) , and π ◦ Y =
( y π ( x 1 ) , y π ( x
, y π ( x n ) ) .
A. MTNP IS A S TOCHASTIC P ROCESS
In the case of MTNP, we observe input and output sequences X =
(( x 1 , t 1 ) , ( x 2 , t 2 )
, · · · , ( x n , t n )) and
Y =
( y ( x 1 ,t 1 ) , y ( x 2 ,t 2 ) , · · · , y ( x n ,t n ) ) of a function h : X ×T
→ (cid:83) t ∈T Y t .
Note that in the main text, we abbreviate x ti =
( x i , t ) and y ti =
y ( x i ,t ) for visibility.
Now we want to show the existence of a stochastic process H :
X × T × Ω → (cid:83) t
∈T Y t , where the data D =
( X, Y ) is generated.
This can be done by showing the following conditions.
(Consistency)
For any m such that 1 ≤ m < n ,
[Equation]
where X i 1 : i 2 =
(( x i 1 , t i 1 ) , · · · , ( x i 2 , t i 2 )) and Y i 1 : i 2 =
( y ( x i
[Equation]
) , · · · , y ( x i 2 ,t i 2 ) ) .
(Exchangability)
For
any permutation π on X 1: n
× T ,
[Equation]
where X 1: n = { x 1 , · · · , x n } , π ◦ X
=
( π (( x 1 , t 1 )) , · · · , π (( x n , t n ))) , and π ◦ Y =
( y π (( x 1 ,t 1 )) , · · · , y π (( x n ,t n )) ) .
Here p ( Y | X, C )
=
ρ X ( Y | C ) is the conditional distribution of Y given any context C .
Note that C is conditioned since we are modeling functional posterior of h , rather than prior .
Now we provide the proof of Proposition 1, which states that the following generative model deﬁnes a stochastic process.
[Equation] [Equation]
To show the conditions of Kolmogorov Extension Theorem, we need two assumptions on the data generating process (Eq. 20).
First, we assume the distribution deﬁned by the data generating process is ﬁnite so that the order of integral can be swapped.
Also, we assume that the conditional distribution p ( y ( x i ,t ) | x i , t, v t ) can implicitly select the per-task latent variable v t among v 1: T using the given task index t , i.e., there exists a distribution
˜ p
such that ˜ p ( y ( x i ,t ) | x i , t, v 1: T ) =
p ( y ( x i ,t ) | x i , t, v t ) .
This means no more than that the latent variables v 1: T are indeed task-speciﬁc , such that each v t corresponds to task f t .
Note that our neural-network model of MTNP (Figure 2) indeed satisﬁes the second assumption, since the decoder selects the corresponding per-task latent variable v t given the task index t ∈ T .
Proof.
We ﬁrst show the consistency condition.
From the data generating process (Eq. 20),
[Equation] [Equation]
[Equation]
Next, we show the exchangability condition.
Let π 1 , π 2 be the values of ﬁrst and second coordinate of π , such that π (( x i , t i ))
=
( π 1 (( x i , t i )) , π 2 (( x i , t i ))) .
Then p
[Equation]
X
[Equation]
[Equation]
= p ( Y | X, C )
[Equation]
[Equation]
Here we used the assumption about p ( y ( x i ,t ) | x i , t, v t ) such that ˜ p
( y π (( x i ,t i ))
| π (( x i , t i )) , v 1: T ) =
p ( y π (( x i ,t i ))
| π (( x i , t i )) , v π 2 (( x i ,t i )) ) .
Since 1 ≤ m < n and π are arbitrarily chosen, the data generating process (Eq. 20) satisﬁes the conditions of the Kolmogorov Extension Theorem.
Thus there exists a stochastic process H : X × T × Ω
→ (cid:83) t ∈T Y t , whose realizations are functions of the form h : X × T →
(cid:83) t ∈T Y t .
Note that the latent variable model of MTNP (Eq. 5) is a special case of the data generating process (Eq. 20), where p ( v t | z, t, C )
= p ( v t | z, C t ) .
Thus MTNP is a stochastic process over the functions of form h : X × T →
(cid:83) t ∈T Y t .
A. ELBO D ERIVATION FOR MTNP
We derive the evidence lower bound (ELBO) for log p θ ( Y 1: T D | X 1: TD , C ) .
Recall that C
=
( X 1: TC , Y 1: T C )
= (cid:83) t ∈T C t where C t =
( X tC , Y tC )
= { ( x ti , y ti ) } i ∈I ( C t ) and D =
( X 1: TD , Y 1: T D )
=
(cid:83) t ∈T D t where D t =
( X tD , Y tD ) =
{ ( x ti , y ti ) } i ∈I ( D t ) .
For simplicity, we assume C t ⊂ D t for all t
so that C ⊂ D as well.
Also, to avoid confusion, for this derivation we denote the conditional prior networks as p θ ( z | C t ) and p θ ( v t | z, C t ) and then replace them with q φ ( z | C t ) and q φ ( v t | z, C t ) respectivelly, when we introduce parameter-sharing between prior and variational posterior networks.
First, the conditional log-likelihood has a lower bound
[Equation]
where p θ ( Y 1: T D | X 1: TD , C, z ) in Eq.
36 can be further expanded by
[Equation]
On Eq.
34 and Eq.
40, we use the conditional independence relation follows from the latent variable model (Eq. 5)
By combining Eq.
36 and Eq. 43, and also by sharing the parameters of conditional priors p θ ( z | C )
and p θ ( v t | z, C t ) with variational posteriors q φ ( z | C )
q φ ( v t | z, C t ) , we get
[Equation]
A. ELBO FOR STNP AND JTNP
STNP is no more than a collection of independent Neural Processes (NPs), where each NP corresponds to each task.
Using T encoders and decoders { ( p θ t , q φ t ) }
Tt =1 , the objective for STNP can be derived by summing up the NP objectives (Eq. 2).
We omit the ELBO derivation for NP.
Note that the parameter sharing is used for the conditional prior network p θ t ( v t | C t ) and the variational posterior network q φ t ( v t | D t ) .
[Equation]
On the other hand
[Equation]
JTNP is a single NP that models all tasks jointly, by concatenating the output variables into a single vector.
Using an encoder q φ and a decoder p θ , the objective for JTNP is the same as the NP objective.
Again, the encoder q φ serves as both conditional prior and variational posterior.
[Equation]
Note that STNP and JTNP model functions with input space X , so there is no superscript t in the input variables.
B A RCHITECTURAL AND T RAINING D ETAILS
In this section, we provide architectural and training details about models used in the experiments (Section 5).
B. A TTENTIVE N EURAL P ROCESS
As a strong NP baseline, we adopt Attentive Neural Processes (ANPs) (Kim et al., 2019) architecture for STNP and JTNP.
The encoder of ANP consists of a latent path and a deterministic path, each computes a latent variable z and a deterministic representation r i speciﬁc to each target example x i .
Then the decoder produces a distribution for the target output y i , which is assumed to be Normal.
The general architecture of ANP follows Kim et al.
(2019), while two major modiﬁcations have made as follows.
First, we replace the average pooling operation in stochastic path by a Pooling by
Multihead Attention (PMA) layer which is introduced in Lee et al.
Next, since we have a categorical task (Segment) in 2D experiment, we modify the decoder for each Segment task to output logits for the Categorical distribution it models.
B. ANP M ODEL FOR JTNP
This section presents a detailed description of the JTNP architecture used in the experiments.
The
JTNP consists of a single ANP, which consists of a latent encoder, a deterministic encoder, and a decoder.
Then JTNP produces target output distribution by conditioning on the context set C
= { ( x i , y 1: T i ) } i ∈I ( C ) where y 1: T i =
( y 1 i , · · · , y Ti ) .
Latent Encoder
The latent encoder samples a global latent z .
For each context example ( x i , y 1: T i ) ∈ C , we ﬁrst project it to a hidden representation s 1: T i =
ψ s ( x i , y 1: T i ) using a single
MLP ψ s .
Then we aggregate them to a global representation s via self-attention followed by a pooling operation.
[Equation]
Then the global latent is sampled via two MLPs.
[Equation]
Deterministic Encoder
The deterministic encoder produces local representation r i for each i ∈ D .
We ﬁrst project each context example ( x i , y 1: T i )
∈
C t to a hidden representation d 1: T i
[Equation]
x
[Equation] [Equation]
i ) that serves as value embedding in cross-attention.
Then by using the context and target input x i as key and query embeddings, we apply a cross-attention along the example axis (per-task).
[Equation]
Decoders Finally, the decoder produces predictive distribution for each joint target output y 1: T i .
We ﬁrst project the input to w i = ψ w ( x i ) , then concatenate it with the global latent variable z and deterministic representation r 1: T i .
To compute the output distribution, we ﬁrst apply two MLPs on the triple ( w i , z, r 1: T i ) .
[Equation]
[Equation] [Equation]
Then for each dimension, we construct the predictive distributions as Normal or Categorical, depending on the corresponding task type.
[Equation]
58) where ( µ i ) Y t (or ( σ 2 i )
Y t ) ) denotes the projection of µ i (or σ 2 i ) into the task-speciﬁc output space Y t , by indexing the corresponding dimension from µ i .
For example, if all tasks are one-dimensional, then this corresponds to selecting t -th coordinate of µ i .
B. ANP M ODEL FOR STNP
This section presents a detailed description of the STNP architecture used in the experiments.
The
STNP consists of T independent ANPs, which consists of T latent encoders, T deterministic encoders, and T decoders.
Then STNP produces target output distribution by conditioning on the context set C
= (cid:83) t ∈T C t where C t = { ( x i , y ti ) } t ∈T .
In the following, we deonte a stacked multi-head attention block (Parmar et al., 2018) by Attn ( Q, K, V ) and a MLP by ψ ( x ) , as in Section 3.3.
Latent Encoders
The latent encoders sample per-task latents v 1: T = ( v 1 , · · · , v T ) .
For each context example ( x i , y ti )
∈
C t , we ﬁrst project it to a hidden representation s ti = ψ ts ( x i , y ti ) using a MLP ψ ts speciﬁc to task f t .
Then we aggregate them to a task-speciﬁc representation s t via self-attention followed by a pooling operation.
[Equation]
Note that each attention is applied along the example axis (per-task) and independent to each task.
Then the per-task latent variables are sampled independently, via MLPs.
[Equation]
Deterministic Encoders
The deterministic encoders produce local representation r ti for each i ∈ D and t ∈ T .
We ﬁrst project each context example ( x i , y ti )
∈ C t to a hidden representation d ti = ψ td
( x i , y ti ) that serves as value embedding in cross-attention.
Then by using the context and target input x i as key and query embeddings, we apply T independent cross-attention along the example axis (per-task).
[Equation]
Decoders Finally, the decoders produce predictive distributions for each target output y ti .
We ﬁrst project the input to w ti = ψ tw ( x i ) , then concatenate it with the corresponding latent variable v t and deterministic representation r ti .
The output distribution is computed similar to MTNP described in
Section 3.3.
[Equation]
We use the hidden dimension d = 128 for all models in synthetic and CelebA experiments and d = 64 and in weather experiments.
B. A RCHITECTURAL H YPER -P ARAMETERS
Table 5 summarizes the number of layers for each module in three models (STNP, JTNP, MTNP) used in the experiments.
We use the same number of layers in all experiments.
[Table]
, JTNP, and MTNP used in the experiments.
Table 6 summarizes the hidden dimension dim hidden of all models used in each experiment.
In our implementation, all layers (including the positional embedding) except the input and output layers have the dimension dim hidden .
[Table]
B. T RAINING D ETAILS
For all three models, we schedule learning rate lr by
[Equation]
where n_iters is the number of total iterations and base_lr is the base learning rate.
We also introduce beta coefﬁcient on the ELBO objective following Higgins et al.
(2017), which is multiplied by each
KL term.
The beta coefﬁcient is scheduled to be linearly increased from 0 to 1 during the ﬁrstiters, then ﬁxed to 1 .
We summarize the training hyper-parameters of models used in the experiments in Table 7.
[Table]
B. P ARAMETER S HARING IN MTNP
The overall description of our neural network model for MTNP is provided in Section 3.3.
We use different parameter sharing techniques in the datasets, depending on whether the tasks are homogeneous or not.
In synthetic and weather tasks, all output values are one-dimensional.
Thus we tie the parameters of the per-task paths in encoder and decoder, which makes more efﬁcient parametrization compared to per-task encoders and decoders.
In visual tasks, however, the tasks have different output dimensionalities.
Thus in this case, we separate the parameters of all per-task paths.
As task identity is implicitly encoded by the separation of task-speciﬁc paths, we do not employ task embeddings
B. O THER B ASELINES
We include two Multi-Output Gaussian Process (MOGP) baselines, MOSM (Parra & Tobar, 2017) and CSM (Ulrich et al., 2015).
To make use of training set of tasks, we consider pretraining MOGPs with respect to the kernel parameters using the same meta-training dataset with MTNP, and transfer the learned kernel parameters as prior in meta-testing.
This allows both MOGPs and MTNPs to be trained and evaluated under the same setting.
To prevent overﬁtting, we early-stopped the pretraining based on NLL.
We observe that such pretraining is effective in synthetic tasks but not in weather tasks, thus we report the pretrained version for results on synthetic tasks and non-pretrained version for results on weather tasks.
We also include two gradient-based meta-learning baselines, MAML (Finn et al., 2017) and Reptile (Nichol et al., 2018) that use the same meta-train/meta-test data with our method.
We chose these models as they are model-agnostic meta-learning methods that can be applied to our multi-task regression setting with incomplete data.
Applied to our problem, the meta-training involves bi-level optimization where the inner loop optimizes the loss for context data and the outer loop optimizes the loss for target data.
We employ a similar architecture to MTNP for the baselines that consists of a 4-layer MLP encoder network shared by all tasks and task-speciﬁc 4-layer MLP decoder networks.
For fair comparisons, we controlled the total number of parameters of the models similar to NP baselines (STNP, JTNP, MTNP).
C E XPERIMENTAL D ETAILS OF 1D S YNTHETIC F UNCTION R EGRESSION
In this section, we describe details of the data generating process and experimental settings of 1D function regression on synthetic tasks.
C. S YNTHETIC D ATASET
As discussed in the paper, we simulate synthetic tasks which are correlated by a set of parameters a, b, c, w ∈ R as follow:
[Equation]
where Sine, Tanh, Sigmoid are sine, hyperbolic tangent, logistic sigmoid function, respectively, and
Gaussian ( x ) is deﬁned as exp( − x 2 ) .
Rather than sharing the exactly same parameters a, b, c, w across tasks, we add a task-speciﬁc noise to each parameter, to control the amount of correlation across tasks as follow:
[Equation]
Thus in fact the input-output pairs of each task is generated as follow:
[Equation]
We split the 1,000 functions into 800 training, 100 validation, and 100 test sets of four correlated tasks.
Then we construct a training dataset, a validataion dataset, and a test dataset using the corresponding set of generated tasks.
For each training and validation data, we sample 200 input points uniformly within the interval [ − 5 , 5] , and applied the corresponding tasks to generate multi-task output values.
For each test data, we choose 1000 input points in the uniform grid of the interval [ − 5 , 5] , and generate the multi-task output values similarly.
Finally, simulating the incomplete data is achieved by randomly dropping each output value y ti with probability γ ∈ [0 , 1] .
C. E VALUATION P ROTOCOL
For evaluation, we average the normalized MSE MSE = 1 n (cid:80) ni =
( y ti − ˆ y ti ) 2 /a 2 on test dataset.
For prediction ˆ Y 1:4 , we approximate the predictive posterior mean with Monte Carlo sampling.
For example in MTNP,
[Equation]
We use N = M = 5 , resulting total 25 samples.
For STNP (or JTNP), we sample each latent v t (or z ) 5 times, since there is no hierarchy.
Since all the output distributions are Gaussian, the posterior predictive mean can be computed by averaging the means of each sample distribution p
( y ti | x i , v tk,l ) .
To plot the predictions in Figure 3 (a), we use the posterior means for both z and v t (which corresponds to the Maximum A Posteriori estimation) and plot the mean and variance of resulting p ( y ti | x i , v t ) .
D A DDITIONAL R ESULTS ON S YNTHETIC 1D F UNCTION R EGRESSION D. A DDITIONAL R ESULTS ON C OMPLETE AND P ARTIALLY I NCOMPLETE D ATA
In this section
[Figure]
we provide additional results on the synthetic experiment, with various missing rates γ and also with standard deviation from 5 different random seeds.
When the data is incomplete and missing some task labels (i
[Figure]
e., γ = 0 .
75 ), we can see that MTNP clearly outperforms the baselines in almost all cases.
When the complete data ( γ = 0 ) is given, MTNP still outperforms almost all baselines while achieves at least competitive performance to JTNP.
Figure 6 and 7 shows that MTNP is the most robust against both context size and quality (incompleteness)
[Figure]
[Figure]
[Table]
[Table]
[Table]
[Table] S T N P M T
N P
J T N
P
[Figure]
O
S M
C S M
R e p t
i l e
M A
M
[Figure] S T N P M T
N P
S + J T N P [Figure]
M O
S M
C S M
R e p t
i l e
M A
M L
S T N
P
M T
N P
S + J T N P [Figure]
M O
S M
C S M
R e p t
i l e
M A
M L
S T N
P
M T
N P
S +
J T N
P
[Figure]
M O
S M
C S M
R e p t
i l e
M A
M L
[Figure]
D. A DDITIONAL R ESULTS ON T OTALLY I NCOMPLETE D ATA
In this section, we explore the totally incomplete setting where no two task output values are observed at the same input point ( I ( C t ) ∩ I ( C t (cid:48) )
= ∅ , ∀ t (cid:54) =
t (cid:48) ), to further validate the practical effectiveness of our method.
To generate the totally incomplete dataset, we randomly sample context input points for each task independently, then compute the corresponding output points according to the Eq.and Eq. 66.
Note that in this case we do not drop any output points, so that the context size is m = |I
( C ) | = (cid:80) t ∈T |I
( C t ) | .
We evaluate the baselines and ours in two different scenarios: (1) training on partially incomplete or complete dataset then testing on totally incomplete dataset and (2) both training and testing on totally incomplete dataset.
We observe that in both scenarios, MTNP outperforms the baselines.
[Table] [Table]
E A DDITIONAL R ESULTS ON W EATHER T IME -S ERIES R EGRESSION
In this section, we provide additional results on the time-series regression experiment on weather data, with various missing rates γ and also with standard deviation from 5 different random seeds.
In this section, we provide additional results on the synthetic experiment, with various missing rates γ and also with standard deviation from 5 different random seeds.
When the data is highly incomplete (i.e., γ = 0 . 5 , 0 . 75 ), we can see that MTNP clearly outperforms the baselines in general.
When the complete or less incomplete data ( γ = 0 , 0 . 25 ) is given, MTNP still outperforms the gradient-based meta-learning baselines and MOSM while achieves at least competitive performance to CSM, STNP and JTNP.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
F E XPERIMENTAL D ETAILS OF I MAGE 2D F UNCTION R EGRESSION
In this section, we describe details of the data generating process and experimental settings of the image function regression experiment.
F. D ATASET
We use 30,000 RGB images from CelebA HQ dataset (Liu et al., 2015) for RGB task and the corresponding semantic segmentation masks among 19 semantic classes from CelebA Mask-HQ dataset (Lee et al., 2020) for Segment task.
For Edge task, we apply the Sobel ﬁlter (Kanopoulos et al., 1988) on the RGB images to generate continuous-valued edges.
This corresponds to the
Canny edge (Canny, 1986) without non-maximum suppression, which is also used in Zamir et.
al.
(2018) (Zamir et al., 2018).
For PNCC task, we apply a pretrained 3D face reconstruction model on the RGB images to generate PNCC label maps (Guo et al., 2020a).
At each pixel, the PNCC label consists of the ( x, y, z ) coordinate of the facial keypoint located at the pixel.
In summary, labels for RGB are 3-dimensional, Edge are 1-dimensional, Segment are 19-dimensional, and PNCC are 3-dimensional vectors.
We split the 30,000 images into 27,000 train, 1,500 valid, and 1,500 test images.
F. E VALUATION P ROTOCOL
To evaluate the accuracy of the multi-task prediction, we average MSE MSE = 1 n (cid:80) ni
=
( y ti − ˆ y ti )on the test images for continuous tasks (RGB, Edge, PNCC), and average mean IoU on the test images for discrete task (Segment).
The predictive posterior mean is computed by Monte Carlo sampling, the same as in the 1D experiment.
For categorical outputs, we discretize the prediction with the argmax operator ˆ y it =
argmax k p ( y ti = k
| x i , v t ) .
To evaluate the consistency of predictions across tasks (coherency), we translate each RGB prediction to other task labels.
For Edge and PNCC, we use the ground-truth label generation algorithm and the pretrained model used to generate the ground-truth labels for the translation, respectively.
For
Segment, we ﬁne-tuned DeeplabV3+ (Chen et al., 2018) with ImageNet (Krizhevsky et al., 2012)
pretrained ResNet-50 (He et al., 2016) backbone.
We refer to the github repository of Yakubovskiy (2019)
(Yakubovskiy, 2020) for the DeeplabV3+ model.
After the translation, we measure MSE and 1 - mIoU for continuous and discrete tasks respectively, to evaluate the disagreement (as oppose to the coherency) between the predictions.
To examine the learned correlation across tasks by MTNP (Table 3 in the main paper), we compare the performance before and after MTNP observes a set of source data.
The source data consist of all examples labeled with the source tasks.
For example, if the target task is RGB and the source tasks are Edge and Segment, we give Edge and Segment labels for all pixels, while no RGB or PNCC labels are given.
Since MTNP requires at least one labeled example for each task, we give a single completely labeled example which is chosen randomly to MTNP as a base context, before MTNP observes the source data.
There are total (cid:0) 41 (cid:1) + (cid:0) 42 (cid:1) + (cid:0) 43 (cid:1) =
14 different combinations of source tasks exist.
By excluding the case where target task is in the set of source tasks, total 7 different combinations of source tasks remain for each target task.
To measure the performance gain from task f 1 to f 2 , we average the performance gain of f 2 from all sets of source tasks that containing f 1 .
For example, the performance gain from Edge to RGB is computed by averaging performance gains δ Edge → RGB , δ Edge,Segment → RGB , δ Edge,PNCC → RGB , and δ Edge,Segment,PNCC → RGB , where we denote δ A → B by the performance gain from source tasks A to target task B .
G A DDITIONAL R ESULTS ON I MAGE 2D F UNCTION R EGRESSION
In this section, we provide additional results on the image regression experiment, with various missing rates γ and also with standard deviation from 5 different random seeds.
[Table]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
m =
[Figure]
m =1m =
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
STNP
MTNP m = GT S+JTNP m =1m = [Table]
[Figure]
STNP
MTNP m =
GT
S+JTNP m =1m =
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
STNP
MTNP m =
GT
S+JTNP m =1m =
H A BLATION S TUDY
In this section, we provide results of ablation study on the implementations of MTNP.
We conduct experiments on synthetic and weather datasets to analyze the effect of various design choices we made for MTNP.
H. A BLATION S TUDY ON S ELF -A TTENTION LAYERS AND P ARAMETRIC P OOLING
In this study, we explore the effect of using self-attention layers before pooling operations and using
PMA layer for pooling in MTNP.
The variants of MTNP are as follows.
(1) MTNP-A: MTNP without self-attention and using average pooling, (2) MTNP-P: MTNP without self-attention and using PMA, (3) MTNP-SA: MTNP with self-attention and using average pooling, (4) MTNP-SP: MTNP with self-attention and using PMA.
The results are summarized below.
Note that we consistently use MTNP-SP architecture for the experiments in Section 5.
As shown in the result, MTNP with selfattention outperforms the one without self-attention by a large margin, implying that self-attention is critical in MTNP.
[Table] [Table]
[Figure]
To investigate whether the self-attention modules operate as desired in per-task attention stage
[Figure]
we visualize the attention weights assigned to each context point.
For each task, we compute an averaged attention weight placed on a set of context points by averaging dimensions of the attention map of shape ( nL, nH, nQ, nK ) to (1 , 1 , 1 , nK ) where nL and nH denote the number of self-attention layers and heads in each layer, and nQ = nK refer to the number of query ( Q ) and key ( K ) vectors and Q = K .
Note that this averaged attention weight can be interpreted as the averaged importance put on each of the context points by the self-attention module.
Results are visualized in Figure 17.
As shown in the ﬁgure, points located at representative positions or at sparse regions are attended more than others.
This shows that self-attention in per-task paths operates as desired.
H. E FFECT OF P ARAMETER -S HARING IN P ER -T ASK B RANCHES
In this study, we explore the effect of parameter-sharing the per-task encoder networks and decoder networks in STNP and MTNP.
The variants of STNP and MTNP are as follows.
(1) STNP-S: STNP with shared encoder and decoder, (2) STNP-TS: STNP with task-speciﬁc encoders and decoders, (3) MTNP-S: MTNP with shared encoder and decoder in per-task branches, (4) MTNP-TS: MTNP with task-speciﬁc encoders and decoders in per-task branches.
Note that we consistently use STNP-TS and MTNP-S architectures for the 1D experiments (synthetic and weather) and STNP-TS and MTNP-TS architectures for the 2D experiments (CelebA) in Section
As can be seen in the tables, in the weather dataset, we observe that the models with parameter sharing (STNP-S & MTNP-S) show comparable performances to their respective non-sharing baselines (STNP-TS & MTNP-TS).
On the other hand, the parameter sharing technique slightly improves the performances of the models in the 1-D synthetic case.
We conjecture that the utilization of the same architecture and its parameter for all tasks acts as a good inductive bias to the models considering that all tasks share the same global parameter a,b,c,w. Nonetheless, we still ﬁnd that MTNPs consistently outperform STNPs regardless of whether the parameters are shared or not, validating the effectiveness of MTNP to capture and exploit functional correlation for multi-task learning problems.
[Table] [Table]
H. E FFECT OF L ATENT AND D ETERMINISTIC E NCODERS
In this study [Table] we compare STNP, JTNP, and MTNP without using deterministic encoder, each corresponds to STNP-L, JTNP-L
[Table]
and MTNP-L in the table.
Note that these variants correspond to the direct NP implementations of STNP/JTNP/MTNP rather than ANP.
The results are provided in the tables below.
The overall trends are the same as the models with deterministic encoder, which demonstrates that the effectiveness of MTNP does not depend on a speciﬁc choice of architecture (vanilla NP or ANP).
[Table]
[Table]
We also compare MTNP with deterministic encoder only, which corresponds to MTNP-D in the table below.
To emphasize the beneﬁts of generative modeling of MTNPs, we include MTNP evaluated on the best sample among 25 predictive samples, which corresponds to MTNP-best in the table.
We observe that MTNP and MTNP-D are comparable in synthetic and weather datasets, which seems reasonable as we designed the deterministic encoder to mimic the latent encoder of MTNP ( e.g. , they employ both per-task and across-task inferences).
However, we can see that MTNP-best clearly outperforms MTNP-D, which implies that MTNP can generate more accurate samples while MTNP-D cannot.
[Table]
[Table]
H. E FFECT OF T ASK E MBEDDINGS
In this study
[Table]
we compare to different types of task embeddings for MTNP.
MTNP-Onehot uses one-hot encoded vector for task embedding e t while MTNP-learnable uses learnable vector for the task embedding.
Note that we consistently use MTNP-learnable for the 1D experiments in Section
[Table] [Table]
We observe that MTNP with one-hot embedding is comparable to MTNP with learnable embedding.
To further investigate the effect of learnable embedding
[Table] we visualize the learned task embedding by MTNP using the t-SNE algorithm.
We include the visualization results in Figure 18 and 19.
As shown in the ﬁgure [Table] we ﬁnd that the learned task embeddings are well-separated from each other and uniformly distributed on the embedding space.
From the observations, we conjecture that well-separated task embeddings are sufﬁcient task information for MTNP.
[Figure] [Figure]