M ETA - LEARNING USING PRIVILEGED INFORMATION FOR DYNAMICS
A BSTRACT
Neural ODE Processes approach the problem of meta-learning for dynamics using a latent variable model, which permits a ﬂexible aggregation of contextual information.
This ﬂexibility is inherited from the Neural Process framework and allows the model to aggregate sets of context observations of arbitrary size into a ﬁxedlength representation.
In the physical sciences, we often have access to structured knowledge in addition to raw observations of a system, such as the value of a conserved quantity or a description of an understood component.
Taking advantage of the aggregation ﬂexibility, we extend the Neural ODE Process model to use additional information within the Learning Using Privileged Information setting, and we validate our extension with experiments showing improved accuracy and calibration on simulated dynamics tasks
I NTRODUCTION & BACKGROUND
Learning using privileged information (LUPI) is a machine learning paradigm where we have access to additional information during training that may not be available at test time (Vapnik & Vashist, 2009; Vapnik & Izmailov, 2015).
Typically this information is higher quality in some way, often it conveys some expert understanding we have about the system being modelled.
To paraphrase an example of Vapnik & Vashist (2009), we might have access to extensive patient records associated with biopsy scans in a training set but wish to deploy our trained model ‘on-the-front-line’ where records are incomplete or unavailable.
We would like to improve the model by using this information without coming to rely on it; to provide an accurate prognosis without human help.
An analogy can be made to the role of a teacher.
Besides providing corrections, skilled teachers accelerate the understanding of their students through explanations and insights.
This is important in the context of learning about physics, where a new perspective often provides traction with difﬁcult problems.
In this work we propose a method for conveying such insight in the case of modelling dynamics.
Our main contribution is a new training-mode architecture that allows privileged information to guide learning, that results in more accurate predictions and better calibrated uncertainty estimation.
Neural Processes.
Deep neural networks are excellent function approximators that are cheap to evaluate and straightforward to train, but typically only provide point estimates and require retraining to make use of information gained at test time.
However, meta-learning (occasionally, ‘learning-tolearn’ ) with neural networks is enjoying a resurgence in popularity as an answer to the serialised learning question, and has been applied in a range of domains (Hospedales et al., 2020).
Gaussian
Processes have markedly different advantages, handling uncertainty in a principled way and adapting to new data at test time (Rasmussen, 2003), at the cost of computational expense.
Neural Processes (NP) aim to combine the best of both by learning to model a distribution over functions, framing the meta-learning problem as a latent variable model (Garnelo et al., 2018a;b).
Neural ODEs.
Dynamical systems are a fundamental object of study in physics and are often most elegantly described using ordinary differential equations (ODE).
Neural ODEs (NODE) (Chen et al., 2018) combine the representation learning capabilities of neural networks with an ODE structure to allow ODEs to be learned directly from observational data.
Follow-up works have made improvements to expressivity (Dupont et al., 2019), investigated time series modelling (Kidger et al., 2020;
Neural ODE Processes.
An alternative to the work of C¸a˘gatay
Yıldız et al.
for modelling uncertainty in dynamics is the Neural ODE Process (NDP) which uses a neural process derived formulation to learn a distribution over dynamics (Norcliffe et al., 2021).
These models are able to fast-adapt to new data points at test time, unlike vanilla-NODEs, whilst inheriting superior timeseries modelling as compared to vanilla-NPs.
The key components of the NDP are an observation encoder, representation aggregator, latent ODE, and decoder.
Learning using privileged information.
The LUPI framework, introduced by Vapnik & Vashist (2009) and expanded by Vapnik & Izmailov (2015), formalises a learning setup in which a teacher is able to provide the student learner with structured explanations, comments, comparisons, etc. beyond direct supervision.
Hern´andez-Lobato et al.
(2014) show that privileged information can be used effectively by GP classiﬁers, whilst Lambert et al.
(2018) apply LUPI to deep neural networks by setting the dropout rate to be a function of the privileged information.
L EARNING USING PRIVILEGED INFORMATION WITH NDP S
Problem statement.
Formally, our setting closely matches that of the NDP: we consider modelling random functions over time, F : T → Y , where F has distribution D induced by a second distribution, D (cid:48) , over some underlying dynamics 1 .
We are provided with a set of labelled samples from an instantiation F of F referred to as the context set, indexed by I C and denoted C =
{ ( t C i , y C i ) } i ∈ I C .
The task is to predict the values { y T j } j ∈ I T taken by F at a set of target times { t T j } j ∈ I T , indexed by
I T , which together form the target set, T = { ( t T j , y T j ) } j ∈ I T .
To ensure the model is able to learn the underlying distribution over dynamics, and how this manifests as a distribution over functions, training assumes access to a set of time-series sampled from F .
Diverging from the NDP setting, during training the model also has access to privileged information relating to each instantiation F of F , π F .
The privileged information could be some physical property or conserved quantity of the system, such as the spring stiffness, as in Figure 2.
At test time no privileged information is provided and there is no difference from the vanilla-NDP setting.
Model overview.
The differences introduced by the LUPI framework affect the training procedure, introducing a second source of information with which to form the global latent variable z .
To incorporate the privileged information within the LUPI-NDP we introduce a second encoder to produce a representation r π = f πe ( π ) , where f πe is parametrised as a fully-connected neural network.
During training, an additional aggregation step is introduced to combine the observations derived representation, r o , with that of the privileged information to form a global representation, r train =
g ( r o , r π ) .
Practically, as the test-time global representation is that formed using only the observations, i.e. r test =
r o , we choose to parametrise g as a residual network (He et al., 2016), that is, g ( r o , r π ) =
r o + g (cid:48) ( r o , r π ) , such that the privileged information is explicitly used as a correction term.
At test time, the model we propose is equivalent to a vanilla-NDP.
Learning and Inference.
The true posterior is intractable and, as is the NP custom, the model is trained using amortised variational inference.
We select an objective that reﬂects the intended test
Typically, in time behaviour of making predictions based solely on the context set, given by
[Equation]
with variational posterior q (a derivation is provided in Appendix A).
Note that during training C is a subset of T. The original NDP model is recovered by setting π to null (dropping it).
E XPERIMENTS
We compare our proposed LUPI-NDP model with a vanilla-NDP that does not make use of the privileged information and, as there are no architectural changes, at test time the models are distinguishable only by the value of their weights.
As explained in Section 2, both models are NDPs, so we refer to them as LUPI and NoPI (no-privileged-information) for clarity.
Full model and training details are provided in Appendix C.
[Figure]
time, t 0. P o p u l a t i o n s LV trajectories uv
Metrics.
Besides providing high quality predictions, NDPs, and
NPs generally, are interesting because the way they approach the meta-learning problem (as a latent variable model) produces uncertainty estimates.
We measure the quality of these estimates by the calibration error and sharpness.
Calibration measures the degree to which uncertainty estimates are commensurate with residuals: if the model estimates an outcome to happen one time in every ten, does it actually occur that frequently?
Sharpness is simply how low the uncertainty estimates are: between equally well-calibrated models we should favour the sharper model as being more informative.
Practically speaking, we value accuracy over calibration over sharpness.
We follow the deﬁnitions of Kuleshov et al.
(2018) for the calibration error and sharpness, detailed in Appendix B.
Damped coupled oscillators.
We ﬁrst consider modelling a system of two masses attached by identical springs in series between parallel walls, described by the second-order differential equations
[Equation]
with spring and damping constants, k and c , respectively.
A distribution over dynamics is induced by sampling some parameters of the system, and this parameter forms the privileged information.
Examples of the trajectories induced by sampling over the drag coefﬁcient as c ∼ U (0 . 5 , 2) are presented in Figure 3.
[Figure]
LUPI
[Figure]
Predictions
[Figure]
an oscillators task example
[Figure]
[Figure]
[Table]
Lotka-Volterra.
We investigate modelling a second two dimensional system, the Lotka-Volterra equations (L-V).
The populations of ‘predator’ and ‘prey’ species, v and u , are governed by
[Equation]
In these examples we use the values α = 2 / 3 , β = 4 / [Table] γ = [Table]
δ = 1 . To produce a range of dynamics we uniformly sample different initial populations u 0 ∼ U (0 . 2 , 1) , v 0 ∼ U (0 . 1 , 0 . 5) as shown in Figure 3.
For these equations V = δu − γ ln( u ) + βv − α ln( v ) is a conserved quantity, and is provided as privileged information.
D ISCUSSION
Training in the LUPI setting produces signiﬁcantly more accurate and better calibrated models in each task.
Table 1 presents numerical results, Figure 4 provides more detail for the varying stiffness task, and further plots are provided in the Appendix.
The models are mostly less sharp ( 5 / 6 ) but sharpness is a secondary measure to calibration, especially when the sharper model is less accurate, as is the case here.
We would also highlight the subtle difference in the kinds of information being incorporated by the models—stiffness and damping are independent variables, whereas V is a conserved quantity that arises from the dynamical system directly—and suggest this is a promising result for the wider applicability of the model.
Future work could explore the effects LUPI has on generalisation, recovering estimates of the privileged information at test-time, and, as the model can be applied as-is to any Neural Process, tasks other than dynamics. A O BJECTIVE DERIVATION At test time we want to (accurately) predict the targets y j ∈
I T at known times t j ∈
I T given the context set C, which means maximising p ( y j ∈ I T | t j ∈ I T , C ) during training. Knowing that we want to end up with something similar to the objectives used by Garnelo et al. (2018b) and Norcliffe et al.
(2021), i.e. something like
[Equation]
we start with the marginal
[Equation]
Noting p ( y j ∈ I T | t j ∈ I T , z , C )
=
p ( y j ∈ I T | t j ∈ I T , z ) and p ( z | t j ∈ I T , C ) =
p ( z | C ) , we multiply by q ( z | T , π ) /q
( z | T , π ) = 1 to get
[Equation]
y
[Equation]
T
[Equation]
j
[Equation]
As usual, p ( z | C ) is intractable, and we approximate it with q (cid:48)
( z | C ) =
q ( z | C , π ← null ) .
Finally, applying Jensen’s inequality produces our objective
[Equation]
This objective is similar to the evidence lower-bound but better reﬂects the intended test time model behaviour.
B C ALIBRATION & SHARPNESS FOR REGRESSION
These descriptions and deﬁnitions borrow heavily from Kuleshov et al.
and are included for completeness.
B. C ALIBRATION
Classiﬁcation.
In the classiﬁcation setting, we say a forecaster (making a large number of predictions) is calibrated if events that are assigned some probability occur about that frequently.
Formally, a forecaster H is calibrated if
[Equation]
that is, in the long run ( T → ∞ ) predictions assigned probability p (H ( x t )
= p ) occur ( y t = 1 ) with frequency p .
Regression.
In the regression setting, the forecaster outputs a cumulative distribution function F t targeting y t .
Calibration here means that y t should fall within a 90% conﬁdence interval approximately 90% of the time.
We can formalise this using the quantile function, deﬁned as returning the threshold value that random draws from F t would fall below p of the time.
For quantile function
Q t ( p ) =
inf { y : p ≤ F t ( y ) } , we deﬁne calibration to mean [Equation] { y t [Equation]
p ) }
T
[Equation]
p
[Equation]
p
[Equation] [Equation] 1] as T → ∞ . That is, in the long-run ( T → ∞ ) targets fall below the quantile function at p ( y t ≤
To produce a calibration score, we consider how far the empirical calibration deviates from a perfectly calibrated model.
This is measured by computing the empirical frequency at a set of conﬁdence levels 0 ≤ p 1 < p 2 < ... <
p m ≤ 1 as
[Equation]
) i.e. how often do the targets fall at a conﬁdence level that is less than the threshold p j , and computing the score
[Equation]
(which is the mean-squared-error of the cumulative histogram of conﬁdence over empirical frequency from the identity.)
B. S HARPNESS
A sharp forecast has conﬁdence intervals that are tightly bound, and the sharpness score can be more straightforwardly deﬁned as the mean variance of the cumulative distribution F t ,
[Equation]
C A RCHITECTURAL AND TRAINING DETAILS
The context/targets distinction is related to training and evaluation and is separate from the architecture of the model.
As such, for this explanation we refer to the set of observations as
X = { ( t X i , y X i ) } i ∈ I X indexed by I X , for which we can substitute C or T as appropriate.
We conceive of the model as consisting of
an observation encoder mapping observations to representations r i = f obs ( t i , y i ) 2.
an aggregator that combines observation representations into a ﬁxed length representation r X = ⊕ i ∈
I X ( r i ) 3.
a privileged information encoder mapping the privileged information to a representation r π = f π ( π ) 4.
a second aggregator that combines r X and r π , r = g ( r X , r π ) , where r and r X have the same dimensionality 5.
a pair of functions, µ ( r ) , σ ( r ) , that parametrise the global latent variable z as a function of either r or r X , z ∼ N ( µ ( r ) , σ ( r )) 6.
a function to initialise the latent ODE state from a sample from the global latent variable, L (0) =
f init ( z (cid:48) ) 7.
the neural ODE derivative as a function of time, the instantaneous latent state and the global latent sample, dL dt = f ODE ( L ( t ) , z (cid:48) , t ) 8. and a decoder that maps from the latent ODE state back to observation space and also depends on the sample from the global latent, ˆ y ( t ) =
f dec ( L ( t ) , z (cid:48) )
as is the fashion, we chose to use neural networks for every parametric function, that is all but the ﬁrst aggregation ⊕ .
Now we know what the parts are for, the architecture we used is
f obs : a three-layer MLP with ReLU activations on the hidden layers (not the output) and hidden dimension2.
⊕ : the mean concatenated with the LogSumExp (a smooth approximation of max) 3.
f π : a three-layer MLP with ReLU activations on the hidden layers (not the output) and hidden dimension
g : a ResNet with an input-to-output skip connection for r X and a residual connection consisting of a three-layer MLP with ReLU activations on the hidden layers (not the output) and hidden dimension5.
µ, σ : two three-layer MLPs with ReLU activations on the hidden layers (not the output) and hidden dimension 16, with shared weights in the ﬁrst two hidden layers 6.
f init : a three-layer MLP with ReLU activations on the hidden layers (not the output) and hidden dimension7.
f ODE : a three-layer MLP with softplus ( x ) = log (1 + exp( x )) activations on the hidden layers (not the output) and hidden dimension8.
f dec : a three-layer MLP with ReLU activations on the hidden layers (not the output) and hidden dimension 16.
We did not undertake any extensive hyperparameter tuning though we can report that neither model is able to learn if the hidden dimension is set to be small ( < 4 ).
We follow the best practices established by Le et al.
(2018) for training NPs, and during training use a learned uncertainty estimate rather than resampling.
This estimate is produced as an additional output from the decoder.
We use Adam with a learning rate of 0 .
001 = 10 − 3 and otherwise default PyTorch settings, and train for 100 epochs with a training set of 500 examples split 80/20 between train/validation.
We do not use early stopping as we found that the models are stable at convergence i.e. they do not diverge when “overtrained”.
The test sets consist of a further 500 examples (of 100 time steps each) over which the reported evaluation metrics were calculated.
A full model with data generators and Google colab notebook will be released after review.
[Figure]
Predictions on an LV example
[Figure]
D L OTKA -V OLTERRA PLOTS
Figure 5 shows additional details for the Lotka-Volterra experiment.