C ONVOLUTIONAL C ONDITIONAL N EURAL P ROCESSES
wpb23@
Turner
Microsoft Research ret26@cam.ac.uk
A BSTRACT
We introduce the Convolutional Conditional Neural Process (C ONV CNP), a new member of the Neural Process family that models translation equivariance in the data.
Translation equivariance is an important inductive bias for many learning problems including time series modelling, spatial data, and images.
The model embeds data sets into an inﬁnite-dimensional function space as opposed to a ﬁnitedimensional vector space.
To formalize this notion, we extend the theory of neural representations of sets to include functional representations , and demonstrate that any translation-equivariant embedding can be represented using a convolutional deep set .
We evaluate C ONV CNPs in several settings, demonstrating that they achieve state-of-the-art performance compared to existing NPs.
We demonstrate that building in translation equivariance enables zero-shot generalization to challenging, out-of-domain tasks.
I NTRODUCTION
Neural Processes (NPs; Garnelo et al., 2018b;a) are a rich class of models that deﬁne a conditional distribution p ( y | x , Z, θ ) over output variables y given input variables x , parameters θ , and a set of observed data points in a context set Z
= { x m , y m } Mm =1 .
A key component of NPs is the embedding of context sets Z into a representation space through an encoder Z (cid:55)→ E ( Z ) , which is achieved using a D EEP S ETS function approximator (Zaheer et al., 2017).
This simple model speciﬁcation allows
NPs to be used for (i) meta-learning (Thrun & Pratt, 2012; Schmidhuber, 1987), since predictions can be generated on the ﬂy from new context sets at test time; and (ii) multi-task or transfer learning (Requeima et al., 2019), since they provide a natural way of sharing information between data sets.
Moreover, conditional NPs (CNPs; Garnelo et al., 2018a), a deterministic variant of NPs, can be trained in a particularly simple way with maximum likelihood learning of the parameters θ , which mimics how the system is used at test time, leading to strong performance (Gordon et al., 2019).
Natural application areas of NPs include time series, spatial data, and images with missing values.
Consequently, such domains have been used extensively to benchmark current NPs (Garnelo et al., 2018a;b; Kim et al., 2019).
Often, ideal solutions to prediction problems in such domains should be translation equivariant: if the data are translated in time or space, then the predictions should be translated correspondingly (Kondor & Trivedi, 2018; Cohen & Welling, 2016).
This relates to the notion of stationarity.
As such, NPs would ideally have translation equivariance built directly into the modelling assumptions as an inductive bias.
Unfortunately, current NP models must learn this structure from the data set instead, which is sample and parameter inefﬁcient as well as impacting the ability of the models to generalize.
The goal of this paper is to build translation equivariance into NPs.
Famously, convolutional neural networks (CNNs) added translation equivariance to standard multilayer perceptrons (LeCun et al., 1998; Cohen & Welling, 2016).
However, it is not straightforward to generalize NPs in an analogous way: (i) CNNs require data to live “on the grid” (e.g. image pixels form a regularly spaced grid), while many of the above domains have data that live “off the grid” (e.g. time series data may be observed irregularly at any time t ∈ R ).
(ii) NPs operate on partially observed context sets whereas
CNNs typically do not.
(iii) NPs rely on embedding sets into a ﬁnite-dimensional vector space for which the notion of equivariance with respect to input translations is not natural, as we detail in
Section 3.
In this work, we introduce the C ONV CNP, a new member of the NP family that accounts for translation equivariance.
This is achieved by extending the theory of learning on sets to include functional representations , which in turn can be used to express any translation-equivariant NP model.
Our key contributions can be summarized as follows.
(i)
We provide a representation theorem for translation-equivariant functions on sets, extending a key result of Zaheer et al.
(2017) to functional embeddings , including sets of varying size.
(ii)
We extend the NP family of models to include translation equivariance.
(iii)
We evaluate the C ONV CNP and demonstrate that it exhibits excellent performance on several synthetic and real-world benchmarks.
B ACKGROUND AND F ORMAL P ROBLEM S TATEMENT
In this section we introduce the notation and precisely deﬁne the problem this paper addresses.
Notation.
In the following, let X
=
R d and Y ⊆ R d (cid:48)
(with Y compact) be the spaces of inputs and outputs respectively.
To ease notation, we often assume scalar outputs Y ⊆ R .
Deﬁne Z M =
( X × Y )
M as the collection of M input–output pairs, Z ≤ M = (cid:83) Mm =
Z m as the collection of at most M pairs, and Z = (cid:83) ∞ m =1 Z m as the collection of ﬁnitely many pairs.
Since we will consider permutation-invariant functions on Z (deﬁned later in Property 1), we may refer to elements of Z as sets or data sets.
Furthermore, we will use the notation [ n ] = { 1 , . . . , n } .
Conditional Neural Processes (CNPs) .
CNPs model predictive distributions as p ( y | x , Z )
= p ( y | Φ( x , Z ) , θ ) , where Φ is deﬁned as a composition ρ ◦ E of an encoder E : Z → R e mapping into the embedding space R e and a decoder ρ : R e → C b ( X , Y ) .
Here E ( Z ) ∈ R e is a vector representation of the set Z , and C b ( X , Y ) is the space of continuous, bounded functions X
→ Y endowed with the supremum norm.
While NPs (Garnelo et al., 2018b) employ latent variables to indirectly specify predictive distributions, in this work we focus on CNP models which do not.
As noted by Lee et al.
(2019); Bloem-Reddy & Teh (2019), since E is a function on sets, the form of Φ in CNPs tightly relates to the growing literature on learning and representing functions on sets (Zaheer et al., 2017; Qi et al., 2017a;
Wagstaff et al., 2019).
Central to this body of work is the notion that, because the elements of a set have no order, functions on sets are naturally permutation invariant.
Hence, to view functions on Z as functions on sets, we require such functions to be permutation invariant.
This notion is formalized in Property 1.
Property 1 ( S n -invariant and S -invariant functions) .
Let S n be the group of permutations of n symbols for n ∈ N .
A function Φ on Z n is called S n -invariant if
[Equation]
all π
[Equation]
and Z n
[Equation]
, where the application of π to Z n is deﬁned as πZ n = (( x π (1) , y π (1) ) , . . .
, ( x π ( n ) , y π ( n ) )) .
A function Φ on Z is called S -invariant if the restrictions Φ | Z n are S n -invariant for all n .
Zaheer et al.
(2017) demonstrate that any continuous S M -invariant function f : Z M → R has a sum-decomposition (Wagstaff et al., 2019), i.e. a representation of the form f ( Z )
=
ρ ( (cid:80)
z
∈ Z φ ( z )) for appropriate ρ and φ (though this could only be shown for ﬁxed-sized sets).
This is indeed the form employed by the NP family for the encoder that embeds sets into a latent representation.
Translation equivariance.
The focus of this work is on models that are translation equivariant : if the input locations of the data are translated by an amount τ , then the predictions should be translated correspondingly.
Translation equivariance for functions operating on sets is formalized in Property 2.
Property 2 (Translation equivariant mappings on sets) .
Let H be an appropriate space of functions on X , and deﬁne T and T (cid:48) as follows:
[Equation]
Then a mapping Φ: Z → H is called translation equivariant if Φ( T τ Z )
= T (cid:48)
τ
Φ( Z ) for all τ ∈ X and Z ∈ Z .
Having formalized the problem, we now describe how to construct CNPs that translation equivariant.
C ONVOLUTIONAL D EEP S ETS
We are interested in translation equivariance (Property 2) with respect to translations on X .
The
NP family encoder maps sets Z to an embedding in a vector space R d , for which the notion of equivariance with respect to input translations in X is not well deﬁned.
For example, a function f on X can be translated by τ ∈ X : f ( · − τ ) .
However, for a vector x ∈ R d , which can be seen as a function [ d ] → R , x ( i ) = x i , the translation x ( · − τ ) is not well-deﬁned.
To overcome this issue, we enrich the encoder E : Z → H to map into a function space H containing functions on X .
Since functions in H map from X , our notion of translation equivariance (Property 2) is now also well deﬁned for E ( Z ) .
As we demonstrate below, every translation-equivariant function on sets has a representation in terms of a speciﬁc functional embedding.
Deﬁnition 1 (Functional mappings on sets and functional representations of sets) .
Call a map
E : Z → H a functional mapping on sets if it maps from sets Z to an appropriate space of functions
H .
Furthermore, call E ( Z ) the functional representation of the set Z .
Considering functional representations of sets leads to the key result of this work, which can be summarized as follows.
For Z (cid:48) ⊆
Z appropriate, a continuous function Φ: Z (cid:48)
→ C b ( X , Y ) satisﬁes
Properties 1 and 2 if
and only if it has a representation of the form
[Equation]
for some continuous and translation-equivariant ρ : H → C b ( X , Y ) , and appropriate φ and ψ .
Note that ρ is a map between function spaces.
We also remark that continuity of Φ is not in the usual sense; we return to this below.
Equation (1) deﬁnes the encoder used by our proposed model, the C ONV CNP.
In Section 3.1, we present our theoretical results in more detail.
In particular, Theorem 1 establishes equivalence between any function satisfying Properties 1 and 2 and the representational form in Equation (1).
In doing so, we provide an extension of the key result of Zaheer et al.
(2017) to functional representations on sets, and show that it can naturally be extended to handle varying-size sets.
The practical implementation of C ONV CNPs — the design of ρ , φ , and ψ — is informed by our results in Section 3.1 (as well as the proofs, provided in Appendix A), and is discussed for domains of interest in Section 4.
3. R EPRESENTATIONS OF T RANSLATION E QUIVARIANT F UNCTIONS ON S ETS
In this section we establish the theoretical foundation of the C ONV CNP.
We begin by stating a deﬁnition that is used in our main result.
We denote [ m ] = { 1 , . . .
, m } .
Deﬁnition 2 (Multiplicity) .
A collection Z (cid:48) ⊆
Z is said to have multiplicity K if, for every set
Z ∈ Z (cid:48) , every x occurs at most K times:
[Equation]
[Table]
[Equation]
For example, in the case of real-world data like time series and images, we often observe only one (possibly multi-dimensional) observation per input location, which corresponds to multiplicity one.
We are now ready to state our key theorem.
Theorem 1.
Consider an appropriate 2 collection Z (cid:48)≤ M ⊆ Z ≤ M with multiplicity K .
Then a function Φ: Z (cid:48)≤ M → C b ( X , Y ) is continuous 3 , permutation invariant (Property 1), and translation
Context set Z c = ( x n ,y n ) Nn =1 y x
Functional representation h (0) =
P ψ ( · − x n ) h (1) =
P y n ψ ( · − x n )
P ψ ( · − x n ) (density channel) x
Evaluate at discretization ( t i )
Ti = x
Apply CNN and predict (cid:20) µ ( x ∗ ) σ ( x ∗ ) (cid:21)
=
T
X i =1 (cid:20) f µ ( t i ) e fσ ( ti ) (cid:21) ψ ρ ( x ∗ − t i ) x ∗ 1 x ∗x ∗x ∗p ( y ∗ 3 | x ∗ 3 ,Z c ) (a) require: ρ = ( CNN , ψ ρ ) , ψ , and density γ require: context ( x n , y n ) Nn =1 , target ( x ∗ m )
Mm = 1 begin lower, upper ← range (cid:0) ( x n ) Nn =1 ∪ ( x ∗ m )
Mm =
[Figure]
(cid:1)
[Figure]
Ti
[Figure]
← uniform_grid(lower, upper ; γ )
[Figure]
( µ
[Figure] [Figure]
m )
Mm =
[Figure]
10 end (
[Figure]
) require: ρ = CNN and E =
CONV θ require: image I , context M c , and target mask M t 1 begin //
We discretize at the pixel locations.
Z c ← M c (cid:12)
I // Extract context set.
h ← CONV
θ ([M c , Z c ] (cid:62) )
h (1: C ) ← h (1: C ) / h (0)
f t ← M t (cid:12) CNN ( h ) µ ← f (1: C ) t σ ← pos ( f ( C +1:2 C ) t ) return ( µ , σ ) 10 end (c) equivariant (Property 2)
if and only if it has a representation of the form
[Equation]
for some continuous and translation-equivariant ρ : H → C b ( X , Y ) and some continuous φ : Y → R K +1 and ψ : X → R , where H is an appropriate space of functions that includes the image of E .
We call a function Φ of the above form C ONV D EEP S ET .
The proof of Theorem 1 is provided in Appendix A. We here discuss several key points from the proof that have practical implications and provide insights for the design of C ONV CNPs: (i) For the construction of ρ and E , ψ is set to a ﬂexible positive-deﬁnite kernel associated with a Reproducing Kernel Hilbert Space (RKHS; Aronszajn (1950)), which results in desirable properties for
E .
(ii) Using the work by Zaheer et al.
(2017), we set φ ( y ) =
( y 0 , y 1 , · · · , y K ) to be the powers of y up to order K .
(iii) Theorem 1 requires ρ to be a powerful function approximator of continuous, translation-equivariant maps between functions.
In Section 4, we discuss how these theoretical results inform our implementations of C ONV CNPs.
Theorem 1 extends the result of Zaheer et al.
(2017) discussed in Section 2 by embedding the set into an inﬁnite-dimensional space — the RKHS — instead of a ﬁnite-dimensional space.
Beyond allowing the model to exhibit translation equivariance, the RKHS formalism allows us to naturally deal with ﬁnite sets of varying sizes, which turns out to be challenging with ﬁnite-dimensional embeddings.
Furthermore, our formalism requires φ ( y ) =
( y 0 , y 1 , y 2 , . . .
, y K ) to expand up to order no more than the multiplicity of the sets K ; if K is bounded, then our results hold for sets up to any arbitrarily large ﬁnite size M , while ﬁxing φ to be only ( K + 1) -dimensional.
C ONVOLUTIONAL C ONDITIONAL N EURAL P ROCESSES
In this section we discuss the architectures and implementation details for C ONV CNPs.
Similar to
NPs, C ONV CNPs model the conditional distribution as p
[Equation]
Σ
[Equation]
) with ( µ n , Σ n ) =
Φ θ ( Z )( x n ) , (2) where Z is the observed data and Φ a C ONV D EEP S ET .
The key considerations are the design of ρ , φ , and ψ for Φ .
We provide separate models for data that lie on-the-grid and data that lie off-the-grid.
Form of φ .
The applications considered in this work have a single (potentially multi-dimensional) output per input location, so the multiplicity of Z is one (i.e., K = 1 ).
It then sufﬁces to let φ be a power series of order one, which is equivalent to appending a constant to y in all data sets, i.e. φ ( y ) =
[1 y ] (cid:62) .
The ﬁrst output φ 1 thus provides the model with information regarding where data has been observed, which is necessary to distinguish between no observed datapoint at x and a datapoint at x with y = 0 .
Denoting the functional representation as h , we can think of the ﬁrst channel h (0) as a “density channel”.
We found it helpful to divide the remaining channels h
(1:) by h (0) (Figure 1b, line 5), as this improved performance when there is large variation in the density of input locations.
In the image processing literature, this is known as normalized convolution (Knutsson & Westin, 1993).
The normalization operation can be reversed by ρ and is therefore not restrictive.
C ONV CNPs for off-the-grid data.
Having speciﬁed φ , it remains to specify the form of ψ and ρ .
Our proof of Theorem 1 suggests that ψ should be a stationary, non-negative, positive-deﬁnite kernel.
The exponentiated-quadratic (EQ) kernel with a learnable length scale parameter is a natural choice.
This kernel is multiplied by φ to form the functional representation E ( Z ) (Figure 1b, line 4; and
Figure 1a, arrow 1).
Next, Theorem 1 suggests that ρ should be a continuous, translation-equivariant map between function spaces.
Kondor & Trivedi (2018) show that, in deep learning, any translation-equivariant model has a representation as a CNN.
However, CNNs operate on discrete (on-the-grid) input spaces and produce discrete outputs.
In order to approximate ρ with a CNN, we discretize the input of ρ , apply the CNN, and ﬁnally transform the CNN output back to a continuous function X → Y .
To do this, for each context and test set, we space points ( t i ) ni
=1 ⊆ X on a uniform grid (at a pre-speciﬁed density) over a hyper-cube that covers both the context and target inputs.
We then evaluate ( E ( Z )( t i ))
ni =
(Figure 1b, lines 2–3; Figure 1a, arrow 2).
This discretized representation of E ( Z ) is then passed through a CNN (Figure 1b, line 6; Figure 1a, arrow 3).
To map the output of the CNN back to a continuous function X → Y , we use the CNN outputs as weights for evenly-spaced basis functions (again employing the EQ kernel), which we denote by ψ ρ (Figure 1b, lines 7–8; Figure 1a, arrow 3).
The resulting approximation to ρ is not perfectly translation equivariant, but will be approximately so for length scales larger than the spacing of ( E ( Z )( t i ))
ni =1 .
The resulting continuous functions are then used to generate the (Gaussian) predictive mean and variance at any input.
This, in turn, can be used to evaluate the log-likelihood.
C ONV CNP for on-the-grid data.
While C ONV CNP is readily applicable to many settings where data live on a grid, in this work we focus on the image setting.
As such, the following description uses the image completion task as an example, which is often used to benchmark NPs (Garnelo et al., 2018a;
Kim et al., 2019).
Compared to the off-the-grid case, the implementation becomes simpler as we can choose the discretization ( t i )
ni
=1 to be the pixel locations.
Let I ∈
R H × W × C be an image — H, W, C denote the height, width, and number of channels, respectively — and let M c be the context mask, which is such that [M c ] i,j = 1 if pixel location ( i, j ) is in the context set, and 0 otherwise.
To implement φ , we select all context points, Z c :=
M c (cid:12) I , and prepend the context mask: φ = [M c , Z c ] (cid:62) (Figure 1c, line 4).
Next, we apply a convolution to the context mask to form the density channel: h (0) =
CONV θ (M c ) (Figure 1c, line 4).
To all other channels, we apply a normalized convolution: h (1: C ) =
CONV θ ( y ) / h (0) (Figure 1c, line 5), where the division is element-wise.
The ﬁlter of the convolution is analogous to ψ , which means that h is the functional representation, with the convolution performing the role of E (the summation in Figure 1b, line 4).
Although the theory suggests using a non-negative, positive-deﬁnite kernel, we did not ﬁnd signiﬁcant empirical differences between an
EQ kernel and using a fully trainable kernel restricted to positive values to enforce non-negativity (see Appendices D.4 and D.5 for details).
Lastly, we describe the on-the-grid version of ρ ( · ) , which consists of two stages.
First, we apply a
CNN to E ( Z ) (Figure 1c, line 6).
Second, we apply a shared, pointwise MLP that maps the output of the CNN at each pixel location in the target set to R 2 C , where we absorb MLP into the CNN ( MLP can be viewed as an 1 × 1 convolution).
The ﬁrst C outputs are the means of a Gaussian predictive distribution and the second C the standard deviations, which then pass through a positivity-enforcing function (Figure 1c, line 7–8).
To summarise, the on-the-grid algorithm is given by
[Equation]
multiplies
[Equation]
where ( µ , σ ) are the image mean and standard deviation, ρ is implemented with CNN , and E is implemented with the mask M c and convolution CONV .
Training.
Denoting the data set D = { Z n } Nn =1 ⊆ Z and the parameters by θ , maximum-likelihood training involves (Garnelo et al., 2018a;b)
[Equation]
where we have split Z n into context ( Z n,c ) and target ( Z n,t ) sets.
This is standard practice in the
NP (Garnelo et al., 2018a;b) and meta-learning settings (Finn et al., 2017; Gordon et al., 2019) and relates to neural auto-regressive models (Requeima et al., 2019).
Note that the context set and target set are disjoint ( Z n,c ∩ Z n,t = ∅ ), which differs from the protocol for the NP (Garnelo et al., 2018a).
Practically, stochastic gradient descent methods (Bottou, 2010) can be used for optimization.
E XPERIMENTS AND R ESULTS
We evaluate the performance of C ONV CNPs in both on-the-grid and off-the-grid settings focusing on two central questions: (i) Do translation-equivariant models improve performance in appropriate domains?
(ii) Can translation equivariance enable C ONV CNPs to generalize to settings outside of those encountered during training?
We use several off-the-grid data-sets which are irregularly sampled time series ( X = R ), comparing to Gaussian processes (GPs; Williams & Rasmussen (2006)) and A TTN CNP(which is identical to the ANP (Kim et al., 2019), but without the latent path in the encoder), the best performing member of the CNP family.
We then evaluate on several on-the-grid image data sets ( X = Z 2 ).
In all settings we demonstrate substantial improvements over existing neural process models.
For the CNN component of our model, we propose a small and large architecture for each experiment (in the experimental sections named C ONV CNP and
C ONV CNPXL, respectively).
We note that these architectures are different for off-the-grid and on-the-grid experiments, with full details regarding the architectures given in the appendices.
5. S YNTHETIC 1D E XPERIMENTS
First we consider synthetic regression problems.
At each iteration, a function is sampled, followed by context and target sets.
Beyond EQ-kernel GPs (as proposed in Garnelo et al.
(2018a);
Kim et al.
, we consider more complex data arising from Matern– 52 and weakly-periodic kernels, as well as a challenging, non-Gaussian sawtooth process with random shift and frequency (see Figure 2, for example).
C ONV CNP is compared to CNP (Garnelo et al., 2018a) and A TTN CNP.
Training and testing procedures are ﬁxed across all models.
Full details on models, data generation, and training procedures are provided in Appendix C.
[Table] [Table]
Table 1 reports the log-likelihood means and standard errors of the models over 1000 tasks.
The context and target points for both training and testing lie within the interval [ − 2 , 2] where training data was observed (marked “training data range” in Figure 2).
Table 1 demonstrates that, even when extrapolation is not required, C ONV CNP signiﬁcantly outperforms other models in all cases, despite having fewer parameters.
Figure 2 demonstrates that C ONV CNP generates excellent ﬁts, even for challenging functions such as from the Matern– 5 2 kernel and sawtooth.
Moreover, Figure 2 compares the performance
[Figure]
of C ONV CNP and A TTN
CNP when data is observed outside the range where the models were trained: translation equivariance enables C ONV CNP to elegantly generalize to this setting, whereas
A TTN CNP is unable to generate reasonable predictions.
5. PLA S T I CC E XPERIMENTS
The PLAsTiCC data set (Allam Jr et al., 2018) is a simulation of transients observed by the LSST telescope under realistic observational conditions.
The data set contains 3,500,734 “light curves”, where each measurement is of an object’s brightness as a function of time, taken by measuring the photon ﬂux in six different astronomical ﬁlters.
The data can be treated as a six-dimensional time series.
The data set was introduced in a Kaggle competition, 4 where the task was to use these light curves to classify the variable sources.
The winning entry (Avocado, Boone, 2019) modeled the light curves with GPs and used these models to generate features for a gradient boosted decision tree classiﬁer.
We compare a multi-input–multi-output C ONV CNP with the GP models used in
Avocado.
5 C ONV CNP accepts six channels as inputs, one for each astronomical ﬁlter, and returnsoutputs — the means and standard deviations of six Gaussians.
Full experimental details are given in
Appendix C.3.
The mean squared error of both approaches is similar, but the held-out log-likelihood from the C ONV CNP is far higher (see Table 2).
[Table]
5. P REDATOR -P REY M ODELS : S IM 2R EAL
The C ONV CNP model is well suited for applications where simulation data is plentiful, but real world training data is scarce (Sim2Real).
The C ONV CNP can be trained on a large amount of simulation data and then be deployed with real-world training data as the context set.
We consider the
Lotka–Volterra model (Wilkinson, 2011)
[Table]
which is used to describe the evolution of predator–prey populations.
This model has been used in the Approximate Bayesian Computation literature where the task is to infer the parameters from samples drawn from the Lotka–Volterra process (Papamakarios & Murray, 2016).
These methods do not simply extend to prediction problems such as interpolation or forecasting.
In contrast
[Table]
we train C ONV CNP on synthetic data sampled from the Lotka–Volterra model and can then condition on real-world data from the Hudson’s Bay lynx–hare data set (Leigh, 1968) to perform interpolation (see Figure 3; full experimental details are given in Appendix C.4).
The C ONV CNP performs accurate interpolation as shown in Figure 3.
We were unable to successfully train the A TTN CNP for this task.
We suspect this is because the simulation data are variable lengthtime series, which requires models to leverage translation equivariance at training time.
As shown in
Section 5.1, the A TTN CNP struggles to do this (see Appendix C.4 for complete details).
5. 2D I MAGE C OMPLETION E XPERIMENTS
To test C ONV CNP beyond one-dimensional features, we evaluate our model on on-the-grid image completion tasks and compare it to A TTN CNP.
Image completion can be cast as a prediction of pixel intensities y ∗ i ( ∈ R 3 for RGB, ∈ R for greyscale) given a target 2D pixel location x ∗ i conditioned on an observed (context) set of pixel values
Z = ( x n , y n ) Nn =1 .
In the following experiments, the context set can vary but the target set contains all pixels from the image.
Further experimental details are in Appendix D.1.
[Table]
Standard benchmarks.
We ﬁrst evaluate the model on four common benchmarks: MNIST (LeCun et al., 1998), SVHN (Netzer et al., 2011), and 32 × 32 and 64 × 64 CelebA (Liu et al., 2018).
Importantly, these data sets are biased towards images containing a single, well-centered object.
As a result, perfect translation-equivariance might hinder the performance of the model when the test data are similarly structured.
We therefore also evaluated a larger C ONV CNP that can learn such non-stationarity, while still sharing parameters across the input space (C ONV CNPXL).
Table 3 shows that C ONV CNP signiﬁcantly outperforms A TTN CNP when it has a large receptive ﬁeld size, while being at least as good with a small receptive ﬁeld size.
Qualitative samples for various context sets can be seen in Figure 5.
Further qualitative comparisons and ablation studies can be found in Appendix D.3 and Appendix D.4 respectively.
Generalization to multiple, non-centered objects.
The data sets from the previous paragraphs were centered and contained single objects.
Here we test whether C ONV CNPs trained on such data can generalize to images containing multiple, non-centered objects.
The last column of Table 3 evaluates the models in a zero shot multi-MNIST (ZSMM) setting, where images contain multiple digits at test time (Appendix D.2).
C ONV CNP signiﬁcantly outperforms
A TTN CNP on such tasks.
Figure 4a shows a histogram of the image log-likelihoods for C ONV CNP and A TTN CNP, as well as qualitative results at different percentiles of the C ONV CNP distribution.
C ONV CNP is able to extrapolate to this out-of-distribution test set, while A TTN CNP appears to model the bias of the training data and predict a centered “mean” digit independently of the context.
Interestingly, C ONV CNPXL does not perform as well on this task.
In particular, we ﬁnd that, as the receptive ﬁeld becomes very large, performance on this task decreases.
We hypothesize that this has to do with behavior of the model at the edges of the image.
CNNs with larger receptive ﬁelds — the region of input pixels that affect a particular output pixel — are able to model non-stationary behavior by looking at the distance from any pixel to the image boundary.
We expand on this discussion and provide further experimental evidence regarding the effects of receptive ﬁeld on the ZSMM task in
Appendix D.6.
Although ZSMM is a contrived task, note that our ﬁeld of view usually contains multiple independent objects, thereby requiring translation equivariance.
As a more realistic example, we took a C ONV CNP model trained on CelebA and tested it on a natural image of different shape which contains multiple people (Figure 4b).
Even with 95% of the pixels removed, the C ONV CNP was able to produce a qualitatively reasonable reconstruction.
A comparison with A TTN CNP is given in Appendix D.3.
Computational efﬁciency.
Beyond the performance and generalization improvements, a key advantage of the C ONV CNP is its computational efﬁciency.
The memory and time complexity of a single self-attention layer grows quadratically with the number of inputs M (the number of pixels for images) but only linearly for a convolutional layer.
Empirically, with a batch size of 16 on 32 × 32 MNIST, C ONV CNPXL requires 945MB of VRAM, while A TTN CNP requires 5839 MB.
For the 56 × 56 ZSMM C ONV CNPXL increases its requirements to 1443 MB, while A TTN CNP could not ﬁt onto a 32GB GPU.
Ultimately, A TTN CNP had to be trained with a batch size of 6 (using 19139 MB) and we were not able to ﬁt it for CelebA64.
Recently, restricted attention has been proposed to overcome this computational issue (Parmar et al., 2018), but we leave an investigation of this and its relationship to C ONV CNPs to future work.
R ELATED W ORK AND D ISCUSSION
We have introduced C ONV CNP, a new member of the CNP family that leverages embedding sets into function space to achieve translation equivariance.
The relationship to (i) the NP family, and (ii) representing functions on sets, each imply extensions and avenues for future work.
Deep sets.
Two key issues in the existing theory on learning with sets (Zaheer et al., 2017; Qi et al., 2017a; Wagstaff et al., 2019) are (i) the restriction to ﬁxed-size sets, and (ii) that the dimensionality of the embedding space must be no less than the cardinality of the embedded sets.
Our work implies that by considering appropriate embeddings into a function space, both issues are alleviated.
In future work, we aim to further this analysis and formalize it in a more general context.
Point-cloud models.
Another line of related research focuses on 3D point-cloud modelling (Qi et al., 2017a;b).
While original work focused on permutation invariance (Qi et al., 2017a; Zaheer et al., 2017), more recent work has considered translation equivariance as well (Wu et al., 2019), leading to a model closely resembling C ONV D EEP S ETS .
The key differences with our work are the following: (i) Wu et al.
(2019) implement ψ as an MLP with learned weights, resulting in a more ﬂexible parameterization of the convolutional weights.
(ii) Wu et al.
(2019) interpret the computations as Monte Carlo approximations to an underlying continuous convolution, whereas we consider the problem of function approximation directly on sets.
(iii) Wu et al.
(2019) only consider the point-cloud application, whereas our derivation and modelling work considers general sets.
Correlated samples and consistency under marginalization.
In the predictive distribution of C ONV CNP
(Equation (2)), predicted y s are conditionally independent given the context set.
Consequently, samples from the predictive distribution lack correlations and appear noisy.
One solution is to instead deﬁne the predictive distribution in an autoregressive way, like e.g. PixelCNN++ (Salimans et al., 2017).
Although samples are now correlated, the quality of the samples depends on the order in which the points are sampled.
Moreover, the predicted y s are then not consistent under marginalization (Garnelo et al., 2018b; Kim et al., 2019).
Consistency under marginalization is more generally an issue for neural autoregressive models (Salimans et al., 2017; Parmar et al., 2018), although consistent variants have been devised (Louizos et al., 2019).
To overcome the consistency issue for
C ONV CNP, exchangeable neural process models (e.g. Korshunova et al., 2018;
Louizos et al., 2019) may provide an interesting avenue.
Another way to introduce dependencies between y s is to employ latent variables as is done in neural processes (Garnelo et al., 2018b).
However, such an approach only achieves conditional consistency : given a context set, the predicted y s will be dependent and consistent under marginalization, but this does not lead to a consistent joint model that also includes the context set itself. A CKNOWLEDGEMENTS
We would like to thank Mark Rowland for help with checking the proofs, and David R. Burt, Will
Tebbutt, Robert Pinsler and Cozmin Ududec for helpful comments on the manuscript.
Andrew
Y. K. Foong is supported by a Trinity Hall Research Studentship and the George and Lilian Schiff Foundation. Richard E. Turner is supported by Google, Amazon, ARM, Improbable and EPSRC grants EP/M0269571 and EP/L000776/ A T HEORETICAL R ESULTS AND P ROOFS
In this section, we provide the proof of Theorem 1.
Our proof strategy is as follows.
We ﬁrst deﬁne an appropriate topology for ﬁxed-sized sets (Appendix A.1).
With this topology in place, we demonstrate that our proposed embedding into function space is homeomorphic (Lemmas 1 and 2).
We then show that the embeddings of ﬁxed-sized sets can be extended to varying-sized sets by “pasting” the embeddings together while maintaining their homeomorphic properties (Lemma 3).
Following this, we demonstrate that the resulting embedding may be composed with a continuous mapping to our desired target space, resulting in a continuous mapping between two metric spaces (Lemma 4).
Finally, in Appendix A.3 we combine the above-mentioned results to prove Theorem 1.
We begin with deﬁnitions that we will use throughout the section and then present our results.
Let
X = R d and let Y ⊆
R be compact.
Let ψ be a symmetric, positive-deﬁnite kernel on X .
By the
Moore–Aronszajn Theorem, there is a unique Hilbert space ( H , (cid:104) · , · (cid:105) H ) of real-valued functions on X for which ψ is a reproducing kernel.
This means that (i) ψ ( · , x ) ∈ H for all x ∈ X and (ii) (cid:104) f, ψ ( · , x ) (cid:105) H = f
( x ) for all f ∈ H and x ∈ X (reproducing property).
For ψ : X × X → R ,
X = ( x 1 , . . .
, x n ) ∈ X n , and X (cid:48)
=
( x (cid:48) 1 , . . .
, x (cid:48) n ) ∈ X n , we denote
[Equation]

[Equation]
Deﬁnition 3 (Interpolating RKHS) .
Call H interpolating if it interpolates any ﬁnite number of points: for every (( x i , y i ))
ni
=1 ⊆
X × Y with ( x i ) ni
=1 all distinct, there is an f ∈ H such that f ( x 1 ) =
y 1 , . .
f ( x n )
=
y n .
For example, the RKHS induced by any strictly positive-deﬁnite kernel, e.g. the exponentiated quadratic (EQ) kernel ψ ( x , x (cid:48) )
=
σ 2 exp( − 12 (cid:96) 2 (cid:107) x − x (cid:48) (cid:107) 2 ) , is interpolating: Let c = ψ ( X , X ) − 1 y and consider f = (cid:80)
ni
=1 c i ψ ( · , x i ) ∈ H .
Then f ( X ) =
ψ ( X , X ) c
= y .
A. T HE Q UOTIENT S PACE A n / S n
Let A be a Banach space.
For x = ( x 1 , . . .
, x n ) ∈
A n and y =
( y 1 , . . .
, y n ) ∈
A n , let x ∼ y if x is a permutation of y ; that is, x ∼ y
if and only if x = π y for some π ∈ S n where
[Equation]
Let A n / S n be the collection of equivalence classes of ∼ .
Denote the equivalence class of x by [ x ] ; for A ⊆
A n , denote [ A ] = { [ a ] : a ∈ A } .
Call the map x
(cid:55)→ [ x ]: A n → A n / S n the canonical map.
The natural topology on A n / S n is the quotient topology, in which a subset of A n / S n is open if and only if its preimage under the canonical map is open in A n .
In what follows, we show that the quotient topology is metrizable.
On A n , since all norms on ﬁnite-dimensional vector spaces are equivalent, without loss of generality consider
[Equation]
Note that (cid:107) · (cid:107)
A n is permutation invariant: (cid:107) π · (cid:107)
A n = (cid:107) · (cid:107) A n for all π ∈ S n .
On A n / S n , deﬁne d
[Equation]
Call a set
[ A ] ⊆
A n / S n bounded if { d ([ x ] , [0]) :
[ x ] ∈ [ A ] } is bounded.
Proposition 1.
The function d is a metric.
Proof.
We ﬁrst show that d is well deﬁned on A n / S n .
Assume x ∼ x (cid:48) and y ∼ y (cid:48) .
Then, x (cid:48) =
π x x and y (cid:48)
=
π y y .
Using the group properties of S n and the permutation invariance of (cid:107) · (cid:107)
A n : d ([ x (cid:48) ] , [ y (cid:48) ])
=
min π ∈ S n (cid:107) π x x − ππ y y (cid:107) A n = min π ∈ S n (cid:107) π x x − π y (cid:107) A n = min π ∈ S n (cid:107) x − π − 1 x π y (cid:107) A n = min π ∈ S n (cid:107) x − π y (cid:107)
A n = d ([ x ] , [ y ]) .
It is clear that d ([ x ] , [ y ]) =
d ([ y ] , [ x ]) and that d ([ x ] , [ y ]) =
if and only if [ x ] =
[ y ] .
To show the triangle inequality, note that
[Equation]
using permutation invariance of (cid:107) · (cid:107) A n .
Hence, taking the minimum over π 1 ,
[Equation]
so taking the minimum over π 2 gives the triangle inequality for d .
Proposition 2.
The canonical map A n → A n / S n is continuous under the metric topology induced by d .
Proof.
Follows directly from d ([ x ] , [ y ]) ≤ (cid:107) x
− y (cid:107) A n .
Proposition 3.
Let A ⊆
A n be topologically closed and closed under permutations.
Then [ A ] is topologically closed in A n / S n under the metric topology.
Proof.
Recall that a subset [ A ] of a metric space is closed iff every limit point of [ A ] is also in [ A ] .
Consider a sequence ([ a n ]) ∞ n =1 ⊆
[ A ] converging to some [ x ] ∈ A n / S n .
Then there are permutations ( π n ) ∞ n =1 ⊆ S n such that π n a n → x .
Here π n a n ∈ A , because A is closed under permutations.
Thus x ∈ A , as A is also topologically closed.
We conclude that [ x ] ∈
[ A ] .
Proposition 4.
Let A ⊆
A n be open.
Then [ A ] is open in A n / S n under the metric topology.
In other words, the canonical map is open under the metric topology.
Proof.
Let [ x ] ∈
[ A ] .
Because A is open, there is some ball B ε ( y ) with ε > 0 and y ∈
A such that x ∈ B ε ( y ) ⊆
A .
Then [ x ] ∈ B ε ([ y ]) , since d ([ x ] , [ y ]) ≤ (cid:107) x − y (cid:107)
A n < ε , and we claim that
B ε ([ y ]) ⊆
[ A ] .
Hence [ x ] ∈ B ε ([ y ]) ⊆
[ A ] , so [ A ] is open.
To show the claim, let [ z ] ∈ B ε ([ y ]) .
Then d ( π z , y ) < ε for some π ∈ S n .
Hence π z ∈ B ε ( y ) ⊆
A , so π z ∈ A .
Therefore, [ z ] =
[ π z ] ∈
[ A ] .
Proposition 5.
The quotient topology on A n / S n induced by the canonical map is metrizable with the metric d .
Proof.
Since the canonical map is surjective, there exists exactly one topology on A n / S n relative to which the canonical map is a quotient map: the quotient topology (Munkres, 1974).
Let p : A n → A n / S n denote the canonical map.
It remains to show that p is a quotient map under the metric topology induced by d ; that is, we show that U ⊂ A n / S n is open in A n / S n under the metric topology if and only if p − 1 ( U ) is open in A n .
Let p − 1 ( U ) be open in A n .
We have that U
= p ( p − 1 ( U )) , so U is open in A n / S n under the metric topology by Proposition 4.
Conversely, if U is open in A n / S n under the metric topology, then p − 1 ( U ) is open in A n by continuity of the canonical map under the metric topology.
A. E MBEDDINGS OF S ETS I NTO AN RKHS
Whereas A previously denoted an arbitrary Banach space, in this section we specialize to A = X ×Y .
We denote an element in A by ( x , y ) and an element in Z M =
A M by (( x 1 , y 1 ) , . . .
, ( x M , y M )) .
Alternatively, we denote (( x 1 , y 1 ) , . . .
, ( x M , y M )) by ( X , y )
where X =
( x 1 , . .
, x M ) ∈
X M and y
=
( y 1 , . . .
, y M ) ∈ Y M .
We clarify that an element in Z M =
A M is permuted as follows: for π ∈ S M , π ( X , y )
=
π (( x 1 , y 1 ) , . . .
, ( x M , y M )) =
(( x π (1) , y π (1) ) , . . .
, ( x π ( n ) , y π ( n ) )) =
( π X , π y ) .
Note that permutation-invariant functions on Z M are in correspondence to functions on the quotient space induced by the equivalence class of permutations,
Z M / S m
The latter is a more natural representation.
Lemma 3 states that it is possible to homeomorphically embed sets into an RKHS.
This result is key to proving our main result.
Before proving Lemma 3, we provide several useful results.
We begin by demonstrating that an embedding of sets of a ﬁxed size into a RKHS is continuous and injective.
Lemma 1.
Consider a collection Z (cid:48)
M ⊆
Z M that has multiplicity K .
Set
[Equation]
and let ψ be an interpolating, continuous positive-deﬁnite kernel.
Deﬁne
[Equation]
where H K +1 = H × · · · × H is the ( K + 1) -dimensional-vector–valued–function Hilbert space constructed from the RKHS H for which ψ is a reproducing kernel and endowed with the inner product (cid:104) f, g (cid:105)
H K +1 =
(cid:80)
K +1 i =1 (cid:104) f
i , g i (cid:105) H .
Then the embedding
[Equation]
is injective, hence invertible, and continuous.
Proof.
First, we show that E M is injective.
Suppose that
[Equation]
Denote X = ( x 1 , . .
, x M ) and y =
( y 1 , . . .
, y M ) , and denote X (cid:48) and y (cid:48) similarly.
Taking the inner product with any f ∈ H on both sides and using the reproducing property of ψ , this implies that
M
[Equation]
for all f ∈ H .
In particular, since by construction φ 1 ( · ) = 1 ,
M (cid:88) i =
f
( x i ) =
M (cid:88) i =
f ( x (cid:48) i ) for all f ∈ H .
Using that H is interpolating, choose a particular ˆ x ∈ X ∪ X (cid:48) , and let f ∈ H be such that f (ˆ x ) = 1 and f ( · ) =
at all other x i and x (cid:48)
i .
Then (cid:88)
[Equation]
so the number of such ˆ x in X and the number of such ˆ x in X (cid:48) are the same.
Since this holds for every ˆ x , X is a permutation of X (cid:48) :
X = π
( X (cid:48) ) for some permutation π ∈ S M .
Plugging in the permutation, we can write
M (cid:88) i =
φ
[Equation]
Then, by a similar argument, for any particular ˆ x ,
[Equation]
Let the number of terms in each sum equal S .
Since Z (cid:48)
M has multiplicity K , S ≤ K .
By Lemma 4 from Zaheer et al.
(2017), the ‘sum-of-power mapping’ from { y i : x i =
ˆ x } to the ﬁrst S +elements of (cid:80) i : x i
=ˆ x φ ( y i ) , i.e. (cid:0)(cid:80) i : x i =ˆ x y
i , . .
, (cid:80) i : x i =
ˆ x y Si (cid:1) , is injective.
Therefore, ( y i )
i : x i =
ˆ x is a permutation of ( y (cid:48) π ( i ) )
i : x i =ˆ x .
Note that x i
=
ˆ x for all above y i .
Furthermore, note that also x (cid:48)
π ( i ) =
x i =
ˆ x for all above y (cid:48)
π ( i ) .
We may therefore adjust the permutation π such that y i = y (cid:48)
π ( i ) for all
i such that x
i =
ˆ x whilst retaining that x
= π
( x (cid:48) ) .
Performing this adjustment for all ˆ x , we ﬁnd that y
=
π ( y (cid:48) ) and x = π
( x (cid:48) ) .
Second, we show that E M is continuous.
Compute
[Equation]
i
[Equation]
=
[Equation]
which goes to zero if [ X (cid:48) , y (cid:48) ]
→ [ X , y ] by continuity of ψ .
Having established the injection, we now show that this mapping is a homeomorphism, i.e. that the inverse is continuous.
This is formalized in the following lemma.
Lemma 2.
Consider Lemma 1.
Suppose that Z (cid:48)
M is also topologically closed in A M and closed under permutations, and that ψ also satisﬁes (i) ψ ( x , x (cid:48) ) ≥ 0 , (ii) ψ ( x , x ) =
σ 2 > 0 , and (iii) ψ ( x , x (cid:48) )
→ 0 as (cid:107) x (cid:107) → ∞ .
Then H M is closed in H K +1 and E − 1 M is continuous.
Remark 1.
To deﬁne Z (cid:48) 2 with multiplicity one, one might be tempted to deﬁne
[Equation]
which indeed has multiplicity one.
Unfortunately, Z (cid:48) 2 is not closed: if [0 , 1] ⊆ X and [0 , 2] ⊆
Y , then ((0 , 1) , (1 /n, 2)) ∞
n =1 ⊆ Z (cid:48) 2 , but ((0 , 1) , (1 /n, 2))
→ ((0 , 1) , (0 , 2)) / ∈ Z (cid:48) 2 , because 0 then has two observations 1 and 2 .
To get around this issue, one can require an arbitrarily small, but non-zero spacing (cid:15) > 0 between input locations:
[Equation]
This construction can be generalized to higher numbers of observations and multiplicities as follows:
[Equation]
Remark 2.
Before moving on to the proof of Lemma 2, we remark that Lemma 2 would directly follow if Z (cid:48)
M were bounded: then Z (cid:48)
M is compact, so E M is a continuous, invertible map between a compact space and a Hausdorff space, which means that E − 1 M must be continuous.
The intuition that the result must hold for unbounded Z (cid:48)
M is as follows.
Since φ 1 ( · ) = 1 , for every f ∈ H M , f 1 is a summation of M “bumps” (imagine the EQ kernel) of the form ψ ( · , x i ) placed throughout X .
If one of these bumps goes off to inﬁnity, then the function cannot uniformly converge pointwise, which means that the function cannot converge in H (if ψ is sufﬁciently nice).
Therefore, if the function does converge in H , ( x i ) Mi
=1 must be bounded, which brings us to the compact case.
What makes this work is the density channel φ 1 ( · ) = 1 , which forces ( x i )
Mi
=1 to be well behaved.
The above argument is formalized in the proof of Lemma 2.
Proof.
Deﬁne
[Equation]
which is compact in A M as a closed subset of the compact set ([ − J, J ] d × Y ) M .
We aim to show that H M is closed in H K +1 and E − 1 is continuous.
To this end, consider a convergent sequence f ( n )
=
(cid:80) Mi
=1 φ ( y ( n ) i ) ψ ( · , x ( n ) i ) → f ∈ H K +1 .
Denote X ( n ) =
( x ( n ) 1 , . . .
, x ( n ) M ) and y ( n ) =
( y ( n ) 1 , . . .
, y ( n ) M ) .
Claim: ( X ( n ) ) ∞ n =1 is a bounded sequence, so ( X ( n ) ) ∞ n =1 ⊆
[ − J, J ] dM for J large enough, which means that ( X ( n ) , y ( n ) ) ∞ n =1 ⊆
Z J where Z J is compact.
Note that [ Z J ] is compact in A M / S M by continuity of the canonical map.
First, we demonstrate that, assuming the claim, H M is closed.
Note that by boundedness of ( X ( n ) , y ( n ) ) ∞ n =1 , ( f ( n ) ) ∞ n =1 is in the image of E M | [ Z J ] :
[ Z J ] → H M .
By continuity of E M | [ Z J ] and compactness of [ Z J ] , the image of E M | [ Z J ] is compact and therefore closed, since every compact subset of a metric space is closed.
Therefore, the image of E M | [ Z J ] contains the limit f .
Since the image of E M | [ Z J ] is included in H M , we have that f ∈ H M , which shows that H M is closed.
Next, we prove that, assuming the claim, E − 1 M is continuous.
Consider E M | [ Z J ] :
[ Z J ] → E M ([ Z J ]) restricted to its image.
Then ( E M | [ Z J ] ) − 1 is continuous, because a continuous bijection from a compact space to a metric space is a homeomorphism.
Therefore
[Equation]
By continuity and invertibility of E M , then f ( n ) → E M ( X , y ) , so E M ( X , y ) =
f by uniqueness of limits.
We conclude that
E − 1 M ( f ( n ) ) → E − 1 M ( f ) , which means that E − 1 M is continuous.
It remains to show the claim.
Let f 1 denote the ﬁrst element of f , i.e. the density channel.
Using the reproducing property of ψ ,
[Equation]
so f ( n )→ f 1 in H means that it does so uniformly pointwise (over x ).
Hence, we can let N ∈ N be such that n ≥ N implies that | f ( n ) 1 ( x ) − f 1 ( x ) | < 13 σ 2 for all x .
Let R be such that | ψ ( x , x ( N ) i ) | < 13 σ 2 /M for (cid:107) x (cid:107) ≥ R and all
i ∈
[ M ] .
Then, for (cid:107) x (cid:107) ≥ R , | f
[Equation]
N
[Equation] [Equation]
At the same time, by pointwise non-negativity of ψ , we have that
[Equation]
Towards contradiction, suppose that ( X ( n ) ) ∞ n =1 is unbounded.
Then ( x ( n ) i ) ∞ n =1 is unbounded for some i ∈
[ M ] .
Therefore, (cid:107) x ( n ) i (cid:107) ≥ R for some n ≥ N , so
[Equation]
which is a contradiction.
The following lemma states that we may construct an encoding for sets containing no more than M elements into a function space, where the encoding is injective and every restriction to a ﬁxed set size is a homeomorphism.
Lemma 3.
For every m ∈ [ M ] , consider a collection Z (cid:48) m ⊆
Z m that (i) has multiplicity K , (ii) is topologically closed, and (iii) is closed under permutations.
Set
[Equation]
and let ψ be an interpolating, continuous positive-deﬁnite kernel that satisﬁes (i) ψ ( x , x (cid:48) ) ≥ 0 , (ii) ψ ( x , x ) = σ 2 > 0 , and (iii) ψ ( x , x (cid:48) )
→ 0 as (cid:107) x (cid:107) → ∞ .
Deﬁne
[Equation]
where H K +1 = H × · · · × H is the ( K + 1) -dimensional-vector–valued–function Hilbert space constructed from the RKHS H for which ψ is a reproducing kernel and endowed with the inner product (cid:104) f, g (cid:105)
H K +1 =
(cid:80)
K +1 i =1 (cid:104) f
i , g i (cid:105) H .
Denote
[Equation]
M (cid:91) m =
[ Z (cid:48) m ] and H ≤ M =
M (cid:91) m =
H m .
Then ( H m )
Mm =1 are pairwise disjoint.
It follows that the embedding E
[Equation]
is injective, hence invertible.
Denote this inverse by E − 1 , where E − 1 ( f ) =
E − 1 m ( f ) if f ∈ H m .
Proof.
Recall that E m is injective for every m ∈
[ M ] .
Hence, to demonstrate that E is injective it remains to show that ( H m )
Mm =1 are pairwise disjoint.
To this end, suppose that
[Equation]
for m (cid:54) = m (cid:48) .
Then, by arguments like in the proof of Lemma 1,
[Equation]
Since φ 1 ( · ) = 1 , this gives m = m (cid:48) , which is a contradiction.
Finally, by repeated application of
Lemma 2, E − 1 m is continuous for every m ∈ [ M ] .
Lemma 4.
Let Φ: [ Z (cid:48)≤ M ] → C b ( X , Y ) be a map from [ Z (cid:48)≤ M ] to C b ( X , Y ) , the space of continuous bounded functions from X to Y , such that every restriction Φ | [ Z (cid:48) m ] is continuous, and let E be from Lemma 3.
Then
[Equation]
is continuous.
Proof.
Recall that, due to Lemma 1, for every m ∈
[ M ] , E − 1 m is continuous and has image [ Z (cid:48) m ] .
By the continuity of Φ | [ Z (cid:48) m ] , then Φ | [ Z (cid:48) m ] ◦ E − 1 m is continuous for every m ∈
[ M ] .
Since Φ ◦ E − 1 | H m =
Φ | [ Z (cid:48) m ] ◦ E − 1 m for all m ∈ [ M ] , we have that Φ ◦ E − 1 | H m is continuous for all m ∈
[ M ] .
Therefore, as H m is closed in H ≤ M for every m ∈ [ M ] , the pasting lemma (Munkres, 1974) yields that Φ ◦ E −is continuous.
From here on, we let ψ be a stationary kernel, which means that it only depends on the difference of its arguments and can be seen as a function X → R .
A. P ROOF OF T HEOREM
With the above results in place, we are ﬁnally ready to prove our central result, Theorem 1.
Theorem 1.
For every m ∈ [ M ] , consider a collection Z (cid:48) m ⊆
Z m that (i) has multiplicity K , (ii) is topologically closed, (iii) is closed under permutations, and (iv) is closed under translations.
Set
[Equation]
and let ψ be an interpolating, continuous positive-deﬁnite kernel that satisﬁes (i) ψ ( x , x (cid:48) ) ≥ 0 , (ii) ψ ( x , x ) = σ 2 > 0 , and (iii) ψ ( x , x (cid:48) )
→ 0 as (cid:107) x (cid:107) → ∞ .
Deﬁne
[Equation]
m
[Equation]
where H K +1 = H × · · · × H is the ( K + 1) -dimensional-vector–valued–function Hilbert space constructed from the RKHS H for which ψ is a reproducing kernel and endowed with the inner product (cid:104) f, g (cid:105)
H K +1 =
(cid:80)
K +1 i =1 (cid:104) f
i , g i (cid:105) H .
Denote
Z (cid:48)≤ M =
M (cid:91) m =
Z (cid:48) m and H ≤ M =
M (cid:91) m =
H m .
Then a function Φ: Z (cid:48)≤ M → C b ( X , Y ) satisﬁes
(i) continuity of the restriction Φ | Z m for every m ∈
[ M ] , (ii) permutation invariance (Property 1), and (iii) translation equivariance (Property 2)
if and only if it has a representation of the form
[Equation]
where ρ : H ≤ M → C b ( X , Y ) is continuous and translation equivariant.
Proof of sufﬁciency.
To begin with, note that permutation invariance (Property 1) and translation equivariance (Property 2) for Φ are well deﬁned, because Z (cid:48)≤ M is closed under permutations and translations by assumption.
First, Φ is permutation invariant, because addition is commutative and associative.
Second, that Φ is translation equivariant (Property 2) follows from a direct veriﬁcation and that ρ is also translation equivariant:
[Equation]
Proof of necessity.
Our proof follows the strategy used by Zaheer et al.
; Wagstaff et al.
To begin with, since Φ is permutation invariant (Property 1), we may deﬁne
[Equation]
for which we verify that every restriction Φ | [ Z (cid:48) m ] is continuous.
By invertibility of E from Lemma 3, we have [ Z ] =
E − 1 ( E ([ Z ])) .
Therefore,
[Equation]
Deﬁne ρ : H ≤ M → C b ( X , Y ) by ρ =
Φ ◦ E − 1 .
First, ρ is continuous by Lemma 4.
Second, E −is translation equivariant, because ψ is stationary.
Also, by assumption Φ is translation equivariant (Property 2).
Thus, their composition ρ is also translation equivariant.
Remark 3.
The function ρ : H ≤ M → C b ( X , Y ) may be continuously extended to the entirety of
H K +1 using a generalisation of the Tietze Extension Theorem by Dugundji et al. (1951).
There are variants of Dugundji’s Theorem that also preserve translation equivariance.
B B ASELINE N EURAL P ROCESS M ODELS
In both our 1d and image experiments, our main comparison is to conditional neural process models.
In particular, we compare to a vanilla CNP (1d only; Garnelo et al.
(2018a))
and an A TTN CNP (Kim et al., 2019).
Our architectures largely follow the details given in the relevant publications.
CNP baseline.
Our baseline CNP follows the implementation provided by the authors.
The encoder is a 3-layer MLP with 128 hidden units in each layer, and R E LU non-linearities.
The encoder embeds every context point into a representation, and the representations are then averaged across each context set.
Target inputs are then concatenated with the latent representations, and passed to the decoder.
The decoder follows the same architecture, outputting mean and standard deviation channels for each input.
Attentive CNP baseline.
The A TTN CNP we use corresponds to the deterministic path of the model described by Kim et al.
for image experiments.
Namely, an encoder ﬁrst embeds each context point c to a latent representation ( x ( c ) , y ( c ) )
(cid:55)→ r ( c ) xy ∈ R 128 .
For the image experiments, this is achieved using a 2-hidden layer MLP of hidden dimensions 128 .
For the 1d experiments, we use the same encoder as the CNP above.
Every context point then goes through two stacked self-attention layers.
Each self-attention layer is implemented with an 8-headed attention, a skip connection, and two layer normalizations (as described in Parmar et al.
(2018), modulo the dropout layer).
To predict values at each target point t , we embed x ( t ) (cid:55)→ r ( t ) x and x ( c ) (cid:55)→ r ( c ) x using the same single hidden layer MLP of dimensions 128 .
A target representation r ( t ) xy is then estimated by applying cross-attention (using an 8-headed attention described above) with keys K :=
{ r ( c ) x } Cc =1 , values V :=
{ r ( c ) xy } Cc =1 , and query q :=
r ( t ) x .
Given the estimated target representation ˆ r ( t ) xy , the conditional predictive posterior is given by a Gaussian pdf with diagonal covariance parametrised by ( µ ( t ) , σ ( t ) pre ) = decoder( r ( t ) xy ) where µ ( t ) , σ ( t ) pre ∈ R 3 and decoder is a 4 hidden layer MLP withhidden units per layer for the images, and the same decoder as the CNP for the 1d experiments.
Following Le et al.
(2018), we enforce we set a minimum standard deviation σ ( t ) min =
[0 . 1; 0 . 1; 0 . 1] to avoid inﬁnite log-likelihoods by using the following post-processed standard deviation:
[Equation]
C 1-D IMENSIONAL E XPERIMENTS
In this section, we give details regarding our experiments for the 1d data.
We begin by detailing model architectures, and then provide details for the data generating processes and training procedures.
The density at which we evaluate the grid differs from experiment to experiment, and so the values are given in the relevant subsections.
In all experiments, the weights are optimized using Adam (Kingma & Ba, 2015) and weight decay of 10 − 5 is applied to all model parameters.
The learning rates are speciﬁed in the following subsections.
C. CNN A RCHITECTURES
Throughout the experiments (Sections 5.1 to 5.3), we consider two models: C ONV CNP (which utilizes a smaller architecture), and C ONV CNPXL (with a larger architecture).
For all architectures, the input kernel ψ was an EQ (exponentiated quadratic) kernel with a learnable length scale parameter, as detailed in Section 4, as was the kernel for the ﬁnal output layer ψ ρ .
When dividing by the density channel, we add ε = 10 − 8 to avoid numerical issues.
The length scales for the EQ kernels are initialized to twice the spacing 1 /γ 1 /d between the discretization points ( t i )
Ti =1 , where γ is the density of these points and d is the dimensionality of the input space X .
Moreover, we emphasize that the size of the receptive ﬁeld is a product of the width of the CNN ﬁlters and the spacing between the discretization points.
Consequently, for a ﬁxed width kernel of the CNN, as the number of discretization points increases, the receptive ﬁeld size decreases.
One potential improvement that was not employed in our experiments, is the use of depthwise-separable convolutions (Chollet, 2017).
These dramatically reduce the number of parameters in a convolutional layer, and can be used to increase the CNN ﬁlter widths, thus allowing one to increase the number of discretization points without reducing the receptive ﬁeld.
The architectures for C ONV CNP and C ONV CNPXL are described below.
C ONV CNP
For the 1d experiments, we use a simple, 4-layer convolutional architecture, with R E LU nonlinearities.
The kernel size of the convolutional layers was chosen to be 5, and all employed a stride of length 1 and zero padding of 2 units.
The number of channels per layer was set to [16 , 32 , 16 , 2] , where the ﬁnal channels where then processed by the ﬁnal, EQ-based layer of ρ as mean and standard deviation channels.
We employ a SOFTPLUS nonlinearity on the standard deviation channel to enforce positivity.
This model has 6,537 parameters.
C ONV CNPXL.
Our large architecture takes inspiration from UNet (Ronneberger et al., 2015).
We employ a 12-layer architecture with skip connections.
The number of channels is doubled every layer for the ﬁrst 6 layers, and halved every layer for the ﬁnal 6 layers.
We use concatenation for the skip connections.
The following describes which layers are concatenated, where L i ← [ L j , L k ] means that the input to layer i is the concatenation of the activations of layers j and k :
• L 8 ← [ L 5 , L 7 ] , • L 9 ←
[ L 4 , L 8 ] , • L 10 ← [ L 3 , L 9 ] , • L 11 ← [ L 2 , L 10 ] , • L 12 ← [ L 1 , L 11 ] .
Like for the smaller architecture, we use R E LU nonlinearities, kernels of size 5, stride 1, and zero padding for two units on all layers.
C. S YNTHETIC 1 D E XPERIMENTAL D ETAILS AND A DDITIONAL R ESULTS
The kernels used for the Gaussian Processes which generate the data in this experiment are deﬁned as follows:
• EQ: k
[Equation]
x
[Equation]
x
[Equation]
e −
[Equation] [Equation]
x − x
[Equation]
C ONV
C N P
A TT N C
N P
C N P
• weakly periodic:
[Equation]
with f 1 ( x ) = cos(8 πx ) and f 2 ( x ) = sin(8 πx ) , and • Matern–
[Equation] [Equation]
with d = 4 | x − x (cid:48) | .
During the training procedure, the number of context points and target points for a training batch are each selected randomly from a uniform distribution over the integers between 3 and 50.
This number of context and target points are randomly sampled from a function sampled from the process (a Gaussian process with one of the above kernels or the sawtooth process), where input locations are uniformly sampled from the interval [ − 2 , 2] .
All models in this experiment were trained forepochs using 256 batches per epoch of batch size 16.
We discretize E ( Z ) by evaluating 64 points per unit in this setting.
We use a learning rate of 3e − 4 for all models, except for C ONV CNPXL on the sawtooth data, where we use a learning rate of 1e − 3 (this learning rate was too large for the other models).
The random sawtooth samples are generated from the following function:
[Equation]
where A is the amplitude, f is the frequency, and t is “time”.
Throughout training, we ﬁx the amplitude to be one.
We truncate the series at an integer K .
At every iteration, we sample a frequency uniformly in [3 , 5] , K in [10 , 20] , and a random shift in [ − 5 , 5] .
As the task is much harder, we sample context and target set sizes over [3 , 100] .
Here the CNP and A TTN
CNP employ learning rates of 10 − 3 .
All other hyperparameters remain unchanged.
C ONV
C N P
A TT N C
N P
C N P
[Table]
We include additional ﬁgures showing the performance of C ONV CNPs, A TTN CNPs and CNPs on
GP and sawtooth function regression tasks in Figures 6 to 8.
C. PLA S T I CC E XPERIMENTAL D ETAILS
The C ONV CNP was trained for 200 epochs using 1024 batches of batch size 4 per epoch.
For training and testing, the number of context points for a batch are each selected randomly from a uniform distribution over the integers between 1 and the number of points available in the series (usually between 10–30 per bandwidth).
The remaining points in the series are used as the target set.
For testing, a batch size of 1 was used and statistics were computed over 1000 evaluations.
We compare C ONV CNP to the GP models used in (Boone, 2019) using the implementation in https:// github.com/kboone/avocado .
The data used for training and testing is normalized according to t ( v ) =
( v − m ) /s with the values in Table 4.
These values are estimated from a batch sampled from the training data.
To remove outliers in the GP results, log-likelihood values less than − 10 are removed from the evaluation.
These same datapoints were removed from the C ONV CNP results as well.
C ONV
C N P
A TT N C
N P
C N P
C. P REDATOR –P REY E XPERIMENTAL D ETAILS
We describe the way simulated training data for the experiment in Section 5.3 was generated from the
Lotka–Volterra model.
The description is borrowed from (Wilkinson, 2011).
Let X be the number of predators and Y the number of prey at any point in our simulation.
According to the model, one of the following four events can occur:
A : A single predator is born according to rate θ 1 XY , increasing X by one.
B : A single predator dies according to rate θ 2 X , decreasing X by one.
C :
A single prey is born according to rate θ 3 Y , increasing Y by one.
D :
A single prey dies (is eaten) according to rate θ 4 XY , decreasing Y by one.
The parameter values θ 1 , θ 2 , θ 3 , and θ 4 , as well as the initial values of X and Y govern the behavior of the simulation.
We choose θ 1 = 0 .
01 , θ 2 = 0 .
5 , θ 3 = 1 , and θ 4 = 0 .
01 , which are also used in (Papamakarios & Murray, 2016) and generate reasonable time series.
Note that these are likely not the parameter values that would be estimated from the Hudson’s Bay lynx–hare data set (Leigh, 1968), but they are used because they yield reasonably oscillating time series.
Obtaining oscillating time series from the simulation is sensitive to the choice of parameters and many parametrizations result in populations that simply die out.
Time series are simulated using Gillespie’s algorithm (Gillespie, 1977):
Draw the time to the next event from an exponential distribution with rate equal to the total rate θ 1 XY + θ 2 X
+ θ 3 Y + θ 4 XY .
Select one of the above events A , B , C , or D at random with probability proportional to its rate.
Adjust the appropriate population according to the selected event, and go to 1.
The simulations using these parameter settings can yield a maximum population of approximately 300 while the context set in the lynx–hare data set has an approximate maximum population of about 80 so we scaled our simulation population by a factor of 2 / 7 .
We also remove time series which are longer than 100 units of time, which have more than 10000 events, or where one of the populations is entirely zero.
The number of context points n for a training batch are each selected randomly from a uniform distribution between 3 and 80, and the number of target points is 150 − n .
These target and context points are then sampled from the simulated series.
The Hudson’s Bay lynx–hare data set has time values that range from 1845 to 1935 .
However, the values supplied to the model range from 0 to 90 to remain consistent with the simulated data.
For evaluation, an interval of 18 points is removed from the the Hudson’s Bay lynx–hare data set to act as a target set, while the remaining 72 points act as the context set.
This construction highlights the model’s interpolation as well as its uncertainty in the presence of missing data.
Models in this setting were trained for 200 epochs with 256 batches per epoch, each batch containing 50 tasks.
For this data set, we only used the C ONV CNP, as we found the C ONV CNPXL to overﬁt.
The learning rate was set to 10 − 3 , and we discretize E ( Z ) by evaluating 100 points per unit.
We attempted to train an A TTN CNP for comparison, but due to the nature of the synthetic data generation, many of the training series end before 90 time units, the length of the Hudson’s Bay lynx-hare series.
Effectively, this means that the A TTN CNP was asked to predict outside of its training interval, a task that it struggles with, as shown in Section 5.1.
The plots in Figure 9 show that the A TTN CNP is able to learn the ﬁrst part of the time series but is unable to model data outside of the ﬁrst 20 or so time units.
Perhaps with more capacity and training epochs the A TTN CNP training would be more successful.
Note from Figure 3 that our model does better on the synthetic data than on the real data.
This could be due to the parameters of the Lotka–Volterra model used being a poor estimate for the real data.
D I MAGE E XPERIMENTAL D ETAILS AND A DDITIONAL R ESULTS D. E XPERIMENTAL D ETAILS
Training details.
In all experiments, we sample the number of context points uniformly from
U ( n total 100 , n total 2 ) , and the number of target points is set to n total .
The context and target points are sampled randomly from each of the 16 images per batch.
The weights are optimised using Adam (Kingma & Ba, 2015) with learning rate 5 × 10 − 4 .
We use a maximum of 100 epochs, with early stopping of 15 epochs patience.
All pixel values are divided by 255 to rescale them to the [0 , 1] range.
In the following discussion, we assume that images are RGB, but very similar models can be used for greyscale images or other gridded inputs (e.g. 1d time series sampled at uniform intervals).
Proposed convolutional CNP.
Unlike A TTN CNP and off-the-grid C ONV CNP, on-the-grid C ONV CNP takes advantage of the gridded structure.
Namely, the target and context points can be speciﬁed in terms of the image, a context mask M c , and a target mask M t instead of sets of input–value pairs.
Although this is an equivalent formulation, it makes it more natural and simpler to implement in standard deep learning libraries.
In the following, we dissect the architecture and algorithmic steps succinctly summarized in Section 4.
Note that all the convolutional layers are actually depthwise separable (Chollet, 2017); this enables a large kernel size (i.e. receptive ﬁelds) while being parameter and computationally efﬁcient.
Let I denote the image.
Select all context points signal :=
M c (cid:12) I and append a density channel density := M c , which intuitively says that “there is a point at this position”: [signal , density] (cid:62) .
Each pixel value will now have 4 channels: 3 RGB channels anddensity channel M c .
Note that the mask will set the pixel value to 0 at a location where the density channel is
0, indicating there are no points at this position (a missing value).
Apply a convolution to the density channel density (cid:48)
=
CONV θ (density) and a normalized convolution to the signal signal (cid:48) :
=
CONV θ (signal) / density (cid:48) .
The normalized convolution makes sure that the output mostly depends on the scale of the signal rather than the number of observed points.
The output channel size is 128 dimensional.
The kernel size of CONV θ depends on the image shape and model used (Table 5).
We also enforce element-wise positivity of the trainable ﬁlter by taking the absolute value of the kernel weights θ before applying the convolution.
As discussed in Appendix D.4, the normalization and positivity constraints do not empirically lead to improvements for on-the-grid data.
Note that in this setting, E ( Z ) is [signal (cid:48) , density (cid:48) ] (cid:62) .
Now we describe the on-the-grid version of ρ ( · ) , which we decompose into two stages.
In the ﬁrst stage, we apply a CNN to [signal (cid:48) , density (cid:48) ] (cid:62) .
This CNN is composed of residual blocks (He et al., 2016), each consisting of 1 or 2 (Table 5) convolutional layers with ReLU activations and no batch normalization.
The number of output channels in each layer is 128.
The kernel size is the same across the whole network, but depends on the image shape and model used (Table 5).
In the second stage of ρ ( · ) , we apply a shared pointwise MLP : R 128 → R 2 C (we use the same architecture as used for the A TTN CNP decoder) to the output of the ﬁrst stage at each pixel location in the target set.
Here C denotes the number of channels in the image.
The ﬁrst C outputs of the MLP are treated as the means of a Gaussian predictive distribution, and the last C outputs are treated as the standard deviations.
These then pass through the positivity-enforcing function shown in Equation (
[Table]
[Figure]
In the real world, it is very common to have multiple objects in our ﬁeld of view which do not interact with each other.
Yet, many image data sets in machine learning contain only a single, well-centered object.
To evaluate the translation equivariance and generalization capabilities of our model, we introduce the zero-shot multi-MNIST setting.
The training set contains all 60000 28 × 28 MNIST training digits centered on a black 56 ×background.
(Figure 10a).
For the test set, we randomly sample with replacement 10000 pairs of digits from the MNIST test set, place them on a black 56 × 56 background, and translate the digits in such a way that the digits can be arbitrarily close but cannot overlap (Figure 10b).
Importantly, the scale of the digits and the image size are the same during training and testing.
D. A TTN CNP AND C ONV CNP Q UALITATIVE C OMPARISON
Figure 11 shows the test log-likelihood distributions of an A TTN CNP and C ONV CNP model as well as some qualitative comparisons between the two.
Although most mean predictions of both models look relatively similar for SVHN and CelebA32, the real advantage of C ONV CNP becomes apparent when testing the generalization capacity of both models.
Figure 12 shows C ONV CNP and A TTN CNP trained on CelebA32 and tested on a downscaled version of Ellen’s famous Oscar selﬁe.
We see that C ONV CNP generalizes better in this setting.
D.
[Figure]
[Table] [Table]
no density
[Table]
[Figure]
To understand the importance of the different components of the ﬁrst layer, we performed an ablation study by removing the density normalization (C ONV CNP no norm.), removing the density channel (C ONV CNP no dens.), removing the positivity constraints (C ONV CNP no abs.), removing the positivity constraints and the normalization (C ONV CNP no abs.
norm.)
and replacing the fully trainable ﬁrst layer by an EQ kernel similar to the continuous case (C ONV CNP EQ).
Table 6 shows the following: (i) Appending a density channel helps.
(ii) Enforcing the positivity constraint is only important when using a normalized convolution.
(iii) Using a less expressive EQ ﬁlter does not signiﬁcantly decrease performance, suggesting that the model might be learning similar ﬁlters (Appendix D.5).
As discussed in Appendix D.4, using a less expressive EQ ﬁlter does not signiﬁcantly decrease performance.
Figure 13 shows that this happens because the fully trainable kernel learns to approximate the EQ ﬁlter.
D. E FFECT OF R ECEPTIVE F IELD ON T RANSLATION E QUIVARIANCE
As seen in Table 3, a C ONV CNPXL with large receptive ﬁeld performs signiﬁcantly worse on the
ZSMM task than C ONV CNP, which has a smaller receptive ﬁeld.
Figure 14 shows a more detailed comparison of the models, and suggests that C ONV CNPXL learns to model non-stationary behaviour, namely that digits in the training set are centred.
We hypothesize that this issue stems from the the treatment of the image boundaries.
Indeed, if the receptive ﬁeld is large enough and the padding values are signiﬁcantly different than the inputs to each convolutional layer, the model can learn position-dependent behaviour by “looking” at the distance from the padded boundaries.
For ZSMM, Figure 15 suggests that “circular” padding, where the padding is implied by tiling the image, helps prevent the model from learning non-stationarities, even as the size of the receptive ﬁeld becomes larger.
We hypothesize that this is due to the fact that “circularly” padded values are harder to distinguish from actual values than zeros.
We have not tested the effect of padding on other datasets, and note that “circular” padding could result in other issues.
E A UTHOR C ONTRIBUTIONS
Richard, Jonathan, and Wessel formulated the project.
Richard and Jonathan helped coordinate the team and were closely involved with all aspects of the project.
Andrew realised that a density channel was necessary in the model.
Yann proposed using normalized convolutions and showed that they led to improvements.
All authors contributed to the writing and editing of the paper.
Wessel and Jonathan developed the theory.
Andrew veriﬁed the proof and suggested improvements.
James also suggested improvements.
Wessel wrote the majority of Appendix A with assistance from Andrew and Jonathan.
Andrew and Jonathan performed the initial experiments on simple time-series.
Wessel, Jonathan, and James reﬁned these experiments and produced the ﬁnal versions for the paper.
James wrote this section of the paper.
James led the experiments on complex time-series and wrote this section of the paper.
Andrew worked on the ﬁrst on-the-grid experiments.
Yann redesigned and reimplemented the on-the-grid computational framework and performed all the image experiments shown in the paper.
Yann wrote this section of the paper.