Paper:
We introduce the Convolutional Conditional Neural Process (CONVCNP), a new member of the Neural Process family that models translation equivariance in the data. Translation equivariance is an important inductive bias for many learning problems including time series modelling, spatial data, and images. The model embeds data sets into an infinite-dimensional function space as opposed to a finitedimensional vector space. To formalize this notion, we extend the theory of neural representations of sets to include functional representations, and demonstrate that any translation-equivariant embedding can be represented using a convolutional deep set. We evaluate CONVCNPs in several settings, demonstrating that they achieve state-of-the-art performance compared to existing NPs. We demonstrate that building in translation equivariance enables zero-shot generalization to challenging, out-of-domain tasks.

Review:
The paper introduces ConvCNP, a new member of the neural process(NP) family that models translational equivariance in the data, which uses convolutions and stationary kernels to aggregate the context data into a functional representation.
This problem is well-motivated as there are various domains where such an inductive bias is desirable, such as spatio-temporal data and images, and will help especially with predictions for out-of-distribution tasks. This inductive bias was never built into NPs, and it remained unanswered whether the NP can learn such a behaviour. This paper shows that the answer is negative and that one needs to make modifications to create such inductive bias.
The architecture of the ConvCNP is motivated by theory that completely characterises the set of translation equivariant functions Phi that maps sets of (x,y) pairs to bounded continuous functions that map x to y (disclaimer: I haven’t read through the proof in the appendix, so will not make any claims on its correctness). Theorem 1 defines the set of such functions using rho, phi and psi, and the choices for each on on-the-grid data and off-the-grid data are listed in Section 4. There are ablation studies in Appendix D.4 that justify the choices.
Overall the paper is very well-written and clear for the most part, with helpful pseudo-code and well-laid out quantitative + qualitative results, and a very detailed appendix that allows replicating the setup. The evaluation is extensive, and the results are significant.
    - The results on 1D synthetic data show a noticeable improvement of the ConvCNP compared to the AttnCNP, with improved interpolation as well as accurate extrapolation for the weakly periodic function. I do think however that a more competitive baseline for AttnCNP would have been to parameterise the logits of the attention weights as a periodic function with learnable length scale (e.g. stationary periodic kernel), since this is another way of building in periodicity into the model. Arguably this is more explicit and restrictive than the translational equivariance built into ConvCNP, but would have made for a more interesting comparison.
    - Having said that, I like how the evaluation was performed on a variety of stochastic processes - previous literature only used GP + EQ kernel, but here more challenging non-smooth functions such as GP + Matern kernels and sawtooth functions are explored - and it’s very convincing to see the outstanding performance of ConvCNPs here.
    - It’s also nice to see results on regression tasks on real data (sections 5.2, 5.3), which was never explored in the NP literature as far as I know. 5.2 shows that ConvCNPs can be competitive against other methods that model stochastic processes, and 5.3 shows an instance of where ConvCNPs do a reasonable job whereas (Attn)CNP fails.
    - The results on images is also extensive, covering 6 different datasets (including the 2 zero shot tasks), and show convincing qualitative and quantitative results. The zero shot tasks are nice examples that explicitly show the consequences of not being able to model translation equivariance in more realistic images composed of multiple objects/faces.
I have several comments/questions regarding the disccusion & related work section:
    - One link that might be worth pointing out regarding functional representation of context is that ANP (or AttenCNP) can also be seen as giving a functional representations of the context; the ANP computes a target-specific representation of the context, which can be seen as a function of the target inputs.
    - I think it’s incorrect to say that latent-variable extensions enforce consistency. Even with the latent variable, if the encoder is seen as part of the model, then the NP isn’t consistent (pointed out in the last paragraph of section 2.1 in the ANP paper). So there still are issues regarding AR sampling. There does however seem to exist variants of NPs that satisfy consistency e.g. https://arxiv.org/abs/1906.08324
    - What is preventing the incorporation of a latent variable in the ConvCNP? Is this just something that can be easily done but you haven’t tried, or do you see any non-trivial issues that arise when doing so e.g. maintaining translation equivariance?
Other minor comments:
    - Are there any guidelines on choice of filter size of CNN in the image case? E.g. have you chosen the filter size of ConvCNP such that the receptive field is smaller than the image, whereas it’s bigger for ConvCNPXL? It’s not clear why having a bigger receptive field allows to capture non-stationarity, and it would be helpful to expand on that, perhaps in the appendix.
    - Also it’d help for the sake of clarity to explain why AttnCNP uses significantly more memory than ConvCNP, i.e. because memory for self-attention is O(N^2) where N=HW is the number of inputs, whereas for convolutions it’s O(HW).
    - I think it’d also help to state explicitly in the body that AttnCNP is ANP without the latent path when it is introduced.
    - typos: first paragraph of Section 2: Z_M <- Z_m (twice), finitely <- infinitely, Appendix D.1: separabe <- separable
Overall, I think this is a very strong submission and I vote for its acceptance.