V ERSATILE N EURAL P ROCESSES FOR L EARNING I MPLICIT N EURAL R EPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Representing a signal as a continuous function parameterized by neural network (a.k.a.
Implicit Neural Representations, INRs) has attracted increasing attention in recent years.
Neural Processes (NPs), which model the distributions over functions conditioned on partial observations (context set), provide a practical solution for fast inference of continuous functions.
However, existing NP architectures suffer from inferior modeling capability for complex signals.
In this paper, we propose an efficient NP framework dubbed Versatile Neural Processes (VNP), which largely increases the capability of approximating functions.
Specifically, we introduce a bottleneck encoder that produces fewer and informative context tokens, relieving the high computational cost while providing high modeling capability.
At the decoder side, we hierarchically learn multiple global latent variables that jointly model the global structure and the uncertainty of a function, enabling our model to capture the distribution of complex signals.
We demonstrate the effectiveness of the proposed VNP on a variety of tasks involving 1D, 2D and 3D signals.
Particularly, our method shows promise in learning accurate INRs w.r.t.
a 3D scene without further finetuning.
I NTRODUCTION
A recent line of research on learning representations is to model a signal ( e .
g ., image, 3D scene)
as a continuous function that map the input coordinates into the corresponding signal values.
By parameterizing a continuous function with neural networks, such implicitly defined representations, i .
e ., implicit neural representations (INRs), offer many benefits over conventional discrete ( e .
g ., grid-based) representations, such as the compactness and memory-efficiency (Sitzmann et al., 2020b;
Tancik et al., 2020; Mildenhall et al., 2020; Chen et al., 2021a).
Characterizing/parameterizing a signal by a corresponding set of network parameters generally requires re-training the neural network, which is computationally costly.
In practice, at test time, it is desired to have models that support fast adaptation to partial observations of a new signal without finetuning.
In fact, the Neural Processes (NPs) family (Jha et al., 2022) supports such merit.
It meta-learns the implicit neural representations of a probabilistic function conditioned on partial signal observations.
During test-time inference, it enables the prediction of the function values at target points within a single forward pass.
Naturally, given partial observations of a signal, there exists uncertainty inside its continuous function since there are many possible ways to interpret these observations ( i .
e ., context set).
The NP methods (Garnelo et al., 2018a;b) learn to map a context set of observed input-output pairs to a conditional distribution over functions (with uncertainty modeling).
However, it has been observed that NPs are prone to underfit the data distribution.
Following the spirits of variational auto-encoders (Kingma & Welling, 2014), the work of (Garnelo et al., 2018b) introduces a global latent variable to better capture the uncertainty in the overall structure of the function, which still suffers from the inferior capability for modeling complex signals.
Attentive Neural Processes (ANP) (Kim et al., 2019) can further alleviate this issue, which leverages the permutation-invariant attention mechanism (Vaswani et al., 2017) to reweight the context points and the target predictions.
However, taking each context point as a token, ANP has troubles in processing complex signals that requires abundant context points as condition ( e .
g ., image with high resolution), where the computational cost is very expensive.
Moreover, for complex signals, modeling the global structure and uncertainty of the function with a single latent Gaussian variable may be suboptimal.
It is worthwhile to explore an efficient framework to excavate the potential of NPs in modeling complex signals.
Partial Signal Observations 1D 2D 3D
Bottleneck Encoder
Modulated MLPBlock
Predicted Target Signals 1D 2D 3D 𝑿 𝑻
S e l f
A tt e n t i o n
C r o ss A tt e n t i o n
Q
K
V
Context Tokens
Blocks ×𝐿 𝐾 𝑫 𝑻 =
(𝑿 𝑻 ,𝒀 𝑻 )
S e t
T o k e n i z e r
Hierarchical Latent Variables ×𝐿 𝑠 ×𝐿 𝑐 * * * * * *
Context Set 𝑫 𝑪 =
(𝑿 𝑪 ,𝒀 𝑪 )
Q
K
V
Or Or
Or Or
M L P s 𝑝 𝜃 (𝑌 𝑇 |𝑧,𝑋 𝑇 ,𝐷 𝐶 )
Decoder
Modulated MLP Block 𝒛𝒛 𝑳 𝑲 … …
In this paper, we propose Versatile Neural Processes (VNP), an efficient and flexible framework for meta-learning of implicit neural representations.
Figure 1 shows the framework of VNP.
Specifically,
VNP consists of a bottleneck encoder and a hierarchical latent modulated decoder.
The bottleneck encoder powered by the set tokenizer and self-attention blocks encodes the set of context points into fewer and informative context tokens, refraining from high computational cost especially on complex signals while attaining higher modeling capability.
At the decoder, we hierarchically learn multiple latent Gaussian variables for jointly modeling the global structure and uncertainty of the function distributions.
Particularly, we sample from the latent variables and use them to modulate the parameters of the MLP modules.
Our VNP has high expressiveness to complex signals ( e .
g ., 2D images and 3D scenes) and significantly outperforms existing NPs approaches on 1D synthetic data.
We summarize our main contributions as below:
• We propose Versatile Neural Processes (VNP) that is capable of learning accurate INRs for approximating the function of a complex signal.
• We introduce a bottleneck encoder to produce compact yet representative context tokens, facilitating the processing of complex signals with tolerable computational complexity.
• We design a hierarchical latent modulated decoder that can better capture and describe the structure and uncertainty of function through the joint modulation from the multiple global latent variables.
• We implement the VNP framework on 1D, 2D, and 3D signals respectively, demonstrating the state-of-the-art performance on a variety of tasks.
Particularly, our method shows promise in learning accurate INRs of 3D scenes without further finetuning.
R ELATED WORK
Implicit Neural Representations (INRs).
INRs aim at parameterizing a signal by a differentiable neural network, i .
e ., learning a continuous mapping function w.r.t.
the signal (Stanley, 2007; Sitzmann et al., 2020b; Tancik et al., 2020).
In the seminal work CPPN (Stanley, 2007), a neural network is trained to learn the implicit function that fits a signal, e .
g ., an image.
Given any spatial position identified by a 2D coordinate, the model that acts as a function, outputs the color value of this position.
Such continuous representations, as a powerful paradigm, have a wide range of applications such as 2D image super-resolution (Chen et al., 2021b), modeling shapes (Chen & Zhang, 2019; Park et al., 2019) and textures (Oechsle et al., 2019), and 3D scene reconstruction (Mildenhall et al., 2020;
Martin-Brualla et al., 2021; Niemeyer & Geiger, 2021).
Most of these methods require re-training the neural network to model/overfit a new signal, which is computationally costly (Sitzmann et al., 2020b; Tancik et al., 2020;
Mildenhall et al., 2020; Chen et al., 2021a).
In practice, it is desired to have models that support fast adaptation to a new signal, i .
e ., approaching the continuous function of this signal without abundant steps of optimization in inference.
Some works (Chen et al., 2021b; Lee et al., 2021; Sitzmann et al., 2020a) adopt standard gradient-based meta-learning algorithms to learn the initial weight parameters of the network (Finn et al., 2017).
However, they still require a few gradient decent steps to fit for the new signals.
Neural Processes (NPs).
Neural Processes actually can learn the continuous function conditioned on partial observations of a signal, enabling fast adaptation to a new signal (not requiring finetuning in inference).
The series of NPs methods (Garnelo et al., 2018a;b) provide probabilistic solutions in predicting continuous functions from partial observations.
NPs approximate the distributions in the function space, which formulates Stochastic Processes (Ross et al., 1996), by introducing stochasticity in function realization.
A line of researches on neural processes has been introduced, targeting at improving the prediction accuracy (Kim et al., 2019; Lee et al., 2020; Wang & Van Hoof, 2020; Volpp et al., 2021), preserving the stationarity of stochastic processes (Gordon et al.;
Foong et al., 2020), and generalizing to observation noise (Kim et al., 2022).
The work of Neural Processes (Garnelo et al., 2018b) learns a latent variable distribution to capture the global uncertainty in the overall structure of the function, which is optimized with variational inferece (Kingma & Welling, 2014).
Attentive Neural
Processes (ANP) leverages the attention mechanism to enhance the representation of each context point and alleviate the underfitting problem (Kim et al., 2019).
Transformer Neural Processes (TNP) (Nguyen & Grover, 2022) similarly takes each context point as a token and leverages transformer architecture to approximate the function.
However, for complex signals that requiring abundant context points as condition ( e . g ., image with high resolution), the computational complexities of ANP and TNP are very high which are quadratic with respect to the number of context points.
It is desired to have a framework that can effectively approximate the functions of complex signals.
In this work, we introduce a strong NP framework, Versatile Neural Processes (VNP), which leverages informative context tokens and explores the hierarchical global latent variables for modulation, leading to superior approximation of function distributions.
R EVISITING THE P ROBLEM F ORMULATION OF NP S
Neural Processes (NPs) (Garnelo et al., 2018b;a) are a class of methods that approximate the probabilistic distribution of the continuous function conditioned on partial observations.
Suppose we have a labeled context set D C
= ( X C , Y C ) :
= ( x i , y i )
i ∈ C sampled from a continuous function with inputs x i and outputs y i , NPs target at predicting an arbitrary and finite number of target points
D T =
( X T , Y T ) :
= ( x i , y i ) i ∈ T by learning the input-output mapping function f .
Given some observations (a set of context points) of a signal, many possible functions may match well to these observations and thus naturally there exists uncertainty on the functions.
The conditional distributions of targets points can be modeled as:
[Equation]
To generate coherent function predictions and better model the function distributions, Garnelo et al .
(Garnelo et al., 2018b) introduce the (Latent) Neural Process by encoding the global structure and uncertainty of the function into a latent Gaussian variable via Bayesian inference:
[Equation]
Due to the intractable log-likelihood, some previous works adopt amortized variational inference (Kingma & Welling, 2014), which is also used to optimize our proposed framework.
We can derive the evidence lower bound (ELBO) on log p θ ( Y T | X T , z ) , where the ELBO can be viewed as a combination of the reconstruction term (the first term) and the KL term (the second term):
[Equation]
Here, ψ and θ refer to the parameters of conditional prior encoder and the decoder, similar to conditional VAEs (Sohn et al., 2015; Ivanov et al., 2019).
q ϕ ( z | D T ) denotes the posterior distribution of the latent variable given the ground truth target points, which is only used in training and not accessed in inference.
So far, we have revisited the problem formulation of NPs.
Since NPs approximate the distribution over functions with some function observations, they can also be regarded as probabilistic, continuous, conditional generative models, where the generation result is a continuous function.
M o d - F C M o d - F C M L P s 𝑫 𝑻 =
(𝑿 𝑻 ,𝒀 𝑻 )
Hierarchical Latent Variables 𝒛 𝒌 + +
MLPs
MLPsPooling
M L P s M L P s P oo li n g ෡𝒀 𝒌− ෡𝒀 𝒌 M o d - F C
M o d - F C M L P s C r o ss
A tt e n t i o n
Q
K
V
PoolingModule 𝒛×𝐿 𝑐 + +
M o d - F C M o d - F C M L P s 𝒛 𝒌 +
+ ContextTokens 𝑿 𝑻 ෡𝒀 𝒌−𝟏 ෡𝒀෡𝒀 𝒌 𝑝 𝜓 𝑞 𝜙 𝑫 𝑻 =
(𝑿 𝑻 ,𝒀 𝑻 ) 𝑹 𝑻×𝒅 𝑹 𝟏×𝒅
𝒛 𝑹 𝑻×𝒅 (a
[Figure]
Pipeline of the decoder.
M o d - F C M o d - F C M L P s 𝑫 𝑻 =
(𝑿 𝑻 ,𝒀 𝑻 )
Hierarchical Latent Variables 𝒛 𝒌 + +
MLPs MLPsPooling
M L P s M L P s P oo li n g ෡𝒀 𝒌− ෡𝒀 𝒌 M o d - F C
M o d - F C M L P s C r o ss A tt e n t i o n
Q K
V
PoolingModule 𝒛×𝐿 𝑐 + +
M o d - F C M o d - F C M L P s 𝒛 𝒌 +
+ ContextTokens 𝑿 𝑻 ෡𝒀 𝒌−𝟏 ෡𝒀෡𝒀 𝒌 𝑝 𝜓 𝑞 𝜙 𝑫 𝑻 =
(𝑿 𝑻 ,𝒀 𝑻 ) 𝑹 𝑻×𝒅 𝑹 𝟏×𝒅
𝒛 𝑹 𝑻×𝒅 (
[Figure]
V ERSATILE N EURAL P ROCESSES
We propose an efficient NP framework, Versatile Neural Processes (VNP), which provides much improved capability in approximating the function of a various signal.
Figure 1 provides an overview of our proposed VNP.
VNP is generic that can be used to generate the functions for 1D, 2D or 3D signals.
For test-time inference, the inputs are a set of context points D C
=
( X C , Y C ) and the coordinates of target points X T .
The outputs are the predicted values of target points Y T .
The continuous function is parameterized by the network weights.
The proposed VNP consists of a bottleneck encoder that efficiently encodes the context points into fewer informative context tokens, refraining from high computational cost especially on complex signals.
At the decoder side, crossattention layers are utilized to exploit the contexts relevant to the given target followed by a stack of modulated MLP blocks.
Particularly, we introduce multiple global latent variables to jointly modulate the MLP parameters, which facilitates the modeling of complex signal.
4. B OTTLENECK E NCODER IN VNP
Some NP methods encode the context points to produce local feature representations at target coordinates.
The early work (Garnelo et al., 2018b) uses simple MLP layers to learn features of the context set and suffers from underfitting problem.
Attentive NP (Kim et al., 2019) and Transformer
NP (Nguyen & Grover, 2022) enhances the feature representation by introducing point-wise selfattention with each context point taken as a token.
However, when the number of context points is large ( e .
g .in order to model an image signal with many details), the computational burden is heavy and such a framework is impractical.
We address this issue by simply introducing a set tokenization module (set tokenizer) followed by self-attention layers.
The set tokenizer is instantiated by a set convolution layer (Zaheer et al., 2017) which transforms the neighboring context points to a token.
Taking a 2D image as an example, with the kernel size of k × k and stride of k , the set tokenizer can reduce the number of sample points by k 2 times (assume the image resolution is an integer multiple of k ).
By using the set tokenzier, the complexity of attention layers is reduced from the first term to the second term as below:
[Equation]
where L s and L c are the number of the self-attention layers and the cross-attention layers, N C and
N T are the number of context points and target points.
The difference between the set convolution and the conventional convolution is that the former can handle missing points ( e .
g .
, in the application of inpainting) and the data that live "off the grid" ( e .
g .
time series data that observed irregularly at any time).
The combination of set tokenizer and attention would empower our framework the flexibility in processing 3D signals, which is infeasible for ConvNP (Gordon et al.
; Foong et al., 2020) because ConvNP requires to preserve all the 3D grids, which is very inefficient in sparse 3D space.
4. H IERARCHICAL L ATENT M ODULATED MLP S
In the previous NP methods (Garnelo et al., 2018b; Kim et al., 2019), they learn a single global latent
Gaussian variable from the observed context set to model the distributions of a function.
However, it potentially limits the expressiveness of the model.
Intuitively, increasing the dimension of the latent variables may increase such flexibility, but in practice, this is not sufficient (Lee et al., 2020).
We design a hierarchical latent modulated decoder in order to better model the complex distribution of a function.
We sequentially learn multiple global latent variables to modulate the parameters of
MLP blocks.
Figure 2 shows the details of the designed hierarchical structure.
The decoder consists of L c cross-attention blocks and L K modulated MLP blocks.
With the target location X T as query and the context tokens from encoder as the keys and values, the cross-attention blocks output the target location features ˆ Y 0 ∈ R T
×
d , where T is the number of target coordinates and d is the feature dimension.
The sequential modulated MLP blocks enable the exploitation of hierarchical global latent variables for the approximation of a complex signal.
The output of the final ( L thK ) modulated MLP block goes through two MLP layers to estimate the probability of Y T .
Figure 2b illustrates the detailed network structure of a modulated MLP block.
We build a Modulated
MLP block by stacking two modulated MLP layers and two unmodulated MLP layers.
This block refines ˆ Y k − 1 to output features ˆ Y k , which is the input of the next Modulated MLP block.
At the heart of each block is a latent variable z ∈ R 1 ×
d z modeled by a Gaussian distribution.
The generation process of z k (of the k th block) can be formulated as follows:
[Equation]
where MLPs refer to two MLP layers and an intermediate ReLU activation layer.
The prediction results Y k − 1 from the previous block ( i .
e ., the ( k − 1) th block) are used for calculating the conditional prior distribution of z k , i .
e ., p ψ ( z k | ˆ Y <k , D C , X T ) .
During training, the conditional posterior distribution q ϕ ( z k |
ˆ Y <k , D C , D T ) can be calculated as well, by incorporating the ground truth target signal values Y T :
[Equation]
As marked with dashed lines in our figures, Y T only participates in training and cannot be accessed during inference.
After sampling the low-dimensional latent variable z k , we use the modulated fully-connected (ModFC) layer (Karras et al., 2020; 2021) to adjust the parameters of modulated MLP layers, taking a sampled realization of z k as the style vector input.
Unlike previous NP methods that concatenate the latent variable to every target coordinate, modulating the MLP parameters from the low-dimensional latent variables is a more flexible way to represent continuous functions (Sitzmann et al., 2020b).
Please see Appendix A for more details for the mechanism of the modulated MLP.
We model the function representations by multiple latent variables ( z 1 , z 2 , ..., z L K ) .
The KL term in Eq. 3, which measures the mismatch between the approximated distribution and ground truth distribution, takes the hierarchical format as
[Equation]
This decomposed KL term together with the original reconstruction term formulates our final optimization objective as
[Equation]
where β denotes a weight to balance the importance between the two terms.
In our experiments on 2D and 3D signals, we multiply the KL term with a small weight β to better capture the uncertainty of function distributions (Higgins et al.).
In addition, we emphasize that although there are some prior works designing hierarchical architecture for VAEs (Sønderby et al., 2016; Vahdat & Kautz, 2020; Child, 2021) or double latent variable models for NP (Wang & Van Hoof, 2020), our proposed hierarchical architecture is different in principle.
Here, every latent variable is a low-dimensional vector obtained after average pooling.
Therefore, the designed hierarchical architecture can deal with arbitrary target coordinates and thereby can be used for boosting the performance of approximating the global structure of continuous functions.
E XPERIMENTS
The proposed Versatile Neural Process (VNP), as an efficient meta-learner of implicit neural representations, can be implemented into a variety of tasks.
We evaluate the effectiveness of VNP on 1D function regression (subsection 5.1), 2D image completion and superresolution (subsection 5.2), and view synthesis for 3D scenes (subsection 5.3), respectively.
[Table]
5. 1D S IGNAL R EGRESSION
We implement the proposed VNP to learn the implicit neural representations for 1D signal regression.
This classical 1D regression aims at predicting the function values at given target locations, conditioned on several observations of the samples from the function.
Following (Kim et al., 2019), we measure the performance by considering the context set likelihood and target set likelihood, which reflects the context reconstruction error and target prediction error, respectively.
Settings.
We train the models on synthetic functions drawn from prior function distributions synthesized with different kernels (RBF, Matern) by following (Gordon et al.;
Kim et al., 2022).
For the evaluated methods, we employ importance weighted sampling (Burda et al., 2016) to evaluate the log likelihood, where the last four methods in Table 1 are measured by sampling latent variables from the posterior distribution and calculating the tighter ELBO with importance weighted sampling.
For fair comparison, we manage to keep the network size comparable with that of other methods, which is completed by adjusting channel number.
Please refer to Appendix B for detailed settings.
Comparison with the State-of-the-Arts .
We compare our method with conditional neural process (CNP) (Garnelo et al., 2018a), (stacked) attentive neural process (ANP) (Kim et al., 2019) (with stacked self-attention layers), bootstrapping attentive neural process (BANP)
(Lee et al., 2020), and convolutional neural process (ConvNP) (Foong et al., 2020).
Table 1 shows the results in terms of log likelihood.
Most of the models can provide satisfactory reconstruction results for context points, except CNP which suffers from underfitting problem (Garnelo et al., 2018b).
On the context points, ours also achieves comparable performance.
However, the prediction results of other methods on the unseen target points are much worse than that on the context sets.
In contrast, our final model, VNP, outperforms previous approaches by a large margin at the target points on both of the function distributions.
Note that although BANP attempts to increase the expressiveness of function representations by using latent variable bootstrapping from the perspective data resampling.
Ablation Study.
We conduct a group of ablation study on this 1D regression task to investigate the effectiveness of different components.
Based on ANP, we first build a Hierarchical Neural
Process (HNP) with hierarchical latent variables.
Note that similar to ANP, the implemented HNP still concatenates every target coordinate with the global latent variables.
We observe significant improvements in Table 1 by comparing HNP with ANP, demonstrating that the hierarchical global latent variables boost the performance of function approximation.
Secondly, the modulated MLP layer for implicit function parameterization can be further equipped in HNP, referred to as HNPMod.
Compared with the concatenation of latent variable, using low-dimensional latent variables to modulate the network parameters enables more flexible function approximating.
Based on HNP-Mod, we finally add the bottleneck encoder to learn conditional network inputs that are adaptive to target locations, building our final model VNP.
In Table 1, we can find this powerful pre-processing encoder can further improve the performance, because the set tokenizer (set convolution here) can preserve locality to learn more appropriate features.
In Table 2, we further ablate on the detailed hierarchical structures in our model.
As we can observe, when we increase the number of Modulated MLP blocks, the prediction performance (at target points)
[Table]
[Figure]
Visualizations.
We visualize the obtained function distributions from stacked ANP (Kim et al., 2019) and our VNP, by sampling the latent variables 20 times.
Given partial observations of a signal, there exists uncertainty on the continuous function since there are many possible ways to interpret these observations (i.e., context set).
An excellent Neural Process model should be able to model such uncertainty through the fitted functions.
In other words, the generated functions conditioned on the context set should approximate the data distribution and cover the target set points.
As shown in
Figure 3, we can see that the generated functions from the stacked ANP cannot predict the target points accurately, e .
g .
, those regions marked by orange.
In contrast, our method provide good approximations for the continuous functions, where the distribution covers the groundtruth function.
5. 2D I MAGES C OMPLETION AND S UPERRESOLUTION
A 2D image can be modeled by a continuous function that maps the 2D pixel coordinates to the color values.
We implement our VNP framework for learning the continuous function of 2D images, which supports tasks such as image completion and super-resolution to arbitrary size.
Settings.
We conduct experiments on CelebA dataset (Liu et al., 2015), mainly with the resized resolution of 64 × 64 .
Due to the limited representation capability or the high requirement on computation resources, most previous NP methods (Kim et al., 2019; Lee et al., 2020) are in general trained and evaluated on relatively low resolution such as 32 × 32 .
Our framework enables the processing of complex signal with higher resolution.
We train a single model that supports image completion and super-resolution tasks.
During training, we control the context ratio as 0.03, which means 3% pixels are taken as context set.
The target ratio is set as 0.15 and the target set has partial intersection with the context set.
More detailed experimental settings can be found in Appendix B.
Visualizations .
We compare our VNP with stacked ANP (Kim et al., 2019) tested with three different context ratios respectively as shown in Figure 4.
Our VNP generates diverse and realistic image completion results with fine details.
In comparison, the results of Stacked ANP are blurred.
In Figure 5, we show that the proposed VNP achieves satisfactory results for image completion and superresolution on CelebA64 dataset.
As a unified framework, with a single model, it can support image completion, superresolution, and the concurrent of completion and superresolution.
[Figure]
CelebA
NLL (lower is better) context ratio = 0.
Stacked ANP 2.
VNP, ks = 1, L K =2. VNP, ks = 1, L K =2.
VNP, ks = 2, L K =2.
VNP, ks = 4, L K =2.
ks is the kernel size in set
L K is the number of Modulated MLP
[Figure]
GFLOPs
CelebA CelebA
[Figure]
Stacked ANP 17.
[Figure] [Figure]
[Figure]
Our VNP 27.
Influence of Patch/Kernel Size .
We conduct an ablation study to investigate the influence of the patch/kernel size in our set tokenizer.
We set the stride the same as the kernel size.
The results are shown in Table 3.
It is observed that using larger kernel size does not degrade the performance obviously.
In addition, using hierarchical structure brings gains.
Complexity Comparisons.
Thanks to our bottleneck encoder design, the proposed VNP is a practical framework for complex signal modeling, where the number of context points as condition is usually large.
For comparison, we calculate the GFLOPs of stacked ANP and our VNP when target ratio is 0.25 on CelebA64 and CelebA178 respectively.
Here, the kernel size of set tokenizer is 4 on
CelebA64 and 10 on CelebA178.
The computational complexity comparisons in terms of GFLOPs are shown in Table 4.
Since we use set tokenizer to reduce the number of tokens and process the context set in image grids, the GFLOPs of VNP would be smaller than that of Stacked ANP, especially when the context ratio is high.
5. V IEW SYNTHESIS ON 3D S CENE
Implicit neural representations excel at representing 3D signals.
However, since 3D signals are usually much more complex, most previous works (Mildenhall et al., 2020; Martin-Brualla et al., 2021) require expensive optimization to fit the neural networks to represent the scenes.
We focus on the task of view synthesis in this section to evaluate our proposed VNP.
[Figure] [Figure]
dB)
Cars
Lamps
Chairs
Deterministic
Learned Init
[Table]
Chen et al., 2021b) 22.
[Table] [Table]
Trans
[Table]
Wang,
[Table] [Table]
[Table] [Table]
Probabilistic
NeRF-VAE
[Table]
Kosiorek et al., 2021) 21.
Our VNP
[Table]
[Table] [Table]
Settings.
We follow the spirits of NeRF (Mildenhall et al., 2020) that fits a network to map the world coordinate into the corresponding RGB values and volume density.
We use bottleneck encoder (tokenization from the image patches) to calculate the adaptive input of the MLPs, queried by target world coordinate.
Then the hierarchically learned global latent variable will modulate the parameters of MLPs to predict the RGB values and volume density.
Note that there is an extra volume rendering process inside each decoding block before the pooling module, because it requires transferring the world coordinate to the image coordinate to compute the latent distribution of z k .
We conduct experiments on ShapeNet (Chang et al., 2015) objects, including three sub-dataset: cars, lamps, and chairs.
More details on the network structures and hyper parameters can be found in Appendix B.
Comparisons.
Our VNP model can generate probabilistic implicit neural representations for previous unseen scene in a single forward pass.
In this 3D task, one prior work NeRF-VAE (Kosiorek et al., 2021) can also generate the scene from the randomly sampled latent variable.
We make quantitative and qualitative comparisons with NeRF-VAE.
In addition, we compare with another two works, including (Chen et al., 2021b) and (Chen & Wang, 2022) that learn implicit neural representations for previous unseen scene using a few steps, although they are not in the family of probabilistic models and thus cannot model distributions of function from priors.
As shown in Table 5, our method achieves the best performance in terms of Peak Signal-to-Noise Ratio (PSNR) on Cars and Lamps and comparable performance with Trans-INR (Chen & Wang, 2022) on Chairs.
The qualitative comparisons in Figure 6a show that our method VNP produces much better prediction of the novel view than NeRF-VAE.
Note that during the optimization, VAE estimates the distribution over the dataset given a sample.
In contrast, NPs estimate the conditional distribution over the labels of the target points given the set of labeled context points, which is more adaptive to the signal.
C ONCLUSION AND D ISCUSSION
The Neural Processes family provides an efficient way for learning implicit neural representations by approximating the function distribution when only partial observation of a signal is given.
We propose an efficient NP framework, Versatile Neural Processes (VNP), that largely increases the capability of approximating functions.
Our bottleneck encoder and hierarchical latent modulated decoder enables strong modeling capability to the complex signals.
Through comprehensive experiments, we show the effectiveness of the proposed VNP on 1D, 2D and 3D signals.
Our work demonstrates the potential of neural process as a promising solution for efficient learning of INRs in complex 3D scenes.
A PPENDIX A: M ORE D ETAILS ON M ODULATED MLP L AYER
The modulated MLP layer used in our paper is similar to the style modulation in Karras et al.
Mathematically, we denote the weights of a MLP layer (1x1 convolution) by W ∈ R d in × d out , where d in , d out , and w ij denote the dimension of input and output, the element in i th row and j th column of W , respectively. We obtain the style vector s ∈ R d in by passing the latent variable z into two MLP layers. The i th element of style vector s i is thus used to modulate the parameter of W as,
[Equation]
(9) where w ij and w ′ ij denote the original and modulated weights, respectively.
The modulated weights are normalized to preserve training stability,
[Equation]
where ϵ is a very small constant to prevent the denominator to be zero.
These two equations describe the mechanism of the modulate MLP used in our method.
A PPENDIX B: D ETAILED E XPERIMENTAL S ETTINGS
Here, we provide the detailed hyper parameters used in all the experiments in Table 6.
Among them, some hyper parameters denote to, L s number of self-attention layers in bottleneck encoder, L c number of cross-attention layers, K number of hierarchical blocks, d the feature dimension through the entire network.
The row of Fourier features describes whether we embed the coordinate into high-frequency embeddings as suggested in (Tancik et al., 2020).
β is the hyper parameter used to balance the losses of reconstruction and KL divergence.
The reconstruction term describes how we calculate the reconstruction loss
E z ∼ q ϕ
( z | D T ) [ − log p θ ( Y T | z, X T , D C )] . 1D synthetic functions
[Equation]
[Table]
In addition, when training for 1D synthetic functions, we will randomly sample 5 to 15 context points and 15 to 25 target points on Matern and RBF kernel.
Training a VNP model for the 1D regression task requires about 5 hours.
Training a 2D VNP model on CelebA64 requires about 24 hours.
Training a 3D VNP model for novel view synthesis takes around 40 hours.
For the inference speed, VNP requires very short time for 1D and 2D tasks.
On 1D regression task, VNP takes 0.285 second for testing a batch with batch size of 2000.
On the CelebA dataset (2D task), our VNP takes around 0.112 second to infer a single batch of size 8.
On the 3D
Cars dataset, our VNP takes 5.28 second to render an image with a novel view (reasons explained in
Appendix C).
All these results are tested with a single V100 GPU.
A PPENDIX C: C OMPARISONS WITH THE O PTIMIZATION - BASED M ETHOD
In this section, we provide the comparison of our VNP and the previous optimization-based method.
We take a representative optimization-based method, SIREN (Sitzmann et al., 2020b) for comparison.
We conduct experiments on the 3D Car dataset Chang et al.
and measure both the inference speed and the PSNR of the novel synthesized view.
Note that since SIREN requires to optimize the network to fit for a specific signal, the iteration time of SIREN is a part of the inference time.
We test the inference speed with a single V100 GPU for both schemes.
The results are shown in Table 7.
It is observed that for the task of novel view synthesis conditioned on a single view, our proposed VNP provides much better prediction performance compared with the optimized-based method SIREN, even if SIREN is optimized for many iterations for a given test image.
One reason is that VNP can learn the dataset prior, which complements many useful information to predict a specific 3D signal.
However, if we optimize the SIREN network to fit the known single view, the network will be initially optimized for the right direction, but will then tend to overfit to this single view after many iterations.
As a result, it is observed that when the SIREN network gets better prediction performance as the iteration number increased from 1 to 100.
However, the performance decreases if we apply more iterations (such as 300 iterations), compared with that of 100 iterations.
As for the inference speed, we can see that our VNP takes 5.28s to render an image in the novel view with a single forward pass.
The reason for such a long inference time is that we use a large number of sampling points in 3D space.
If an image is with the resolution of 128 × 128 , to render the RGB value of every pixel, there are 128 × 128 × p target points, where p is the number of sampled point along each ray (set as p = 128 in our experiment).
Therefore, processing all these target points with cross-attention requires huge GPU memory.
We have to divide the image into several pixel groups, and render these pixel groups sequentially.
Currently, we have not implement any optimization of our code.
We will improve the implementation of method on tasks involving 3D signals in the future.
Note our VNP only presents slow inference speed on this 3D novel view synthesis task.
On the 1D and 2D tasks, since there are not so many target points and the model does not incorporate rendering process, the inference speed of our method is fast.
[Table]
A PPENDIX D: L IMITATION
The proposed VNP inherits the limitation of NP family.
NPs are interesting techniques for metalearning implicit neural representations due to their reduction of the high cost of training.
NPs learns the common knowledge shared by the dataset, enabling fast inference of an unseen signal without the need of finetuning.
NPs still cannot work well for dataset including diverse objects (with less shared knowledge), e.g., ImageNet.