Generalised Implicit Neural Representations
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider the problem of learning implicit neural representations (INRs) forsignals on non-Euclidean domains.
In the Euclidean case, INRs are trained ona discrete sampling of a signal over a regular lattice.
Here, we assume that thecontinuous signal exists on some unknown topological space from which we samplea discrete graph.
In the absence of a coordinate system to identify the samplednodes, we propose approximating their location with a spectral embedding of thegraph.
This allows us to train INRs without knowing the underlying continuousdomain, which is the case for most graph signals in nature, while also making the
INRs independent of any choice of coordinate system.
We show experiments withour method on various real-world signals on non-Euclidean domains.
Introduction
Implicit neural representations (INRs) are a class of techniques to parametrise signals using neuralnetworks [48, 41, 45, 34].
INRs are trained to map each point in a given domain to the correspondingvalue of a signal at that point.
For example, INRs for images learn to map the 2D coordinates ofpixels to their corresponding RGB values.
INRs can also be conditioned on a latent vector, typicallylearned end-to-end as part of the model, that allows them to represent different signals on the samedomain.
INRs have been successfully applied to model complex signals like images [48, 45], signeddistance functions [41], and radiance fields [40].
By expressing a signal as an INR, we obtain a continuous approximation of the signal on the wholedomain.
This allows us to compute a higher-resolution approximation of the signal by sampling morepoints on the domain ( e.g. , a finer grid of pixels).
Additionally, since INRs are fully differentiable, the latent vector of a conditional INR can be optimised with backpropagation and gradient descent toobtain a signal with some desired characteristics.
For example, a conditional INR for signed distancefunctions can be used to design surfaces with a target aerodynamic profile [42].
So far, the literature on INRs has only focused on signals on Euclidean domains.
However, signalsdefined on non-Euclidean domains are ubiquitous in nature and are especially relevant in artificialintelligence, as demonstrated by the recent rise of geometric deep learning [12].
In this paper, wepropose an extension of the INR setting to signals on arbitrary non-Euclidean domains.
Contributions
We formulate the generalised INR problem as the task of learning an implicitrepresentation for a signal on an arbitrary topological space M where, instead of observing the signalsampled on a regular lattice, we observe a graph ( i.e. , a discretisation of M ) and the correspondinggraph signal.
In most practical cases, M is unknown and we cannot represent the sampled verticesin a coordinate system.
Even when M is known (as in the Euclidean case), training an INR ona fixed coordinate system means that the model will depend on this choice.
We solve both theseissues by identifying the sampled nodes with an intrinsic spectral embedding obtained from theeigenvectors of the graph Laplacian.
We then train a neural network to map the spectral embeddingsto the corresponding signal values.
Figure 1 shows a schematic view of the method.
Since the eigenvectors of the graph Laplacian are a discrete approximation of the continuous eigenfunctions of the Laplace operator on M (when appropriately rescaled) [5, 10, 7], at inference timewe can map arbitrary points on M to the corresponding approximation of the signal, as long as thediscrete graph signal is sampled consistently.
This allows us to compute higher-resolution signals orto estimate the signal on different graph realisations of the same phenomenon ( e.g. , different socialnetworks with a similar structure).
Results
In the experiments section, we show concrete examples of learning INRs for signals ongraphs and manifolds, using real-world data from biology and social networks.
We also show that
INRs trained on one instance of a graph can be transferred to different graph realisations of theunderlying domain.
Then, we show applications of conditional generalised INRs to model differentsignals on the same irregular domain, and also different signals on different domains.
Finally,we conclude the paper with an experiment that consolidates all our results, modelling real-worldmeteorological signals on the spherical surface of the Earth.
Related works
INRs have recently attracted the attention of the machine learning community for their abilityto represent complex signals, especially in computer vision, 3D rendering, and image synthesisapplications [41, 38, 2, 22, 40, 46, 32].
Recent work has highlighted that INRs benefit from computingsinusoidal transformations of the coordinates, typically called Fourier features or positional encodings(PEs) [52, 50, 40, 53, 8]. Since the PEs typically used in INRs are also the eigenfunctions of the
Laplace operator in Euclidean domains, our method is a generalisation of this approach.
However, these transformations are not applicable to non-Euclidean domains and still depend on achoice of coordinate system.
A proposal for making INRs equivariant under the SO(3) group has beenpresented by Deng et al.
[15], although this technique does not apply in general to non-Euclideandomains.
The field of geometric deep learning has also explored the use of graph PEs similar to theones we use in this paper, showing their usefulness as node features for graph neural networks [47,36, 17, 16].
Recent research has also investigated the idea of learnable graph PEs [18, 13].
Related to our work, Belkin and Niyogi [5], Boguna et al.
[11] and Levie et al.
[30] have studiedthe geometrical link between graphs and topological spaces.
Keriven et al.
[25, 26] investigated theconvergence, in the limit, of graph neural networks trained on large graphs.
An alternative view ofgraphs as discrete samples of continuous spaces is the concept of graphon, which can be seen both asa generative model and as the limit of a sequence of graphs [37].
We also mention the work of Koestler et al.
[28], who propose a similar idea to ours for learning fieldson meshes. In this work, the authors train a neural network to map the eigenfunctions of the Laplacianto an RGB texture. By contrast, we focus on a wider range of tasks and experiments, includingtransferring to completely different graphs, conditioning the INRs, and modelling dynamical systems. Our scheme to compute generalised embeddings is also different from Koestler et al.
’s, not relying onknowing node coordinates.  MethodStandard setting In the standard INR setting, we consider a signal f : X → Y with X ⊆ R d and
Y ⊆ R p .
We observe a discrete realisation of the signal f ( x i ) for i = 1 , . . . , n , where points x i aresampled on a regular lattice on X .
Then, we train a neural network f θ : X → Y , with parametersθ , on input-output pairs ( x i , f ( x i )) .
Since we know that the signal domain is a subset of R d , atinference time we can sample points anywhere on X to compute the approximated value of the signalat those points.
For example, an INR for images maps equispaced points in the unit square (pixelcoordinates) to points in the unit cube (RGB values normalised between 0 and 1).
The specific imageon which we train the INR is a realisation of one such signal at a given resolution, and at inferencetime we can sample a finer lattice to super-resolve the image [48, 45].
Generalised setting
In the generalised setting, we consider a continuous signal f : M → Y , with M an arbitrary topological space.
We observe a discrete graph signal f ( v i ) on an undirected graph G =
( V, E ) , with node set V = { v i } for i
= 1 , . .
. , n and edge set E ⊆
V × V .
Note that the graphcan be weighted if a metric is available on M . The meaning of sampling G from M is intuitive inthe case of geometric meshes or other physical structures like proteins (in which case we also knowthe coordinates of v i ), but the same reasoning also applies to more complex domains with an abstractmeaning ( e.g. , the space of all possible papers and their citations).
We generally assume that thegraph describes some measure of closeness between uniformly sampled points on M , be it a functionof the coordinates ( e.g. , a kernel) or some logical or functional relation that is given as part of thedata.
In general, we don’t assume to know the true M from which G is sampled or the coordinates ofv i .
We only observe G and the associated graph signal.
For an in-depth discussion on the relationbetween graphs and topological spaces, see references [5], [11], and [30].
Proposed method
We approach the generalised INR problem by mapping v i to f ( v i ) through aspectral embedding obtained from the graph Laplacian.
Let A ∈ R n × n be the weighted adjacencymatrix of graph G , D the diagonal degree matrix, and
L = D − A the combinatorial Laplacian.
Theeigendecomposition of the Laplacian yields an orthonormal basis of eigenvectors { u k ∈ R n } fork = 1 , . . .
, n , with a canonical ordering given by their associated eigenvalues λ 1 ≤ λ 2 ≤ · · · ≤ λ n .
We define a generalised spectral embedding of size k for node v i as
[Figure] [Equation]
[Figure] [Equation] [Figure] [Equation]
[Figure]
[Equation]
[Figure]
The generalised spectral embeddings are a discreteapproximation of the eigenfunctions of the continuous Laplace-Beltrami operator on M and convergeto them for n → ∞ .
See the discussion in references [5], [10], [9], and [7] for a formal analysis.
Inpractice, rescaling the eigenvectors by √ n ensuresthat the embeddings for graphs of different sizes willhave the same range component-wise and that similarnodes will have similar embeddings regardless of thegraph size.
As an example, we show a comparisonbetween the eigenvectors and the generalised spectralembeddings of a path graph in Figure 2.
We train a neural network f θ : e i (cid:55)→ f ( v i ) on theobserved graph signal.
Since the generalised spectralembeddings are an intrinsic property of the graph, the INR will not depend on any choice of coordinatesystem but only on the topology of the underlying continuous domain.
At inference time, we can compute the approximated value of the signal at an arbitrary location on Mby computing the associated spectral embedding.
If we know M , or if we estimate it, we can samplenew vertices directly from the domain and apply the same procedure used to construct the traininggraph.
If M is unknown ( e.g. , in the case of natural graphs that describe some abstract concept likecitations, social interactions, or biological relations), then we just assume to observe a similar graphsampled from M .
An important difference between generalised INRs and Euclidean INRs is thatgenerally we must observe the full graph in order to compute the inputs.
This is necessary because,while in the Euclidean case we completely know the domain of the signal a priori , in the generalisedcase we need to estimate the topology of the domain by sampling.
Complexity
Although computing the full eigendecomposition of the Laplacian has a complexityof O ( n 3 ) , here we are only interested in the first k eigenvectors.
Also, the Laplacian of most graphsis very sparse, with nodes having an average degree ¯ d ≪ n .
We can therefore use the implicitlyrestarted Lanczos method for eigendecomposition implemented by the ARPACK software, whichhas a complexity of O ( ¯ dn 2 ) and can be easily parallelised [29].
In practice, all computations for thispaper were easily managed on a commercial laptop with 10 CPU cores, scaling up to graphs in theorder of 10 5 nodes.
We also note that at inference time, depending on how the edges are constructed, it could be possibleto add new nodes and edges to the training graph instead of sampling an entirely new graph from M .
In this case, the spectral embeddings for the new nodes can be estimated using the Nyström methodwithout needing to compute the full eigendecomposition [3, 9].
Alternative approaches
The generalised Laplacian embeddings used in this paper are only one ofmany possibilities to represent nodes sampled from M .
To name a few, locally linear embeddings [43],
Isomap [51], Laplacian eigenmaps [6] and diffusion maps [14] are all based on the idea of embeddingdata using the first few principal eigenvectors of a similarity matrix.
Here we focus on the Laplaciansince it is well known, sparse, and easy to compute, and its eigendecomposition is stable to graphperturbations.
One disadvantage of Laplacian eigenvectors is that they are only unique up to sign,leading to 2 k possible eigenbases for a given M and the consequent ambiguity when transferringan INR to a different graph realisation.
Additionally, eigenvalues of multiplicity greater than 1also introduce ambiguity in the eigenvectors, since all rotations and reflections of the associatedeigenspace are valid choices.
However, simple heuristics can be used to eliminate sign ambiguity,and we did not encounter any other practical issue in this regard.
Alternative ways to resolve theambiguity is to use PEs based on random walks [18, 39, 31], heat kernels [49, 19], or the more recentsign-and-basis invariant neural networks [33].
We leave the exploration of these alternatives to futurework, since they do not significantly impact our main contributions.
Experiments
Setting
We implement the generalised INR as a SIREN multi-layer perceptron [45].
The model has6 layers with 512 hidden neurons and a skip connection from the input to the middle layer.
We use thesame hyperparameters and initialisation scheme suggested by Sitzmann et al.
We train the modelusing Adam [27] with a learning rate of 10 − 4 and an annealing schedule that halves the learning rateif the loss does not improve for 1000 steps.
At each step, we sample 5000 nodes randomly from thegraph as a mini-batch.
We use spectral embeddings of size k = 100 , a value that we found empiricallyas a good trade-off between complexity and performance.
We report an analysis on the effect of k in
Section 4.1.
We ran all experiments on an Nvidia Tesla V100 GPU.
The code to reproduce our resultsand the high-resolution version of all figures are available at https://github.com/<anonymous> .
All deviations from the default setting are documented in the supplementary material, where we alsoreport an additional experiment on solving differential equations with generalised INRs.
4.1 Learning generalised INRs
We begin by verifying that a typical INR architecture can successfully learn arbitrary graph signals inthe generalised setting.
We consider three datasets as test cases.
Bunny
We generate a texture on the Stanford bunny mesh 1 using the Gray-Scott reaction-diffusionmodel given by:
[Equation]
for state vectors a , b ∈ R n and parameters D a , D b , F, K ∈ R , where ⊙ indicates elementwise product [21].
The mesh has 34834 nodes and 104288 edges.
We configure the systemto yield “coral” patterns 2 and evolve the simulation for 10 4 steps starting from random positive values for a , b .
At the end of the evolution, we use vector a as target graph signal. [Table]
Protein
As a real-world domain, we considerthe solvent excluded surface of a protein structure.
The continuous signal is the value of theelectrostatic field generated by the amino acidresidues at the surface.
Protein function can belargely understood by studying their surfacesand the corresponding chemical features, likethe electrostatic charge, so this represents a potentially interesting application of INRs to biology.
We use a combination of the MSMS [44] and APBS [24] software packages to estimate the surface,sample the nodes, and get the corresponding signal, following the same workflow of Gainza et al.[20].
The final graph has 11966 nodes and 35892 edges. [Table]
US election
Finally, we consider a social network dataset introduced by Jia and Benson [23],in which nodes represent United States counties and edges are estimated using the Facebook
Social Connectedness Index.
The target signalrepresents the county-wise outcome of the {+
+} US presidential election, as values in the range[ − 1 , 1] (indicating the proportion of votes infavour of one candidate vs. the other).
The graph has 3106 nodes and 22574 edges.
Results
In all cases, the generalised INR achieves an R 2 close to 1 indicating that the model canindeed learn non-trivial signals on non-Euclidean domains.
We report the R 2 and mean squared errorfor all datasets in Tab. 1.
We compare the true graph signals and those learned by the INR, for thebunny and protein, in Fig. As a second test, we compare typical INRs trained on node coordinateswith generalised INRs trained on the spectral embeddings. We consider the bunny and protein sincethey have input coordinates for the typical INRs. We also test different activation functions—ReLUor SIREN.
We report in Tab. 2 the R 2 for a held-out set of nodes (to evaluate whether the INRs areoverfitting instead of learning a meaningful representation).
4 All results are averaged over 5 runsand have negligible standard deviations.
We see that the generalised INRs perform almost alwaysbetter than the typical ones. Note that these experiments only verify that the model can indeed learnthe target signals.
We will test the transferability of the model in-depth in Sec. 4.2.
In the followingsections, we mostly focus on known spaces M or synthetic graphs so that we can better answer otherresearch questions by having full control of the data.
Size of the spectral embeddings
Typical INRs take as input the d -dimensional coordinates ofpoints on the sampled lattice.
In our general setting, the dimension k of the spectral embeddings is ahyperparameter of the method.
To evaluate the impact of k on the performance of the INR, we trainthe same model for different values of k on the Stanford bunny.
We report the results in Figure 4, showing that the model fails to learn a meaningful representation for k ≤ 3 .
This is reasonable because at least 3 non-trivial eigenvectors areneeded to correctly distinguish the main structural features of the bunny (ears, tail, etc.), asshown in Figure 5.
We also see that, while the INR has no issues in learning the signalfor k = 100 , for lower values of k the model struggles to represent the signal on the ears. [Figure]
Figure 5 again provides a possible explanation for this result,since the first few eigenvectors tend to collapse on the ears andhigher frequency eigenvectors are needed to distinguish pointstowards the narrow tips.
We will show another consequenceof this in Section 4.2.
We also mention that the drop in performance for lower values of k on the other two datasets is lesspronounced than on the bunny, likely because of their lack ofany high-frequency features.
4.2 Transferability of generalised INRs
Because the generalised spectral embeddings are a discreteapproximation of the continuous eigenfunctions of the Laplaceoperator on M , we can apply a trained INR to a different graphrealisation of M as long as its spectral structure is consistent with the training data.
In practical terms, we need to know 1) how to sample new nodes and 2) how the nodes are connected.
If we know M , then the problem simply boils down to choosing a sampling strategy for the pointsand a suitable measure of closeness ( e.g. a kernel or edge-generating function like the k-nearestneighbours algorithm).
If M is unknown, then we assume to observe a new graph through the sameprocess that generated the training data ( e.g. , we observe new social interactions among the same or asimilar set of people).
Note that, when sampling a new graph from M , the spectral embeddings will be a slightly different(possibly better) approximation of the continuous eigenfunctions.
As this effect is more evident onhigh-frequency eigenvectors, we empirically observed that training the INR with a smaller k and ReLU activations improved its transferability ( cf. Table 2).
See the supplementary material for moredetails.
We investigate the transferability of generalised INRs in two settings.
Transferring to similar graphs
As a first example, we study the case in which M is unknownand we only observe the discrete graphs.
We consider a stochastic block model (SBM) with twocommunities of equal size (in total, n = 1000 ), parametrised by an inter-connection probability pand an intra-connection probability r .
We train an INR on a graph with a strong community structure ( p = 0 .
1 , r = 0 . 5 ) and a graph signalindicating the community of each node (effectively a node classification task).
Then, we test the INRon graphs sampled in the ranges p ∈
[0 . 1 , 1] and r ∈ [0 . 1 , 1] to test the robustness of the model tochanges in the community structure.
We report our results in Figure 6.
First, we note that perfectly recovering the correct clusters is onlytheoretically guaranteed in certain regions of the parameter space [1], delimited with an orange linein the figure.
We see that the trained INR transfers perfectly to all graphs sampled in this region andthat its performance sharply drops only for p > r , where graphs have no community structure (see
Figure 6 a, b).
Additionally, we see that the INR maintains a good performance also in the regionbetween the theoretical boundary and the line p = r , i.e. , a region in which graphs still exhibit somecommunity structure although perfect recovery is not guaranteed.
Overall, this indicates that trainingthe INR on an instance of a graph allows us to transfer it to graphs with a similar structure.
[-We consideragain the Stanford bunny mesh with the same texture graph signal as Section 4.1 and we apply Loop’smethod for mesh subdivision to obtain a higher-resolution mesh with 139122 nodes and 
-]
0 .0 .0 .1 .p
(inter) 0.
r ( i n t r a ) 0. uu(a) p = 0 .
1 , r = 0 .uu(b)
p = 0 .
5 , r = 0 .
Squared error
[Figure]
i t
[Figure]
As a second test case, we consider the super-resolution of a signal. We consideragain the Stanford bunny mesh with the same texture graph signal as Section 4.1 and we apply Loop’smethod for mesh subdivision to obtain a higher-resolution mesh with 139122 nodes and 416929edges [35]. Then, we compute the spectral embeddings for the new graph and predict the signal withthe INR trained on the low-resolution graph.
Because the unit-norm eigenvectors are only unique up to sign, the INR will transfer to a differentgraph only if its eigenvectors are aligned to the training ones ( i.e. , similar nodes must have similarspectral embeddings — in the previous experiment, we did not have to deal with this problem due tosymmetry in the SBM).
However, in many cases, a simple comparison of the eigenvectors’ histogramsprovides a useful heuristic for automatically aligning them. For simplicity, here we also manuallyverify the alignment of the two eigenbases manually.
We show in Figure 7 a qualitative comparison between the training and super-resolved signals,from which we see that the INR can predict the graph signal correctly.
Quantitatively, since meshsubdivision does not remove the original nodes from the mesh, we can compute the R 2 between thetraining signal and part of the super-resolved signal.
The model achieves an R 2 of 0.39 compared to atraining R 2 of 0.99.
However, this apparent drop in performance is mostly due to the high-frequencyears of the bunny, for which the model evidently overfits and cannot tolerate even small changesin the eigenvectors.
Figure 8 shows the distribution of the squared error between the ground truthand predicted signal, from which we see that the error is close to zero for most of the nodes (notethat the y-axis is logarithmic).
By ignoring the nodes above the 90th percentile of the squared errordistribution, the R 2 score is 0.94.
Increasing k resulted in a better prediction of the ears region ( cf.
Figure 4) at the cost of worse overall transferability. [Figure]
4.3 Conditional generalised INRs
We also consider the setting in which a generalised INR is conditioned on a latent vector z ∈ R qto allow the representation of different signals by the same neural network f θ ( e i , z ) .
Unlike the
Euclidean setting, here we can investigate two cases: 1) learning a conditional INR for differentsignals on the same domain (the typical setting), and 2) learning a conditional INR for differentsignals on different domains.
Reaction-diffusion process
For the first case, we train a generalised INR to approximate the
Gray-Scott reaction-diffusion process that we used to generate the graph signal in Section 4.1, givenby Equation (2).
We consider a signal f
( v i , t ) , where t ∈ N is the number of updates since the startof the simulation.
We evolve the system for 3000 steps starting from the same random initialisationused in Section 4.1 and we train the INR f θ ( e i , t ) using 300 time steps sampled at t ≡ 0 (mod 10) .
The graph is fixed throughout the evolution, so the INR must learn to exploit the conditioning input tto correctly predict the signal at different times. [Figure]
At test time, we predict the signal for all the remaining time stepsand compare the prediction with the ground truth signals.
The INRachieves a test R 2 of 0 .
999 ± 0 .
006 indicating that it successfullylearned to approximate the true dynamics.
We report in Figure 9 afew samples predicted by the INR at test time.
By comparison, asimple baseline using linear interpolation between training samplesachieves a similar R 2 , while requiring to keep all 300 training stepsin memory. The INR, on the other hand, represents the signal and itsevolution with a single set of weights and a scalar control parameter.
Multi-protein INR
For the second case, we consider a dataset
D = { ( M m , f m ) } of protein surfaces M m with associated electrostatic signals f m : M m → R for m = 1 , . .
. , |D| , obtainedfollowing the same protocol as Section 4.1.
We configure the neuralnetwork f θ ( e i , z m ) as an autodecoder [41] where z m ∈ R
z is a vector of free parameters learnedend-to-end alongside the weights of the neural network.
We assign a separate vector z m to eachtraining sample, allowing the model to find the best latent representation for the dataset.
Since thecontinuous surface and the associated signals are unique to each protein in the dataset, the latentvector z m must capture both aspects.
We set the latent vector size z = 8 , which we empiricallyfound to give consistently good performance across datasets.
Larger values of z did not significantlyimprove the performance.
We test the ability of the model to represent datasets of different sizes, by sampling random proteinsfrom the dataset used in reference [20].
Figure 10 shows the R 2 of the model for different valuesof |D| .
Understandably, the performance decreases as the number of samples grows, although themodel achieves R 2 ≥ 0 .
9 for datasets of up to 500 different proteins.
Overall, these results indicatethat the model can represent many domains and signals with a single set of weights.
[Figure]
An animated and uncompressed version of this figure is available in the supplementary material.
4.4 Weather modelling
As a final experiment, we consolidate all results from previous sections to showcase an applicationof generalised INRs in a real-world setting.
We consider signals f ( v i , t ) representing hourly meteorological measurements on the spherical domain of the Earth’s surface.
Our goal is to train aconditional INR at a given spatial and temporal resolution and then use it to super-resolve the signalover space and time.
Data
We collected data from the National Oceanic and Atmospheric Administration (NOAA)
Operational Model Archive and Distribution System, specifically from the Global Forecast System(GFS).
The data consists of three different signals representing: the dew point temperature 2mabove-ground measured in K ( dpt2m ), the atmosphere’s total cloud cover percentage ( tcdcclm ), andthe surface wind speed in m/s ( gustsfc ).
We consider a 24-hour period sampled at 1-hour incrementsover an equiangular spherical grid sampled at 1° voxels.
6 We generate a spherical mesh from the65160 locations of the grid by computing their convex hull with the Qhull software [4], for a total of193320 edges.
Results
We train a conditional INR f θ ( e i , t ) at the given spatial and temporal resolution, fort = 0 , . .
Then, we evaluate its predictions on a high-resolution spherical mesh with 258480nodes and 773280 edges (obtained with the same procedure of Section 4.2), and at time incrementsof 30 minutes.
We report in Figure 11 the prediction of the INR for a random test time, on both theoriginal mesh and the high-resolution one.
We refer the reader to the animated version of the figureavailable in the supplementary material for a better appreciation of the results.
Overall, our resultsconfirm that generalised INRs can model complex real-world signals on non-Euclidean domains,learning a realistic continuous approximation of the signals from low-resolution samples.
Conclusion
We have presented the problem of learning generalised implicit neural representations for signalson non-Euclidean domains.
Our method learns to map a spectral embedding of the domain to thevalue of the signal, without relying on a choice of coordinate system.
We have shown applications ofour method on biological, social, and meteorological data, highlighting the potential usefulness ofsuch INRs in a wide range of scientific fields.
We hope that future work will explore more real-worldapplications of our method.