Diffusion Curvature for Estimating Local Curvature in High Dimensional Data
Anonymous Author(s)
AffiliationAddress email
Abstract
We introduce a new intrinsic measure of local curvature on point-cloud data calleddiffusion curvature .
Our measure uses the framework of diffusion maps, includingthe data diffusion operator, to structure point cloud data and define local curvaturebased on the laziness of a random walk starting at a point or region of the data.
We show that this laziness directly relates to volume comparison results from
Riemannian geometry.
We then extend this scalar curvature notion to an entirequadratic form using neural network estimations based on the diffusion map ofpoint-cloud data.
We show applications of both estimations on toy data, single-celldata and on estimating local Hessian matrices of neural network loss landscapes.
Introduction
With the advent of high-throughput high-dimensional point-cloud data in many fields includingbiomedicine, social science, physics, and finance, there is an increasing need for methods to extractstructure and patterns.
A key framework for exploring such data is the manifold hypothesis, whichstates that high-dimensional data arise as samples from an underlying lower dimensional manifold.
The diffusion geometry and diffusion map framework first introduced by Coifman et al.
[1] hasproven to be a useful framework for the shape and structure of the data.
Diffusion geometry involvesconverting data into an affinity graph which is then Markov normalized to form a data diffusionoperator.
This diffusion operator allows for the understanding of geometric features of the data suchas manifold-intrinsic distances, and intrinsic manifold eigendimensions (also known as diffusionmaps).
However, curvature, a salient feature of Riemannian geometry, has received less attention.
Being an inherently smooth quantity, its extension to the discrete case is not straightforward.
Themost authoritative definitions of curvature in the discrete setting have been that of Ollivier–Ricci [2]and of Forman [3], which compute curvature on edges of a graph.
By contrast, here we propose amethod of estimating curvature at points or regions in point-cloud data, where there are no inherentedges to consider—one of the first such formulations to our knowledge.
To achieve this, we utilize the connection between data diffusion and volume to define a newformulation of curvature known as diffusion curvature based on the volume growth of graph diffusionstarting from a vertex.
Starting from a unit mass placed at one or several points, we show that incases of positive curvature the volume growth tends to be slower with more of the mass remaining inthe original point, whereas in cases of negative curvature the mass tends to disperse faster to otherpoints.
Therefore the ratio of mass in the original point, compared to the remainder of the cloud givesa scalar value of curvature.
We note that this method avoids the complexity of having to consider all(potentially exponentially many) edges that would arise by simply attempting to directly translatenotions of edge curvature to the data affinity graph.
In addition to the scalar notion of curvature, wealso show that we are able to compute a quadratic form of arbitrarily high dimensions using neuralnetwork training based on diffusion probabilities as the input.
We validate our notion of curvature on toy datasets as well as real single-cell point cloud data ofvery high dimensionality.
Then, we study both the scalar diffusion as well as the diffusion basedquadratic form in studying the area surrounding found minima in neural network loss landscapes.
The Hessian of the parameter space is generally difficult to compute as it contains O ( n 2 ) entriesfor a neural network with n parameters.
However, restricting ourselves to the area around a localminima yields a lower dimensional space, where only a subset of parameter combinations changein the vicinity.
We sample in this lower dimensional space and compute a diffusion operator whichallows us to assess the curvature of the minima.
First, we can assess if the point of convergence isindeed a minimum or potentially a saddle.
Further, studies have shown that flat minima, i.e., minimasituated in a neighborhood of the loss landscape with roughly similar error, tend to generalize betterthan sharp minima.
These qualities can be assessed from curvature estimations of minima regions.
Our key contributions include:
• The formulation of diffusion curvature, an intrinsic scalar valued curvature for regions ofhigh dimensional data sampled from an underlying manifold, computed from data diffusion.•
Proof that our notion of curvature will result in higher values in positive curvature based onvolume comparison results from Riemannian geometry.•
A neural network version that computes a quadratic form starting from a diffusion map.
• Validation of our framework on toy and single cell datasets.• Application to Hessian estimation in neural networks.
Background2.1 High Dimensional Point Clouds and the Manifold Assumption
A useful assumption in representation learning is that high-dimensional data is sampled from anintrinsic low-dimensional manifold that is mapped via nonlinear functions to observable highdimensional measurements; this is commonly referred to as the manifold assumption.
Formally,let M d be a hidden d dimensional manifold that is only observable via a collection of n ≫ dnonlinear functions f 1 , . . .
, f n : M d → R that enable its immersion in a high-dimensional ambientspace as F ( M d ) = { f ( z ) =
( f 1 ( z ) , . . .
, f n ( z )) T : z ∈ M d } ⊂ R n from which data is collected.
Conversely, given a dataset X = { x 1 , . . .
, x N } ⊂ R n of high-dimensional observations, manifoldlearning methods assume data points originate from a sampling Z
= { z i } Ni =1 ⊂ M d of the underlyingmanifold via x i = f
( z i ) , and aim to learn a low dimensional intrinsic representation that approximatesthe manifold geometry of M d .
2.1.1 Diffusion Maps
To learn a manifold geometry from collected point cloud data, we use the popular diffusion maps construction [1].
This construction starts by considering local similarities defined via a kernel K ( x, y ) , with x, y ∈ F ( M d ) , that captures local neighborhoods in the data.
While the Gaussian kernel is a popular choice for K , it encodes sampling density information in its computation.
Hence, to construct a diffusion geometry that is robust to sampling density variations, we use an anisotropic kernel
[Equation]
with ∥ · ∥ 1 the 1 -norm (or row sum) and G ( x, y ) =
e − ∥ x − y ∥ 2 / σ as proposed in [1], where 0 ≤ α ≤ 1controls the separation of geometry from density, with α = [-
-]0 yielding the classic Gaussian kernel,and α = 1 completely removing density and providing a geometric equivalent to uniform samplingof the underlying manifold.
In [1], it is shown that in the limit of infinitely many points this becomesequivalent to a Laplace–Beltrami operator, which encodes geometric properties of the underlyingmanifold.
Next, the similarities encoded by K are normalized to define transition probabilitiesp ( x, y )
=
K ( x,y ) /
∥K ( x, · ) ∥ 1 that are organized in an N × N row stochastic matrix P ij = p ( x i , x j ) thatdescribes a Markovian diffusion process over the intrinsic geometry of the data.
Finally, a diffusionmap is defined by taking the eigenvalues 1 = λ 1 ≥ λ 2 ≥ · · · ≥ λ N and (corresponding) eigenvectors{ ϕ j } Nj =1 of P , and mapping each data point x i ∈
X to an N dimensional vector:
[Equation]
Here t represents a diffusion-time.
In general, as t increases, most of the eigenvalues becomenegligible; truncated diffusion map coordinates can thus be used for dimensionality reduction, and
Euclidean distances in this space are a manifold-intrinsic distance [1, 4].
Note that the trivialeigenvector, ϕ 1 (with corresponding eigenvalue λ 1 = 1 ), is often omitted since it represents thestationary distribution of the Markov random walk and does not contain information about the data.
The eigenvalues of the Laplace-Beltrami operator, as well as the eigenvalues of the diffusion map,encode geometry information about a compact Riemannian manifold [1], as can be seen via theasymptotic expansion of the trace of the heat kernel [5].
Dimension, volume, and total scalarcurvature are spectrally determined properties of the manifold.
Given the relationship betweendiffusion coordinates and the eigenvectors of the Laplace-Beltrami operator, one can surmise that thediffusion operator also encodes geometric information.
These relations motivate our use of diffusionto measure curvature on a data manifold.
Section A5 in the Appendix contains additional details inthe Riemannian setting.
2.2 Discrete Curvature in Riemannian Geometry
There is a long history in Riemannian geometry of using curvature to study geometric and topologicalproperties of a Riemannian manifold.
In particular, lower bounds on Ricci curvature have been relatedto diameter (Bonnet-Myers [6]), volume (Bishop-Gromov [7]), the Laplacian (Cheng-Yau [8]), theisoperimetric constant (Buser [9]), and topological properties of the manifold (Hamilton’s Ricciflow [10]).
Although somewhat paradoxical, there has recently been work to extend these smooth
Riemannian ideas to the discrete setting, in particular to graphs and Markov chains.
Especiallynotable in this regard is the definition of discrete Ricci curvature due to Ollivier which makes use ofthe transport distance between probability distributions.
Ollivier’s Ricci curvature [2] starts with a metric space X equipped with a random walk (a probabilitymeasure m x ( · ) for each x ∈ X ) and assigns the edge-wise scalar curvature k OR ( x, y ) =
1 − 102 W 1 ( m x ,m y ) /
d ( x,y ) where W 1 ( · , · ) is the L 1 transportation distance (also known as earth mover’sdistance or Wasserstein distance).
In the Riemannian setting, if one defines the random walk to bedm rx ( y ) = dvol ( y ) /
volB ( x,r ) then Ollivier demonstrates that k OR ( x, y ) =
r 2 Ric ( v,v ) / 2 d +2 + O
( r 3 +d ( x, y ) r 2 ) where v is a unit tangent vector,
and y is a point on the geodesic issuing from v withd ( x, y ) small enough.
Ollivier (and others [11], [12]) have used lower bounds on this notion ofcurvature to study global properties of the space (i.e. diameter, volume growth, spectral gap).
The main downsides of Ollivier’s Ricci curvature are that it is an edge-wise notion and that it requiresknowledge of the transport distance.
A priori point cloud data does not come with a graph structure.
We therefore propose a diffusion based method for prescribing a point-wise scalar curvature to pointcloud data.
Like Ollivier’s notion, this diffusion curvature is based on the idea that the spread ofgeodesics is influenced by Riemannian curvature.
We prove theoretical bounds on the diffusioncurvature by appealing to the Bishop-Gromov volume comparison theorem.
2.3 The Loss Landscape and Hessian of Neural Networks
In the context of neural networks, second-order information about the loss function contains usefulinformation about the generalizability of minima, the heuristic being that flat minima generalize betterthan sharp minima. Previous studies have sought to directly examine the loss landscape of a neuralnetwork. Li et al. [13] used random samples around found minima to visualize and quantify
sharpnessof minima, Horoi et al.
[14] utilized jump-and-retrain sampling to visualize and classify “good” and“bad” minima.
However, computing the full Hessian of the parameter space is only possible in specialcases such as variational quantum classifiers [15] and it is generally intractable for high-dimensionalsystems (such as the parameter space of a neural network).
Consequently, previous work has soughtto approximate the largest (and smallest) eigenvalues of the Hessian [16, 17], yielding an incompletepicture.
To circumvent this, we estimate a low dimensional Hessian of the loss function around itsoptimum using a quadratic approximation.
Restricting ourselves to the area around the optimumyields a lower dimensional space, where only a subset of parameter combinations change in thevicinity.
This Hessian approximation is achieved by sampling around the optimum, constructing adiffusion operator based on these sampled points, and using a neural network to learn the coefficientsof the quadratic approximation.
Neural networks are trained such that their output function f
( X, θ ) matches a given function y ( X ) on training data X by performing, e.g., stochastic gradient descent on a loss function comparing y and f with respect to the parameters θ .
A typical example of a loss function is mean squared error
L ( X, θ ) = (cid:80) x ∈ X
|| f ( x ) − y ( x )
|| 2 .
The loss function defines a loss landscape for fixed training data
X .
Here, we use data diffusion to directly estimate the Hessian of the loss landscape, with respect to the parameters of a neural network, around found minima, providing a quantitative measure of the region in terms of its spectrum and condition number.
In general, the Hessian of a scalar-valued function, L ( θ ) is a symmetric matrix of partial derivatives with the quadratic form:
[Equation]
 
[Equation]   Since the gradient is zero at critical points, i.e., ∇L ( θ 0 ) = 0 , the quadratic approximation to thefunction around its critical points is given by: L ( θ ) ≈ L ( θ 0 ) + 1 / 2 H L ( θ − θ 0 ) .
We see thatthe signature of the Hessian (the signs of its eigenvalues) precisely classifies the critical points( ∇L ( θ 0 ) = 0) as a local maximum (all eigenvalues are negative), minimum (all eigenvalues arepositive) or saddle (eigenvalues have mixed signs).
Computing a local Hessian yields rich informationabout the local curvature around the optimal point in the neural network parameter space.
Methods
3.1 Diffusion Curvature
We now illustrate how diffusion can be used to measure the relative spreading of geodesics under the influence of Riemannian curvature.
To build intuition, we first discuss the case of surfaces (dimension n = 2 ).
Three canonical surfaces are the sphere, the cylinder, and the saddle.
The Gaussian curvature of these surfaces are positive, zero, and negative, respectively.
Imagine taking a sticker and trying to adhere it to one of these surfaces.
The sticker, having been printed on a flat piece of paper, has zero Gaussian curvature.
It will adhere perfectly to the cylinder, will bunch up (there will be too much sticker material) when trying to adhere it to the sphere, and will rip (there will be too little sticker material) when trying to adhere it to the saddle.
This example showcases the area comparison definition [18, pp.
225–226] of Gaussian curvature, k , where one computes the limiting difference between the area A ( r ) of a geodesic disk on the manifold and a standard Euclidean disk,
[Equation]
For example, Gaussian curvature will be positive when πr 2 > A ( r ) , i.e., when the area of the stickerexceeds the area of the corresponding geodesic disk on the sphere.
The Bishop-Gromov volumecomparison theorem formalizes how to extend this into dimensions n > 2 .
Here we will compareto M nk , the complete n -dimensional simply-connected Riemannian manifold of constant sectionalcurvature k , i.e the sphere of radius 1 / √ k when k > 0 , Euclidean n -space when k = 0 , and theappropriately scaled version of hyperbolic n -space when k < 0 .
Theorem 1 (Bishop-Gromov) .
Let ( M n , g ) be a complete Riemannian manifold with Ricci curvaturebounded below by ( n − 1) k .
Let B ( p, r ) denote the ball of radius r about a point p ∈ M and¯ B ( p k , r ) denote a ball of radius r about a point p k ∈ M nk (as defined above).
Then ϕ
( r ) = {+
+}
Vol ( B ( p,r )) /
Vol ( ¯ B ( p k ,r )) is a non-increasing function on (0 , ∞ ) which tends to 1 as r → 0 .
In particular, Vol ( B ( p, r )) ≤ Vol ( ¯ B ( p k , r )) .
Theorem 1 captures the sticker phenomenon: as curvature increases, the volume of comparablegeodesic balls decreases.
Positive curvature corresponds to geodesic convergence and smallervolumes, whereas negative curvature corresponds to geodesic divergence (spread) and larger volumes.
The discrete nature of the data manifold makes it impossible to compare volumes as the distancescale (radius) goes to zero.
Theorem 2 gives us access to the intrinsic distance between data points,i.e., the diffusion distance, which we then use to define metric balls on the Riemannian manifold M .
Theorem 2 (Coifman et al.
The diffusion map Φ t ( x i ) =
[ λ t 1 ϕ 1 ( x i ) , . . .
, λ tN ϕ N ( x i )]
T embedsdata into a Euclidean space where the Euclidean distance is equal to the diffusion distance D m , i.e. D 2 m ( x, [Equation] ) =
∥
Φ t ( x ) − Φ t ( y ) ∥ 2 (
[Equation]
+ O ( e − αm )) .
We can now define B m ( x, r ) =
{ y ∈ M : D m ( x, y ) ≤ r } ⊂ M to be the ball centered at x ∈ Mwith diffusion radius r .
Let B ( x, r ) denote the set of sampled points from B m ( x, r ) and let | B ( x, r ) |denote the cardinality of this set.
To define diffusion curvature, we generate a random walk from apoint x by diffusing a Dirac based at x , i.e., m x ( · )
=
δ x P t where δ x is the one-hot vector, so thatm x ( y ) =
P t ( x, y ) is the transition probability from x to y .
Definition 1.
The pointwise Diffusion Curvature C ( x ) is the average probability that a random walkstarting from a point x ends within B ( x, r ) after t steps of data diffusion, i.e.,
[Equation]
We can extend this definition to a contiguous region U of the manifold M consisting of neighboringpoints
U = { x j } kj =1 by defining m U ( · ) =
δ U P t , where δ U is the indicator vector on the set U .
We have C ( x ) ∈ [0 , 1 /N ] where N = | B ( x, r ) | with larger values indicating higher curvaturerelative to lower values.
The idea is to use diffusion probabilities to capture the relative spreading ofgeodesics.
Intuitively, a random walker is more likely to return to their starting point in a region ofpositive curvature (where paths converge) than negative curvature (where paths diverge).
More precisely, in negatively curved regions, the random walker can get lost in the various divergent(disconnected) branches, whereas in positively curved regions the paths of the random walker exhibitmore inter-connectivity, so that the return probability of a walk is higher.
We make this idea moreformal in the following:
Theorem 3.
If we sample uniformly from a Riemannian manifold M n with Ricci curvature boundedbelow by k ( n − 1) then C ( x ) ≥ C ( x k ) , where x ∈ M n and x k ∈ M nk (as defined above).
Proof.
From Theorem 1 we have vol ( B m ( x, r )) ≤ vol ( ¯ B m ( x k , r )) and together with uniformsampling this yields | B ( x, r ) | ≤ | ¯ B ( x k , r ) | .
We then have 1 / | B ( x, r ) | ≥ 1 / | ¯ B ( x k , r ) | so that evenwith uniform transition probability measures we achieve the desired inequality.
Moreover, as theprobability measure m x ( · ) is constructed by diffusing a Dirac according to affinities based on a
Gaussian kernel, the measure decays with relative diffusion distance from x .
Hence for sets centeredat x , larger cardinality implies lower average transition probability.
We may therefore conclude that
[Equation]
Limitations.
While diffusion curvature can be computed easily, it also necessitates an accuratesampling procedure, which can be problematic for difficult spaces.
Moreover, due to its formulation,diffusion curvature only provides relative (non-negative) values.
3.2 Neural Network Implementation of Quadratic Forms
In the previous section, we have shown that the diffusion operator and associated diffusion probabilities contain information for computation of curvature from point cloud data.
In this section, weextend this measure from a scalar quantity to an entire quadratic form by training a neural network onthe diffusion map embedding of the point cloud.
A schematic of our neural network, called CurveNet,is shown in Figure 2.
CurveNet takes as input points { X i =
( x i , y i ) } Ni =1 sampled near a local minimum or saddle point( x s , y s ) .
Here x i ∈
R k are input parameters for an objective function or loss, f , with y i =
f ( x i )and ∇ f ( x s )
= 0 .
We map each data point, X i , to a N dimensional diffusion vector, Φ t
( X i ) =[
λ t 1 ϕ 1 ( X i ) , . . .
, λ tN ϕ n ( X i )] T .
Recall from Equation 1 that the diffusion map is an eigendecomposition of the diffusion operator
P t .
Hence, without reduction of dimensionality, it encodes all information present in the diffusionoperator, and contains N dimensions for a sampling of N points.
Here we utilize this to estimate ak × k quadratic form of the sampled point cloud, y i
= x T i ˆ Q x i , using a neural network.
Note thatwhen
the point cloud is a sample of a local region in the manifold, this gives a localized quadraticform describing 2 nd order information in the region sampled.
Since our focus is on local quadratic approximation, we train CurveNet on samples from idealizedsurfaces of the form f ( x ) = x T Qx where Q is a symmetric matrix with dimensionality dependenton the cardinality of the point cloud; see Figure 3 for some examples of generated 2 -D surfaces.
Wegenerate a series of such surfaces by varying parameters within Q and sampling points.
We thenprovide the diffusion map embedding of the sampled points to the neural network and train it topredict Q using a mean-squared error loss.
We also include an L 1 regularization term to promotesparsity in the coefficients, which estimates the dimensionality of the input data, leading to
[Equation]
Curvature and the Hessian.
In the classical setting of surfaces, a relevant notion of curvature is the Gaussian curvature κ .
In the special case that M is realized as the graph of a function f ( x, y ) , [Table] the Gaussian curvature can be calculated explicitly from the Hessian matrix H of f by the formula
[Equation]
Note that at a local optimum of f (where ∇ f = 0 ) the Gaussian curvature is completely characterizedby the eigenvalues of the Hessian.
In higher dimensions, by contrast, a single number is no longersufficient to capture the full curvature information of a Riemannian manifold, and one insteadappeals to the Riemann curvature tensor.
In the special case that M is realized as the graph of afunction f : R n → R , the spectrum of the Hessian of f again contains important (albeit not complete)curvature information.
As we are interested in measuring curvature in the setting of loss landscapes,we apply CurveNet described above to directly compute the Hessian of the parameters with respectto the loss function of a neural network in question.
Results
We validate both diffusion curvature and CurveNet using (a) synthetic test cases, and (b) single-cell
RNA sequencing data (scRNA-seq).
Moreover, we analyze the quality of our curvature estimationmethods in a data sampling scenario, where we sample points in the vicinity of an optimum.
All testcases are chosen to assess the efficiency and accuracy of our methods.
We trained CurveNet on N = 1000 samples from idealized surfaces, since this defines the dimensionsof the diffusion map used for curvature estimation on loss landscapes.
As the parameter spaceof neural networks can be large, sampling points around the minima is a method of reducingdimensionality and restricting the estimation to a less complex space.
We used intrinsic dimensionsof k = { 2 , 3 , . . .
20 } such that the same neural network can learn higher or lower dimensionalspaces.
We trained on 5000 different randomly generated quadrics, all sampled using N = 1000points, for each intrinsic dimension.
CurveNet then outputs a quadratic form with K ( K + 1) / 2entries, where ( K = max( k ) = 20 ), which we compare with the known ground truth using meansquared error (Table 1).
Training and testing were done on 8 core Tesla K80 GPUs with 24 GBmemory/chip.
Architectural details and code for Curvenet and the implementation of diffusioncurvature are available via an anonymized GitHub URL provided in the Appendix (Section A6).
4.1 Toy test cases for curvature estimation
We generated a series of synthetic datasets where the primary objective is to estimate curvature atcentral points (which are not affected by edge effects).
Figure 3 shows a series of artificially generated3 D surfaces whose curvature varies in two principal directions from positive to negative.
In the leftcolumn we show the curvature estimate given by Diffusion Curvature (Eqn. 2).
The second, third andfourth columns show comparisons to Gaussian curvature (defined in Section 3.1), mean curvatureand Ollivier’s Ricci curvature (defined in Section 2.2) respectively. The last column contains biaxialplots showing the correlation between Gaussian and Diffusion curvature.
We observe that Diffusion Curvature is capable of approximating/capturing Gaussian Curvature:despite being scaled differently, Diffusion Curvature highlights essentially the same structures as
Gaussian Curvature.
This qualitative observation is quantified by the correlation.
Overall, weobtain high correlations for the different surfaces.
The last surface is slightly different, as highvalues measured by Diffusion Curvature are more concentrated within the “cusp”, whereas Gaussian curvature spreads out such values over a larger part of the surface.
This example demonstrates thebenefits of Diffusion Curvature, though: in the context of loss landscape analysis, Diffusion Curvatureis much more sensitive to such sharper minima, thus facilitating their detection.
4.2 Curvature estimation for single-cell data
We also estimated the curvature of a publicly available single-cell point cloud dataset obtainedusing mass cytometry of mouse fibroblasts.
The dataset does not contain any identifiable or humaninformation.
Mass cytometry is used to quantitatively measure 2005 mouse fibroblast cells inducedto undergo reprogramming into stem cell state using using 33 channels representing various proteinbiomarkers.
Such a system is often called induced pluripotent stem cell (iPSC) reprogramming [20].
This dataset shows a progression of the fibroblasts to a point of divergence where two lineages emerge,one lineage which successfully reprograms and another lineage that undergoes apoptosis [4].
We notethat our model correctly identifies the initial branching point (with cells that don’t survive) as havinglow values of diffusion curvature indicating relatively negative curvature due to divergent paths outof the point (resulting in divergent random walks, see Figure 4).
On the other hand it shows highervalues indicating flat curvature along the horizontal branch.
Diffusion curvature of another single-celldataset, measuring embryonic stem cell differentiation, is provided in Section A1 of the Appendix.
4.3 Hessian estimation and sampling
To obtain second-order information around a critical point, x s , we estimate a local Hessian: wefirst (for both toy test cases and neural networks) sample n points, x 1 , x 2 . . .
x n , at random in theneighborhood of the critical point. Here x i ∈ R k , , where k is the full dimensionality of the domain
ofthe toy functions or the neural network: f : R k → R and the value of the objective function or loss,f ∈ R can be obtained by evaluating f ( x i ) .
In both settings, this allows us to obtain an R n × ( k +
matrix a sampled point x i ∈ R k with its associated loss or objective function inthe this column a loss axis use it an input the neural network the coefficients along with the diffusion axes,
The diffusion axes were obtained by constructing a diffusion map in R n based on the sampled pointsaround the optimum, i.e., the minimum.
We uniformly sampled 1000 points on a k-dimensionalhypersphere which was then scaled by the parameter space of the optimum or saddle, as well as thegradient at that point.
Special care was taken to ensure the points were sampled locally by evaluatingat the relative difference between the evaluated loss at locally sampled points around the optimum orsaddle and the actual loss at the optimum or saddle.
We then use these diffusion coordinates Φ( x )and the value at the sampled point f ( x ) ∈ R as an input pair (Φ( x ) , f ( x )) to the neural network,which we ultimately use to estimate the Hessian.
Figure 5 shows the eigenspectrum of the Hessian estimated using CurveNet for a feed-forward neuralnetwork trained to classify MNIST digits. We observe that the number of negative eigenvalues of the Hessian matrix (indicative of a maximum in the loss landscape) decrease over training epochs.
In Figure 5, when comparing epoch 25 and epoch 200 , we observe a marked shift in the (cumulative)density of eigenvalues towards positive eigenvalues, showing that the feed-forward neural network isapproaching a minima in the parameter space.
Hessian eigenspectrums estimated using CurveNet fora convolutional neural network and ResNet-18 are provided in the Appendix (Sections A2 and A3).
Conclusion
We proposed diffusion curvature , a new measure of intrinsic local curvature of point clouds.
Ourmeasure leverages the laziness of random walks, obtained using the diffusion maps framework.
Welink diffusion curvature to existing volume comparison results from Riemannian geometry.
In contrastto these notions, diffusion curvature can be computed effectively via neural networks even for highdimensional data.
While we demonstrated the effectiveness of such a curvature measure by analyzingnumerous datasets of varying complexities, our formulation also leads to new research directions.
Of particular interest will be proving additional results about our measure, relating it to existingquantities such as the Laplace–Beltrami operator, as well as formally proving its stability properties.
We also want to develop new methods that use diffusion curvature to compare different datasets; beinga quantity that is invariant under transformations (such as rotations of a point cloud), we considerdiffusion curvature to be a suitable candidate for assessing the similarity of high-dimensional complexpoint clouds.
Negative Impact: We cannot envision any negative societal impact of this work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](
b) Did you describe the limitations of your work?
[Yes] See Section 3.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Section 4,additional details are provided in the Supplementary Materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4, additional details are provided in the Supplementary
Materials.
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[Yes] Mean and standard deviation of model predictions is reported Table 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We specify number and type of
GPUs in Section 4, training time is provided in Supplementary Materials.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We make our data available as part of the code we release.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]