A CCELERATING O PTIMIZATION USING N EURAL R EPARAMETRIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We tackle the problem of accelerating non-linear non-convex optimization problems. We discover that reparametrizing the optimization variables as the output of a neural network can lead to signiﬁcant speedup.
We examine the dynamics of gradient ﬂow of such neural reparametrization.
We ﬁnd that to obtain the maximum speed up, the neural network architecture needs to be a specially designed graph convolutional network (GCN). The aggregation function of the GCN is constructed from the gradients of the loss function and reduces to the Hessian in early stages of the optimization.
We show the utility of our method on two optimization problems: network synchronization and persistent homology optimization and ﬁnd an impressive speedup, with our method being 4 ∼ 80 × faster
I NTRODUCTION
Gradient-based optimization lies at the heart of machine learning.
For non-linear, non-convex largescale optimization problems, gradient-based optimization can suffer from slow convergence and signiﬁcant computational bottleneck. In linear systems, preconditioning Axelsson (1996);
Saad &
Van Der Vorst (2000) can accelerating convergence by multiplying a symmetric positive-deﬁnite preconditioner matrix to the original problem.
In effect, preconditioning re-scales the loss landscape to have more circle-like level sets, hence changing the condition number of the system. Adaptive gradient optimizers such as AdaGrad Duchi et al. and Adam Kingma & Ba (2014) uses the same idea but updates the preconditioner during iterative optimization with adaptive step size, which leads to great improvement in convergence speed. Our goal is to improve the convergence of non-convex optimization further.
First, we examine the dynamics of the gradient ﬂow and identify the condition number that dictates the speed of convergence.
We discover that changing the condition number naturally requires the reparametrization to be a type of graph neural networks (GNN). Based on the analysis, we introduce a novel neural reparametrization scheme, which generalizes preconditioning to non-linear systems. Note that our method is complementary to existing gradient-based optimization algorithms and can be used in conjunction with any adaptive gradient optimizer (e.g. Adam). We test our hypothesis on two highly nonlinear optimization problems and obtain impressive speedups.
A few earlier work Sosnovik & Oseledets (2019); Hoyer et al. (2019) have demonstrated that convolutional neural networks can serve as priors to improve the parameterization of a class of structural optimization problems. But the theoretical foundation behind these techniques are not well understood. Concurrently, the implicit acceleration of over-parametrization in linear neural networks have been analyzed in Arora et al.
; Tarmoun et al. Speciﬁcally, Arora et al. (2018) shows that reparametrizing the linear weights with deep linear networks impose a pre-conditioning scheme on gradient descent. Tarmoun et al.
(2021) further exploits the conservation laws in gradient ﬂow and derive convergence rate for two-layer linear networks. In contrast, our method applies to highly nonlinear problems, guided by theoretical insights. Our contributions can be summarized as follows We analyze the dynamics of the gradient ﬂow and derive how a neural reparametrization (NR) can lead to a speedup.
We prove that maximum speedup is achieved if the NR is a special GCN which uses gradient of the loss in its aggregation function. For early steps, we show that the NR should use the Hessian of the loss, leading to an efﬁcient hybrid implementation. We showcase NP on two optimization tasks: network synchronization and persistent homology and observe consistent speedup.
Our method also demonstrates a novel usage and unique perspective for GNN, speciﬁcally GCN (Kipf & Welling, 2016). The majority of literature treat GNNs as a representation learning tool for graph-structure data (see surveys and the references in Bronstein et al. ; Zhang et al. Wu et al.
; Goyal & Ferrara (2018)). However, our analysis shows that GNN can be used to modify optimization dynamics in problems without explicit graph data.
M ETHODOLOGY
We consider the problem of minimizing a smooth, lower-bounded loss function L ( w ) ∈ R on abounded variable w ∈ R n :
[Equation]
When the optimization variables are matrices or tensors, w represents a ﬂattened vector containing all variables. w can also represent all trainable parameters in a deep neural network. For supervised learning with a dataset Z = { ( x i , y i )
Ni =1 } , the loss function L ( w ) becomes L ( w ; Z ) .
The dataset and the model class (e.g. architecture of a neural network) together deﬁne a loss landscape as a function of the optimization variables w . Gradient-based optimization updates the variables w by taking repeated steps in the direction of the steepest descent w t +1 = w t − ε ∂ L ∂ w t for a learning rate ε .
With very small time steps δt , we examine the dynamics of w ( t ) ∈ R n during the optimization process, treating t as a continuous time.
Using steepest descent for the optimization, d w /dt follows the gradient ﬂow: [Equation] [Equation] where denote the learning rate ε as a diagonal matrix of learning rates, because modern optimizers such as RMSProp (Tieleman et al., 2012) or Adam (Kingma & Ba, 2014) choose ε adaptively for each w i .
Speciﬁcally, in Adam and RMSProp, which we use in experiments, we have
[Equation]
where δ ij is the Kronecker delta, and η and ξ are small numbers. The expected value E is calculated using stochastic samples for ∇ i L over multiple iteration steps, with more weights given to more recent steps. Equation 1 is also an ordinary differential equation (ODE).
The local minima of L ( w ) correspond to the stationary solutions of the ODE, i.e. d w /dt = 0 .
Convergence rate We are interested in the expected convergence rate E [ d L /dt ] of the loss function. This expectation can be either approximated in a fashion similar to adaptive gradient optimizers (i.e. exponential time average) or be over random initializations under a distribution P ( w (0)) .
We will assume the expectation value for ε and E [ d L /dt ] are evaluated with the same method, meaning we assume E [ ε ] = ε . The convergence rate is then given by
[Equation]
where we used our assumption to get E [ εM ] = ε E [ M ] , with M ij = ∇ i L∇ j L being the “square gradients” matrix and M = E [ M ] .
With this deﬁnition, equation 2 becomes ε = η/ (cid:113) diag[ M ] + ξ . Note that M is time-dependent and can change during optimization.
If we know the distribution P ( w ( t )) for some t , we can approximate M . One way to do this is similar to Adam as a weighted average over the past few steps.
Also, if we know the distribution of the initialization P ( w (0)) , we can evaluate M for early stages, which is what we [Figure] will use in the current paper.
When using Xavier initialization (Glorot & Bengio, 2010), we have w (0) ∼ N (0 , n − 1 / 2 ) . In this case we can show (see appendix A.1) [Equation] where H ij ( w ) ≡ ∂ 2 L ( w ) /∂
w i ∂ w j is the Hessian of the loss at t = 0 . The eigenvectors of M with zero eigenvalue are modes that do not evolve during optimization. As gradients are zero in these directions, GD can never ﬁnd and explore those directions. Therefore, we may exclude them from the beginning and claim that M is full rank.
The rate d w i /dt can be different along different directions of the parameter space, with fast and slow modes. Next, we quantify this difference in rates and suggest ways for acceleration. 2. F AST AND SLOW D YNAMICS OF G RADIENT F LOW M is positive semi-deﬁnite because for any vector v ∈ R n we have v T M v = E (cid:104)(cid:0) v T ∇L (cid:1) 2 (cid:105) ≥ 0 .
Thus, M is Hermitian with a spectral expansion M = (cid:80) i m i ψ i ψ Ti .
Using equation 1, we can show that the dynamics of w along different ψ i are orthogonal to each other
[Equation]
If ε is constant and not adaptive, w evolves faster along modes ψ i with larger eigenvalues m i , since if m i > m j we have
[Equation]
This is precisely what optimizers such as Adam try to address.
Ideally, having ε i = η/ √ m i would make up for the rate difference, resulting in a uniform GD where no mode ψ i is evolving more slowly than others. However, calculating m i is O ( n 3 ) and can be quite expensive during optimization, hence the approximate version equation 2 is used. We show here that in many cases it is worthwhile to do a full correction to GD using M , at least in early stages of the optimization.
The eigenvalues of M determine the rate of evolution of the overlaps along each of its eigenvectors.
In particular, if some eigenvalue m slow (cid:28) mean i [ m i ] is much smaller than the mean, the evolution of w along ψ slow will be much slower than other directions.
Therefore, we will refer to all ψ slow as the slow modes of the optimization problem.
Conversely, the ψ fast where m fast (cid:29) mean i [ m i ] will be referred to as the fast modes .
When running GD, the maximum change in w is bounded to ensure numerical stability.
Because of the orthogonality in equation 5, we can enforce a numerical bound η via
[Equation]
where m max = max i m i is the largest eigenvalue of M .
Therefore, the learning rate ε is bounded by the largest eigenvalues (fastest modes).
Thus, in order to speed up convergence , we need to focus on the slow modes, which can be achieve with a reparametrization of w .
2. N EURAL P ARAMETRIZATION OF O PTIMIZATION P ROBLEMS
Using neural networks to reparametrize optimization problems have been suggested in (Hoyer et al., 2019; Sosnovik & Oseledets, 2019).
The idea is that the optimization variable w can be deﬁned as the output of a deep neural network w ( θ ) ≡ f ( θ ) with trainable parameters θ .
Then, rather than optimizing over the original variables w directly, we can instead optimize over the neural network parameters θ .
We will take this idea further and analyze the type of neural network architectures that can accelerate the optimization.
Modiﬁed convergence rate The neural reparametrization of the variables w ( θ ) leads the following updating rule for θ :
[Equation]
The convergence rate is then given by
[Equation]
Here K is the neural tangent kernel (NTK) (Jacot et al., 2018) of w ( θ ) , and M ij is the squared gradients. Note, we have absorbed ˆ ε a into K for convenience of notation.
Expected convergence rates The expected convergence rate E
[ d L /dt ] equation 3 depended on M = E [ M ] , the reparametrized one from equation 9 depends on E [ MK ] .
In order to quantify any improvements in E [ d L /dt ] from equation 9, we need to be able to express E [ MK ] in terms of E [ M ] , which is not possible for a general K . If we assume M and K to be independent from each other, i.e. E [ MK ] becomes E [ M ] E [ K ] .
Since M is a function of w , K must not be a function of w . One way is to have w = σ ( Aθ + b ) or any other similar neural network. We will discuss explicit choices for w below in sec.
In summary, an important constraint that we will put on w ( θ ) is that it yields a K independent of w choose w [Equation]
Now we can compute the expected rate of change of w ( θ ) along the previous fast and slow mode eigenvectors ψ i of M , similar to equation 6.
Using equation 9 and equation 8 we have
[Equation]
where ∂ L /∂ w = ∇ w L is the gradient vector.
Multiplying equation 11 by its transpose, denoting
K ≡ E [ K ] , and using equation 10 the expected value becomes
[Equation]
which shows how the neural reparametrization may speed up the convergence.
First, note that NR changes the rate of evolution of modes ψ i in equation 7.
We can quantify this change when [ K, M ] = 0 .
[-η , if [ K, M ] = 
 we have [Equation] where k i is the eigenvalue of-]In this case, K and M can be diagonalized simultaneously with shared eigenvectors ψ i .
Plugging the spectral expansion K =
(cid:80)
i k
i ψ i ψ Ti into equation 12 and using ε <
η/ √ m max (equation 7) we have
[Equation] Thus, one way to speed up the convergence, is to have [ K, M ] = 0 and have K increase the slowest rates (cid:107) ψ slow d w /dt (cid:107) , while keeping the fastest rate mostly unchanged. As a result, we have:
Theorem 1.
With a neural reparametrization yielding (cid:2) K, M (cid:3) = 0 , to make all modes ψ i reach the ideal maximum rate max i (cid:107) ψ Ti d w /dt (cid:107) <
η , we need to have
[Equation]
Proof. This is because plugging k 2 i ≈ m max / ( m i + ξ ) ( ξ (cid:28) 1 ) into equation 13 makes the r.h.s. equal to η 2 .
Next, we discuss how we can achieve this in practice.
Speciﬁcally, we derive what architecture for the neural network w ( θ ) =
f ( θ ) will result in a K satisfying the condition in equation 14.
2. P RACTICAL I MPLEMENTATION
To ﬁnd a neural network model that satisﬁes equation 14, we need to consider the fact that M ( w ( t )) changes during optimization.
To minimize the overhead from constructing w ( θ ) (and hence K ) as a function of M , the architecture has to be as simple as possible and involve steps which do not increase the time complexity of each iteration signiﬁcantly.
We will now discuss when such neural reparametrization is feasible and how to implement equation 14 in practice.
Hybrid optimization To use equation 14 at points other than the initial steps, we need to dynamically estimate and update M . Though estimating M ( t ) efﬁciently with ideas from Adam is possible, in the current work we focus on the speeding up at early stages using equation 4.
Thus, our main proposal is to use a two stage hybrid optimization: 1) use the Hessian to reparametrize w ( θ ) in early stages for some number of iterations and then 2) switch to the original optimization over w . Simple implementation Because M may depend on w ( θ ) , as in equation 10, we need a K that is independent of θ to ensure E [ MK ] =
E [ M ] E [ K ] . The simplest architecture is a linear model w i
= (cid:80) a κ ia θ a , which yields K
=
κκ
T ≈ ( M/m max ) − 1 / 2 .
We need K to be a full rank matrix, meaning that κ needs to be at least n × n .
Since we need the least computationally expensive K , we choose θ ∈ R n , and let κ be a symmetric n × n matrix.
This way, κ = √ K =
( M/m max ) − 1 / 4 .
For the initial stages, where M ≈ H 2 /n we have κ =
( H /h max ) − 1 / 2 , with H being the Hessian.
Per-step time complexity Since w ∈ R n , the complexity of computing L ( w ) is O ( np ) where p depends on the sparsity structure of L . M is an n × n matrix and computing M − 1 / 2 is O ( n 3 ) , which is very expensive when n (cid:29) 1 . The eigenvalue m max can be estimated iteratively in O ( qn 2 ) . Hence, computing K by exactly following equation 14 is not feasible, unless M is ﬁxed and can be calculated ofﬂine.
Even then, evaluating M − 1 / 2 only makes sense if the expected GD steps for convergence is more than O ( n ) , otherwise we don’t gain any speedup. We may instead try to approximate M − 1 / 2 and m max iteratively with complexity O ( qn 2 ) where q (cid:28) n is the number of iterations. Hence we note that using equation 14 to speedup the optimization may generally be useful if the the per step complexity of GD on L has a time complexity of least O ( qn 2 ) . Implementation constraints In summary, we have the following constraints on the structure of the neural network w ( θ )
Computing w ( θ ) ∈
R n must be at most O ( qn 2 ) with q (cid:28) n 2.
To have K independent of M , ∂ w /∂θ must be independent of w and hence, only linear maps w ( θ ) = κθ are allowed, with κ independent of w . 3. θ ∈ R m with m ≥ n . We choose m = n , which yields κ = √ K
=
( M/m max ) − 1 /4. Instead of κ = ( M/m max ) −
Relation to GCN Since both θ ∈ R n and w ∈ R n , our linear w = κθ architecture is essentially a Graph Convolutional Network (GCN) (Kipf & Welling, 2016) with the aggregation rule κ = ( M/m max ) − 1 / 4 , or a weighted adjacency matrix.
In fact, all of our derivations remain unchanged if w ∈ R n × d and θ ∈ R n × m (i.e. if there are d and m features per node in w and θ ).
Thus, our claim is that we can speed up the optimization process by reparametrizing w using a GCN with linear activation, as w = κθ , where θ are trainable and κ is the aggregation function derived from the loss gradients.
However, evaluating this κ during GD can be quite expensive ( O ( n 3 ) ) for large n (GD steps ∼ O ( n 2 ) ).
Ideally we want an approximation for κ which is O ( qn 2 ) with q ∼ O (1) .
Computationally efﬁcient implementation We will focus on speeding up the early stages, in which we use the Hessian H in M ≈ H 2 /n . Deﬁne H ≡ (1 − ξ ) H / √ h max , where h max = √ m max and ξ (cid:28) 1 .
To get an O ( qn 2 ) approximation for κ = {+( M/m max ) − 1 / 
 =+}
H − 1 / 2 we can take the ﬁrst q terms in the binomial expansion (see Appendix A.2). A rough estimate would be
[Equation]
Since H is positive semi-deﬁnite and its largest eigenvalue is 1 − ξ < 1 , the sum in equationcan be truncated after q ∼ O (1) terms.
The ﬁrst q terms of equation 25 can be implemented as a q layer GCN with aggregation function f ( M ) =
I − M 0 and residual connections.
We choose q to be as small as 1 or 2 , as larger q may actually slow down the optimization.
Note that computing f ( M 0 ) q θ is O ( qn 2 ) because we do not need to ﬁrst compute f
( M ) q (which is O ( qn 3 ) )
and instead use v
i +1 f
( M ) v i ( O ( n 2 ) )
q times with v 1 = θ .
Lastly, to evaluate M 0 we need to estimate the leading eigenvalue m max .
Given any vector v ∈ R n and using spectral expansion, we have
[Equation]
where, since H is positive semi-deﬁnite, for q > 1 the leading eigenvector ψ max quickly dominates the sum and we have H q v ≈ h qmax ( ψ max v ) ψ max .
Here we chose q = 2 to get a crude approximation for h max .
The generalized Perron-Frobenius theorem (Berman & Plemmons, 1994), state the components of the leading eigenvector ψ max should be mostly positive.
Therefore, we chose v = 1 / √ n , where 1 = (1 , . .
. , 1) , which should be close to the actual ψ max .
If we think of M as the weighted adjacency matrix of a graph with n vertices, the vector component D i =
[ H 1 ] i = (cid:80)
j H ij is the weighted degree of node i .
Hence equation 16 states H /m max ≈ (cid:0)(cid:80)
i D i / (cid:107) D (cid:107) 2 (cid:1) H .
This is similar to the graph diffusion operator D − 1 H and D − 1 / 2 H D − 1 / 2 which are used as the aggregation functions in GCN (here D ij =
D i δ ij is the degree matrix).
When using modern optimizers, the adaptove learning rates is a term within K .
This will interfere with the optimal choice for κ and hence our GCN aggregation function.
Therefore, in practice, we do not ﬁx the coefﬁcients for H and H 2 in our GCN.
Instead, we use single or two layer GCN with aggregation function f ( H ) =
D − 1 / 2 H D − 1 / 2 .
E XPERIMENTS
We showcase the acceleration effect of neural reparametrization on two non-convex optimization problems: network of Kuramoto oscillators which are widely-used for modeling synchronization phenomena; and persistent homology, a mathematical tool that computes the topology features of data that persist across scales .
We use Adam Kingma & Ba (2014) optimizer and compare different reparameterization models.
We implemented all models with Pytorch.
3. N ETWORK OF K URAMOTO OSCILLATORS
Kuramoto model Kuramoto (1975; 1984) are widely used for synchronization problems, which have profound impact on engineering, physics and social systems (Pikovsky et al., 2003).
As shown in
Fig.
3(right), Kuramoto model describes the behavior of a large set of coupled oscillators.
Each oscillator is deﬁned by an angle θ i
= ω
i t + φ i , where ω i is the frequency and φ i is the phase.
We
Final
L i n e a r
G C N
R G C N iterations c a b consider the case where ω i = 0 .
The coupling strength is represented by a graph adjacency matrix with elements A ij ∈ R .
Deﬁning ∆ ij ≡ φ i − φ j , the dynamics of the phases φ i ( t ) in the Kuramoto model follows
[Equation]
Hopf-Kuramoto (HK) model
Lauter et al.
(2015) is a more general version of the Kuramoto model.
It includes second neighbor interactions. HK has a rich phase space with qualitatively different solutions. The phase space includes regions where simulations becomes slow and difﬁcult.
We use the HK model to showcase our method in more complex and nonlinear scenarios. The HK model’s dynamics follows [Equation] with the following loss function
[Equation]
Our goal is to minimize the phase drift dφ i /dt to synchronize the oscillators.
Existing numerical methods directly optimize the loss L ( φ ) with gradient-based algorithms, which we refer to as linear . We apply our method to reparametrize the phase variables φ and speed up convergence towards synchronization. Implementation. For early stages, we use a GCN with the aggregation function derived from the
Hessian which for the Kuramoto model simply becomes H ij [Equation] k A ik [Equation]
ij = − L ij , where L = D − A is the graph Laplacian of A . We found that NR in the early stages of the optimization gives more speed up. We implemented the hybrid optimization described earlier, where we reparemtrize the problem in the ﬁrst 100 iterations and then switch to the original linear optimization for the rest of the optimization. We experimented with three Kuramoto oscillator systems with different coupling structures: square lattice, circle graph, and tree graph.
For each system, the phases are randomly initialized between 0 and 2 π from uniform distribution. We let the different models run until the loss converges (patience steps for early stopping, 10 − 15 loss ﬂuctuation limit). Results of Kuramoto Model. Figure 2 shows the results of Kuramoto model on a square lattice.
Additional results on circle graph, and tree graph can be found in Appendix B. Figure 2 (a) shows that NR with one-layer GCN (GCN-1) and GCN with residual connection (RGCN-1) achieves signiﬁcant speedup. In particular, we found 3 . 6 ± . 5 speed improvement for the lattice, 6 . 1 ± .
1 for the circle graph and 2 . 7 ± . 3 for tree graphs. We also experimented with two layer (GCN/RGCN-2) and
S 1 / c S 2 /c  0. b a 0.
A A A 13  three layer (GCN/RGCN-3) GCNs. As expected, the overhead of deeper GCN models slows down optimization and offsets the NR speedup gains.
Figure 2 (b) visualizes the evolution of the oscillator phase φ i on a square lattice over iterations.
We can see that even though different reparametrization models reach the same loss value, the ﬁnal solutions are quite different. The linear version (without NR) arrives at the ﬁnal solution smoothly, while GCN models form dense clusters at the initial steps and reach an organized state before 100 steps. To quantify the level of synchronization, we measure a quantity ρ known as the “global order parameter” (Sarkar & Gupte (2021)): ρ = 1 N (cid:12)(cid:12)(cid:12)(cid:80)
j e iφ j (cid:12)(cid:12)(cid:12) .
Figure 2 (c) shows the convergence of the global order parameter over time.
We can see that one-layer GCN and RGCN gives the highest amount of acceleration, driving the system to synchronization. Results of Hopf-Kuramoto Model.
We report the comparison of NR models on synchronizing
Hopf-Kuramoto dynamics.
According to the Lauter et al.
(2015) paper, we identify two different main patterns on the phase diagram Fig.
8(b): ordered (small s 2 /c , smooth patterns) and disordered (large s 2 /c , noisy) phases.
In all experiments we use the same lattice size 50 × 50 , with the the same stopping criteria ( 10 patience steps and 10 − 10 loss error limit) and switch between the Linear and GCN model after 100 iteration steps.
Fig.
3 (a) shows the loss at convergence versus the speedup.
We compare different GCN models and observe that GCN with A 2 as the propagation rule achieves the highest speedup. Also, we can see that we have different speedups in each region, especially in the disordered phases.
Furthermore, we observed that the Linear and GCN models converge into a different minimum state in a few cases.
However, the patterns remain the same.
(b) shows how the level of ordering changes region by region. If the global order parameter is closer to 0 , we have more of a disordered phase while the parameter is closer to 1 , meaning it is a more organized pattern.
We know that running dynamics in a large lattice are computationally expensive, especially close to the phase transition points where difﬁcult to separate the slow and fast modes.
Persistent homology Edelsbrunner et al.
(2008) is an algebraic tool for measuring topological features of shapes and functions.
It provides precise deﬁnition of qualitative features, is computable with linear algebra and robust to perturbation of input data (Otter et al., 2017), see more details in
Appendix B.2.
An example of persistent homology is point clouds optimization Gabrielsson et al.
(2020); Carriere et al. (2021).
Given a random point cloud φ that lacks any observable characteristics, we aim to produce persistent homological features by optimizing the position of data points. [Equation]
[Equation]
where D = { ( b i , d i ) } i ∈
I k is the persistence diagram, consisting of all the pairs of birth and death ﬁltration value of the set of k-dimensional homological features I k .
π ∆ is the projection onto the diagonal ∆ and (cid:80) i
d ( φ i , S ) constrains the points within the a square centered at the origin with length 2 r .
We will refer r as the range of the point cloud.
Carriere et al.
(2021) optimizes the point cloud positions directly with gradient-based optimization.
We will refer theirs as the baseline linear model.
In contrast, we integrate GCN and reparameterize the point cloud positions, and will refer ours as the GCN model.
[Figure]
[Figure]
Implementation.
To measure the speed-up of GCN model, we used the [-node embedding G = [ G 0 , G 1 , G 2 ] ∈ R N × h where h = h 
-]same Gudhi library for computing persistence diagram as Gabrielsson et al.
[-+ h 
 + h 2 . Finally, we apply a fully connected network to project the h -dimensional embedding G to the original d dimension features X . We allow G 0 to be freely trainable and W ∈ R-]; Carriere et al. (2021).
The runtime of learning persistent homology is dominated by computing persistence diagram in every iteration, which has the time complexity of O ( n 3 ) .
Thus, the runtime per iteration for GCN model and linear model are very similar, and we demonstrate that the GCN model can reduce convergence time by a factor of ∼ 4 .
We ran the experiments for point cloud of 100 , 200 , 300 points, with ranges of 0 . 5 , 1 . 0 , 2 . 0 , 4 . 0 .
The hyperparameters of the GCN model are kept constant, including network dimensions.
The result from each setting consists of outputs from 5 consecutive runs.
Results. Fig. 4 and 5 shows that the speedup of the GCN model is related to point cloud density. Training converges faster as the point cloud becomes more sparse, but the speed-up gain saturates as point cloud density decreases.
On the other hand, time required for initial point cloud ﬁtting increases signiﬁcantly with the range of point cloud.
Consequently, the overall speedup peaks when the range of point cloud is around 4 times larger than what is used be in Gabrielsson et al.
Carriere et al.
(2021), which spans over an area 16 times larger. Further increase in point cloud range causes the speedup to drop as the extra time of initial point cloud ﬁtting outweighs the reduced training time. The loss curve plot in 4 shows the convergence of training loss of the GCN model and the baseline model in one of the settings when GCN is performing well. Figure 5 shows the initial random point cloud and the output from the GCN model.
In the Appendix B.2, we included the results of GCN model hyperparameter search and a runtime comparison of the GCN model under all experiment settings.
Conclusion We propose a neural reparametrization scheme to accelerate a large class of optimization problems.
Our method is grounded on analysis that the dynamics of gradient ﬂow are related to the condition number of the system.
By reparametrizing the optimization problem with a graph convolutional network, we can modify the condition number and obtain the maximum speed up.
The aggregation function of the GCN is constructed from the gradients of the loss function and reduces to the Hessian in early stages of the optimization.
We demonstrate our method on optimizing synchronization problems and persistent homology of point-clouds.
Depending on the experiment, we obtain a best case speed-up that ranges from 4 to 80.