TRANSFERRING OPTIMALITY ACROSS DATA DISTRIBUTIONS VIA HOMOTOPY METHODS
Anonymous authors
Paper under double-blind review
A BSTRACT
Homotopy methods, also known as continuation methods, are a powerful mathematical tool to efﬁciently solve various problems in numerical analysis.
In this work, we propose a novel homotopy-based numerical method that can be used to gradually transfer optimized weights of a neural network across different data distributions.
This method generalizes the widely-used heuristic of pre-training weights on one dataset and then ﬁne-tuning them on another dataset of interest.
We conduct a theoretical analysis showing that, under some assumptions, the homotopy method combined with Stochastic Gradient Descent (SGD) is guaranteed to converge to an r θ -optimal solution for a target task when started from an r θ -optimal solution on a source task. Empirical evaluations on a toy regression dataset and for transferring optimized weights from MNIST to Fashion-MNIST and CIFAR-10 show up to two orders of magnitude speedup over random initialization and substantial improvements over pre-training.
I NTRODUCTION
The homotopy method (Allgower & Georg, 1980), also known as the continuation method, is a powerful mathematical tool to efﬁciently solve various problems in numerical analysis (e.g., TranDinh et al.
(2012), Zanelli et al.
The core idea consists in sequentially solving a series of parametric problems, starting from an easy-to-solve problem and progressively deforming it, via a homotopy function, to the target one.
Homotopy methods are suitable to solve complex non-convex optimization problems where no or only little prior knowledge regarding the localization of the solutions is available.
In addition, in contrast to state-of-the-art algorithms in deep learning (e.g., Bottou (2010), Duchi et al.
(2011), Kingma & Ba (2015)) , these methods often achieve global convergence guarantees by only exploiting local structures of the problem. Concepts, such as curriculum-learning and warm-starting, that are related, though to different degrees, to homotopy methods have been explored both in the deep learning (e.g., Gulcehre et al. , Mobahi (2016), Gulcehre et al.
and in the reinforcement learning (e.g., Narvekar (2017)) communities.
In this work, we propose a novel homotopy-based numerical method to transfer knowledge regarding the localization of an optimum across different task distributions in deep learning.
This method gradually tracks a neural network’s
(close-to-)optimal weights from one data distribution to another one via the homotopy method (Allgower & Georg, 1980) and can be interpreted as a generalization of the very common heuristic of ﬁne-tuning a pre-trained network.
After discussing related work (Section 2) and background on homotopy methods (Section 3), our contributions are as follows:
We provide a general theoretical analysis of the homotopy method when using SGD as an iterative solver, proving that under some assumptions it tracks an r θ -optimal solution from the source task to the target task (Section 4).
We introduce homotopic functions for transferring optimality across data distributions for regression and classiﬁcation tasks (Section 5).
For a toy regression dataset and for transferring optimized weights from MNIST to FashionMNIST and from MNIST to CIFAR-10, we show that our method obtains up to two orders of magnitude better numerical performance than random initialization and substantial improvement of the numerical performance over pre-training (Section 6).
R ELATED W ORK
Deep neural networks have led to establish a new state-of-the-art in many applications.
Despite their great success and the many theoretical studies that have been published in the last years (e.g., Balduzzi et al. (2017), Li et al. , Feizi et al.
, Kunin et al. (2019)), training these deep models remains a big challenge.
Various stochastic optimization algorithms (e.g., Duchi et al. (2011), Kingma & Ba (2015), Reddi et al. and initialization heuristics (e.g., Daniely et al. (2016), Klambauer et al.
(2017), Hanin & Rolnick (2018)) have been recently suggested in order to improve and speed up the training procedure.
We now brieﬂy discuss the state-of-the-art deep learning optimization techniques and initialization strategies that are most related with the proposed homotopy-method, drawing connections with existing and ongoing research works in the ﬁeld.
Curriculum Learning. First introduced by Bengio et al.
and then extended in different works (e.g., Graves et al. (2017), Weinshall et al. (2018), Hacohen & Weinshall (2019)), curriculum learning can also be listed among the optimization heuristics proposed to alleviate the complexity of solving high dimensional and non-convex problems.
In particular, taking inspiration from the fact that humans and animals learn “better” when exposed to progressively more complex situations in an organized manner, curriculum learning techniques guide the training by starting with “easy-tolearn” samples and progressively introducing more “complex-to-learn” ones.
This guided learning process can also be rephrased in a homotopy-like fashion (see Algorithm 1) as solving a sequence of optimization problems where the target training distribution gradually changes from considering only the “easy” examples to the full original training distribution. Meta-Learning and Transfer-Learning.
Due to the massive amount of computational resources required by the development of modern deep learning applications, the community has started to explore the possibility of re-using learned weights across different tasks, leading to the development of many new transfer-learning (e.g., Rohrbach et al. , Wang & Schneider (2014), Cui et al. and meta-learning (e.g., Schmidhuber (1987), Hochreiter et al. (2001), Finn et al.
(2017), Zintgraf et al.
(2019)) algorithms.
The simplest way to transfer knowledge across different tasks consists in using warm-start initialization.
This heuristic is amply used in computer vision applications, where it is also known as the ﬁne-tuning technique (e.g., Krizhevsky et al. (2012), Yosinski et al. (2014), Reyes et al.
(2015), K¨ading et al.
So far, there is no rigorous explanation of why and when ﬁne-tuning works.
However, numerous empirical evaluations on different benchmarks show that warm-starting the weights of deep models often leads to faster convergence and better generalization than using random initialization.
B ACKGROUND
In this work, we will focus on solving problems of the form
[Equation]
where L : R d → R is our target objective function.
Problems as described in 1 arise, for instance, in classiﬁcation and regression scenarios.
In the following section we brieﬂy review the main concepts of homotopy and continuation methods , which the proposed technique to solve problem 1 is based on.
3. H OMOTOPIC F UNCTIONS AND C ONTINUATION M ETHODS FOR O PTIMIZATION
Given two topological spaces Z and Y , a homotopy is a continuous deformation between two continuous functions g, f : Z → Y that fulﬁlls certain properties.
We can formalize this concept with the following deﬁnition
Deﬁnition 3.1.
Let g, f : Z → Y be continuous maps on the topological spaces Z, Y .
A homotopy from g to f is a continuous function H :
Z × [0 , 1] → Y such that
[Equation]
If such H function exists, g is said to be homotopic of f , and this relation is denoted by g (cid:39) f .
It is straightforward to show that, A ⊆ R n being a convex set, any two continuous maps g, f :
Z → A are homotopic (see (Suciu, 2016) for a derivation).
From this fact it follows that any two continuous and real functions are homotopic.
See Figures 4a– 4b in the appendix for a graphical representation of two different homotopy maps between the probability density functions of two
Gaussian distributions, where λ ∈ [0 , 1] denotes the homotopy parameter.
See also Section B in the appendix for details on some of the main properties of homotopic functions.
Continuation methods (also known as homotopy methods) are a widely used mathematical tool to solve complex non-convex optimization problems where no or only very limited prior knowledge regarding the localization of optimal solutions is available (see (Allgower & Georg, 1980) for a full characterization of continuation methods).
The core idea of a homotopy approach consists in deﬁning a homotopy function H ( θ, λ ) with λ ∈ [0 , 1] such that H ( θ, 0) =
L 0 ( θ ) is a trivial to optimize smooth map (or a smooth map of which a surrogate θ 0 of an optimal solution is available) and H ( θ, 1)
=
L ( θ ) is our target objective function.
Instead of directly addressing problem 1, we approximately and sequentially solve γ > 0 parametric optimization problems of the form
[Equation]
for increasing values of the parameter λ for i = 1 , . . .
, γ and warm-starting each problem with the previously derived approximate solution.
Conceptually, Algorithm 1 describes the basic steps of a general homotopy algorithm.
Under appropriate assumptions, if the increment ∆ λ is sufﬁciently small, then the iterative procedure in Algorithm 1 will converge to a neighborhood of an optimal solution of the target objective L that depends in some sense on the number of iterations k >performed, (Allgower & Georg, 1980).
Many different variations of Algorithm 1 exist.
In particular,
Algorithm
1: θ ← θ 0 such that θ
≈ θ ∗ = θ H ( θ, γ > ∈
Z /γ 4: k > 5: for i = 1 , . . .
, γ do 6: λ ← λ + ∆ λ 7: procedure θ ← I TERATIVE S OLVER ( θ, k )
Approximately
H ( θ, λ ) 8: return θ different update schemes for the λ parameter can be adopted (e.g. geometric or sublinear rate of increase), various linear solvers can be used under distinct and speciﬁc assumptions, and, ﬁnally, also diverse level of approximations for the solutions θ ∗ i can be considered, i.e. different k values.
Before going into the details of two concrete formulations of the conceptual homotopy method outlined in Algorithm 1 (see Section 5), we provide a general theoretical analysis in a simpliﬁed setting.
T HEORETICAL A NALYSIS
In this section, we provide a theoretical analysis of homotopy methods when Stochastic Gradient
Descent (SGD) (Bottou, 2010) is used as iterative solver in Algorithm 1.
In particular, we approximately and sequentially solve γ > 0 unconstrained optimization problems of the form
[Equation]
where H ( θ, λ i ) fulﬁlls the assumptions described in Section 4.1 and
λ i ∈
Ω λ .
Let θ i be an approximate solution of the problem associated with parameter λ
i derived by applying k > 0 iterations of SGD (in the limit, k = 1 ) and also the starting point for the problem associated with parameter λ
i +1 , ∀ i ∈ 1 , . .
, γ − 1 .
In addition, let θ
denote the initial point for the problem associated with λ 1 .
In this section we derive error bounds for the numerical error E (cid:2)
||
θ i − θ ∗
i
|| 2 (cid:3) when the changes in the parameter λ are “sufﬁciently” small.
4. A SSUMPTIONS
We now expose the fundamental assumptions for our theoretical analysis on which all the derivations in Section 4.2 rely.
Assumption 4.1 (smoothness of the objective function) .
Assume that H : R d × Ω λ → R is a smooth function w.r.t.
both arguments θ and λ .
Assumption 4.2 (bounded “variance”) .
Let θ k denote the value of θ after k iterations of SGD, and ˜ g k denote an unbiased estimate of the gradient ∇ H ( θ k , λ i ) w.r.t.
θ k .
Assume that there exists a constant C > 0 such that the following bound on the expected squared norm of the estimate of the gradient holds
[Equation]
Remark 4.3.
Assumption 4.2 is standard for proving error bounds on SGD iterates (see (Schmidt, 2014)).
In addition, notice that, since
[Equation]
the C constant is proportional to the variance and the squared expected value of the norm of the gradient estimate.
Therefore, it decreases when the iterates approach an optimum and by reducing the noise in the estimate of the gradient.
In the limit (i.e. exact gradient and convergence to an optimum), C = 0 .
Assumption 4.4 (strong convexity) .
Assume that there exists µ > 0 such that
[Equation]
Remark 4.5.
Assumption 4.4 is rather strong when considered globally.
Nevertheless, the proposed homotopy-based method enables the exploitation of (usually stronger) local properties of the problem unlike other state-of-the-art methods for optimization.
Therefore, with some extra precautions, the analysis we conduct can also be applied locally in a more general context.
In this case, the considered assumptions are only required to hold in a neighborhood of an optimal solution, leading to results applicable in the non-convex scenario too.
We provide some considerations that can be used to derive more general results in Section G of the appendix, where we relax condition 6 to hold only in a neighborhood of an optimal solution.
4. D ERIVATIONS
Proposition 4.6.
Given Assumptions 4.1 and 4.4, there exists (cid:15) > 0 such that θ ∗ ( λ ) is Lipschitz continuous over B ¯ λ ( (cid:15) ) :
= (cid:8) λ s.t. ||
¯ λ − λ
|| < (cid:15) (cid:9) ∀ ¯ λ ∈ Ω λ , i.e. the following inequality holds
[Equation]
Proof.
Proposition 4.6 follows directly from the application of the Implicit Function Theorem (see
Lemma 2.1.8 in (Allgower & Georg, 1980)).
Proposition 4.7.
Let θ i be the starting point for problem described in equation 1, and let θ i +1 denote the iterate after k > 0 SGD steps.
Under Assumptions 4.2 and 4.4 and by setting the learning rate α to a constant value such that 0 < α < 12 µ , the following error bound on the iterates holds
[Equation]
Proof.
See Section E in the appendix.
Remark 4.8.
The expectation in equation 8 is taken w.r.t.
all the random variables, i.e. gradients and initial point θ 0 , involved in the optimization procedure up to the current i + 1 iteration of the algorithm.
Under the considered assumptions, we now derive the error bounds for the problem associated with parameter λ
i +1 based on the iterates
θ i
and θ
i +1 .
In particular, the goal is to show that, if the approximate solution θ i for problem with parameter λ
i is sufﬁciently close to the optimum θ ∗ i in expectation, i.e. E (cid:2)
||
θ i − θ ∗
i
|| 2 (cid:3) ≤ r 2 θ , then, for a “sufﬁciently” small change in the λ i parameter,
E (cid:2)
|| θ i +1 − θ ∗
i +
|| 2 (cid:3) ≤ r 2 θ .
See Figure 9 in the appendix for a graphical representation.
Theorem 4.9.
Consider Algorithm 1 with Stochastic Gradient Descent as solver and let k > 0 be the number of iterations and
0 < α < 12 µ be the step size.
For r θ ∈ R such that
[Equation]
then, if E (cid:2)
||
θ i − θ ∗
i
|| 2 (cid:3) ≤ r 2 θ
and || λ
i − λ
i +1 || ≤ ˜ (cid:15) , where ˜ (cid:15) :
=
min { ¯ (cid:15), (cid:15) } with
[Equation]
the following inequality holds
[Equation]
Proof.
See Section F is the appendix.
Theorem 4.9 shows that, under the assumptions listed in Section 4.1, for small variations of the parameter λ i whose feasible values are inversely proportional to the Lipschitz constant δ of θ ∗ ( λ ) (see Proposition 4.6), which describes how fast the optimal solution changes w.r.t.
variations in the parameter, it is possible to deﬁne balls of radius r θ around the optimal solutions θ ∗ i and θ ∗
i +1 where the iterates θ
i
and θ i +1 are respectively included in expectation (see Figure 9 in the appendix for a graphical representation).
In addition, the lower bound on r θ is proportional to the noise level C in the estimate of the gradient and inversely proportional to the number of iterations k performed.
Consequently, the less noisy the gradient estimate is and the more iterations of SGD are performed, the closer the iterates θ i
and θ i +1 will be in expected value to the respective optimal solutions θ ∗
i and θ ∗
i +1 .
Finally, notice that these results can be applied recursively.
T RANSFERRING O PTIMALITY VIA H OMOTOPY M ETHODS
In this section we describe a possible application of homotopy methods to solve supervised regression and classiﬁcation tasks.
We address the case where deep neural networks are used as models.
We start by introducing the problem framework of supervised learning and then we propose two different homotopy functions for the regression and classiﬁcation scenarios, respectively.
5. P ROBLEM F ORMULATION
Despite the generality of the proposed methodology, in this work we speciﬁcally address the supervised learning framework, and, in particular, when the predictive model is constituted by a deep neural network f ( x ; θ ) parameterized by θ ∈ R d .
In the supervised learning scenario, independently from the type of task τ , we typically dispose of a training set D τ consisting of N pairs of examples ( x j , y j ) .
The goal of the learning process is to ﬁnd the values of θ that minimize an objective function which measures the discrepancy between the outputs produced by the network
ˆ y = f
( x ; θ ) and the target outputs y .
In particular, the learning process consists in minimizing the empirical objective function
[Equation]
whose non-convexity originates from the high non-convexity of our model f .
In the classical setting, L is chosen based on the KL divergence between the target data distribution
Q x,y , with density q x,y = q ( y | x ) q ( x ) , and the learned data distribution P x,y ( θ ) , with density p x,y = p ( y | x ; θ ) q ( x ) , where p ( y | x ; θ ) is modeled via a neural network.
This leads to the following form for the objective function
[Equation]
5. H OMOTOPY F UNCTIONS A CROSS D ATA D ISTRIBUTIONS
Finding a value of θ that attains a local minimum of the objective function in equation 12 is often a hard optimization task, given the high dimensionality and non-convexity of the problem.
In addition, prior knowledge regarding the localization of the solutions is rarely available.
The complexity of minimizing such functions also depends on the task distribution Q x,y that is addressed (e.g., Ionescu et al. (2016), Zendel et al.
For some tasks, convergence to a good approximate solution is achieved after a few epochs, while for other tasks, orders of magnitude more iterations are required to reach the neighborhood of a solution.
In this perspective, different heuristics have been recently introduced in the attempt of re-using the prior knowledge gained from the approximate solution of hard tasks across different data distributions.
The question whether we could exploit easy-to-solve or already-solved tasks to speed up and improve the learning of unsolved hard tasks arises.
The method we propose in this paper addresses this question and attempts to do so by using a rigorous and well-established mathematical framework, with the goal of speeding up the learning processes in presence of hard-to-solve tasks.
In the perspective of homotopy methods, these goals can be achieved under some assumptions by deﬁning a homotopy transformation between starting and target tasks and by following the procedure described by Algorithm 1.
Despite the ﬂexibility and generality of the method, with this work we only focus on homotopy deformations across different task distributions, but similar transformations can be applied in numerous different manners that are also worth exploring, e.g., progressively modifying the architecture of the network or the objective function terms.
Let s be the source task with training data D s of pairs ( x s , y s ) ∼ Q x s ,y s whose good approximate solution θ ∗ s for the minimization of the objective in 12 is available (or cheaply computable), and let τ denote the target task with training data D τ of pairs ( x τ , y τ ) ∼ Q x τ ,y τ whose conditional distribution we aim to learn.
We propose two different homotopy deformations from task s to task τ for regression and classiﬁcation, respectively.
5.2. S UPERVISED R EGRESSION
In the supervised regression scenario, by modeling the density of the conditional learned distribution as p ( y | x ; θ ) =
N (cid:0) y ; f ( x, θ ) , σ 2 I (cid:1) and using the approximate KL divergence objective function described in equation 13, we recover the mean squared error as minimization criterion.
The proposed homotopy deformation is based on the following equations
[Equation]
[Equation]
Notice that the transformation described in equation 14 preserves the unimodality of the conditional distribution (see caption of Figures 4a and 4b), and, when used in combination with the objective function deﬁned in equation 13, leads to the minimization w.r.t.
θ of
[Equation]
[Equation] See Figure 5a in the appendix for a graphical representation of this homotopy deformation when applied to gradually transform a one-dimensional sine wave function with a frequency of 1 radian into a one-dimensional sine wave function with a frequency of 137 radians.
A downside of this homotopy deformation is that the same support for x is required (the absence of the subscripts s and τ on x stands to indicate that the same realization for x s and x τ has to be considered).
Alternatively, it is possible to approximate equation 14 by using a Gaussian ﬁlter (see Figure 5b and Section C in the appendix).
5.2. S UPERVISED C LASSIFICATION
In the case of supervised classiﬁcation, by modeling the density of the conditional learned distribution as p ( y | x ; θ ) =
Multinoulli ( y ; f ( x ; θ )) , and using the approximate KL divergence objective function described in equation 13, we recover the cross-entropy loss function.
A possible homotopy deformation for the classiﬁcation case consists in applying the following transformations
[Equation]
which corresponds to the use of probabilistic labels.
See Figure 6 in the appendix for a graphical representation of the proposed homotopy deformation.
The corresponding label vector for the deformed image represented in Figure 6b is y 0 .
5 =
5 , 0 , 0 , 0 , 0] , given that λ = 0 .and
that the sampled realizations of x s and x τ , represented in Figures 6a and 6c, belong to classand 5 respectively.
E XPERIMENTAL E VALUATION
In this section, we present some experimental evaluations of homotopy methods when applied to solve supervised regression and classiﬁcation tasks.
As homotopy functions we adopt the ones discussed in Section 5.2.
We empirically show that homotopy methods outperform random and warm-start initialization schemes in terms of numerical performance.
In particular, when the target task is complex and/or, in the transfer-learning scenario, when the data distributions are signiﬁcantly different, continuation methods can achieve signiﬁcant speed-up compared to random and warm-start initializations.
We believe that their superior numerical performance relies on the use of homotopy functions that progressively deform the data distribution from an easy-to-solve or alreadysolved task to the target data distribution.
In addition, consistently across all the benchmarks, our homotopy-based method shows faster convergence than random-initialization and faster or comparable convergence than warm-start initialization.
When the source task is “similar” to the target one, there is indeed no need to gradually vary the λ parameter in Algorithm 1, but it sufﬁces to directly set it to 1.
In this extreme case, our homotopy method boils down to warm-start initialization.
6. R EGRESSION
For the supervised regression scenario, the problem we address is how to transfer “optimality knowledge” across two tasks that involve regressing from the input to the output of two different sine wave functions with different values of phase ω .
Each considered dataset has 10000 samples split across training and testing, where both x and y are deﬁned as follows x
[Equation]
The goal is to start with an “easy-to-learn” task, i.e. ω ≈ 1 rad, whose optimum is available by performing only few epochs with a ﬁrst-order optimizer, e.g. SGD, Adam, and progressively transfer the “optimality knowledge” to a more complex task, i.e ω >> 1 rad, by approximately solving the homotopy problems for increasing values of λ as described in Algorithm 1.
We set ω = 1 rad for our source task distribution, and study the performance of the proposed approach with homotopy function as described in equation 14 for different target distributions with ω >> 1 rad.
See Figures 7a and 7b in the appendix for a visualization of the starting data distribution with ω =rad and the target data distribution when ω = 137 rad, respectively.
The regressor is a feedforward neural network with 6 hidden layers of 100 units each and relu as activation function.
In order to make the experiments more robust with respect to the choice of the step size α , we use Adam as optimizer.
For the experiments in Figures 1a–1b, Figures 8a–8b in the appendix, and Figure 2a, we set α = 0 .
001 , γ = 10 , k
= 200 and then performed an additional 500 epochs on the ﬁnal target problem, while for the experiments in Figure 2b, we set γ = 10 , k = 300 and performed an additional 600 epochs on the ﬁnal target problem.
In this last scenario we set α = 0 .
001 and then decrease it with a cosine annealing schedule to observe convergence to an optimum.
As shown in
Figures 1a–1b, Figures 8a–8b in the appendix, and Figures 2a and 2b, the homotopy method leads to faster convergence than the considered baselines by preserving the vicinity to an optimal solution for problems H ( θ, λ ) across the different λ values.
In particular, we achieve a training loss up to two orders of magnitude better than the considered baselines.
6. C LASSIFICATION
For the supervised classiﬁcation scenario, we ﬁrst apply the continuation method with the homotopy deformation described in equation 17 and equation 18 in order to transfer optimality from MNIST, a notoriously “easy-to-learn” task for neural networks, to the FashionMNIST task.
Since the two datasets have the same input dimensionality and the same number of classes, no additional preprocessing of the data is required.
As network architecture, we use a VGG-type of network, (Simonyan & Zisserman, 2015), and Adam as optimizer with a step size of α = 0 . 001 . [Figure] [Figure]
Secondly, we consider CIFAR-10 as target data distribution.
Differently from the previous scenario, padding of the MNIST samples is required in order to apply equation 17.
Also in this case we adopt a VGG-type of network, (Simonyan & Zisserman, 2015), and Adam as optimizer with a step size of α = 0 . 0001 .
As shown in Figures 3a and 3b, in both benchmarks the homotopy method leads to faster convergence than random initialization.
While in the second benchmark our method reaches a lower value of training loss in fewer epochs than warm-start, in the MNIST-to-FashionMNIST case the performance is comparable to using warm-start initialization.
A possible interpretation is that, when the source and target task distributions are “too similar”, as we hypothesize in the MNISTto-FashionMNIST scenario, then there is no need for homotopy deformations to be applied, i.e.
0 < λ < 1 , but we can directly apply λ = 1 in our scheme, which corresponds to warm-start initialization.
C ONCLUSIONS
In this paper we propose a new methodology based on homotopy methods in order to transfer knowledge across different task distributions.
In particular, our homotopy-based method allows one to exploit easy-to-solve or already-solved learning problems to solve new and complex tasks, by approximately and sequentially solving a sequence of optimization problems where the task distribution is [Figure] gradually deformed from the source to the target one.
We conduct a theoretical analysis of a general homotopy method in a simpliﬁed setting, and then we test our method in popular deep learning benchmarks, where it shows superior numerical performance compared to random and warm-start initialization schemes.
The proposed framework, in its limiting case, corresponds to the widely used ﬁne-tuning heuristic, allowing for a new and more rigorous interpretation of the latter.
Finally, the generality of homotopy methods also opens many novel and promising research directions in fundamental deep learning scenarios such as stochastic non-convex optimization, e g. by deﬁnition of homotopy functions across architectures, objective function terms and transfer-learning.
A A PPENDIX B P ROPERTIES OF H OMOTOPIC F UNCTIONS
Among the numerous properties of homotopic functions, for the formulation of our method (see
Section 5.2) we make use of the following properties
Proposition B.1.
Suppose that there exists a homotopy H : Z × [0 , 1] → Y from g to f , i.e. g (cid:39) f .
Then
• g (cid:39) g (reﬂexive property) • g (cid:39) f
= ⇒ f (cid:39) g (symmetric property) • g (cid:39) f and f (cid:39) h
= ⇒ g (cid:39) h (transitive property)
Proof.
See proof of Theorem 1.5 in (Suciu, 2016).
Proposition B.2.
Let g, g (cid:48) :
Z → Y and f, f (cid:48) :
Y → W be continuous maps, and let f ◦ g, f (cid:48) ◦ g (cid:48) :
Z → W be the respective composite maps.
If g (cid:39) g (cid:48) and f (cid:39) f (cid:48) , then f ◦ g (cid:39) f (cid:48) ◦ g (cid:48) .
Proof.
See proof of Proposition 1.7 in (Suciu, 2016).
C A PPROXIMATION VIA G AUSSIAN F ILTER
For the supervised regression, we propose the following homotopy deformation
[Equation]
A downside of this homotopy function is that the same support for x is required (the absence of the subscripts s and τ on x stands to indicate that the same realization for x s and x τ has to be considered).
Alternatively, it is possible to approximate equation 20 by using a Gaussian ﬁlter, as depicted in Figure 5b.
In particular, having sampled one realization j of the pair ( x s , y s ) from the training set D s , M >realizations of the pair ( x τ , y τ ) are sampled from D τ .
Each y τ,i realization is then weighted based on the vicinity of x τ,i to the sampled x s,j realization.
This leads to the following approximation of the j realization of y λ
[Equation]
where ξ > 0 is the standard deviation of the Gaussian ﬁlter.
D A DDITIONAL F IGURES
[Figure]
The homotopy represented Figure 4a results in a mixture of Gaussian distributions, with mixture coefﬁcient given by the homotopy parameter λ .
In Figure 4b the deformation concerns instead the parameters µ and σ of the original distributions.
Preserving unimodality is a desirable property when the homotopy function is used in combination with a continuation method since, as shown in Figure 4b, the location of the optimum moves together with the function deformation, allowing the optimizer to track it and gradually reach the optimum of the ﬁnal target task.
On the contrary, as shown in Figure 4a, deforming the function does not lead to a gradual shift of the optimal solutions.
Consequently, approximately and sequentially solving the problems corresponding to intermediate values of the homotopy parameter λ , i.e. 0 < λ < 1 , will not allow the homotopy method to gradually approach the desired ﬁnal optimal solution.
[Figure]
[Figure] [Figure]
[Figure]
E E RROR B OUNDS ON SGD I TERATES
Considering problem 4 with ﬁxed parameter
λ i we will refer to θ k
= θ
i
k where we drop the subscript i in order to simplify the notation.
The analysis holds for all ﬁxed parameters
λ i .
Let us use SGD to solve the following optimization problem
[Equation]
where the objective function H fulﬁlls Assumptions 4.2 and 4.4.
We now derive error bounds on the iterates of SGD
[Equation] [Equation]
where ˜ g k is an unbiased estimate of ∇ H ( θ k , λ i ) w.r.t.
θ k and 0 < α < 12 µ is the step size.
We start by applying the deﬁnition of SGD iterates
[Equation]
We now take the expectation w.r.t.
θ 0 , ˜ g 0 , . .
˜ g k − 1 , ˜ g k and, considering Assumptions 4.2 and 4.4, we obtain
[Equation]
Applying this result recursively
[Equation]
we derive the following bound on the error associated with SGD iterates
[Equation]
F P ROOF OF T HEOREM 4.
Theorem F.1.
Consider Algorithm 1 with Stochastic Gradient Descent as solver and let k > 0 be the number of iterations and
0 < α < 12 µ be the step size.
For r θ ∈ R such that
[Equation]
then, if E (cid:2)
||
θ i − θ ∗
i
|| 2 (cid:3) ≤ r 2 θ
and || λ
i − λ
i +1 || ≤ ˜ (cid:15) , where ˜ (cid:15) :
=
min { ¯ (cid:15), (cid:15) } with
[Equation]
the following inequality holds
[Equation]
Proof.
[Equation]
We now solve in ˜ (cid:15) the following second degree inequality
[Equation]
The inequality 31 admits solutions if and only if r 2 θ > αC 2 2 µ (1 − (1 − 2 αµ ) k ) .
Considering the conditions on the learning rate α , then the inequality admits solutions for values of r θ such that r 2 θ >
[Equation]
In particular, inequality 31 holds ∀ ˜ (cid:15) ∈ [0 , ¯ (cid:15) ] , where ¯ (cid:15) =
− r θ δ +
[Equation]
G T HEORETICAL A NALYSIS IN THE N ON -C ONVEX S ETTING G. R EALIZATION OF THE I TERATES IN THE S TRONGLY C ONVEX R EGION
Considering problem 4 with ﬁxed parameter
λ i , we will refer to θ k
= θ i,k , where we drop the subscript i in order to simplify the notation.
The analysis holds for all ﬁxed parameters
λ i .
In particular, we address the following optimization problem
[Equation]
Let assume that strong convexity only holds in a ball of radius B around the optimal solution θ ∗ , i.e. local strong convexity.
Let also assume that deterministic gradient descent converges linearly with contraction rate 0 < k d <
[Equation]
for any θ k such that || θ k − θ ∗ || ≤ B , and superscript D denotes iterates obtained by applying the full gradient g k := ∇ H ( θ k , λ i )
[Equation]
[Equation]
We also introduce the assumption that the norm of each gradient component ∇ (cid:96) j is upper bounded by a constant ν ||∇ (cid:96) j
( θ, λ i ) |
[Equation]
≤
[Equation]
=
[Equation]
N
[Equation]
(35) and ∀ θ s.t.
||
θ − θ ∗ || ≤ B .
Let θ k +1 denote the iterate obtained by applying one iteration of stochastic gradient descent
[Equation]
36) where ˜
g k :=
1 M (cid:80) j ∈M ∇ (cid:96) j ( θ k , λ i ) and M is a set of 0 < M < N indexes randomly sampled from 1 , . . . , N .
Given any realization of θ k s.t.
||
θ k − θ ∗
|| <
B and any realization of ˜ g k , we have that
[Equation]
Since we have assumed that the current realization of θ k is in the ball of radius B around the optimal solution θ ∗ , it directly follows from equation 37 that whenever α ( N − M ) ≤ (1 − k d ) B ν , the realization of θ k +1 will also lie in this ball.
These derivations show that when the realization of the current iterate θ k lies in the ball of radius B around the optimum θ ∗ , and α ( N − M ) ≤ (1 − k d ) B ν , then the next iterate θ k +1 will also lie in the ball.
Consequently, if we assume that the initial point θ 0 is in the ball of radius B , then, by applying the derivations recursively, we can show that the iterates will remain in the set where strong convexity holds.
Consequently, we can now apply locally the error bounds on SGD iterates derived in equation 26.
G. G ENERALIZATION OF T HEOREM 4.9 FOR L OCALLY S TRONGLY C ONVEX P ROBLEMS
Let θ i be an approximation of θ ∗ i as deﬁned in 4 and assume that the parameter λ i is subject to a sufﬁciently small variation, i.e || λ i − λ i +
|| ≤ (cid:15) .
In order to apply the results derived in Theorem 4.9, given a realization of θ i , we are interested in showing what the conditions are on || θ i − θ ∗
i ||
such that || θ
i − θ ∗ i +1 || ≤ B .
Proposition G.1.
Let λ i +1 be such
that ||
λ
i − λ
i +
|| ≤ (cid:15) , with 0 ≤ (cid:15) ≤ B δ .
If || θ i − θ ∗ i || ≤ B − δ(cid:15) , then ||
θ i − θ ∗ i +
|| ≤ B .
Proof.
[Equation]
Finally, using the fact that || λ
i − λ
i +
|| ≤ (cid:15) , it follows that, if || θ i − θ ∗
i || ≤ B − δ(cid:15) ,
then || θ i − θ ∗ i +
|| ≤
B .
Given Proposition G.1 and the results from Section G.1, it is straightforward to show that any realization of the iterates will lie in the region around the optimal solution where strong convexity holds by assumption.
In particular, the derivations in Theorem 4.9 are directly applicable and allow us to obtain the same results locally also in the non-convex scenario.