We provide new adaptive first-order methods for constrained convex optimization. Our main algorithm AdaAGD+ is an accelerated method, which is universal in the sense that it achieves nearly-optimal convergence rates for both smooth and non-smooth functions, even when it only has access to stochastic gradients. In addition, it does not require any prior knowledge on how the objective function is parametrized, since it automatically adjusts its per-coordinate learning rate. This can be seen as a truly accelerated AdaGrad method for constrained optimization. We complement it with a simpler algorithm AdaGrad+ which enjoys the same features, and achieves the standard non-accelerated convergence rate. We also present a set of new results involving adaptive methods for unconstrained optimization and monotone operators.
We provide new adaptive first-order methods for constrained convex optimization. Our main algorithms AdaACSA and AdaAGD+ are accelerated methods, which are universal in the sense that they achieve nearly-optimal convergence rates for both smooth and non-smooth functions, even when they only have access to stochastic gradients. In addition, they do not require any prior knowledge on how the objective function is parametrized, since they automatically adjust their per-coordinate learning rate. These can be seen as truly accelerated Adagrad methods for constrained optimization. We complement them with a simpler algorithm AdaGrad+ which enjoys the same features, and achieves the standard non-accelerated convergence rate. We also present a set of new results involving adaptive methods for unconstrained optimization and monotone operators.
R
algorithm
algorithms AdaACSA and
R
is an accelerated method, which is
are accelerated methods, which are
R
it achieves
they achieve
R
it only has
they only have
R
it does
they do
R
it automatically adjusts its
they automatically adjust their
R
This
These
R
a truly accelerated AdaGrad method
truly accelerated Adagrad methods
R
it
them