We study the behavior of stochastic gradient descent applied to \|Ax -b \|_2^2 \rightarrow \min for invertible A \in R^{n \times n}. We show that there is an explicit constant c_{A} depending (mildly) on A such that E ~\left\| Ax_{k+1}-b\right\|^2_{2} \leq \left(1 + c_{A}{\|A\|_F^2}\right) \left\|A x_k -b \right\|^2_{2} - 2{\|A\|_F^2} \left\|A^T (Ax _k - b )\right\|^2_{2}. This is a curious inequality: when applied to a discretization of a partial differential equation like -\Delta u = f, the last term measures the regularity of the residual u_k - u in a higher Sobolev space than the remaining terms: if u _k - u has large fourth derivatives (i.e. bi-Laplacian \Delta^2), then SGD will dramatically decrease the size of the second derivatives (i. e. \Delta) of u_k - u. For symmetric matrices, this inequality has an extension to higher-order Sobolev spaces. This implies a regularization phenomenon: an energy cascade from large singular values to small singular values acts as regularizer .
We study the behavior of stochastic gradient descent applied to \|Ax -b \|_2^2 \rightarrow \min for invertible A \in R^{n \times n}. We show that there is an explicit constant c_{A} depending (mildly) on A such that E ~\left\| Ax_{k+1}-b\right\|^2_{2} \leq \left(1 + c_{A}{\|A\|_F^2}\right) \left\|A x_k -b \right\|^2_{2} - 2{\|A\|_F^2} \left\|A^T A (x _k - x )\right\|^2_{2}. This is a curious inequality: the last term has one more matrix applied to the residual u_k - u than the remaining terms: if x _k - x is mainly comprised of large singular vectors, stochastic gradient descent leads to a quick regularization. For symmetric matrices, this inequality has an extension to higher-order Sobolev spaces. This explains a (known) regularization phenomenon: an energy cascade from large singular values to small singular values smoothes .
R
(Ax
A (x
R
b
x
D
when applied to a discretization of a partial differential equation like -\Delta u = f,
R
measures the regularity of the
has one more matrix applied to the
D
in a higher Sobolev space
R
u
x
R
u has large fourth derivatives (i.e. bi-Laplacian \Delta^2), then SGD will dramatically decrease the size of the second derivatives (i. e. \Delta) of u_k - u.
x is mainly comprised of large singular vectors, stochastic gradient descent leads to a quick regularization.
R
implies a
explains a (known)
R
acts as regularizer
smoothes