確率的勾配降下法
m 0 = v 0 = 0 g t = ∇ Q i ( w ) m t = β 1 m t − 1 + ( 1 − β 1 ) g t v t = β 2 v t − 1 + ( 1 − β 2 ) g t 2 , and V t = diag ( v t ) η ^ t = Clip ( α V t , η l ( t ) , η u ( t ) ) w t = w t − 1 − η ^ t t ∘ m t {\displaystyle {\begin{aligned}m_{0}&=v_{0}=0\\g_{t}&=\nabla Q_{i}(w)\\m_{t}&=\beta _{1}m_{t-1}+(1-\beta _{1})g_{t}\\v_{t}&=\beta _{2}v_{t-1}+(1-\beta _{2})g_{t}^{2},{\text{and }}V_{t}={\text{diag}}(v_{t})\\{\hat {\eta }}_{t}&={\text{Clip}}({\frac {\alpha }{\sqrt {V_{t}}}},\eta _{l}(t),\eta _{u}(t))\\w_{t}&=w_{t-1}-{\frac {{\hat {\eta }}_{t}}{\sqrt {t}}}\circ m_{t}\end{aligned}}}