Module Expr Str ImportFrom alias ImportFrom alias ImportFrom alias alias alias ClassDef bases NameLoad body Expr Str Assign NameStore Call AttributeLoad Str attr FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults Num Num Num Num NameLoad body Expr Str Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad keyword NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Num decorator_list FunctionDef arguments args NameParam defaults body For TupleStore NameStore NameStore AttributeLoad NameLoad attr body Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore BinOpMult NameLoad NameLoad AugAssignSub AttributeStore NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad NameLoad Assign AttributeStore NameLoad attr BinOpAdd BinOpMult NameLoad AttributeLoad NameLoad attr BinOpMult BinOpSub Num NameLoad BinOpPow NameLoad Num Assign NameStore BinOpMult BinOpMult BinOpDiv Call NameLoad BinOpAdd AttributeLoad NameLoad attr NameLoad Call NameLoad BinOpAdd AttributeLoad NameLoad attr NameLoad NameLoad AttributeLoad NameLoad attr AugAssignSub AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr BinOpAdd NameLoad NameLoad Assign AttributeStore NameLoad attr BinOpAdd BinOpMult NameLoad AttributeLoad NameLoad attr BinOpMult BinOpSub Num NameLoad BinOpPow AttributeLoad NameLoad attr Num AugAssignAdd AttributeStore NameLoad attr Num Expr Yield Dict Str Str Str Str AttributeLoad NameLoad attr NameLoad NameLoad NameLoad decorator_list decorator_list
N/A N/A This module provides an implementation of adadelta. __future__ absolute_import base Minimizer mathadapt sqrt ones_like clip Adadelta N/A Minimizer N/A N/A Adadelta optimizer.

    Adadelta [zeiler2013adadelta]_ is a method that uses the magnitude of recent
    gradients and steps to obtain an adaptive step rate. An exponential moving
    average over the gradients and steps is kept; a scale of the learning rate
    is then obtained by their ration.

    Let :math:`f'(\theta_t)` be the derivative of the loss with respect to the
    parameters at time step :math:`t`. In its
    basic form, given a step rate :math:`\alpha`, a decay term
    :math:`\gamma` and an offset :math:`\epsilon` we perform the following
    updates:

    .. math::
       g_t &=& (1 - \gamma)~f'(\theta_t)^2 + \gamma g_{t-1}

    where :math:`g_0 = 0`. Let :math:`s_0 = 0` for updating the parameters:

    .. math::
       \Delta \theta_t &=& \alpha {\sqrt{s_{t-1} + \epsilon} \over \sqrt{g_t + \epsilon}}~f'(\theta_t), \\
       \theta_{t+1} &=& \theta_t - \Delta \theta_t.

    Subsequently we adapt the moving average of the steps:

    .. math::
       s_t &=& (1 - \gamma)~\Delta\theta_t^2 + \gamma s_{t-1}.

    To extend this with Nesterov's accelerated gradient, we need a momentum
    coefficient :math:`\beta` and incorporate it by using slightly different
    formulas:

    .. math::
        \theta_{t + {1 \over 2}} &=& \theta_t - \beta \Delta \theta_{t-1}, \\
       g_t &=& (1 - \gamma)~f'(\theta_{t + {1 \over 2}})^2 + \gamma g_{t-1}, \\
       \Delta \theta_t &=& \alpha {\sqrt{s_{t-1} + \epsilon} \over \sqrt{g_t + \epsilon}}~f'(\theta_{t + {1 \over 2}}).

    In its original formulation, the case :math:`\alpha = 1, \beta = 0` was
    considered only.

    .. [zeiler2013adadelta] Zeiler, Matthew D.
       "ADADELTA: An adaptive learning rate method."
       arXiv preprint arXiv:1212.5701 (2012).
     N/A state_fields N/A N/A n_iter gms sms step step_rate decay offset momentum split __init__ N/A N/A self wrt fprime step_rate decay momentum offset args N/A 1 0.9 0 0.0001 None N/A N/A Create an Adadelta object.

        Parameters
        ----------

        wrt : array_like
            Array that represents the solution. Will be operated upon in
            place.  ``fprime`` should accept this array as a first argument.

        fprime : callable
            Callable that given a solution vector as first parameter and *args
            and **kwargs drawn from the iterations ``args`` returns a
            search direction, such as a gradient.

        step_rate : scalar or array_like, optional [default: 1]
            Value to multiply steps with before they are applied to the
            parameter vector.

        decay : float, optional [default: 0.9]
            Decay parameter for the moving average. Must lie in [0, 1) where
            lower numbers means a shorter "memory".

        momentum : float or array_like, optional [default: 0]
          Momentum to use during optimization. Can be specified analoguously
          (but independent of) step rate.

        offset : float, optional, [default: 1e-4]
            Before taking the square root of the running averages, this offset
            is added.

        args : iterable
            Iterator over arguments which ``fprime`` will be called with.
         N/A N/A N/A N/A super Adadelta self __init__ wrt args args N/A N/A self fprime fprime N/A N/A self step_rate step_rate N/A N/A self decay decay N/A N/A self offset offset N/A N/A self momentum momentum N/A N/A self gms 0 N/A N/A self sms 0 N/A N/A self step 0 N/A _iterate N/A N/A self N/A N/A N/A N/A args kwargs N/A self args N/A N/A step_m1 N/A self step N/A d N/A self decay N/A o N/A self offset N/A m N/A self momentum N/A step1 N/A step_m1 m N/A N/A self wrt step1 N/A gradient N/A N/A self fprime N/A self wrt args kwargs N/A N/A self gms N/A N/A d N/A self gms N/A N/A 1 d N/A gradient 2 N/A step2 N/A N/A N/A N/A sqrt N/A N/A self sms o N/A sqrt N/A N/A self gms o gradient N/A self step_rate N/A N/A self wrt step2 N/A N/A self step N/A step1 step2 N/A N/A self sms N/A N/A d N/A self sms N/A N/A 1 d N/A N/A self step 2 N/A N/A self n_iter 1 N/A N/A N/A n_iter gradient args kwargs N/A self n_iter gradient args kwargs N/A N/A