Module Expr Str ImportFrom alias alias ImportFrom alias ImportFrom alias ImportFrom alias alias alias alias alias alias ImportFrom alias alias ImportFrom alias alias ImportFrom alias alias ImportFrom alias alias alias ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam defaults kwarg body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad decorator_list Call NameLoad keyword ListLoad Str Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Return NameLoad decorator_list Call NameLoad keyword ListLoad Str keyword ListLoad Str FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad kwarg body Expr Str Pass decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Pass decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body If CompareIn NameLoad ListLoad Str Str body Return AttributeLoad NameLoad attr If CompareIn NameLoad ListLoad Str body Return Num Return Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign NameStore BinOpSub NameLoad Call AttributeLoad NameLoad attr keyword Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad If NameLoad body AugAssignMult NameStore NameLoad Assign NameStore BinOpAdd Call AttributeLoad NameLoad attr keyword Num Call AttributeLoad NameLoad attr BinOpSub Num NameLoad keyword Num Return BinOpDiv NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Return Call AttributeLoad BinOpMult Call AttributeLoad NameLoad attr NameLoad NameLoad attr keyword Num decorator_list NameLoad decorator_list ClassDef bases NameLoad NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad kwarg body If UnaryOpNot NameLoad body Assign NameStore Call NameLoad keyword NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call NameLoad keyword SubscriptLoad NameLoad Index Str keyword NameLoad keyword Str If UnaryOpNot NameLoad body Assign NameStore Call NameLoad keyword Str If UnaryOpNot NameLoad body Assign NameStore Call NameLoad keyword Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign NameStore BinOpAdd ListLoad AttributeLoad NameLoad attr NameLoad NameLoad Call AttributeLoad NameLoad attr Str ListLoad Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr keyword NameLoad NameLoad decorator_list Call NameLoad keyword ListLoad Str FunctionDef arguments args NameParam defaults body Assign AttributeStore AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign AttributeStore AttributeLoad NameLoad attr attr ListComp AttributeLoad NameLoad attr comprehension NameStore AttributeLoad NameLoad attr Assign AttributeStore AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign AttributeStore AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign AttributeStore AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign AttributeStore AttributeLoad NameLoad attr attr Num decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body If UnaryOpNot NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad NameLoad Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Slice Num keyword BinOpSub AttributeLoad NameLoad attr Num Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad kwarg body Expr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Return TupleLoad NameLoad AttributeLoad NameLoad attr decorator_list Call NameLoad keyword ListLoad Str Str FunctionDef arguments args NameParam defaults body Return BinOpAdd ListLoad Str Str Str AttributeLoad NameLoad attr decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam NameParam NameParam defaults body Return ListLoad Call AttributeLoad NameLoad attr TupleLoad NameLoad AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr TupleLoad NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Num decorator_list Call NameLoad keyword ListLoad Str Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list Call NameLoad keyword ListLoad Str keyword ListLoad Str FunctionDef arguments args NameParam NameParam defaults body If CompareIn NameLoad ListLoad Str body Return AttributeLoad NameLoad attr If CompareIn NameLoad ListLoad Str body Return Num Return Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad decorator_list decorator_list ClassDef bases NameLoad NameLoad NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults NameLoad kwarg body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr ListLoad AttributeLoad Call NameLoad attr AttributeLoad Call NameLoad keyword NameLoad attr NameLoad decorator_list Call NameLoad FunctionDef arguments args NameParam defaults body Return AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Num attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore SubscriptLoad AttributeLoad NameLoad attr Index Num attr NameLoad decorator_list AttributeLoad NameLoad attr FunctionDef arguments args NameParam defaults body Return AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Num attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore SubscriptLoad AttributeLoad NameLoad attr Index Num attr NameLoad decorator_list AttributeLoad NameLoad attr decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam defaults kwarg body Expr Str Pass decorator_list NameLoad FunctionDef arguments args NameParam defaults kwarg body Expr Str Pass decorator_list NameLoad FunctionDef arguments args NameParam defaults kwarg body Expr Str Pass decorator_list NameLoad decorator_list Call NameLoad NameLoad ClassDef bases NameLoad NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad NameLoad kwarg body Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad AttributeLoad NameLoad attr attr If NameLoad body If UnaryOpNot NameLoad body Assign NameStore Str If UnaryOpNot NameLoad body Assign NameStore Str AugAssignAdd AttributeStore NameLoad attr ListLoad NameLoad NameLoad orelse Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index Num Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index Num If UnaryOpNot NameLoad body Assign NameStore ListComp NameLoad comprehension NameStore AttributeLoad NameLoad attr CompareNotIn Str NameLoad Assign NameStore Call NameLoad NameLoad SubscriptLoad AttributeLoad AttributeLoad NameLoad attr attr Index Num Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr BinOpAdd Str AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Assign AttributeStore NameLoad attr ListComp NameLoad comprehension NameStore AttributeLoad NameLoad attr CompareIn NameLoad AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Assign NameStore ListLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AugAssignAdd NameStore Call AttributeLoad NameLoad attr Str ListLoad Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr keyword NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign AttributeStore AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults kwarg body Expr Str Assign NameStore Call NameLoad NameLoad AttributeLoad NameLoad attr keyword NameLoad Assign NameStore Call NameLoad NameLoad AttributeLoad NameLoad attr keyword NameLoad Assign NameStore Call NameLoad NameLoad AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Call NameLoad NameLoad NameLoad Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults kwarg body Expr Str Assign NameStore ListComp NameLoad comprehension NameStore AttributeLoad NameLoad attr CompareNotIn Str NameLoad Assign NameStore Call NameLoad NameLoad NameLoad keyword NameLoad Assign NameStore Call NameLoad NameLoad AttributeLoad NameLoad attr keyword NameLoad If AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad Call NameLoad Call NameLoad NameLoad NameLoad AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad keyword NameLoad Call NameLoad NameLoad NameLoad Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults kwarg body Expr Str Assign NameStore SubscriptLoad NameLoad Index AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Assign NameStore Call NameLoad NameLoad AttributeLoad NameLoad attr keyword NameLoad Assign NameStore Call NameLoad NameLoad AttributeLoad NameLoad attr keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad Call NameLoad NameLoad NameLoad Dict AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad Call NameLoad NameLoad NameLoad NameLoad NameLoad Return BinOpAdd NameLoad Call NameLoad Call AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults body Return BinOpAdd AttributeLoad NameLoad attr ListLoad AttributeLoad NameLoad attr decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults body Return BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults body Return BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam defaults kwarg body Expr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad NameLoad Index AttributeLoad NameLoad attr Return Call AttributeLoad NameLoad attr Call NameLoad NameLoad Dict AttributeLoad NameLoad attr NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad decorator_list AttributeLoad NameLoad attr FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam NameParam defaults kwarg body Return BinOpAdd Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad SubscriptLoad NameLoad Index AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return AttributeLoad AttributeLoad NameLoad attr attr decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam NameParam defaults body If CompareIn NameLoad AttributeLoad NameLoad attr body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad If CompareEq NameLoad AttributeLoad NameLoad attr body Assign TupleStore NameStore AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad If AttributeLoad NameLoad attr body If CompareEq NameLoad AttributeLoad NameLoad attr body Return Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Index Num If CompareEq NameLoad AttributeLoad NameLoad attr body Return Num Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list decorator_list
N/A N/A Attention mechanisms.

This module defines the interface of attention mechanisms and a few
concrete implementations. For a gentle introduction and usage examples see
the tutorial TODO.

An attention mechanism decides to what part of the input to pay attention.
It is typically used as a component of a recurrent network, though one can
imagine it used in other conditions as well. When the input is big and has
certain structure, for instance when it is sequence or an image, an
attention mechanism can be applied to extract only information which is
relevant for the network in its current state.

For the purpose of documentation clarity, we fix the following terminology
in this file:

* *network* is the network, typically a recurrent one, which
  uses the attention mechanism.

* The network has *states*. Using this word in plural might seem weird, but
  some recurrent networks like :class:`~blocks.bricks.recurrent.LSTM` do
  have several states.

* The big structured input, to which the attention mechanism is applied,
  is called the *attended*. When it has variable structure, e.g. a sequence
  of variable length, there might be a *mask* associated with it.

* The information extracted by the attention from the attended is called
  *glimpse*, more specifically *glimpses* because there might be a few
  pieces of this information.

Using this terminology, the attention mechanism computes glimpses
given the states of the network and the attended.

An example: in the machine translation network from [BCB]_ the attended is
a sequence of so-called annotations, that is states of a bidirectional
network that was driven by word embeddings of the source sentence. The
attention mechanism assigns weights to the annotations. The weighted sum of
the annotations is further used by the translation network to predict the
next word of the generated translation. The weights and the weighted sum
are the glimpses. A generalized attention mechanism for this paper is
represented here as :class:`SequenceContentAttention`.

 abc ABCMeta abstractmethod theano tensor six add_metaclass blocks.bricks Brick Initializable Sequence Feedforward Linear Tanh blocks.bricks.base lazy application blocks.bricks.parallel Parallel Distribute blocks.bricks.recurrent recurrent BaseRecurrent blocks.utils dict_union dict_subset pack AbstractAttention N/A Brick N/A N/A The common interface for attention bricks.

    First, see the module-level docstring for terminology.

    A generic attention mechanism functions as follows. Its inputs are the
    states of the network and the attended. Given these two it produces
    so-called *glimpses*, that is it extracts information from the attended
    which is necessary for the network in its current states

    For computational reasons we separate the process described above into
    two stages:

    1. The preprocessing stage, :meth:`preprocess`, includes computation
    that do not involve the state. Those can be often performed in advance.
    The outcome of this stage is called *preprocessed_attended*.

    2. The main stage, :meth:`take_glimpses`, includes all the rest.

    When an attention mechanism is applied sequentially, some glimpses from
    the previous step might be necessary to compute the new ones.  A
    typical example for that is when the focus position from the previous
    step is required. In such cases :meth:`take_glimpses` should specify
    such need in its interface (its docstring explains how to do that). In
    addition :meth:`initial_glimpses` should specify some sensible
    initialization for the glimpses to be carried over.

    .. todo::

        Only single attended is currently allowed.

        :meth:`preprocess` and :meth:`initial_glimpses` might end up
        needing masks, which are currently not provided for them.

    Parameters
    ----------
    state_names : list
        The names of the network states.
    state_dims : list
        The state dimensions corresponding to `state_names`.
    attended_dim : int
        The dimension of the attended.

    Attributes
    ----------
    state_names : list
    state_dims : list
    attended_dim : int

     __init__ N/A N/A self state_names state_dims attended_dim N/A kwargs N/A N/A N/A self state_names state_names N/A N/A self state_dims state_dims N/A N/A self attended_dim attended_dim N/A N/A N/A N/A super AbstractAttention self __init__ kwargs N/A N/A lazy allocation N/A state_names state_dims attended_dim preprocess N/A N/A self attended N/A N/A N/A Perform the preprocessing of the attended.

        Stage 1 of the attention mechanism, see :class:`AbstractAttention`
        docstring for an explanation of stages. The default implementation
        simply returns attended.

        Parameters
        ----------
        attended : :class:`~theano.Variable`
            The attended.

        Returns
        -------
        preprocessed_attended : :class:`~theano.Variable`
            The preprocessed attended.

         N/A attended N/A N/A application inputs N/A attended outputs N/A preprocessed_attended take_glimpses N/A N/A self attended preprocessed_attended attended_mask N/A None None kwargs N/A N/A Extract glimpses from the attended given the current states.

        Stage 2 of the attention mechanism, see :class:`AbstractAttention`
        for an explanation of stages. If `preprocessed_attended` is not
        given, should trigger the stage 1.

        This application method *must* declare its inputs and outputs.
        The glimpses to be carried over are identified by their presence
        in both inputs and outputs list. The attended *must* be the first
        input, the preprocessed attended *must* be the second one.

        Parameters
        ----------
        attended : :class:`~theano.Variable`
            The attended.
        preprocessed_attended : :class:`~theano.Variable`, optional
            The preprocessed attended computed by :meth:`preprocess`.  When
            not given, :meth:`preprocess` should be called.
        attended_mask : :class:`~theano.Variable`, optional
            The mask for the attended. This is required in the case of
            padded structured output, e.g. when a number of sequences are
            force to be the same length. The mask identifies position of
            the `attended` that actually contain information.
        \*\*kwargs : dict
            Includes the states and the glimpses to be carried over from
            the previous step in the case when the attention mechanism is
            applied sequentially.

         N/A N/A abstractmethod initial_glimpses N/A N/A self batch_size attended N/A N/A N/A Return sensible initial values for carried over glimpses.

        Parameters
        ----------
        batch_size : int or :class:`~theano.Variable`
            The batch size.
        attended : :class:`~theano.Variable`
            The attended.

        Returns
        -------
        initial_glimpses : list of :class:`~theano.Variable`
            The initial values for the requested glimpses. These might
            simply consist of zeros or be somehow extracted from
            the attended.

         N/A N/A abstractmethod get_dim N/A N/A self name N/A N/A N/A N/A name N/A attended preprocessed_attended N/A N/A N/A self attended_dim N/A N/A name N/A attended_mask N/A N/A 0 N/A N/A N/A N/A super AbstractAttention self get_dim name N/A N/A GenericSequenceAttention N/A AbstractAttention N/A N/A Logic common for sequence attention mechanisms. compute_weights N/A N/A self energies attended_mask N/A N/A N/A Compute weights from energies in softmax-like fashion.

        .. todo ::

            Use :class:`~blocks.bricks.Softmax`.

        Parameters
        ----------
        energies : :class:`~theano.Variable`
            The energies. Must be of the same shape as the mask.
        attended_mask : :class:`~theano.Variable`
            The mask for the attended. The index in the sequence must be
            the first dimension.

        Returns
        -------
        weights : :class:`~theano.Variable`
            Summing to 1 non-negative weights of the same shape
            as `energies`.

         N/A energies N/A energies N/A N/A energies max axis 0 N/A unnormalized_weights N/A N/A tensor exp energies N/A attended_mask N/A N/A unnormalized_weights attended_mask N/A normalization N/A N/A N/A unnormalized_weights sum axis 0 N/A N/A tensor all N/A 1 attended_mask axis 0 N/A N/A unnormalized_weights normalization N/A application compute_weighted_averages N/A N/A self weights attended N/A N/A N/A Compute weighted averages of the attended sequence vectors.

        Parameters
        ----------
        weights : :class:`~theano.Variable`
            The weights. The shape must be equal to the attended shape
            without the last dimension.
        attended : :class:`~theano.Variable`
            The attended. The index in the sequence must be the first
            dimension.

        Returns
        -------
        weighted_averages : :class:`~theano.Variable`
            The weighted averages of the attended elements. The shape
            is equal to the attended shape with the first dimension
            dropped.

         N/A N/A N/A N/A N/A N/A tensor shape_padright weights attended sum axis 0 N/A application N/A SequenceContentAttention N/A GenericSequenceAttention Initializable N/A N/A Attention mechanism that looks for relevant content in a sequence.

    This is the attention mechanism used in [BCB]_. The idea in a nutshell:

    1. The states and the sequence are transformed independently,

    2. The transformed states are summed with every transformed sequence
       element to obtain *match vectors*,

    3. A match vector is transformed into a single number interpreted as
       *energy*,

    4. Energies are normalized in softmax-like fashion. The resulting
       summing to one weights are called *attention weights*,

    5. Weighted average of the sequence elements with attention weights
       is computed.

    In terms of the :class:`AbstractAttention` documentation, the sequence
    is the attended. The weighted averages from 5 and the attention
    weights from 4 form the set of glimpses produced by this attention
    mechanism.

    Parameters
    ----------
    state_names : list of str
        The names of the network states.
    attended_dim : int
        The dimension of the sequence elements.
    match_dim : int
        The dimension of the match vector.
    state_transformer : :class:`.Brick`
        A prototype for state transformations. If ``None``,
        a linear transformation is used.
    attended_transformer : :class:`.Feedforward`
        The transformation to be applied to the sequence. If ``None`` an
        affine transformation is used.
    energy_computer : :class:`.Feedforward`
        Computes energy from the match vector. If ``None``, an affine
        transformations preceeded by :math:`tanh` is used.

    Notes
    -----
    See :class:`.Initializable` for initialization parameters.

    .. [BCB] Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio. Neural
       Machine Translation by Jointly Learning to Align and Translate.

     __init__ N/A N/A self match_dim state_transformer attended_transformer energy_computer N/A None None None kwargs N/A N/A N/A state_transformer N/A N/A state_transformer N/A Linear use_bias False N/A N/A self match_dim match_dim N/A N/A self state_transformer state_transformer N/A N/A self state_transformers N/A Parallel input_names N/A kwargs N/A state_names prototype state_transformer name state_trans N/A N/A attended_transformer N/A N/A attended_transformer N/A Linear name preprocess N/A N/A energy_computer N/A N/A energy_computer N/A ShallowEnergyComputer name energy_comp N/A N/A self attended_transformer attended_transformer N/A N/A self energy_computer energy_computer N/A children N/A N/A N/A self state_transformers attended_transformer energy_computer N/A N/A kwargs get children N/A N/A N/A N/A N/A super SequenceContentAttention self __init__ children children kwargs N/A N/A lazy allocation N/A match_dim _push_allocation_config N/A N/A self N/A N/A N/A N/A N/A self state_transformers input_dims N/A self state_dims N/A N/A N/A self state_transformers output_dims N/A N/A self match_dim N/A name N/A self state_names N/A N/A N/A self attended_transformer input_dim N/A self attended_dim N/A N/A N/A self attended_transformer output_dim N/A self match_dim N/A N/A N/A self energy_computer input_dim N/A self match_dim N/A N/A N/A self energy_computer output_dim 1 N/A compute_energies N/A N/A self attended preprocessed_attended states N/A N/A N/A N/A preprocessed_attended N/A N/A preprocessed_attended N/A N/A self preprocess attended N/A transformed_states N/A N/A N/A self state_transformers apply as_dict True states N/A match_vectors N/A sum N/A N/A transformed_states values preprocessed_attended N/A energies N/A N/A N/A N/A N/A self energy_computer apply match_vectors reshape N/A N/A match_vectors shape N/A -1 ndim N/A N/A match_vectors ndim 1 N/A energies N/A application take_glimpses N/A N/A self attended preprocessed_attended attended_mask N/A None None states N/A N/A Compute attention weights and produce glimpses.

        Parameters
        ----------
        attended : :class:`~tensor.TensorVariable`
            The sequence, time is the 1-st dimension.
        preprocessed_attended : :class:`~tensor.TensorVariable`
            The preprocessed sequence. If ``None``, is computed by calling
            :meth:`preprocess`.
        attended_mask : :class:`~tensor.TensorVariable`
            A 0/1 mask specifying available data. 0 means that the
            corresponding sequence element is fake.
        \*\*states
            The states of the network.

        Returns
        -------
        weighted_averages : :class:`~theano.Variable`
            Linear combinations of sequence elements with the attention
            weights.
        weights : :class:`~theano.Variable`
            The attention weights. The first dimension is batch, the second
            is time.

         N/A energies N/A N/A self compute_energies attended preprocessed_attended states N/A weights N/A N/A self compute_weights energies attended_mask N/A weighted_averages N/A N/A self compute_weighted_averages weights attended N/A N/A weighted_averages N/A weights T N/A N/A application outputs N/A weighted_averages weights take_glimpses_inputs N/A N/A self N/A N/A N/A N/A N/A attended preprocessed_attended attended_mask N/A self state_names N/A N/A N/A take_glimpses property inputs initial_glimpses N/A N/A self batch_size attended N/A N/A N/A N/A N/A N/A tensor zeros N/A batch_size N/A self attended_dim N/A N/A tensor zeros N/A batch_size N/A N/A attended shape N/A 0 N/A N/A application outputs N/A weighted_averages weights preprocess N/A N/A self attended N/A N/A N/A Preprocess the sequence for computing attention weights.

        Parameters
        ----------
        attended : :class:`~tensor.TensorVariable`
            The attended sequence, time is the 1-st dimension.

         N/A N/A N/A N/A self attended_transformer apply attended N/A N/A application inputs N/A attended outputs N/A preprocessed_attended get_dim N/A N/A self name N/A N/A N/A N/A name N/A weighted_averages N/A N/A N/A self attended_dim N/A N/A name N/A weights N/A N/A 0 N/A N/A N/A N/A super SequenceContentAttention self get_dim name N/A N/A ShallowEnergyComputer N/A Sequence Initializable Feedforward N/A N/A A simple energy computer: first tanh, then weighted sum.

    Parameters
    ----------
    use_bias : bool, optional
        Whether a bias should be added to the energies. Does not change
        anything if softmax normalization is used to produce the attention
        weights, but might be useful when e.g. spherical softmax is used.

     __init__ N/A N/A self use_bias N/A False kwargs N/A N/A N/A N/A N/A super ShallowEnergyComputer self __init__ N/A N/A N/A Tanh apply N/A N/A Linear use_bias use_bias apply kwargs N/A N/A lazy input_dim N/A N/A self N/A N/A N/A N/A N/A N/A self children N/A 1 input_dim N/A property input_dim N/A N/A self value N/A N/A N/A N/A N/A N/A self children N/A 1 input_dim value N/A N/A input_dim setter output_dim N/A N/A self N/A N/A N/A N/A N/A N/A self children N/A 1 output_dim N/A property output_dim N/A N/A self value N/A N/A N/A N/A N/A N/A self children N/A 1 output_dim value N/A N/A output_dim setter N/A AbstractAttentionRecurrent N/A BaseRecurrent N/A N/A The interface for attention-equipped recurrent transitions.

    When a recurrent network is equipped with an attention mechanism its
    transition typically consists of two steps: (1) the glimpses are taken
    by the attention mechanism and (2) the next states are computed using
    the current states and the glimpses. It is required for certain
    usecases (such as sequence generator) that apart from a do-it-all
    recurrent application method interfaces for the first step and
    the second steps of the transition are provided.

     apply N/A N/A self N/A kwargs N/A N/A Compute next states taking glimpses on the way. N/A N/A abstractmethod take_glimpses N/A N/A self N/A kwargs N/A N/A Compute glimpses given the current states. N/A N/A abstractmethod compute_states N/A N/A self N/A kwargs N/A N/A Compute next states given current states and glimpses. N/A N/A abstractmethod N/A N/A add_metaclass ABCMeta AttentionRecurrent N/A AbstractAttentionRecurrent Initializable N/A N/A Combines an attention mechanism and a recurrent transition.

    This brick equips a recurrent transition with an attention mechanism.
    In order to do this two more contexts are added: one to be attended and
    a mask for it. It is also possible to use the contexts of the given
    recurrent transition for these purposes and not add any new ones,
    see `add_context` parameter.

    At the beginning of each step attention mechanism produces glimpses;
    these glimpses together with the current states are used to compute the
    next state and finish the transition. In some cases glimpses from the
    previous steps are also necessary for the attention mechanism, e.g.
    in order to focus on an area close to the one from the previous step.
    This is also supported: such glimpses become states of the new
    transition.

    To let the user control the way glimpses are used, this brick also
    takes a "distribute" brick as parameter that distributes the
    information from glimpses across the sequential inputs of the wrapped
    recurrent transition.

    Parameters
    ----------
    transition : :class:`.BaseRecurrent`
        The recurrent transition.
    attention : :class:`.Brick`
        The attention mechanism.
    distribute : :class:`.Brick`, optional
        Distributes the information from glimpses across the input
        sequences of the transition. By default a :class:`.Distribute` is
        used, and those inputs containing the "mask" substring in their
        name are not affected.
    add_contexts : bool, optional
        If ``True``, new contexts for the attended and the attended mask
        are added to this transition, otherwise existing contexts of the
        wrapped transition are used. ``True`` by default.
    attended_name : str
        The name of the attended context. If ``None``, "attended"
        or the first context of the recurrent transition is used
        depending on the value of `add_contents` flag.
    attended_mask_name : str
        The name of the mask for the attended context. If ``None``,
        "attended_mask" or the second context of the recurrent transition
        is used depending on the value of `add_contents` flag.

    Notes
    -----
    See :class:`.Initializable` for initialization parameters.

    Wrapping your recurrent brick with this class makes all the
    states mandatory. If you feel this is a limitation for you, try
    to make it better! This restriction does not apply to sequences
    and contexts: those keep being as optional as they were for
    your brick.

    Those coming to Blocks from Groundhog might recognize that this is
    a `RecurrentLayerWithSearch`, but on steroids :)

     __init__ N/A N/A self transition attention distribute add_contexts attended_name attended_mask_name N/A None True None None kwargs N/A N/A N/A self _sequence_names N/A list N/A N/A transition apply sequences N/A N/A self _state_names N/A list N/A N/A transition apply states N/A N/A self _context_names N/A list N/A N/A transition apply contexts N/A add_contexts N/A N/A N/A attended_name N/A N/A attended_name attended N/A N/A attended_mask_name N/A N/A attended_mask_name attended_mask N/A N/A self _context_names N/A attended_name attended_mask_name N/A N/A attended_name N/A N/A self _context_names N/A 0 N/A attended_mask_name N/A N/A self _context_names N/A 1 N/A N/A distribute N/A N/A normal_inputs N/A name N/A name N/A self _sequence_names N/A mask name N/A distribute N/A Distribute normal_inputs N/A N/A N/A attention take_glimpses outputs N/A 0 N/A N/A self transition transition N/A N/A self attention attention N/A N/A self distribute distribute N/A N/A self add_contexts add_contexts N/A N/A self attended_name attended_name N/A N/A self attended_mask_name attended_mask_name N/A N/A self preprocessed_attended_name N/A preprocessed_ N/A self attended_name N/A N/A self _glimpse_names N/A N/A N/A self attention take_glimpses outputs N/A N/A self previous_glimpses_needed N/A name N/A name N/A self _glimpse_names N/A name N/A N/A N/A self attention take_glimpses inputs N/A children N/A N/A self transition N/A self attention N/A self distribute N/A children N/A N/A kwargs get children N/A N/A N/A N/A N/A super AttentionRecurrent self __init__ children children kwargs N/A _push_allocation_config N/A N/A self N/A N/A N/A N/A N/A self attention state_dims N/A N/A N/A self transition get_dims N/A N/A self attention state_names N/A N/A N/A self attention attended_dim N/A N/A self get_dim N/A self attended_name N/A N/A N/A self distribute source_dim N/A N/A N/A self attention get_dim N/A N/A self distribute source_name N/A N/A N/A self distribute target_dims N/A N/A N/A self transition get_dims N/A N/A self distribute target_names N/A take_glimpses N/A N/A self N/A kwargs N/A N/A Compute glimpses with the attention mechanism.

        A thin wrapper over `self.attention.take_glimpses`: takes care
        of choosing and renaming the necessary arguments.

        Parameters
        ----------
        \*\*kwargs
            Must contain the attended, previous step states and glimpses.
            Can optionaly contain the attended mask and the preprocessed
            attended.

        Returns
        -------
        glimpses : list of :class:`~tensor.TensorVariable`
            Current step glimpses.

         N/A states N/A dict_subset kwargs N/A self _state_names pop True N/A glimpses N/A dict_subset kwargs N/A self _glimpse_names pop True N/A glimpses_needed N/A dict_subset glimpses N/A self previous_glimpses_needed N/A result N/A N/A N/A self attention take_glimpses N/A N/A kwargs pop N/A self attended_name N/A N/A kwargs pop N/A self preprocessed_attended_name None N/A N/A kwargs pop N/A self attended_mask_name None N/A dict_union states glimpses_needed N/A result N/A application take_glimpses_outputs N/A N/A self N/A N/A N/A N/A self _glimpse_names N/A N/A N/A take_glimpses property outputs compute_states N/A N/A self N/A kwargs N/A N/A Compute current states when glimpses have already been computed.

        Combines an application of the `distribute` that alter the
        sequential inputs of the wrapped transition and an application of
        the wrapped transition. All unknown keyword arguments go to
        the wrapped transition.

        Parameters
        ----------
        \*\*kwargs
            Should contain everything what `self.transition` needs
            and in addition the current glimpses.

        Returns
        -------
        current_states : list of :class:`~tensor.TensorVariable`
            Current states computed by `self.transition`.

         N/A normal_inputs N/A name N/A name N/A self _sequence_names N/A mask name N/A sequences N/A dict_subset kwargs normal_inputs pop True N/A glimpses N/A dict_subset kwargs N/A self _glimpse_names pop True N/A N/A self add_contexts N/A N/A N/A N/A kwargs pop N/A self attended_name N/A N/A N/A kwargs pop N/A self attended_mask_name None N/A N/A N/A sequences update N/A N/A N/A self distribute apply as_dict True N/A dict_subset N/A dict_union sequences glimpses N/A N/A N/A self distribute apply inputs N/A current_states N/A N/A N/A self transition apply iterate False as_list True N/A dict_union sequences kwargs N/A current_states N/A application compute_states_outputs N/A N/A self N/A N/A N/A N/A self _state_names N/A N/A N/A compute_states property outputs do_apply N/A N/A self N/A kwargs N/A N/A Process a sequence attending the attended context every step.

        In addition to the original sequence this method also requires
        its preprocessed version, the one computed by the `preprocess`
        method of the attention mechanism. Unknown keyword arguments
        are passed to the wrapped transition.

        Parameters
        ----------
        \*\*kwargs
            Should contain current inputs, previous step states, contexts,
            the preprocessed attended context, previous step glimpses.

        Returns
        -------
        outputs : list of :class:`~tensor.TensorVariable`
            The current step states and glimpses.

         N/A attended N/A kwargs N/A N/A self attended_name N/A preprocessed_attended N/A N/A kwargs pop N/A self preprocessed_attended_name N/A attended_mask N/A N/A kwargs get N/A self attended_mask_name N/A sequences N/A dict_subset kwargs N/A self _sequence_names pop True must_have False N/A states N/A dict_subset kwargs N/A self _state_names pop True N/A glimpses N/A dict_subset kwargs N/A self _glimpse_names pop True N/A current_glimpses N/A N/A self take_glimpses as_dict True N/A dict_union states glimpses N/A N/A self attended_name N/A self attended_mask_name N/A self preprocessed_attended_name attended attended_mask preprocessed_attended N/A current_states N/A N/A self compute_states as_list True N/A dict_union sequences states current_glimpses kwargs N/A N/A current_states N/A list N/A N/A current_glimpses values N/A recurrent do_apply_sequences N/A N/A self N/A N/A N/A N/A self _sequence_names N/A N/A N/A do_apply property sequences do_apply_contexts N/A N/A self N/A N/A N/A N/A N/A self _context_names N/A N/A self preprocessed_attended_name N/A N/A N/A do_apply property contexts do_apply_states N/A N/A self N/A N/A N/A N/A N/A self _state_names N/A self _glimpse_names N/A N/A N/A do_apply property states do_apply_outputs N/A N/A self N/A N/A N/A N/A N/A self _state_names N/A self _glimpse_names N/A N/A N/A do_apply property outputs apply N/A N/A self N/A kwargs N/A N/A Preprocess a sequence attending the attended context at every step.

        Preprocesses the attended context and runs :meth:`do_apply`. See
        :meth:`do_apply` documentation for further information.

         N/A preprocessed_attended N/A N/A N/A self attention preprocess N/A kwargs N/A N/A self attended_name N/A N/A N/A self do_apply N/A dict_union kwargs N/A N/A self preprocessed_attended_name preprocessed_attended N/A application apply_delegate N/A N/A self N/A N/A N/A N/A N/A N/A self do_apply __get__ self None N/A N/A apply delegate apply_contexts N/A N/A self N/A N/A N/A N/A self _context_names N/A N/A N/A apply property contexts initial_states N/A N/A self batch_size N/A kwargs N/A N/A N/A N/A pack N/A N/A N/A self transition initial_states batch_size kwargs N/A pack N/A N/A N/A self attention initial_glimpses batch_size N/A kwargs N/A N/A self attended_name N/A application initial_states_outputs N/A N/A self N/A N/A N/A N/A N/A self do_apply states N/A N/A N/A initial_states property outputs get_dim N/A N/A self name N/A N/A N/A N/A name N/A self _glimpse_names N/A N/A N/A N/A N/A self attention get_dim name N/A N/A name N/A self preprocessed_attended_name N/A N/A N/A original_name N/A N/A N/A self attention preprocess outputs N/A N/A N/A N/A self attention get_dim original_name N/A N/A self add_contexts N/A N/A N/A name N/A self attended_name N/A N/A N/A N/A N/A self attention get_dim N/A N/A N/A N/A self attention take_glimpses inputs N/A 0 N/A N/A name N/A self attended_mask_name N/A N/A 0 N/A N/A N/A N/A self transition get_dim name N/A N/A