Module Expr Str Import alias Import alias ImportFrom alias Import alias Assign NameStore Call AttributeLoad NameLoad attr NameLoad ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults kwarg body Expr Call AttributeLoad NameLoad attr Str NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Delete SubscriptDel NameLoad Index Str Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults vararg body If BoolOpOr CompareEq NameLoad Str CompareEq NameLoad Str body Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr orelse If CompareEq NameLoad Str body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str orelse Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list decorator_list ClassDef bases AttributeLoad AttributeLoad NameLoad attr attr body Expr Str FunctionDef arguments args NameParam NameParam defaults kwarg body Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body If UnaryOpNot Call NameLoad NameLoad Str body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Str Return AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body If UnaryOpNot Call NameLoad NameLoad Str body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Str Return AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases AttributeLoad AttributeLoad NameLoad attr attr body Expr Str FunctionDef arguments args NameParam NameParam defaults Num kwarg body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad NameLoad Assign NameStore NameLoad If CompareEq NameLoad Str body If UnaryOpNot AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr NameLoad Assign NameStore CompareEq AttributeLoad NameLoad attr NameLoad orelse If CompareEq NameLoad Str body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Num orelse If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr NameLoad orelse Raise Call NameLoad BinOpAdd Str NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Return NameLoad decorator_list decorator_list
N/A N/A Synchronization of parameters of concurrent training processes.

Use this module to run several Blocks training processes
concurrently, e.g. to perform Asynchronous SGD [DOWNPOUR_].
It is built on top of Platoon, a simple framework that makes it easy
to share parameters and perform communication among multiple training
jobs. Below is the list of the key concepts introduced in Platoon:

- the training process is called *worker*
- a separate process called *controller* is sending/receiving messages
  from/to workers and is responsible for synchronizing their activities
- each process has its *local parameters*, which are usual Theano shared
  variables
- all processes share *global* parameters, which are stored in
  shared memory (not to be confused with shared Theano variables!)

For parallel training to be efficient the workers should regularly
synchronize their local parameters with the global parameters.
Using the components from this module you should be able to quickly
add the required synchronization calls to your Blocks training script.
Please see the example [sync-example_] for more details.

.. [DOWNPOUR] J. Dean et al., *Large Scale Distributed Deep Networks*,
   NIPS 2012

.. [sync-example_]
    https://github.com/rizar/blocks-examples/blob/
    use_synchronize/mnist/__init__.py

 N/A logging N/A numpy blocks.extensions SimpleExtension N/A platoon.channel N/A logger N/A N/A logging getLogger __name__ Synchronize N/A SimpleExtension N/A N/A Synchronize the parameters shared between multiple workers.

    When called, this extensions triggers synchronization
    of global and local parameters. The special cases
    are the callbacks 'before_training', 'on_resumption' and
    'after_training'.

    On 'before_training' and 'on_resumption' it requests initialization
    of the global parameters, for more details see
    :class:`SynchronizedWorker`. On 'after_training', it notifies the
    controller that this worker is done.

    Parameters
    ----------
    worker : instance of :class:`SynchronizeWorker`
        The worker object of this progress.

    Notes
    -----
    This extensions makes it guess about the parameters by asking
    `main_loop.model`, which effectively means that all of them must be
    properly tagged (you get if for granted if you use bricks).

    It is recommended to trigger an additional
    synchronization after every time consuming operation, such as
    validation or checkpointing. Otherwise you might end up using
    very stale parameters.

     __init__ N/A N/A self worker N/A kwargs N/A N/A N/A N/A kwargs setdefault before_training True N/A N/A N/A kwargs setdefault on_resumption True N/A N/A N/A kwargs setdefault after_training True N/A N/A N/A N/A super Synchronize self __init__ kwargs N/A N/A self worker worker N/A __getstate__ N/A N/A self N/A N/A N/A state N/A N/A N/A self __dict__ copy N/A N/A state N/A worker N/A state N/A do N/A N/A self which_callback N/A args N/A N/A N/A N/A which_callback before_training N/A which_callback on_resumption N/A N/A N/A N/A N/A self worker init_shared_params N/A N/A N/A self main_loop model parameters N/A N/A N/A which_callback after_training N/A N/A N/A N/A N/A self worker send_req done N/A N/A N/A N/A N/A self worker sync_params N/A N/A SynchronizeWorker N/A N/A N/A platoon channel Worker N/A N/A Extends Platoon worker to support a simple protocol.

    The particular protocol between :class:`SynchronizeController` and
    :class:`SynchronizedWorker` provided in this module involves
    the following communication:

    - One of the workers is chosen as the main worker
      (see :meth:`is_main_worker`). You might want to have the main worker
      do episodic training-related chores, such as validation and/or
      checkpointing. Meanwhile, other workers can keep training!
    - Each worker receives a unique random seed, which is meant to
      determine the order of data traversal (see :meth:`seed`).

    Parameters
    ----------
    sync_rule : instance of :class:`~platoon.param_sync.ParamSyncRule`
        The rule for parameter synchronization.

    See Also
    --------
    :class:`~platoon.channel.Worker`
        For other parameters.

     __init__ N/A N/A self sync_rule N/A kwargs N/A N/A N/A self sync_rule sync_rule N/A N/A N/A N/A super SynchronizeWorker self __init__ kwargs N/A is_main_worker N/A N/A self N/A N/A N/A N/A N/A hasattr self _is_main_worker N/A N/A N/A self _is_main_worker N/A N/A self send_req is_main_worker? N/A N/A self _is_main_worker N/A property seed N/A N/A self N/A N/A N/A N/A N/A hasattr self _seed N/A N/A N/A self _seed N/A N/A self send_req seed N/A N/A self _seed N/A property init_shared_params N/A N/A self parameters N/A N/A N/A N/A N/A N/A super SynchronizeWorker self init_shared_params parameters N/A self sync_rule N/A N/A SynchronizeController N/A N/A N/A platoon channel Controller N/A N/A Extends Platoon controller to support a simple protocol.

    Communicates with the workers as described in
    :class:`SynchronizedWorker` documentation.

    Parameters
    ----------
    seed_for_seeds : int
        The seed to be used in the random number generator that provides
        the seeds to the workers.

    See Also
    --------
    :class:`~platoon.channel.Controller`
        For other arguments.

     __init__ N/A N/A self seed_for_seeds N/A 1 kwargs N/A N/A N/A N/A N/A super SynchronizeController self __init__ kwargs N/A N/A self main_worker None N/A N/A self seed_generator N/A N/A N/A numpy random RandomState seed_for_seeds N/A handle_control N/A N/A self req worker_id N/A N/A N/A N/A N/A logger info N/A N/A Request: {}, worker_id: {} format req worker_id N/A response None N/A N/A req is_main_worker? N/A N/A N/A N/A self main_worker N/A N/A N/A self main_worker worker_id N/A response N/A N/A self main_worker worker_id N/A N/A N/A req seed N/A N/A response N/A N/A N/A self seed_generator randint 100000 N/A N/A N/A req done N/A N/A N/A N/A self worker_is_done worker_id N/A N/A N/A ValueError N/A Unknown request  req N/A N/A N/A logger info N/A N/A Response: {} format response N/A response N/A N/A