Module Expr Str Import alias Import alias Assign NameStore AttributeLoad NameLoad attr ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str NameLoad Str Expr Call AttributeLoad NameLoad attr Str Num Str Expr Call AttributeLoad NameLoad attr Str Num Str ClassDef bases NameLoad body Expr Str decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults vararg kwarg body Expr Str Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Assign NameStore Yield Call AttributeLoad NameLoad attr BinOpAdd NameLoad Str keyword Dict Str Str Str NameLoad NameLoad NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr If BoolOpAnd AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Str body Raise Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str If BoolOpOr UnaryOpNot AttributeLoad NameLoad attr UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr Str body Raise Call NameLoad BinOpMod Str NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults Num body Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad NameLoad NameLoad If CompareGt NameLoad Num body Expr Yield Call AttributeLoad NameLoad attr NameLoad Assign NameStore ListLoad Num Assign NameStore SubscriptLoad NameLoad Index Str FunctionDef arguments args NameParam defaults body Assign NameStore SubscriptLoad NameLoad Index Num Assign SubscriptStore NameLoad Index Num BinOpAdd NameLoad Num Return Call NameLoad NameLoad NameLoad NameLoad NameLoad decorator_list Assign NameStore SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr keyword NameLoad Assign NameStore NameLoad TryExcept body Expr Yield Call NameLoad NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad Expr Yield Call NameLoad BinOpMod Str TupleLoad AttributeLoad AttributeLoad NameLoad attr attr Call NameLoad NameLoad If BoolOpAnd Call NameLoad NameLoad NameLoad CompareGtE SubscriptLoad NameLoad Index Str AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore NameLoad orelse Return Assign NameStore Dict Str NameLoad If NameLoad body Assign SubscriptStore NameLoad Index Str Str TryExcept body Assign NameStore Yield Call AttributeLoad NameLoad attr BinOpAdd NameLoad Str keyword NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad AttributeLoad AttributeLoad NameLoad attr attr NameLoad orelse If BoolOpAnd AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Str body Expr Call AttributeLoad NameLoad attr Str NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Str NameLoad orelse Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults Num Num body Assign NameStore BinOpMod Str TupleLoad AttributeLoad NameLoad attr NameLoad Assign NameStore ListLoad While UnaryOpNot AttributeLoad NameLoad attr body Assign NameStore BinOpSub NameLoad Call NameLoad NameLoad Assign NameStore ListLoad If CompareGt NameLoad Num body Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad NameLoad TryExcept body Assign NameStore Yield Call AttributeLoad NameLoad attr BinOpAdd NameLoad Str keyword Dict Str NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr Str NameLoad AttributeLoad AttributeLoad NameLoad attr attr NameLoad orelse If AttributeLoad NameLoad attr body If Call AttributeLoad AttributeLoad NameLoad attr attr Str body Expr Call AttributeLoad NameLoad attr Str NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Str orelse If SubscriptLoad AttributeLoad NameLoad attr Index Str body Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index Str For TupleStore NameStore NameStore Call NameLoad NameLoad body Assign NameStore Yield Call NameLoad NameLoad NameLoad NameLoad keyword BinOpMult NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr If NameLoad body Assign NameStore AttributeLoad NameLoad attr Expr Yield Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore NameLoad Slice ListComp NameLoad comprehension NameStore NameLoad UnaryOpNot AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Call NameLoad NameLoad NameLoad NameLoad decorator_list decorator_list
N/A N/A Workers that consumer a release server's work queue. N/A logging N/A gflags N/A FLAGS N/A gflags FLAGS dpxdt.client fetch_worker dpxdt.client timer_worker dpxdt.client workers N/A LOGGER N/A workers LOGGER N/A N/A N/A gflags DEFINE_string queue_server_prefix None URL prefix of where the work queue server is located, such as "https://www.example.com/api/work_queue". This should use HTTPS if possible, since API requests send credentials using HTTP basic auth. N/A N/A N/A gflags DEFINE_integer queue_idle_poll_seconds 60 How often to poll the work queue for new tasks when the worker is currently not processing any tasks. N/A N/A N/A gflags DEFINE_integer queue_busy_poll_seconds 1 How often to poll tasks running locally to see if they have completed and then go back to the server to look for more work. Error N/A Exception N/A N/A Base-class for exceptions in this module. N/A GiveUpAfterAttemptsError N/A Error N/A N/A Exception indicates the task should give up after N attempts. __init__ N/A N/A self max_attempts N/A args kwargs N/A N/A Initializer.

        Args:
            max_attempts: Maximum number of attempts to make for this task,
                inclusive. So 2 means try two times and then retire the task.
            *args, **kwargs: Optional Exception arguments.
         N/A N/A N/A Exception __init__ self args kwargs N/A N/A self max_attempts max_attempts N/A N/A HeartbeatError N/A Error N/A N/A Reporting the status of a task in progress failed for some reason. N/A HeartbeatWorkflow N/A N/A workers WorkflowItem N/A N/A Reports the status of a RemoteQueueWorkflow to the API server.

    Args:
        queue_url: Base URL of the work queue.
        task_id: ID of the task to update the heartbeat status message for.
        message: Heartbeat status message to report.
        index: Index for the heartbeat message. Should be at least one
            higher than the last heartbeat message.
     run N/A N/A self queue_url task_id message index N/A N/A N/A call N/A N/A N/A fetch_worker FetchItem N/A queue_url /heartbeat post N/A task_id message index task_id message index username N/A FLAGS release_client_id password N/A FLAGS release_client_secret N/A N/A N/A call json N/A N/A N/A call json get error N/A N/A N/A HeartbeatError N/A N/A N/A call json get error N/A N/A N/A N/A call json N/A N/A N/A N/A call json get success N/A N/A N/A HeartbeatError N/A Bad response: %r call N/A N/A DoTaskWorkflow N/A N/A workers WorkflowItem N/A N/A Runs a local workflow for a task and marks it done in the remote queue.

    Args:
        queue_url: Base URL of the work queue.
        local_queue_workflow: WorkflowItem sub-class to create using parameters
            from the remote work payload that will execute the task.
        task: JSON payload of the task.
        wait_seconds: Wait this many seconds before starting work.
            Defaults to zero.
     N/A fire_and_forget True run N/A N/A self queue_url local_queue_workflow task wait_seconds N/A 0 N/A N/A N/A N/A LOGGER info Starting work item from queue_url=%r, task=%r, workflow=%r, wait_seconds=%r queue_url task local_queue_workflow wait_seconds N/A N/A wait_seconds 0 N/A N/A N/A N/A N/A timer_worker TimerItem wait_seconds N/A index N/A 0 N/A task_id N/A task N/A task_id heartbeat N/A N/A message N/A N/A N/A next_index N/A index N/A 0 N/A N/A index N/A 0 N/A next_index 1 N/A N/A HeartbeatWorkflow queue_url task_id message next_index N/A N/A payload N/A task N/A payload N/A N/A N/A payload update heartbeat heartbeat N/A error False N/A N/A N/A N/A N/A local_queue_workflow payload N/A N/A N/A Exception N/A e N/A N/A N/A N/A LOGGER exception Exception while processing work from queue_url=%r, task=%r queue_url task N/A N/A N/A heartbeat N/A %s: %s N/A N/A N/A e __class__ __name__ N/A str e N/A N/A N/A isinstance e GiveUpAfterAttemptsError N/A N/A task N/A lease_attempts N/A e max_attempts N/A N/A N/A N/A LOGGER warning Hit max attempts on task=%r, marking task as error task N/A error True N/A N/A N/A finish_params N/A task_id task_id N/A error N/A N/A N/A finish_params N/A error 1 N/A N/A N/A finish_item N/A N/A N/A fetch_worker FetchItem N/A queue_url /finish post finish_params username N/A FLAGS release_client_id password N/A FLAGS release_client_secret N/A N/A N/A Exception N/A e N/A N/A N/A N/A LOGGER error Could not finish work with queue_url=%r, task=%r. %s: %s queue_url task N/A N/A e __class__ __name__ e N/A N/A N/A N/A finish_item json N/A N/A N/A finish_item json get error N/A N/A N/A N/A LOGGER error Could not finish work with queue_url=%r, task=%r. %s queue_url N/A N/A finish_item json N/A error task N/A N/A N/A N/A LOGGER info Finished work item with queue_url=%r, task_id=%r queue_url task_id N/A N/A RemoteQueueWorkflow N/A N/A workers WorkflowItem N/A N/A Fetches tasks from a remote queue periodically, runs them locally.

    Args:
        queue_name: Name of the queue to fetch from.
        local_queue_workflow: WorkflowItem sub-class to create using parameters
            from the remote work payload that will execute the task.
        max_tasks: Maximum number of tasks to have in flight at any time.
            Defaults to 1.
        wait_seconds: How many seconds should be between tasks starting to
            process locally. Defaults to 0. Can be used to spread out
            the load a new set of tasks has on the server.
     run N/A N/A self queue_name local_queue_workflow max_tasks wait_seconds N/A 1 0 N/A N/A queue_url N/A %s/%s N/A N/A FLAGS queue_server_prefix queue_name N/A outstanding N/A N/A N/A N/A self interrupted N/A N/A next_count N/A max_tasks N/A len outstanding N/A next_tasks N/A N/A N/A next_count 0 N/A N/A N/A N/A LOGGER debug Fetching %d tasks from queue_url=%r for workflow=%r next_count queue_url local_queue_workflow N/A N/A N/A next_item N/A N/A N/A fetch_worker FetchItem N/A queue_url /lease post N/A count next_count username N/A FLAGS release_client_id password N/A FLAGS release_client_secret N/A N/A N/A Exception N/A e N/A N/A N/A N/A LOGGER error Could not fetch work from queue_url=%r. %s: %s queue_url N/A N/A e __class__ __name__ e N/A N/A N/A next_item json N/A N/A N/A N/A N/A next_item json get error N/A N/A N/A N/A LOGGER error Could not fetch work from queue_url=%r. %s queue_url N/A N/A next_item json N/A error N/A N/A N/A N/A next_item json N/A tasks N/A N/A next_tasks N/A N/A next_item json N/A tasks N/A N/A index task N/A enumerate next_tasks N/A N/A item N/A N/A DoTaskWorkflow queue_url local_queue_workflow task wait_seconds N/A index wait_seconds N/A N/A N/A outstanding append item N/A poll_time N/A FLAGS queue_idle_poll_seconds N/A outstanding N/A N/A poll_time N/A FLAGS queue_busy_poll_seconds N/A N/A N/A N/A timer_worker TimerItem poll_time N/A N/A outstanding N/A N/A x N/A x outstanding N/A N/A x done N/A N/A N/A LOGGER debug %d items for %r still outstanding: %r N/A len outstanding local_queue_workflow outstanding N/A N/A