Module Expr Str ImportFrom alias ImportFrom alias TryExcept body ImportFrom alias handlers ExceptHandler type NameLoad body ImportFrom alias ImportFrom alias ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults Num body Expr Str Assign AttributeStore NameLoad attr Call NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign AttributeStore NameLoad attr Call NameLoad keyword BinOpMult AttributeLoad NameLoad attr Num For NameStore Call NameLoad AttributeLoad NameLoad attr body Assign NameStore Call NameLoad keyword AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str While NameLoad body TryFinally body Assign TupleStore NameStore NameStore NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call NameLoad NameLoad NameLoad finalbody Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str If CompareEq AttributeLoad NameLoad attr Num body Return Call NameLoad NameLoad NameLoad If CompareIs AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr TupleLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str If CompareIs AttributeLoad NameLoad attr NameLoad body Return Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam defaults body For NameStore NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults vararg kwarg body Expr Str Assign AttributeStore AttributeLoad AttributeLoad NameLoad attr attr attr Call NameLoad AttributeLoad NameLoad attr Str AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore AttributeLoad AttributeLoad NameLoad attr attr attr BoolOpOr AttributeLoad NameLoad attr Call NameLoad TryFinally body TryExcept body Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str keyword Dict Str Str NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad For NameStore AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad finalbody Assign AttributeStore AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad Assign AttributeStore AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad decorator_list FunctionDef arguments args NameParam defaults vararg kwarg body Expr Str Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Return NameLoad decorator_list FunctionDef arguments args NameParam defaults vararg kwarg body Expr Str If CompareIsNot AttributeLoad NameLoad attr NameLoad body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad orelse Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Return Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad NameLoad attr NameLoad decorator_list decorator_list
N/A N/A 
This module holds a simple system for the multi-threaded execution of
scraper code. This can be used, for example, to split a scraper into
several stages and to have multiple elements processed at the same
time.

The goal of this module is to handle simple multi-threaded scrapers,
while making it easy to upgrade to a queue-based setup using celery
later.
 uuid uuid4 time sleep N/A N/A queue Queue N/A N/A N/A ImportError N/A Queue Queue threading Thread TaskManager N/A object N/A N/A  The `TaskManager` is a singleton that manages the threads
    used to parallelize processing and the queue that manages the
    current set of prepared tasks.  __init__ N/A N/A self threads N/A 10 N/A N/A 
        :param threads: The number of threads to be spawned. Values
            ranging from 5 to 40 have shown useful, based on the amount
            of I/O involved in each task.
        :param daemon: Mark the worker threads as daemons in the
            operating system, so that they will not be included in the
            number of application threads for this script.
         N/A N/A self num_threads N/A int threads N/A N/A self queue None N/A _spawn N/A N/A self N/A N/A N/A  Initialize the queue and the threads.  N/A N/A self queue N/A Queue maxsize N/A N/A self num_threads 10 N/A i N/A range N/A self num_threads N/A N/A t N/A Thread target N/A self _consume N/A N/A t daemon True N/A N/A N/A t start N/A _consume N/A N/A self N/A N/A N/A  Main loop for each thread, handles picking a task off the
        queue, processing it and notifying the queue that it is done.
         N/A True N/A N/A N/A N/A N/A task args kwargs N/A N/A N/A self queue get True N/A N/A task args kwargs N/A N/A N/A N/A N/A self queue task_done N/A put N/A N/A self task args kwargs N/A N/A N/A  Add a new item to the queue. An item is a task and the
        arguments needed to call it.

        Do not call this directly, use Task.queue/Task.run instead.
         N/A N/A N/A self num_threads 0 N/A N/A N/A task args kwargs N/A N/A N/A self queue None N/A N/A N/A N/A self _spawn N/A N/A N/A N/A self queue put N/A task args kwargs N/A wait N/A N/A self N/A N/A N/A  Wait for each item in the queue to be processed. If this
        is not called, the main thread will end immediately and none
        of the tasks assigned to the threads would be executed.  N/A N/A N/A self queue None N/A N/A N/A N/A N/A N/A self queue join N/A N/A ChainListener N/A object N/A __init__ N/A N/A self task N/A N/A N/A N/A self task task N/A notify N/A N/A self value N/A N/A N/A N/A N/A N/A self task queue value N/A N/A PipeListener N/A ChainListener N/A notify N/A N/A self value N/A N/A N/A value_item value N/A N/A N/A N/A N/A self task queue value_item N/A N/A Task N/A object N/A N/A  A task is a decorator on a function which helps managing the
    execution of that function in a multi-threaded, queued context.

    After a task has been applied to a function, it can either be used
    in the normal way (by calling it directly), through a simple queue
    (using the `queue` method), or in pipeline mode (using `chain`,
    `pipe` and `run`).
     __init__ N/A N/A self scraper fn task_id N/A None N/A N/A N/A self scraper scraper N/A N/A self fn fn N/A N/A self task_id task_id N/A N/A self _listeners N/A N/A N/A self _source None N/A __call__ N/A N/A self N/A args kwargs N/A N/A  Execute the wrapped function. This will either call it in
        normal mode (returning the return value), or notify any
        pipeline listeners that have been associated with this task.
         N/A N/A N/A N/A self scraper task_ctx name N/A getattr N/A self fn func_name N/A N/A self fn __name__ N/A N/A N/A N/A self scraper task_ctx id N/A N/A self task_id N/A uuid4 N/A N/A N/A N/A N/A N/A N/A N/A N/A self scraper log debug Begin task extra N/A taskArgs taskKwargs args kwargs N/A value N/A N/A self fn args kwargs N/A listener N/A self _listeners N/A N/A N/A N/A listener notify value N/A value N/A N/A N/A Exception N/A e N/A N/A N/A N/A N/A N/A self scraper log exception e N/A N/A N/A N/A N/A self scraper task_ctx name None N/A N/A N/A N/A self scraper task_ctx id None N/A queue N/A N/A self N/A args kwargs N/A N/A  Schedule a task for execution. The task call (and its
        arguments) will be placed on the queue and processed
        asynchronously.  N/A N/A N/A N/A N/A self scraper task_manager put self args kwargs N/A self N/A wait N/A N/A self N/A N/A N/A  Wait for task execution in the current queue to be
        complete (ie. the queue to be empty). If only `queue` is called
        without `wait`, no processing will occur.  N/A N/A N/A N/A N/A self scraper task_manager wait N/A self N/A run N/A N/A self N/A args kwargs N/A N/A  Queue a first item to execute, then wait for the queue to
        be empty before returning. This should be the default way of
        starting any scraper.
         N/A N/A N/A self _source None N/A N/A N/A N/A N/A self _source run args kwargs N/A N/A N/A N/A self queue args kwargs N/A N/A N/A self wait N/A chain N/A N/A self other_task N/A N/A N/A  Add a chain listener to the execution of this task. Whenever
        an item has been processed by the task, the registered listener
        task will be queued to be executed with the output of this task.

        Can also be written as::

            pipeline = task1 > task2
         N/A N/A other_task _source self N/A N/A N/A N/A self _listeners append N/A ChainListener other_task N/A other_task N/A pipe N/A N/A self other_task N/A N/A N/A  Add a pipe listener to the execution of this task. The
        output of this task is required to be an iterable. Each item in
        the iterable will be queued as the sole argument to an execution
        of the listener task.

        Can also be written as::

            pipeline = task1 | task2
         N/A N/A other_task _source self N/A N/A N/A N/A self _listeners append N/A PipeListener other_task N/A other_task N/A __gt__ N/A N/A self other_task N/A N/A N/A N/A N/A self chain other_task N/A __or__ N/A N/A self other_task N/A N/A N/A N/A N/A self pipe other_task N/A N/A