Module ImportFrom alias ImportFrom alias alias ImportFrom alias ImportFrom alias ClassDef bases NameLoad body Pass decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Num Num NameLoad NameLoad ListLoad NameLoad NameLoad Num NameLoad NameLoad NameLoad kwarg body Expr Str If CompareEq NameLoad Str body Raise Call NameLoad Str Assign AttributeStore NameLoad attr NameLoad If BoolOpAnd CompareIs NameLoad NameLoad CompareIs NameLoad NameLoad CompareIs NameLoad NameLoad body Raise Call NameLoad Str If BoolOpAnd CompareIsNot NameLoad NameLoad CompareIsNot NameLoad NameLoad body Raise Call NameLoad Str If BoolOpAnd CompareIsNot NameLoad NameLoad CompareIsNot NameLoad NameLoad body Raise Call NameLoad Str If BoolOpAnd CompareIsNot NameLoad NameLoad CompareIsNot NameLoad NameLoad body Raise Call NameLoad Str If NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr orelse If NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad orelse Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr ListLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Expr Call NameLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Expr Str Return Call NameLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body If NameLoad body Assign AttributeStore NameLoad attr BinOpAdd Call AttributeLoad NameLoad attr Call NameLoad keyword NameLoad Assign AttributeStore NameLoad attr NameLoad orelse Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr Call NameLoad NameLoad Assign AttributeStore NameLoad attr SubscriptLoad NameLoad Index Str decorator_list FunctionDef arguments args NameParam defaults kwarg body Expr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr If CompareIsNot Call AttributeLoad NameLoad attr Str NameLoad body Delete SubscriptDel NameLoad Index Str Assign NameStore Call NameLoad NameLoad If CompareNotIn Str NameLoad body Assign AttributeStore NameLoad attr Num If CompareNotIn Str NameLoad body Assign AttributeStore NameLoad attr BinOpAdd AttributeLoad NameLoad attr Num If CompareNotIn Str NameLoad body Assign AttributeStore NameLoad attr NameLoad If CompareNotIn Str NameLoad body Assign AttributeStore NameLoad attr NameLoad If BoolOpAnd CompareIsNot Call AttributeLoad NameLoad attr Str NameLoad CompareIsNot Call AttributeLoad NameLoad attr Str NameLoad body Raise Call NameLoad Str If BoolOpAnd CompareIsNot Call AttributeLoad NameLoad attr Str NameLoad CompareIsNot Call AttributeLoad NameLoad attr Str NameLoad body Raise Call NameLoad Str If BoolOpAnd CompareIsNot Call AttributeLoad NameLoad attr Str NameLoad CompareIsNot Call AttributeLoad NameLoad attr Str NameLoad body Raise Call NameLoad Str If Call AttributeLoad NameLoad attr Str body Expr Call AttributeLoad NameLoad attr Call AttributeLoad SubscriptLoad NameLoad Index Str attr Delete SubscriptDel NameLoad Index Str orelse If Call AttributeLoad NameLoad attr Str body Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Delete SubscriptDel NameLoad Index Str orelse If Call AttributeLoad NameLoad attr Str body Assign AttributeStore NameLoad attr SubscriptLoad NameLoad Index Str If AttributeLoad NameLoad attr body Assign SubscriptStore AttributeLoad NameLoad attr Index Str SubscriptLoad NameLoad Index Str Delete SubscriptDel NameLoad Index Str For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Expr Call NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Str NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return BinOpMod Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Return CompareLt AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body If BoolOpOr UnaryOpNot AttributeLoad NameLoad attr UnaryOpNot AttributeLoad NameLoad attr body Return NameLoad orelse Return CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body If AttributeLoad NameLoad attr body Return Call NameLoad NameLoad AttributeLoad NameLoad attr orelse If AttributeLoad NameLoad attr body Assign NameStore BinOpMod Str AttributeLoad NameLoad attr If Call NameLoad NameLoad NameLoad body Return Call NameLoad NameLoad NameLoad orelse Return NameLoad decorator_list decorator_list FunctionDef arguments args NameParam defaults body FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list Return NameLoad decorator_list
N/A __future__ absolute_import datetime datetime timedelta grab.spider.error SpiderMisuseError grab.base copy_config BaseTask N/A object N/A N/A N/A Task N/A BaseTask N/A N/A 
    Task for spider.
     __init__ N/A N/A self name url grab grab_config priority priority_is_custom network_try_count task_try_count disable_cache refresh_cache valid_status use_proxylist cache_timeout delay raw callback fallback_name N/A None None None None None True 0 1 False False N/A True None 0 False None None kwargs N/A N/A 
        Create `Task` object.

        If more than one of url, grab and grab_config options are non-empty
        then they processed in following order:
        * grab overwrite grab_config
        * grab_config overwrite url

        Args:
            :param name: name of the task. After successful network operation
                task's result will be passed to `task_<name>` method.
            :param url: URL of network document. Any task requires `url` or
                `grab` option to be specified.
            :param grab: configured `Grab` instance. You can use that option in
                case when `url` option is not enough. Do not forget to
                configure `url` option of `Grab` instance because in this case
                the `url` option of `Task` constructor will be overwritten
                with `grab.config['url']`.
            :param priority: - priority of the Task. Tasks with lower priority
                will be processed earlier. By default each new task is assigned
                with random priority from (80, 100) range.
            :param priority_is_custom: - internal flag which tells if that task
                priority was assigned manually or generated by spider according
                to priority generation rules.
            :param network_try_count: you'll probably will not need to use it.
                It is used internally to control how many times this task was
                restarted due to network errors. The `Spider` instance has
                `network_try_limit` option. When `network_try_count` attribute
                of the task exceeds the `network_try_limit` attribute then
                processing of the task is abandoned.
            :param task_try_count: the as `network_try_count` but it increased
                only then you use `clone` method. Also you can set it manually.
                It is useful if you want to restart the task after it was
                cancelled due to multiple network errors. As you might guessed
                there is `task_try_limit` option in `Spider` instance. Both
                options `network_try_count` and `network_try_limit` guarantee
                you that you'll not get infinite loop of restarting some task.
            :param disable_cache: if `True` disable cache subsystem.
                The document will be fetched from the Network and it will not
                be saved to cache.
            :param refresh_cache: if `True` the document will be fetched from
                the Network and saved to cache.
            :param valid_status: extra status codes which counts as valid
            :param use_proxylist: it means to use proxylist which was
                configured via `setup_proxylist` method of spider
            :param cache_timeout: maximum age (in seconds) of cache record to
                be valid
            :param delay: if specified tells the spider to schedule the task
                and execute    it after `delay` seconds
            :param raw: if `raw` is True then the network response is
                forwarding to the corresponding handler without any check of
                HTTP status code of network error, if `raw` is False (by
                default) then failed response is putting back to task queue or
                if tries limit is reached then the processing of this  request
                is finished.
            :param callback: if you pass some function in `callback` option
                then the network response will be passed to this callback and
                the usual 'task_*' handler will be ignored and no error will be
                raised if such 'task_*' handler does not exist.
            :param fallback_name: the name of method that is called when spider
                gives up to do the task (due to multiple network errors)

            Any non-standard named arguments passed to `Task` constructor will
            be saved as attributes of the object. You can get their values
            later as attributes or with `get` method which allows to use
            default value if attribute does not exist.
         N/A N/A name generator N/A N/A N/A SpiderMisuseError Task name could not be "generator" N/A N/A self name name N/A N/A N/A url None N/A grab None N/A grab_config None N/A N/A N/A SpiderMisuseError Either url, grab or grab_config argument of Task constructor should not be None N/A N/A N/A url None N/A grab None N/A N/A N/A SpiderMisuseError Options url and grab could not be used together N/A N/A N/A url None N/A grab_config None N/A N/A N/A SpiderMisuseError Options url and grab_config could not be used together N/A N/A N/A grab None N/A grab_config None N/A N/A N/A SpiderMisuseError Options grab and grab_config could not be used together N/A grab N/A N/A N/A N/A self setup_grab_config N/A N/A grab dump_config N/A N/A grab_config N/A N/A N/A N/A self setup_grab_config grab_config N/A N/A N/A self grab_config None N/A N/A self url url N/A N/A N/A self process_delay_option delay N/A N/A self fallback_name fallback_name N/A N/A self priority_is_custom priority_is_custom N/A N/A self priority priority N/A N/A self network_try_count network_try_count N/A N/A self task_try_count task_try_count N/A N/A self disable_cache disable_cache N/A N/A self refresh_cache refresh_cache N/A N/A self valid_status valid_status N/A N/A self use_proxylist use_proxylist N/A N/A self cache_timeout cache_timeout N/A N/A self raw raw N/A N/A self origin_task_generator None N/A N/A self callback callback N/A N/A self coroutines_stack N/A N/A N/A key value N/A N/A kwargs items N/A N/A N/A setattr self key value N/A get N/A N/A self key default N/A None N/A N/A 
        Return value of attribute or None if such attribute
        does not exist.
         N/A N/A getattr self key default N/A process_delay_option N/A N/A self delay N/A N/A N/A delay N/A N/A N/A self schedule_time N/A N/A N/A datetime utcnow N/A timedelta seconds delay N/A N/A self original_delay delay N/A N/A N/A self schedule_time None N/A N/A self original_delay None N/A setup_grab_config N/A N/A self grab_config N/A N/A N/A N/A self grab_config N/A copy_config grab_config N/A N/A self url N/A grab_config N/A url N/A clone N/A N/A self N/A kwargs N/A N/A 
        Clone Task instance.

        Reset network_try_count, increase task_try_count.
         N/A attr_copy N/A N/A N/A self __dict__ copy N/A N/A N/A N/A attr_copy get grab_config None N/A N/A N/A attr_copy N/A url N/A task N/A Task attr_copy N/A N/A network_try_count kwargs N/A N/A N/A task network_try_count 0 N/A N/A task_try_count kwargs N/A N/A N/A task task_try_count N/A N/A self task_try_count 1 N/A N/A refresh_cache kwargs N/A N/A N/A task refresh_cache False N/A N/A disable_cache kwargs N/A N/A N/A task disable_cache False N/A N/A N/A N/A N/A kwargs get url None N/A N/A N/A kwargs get grab None N/A N/A N/A SpiderMisuseError Options url and grab could not be used together N/A N/A N/A N/A N/A kwargs get url None N/A N/A N/A kwargs get grab_config None N/A N/A N/A SpiderMisuseError Options url and grab_config could not be used together N/A N/A N/A N/A N/A kwargs get grab None N/A N/A N/A kwargs get grab_config None N/A N/A N/A SpiderMisuseError Options grab and grab_config could not be used together N/A N/A N/A kwargs get grab N/A N/A N/A N/A task setup_grab_config N/A N/A N/A kwargs N/A grab dump_config N/A N/A kwargs N/A grab N/A N/A N/A N/A kwargs get grab_config N/A N/A N/A N/A task setup_grab_config N/A kwargs N/A grab_config N/A N/A kwargs N/A grab_config N/A N/A N/A N/A kwargs get url N/A N/A N/A task url N/A kwargs N/A url N/A N/A task grab_config N/A N/A N/A N/A task grab_config N/A url N/A kwargs N/A url N/A N/A kwargs N/A url N/A N/A key value N/A N/A kwargs items N/A N/A N/A setattr task key value N/A N/A N/A task process_delay_option N/A N/A task get delay None N/A task N/A __repr__ N/A N/A self N/A N/A N/A N/A <Task: %s> N/A self url N/A __lt__ N/A N/A self other N/A N/A N/A N/A N/A self priority N/A other priority N/A __eq__ N/A N/A self other N/A N/A N/A N/A N/A N/A self priority N/A N/A other priority N/A N/A True N/A N/A N/A N/A self priority N/A other priority N/A get_fallback_handler N/A N/A self spider N/A N/A N/A N/A self fallback_name N/A N/A N/A getattr spider N/A self fallback_name N/A N/A N/A self name N/A N/A fb_name N/A task_%s_fallback N/A self name N/A N/A hasattr spider fb_name N/A N/A N/A getattr spider fb_name N/A N/A None N/A N/A inline_task N/A N/A f N/A N/A wrap N/A N/A self grab task N/A N/A N/A origin_task_generator N/A f self grab task N/A N/A N/A self handler_for_inline_task None origin_task_generator N/A N/A wrap N/A