Module Assign NameStore Str ImportFrom alias Import alias ImportFrom alias Expr Str ClassDef bases body Expr Str Assign NameStore Num Assign NameStore Num FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad Assign AttributeStore NameLoad attr Call NameLoad Assign AttributeStore NameLoad attr Call NameLoad Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr decorator_list FunctionDef arguments args NameParam NameParam defaults body If CompareIn NameLoad AttributeLoad NameLoad attr body Raise Call NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Delete SubscriptDel AttributeLoad NameLoad attr Index NameLoad Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index NameLoad attr If CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index NameLoad attr Delete SubscriptDel AttributeLoad NameLoad attr Index NameLoad If UnaryOpNot AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body If CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Return Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr If NameLoad body If CompareIn Str AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr AttributeLoad NameLoad attr TryExcept body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad handlers ExceptHandler body Pass orelse If UnaryOpNot AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad NameLoad Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index AttributeLoad NameLoad attr attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Num NameLoad decorator_list FunctionDef arguments args NameParam defaults vararg kwarg body Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index NameLoad attr NameLoad NameLoad NameLoad decorator_list decorator_list ClassDef bases body Expr Str Assign NameStore Num Assign NameStore Num FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam defaults body If CompareIn NameLoad AttributeLoad NameLoad attr body Raise Call NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad NameLoad Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index NameLoad attr Delete SubscriptDel AttributeLoad NameLoad attr Index NameLoad decorator_list FunctionDef arguments args NameParam defaults body If CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults vararg kwarg body Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr If NameLoad body Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad NameLoad attr Num NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index NameLoad attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str decorator_list
N/A N/A __author__ wcong twisted.internet reactor N/A engine ants.crawl scheduler N/A 
crawl server and client
 CrawlServer N/A N/A N/A 
    cluster get a crawl job
    distribute it to all the node
     N/A STATUS_RUNNING 1 N/A STATUS_STOP 2 __init__ N/A N/A self cluster_manager N/A N/A N/A N/A self settings N/A cluster_manager settings N/A N/A self cluster_manager cluster_manager N/A N/A self spider_manager N/A N/A cluster_manager node_manager spider_manager N/A N/A self status N/A self STATUS_STOP N/A N/A self scheduler N/A N/A scheduler SchedulerServer N/A cluster_manager settings N/A N/A self running_spider_dict N/A dict N/A N/A self init_spider_dict N/A dict N/A N/A self stop_spider_dict N/A dict N/A N/A self distribute_index 0 N/A N/A self distribute_node_list N/A list N/A N/A N/A self cluster_manager cluster_info node_list N/A init_spider_job N/A N/A self spider_name N/A N/A N/A N/A spider_name N/A self running_spider_dict N/A N/A N/A RunningJobException N/A spider N/A N/A N/A self spider_manager create spider_name settings N/A self settings N/A crawl_engine N/A N/A engine EngineServer spider N/A self cluster_manager N/A self scheduler N/A N/A N/A self running_spider_dict N/A spider_name crawl_engine N/A run_engine N/A N/A self spider_name N/A N/A N/A N/A N/A self init_spider_dict N/A spider_name N/A N/A N/A N/A N/A self running_spider_dict N/A spider_name run N/A N/A N/A self status N/A self STATUS_STOP N/A N/A N/A self status N/A self STATUS_RUNNING N/A N/A N/A self distribute N/A init_init_job_dict N/A N/A self spider_name node_list N/A N/A N/A N/A N/A self init_spider_dict N/A spider_name N/A list node_list N/A remove_init_node N/A N/A self spider_name node_info N/A N/A N/A N/A N/A N/A N/A self init_spider_dict N/A spider_name remove node_info N/A init_stop_job_dict N/A N/A self spider_name node_list N/A N/A N/A N/A N/A self stop_spider_dict N/A spider_name N/A list node_list N/A remove_stop_job_dict N/A N/A self spider_name node_info N/A N/A N/A N/A N/A N/A N/A self stop_spider_dict N/A spider_name remove node_info N/A stop_engine N/A N/A self spider_name N/A N/A N/A N/A N/A N/A N/A self running_spider_dict N/A spider_name stop N/A N/A N/A self running_spider_dict N/A spider_name N/A N/A N/A self running_spider_dict N/A N/A N/A self status N/A self STATUS_STOP N/A distribute N/A N/A self N/A N/A N/A N/A N/A self status N/A self STATUS_STOP N/A N/A N/A request N/A N/A N/A self scheduler next_request N/A request N/A N/A N/A follow_cookiejar N/A request meta N/A N/A node_info N/A N/A N/A N/A self cluster_manager cluster_info get_node_by_name N/A request node_name N/A N/A N/A N/A N/A N/A self distribute_node_list remove node_info N/A N/A N/A N/A N/A N/A N/A N/A self distribute_node_list N/A N/A N/A self distribute_node_list N/A list N/A N/A N/A self cluster_manager cluster_info node_list N/A node_info N/A N/A N/A self distribute_node_list pop N/A N/A N/A N/A N/A self cluster_manager node_manager send_request_to_client request node_info N/A N/A N/A N/A N/A self running_spider_dict N/A N/A request spider_name log_distributed_request request node_info N/A N/A N/A reactor callLater 0 self N/A __call__ N/A N/A self N/A args kwargs N/A N/A N/A N/A self distribute N/A accept_request N/A N/A self spider_name request N/A N/A N/A N/A N/A N/A N/A self running_spider_dict N/A spider_name add_request request N/A accept_crawl_result N/A N/A self spider_name node_name request_hash_code msg N/A N/A N/A N/A N/A N/A N/A self running_spider_dict N/A spider_name accept_request_result node_name request_hash_code msg N/A N/A CrawlClient N/A N/A N/A 
    nodes init a spider,and listen the request job
    get a job execute it and return result
     N/A STATUS_RUNNING 1 N/A STATUS_STOP 2 __init__ N/A N/A self node_manager N/A N/A N/A N/A self settings N/A node_manager settings N/A N/A self status N/A self STATUS_STOP N/A N/A self node_manager node_manager N/A N/A self spider_manager N/A N/A self node_manager spider_manager N/A N/A self scheduler N/A N/A scheduler SchedulerClient N/A node_manager settings N/A N/A self running_spider_dict N/A dict N/A spider_list N/A N/A self N/A N/A N/A N/A N/A N/A self spider_manager list N/A init_engine N/A N/A self spider_name N/A N/A N/A N/A spider_name N/A self running_spider_dict N/A N/A N/A RunningJobException N/A spider N/A N/A N/A self spider_manager create spider_name settings N/A self settings N/A crawl_engine N/A N/A engine EngineClient spider N/A self node_manager N/A self scheduler N/A N/A N/A self running_spider_dict N/A spider_name crawl_engine N/A N/A N/A self run N/A stop_engine N/A N/A self spider_name N/A N/A N/A N/A N/A N/A N/A self running_spider_dict N/A spider_name stop N/A N/A N/A self running_spider_dict N/A spider_name N/A run N/A N/A self N/A N/A N/A N/A N/A self status N/A self STATUS_STOP N/A N/A N/A self status N/A self STATUS_RUNNING N/A N/A N/A self distribute N/A __call__ N/A N/A self N/A args kwargs N/A N/A N/A N/A self distribute N/A distribute N/A N/A self N/A N/A N/A request N/A N/A N/A self scheduler next_request N/A request N/A N/A N/A N/A N/A N/A self running_spider_dict N/A N/A request spider_name crawl_request request N/A N/A N/A reactor callLater 0 self N/A accept_request N/A N/A self spider_name request N/A N/A N/A N/A N/A N/A N/A self running_spider_dict N/A spider_name accept_request request N/A N/A RunningJobException N/A Exception N/A N/A 
    job is running
     N/A