Module Expr Str Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias alias ImportFrom alias FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return Call NameLoad NameLoad NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad NameLoad Return Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore ListLoad For NameStore NameLoad body Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore NameLoad For NameStore NameLoad body Assign NameStore Call NameLoad NameLoad Return NameLoad decorator_list Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Expr Str Assign NameStore AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad AttributeLoad NameLoad attr keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore BinOpMod Str TupleLoad Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad Assign AttributeStore NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad If CompareEq NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad Call NameLoad AttributeLoad NameLoad attr Num NameLoad attr Assign NameStore Call NameLoad keyword NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr decorator_list Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad keyword NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call NameLoad keyword NameLoad Expr Call NameLoad Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr decorator_list Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str TryExcept body Expr Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr For NameStore ListLoad NameLoad NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr decorator_list
N/A N/A 
This module contains the default architecture for worker processes. In order to
start a new worker process you should simply call this modules `main` method.

Communication between master -> worker and inside the worker is as follows:

Master              -> PUSH ->              Worker Fetcher

Worker Fetcher      -> PUSH ->              Worker Extractor

Worker Extractor    -> PUB  ->              Master

Each Worker is a ZmqWorker (or ZmqAsyncWorker). The Master pushes new CrawlUris
to the `Worker Fetcher`. This will download the content from the web and `PUSH`
the resulting `CrawlUri` to the `Worker Extractor`. At this stage several
Modules for extracting new URLs are running. The `Worker Scoper` will decide if
the newly extracted URLs are within the scope of the crawl.
 N/A logging N/A os N/A signal N/A socket N/A traceback N/A zmq zmq.core.error ZMQError zmq.eventloop.ioloop IOLoop DelayedCallback zmq.log.handlers PUBHandler spyder.import_util import_class spyder.core.constants ZMQ_SPYDER_MGMT_WORKER spyder.core.constants ZMQ_SPYDER_MGMT_WORKER_AVAIL spyder.core.constants ZMQ_SPYDER_MGMT_WORKER_QUIT spyder.core.constants ZMQ_SPYDER_MGMT_WORKER_QUIT_ACK spyder.core.messages MgmtMessage spyder.core.mgmt ZmqMgmt spyder.core.worker ZmqWorker AsyncZmqWorker spyder.processor.fetcher FetchProcessor create_worker_management N/A N/A settings zmq_context io_loop N/A N/A N/A 
    Create and return a new instance of the `ZmqMgmt`.
     N/A listening_socket N/A N/A zmq_context socket N/A zmq SUB N/A N/A N/A listening_socket setsockopt N/A zmq SUBSCRIBE N/A N/A N/A listening_socket connect N/A settings ZEROMQ_MGMT_MASTER N/A publishing_socket N/A N/A zmq_context socket N/A zmq PUB N/A N/A N/A publishing_socket connect N/A settings ZEROMQ_MGMT_WORKER N/A N/A ZmqMgmt listening_socket publishing_socket io_loop io_loop N/A create_worker_fetcher N/A N/A settings mgmt zmq_context log_handler io_loop N/A N/A N/A 
    Create and return a new `Worker Fetcher`.
     N/A pulling_socket N/A N/A zmq_context socket N/A zmq PULL N/A N/A N/A pulling_socket connect N/A settings ZEROMQ_WORKER_PROC_FETCHER_PULL N/A pushing_socket N/A N/A zmq_context socket N/A zmq PUSH N/A N/A N/A pushing_socket setsockopt N/A zmq HWM N/A settings ZEROMQ_WORKER_PROC_FETCHER_PUSH_HWM N/A N/A N/A pushing_socket bind N/A settings ZEROMQ_WORKER_PROC_FETCHER_PUSH N/A fetcher N/A FetchProcessor settings io_loop N/A N/A AsyncZmqWorker pulling_socket pushing_socket mgmt fetcher log_handler N/A settings LOG_LEVEL_WORKER io_loop N/A create_processing_function N/A N/A settings pipeline N/A N/A N/A 
    Create a processing method that iterates all processors over the incoming
    message.
     N/A processors N/A N/A processor pipeline N/A N/A processor_class N/A import_class processor N/A N/A N/A processors append N/A processor_class settings processing N/A N/A data_message N/A N/A N/A 
        The actual processing function calling each configured processor in the
        order they have been configured.
         N/A next_message data_message N/A processor processors N/A N/A next_message N/A processor next_message N/A next_message N/A N/A processing N/A create_worker_extractor N/A N/A settings mgmt zmq_context log_handler io_loop N/A N/A N/A 
    Create and return a new `Worker Extractor` that will combine all configured
    extractors to a single :class:`ZmqWorker`.
     N/A pipeline N/A settings SPYDER_EXTRACTOR_PIPELINE N/A N/A N/A pipeline extend N/A settings SPYDER_SCOPER_PIPELINE N/A processing N/A create_processing_function settings pipeline N/A pulling_socket N/A N/A zmq_context socket N/A zmq PULL N/A N/A N/A pulling_socket connect N/A settings ZEROMQ_WORKER_PROC_EXTRACTOR_PULL N/A pushing_socket N/A N/A zmq_context socket N/A zmq PUB N/A N/A N/A pushing_socket setsockopt N/A zmq HWM N/A settings ZEROMQ_WORKER_PROC_EXTRACTOR_PUB_HWM N/A N/A N/A pushing_socket connect N/A settings ZEROMQ_WORKER_PROC_EXTRACTOR_PUB N/A N/A ZmqWorker pulling_socket pushing_socket mgmt processing log_handler N/A settings LOG_LEVEL_WORKER io_loop io_loop N/A main N/A N/A settings N/A N/A N/A 
    The :meth:`main` method for worker processes.

    Here we will:

     - create a :class:`ZmqMgmt` instance

     - create a :class:`Fetcher` instance

     - initialize and instantiate the extractor chain

    The `settings` have to be loaded already.
     N/A identity N/A worker:%s:%s N/A N/A N/A socket gethostname N/A N/A os getpid N/A ctx N/A N/A zmq Context N/A io_loop N/A N/A IOLoop instance N/A log_pub N/A N/A ctx socket N/A zmq PUB N/A N/A N/A log_pub connect N/A settings ZEROMQ_LOGGING N/A zmq_logging_handler N/A PUBHandler log_pub N/A N/A zmq_logging_handler root_topic spyder.worker N/A logger N/A N/A logging getLogger N/A N/A N/A logger addHandler zmq_logging_handler N/A N/A N/A logger setLevel N/A settings LOG_LEVEL_WORKER N/A N/A N/A logger info process::Starting up another worker N/A mgmt N/A create_worker_management settings ctx io_loop N/A N/A N/A logger debug process::Initializing fetcher, extractor and scoper N/A fetcher N/A create_worker_fetcher settings mgmt ctx zmq_logging_handler io_loop N/A N/A N/A fetcher start N/A extractor N/A create_worker_extractor settings mgmt ctx zmq_logging_handler io_loop N/A N/A N/A extractor start quit_worker N/A N/A raw_msg N/A N/A N/A 
        When the worker should quit, stop the io_loop after 2 seconds.
         N/A msg N/A MgmtMessage raw_msg N/A N/A ZMQ_SPYDER_MGMT_WORKER_QUIT N/A msg data N/A N/A N/A N/A logger info process::We have been asked to shutdown, do so N/A N/A N/A N/A DelayedCallback N/A io_loop stop 2000 io_loop start N/A ack N/A MgmtMessage topic ZMQ_SPYDER_MGMT_WORKER identity identity data ZMQ_SPYDER_MGMT_WORKER_QUIT_ACK N/A N/A N/A N/A mgmt _out_stream send_multipart N/A N/A ack serialize N/A N/A N/A N/A mgmt add_callback ZMQ_SPYDER_MGMT_WORKER quit_worker N/A N/A N/A mgmt start N/A msg N/A MgmtMessage topic ZMQ_SPYDER_MGMT_WORKER identity identity data ZMQ_SPYDER_MGMT_WORKER_AVAIL N/A N/A N/A N/A mgmt _out_stream send_multipart N/A N/A msg serialize handle_shutdown_signal N/A N/A _sig _frame N/A N/A N/A 
        Called from the os when a shutdown signal is fired.
         N/A msg N/A MgmtMessage data ZMQ_SPYDER_MGMT_WORKER_QUIT N/A N/A quit_worker N/A N/A msg serialize N/A N/A N/A io_loop start N/A N/A N/A N/A signal signal N/A signal SIGINT handle_shutdown_signal N/A N/A N/A signal signal N/A signal SIGTERM handle_shutdown_signal N/A N/A N/A logger info process::waiting for action N/A N/A N/A N/A N/A io_loop start N/A N/A N/A ZMQError N/A N/A N/A N/A logger debug Caught a ZMQError. Hopefully during shutdown N/A N/A N/A logger debug N/A N/A traceback format_exc N/A mod N/A fetcher extractor mgmt N/A N/A N/A N/A mod close N/A N/A N/A logger info process::Houston: Worker down N/A N/A N/A ctx term N/A