Module Import alias ImportFrom alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ClassDef bases AttributeLoad NameLoad attr body FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Num Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Str SubscriptLoad AttributeLoad NameLoad attr Index Str keyword AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr For NameStore Call AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr For NameStore Call AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr Call NameLoad Assign NameStore Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index Str NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call NameLoad NameLoad AttributeLoad NameLoad attr Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr AttributeLoad NameLoad attr Str Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr NameLoad Assign NameStore Str Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index Str NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call NameLoad NameLoad AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr Call NameLoad Assign NameStore Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index Str NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call NameLoad NameLoad AttributeLoad NameLoad attr Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr NameLoad Assign NameStore Str Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index Str NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call NameLoad NameLoad AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list decorator_list ClassDef bases NameLoad body Assign NameStore Num FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr ListLoad TupleLoad Str AttributeLoad AttributeLoad NameLoad attr attr Dict Str NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad Assign NameStore Call NameLoad NameLoad keyword AttributeLoad NameLoad attr Assign NameStore Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Str SubscriptLoad AttributeLoad NameLoad attr Index Str AttributeLoad NameLoad attr NameLoad Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad keyword BinOpMod Str AttributeLoad NameLoad attr keyword BinOpMod Str AttributeLoad NameLoad attr keyword Dict Str Str Assign NameStore Call NameLoad Assign AttributeStore NameLoad attr Str Assign AttributeStore NameLoad attr NameLoad FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Num AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr Str AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr Call AttributeLoad NameLoad attr decorator_list Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr NameLoad Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list decorator_list If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr
N/A N/A logging logging StreamHandler N/A sys N/A os.path N/A time N/A random N/A unittest N/A tornado N/A tornado.httpserver N/A tornado.web N/A zmq zmq.eventloop.ioloop IOLoop zmq.eventloop.zmqstream ZMQStream spyder.core.constants ZMQ_SPYDER_MGMT_WORKER spyder.core.constants ZMQ_SPYDER_MGMT_WORKER_QUIT spyder.core.constants ZMQ_SPYDER_MGMT_WORKER_QUIT_ACK spyder.core.messages DataMessage MgmtMessage spyder.core.mgmt ZmqMgmt spyder.core.worker AsyncZmqWorker spyder.core.settings Settings spyder.processor.fetcher FetchProcessor spyder.thrift.gen.ttypes CrawlUri ZmqTornadoIntegrationTest N/A N/A unittest TestCase N/A setUp N/A N/A self N/A N/A N/A N/A self _io_loop N/A N/A IOLoop instance N/A N/A self _ctx N/A N/A zmq Context 1 N/A N/A N/A self _setup_mgmt_sockets N/A N/A N/A self _setup_data_sockets N/A N/A self _mgmt N/A ZmqMgmt N/A N/A self _mgmt_sockets N/A worker_sub N/A N/A self _mgmt_sockets N/A worker_pub io_loop N/A self _io_loop N/A N/A N/A N/A self _mgmt start N/A N/A N/A N/A self _mgmt add_callback ZMQ_SPYDER_MGMT_WORKER N/A self on_mgmt_end N/A tearDown N/A N/A self N/A N/A N/A N/A N/A N/A self _mgmt stop N/A socket N/A N/A N/A self _mgmt_sockets itervalues N/A N/A N/A N/A socket close N/A socket N/A N/A N/A self _worker_sockets itervalues N/A N/A N/A N/A socket close N/A N/A N/A N/A self _ctx term N/A _setup_mgmt_sockets N/A N/A self N/A N/A N/A N/A self _mgmt_sockets N/A dict N/A mgmt_master_worker inproc://master/worker/coordination/ N/A sock N/A N/A N/A self _ctx socket N/A zmq PUB N/A N/A N/A sock bind mgmt_master_worker N/A N/A N/A self _mgmt_sockets N/A tmp1 sock N/A N/A N/A self _mgmt_sockets N/A master_pub N/A ZMQStream sock N/A self _io_loop N/A N/A N/A self _mgmt_sockets N/A worker_sub N/A N/A N/A self _ctx socket N/A zmq SUB N/A N/A N/A N/A N/A self _mgmt_sockets N/A worker_sub setsockopt N/A zmq SUBSCRIBE N/A N/A N/A N/A N/A self _mgmt_sockets N/A worker_sub connect mgmt_master_worker N/A mgmt_worker_master inproc://worker/master/coordination/ N/A N/A N/A self _mgmt_sockets N/A worker_pub N/A N/A N/A self _ctx socket N/A zmq PUB N/A N/A N/A N/A N/A self _mgmt_sockets N/A worker_pub bind mgmt_worker_master N/A sock N/A N/A N/A self _ctx socket N/A zmq SUB N/A N/A N/A sock setsockopt N/A zmq SUBSCRIBE N/A N/A N/A sock connect mgmt_worker_master N/A N/A N/A self _mgmt_sockets N/A tmp2 sock N/A N/A N/A self _mgmt_sockets N/A master_sub N/A ZMQStream sock N/A self _io_loop N/A _setup_data_sockets N/A N/A self N/A N/A N/A N/A self _worker_sockets N/A dict N/A data_master_worker inproc://master/worker/pipeline/ N/A sock N/A N/A N/A self _ctx socket N/A zmq PUSH N/A N/A N/A sock bind data_master_worker N/A N/A N/A self _worker_sockets N/A tmp3 sock N/A N/A N/A self _worker_sockets N/A master_push N/A ZMQStream sock N/A self _io_loop N/A N/A N/A self _worker_sockets N/A worker_pull N/A N/A N/A self _ctx socket N/A zmq PULL N/A N/A N/A N/A N/A self _worker_sockets N/A worker_pull connect data_master_worker N/A data_worker_master inproc://worker/master/pipeline/ N/A N/A N/A self _worker_sockets N/A worker_pub N/A N/A N/A self _ctx socket N/A zmq PUB N/A N/A N/A N/A N/A self _worker_sockets N/A worker_pub bind data_worker_master N/A sock N/A N/A N/A self _ctx socket N/A zmq SUB N/A N/A N/A sock setsockopt N/A zmq SUBSCRIBE N/A N/A N/A sock connect data_worker_master N/A N/A N/A self _worker_sockets N/A tmp4 sock N/A N/A N/A self _worker_sockets N/A master_sub N/A ZMQStream sock N/A self _io_loop N/A on_mgmt_end N/A N/A self _msg N/A N/A N/A N/A N/A N/A self _io_loop stop N/A N/A SimpleFetcherTestCase N/A ZmqTornadoIntegrationTest N/A N/A port 8085 setUp N/A N/A self N/A N/A N/A N/A N/A ZmqTornadoIntegrationTest setUp self N/A path N/A N/A N/A os path join N/A N/A N/A os path dirname __file__ static N/A application N/A N/A N/A tornado web Application N/A N/A /(.*) N/A N/A tornado web StaticFileHandler N/A path path N/A N/A self _server N/A N/A N/A tornado httpserver HTTPServer application io_loop N/A self _io_loop N/A N/A N/A N/A self _server listen N/A self port N/A tearDown N/A N/A self N/A N/A N/A N/A N/A ZmqTornadoIntegrationTest tearDown self N/A N/A N/A N/A self _server stop N/A test_fetching_etag_works N/A N/A self N/A N/A N/A settings N/A Settings N/A fetcher N/A FetchProcessor settings io_loop N/A self _io_loop N/A worker N/A AsyncZmqWorker N/A N/A self _worker_sockets N/A worker_pull N/A N/A self _worker_sockets N/A worker_pub N/A self _mgmt fetcher N/A StreamHandler N/A sys stdout N/A logging DEBUG N/A self _io_loop N/A N/A N/A worker start N/A curi N/A CrawlUri url N/A http://localhost:%s/robots.txt N/A self port effective_url N/A http://127.0.0.1:%s/robots.txt N/A self port req_header N/A Etag "3926227169c58185234888b60000c6eb1169577d" N/A msg N/A DataMessage N/A N/A msg identity me N/A N/A msg curi curi assert_expected_result_and_stop N/A N/A raw_msg N/A N/A N/A msg N/A DataMessage raw_msg N/A N/A N/A self assertEqual 304 N/A N/A msg curi status_code N/A N/A N/A self assertEqual N/A N/A msg curi content_body N/A death N/A MgmtMessage topic ZMQ_SPYDER_MGMT_WORKER data ZMQ_SPYDER_MGMT_WORKER_QUIT N/A N/A N/A N/A N/A self _mgmt_sockets N/A master_pub send_multipart N/A N/A death serialize N/A N/A N/A N/A N/A N/A self _worker_sockets N/A master_sub on_recv assert_expected_result_and_stop N/A N/A N/A N/A N/A self _worker_sockets N/A master_push send_multipart N/A N/A msg serialize N/A N/A N/A N/A self _io_loop start N/A N/A N/A N/A __name__ __main__ N/A N/A N/A N/A unittest main