Module ImportFrom alias ImportFrom alias Import alias ImportFrom alias ImportFrom alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ClassDef bases NameLoad body Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Dict Str Str Str Str Str Str Str Call NameLoad Num Num Num Num Num Num Assign NameStore Call NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad NameLoad Assign AttributeStore NameLoad attr Call NameLoad NameLoad Assign AttributeStore NameLoad attr Call NameLoad NameLoad Assign AttributeStore NameLoad attr Call NameLoad NameLoad Assign AttributeStore NameLoad attr Call NameLoad NameLoad Assign AttributeStore NameLoad attr Call NameLoad NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword Str keyword Str keyword AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad NameLoad attr TryExcept body Expr Call AttributeLoad AttributeLoad NameLoad attr attr handlers ExceptHandler type NameLoad body Pass decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call NameLoad keyword Num AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad AugAssignAdd SubscriptStore AttributeLoad NameLoad attr Index Str Num decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad AugAssignAdd SubscriptStore AttributeLoad NameLoad attr Index Str Num decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad AugAssignAdd SubscriptStore AttributeLoad NameLoad attr Index Str Num decorator_list FunctionDef arguments args NameParam NameParam defaults body AugAssignAdd SubscriptStore AttributeLoad NameLoad attr Index Str Num If CompareIn SubscriptLoad SubscriptLoad NameLoad Index Num Index Num ListLoad Str Str body Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Num If CompareEq NameLoad Str body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Return If CompareEq NameLoad Str body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Return Raise Call NameLoad Str decorator_list FunctionDef arguments args NameParam NameParam defaults body If CompareIn SubscriptLoad SubscriptLoad NameLoad Index Num Index Num ListLoad Str Str body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad AugAssignAdd SubscriptStore AttributeLoad NameLoad attr Index Str Num decorator_list FunctionDef arguments args NameParam NameParam defaults body If CompareIn SubscriptLoad SubscriptLoad NameLoad Index Num Index Num ListLoad Str Str body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad AugAssignAdd SubscriptStore AttributeLoad NameLoad attr Index Str Num decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str If CompareEq Call NameLoad NameLoad Num body Return Call NameLoad Str NameLoad orelse If CompareEq Call NameLoad NameLoad Num body Assign TupleStore NameStore NameStore Call NameLoad Str NameLoad Return TupleLoad NameLoad NameLoad NameLoad Raise Call NameLoad Str decorator_list decorator_list FunctionDef arguments args defaults body Assign NameStore Call NameLoad keyword Str Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Str Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword NameLoad keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad keyword AttributeLoad NameLoad attr Assign NameStore IfExp AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Str Assign NameStore IfExp AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Str Assign NameStore Call NameLoad NameLoad NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr decorator_list If CompareEq NameLoad Str body Expr Call NameLoad
N/A time time datetime timedelta N/A logging argparse ArgumentParser struct unpack N/A zmq zmq.eventloop.ioloop IOLoop zmq.eventloop.zmqstream ZMQStream distributed_frontera.settings Settings socket_config SocketConfig Server N/A object N/A N/A ctx None N/A loop None N/A stats None N/A spiders_in None N/A spiders_out None N/A sw_in None N/A sw_out None N/A db_in None N/A db_out None __init__ N/A N/A self hostname base_port N/A N/A N/A N/A self ctx N/A N/A zmq Context N/A N/A self loop N/A N/A IOLoop instance N/A N/A self stats N/A started spiders_out_recvd spiders_in_recvd db_in_recvd db_out_recvd sw_in_recvd sw_out_recvd N/A time 0 0 0 0 0 0 N/A socket_config N/A SocketConfig hostname base_port N/A spiders_in_s N/A N/A N/A self ctx socket N/A zmq XPUB N/A spiders_out_s N/A N/A N/A self ctx socket N/A zmq XSUB N/A sw_in_s N/A N/A N/A self ctx socket N/A zmq XPUB N/A sw_out_s N/A N/A N/A self ctx socket N/A zmq XSUB N/A db_in_s N/A N/A N/A self ctx socket N/A zmq XPUB N/A db_out_s N/A N/A N/A self ctx socket N/A zmq XSUB N/A N/A N/A spiders_in_s bind N/A N/A socket_config spiders_in N/A N/A N/A spiders_out_s bind N/A N/A socket_config spiders_out N/A N/A N/A sw_in_s bind N/A N/A socket_config sw_in N/A N/A N/A sw_out_s bind N/A N/A socket_config sw_out N/A N/A N/A db_in_s bind N/A N/A socket_config db_in N/A N/A N/A db_out_s bind N/A N/A socket_config db_out N/A N/A self spiders_in N/A ZMQStream spiders_in_s N/A N/A self spiders_out N/A ZMQStream spiders_out_s N/A N/A self sw_in N/A ZMQStream sw_in_s N/A N/A self sw_out N/A ZMQStream sw_out_s N/A N/A self db_in N/A ZMQStream db_in_s N/A N/A self db_out N/A ZMQStream db_out_s N/A N/A N/A N/A self spiders_out on_recv N/A self handle_spiders_out_recv N/A N/A N/A N/A self sw_out on_recv N/A self handle_sw_out_recv N/A N/A N/A N/A self db_out on_recv N/A self handle_db_out_recv N/A N/A N/A N/A self sw_in on_recv N/A self handle_sw_in_recv N/A N/A N/A N/A self db_in on_recv N/A self handle_db_in_recv N/A N/A N/A N/A self spiders_in on_recv N/A self handle_spiders_in_recv N/A N/A N/A logging basicConfig format %(asctime)s %(message)s datefmt %Y-%m-%d %H:%M:%S level N/A logging INFO N/A N/A self logger N/A N/A logging getLogger distributed_frontera.messagebus.zeromq.broker.Server N/A start N/A N/A self N/A N/A N/A N/A N/A N/A self logger info Distributed Frontera ZeroMQ broker is started. N/A N/A N/A self log_stats N/A N/A N/A N/A N/A N/A self loop start N/A N/A N/A KeyboardInterrupt N/A N/A N/A log_stats N/A N/A self N/A N/A N/A N/A N/A N/A self logger info N/A self stats N/A N/A N/A N/A self loop add_timeout N/A timedelta seconds 10 N/A self log_stats N/A handle_spiders_out_recv N/A N/A self msg N/A N/A N/A N/A N/A N/A self sw_in send_multipart msg N/A N/A N/A N/A self db_in send_multipart msg N/A N/A N/A self stats N/A spiders_out_recvd 1 N/A handle_sw_out_recv N/A N/A self msg N/A N/A N/A N/A N/A N/A self db_in send_multipart msg N/A N/A N/A self stats N/A sw_out_recvd 1 N/A handle_db_out_recv N/A N/A self msg N/A N/A N/A N/A N/A N/A self spiders_in send_multipart msg N/A N/A N/A self stats N/A db_out_recvd 1 N/A handle_db_in_recv N/A N/A self msg N/A N/A N/A N/A N/A self stats N/A db_in_recvd 1 N/A N/A N/A N/A msg N/A 0 N/A 0 N/A N/A N/A N/A action identity partition_id N/A N/A self decode_subscription N/A msg N/A 0 N/A N/A identity sl N/A N/A N/A N/A N/A self spiders_out send_multipart msg N/A N/A N/A identity us N/A N/A N/A N/A N/A self sw_out send_multipart msg N/A N/A N/A AttributeError Unknown identity in channel subscription. N/A handle_sw_in_recv N/A N/A self msg N/A N/A N/A N/A N/A N/A msg N/A 0 N/A 0 N/A N/A N/A N/A N/A N/A self spiders_out send_multipart msg N/A N/A N/A self stats N/A sw_in_recvd 1 N/A handle_spiders_in_recv N/A N/A self msg N/A N/A N/A N/A N/A N/A msg N/A 0 N/A 0 N/A N/A N/A N/A N/A N/A self db_out send_multipart msg N/A N/A N/A self stats N/A spiders_in_recvd 1 N/A decode_subscription N/A N/A self msg N/A N/A N/A 

        :param msg:
        :return: tuple of action, identity, partition_id
        where
        action is 1 - subscription, 0 - unsubscription,
        identity - 2 characters,
        partition_id - 8 bit unsigned integer (None if absent)
         N/A N/A N/A len msg 4 N/A N/A N/A unpack >B2sB msg N/A N/A N/A N/A len msg 3 N/A N/A N/A action identity N/A unpack >B2s msg N/A N/A action identity None N/A N/A ValueError Can't decode subscription correctly. N/A N/A main N/A N/A N/A N/A N/A parser N/A ArgumentParser description Crawl frontier worker. N/A N/A N/A parser add_argument --config type str help Settings module name, should be accessible by import. N/A N/A N/A parser add_argument --hostname type str help Hostname or IP address to bind. Default is 127.0.0.1 N/A N/A N/A parser add_argument --log-level -L type str default INFO help Log level, for ex. DEBUG, INFO, WARN, ERROR, FATAL. Default is INFO. N/A N/A N/A parser add_argument --port type int help Base port number, server will bind to 6 ports starting from base. Default is 5550 N/A args N/A N/A parser parse_args N/A settings N/A Settings module N/A args config N/A hostname N/A N/A args hostname N/A args hostname N/A N/A settings get ZMQ_HOSTNAME N/A port N/A N/A args port N/A args port N/A N/A settings get ZMQ_BASE_PORT N/A server N/A Server hostname port N/A N/A N/A N/A server logger setLevel N/A args log_level N/A N/A N/A server start N/A N/A N/A __name__ __main__ N/A N/A N/A main