Module Import alias ImportFrom alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias identifier Import alias identifier Import alias identifier Import alias identifier Assign NameStore Num Assign NameStore Str Assign NameStore Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr Num Assign NameStore Call AttributeLoad NameLoad attr Num Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore Call AttributeLoad NameLoad attr Num Assign NameStore Call AttributeLoad NameLoad attr Num Assign NameStore NameLoad Assign NameStore ListLoad ClassDef bases body FunctionDef arguments args NameParam NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Dict Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Num decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Num Expr Call AttributeLoad AttributeLoad NameLoad attr attr TupleLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Num Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd BinOpAdd Str Call NameLoad AttributeLoad NameLoad attr Str Call NameLoad AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr TupleLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr TupleLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr TupleLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Print Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str While AttributeLoad NameLoad attr body TryExcept body Assign TupleStore NameStore NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad handlers ExceptHandler body Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad Str attr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad NameLoad Global identifier If BoolOpAnd CompareGt Call NameLoad AttributeLoad NameLoad attr Num CompareEq NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Assign NameStore NameLoad TryFinally body TryExcept body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr While AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Num handlers ExceptHandler type NameLoad body Pass ExceptHandler body Assign AttributeStore NameLoad attr NameLoad Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpAdd Str Call AttributeLoad Str attr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad finalbody Expr Call AttributeLoad NameLoad attr Delete SubscriptDel AttributeLoad NameLoad attr Index NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str While AttributeLoad NameLoad attr body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad If CompareNotIn NameLoad NameLoad body Assign SubscriptStore NameLoad Index NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler body Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpAdd Str Call AttributeLoad Str attr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str For NameStore AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr body Assign NameStore Call AttributeLoad NameLoad attr ListLoad Call NameLoad NameLoad AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Assign SubscriptStore NameLoad Index NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad If BoolOpAnd CompareEq AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr CompareNotEq AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Num body If CompareNotEq AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Num body Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr While AttributeLoad NameLoad attr body TryExcept body If CompareEq AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr ListLoad Call NameLoad NameLoad AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr BinOpAdd AttributeLoad NameLoad attr Num If CompareNotEq AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Num body Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr If BoolOpAnd CompareNotEq AttributeLoad NameLoad attr Num CompareGt BinOpAdd Call NameLoad NameLoad Num AttributeLoad NameLoad attr body Break orelse If CompareEq AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr body If CompareEq Call NameLoad BinOpAdd NameLoad NameLoad Call NameLoad AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr body Break orelse Expr Call AttributeLoad NameLoad attr Num handlers ExceptHandler body Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpAdd Str Call AttributeLoad Str attr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str While AttributeLoad NameLoad attr body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr ListLoad NameLoad If UnaryOpNot NameLoad body Continue For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr handlers ExceptHandler body Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpAdd Str Call AttributeLoad Str attr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str For NameStore AttributeLoad NameLoad attr body If AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index NameLoad attr body Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index NameLoad attr decorator_list decorator_list ClassDef bases body FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr BinOpAdd Call AttributeLoad NameLoad attr Call NameLoad BinOpAdd BinOpAdd Call NameLoad SubscriptLoad NameLoad Index Num Str Call NameLoad SubscriptLoad NameLoad Index Num Str Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Str Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Num If CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr body If CompareEq AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr Str Return orelse Assign AttributeStore NameLoad attr NameLoad Raise Call NameLoad Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr TupleLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr TupleLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr Str While AttributeLoad NameLoad attr body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type NameLoad body Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str While AttributeLoad NameLoad attr body If CompareGt AttributeLoad NameLoad attr Num body Expr Call AttributeLoad NameLoad attr Num Continue Assign NameStore Call AttributeLoad NameLoad attr ListLoad NameLoad If UnaryOpNot NameLoad body Continue For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr BinOpAdd AttributeLoad NameLoad attr Num Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str Call NameLoad SubscriptLoad AttributeLoad AttributeLoad NameLoad attr attr Index Num decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str If CompareIs NameLoad NameLoad body Return Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd Str Call NameLoad AttributeLoad NameLoad attr If CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr Str orelse If CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr body If CompareEq AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str Call AttributeLoad Call NameLoad Call NameLoad AttributeLoad AttributeLoad NameLoad attr attr attr Num Str Call AttributeLoad Call NameLoad Call NameLoad NameLoad attr Num Str Call AttributeLoad Call NameLoad Call NameLoad NameLoad attr Num For NameStore AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr NameLoad If CompareEq AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr body Assign AttributeStore NameLoad attr BinOpSub AttributeLoad NameLoad attr Num For NameStore AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad If Call NameLoad AttributeLoad NameLoad attr Str body If CompareIsNot AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr If CompareEq AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr body Assign AttributeStore NameLoad attr BinOpSub AttributeLoad NameLoad attr Num For NameStore AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr NameLoad If Call NameLoad AttributeLoad NameLoad attr Str body If CompareIsNot AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr If CompareEq AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Num body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str NameLoad AttributeLoad NameLoad attr orelse If CompareEq AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Num body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str NameLoad AttributeLoad NameLoad attr orelse If CompareEq AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Num body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str NameLoad AttributeLoad NameLoad attr orelse Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Call NameLoad AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str NameLoad AttributeLoad NameLoad attr orelse Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str NameLoad AttributeLoad NameLoad attr orelse Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd Str Call NameLoad AttributeLoad NameLoad attr Str Return decorator_list FunctionDef arguments args NameParam NameParam defaults body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str Call NameLoad Call NameLoad BinOpAdd NameLoad NameLoad Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr BinOpAdd NameLoad NameLoad handlers ExceptHandler body Raise Call NameLoad Str decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore AttributeLoad NameLoad attr If CompareIn Str NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr Call AttributeLoad Str attr SubscriptLoad NameLoad Slice Num Return Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Num While AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore BinOpAdd NameLoad NameLoad If UnaryOpNot NameLoad body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad If CompareIn Str NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr Call AttributeLoad Str attr SubscriptLoad NameLoad Slice Num Break If CompareEq AttributeLoad NameLoad attr NameLoad body Return Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str Call NameLoad Call NameLoad SubscriptLoad NameLoad Index Num Str Return Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Num decorator_list FunctionDef arguments args NameParam defaults body Expr Str If CompareNotEq AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr NameLoad decorator_list decorator_list FunctionDef arguments args defaults body Expr Str TryExcept body Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore BinOpMult BinOpDiv Call NameLoad NameLoad Call NameLoad BinOpAdd NameLoad NameLoad Num Print Str Print BinOpAdd Str Call NameLoad NameLoad Print BinOpAdd Str Call NameLoad NameLoad Print BinOpAdd Str Call NameLoad NameLoad Print Str Print BinOpAdd Call NameLoad NameLoad Str handlers ExceptHandler body Pass Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call NameLoad decorator_list FunctionDef arguments args defaults body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr BinOpAdd Str Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr If CompareEq AttributeLoad NameLoad attr Str body Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr TupleLoad While AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Num Expr Call AttributeLoad NameLoad attr Expr Call NameLoad decorator_list If CompareEq NameLoad Str body Expr Call NameLoad
N/A N/A datetime distutils.command.config config N/A logging N/A pickle N/A Queue N/A signal N/A socket N/A sys N/A time N/A thread N/A traceback N/A uuid N/A modules.storage storage N/A modules.logger logger N/A modules.protocol protocol N/A modules.configuration configuration N/A buffSize 524288 N/A delimiter 

12345ZEEK6789
 N/A urlVisited N/A dict N/A urlPool N/A N/A Queue Queue 0 N/A urlToVisit N/A N/A Queue Queue 0 N/A scrappedURLlist N/A N/A visitedURLlist N/A N/A skippedURLlist N/A N/A outputQueue N/A N/A Queue Queue 200 N/A sessionStorageQueue N/A N/A Queue Queue 0 N/A serverRunning False N/A skippedSessions N/A Server N/A N/A __init__ N/A N/A self host port N/A N/A N/A N/A self host host N/A N/A self port port N/A N/A self s None N/A N/A self clientDict N/A N/A N/A self isActive True N/A N/A self requestLimit 0 N/A N/A self requestCount 0 N/A setup N/A N/A self configuration N/A N/A N/A Basic setup operation (socket binding, listen, etc) N/A N/A N/A logger log N/A logging DEBUG Socket initialization N/A N/A self s N/A N/A socket socket N/A socket AF_INET N/A socket SOCK_STREAM N/A N/A N/A N/A self s setsockopt N/A socket SOL_SOCKET N/A socket SO_REUSEADDR 1 N/A N/A N/A N/A self s bind N/A N/A self host N/A self port N/A N/A N/A N/A self s listen 5 N/A N/A N/A logger log N/A logging INFO N/A N/A N/A N/A Listening on [ N/A str N/A self host : N/A str N/A self port ] N/A N/A self configurationPayload configuration N/A N/A self requestLimit N/A N/A configuration config requestLimit N/A run N/A N/A self N/A N/A N/A Launches the urlDispatcher and mainRoutine threads N/A N/A N/A logger log N/A logging DEBUG Starting beginCrawlingProcedure N/A N/A N/A thread start_new_thread N/A self urlDispatcher N/A N/A N/A N/A thread start_new_thread N/A self mainRoutine N/A N/A N/A N/A thread start_new_thread N/A self storageRoutine N/A N/A listen N/A N/A self N/A N/A N/A Waits for new clients to connect and launches a new client thread accordingly N/A - - - - - - - - - - - - - - - N/A N/A N/A logger log N/A logging INFO Waiting for working nodes to connect... N/A N/A self isActive N/A N/A N/A N/A N/A client address N/A N/A N/A self s accept N/A N/A N/A thread start_new_thread N/A self connectionHandler N/A client address N/A N/A N/A N/A N/A exc_type exc_value exc_traceback N/A N/A sys exc_info N/A message N/A N/A join N/A N/A traceback format_exception exc_type exc_value exc_traceback N/A N/A N/A logger log N/A logging CRITICAL message N/A N/A self isActive False N/A connectionHandler N/A N/A self socket address N/A N/A N/A Creates a server-side client object and makes it listen for inputs N/A clientID N/A N/A uuid uuid4 N/A client N/A SSClient clientID socket address N/A N/A N/A self clientDict N/A clientID client N/A serverRunning N/A N/A N/A N/A len N/A self clientDict 0 N/A serverRunning False N/A N/A N/A N/A self run N/A serverRunning True N/A N/A N/A N/A N/A N/A N/A client sendConfig N/A self configurationPayload N/A N/A N/A client run N/A N/A client isActive N/A N/A N/A N/A time sleep 0.3 N/A N/A N/A EOFError N/A N/A N/A N/A N/A N/A client isActive False N/A N/A exc_type exc_value exc_traceback N/A N/A sys exc_info N/A message N/A N/A N/A join N/A N/A traceback format_exception exc_type exc_value exc_traceback N/A N/A N/A logger log N/A logging ERROR message N/A N/A N/A N/A client disconnect N/A N/A N/A self clientDict N/A clientID N/A urlDispatcher N/A N/A self N/A N/A N/A Reads from the urlPool, makes sure the url has not been visited and adds it to the urlToVisit Queue N/A N/A N/A logger log N/A logging INFO Starting server urlDispatcher N/A N/A self isActive N/A N/A N/A N/A url N/A N/A urlPool get True N/A N/A url urlVisited N/A N/A N/A urlVisited N/A url True N/A N/A N/A urlToVisit put url N/A N/A N/A scrappedURLlist append url N/A N/A N/A N/A N/A exc_type exc_value exc_traceback N/A N/A sys exc_info N/A message N/A N/A N/A join N/A N/A traceback format_exception exc_type exc_value exc_traceback N/A N/A N/A logger log N/A logging ERROR message N/A mainRoutine N/A N/A self N/A N/A N/A To Come in da future. For now, no use N/A N/A N/A logger log N/A logging INFO Starting server mainRoutine N/A url N/A N/A N/A self configurationPayload config rootUrls N/A N/A payload N/A N/A protocol URLPayload N/A N/A str url N/A N/A protocol URLPayload TOVISIT N/A packet N/A N/A protocol Packet N/A protocol URL payload N/A N/A urlVisited N/A url True N/A N/A N/A outputQueue put packet N/A N/A N/A N/A N/A self configurationPayload crawlingType N/A N/A protocol ConfigurationPayload STATIC_CRAWLING N/A N/A N/A N/A self configurationPayload config crawlDelay 0 N/A N/A N/A N/A N/A N/A self configurationPayload config crawlDelay 0 N/A N/A N/A N/A time sleep N/A N/A N/A self configurationPayload config crawlDelay N/A N/A self isActive N/A N/A N/A N/A N/A N/A N/A self configurationPayload crawlingType N/A N/A protocol ConfigurationPayload DYNAMIC_CRAWLING N/A N/A url N/A N/A urlToVisit get True N/A payload N/A N/A protocol URLPayload N/A N/A str url N/A N/A protocol URLPayload TOVISIT N/A packet N/A N/A protocol Packet N/A protocol URL payload N/A N/A N/A outputQueue put packet N/A N/A self requestCount N/A N/A self requestCount 1 N/A N/A N/A N/A N/A self configurationPayload config crawlDelay 0 N/A N/A N/A N/A time sleep N/A N/A N/A self configurationPayload config crawlDelay N/A N/A N/A N/A self requestLimit 0 N/A N/A N/A len visitedURLlist 1 N/A self requestLimit N/A N/A N/A N/A N/A N/A N/A self configurationPayload crawlingType N/A N/A protocol ConfigurationPayload STATIC_CRAWLING N/A N/A N/A N/A len N/A skippedURLlist visitedURLlist N/A len N/A N/A N/A self configurationPayload config rootUrls N/A N/A N/A N/A N/A N/A time sleep 0.3 N/A N/A N/A N/A N/A exc_type exc_value exc_traceback N/A N/A sys exc_info N/A message N/A N/A N/A join N/A N/A traceback format_exception exc_type exc_value exc_traceback N/A N/A N/A logger log N/A logging ERROR message N/A N/A N/A logger log N/A logging INFO Scrapping complete. Terminating... N/A N/A N/A self disconnectAllClient N/A N/A self isActive False N/A storageRoutine N/A N/A self N/A N/A N/A Stores session and data N/A N/A N/A logger log N/A logging INFO Starting server storageRoutine N/A N/A self isActive N/A N/A N/A N/A sessions N/A N/A protocol deQueue N/A sessionStorageQueue N/A N/A sessions N/A N/A N/A session sessions N/A N/A N/A N/A storage writeToFile session N/A session dataContainer N/A N/A N/A N/A N/A exc_type exc_value exc_traceback N/A N/A sys exc_info N/A message N/A N/A N/A join N/A N/A traceback format_exception exc_type exc_value exc_traceback N/A N/A N/A logger log N/A logging ERROR message N/A disconnectAllClient N/A N/A self N/A N/A N/A Disconnects all clients N/A connectedClient N/A self clientDict N/A N/A N/A N/A N/A self clientDict N/A connectedClient isActive N/A N/A N/A N/A N/A N/A self clientDict N/A connectedClient disconnect N/A N/A SSClient N/A N/A __init__ N/A N/A self cId socket address N/A N/A N/A N/A self id cId N/A N/A self socket socket N/A N/A self address address N/A N/A self isActive True N/A N/A self formattedAddr N/A N/A N/A logger formatBrackets N/A str N/A N/A N/A str N/A address N/A 0 : N/A str N/A address N/A 1 N/A N/A self sentCount 0 N/A N/A self data N/A N/A self configuration None N/A N/A N/A logger log N/A logging INFO N/A N/A N/A N/A logger GREEN N/A self formattedAddr Working node connected N/A logger NOCOLOR N/A sendConfig N/A N/A self configuration N/A N/A N/A Sends the configuration to the client N/A N/A N/A logger log N/A logging DEBUG N/A N/A self formattedAddr Sending configuration N/A N/A self configuration configuration N/A packet N/A N/A protocol Packet N/A protocol CONFIG N/A self configuration N/A N/A N/A self writeSocket packet N/A N/A N/A logger log N/A logging DEBUG N/A N/A self formattedAddr Configuration sent waiting for ACK N/A packet N/A N/A self readSocket 5 N/A N/A N/A packet type N/A protocol INFO N/A N/A N/A N/A N/A packet payload info N/A N/A protocol InfoPayload CLIENT_ACK N/A N/A N/A N/A logger log N/A logging DEBUG N/A N/A self formattedAddr Working node ACK received (configuration) N/A N/A N/A N/A self isActive False N/A N/A Exception Unable to transmit configuration N/A run N/A N/A self N/A N/A N/A Launched the input and output thread with the client itself N/A N/A N/A thread start_new_thread N/A self inputThread N/A N/A N/A N/A thread start_new_thread N/A self outputThread N/A N/A inputThread N/A N/A self N/A N/A N/A Listens for inputs from the client N/A N/A N/A logger log N/A logging DEBUG N/A N/A self formattedAddr Listening for packets N/A N/A self isActive N/A N/A N/A N/A deserializedPacket N/A N/A self readSocket N/A N/A N/A self dispatcher deserializedPacket N/A N/A N/A EOFError N/A N/A N/A self isActive False N/A outputThread N/A N/A self N/A N/A N/A Checks if there are messages to send to the client and sends them N/A N/A self isActive N/A N/A N/A N/A self sentCount 5 N/A N/A N/A N/A time sleep 0.03 N/A N/A packetToBroadCast N/A N/A protocol deQueue N/A outputQueue N/A N/A packetToBroadCast N/A N/A N/A packet packetToBroadCast N/A N/A N/A N/A self writeSocket packet N/A N/A self sentCount N/A N/A self sentCount 1 N/A N/A N/A logger log N/A logging DEBUG N/A N/A N/A self formattedAddr Sending URL  N/A str N/A N/A N/A packet payload urlList N/A 0 N/A dispatcher N/A N/A self packet N/A N/A N/A Dispatches packets to the right packet queue or takes action if needed (ie: infoPacket) N/A N/A packet None N/A N/A N/A N/A N/A logger log N/A logging DEBUG N/A Dispatching packet of type:  N/A str N/A packet type N/A N/A N/A packet type N/A protocol INFO N/A N/A N/A N/A logger log N/A logging DEBUG N/A N/A self formattedAddr Received INFO packet N/A N/A N/A N/A packet type N/A protocol URL N/A N/A N/A N/A N/A packet payload type N/A N/A protocol URLPayload SCRAPPED_URL N/A N/A N/A N/A logger log N/A logging INFO N/A N/A N/A N/A N/A N/A N/A self formattedAddr Receiving scrapped URLs :  N/A N/A N/A str N/A len N/A N/A packet payload urlList center 5  /  N/A N/A N/A str N/A len scrappedURLlist center 7  -  N/A N/A N/A str N/A len skippedURLlist center 5 N/A url N/A N/A packet payload urlList N/A N/A N/A N/A urlPool put url N/A N/A N/A N/A packet payload type N/A N/A protocol URLPayload VISITED N/A N/A N/A self sentCount N/A N/A self sentCount 1 N/A url N/A N/A packet payload urlList N/A N/A N/A N/A logger log N/A logging INFO N/A N/A self formattedAddr Receiving scrapped data N/A N/A N/A logger log N/A logging DEBUG N/A N/A N/A self formattedAddr Receiving scrapped data url N/A N/A N/A visitedURLlist append url N/A N/A hasattr N/A packet payload session N/A N/A N/A N/A N/A packet payload session None N/A N/A N/A N/A sessionStorageQueue put N/A N/A packet payload session N/A N/A N/A N/A packet payload type N/A N/A protocol URLPayload SKIPPED N/A N/A N/A self sentCount N/A N/A self sentCount 1 N/A url N/A N/A packet payload urlList N/A N/A N/A N/A skippedURLlist append url N/A N/A hasattr N/A packet payload session N/A N/A N/A N/A N/A packet payload session None N/A N/A N/A N/A sessionStorageQueue put N/A N/A packet payload session N/A N/A N/A N/A N/A packet payload session returnCode -1 N/A N/A N/A N/A logger log N/A logging INFO N/A N/A N/A N/A N/A logger PINK N/A self formattedAddr Skipped (timeout) :  url N/A logger NOCOLOR N/A N/A N/A N/A N/A N/A packet payload session returnCode -2 N/A N/A N/A N/A logger log N/A logging INFO N/A N/A N/A N/A N/A logger PINK N/A self formattedAddr Skipped (request not allowed - robot parser) :  url N/A logger NOCOLOR N/A N/A N/A N/A N/A N/A packet payload session returnCode -100 N/A N/A N/A N/A logger log N/A logging INFO N/A N/A N/A N/A N/A logger YELLOW N/A self formattedAddr Skipped (unknown error) :  url N/A logger NOCOLOR N/A N/A N/A N/A logger log N/A logging INFO N/A N/A N/A N/A N/A N/A N/A logger BLUE N/A self formattedAddr Skipped (html error  N/A str N/A N/A N/A packet payload session returnCode ) :  url N/A logger NOCOLOR N/A N/A N/A N/A logger log N/A logging INFO N/A N/A N/A N/A N/A logger RED N/A self formattedAddr No session returned url N/A logger NOCOLOR N/A N/A N/A N/A logger log N/A logging CRITICAL N/A N/A Unrecognized packet type :  N/A str N/A packet type . This packet was dropped N/A N/A writeSocket N/A N/A self obj N/A N/A N/A N/A N/A serializedObj N/A N/A pickle dumps obj N/A N/A N/A logger log N/A logging DEBUG N/A N/A N/A N/A self formattedAddr Sending  N/A str N/A len N/A serializedObj delimiter  bytes N/A N/A N/A N/A self socket sendall N/A serializedObj delimiter N/A N/A N/A N/A N/A Exception Unable to write to socket (client disconnected) N/A readSocket N/A N/A self timeOut N/A None N/A N/A N/A N/A N/A self socket settimeout timeOut N/A data N/A self data N/A N/A 

12345ZEEK6789
 data N/A N/A data N/A N/A data split 

12345ZEEK6789
 N/A N/A self data N/A N/A 

12345ZEEK6789
 join N/A data N/A 1 N/A N/A N/A pickle loads N/A data N/A 0 N/A N/A self isActive N/A N/A buffer N/A N/A N/A self socket recv buffSize N/A data N/A data buffer N/A N/A buffer N/A N/A N/A N/A logger log N/A logging INFO N/A N/A N/A N/A logger RED N/A self formattedAddr Lost connection N/A logger NOCOLOR N/A N/A self isActive False N/A N/A 

12345ZEEK6789
 data N/A N/A data N/A N/A data split 

12345ZEEK6789
 N/A N/A self data N/A N/A 

12345ZEEK6789
 join N/A data N/A 1 N/A N/A N/A N/A self isActive False N/A N/A N/A N/A N/A logger log N/A logging DEBUG N/A N/A N/A N/A self formattedAddr Receiving  N/A str N/A len N/A data N/A 0  bytes N/A N/A N/A pickle loads N/A data N/A 0 N/A disconnect N/A N/A self N/A N/A N/A Disconnects the client N/A N/A N/A self socket None N/A N/A N/A N/A logger log N/A logging INFO N/A N/A N/A N/A logger RED N/A self formattedAddr Disconnecting N/A logger NOCOLOR N/A N/A self isActive False N/A N/A N/A N/A self socket close N/A N/A self socket None N/A N/A ending N/A N/A N/A N/A N/A Temporary ending routine N/A N/A N/A scrapped N/A len scrappedURLlist N/A skipped N/A len skippedURLlist N/A visited N/A len visitedURLlist N/A skipRate N/A N/A N/A float skipped N/A float N/A skipped visited 100 N/A 

------------------------- N/A N/A Scrapped :  N/A str scrapped N/A N/A Skipped :  N/A str skipped N/A N/A Visited :  N/A str visited N/A ------------------------- N/A N/A N/A str skipRate % skipping rate
 N/A N/A N/A N/A N/A N/A N/A sys exit N/A handler N/A N/A signum frame N/A N/A N/A N/A ending N/A main N/A N/A N/A N/A N/A N/A N/A signal signal N/A signal SIGINT handler N/A N/A N/A logger printAsciiLogo N/A config N/A N/A configuration configParser N/A N/A N/A logger init N/A config logPath N/A server- N/A str N/A N/A N/A datetime datetime now N/A N/A logger debugFlag N/A config verbose N/A N/A N/A config crawling dynamic N/A N/A nodeConfig N/A N/A protocol ConfigurationPayload N/A N/A protocol ConfigurationPayload DYNAMIC_CRAWLING config N/A N/A nodeConfig N/A N/A protocol ConfigurationPayload N/A N/A protocol ConfigurationPayload STATIC_CRAWLING config N/A server N/A Server N/A config host N/A config port N/A N/A N/A server setup nodeConfig N/A N/A N/A thread start_new_thread N/A server listen N/A N/A N/A server isActive N/A N/A N/A N/A time sleep 0.5 N/A N/A N/A server disconnectAllClient N/A N/A ending N/A N/A N/A __name__ __main__ N/A N/A N/A main