Module ImportFrom alias Import alias Import alias Import alias Import alias ImportFrom alias ClassDef bases NameLoad body Assign NameStore Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr keyword SubscriptLoad NameLoad Index Str keyword SubscriptLoad NameLoad Index Str TryExcept body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str handlers ExceptHandler type NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad NameLoad attr Num decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Assign NameStore Call AttributeLoad Str attr keyword SubscriptLoad NameLoad Index Str keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword Num Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad UnaryOpUSub SubscriptLoad NameLoad Index Str If BoolOpAnd CompareIn Str NameLoad CompareNotEq SubscriptLoad NameLoad Index Str Num body Assign NameStore Call AttributeLoad Str attr keyword SubscriptLoad NameLoad Index Str keyword SubscriptLoad NameLoad Index Str keyword SubscriptLoad NameLoad Index Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad SubscriptLoad NameLoad Index Str Assign SubscriptStore NameLoad Index Str NameLoad Assign SubscriptStore NameLoad Index Str NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad decorator_list decorator_list
N/A base_handler BaseHandler N/A tldextract N/A redis N/A pickle N/A sys redis.exceptions ConnectionError ScraperHandler N/A BaseHandler N/A N/A schema scraper_schema.json setup N/A N/A self settings N/A N/A N/A 
        Setup redis and tldextract
         N/A N/A self extract N/A N/A tldextract TLDExtract N/A N/A self redis_conn N/A N/A redis Redis host N/A settings N/A REDIS_HOST port N/A settings N/A REDIS_PORT N/A N/A N/A N/A N/A N/A self redis_conn info N/A N/A N/A N/A self logger debug Connected to Redis in ScraperHandler N/A N/A N/A ConnectionError N/A N/A N/A N/A N/A self logger error Failed to connect to Redis in ScraperHandler N/A N/A N/A sys exit 1 N/A handle N/A N/A self dict N/A N/A N/A 
        Processes a vaild crawl request

        @param dict: a valid dictionary object
         N/A ex_res N/A N/A self extract N/A dict N/A url N/A key N/A N/A {sid}:{dom}.{suf}:queue format sid N/A dict N/A spiderid dom N/A ex_res domain suf N/A ex_res suffix N/A val N/A N/A pickle dumps dict protocol -1 N/A N/A N/A N/A self redis_conn zadd key val N/A N/A dict N/A priority N/A N/A N/A expires dict N/A N/A dict N/A expires 0 N/A N/A key N/A N/A timeout:{sid}:{appid}:{crawlid} format sid N/A dict N/A spiderid appid N/A dict N/A appid crawlid N/A dict N/A crawlid N/A N/A N/A N/A self redis_conn set key N/A dict N/A expires N/A N/A dict N/A parsed True N/A N/A dict N/A valid True N/A N/A N/A N/A self logger info Added crawl to Redis extra dict N/A N/A