Module Expr Str Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body For NameStore NameLoad body If Call NameLoad NameLoad NameLoad body If BoolOpOr AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad NameLoad body Expr Yield NameLoad orelse Assign NameStore AttributeLoad Call NameLoad NameLoad attr If BoolOpAnd NameLoad CompareNotIn NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd Str Call NameLoad NameLoad Str AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad orelse Expr Yield NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body If UnaryOpNot Call NameLoad NameLoad Str body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore BoolOpOr AttributeLoad Call NameLoad NameLoad attr Str Return Call NameLoad Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad Str NameLoad If UnaryOpNot NameLoad body Return Call AttributeLoad NameLoad attr Str Assign NameStore BinOpMod Str Call AttributeLoad Str attr GeneratorExp Call AttributeLoad NameLoad attr NameLoad comprehension NameStore NameLoad CompareIsNot NameLoad NameLoad Return Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call NameLoad decorator_list decorator_list
N/A N/A 
Offsite Spider Middleware

See documentation in docs/topics/spider-middleware.rst
 N/A re ants signals ants.http Request ants.utils.httpobj urlparse_cached ants.utils log OffsiteMiddleware N/A object N/A __init__ N/A N/A self stats N/A N/A N/A N/A self stats stats N/A from_crawler N/A N/A cls crawler N/A N/A N/A o N/A cls N/A crawler stats N/A N/A N/A N/A crawler signals connect N/A o spider_opened signal N/A signals spider_opened N/A o N/A classmethod process_spider_output N/A N/A self response result spider N/A N/A N/A x result N/A N/A N/A isinstance x Request N/A N/A N/A N/A x dont_filter N/A N/A self should_follow x spider N/A N/A N/A x N/A N/A domain N/A N/A urlparse_cached x hostname N/A N/A domain N/A domain N/A self domains_seen N/A N/A N/A N/A N/A self domains_seen add domain N/A N/A N/A log spider_log N/A N/A N/A Filtered offsite request to  N/A str domain : N/A x url level N/A log DEBUG spider spider N/A N/A N/A N/A self stats inc_value offsite/domains spider spider N/A N/A N/A N/A self stats inc_value offsite/filtered spider spider N/A N/A N/A x N/A should_follow N/A N/A self request spider N/A N/A N/A N/A N/A hasattr self host_regex N/A N/A N/A N/A self spider_opened spider N/A regex N/A self host_regex N/A host N/A N/A N/A urlparse_cached request hostname N/A N/A bool N/A N/A regex search host N/A get_host_regex N/A N/A self spider N/A N/A N/A Override this method to implement a different offsite policy N/A allowed_domains N/A getattr spider allowed_domains None N/A N/A allowed_domains N/A N/A N/A N/A re compile N/A regex N/A ^(.*\.)?(%s)$ N/A N/A | join N/A N/A N/A re escape d N/A d allowed_domains N/A d None N/A N/A N/A re compile regex N/A spider_opened N/A N/A self spider N/A N/A N/A N/A self host_regex N/A N/A self get_host_regex spider N/A N/A self domains_seen N/A set N/A N/A