Module Import alias Import alias ImportFrom alias ImportFrom alias alias ImportFrom alias alias ImportFrom alias TryExcept body Import alias Import alias Assign NameStore NameLoad handlers ExceptHandler type NameLoad body Assign NameStore NameLoad FunctionDef arguments args NameParam defaults body If BoolOpAnd AttributeLoad NameLoad attr CompareIn AttributeLoad AttributeLoad NameLoad attr attr TupleLoad Str Str body Return ListLoad Call AttributeLoad NameLoad attr Return ListLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore ListLoad If UnaryOpNot NameLoad body Raise Call NameLoad Str TryExcept body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad NameLoad attr Str If BoolOpAnd NameLoad Call AttributeLoad SubscriptLoad NameLoad Index Num attr Str body Assign NameStore Call AttributeLoad SubscriptLoad NameLoad Index Num attr Str orelse Assign NameStore AttributeLoad NameLoad attr Assign NameStore ListComp Call AttributeLoad Call AttributeLoad NameLoad attr Str attr comprehension NameStore Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr Str AugAssignAdd NameStore ListComp Call AttributeLoad Call AttributeLoad NameLoad attr Str attr comprehension NameStore Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr Str For NameStore NameLoad body TryExcept body Expr Call AttributeLoad NameLoad attr Str handlers ExceptHandler type NameLoad body Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr Str keyword Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad If CompareIn AttributeLoad NameLoad attr TupleLoad Str Str body TryExcept body Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad handlers ExceptHandler body Print Str NameLoad Continue orelse If BoolOpOr CompareIn AttributeLoad NameLoad attr TupleLoad Str Str Call AttributeLoad NameLoad attr Str body Continue orelse If BoolOpAnd CompareEq AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr BinOpAdd SubscriptLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Slice Num SubscriptLoad Call AttributeLoad NameLoad attr NameLoad Slice Num Expr Call AttributeLoad NameLoad attr NameLoad orelse If CompareNotIn AttributeLoad NameLoad attr TupleLoad Str Str Str Str Str body Print BinOpAdd Str NameLoad AttributeLoad NameLoad attr handlers ExceptHandler type AttributeLoad AttributeLoad NameLoad attr attr body Pass Return Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults Num NameLoad NameLoad NameLoad NameLoad body Expr Str Assign NameStore Num Assign NameStore ListLoad If Call NameLoad NameLoad NameLoad body Assign NameStore Call NameLoad ListLoad NameLoad Assign NameStore ListLoad AttributeLoad NameLoad attr orelse If Call NameLoad NameLoad NameLoad body Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad Call NameLoad Call AttributeLoad NameLoad attr Str orelse Raise Call NameLoad Str If NameLoad body AugAssignAdd NameStore NameLoad TryExcept body While BoolOpAnd CompareNotEq NameLoad NameLoad NameLoad body Assign NameStore ListLoad Assign NameStore Call AttributeLoad NameLoad attr Print BinOpAdd BinOpAdd Call NameLoad Call NameLoad NameLoad Str Call NameLoad Call NameLoad NameLoad Expr Call NameLoad If UnaryOpNot AttributeLoad NameLoad attr body Expr Call NameLoad If AttributeLoad AttributeLoad NameLoad attr attr body If Call AttributeLoad NameLoad attr Str AttributeLoad AttributeLoad NameLoad attr attr body AugAssignAdd NameStore Call NameLoad NameLoad AugAssignAdd NameStore Call NameLoad NameLoad orelse Print Str AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr NameLoad If NameLoad body Expr Call NameLoad NameLoad For NameStore NameLoad body If CompareNotIn AttributeLoad NameLoad attr NameLoad body Continue If BoolOpAnd NameLoad Call NameLoad NameLoad body Continue If BoolOpAnd UnaryOpNot NameLoad Call NameLoad GeneratorExp CompareEq NameLoad NameLoad comprehension NameStore BinOpAdd NameLoad Call NameLoad NameLoad body Continue If BoolOpAnd NameLoad Call NameLoad GeneratorExp Call AttributeLoad NameLoad attr NameLoad comprehension NameStore BinOpAdd NameLoad Call NameLoad NameLoad body Continue Expr Call AttributeLoad NameLoad attr NameLoad AugAssignAdd NameStore Num handlers ExceptHandler type NameLoad body Print BinOpAdd BinOpAdd Call NameLoad Call NameLoad NameLoad Str Call NameLoad Call NameLoad NameLoad Return Call NameLoad NameLoad decorator_list Assign NameStore NameLoad
N/A N/A re N/A urlparse collections deque burst.http Request RequestSet burst.utils e clear_line burst.color * N/A N/A N/A lxml.html N/A lxml.etree N/A has_lxml True N/A N/A N/A ImportError N/A N/A has_lxml False _follow_redirect N/A N/A r N/A N/A N/A N/A N/A r response N/A N/A N/A r response status N/A 301 302 N/A N/A N/A N/A N/A r follow N/A N/A N/A _get_links N/A N/A r N/A N/A N/A new_reqs N/A N/A N/A has_lxml N/A N/A N/A Exception To use the spider, you need lxml N/A N/A N/A root N/A N/A N/A lxml html fromstring N/A N/A r response content N/A base_tag N/A N/A root xpath //base N/A N/A base_tag N/A N/A N/A base_tag N/A 0 get href N/A N/A base N/A N/A N/A base_tag N/A 0 get href N/A N/A base N/A r url N/A links N/A N/A N/A N/A N/A x get href strip N/A x N/A N/A root xpath //a|//area N/A N/A x get href N/A links N/A N/A N/A N/A N/A x get src strip N/A x N/A N/A root xpath //frame N/A N/A x get src N/A l links N/A N/A N/A N/A N/A N/A l encode ascii N/A N/A N/A UnicodeEncodeError N/A N/A l N/A e N/A N/A l encode utf-8 safe / N/A url_p N/A N/A urlparse urlparse l N/A N/A N/A url_p scheme N/A http https N/A N/A N/A N/A N/A N/A new_reqs append N/A create l N/A N/A N/A N/A Invalid link: l N/A N/A N/A N/A N/A N/A url_p scheme N/A javascript mailto N/A N/A l startswith # N/A N/A N/A N/A N/A N/A N/A url_p scheme N/A url_p path N/A N/A nr N/A N/A r copy N/A n_path N/A N/A urlparse urljoin base l N/A N/A nr url N/A N/A urlparse urlunparse N/A N/A N/A N/A urlparse urlparse N/A r url N/A 2 N/A N/A N/A urlparse urlparse n_path N/A 2 N/A N/A N/A new_reqs append nr N/A N/A N/A N/A url_p scheme N/A ftp irc xmpp mms tel N/A N/A N/A UNKNOWN PROTOCOL Miam!?: l N/A url_p scheme N/A N/A N/A N/A N/A lxml etree XMLSyntaxError N/A N/A N/A N/A RequestSet new_reqs N/A spider N/A N/A init max ignore_qs post_func excluded_func hosts N/A -1 False None None None N/A N/A 
  Spider a request by following some links.

  init    - The initial request(s)
  max       - The maximum of request to execute
  post_func - A hook to be executed after each new page fetched
  hosts     - A lists of authorised hosts to spider on. By default
              only the hostname of r_init is allowed.
  excluded_func - A predicate that must indicates if a Request should
                  be executed.
   N/A nb 0 N/A checked N/A N/A N/A isinstance init Request N/A N/A q N/A deque N/A init N/A hs N/A N/A init hostname N/A N/A N/A isinstance init RequestSet N/A N/A q N/A deque init N/A hs N/A list N/A set N/A N/A init extract hostname N/A N/A N/A TypeError init must be a Request or a RequestSet N/A hosts N/A N/A hs hosts N/A N/A N/A N/A N/A nb max q N/A N/A to_add N/A N/A r N/A N/A q popleft N/A N/A N/A N/A str N/A len checked / N/A str N/A len q N/A N/A clear_line N/A N/A N/A r response N/A N/A N/A r N/A N/A N/A r response content_type N/A N/A N/A N/A re match text/html N/A N/A r response content_type N/A N/A to_add N/A _follow_redirect r N/A to_add N/A _get_links r N/A N/A 
Ignoring N/A N/A r response content_type N/A N/A N/A checked append r N/A post_func N/A N/A N/A post_func r N/A nr to_add N/A N/A N/A N/A nr hostname hs N/A N/A N/A N/A excluded_func N/A excluded_func nr N/A N/A N/A N/A N/A ignore_qs N/A any N/A N/A nr rc N/A rc N/A checked N/A list q N/A N/A N/A N/A ignore_qs N/A any N/A N/A N/A nr similar rc N/A rc N/A checked N/A list q N/A N/A N/A N/A N/A q append nr N/A nb 1 N/A N/A N/A KeyboardInterrupt N/A N/A N/A N/A N/A str N/A len checked / N/A str N/A len q N/A N/A RequestSet checked N/A N/A s spider