Module Expr Str Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias If NameLoad body ImportFrom alias alias ImportFrom alias orelse ImportFrom alias alias ImportFrom alias FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Return NameLoad decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Str BinOpBitOr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign NameStore Call NameLoad ListLoad Str Str Str Str Str Str Assign NameStore Call NameLoad ListLoad Str Str FunctionDef arguments args NameParam NameParam defaults body FunctionDef arguments args NameParam defaults body Return IfExp CompareIs NameLoad NameLoad Str NameLoad decorator_list Return Call NameLoad BoolOpOr Call NameLoad Call AttributeLoad NameLoad attr Num Call NameLoad Call AttributeLoad NameLoad attr Num Call NameLoad Call AttributeLoad NameLoad attr Num decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str For NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Assign TupleStore NameStore NameStore TupleLoad Call AttributeLoad NameLoad attr Num Call AttributeLoad NameLoad attr Num If CompareNotIn NameLoad AttributeLoad NameLoad attr body Continue Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad If Call NameLoad GeneratorExp Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad comprehension NameStore AttributeLoad NameLoad attr body Continue Expr Yield NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str For NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Yield Call AttributeLoad NameLoad attr NameLoad decorator_list NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return TupleLoad Call NameLoad Lambda arguments args NameParam defaults UnaryOpNot Call NameLoad NameLoad NameLoad Call NameLoad Lambda arguments args NameParam defaults Call NameLoad NameLoad NameLoad decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Call NameLoad FunctionDef arguments args NameParam defaults body Expr Str Assign AttributeStore NameLoad attr Call NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr NameLoad keyword Num Return TupleLoad Call NameLoad Call NameLoad Assign TupleStore NameStore NameStore Call NameLoad ListComp Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr NameLoad comprehension NameStore NameLoad AttributeLoad AttributeLoad NameLoad attr attr Return TupleLoad Call NameLoad Call NameLoad AttributeLoad NameLoad attr NameLoad Call NameLoad Call NameLoad AttributeLoad NameLoad attr NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type AttributeLoad NameLoad attr name NameStore body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr NameLoad keyword Num Return TupleLoad Call NameLoad Call NameLoad Assign NameStore Call NameLoad GeneratorExp Call AttributeLoad NameLoad attr NameLoad comprehension NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad GeneratorExp Call AttributeLoad NameLoad attr NameLoad comprehension NameStore Call AttributeLoad NameLoad attr NameLoad Return TupleLoad NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body If AttributeLoad NameLoad attr body Return Call AttributeLoad NameLoad attr NameLoad orelse If AttributeLoad NameLoad attr body Return Call AttributeLoad NameLoad attr NameLoad NameLoad orelse Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return TupleLoad Call NameLoad Call NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad Num body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr BoolOpOr NameLoad Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Return BinOpAdd TupleLoad NameLoad Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad If CompareIs NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Assign TupleStore NameStore NameStore TupleLoad Call NameLoad Call NameLoad Assign NameStore Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr FunctionDef arguments args defaults body While UnaryOpNot Call AttributeLoad NameLoad attr body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr keyword Num handlers ExceptHandler type NameLoad body Continue If CompareNotIn NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad TryExcept body Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad For NameStore Call AttributeLoad Call AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Continue Expr Call AttributeLoad NameLoad attr NameLoad If NameLoad body For NameStore NameLoad body If CompareNotIn NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr decorator_list For TupleStore NameStore NameStore Call NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad NameLoad keyword Num Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore ListLoad For NameStore Call NameLoad AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Return NameLoad decorator_list decorator_list
N/A N/A Support for webpage parsing and crawling. N/A os N/A re N/A threading N/A traceback compatibility PY3 http Context link Link tracer TRACER util Memoizer N/A PY3 N/A queue Empty Queue urllib.parse urlparse N/A Queue Empty Queue urlparse urlparse unescape N/A N/A s N/A N/A N/A Unescapes html. Taken from https://wiki.python.org/moin/EscapingHtml N/A s N/A N/A s replace &lt; < N/A s N/A N/A s replace &gt; > N/A s N/A N/A s replace &amp; & N/A s N/A PageParser N/A object N/A N/A A helper class to extract and differentiate ordinary and download links from webpages. N/A HREF_RE N/A N/A re compile href=(?:"([^"]*)"|\'([^\']*)\'|([^>\s\n]*)) N/A N/A re I N/A re S N/A REL_RE N/A N/A re compile <[^>]*\srel\s*=\s*['"]?([^'">]+)[^>]*> N/A re I N/A REL_SKIP_EXTENSIONS N/A frozenset N/A .zip .tar .tar.gz .tar.bz2 .tgz .exe N/A REL_TYPES N/A frozenset N/A homepage download href_match_to_url N/A N/A cls match N/A N/A pick N/A N/A group N/A N/A N/A N/A N/A group None group N/A N/A N/A unescape N/A N/A pick N/A N/A match group 1 N/A pick N/A N/A match group 2 N/A pick N/A N/A match group 3 N/A classmethod rel_links N/A N/A cls page N/A N/A N/A return rel= links that should be scraped, skipping obviously data links. N/A match N/A N/A N/A cls REL_RE finditer page N/A N/A N/A href rel N/A N/A N/A match group 0 N/A N/A match group 1 N/A N/A rel N/A cls REL_TYPES N/A N/A N/A href_match N/A N/A N/A cls HREF_RE search href N/A href_match N/A N/A href N/A N/A cls href_match_to_url href_match N/A parsed_href N/A urlparse href N/A N/A any N/A N/A N/A N/A parsed_href path endswith ext N/A ext N/A cls REL_SKIP_EXTENSIONS N/A N/A N/A N/A href N/A classmethod links N/A N/A cls page N/A N/A N/A return all links on a page, including potentially rel= links. N/A match N/A N/A N/A cls HREF_RE finditer page N/A N/A N/A N/A N/A cls href_match_to_url match N/A classmethod N/A partition N/A N/A L pred N/A N/A N/A N/A N/A filter N/A N/A N/A v N/A N/A N/A pred v L N/A filter N/A N/A N/A v N/A N/A pred v L N/A Crawler N/A object N/A N/A A multi-threaded crawler that supports local (disk) and remote (web) crawling. N/A _CRAWL_CACHE N/A Memoizer reset_cache N/A N/A cls N/A N/A N/A Reset the internal crawl cache. This is intended primarily for tests. N/A N/A cls _CRAWL_CACHE N/A Memoizer N/A classmethod crawl_local N/A N/A cls link N/A N/A N/A N/A N/A dirents N/A N/A os listdir N/A link local_path N/A N/A N/A OSError N/A e N/A N/A N/A N/A TRACER log N/A Failed to read %s: %s N/A N/A link local_path e V 1 N/A N/A N/A set N/A set N/A N/A files dirs N/A partition N/A N/A N/A N/A os path join N/A link local_path fn N/A fn dirents N/A N/A os path isdir N/A N/A N/A set N/A map N/A Link from_filename files N/A set N/A map N/A Link from_filename dirs N/A classmethod crawl_remote N/A N/A cls context link N/A N/A N/A N/A N/A content N/A N/A context content link N/A N/A N/A N/A context Error N/A e N/A N/A N/A N/A TRACER log N/A Failed to read %s: %s N/A N/A link url e V 1 N/A N/A N/A set N/A set N/A links N/A set N/A N/A N/A link join href N/A href N/A N/A PageParser links content N/A rel_links N/A set N/A N/A N/A link join href N/A href N/A N/A PageParser rel_links content N/A N/A links rel_links N/A classmethod crawl_link N/A N/A cls context link N/A N/A N/A N/A link local N/A N/A N/A N/A cls crawl_local link N/A N/A N/A link remote N/A N/A N/A N/A cls crawl_remote context link N/A N/A N/A N/A TRACER log N/A Failed to crawl %s: unknown scheme %s N/A N/A link url N/A link scheme N/A N/A N/A set N/A set N/A classmethod __init__ N/A N/A self context threads N/A None 1 N/A N/A N/A self _threads threads N/A N/A self context N/A context N/A N/A Context get N/A _make_cache_key N/A N/A self links follow_links N/A N/A N/A N/A N/A follow_links N/A tuple links N/A crawl N/A N/A self link_or_links follow_links N/A False N/A N/A links N/A list N/A N/A Link wrap_iterable link_or_links N/A cache_key N/A N/A self _make_cache_key links follow_links N/A result N/A N/A N/A self _CRAWL_CACHE get cache_key N/A N/A result None N/A N/A result N/A N/A self _crawl links follow_links N/A N/A N/A N/A self _CRAWL_CACHE store cache_key result N/A result N/A _crawl N/A N/A self link_or_links follow_links N/A N/A N/A N/A links seen N/A N/A set N/A set N/A queue N/A Queue N/A converged N/A N/A threading Event execute N/A N/A N/A N/A N/A N/A N/A N/A converged is_set N/A N/A N/A N/A link N/A N/A queue get timeout 0.01 N/A N/A N/A Empty N/A N/A N/A N/A link seen N/A N/A N/A N/A seen add link N/A N/A N/A N/A roots rels N/A N/A self crawl_link N/A self context link N/A N/A N/A Exception N/A e N/A N/A N/A N/A TRACER log N/A Unknown exception encountered: %s e N/A line N/A N/A N/A N/A traceback format_exc splitlines N/A N/A N/A N/A TRACER log line N/A N/A N/A queue task_done N/A N/A N/A N/A links update roots N/A follow_links N/A N/A rel rels N/A N/A N/A rel seen N/A N/A N/A N/A queue put rel N/A N/A N/A queue task_done N/A N/A N/A i link N/A enumerate link_or_links N/A N/A N/A N/A TRACER log N/A crawling link i=%s link=%s follow_links=%s N/A i link follow_links V 3 N/A N/A N/A queue put link N/A workers N/A N/A _ N/A range N/A self _threads N/A N/A worker N/A N/A threading Thread target execute N/A N/A N/A workers append worker N/A N/A worker daemon True N/A N/A N/A worker start N/A N/A N/A queue join N/A N/A N/A converged set N/A links N/A N/A