Module Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias Import alias TryExcept body Import alias ImportFrom alias handlers ExceptHandler type NameLoad body Print Str ClassDef bases body FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Dict Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body If CompareIsNot NameLoad NameLoad body Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Pass decorator_list FunctionDef arguments args NameParam defaults body Pass decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Return Call AttributeLoad NameLoad attr keyword ListLoad Str Str decorator_list FunctionDef arguments args NameParam NameParam defaults Num body Pass decorator_list FunctionDef arguments args NameParam NameParam defaults body Pass decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad Num body If BoolOpOr UnaryOpNot AttributeLoad NameLoad attr NameLoad body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad AttributeLoad NameLoad attr keyword Lambda arguments args NameParam defaults Call NameLoad SubscriptLoad NameLoad Index Str Assign NameStore Call NameLoad AttributeLoad NameLoad attr keyword Lambda arguments args NameParam defaults Call NameLoad Call AttributeLoad Str attr SubscriptLoad NameLoad Index Str Assign NameStore Call NameLoad AttributeLoad NameLoad attr keyword Lambda arguments args NameParam defaults Call NameLoad SubscriptLoad NameLoad Index Str Assign NameStore Call NameLoad AttributeLoad NameLoad attr keyword Lambda arguments args NameParam defaults Call NameLoad SubscriptLoad NameLoad Index Str Return DictComp NameLoad SubscriptLoad ListComp TupleLoad SubscriptLoad NameLoad Index Str SubscriptLoad NameLoad Index NameLoad comprehension NameStore NameLoad Slice NameLoad comprehension TupleStore NameStore NameStore Call NameLoad ListLoad Str Str Str Str ListLoad NameLoad NameLoad NameLoad NameLoad decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam defaults body With Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad AttributeLoad NameLoad attr Index Str Str NameStore body Return Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body With Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad AttributeLoad NameLoad attr Index Str AttributeLoad NameLoad attr NameStore body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body For NameStore NameLoad body With Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad AttributeLoad NameLoad attr Index Str Call NameLoad SubscriptLoad SubscriptLoad NameLoad Index NameLoad Index Str Str NameStore body Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults Num body Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign NameStore Dict For TupleStore NameStore NameStore Call NameLoad Call AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str body If CompareEq NameLoad NameLoad body Break If CompareIn NameLoad NameLoad body With Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad AttributeLoad NameLoad attr Index Str NameLoad NameStore body Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign NameStore Call NameLoad For NameStore Call AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str body If CompareIn NameLoad NameLoad body With Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad AttributeLoad NameLoad attr Index Str NameLoad NameStore body Expr Call AttributeLoad NameLoad attr SubscriptLoad Call AttributeLoad NameLoad attr NameLoad Index Str Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body With Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad AttributeLoad NameLoad attr Index Str AttributeLoad NameLoad attr Str NameStore body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad NameLoad NameLoad NameLoad Assign AttributeStore NameLoad attr DictComp NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str NameLoad comprehension NameStore ListLoad Str Str Str decorator_list FunctionDef arguments args NameParam defaults body Return Call AttributeLoad Call AttributeLoad Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr Str attr attr decorator_list FunctionDef arguments args NameParam defaults body Return Call AttributeLoad Call AttributeLoad Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr AttributeLoad NameLoad attr attr attr decorator_list FunctionDef arguments args NameParam NameParam defaults body For NameStore NameLoad body Assign SubscriptStore SubscriptLoad NameLoad Index NameLoad Index Str Call NameLoad NameLoad For NameStore Call NameLoad Call AttributeLoad NameLoad attr Num body If UnaryOpNot NameLoad body Return Expr Call AttributeLoad Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr ListComp NameLoad comprehension NameStore NameLoad CompareIsNot NameLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults Num body Assign NameStore Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad Str attr NameLoad NameLoad Return DictComp SubscriptLoad SubscriptLoad NameLoad Index Str Index Str SubscriptLoad NameLoad Index Str comprehension NameStore SubscriptLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr attr NameLoad attr attr Index Str BoolOpAnd CompareIn Str SubscriptLoad NameLoad Index Str CompareIn AttributeLoad NameLoad attr SubscriptLoad SubscriptLoad NameLoad Index Str Index Str decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr For NameStore NameLoad body Assign SubscriptStore SubscriptLoad NameLoad Index NameLoad Index Str NameLoad Expr Call AttributeLoad Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr ListComp SubscriptLoad NameLoad Index NameLoad comprehension NameStore NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Str Assign NameStore Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr Str attr Str attr NameLoad attr attr If CompareIn Str NameLoad body Return Call NameLoad ListComp SubscriptLoad NameLoad Index Str comprehension NameStore SubscriptLoad NameLoad Index Str CompareIn AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str orelse Print TupleLoad Str AttributeLoad NameLoad attr Return Call NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad Call AttributeLoad Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr AttributeLoad NameLoad attr attr attr If CompareNotIn Str NameLoad body Assign SubscriptStore NameLoad Index Str SubscriptLoad NameLoad Index Str Assign SubscriptStore NameLoad Index Str SubscriptLoad NameLoad Index Str Assign SubscriptStore SubscriptLoad AttributeLoad NameLoad attr Index Str Index AttributeLoad NameLoad attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad NameLoad NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr keyword Str keyword Str keyword BinOpAdd AttributeLoad NameLoad attr Str Index Str decorator_list FunctionDef arguments args NameParam defaults body Return SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr keyword Str keyword BinOpAdd AttributeLoad NameLoad attr Str Index Str decorator_list FunctionDef arguments args NameParam NameParam defaults body For NameStore NameLoad body Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad keyword SubscriptLoad NameLoad Index NameLoad keyword Str keyword BinOpAdd AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam NameParam defaults Num body Assign NameStore Dict Str Dict Str Dict Str Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad keyword Str keyword BinOpAdd AttributeLoad NameLoad attr Str Return SubscriptLoad SubscriptLoad NameLoad Index Str Index Str decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Dict Str Str Dict Str Dict Str Call AttributeLoad Str attr AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad keyword Str keyword BinOpAdd AttributeLoad NameLoad attr Str Return Call NameLoad ListComp SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Num comprehension NameStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr keyword NameLoad keyword Str keyword BinOpAdd AttributeLoad NameLoad attr Str decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam defaults body Return SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str decorator_list FunctionDef arguments args NameParam defaults body Return SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body For NameStore NameLoad body Assign SubscriptStore SubscriptLoad AttributeLoad NameLoad attr Index Str Index Call NameLoad NameLoad SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults Num body Return SubscriptLoad AttributeLoad NameLoad attr Index Str decorator_list FunctionDef arguments args NameParam defaults body Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call NameLoad Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Return Call NameLoad ListComp SubscriptLoad SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index NameLoad Index Str comprehension NameStore SubscriptLoad AttributeLoad NameLoad attr Index Str CompareIn AttributeLoad NameLoad attr SubscriptLoad SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index NameLoad Index Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign SubscriptStore SubscriptLoad AttributeLoad NameLoad attr Index Str Index AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body Import alias FunctionDef arguments args NameParam NameParam defaults body Raise Call NameLoad Str decorator_list FunctionDef arguments args NameParam defaults body Raise Call NameLoad Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Raise Call NameLoad Str decorator_list FunctionDef arguments args NameParam defaults body Raise Call NameLoad Str decorator_list FunctionDef arguments args NameParam defaults body Raise Call NameLoad Str decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Print Str Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr If NameLoad body Print Str Expr Call AttributeLoad NameLoad attr Assign NameStore ListLoad orelse Print Str Assign NameStore Call AttributeLoad NameLoad attr Assign SubscriptStore AttributeLoad NameLoad attr Index Str NameLoad Print Str Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call AttributeLoad NameLoad attr Print Str If CompareIn Str AttributeLoad NameLoad attr body Assign NameStore Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Str orelse Assign NameStore Num Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad keyword AttributeLoad NameLoad attr Print Str Expr Call AttributeLoad NameLoad attr NameLoad Print TupleLoad Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str decorator_list decorator_list ClassDef bases NameLoad NameLoad body FunctionDef arguments args NameParam NameParam defaults body With Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad AttributeLoad NameLoad attr Index Str Call NameLoad SubscriptLoad NameLoad Index Str Str NameStore body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad AttributeLoad NameLoad attr Index Str AttributeLoad NameLoad attr If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body With Call NameLoad NameLoad NameStore body Assign NameStore DictComp Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad Call AttributeLoad NameLoad attr NameLoad attr UnaryOpNot Call AttributeLoad NameLoad attr Str orelse Assign NameStore Dict Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body If NameLoad body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad AttributeLoad NameLoad attr Index Str AttributeLoad NameLoad attr With Call NameLoad NameLoad Str NameStore body Expr Call AttributeLoad NameLoad attr DictComp Call NameLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad decorator_list decorator_list ClassDef bases NameLoad NameLoad body FunctionDef arguments args NameParam NameParam defaults body Assign SubscriptStore SubscriptLoad AttributeLoad NameLoad attr Index Str Index Call NameLoad SubscriptLoad NameLoad Index Str NameLoad Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body If BoolOpOr CompareNotIn Str AttributeLoad NameLoad attr CompareNotIn AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str body Assign NameStore Call NameLoad orelse Assign NameStore SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body If NameLoad body Assign SubscriptStore SubscriptLoad AttributeLoad NameLoad attr Index Str Index AttributeLoad NameLoad attr Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad NameLoad body FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword BinOpAdd AttributeLoad NameLoad attr Str keyword Str keyword Call NameLoad SubscriptLoad NameLoad Index Str keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body TryExcept body Assign NameStore SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr keyword BinOpAdd AttributeLoad NameLoad attr Str keyword Str keyword AttributeLoad NameLoad attr Index Str Assign NameStore DictComp Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr handlers ExceptHandler body Assign NameStore Dict Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body If NameLoad body TryExcept body Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword BinOpAdd AttributeLoad NameLoad attr Str keyword Str keyword AttributeLoad NameLoad attr keyword Call AttributeLoad NameLoad attr DictComp Call NameLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr handlers ExceptHandler body Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword BinOpAdd AttributeLoad NameLoad attr Str keyword Str keyword AttributeLoad NameLoad attr keyword Call AttributeLoad NameLoad attr Dict Str DictComp Call NameLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad NameLoad body FunctionDef arguments args NameParam NameParam defaults body TryExcept body Assign SubscriptStore SubscriptLoad AttributeLoad NameLoad attr Index Str Index Call NameLoad SubscriptLoad NameLoad Index Str NameLoad handlers ExceptHandler type AttributeLoad AttributeLoad NameLoad attr attr body Print TupleLoad Str Call NameLoad SubscriptLoad NameLoad Index Str decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad Call AttributeLoad Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr AttributeLoad NameLoad attr attr attr If CompareIn Str NameLoad body Assign NameStore Dict orelse Assign NameStore DictComp Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr UnaryOpNot Call AttributeLoad NameLoad attr Str Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body If NameLoad body Assign NameStore Call AttributeLoad Call AttributeLoad Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr AttributeLoad NameLoad attr attr attr If CompareIn Str NameLoad body Assign NameStore Dict Expr Call AttributeLoad NameLoad attr DictComp Call NameLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Assign SubscriptStore SubscriptLoad AttributeLoad NameLoad attr Index Str Index AttributeLoad NameLoad attr NameLoad decorator_list decorator_list
N/A N/A json N/A os sky.scraper Scraper sky.crawler crawl sky.crawler.crawling NewsCrawler sky.helper slugify sky.helper chunker N/A requests N/A N/A N/A transaction BTrees.OOBTree OOBTree N/A N/A N/A ImportError N/A N/A Optional ZODB not possible as backend. Use `pip3 install ZODB zodbpickle` CrawlPlugin N/A N/A __init__ N/A N/A self project_name server plugin_name cache N/A None None None N/A N/A N/A self project_name project_name N/A N/A self plugin_name plugin_name N/A N/A self server server N/A N/A self cache N/A N/A self prepare_cache cache N/A N/A self crawl_config None N/A N/A self data N/A N/A N/A self documents N/A N/A N/A self template_dict None N/A prepare_cache N/A N/A self cache N/A N/A N/A N/A cache None N/A N/A N/A cache plugin_name N/A self plugin_name N/A N/A cache project_name N/A self project_name N/A N/A N/A cache setup N/A cache N/A get_default_plugin N/A N/A self N/A N/A N/A N/A get_specific_plugin N/A N/A self N/A N/A N/A N/A start_crawl N/A N/A self N/A N/A N/A N/A N/A crawl start N/A self crawl_config cache N/A self cache N/A scrape_data N/A N/A self N/A N/A N/A skindex N/A Scraper N/A self crawl_config N/A N/A N/A skindex load_local_pages N/A N/A N/A skindex add_template_elements N/A N/A N/A skindex process_all exclude_data N/A cleaned author N/A get_documents N/A N/A self maximum_number_of_documents N/A 10000 N/A N/A N/A save_bulk_data N/A N/A self data N/A N/A N/A N/A run N/A N/A self delete_existing_documents N/A False N/A N/A N/A self crawl_config N/A N/A self get_default_plugin N/A N/A N/A self start_crawl N/A N/A self data N/A N/A self scrape_data N/A N/A N/A self save_bulk_data N/A self data N/A get_bad_summary N/A N/A self force_get_documents n N/A False 5 N/A N/A N/A N/A N/A self documents force_get_documents N/A N/A N/A self documents N/A N/A self get_documents N/A title_sort N/A sorted N/A self documents key N/A N/A N/A doc N/A N/A len N/A doc N/A title N/A body_sort N/A sorted N/A self documents key N/A N/A N/A doc N/A N/A len N/A N/A join N/A doc N/A body N/A date_sort N/A sorted N/A self documents key N/A N/A N/A doc N/A N/A len N/A doc N/A publish_date N/A url_sort N/A sorted N/A self documents key N/A N/A N/A doc N/A N/A len N/A doc N/A url N/A N/A k N/A N/A N/A N/A d N/A url N/A d N/A k N/A d sorted_type N/A n N/A N/A k sorted_type N/A zip N/A title body publish_date url N/A title_sort body_sort date_sort url_sort N/A N/A CrawlFilePlugin N/A CrawlPlugin N/A get_default_plugin N/A N/A self N/A N/A N/A N/A open N/A N/A N/A os path join N/A N/A self server N/A plugins default f N/A N/A N/A N/A json load f N/A get_specific_plugin N/A N/A self N/A N/A N/A N/A open N/A N/A N/A os path join N/A N/A self server N/A plugins N/A self plugin_name f N/A N/A specific_config N/A N/A json load f N/A specific_config N/A save_bulk_data N/A N/A self data N/A N/A N/A row data N/A N/A N/A open N/A N/A N/A os path join N/A N/A self server N/A documents N/A slugify N/A N/A data N/A row N/A url w f N/A N/A N/A N/A json dump N/A data N/A row f N/A get_documents N/A N/A self maximum_number_of_documents N/A 1000000 N/A N/A slugged_url N/A slugify N/A self plugin_name N/A results N/A N/A N/A num fn N/A enumerate N/A N/A os listdir N/A N/A self server N/A documents N/A N/A N/A num maximum_number_of_documents N/A N/A N/A N/A slugged_url fn N/A N/A N/A open N/A N/A N/A os path join N/A N/A self server N/A documents fn f N/A N/A N/A results N/A slugged_url N/A N/A json load f N/A results N/A get_seen_urls N/A N/A self N/A N/A N/A slugged_url N/A slugify N/A self plugin_name N/A seen_urls N/A set N/A fn N/A N/A os listdir N/A N/A self server N/A documents N/A N/A N/A slugged_url fn N/A N/A N/A open N/A N/A N/A os path join N/A N/A self server N/A documents fn f N/A N/A N/A N/A seen_urls add N/A N/A N/A json load f N/A url N/A seen_urls N/A save_config N/A N/A self config N/A N/A N/A N/A open N/A N/A N/A os path join N/A N/A self server N/A plugins N/A self plugin_name w f N/A N/A N/A N/A json dump config f N/A N/A CrawlCloudantPlugin N/A CrawlPlugin N/A __init__ N/A N/A self project_name server plugin_name cache N/A N/A N/A N/A N/A N/A super CrawlCloudantPlugin self __init__ project_name server plugin_name cache N/A N/A self dbs N/A x N/A N/A N/A self server database N/A N/A N/A self project_name -crawler- x N/A x N/A plugins documents template_dict N/A get_default_plugin N/A N/A self N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A self dbs N/A plugins get default result json N/A get_specific_plugin N/A N/A self N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A self dbs N/A plugins get N/A self plugin_name result json N/A save_bulk_data N/A N/A self data N/A N/A N/A url_id data N/A N/A N/A N/A data N/A url_id N/A _id N/A slugify url_id N/A chunk N/A chunker N/A N/A data values 10000 N/A N/A N/A chunk N/A N/A N/A N/A N/A N/A N/A N/A N/A self dbs N/A documents bulk_docs N/A x N/A x chunk N/A x None result N/A get_documents N/A N/A self maximum_number_of_documents N/A 1000000 N/A N/A query N/A N/A query={} format N/A self plugin_name N/A params N/A N/A ?include_docs=true&limit={}&{} format maximum_number_of_documents query N/A N/A N/A N/A x N/A doc N/A _id N/A x N/A doc N/A x N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A self dbs N/A documents all_docs get params result json N/A rows N/A N/A url N/A x N/A doc N/A N/A self plugin_name N/A N/A x N/A doc N/A url N/A delete_existing_documents N/A N/A self N/A N/A N/A docs N/A N/A self get_documents N/A x docs N/A N/A N/A N/A docs N/A x N/A _deleted True N/A N/A N/A N/A N/A N/A N/A self dbs N/A documents bulk_docs N/A N/A docs N/A x N/A x docs result N/A get_seen_urls N/A N/A self N/A N/A N/A params N/A udocs N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A self dbs N/A documents design urlview view view1 get params result json N/A N/A rows udocs N/A N/A N/A set N/A N/A udoc N/A key N/A udoc N/A udocs N/A rows N/A N/A self plugin_name N/A udoc N/A key N/A N/A N/A no seen urls for plugin N/A self plugin_name N/A N/A set N/A save_config N/A N/A self config N/A N/A N/A doc N/A N/A N/A N/A N/A N/A N/A N/A self dbs N/A plugins get N/A self plugin_name result json N/A N/A error doc N/A N/A N/A config N/A _id N/A doc N/A _id N/A N/A config N/A _rev N/A doc N/A _rev N/A N/A N/A N/A self dbs N/A plugins N/A N/A self plugin_name config N/A N/A CrawlElasticSearchPlugin N/A CrawlPlugin N/A __init__ N/A N/A self project_name server plugin_name cache N/A N/A N/A N/A N/A N/A super CrawlElasticSearchPlugin self __init__ project_name server plugin_name cache N/A N/A self es server N/A get_default_plugin N/A N/A self N/A N/A N/A N/A N/A N/A N/A self es get id default doc_type plugin index N/A N/A self project_name -crawler-plugins N/A _source N/A get_specific_plugin N/A N/A self N/A N/A N/A N/A N/A N/A N/A self es get id N/A self plugin_name doc_type plugin index N/A N/A self project_name -crawler-plugins N/A _source N/A save_bulk_data N/A N/A self data N/A N/A N/A url_id data N/A N/A doc_id N/A slugify url_id N/A N/A N/A N/A self es index id doc_id body N/A data N/A url_id doc_type document index N/A N/A self project_name -crawler-documents N/A get_documents N/A N/A self maximum_number_of_documents N/A 1000000 N/A N/A query N/A query N/A wildcard N/A url N/A N/A *{}* format N/A self plugin_name N/A res N/A N/A N/A self es search body query doc_type document index N/A N/A self project_name -crawler-documents N/A N/A N/A res N/A hits N/A hits N/A get_seen_urls N/A N/A self N/A N/A N/A query N/A query fields N/A wildcard N/A url N/A N/A *{}* format N/A self plugin_name url N/A res N/A N/A N/A self es search body query doc_type document index N/A N/A self project_name -crawler-documents N/A N/A set N/A N/A N/A N/A x N/A fields N/A url N/A 0 N/A x N/A N/A res N/A hits N/A hits N/A save_config N/A N/A self config N/A N/A N/A N/A N/A N/A self es index id N/A self plugin_name body config doc_type plugin index N/A N/A self project_name -crawler-plugins N/A N/A CrawlZODBPlugin N/A CrawlPlugin N/A get_default_plugin N/A N/A self N/A N/A N/A N/A N/A N/A self server N/A plugins N/A default N/A get_specific_plugin N/A N/A self N/A N/A N/A N/A N/A N/A self server N/A plugins N/A N/A self plugin_name N/A save_bulk_data N/A N/A self data N/A N/A N/A url_id data N/A N/A N/A N/A N/A self server N/A documents N/A N/A slugify url_id N/A data N/A url_id N/A N/A N/A transaction commit N/A get_documents N/A N/A self maximum_number_of_documents N/A 1000000 N/A N/A N/A N/A self server N/A documents N/A delete_existing_documents N/A N/A self N/A N/A N/A N/A N/A self server N/A documents N/A OOBTree N/A N/A N/A transaction commit N/A get_seen_urls N/A N/A self N/A N/A N/A N/A set N/A N/A N/A N/A N/A self server N/A documents N/A s N/A url N/A s N/A N/A self server N/A documents N/A N/A self plugin_name N/A N/A N/A N/A self server N/A documents N/A s N/A url N/A save_config N/A N/A self config N/A N/A N/A N/A N/A N/A self server N/A plugins N/A N/A self plugin_name config N/A N/A N/A transaction commit N/A N/A CrawlPluginNews N/A CrawlPlugin N/A N/A ast save_data N/A N/A self data N/A N/A N/A N/A NotImplementedError save_data required N/A get_template_dict N/A N/A self N/A N/A N/A N/A NotImplementedError get_template_dict required N/A save_template_dict N/A N/A self templated_dict N/A N/A N/A N/A NotImplementedError save_template_dict required N/A get_seen_urls N/A N/A self N/A N/A N/A N/A NotImplementedError seen_urls required N/A delete_existing_documents N/A N/A self N/A N/A N/A N/A NotImplementedError delete_existing_documents required N/A run N/A N/A self delete_existing_documents N/A False N/A N/A getting crawl plugin info N/A N/A self crawl_config N/A N/A self get_default_plugin N/A N/A N/A N/A self crawl_config update N/A N/A self get_specific_plugin N/A delete_existing_documents N/A N/A deleting documents before crawling N/A N/A N/A self delete_existing_documents N/A seen_urls N/A N/A N/A getting seen urls N/A seen_urls N/A N/A self get_seen_urls N/A N/A N/A self crawl_config N/A seen_urls seen_urls N/A getting template dict N/A N/A N/A self crawl_config N/A template_dict N/A N/A self get_template_dict N/A starting crawl N/A N/A logging_level N/A self crawl_config N/A N/A logging_level N/A int N/A N/A self crawl_config N/A logging_level N/A N/A logging_level 3 N/A templated_dict N/A N/A crawl start N/A self crawl_config NewsCrawler N/A self save_data N/A self save_bulk_data logging_level logging_level cache N/A self cache N/A saving template.. N/A N/A N/A self save_template_dict templated_dict N/A N/A crawling/scraping N/A self project_name N/A self plugin_name DONE N/A N/A CrawlFilePluginNews N/A CrawlFilePlugin CrawlPluginNews N/A save_data N/A N/A self data N/A N/A N/A N/A open N/A N/A N/A os path join N/A N/A self server N/A documents N/A slugify N/A data N/A url w f N/A N/A N/A N/A json dump data f N/A get_template_dict N/A N/A self N/A N/A N/A temp_dict_fn N/A N/A N/A os path join N/A N/A self server N/A template_dict N/A self plugin_name N/A N/A N/A N/A os path isfile temp_dict_fn N/A N/A N/A open temp_dict_fn f N/A N/A template_dict N/A N/A N/A N/A self ast literal_eval k v N/A N/A k v N/A N/A N/A N/A json load f items N/A N/A N/A k startswith _ N/A N/A template_dict N/A N/A template_dict N/A save_template_dict N/A N/A self templated_dict N/A N/A N/A templated_dict N/A N/A temp_dict_fn N/A N/A N/A os path join N/A N/A self server N/A template_dict N/A self plugin_name N/A N/A open temp_dict_fn w f N/A N/A N/A N/A json dump N/A N/A repr k v N/A N/A k v N/A N/A templated_dict items f N/A N/A CrawlZODBPluginNews N/A CrawlZODBPlugin CrawlPluginNews N/A save_data N/A N/A self data N/A N/A N/A N/A N/A N/A self server N/A documents N/A N/A slugify N/A data N/A url data N/A N/A N/A transaction commit N/A get_template_dict N/A N/A self N/A N/A N/A N/A N/A template_dict N/A self server N/A N/A self plugin_name N/A N/A self server N/A template_dict N/A N/A template_dict N/A OOBTree N/A N/A template_dict N/A N/A N/A self server N/A template_dict N/A N/A self plugin_name N/A template_dict N/A save_template_dict N/A N/A self templated_dict N/A N/A N/A templated_dict N/A N/A N/A N/A N/A self server N/A template_dict N/A N/A self plugin_name N/A OOBTree templated_dict N/A N/A N/A transaction commit N/A N/A CrawlElasticSearchPluginNews N/A CrawlElasticSearchPlugin CrawlPluginNews N/A save_data N/A N/A self data N/A N/A N/A N/A N/A N/A self es index index N/A N/A self project_name -crawler-documents doc_type document id N/A slugify N/A data N/A url body data N/A get_template_dict N/A N/A self N/A N/A N/A N/A N/A templ_dict N/A N/A N/A N/A self es get index N/A N/A self project_name -crawler-template_dict doc_type template_dict id N/A self plugin_name N/A _source N/A templ_dict N/A N/A N/A N/A self ast literal_eval k v N/A N/A k v N/A N/A templ_dict items N/A N/A N/A N/A templ_dict N/A N/A templ_dict N/A save_template_dict N/A N/A self templ_dict N/A N/A N/A templ_dict N/A N/A N/A N/A N/A N/A N/A self es index index N/A N/A self project_name -crawler-template_dict doc_type template_dict id N/A self plugin_name body N/A N/A json dumps N/A N/A repr k v N/A N/A k v N/A N/A templ_dict items N/A N/A N/A N/A N/A N/A N/A self es update index N/A N/A self project_name -crawler-template_dict doc_type template_dict id N/A self plugin_name body N/A N/A json dumps N/A doc N/A N/A repr k v N/A N/A k v N/A N/A templ_dict items N/A N/A CrawlCloudantPluginNews N/A CrawlCloudantPlugin CrawlPluginNews N/A save_data N/A N/A self data N/A N/A N/A N/A N/A N/A N/A N/A self dbs N/A documents N/A N/A slugify N/A data N/A url data N/A N/A N/A N/A N/A requests exceptions HTTPError N/A N/A N/A conflict error N/A slugify N/A data N/A url N/A get_template_dict N/A N/A self N/A N/A N/A template_dict N/A N/A N/A N/A N/A N/A N/A N/A self dbs N/A template_dict get N/A self plugin_name result json N/A N/A error template_dict N/A N/A template_dict N/A N/A N/A template_dict N/A N/A N/A N/A self ast literal_eval k v N/A N/A k v N/A N/A template_dict items N/A N/A N/A k startswith _ N/A template_dict N/A save_template_dict N/A N/A self templated_dict N/A N/A N/A templated_dict N/A N/A doc N/A N/A N/A N/A N/A N/A N/A N/A self dbs N/A template_dict get N/A self plugin_name result json N/A N/A error doc N/A N/A doc N/A N/A N/A N/A doc update N/A N/A repr k v N/A N/A k v N/A N/A templated_dict items N/A N/A N/A N/A self dbs N/A template_dict N/A N/A self plugin_name doc N/A N/A