Module Import alias ImportFrom alias ImportFrom alias alias identifier alias identifier ImportFrom alias ImportFrom alias identifier ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore Call NameLoad Assign AttributeStore NameLoad attr Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Num Assign NameStore ListLoad Str Str Str Assign NameStore Call AttributeLoad NameLoad attr Str ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Call NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Call NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr decorator_list NameLoad decorator_list ClassDef bases NameLoad body Assign NameStore NameLoad Assign NameStore NameLoad decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr Dict decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call NameLoad NameLoad Str Assign SubscriptStore AttributeLoad NameLoad attr Index NameLoad Call NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call NameLoad For TupleStore NameStore NameStore Call AttributeLoad AttributeLoad NameLoad attr attr body If CompareLt BinOpSub NameLoad NameLoad Num body Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list decorator_list FunctionDef arguments args NameParam defaults body Return ListComp Call NameLoad AttributeLoad NameLoad attr NameLoad comprehension NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list Expr Str If CompareEq NameLoad Str body Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr ListComp Call NameLoad keyword Call AttributeLoad NameLoad attr comprehension NameStore NameLoad While NameLoad body FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad NameLoad attr NameLoad Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults kwarg body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore ListComp Call NameLoad keyword NameLoad comprehension NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad decorator_list Assign NameStore Dict Str Str Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Assign NameStore Call AttributeLoad NameLoad attr keyword SubscriptLoad NameLoad Index Str keyword SubscriptLoad NameLoad Index Str If UnaryOpNot NameLoad body Continue For NameStore NameLoad body Assign SubscriptStore AttributeLoad NameLoad attr Index Str Dict Str NameLoad Expr Call NameLoad NameLoad keyword Num
N/A N/A re time time grequests AsyncRequest get grequests_get map grequests_map urlparse urljoin frontera.core.models Request FrontierRequest frontera.utils.converters BaseRequestConverter frontera.contrib.requests.converters ResponseConverter frontera.utils.managers FrontierManagerWrapper frontera.core get_slot_key frontera Settings N/A SETTINGS N/A Settings N/A N/A SETTINGS BACKEND frontera.contrib.backends.memory.MemoryRandomOverusedBackend N/A N/A SETTINGS LOGGING_MANAGER_ENABLED True N/A N/A SETTINGS LOGGING_BACKEND_ENABLED False N/A N/A SETTINGS MAX_REQUESTS 0 N/A N/A SETTINGS MAX_NEXT_REQUESTS 40 N/A SEEDS N/A http://www.imdb.com http://www.bbc.com/ http://www.amazon.com/ N/A LINK_RE N/A N/A re compile href="(.*?)" GRequestsConverter N/A BaseRequestConverter N/A N/A Converts between frontera and grequests request objects to_frontier N/A N/A cls request N/A N/A N/A request: AsyncRequest > Frontier N/A N/A FrontierRequest url N/A request url method N/A request method N/A classmethod from_frontier N/A N/A cls request N/A N/A N/A request: Frontier > AsyncRequest N/A N/A AsyncRequest method N/A request method url N/A request url N/A classmethod N/A GRequestsFrontierManager N/A FrontierManagerWrapper N/A N/A request_converter_class GRequestsConverter N/A response_converter_class ResponseConverter N/A HostnameStatistics N/A object N/A __init__ N/A N/A self N/A N/A N/A N/A self stats N/A N/A on_request N/A N/A self request N/A N/A N/A key N/A get_slot_key request domain N/A N/A N/A self stats N/A key N/A time N/A collect_overused_keys N/A N/A self overused N/A N/A N/A ts N/A time N/A N/A key timestamp N/A N/A N/A self stats iteritems N/A N/A N/A N/A ts timestamp 5.0 N/A N/A N/A N/A overused append key N/A overused N/A N/A extract_page_links N/A N/A response N/A N/A N/A N/A N/A urljoin N/A response url link N/A link N/A N/A LINK_RE findall N/A response text N/A N/A 
The idea is to send requests to each domain with at least 5 seconds of delay. grequests only allows us to limit the
number of simultaneous requests. So, we basically performing checks every frontier iteration and limiting the contents
of new frontier batch by sending overused keys in `info` argument to get_next_requests. Therefore, we're getting to 5
seconds delays per batch.
 N/A N/A __name__ __main__ N/A N/A frontier N/A GRequestsFrontierManager SETTINGS N/A stats N/A HostnameStatistics N/A N/A N/A frontier add_seeds N/A N/A grequests_get url N/A N/A url strip N/A url SEEDS N/A True N/A error_handler N/A N/A request exception N/A N/A N/A N/A N/A frontier request_error request N/A str exception N/A callback N/A N/A response N/A kwargs N/A N/A N/A N/A stats on_request N/A response request N/A links N/A N/A grequests_get url url N/A url N/A extract_page_links response N/A N/A N/A frontier page_crawled response response links links N/A N/A dl_info N/A type domain N/A N/A N/A stats collect_overused_keys N/A dl_info N/A overused_keys N/A next_requests N/A N/A frontier get_next_requests key_type N/A dl_info N/A type overused_keys N/A dl_info N/A overused_keys N/A N/A next_requests N/A N/A N/A r next_requests N/A N/A N/A N/A r kwargs N/A hooks N/A response callback N/A N/A grequests_map next_requests size 10