Module ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias alias alias ImportFrom alias ImportFrom alias ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr Call NameLoad NameLoad keyword NameLoad Assign AttributeStore NameLoad attr Call NameLoad ListComp TupleLoad NameLoad Call NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr If NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr For TupleStore NameStore NameStore Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str Call AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str Call AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Return Call NameLoad AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list NameLoad decorator_list ClassDef bases NameLoad body Assign NameStore Str FunctionDef arguments args NameParam NameParam defaults body Return Call NameLoad AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str Call AttributeLoad NameLoad attr Str keyword Str decorator_list decorator_list ClassDef bases NameLoad body Assign NameStore Str FunctionDef arguments args NameParam NameParam defaults body Return Call NameLoad AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str Call AttributeLoad NameLoad attr Str keyword Str decorator_list decorator_list ClassDef bases NameLoad body Assign NameStore Str FunctionDef arguments args NameParam NameParam defaults body Return Call NameLoad AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Return UnaryOpUSub SubscriptLoad AttributeLoad NameLoad attr Index Str decorator_list decorator_list ClassDef bases NameLoad body Assign NameStore Str FunctionDef arguments args NameParam NameParam defaults body Return Call NameLoad AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Return SubscriptLoad AttributeLoad NameLoad attr Index Str decorator_list decorator_list Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr Call NameLoad NameLoad keyword NameLoad Assign AttributeStore NameLoad attr Call NameLoad ListComp TupleLoad NameLoad Call NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call NameLoad NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index Str If NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr Str Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call NameLoad NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index Str Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index Str If NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr Str Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body Return AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body For NameStore ListLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr body If NameLoad body Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body For NameStore ListLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr body If NameLoad body Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults kwarg body Assign NameStore Call AttributeLoad NameLoad attr Str ListLoad Num Assign NameStore ListLoad For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return NameLoad decorator_list decorator_list
N/A __future__ absolute_import sqlalchemy create_engine sqlalchemy.orm sessionmaker frontera.core.components DistributedBackend frontera.contrib.backends CommonBackend frontera.contrib.backends.sqlalchemy.components Metadata Queue States frontera.contrib.backends.sqlalchemy.models DeclarativeBase frontera.utils.misc load_object SQLAlchemyBackend N/A CommonBackend N/A __init__ N/A N/A self manager N/A N/A N/A N/A self manager manager N/A settings N/A manager settings N/A engine N/A N/A settings get SQLALCHEMYBACKEND_ENGINE N/A engine_echo N/A N/A settings get SQLALCHEMYBACKEND_ENGINE_ECHO N/A drop_all_tables N/A N/A settings get SQLALCHEMYBACKEND_DROP_ALL_TABLES N/A clear_content N/A N/A settings get SQLALCHEMYBACKEND_CLEAR_CONTENT N/A models N/A N/A settings get SQLALCHEMYBACKEND_MODELS N/A N/A self engine N/A create_engine engine echo engine_echo N/A N/A self models N/A dict N/A N/A name N/A load_object klass N/A N/A name klass N/A N/A models items N/A drop_all_tables N/A N/A N/A N/A N/A DeclarativeBase metadata drop_all N/A self engine N/A N/A N/A N/A DeclarativeBase metadata create_all N/A self engine N/A N/A self session_cls N/A sessionmaker N/A N/A N/A N/A self session_cls configure bind N/A self engine N/A clear_content N/A N/A session N/A N/A self session_cls N/A N/A name table N/A N/A N/A N/A DeclarativeBase metadata tables items N/A N/A N/A N/A session execute N/A N/A table delete N/A N/A N/A session close N/A N/A self _metadata N/A Metadata N/A self session_cls N/A N/A self models N/A MetadataModel N/A N/A settings get SQLALCHEMYBACKEND_CACHE_SIZE N/A N/A self _states N/A States N/A self session_cls N/A N/A self models N/A StateModel N/A N/A settings get STATE_CACHE_SIZE_LIMIT N/A N/A self _queue N/A N/A self _create_queue settings N/A frontier_stop N/A N/A self N/A N/A N/A N/A N/A N/A super SQLAlchemyBackend self frontier_stop N/A N/A N/A N/A self engine dispose N/A _create_queue N/A N/A self settings N/A N/A N/A N/A Queue N/A self session_cls N/A N/A self models N/A QueueModel N/A N/A settings get SPIDER_FEED_PARTITIONS N/A queue N/A N/A self N/A N/A N/A N/A self _queue N/A property metadata N/A N/A self N/A N/A N/A N/A self _metadata N/A property states N/A N/A self N/A N/A N/A N/A self _states N/A property N/A FIFOBackend N/A SQLAlchemyBackend N/A N/A component_name SQLAlchemy FIFO Backend _create_queue N/A N/A self settings N/A N/A N/A N/A Queue N/A self session_cls N/A N/A self models N/A QueueModel N/A N/A settings get SPIDER_FEED_PARTITIONS ordering created N/A N/A LIFOBackend N/A SQLAlchemyBackend N/A N/A component_name SQLAlchemy LIFO Backend _create_queue N/A N/A self settings N/A N/A N/A N/A Queue N/A self session_cls N/A N/A self models N/A QueueModel N/A N/A settings get SPIDER_FEED_PARTITIONS ordering created_desc N/A N/A DFSBackend N/A SQLAlchemyBackend N/A N/A component_name SQLAlchemy DFS Backend _create_queue N/A N/A self settings N/A N/A N/A N/A Queue N/A self session_cls N/A N/A self models N/A QueueModel N/A N/A settings get SPIDER_FEED_PARTITIONS N/A _get_score N/A N/A self obj N/A N/A N/A N/A N/A N/A obj meta N/A depth N/A N/A BFSBackend N/A SQLAlchemyBackend N/A N/A component_name SQLAlchemy BFS Backend _create_queue N/A N/A self settings N/A N/A N/A N/A Queue N/A self session_cls N/A N/A self models N/A QueueModel N/A N/A settings get SPIDER_FEED_PARTITIONS N/A _get_score N/A N/A self obj N/A N/A N/A N/A N/A obj meta N/A depth N/A N/A N/A BASE CommonBackend N/A LIFO LIFOBackend N/A FIFO FIFOBackend N/A DFS DFSBackend N/A BFS BFSBackend Distributed N/A DistributedBackend N/A __init__ N/A N/A self manager N/A N/A N/A N/A self manager manager N/A settings N/A manager settings N/A engine N/A N/A settings get SQLALCHEMYBACKEND_ENGINE N/A engine_echo N/A N/A settings get SQLALCHEMYBACKEND_ENGINE_ECHO N/A models N/A N/A settings get SQLALCHEMYBACKEND_MODELS N/A N/A self engine N/A create_engine engine echo engine_echo N/A N/A self models N/A dict N/A N/A name N/A load_object klass N/A N/A name klass N/A N/A models items N/A N/A self session_cls N/A sessionmaker N/A N/A N/A N/A self session_cls configure bind N/A self engine N/A N/A self _metadata None N/A N/A self _queue None N/A N/A self _states None N/A strategy_worker N/A N/A cls manager N/A N/A N/A b N/A cls manager N/A settings N/A manager settings N/A drop_all_tables N/A N/A settings get SQLALCHEMYBACKEND_DROP_ALL_TABLES N/A clear_content N/A N/A settings get SQLALCHEMYBACKEND_CLEAR_CONTENT N/A model N/A N/A b models N/A StateModel N/A drop_all_tables N/A N/A N/A N/A N/A model __table__ drop bind N/A b engine N/A N/A N/A N/A model __table__ create bind N/A b engine N/A clear_content N/A N/A session N/A N/A b session_cls N/A N/A N/A session execute N/A N/A N/A model __table__ delete N/A N/A N/A session close N/A N/A b _states N/A States N/A b session_cls model N/A N/A settings get STATE_CACHE_SIZE_LIMIT N/A b N/A classmethod db_worker N/A N/A cls manager N/A N/A N/A b N/A cls manager N/A settings N/A manager settings N/A drop N/A N/A settings get SQLALCHEMYBACKEND_DROP_ALL_TABLES N/A clear_content N/A N/A settings get SQLALCHEMYBACKEND_CLEAR_CONTENT N/A metadata_m N/A N/A b models N/A MetadataModel N/A queue_m N/A N/A b models N/A QueueModel N/A drop N/A N/A N/A N/A N/A metadata_m __table__ drop bind N/A b engine N/A N/A N/A N/A queue_m __table__ drop bind N/A b engine N/A N/A N/A N/A metadata_m __table__ create bind N/A b engine N/A N/A N/A N/A queue_m __table__ create bind N/A b engine N/A clear_content N/A N/A session N/A N/A b session_cls N/A N/A N/A session execute N/A N/A N/A metadata_m __table__ delete N/A N/A N/A session execute N/A N/A N/A queue_m __table__ delete N/A N/A N/A session close N/A N/A b _metadata N/A Metadata N/A b session_cls metadata_m N/A N/A settings get SQLALCHEMYBACKEND_CACHE_SIZE N/A N/A b _queue N/A Queue N/A b session_cls queue_m N/A N/A settings get SPIDER_FEED_PARTITIONS N/A b N/A classmethod queue N/A N/A self N/A N/A N/A N/A self _queue N/A property metadata N/A N/A self N/A N/A N/A N/A self _metadata N/A property states N/A N/A self N/A N/A N/A N/A self _states N/A property frontier_start N/A N/A self N/A N/A N/A component N/A N/A self metadata N/A self queue N/A self states N/A N/A component N/A N/A N/A N/A component frontier_start N/A frontier_stop N/A N/A self N/A N/A N/A component N/A N/A self metadata N/A self queue N/A self states N/A N/A component N/A N/A N/A N/A component frontier_stop N/A add_seeds N/A N/A self seeds N/A N/A N/A N/A N/A N/A self metadata add_seeds seeds N/A get_next_requests N/A N/A self max_next_requests N/A kwargs N/A N/A partitions N/A N/A kwargs pop partitions N/A 0 N/A batch N/A N/A partition_id partitions N/A N/A N/A N/A batch extend N/A N/A N/A self queue get_next_requests max_next_requests partition_id kwargs N/A batch N/A page_crawled N/A N/A self response links N/A N/A N/A N/A N/A N/A self metadata page_crawled response links N/A request_error N/A N/A self request error N/A N/A N/A N/A N/A N/A self metadata request_error request error N/A finished N/A N/A self N/A N/A N/A NotImplementedError N/A N/A