Module Import alias Import alias ImportFrom alias ImportFrom alias alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad FunctionDef arguments args defaults vararg kwarg body FunctionDef arguments args NameParam defaults body FunctionDef arguments args defaults vararg kwarg body TryExcept body Return Call NameLoad NameLoad NameLoad handlers ExceptHandler type Call AttributeLoad NameLoad attr Str NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Raise Call AttributeLoad NameLoad attr keyword NameLoad decorator_list Call AttributeLoad NameLoad attr NameLoad NameLoad Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Assign NameStore BoolOpOr NameLoad BinOpSub Call AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr Assign NameStore BoolOpOr NameLoad Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Call NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad BinOpBitOr NameLoad NameLoad attr decorator_list AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad body Assign NameStore Call NameLoad Assign NameStore SubscriptLoad NameLoad Index NameLoad Assign NameStore BoolOpOr NameLoad BinOpSub Call AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr Assign NameStore BoolOpOr NameLoad Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Return TupleLoad NameLoad Dict Str Str Str Call NameLoad NameLoad NameLoad decorator_list AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign TupleStore NameStore NameStore NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr Call NameLoad NameLoad NameLoad For NameStore NameLoad body Expr Call NameLoad NameLoad NameLoad NameLoad decorator_list AttributeLoad NameLoad attr FunctionDef arguments args NameParam NameParam NameParam defaults body Assign SubscriptStore NameLoad Index Str NameLoad Assign SubscriptStore SubscriptLoad NameLoad Index Str Index Str Call NameLoad Assign NameStore BinOpBitOr Call AttributeLoad NameLoad attr NameLoad NameLoad Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad decorator_list Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr FunctionDef arguments args NameParam defaults kwarg body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list Call NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call NameLoad Assign NameStore SubscriptLoad NameLoad Index NameLoad Assign NameStore Call Call NameLoad AttributeLoad NameLoad attr NameLoad If UnaryOpNot NameLoad body Raise Call AttributeLoad NameLoad attr Call AttributeLoad Str attr SubscriptLoad NameLoad Index Str Assign SubscriptStore NameLoad Index Str Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Return NameLoad decorator_list Call NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr FunctionDef arguments args NameParam NameParam defaults kwarg body If UnaryOpNot NameLoad body Raise Call AttributeLoad NameLoad attr Call AttributeLoad Str attr SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad decorator_list Call NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad Call NameLoad NameLoad NameLoad Num kwarg body Assign NameStore BoolOpOr NameLoad AttributeLoad NameLoad attr Assign NameStore AttributeLoad Call AttributeLoad NameLoad attr NameLoad attr Assign NameStore BoolOpOr NameLoad Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad If NameLoad body Assign NameStore Call NameLoad Call NameLoad NameLoad NameLoad While NameLoad body Expr Call AttributeLoad Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword NameLoad keyword NameLoad NameLoad attr Assign NameStore Call NameLoad Call NameLoad NameLoad NameLoad orelse For NameStore NameLoad body Expr Call NameLoad TupleLoad NameLoad keyword NameLoad keyword NameLoad NameLoad If NameLoad body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr Call NameLoad NameLoad decorator_list AttributeLoad NameLoad attr
N/A N/A logging N/A functools itertools islice datetime date timedelta celery Celery scrapi util scrapi events scrapi settings scrapi registry scrapi processing scrapi.util timestamp scrapi.base.helpers null_on_error N/A app N/A Celery N/A N/A N/A app config_from_object settings N/A logger N/A N/A logging getLogger __name__ task_autoretry N/A N/A N/A args_task kwargs_task N/A actual_decorator N/A N/A func N/A N/A wrapper N/A N/A N/A args kwargs N/A N/A N/A N/A N/A func args kwargs N/A N/A N/A N/A N/A kwargs_task get autoretry_on Exception N/A exc N/A N/A N/A N/A logger info N/A N/A Retrying with exception {} format exc N/A N/A N/A wrapper retry exc exc N/A N/A N/A app task args_task kwargs_task N/A N/A functools wraps func N/A wrapper N/A N/A actual_decorator N/A run_harvester N/A N/A harvester_name start_date end_date N/A None None N/A N/A N/A N/A logger info N/A N/A Running harvester "{}" format harvester_name N/A start_date N/A start_date N/A N/A N/A date today N/A timedelta N/A settings DAYS_BACK N/A end_date N/A end_date N/A N/A date today N/A normalization N/A N/A begin_normalization s harvester_name N/A start_harvest N/A N/A harvest si harvester_name N/A timestamp start_date start_date end_date end_date N/A N/A N/A N/A start_harvest normalization apply_async N/A N/A app task N/A N/A events creates_task N/A events HARVESTER_RUN harvest N/A N/A harvester_name job_created start_date end_date N/A None None N/A N/A harvest_started N/A timestamp N/A harvester N/A registry N/A harvester_name N/A start_date N/A start_date N/A N/A N/A date today N/A timedelta N/A settings DAYS_BACK N/A end_date N/A end_date N/A N/A date today N/A N/A N/A logger info N/A N/A Harvester "{}" has begun harvesting format harvester_name N/A result N/A N/A harvester harvest start_date start_date end_date end_date N/A N/A result N/A harvestFinished harvestTaskCreated harvestStarted N/A timestamp job_created harvest_started N/A N/A app task N/A N/A events logged N/A events HARVESTER_RUN begin_normalization N/A N/A raw_docs_timestamps harvester_name N/A N/A N/A harvest_ret is harvest return value:
        a tuple contaiing list of rawDocuments and
        a dictionary of timestamps
     N/A N/A raw_docs timestamps raw_docs_timestamps N/A N/A N/A logger info N/A N/A Normalizing {} documents for harvester "{}" format N/A len raw_docs harvester_name N/A raw raw_docs N/A N/A N/A spawn_tasks raw timestamps harvester_name N/A N/A app task spawn_tasks N/A N/A raw timestamps harvester_name N/A N/A N/A N/A raw N/A timestamps timestamps N/A N/A N/A raw N/A timestamps N/A normalizeTaskCreated N/A timestamp N/A chain N/A N/A N/A normalize si raw harvester_name N/A N/A process_normalized s raw N/A N/A N/A chain apply_async N/A N/A N/A process_raw delay raw N/A N/A N/A events creates_task N/A events PROCESSING N/A N/A events creates_task N/A events NORMALIZATION process_raw N/A N/A raw_doc N/A kwargs N/A N/A N/A N/A processing process_raw raw_doc kwargs N/A N/A task_autoretry default_retry_delay N/A settings CELERY_RETRY_DELAY max_retries N/A settings CELERY_MAX_RETRIES N/A N/A events logged N/A events PROCESSING raw normalize N/A N/A raw_doc harvester_name N/A N/A N/A normalized_started N/A timestamp N/A harvester N/A registry N/A harvester_name N/A normalized N/A N/A null_on_error N/A harvester normalize raw_doc N/A N/A normalized N/A N/A N/A N/A events Skip N/A N/A Did not normalize document with id {} format N/A raw_doc N/A docID N/A N/A normalized N/A timestamps N/A N/A util stamp_from_raw raw_doc normalizeStarted normalized_started N/A normalized N/A N/A task_autoretry default_retry_delay N/A settings CELERY_RETRY_DELAY max_retries N/A settings CELERY_MAX_RETRIES throws N/A events Skip N/A N/A events logged N/A events NORMALIZATION process_normalized N/A N/A normalized_doc raw_doc N/A kwargs N/A N/A N/A normalized_doc N/A N/A N/A N/A events Skip N/A N/A Not processing document with id {} format N/A raw_doc N/A docID N/A N/A N/A processing process_normalized raw_doc normalized_doc kwargs N/A N/A task_autoretry default_retry_delay N/A settings CELERY_RETRY_DELAY max_retries N/A settings CELERY_MAX_RETRIES throws N/A events Skip N/A N/A events logged N/A events PROCESSING normalized migrate N/A N/A migration source_db sources async dry group_size N/A None N/A tuple False True 1000 kwargs N/A N/A source_db N/A source_db N/A settings CANONICAL_PROCESSOR N/A documents N/A N/A N/A processing get_processor source_db documents N/A doc_sources N/A sources N/A N/A registry keys N/A docs N/A documents doc_sources N/A async N/A N/A segment N/A list N/A islice docs group_size N/A segment N/A N/A N/A N/A N/A N/A migration s segment sources sources dry dry source_db source_db kwargs apply_async N/A segment N/A list N/A islice docs group_size N/A N/A doc docs N/A N/A N/A migration N/A doc sources sources dry dry kwargs N/A dry N/A N/A N/A N/A logger info Dry run complete N/A N/A N/A logger info N/A N/A Documents processed for migration {} format N/A str migration N/A N/A app task