Module Import alias Import alias Import alias ImportFrom alias ImportFrom alias Import alias ImportFrom alias ImportFrom alias alias ImportFrom alias alias Assign NameStore Call AttributeLoad NameLoad attr NameLoad FunctionDef arguments args defaults body TryExcept body Return DictComp SubscriptLoad NameLoad Index Str Call NameLoad NameLoad comprehension NameStore Call NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr Str If AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args defaults body Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr attr While NameLoad body For NameStore SubscriptLoad NameLoad Index Str body Expr Yield NameLoad If SubscriptLoad NameLoad Index Str body Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str attr orelse Break decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad NameLoad kwarg body Assert BoolOpAnd NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad ImportFrom alias Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str NameLoad keyword Dict Str Str Str Str NameLoad NameLoad NameLoad NameLoad keyword NameLoad keyword NameLoad Return Call NameLoad Call AttributeLoad Str attr Call AttributeLoad Call AttributeLoad NameLoad attr attr TupleLoad NameLoad Call NameLoad keyword NameLoad keyword NameLoad keyword NameLoad decorator_list ClassDef bases NameLoad body Assign NameStore Str FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad NameLoad body Assign NameStore BoolOpOr NameLoad BinOpSub Call AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr Assign NameStore BoolOpOr NameLoad Call AttributeLoad NameLoad attr Return ListComp Call NameLoad Dict Str Str Str Str Call AttributeLoad NameLoad attr NameLoad SubscriptLoad NameLoad Index Str SubscriptLoad NameLoad Index Str Str comprehension NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr keyword Dict Str Str Str Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr For NameStore SubscriptLoad NameLoad Index Str body Expr Yield NameLoad While Call AttributeLoad NameLoad attr Str body Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str attr For NameStore SubscriptLoad NameLoad Index Str body Expr Yield NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Assign NameStore SubscriptLoad NameLoad Index Str For NameStore SubscriptLoad NameLoad Index Str body If CompareEq Call AttributeLoad NameLoad attr Str NameLoad Str body Delete SubscriptDel NameLoad Index Str If CompareEq SubscriptLoad NameLoad Index Str Str body Assign SubscriptStore SubscriptLoad NameLoad Index Str Index Str Str Return Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return Dict Str Str decorator_list NameLoad decorator_list Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad
N/A N/A six N/A json N/A logging datetime date datetime timedelta N/A requests scrapi settings scrapi.base BaseHarvester HarvesterMeta scrapi.linter.document RawDocument NormalizedDocument N/A logger N/A N/A logging getLogger __name__ gen_harvesters N/A N/A N/A N/A N/A N/A N/A N/A N/A source N/A shortname N/A gen_harvester source N/A source N/A get_sources N/A N/A N/A Exception N/A e N/A N/A N/A N/A logger warn Could not generate harvesters from push api N/A N/A settings DEBUG N/A N/A N/A N/A logger exception e N/A get_sources N/A N/A N/A N/A N/A response N/A N/A N/A N/A requests get N/A N/A {}/provider_list format N/A settings SHARE_REG_URL json N/A True N/A N/A source N/A response N/A results N/A N/A N/A source N/A N/A response N/A next N/A N/A response N/A N/A N/A N/A requests get N/A response N/A next json N/A N/A N/A gen_harvester N/A N/A shortname longname url favicon_dataurl N/A None None None None kwargs N/A N/A N/A shortname longname url favicon_dataurl N/A N/A N/A logger info N/A N/A Generating harvester {} format shortname scrapi.processing.elasticsearch DatabaseManager N/A dm N/A DatabaseManager N/A N/A N/A dm setup N/A es N/A dm es N/A N/A N/A es index share_providers shortname body N/A favicon short_name long_name url favicon_dataurl shortname longname url id shortname refresh True N/A N/A type N/A N/A {}Harvester format N/A N/A N/A N/A shortname lower capitalize N/A PushApiHarvester N/A dict short_name shortname long_name longname url url N/A PushApiHarvester N/A BaseHarvester N/A N/A file_format json harvest N/A N/A self start_date end_date N/A None None N/A N/A start_date N/A start_date N/A N/A N/A date today N/A timedelta N/A settings DAYS_BACK N/A end_date N/A end_date N/A N/A date today N/A N/A N/A RawDocument N/A doc source docID filetype N/A N/A json dumps record N/A record N/A source N/A record N/A docID json N/A record N/A N/A self get_records start_date end_date N/A get_records N/A N/A self start_date end_date N/A N/A N/A response N/A N/A requests get N/A N/A {}/established format N/A settings SHARE_REG_URL params N/A from to source N/A N/A start_date isoformat N/A N/A end_date isoformat N/A self short_name N/A response N/A N/A response json N/A record N/A response N/A results N/A N/A N/A record N/A N/A N/A response get next N/A N/A response N/A N/A N/A N/A requests get N/A response N/A next json N/A record N/A response N/A results N/A N/A N/A record N/A normalize N/A N/A self raw N/A N/A N/A raw_data N/A N/A json loads N/A raw N/A doc N/A document N/A raw_data N/A jsonData N/A contributor N/A document N/A contributors N/A N/A N/A N/A N/A contributor get email None N/A N/A N/A contributor N/A email N/A N/A N/A raw_data N/A status deleted N/A N/A N/A N/A document N/A shareProperties N/A status deleted N/A N/A NormalizedDocument document N/A run_at N/A N/A self N/A N/A N/A N/A minute */15 N/A property N/A N/A N/A six add_metaclass HarvesterMeta N/A harvesters N/A gen_harvesters