Module Import alias Import alias identifier Import alias Import alias Import alias ImportFrom alias alias ImportFrom alias ImportFrom alias ImportFrom alias ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Num Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad Return Call NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str If Call NameLoad NameLoad NameLoad body Assign NameStore Call NameLoad NameLoad Delete SubscriptDel NameLoad Index Str Delete SubscriptDel NameLoad Index Str Delete SubscriptDel NameLoad Index Str Delete SubscriptDel NameLoad Index Str Assign SubscriptStore NameLoad Index Str Call AttributeLoad AttributeLoad NameLoad attr attr Assign SubscriptStore NameLoad Index Str Str Assign SubscriptStore NameLoad Index Str AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad Return NameLoad orelse If Call NameLoad NameLoad body Assign SubscriptStore NameLoad Index Str Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad Return NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Num Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad TryExcept body Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Str Assign NameStore Call NameLoad NameLoad handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Num Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore SubscriptLoad NameLoad Index Str Return Call NameLoad NameLoad NameLoad NameLoad NameLoad keyword NameLoad keyword NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body TryExcept body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign NameStore Call NameLoad NameLoad Assign SubscriptStore NameLoad Index Str Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr Assign NameStore AttributeLoad NameLoad attr TryExcept body If AttributeLoad NameLoad attr body Assign SubscriptStore NameLoad Index Str Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler body Assign NameStore Str Assign NameStore Call AttributeLoad Str attr keyword NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad If AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad Str attr keyword NameLoad keyword SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad Assign SubscriptStore NameLoad Index Str NameLoad handlers ExceptHandler type NameLoad body Assign SubscriptStore NameLoad Index Str NameLoad Assign SubscriptStore NameLoad Index Str Call AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body If CompareNotIn NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Num Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad Return Call NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str If Call NameLoad NameLoad NameLoad body Assign NameStore Call NameLoad NameLoad Delete SubscriptDel NameLoad Index Str Delete SubscriptDel NameLoad Index Str Delete SubscriptDel NameLoad Index Str Delete SubscriptDel NameLoad Index Str Delete SubscriptDel NameLoad Index Str Delete SubscriptDel NameLoad Index Str Assign SubscriptStore NameLoad Index Str Str Assign SubscriptStore NameLoad Index Str Call AttributeLoad AttributeLoad NameLoad attr attr Assign SubscriptStore NameLoad Index Str AttributeLoad NameLoad attr If SubscriptLoad NameLoad Index Str body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad orelse Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad Return NameLoad decorator_list decorator_list
N/A N/A json N/A datetime dt N/A sys N/A traceback N/A base64 kafka KafkaClient SimpleProducer kafka.common KafkaUnavailableError crawling.items RawResponseItem scutils.log_factory LogFactory LoggingBeforePipeline N/A object N/A N/A 
    Logs the crawl, currently the 1st priority of the pipeline
     __init__ N/A N/A self logger N/A N/A N/A N/A self logger logger N/A N/A N/A N/A self logger debug Setup before pipeline N/A from_settings N/A N/A cls settings N/A N/A N/A my_level N/A N/A settings get SC_LOG_LEVEL INFO N/A my_name N/A N/A settings get SC_LOGGER_NAME sc-logger N/A my_output N/A N/A settings get SC_LOG_STDOUT True N/A my_json N/A N/A settings get SC_LOG_JSON False N/A my_dir N/A N/A settings get SC_LOG_DIR logs N/A my_bytes N/A N/A settings get SC_LOG_MAX_BYTES 10MB N/A my_file N/A N/A settings get SC_LOG_FILE main.log N/A my_backups N/A N/A settings get SC_LOG_BACKUPS 5 N/A logger N/A N/A LogFactory get_instance json my_json name my_name stdout my_output level my_level dir my_dir file my_file bytes my_bytes backups my_backups N/A N/A cls logger N/A classmethod from_crawler N/A N/A cls crawler N/A N/A N/A N/A N/A cls from_settings N/A crawler settings N/A classmethod process_item N/A N/A self item spider N/A N/A N/A N/A N/A N/A self logger debug Processing item in LoggingBeforePipeline N/A N/A isinstance item RawResponseItem N/A N/A item_copy N/A dict item N/A N/A item_copy N/A body N/A N/A item_copy N/A links N/A N/A item_copy N/A response_headers N/A N/A item_copy N/A request_headers N/A N/A item_copy N/A logger N/A N/A N/A self logger name N/A N/A item_copy N/A action emit N/A N/A item_copy N/A spiderid N/A spider name N/A N/A N/A N/A self logger info Scraped page extra item_copy N/A item N/A N/A N/A isinstance item N/A N/A N/A item N/A logger N/A N/A N/A self logger name N/A N/A N/A N/A self logger error Scraper Retry extra item N/A None N/A N/A KafkaPipeline N/A object N/A N/A 
    Pushes a serialized item to appropriate Kafka topics.
     __init__ N/A N/A self producer topic_prefix aKafka logger appids use_base64 N/A N/A N/A N/A self producer producer N/A N/A self topic_prefix topic_prefix N/A N/A self topic_list N/A N/A N/A self kafka aKafka N/A N/A self appid_topics appids N/A N/A self logger logger N/A N/A N/A N/A self logger debug Setup kafka pipeline N/A N/A self use_base64 use_base64 N/A from_settings N/A N/A cls settings N/A N/A N/A my_level N/A N/A settings get SC_LOG_LEVEL INFO N/A my_name N/A N/A settings get SC_LOGGER_NAME sc-logger N/A my_output N/A N/A settings get SC_LOG_STDOUT True N/A my_json N/A N/A settings get SC_LOG_JSON False N/A my_dir N/A N/A settings get SC_LOG_DIR logs N/A my_bytes N/A N/A settings get SC_LOG_MAX_BYTES 10MB N/A my_file N/A N/A settings get SC_LOG_FILE main.log N/A my_backups N/A N/A settings get SC_LOG_BACKUPS 5 N/A my_appids N/A N/A settings get KAFKA_APPID_TOPICS False N/A logger N/A N/A LogFactory get_instance json my_json name my_name stdout my_output level my_level dir my_dir file my_file bytes my_bytes backups my_backups N/A N/A N/A kafka N/A KafkaClient N/A settings N/A KAFKA_HOSTS N/A producer N/A SimpleProducer kafka N/A N/A N/A KafkaUnavailableError N/A N/A N/A N/A logger error Unable to connect to Kafka in Pipeline, raising exit flag. N/A N/A N/A sys exit 1 N/A topic_prefix N/A settings N/A KAFKA_TOPIC_PREFIX N/A use_base64 N/A settings N/A KAFKA_BASE_64_ENCODE N/A N/A cls producer topic_prefix kafka logger appids my_appids use_base64 use_base64 N/A classmethod from_crawler N/A N/A cls crawler N/A N/A N/A N/A N/A cls from_settings N/A crawler settings N/A classmethod process_item N/A N/A self item spider N/A N/A N/A N/A N/A N/A N/A N/A self logger debug Processing item in KafkaPipeline N/A datum N/A dict item N/A N/A datum N/A timestamp N/A N/A N/A N/A N/A dt datetime utcnow isoformat N/A prefix N/A self topic_prefix N/A N/A N/A N/A self use_base64 N/A N/A N/A datum N/A body N/A N/A base64 b64encode N/A datum N/A body N/A message N/A N/A json dumps datum N/A N/A N/A N/A message json failed to parse N/A firehose_topic N/A N/A {prefix}.crawled_firehose format prefix prefix N/A N/A N/A self checkTopic firehose_topic N/A N/A N/A N/A self producer send_messages firehose_topic message N/A N/A self appid_topics N/A N/A appid_topic N/A N/A {prefix}.crawled_{appid} format prefix prefix appid N/A datum N/A appid N/A N/A N/A self checkTopic appid_topic N/A N/A N/A N/A self producer send_messages appid_topic message N/A N/A item N/A success True N/A N/A N/A Exception N/A N/A N/A item N/A success False N/A N/A item N/A exception N/A N/A traceback format_exc N/A item N/A checkTopic N/A N/A self topicName N/A N/A N/A N/A topicName N/A self topic_list N/A N/A N/A N/A N/A self kafka ensure_topic_exists topicName N/A N/A N/A N/A self topic_list append topicName N/A N/A LoggingAfterPipeline N/A object N/A N/A 
    Logs the crawl for successfully pushing to Kafka
     __init__ N/A N/A self logger N/A N/A N/A N/A self logger logger N/A N/A N/A N/A self logger debug Setup after pipeline N/A from_settings N/A N/A cls settings N/A N/A N/A my_level N/A N/A settings get SC_LOG_LEVEL INFO N/A my_name N/A N/A settings get SC_LOGGER_NAME sc-logger N/A my_output N/A N/A settings get SC_LOG_STDOUT True N/A my_json N/A N/A settings get SC_LOG_JSON False N/A my_dir N/A N/A settings get SC_LOG_DIR logs N/A my_bytes N/A N/A settings get SC_LOG_MAX_BYTES 10MB N/A my_file N/A N/A settings get SC_LOG_FILE main.log N/A my_backups N/A N/A settings get SC_LOG_BACKUPS 5 N/A logger N/A N/A LogFactory get_instance json my_json name my_name stdout my_output level my_level dir my_dir file my_file bytes my_bytes backups my_backups N/A N/A cls logger N/A classmethod from_crawler N/A N/A cls crawler N/A N/A N/A N/A N/A cls from_settings N/A crawler settings N/A classmethod process_item N/A N/A self item spider N/A N/A N/A N/A N/A N/A self logger debug Processing item in LoggingAfterPipeline N/A N/A isinstance item RawResponseItem N/A N/A item_copy N/A dict item N/A N/A item_copy N/A body N/A N/A item_copy N/A links N/A N/A item_copy N/A response_headers N/A N/A item_copy N/A request_headers N/A N/A item_copy N/A status_code N/A N/A item_copy N/A status_msg N/A N/A item_copy N/A action ack N/A N/A item_copy N/A logger N/A N/A N/A self logger name N/A N/A item_copy N/A spiderid N/A spider name N/A N/A item N/A success N/A N/A N/A N/A N/A self logger info Sent page to Kafka extra item_copy N/A N/A N/A N/A N/A self logger error Failed to send page to Kafka extra item_copy N/A item N/A N/A