Module Import alias identifier ImportFrom alias ImportFrom alias alias alias ImportFrom alias ImportFrom alias alias ImportFrom alias alias alias alias alias alias alias ImportFrom alias ImportFrom alias alias alias alias alias alias alias Assign NameStore Num Assign NameStore ListLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Dict For NameStore NameLoad body If CompareIn NameLoad NameLoad body Assign SubscriptStore NameLoad Index Call AttributeLoad NameLoad attr Str Str SubscriptLoad NameLoad Index NameLoad Delete SubscriptDel NameLoad Index NameLoad If CompareIn Str NameLoad body Delete SubscriptDel NameLoad Index Str Assign SubscriptStore NameLoad Index Str Call AttributeLoad NameLoad attr Return TupleLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str TryExcept body Return Call AttributeLoad AttributeLoad NameLoad attr attr keyword SubscriptLoad NameLoad Index Str handlers ExceptHandler body Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str If CompareIn NameLoad NameLoad body Delete SubscriptDel NameLoad Index NameLoad TryExcept body Assign TupleStore NameStore NameStore Call NameLoad Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad If UnaryOpNot NameLoad body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign SubscriptStore NameLoad Index Str NameLoad If BoolOpAnd Call NameLoad NameLoad Str CompareEq AttributeLoad NameLoad attr Num body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad orelse Assign SubscriptStore NameLoad Index Str AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad If CompareIn Str NameLoad body Assign NameStore Dict Str Str Str NameLoad SubscriptLoad NameLoad Index Str SubscriptLoad NameLoad Index Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call NameLoad keyword AttributeLoad NameLoad attr keyword SubscriptLoad NameLoad Index Str keyword NameLoad keyword NameLoad keyword Str keyword NameLoad decorator_list AttributeLoad NameLoad attr ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults body Return BoolOpAnd Call NameLoad NameLoad BinOpAdd NameLoad Str CompareNotEq Call NameLoad NameLoad BinOpAdd NameLoad Str Str decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body For NameStore AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr Str If BoolOpAnd UnaryOpNot Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad body If CompareEq NameLoad Str body Expr Call NameLoad keyword AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword Str keyword NameLoad Raise NameLoad orelse Expr Call NameLoad keyword AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword Str keyword NameLoad Return NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults body Print BinOpMod Str TupleLoad AttributeLoad NameLoad attr SubscriptLoad SubscriptLoad NameLoad Index Str Slice Num BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr IfExp CompareIn Str NameLoad SubscriptLoad NameLoad Index Str Str Return NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Call NameLoad NameLoad NameLoad Return NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults body Assign SubscriptStore AttributeLoad NameLoad attr Index AttributeLoad NameLoad attr Call NameLoad ListLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Delete SubscriptDel AttributeLoad NameLoad attr Index AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body If UnaryOpNot CompareIn SubscriptLoad NameLoad Index NameLoad SubscriptLoad AttributeLoad NameLoad attr Index AttributeLoad NameLoad attr body Expr Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index AttributeLoad NameLoad attr attr SubscriptLoad NameLoad Index NameLoad Return NameLoad orelse Raise NameLoad decorator_list decorator_list
N/A N/A django.utils.timezone tz scrapy.exceptions DropItem colorama Fore Back Style django.db transaction errors raise_missing_value DATABASE_ERROR core.models TestItem SITE_MODEL ITEM_MODEL MERCHANT_MODEL SoldCount STATUS_CRITICAL STATUS_WARNING core.general FIELD_FINISHED spiders.fields F_M_NAME F_M_LAT F_M_LON F_M_ADDRESS F_M_PHONE F_M_WEBSITE F_M_EMAIL N/A SOLD_COUNT_LOG_PERIOD 15 N/A MERCHANT_FIELDS N/A F_M_NAME F_M_LAT F_M_LON F_M_ADDRESS F_M_PHONE F_M_WEBSITE F_M_EMAIL extract_item_data N/A N/A deal N/A N/A N/A Separate deal and merchant data. N/A merchant N/A N/A field MERCHANT_FIELDS N/A N/A N/A field deal N/A N/A N/A merchant N/A N/A N/A field replace merchant_ N/A deal N/A field N/A N/A deal N/A field N/A N/A merchant_city deal N/A N/A N/A deal N/A merchant_city N/A N/A deal N/A date_time N/A N/A tz now N/A N/A deal merchant N/A find_existing_merchant N/A N/A merchant N/A N/A N/A Find a Merchant object from the data in a merchant data dictionary. N/A N/A N/A N/A N/A N/A MERCHANT_MODEL objects get name N/A merchant N/A name N/A N/A N/A N/A None N/A store_item N/A N/A item spider N/A N/A N/A Store an item in the database. N/A N/A FIELD_FINISHED item N/A N/A N/A item N/A FIELD_FINISHED N/A N/A N/A N/A deal merchant N/A extract_item_data N/A dict item N/A merchant_obj N/A find_existing_merchant merchant N/A N/A merchant_obj N/A N/A merchant_obj N/A N/A N/A MERCHANT_MODEL objects create merchant N/A N/A deal N/A merchant merchant_obj N/A N/A N/A hasattr spider site_id N/A N/A spider site_id -1 N/A N/A N/A N/A N/A TestItem objects create deal N/A N/A N/A deal N/A site_id N/A spider site_id N/A deal_id N/A N/A N/A ITEM_MODEL objects create deal N/A N/A sold_count deal N/A N/A log_data N/A deal date value deal_id N/A deal N/A date_time N/A deal N/A sold_count N/A log N/A N/A N/A SoldCount objects create log_data N/A N/A N/A Exception N/A e N/A N/A N/A raise_missing_value spider_name N/A spider name url N/A item N/A url exception e level STATUS_CRITICAL message Failed to store item. category DATABASE_ERROR N/A N/A transaction autocommit CheckMissingValuesPipeline N/A object N/A N/A Check if the item is missing key values, and if so, log the error and drop
    it. defines_xpath N/A N/A self spider key N/A N/A N/A N/A N/A hasattr spider N/A key _xpath N/A N/A getattr spider N/A key _xpath N/A process_item N/A N/A self item spider N/A N/A N/A key N/A item fields N/A N/A url N/A N/A item get url N/A N/A N/A N/A N/A item get key N/A N/A self defines_xpath spider key N/A N/A N/A key hash_id N/A N/A N/A raise_missing_value spider_name N/A spider name field_name key url url level STATUS_CRITICAL message Missing Item hash ID. category DATABASE_ERROR N/A DropItem N/A N/A N/A raise_missing_value spider_name N/A spider name field_name key url url level STATUS_WARNING message Missing field. category DATABASE_ERROR N/A item N/A N/A PrintPipeline N/A object N/A N/A When processing a deal, pretty-print the important information to the console
    with some color.
     process_item N/A N/A self item spider N/A N/A N/A N/A %s%s%s %s%s (%s %s%s%s, %s) N/A N/A Fore GREEN N/A N/A item N/A hash_id N/A 10 N/A N/A Fore RESET N/A Style BRIGHT N/A item N/A offer N/A Style RESET_ALL N/A item N/A currency N/A N/A Fore YELLOW N/A Style DIM N/A item N/A price N/A N/A Fore RESET N/A Style RESET_ALL N/A N/A discount item N/A item N/A discount ? N/A item N/A N/A StorePipeline N/A object N/A N/A Store every item that comes along into the database. process_item N/A N/A self item spider N/A N/A N/A N/A store_item item spider N/A item N/A N/A DuplicatesPipeline N/A object N/A N/A Filter out duplicate objects. spider_opened N/A N/A self spider N/A N/A N/A N/A N/A self duplicates N/A N/A spider name N/A set N/A N/A spider_closed N/A N/A self spider N/A N/A N/A N/A N/A self duplicates N/A N/A spider name N/A process_item N/A N/A self item spider N/A N/A N/A N/A N/A N/A item N/A FIELD_HASH_ID N/A N/A self duplicates N/A N/A spider name N/A N/A N/A N/A N/A N/A self duplicates N/A N/A spider name add N/A item N/A FIELD_HASH_ID N/A item N/A N/A DropItem N/A N/A