Module ImportFrom alias alias alias alias ImportFrom alias ImportFrom alias ImportFrom alias alias ImportFrom alias Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias alias alias ImportFrom alias ImportFrom alias alias ImportFrom alias Assign NameStore Call AttributeLoad NameLoad attr NameLoad ClassDef bases NameLoad body Assign NameStore Str Assign NameStore Str FunctionDef arguments args NameParam NameParam defaults kwarg body Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr ListLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body For NameStore NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body If UnaryOpNot NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad If NameLoad body Return Call NameLoad Call AttributeLoad NameLoad attr Num orelse Return Num decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad kwarg body Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr If NameLoad body Expr Call AttributeLoad NameLoad attr Dict Str Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults Num kwarg body FunctionDef arguments args NameParam defaults body While NameLoad body If Call AttributeLoad NameLoad attr NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Return NameLoad Expr Call AttributeLoad NameLoad attr Num decorator_list If CompareGt NameLoad Num body Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore BinOpSub Call NameLoad NameLoad Num Assign NameStore BinOpDiv NameLoad NameLoad Assign NameStore NameLoad If CompareLt BinOpMult NameLoad BinOpAdd NameLoad Num NameLoad body Assign NameStore BinOpAdd NameLoad Num Assign NameStore Num Assign NameStore IfExp CompareLtE NameLoad NameLoad NameLoad NameLoad For NameStore Call NameLoad Num NameLoad body Assign TupleStore NameStore NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Call NameLoad keyword AttributeLoad NameLoad attr keyword TupleLoad NameLoad SubscriptLoad NameLoad Slice NameLoad BinOpAdd NameLoad Num NameLoad keyword NameLoad Assign NameStore BinOpAdd NameLoad Num Assign NameStore BinOpAdd BinOpAdd NameLoad NameLoad Num If BoolOpAnd CompareEq NameLoad BinOpSub NameLoad Num CompareGt NameLoad NameLoad body Assign NameStore NameLoad For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr For NameStore NameLoad body Assign NameStore Call NameLoad NameLoad For NameStore SubscriptLoad NameLoad Index Str body Expr Call AttributeLoad NameLoad attr NameLoad For NameStore SubscriptLoad NameLoad Index Str body Expr Call AttributeLoad NameLoad attr NameLoad For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr Return Dict Str Str NameLoad NameLoad orelse Return Call NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad Str NameLoad Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad keyword Call NameLoad keyword Call AttributeLoad NameLoad attr Str Assign NameStore ListComp NameLoad comprehension NameStore NameLoad Assign NameStore Num Assign NameStore Num If CompareGt Call NameLoad NameLoad Num body Assign NameStore SubscriptLoad NameLoad Index Num Assign NameStore Call AttributeLoad SubscriptLoad NameLoad Slice Call AttributeLoad NameLoad attr Str attr Str Str Assign NameStore Call NameLoad NameLoad Assign NameStore SubscriptLoad NameLoad Index Num Assign NameStore SubscriptLoad NameLoad Slice BinOpAdd Call AttributeLoad NameLoad attr Str Num BinOpSub Call AttributeLoad NameLoad attr Str Num Assign NameStore Call NameLoad NameLoad Assign NameStore NameLoad Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Call NameLoad Num BinOpAdd Call AttributeLoad NameLoad attr NameLoad Num NameLoad AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Assign NameStore Call NameLoad ListComp SubscriptLoad SubscriptLoad NameLoad Index Str Index Str comprehension NameStore AttributeLoad NameLoad attr Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign NameStore NameLoad If BoolOpAnd CompareGt NameLoad Num CompareGt NameLoad Num CompareLtE BinOpDiv NameLoad NameLoad Num CompareLtE BinOpDiv NameLoad NameLoad Num body Assign NameStore NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Assign NameStore Call NameLoad Dict Str Str Str Str Str Str Str Str NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad NameLoad NameLoad NameLoad Str Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad keyword NameLoad Assign AttributeStore NameLoad attr SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Print AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad IfExp NameLoad Str Str BinOpSub Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad NameLoad NameLoad NameLoad Call NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam NameParam defaults kwarg body Expr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr ListLoad decorator_list FunctionDef arguments args NameParam defaults body TryExcept body Assign TupleStore AttributeStore NameLoad attr AttributeStore NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr handlers ExceptHandler body Import alias Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad SubscriptLoad Call AttributeLoad NameLoad attr Index Num attr SubscriptLoad Call AttributeLoad NameLoad attr Index Num Expr Call AttributeLoad AttributeLoad NameLoad attr attr Dict Str Str Str Call NameLoad AttributeLoad SubscriptLoad Call AttributeLoad NameLoad attr Index Num attr Call NameLoad SubscriptLoad Call AttributeLoad NameLoad attr Index Num Call NameLoad Call AttributeLoad NameLoad attr SubscriptLoad Call AttributeLoad NameLoad attr Index Num decorator_list decorator_list
N/A BeautifulSoup BeautifulSoup ResultSet SoupStrainer Tag django.core.management.base BaseCommand django.db.models Max multiprocessing Pipe Process threading Thread N/A datetime N/A logging N/A re N/A sys N/A time N/A urllib2 utils.pid Pid crawler_callbacks_data callback_allhit callback_details callback_add_crawlfk crawler_callbacks_save callback_database crawler_common get_allhit_url get_group_url mturk.main.models Crawl N/A log N/A N/A logging getLogger __name__ Command N/A BaseCommand N/A N/A help Runs the MTurk crawler N/A args number of crawling processes handle N/A N/A self processes_count N/A options N/A N/A processes_count N/A int processes_count N/A crawler N/A Crawler processes_count N/A N/A N/A crawler start N/A N/A N/A crawler join N/A N/A Crawler N/A Thread N/A __init__ N/A N/A self processes_count N/A N/A N/A N/A N/A Thread __init__ self N/A N/A self processes_count processes_count N/A N/A self data N/A N/A N/A self errors N/A N/A append_errors N/A N/A self errors N/A N/A N/A error errors N/A N/A N/A N/A N/A self errors append error N/A get_max_page N/A N/A self html N/A None N/A N/A N/A html N/A N/A response N/A N/A urllib2 urlopen N/A get_allhit_url N/A html N/A N/A response read N/A pattern N/A N/A re compile <a href=\".*pageNumber=([0-9]+).{150,200}Last</a> N/A max_page N/A N/A re search pattern html N/A max_page N/A N/A N/A int N/A N/A max_page group 1 N/A N/A 1 N/A launch_worker N/A N/A self callback callback_arg conn N/A None kwargs N/A N/A worker N/A Worker callback callback_arg kwargs N/A N/A N/A worker start N/A N/A N/A worker join N/A conn N/A N/A N/A N/A conn send N/A data errors N/A worker data N/A worker errors N/A process_values N/A N/A self values callback processes_count N/A 1 kwargs N/A receive_from_pipe N/A N/A conn N/A N/A N/A True N/A N/A N/A N/A conn poll None N/A N/A data N/A N/A conn recv N/A N/A N/A conn close N/A data N/A N/A N/A time sleep 1 N/A N/A N/A processes_count 1 N/A N/A data N/A N/A errors N/A N/A conns N/A N/A processes N/A N/A max_value N/A N/A len values 1 N/A interval N/A max_value processes_count N/A values_range processes_count N/A N/A N/A processes_count N/A interval 1 max_value N/A N/A values_range N/A values_range 1 N/A values_from 0 N/A values_to N/A N/A interval max_value interval max_value N/A i N/A range 0 values_range N/A N/A N/A parent_conn child_conn N/A Pipe False N/A N/A N/A conns append parent_conn N/A N/A N/A processes append N/A Process target N/A self launch_worker args N/A callback N/A values N/A values_from N/A values_to 1 child_conn kwargs kwargs N/A values_from N/A values_to 1 N/A values_to N/A N/A values_to interval 1 N/A N/A N/A i N/A values_range 2 N/A values_to max_value N/A N/A values_to max_value N/A process processes N/A N/A N/A N/A process start N/A conn conns N/A N/A result N/A receive_from_pipe conn N/A record N/A result N/A data N/A N/A N/A N/A data append record N/A error N/A result N/A errors N/A N/A N/A N/A errors append error N/A process processes N/A N/A N/A N/A process join N/A N/A data errors data errors N/A N/A N/A callback values kwargs N/A run N/A N/A self N/A N/A N/A pid N/A Pid mturk_crawler True N/A N/A N/A log info Crawler started N/A start_time N/A N/A N/A datetime datetime now N/A N/A N/A log debug Fetching stats N/A main_response N/A N/A urllib2 urlopen N/A get_allhit_url N/A main_html N/A N/A main_response read N/A main_soup N/A BeautifulSoup main_html parseOnlyThese N/A SoupStrainer text N/A N/A re compile (^[0-9,]+ HITs|of [0-9]+ Results) N/A main_stats N/A tag N/A tag main_soup N/A hits_available -1 N/A groups_available -1 N/A N/A N/A len main_stats 1 N/A N/A hits_available_tmp N/A main_stats N/A 0 N/A hits_available_tmp N/A N/A N/A hits_available_tmp N/A N/A N/A hits_available_tmp find replace , N/A hits_available N/A int hits_available_tmp N/A groups_available_tmp N/A main_stats N/A 1 N/A groups_available_tmp N/A groups_available_tmp N/A N/A N/A N/A groups_available_tmp find of 3 N/A N/A N/A groups_available_tmp find Results 1 N/A groups_available N/A int groups_available_tmp N/A main_soup None N/A N/A N/A log debug Allhit processing N/A result_allhit N/A N/A self process_values N/A range 1 N/A N/A N/A self get_max_page main_html 1 callback_allhit N/A self processes_count N/A N/A self data N/A result_allhit N/A data N/A N/A N/A self append_errors N/A result_allhit N/A errors N/A N/A N/A log debug Details processing N/A result_details N/A N/A self process_values N/A self data callback_details N/A self processes_count N/A N/A self data N/A result_details N/A data N/A N/A N/A self append_errors N/A result_details N/A errors N/A hits_downloaded N/A sum N/A N/A N/A hgs N/A HitGroupStatus N/A hits_available N/A hgs N/A self data N/A groups_downloaded N/A len N/A self data N/A success False N/A N/A N/A groups_downloaded 0 N/A hits_downloaded 0 N/A N/A groups_available groups_downloaded 1.5 N/A N/A hits_available hits_downloaded 1.5 N/A N/A success True N/A N/A N/A log debug N/A Crawl finished with success=%s. Saving main_crawl entry success N/A crawl N/A Crawl N/A start_time end_time success hits_available hits_downloaded groups_available groups_downloaded errors start_time N/A N/A N/A datetime datetime now success hits_available hits_downloaded groups_available groups_downloaded N/A N/A N/A crawl save N/A N/A N/A log debug Adding FKs N/A result_add_crawlfk N/A N/A self process_values N/A self data callback_add_crawlfk crawl crawl N/A N/A self data N/A result_add_crawlfk N/A data N/A N/A N/A self append_errors N/A result_add_crawlfk N/A errors N/A N/A N/A log debug Saving results N/A result_save_database N/A N/A self process_values N/A self data callback_database N/A N/A N/A self append_errors N/A result_save_database N/A errors N/A N/A self errors N/A N/A N/A log info N/A Crawler finished %ssuccessfully in %s with %d results, %d HITs (of %d and %d) and %d errors N/A N/A success un N/A N/A N/A N/A datetime datetime now start_time groups_downloaded hits_downloaded groups_available hits_available N/A len N/A self errors N/A N/A N/A pid remove_pid N/A N/A Worker N/A Thread N/A __init__ N/A N/A self callback callback_arg N/A callback_kwargs N/A N/A N/A N/A Thread __init__ self N/A N/A self callback callback N/A N/A self callback_arg callback_arg N/A N/A self callback_kwargs callback_kwargs N/A N/A self data N/A N/A N/A self errors N/A N/A run N/A N/A self N/A N/A N/A N/A N/A N/A N/A self data N/A self errors N/A N/A self callback N/A self callback_arg N/A self callback_kwargs N/A N/A N/A N/A traceback N/A N/A N/A log error N/A %s: %s N/A N/A N/A N/A N/A sys exc_info N/A 0 __name__ N/A N/A N/A sys exc_info N/A 1 N/A N/A N/A N/A self errors append N/A type value traceback N/A str N/A N/A N/A N/A sys exc_info N/A 0 __name__ N/A str N/A N/A N/A sys exc_info N/A 1 N/A unicode N/A N/A traceback extract_tb N/A N/A N/A sys exc_info N/A 2 N/A N/A