Module Expr Str Import alias Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias alias alias ImportFrom alias ImportFrom alias ImportFrom alias alias alias Assign NameStore BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd Str Str Str Str Str Str Str Str Str Str Str Str Str Assign NameStore Str FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Dict Str Str Str Str NameLoad IfExp NameLoad Call AttributeLoad NameLoad attr Str Str IfExp NameLoad Call AttributeLoad NameLoad attr Str Str Num Assign NameStore Num Assign NameStore ListLoad TryExcept body Expr Call AttributeLoad AttributeLoad NameLoad attr attr BinOpMod Str NameLoad While NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore BinOpMod NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad If CompareEq AttributeLoad NameLoad attr Num body Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Str Assign NameStore SubscriptLoad NameLoad Index Str For NameStore SubscriptLoad NameLoad Index Str body Assign TupleStore NameStore NameStore NameStore SubscriptLoad NameLoad Index Str Assign NameStore Call NameLoad NameLoad If NameLoad body Expr Call AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad AugAssignAdd NameStore Num If CompareEq NameLoad NameLoad body Break orelse AugAssignAdd SubscriptStore NameLoad Index Str Num orelse Raise Call NameLoad BinOpMod Str Call NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Return NameLoad handlers ExceptHandler type AttributeLoad NameLoad attr name NameStore body Print BinOpMod Str NameLoad Print BinOpMod Str NameLoad Raise decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr Lambda arguments args defaults Str TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad If CompareEq AttributeLoad NameLoad attr Num body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Str Return TupleLoad AttributeLoad NameLoad attr NameLoad orelse Return TupleLoad AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type AttributeLoad NameLoad attr name NameStore body Return TupleLoad AttributeLoad NameLoad attr NameLoad decorator_list Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str For TupleStore NameStore NameStore NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call NameLoad NameLoad If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Continue Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str FunctionDef arguments args NameParam defaults body Return SubscriptLoad SubscriptLoad NameLoad Index Num Index Num decorator_list FunctionDef arguments args NameParam defaults body Return SubscriptLoad SubscriptLoad NameLoad Index Num Index Num decorator_list FunctionDef arguments args NameParam defaults body Return SubscriptLoad NameLoad Index Num decorator_list FunctionDef arguments args NameParam defaults body Return CompareEq Call NameLoad NameLoad Num decorator_list FunctionDef arguments args NameParam defaults body Return CompareNotEq Call NameLoad NameLoad Num decorator_list Assign NameStore Lambda arguments args NameParam NameParam defaults BinOpAdd NameLoad Num Assign NameStore BinOpRShift BinOpRShift NameLoad Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad Num Assign NameStore BinOpRShift BinOpRShift BinOpRShift NameLoad Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore BinOpRShift BinOpRShift NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad If CompareGt Call NameLoad NameLoad Num body Print Print Str For TupleStore NameStore NameStore Call NameLoad NameLoad body Print BinOpMod Str TupleLoad NameLoad Call NameLoad NameLoad Call NameLoad NameLoad If CompareGt NameLoad Num body Print Print BinOpMod Str TupleLoad Call NameLoad NameLoad NameLoad Print Print Str Assign NameStore Call NameLoad Str Return CompareIn Call AttributeLoad NameLoad attr TupleLoad Str Str Str orelse Print Print Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad SubscriptLoad NameLoad Slice Num decorator_list FunctionDef arguments args NameParam defaults body Return BinOpAdd NameLoad Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad NameLoad TryExcept body Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad If CompareEq AttributeLoad NameLoad attr Num body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad With Call NameLoad NameLoad Str NameStore body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr orelse Raise Call NameLoad BinOpMod Str TupleLoad Call NameLoad AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Raise decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str If CompareIs NameLoad NameLoad body Return NameLoad Return SubscriptLoad NameLoad Index Str decorator_list FunctionDef arguments args NameParam defaults body While CompareGt Call NameLoad NameLoad Num body Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Num Assign TupleStore NameStore NameStore NameLoad If CompareEq NameLoad Num body Expr Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body With Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Str NameStore body For TupleStore NameStore NameStore NameStore NameLoad body Assign TupleStore NameStore NameStore NameLoad If CompareNotEq NameLoad Num body Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str decorator_list FunctionDef arguments args defaults body Assign NameStore Call NameLoad If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body With Call NameLoad NameLoad Str NameStore body Return Call AttributeLoad NameLoad attr NameLoad orelse Print Print Str Print Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad Assign NameStore ListComp NameLoad comprehension NameStore NameLoad CompareEq SubscriptLoad SubscriptLoad NameLoad Index Num Index Num Num If CompareGt Call NameLoad NameLoad Num body With Call NameLoad NameLoad Str NameStore body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad orelse If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args defaults body Assign NameStore Call NameLoad If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body TryExcept body Assign NameStore Call NameLoad handlers ExceptHandler type NameLoad body Print Str Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Print Str Print BinOpMod Str TupleLoad NameLoad Call NameLoad NameLoad Print Str Print Str Assign NameStore Call NameLoad Str If Call AttributeLoad Str attr Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad orelse Return NameLoad orelse Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad NameLoad body Assign NameStore ListLoad TryFinally body TryExcept body For NameStore NameLoad body Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Expr Call NameLoad NameLoad NameLoad Expr Call NameLoad NameLoad If Call NameLoad NameLoad body Expr Call NameLoad NameLoad handlers ExceptHandler type NameLoad body Print Print Str finalbody Expr Call NameLoad NameLoad decorator_list FunctionDef arguments args defaults body Assign NameStore Call NameLoad TryFinally body TryExcept body If BoolOpAnd NameLoad Call NameLoad NameLoad body Expr Call NameLoad NameLoad handlers ExceptHandler type NameLoad body Print Print Str finalbody Expr Call NameLoad NameLoad decorator_list
N/A N/A 
Enumerates and downloads agency submission files from USASpending.gov.

Enumeration is done by using the AJAX JSON API that the site uses to
construct the table of files.
 N/A os N/A sys N/A pickle N/A urllib N/A urllib2 N/A demjson N/A stream datetime date time datetime BeautifulSoup BeautifulSoup utils pretty_bytes timeliness DATA_DIR DOWNLOAD_DIR FISCAL_YEARS N/A URL_PATTERN N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A http://www.usaspending.gov/trendslist ?type=1 &trendreport=assistance &fiscalyear=%(fiscalyear)s &agencySelect= &startdate=%(startdate)s &enddate=%(enddate)s &page=%(page)s &rp=50 &sortname=agency_submission_date &sortorder=desc &query= &qtype= N/A DATA_URL_BASE http://www.usaspending.gov request_listing N/A N/A fy startdate enddate N/A N/A N/A json N/A N/A demjson JSON N/A params N/A fiscalyear startdate enddate page fy N/A startdate N/A N/A startdate strftime %Y-%m-%d N/A enddate N/A N/A enddate strftime %Y-%m-%d 1 N/A rows_processed 0 N/A listing N/A N/A N/A N/A N/A N/A N/A sys stdout write N/A Listing agency submission files for fiscal year %s fy N/A True N/A N/A N/A N/A N/A sys stdout write . N/A N/A N/A N/A sys stdout flush N/A url N/A URL_PATTERN params N/A response N/A N/A urllib2 urlopen url N/A N/A N/A response code 200 N/A N/A response_obj N/A N/A json decode N/A N/A response read N/A row_count N/A len N/A response_obj N/A rows N/A total_rows N/A response_obj N/A total N/A row N/A response_obj N/A rows N/A N/A N/A agency html datetime N/A row N/A cell N/A href N/A extract_href html N/A href N/A N/A N/A N/A listing append N/A href datetime N/A rows_processed 1 N/A N/A rows_processed total_rows N/A N/A N/A N/A N/A params N/A page 1 N/A N/A N/A Exception N/A Unexpected response status code: %s N/A str N/A response code N/A N/A N/A N/A sys stdout write N/A listing N/A N/A N/A N/A urllib2 URLError N/A err N/A N/A N/A URLError encountered for fiscal year %s fy N/A N/A Problematic URL: %s url N/A N/A update_download_schedule N/A N/A schedule listing N/A N/A N/A schedule is a list of 3-tuples:
         (href, datetime, (head-status, content-length))
       listing is a list of 2-tuples:
         (href, datetime)
       This function submits a HEAD request for each href,
       storing the the status code and content-length value
       obtained from the server.
        obtain_head N/A N/A href N/A N/A N/A url N/A url_for_href href N/A request N/A N/A urllib2 Request url N/A N/A request get_method N/A N/A N/A N/A HEAD N/A N/A N/A response N/A N/A urllib2 urlopen request N/A N/A N/A response code 200 N/A N/A response_info N/A N/A response info N/A content_length N/A int N/A response_info N/A Content-Length N/A N/A N/A response code content_length N/A N/A N/A N/A response code None N/A N/A N/A N/A urllib2 HTTPError N/A http_err N/A N/A N/A N/A http_err code None N/A N/A N/A N/A N/A sys stdout write Determining files to download N/A N/A href datetime listing N/A N/A N/A N/A N/A sys stdout write . N/A N/A N/A N/A sys stdout flush N/A destination_path N/A data_file_destination_path href N/A N/A N/A N/A os path exists destination_path N/A N/A N/A head N/A obtain_head href N/A N/A N/A schedule append N/A href datetime head N/A N/A N/A N/A sys stdout write N/A confirm_download_schedule N/A N/A schedule N/A N/A N/A Reports the total number of bytes and total number of files
    to download. Also lists the inaccessible files (based on HEAD
    response). Then asks user to confirm downloading.
     content_length N/A N/A tpl N/A N/A N/A N/A N/A tpl N/A 2 N/A 1 N/A status_code N/A N/A tpl N/A N/A N/A N/A N/A tpl N/A 2 N/A 0 N/A href N/A N/A tpl N/A N/A N/A N/A tpl N/A 0 N/A is_OK N/A N/A tpl N/A N/A N/A N/A N/A status_code tpl 200 N/A not_OK N/A N/A tpl N/A N/A N/A N/A N/A status_code tpl 200 N/A N/A increment N/A N/A N/A x _ N/A N/A x 1 N/A file_count N/A N/A schedule N/A N/A stream filter is_OK N/A N/A stream reduce increment 0 N/A bytes_to_download N/A N/A N/A schedule N/A N/A stream filter is_OK N/A N/A stream map content_length sum N/A inaccessible_files N/A N/A schedule N/A N/A stream filter not_OK list N/A N/A N/A len inaccessible_files 0 N/A N/A N/A Some files are inaccessible: N/A N/A idx sched N/A enumerate inaccessible_files N/A N/A N/A %d: %d %s N/A idx N/A status_code sched N/A href sched N/A N/A bytes_to_download 0 N/A N/A N/A N/A Need to download %s in %d files. N/A N/A pretty_bytes bytes_to_download file_count N/A N/A Are you sure you want to continue? [Y/n] N/A user_input N/A raw_input >  N/A N/A N/A N/A user_input upper N/A Y YES N/A N/A N/A Nothing to download. N/A False N/A data_file_destination_path N/A N/A href N/A N/A N/A N/A N/A N/A os path join DOWNLOAD_DIR N/A href N/A 1 N/A url_for_href N/A N/A href N/A N/A N/A N/A DATA_URL_BASE N/A N/A urllib quote href N/A download_data_file N/A N/A href N/A N/A N/A destination_path N/A data_file_destination_path href N/A N/A N/A url N/A url_for_href href N/A response N/A N/A urllib2 urlopen url N/A N/A N/A response code 200 N/A N/A destination_dir N/A N/A N/A os path dirname destination_path N/A N/A N/A N/A N/A os path exists destination_dir N/A N/A N/A N/A os makedirs destination_dir N/A N/A file destination_path w destination_file N/A N/A N/A N/A destination_file write N/A N/A response read N/A N/A N/A Exception N/A Unexpected response code %s for data file: %s N/A N/A str N/A response code destination_path N/A N/A N/A KeyboardInterrupt N/A N/A N/A N/A os remove destination_path N/A N/A extract_href N/A N/A html N/A N/A N/A soup N/A BeautifulSoup html N/A anchor N/A N/A soup find a N/A N/A anchor None N/A N/A None N/A N/A anchor N/A href N/A exec_download_schedule N/A N/A schedule N/A N/A N/A N/A N/A len schedule 0 N/A N/A N/A href datetime head N/A N/A schedule pop 0 N/A N/A code content_length head N/A N/A code 200 N/A N/A N/A download_data_file href N/A log_inaccessible_files N/A N/A schedule N/A N/A N/A N/A file N/A N/A N/A os path join DATA_DIR missing.txt w missing_file N/A N/A N/A href datetime head schedule N/A N/A N/A code content_length head N/A N/A code 200 N/A N/A N/A N/A missing_file write N/A url_for_href href N/A N/A N/A missing_file write N/A schedule_file_path N/A N/A N/A N/A N/A N/A N/A N/A os path join DATA_DIR schedule.pickle N/A restore_schedule N/A N/A N/A N/A N/A schedule_path N/A schedule_file_path N/A N/A N/A N/A os path exists schedule_path N/A N/A N/A file schedule_path r schedule_file N/A N/A N/A N/A pickle load schedule_file N/A N/A N/A No download schedule found. N/A Try the crawl or refresh commands. N/A None N/A save_schedule N/A N/A schedule N/A N/A N/A schedule_path N/A schedule_file_path N/A schedule N/A entry N/A entry schedule N/A N/A N/A entry N/A 2 N/A 0 200 N/A N/A N/A len schedule 0 N/A N/A N/A file schedule_path w schedule_file N/A N/A N/A N/A pickle dump schedule schedule_file N/A N/A N/A N/A N/A os path exists schedule_path N/A N/A N/A N/A os remove schedule_path N/A offer_resume N/A N/A N/A N/A N/A schedule_path N/A schedule_file_path N/A N/A N/A N/A os path exists schedule_path N/A N/A N/A N/A schedule N/A restore_schedule N/A N/A N/A EOFError N/A N/A Deleting corrupt download schedule. N/A N/A N/A os remove schedule_path N/A False N/A size N/A N/A N/A os path getsize schedule_path N/A A download schedule file exists: N/A N/A     %s (%s) N/A schedule_path N/A pretty_bytes size N/A You can either resume or delete this schedule. N/A What would you like to do? N/A user_input N/A raw_input [r]esume or [d]elete>  N/A N/A N/A DELETE startswith N/A N/A user_input upper N/A N/A N/A N/A os remove schedule_path N/A False N/A N/A True N/A N/A False N/A crawl_main N/A N/A startdate enddate N/A None None N/A N/A download_schedule N/A N/A N/A N/A N/A N/A fy FISCAL_YEARS N/A N/A listing N/A request_listing fy startdate enddate N/A N/A update_download_schedule download_schedule listing N/A N/A log_inaccessible_files download_schedule N/A N/A confirm_download_schedule download_schedule N/A N/A N/A exec_download_schedule download_schedule N/A N/A N/A KeyboardInterrupt N/A N/A N/A Exiting due to CTRL-C N/A N/A N/A save_schedule download_schedule N/A resume_main N/A N/A N/A N/A N/A download_schedule N/A restore_schedule N/A N/A N/A N/A N/A N/A download_schedule N/A confirm_download_schedule download_schedule N/A N/A N/A exec_download_schedule download_schedule N/A N/A N/A KeyboardInterrupt N/A N/A N/A Exiting due to CTRL-C N/A N/A N/A save_schedule download_schedule N/A