Module Expr Str Import alias Import alias Import alias Import alias Import alias identifier Import alias Import alias Expr Call AttributeLoad NameLoad attr Num ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr BinOpAdd BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr NameLoad Call AttributeLoad Call AttributeLoad Str attr AttributeLoad AttributeLoad NameLoad attr attr attr Str Str Str Call AttributeLoad Call AttributeLoad Str attr AttributeLoad AttributeLoad NameLoad attr attr attr Str Str Assign AttributeStore NameLoad attr BinOpAdd AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr BinOpAdd AttributeLoad NameLoad attr Str TryExcept body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword NameLoad keyword BinOpMult AttributeLoad NameLoad attr Num keyword AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr AttributeLoad NameLoad attr If CompareNotEq AttributeLoad NameLoad attr Num body Assign AttributeStore NameLoad attr BinOpDiv BinOpSub AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr orelse Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr Str Num Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr Str Num Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Assign NameStore Call AttributeLoad Str attr NameLoad NameLoad AttributeLoad NameLoad attr BinOpDiv AttributeLoad NameLoad attr Call NameLoad NameLoad NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr If CompareIsNot AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad Str attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr If CompareIsNot AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Num AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Import alias identifier For NameStore Call NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Num AttributeLoad NameLoad attr BinOpAdd NameLoad Num Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad ExtSlice Index NameLoad Slice Slice keyword Str keyword Str Expr Call AttributeLoad NameLoad attr keyword Str keyword Str keyword Num For NameStore Call NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Num AttributeLoad NameLoad attr BinOpAdd NameLoad Num Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad ExtSlice Index NameLoad Slice Slice keyword Str keyword Str Expr Call AttributeLoad NameLoad attr keyword Str keyword Str keyword Num Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str AugAssignAdd AttributeStore NameLoad attr Num If AttributeLoad NameLoad attr body AugAssignAdd AttributeStore NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Num NameLoad Call AttributeLoad NameLoad attr NameLoad Num Num orelse If CompareGt Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr BinOpSub AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad Num Num If CompareEq BinOpMod AttributeLoad NameLoad attr AttributeLoad NameLoad attr Num body Assign NameStore Call AttributeLoad NameLoad attr AugAssignAdd AttributeStore NameLoad attr Num Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad Num Num Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad NameLoad If CompareGtE AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad orelse Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Num AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign TupleStore NameStore NameStore NameStore NameStore NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Expr Str AugAssignAdd AttributeStore NameLoad attr NameLoad AugAssignAdd AttributeStore NameLoad attr Num Assign NameStore BinOpSub Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr If AttributeLoad NameLoad attr body If BoolOpOr NameLoad CompareEq AttributeLoad NameLoad attr Num body AugAssignAdd AttributeStore NameLoad attr Num AugAssignAdd AttributeStore NameLoad attr AttributeLoad NameLoad attr orelse Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Num Num NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr BinOpDiv AttributeLoad NameLoad attr NameLoad If CompareGt AttributeLoad NameLoad attr Num body Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore BinOpAdd BinOpAdd Str Call NameLoad NameLoad Str Assign NameStore BinOpAdd AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Num Expr Call AttributeLoad NameLoad attr If CompareIsNot AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Num decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign NameStore Num If BoolOpAnd CompareIs AttributeLoad NameLoad attr NameLoad CompareGt Call NameLoad AttributeLoad NameLoad attr NameLoad body Assign AttributeStore NameLoad attr SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Index Num Assign NameStore Num If CompareIsNot AttributeLoad NameLoad attr NameLoad body For NameStore Call NameLoad NameLoad body AugAssignAdd NameStore Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad AttributeLoad NameLoad attr ExtSlice Index NameLoad Ellipsis Expr Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr BinOpDiv NameLoad NameLoad decorator_list decorator_list If CompareEq NameLoad Str body Pass
N/A N/A 
The NeuralAgent class wraps a deep Q-network for training and testing
in the Arcade learning environment.

Author: Nathan Sprague

 N/A os N/A cPickle N/A time N/A logging N/A numpy np N/A ale_data_set N/A sys N/A N/A N/A sys setrecursionlimit 10000 NeuralAgent N/A object N/A __init__ N/A N/A self q_network epsilon_start epsilon_min epsilon_decay replay_memory_size exp_pref replay_start_size update_frequency rng s3_utility N/A N/A N/A N/A self network q_network N/A N/A self epsilon_start epsilon_start N/A N/A self epsilon_min epsilon_min N/A N/A self epsilon_decay epsilon_decay N/A N/A self replay_memory_size replay_memory_size N/A N/A self exp_pref exp_pref N/A N/A self replay_start_size replay_start_size N/A N/A self update_frequency update_frequency N/A N/A self rng rng N/A N/A self s3_utility s3_utility N/A N/A self phi_length N/A N/A self network num_frames N/A N/A self image_width N/A N/A self network input_width N/A N/A self image_height N/A N/A self network input_height N/A time_str N/A N/A time strftime _%m-%d-%H-%M_ N/A N/A time gmtime N/A N/A self exp_dir N/A N/A N/A N/A N/A self exp_pref time_str N/A N/A N/A N/A {} format N/A N/A self network lr replace . p _ N/A N/A N/A N/A {} format N/A N/A self network discount replace . p N/A N/A self results_filepath N/A N/A self exp_dir /results.csv N/A N/A self learning_filepath N/A N/A self exp_dir /learning.csv N/A N/A N/A N/A N/A os stat N/A self exp_dir N/A N/A N/A OSError N/A N/A N/A N/A os makedirs N/A self exp_dir N/A N/A self num_actions N/A N/A self network num_actions N/A N/A self data_set N/A N/A ale_data_set DataSet width N/A self image_width height N/A self image_height rng rng max_steps N/A self replay_memory_size phi_length N/A self phi_length N/A N/A self test_data_set N/A N/A ale_data_set DataSet width N/A self image_width height N/A self image_height rng rng max_steps N/A N/A self phi_length 2 phi_length N/A self phi_length N/A N/A self epsilon N/A self epsilon_start N/A N/A N/A self epsilon_decay 0 N/A N/A N/A self epsilon_rate N/A N/A N/A self epsilon_start N/A self epsilon_min N/A self epsilon_decay N/A N/A N/A self epsilon_rate 0 N/A N/A self testing False N/A N/A N/A self _open_results_file N/A N/A N/A self _open_learning_file N/A N/A self episode_counter 0 N/A N/A self batch_counter 0 N/A N/A self holdout_data None N/A N/A self last_img None N/A N/A self last_action None N/A _open_results_file N/A N/A self N/A N/A N/A N/A self results_file N/A open N/A self results_filepath w 0 N/A N/A N/A N/A self results_file write epoch,num_episodes,total_reward,reward_per_epoch,mean_q
 N/A N/A N/A N/A self results_file flush N/A _open_learning_file N/A N/A self N/A N/A N/A N/A self learning_file N/A open N/A self learning_filepath w 0 N/A N/A N/A N/A self learning_file write mean_loss,epsilon
 N/A N/A N/A N/A self learning_file flush N/A _update_results_file N/A N/A self epoch num_episodes holdout_sum N/A N/A N/A out N/A N/A {},{},{},{},{}
 format epoch num_episodes N/A self total_reward N/A N/A self total_reward N/A float num_episodes holdout_sum N/A N/A N/A N/A self results_file write out N/A N/A N/A N/A self results_file flush N/A N/A N/A self s3_utility None N/A N/A N/A N/A N/A self s3_utility upload_file results.csv N/A self results_filepath N/A _update_learning_file N/A N/A self N/A N/A N/A out N/A N/A {},{}
 format N/A N/A np mean N/A self loss_averages N/A self epsilon N/A N/A N/A N/A self learning_file write out N/A N/A N/A N/A self learning_file flush N/A N/A N/A self s3_utility None N/A N/A N/A N/A N/A self s3_utility upload_file learning.csv N/A self learning_filepath N/A start_episode N/A N/A self observation N/A N/A N/A 
        This method is called once at the beginning of each episode.
        No reward is provided, because reward is only available after
        an action has been taken.

        Arguments:
           observation - height x width numpy array

        Returns:
           An integer action
         N/A N/A self step_counter 0 N/A N/A self batch_counter 0 N/A N/A self episode_reward 0 N/A N/A self loss_averages N/A N/A N/A self start_time N/A N/A time time N/A return_action N/A N/A N/A self rng randint 0 N/A self num_actions N/A N/A self last_action return_action N/A N/A self last_img observation N/A return_action N/A _show_phis N/A N/A self phi1 phi2 N/A N/A N/A matplotlib.pyplot plt N/A p N/A range N/A self phi_length N/A N/A N/A N/A plt subplot 2 N/A self phi_length N/A p 1 N/A N/A N/A plt imshow N/A phi1 N/A N/A p N/A N/A interpolation none cmap gray N/A N/A N/A plt grid color r linestyle - linewidth 1 N/A p N/A range N/A self phi_length N/A N/A N/A N/A plt subplot 2 N/A self phi_length N/A p 5 N/A N/A N/A plt imshow N/A phi2 N/A N/A p N/A N/A interpolation none cmap gray N/A N/A N/A plt grid color r linestyle - linewidth 1 N/A N/A N/A plt show N/A step N/A N/A self reward observation N/A N/A N/A 
        This method is called each time step.

        Arguments:
           reward      - Real valued reward.
           observation - A height x width numpy array

        Returns:
           An integer action.

         N/A N/A self step_counter 1 N/A N/A self testing N/A N/A N/A self episode_reward reward N/A action N/A N/A self _choose_action N/A self test_data_set 0.05 observation N/A N/A np clip reward -1 1 N/A N/A N/A N/A len N/A self data_set N/A self replay_start_size N/A N/A N/A self epsilon N/A max N/A self epsilon_min N/A N/A self epsilon N/A self epsilon_rate N/A action N/A N/A self _choose_action N/A self data_set N/A self epsilon observation N/A N/A np clip reward -1 1 N/A N/A N/A N/A self step_counter N/A self update_frequency 0 N/A N/A loss N/A N/A self _do_training N/A N/A self batch_counter 1 N/A N/A N/A N/A self loss_averages append loss N/A N/A action N/A N/A self _choose_action N/A self data_set N/A self epsilon observation N/A N/A np clip reward -1 1 N/A N/A self last_action action N/A N/A self last_img observation N/A action N/A _choose_action N/A N/A self data_set epsilon cur_img reward N/A N/A N/A 
        Add the most recent data to the data set and choose
        an action based on the current policy.
         N/A N/A N/A data_set add_sample N/A self last_img N/A self last_action reward False N/A N/A N/A self step_counter N/A self phi_length N/A N/A phi N/A N/A data_set phi cur_img N/A action N/A N/A N/A self network choose_action phi epsilon N/A N/A action N/A N/A N/A self rng randint 0 N/A self num_actions N/A action N/A _do_training N/A N/A self N/A N/A N/A 
        Returns the average loss for the current batch.
        May be overridden if a subclass needs to train the network
        differently.
         N/A N/A states actions rewards next_states terminals N/A N/A N/A self data_set random_batch N/A N/A self network batch_size N/A N/A N/A N/A self network train states actions rewards next_states terminals N/A end_episode N/A N/A self reward terminal N/A True N/A N/A 
        This function is called once at the end of an episode.

        Arguments:
           reward      - Real valued reward.
           terminal    - Whether the episode ended intrinsically
                         (ie we didn't run out of steps)
        Returns:
            None
         N/A N/A self episode_reward reward N/A N/A self step_counter 1 N/A total_time N/A N/A N/A time time N/A self start_time N/A N/A self testing N/A N/A N/A terminal N/A N/A self episode_counter 0 N/A N/A N/A self episode_counter 1 N/A N/A self total_reward N/A self episode_reward N/A N/A N/A N/A N/A self data_set add_sample N/A self last_img N/A self last_action N/A N/A np clip reward -1 1 True N/A N/A N/A logging info N/A N/A steps/second: {:.2f} format N/A N/A self step_counter total_time N/A N/A N/A self batch_counter 0 N/A N/A N/A N/A self _update_learning_file N/A N/A N/A logging info N/A N/A average loss: {:.4f} format N/A N/A np mean N/A self loss_averages N/A finish_epoch N/A N/A self epoch N/A N/A N/A net_filename N/A N/A /network_file_ N/A str epoch .pkl N/A net_filepath N/A N/A self exp_dir net_filename N/A net_file N/A open net_filepath w N/A N/A N/A cPickle dump N/A self network net_file -1 N/A N/A N/A net_file close N/A N/A N/A self s3_utility None N/A N/A N/A N/A N/A self s3_utility upload_file net_filename net_filepath N/A start_testing N/A N/A self N/A N/A N/A N/A self testing True N/A N/A self total_reward 0 N/A N/A self episode_counter 0 N/A finish_testing N/A N/A self epoch N/A N/A N/A N/A self testing False N/A holdout_size 3200 N/A N/A N/A N/A self holdout_data None N/A N/A len N/A self data_set holdout_size N/A N/A N/A self holdout_data N/A N/A N/A N/A self data_set random_batch holdout_size N/A 0 N/A holdout_sum 0 N/A N/A N/A self holdout_data None N/A N/A i N/A range holdout_size N/A N/A holdout_sum N/A N/A np max N/A N/A N/A self network q_vals N/A N/A self holdout_data N/A N/A i N/A N/A N/A N/A self _update_results_file epoch N/A self episode_counter N/A holdout_sum holdout_size N/A N/A N/A N/A __name__ __main__ N/A N/A