Module Assign NameStore Str Expr Str ImportFrom alias alias alias alias alias alias alias alias ImportFrom alias ImportFrom alias FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assert CompareLt NameLoad Num Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad NameLoad For NameStore Call NameLoad Num Call NameLoad BinOpDiv Num BinOpSub Num NameLoad body Assign NameStore Call NameLoad NameLoad NameLoad AugAssignAdd NameStore BinOpMult BinOpPow NameLoad NameLoad Call NameLoad NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Assign NameStore BinOpAdd BinOpMult NameLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad TupleLoad NameLoad NameLoad For NameStore Call NameLoad NameLoad body For NameStore Call NameLoad NameLoad body Assign SubscriptStore NameLoad Index TupleLoad NameLoad NameLoad Call NameLoad SubscriptLoad SubscriptLoad NameLoad Index NameLoad Index NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num For TupleStore NameStore NameStore Call NameLoad AttributeLoad NameLoad attr body AugAssignAdd NameStore BinOpMult SubscriptLoad NameLoad Index NameLoad AttributeLoad Call NameLoad NameLoad NameLoad Num attr Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore BinOpAdd BinOpMult NameLoad NameLoad NameLoad Assign NameStore Call NameLoad TupleLoad NameLoad NameLoad For NameStore Call NameLoad NameLoad body Assign NameStore Call NameLoad ListComp Call NameLoad SubscriptLoad NameLoad ExtSlice Index NameLoad Slice NameLoad comprehension NameStore NameLoad For NameStore NameLoad body Assign SubscriptStore NameLoad Index TupleLoad NameLoad NameLoad BinOpDiv Num Call NameLoad NameLoad Return TupleLoad NameLoad Call NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num Assign NameStore Call NameLoad TupleLoad NameLoad NameLoad For NameStore Call NameLoad NameLoad body Assign NameStore Call NameLoad SubscriptLoad NameLoad Index NameLoad For NameStore NameLoad body Assign SubscriptStore NameLoad Index TupleLoad NameLoad NameLoad BinOpDiv Num Call NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num Return TupleLoad BinOpDiv Call NameLoad TupleLoad NameLoad NameLoad Call NameLoad NameLoad Call NameLoad Call NameLoad NameLoad keyword Num decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad SubscriptLoad NameLoad Index Num Assign NameStore Call AttributeLoad BinOpMult Call NameLoad NameLoad NameLoad attr NameLoad Assign NameStore Call NameLoad TupleLoad NameLoad NameLoad For TupleStore NameStore NameStore NameLoad body Assign SubscriptStore NameLoad Index TupleLoad NameLoad NameLoad Num Return TupleLoad NameLoad Call NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad Num body Expr Str If CompareIs NameLoad NameLoad body Assign TupleStore NameStore NameStore Call NameLoad NameLoad orelse Assign NameStore NameLoad Assign NameStore Call NameLoad NameLoad NameLoad If CompareIs NameLoad NameLoad body Assign NameStore Lambda arguments args NameParam defaults Call NameLoad NameLoad NameLoad NameLoad While CompareGt NameLoad Num body Assign NameStore Call NameLoad NameLoad Assign TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad NameLoad NameLoad If CompareLt Call NameLoad Call NameLoad Call NameLoad BinOpSub NameLoad NameLoad Num body Return TupleLoad NameLoad NameLoad Assign NameStore NameLoad AugAssignSub NameStore Num Return TupleLoad NameLoad NameLoad decorator_list
N/A N/A __author__ Tom Schaul, tom@idsia.ch N/A  
Doing RL when an environment model (transition matrices and rewards) are available.


Representation:
 - a policy is a 2D-array of probabilities, 
    one row per state (summing to 1), one column per action.
 
 - a transition matrix (T) maps from originating states to destination states 
    (probabilities in each row sum to 1).
 
 - a reward vector (R) maps each state to the reward value obtained when entering (or staying in) a state.
 
 - a feature map (fMap) is a 2D array of features, one row per state.
 
 - a task model is defined by a list of transition matrices (Ts), one per action, a 
    reward vector R, a discountFactor
    
Note: a task model combined with a policy is again a transition matrix ("collapsed" dynamics).
        
 - a value function (V) is a vector of expected discounted rewards (one per state).
 
 - a set of state-action values (Qs) is a 2D array, one row per action.
 
 scipy dot zeros zeros_like ones mean array ravel rand numpy.matlib repmat pybrain.utilities all_argmax trueValues N/A N/A T R discountFactor N/A N/A N/A  Compute the true discounted value function for each state,
    given a policy (encoded as collapsed transition matrix).  N/A N/A discountFactor 1 N/A distr N/A N/A T copy N/A res N/A dot T R N/A i N/A range 1 N/A int N/A 10 N/A 1.0 discountFactor N/A N/A distr N/A dot distr T N/A res N/A N/A discountFactor i N/A dot distr R N/A res N/A trueQValues N/A N/A Ts R discountFactor policy N/A N/A N/A  The true Q-values, given a model and a policy.  N/A T N/A collapsedTransitions Ts policy N/A V N/A trueValues T R discountFactor N/A Vnext N/A N/A V discountFactor R N/A numA N/A len Ts N/A dim N/A len R N/A Qs N/A zeros N/A dim numA N/A si N/A range dim N/A N/A a N/A range numA N/A N/A N/A Qs N/A N/A si a N/A dot N/A N/A Ts N/A a N/A si Vnext N/A Qs N/A collapsedTransitions N/A N/A Ts policy N/A N/A N/A  Collapses a list of transition matrices (one per action) and a list 
        of action probability vectors into a single transition matrix. N/A res N/A zeros_like N/A Ts N/A 0 N/A dim N/A len N/A Ts N/A 0 N/A N/A ai ap N/A enumerate N/A policy T N/A N/A res N/A N/A Ts N/A ai N/A N/A repmat ap dim 1 T N/A res N/A greedyPolicy N/A N/A Ts R discountFactor V N/A N/A N/A  Find the greedy policy, (soft tie-breaking)
    given a value function and full transition model.  N/A dim N/A len V N/A numA N/A len Ts N/A Vnext N/A N/A V discountFactor R N/A policy N/A zeros N/A dim numA N/A si N/A range dim N/A N/A actions N/A all_argmax N/A N/A dot N/A T N/A N/A si N/A Vnext N/A T Ts N/A a actions N/A N/A N/A policy N/A N/A si a N/A 1.0 N/A len actions N/A N/A policy N/A collapsedTransitions Ts policy N/A greedyQPolicy N/A N/A Qs N/A N/A N/A  Find the greedy deterministic policy, 
    given the Q-values.  N/A dim N/A len Qs N/A numA N/A len N/A Qs N/A 0 N/A policy N/A zeros N/A dim numA N/A si N/A range dim N/A N/A actions N/A all_argmax N/A Qs N/A si N/A a actions N/A N/A N/A policy N/A N/A si a N/A 1.0 N/A len actions N/A policy N/A randomPolicy N/A N/A Ts N/A N/A N/A  Each action is equally likely.  N/A numA N/A len Ts N/A dim N/A len N/A Ts N/A 0 N/A N/A N/A N/A ones N/A dim numA N/A float numA N/A mean N/A array Ts axis 0 N/A randomDeterministic N/A N/A Ts N/A N/A N/A  Pick a random deterministic action for each state.  N/A numA N/A len Ts N/A dim N/A len N/A Ts N/A 0 N/A choices N/A N/A N/A N/A rand dim numA astype int N/A policy N/A zeros N/A dim numA N/A N/A si a choices N/A N/A N/A policy N/A N/A si a 1 N/A N/A policy N/A collapsedTransitions Ts policy N/A policyIteration N/A N/A Ts R discountFactor VEvaluator initpolicy maxIters N/A None None 20 N/A N/A  Given transition matrices (one per action),
    produce the optimal policy, using the policy iteration algorithm.
    
    A custom function that maps policies to value functions can be provided.  N/A N/A initpolicy None N/A N/A N/A policy T N/A randomPolicy Ts N/A N/A policy initpolicy N/A T N/A collapsedTransitions Ts policy N/A N/A VEvaluator None N/A N/A VEvaluator N/A N/A N/A T N/A N/A trueValues T R discountFactor N/A N/A maxIters 0 N/A N/A V N/A VEvaluator T N/A N/A newpolicy T N/A greedyPolicy Ts R discountFactor V N/A N/A N/A sum N/A ravel N/A abs N/A newpolicy policy 0.001 N/A N/A N/A policy T N/A policy newpolicy N/A maxIters 1 N/A N/A policy T N/A