Original: Bayesian Risk Markov Decision Processes
Revised: Bayesian Risk Markov Decision Processes

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We consider finite-horizon Markov Decision Processes wheredistributional parameters, such as transition probabilities, are unknown and estimated from data.
Revised: We consider finite-horizon Markov Decision Processes where parameters, such astransition probabilities, are unknown and estimated from data.

Original: Thepopular distributionally robust approach to addressing the parameter uncertaintycan sometimes be overly conservative.
Revised: The popular distributionally robust approach to addressing the parameter uncertainty can sometimesbe overly conservative.

Original: In this paper, we propose a new formulation, Bayesian risk Markov decision process (BR-MDP), to address parameter uncertainty in MDPs, where a risk functional is applied in nested form to the expectedtotal cost with respect to the Bayesian posterior distributions of the unknown parameters.
Revised: In this paper, we propose a new formulation, Bayesianrisk Markov decision process (BR-MDP), to address parameter uncertainty in MDPs, where a risk functional is applied in nested form to the expected total costwith respect to the Bayesian posterior distributions of the unknown parameters.

Original: The proposed formulation provides more flexible risk attitudes towardsparameter uncertainty and takes into account the availability of data in future timestages.
Revised: The proposed formulation provides more flexible risk attitudes towards parameteruncertainty and takes into account the availability of data in future time stages.

Original: To solve the proposed formulation with the conditional value-at-risk (CVaR)risk functional, we propose an efficient approximation algorithm by deriving ananalytical approximation of the value function and utilizing the convexity of CVaR.
Revised: To solve the proposed formulation with the conditional value-at-risk (CVaR) riskfunctional, we propose an efficient approximation algorithm by deriving an analytical approximation of the value function and utilizing the convexity of CVaR.

Original: We demonstrate the empirical performance of the BR-MDP formulation and theproposed algorithms on a gambler’s betting problem and an inventory controlproblem.
Revised: Wedemonstrate the empirical performance of the BR-MDP formulation and proposedalgorithms on a gambler’s betting problem and an inventory control problem.

Original:  Introduction
Revised:  Introduction

Original: Markov decision process (MDP) is a paradigm for modeling sequential decision making underuncertainty.
Revised: Markov decision process (MDP) is a paradigm for modeling sequential decision making underuncertainty.

Original: From a modeling perspective, some parameters of MDPs are unknown and need tobe estimated from data.
Revised: From a modeling perspective, some parameters of MDPs are unknown and need tobe estimated from data.

Original: In this paper, we consider MDPs where transition probability and costparameters are not known.
Revised: In this paper, we consider MDPs where transition probability and costparameters are not known.

Original: A natural question would be: given a finite and probably small set of data,how does a decision maker find an “optimal” policy that minimizes the expected total cost under theuncertain transition probability and cost parameters?
Revised: A natural question would be: given a finite and probably small set of data,how does a decision maker find an “optimal” policy that minimizes the expected total cost under theuncertain transition probability and cost parameters?

Original: A possible approach that mitigates the parameter uncertainty lies in the framework of  robust MDPs(e.g.
Revised: A possible approach that mitigates the parameter uncertainty (also known as epistemic uncertainty)lies in the framework of distributionally robust MDPs (DR-MDPs, [1]).

Original: In robust MDPs, parameters are usually assumed to belong to a known setreferred to as the ambiguity set, and the optimal decisions are chosen according to their performanceunder the worst possible parameter realizations within the ambiguity set. Later [6] extends thedistributionally robust approach to MDPs (DR-MDPs) with parameter uncertainty and utilizes theprobabilistic information of the unknown parameters. The distributionally robust approach originatesfrom stochastic optimization (e.g. [7], [8]), which regards the unknown parameters as randomvariables and assumes the associated distributions belong to an ambiguity set that is constructed fromthe data.
Revised: DR-MDP regards theunknown parameters as random variables and assumes the associated distributions belong to anambiguity set that is constructed from the data.

Original: DR-MDP then finds the optimal policy that minimizes the expected total cost with theparameters following the most adversarial(worst-case) distribution within the ambiguity set.
Revised: DR-MDP then finds the optimal policy that minimizesthe expected total cost with the parameters following the most adversarial distribution within theambiguity set.

Original: However, distributionally robust approaches might yield overly conservative solutions that performpoorly for scenarios that are more likely to happen than the worst case.
Revised: However, distributionally robust approaches might yield overly conservative solutionsthat perform poorly for scenarios that are more likely to happen than the worst case.

Original: In view of such drawback, [9,10] propose a Bayesian risk optimization (BRO) framework that reformulates a stochastic optimization problem with parameter uncertainty. They quantify theparameter uncertainty using a Bayesianposterior distribution as opposed to an ambiguity set, and impose a risk functional on  the objectivefunction with respect to  the posterior distribution.
Revised: Moreover, aspointed out in [2], DR-MDP does not explicitly specify the dynamics of the considered problem inthe sense that the distribution of the unknown parameters does not depend on realizations of the dataprocess, and therefore is generally not time consistent (we refer the reader to [2] for details on timeconsistency). In view of the aforementioned drawbacks of DR-MDP, we propose a new formulation named as

Original: The risk functionals provide more flexible attitudestowards risk than the conservative worst-case measure. BRO is essentially seeking a trade-off betweenthe posterior expected performance and the robustness in the actual performance, which is measuredby the width of confidence intervals for the solution’s actual performance.
Revised: Bayesian risk MDP (BR-MDP), to address the parameter uncertainty in MDPs. BR-MDP takesa similar perspective as Bayesian risk optimization (BRO), which is a new framework proposedby [3, 4] for static (single stage) optimization.

Original: Intuitively speaking,
Revised: They quantify the parameter uncertainty using a

Original: BRO framework tries to avoid a scenario where a solution performs well under the estimated modelbut performs badly under the true model, by possibly giving up some good expected performanceand trading for more confidence about the actual performance of a solution (in terms of width ofconfidence interval).
Revised: Bayesian posterior distribution to replace the ambiguity set in DRO, and impose a risk functional onthe objective function with respect to the posterior distribution.

Original: Apart from the overly conservative concern, as pointed out in [11], DR-MDPdoes not explicitly specify the dynamics of the considered problem, in the sense that the distributionof the unknown parameters does not depend on realizations of  the data process. In this paper, we take a similar perspective as BRO in [9, 10] and propose a new Bayesian riskformulation for MDPs (BR-MDPs) toaddress the parameter uncertainty in MDPs.
Revised: To extend to BRO to MDPs and alsoensure time consistency, we propose to use a nested risk functional taken with respect to the processof Bayesian posterior distributions, where the posterior is updated with all realizations of randomnessup to each time stage. We show the proposed BR-MDP formulation is time consistent, and derivethe corresponding dynamic programming equation with an augmented state that incorporates theposterior information. The proposed framework works for an offline planning problem, where thedecision maker, after loaded with the learned optimal policy and put to the real environment, acts likeit adapts to the environment.

Original: To clearly specifythe dynamics and the randomness in the system, we take the perspective of stochastic optimal controlto model the state transition bya difference equation that involves current state, next state, action, andthe randomness in the system (see Chapter 3.5 in [12]). We assume the distribution of the randomnessbelongs to some parametric family, and model the uncertainty over the unknown parameters via a 
Revised: To solve the proposed BR-MDP formulation, we develop an efficient algorithm by drawing aconnection between BR-MDP and partially observable MDP (POMDP) and utilizing the convexityof CVaR. The connection between BR-MDP and POMDP is motivated by the observation that theposterior distribution in BR-MDP is exactly like the belief state (which is the posterior distributionof the unobserved state given the history of observations) in a POMDP. The optimal value functionof a POMDP (in a minimization problem) can be expressed as an lower envelope of a set of linearfunctions (also called α -functions, see [5]).

Original: Bayesian posterior distribution that is updated based on the realization of the randomness at everytime stage. It is worth noting that applying the Bayesian approach to MDPs has been consideredin [13], which proposes a Bayes-adaptive MDP (BAMDP) formulation with an augmented statecomposed of  the underlying MDP state and the posterior distribution of the unknown parameters.
Revised: We show a similar α -function representation of thevalue function for BR-MDP, where the number of α -functions grow exponentially over time. Tohave computationally feasible algorithm, we further derive an analytical approximation of the valuefunction that keeps a constant number of α -functions over time. To summarize, the contributions of this paper are two folds.

Revised: First, we propose a new time-consistentformulation BR-MDP to handle the parameter uncertainty in MDPs.

Original:  In 
Revised: Second, we propose an efficientalgorithm to solve the proposed formulation with a CVaR risk functional, and the algorithm can beeasily extended to other coherent risk measures (see [6] for an overview on coherent risk measures).  Related Literature One possible approach that mitigates the parameter uncertainty in MDPs lies in the framework ofrobust MDPs (e.g. [7, 8, 9, 10, 11]). In robust MDPs, parameters are usually assumed to belong to aknown set referred to as the ambiguity set, and the optimal decisions are chosen according to theirperformance under the worst possible parameter realizations within the ambiguity set.

Revised: [1] furtherextends the distributionally robust approach to MDPs (DR-MDPs) with parameter uncertainty and utilizes the probabilistic information of the unknown parameters.

Revised: Different from all the aforementionedworks, we take a Bayesian perspective in the BR-MDP formulation and seek a trade-off between theposterior expected performance and the robustness in the actual performance.

Revised: It is worth noting that applying the Bayesian approach to MDPs has been considered in [12], whichproposes a Bayes-adaptive MDP (BAMDP) formulation with an augmented state composed of theunderlying MDP state and the posterior distribution of the unknown parameters.

Original:  BAMDP, each transition probability is treated as an unknown parameter associated with a Dirichletprior distribution, and an expectation is taken with respect to the Dirichlet posterior on the expectedtotal cost.
Revised: In BAMDP, eachtransition probability is treated as an unknown parameter associated with a Dirichlet prior distribution,and an expectation is taken with respect to the Dirichlet posterior on the expected total cost.

Original: In contrast, our BR-MDP formulation imposes a risk functional, taken with respect to theposterior distribution (which could be chosen as Dirichlet distribution but is more general), on theexpected total cost in a nested form.
Revised: Incontrast, our BR-MDP formulation imposes a risk functional, taken with respect to the posteriordistribution (which could be chosen as Dirichlet distribution but is more general), on the expectedtotal cost in a nested form.

Original: On a related note, risk-averse decision making has been widely studied in MDPs.
Revised: On a related note, risk-averse decision making has been widely studied in MDPs.

Original: Apart from therobust MDPs that address the parameter uncertainty (also termed as epistemic uncertainty), risksensitive MDPs (e.g. [14, 15, 16, 17]) address the intrinsic uncertainty (also termed as aleatoricuncertainty) that is due to the inherent stochasticity of the underlying MDP, by replacing the riskneutral expectation (with respect to the state transition) by general risk measures, such as conditionalvalue-at-risk (CVaR, see [18]).
Revised: Apart from the robust MDPs that address the parameter uncertainty, risk-sensitive MDPs (e.g.[13, 14, 15, 16]) address the intrinsic uncertainty (also known as aleatoric uncertainty) that is dueto the inherent stochasticity of the underlying MDP, by replacing the risk-neutral expectation (withrespect to the state transition) by general risk measures, such as conditional value-at-risk (CVaR, see[17]).

Original: Most of the existing literature on risk-sensitive MDPs consider a staticrisk functional applied to the total return (e.g. [19, 20, 21, 22]).
Revised: Most of the existing literature on risk-sensitive MDPs consider a static risk functional appliedto the total return (e.g. [18, 19, 20, 21]).

Original: There are two recent works closelyrelated to ours, both of which apply a risk functional to BAMDP.
Revised: There are two recent works closely related to ours, both ofwhich apply a risk functional to BAMDP.

Original: Specifically, [23] formulates the risksensitive planning problem as a two-player zero-sum game and then applies a (static) risk functionalto the expected total cost, which adds robustness to the incorrect priors over model parameters; [24]optimizes a CVaR risk functional over the total cost and simultaneously addresses both epistemic andaleatoric uncertainty.
Revised: Specifically, [22] formulates the risk-sensitive planningproblem as a two-player zero-sum game and then applies a static risk functional to the expected totalcost, which adds robustness to the incorrect priors over model parameters; [23] optimizes a CVaR riskfunctional over the total cost and simultaneously addresses both epistemic and aleatoric uncertainty.

Original: In contrast, we consider a nested risk functional and seek robustness with respectto the epistemic uncertainty. A primary motivation for considering such nested risk functional is theissue of time consistency (see [25, 26, 27, 11]) in the sense that the optimal solution to the formulatedproblem keeps optimal at every stage of the decision process with respect to the conditional riskfunctional. Optimizing a static risk measure can lead to“time-inconsistent” behavior, where theoptimal policy at the current time stage can become suboptimal in the next time stage simply becausea new piece of information is revealed (see [25, 26]). In this work, we show theproposed BR-MDPformulation is time consistent, and derive the corresponding dynamic programming equation withan augmented state that incorporates the posterior information.
Revised: In contrast, we consider a nested risk functional due to the time consistency consideration discussedin previous section.

Original: The proposed framework works foran offline planning problem, where the decision maker, after loaded with the learned optimal policyand put to the real environment, acts like it adapts to the environment.

Original: Also note that our problemsetting relies on partial knowledge of the model (state transition equation, cost function etc.) andworks in an offline planning setting with no interaction with the environment, and hence is differentfrom Bayesian reinforcement learning (e.g. [28, 29]).
Revised: Also note that our problem setting relies on partial knowledge of the model (state transition equation, cost function etc.) and works in an offline planning setting with no interactionwith the environment, and hence is different from Bayesian reinforcement learning (e.g. [24, 25]).

Original: To solve the proposed BR-MDP formulation, we develop an efficient algorithm by drawing asimilarity between BR-MDP and partially observable MDP (POMDP) and utilizing the convexityof CVaR.

Original: In a POMDP, the optimal value function (in a minimization problem) can be expressedas an lower envelope of a set of linear functions (also called α -functions, see [30]).

Original: We show asimilar α -function representation of the value function for BR-MDP, where the number of α -functionsgrow exponentially over time.

Original: To have computationally feasible algorithm, we derive an analyticalapproximation of the value function that keeps a constant number of α -functions over time.

Original: Notethat [31] also reformulates the BAMDP as a POMDP and proposes a point-based value iterationalgorithm.

Original: However, their approach is not applicable for the risk-averse case, as it relies on thepiecewise linearity of the optimal value function, which does not hold for risk functionals other thanexpectation.

Original: Additionally, [24] proposes an approximate algorithm based on Monte Carlo tree searchand Bayesian optimization to solve risk-averse BAMDP, but it works only for a static risk functionaland does not easily generalize to our formulation with a nested risk functional.

Original: To summarize, the contributions of this paper are two folds.

Original: First, we propose a Bayesian risk MDPformulation to handle the parameter uncertainty in MDPs.

Original: Second, we propose an efficient algorithmto solve the proposed formulation with a CVaR risk functional, and the algorithm can be easilyextended to other coherent risk measures (see [32] for an overview on coherent risk measures).

Original:  Preliminaries and problem formulation
Revised:  Preliminaries and Problem Formulation3.1 Preliminaries: BRO and CVaR

Original: 2.1 Preliminaries: Bayesian risk optimization and CVaR

Original: Bayesian risk optimization (BRO, see [9, 10]) considers a general stochastic optimization problem:min x E P c
Revised: Bayesian risk optimization (BRO, see [3, 4]) considers a general stochastic optimization problem:min x E P c

Original: [ h ( x, ξ )] , where x is the decision vector, ξ is a random vector with distribution P c , h isa deterministic cost function.
Revised: [ h ( x, ξ )] , where x is the decision vector, ξ is a random vector with distribution P c , h isa deterministic cost function.

Original: The true distribution P c is rarely known in practice and often needsto be estimated from data.
Revised: The true distribution P c is rarely known in practice and often needsto be estimated from data.

Original: It is very likely that the solution obtained from solving an estimatedmodel performs badly under the true model.
Revised: It is very likely that the solution obtained from solving an estimatedmodel performs badly under the true model.

Original: To avoid such a scenario, BRO seeks robustness inthe actual performance by imposing a risk functional to the expected cost function and solving thefollowing problem: min x ρ P n { E P θ [ h ( x, ξ )] } , where ρ is a risk functional, and P n is the posteriordistribution of θ after observing n data points.
Revised: To avoid such a scenario, BRO seeks robustness inthe actual performance by imposing a risk functional to the expected cost function and solving thefollowing problem: min x ρ P n { E P θ [ h ( x, ξ )] } , where ρ is a risk functional, and P n is the posteriordistribution of θ after observing n data points.

Original: It is assumed that the unknown distribution P c belongsto a parametric family { P θ | θ ∈ Θ } , where Θ is the parameter space and θ c ∈ Θ is the true parameter.
Revised: It is assumed that the unknown distribution P c belongsto a parametric family { P θ | θ ∈ Θ } , where Θ is the parameter space and θ c ∈ Θ is the unknown trueparameter value.

Original: Through a Bayesian perspective, θ c can be viewed as a realization of a random variable θ .
Revised: Taking a Bayesian perspective, θ c is viewed as a realization of a random vector θ .

Original: In particular, conditional value at risk (CVaR), a common coherent risk measure (see [32]), isconsidered for the risk functional.
Revised: In particular, conditional value-at-risk (CVaR), a common coherent risk measure (see [6]), is considered for the risk functional.

Original: For a random variable X defined on a probability space (Ω , F , P ) , VaR α ( X ) is defined as the α -quantile of X , i.e., VaR α ( X ) :
Revised: For a random variable X defined on a probability space (Ω , F , P ) ,value-at-risk VaR α ( X ) is defined as the α -quantile of X , i.e., VaR α ( X ) :

Original: = inf { t : P ( X ≤ t ) ≥ α } , wherethe confidence level α takes value in (0 , 1) .
Revised: = inf { t : P ( X ≤ t ) ≥ α } ,where the confidence level α ∈ (0 , 1) .

Original: Assuming there is no probability atom at VaR α ( X ) , CVaR at confidence level α is defined as the mean of the α -tail distribution of Z , i.e., CVaR α ( X ) 
Revised: Assuming there is no probability atom at VaR α ( X ) , CVaRat confidence level α is defined as the mean of the α -tail distribution of X , i.e., CVaR α ( X ) =

Original: = 121 E [ X | X ≥ VaR α ( X )] .
Revised: 107 E [ X | X ≥ VaR α ( X )] .

Original: It is shown in [18] that CVaR can be written as a convex optimization:
Revised: It is shown in [17] that CVaR can be written as a convex optimization:

Original: [Equation]
Revised: [Equation]

Revised: where ( · ) + stands for max(0 , · ) .

Original: 2.2 New formulation: Bayesian risk MDPs 
Revised: 3.2 New formulation: Bayesian risk MDPs (BR-MDPs)

Original: Consider a finite-horizon MDP defined as ( S , A , P , C ) , where S is the state space, A is the actionspace, P is the transition probability with P ( s t +1 | s t , a t ) denoting the probability of transitioning tostate s t +1 from state s t when action a t is taken, C is the cost function with C t ( s t , a t , s t +1 ) denotingthe cost at time stage t when action a t is taken and state transitions from s t to s t +1 .
Revised: Consider a finite-horizon MDP defined as ( S , A , P , C ) , where S is the state space, A is the actionspace, P is the transition probability with P ( s t +1 | s t , a t ) denoting the probability of transitioning tostate s t +1 from state s t when action a t is taken, C is the cost function with C t ( s t , a t , s t +1 ) denotingthe cost at time stage t when action a t is taken and state transitions from s t to s t +1 .

Original: A Markoviandeterministic policy π is a function mapping from S to A .
Revised: A Markoviandeterministic policy π is a function mapping from S to A .

Original: Given an initial state s 0 , the goal is tofind an optimal policy that minimizes the expected total cost: minπ E π, P , C (cid:104)(cid:80) [Equation] s
Revised: Given an initial state s 0 , the goal is tofind an optimal policy that minimizes the expected total cost: min

Original: [Equation]
Revised: [Equation]

Original: ) (cid:105) ,where E π, P , C is the expectation with policy π when the transition probability is P and the cost is C .
Revised: where E π, P , C is the expectation with policy π when the transition probability is P and the cost is C .

Original: In practice, P and C are often unknown and estimated from data.
Revised: In practice, P and C are often unknown and estimated from data.

Original: To deal with the parameter uncertainty in MDPs, we propose a  Bayesian risk MDP formulation thatextends the BROformulation in stochastic optimization to MDPs.
Revised: To deal with the parameter uncertainty in MDPs, we propose a new formulation, Bayesian risk MDP (BR-MDP), by extending BRO in static optimization to MDPs.

Original: We assume the state transitionis specified by the state equation s t +1 =
Revised: We assume the state transitionis specified by the state equation s t +1 =

Original: g t ( s t , a t , ξ t ) with a known transition function g t , whichinvolves state s t ∈ S  , action a t ∈ A  , andthe randomness ξ t ∈ Ξ in the system.
Revised: g t ( s t , a t , ξ t ) with a known transition function g t , whichinvolves state s t ∈ S ⊆ R s , action a t ∈ A ⊆ R a , and randomness ξ t ∈ Ξ ⊆ R k , wheres, a, k are the dimensions of the state, action, and randomness, respectively.

Original: We assume  ξtat time stages t = 0 , · · · , T − 1 to be independent and identically distributed.
Revised: We assume { ξ t , t =0 , · · · , T − 1 } are independent and identically distributed (i.i.d.).

Original: Note that the stateequation together with the distribution of ξ  uniquely determines the transition probability of the MDP,i.e., P ( s t +1  | s t , a t ) =
Revised: Note that the state equationtogether with the distribution of ξ t uniquely determines the transition probability of the MDP, i.e., P ( s t +1 ∈ S ′ | s t , a t ) =

Original: P (Ξ t ∈ { ξ t ∈ Ξ :  s t [-+
-]
Revised: P ( { ξ t ∈ Ξ : g t ( s t [-+
-], a t , ξ t )

Original: = g t ( s t , a t, ξ t ) } )  .
Revised: ∈ S ′ }| s t , a t ) , where S ′ is a measurable set in S .

Original: We refer the readers to Chapter3.5 in [12] for the equivalence between stochastic optimal control and MDP formulation.
Revised: We refer the readers to Chapter 3.5 in [26] for the equivalence between stochastic optimal control and MDP formulation.

Original: We usethe representation of state equation instead of transition probability in MDPs, for the purpose ofdecoupling the randomness and the policy, leading to a cleaner formulation in the nested form.
Revised: We use the representation of state equations instead of transition probabilities in MDPs, for the purpose of decoupling the randomness and the policy, leading to a cleaner formulationin the nested form.

Original: Weassume the distribution of ξ  , denoted by f ( · ; θ c ) , belongs to a parametric family { f ( · ; θ ) | θ ∈ Θ } ,where Θ  is the parameter space,  and θ c ∈ Θ is the true but unknown parameter value.
Revised: We assume the distribution of ξ t , denoted by f ( · ; θ c ) , belongs to a parametricfamily { f ( · ; θ ) | θ ∈ Θ } , where Θ ⊆ R d is the parameter space, d is the dimension of the parameterθ , and θ c ∈ Θ is the true but unknown parameter value.

Original: The parametricassumption implies that we have partial knowledge of the problem structure. For example, in inventorycontrol problems, the customer demand isunknown but often assumed to follow a Poisson process(see [33]).
Revised: The parametric assumption is satisfied inmany problems; for example, in inventory control the customer demand is often assumed to followa Poisson process (see [27]) with unknown arrival rate.

Original: The cost at time stage t is assumed to be a function of state s t , action a t , and randomnessξ t , i.e., C t ( s t , a t , ξ t ) .
Revised: The cost at time stage t is assumed to be afunction of state s t , action a t , and randomness ξ t , i.e., C t ( s t , a t , ξ t ) .

Original: Now given adata set, we can compute the prior distribution µ 0 over the support Θ , which captures theinitial uncertainty  in the parameter estimate.
Revised: We start with a prior distribution µ 0 over the parameter space Θ , which captures the initial uncertainty 136 in the parameter estimate from an initial data set and can also incorporate expert opinion.

Original: Then given an observed realization of the data process,we update the posterior distribution µ t according to the  Bayes’ rule.
Revised: Then givenan observed realization of the data process, we update the posterior distribution µ t according to the 138 Bayes’ rule.

Original: Let the policy be a sequence ofmappings fromphysical state s t and posterior µ t to the action space, i.e., π = { π t |π t : S × M t →
Revised: Let the policy be a sequence of mappings from state s t and posterior µ t to the actionspace, i.e., π = { π t |

Original:  A , t = 0 , · · · , T − 1 } , where M t is the space of posterior  distributions at time stage t .
Revised: π t : S × M t → A , t = 0 , · · · , T − 1 } , where M t is the space of posterior 140 distributions at time stage t .

Original: The details ofthe BR-MDP formulationis presented below.
Revised: Now we present the BR-MDP formulation below.

Original: [Equation]
Revised: [Equation]

Original: The cost at the last time stage only depends on the state, which is denoted by C T ( s T ) . Also notea t = π t ( s t , µ t ) ,  E f ( · ; θ t ) denotes the expectation with respect to ξ t ∼ f ( · ; θ t ) conditional on θ t , andρ µ t denotes a risk functionaltaken with respect to θ t ∼ µ t . 
Revised: where a t = π t ( s t , µ t ) , θ t is a random vector following distribution µ t , E f ( · ; θ t ) denotes the expectation with respect to ξ t ∼ f ( · ; θ t ) conditional on θ t , and ρ µ t denotes a risk functional with respectto θ t ∼ µ t . We assume the last-stage cost only depends on the state, hence denoted by C T ( s T ) .

Original: Equation (3) is the transition of thephysical state s t , and without loss of generality we assumethat the initial state s 0 takes a deterministicvalue.
Revised: Equation (3) is the transition of the state s t , and without loss of generality we assume the initial states 0 takes a deterministic value.

Original: Equation (4) is the update of the posterior µ t, given theinitial prior µ 0 that is constructed fromthe historical data set.
Revised: Equation (4) is the updating of the posterior µ t given the prior µ 0 .

Original: 2.3 Time consistency and dynamic programming
Revised: 3.3 Time consistency and dynamic programming

Original: It is important to note that the BR-MDP formulation (2) takes a nested form of the risk functional toensure time consistency of its optimal policy.
Revised: It is important to note that the BR-MDP formulation (2) takes a nested form of the risk functional.

Original: Time consistency, simply put, means that the optimalpolicy solved at time stage 0 is still optimal for any remaining time stage t ≥ 1  even when the realization of the randomness ξ t is revealed up to that time stage.To illustrate, consider the simple caseof a two-stage problem where the risk functional ρ is CVaR with confidence level α .
Revised: Aprimary motivation for considering such nested risk functional is the issue of time consistency (see[28, 29, 30, 2]), which means that the optimal policy solved at time stage 0 is still optimal for anyremaining time stage t ≥ 1 with respect to the conditional risk functional, even when the realizationof the randomness ξ t is revealed up to that time stage.

Original: Our BR-MDPsolves a nested formulation min a 0 ,a 1 ρ µ 0 E f ( · ; θ 0 ) [ C 0 ( s 0 , a 0 , ξ 0 ) + ρ µ 1 E f ( · ; θ 1 ) [ C 1 ( s 1 , a 1 , ξ 1 )]] , whilethe non-nested counterpart solves min a 0 ,a 1 ρ µ 0 E [ C 0 ( s 0 , a 0 , ξ 0 ) + C 1 ( s 1 , a 1 , ξ 1 )] , where  the expectation is taken with respect to (w.r.t.)
Revised: In contrast, optimizing a static risk functional can lead to “time-inconsistent” behavior, where theoptimal policy at the current time stage can become suboptimal in the next time stage simply becausea new piece of information is revealed (see [28, 29]).

Original: the joint distribution ofξ 0 and ξ 1 , and the static  risk functional isapplied to the total cost. Then we have the following relation between the two formulations:
Revised: Another drawback of the static formulationfor a general risk functional is the lack of dynamic programming equation.

Original: [Equation] where the inequality is justified by CVaR being the right tail average of the distribution, and theequality follows from the tower property of conditional expectation. Note that for the non-nested formulation, the derivation of the dynamic programming equation isbased on the interchangeability principle and the decomposability property of the risk functional, where such decomposability property holds only for expectation and max-type risk functionals (see[34]).
Revised: In particular, [2] pointsout that for the static formulation, the derivation of the dynamic programming equation is based onthe interchangeability principle and the decomposability property of the risk functional, where suchdecomposability property holds only for expectation and max-type risk functionals.

Original: On the other hand, for our nested formulation (2), the corresponding dynamic programmingequation is easily obtained as follows.
Revised: On the otherhand, for our nested formulation (2), the corresponding dynamic programming equation is easilyobtained as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where s t +1 = g t ( s t , a t , ξ t ) , µ t +1 ( θ ) = µ t ( θ ) f ( ξ t ; θ ) (cid:82) Θ µ t ( θ ) f ( ξ t ; θ ) dθ , for t = 0 , · · · , T − 1 .
Revised: where s t and µ t follow equation (3) and (4), respectively.

Original: Therefore, our BRMDP formulation provides a time-consistent risk-averse framework to deal with epistemic uncertaintyin MDPs.
Revised: Therefore, our BR-MDP formulationprovides a time-consistent risk-averse framework to deal with epistemic uncertainty in MDPs.

Original: The exact dynamic programming is summarized in Algorithm 1.
Revised: Theexact dynamic programming is summarized in Algorithm 1, for benchmark purpose.

Original: input : finite horizon T , initial state s 0 , prior distribution µ 0 ; output : optimal value function V ∗ 0 ( s 0 , µ 0 ) and corresponding optimal policy π ∗ ; set V ∗ T ( s T , µ T ) =
Revised: input : finite horizon T , initial state s 0 , prior distribution µ 0 ; output : optimal value function V ∗ 0 ( s 0 , µ 0 ) and corresponding optimal policy π ∗ ; set V ∗ T ( s T , µ T ) =

Original: C T ( s T ) , ∀ ( s T , µ T ) ∈ S
Revised: C T ( s T ) , ∀ ( s T , µ T ) ∈ S

Original: × M t ; for t ← T − 1 to 0 do for each ( s t , µ t ) ∈ S
Revised: × M t ; for t ← T − 1 to 0 do for each ( s t , µ t ) ∈ S

Original: × M t do solve dynamic programming equation (5); set π ∗ t ( s t , µ t ) :
Revised: × M t do solve dynamic programming equation (5); set π ∗ t ( s t , µ t ) :

Original: = a ∗ t , where a ∗ t attains V ∗ t ( s t , µ t ) ; end end
Revised: = a ∗ t , where a ∗ t attains V ∗ t ( s t , µ t ) ; end end

Original:  An analytical approximate solution to BR-MDPs
Revised:  An Analytical Approximate Solution to BR-MDPs

Original: Although the exact dynamic programming works for a general risk functional, there are two challengesto carry it out.
Revised: Although the exact dynamic programming works for a general risk functional, there are two challengesto carry it out.

Original: First, the expectation and the risk functional are generally impossible to computeanalytically and estimation by (nested) Monte Carlo simulation can be computationally expensive.
Revised: First, the expectation and the risk functional are generally impossible to computeanalytically and estimation by (nested) Monte Carlo simulation can be computationally expensive.

Original: Second, the update of the posterior distribution µ t does not have a closed form in general and oftenresults in an infinite-dimensional posterior.
Revised: Second, the update of the posterior distribution µ t does not have a closed form in general and often results in an infinite-dimensional posterior.

Original: We circumvent the latter difficulty by using conjugatefamilies of distributions (see Chapter 5 in [35]), where the posterior distribution falls into the sameparametric family as the prior distribution, and thus maintain the dimensionality of the posterior to bethe finite (and often small) dimension of the parameter space of the conjugate distribution.
Revised: We circumvent the latter difficulty by using conjugatefamilies of distributions (see Chapter 5 in [31]), where the posterior distribution falls into the sameparametric family as the prior distribution, and thus maintain the dimensionality of the posterior to bethe finite (and often small) dimension of the parameter space of the conjugate distribution.

Original: However,the posterior parameters usually take continuous values. Therefore, BR-MDP with the augmentedstate ( s, µ ) isin fact a continuous-state MDP.
Revised: However,the posterior parameters usually take continuous values and hence BR-MDP with the augmented state( s, µ ) is a continuous-state MDP.

Original: The computational cost for solving a continuous-state MDP increases exponentially indimension if we simply discretize the continuous state.
Revised: We note that this continuous-state MDP resembles a belief-MDP,which is the equivalent transformation of a POMDP by regarding the posterior distribution of theunobserved state as a (belief) state (see [32]), and our posterior distribution µ t is just like the beliefstate in POMDP in the sense that both are posterior distributions updated via Baye’s rule.

Original: In view of the computational challenges in carrying out the exact dynamic programming, we derive anefficient approximation algorithm for the BR-MDP with the risk functional CVaR.
Revised: Thisobservation motivates our algorithm development in this section. Specifically, we derive an efficient approximation algorithm for the BR-MDP with the risk functional CVaR.

Original: The main idea isthat for CVaR, once we know the variable u in (1), it is reduced to an expectation of a convex function.
Revised: The main idea is that for CVaR, once we know the variable u in (1), it is reduced to anexpectation of a convex function.

Original: If there is a way to approximate the value function for a given u , we can utilize the convexity of CVaR and apply gradient descent to solve for u .
Revised: If there is a way to approximate the value function for a given u , wecan utilize the convexity of CVaR and apply gradient descent to solve for u .

Original: Hence, we proceed by first deriving the approximatevalue function for a fixed u and time stage.
Revised: Hence, we proceed byfirst deriving the approximate value function for a fixed u and time stage.

Original: Similar to POMDP, by viewing the unknown parameterθ as unobservable state and physical state s as observable state, we maintain a belief (i.e. posteriordistribution) µ on θ .

Original: It is well known that the value function in a POMDP  can be represented bya set of so-called α -functions. We first show an α -function representation of the value function in BR-MDP, and then derive the approximate value function based on this representation.
Revised: Similar to POMDP whosevalue function can be represented by a set of so-called α -functions, we first show an α -functionrepresentation of the value function in BR-MDP, and then derive the approximate value functionbased on this representation.

Revised: All proofs of the propositions/theorems below can be found in Appendix.

Original: 3.1 α -function representation 
Revised: 4.1 α -function representation of the value function

Original: By the definition of CVaR (see (1)) with a confidence level α , we can rewrite the dynamic program- 204 ming equation (5) for the BR-MDP as:
Revised: By the definition of CVaR (see (1)) with a confidence level α , we can rewrite the dynamic program- 191 ming equation (5) for the BR-MDP as:

Original: [Equation]
Revised: [Equation]

Original: where we assume that ξ takes continuous values and the parameter space Θ is a finite set.
Revised: where we assume that ξ and θ take continuous values, and the integration can be numericallyapproximated by Monte Carlo sampling.

Original: If ξ isdiscrete, we can replace the integral by summation.
Revised: If ξ and θ are discrete, the integral can be replaced bysummation.

Original: The assumption of a finite parameter space is practical in many real-world problems.

Original: It also can be viewed as a discrete approximation of acontinuous parameter set and the discretization can be of any precision.

Original: The next proposition showsthat the optimal value function corresponds to the lower envelope of a set of α -functions.
Revised: The next proposition shows that the optimal value function corresponds to the lowerenvelope of a set of α -functions.

Original: Proposition 3.1 ( α -function representation) .
Revised: Proposition 4.1 ( α -function representation) .

Original: The optimal value function in (6) can be represented bythe lower envelope of a set of α -functions denoted by Γ t = { α t } a t ∈A , i.e.,
Revised: The optimal value function in (6) can be represented bythe lower envelope of a set of α -functions denoted by Γ t = { α t } a t ∈A , i.e.,

Original: [Equation]
Revised: [Equation]

Revised: where α t ( s t , θ ) =

Original: The detailed proof of Proposition 3.1 can be found in the appendix.
Revised: u t + 1 1 − α (cid:16)(cid:82) Ξ f ( ξ ; θ ) (cid:16) C t ( s t , a t , ξ ) + min α t +1 (cid:82) Θ α [Equation] − u

Original:  Note that there is a majordistinction between the α -function representations of a POMDP and a BR-MDP: the optimal valuefunction in the risk-neutral POMDP is piecewise linear and convex in the belief state (see [30]),whereas the optimal value function in BR-MDP is no longer piecewise linear in the posterior due tothe ( · ) + operator in CVaR.
Revised: [Equation] + . Note that there is a major distinction between the α -function representations of a POMDP and a BR-MDP: the optimal value function in the risk-neutral POMDP is piecewise linear and convex inthe belief state (see [5]), whereas the optimal value function in BR-MDP is no longer piecewise linearin the posterior due to the ( · ) + operator in CVaR.

Original: In addition, it is computationally impossible to obtain the α -functionsexcept for the last time stage.
Revised: In addition, it is computationally impossible toobtain the α -functions except for the last time stage.

Original: Specifically, denote the cardinality of the α t +1 set as | Γ t +1 | .
Revised: Specifically, denote the cardinality of the α t +1set as | Γ t +1 | .

Original: Since foreach realization of ξ there are | Γ t +1 | candidates for  α  t +1 , there are a total of |A|| Γ t +1 | | Ξ | candidatesfor α t , let alone the optimization over u t .When Ξ is uncountable (i.e., when ξ is continuous randomvariable), the set Γ t is also uncountable.
Revised: Note that for each realization of ξ there are | Γ t +1 | candidates for the minimizer α ∗ ( ξ ) t +1 ,which attains the minimum of (cid:82) Θ α t +1 ( s t +1 , θ ) µ t ( θ ) f ( ξ ; θ ) (cid:82) Θ µ t ( θ ) f ( ξ ; θ ) dθ dθ . There are a total of |A|| Γ t +1 | | Ξ |candidates for Γ t , let alone the optimization over u t .

Original: To deal with the difficulty in computing the α -functionsin POMDPs, [36] proposes several approximation algorithms, and later [37] extends the analysisto a continuous-state optimal stopping problem by applying Jensen’s inequality to the exact valueiteration in different ways and obtains a more computationally efficient approximation to the optimalvalue function.
Revised: To deal with the difficulty in computing theα -functions in POMDPs, [33] proposes several approximation algorithms, and later [34] extends theanalysis to a continuous-state optimal stopping problem by applying Jensen’s inequality to the exactvalue iteration in different ways and obtains a more computationally efficient approximation to theoptimal value function.

Original: Inspired by these works, we derive the approximation approach in the next section.
Revised: Inspired by these works, we derive an approximation approach next.

Original: 3.2 α -function approximation
Revised: 4.2 α -function approximation

Original: In this section, we derive the α -function approximation for a fixed vector ( u 0 , u 1 , · · · , u T − 1 ) .
Revised: In this section, we derive the α -function approximation for a fixed vector ( u 0 , u 1 , · · · , u T − 1 ) .

Original: Without loss of generality, we assume the cost function at each time stage is nonnegative (otherwise add a large constant to the cost at each time stage).
Revised: Without loss of generality, we assume the cost function at each time stage is non-negative (otherwise add a large constant to the cost at each time stage).

Original: For theease of exposition, we rewrite (6) as V ∗ t ( s t , µ t ) = min a t ∈A ,u t ∈ R Q ∗ t ( s t , µ t , a t , u t ) , where Q [Equation] s 
Revised: For ease of exposition,we rewrite (6) as V ∗ t ( s t , µ t ) = min a t ∈A ,u t ∈ R Q ∗ t ( s t , µ t , a t , u t ) , where Q ∗ t ( s t , µ t , a t , u t ) =

Original: [Equation] + [Equation] + 
Revised: u t + 1 1 − α (cid:82) Θ µ t ( θ ) (cid:0)(cid:82)

Original: [Equation] ( s [Equation] + 
Revised: Ξ f ( ξ ; θ ) (cid:0) C t ( s t , a t , ξ ) + V ∗ t +1 ( s t +1 , µ t +1 ) (cid:1) dξ − u t (cid:1) + dθ .

Original: [Equation] , µ [Equation] (cid:1)
Revised: Let V t ( s t , µ t ) :

Original: [Equation] − u t (cid:1) + . Let V t ( s t , µ t ) : = min a t Q ∗ t ( s t , µ t , a t , u t ) be the “optimal” value function.
Revised: =min a t Q ∗ t ( s t , µ t , a t , u t ) be the “optimal” value function for a given u t .

Original: Let V t ( s t , µ t ) :
Revised: Let V t ( s t , µ t ) :

Original: =min α t ∈ Γ t (cid:80) θ ∈ Θ α t ( s t , θ ) µ t ( θ )  , where Γ t = { α t } a t ∈A and
Revised: =min α t ∈ Γ t (cid:82) Θ α t ( s t , θ ) µ t ( θ ) dθ , where Γ t = { α t } a t ∈A andα t ( s t , θ ) =

Original: [Equation]
Revised: u t + 1 − α (cid:90) Ξ (cid:18) C t ( s t , a t , ξ ) f ( ξ ; θ ) − u t + min α t +1 α t +1 ( s t +1 , θ ) f ( ξ ; θ ) (cid:19) dξ.

Original: V t ( s t , µ t ) serves as a lower bound for V t ( s t , µ t ) (see Proposition 3.2), and is similar to the fastinformed bound in POMDP (see [36]).
Revised: V t ( s t , µ t ) serves as a lower bound for V t ( s t , µ t ) (see Proposition 4.2), and is similar to the fastinformed bound in POMDPs (see [33]).

Original: Note that the set Γ t has a constant cardinality of |A| .
Revised: Note that the set Γ t has a constant cardinality of |A| .

Original: However,it involves a minimum within an integral, which can be hard to compute numerically.
Revised: However, it involves a minimum within an integral, which can be hard to compute numerically.

Original: Also, thelower bound is loose in the sense that it could be negative, while the true CVaR value is alwaysnon-negative (due to the non-negative cost).
Revised: Also,the lower bound is loose in the sense that it could be negative, while the true CVaR value is alwaysnon-negative (due to the non-negative cost).

Original: Next, let ¯ V t ( s t , µ t ):
Revised: Next, let ¯ V t ( s t , µ t )

Original:  = 
Revised: [Equation] = min [Equation] ¯Γ

Revised: [Equation]

Revised: s

Original: min ¯ α t ∈ ¯Γ t (cid:80) θ ∈ Θ ¯ α t ( s t , θ ) µ t ( θ )  ,where ¯Γ t = { ¯ α t } a t ∈A and
Revised: [Equation] , θ ) µ [Equation] ( θ ) dθ ,where ¯Γ t = { ¯ α t } a t ∈A and¯ α t ( s t , θ )

Original: [Equation]
Revised: [Equation]

Original:  ¯ V t ( s t , µ t ) serves as an upper bound for V t ( s t , µ t ) (see Proposition 3.2), and is similar to the unobservable MDP bound in POMDP (see [36]), obtained by discarding all observations available tothe decision maker.
Revised: t + [Equation] ¯ V t ( s t , µ t ) serves as an upper bound for V t ( s t , µ t ) (see Proposition 4.2), and is similar to the unobservable MDP bound in POMDPs (see [33]), obtained by discarding all observations available tothe decision maker.

Original: Suppose the cardinality at time stage t + 1 is | ¯Γ t +1 | , then ¯Γ t has a cardinalityof |A|| ¯Γ t +1 | .
Revised: Suppose the cardinality at time stage t + 1 is | ¯Γ t +1 | , then ¯Γ t has a total numberof |A|| ¯Γ t +1 | candidates.

Original: In the following, we derive another approximate value function (cid:101) V t that is boundedby V t and ¯ V t , and is at least better than one of the above bounds.
Revised: In the following, we derive another approximate value function (cid:101) V t that isbounded by V t and ¯ V t , and is at least better than one of the above bounds.

Original: It keeps a constant number|A| of α -functions at each time stage, thus drastically reducing the computational complexity.
Revised: It keeps a constant number|A| of α -functions at each time stage, thus drastically reducing the computational complexity.

Original: Let
Revised: Let

Original: [Equation]

Original: V t

Original: [Equation]

Original: s t ,

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: =
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: t ( s t , θ ) µ t ( θ ) , where (cid:101) Γ t = { (cid:101) α t } a t ∈A and(cid:101) [Equation] t ( s t , θ [Equation]
Revised: dθ , where (cid:101) Γ t = { (cid:101) α t } a t ∈A and

Original: =

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.2.
Revised: Proposition 4.2.

Original: For all t < T and any given u t ∈ R , the following inequalities hold:
Revised: For all t < T and any given u t ∈ R , the following inequalities hold:

Original: [Equation]
Revised: [Equation]

Original: The detailed proof of Proposition 3.2 can be found in the appendix.

Original: To have an implementablealgorithm, we need to use the approximate updating of α -functions iteratively and replace thetrue α -functions at the t + 1 time stage in (7) by the approximate (cid:101) α -functions from the previousiteration.
Revised: To have an implementable algorithm, we need to use the approximate updating of α -functionsiteratively and replace the true α -functions at the t + 1 time stage in (7) by the approximate (cid:101) αfunctions from the previous iteration.

Original: It is clear that the iterative approximations preserve the directions of the inequalities.
Revised: It is clear that the iterative approximations preserve thedirections of the inequalities.

Original: Defineu t : := ( u t , · · · , u T − 1 ) .
Revised: Define u t : := ( u t , · · · , u T − 1 ) .

Original: The approximate value function at time stage t for a given u t : is given by(cid:101)
Revised: The approximate value functionat time stage t for a given u t : is given by (cid:101) V t

Original: V t ( s t , µ t , u t : )=
Revised: ( s t , µ t , u t : )

Original:  min ˜
Revised: = min ˜

Original: α t ∈ ˜Γ t (cid:80) θ ∈ Θ (cid:101) α t ( s t , θ ) µ t ( θ )  , where ˜Γ t = { ˜ α t } a t ∈A and[Equation] t [Equation]
Revised: α t ∈ ˜Γ t (cid:82) Θ (cid:101) α t ( s t , θ ) µ t ( θ ) dθ , where˜Γ t = { ˜ α t } a t ∈A and

Original: s

Original: [Equation]

Original: [Equation]

Original: ) = u

Original: [Equation]
Revised: [Equation]

Original: 3.3 Approximate dynamic programming with gradient descent
Revised: 4.3 Approximate dynamic programming with gradient descent

Original: In this section, we incorporate α -function approximation with gradient descent on ( u 0 , u 1 , · · · , u T − 1 )based on the convexity of the approximate value function w.r.t.
Revised: In this section, we incorporate α -function approximation with gradient descent on ( u 0 , u 1 , · · · , u T − 1 )based on the convexity of the approximate value function w.r.t.

Original: ( u 0 , u 1 , · · · , u T − 1 ) , as formallyshown in the theorem below.
Revised: ( u 0 , u 1 , · · · , u T − 1 ) , as formallyshown in the theorem below.

Original: Theorem 3.3.
Revised: Theorem 4.3.

Original: Suppose the cost function C t ( s, a, ξ ) is jointly convex in ( s, a ) for any fixed ξ , and thestate transition function g t ( s, a, ξ ) is jointly convex in ( s, a ) for any fixed ξ .
Revised: Suppose the cost function C t ( s, a, ξ ) is jointly convex in ( s, a ) for any fixed ξ , and thestate transition function g t ( s, a, ξ ) is jointly convex in ( s, a ) for any fixed ξ .

Original: Then the approximatevalue function (cid:101) V t ( s t , µ t , u t : ) is convex in u t :  .
Revised: Then the approximatevalue function (cid:101) V t ( s t , µ t , u t : ) is convex in u t : , for all t < T .

Original: The detailed proof of Theorem 3.3 can be found in the appendix.
Revised: The jointly convex assumption in Theorem 4.3 is common for gradient-based algorithms for solvingmulti-stage decision making problems (e.g. [35]). It is satisfied in many real-world problems such as while some stopping criterion is met inventory control (e.g. [36]) and portfolio optimization (e.g. [37]).

Original: We give the full algorithm in
Revised: We present the full algorithm in

Original: Algorithm 2.
Revised: Algorithm 2.

Original:  Algorithm 2: Approximate dynamic programming for finite-horizon CVaR BR-MDPs. input : finite horizon T , initial state s 0 , prior distribution µ 0 , initial vector u 0 = ( u 00 , u 01 , · · · , u 0 T − 1 ) , gradient descent step size η k for k = 0 , 1 , · · · ; initialization : set (cid:101) α T ( s T , θ ) = C T ( s T ) , ∀ s T ∈ S , ∀ θ ∈ Θ ; set k = 0 . while some stopping criterion is met do for t ← T − 1 to0 do for each action a t ∈ A , compute (cid:101)
Revised: Note that in Algorithm 2, when applying the gradient descent, we need to compute the gradient of theapproximate value function w.r.t.

Original: α t ( s t , θ ) according to (8); end approximate the value function (cid:101) V 0 ( s 0 , µ 0 , u k ) : = min (cid:101) α 0 (cid:80) θ ∈ Θ (cid:101) α 0 ( s 0 , [-θ ) µ 0 ( θ ) ; compute the gradient ∂ (cid:101) V 
 ∂u k-] ,update the vector u [-k +
 =-]
Revised: the vector ( u 0 , u 1 , [-θ ) µ 0 ( θ ) ; compute the gradient ∂ (cid:101) V 
 ∂u k-]· · · , u [-k +
 =-]T − 1 ) .

Original: u k − η k ∂ (cid:101) V 0 ∂u k , set k
Revised: For ∂ (cid:101) V 0 ∂u 0 , we have∂ (cid:101)

Original: = k + 1 . end output the approximate value function (cid:101) V 
( s 0 , µ 0 , u k ) ; output the optimal policy (cid:101) π t ( s t , µ t ) :=
Revised: V∂u

Original: arg min a t ∈A (cid:80) θ ∈ Θ (cid:101) α t ( s t , a t , θ ) µ t ( θ )
Revised: = (cid:90)

Original: [Equation]
Revised: [Equation]

Original: Note that in Algorithm 2, when applying the gradient descent, we need to compute the gradient of theapproximate value function w.r.t. the vector ( u 0 , u 1 , · · · , u T − 1 ) .
Revised: [Equation]

Original: For ∂ (cid:101) V 0 ∂u 0 , we have
Revised: (cid:26)(cid:90) Ξ

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) a ∗ 0 = arg min a 0 ∈A (cid:80) θ ∈ Θ (cid:101)
Revised: where (cid:101) a ∗ 0 = arg min a 0 ∈A (cid:82) Θ (cid:101)

Original: α 

Revised: α

Original: ( s 0 , θ ) µ 0 ( θ )  .
Revised: ( s 0 , θ ) µ 0 ( θ ) dθ .

Original: For ∂ (cid:101) V 0 ∂u t , t = 1 , · · · , T − 1 , we have
Revised: For ∂ (cid:101) V 0 ∂u t , t = 1 , · · · , T − 1 , we have∂

Original: [Equation]
Revised: [Equation]

Revised: V

Original: [Table]
Revised: [Equation]

Original: where ∂ (cid:101) α l ∂u t can be computed recursively from ∂ (cid:101)
Revised: where ∂ (cid:101) α l ∂u t can be computed recursively from ∂ (cid:101)

Original: α l +1 ∂u t for l = 1 , · · · , t − 1 .
Revised: α l +1 ∂u t for l = 1 , · · · , t − 1 .

Original: The approximate value function output by Algorithm 2 provides an upper bound on the optimal valuefunction, which is shown in the theorem below.
Revised: The approximate value function output by Algorithm 2 provides an upper bound on the optimal valuefunction, which is shown in the theorem below.

Original: Theorem 3.4.
Revised: Theorem 4.4.

Original: min u t : (cid:101) V t ( s t , µ t , u t : ) is an upper bound for the optimal value function V ∗ t ( s t , µ t ) .
Revised: min u t : (cid:101) V t ( s t , µ t , u t : ) is an upper bound for the optimal value function V ∗ t ( s t , µ t ) .

Original: The detailed proof of Theorem 3.4 can be found in the appendix.

Original: We will later show in the numericalexperiments that even though the approximate value function is an upper bound on the optimalvalue function,  the gap between these two is small.
Revised: Even though the approximate value function is an upper bound on the optimal value function, wewill later show in the numerical experiments that the gap between these two is small.

Original: As a final note, even though we develop thealgorithm for the risk functional CVaR, it can be extended easily to other coherent risk measures.
Revised: As a final note,even though we develop the algorithm for the risk functional CVaR, it can be extended easily to othercoherent risk measures.

Original: More specifically, for a class of coherent risk measures which can be represented in the followingparametric form R ( Z ) : = inf λ ∈ Λ E [Ψ( Z, λ )] , where Ψ : R × Λ
Revised: The detailed discussion can be found in Appendix.

Original: → R is a real-valued function andΨ( z, λ ) is convex in ( z, λ ) , we can apply the same technique to approximate the α -functions for agiven vector λ 0 , · · · , λ T − 1 , and then apply gradient descent on the approximate value function.

Original: Theconvergence is guaranteed due to the convexity.

Original: We refer the readers to [38] for more discussion onthis class of coherent risk measures.

Original:  Numerical experiments
Revised:  Numerical experiments

Original: Code in Python for the numerical experiments is included in the supplementary.

Original: Computational timeis reported for a 1.4 GHz Intel Core i5 processor with 8 GB memory.

Original: We illustrate the performanceof our proposed formulation and algorithms with two offline planning problems.
Revised: We illustrate the performance of our proposed formulation and algorithms with two offline planningproblems.

Original: (1) Gambler’sbetting problem .
Revised: Gambler’s betting problem .

Original: Consider a gambler betting in the casino with initial money of s 0 .
Revised: Consider a gambler betting in the casino with initialmoney of s 0 .

Original: At each timestage the gambler chooses how much to bet from a set { 0 , 1 , 2 , 5 , 10 } .
Revised: At each time stage the gambler chooses how much to bet from a set { 0 , 1 , 2 , 5 , 10 } .

Original: To ensure the cost for eachtime stage is non-negative, we add a constant c = 10 to each time stage, and the cost isgiven byc − a · ξ , where a stands for action of how much to bet, ξ = 1 stands for a win and ξ = − 1 standsfor a loss, and the winning rate θ c = P ( ξ = 1) is unknown. The gambler bets for T = 6 rounds.
Revised: The gambler bets for T = 6 rounds. The cost at each time stage is − a · ξ , where a stands foraction of how much to bet, ξ = 1 stands for a win, ξ = − 1 stands for a loss, and the winningrate θ c = P ( ξ = 1) is unknown. We add a constant c = 10 to make the adjusted cost c − a · ξnon-negative to run our algorithm (since the algorithm requires non-negative stage-wise cost),and then subtract cT from the resultant total adjusted cost to recover the total cost.

Original: The data set consists of historical betting record with size 10.
Revised: The data setconsists of historical betting records with size N .

Original: (2) Inventory control problem .
Revised: (2) Inventory control problem .

Original: Consider a warehouse manager with initial inventory level s 0 .
Revised: Consider awarehouse manager with initial inventory level s 0 .

Original: At each time stage the managerchooses how much to replenish from the set { 0 , 1 , · · · , M − s } , where M  is the storage capacity.
Revised: At each time stage the manager chooses howmuch to replenish from the set { 0 , 1 , · · · , M − s } , where M = 15 is the storage capacity.

Original: The customer demand is a random variable ξ , which is assumed to follow a Poisson distributionwith parameter θ c that is truncated below M C  ,  the maximal customer demand the warehouse canhandle.
Revised: Thecustomer demand is a random variable ξ following a Poisson distribution with parameter θ c truncatedbelow M C = 20 , which is the maximal customer demand the warehouse can handle.

Original: The state transition is given by s t +1 = 
Revised: The statetransition is given by s t +1 = max

Original: ( s t + a t − ξ t ) + , the cost function is given by C t ( s t , a t , ξ t ) =
Revised: ( s t + a t − ξ t , 0) , the cost function is given by C t ( s t , a t , ξ t ) =

Original: h t · ( s t + a t − ξ t ) + + p t · ( ξ t − s t − a t ) + , where h t is the holding cost and p t is thepenalty cost.
Revised: h t · max( s t + a t − ξ t , 0) + p t · max( ξ t − s t − a t , 0) , where h t is the holding cost and p t is the penalty cost.

Original: The final stage cost is set to 0 for simplicity.
Revised: The final stage cost is set to 0 for simplicity.

Original: The manager has to plan for T = 6 timestages.
Revised: The manager has to plan for T = 6 timestages.

Original: The data set consists of historical customer demand with size 10.
Revised: The data set consists of historical customer demands with size N .

Original: We compare the following approaches.
Revised: We compare the following approaches.

Original: (1)CVaR BR-MDP (exact): exact dynamic programming (Algorithm 1) presented in this paper.
Revised: (1) BR-MDP (e.): exact dynamic programming (Algorithm 1),where the state s t and µ t are discretized to a fine grid to carry out the dynamic programming (see

Original:  (2)CVaR BR-MDP (approx): approximate dynamic programming(Algorithm 2) presented in this paper.
Revised: Appendix for details). (2) BR-MDP (a.): approximate dynamic programming (Algorithm 2).

Original: (3) Nominal: maximal likelihood estimation (MLE) estimatorθ MLE is computed from the given data. The policy is obtained by solving an MDP with parameterθ MLE .
Revised: Nominal: maximal likelihood estimation (MLE) estimator θ MLE is computed from the given data, andthen a policy is obtained by solving the MDP with parameter θ MLE .

Original: (4) DR-MDP: distributionally robust MDP presented in [6].
Revised: (4) DR-MDP: distributionallyrobust MDP presented in [1].

Original: For each of the consideredapproaches, we obtain the corresponding optimal policy for a given data set, and then evaluate theactual performance of the obtained policy on the true system, i.e., MDP with the true distributionalparameter θ c .
Revised: For each of the considered approaches, we obtain the correspondingoptimal policy for a same data set, and then evaluate the actual performance of the obtained policyon the true system, i.e., MDP with the true parameter θ c .

Original: This is referred to as one replication, and we repeat the experiments for 100 replications.
Revised: This is referred to as one replication, andwe repeat the experiments for 100 replications on different independent data sets.

Original: Results for the gambler’s betting problem can be found in Table 1a,  Figure 1 and Figure 1b.
Revised: Results for thegambler’s betting problem can be found in Table 1a, Table 2, Figure 1a and Figure 1b.

Original: Resultsfor the inventory control problem can be found in Table 1b.
Revised: Results for theinventory control problem can be found in Table 1b.

Original:  Table 1 reports the average  time to obtain the optimal policy and the average actual performance ofthe obtained policy over the 100 replications.
Revised: [Table] [Table] Table 1 reports the average computation time to obtain the optimal policy and the mean and varianceof the actual performance of the obtained policy over the 100 replications.

Original: We have the following observations: (1) The nominalapproach has the largest variability (see standard deviation of the actual performance).
Revised: We have the followingobservations: (1) The nominal approach has the largest variability (see variance of the actualperformance).

Original: (2) DR-MDPis the most conservative since the standard deviation is 0, but this conservativeness is not alwaysdesirable.
Revised: (2) DR-MDP is the most conservative since the variance of actual performance is 0,but this conservativeness is not always desirable.

Original: In the betting problem, the conservative action is not to bet.
Revised: In the betting problem, the conservative action isnot to bet.

Original: DR-MDPwill always choosenot to bet, which is not optimal when θ c > 0 .
Revised: DR-MDP always chooses not to bet, which is not optimal when θ c > 0 .

Original: This can be seen from the worst actual performanceof DR-MDP in the betting problem when θ c = 0 .
Revised: This can be seenfrom the worst actual performance of DR-MDP in the betting problem when θ c = 0 .

Original: 55 and in the inventory control problem when
Revised: 55 and in theinventory control problem.

Original: [Table] [Figure] [Figure]
Revised: (3) For BR-MDP, the computation time for the approximation algorithmis greatly reduced compared to the exact algorithm.

Original: [Figure] [Figure] [Figure] [Table]
Revised: Table 2 reports the actual performance of different formulations over 100 replications with differentdata size N = 5 , 10 , 100 .

Original: [Figure] [Table] [Figure] θ c = 12 .
Revised: We have the following observations: (1) When the data size is very small( N = 5 ), our proposed formulation provides more robustness (much smaller variance) than thenominal approach, which does not hedge against the distributional shift at all.

Original: (3) For CVaR BR-MDP, the computational time for the approximate algorithm is greatlyreduced compared to the exact algorithm.
Revised: (2) When the data sizegrows larger, the robustness and the benefit of considering future data realization in our proposedformulation decrease compared to the nominal approach, as both the posterior distribution and the MLE estimator converges to the true parameter as data size goes to infinity.

Original: Figure 1a shows the histogram of actual performance over100 replications for the nominal approach and CVaR BR-MDP (exact) with different confidence levelsα = 0 .
Revised: Figure 1a shows the histogram of actual performance over 100 replications for the nominal approachand CVaR BR-MDP (exact) with different confidence levels α = 0 .

Original: 1 , 0 . 5 , 0 . 99 underdistributional parameter θ c = 0 .
Revised: under θ c = 0 .

Original: We have the following observationscombining Table 1 and Figure 1a: (1) OurCVaR BR-MDP formulation (both exact and approximate)produces more consistent solution performance across a wide range of input data compared to thenominal approach, which can be seen from the smaller standard deviation in Table 1 and moreconcentrated distribution of the actual performance in Figure 1a.
Revised: Wehave the following observations combining Table 1 and Figure 1a: (1) Our BR-MDP formulation(both exact and approximate) produce more consistent solution performance across a wide range of input data compared to the nominal approach, which can be seen from the smaller variance in Table 1 and more concentrated distribution of the actual performance in Figure 1a.

Original: (2) Compared with DR-MDP, our BR-MDP formulation provides less conservative solution by not fixating on the worst-case scenario,which can be seen from the better actual performance in the inventory control problem (see Table 1b)and the betting problem (see Table 1a when θ c = 0 . 55 ).
Revised: (2) Comparedwith DR-MDP, our BR-MDP formulation provides less conservative solution by not fixating on theworst-case scenario, which can be seen from the better actual performance in the inventory controlproblem (see Table 1b) and the betting problem (see Table 1a when θ c = 0 . 55 ).

Original: As such, our CVaR BR-MDP formulationbalances the trade-off between the expected performance and robustness in the actual performance.
Revised: Our BR-MDPformulation balances the trade-off between the expected performance and variability in the actualperformance.

Original:  Figure 1a shows that in the betting problem with θ c = 0 .
Revised: (3) Figure 1a shows that in the betting problem with θ c = 0 .

Original: 45 , the nominal approach has 20 replicationswhere MLE estimator θ MLE > 0 .
Revised: 45 , the nominal approachhas 20 replications where MLE estimator θ MLE > 0 .

Original: 5 and the gambler chooses to bet, which is not the optimal action.
Revised: 5 and the gambler chooses to bet, which isnot the optimal action.

Original: In contrast,CVaR BR-MDP formulation will learn from the future data realization and update itsposterior distribution on θ .
Revised: In contrast, BR-MDP formulation will learn from the future data realizationand update its posterior distribution on θ .

Original: As a result, in those 20 replications, the gambler initially chooses tobet, but after some time chooses not to bet, which results in the left-shift of the actual performancedistribution.
Revised: As a result, in those 20 replications, the gambler initiallychooses to bet, but after some time chooses not to bet, which results in the left-shift of the actualperformance distribution.

Original: This illustrates time consistency (or in other words, adaptivity to the data process) of our BR-MDP formulation.
Revised: This illustrates time consistency (or in other words, adaptivity to the dataprocess) of our BR-MDP formulation.

Original: This illustration is even more evident by the comparison between DR-MDPand CVaR BR-MDP with α = 1 (CVaR with α = 1 corresponds to the worst-case measure), wherethe only difference is that BR-MDP takes a nested form of risk functional while DR-MDP uses astatic one.
Revised: This illustration is even more evident by the comparisonbetween DR-MDP and BR-MDP with α = 1 (CVaR with α = 1 corresponds to the worst-casemeasure), where the only difference is that BR-MDP takes a nested form of risk functional while DR-MDP uses a static one.

Original: The confidence level α affects the conservativeness of the CVaR BR-MDP formulation.
Revised: The confidence level α affects the conservativeness of the CVaR BR-MDP formulation.

Original: As α increases, the gambler is more likely to take a conservative action (which is not to bet), so theactual performance distribution will shift more to the left.
Revised: As α increases, the gambler is more likely to take a conservative action (whichis not to bet), so the actual performance distribution will shift more to the left.

Original: Lastly, Figure 1b plots the value function
Revised: Lastly, Figure 1bplots

Original:  V ∗ 0 ( s 0 , µ 0 ) of CVaR BR-MDP (exact) and ˜ V ∗ 0 ( s 0 , µ 0 ) of CVaR BR-MDP (approx) under differentprior distributions µ 0 with θ c = 0 . 45 , verifying Theorem 3.4 that ˜ V ∗ 0 ( s 0 , µ 0 ) is indeed an upperbound for V ∗ 0 ( s 0 , µ 0 ) but the difference between these two is small. 
Revised: the value function V ∗ 0 ( s 0 , µ 0 ) of CVaR BR-MDP (exact) and ˜ V ∗ 0 ( s 0 , µ 0 ) of CVaR BR-MDP(approx) under different prior distributions µ 0 with θ c = 0 . 45 , verifying Theorem 4.4 that ˜ V ∗ 0 ( s 0 , µ 0 )is indeed an upper bound for V ∗ 0 ( s 0 , µ 0 ) but the difference between these two is small. (a) Histogram of actual performance over 100 repli- cations for CVaR BR-MDP (exact) with different α . [Figure] b) Value functions of CVaR BR-MDP (exact and approx) under different priors.

Revised:  Conclusion

Revised: In this paper, we propose a new formulation, coined as Bayesian Risk MDP (BR-MDP), to providerobustness against parameter uncertainty in MDPs.

Original: Checklist
Revised: BR-MDP is a time-consistent formulation with adynamic risk functional that seeks the trade-off between the posterior expected performance and therobustness in the actual performance. For finite-horizon BR-MDP with the CVaR risk functional,we develop an efficient approximation algorithm by drawing a connection between BR-MDPs and POMDPs and deriving an approximate alpha-function representation that remains a low computationalcost. Our experiment results demonstrate the efficiency of the proposed approximate algorithm, andshow the robustness and the adaptivity to future data realization of the BR-MDP formulation.

Revised: One of the limitations of our work is the parametric assumption on the distribution of randomness.

Revised: In 342 the future work, we wish to extend the BR-MDP formulation to non-parametric Bayesian setting, andevaluate the performance of the proposed formulation and algorithm on real-world data sets in more 344 challenging problems.

Revised: In addition, the proposed alpha-function approximation algorithm provides anupper bound of the exact value, while there is no theoretical guarantee on the gap between the two.

Revised: In 346 future we will develop more efficient approximation algorithms with a convergence guarantee, such asmethods based on stochastic dual dynamic programming.

Revised: There are also other interesting directions, 348 such as extending the BR-MDP formulation to an infinite horizon problem and utilizing functionapproximation to improve the scalability of the proposed approach to more complex domains.

Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] 
Revised: [Yes] We describe the limitations in the conclusionsection.(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [No] 
Revised: [No] We report no

Original: This work is mostlytheoretical, we foresee negative work
Revised: negative societal impacts, including but not limited to potential malicious or unintended uses, environmental impact, fairness considerations, privacy considerations, or security considerations.

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Revised: If you are including theoretical results...

Original: (b) Did you include complete proofs of all theoretical results? [Yes] If you ran experiments...
Revised: (a) Did you state the full set of assumptions of all theoretical results? [Yes](b) Did you include complete proofs of all

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental

Original: results (either in the supplemental material or as a URL)?
Revised: results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments

Original: multiple times)?
Revised: multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,

Original: internal cluster, or cloud provider)?
Revised: internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A](d)
Revised: [N/A]

Original:  Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re us- [Table] ing/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: informa-
Revised: informa-

Original: tion or offensive content?
Revised: tion or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)

Original: approvals, if applicable?
Revised: approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on

Original: participant compensation?
Revised: participant compensation?

Original: [N/A]
Revised: [N/A]