Devising guidance on how to assign individuals to treatment is an important goal of empirical research. In practice  individuals often arrive sequentially, and the planner faces various constraints such as limited budget/capacity, or borrowing constraints, or the need to place people in a queue. For instance, a governmental body may receive a budget outlay at the beginning of an year, and it may need to decide how best to allocate resources within the year to individuals who arrive sequentially. In this and other examples involving inter-temporal trade-offs, previous work on devising optimal policy rules in a static context is either not applicable, or is sub-optimal. Here we show how one can use offline observational data to estimate an optimal policy rule that maximizes ex-ante expected welfare in this dynamic context. We allow the class of policy rules to be restricted for computational, legal or incentive compatibility reasons. The problem is equivalent to one of optimal control under a constrained policy class, and we exploit recent developments in Reinforcement Learning (RL) to propose an algorithm to solve this. The algorithm is easily implementable and computationally efficient, with speedups achieved through multiple RL agents learning in parallel processes. We also characterize the statistical regret from using our estimated policy rule . To do this, we show that a Partial Differential Equation (PDE) characterizes the evolution of the value function under each policy . The data enables us to obtain a sample version of the PDEthat provides estimates of these value functions. The estimated policy rule is the one with the maximal estimated value function. Using the theory of viscosity solutions to PDEs we show that the policy regret decays at a n^{-1/2} rate in most examples; this is the same rate as that obtained in the static case.
Devising guidance on how to assign individuals to treatment is an important goal in empirical research. In practice , individuals often arrive sequentially, and the planner faces various constraints such as limited budget/capacity, or borrowing constraints, or the need to place people in a queue. For instance, a governmental body may receive a budget outlay at the beginning of a year, and it may need to decide how best to allocate resources within the year to individuals who arrive sequentially. In this and other examples involving inter-temporal trade-offs, previous work on devising optimal policy rules in a static context is either not applicable, or sub-optimal. Here we show how one can use offline observational data to estimate an optimal policy rule that maximizes expected welfare in this dynamic context. We allow the class of policy rules to be restricted for legal, ethical or incentive compatibility reasons. The problem is equivalent to one of optimal control under a constrained policy class, and we exploit recent developments in Reinforcement Learning (RL) to propose an algorithm to solve this. The algorithm is easily implementable with speedups achieved through multiple RL agents learning in parallel processes. We also characterize the statistical regret from using our estimated policy rule by casting the evolution of the value function under each policy in a Partial Differential Equation (PDE) form and using the theory of viscosity solutions to PDEs . We find that the policy regret decays at a n^{-1/2} rate in most examples; this is the same rate as in the static case.
R
of
in
A
,
R
an
a
D
is
D
ex-ante
R
computational, legal
legal, ethical
D
and computationally efficient,
R
. To do this, we show that a Partial Differential Equation (PDE) characterizes
by casting
R
. The data enables us to obtain a sample version of the PDEthat provides estimates of these value functions. The estimated policy rule is the one with the maximal estimated value function. Using the
in a Partial Differential Equation (PDE) form and using the
R
we show
. We find
D
that obtained