We propose a novel method---based on local moment matching---to approximate the value function of a Markov Decision Process . The method is grounded in recent work by Braverman et al (2020 ) that relates the solution of the Bellman equation to that of a PDE where, in the spirit of the central limit theorem, the transition matrix is reduced to its local first and second moments. Solving the PDE is not required by our method. Instead  we construct a "sister"  Markov chain whose two local transition moments are (approximately ) identical with those of the focal chain. Because they share these moments , the original chain and its "sister" are coupled through the PDE, a coupling that facilitates optimality guarantees. We show how this view can be embedded into the existing aggregation framework of ADP, providing a disciplined mechanism to tune the aggregation and disaggregation probabilities . This embedding into aggregation also reveals how the approximation's accuracy depends on a certain local linearity of the value function . The computational gains arise from the reduction of the effective state space from N to N^{\frac{1}{2}+\epsilon} is as one might intuitively expect from approximations grounded in the central limit theorem.
We introduce a framework to approximate a Markov Decision Process that stands on two pillars: state aggregation -- as the algorithmic infrastructure; and central-limit-theorem-type approximations -- as the mathematical underpinning of optimality guarantees. The theory is grounded in recent work Braverman et al (2020 that relates the solution of the Bellman equation to that of a PDE where, in the spirit of the central limit theorem, the transition matrix is reduced to its local first and second moments. Solving the PDE is not required by our method. Instead , we construct a "sister" (controlled) Markov chain whose two local transition moments are approximately identical with those of the focal chain. Because of thismoment matching , the original chain and its "sister" are coupled through the PDE, a coupling that facilitates optimality guarantees. Embedded into standard soft aggregation algorithms, moment matching provided a disciplined mechanism to tune the aggregation and disaggregation probabilities . The computational gains arise from the reduction of the effective state space from N to N^{\frac{1}{2}+\epsilon} is as one might intuitively expect from approximations grounded in the central limit theorem.
R
propose a novel method---based on local moment matching---to approximate the value function of
introduce a framework to approximate
R
. The method
that stands on two pillars: state aggregation -- as the algorithmic infrastructure; and central-limit-theorem-type approximations -- as the mathematical underpinning of optimality guarantees. The theory
D
by
D
)
R
not
not
A
,
A
(controlled)
R
(approximately )
approximately
R
they share these moments
of this
A
moment matching
R
We show how this view can be embedded into the existing aggregation framework of ADP, providing
Embedded into standard soft aggregation algorithms, moment matching provided
D
. This embedding into aggregation also reveals how the approximation's accuracy depends on a certain local linearity of the value function