  Reinforcement learning (RL) algorithms usually require a substantial amount
of interaction data and perform well only for specific tasks in a fixed
environment. In some scenarios such as healthcare, however, usually only few
records are available for each patient, and patients may show different
responses to the same treatment, impeding the application of current RL
algorithms to learn optimal policies. To address the issues of mechanism
heterogeneity and related data scarcity, we propose a data-efficient RL
algorithm that exploits structural causal models (SCMs) to model the state
dynamics, which are estimated by leveraging both commonalities and differences
across subjects. The learned SCM enables us to counterfactually reason what
would have happened had another treatment been taken. It helps avoid real
(possibly risky) exploration and mitigates the issue that limited experiences
lead to biased policies. We propose counterfactual RL algorithms to learn both
population-level and individual-level policies. We show that counterfactual
outcomes are identifiable under mild conditions and that Q- learning on the
counterfactual-based augmented data set converges to the optimal value
function. Experimental results on synthetic and real-world data demonstrate the
efficacy of the proposed approach.
