  To evaluate prospective contextual bandit policies when experimentation is
not possible, practitioners often rely on off-policy evaluation, using data
collected under a behavioral policy. While off-policy evaluation studies
typically focus on the expected return, practitioners often care about other
functionals of the reward distribution (e.g., to express aversion to risk). In
this paper, we first introduce the class of Lipschitz risk functionals, which
subsumes many common functionals, including variance, mean-variance, and
conditional value-at-risk (CVaR). For Lipschitz risk functionals, the error in
off-policy risk estimation is bounded by the error in off-policy estimation of
the cumulative distribution function (CDF) of rewards. Second, we propose
Off-Policy Risk Assessment (OPRA), an algorithm that (i) estimates the target
policy's CDF of rewards; and (ii) generates a plug-in estimate of the risk.
Given a collection of Lipschitz risk functionals, OPRA provides estimates for
each with corresponding error bounds that hold simultaneously. We analyze both
importance sampling and variance-reduced doubly robust estimators of the CDF.
Our primary theoretical contributions are (i) the first concentration
inequalities for both types of CDF estimators and (ii) guarantees on our
Lipschitz risk functional estimates, which converge at a rate of O(1/\sqrt{n}).
For practitioners, OPRA offers a practical solution for providing
high-confidence assessments of policies using a collection of relevant metrics.
