We theoretically and experimentally compare estimators for off-policy evaluation (OPE) using dependent samples obtained via multi-armed bandit (MAB) algorithms. The goal of OPE is to evaluate a new policy using historical data. Because the MAB algorithms sequentially update the policy based on past observations, the generated samples are not independent and identically distributed. To conduct OPE from dependent samples , we need to use some techniques for constructing the estimator with asymptotic normality. In particular, we focus on a doubly robust (DR ) estimator , which consists of an inverse probability weighting (IPW) component and an estimatorof the conditionally expected outcome. We first summarize existing and new theoretical results for such OPE estimators . Then, we compare their empirical properties using benchmark datasets with other estimators, such as an estimator with cross-fitting .
A doubly robust (DR) estimator is crucial in causal inference, which consists of two nuisance parameters: the conditional mean outcome and logging policy (probability of choosing an action). This paper provides a DR estimator for dependent samples obtained in adaptive experiments and introduces two related topics. First, we propose adaptive-fitting as a variant of sample-splitting for showing an asymptotically normal semiparametric estimator from dependent samples without non-Donsker nuisance estimators. Second, we report an empirical paradox that a DR estimator shows better performances than other estimators using the true logging policy instead of its estimator . While a similar phenomenon is also known for estimators with i.i.d. samples, we hypothesize that traditional explanations based on asymptotic efficiency cannot elucidate our case with dependent samples. We confirm this hypothesis through simulation studies .
R
We theoretically and experimentally compare estimators for off-policy evaluation (OPE) using dependent samples obtained via multi-armed bandit (MAB) algorithms. The goal of OPE is to evaluate a new policy using historical data. Because the MAB algorithms sequentially update the policy based on past observations, the generated samples are not independent and identically distributed. To conduct OPE from dependent samples , we need to use some techniques for constructing the estimator with asymptotic normality. In particular, we focus on a doubly robust (DR ) estimator , which consists of an inverse probability weighting (IPW) component and an estimatorof the conditionally expected outcome. We first summarize existing and new theoretical results for such OPE estimators
A doubly robust (DR) estimator is crucial in causal inference, which consists of two nuisance parameters: the conditional mean outcome and logging policy (probability of choosing an action). This paper provides a DR estimator for dependent samples obtained in adaptive experiments and introduces two related topics. First, we propose adaptive-fitting as a variant of sample-splitting for showing an asymptotically normal semiparametric estimator from dependent samples without non-Donsker nuisance estimators. Second, we report an empirical paradox that a DR estimator shows better performances than other estimators using the true logging policy instead of its estimator
R
Then, we compare their empirical properties using benchmark datasets with other estimators, such as an estimator with cross-fitting
While a similar phenomenon is also known for estimators with i.i.d. samples, we hypothesize that traditional explanations based on asymptotic efficiency cannot elucidate our case with dependent samples. We confirm this hypothesis through simulation studies