Bandit algorithms are increasingly used in real world sequential decision making problems, from online advertising to mobile health. As a result, there are more datasets collected using bandit algorithms and with that an increased desire to be able to use these datasets to answer scientific questions like: Did one type of ad increase the click-through rate more or lead to more purchases? In which contexts is a mobile health intervention effective? However, it has been shown that classical statistical approaches , like those based on the ordinary least squares estimator, fail to provide reliable confidence intervals when used with bandit data . Recently methods have been developed to conduct statistical inference using simple models fit to data collected with multi-armed bandits.However there is a lack of general methods for conducting statistical inference using more complex models  . In this work, we develop theory justifying the use of M-estimation (Van der Vaart, 2000), traditionally used with i.i.d data, to provide inferential methods for a large class of estimators -- including least squares and maximum likelihood estimators -- but now with data collected with  (contextual) bandit algorithms. To do this we generalize the use of adaptive weights pioneered by Hadad et al. (2019) and Deshpande et al. (2018). Specifically, in settings in which the data is collected via a (contextual) bandit algorithm, we prove that certain adaptively weighted M-estimators are uniformly asymptotically normal and demonstrate empirically that we can use their asymptotic distribution to construct reliable confidence regions for a variety of inferential targets.
Bandit algorithms are increasingly used in real-world sequential decision-making problems. Associated with this is an increased desire to be able to use the resulting datasets to answer scientific questions like: Did one type of ad lead to more purchases? In which contexts is a mobile health intervention effective? However, classical statistical approaches fail to provide valid confidence intervals when used with data collected with bandit algorithms. Alternative methods have recently been developed for simple models (e.g., comparison of means). Yet there is a lack of general methods for conducting statistical inference using more complex models on data collected with (contextual) bandit algorithms; for example, current methods cannot be used for valid inference on parameters in a logistic regression model for a binary reward . In this work, we develop theory justifying the use of M-estimators -- which includes estimators based on empirical risk minimization as well as maximum likelihood -- on data collected with adaptive algorithms, including (contextual) bandit algorithms. Specifically, we show that M-estimators , modified with particular adaptive weights, can be used to construct asymptotically valid confidence regions for a variety of inferential targets.
R
real world sequential decision making problems, from online advertising to mobile health. As a result, there are more datasets collected using bandit algorithms and with that
real-world sequential decision-making problems. Associated with this is
R
these
the resulting
D
increase the click-through rate more or
D
it has been shown that
D
, like those based on the ordinary least squares estimator,
R
reliable
valid
R
bandit data . Recently methods have been developed to conduct statistical inference using simple models fit to data collected with multi-armed bandits.However
data collected with bandit algorithms. Alternative methods have recently been developed for simple models (e.g., comparison of means). Yet
A
on data collected with (contextual) bandit algorithms; for example, current methods cannot be used for valid inference on parameters in a logistic regression model for a binary reward
R
M-estimation (Van der Vaart, 2000), traditionally used with i.i.d data, to provide inferential methods for a large class of estimators
M-estimators
R
including least squares and maximum likelihood estimators
which includes estimators based on empirical risk minimization as well as maximum likelihood
R
but now with
on
A
adaptive algorithms, including
R
To do this we generalize the use of adaptive weights pioneered by Hadad et al. (2019) and Deshpande et al. (2018). Specifically, in settings in which the data is collected via a (contextual) bandit algorithm, we prove that certain adaptively weighted
Specifically, we show that
R
are uniformly asymptotically normal and demonstrate empirically that we can use their asymptotic distribution to construct reliable
, modified with particular adaptive weights, can be used to construct asymptotically valid