  We offer a theoretical characterization of off-policy evaluation (OPE) in
reinforcement learning using function approximation for marginal importance
weights and $q$-functions when these are estimated using recent minimax
methods. Under various combinations of realizability and completeness
assumptions, we show that the minimax approach enables us to achieve a fast
rate of convergence for weights and quality functions, characterized by the
critical inequality \citep{bartlett2005}. Based on this result, we analyze
convergence rates for OPE. In particular, we introduce novel alternative
completeness conditions under which OPE is feasible and we present the first
finite-sample result with first-order efficiency in non-tabular environments,
i.e., having the minimal coefficient in the leading term.
