  This work introduces Bilinear Classes, a new structural framework, which
permit generalization in reinforcement learning in a wide variety of settings
through the use of function approximation. The framework incorporates nearly
all existing models in which a polynomial sample complexity is achievable, and,
notably, also includes new models, such as the Linear $Q^*/V^*$ model in which
both the optimal $Q$-function and the optimal $V$-function are linear in some
known feature space. Our main result provides an RL algorithm which has
polynomial sample complexity for Bilinear Classes; notably, this sample
complexity is stated in terms of a reduction to the generalization error of an
underlying supervised learning sub-problem. These bounds nearly match the best
known sample complexity bounds for existing models. Furthermore, this framework
also extends to the infinite dimensional (RKHS) setting: for the the Linear
$Q^*/V^*$ model, linear MDPs, and linear mixture MDPs, we provide sample
complexities that have no explicit dependence on the explicit feature dimension
(which could be infinite), but instead depends only on information theoretic
quantities.
