  We introduce active testing: a new framework for sample-efficient model
evaluation. While approaches like active learning reduce the number of labels
needed for model training, existing literature largely ignores the cost of
labeling test data, typically unrealistically assuming large test sets for
model evaluation. This creates a disconnect to real applications where test
labels are important and just as expensive, e.g. for optimizing
hyperparameters. Active testing addresses this by carefully selecting the test
points to label, ensuring model evaluation is sample-efficient. To this end, we
derive theoretically-grounded and intuitive acquisition strategies that are
specifically tailored to the goals of active testing, noting these are distinct
to those of active learning. Actively selecting labels introduces a bias; we
show how to remove that bias while reducing the variance of the estimator at
the same time. Active testing is easy to implement, effective, and can be
applied to any supervised machine learning method. We demonstrate this on
models including WideResNet and Gaussian processes on datasets including
CIFAR-100.
