  Metafeatures, or dataset characteristics, have been shown to improve the
performance of hyperparameter optimization (HPO). Conventionally, metafeatures
are precomputed and used to measure the similarity between datasets, leading to
a better initialization of HPO models. In this paper, we propose a cross
dataset surrogate model called Differentiable Metafeature-based Surrogate
(DMFBS), that predicts the hyperparameter response, i.e. validation loss, of a
model trained on the dataset at hand. In contrast to existing models, DMFBS i)
integrates a differentiable metafeature extractor and ii) is optimized using a
novel multi-task loss, linking manifold regularization with a dataset
similarity measure learned via an auxiliary dataset identification meta-task,
effectively enforcing the response approximation for similar datasets to be
similar. We compare DMFBS against several recent models for HPO on three large
meta-datasets and show that it consistently outperforms all of them with an
average 10% improvement. Finally, we provide an extensive ablation study that
examines the different components of our approach.
