  Detecting out-of-distribution (OOD) samples is an essential requirement for
the deployment of machine learning systems in the real world. Until now,
research on energy-based OOD detectors has focused on the softmax confidence
score from a pre-trained neural network classifier with access to class labels.
In contrast, we propose an unsupervised energy-based OOD detector leveraging
the Stiefel-Restricted Kernel Machine (St-RKM). Training requires minimizing an
objective function with an autoencoder loss term and the RKM energy where the
interconnection matrix lies on the Stiefel manifold. Further, we outline
multiple energy function definitions based on the RKM framework and discuss
their utility. In the experiments on standard datasets, the proposed method
improves over the existing energy-based OOD detectors and deep generative
models. Through several ablation studies, we further illustrate the merit of
each proposed energy function on the OOD detection performance.
