  Autoencoders are widely used in machine learning applications, in particular
for anomaly detection. Hence, they have been introduced in high energy physics
as a promising tool for model-independent new physics searches. We scrutinize
the usage of autoencoders for unsupervised anomaly detection based on
reconstruction loss to show their capabilities, but also their limitations. As
a particle physics benchmark scenario, we study the tagging of top jet images
in a background of QCD jet images. Although we reproduce the positive results
from the literature, we show that the standard autoencoder setup cannot be
considered as a model-independent anomaly tagger by inverting the task: due to
the sparsity and the specific structure of the jet images, the autoencoder
fails to tag QCD jets if it is trained on top jets even in a semi-supervised
setup. Since the same autoencoder architecture can be a good tagger for a
specific example of an anomaly and a bad tagger for a different example, we
suggest improved performance measures for the task of model-independent anomaly
detection. We also improve the capability of the autoencoder to learn
non-trivial features of the jet images, such that it is able to achieve both
top jet tagging and the inverse task of QCD jet tagging with the same setup.
However, we want to stress that a truly model-independent and powerful
autoencoder-based unsupervised jet tagger still needs to be developed.
