  Anomaly detection aims at identifying unexpected fluctuations in the expected
behavior of a given system. It is acknowledged as a reliable answer to the
identification of zero-day attacks to such extent, several ML algorithms that
suit for binary classification have been proposed throughout years. However,
the experimental comparison of a wide pool of unsupervised algorithms for
anomaly-based intrusion detection against a comprehensive set of attacks
datasets was not investigated yet. To fill such gap, we exercise seventeen
unsupervised anomaly detection algorithms on eleven attack datasets. Results
allow elaborating on a wide range of arguments, from the behavior of the
individual algorithm to the suitability of the datasets to anomaly detection.
We conclude that algorithms as Isolation Forests, One-Class Support Vector
Machines and Self-Organizing Maps are more effective than their counterparts
for intrusion detection, while clustering algorithms represent a good
alternative due to their low computational complexity. Further, we detail how
attacks with unstable, distributed or non-repeatable behavior as Fuzzing, Worms
and Botnets are more difficult to detect. Ultimately, we digress on
capabilities of algorithms in detecting anomalies generated by a wide pool of
unknown attacks, showing that achieved metric scores do not vary with respect
to identifying single attacks.
