  Anomalies or failures in large computer systems, such as the cloud, have an
impact on a large number of users that communicate, compute, and store
information. Therefore, timely and accurate anomaly detection is necessary for
reliability, security, safe operation, and mitigation of losses in these
increasingly important systems. Recently, the evolution of the software
industry opens up several problems that need to be tackled including (1)
addressing the software evolution due software upgrades, and (2) solving the
cold-start problem, where data from the system of interest is not available. In
this paper, we propose a framework for anomaly detection in log data, as a
major troubleshooting source of system information. To that end, we utilize
pre-trained general-purpose language models to preserve the semantics of log
messages and map them into log vector embeddings. The key idea is that these
representations for the logs are robust and less invariant to changes in the
logs, and therefore, result in a better generalization of the anomaly detection
models. We perform several experiments on a cloud dataset evaluating different
language models for obtaining numerical log representations such as BERT,
GPT-2, and XL. The robustness is evaluated by gradually altering log messages,
to simulate a change in semantics. Our results show that the proposed approach
achieves high performance and robustness, which opens up possibilities for
future research in this direction.
