Can Language Models Help in System Security? Investigating Log Anomaly Detection using BERT (2022)

Abstract: The log files generated by networked computer systems contain valuable information that can be used to monitor system security and stability. Recently, techniques based on Deep Learning and Natural Language Processing have been proven effective in detecting anomalous activities from system logs. The current approaches, however, have limited practical application because they rely on log templates which cannot handle variability in log content, or they require supervised training to be effective. In this paper, a novel log anomaly detection approach named LogFiT is proposed. The LogFiT model inherits the linguistic “knowledge” encoded within a pretrained BERT-based language model and fine-tunes it towards learning the linguistic structure of system logs. The LogFiT model is trained in a self-supervised manner using normal log data only. Using masked token prediction and centroid distance minimisation as training objectives, the LogFiT model learns to recognise the linguistic patterns associated with the normal log data. During inference, a discriminator function uses the LogFiT model’s top-k token prediction accuracy and computed centroid distance to determine if the input is normal or anomaly. Experiments show that LogFiT’s F1 score and specificity exceeds that of baseline models on the HDFS dataset and comparable on the BGL dataset.

## introduction cybercrime costs businesses billions of dollars annually (riskiq, 2019; australia department of home affairs, 2020; international business machines, 2022). log anomaly detection helps to protect businesses' digital infrastructure from cyberattacks by providing the ability to detect abnormal activities, such as network intrusions, from large volumes of event logs generated by networked computer systems. recently, approaches based on deep learning and natural language processing (nlp) have been applied to address the log anomaly detection problem. a review of the literature indicates that long short-term memory (lstm), represented by the deeplog model @xcite , and transformers, represented by the logbert model @xcite , are the deep learning architectures used in the state of the art research in this domain. a practical consideration in log anomaly detection using deep learning is the availability of labeled data to be used in training predictive models. because of the high cost of preparing labeled data, classification-based approaches such as logsy @xcite are of limited value in production settings. thus, a majority of log anomaly detection approaches focus on the zero-positive training scenario, in which predictive models are trained in a self-supervised manner using normal log data only @xcite . @xcite identifies two general categories of self-supervised models for anomaly detection: (1) forecasting-based, which attempts to predict the next log entry given previous log entries; @xcite (2) reconstruction-based, which recomposes log sequences that have been intentionally corrupted. the deeplog model adopts the forecasting-based approach, while the logbert model uses the reconstruction-based approach. we focus on log data that consists of sequences of log sentences. a key factor affecting the effectiveness of log anomaly detection models is how well it encodes representations of sequences of log sentences, especially as the content of the log sentences changes over time @xcite . a common approach is to encode log sentences by first converting them to log templates @xcite . however, this method is shown to negatively affect model effectiveness due to sub-optimal vector representation of the log sequences, and its inability to handle unexpected variability in the content of log sentences over time @xcite @xcite . to address the limitations of current approaches, we make the following contributions: ## related work system log data consists of log sentences representing events that occur within computer systems. several log anomaly detection methods use log parsing as its initial step, in which the log data is converted into a standardised format called "log templates" @xcite @xcite , such that every log sentence can be mapped to a specific log template. the list of log templates thus forms the vocabulary of the model, instead of words or tokens as is typical in nlp. an example of system log data as it is converted to log templates is shown in figure 1 . the deeplog and logbert approaches are illustrated in figure 2 . in both of these approaches, the input log data is pre-processed to convert them into log sentence templates, which form the vocabulary of these models. the input to the model is a sequence of log keys, which are indexes used to look up the corresponding log sentence template from the vocabulary. in the case of deeplog, the last log key is removed from the input, and the model is trained to predict the missing log key given the previous log keys. in the case of log-bert, some percentage of log keys are masked in the input, and the model is trained to predict what the masked log keys are. some studies @xcite @xcite suggest that log templates often result in significant loss of contextual information that is beneficial to a predictive model's performance. the problem with log templates is that it assumes the list of log templates invariant. however, changes in the content of log sentences will naturally happen over time. thus models that rely on log templates will not be able to map new log sentences to an entry in the list of log templates. consequently, logsy @xcite , neuralog @xcite and a2log @xcite do not use log templates; instead the log data is pre-processed using simple cleanup scripts to remove unnecessary details such as specific ip addresses, file paths, port numbers, and urls. recently, the linguistic capabilities of pretrained lms such as bert @xcite has been a subject of increasing interest. several studies have concluded that bert-based language models learn syntactic and semantic information that can be used to increase the effectiveness of downstream nlp tasks @xcite @xcite . the logfit model therefore leverages a pre-trained bert lm to accurately "understand" the linguistic structure and sequential properties of normal system logs. ## method logfit is trained on normal log data which is first transformed into semantic vectors before being passed to the anomaly detection model. in contrast to deeplog and logbert, the logfit model does not require the extraction of log templates during the pre-processing step. by inheriting from a bert-based language model, logfit has the capabilities of an auto-encoder that can reconstruct log data that have been intentionally corrupted via masking. specifically, logfit uses the longformer @xcite variant of the bert family of models. the longformer model allows logfit to handle log paragraphs that contain up to 4096 tokens, much higher than bert's limit of 512 tokens. the input to the logfit model is a log paragraph consisting of individual log sentences joined together with a line separator character. logfit supports up to 4096 tokens, which follows from the limit of the longformer model. it is noted that the "tokens" in logfit differs from the "tokens" in deeplog and logbert -in logfit the tokens are words or sub-words, while in deeplog and logbert the tokens are log sentence templates. the output of the final layer of logfit are 768-dimension vectors that are the learned contextual representations of the input tokens. of interest are the [cls] token vector cv and the masked token prediction vector mv. by convention, in bertbased models, the [cls] token is the first token in the input sequence, and is typically used for classification tasks. the cv vector corresponds to the representation of the entire log paragraph, while the masked token prediction vectors mv correspond to the model's predictions for the masked tokens. at the beginning of each training epoch, the cv vector is used to compute the centroid of all normal training data. during training proper, the cv vector of each log paragraph is used to compute its distance from the current centroid. in contrast, the mv vector is used to compute the masked token prediction loss (cross-entropy loss) following the bert masked language modeling algorithm. an important detail related to logfit's use of longformer is the use of global attention for the [cls] token and all line separator characters only, while all other tokens are limited to local attention with a window size of 16 to 32 -this value is based on findings discussed in @xcite . ## datasets and experimental setup we use three public datasets: hdfs @xcite , bgl @xcite and thunderbird @xcite . these datasets are selected because they are used by the baseline models. some statistics on these datasets are shown in the hdfs dataset consists of log entries (sentences) that are grouped into sessions, identified by the block id field. in contrast the bgl and thunderbird datasets do not have session identifiers, so a time-based grouping of log sentences is used. during deployment logfit is intended to be used in an online mode (as opposed to batch) therefore for datasets where the grouping of log sentences is based on time window, the chosen interval is 30 seconds so that a system utilising logfit can provide timely feedback to system operators. each group of log sentences (i.e., a log paragraph) becomes a single sample that is then fed in batches to the models during training, tuning and evaluation. the datasets are split into training/validation, tuning, and evaluation sets. the training/validation set is created from 6,000 normal samples for training, and 5,000 normal plus 1,000 anomaly samples for parameter tuning. the evaluation set is created from 5,000 normal plus 1,000 anomaly samples. no random shuffling is performed on the datasetsthe chronological order of the logs is used; this is to prevent models from "peeking into the future" during training. the evaluation set consists of log data that appear after (in chronological order) the train/validation set. ## results and discussion log anomaly detection performance. effect of delaying centroid distance computation. figure 4 shows how the two threshold parameters, top-k token prediction accuracy and centroid distance, contributes to the anomaly decision for the hdfs evaluation set (which consists of 5,000 normal samples and 1,000 anomaly samples). the figure indicates that centroid distance is not an important decision factor for discriminating normal and anomaly hdfs log paragraphs. transfer learning. due to transfer learning, the logfit model starts training with an inherited knowledge of the linguistic characteristics of the english language, while neither deeplog or log-bert have this benefit. this allows logfit training to converge in fewer number of epochs compared to the two baseline models. furthermore, because logfit uses a large pre-trained language model to vectorise log paragraphs, it is more robust to changes in the content of the log data @xcite . log parsing. unlike deeplog, logbert and other approaches that depend on a log parsing step, logfit works directly with the text data. figure 5 shows an example of logfit's input log paragraph which have been masked according to the bert masking algorithm, and shows the two criteria (topk accuracy and centroid distance) used by log-fit to decide whether the input is normal or an anomaly. note that in our initial experiments on the thunderbird dataset, logfit's effectiveness was below that of baselines. this was attributed to the length of the log paragraphs being input to the logfit model. after reducing the time window from 60 seconds to 30 seconds, logfit's f1 score and specificity were comparable to that of logbert. the same reduction in time window was applied to the baselines as well. statistical significance. figure 6 shows the predictions of the logfit model compared against the predictions of the logbert model on the hdfs dataset, presented in a mcnemar contingency table. applying mcnemar's test with continuity correction and a significance level of α = 0.05 produces χ 2 = 2553.83 and @xmath0.0 which confirms that logfit performs better than logbert. ## limitations due to the size and computational requirements of the longformer model, training on log data where the length of a paragraph is longer than 2048 tokens takes a long time to complete. further, it can be prone to out-of-memory errors even when training on an nvidia rtx a6000 with 48 gb of gpu memory. addressing this limitation will be the subject of a follow up study. ## conclusions detecting abnormal computer system behavior from the log files that the system generates is an important capability in today's hyper-connected world. natural language processing techniques and in particular transformer-based models using bert are investigated for anomaly detection in system logs. we presented a novel log anomaly detection model named logfit. the logfit model leverages the general knowledge embodied in the pretrained weights of a bert-based language model and fine-tuned it to learn the specific linguistic patterns of system logs. logfit is trained in a selfsupervised manner using only the normal logs and combining two training objectives: masked token prediction and centroid distance minimisation. it learns to recognise only the linguistic structure of normal system logs and can reconstruct normal log data that have been intentionally corrupted. log-fit flags as anomalies any log sample that it fails to reconstruct. we showed that our method outperform baseline models on the hdfs and bgl datasets, and produces comparable performance on the thunderbird dataset. finally, logfit is robust to future changes in the syntactic structure of log paragraphs because of its built-in ability to handle out-of-vocabulary tokens.