Comparative Analysis of Natural Language Processing Models for Malware Spam Email Identification (2024)

Abstract: Spam email is one of the main vectors of cyberattacks containing scams and spreading malware. Spam emails can contain malicious and external links and attachments with hidden malicious code. Hence, cybersecurity experts seek to detect this type of email to provide earlier and more detailed warnings for organizations and users. This work is based on a binary classification system (with and without malware) and evaluates models that have achieved high performance in other natural language applications, such as fastText, BERT, RoBERTa, DistilBERT, XLM-RoBERTa, and Large Language Models such as LLaMA and Mistral. Using the Spam Email Malware Detection (SEMD-600) dataset, we compare these models regarding precision, recall, F1 score, accuracy, and runtime. DistilBERT emerges as the most suitable option, achieving a recall of 0.792 and a runtime of 1.612 ms per email.

## introduction spam email has been a challenge since the creation of email services. spam is known as a synonym for annoying and unwanted emails, which result in a loss of time and productivity for users. moreover, spam is currently one of the most common sources for incoming scams @xcite , and also a frequent medium to spread malicious files like ransomware, viruses, and malware. malicious files can take control of the devices for a harmful and undesirable effect on host machines @xcite . criminals often demand financial rewards from individuals or organizations to release the infected devices. cybersecurity organizations develop anti-spam filters focusing on fraudulent activities such as phishing or spoofing @xcite . however, little work has been done to detect those spam emails with highly suspicious indicators that may contain malware, either through external links or attached files @xcite . filtering these emails may enhance the identification by computer security incident response teams (csirt), cybersecurity companies like the spanish national cybersecurity institute (incibe), or users, as well as alerting and providing insight for further investigation. additionally, spammers, users who send spam emails, counteract this type of system through various sophisticated strategies like introducing obfuscated words. consequently, there is a back-andforth battle between both parties, which causes a deterioration of datasets and models trained with them over the years @xcite . this adversarial dataset shift leads developers to update the filters with newer data constantly. the lack of public and annotated data hinders the periodic update of anti-spam systems for some trending and malicious scams. nevertheless, the rise of natural language processing (nlp) models such as transformers @xcite or large language models (llms) @xcite allows the specialization of pre-trained models using a smaller number of examples. these models may enhance and accelerate the adaptation of filters to new trends. in this context, we propose to evaluate a selection of the most used nlp models to detect spam emails with suspicious files from traditional pipelines to the application of transformers and llms. following the work of @xcite , we classify spam email using only the textual information, i.e., through a text classification approach, as either with or without malware files. due to the lack of a publicly available dataset, we leverage the previous dataset built by @xcite to obtain the performance results. this small dataset allows us to provide evidence for our hypothesis. finally, this work can offer an initial recommendation about the most suitable model and its configuration that cyberse-curity companies may use if they would decide to implement this filter. the rest of the paper is organized as follows. section 2 reviews the background of malware detection, especially in spam emails. section 3 explains the spam email malware detection (semd-600) dataset and the seven classifiers to be evaluated. section 4 presents the evaluation and discussion of the classifier performance. section 5 sums up the contributions of our work and identifies future work. ## background malware detection has been studied in the literature @xcite in recent years using nlp techniques by exploring different learning machines such as support vector machine (svm) or long short-term memory (lstm) following a hybrid approach. @xcite enhanced the techniques to make accessible the potentially malicious code for nlp techniques, in particular semantic similarities. these works aimed at detecting malware in several environments, like android applications, by directly analyzing the code. however, transferring this methodology to spam email can increase the analysis runtime, as spam emails usually contain multiple potentially malicious resources, urls, or attachments. although some works in the literature focus on detecting malware in files, we only focus on detecting spam emails containing such files. delving into spam email, abu qbeitah and aldwairi (2018) dynamically analyzed the automatic anomaly detection and active signature generation based on the observed behavior of new malware in phishing emails. @xcite investigated malware propagation patterns to define features to spot malicious webmail attachments. while @xcite focused on surveillance against malware by developing a robust traffic classification system, using principal component analysis (pca) and artificial neural network (ann). nevertheless, we aim to leverage quick and secure analysis of the textual information to process the largest possible number of spam emails. the work of redondo-gutierrez et al. (2022) laid the foundation for targeted detection of spam emails with malware content. they sought to analyze the textual information from the email to avoid opening the potential malicious resource throughout a binary text classification. in this way, they proposed a faster and more secure system to detect these emails and a custom and novel dataset available on request (semd-600). despite the novelty of the work, they only carried out the challenge through a traditional approach using term frequency -inverse document frequency (tf-idf) and bag of words (bow) as vectorizers and svm, logistic regression (lr) and random forest as classifiers. thus, exploring trends and current alternatives can improve the performance of the system, considering they achieved their best performance using tf-idf along with lr. during the latest years, there has been a rise in the nlp from word embedding based models such as word2vect and fasttext, the attentionbased models -transformers -from bert and roberta to llms like chatgpt @xcite . the attention-based models represent the state-of-the-art in most nlp applications, including text classification. transformers achieved high overall performance in text classification using pre-trained models as a single pipeline containing all stages of preprocessing, feature extraction, selection, and classification. ## methodology in this paper, we follow a text classification approach to classify spam emails based only on their textual content, focusing specifically on whether they contain malware or not. redondo-gutierrez et al. (2022) also adopted this perspective in their work; thereby, we take their work as the baseline to compare their best model with transformers and llms and using their custom and only publicly available dataset in the literature, spam email malware detection 600 (semd-600) foot_0 . redondo-gutierrez et al. (2022) built semd-600 using virustotal reports to find spam emails with malware. they obtained the resources, i.e., spam emails, from the public repository spam archive of bruce guenter 2 . authors randomly selected examples between january 2021 and april 2022, building a dataset comprising 300 spam emails with malware and 300 without malware, written in english only. due to the rise of attention-based models and the recent emergence of llms, we compare the best model based on traditional techniques (tf-idf with lr) from @xcite against the most popular current methods in the task of text classification. by doing this, we are providing new baseline results for the task of spam malware detection using only the text of the email. these encompass a word embedding solution -fasttext -, four early attention models based on bert architecture ( ameer et al., 2023 ## conclusions in this work, we evaluated a set of one-word embedding and six attention-based models (two out of six are llms) against the results obtained by redondo-gutierrez et al. (2022) using traditional techniques to detect spam emails containing malware. we followed a binary classification (with or without malware) and trained our model in the semd-600 dataset. this small dataset can help determine the effectiveness of using a pre-trained model with few examples. the results show that distilbert achieved the highest recall and was the third fastest model. although distilbert outperformed the previous best model, the overall recall was less than 0.800, indicating a wide range of improvements. the performance gap between state-of-the-art nlp models and more traditional models is not as wide as initially expected, and considering the easy portability of the traditional models, they prove to be a suitable option for cybersecurity organizations. for future work, it would be interesting to evaluate different sets of parameters in transformer models and extract features and patterns common in spam emails with malware. in addition, extending the number of examples in both classes (with or without malware) of the semd-600 dataset can help to determine if the size of the dataset plays a crucial role in this task.