Towards Reliable and Practical Phishing Detection (2025)

Abstract: As the prevalence of phishing attacks continues to rise, there is an increasing demand for more robust detection technologies. With recent advances in AI, we discuss how to construct a reliable and practical phishing detection system using language models. For this system, we introduce the first large-scale Korean dataset for phishing detection, encompassing six types of phishing attacks. We consider multiple factors for building a real-time detection system for edge devices, such as model size, Speech-To-Text quality, split length, training technique and multi-task learning. We evaluate the model’s ability twofold: in-domain, and unseen attack detection performance which is referred to as zero-day performance. Additionally, we demonstrate the importance of accurate comparison groups and evaluation datasets, showing that voice phishing detection performs reasonably well while smishing detection remains challenging. Both the dataset and the trained model will be available upon request.

## introduction phishing is an act of deceiving individuals into disclosing sensitive information or installing malicious software. with a huge amount of global financial damage, the demand for advancing phishing detection is larger than ever before. for instance, in 2022, the total loss amounts to 107 million dollars in south korea (kisa, 2022) and a total loss of 52 million dollars was reported in the us @xcite . phishing poses significant detection challenges due to their subtle mimicry of legitimate communications and their ability to adapt rapidly, evading traditional defenses. addressing these challenges requires detection systems that excel in two critical capabilities: (1) distinguishing nuanced differences between phishing and legitimate samples (imitation detection) and (2) generalizing to novel and unseen attack types (zero-day detection @xcite ). these requirements highlight the need for robust datasets and development of methodologies that bridge the gap between academic research and practical deployment. while previous research has advanced phishing detection, challenges remain for real-world application. existing datasets lack size and diversity, with only 609 voice phishing samples available in @xcite and 638 smishing instances in english @xcite . moreover, current approaches often overlook practical issues, such as the need for real-time detection during calls, rather than post-call decisions, and other deployment challenges. additionally, these methods fail to address zero-day attacks-new and unseen phishing techniques-which are critical for building robust detection systems. in this paper, we present a comprehensive approach to building reliable phishing detection systems, underpinned by the introduction of the first large-scale dataset for smishing and vishing detection. this dataset comprises 94,602 phishing samples and 205,870 non-phishing samples, spanning six distinct attack types across multiple modalities. each phishing type reflects the diverse strategies attackers employ, such as impersonating government agencies, financial institutions, parcel services, and even personal contacts. the dataset not only enables high-fidelity imitation detection but also includes carefully curated non-phishing samples to enhance robustness. these non-phishing examples are collected through crowdsourcing and are designed to mirror phishing characteristics, adhering to criteria such as thematic alignment, exclusion of impersonation targets, and the inclusion of phishing-related keywords to prevent overfitting. to enable real-world deployment, we investigate practical considerations in system design. we focus on edge-compatible, small to medium-sized language models, such as distilkobert @xcite and mbert-base @xcite , in conjunction with automatic speech recognition (asr) models like wav2vec2 @xcite and whisper @xcite . advanced training techniques, including parameter-efficient fine-tuning (peft) and task-adaptive pretraining (tapt), are applied to enhance performance while maintaining computational efficiency. we also address the challenge of handling real-time streaming data, a critical aspect of vishing detection, where timely detection can prevent significant harm. we evaluate the models using a robust framework that prioritizes imitation and zero-day detection performance, as well as recall rates. the dataset and detection systems will be made available for further research, with the potential to generalize insights across languages and regions. this study not only advances the state of phishing detection but also contributes broadly to fraud prevention and cybersecurity. ## dataset construction we constructed a dataset comprising 94,602 phishing samples and 205,870 non-phishing samples. each sample includes the following attributes: (1) text, (2) collection date, (3) phishing type, (4) label (phishing/non-phishing), and (5) modality (text or voice). ## task setup to evaluate the challenges of phishing detection comprehensively, we define two key performance aspects and corresponding evaluation metrics. ## implementation details this section outlines the key considerations and methods for building a practical and robust realtime phishing detection system. ## experiment results this section presents the detection performance for vishing and smishing across various experimental setups, focusing on identifying the most effective detection system. notably, the evaluations in this section utilize our challenging dataset, specifically designed to assess the model's robustness under difficult conditions. for results on validation sets derived from proportional splits of the full dataset, refer to appendix i. ## conclusion in this paper, we conduct a comprehensive study to create a reliable and practical phishing detection model. we develop the first large-scale phishing dataset, which serves as the foundation for training a robust and practical detection system. we then conduct experiments considering various factors that can affect the performance. we define the challenges of phishing detection, focusing on imitation and zero-day attacks, and evaluate each model based on them. we believe that our phishing dataset and propose methodology will facilitate research in phishing detection and, more broadly, fraud detection.