NLP - ADB ench: NLP Anomaly Detection Benchmark (2025)

Abstract: Anomaly detection (AD) is an important machine learning task with applications in fraud detection, content moderation, and user behavior analysis. However, AD is relatively understudied in a natural language processing (NLP) context, limiting its effectiveness in detecting harmful content, phishing attempts, and spam reviews. We introduce NLP-ADBench, the most comprehensive NLP anomaly detection (NLP-AD) benchmark to date, which includes eight curated datasets and 19 state-of-the-art algorithms. These span 3 end-to-end methods and 16 two-step approaches that adapt classical, non-AD methods to language embeddings from BERT and OpenAI. Our empirical results show that no single model dominates across all datasets, indicating a need for automated model selection. Moreover, two-step methods with transformer-based embeddings consistently outperform specialized end-to-end approaches, with OpenAI embeddings outperforming those of BERT. We release NLP-ADBench at https://github.com/USC-FORTIS/NLP-ADBench, providing a unified framework for NLP-AD and supporting future investigations.

## introduction anomaly detection (ad) is a fundamental area in machine learning with diverse applications in web systems, such as fraud detection, content moderation, and user behavior analysis @xcite . substantial progress has been achieved in ad for structured data such as tabular, graph, and time series @xcite @xcite , but its extension to natural language processing (nlp) remains relatively underexplored @xcite . this gap limits our ability to identify harmful content, phishing attempts, and spam reviews. for instance, detecting abusive or threatening language is crucial for ensuring that social media platforms and online forums remain safe environments for users @xcite . likewise, detecting anomalous product reviews or descriptions in e-commerce is important for preserving user trust and platform credibility @xcite . however, many standard ad methods are designed for numeric or categorical data and are not easily adapted to unstructured text @xcite . existing studies on nlp-specific ad are limited in both dataset variety and algorithmic range @xcite @xcite , leaving open questions about which approaches work best under different conditions. these gaps lead to a central research question: how can we systematically evaluate and compare diverse ad methods across real-world text datasets, and what insights can be gained to guide future development in nlp-based ad? our proposal and key contributions. we introduce nlp-adbench, the most comprehensive benchmark for nlp-ad tasks. nlp-adbench offers four major benefits compared to prior work @xcite : (i) eight real-world datasets covering a wide range of web use cases; (ii) 19 advanced methods that apply standard ad algorithms to language embeddings or use end-to-end neural architectures; (iii) detailed empirical findings that highlight new directions for nlp-ad; and (iv) fully open-source resources, including datasets, algorithm implementations, and more, aligns with the resources and evaluation track. key insights/takeaways (see details in §3). our comprehensive experiments reveal: (i) no single model dominates across all datasets, showing the need for model selection; (ii) transformer-based embeddings substantially boost two-step ad methods (e.g., lunar @xcite and lof @xcite ) relative to end-to-end approaches; (iii) high-dimensional embeddings (e.g., from openai) improve detection performance, but also raise computational overhead; and (iv) datasetspecific biases and human-centered anomaly definitions remain challenging for building robust and widely applicable nlp-ad systems. ## conclusion we present nlp-adbench, the most comprehensive benchmark for contextual nlp anomaly detection (nlp-ad), evaluating 19 state-of-the-art algorithms across 8 diverse datasets. our findings establish the superiority of two-step methods leveraging transformer-based embeddings, such as openai + lunar, over end-to-end approaches, demonstrating the power of hybrid strategies for handling complex nlp anomaly detection tasks. by combining advanced text embeddings with traditional anomaly detection methods, nlp-adbench provides a robust and flexible framework that sets a new standard for evaluating nlp-ad systems. additionally, we offer actionable insights into model performance, dataset variability, and embedding utilization, paving the way for future research.