Pangram at G en AI Detection Task 3: An Active Learning Approach to Machine-Generated Text Detection (2025)

Abstract: We pretrain an autoregressive LLM-based detector on a wide variety of datasets, domains, languages, prompt schemes, and LLMs used to generate the AI portion of the dataset. We aggressively employ several augmentation strategies and preprocessing strategies to improve robustness. We then mine the RAID train set for the AI examples with the largest error based on the original classifier, and mix those examples and their human-written counterparts back into the training set. We then retrain the detector until convergence.

## introduction we employ the same approach to the coling machine-generated text detection task @xcite as we describe for our main production model at pangram labs @xcite . as we describe in our technical report, classifier accuracy when training deep learning models to detect ai-generated text does not naturally obey scaling laws. classifier performance scales sublinearly as the size of the dataset used to train ai-generated text classifiers increases, ultimately even converging to a regime in which adding more data to the classifier no longer helps performance at all (e.g., validation loss stops decreasing even before the first epoch concludes). we hypothesize that the reason for this "saturation of scaling laws" occurs for multiple reasons. first, we notice during training on a random distribution of human and ai-generated examples that the loss curves are very spiky. there are several batches in a row with nearly-zero loss followed by single batches with very high loss, causing gradient norms to reach very high values. as a result, low learning rates and aggressive gradient clipping are required for stable convergence. second, many ai examples follow very simple patterns that make detection very obvious. for example, llms responses often begin with "certainly!" or "sure, here is a...", which are very obvious tells that a piece of text is llm-generated. these easy examples flood the training set and cause learning to end prematurely as the model overfits to these simple patterns. our intuition is that a naively sampled dataset is flooded with examples that are too easy and have such giveaway patterns. to fix this problem, we resample the dataset with "hard" examples oversampled. to define a hard example, we first train a model on the randomly sampled dataset. we then use this model to mine for high error examples on the raid dataset. we then take these examples, and their opposite label pairs, and retrain the model, which results in much more stable convergence behavior. ## results pangram places first overall in detecting gpt4, chatgpt, and llama (state-of-the-art models) with no adversarial attacks, second overall in detecting all models without adversarial attacks, and tied for first overall in detecting ai-generated text with adversarial attacks. the full results are posted publicly on the raid website. we refer the reader to the leaderboard for full details. @xcite ## discussion we believe our method is a general framework for scaling the deep learning approach to detecting aigenerated text, and this prototype model is only a starting point. for example, the framework could be extended beyond the current dataset to even more domains and data and larger models, or customized to private data or domains in which open data is not readily available at scale, such as messaging, email, or other data with high amounts of pii. ## conclusion in this work, we have demonstrated a general framework for domain and model adaptation of deep-learning based ai detectors based on active learning and mirroring. we further argue that active learning is necessary for scaling performance both in terms of model size and data and present a model larger and more accurate than the other methods in the raid benchmark, without the need for using perplexity-based features or otherwise handcrafted feature engineering.