Does D etect GPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better (2024)

Abstract: The burgeoning generative capabilities of large language models (LLMs) have raised growing concerns about abuse, demanding automatic machine-generated text detectors. DetectGPT, a zero-shot metric-based detector, first introduces perturbation and shows great performance improvement. However, in DetectGPT, the random perturbation strategy could introduce noise, and logit regression depends on the threshold, harming the generalizability and applicability of individual or small-batch inputs. Hence, we propose a novel fine-tuned detector, PECOLA, bridging metric-based and fine-tuned methods by contrastive learning on selective perturbation. Selective strategy retains important tokens during perturbation and weights for multi-pair contrastive learning. The experiments show that PECOLA outperforms the state-of-the-art (SOTA) by 1.20% in accuracy on average on four public datasets. And we further analyze the effectiveness, robustness, and generalization of the method.

## introduction machine-generated text (mgt) detection is to discriminate mgt from human-written texts (hwt), preventing abuse of large language models (llms), including academic misconduct @xcite , spam synthesis @xcite , untrustworthy news @xcite , etc. currently, existing mgt detection methods can be mainly classified into two categories @xcite , i.e., fine-tuned methods @xcite @xcite @xcite and zero-shot metric-based methods @xcite @xcite @xcite . in general terms, finetuned detector methods can achieve better accuracy yes, the connection is bone! you vibrate the bones in your jaw, and that vibration carries into your ear. sound doesn't just travel in air, it travels in all materials. pecola random perturb selective perturb perturb pecola keeps important words while perturbation compared to detectgpt. than zero-shot metric-based methods, especially generalizable to black-box generators, but are more costly during data collection, fine-tuning, and running, in most cases. on the other hand, zero-shot metric-based methods show better interpretability than fine-tuned ones. detectgpt @xcite , as an unsupervised zero-shot metric-based method, first introduces perturbation in mgt detection. specifically, it applies random masking to the original input sample and uses t5 @xcite to fill in. it posits that minor perturbations of mgt tend to have lower log probability under the base model than the original sample. the introduction of perturbation in detectgpt surpasses the vanilla logprobability-based method @xcite in white-box settings. however, detectgpt still has three significant defects: (i) detectgpt's reliance on the logit regression module's threshold compromises its generalization in zero-shot settings and limited to large batch input, failing on individual inputs. (ii) de-tectgpt does not fully utilize the perturbation. as a metrics-based method, it only considers the probability difference caused by perturbation, which is overly simplified and slightly indistinguishable. perturbation should indeed be a stronger augment that carries implicit language pattern information. (iii) detectgpt perturbs the original sample randomly and unrestricted, which could introduce more noise and negatively impact the performance @xcite . for example, @xcite find entity-relationship plays a role in the detection, which might be destroyed in random perturbation of detectgpt. in this paper, we thus propose a perturbationbased contrastive learning model, pecola, for mgt detection, toward the defects with two stages, i.e., selective strategy perturbation (sec. 3.1) and token-level weighted multi-pairwise contrastive learning (sec. 3.2). firstly, selective strategy perturbation is a token-level rewriting method with restrictions on modifying important texts @xcite to reduce noise. the motivation is to simulate the human behavior of modification @xcite @xcite . the perturbation strategy consists of token removal and substitution, as shown in fig. 1 . the experiments show that the selective strategy perturbation method can improve the performance of both metrics-based (i.e., detectgpt) and model-based methods. secondly, we propose a multi-pairwise contrastive learning model to process the perturbed texts. different from the logit regression module in detectgpt, the trained model is generalizable without any threshold setting, and it can deal with individual inputs. moreover, by utilizing multi-pairwise contrastive learning, the model could better utilize perturbation to focus on the language pattern gap between hwt and mgt. the importance weight from the perturbation stage is also reused as contrastive learning weight. notably, by using contrastive learning, pecola is a strong few-shot fine-tuning method, which effectively bridges and integrates metric-based and fine-tuned detector categories. finally, extensive experiments show pecola is significantly superior to baseline and sota methods on four datasets, pecola improves by 1.20% to sota on average under few-shot settings, surpassing the latest methods by 3.84% among metric-based detectors and by 1.62% among fine-tuned detectors. further experiments show that pecola is as well better at generalization, robustness, and effectiveness. our contributions are summarized as follows: we utilize a novel fine-tuned contrastive learning module to replace the logit regression of detectgpt (metric-based), which frees the detector from setting the threshold, enables it to deal with individual input, and can be generalizable and effective on the few-shot setting by contrasting perturbed texts with origin ones. perturbation. data perturbation methods find frequent application in text classification tasks @xcite , which is commonly employed through the technique of consistency regularization @xcite . input texts 0.0646 0.0767 0.0767 0.1069 0.1069 0.1069 0.106 0.10 token importance selective strategy perturbation plm-encoder multi-contrastive objective 𝓛 = 𝓛 𝒄𝒆 + 𝝀𝓛 𝒄𝒐𝒏 joint optimization via sgd fine-tuning token-level weighted multi-pairwise contrastive learning the inner ear is made up of tin y sensors called hair cells that detect the movement of the bo nes in our head and send signal s to the brain, which interprets these signals as sound. … by machine? or by human? <@xmath0, <@xmath1. … yake select & rank selective masking strategies mask-filling lm* substitution original (6) (6) (6) (6) (7) 𝑠 𝑗 𝑓𝑖𝑙𝑙 𝑠 𝑖 𝑚𝑎𝑠𝑘 𝑠 𝑗 𝑚𝑎𝑠𝑘 s 𝑗 𝑠 𝑖 𝑓𝑖𝑙𝑙 s 𝑖 (7) (6) (6) (7) nevertheless, in mgt detection, previous perturbation methods have exhibited certain limitations. for instance, they often resort to randomly selecting target tokens for synonym replacement @xcite , deletion, insertion @xcite , rewriting by llms @xcite , and fine-tuning pre-trained language models (plms) to fill text spans of variable lengths @xcite . while these methods do enhance text diversity, the indiscriminate replacement of tokens without guided rules can lead to the generation of less reliable texts. @xcite utilize perturbations as stress test approaches for the robustness of mgt detectors to show their loopholes. these limitations motivate us to devise data perturbation methods tailored for mgt detection. our approach, with selective perturbation, aims to better represent meaningful recombination spaces while preserving the inherent semantic features of the text, ultimately enhancing the diversity of samples. constrastive learning. contrastive learning is an effective solution method to the issues that solely relying on cross-entropy classification loss would lead to a lack of robustness and suboptimal generalization @xcite . in limited labeled data task @xcite , introduce a robust contrastive learning method to capture the similarities between the same instances in the representation space while separating those of different classes. similarly, out-of-distribution (ood) usually leads to severe semantic shift issues during inference, prompting another approach based on margin contrastive learning @xcite . differently, our method focuses more on the changes of the rephrase space in data distribution after perturbation, and strives to reduce reliance on the mask-filling models in few-shot learning. ## methodology as shown in fig. 2 , the workflow of pecola mainly consists of two stages: selective strategy perturbation and supervised contrastive learning, which joined the advantage of metric-based and model-based detection methods, respectively. ## conclusion in this paper, we introduce pecola, a novel machine-generated text detection method that effectively bridges and integrates metric-based and fine-tuned detectors for mgt detection. to relieve the information loss caused by the random masking used in detectgpt, we present a tokenlevel selective strategy perturbation method. to better distinguish meaningful recombination spaces and reduce reliance on the mask-filling models, we present a token-level weighted multi-pairwise contrastive learning method. in few-shot settings, experimental results show that pecola significantly enhances the performance of plms in mgt detection. subsequent analytical experiments validate pecola's effectiveness, robustness, generalization, and capability in detecting short texts.