DART : An AIGT Detector using AMR of Rephrased Text (2025)

Abstract: As large language models (LLMs) generate more human-like texts, concerns about the side effects of AI-generated texts (AIGT) have grown. So, researchers have developed methods for detecting AIGT. However, two challenges remain. First, the performance of detecting black-box LLMs is low because existing models focus on probabilistic features. Second, most AIGT detectors have been tested on a single-candidate setting, which assumes that we know the origin of an AIGT and which may deviate from the real-world scenario. To resolve these challenges, we propose DART, which consists of four steps: rephrasing, semantic parsing, scoring, and multiclass classification. We conducted three experiments to test the performance of DART. The experimental result shows that DART can discriminate multiple black-box LLMs without probabilistic features and the origin of AIGT.

## introduction as large language models (llms) continue to advance, it becomes increasingly difficult for humans to discern ai-generated text (aigt). this poses issues in society and research, such as spreading fake news and tainting ai training data. researchers have developed aigt detectors to address these issues. despite their success, two challenges related to real-world applicability persist. one challenge with applying aigt detectors is low performance in detecting recent black-box llms. traditionally, aigt detectors rely on probabilistic features such as logits. however, in commercial black-box models, including gpt (ope-nai, 2024a,b) or gemini @xcite , we cannot access their computation procedure which provides logits. that is, traditional approaches cannot detect such black-box models. so, researchers † equal contribution. the other challenge is the vagueness of the origin of aigts. in the inference time of a detector, it receives a text without any information about its origin. so, similar to the inference scenario, we should verify whether a detector can successfully discriminate aigt regardless of source models. however, existing studies mainly tested their detectors under the assumption that a candidate llm is known in advance; they tested whether a binary detector can distinguish a 'human-written text' from an 'aigt by the predefined candidate.' so, whether existing detectors can detect the origin without the assumption is questionable. to address these challenges, we propose a detector using amr of rephrased text (dart ). dart utilizes the semantic gap between given input and rephrased text, using abstract meaning representation (amr). this rephrasing idea was first introduced by raidar @xcite ; we adopted a similar idea to reveal such a semantic gap. to examine the real-world detection performance, we assess dart in three settings: single-candidate, multi-candidate, and leave-one-out. experimental results show that dart can successfully discriminate humans from four cutting-edge llms, including gpt-3.5-turbo, gpt-4o, llama 3-70b @xcite , and gemini-1.5-flash. thus, this paper has the following contributions: ## background in this section, we categorize existing studies regarding numbers (single or multi) and transparency (white box or black box) of candidate llms. single white-box candidates aigt detectors first attempted to extract candidate-specific features. as the candidate is a known white-box model, some researchers designed algorithms adopting probabilistic features from the model @xcite . for example, detect-gpt @xcite used log probabilities of tokens as features. other researchers used neural models that can learn features from the given texts @xcite . however, as many black-box llms recently emerged, the performance of existing detectors should be revalidated on those llms. single black-box candidates some aigt detectors then attempted to extract features regardless of the candidate @xcite @xcite , as black-box candidates may not provide probabilistic features. fast-detectgpt @xcite extended de-tectgpt by extracting probabilistic features from a proxy white-box model (e.g., gpt-j). since such a proxy can provide less accurate results, other studies used syntactic or surface-level features without using a proxy @xcite . for example, dna-gpt @xcite used n-grams from multiple paraphrased texts generated by the candidate. however, such syntactic features are insufficient to detect recent llms because recent models generate text with human-level syntax. multiple candidates as a single-candidate performance is far from real-world scenarios, recent aigt detectors were designed to detect multiple candidates @xcite @xcite @xcite . for example, poger @xcite extends resampling methods to estimate probability using about 100 paraphrases. because of such an excessive regeneration, poger incurs high computational costs. besides, seqxgpt @xcite used a transformer-based detector with a proxy model estimating probabilistic features. however, these studies mainly focused on surface-level features and the limited range of llms (e.g., gpt family), raising questions about detecting other cutting-edge llms. ## the dart framework as shown in figure 1 , dart utilizes semantic gaps between a given text and rephrased texts. to train a detector capturing such gaps, dart uses a four-step procedure: rephrasing, semantic parsing, semantic gap scoring, and classification. step 1, rephrasing: we hypothesized that rephrasing texts could reveal the difference between humans and ai in the way they express semantics. to obtain the rephrased texts, dart uses a rephraser module that generates semantically closer text t 1 from a given text t 0 . further, we let the rephraser generate another rephrased text t 2 by giving t 1 to attain additional features. to avoid generating rephrased texts irrelevant to the given input, we need a reliable rephraser that can preserve semantics. so, we adopted gpt-4o-20240513 as our rephraser because the model showed the highest performance in semantics-related tasks (ope-nai, 2024a). appendix a.2 details the prompts used in the rephrasing step. step 2, semantic parsing: dart adopts a semantic parser to transform texts into semantic representations. we especially adopted amr as a semantic representation because amr has widely been adopted to abstract the given text into semantics @xcite . for the parser, we adopted @xcite . as a result, the parser constructs an amr graph a i from each t i . step 3, semantic gap scoring: dart uses metrics for semantic parsers to measure semantic gaps between texts. as we adopted amr as a semantic representation in the previous step, we utilize a fast and efficient algorithm for scoring amr similarity called sema @xcite . to obtain semantic gaps between a 0 and a i (@xmath0, dart computes precision p i and recall r i scores generated by sema, resulting a feature step 4, classification: dart has a classifier that predicts one possible origin of t 0 . dart uses interpretable classifiers, including support vector machine (svm) or decision tree (dt), though any classifier that maps v to origins can be used. ## experiments to evaluate the performance of dart, we conducted three experiments: (1) single-candidate, (2) multi-candidate, and (3) leave-one-out settings. first, in the single-candidate setting, we formulate aigt detection as a binary classification task. assuming that aigts are exclusively produced by a specific llm, a detector should predict whether the given text is produced by the llm. second, in the multi-candidate setting, we formulate the task as a multi-label classification. after training on aigts from multiple candidate sources, a detector should decide the source of the given input text among the candidates. third, in the leave-one-out setting, we test the generalizability of detectors. we examined whether a detector can successfully classify aigts from models that were unseen during the training. we ran each experiment 10 times for each experiment to achieve reproducibility. further, we analyzed dart's training efficiency by examining the decreasing rate of detecting performance as the size of the training dataset. ## result and discussion single-candidate experiment: dart outperformed existing models. as shown in multi-candidate experiment: dart also outperformed seqxgpt. as shown in we suspect how the detectors extract features using an llm affects the performance. we present a contingency table of seqxgpt and dart dt to support this claim, as shown in figure 2 . the figure shows that (i) seqxgpt struggled in distinguishing models other than llama 3, and (ii) dart dt struggled in distinguishing the gpt family and humans. since seqxgpt in our experiment used gpt-2 as a proxy model, and dart dt used gpt-4o as a rephraser module, the characteristics of the used llms affected the detection performances. for example, as dart dt utilizes semantic gaps between the original and rephrased texts, origins should reveal distinguishable gaps to identify them successfully. so, when the gaps are too similar between origins to discriminate them, dart dt faces difficulty in the classification step. since gpt-4o has a similar language understanding ability to humans (openai, 2024a), gpt-4o and humans may be less distinguishable through macro f1 gpt-3.5-turbo ## conclusion we introduced an aigt framework, dart to tackle challenges in applying aigt detectors to real-world scenarios. dart employed rephraser and semantic gap scoring module to address the challenges of black-box models. to evaluate whether dart can address vagueness of origin, we assessed dart in three experimental settings: single-candidate, multi-candidate, and leave-oneout settings. as a result, dart achieved outstanding performance compared to existing aigt detectors, demonstrating successful capture of differences across origins with semantic gaps.