Tesla at G en AI Detection Task 2: Fast and Scalable Method for Detection of Academic Essay Authenticity (2025)

Abstract: This paper describes a simple yet effective method to identify if academic essays have been written by students or generated through the language models in English language. We extract a set of style, language complexity, bias and subjectivity, and emotion-based features that can be used to distinguish human-written essays from machine-generated essays. Our methods rank 6th on the leaderboard, achieving an impressive F1-score of 0.986.

## introduction the emergence of large language models (llms) such as chatgpt, gpt-4, claude, and other similar applications has revolutionized text generation, creating highly coherent and human-like essays. while these advancements hold promise for educational tools and creative writing, they also pose significant challenges to academic integrity. specifically, misusing machine-generated essays in educational settings threatens to undermine genuine learning and assessment. to address this issue, the task of distinguishing between human-authored and machine-generated essays has gained critical importance. this binary classification task focuses on identifying english-language essays as machinegenerated or human-written. the task's details are outlined in the task overview paper @xcite . an ideal text detector should be accurate, easy to train and should be robust to any adversarial attacks. it should be easy to train, inference should be fast and probably is agnostic to the machine learning model that is used to generate the text. our approach involves identifying key textual features based on style and statistical counts of specific words, words involving complexity, bias, affect, and moral features. by transforming the text into various feature spaces and using the transformed vectors to train simple classical ml algorithms. participation in this task revealed several key insights. our system achieved 6th position on the leaderboard. however, challenges persist in distinguishing highly sophisticated machine-generated texts from essays written by proficient human authors, particularly in cases where llms simulate idiosyncratic human writing styles. this highlights the need for further exploration of subtle linguistic markers and the inclusion of diverse training data. quantitatively, our system achieved an f1 score of 0.986 on the test set, achieving 6th position on the leaderboard. we have made our codebase publicly available foot_0 to facilitate future research in this area. we hope this encourages further innovation and collaboration in safeguarding academic integrity through advancements in machine-generated text detection. ## related work ever since large language models have started to generate coherent, human like text, the task of identifying machine generated task has gained significant attention. as the complete list of works for detecting machine generated text is exhaustive, we list some of the key works that attempt to identify machine generated text. primarily, the methods can be broadly categorized into 3 categories: methods involving language specific features with simple ml models. these models usually use a simple bag of words or extract specific linguistic features and train traditional classification models like logistic regression, svm, random forests or a simple neural network on the extracted feature values. the advantages with these kind of models is that they are quick to train and evaluate. methods involving fine-tuning a transformer (encoder or decoder only) models. these models usually use a transformer encoder architecture like bert, roberta or a decoder only architecture like gpt to learn a classifier. this involves finetuning the model on both the classes of data. as the models are originally pretrained on large quantities of text, these models can understand the structure of the text. methods that have the text generation model available. these methods use a language model and compute per-token probability and per-token ranks in the predicted next token distribution. these methods use these probabilities to train a simple classifier that are used to train classifier models. the primary disadvantage of these methods is that they require the text generation model to compute the token probabilities. in the real-world, access to the model that is used to generate the text may not be possible. the creators of gpt model @xcite develop a simple model that uses tf-idf features on unigrams and bigrams fed to a logistic regression model for identifying text generated from gpt-2 models with an accuracy of 88%. @xcite train simple models on psychological features with simple neural network architectures to determine if an article is written by a human or a language model. @xcite develop a model on top of grover model that can identify fake news articles written by gpt-2. @xcite fine-tune roberta model for the task to identify the texts generated by the largest variant of the gpt-2 model with an accuracy of 95%. @xcite show that a fine-tuned roberta model can spot machine generated tweets from human tweets with over 90% accuracy. gehrmann et al. (2019) develop a tool gltr foot_1 that uses the per-token probability and per-token rank in the predicted next token distribution and the entropy of the predicted next token to determine if the text is human written or machine-generated. @xcite introduced detectgpt, a zero-shot detection method that leverages the curvature of a language model's probability function to identify machine-generated text without requiring any training data. this approach demonstrated effectiveness across various llms and datasets. yang et al. (2023) introduced dna-gpt, a training-free detection technique that utilizes divergent n-gram analysis. by comparing regener-ated text segments with original ones, this method effectively identified discrepancies indicative of machine generation, offering a promising direction for explainable detection. @xcite developed m4, a benchmark dataset with texts generated from various generators, domains, and languages. their empirical study revealed that existing detectors often misclassify machine-generated text as human-written, particularly when encountering unseen generators or domains, indicating significant room for improvement. @xcite presented fast-detectgpt, an efficient zero-shot detection method that reduces computational costs associated with previous approaches like detectgpt. by introducing conditional probability curvature, this method offers a scalable solution for real-time detection applications. @xcite presented mage, a comprehensive testbed evaluating detection methods across diverse domains and llms. they fine-tune a longformer @xcite that can detect machine generated content with 86.61% on unseen models. @xcite propose raid, a benchmark designed to evaluate the robustness of machinegenerated text detectors against adversarial attacks and unseen models. their study demonstrated that many detectors can easily be circumvented, setting a new standard for evaluating and improving detection methodologies. ## system overview we formulate the problem of identifying the given essay in english as human-written vs. machinegenerated as a binary text classification problem. we identify a comprehensive set of linguistically motivated and statistical features for text analysis. we transform the essay into this feature space and use the feature matrix to train classical machine learning algorithms. we use a set of style, language complexity, bias and subjectivity, and emotion-based features of the text to train machine learning models on these features. these features capture the style, syntax, fluency and other psycholinguistic characteristics of the text. the features have been used to identify fake news @xcite . the code to extract these features has been packaged into a python package that is easy and fast. 3 these features fall broadly into these categories: ## experimental setup the task organizers have shared a dataset that can be used to train the various machine-learning models. the dataset contained a predefined split to be used for training and evaluation. the organizers shared the unlabeled test set for making predictions with the trained models. as the test set labels are not publicly released, we do not know the exact number of essays that were machine-generated vs written by human in the test set. we experimented with 4 different machine learning algorithms: logistic regression, random forest, randomized decision trees(extra trees), and xgboost. we used scikit-learn for training our models. we have used the default hyperparameters provided by the scikit-learn while training our models. the official metric for this task is the macro f1 score. ## results the official test set results scored on codalab have been presented below in since only 1 submission is considered for the final evaluation, we used the model that gave the best f1-score on the development set. from the results of the development set, we observe that the complexity features were most helpful for this task. using just the complexity features alone gave very good results, with an f1 score of 0.9916 using logistic regression model. style features gave an f1-score of 0.9442 using xgboost. complexity features again gave high performance of f1-score of 0.9916 with logistic regression model. moral and affect features did not perform as much as the other feature group. we can observe that models trained with all concatenated features gave higher accuracy and f1 metric than using any one of the feature groups. motivated by the high performance of 'all features' with extra trees model, we trained a model using all the training and development data. we used this model to make inferences on the test data. however, after the shared task deadline has passed, the organizers have allowed for submitting the predictions of other models on the official test set for comparison. the official test set results place us in the 6th position of the leaderboard with an accuracy of 0.9876 and a macro f1 score of 0.9859. ## discussion from the trained models, we found the most important features that are useful for discriminating between machine-generated essays and humanwritten essays. rank feature 1 fraction of stop words 2 average word length 3 fraction of punctuations 4 coleman liau index 5 lix readability index 6 fraction of existential there 7 smog index 8 fraction of coordinating conjunctions 9 flesch kincaid grade level 10 type token ratio section a, shows the histogram of the top 10 features among both the classes. we can observe the differences in the distribution of the top 10 features. ## conclusion we conclude that the nela features can be used to identify machine-generated text with high accuracy, as shown through the evaluation of the test set. we feel that our model is robust to adversarial inputs through perturbations. as a future work, we plan to evaluate our model by adversely perturbing the input. we plan to explore the robustness of our methods to domain shifts. figures 1 to 10 show the histograms of the top features across the two classes of the essays. figure 2: histogram of average word length figure 3: histogram of the fraction of all punctuation words figure 5: histogram of lix readability index figure 6: histogram of fraction of existential 'there' figure 7: histogram of smog index figure 8: histogram of fraction of coordinating conjunctions figure 9: histogram of flesch kincaid grade level figure 10: histogram of type token ratio