Kathlalu at S em E val-2024 Task 8: A Comparative Analysis of Binary Classification Methods for Distinguishing Between Human and Machine-generated Text (2024)

Abstract: This paper investigates two methods for constructing a binary classifier to distinguish between human-generated and machine-generated text. The main emphasis is on a straightforward approach based on Zipf’s law, which, despite its simplicity, achieves a moderate level of performance. Additionally, the paper briefly discusses experimentation with the utilization of unigram word counts.

## introduction this paper addresses the task of classifying textual data as human or machine-generated, focusing on subtask a @xcite with monolingual english data. the rise of technologies like chatgpt has led to a surge in the use of machine-generated content in academia and workplaces. the task is crucial for ensuring the authenticity of texts, especially as individuals may potentially claim authorship of machine-generated content as their own work, raising concerns about academic integrity. by focusing on subtask a and utilizing englishlanguage data, this research addresses the challenges associated with the increasing prevalence of machine-generated text in academic and professional contexts, offering effective classification methods. in this paper, we use simple methods based on linguistic intuition to distinguish between human and machine-generated text. our primary approaches involve leveraging zipf's law as one method, and employing word unigram counts as another. we explored multiple approaches, ultimately narrowing our focus to two strategies. although we submitted only one approach for leaderboard consideration, we believe the other one offers valuable insights as well. surprisingly, we found that our simple methods based on linguistic intuition can rival the performance of large language models in the same task. while this approach could have been applied to the multilingual track, regrettably, time constraints prevented us from pursuing this direction. we ranked in the middle compared to other teams participating in this task. ## background in configuring our task, we utilized the subtask a monolingual training data to train all three approaches, encompassing texts from both humans and models such as chatgpt openai (2022), cohere cohere (n.d.), davinci openai (n.d.), and dolly hugging face (n.d.). each data entry included the text, its source, the model used, the assigned label (either 0 or 1), and a unique identifier. for development purposes, we employed the subtask a monolingual development data, which featured texts generated by humans and the bloomz bigscience (n.d.) model, along with their corresponding source, model, label, and id. the final test data exclusively included texts and their respective ids, with all other information omitted. the output data comprised jsonl files containing only the text ids and their predicted labels (either 0 or 1). these files were generated once using the development data to refine our approach and again for the final test data, aligning with the task objective of predicting the label for a given text using our approach. our research delved into the practical implementation of zipf's law for binary classification. while consulting linders and louwerse's paper @xcite , as well as nguyen-son et al. @xcite , we found theoretical mentions of its potential application. however, none of these sources provided an actual approach. in contrast, our approach involves the concrete implementation of zipf's law, resulting in a functioning system. ## experimental setup for implementing the zipf's law and word unigram approach, we relied on counter python software foundation (2022), numpy numpy (2022), and curve_fit from scipy.optimize scipy (2022), without requiring any additional external tools or libraries. we utilized the provided data without creating additional splits. during testing on the development data, we employed the function f1_score from sklearn.metrics scikit-learn (2022). ## results our initial two approaches exhibit commendable performance in accurately predicting labels. the f1 score for the labels predicted by the zipf's law approach was 0.729 in the official ranking, with the task organizers' baseline set at 0.884. our submission secured the 83rd position out of 137 in the official rankings. the preliminary f1 score for the unigram approach was 0.60, reflecting the test phase; however, this result was not included in the final submission. ## conclusion in conclusion, we are satisfied with the performance of our zipf's law system in the shared task, particularly given its simplicity compared to other model-based approaches. the unigram system demonstrated commendable performance as well. we also explored training a linear support vector classifier (svc) scikit-learn (n.d.a) using character n-grams and employing a sublinear tfidf scikit-learn (n.d.b) approach. we integrated several models, partitioning the training data into distinct files representing specific models used, such as chatgpt, cohere, davinci, and dolly, ensuring a balanced distribution of human-generated texts across all model categories. despite our careful preparations, all four models unexpectedly produced identical labels for all texts during prediction, rendering the system ineffective. this unexpected outcome highlights the necessity of rigorous testing and debugging to ensure the reliability of our methods. identifying and resolving the underlying issues will be crucial for future improvements in model performance and credibility. moving forward, we intend to enhance both the zipf's law and unigram systems through a comprehensive review of relevant literature. additionally, we're dedicated to fixing the bug in our tf-idf vectorizer to maximize its potential in future iterations.