TURINGBENCH : A Benchmark Environment for T uring Test in the Age of Neural Text Generation (2021)

Abstract: Recent progress in generative language models has enabled machines to generate astonishingly realistic texts. While there are many legitimate applications of such models, there is also a rising need to distinguish machine-generated texts from human-written ones (e.g., fake news detection). However, to our best knowledge, there is currently no benchmark environment with datasets and tasks to systematically study the so-called ”Turing Test” problem for neural text generation methods. In this work, we present the TURINGBENCH benchmark environment, which is comprised of (1) a dataset with 200K human- or machine-generated samples across 20 labels Human, GPT-1, GPT-2_small, GPT-2_medium, GPT-2_large,GPT-2_xl, GPT-2_PyTorch, GPT-3, GROVER_base, GROVER_large, GROVER_mega, CTRL, XLM, XLNET_base, XLNET_large, FAIR_wmt19, FAIR_wmt20, TRANSFORMER_XL, PPLM_distil, PPLM_gpt2, (2) two benchmark tasks–i.e., Turing Test (TT) and Authorship Attribution (AA), and (3) a website with leaderboards. Our preliminary experimental results using TURINGBENCH show that GPT-3 and FAIR_wmt20 are the current winners, among all language models tested, in generating the most human-like indistinguishable texts with the lowest F1 score by five state-of-the-art TT detection models. The TURINGBENCH is available at: https://turingbench.ist.psu.edu/

## introduction recently, the field of natural language generation (nlg) has seen a massive improvement. while the field of nlg has existed for some time since even before the onset of the first chatbot eliza @xcite , the recent neural architecture transformers @xcite has led to speedy improvement in the generation of long coherent texts. gpt-1 @xcite created by openai is the first installment of these prompt: some investigators struggle to reach comey human: some congressional investigators want to bring former fbi director james comey to the hill to tell them about his meetings with president donald trump but they can't find him. gpt-2: some investigators struggle to reach comey and his team because the fbi's headquarters is in northern virginia. but the fbi has a lot of other people who can help. fair: we have a duty to be vigilant. james comey's testimony today should make him think twice about it. the white house said trump had spoken with comey during the hour-long meeting and that all the senior u.s. officials were given a "heads up" on the situation. gpt-3: "like louis xvi, he doesn't see the storm growing in the distance," says the democratic operative. the lack of specifics, even from surrogates on trump's behalf, forces well-known democrats to point out the obvious. new waves of text-generators. in fact, gpt-1 was built with 117 million parameters, however, in less than 3 years, google's switch transformer @xcite was the largest language model with 1.6 trillion parameters as of january-june 2021. currently, the largest language model is beijing academy of artificial intelligence's (baai) wu-dao 2.0 with 1.75 trillion parameters. even more alarming, since the birth of gpt-1, the field of nlg has grown exponentially such that hugging face's model repo houses more than 9k english and non-english language models (of which over 2k are text-generators). see figure 2 for evolution of neural text-generators. naturally, these newer language models are able to generate texts that can be easily misconstrued as human-written. thus, due to the superior quality of recent generated texts and how easily such text-generators can be used, the potential for misuse is great. this misuse includes but is not limited to the spread of misinformation @xcite and political propaganda @xcite . therefore, it is urgent that we tackle ways to automatically distinguish machine-generated texts from human-written ones accurately. to build accurate detectors of machinegenerated texts, sufficient data is required but lacking. therefore, we create a benchmark environment, turingbench, to combat the obvious security issue language models could pose. just in line with benchmark environments such as squad @xcite and glue @xcite that tremendously facilitate the progress of natural language understanding, we build the first benchmark for authorship attribution in the form of the turing test by including humans and neural language models. the turingbench environment comprises benchmark datasets, benchmark tasks, and a website to host leaderboards. this benchmark dataset is created by collecting 10k news articles (mostly in politics) written by journalists in media outlets such as cnn, washington post, etc. using the title of each article, we prompt 19 selected neural text-generators to generate an article similar to the human-written one. this creates 200k articles with 20 labels (or authors). next, we have two benchmark tasks -turing test and authorship attribution. the turing test task is modeled after the turing test concept @xcite , where if a machine shows intelligent behavior or characteristics usually attributed to a human, then the machine has passed the test. in this scenario, the goal is to cause the machine to fail the turing test. thus, we define this benchmark task as a binary classification problem with human and machine labels. given 19 neural text-generators, there are 19 turing test subtasks with 19 human-machine pairs. furthermore, we understand that due to the ubiquitous nature of these neural language models, distinguishing machine-generated texts from humanwritten ones is no longer sufficient. it is now also important we inquire as to which particular neural text-generator authored a piece of text. to this end, the authorship attribution task aims to assign authorship to one of the many text-generators. we study 20 authors for this task, however, as we have observed, this can easily become 2k authors very soon which will grossly exacerbate the difficulty of this task. finally, to host all these tasks and datasets, we build a turingbench website with leaderboards for each benchmark task and call for participation in tackling this very relevant and nontrivial problem. lastly, we compare state-of-the-art (sota) and baseline turing test and authorship attribution models. from the experimental results, we observe that we need more complex models to accurately distinguish machine-generated texts from humanwritten ones, including text-generators that are yet to be created. ## related work neural text generation recent advances in neural network-based language modeling have demonstrated promising results in text generation @xcite . current state-ofthe-art neural text generation models can produce texts approaching the quality of human-written ones, especially in terms of grammar, fluency, coherency, and usage of real world knowledge @xcite @xcite @xcite . the progress in neural text generation has facilitated a wide range of applications: dialog response generation @xcite , storytelling @xcite , table-to-text generation @xcite , code comment generation @xcite , medical report generation @xcite . however, as these language models can generate text indistinguishable from human-written text, they can also be misused by adversaries to generate fake news @xcite @xcite @xcite , fake produce reviews @xcite , spam emails @xcite . automatic detection of generated text given the potential malicious applications of text generation @xcite , it is thus vital to build detectors to distinguish text generated by machines from humans @xcite @xcite @xcite . most current work focus on fake news detection @xcite @xcite @xcite . despite this progress, it remains a challenging task to build generalizable, interpretable, and robust detectors @xcite . authorship attribution authorship attribution (aa) aims to decide the author of a given text from a set of candidates @xcite @xcite . aa has a broad range of applications including author profiling (lópez-monroy et al., 2020), computer forensics (lambers and veenman, 2009), and plagiarism detection @xcite . previous work on aa has explored and combined various features and representations at different levels including n-grams @xcite @xcite , pos-tags @xcite psycholinguistics features @xcite , while recent approaches also build deep neural network based classifiers such as feed-forward nnlms @xcite , cnns @xcite , lstms (jafariakinabad and hua, 2019, 2020), and bert-based models @xcite . however, previous aa work largely focuses on authorship attribution among humans, while only a few papers @xcite @xcite study neural generated text. our work aims to provide the first benchmark for authorship attribution in the form of the turing test by including humans and neural language models. ## the turingbench environment figure 3 overviews the framework of the turing-bench environment. ## experiments we experiment with several sota and baseline models as summarized in ## discussion we present several observations from our experimental results. 1. both tt and aa tasks are non-trivial: the average f1 score for each human vs. machine subtask and tt model is below 0.87, with fair_wmt20 achieving the lowest (0.49). fair_wmt20 is the newest text-generator in our list and before that we have gpt-3 which achieves the second lowest average f1 score (0.55). this suggests a trend that as newer textgenerators get built, generated texts will become even more human-like, making the tt and aa tasks more difficult. additionally, the difficulty of the aa task is further demonstrated by the pca plot of linguistic features liwc of the turingbench dataset in figure 8 . using liwc to capture stylistic signatures of authors has been studied @xcite . however, we observe that there are quite a few overlaps in linguistic features across different authors (i.e., language models). this makes these authors' writing styles linearly inseparable. based on the average accuracies of two human tests, by and large, we observe that humans currently differentiate machine-generated texts from human-written ones, not much better (i.e., 0.535 and 0.513) than the level of random guessing (i.e., 0.5). ## conclusion in this paper, we have introduced the turing-bench environment and its preliminary results for both turing test (tt) and authorship attribution (aa) tasks. while varied, overall, (1) many contemporary language models can generate texts whose qualities are, to human eyes, indistinguishable from human-written texts, and (2) while some computational solutions for both tt and aa tasks can differentiate human-written texts from machine-generated ones much better than random guessing, overall, the community needs to research and develop better solutions for mission-critical applications. we hope that the turingbench environment will provide a platform on which insights into ways to tackle this urgent issue can be developed and shared.