Do Stochastic Parrots have Feelings Too? Improving Neural Detection of Synthetic Text via Emotion Recognition (2023)

Abstract: Recent developments in generative AI have shone a spotlight on high-performance synthetic text generation technologies. The now wide availability and ease of use of such models highlights the urgent need to provide equally powerful technologies capable of identifying synthetic text. With this in mind, we draw inspiration from psychological studies which suggest that people can be driven by emotion and encode emotion in the text they compose. We hypothesize that pretrained language models (PLMs) have an affective deficit because they lack such an emotional driver when generating text and consequently may generate synthetic text which has affective incoherence i.e. lacking the kind of emotional coherence present in human-authored text. We subsequently develop an emotionally aware detector by fine-tuning a PLM on emotion. Experiment results indicate that our emotionally-aware detector achieves improvements across a range of synthetic text generators, various sized models, datasets, and domains. Finally, we compare our emotionally-aware synthetic text detector to ChatGPT in the task of identification of its own output and show substantial gains, reinforcing the potential of emotion as a signal to identify synthetic text. Code, models, and datasets are available at https: //github.com/alanagiasi/emoPLMsynth

## introduction modern plms can surpass human-level baselines across several tasks in general language understanding @xcite and can produce synthetic text that can exceed human level quality, such as synthetic propaganda thought to be more plausible than human written propaganda @xcite . plms have been used to generate disinformation @xcite , left-or right-biased news @xcite , fake comments @xcite , fake reviews @xcite , and plagiarism @xcite and can generate synthetic text at scale, across domains, and across languages. the increasing high quality of synthetic text from larger and larger plms brings with it an increasing risk of negative impact due to potential misuses. in this work, we focus on the task of synthetic text detection. due to the potentially profound consequences of global synthetic disinformation we focus mainly, but not exclusively, on the detection of synthetic text in the news domain. 1 synthetic news has already been published on one highly reputable media website, only later to be withdrawn and apologies issued for the "breach of trust" @xcite . current approaches to synthetic text detection tend to focus on learning artefacts from the output distribution of plms @xcite @xcite , e.g. increased perplexity caused by nucleus sampling @xcite . however, plm distributions are dependent on training data and numerous hyperparameter choices including model architecture and sampling strategy. this gives rise to a combinatorial explosion of possible distributions and makes the task of synthetic text detection very difficult. furthermore, it is not unexpected that performance decreases when classifying out-of-distribution instances, and there is a growing field of work investigating this shortcoming @xcite . in this work, we consider not only the plm output distribution, but also the other side of the synthetic text detection coin -human factors. we present a novel approach to the task of synthetic text detection which aims to exploit any difference between expression of emotion in human-authored and synthetic text. neural word representations can have difficulty with emotion words, and plm sampling strategies are stochastic rather than driven by emotion -we use the term affective deficit to refer to these shortcomings. thus, the resulting synthetic text can express emotion in an incoherent way, and we introduce the term affective incoherence to refer to this type of limitation. to be clear, we do not contend that synthetic text is devoid of emotion, rather that the emotional content of synthetic text may be affectively incoherent, and that this affective incoherence stems from the underlying affective deficit of the plm. for the purpose of demonstration of the affective deficit that we believe to be characteristic of text produced by plms, we provide the following simple example of human-versus machineauthored text with positive emotion words highlighted in orange and negative emotion words in pink. one shows coherent emotion expected of human-authored text, while the other demonstrates affective incoherence (see footnote foot_2 to reveal which was synthetic/human-authored text). in this simple example, we have demonstrated one kind of affective incoherence present in synthetic text but we suspect that fine-tuning an emotionallyaware plm could detect additional and more complex emotional patterns that might go undetected by humans. we hypothesise that the affective deficit of plms could result in synthetic text which is affectively incoherent, which could be useful in distinguishing it from human text. we use a transfer learning (pan and yang, 2010) method to train an "emotionally-aware" detector model. by fine-tuning a plm first on emotion classification and then on our target task of synthetic text detection, we demonstrate improvements across a range of synthetic text generators, various sized models, datasets and domains. furthermore, our emotionally-aware detector proves to be more accurate at distinguishing between human and chatgpt text than (zero-shot) chatgpt itself. finally, we create two new datasets: newssynth, a dataset of 20k human and synthetic news articles, and chatgpt100, a testset of 100 human and chat-gpt texts on a range of topics. we make all code, models and datasets publicly available to aid future research. 3 2 related work people are relatively poor at detecting synthetic text, and have been shown to score just above random chance @xcite . hybrid systems, such as gltr @xcite for example, use automation to provide information to aid human classification, highlighting a text sequence using colours to represent likeness to the plm output distribution such as gpt-2 ( radford et al., 2019). gehrmann et al. (2019) reported an increase in detection accuracy of approximately 18% (from 54% to 72%) using gltr, while @xcite report an f1 score of 46% using gltr with a heuristic based on an analysis of human text. both human and hybrid approaches involve human decisions, which can be slow, expensive, susceptible to bias, and inconsistent. automatic detection produces the best results for synthetic text detection. this usually involves training plms to detect other plms, but zero-shot detection methods also exist, e.g. detectgpt @xcite . potentially the best supervised detector, bert, can detect synthetic text from 19 different generators with a mean f1 of 87.99%, compared to 56.81% for hybrid, and worst of all humans at 53.58% @xcite . performance of sota detectors can however be inconsistent and unpredictable due to several factors specific to both the detector and generator, including: model size and architecture, training data and domain thereof, sampling strategy, hyperparameter selection, and sentence length. as mentioned above, @xcite showed the best of these models (bert) achieves a mean f1 of 87.99% on 19 different synthetic text generators. however, the mean score hides the wide range (≈53%) of f1's, ranging from as low as 47.01% to 99.97%, for distinct synthetic text generators. this volatility may be due in part to the detector simply learning artefacts of the generator distribution. consequently, the task of synthetic text detection is somewhat of an arms race with detectors playing catch-up, forced to learn ever-changing distributions due to the numerous factors that can potentially change those distributions. existing approaches to synthetic text detection exploit properties of synthetic text. synthetic text can be incoherent and degrade as the length of generated text increases @xcite , perplexity increases with increasing length unlike human text @xcite , and plms are susceptible to sampling bias, induction bias, and exposure bias @xcite . for example, exposure bias can contribute to brittle text which is repetitive, incoherent, even containing hallucinations @xcite . synthetic text can have an inconsistent factual structure, such as mentioning irrelevant entities @xcite . perhaps unsurprisingly, synthetic text detection is less difficult with longer excerpts of generated text, for both humans and machines @xcite . one aspect of writing that has not, up to now, been a focus of synthetic text detection efforts is the expression of emotion. the problem of encoding emotion was first identified in neural nlp with static embeddings such as word2vec @xcite . static word embeddings have difficulty distinguishing antonymns from synonyms @xcite . this deficit is present in embeddings for words which represent opposing emotions (e.g. joy-sadness) @xcite . furthermore, words representing opposing emotions can have closer embeddings relative to words representing similar emotions @xcite . there have been various approaches to address this affective deficit in embeddings, such as transfer learning from sentiment analysis @xcite , an additional training phase using an emotional lexicon and psychological model of emotions @xcite , and combining separately-learned semantic and sentiment embedding spaces @xcite . addressing potential affective deficits of plms is also the goal of work aiming to make dialogue systems more empathetic. for example @xcite force dialogue generation to express emotion based on the emotion detected in an utterance, while @xcite follow a similar approach with a transformer architecture to make the system more empathetic. in contrast, @xcite report that human text can display consistency in emotional content whereby similar emotions tend to occur adjacent to each other while dissimilar emotions seldom do. 4 past work in synthetic text detection has focused on the properties of synthetic text generators and is yet to take advantage of the factors that potentially influence human-authored text, such as the emotions humans express in the text they write. our work exploits this plm affective deficit to improve synthetic text detection. ## equipping plms with emotional intelligence our method is illustrated in figure 1 . the process works as follows: we conduct experiments using various plm sizes, architectures, datasets, and domains for synthetic text generation and detection. ## chatgpt experiments all experiments so far have involved plms pretrained with the self-supervised objective of predicting the next token or a masked token. we conduct a final experiment with chatgpt, a more human- ## conclusion we conducted experiments investigating the role that emotion recognition can play in the detection 8 chatgpts responses suggest it may use fact-checking as a proxy during synthetic text detection. is it possible that some other proxy for synthetic text is at play? we ruled out some potential proxies related to article length in §4.7.1. in ablation studies in §4.7.3, we showed that the emotion labels result in an improvement in performance compared to randomized labels for the same emotion dataset. other potential proxies are nonsensical sentences, repetitive text, etc. however, we account for these by comparing our emotionally-aware plms (emoplmsynth) against standard plms fine-tuned on synthetic text detection only (plmsynth). thus, any advantage or disadvantage of sentences without meaning (or any other factor) is also available to the non-emotionally-aware model against which we compare our emotionally-aware model. future work will investigate further the affective profile (i.e. emotional content and characteristics) of human and synthetic text; and attempt to determine if there are measurable differences which may prove useful in the task of synthetic text detection.