Understanding and Explicitly Measuring Linguistic and Stylistic Properties of Deception via Generation and Translation (2020)

Abstract: Massive digital disinformation is one of the main risks of modern society. Hundreds of models and linguistic analyses have been done to compare and contrast misleading and credible content online. However, most models do not remove the confounding factor of a topic or narrative when training, so the resulting models learn a clear topical separation for misleading versus credible content. We study the feasibility of using two strategies to disentangle the topic bias from the models to understand and explicitly measure linguistic and stylistic properties of content from misleading versus credible content. First, we develop conditional generative models to create news content that is characteristic of different credibility levels. We perform multi-dimensional evaluation of model performance on mimicking both the style and linguistic differences that distinguish news of different credibility using machine translation metrics and classification models. We show that even though generative models are able to imitate both the style and language of the original content, additional conditioning on both the news category and the topic leads to reduced performance. In a second approach, we perform deception style “transfer” by translating deceptive content into the style of credible content and vice versa. Extending earlier studies, we demonstrate that, when conditioned on a topic, deceptive content is shorter, less readable, more biased, and more subjective than credible content, and transferring the style from deceptive to credible content is more challenging than the opposite direction.

## introduction as online social media usage continues to grow, it is becoming easier to access a much wider va-riety of news sources than ever before. around two-thirds of u.s. adults get at least some of their news from sources on social media. 1 however, with the lack of traditional fact-checking and verification processes that accompany more standard news sources, this leads to a significant potential for the spread of false, misleading, and harmful information. the growing impact of such information in the online environment has lead to increased attention, awareness, and efforts to understand and combat its spread @xcite . in social media, a great deal of attention has been dedicated to detect and measure the spread and impact of deceptive news @xcite . many researchers @xcite @xcite @xcite have analyzed linguistic differences to build models that classify types of deceptive news content. there are several limitations to the use of classification models to understand differences between deceptive and trustworthy content. firstly, such models may learn to rely on the most prominent distinguishing lexical features between news categories but may not learn to model more subtle stylistic differences. secondly, these models may learn context-dependent features that will evolve as the topics and news events change over time. our major contribution is to understand and explicitly measure linguistic and stylistic properties of content from misleading versus credible news sources while mitigating the topical bias. for that, we develop and rigorously evaluate two separate strategies: a generation approach to generate con-tent of different credibility news source categories and a translation approach, which involves converting deceptive tweets to credible and vice versa while preserving the meaning. our goal is to generate corresponding text in the style of credible and misleading news sources that will allow analysis of stylistic differences only, controlling for topical differences that would exist in the full corpus of real tweets. first, for the text generation task, we aim to learn a generative language model that can produce news content characteristic of news sources of varied credibility in order to separate the linguistic and stylistic differences of these news sources. we aim to determine whether such models can reliably produce text that is characteristic of news sources of varied credibility, whether we can additionally control the topic of such generated text, and how we can best evaluate the performance of the generative models. we demonstrate that we can effectively generate text of different news source categories but that performance is reduced when additionally conditioning on topical indicators. therefore, to perform linguistic analysis controlling for topic we turn to the translation task, for which we first create a parallel corpus and then take advantage of encoder-decoder architectures to train a transformation function to convert misleading to credible content and vice versa. rather than aiming to modify the information content of the text, we aim to modify the stylistic properties while leaving the content as unchanged as possible. ## related work efforts related to the detection of deception in written text has examined deceptive language in several domains such as fake news @xcite , political speeches, online opinions about topics such as abortion or death penalty @xcite . most of the existing models for deception detection rely on linguistic features such as n-grams, language complexity, part-of-speech tags, and syntactic and semantic features @xcite @xcite . linguistic inquiry and word count (liwc) @xcite features were used to show that deceptive texts had fewer self-references, to avoid own self involved in the lies, and more words related to certainty (always, all, truly), possibly due to the need of the speaker to emphasize the "fake" truth @xcite . recent work in deception detection focused on developing predictive models to classify fake and verified news @xcite , assess information credibility (wang, 2017), understand linguistic cues that distinguish fake and real news stories @xcite , and characterize the signatures of coordination @xcite . recently, linguistically infused neural network architectures were developed for (a) classifying social media posts as credible or deceptive or as different types of deceptive news @xcite , (b) factuality assessments of statements with different levels of credibility @xcite , and (c) classifying deceptive strategies and understanding intent behind deception @xcite . text generation models leveraging deep learning architectures and neural language models have been applied to domains ranging from biographies @xcite to conversational text @xcite , and recently to generating rumors @xcite , fake news @xcite and stylometry @xcite . large-scale pre-trained language models, such as the generative pretrained transformer models, gpt-2 @xcite and @xcite , have achieved excellent performance in freeform text generation, while approaches specifically for controllable text generation have been developed e.g., plug and play @xcite and the conditional transformer language (ctrl) model @xcite . unlike any prior work on understanding the language of deception, we propose to disentangle the topic bias via translation and generation approaches to understand and measure stylistic and linguistic differences between misleading and credible news of different levels of credibility ranging from disinformation to credible news. ## data while we would ideally like to identify deceptiveness at the level of individual news stories, due to the difficulty of this task and the high potential for misclassification, we instead focus on the classification of text at the news source level similar to @xcite . the news sources that we study are classified into the following categories based on publicly available lists of news sources annotated by experts. • disinformation sources provide partial, distorted, or false depictions of reality. topical indicators for generation task we aim to develop generation models that can generate text conditional on both news source type as well as specified topical content. as a proxy for content topic, we condition on different combinations of named entities mentioned in the tweets. we detect named entities in the tweets using the allennlp models foot_1 and annotate each tweet with a one-hot encoded indicator of the presence of each of the most common 250 named entities in the corpus. parallel corpus creation to support the translation task, we aligned tweet pairs from credible and deceptive news sources when they both were published on the same day (temporal proximity) and when the pair meets our threshold of content similarity. that is, when (a) all named entities and at least 70% (manually determined threshold) of the nouns in the deceptive tweet also occurs in the credible tweet, and (b) the (subject, verb, object) tuples obtained using syntaxnet dependency parser in both deceptive and credible tweets match. however, we do not want to have pairs with perfect similarities, as it would not serve our goal to study the differences between the style of credible and deceptive news sources. to obtain the final set of tweet pairs from the aligned set, we compute the text similarities for each pair using edit distance and cosine distances of word2vec, doc2vec, and tfidf. after rigorous manual inspection we retain only those pairs with edit similarity that falls within the (0.35, 0.7) range. this results in 105,365 aligned (credible, deceptive) tweet pairs. 3 4 approach generation models to support our feasibility analysis, we leverage a three layer long short-term memory (lstm) word-level language model for text generation. to create the conditional generative models, we add an embedding layer for the news source category that learns a dense representation for each of the six categories. the relevant embedding is appended to each token embedding in the input sequence, such that the model is tasked with predicting the next token given a modified version of the input token representation that is specific to the class of the input. when conditioning on named entity in addition to the news source, we add an additional component to the embedding by applying a dense layer to a one hot encoded representation of 250 most frequent named entities present in the tweet. for all models we use the adam optimizer, 256-dimensional hidden layers, a batch size of 256, and 200 epochs trained on a tesla p40 gpu. translation models sequence-to-sequence neural models have significantly advanced the state-ofthe-art in a variety of natural language processing tasks such as machine translation, speech recognition, and text summarization @xcite . we propose multiple variations of encoderdecoder models that learn a function that transforms tweets from deceptive news sources to the style of credible news sources, and the other way around. we run experiments with two commonlyused types of recurrent layers -lstms and gated recurrent units (grus) -for the encoders and decoders. for both lstms and grus we used the adam optimizer with a learning rate of 0.001, 256-dimensional embeddings, categorical crossentropy loss, a batch size of 64, and 100 epochs with early stopping. given the success of attention mechanisms for language tasks, we additionally experiment with multiplicative- @xcite and self-attention @xcite mechanisms.we conduct translations in both directions: (1) forward: credible to deceptive, and (2) reverse: deceptive to credible. a conceptual schematic of both the generation and translation approaches is shown in figure 1 . ## generation results for the generation task, we aim to evaluate both the quality of the generated text and the similarity of the generated text to the desired conditional news category. the sign of good model performance is when the text similarity is higher between generated tweets of a specific category and real tweets of the same category than it is with real tweets of a different category. we first evaluate the performance of the models that were conditioned on only the news source category. examples of tweets of each category that were generated using these models can be found in generation evaluation with bleu in order to determine the textual similarity of tweets both within and across categories, we calculate the bleu scores of both real and generated tweets from a given news source category with the same category and with different categories. because we do not have one-to-one correspondence between real and generated text examples for the generation task, we calculate the bleu score of each generated example against a sample of the full real corpus. we equivalently generate a bleu score for a sample of real tweets against the real corpus to determine the typical n-gram similarity of real tweets to other real tweets. by comparing these two scores, we can determine how the similarity of generated samples compares with the expected similarity of real samples. figure 2 summarizes the bleu score results for both real and generated content. we find that the bleu score similarity to the target category is similar to what is observed for real content from the same category. however, the bleu score similarity to content from categories other than the target category is significantly higher than what is observed for real cross-category content. for all categories we find that the bleu score ratio of the generated content is greater than one, indicating that generated content is more similar to the target category than the other categories. however, in all cases the ratio is still significantly less than what is observed for the real content indicating that the models may be "hedging" and skewing their results towards similarity to the full corpus. we find that generated content that are characteristic of disinformation news sources come the closest to matching the bleu-score ratio of real tweets. for models which were conditioned on named entities, we can see that the raw bleu scores are lower both within category and across category compared with the simpler model. this indicates that this model produces text that differs more from the real tweet corpus as a whole. additionally, the bleu score ratio has been reduced to below 1.0 for the hoax news category, indicating that these generated tweets are not more similar to real hoax tweets than real tweets of other categories. these results might be explained by the additional conditioning causing the generated distribution to differ from the real distribution. because our goal is to generate text that stylistically matches the different news categories, even though we force the model to focus on topic areas that are not typical of that category, we aim to probe the stylistic similarity of real and generated text. below we explore additional dimensions of evaluation to probe the stylistic quality of the conditional generation. ## translation results to evaluate the translation task, we rely on a set of metrics used to evaluate machine translation and text generation systems: bleu @xcite , ibleu @xcite , fk-bleu @xcite , sari @xcite , and gleu @xcite . we present performance results of translations between deceptive and credible aligned tweets in ## conclusions and future work we have presented a novel understanding of linguistic and stylistic properties deception using translation and generation approaches designed to disentangle from the topic bias. we have demonstrated progress towards developing generative models for this purpose, with our generative models being able to imitate both the style and the content of the real tweets. we have evaluated our generative models using bleu scores, linguistic analysis, and classification. however, with additional topical conditioning dimensions, we find significantly reduced performance on maintaining the observed stylistic and content differences. in order to improve the performance of these generative models, future work includes the application of more advanced controllable text generation model transformer-based and other recently emerged neural architectures @xcite @xcite @xcite . we will focus on improving the ability to perform multi-dimensional conditioning to enable the desired topic-controlled misleading versus credible news source style analysis. we have shown that translating from misleading to credible content is more difficult than the opposite direction. this may be because deceptive content has a higher level of stylistic and lexical variation making it more difficult for the models correctly anticipate its style, while verified content has a more factual style. our translation results clearly demonstrate that, when conditioned on a topic, content from deceptive news sources is shorter, less readable, more biased, and more subjective than content from deceptive news sources. leveraging these models to allow topic-agnostic comparison of the style of deceptive and credible news sources, we demonstrate several key stylistic differences.