Mode Effects’ Challenge to Authorship Attribution (2021)

Abstract: The success of authorship attribution relies on the presence of linguistic features specific to individual authors. There is, however, limited research assessing to what extent authorial style remains constant when individuals switch from one writing modality to another. We measure the effect of writing mode on writing style in the context of authorship attribution research using a corpus of documents composed online (in a web browser) and documents composed offline using a traditional word processor. The results confirm the existence of a “mode effect” on authorial style. Online writing differs systematically from offline writing in terms of sentence length, word use, readability, and certain part-of-speech ratios. These findings have implications for research design and feature engineering in authorship attribution studies.

## introduction that authorship attribution techniques work as reliably as they do has been attributed to the fact that each individual has a distinctive writing style. texts written by the same author can be recognized by analyzing lexical and syntactic features in documents @xcite . this principle is practically successful in a variety of settings @xcite @xcite . in some cases, however, authorial style is challenging to detect. for example, authorial style fades as time goes by @xcite , varies considerably in collaborative environments @xcite @xcite , and drifts depending on document genre @xcite @xcite . the aforementioned changes are conspicuous due to the fact that there are certain markers indicating that a document may have been written in a fashion that will lead to stylistic variation. for instance, the presence of genre-specific words indicates a document may present a puzzle for standard authorship attribution techniques. sometimes, however, documents which may challenge an analysis of authorial style can be unannounced. this paper shows that authorial style changes with respect to sentence length, word use, readability, and certain part-of-speech ratios when the writing environment switches from traditional word processing software to an input box of a web browser. ## mode effects originally developed in survey research and educational testing, a "mode effect" describes the following phenomenon: a respondent may answer the same question differently depending on how a survey is administered (e.g., online vs. phone) @xcite . although discussion continues about mode effects' underlying mechanism @xcite , contextualized magnitude @xcite , and adjustment methods @xcite , a consensus has been reached that mode effects can impair survey validity. for instance, @xcite compiled six studies investigating illicit drug use with self-administered and intervieweradministered surveys. the results showed that illicit drug use was reported at higher rates when questions were administered without an interviewer present. functionally, writing modality resembles survey modality: the style observed in an individual's writing may vary depending on how the writer composes the document. a document written by hand may vary from a document composed using traditional word processing software. further variation may be observed if the document is typed into a text box in a web browser. therefore, this research uses "mode effect" to label such differences. ## data participants in this experiment were recruited on amazon mechanical turk (mturk). two distinct types of writing were collected from each of the 18 participants: (1) ca. 6,500 words of pre-existing formal writing and (2) a short ca. 500-word openended response to an essay prompt. for the preexisting writing samples, participants were asked to "submit at least 6500 words total from multiple documents of your own writing that was done for a formal purpose (school essays, grant proposals, etc)." for the 500-word essay, respondents were asked to describe [their] neighborhood to someone who has never been there before as part of a college application. 1 respondents also completed a demographic questionnaire, reporting their gender and age bracket. responses that were not in english or which seemed very likely to be inauthentic were excluded. @xcite discusses challenges dealing with mturk surveys). we also excluded one response which appeared to contain writing copied (without attribution) from online sources. the preexisting writing samples were further processed in order to remove personally identifying information. lengthy quotations, headings, tables, and figures were also removed. the data for this experiment are a subset of data collected as part of research seeking to replicate results in @xcite . in the full replication experiment, respondents were randomly given one of four essay prompts. in this paper, we only used the responses by respondents randomly assigned to the "control" condition. these respondents provided pre-existing writing samples and a response to the essay prompt mentioned above. responses were collected between march 29th and june 1st, 2019. 14 out of 18 respondents reported their age as "18-34". self-reported gender was also collected. ten of the respondents were men and eight were women. the overwhelming majority of pre-existing writing samples collected were essays written for undergraduate courses. many essays discussed films and literary works. many appeared to be written for political science and business courses. the writing prompt generally elicited the desired response: respondents wrote about their neighborhood using formal or semi-formal prose. to check that all writing collected exhibited approximately the same degree of formality, we compared the formality of the writing in the offline corpus with the formality of the writing in the online corpus using a formality score developed by @xcite . we found that the formality scores in each corpus were similar. 2 4 method this paper focuses on stylistic differences introduced by seemingly innocuous variation in the mode used to enter a text-offline composition vs. online typing into a text box. we are interested in whether a mode effect occurs in writing. to the extent that it is observed, we wish to know if its impact on an author's style is predictable. does writing mode induce similar changes in the writing style of different individuals? to answer these questions, we extract linguistic features from the documents written by the 18 participants. we then use a bayesian hierarchical model to estimate differences in the rates at which the linguistic features appear in texts written using different writing modes. ## results before summarizing differences between online and offline writing across all individuals and features, we first consider how a single feature varies in writing from the 18 subjects. this analysis demonstrates how we use bayesian methods to infer collective tendencies in the data. ## mode effects and authorship attribution accuracy another way to understand the magnitude of mode effects is to check if standard authorship attribution techniques have a harder time identifying an individual's writing when presented with the same individual's writing composed in a different mode. that is, we can compare the rate at which an authorship attribution model identifies the correct individual when presented with an unsigned document written offline with the rate at which the model identifies the correct individual when given a document written offline. (the model is trained using writing samples written in the offline mode.) this approach has the virtue of allowing us to answer the question we began with: does the mode effect make authorship attribution more difficult? for this experiment, we use two authorship attribution models featured in @xcite . one is extremely basic, making use of nine features and a simple feed-forward neural network. the second model uses a larger feature set ("writeprints static") and a support vector machine classifier with a linear kernel. 3 authorship attribution accuracy is calculated in the following steps: the svm with a linear kernel used a maximum iteration of 100,000. the neural network used half of the sum of the author count and feature size (which is nine) as the hidden layer count, 100 neurons per layer, and a maximum iteration of 100. all experiments were performed using scikit-learn 0.22.1. figure 5: 95% hpd, rope, and decisions for differences of group scale (η online -η offline ) 3. repeat the previous steps 1,000 times using a different author subsets. calculate the average accuracy over these replications. calculating attribution accuracy for the online writing samples follows similar steps. step 2 differs. the model is trained on pre-existing documents and asked to predict the authorship of a document written online. figure 7 shows our results. it is clear that the accuracy suffers when applying offline-writingtrained classifiers to online writings. that is, changes in authorial style are big enough to confuse the classifiers. writing mode differences make authorship attribution more difficult. ## discussion this study investigated whether individuals' writing style varies by "mode": does the mode used to ). the ropes are set with (-0.1, 0.1) associated with cohen's "small" effect (@xmath0.2). compose a document (word processor (offline) versus web browser text entry (online)) affect measurements of individuals' writing style? our findings confirmed the existence of mode effects. in online writing, respondents tend to use shorter sentences, shorter words, more adjectives (relative to nouns) and pronouns (relative to nouns). sentence lengths exhibit lower variability as well. therefore, we suggest authorship attribution researchers should exercise caution when dealing texts written using different modes. for example, in @xcite , the au-thors attributed lower accuracy in an authorship attribution task to the fact that writers employed authorship attribution circumvention techniques. our research suggests that this lower accuracy may be due in part to differences in writing mode. (the circumvention techniques were only used in online writing.) systematic differences in writing associated with different modes may complicate a broader range of experiments. just as researchers appropriately anticipate genre-dependent stylistic differences in individuals' writing (e.g., fiction vs. non-fiction prose), experiments should also anticipate modedependent differences.