Linguistically Motivated Features for Classifying Shorter Text into Fiction and Non-Fiction Genre (2022)

Abstract: This work deploys linguistically motivated features to classify paragraph-level text into fiction and non-fiction genre using a logistic regression model and infers lexical and syntactic properties that distinguish the two genres. Previous works have focused on classifying document-level text into fiction and non-fiction genres, while in this work, we deal with shorter texts which are closer to real-world applications like sentiment analysis of tweets. Going beyond simple POS tag ratios proposed in Qureshi et al.(2019) for document-level classification, we extracted multiple linguistically motivated features belonging to four categories: Lexical features, POS ratio features, Syntactic features and Raw features. For the task of short-text classification, a model containing 28 best-features (selected via Recursive feature elimination with cross-validation; RFECV) confers an accuracy jump of 15.56 % over a baseline model consisting of 2 POS-ratio features found effective in previous work (cited above). The efficacy of the above model containing a linguistically motivated feature set also transfers over to another dataset viz, Baby BNC corpus. We also compared the classification accuracy of the logistic regression model with two deep-learning models. A 1D CNN model gives an increase of 2% accuracy over the logistic Regression classifier on both corpora. And the BERT-base-uncased model gives the best classification accuracy of 97% on Brown corpus and 98% on Baby BNC corpus. Although both the deep learning models give better results in terms of classification accuracy, the problem of interpreting these models remains unsolved. In contrast, regression model coefficients revealed that fiction texts tend to have more character-level diversity and have lower lexical density (quantified using content-function word ratios) compared to non-fiction texts. Moreover, subtle differences in word order exist between the two genres, i.e., in fiction texts Verbs precede Adverbs (inter-alia).

## introduction written text can be classified into various categories based on its content or writing style. this paper focuses on classifying shorter texts into fiction and non-fiction genres based on their writing style. in general, the fiction writing has an imaginative writing style and involves non-factual prose content. in contrast, the non-fiction writing deals with actual events, places, and persons and is written purely based on the facts. in some cases, distinguishing between these two writing categories is easy due to the content of the text. however, classification becomes challenging in many instances due to blurry boundaries between them. for example, a short story may contain imaginary characters situated in real-life settings. therefore it is essential to factor in the writing style of texts while classifying them into fiction and non-fiction genres. the problem of genre identification using linguistically motivated features has been extensively investigated in nlp (see references in the rest of this section). however, the particular problem of fiction vs. non-fiction genre classification has started receiving serious attention only in recent years @xcite . in the cited works, different features have been studied for classifying document-level texts (or long texts) into fiction and non-fiction genres. in contrast, very little effort has been expended to investigate the set of relevant features which are effective for the classification of shorter texts, i.e., paragraph-level texts into fiction and non-fiction genres. shorter texts or paragraph-level texts are more common on the internet and have several important practical applications like breaking news detection, opinion mining, micro-blog summarization, and discovering trending topics @xcite . genre identification tools for shorter text can potentially be deployed to filter out specific categories of tweets, news headlines, product reviews, and online app reviews which have been written to manipulate or influence the users/customers. for such applications, fiction vs. non-fiction classification technology capable of analyzing writing styles can play a crucial role. the main objective of this paper is to identify the most relevant features that not only enable one to build an effective classifier but also provide deeper insights about the properties that can be used to distinguish these two genres in shorter texts. to this end, we deployed features belonging to four categories, i.e., raw features, pos-ratio features, lexical features and syntactic features @xcite @xcite . raw text features quantify the basic properties of the text, like sentence length and variation in sentence length within a paragraph. lexical features are based on the statistics of words or characters present in the corpus. in writing, vocabulary plays an important role as it involves the coordination of many higher levels and lower levels of cognitive skills @xcite . previous studies have also used various measures of lexical diversity to discern differences between genres @xcite . we deployed a character-level diversity estimate and a lexical density estimate (ratio of content to function words). pos-ratio features proposed by @xcite compute the ratios of different parts of speech tags present in the corpus, e.g. adverb/noun, adjective/verb, and verb/pronoun. they found these ratios to be very effective for document-level fiction vs non-fiction classification (accuracy of 96.31 % on brown corpus text @xcite ). while most of our features were adapted from prior work, we introduce 3 novel syntactic features extracted from parse trees in this paper. prominently, we modelled word order variation across genres by extracting head-dependent bigrams (containing linear order precedence as well). inspired from the theoretical psycholinguistics literature, we also incorporated features quantifying syntactic complexity @xcite as well as argument-adjunct patterns @xcite . we extracted the aforementioned features 1 from brown corpus paragraphs and performed feature selection experiments using the recursive feature elimination cross-validation algorithm (rfecv @xcite on individual and combined feature sets. we report the performance of different classification models trained on several feature combinations and compare them with a baseline 1 scripts to extract various linguistic features used in this work can be accessed here: https://github.com/ armankazmi/linguistic-features-of-text model with only two pos-ratio features found effective in prior work @xcite . our classification model containing the best 28 features confers an accuracy score of 91.89% on brown corpus @xcite paragraphs with an accuracy jump of 15.56% over the baseline model containing qureshi et al.'s simple pos ratio features (76% accuracy on short-text classification in the brown corpus). in order to check the transferability and generalizability of our results, we used the aforementioned model trained on the brown corpus (american english text) to classify shorter texts obtained from the baby bnc corpus of british english @xcite . our model obtained an accuracy score of 94% which attests its utility for novel text and demonstrates how it is not biased w.r.t. language variety, i.e., american english (brown) vs british english (baby bnc). following previous work in the nlp literature @xcite @xcite , we also compared our classification results based on a traditional logistic regression model (containing handcrafted features) with 2 deep learning models. on shorter text from both brown and bnc corpora, a 1d cnn model induces a 2% increase in accuracy score over the logistic regression classifier. finally, we used a pre-trained bert-base-uncased model @xcite resulting in state-of-theart accuracy of 97% on brown corpus and 98% on baby bnc corpus respectively. although both the deep learning models (cnn models and the bert-base-uncased models) result in better results in terms of classification accuracy, they are not easily interpretable i.e., linguistic properties captured by these deep learning models are not obvious. another issue is that cnn and bert models are expensive to train from scratch and are more prone to overfitting when compared to the logistic regression classifier. on the other hand, our experimental results using simple logistic regression models are interpretable in terms of the impact of specific features. our regression coefficients indicate that fiction texts tend to be more diverse in terms of characters and have lower lexical density than non-fiction texts. subtle differences in word order between the two genres can also be inferred from the coefficients our dependency bigram features. for example, verbs tend to precede adverbs and pronouns in the case of fiction texts, in contrast to the opposite pattern in non-fiction texts. our main contribution is that we extend the work of @xcite on document-level genre classification to the problem of genre-identification for shorter text by incorporating theoretically and cognitively motivated features. our bestperforming model containing linguistically motivated features substantially outperformed their best-performing model for this novel task. the features deployed by @xcite (another recent work on fiction vs non-fiction classification cited earlier) are very elaborate but are not directly connected to cognitive theories. earlier works like @xcite and mendhakar (2022) analyzed various linguistic characteristics of fictional and non-fictional text but focused more on sub-genre classification within fiction and non-fiction genres. the rest of the paper is organized as follows. in section 2 we present details of the data sets used in this study. section 3 provides the motivation and descriptions of the linguistic features used in this work. section 4 describes the machine learning algorithms we deployed and the results of genre classification experiments using those algorithms. section 5 discusses the implications of our findings. finally, in section 6, we summarize all the results and provide pointers for future research. ## data and methods our dataset consists of paragraphs from brown corpus @xcite and baby british national corpus @xcite . these corpora contain text from fiction and non-fiction genres, thus serve an important resource for our research. we set up a binary classification task to predict shorter texts into fiction and non-fiction genres. therefore, every long document in these corpora was split into separate paragraphs based on the default paragraph annotation provided. after that, each paragraph was tagged to the class based on the class label of their parent document. to mitigate the data imbalance between the two classes since different paragraphs may have varying lengths in terms of the number of sentences, we chose only those paragraphs that had 5 or 6 sentences, and the rest were discarded. as a pre-processing step, we automatically tagged and parsed the paragraphs in our dataset using state-of-the-art taggers and parsers. we used stanza @xcite for parts-of-speech tagging and stanford corenlp toolkit for dependency and constituency parsing @xcite . the punctuation marks in the paragraphs were stripped off prior to their parsing. we then extracted a wide variety of linguistic features from the tagged and parsed text, thus creating a vector representation of each paragraph. the set of features used for the classification task and the underlying motivation behind using them is described in the subsequent section. we used a traditional machine learning model (logistic regression) as well as two deep learning models (cnn and bert) for our classification task as described in section 4. to further our understanding of our classification models, we tested the model's applicability in british english, where we use british national corpus @xcite . this way, we perform transfer learning where the model is learned on one corpus, and its applicability is tested on another corpus. this also provides a more robust way of analyzing our model's predictions. baby bnc corpus consists of four categories: fiction, newspaper, spoken, and academic. following @xcite , we considered academic documents in nonfiction category and fiction documents in fiction category, and rest others were excluded from our primary analyses. as they mention, the news genre lacks a clear demarcation foot_0 in either category. ## linguistic features for genre classification of shorter texts, we deployed the following four distinct categories of features in our work: 1. raw text 2. lexical 3. pos ratios 4. syntactic features. these features and their motivation are described below. ## experiments and results this section presents the results of our experiments for classifying shorter texts into fiction and nonfiction genre. the following subsections describe our classification results using a traditional machine learning model (logistic regression) and two deep learning models (cnn and bert). ## discussion overall, our classification results show that deep learning models achieve better accuracy than the traditional machine learning model for the task of genre identification in shorter text. however, one of the main objectives of this work is to identify the properties of the text that help distinguish between fiction and non-fiction genres. this objective is accomplished using the traditional logistic regression model as the regression coefficients enable interpretation of the features used in the model. though the deep learning models confer better accuracy on our task, it is difficult to interpret them. the cnn and bert models used in our work were trained on thousands of parameters. word embedding vectors capture the properties of the words in the texts, but the complex structure of the internal representations of these models make it difficult to discern the exact generalizations learned by the models in question. in order to understand the impact of various features in distinguishing between fiction and nonfiction genres, we interpret the regression coefficients of our logistic regression model (depicted in dependency relations also provide important cues that help distinguish between fiction and nonfiction genres. fiction is characterized by a greater number of syntactic subjects, oblique noun modifiers, 's possessives, and discourse markers compared to non-fiction. in contrast, non-fiction texts are characterized by a greater frequency of numeral modifiers, passive voice sentences, relative clauses, and multi-word expressions. further, subtle word order differences between the two genres act as effective predictors of paragraph-level genres. in fictional paragraphs, verbs tend to precede adverbs and pronouns while proper nouns are likely to occur before verbs. in non-fiction paragraphs, proper nouns (propn) and verbs tend to precede numbers (num) and pronouns precede verbs. see figure 1 (a) for a visual illustration of the above patterns in the brown corpus genres. similarly, figure 1(b ) depicts the genre-wise percentage of content and function words computed over all the words in the brown corpus. it suggests that the content and function word percentages in the non-fiction genre are greater than in the fiction genre. further, figure 2 represents the percentage of genre-specific parts of speech tags (computed over the total number of brown corpus parts of speech tags), where the percentage of nouns in fiction is greater than that in non-fiction while adverbs and verbs have similar distributions across both genres. in the rest of this section, we illustrate the impor-tance of our features using linguistic examples (the verb is shown in bold, and the adverb is in italics). given below are the first two sentences taken from the brown corpus fiction paragraphs (fileid: cn13, paragraph number: 22): these sentences depict the case where verbs precede adverbs in fictional text. these examples also indicate how the later adverb plays a crucial role in providing extra information about the verb, thus augmenting the imaginative quotient of the text. we prove two further examples below from nonfiction texts taken from brown corpus (fileid: ce16, paragraph number: 17), which shows the genrespecific tendency of adverbs preceding verbs: finally, we also checked the performance of all the models (traditional as well as deep learning models) on newswire text from the baby bnc corpus (605 paragraphs from the 97 news category documents) individually as well as combined with our baby bnc dataset (described in section 2 used in previous experiments). ## conclusions and future work in this work, we classified paragraph-level text into fiction and non-fiction genres using a traditional machine learning model (logistic regression) and two different deep learning models. for shorttext genre identification, we show that the traditional model containing hand-crafted features (raw text, pos ratios, lexical and syntactic features) significantly outperformed a baseline model containing pos-ratio features, (originally proposed by @xcite for the task of documentlevel genre classification). we also obtained the insight that subtle differences in word order exist between the two genres, i.e., in fiction texts verbs precede adverbs (inter-alia) compared to non-fiction texts. finally, we showed that deep learning models (viz., cnn and bert) perform significantly better than our traditional model. we obtained state-of-the-art results for the task of shorttext genre identification using a pre-trained bert model fine-tuned on the brown corpus. in future work, we intend to investigate the efficacy of the hand-crafted features on a larger data set and also plan to create a gold standard corpus of human-annotated fiction and non-fiction paragraphs for fine-grained evaluation. future research needs to investigate whether syntactic complexity and arguments/adjuncts patterns (not having any impact using our current machine learning setup) are effective predictors of genre shorter texts using other learning algorithms. further, our finding that journalistic prose (as in news) is not purely non-fiction in nature and might contain fictional elements, needs more systematic investigation. another line of future inquiry is to combine traditional models (encoding linguistic features) with state-ofthe-art deep learning models. finally, it would be interesting to investigate if causality expressed in the natural language text plays an essential role in classifying text into fiction and non-fiction genres. b syntactic depth-based feature calculation for the example sentence, "the children ate the cake with a spoon", we describe the method to compute @xcite measure of syntactic complexity. figure 3 illustrates the constituency parse tree of the example sentence. now, we define the lineage of a word as the class of nodes, including the leaf node (terminal node) associated with that word, the root node of its tree, and all the intermediate nodes on the unique path between leaf and root nodes. now, according to sampson depth of a terminal node is defined as the total number of those non-terminal nodes in the word's lineage with at least one younger sister. 6 the depth of the word 'cake' in the example sentence according to the above definition is 1. similarly, the depth of each terminal node or the word in the sentence in sequence from left to right is 1,1,0,1,1,0,0,0, which sums to 4. the depth-based measure is the average over the leaf nodes; hence, the value is 0.5 in this example.