Syntax and Themes: How Context Free Grammar Rules and Semantic Word Association Influence Book Success (2021)

Abstract: In this paper, we attempt to improve upon the state-of-the-art in predicting a novel’s success by modeling the lexical semantic relationships of its contents. We created the largest dataset used in such a project containing lexical data from 17,962 books from Project Gutenberg. We utilized domain specific feature reduction techniques to implement the most accurate models to date for predicting book success, with our best model achieving an average accuracy of 94.0%. By analyzing the model parameters, we extracted the successful semantic relationships from books of 12 different genres. We finally mapped those semantic relations to a set of themes, as defined in Roget’s Thesaurus and discovered the themes that successful books of a given genre prioritize. At the end of the paper, we further showed that our model demonstrate similar performance for book success prediction even when Goodreads rating was used instead of download count to measure success.

## introduction since its publication in 1868, approximately 1.78 million copies of louisa may alcott's little women have been sold, which equates to about 1,000 copies a month for 152 years. every publisher in the industry hopes to find a manuscript that can sell even 10,000 copies in its lifetime. this begs the question: what makes little women a timeless success? recently, researchers have attempted to use machine learning and natural language processing to answer this question, among others. predicting the success of a novel by analyzing its content is a challenging research problem. thousands of new books are published every year, and * both authors contributed equally to this research. figure 1 : this figure represents the lexical production rules of context free grammar observed in both successful and unsuccessful books classified by goodreads ratings. we present the count of 20 lexical rules, each normalized with respect to it's highest occurrence in any book, for 10 successful and 10 unsuccessful books from children genre. we see that certain lexical rules occur more frequently in successful books than unsuccessful books. only a fraction of them achieve wide popularity. therefore, the ability to predict a book's success prior to publication would be exceptionally useful to the publishing industry and enable editors to make better decisions. many factors contribute to a book's success including, but not limited to plot, setting, character development, etc. additionally, there are some other factors that contribute to a book's popularity that an author and publisher cannot control like the time when a book is published, the author's reputation, and the marketing strategy. in this paper, we only focus on the content of the book to predict its popularity. in this paper, we explore whether novels in a spe-cific genre have certain dominant themes in common based on stylometric features, and if so, what meaning we can attribute to those themes as it relates to a book's success. to attain this objective, we investigated ways to enable using the entire book's content for stylistic modeling using frequencies of lexical production rules of cfgs for each novel (see figure 1 ) followed by semantic word association of those rules to roget's categories and themes for better interpretation. in this work, we followed widely used feature reduction technique for svm modeling to reduce the large lexical feature space for lengthy novels and experiment with different techniques using pos, unigram, word-net, and lexical production rules. in this article we present the following contributions: ## related works roget's thesaurus is a widely used englishlanguage thesaurus. a british lexicographer, peter mark roget (1779-1869), created the thesaurus in 1805. the first version of the thesaurus comprised of nearly 15,000 words and was released to the public on 29 april 1852 @xcite . since then each successive edition was improved with more words, with the most recent edition containing more than 100,000 words. in previous work, @xcite showed that roget's is an excellent resource for measuring semantic similarity and roget's word clusters have higher correlation than many other prominent word groups e.g., wordnet @xcite . syntactic features, such as cfg productions have been found to be very effective in different nlp tasks. @xcite used cfgs for authorship attribution achieving very high accuracy such as 96%. @xcite presented systematic analyses based on lexical and syntactic features for genre detection of a literary works showing that novels involve more use of verbs and adverbs. on the other hand, @xcite showed that informative writing tend to use nouns, prepositions, determiners and coordinating conjunctions more. cfgs were also used in several other works, such as gender attribution by tracing stylometric evidence by @xcite , and native language detection by exploiting parse structures @xcite . in the earlier work, ashok et al. ( 2013 ) used stylistic approaches, such as unigram, bigram, part-of-speech distribution, grammatical rules, constituents, sentiment, and connotation as features and used liblinear svm @xcite for the book success classification task. they used books from 8 genres, and they were able to achieve an average accuracy of 73.50% across all genres. @xcite used a set of hand-crafted features in combination with a recurrent neural network and generated feature representation to predict success. they obtained an average f1-score of 73.50% for 8 genres. in a more recent work by @xcite , they used the flow of emotion throughout a book for success prediction and obtained an f1-score of 69%. in this paper, we used widely used feature reduction technique for svm modeling. @xcite used svm weights for assigning ranks in the feature selection process. they verified that the topranked genes found by svm have biological relevance to cancer and the svm classifier with svm selected features worked better than other classifiers in determining the relevant features along with the classification task. ## experimental results using the original small dataset @xcite , the prediction accuracy for each model by genre is presented in the prediction accuracy for each model across all books, and each model by genre, both before and after feature reduction are shown in in ## interpreting book success prediction while our reduced γ g and wordnet models display excellent performance in both test settings (by genre and independent of genre), the resulting feature sets are not self-explanatory. in other words, the respective lexical production rules and synsets that the models deem most important do not necessarily highlight some interesting aspect of successful books. this is where roget's thesaurus proves most valuable. we figured that if we looked up the roget theme of the rhs for each lexical production rule and the roget theme for each wordnet synset we would find that the successful and unsuccessful books prioritize different themes. with this hypothesis in mind, we mapped the reduced wordnet and reduced γ g models to new roget models by first looking up the roget category of each synset and rhs, respectively, from the reduced feature sets, and then summing the frequencies in each group of sysnets/symbols. as we did with each previous model, we reduced the new wnrc and γ g rc models. from the wnrc r and γ g rc r models we mapped again, this time from roget categories to the 23 roget themes, which produced the wnrt and γ g rt models. mapping examples are given in we did not expect the performance of the wnrc and γ g rc models, since they were conceived strictly as intermediary maps between wordnet/γ g and roget themes. γ g rc produced the highest baseline results of all the models without any feature reduction used in our experiments with 81.9% average accuracy by genre. furthermore, γ g rc r accurately predicts success classification per genre at an average rate of 88.4%. what's impressive about the accuracy of γ g rc r , when compared to that of γ g , is the large difference in number of features used in each model as shown when predicting detective novels in with these impressive results from γ g rc r , we expected γ g rt and γ g rt r to follow suit despite learning with a feature set of at most 13 features. however, this was not the case as γ g rt r predicts the success of a book by its genre with an average accuracy of only 69.9%. as previously stated, the motivation for the construction of wnrt and γ g rt was strictly to find a common thread between successful novels in each genre. therefore, the poor performance of the wnrt r and γ g rt r models does not undercut the reasoning behind its conception, and the high accuracy of wordnet r , wnrc r , γ g r , and γ g rc r supports our claim that each is a general model that can reveal underlying characteristics of successful books. additionally, wnrt and γ g rt do not improve performance after feature reduction when classifying independent of genre. this outcome also supports our original hypothesis as it shows that the models require each of the 23 roget themes in order to make the most accurate prediction. the lack of improvement in wnrt r and γ g rt r when predicting success class independent of genre also demonstrates the relationship between a novel's genre and its prioritization of certain themes. ## experiments with goodreads rating the discoveries made in our research are just the beginning of what can be done with our dataset. in addition to the data utilized for this project, we also extracted goodreads rating 3 as proposed in @xcite . we could collect the rating for 7,541 books from a total of 17,962 books scraped from project gutenberg, where each book has been rated by at least 5 readers. we labeled all the books having average @xmath0.5 as successful, @xmath1.5 as unsuccessful (presented in when predicting novel success by genre, γ g shows the best results predicting a book's success class with an average weighted f1-score of 92.2% outperforming previous stateof-the-art results(75% for the genre attention with rnn method (maharjan et al., 2018b)) as well. this result validates the applicability of our proposed model for book success prediction. ## conclusion we created the largest dataset for evaluating book success, and presented a novel study of how context free grammar rules and semantic word association of influence a book's success. our empirical results demonstrate that our large dataset combined with our feature reduction technique can predict a book's success with better accuracy than the current stateof-the-art methods. the analysis performed in this project shows the relationship between thematic word groups and a book's popularity, with our best model that uses context free grammar lexical production rules (γ g r ) achieving a prediction accuracy of 94.0%. finally, we illustrated that readers expect certain themes to be prioritized over others based on a book's genre, and the proper use of those themes directly contributes to a book's popularity.