Screenplay Quality Assessment: Can We Predict Who Gets Nominated? (2020)

Abstract: Deciding which scripts to turn into movies is a costly and time-consuming process for filmmakers. Thus, building a tool to aid script selection, an initial phase in movie production, can be very beneficial. Toward that goal, in this work, we present a method to evaluate the quality of a screenplay based on linguistic cues. We address this in a two-fold approach: (1) we define the task as predicting nominations of scripts at major film awards with the hypothesis that the peer-recognized scripts should have a greater chance to succeed. (2) based on industry opinions and narratology, we extract and integrate domain-specific features into common classification techniques. We face two challenges (1) scripts are much longer than other document datasets (2) nominated scripts are limited and thus difficult to collect. However, with narratology-inspired modeling and domain features, our approach offers clear improvements over strong baselines. Our work provides a new approach for future work in screenplay analysis.

## introduction the motion picture industry is a multi-billion dollar business worldwide @xcite . decisions in selecting movies to be produced are critical to the profitability of a movie studio. however, the selection of the screenplay that happens at the initial phase of a movie production pipeline and has a large influence on the financial budget and quality of the final movie production, has a large subjective element. for example, a typical script review service costs a studio @xmath0150 to receive a report containing a short summary of the script and opinion as to its quality @xcite . considering the amount of scripts a studio needs to filter through, it can be overwhelming. thus, an objective and reliable tool to help evaluate and narrow down the candidate scripts is of vital importance to aid the "green-lighting" (deciding which scripts to turn into movies) process. consider this scenario, if a tool can facilate the script review process and provide the chance of success, wouldn't this make an impact and cut down lots of budgeting decisions in the production process? the main idea of this work is to develop such a tool which gather custom analyses from various aspects, e.g., screenplay writing theory, characterfocused linguistic behavior, to help assess the quality of the script. in general, movie script writing can follow a well-defined three act structure @xcite . also, weiland @xcite specifies a more fine-grained storytelling plan, starting from hook, inciting event, 1st plot point, 1st pinch point, midpoint, 2nd pinch point, 3rd plot point, climax to resolution, what are called structural points (sp). we believe knowledge like the above in strucuring a screenplay can bring benefits in selecting the most relevant textual properties for the prediction of script quality. aside from the event positioning, @xcite reported that how writers develop characters and events, i.e., characterization and plot, are two main foci of industry reviewers. we thus devise our domain specific features in these two aspects. we hope to offer an enhanced understanding of the essential elements in high-quality movie scripts. to perform quality assessment, based on an assumption that the nominated scripts are recognized writings and thus should have had higher chance of passing green-lighting, we propose to perform an evaluation in a two-fold approach. first, we use award-nomination prediction as a proxy to the green-lighting process. second, we examine our domain features and models by integrating them into existing document classification methods. we acknowlodge the constraints of our metric in that the number of award venues has its limits, and not necessarily those without nomination would be any worse than the nominated. but due to the difficulty in collecting unproduced scripts with peer reviews, we adopt our current approach. our main contributions are as follows: (1) we defined a quality metric for screenplays and collected ground truths from peer-reviewed venues. a noteworthy attempt in measuring quality of literary works we know of is made by @xcite , who quantitatively analyze various indicators for discerning professional poems from amateurs'. however, in script writing, the cinematic success criteria lack evaluative consensus (simonton, 2009) -previous works on evaluation of movies have largely focused on forecasting revenue or profit of movies using production, distribution, and advertising data @xcite or basic textual and human annotated features @xcite . the main differences between our work and previous works are: (1) our approach aims to process automatically without human annotated features. problem setup. our work focuses on measuring quality as whether or not a movie would be nominated at a peer-reviewed venue. the basic assumption for using this approach as success metrics is simple -a screenplay that receives nominations by critical reviewers should have had higher chance of getting through green-lighting. challenges. by nature, a movie should be tough to be cleanly categorized, due to its length, complex storyline and turns, and the lack of evaluative criteria. prior works in document classification @xcite @xcite evaluated on datasets with small document size (reuters, imdb, yelp, etc.). however, our document size on average is at least 65 times longer, which may be challenging for nn-based models to train due to long sequences and the associated computational burden. besides, the number of training data we have is at most 1000 times smaller than other datasets. with our datasets being long, fewer and skewed, state-ofthe-art deep learning techniques may not work well. summary of the comparisons is shown in ## analysis of domain features in this section, we introduce our domain features that are divised to achieve our goal and provide analysis based on our problem setup. characterization and plot are major aspects of focus in the industry; inspired by which, we devised 6 novel features. for each, we provide intuitive motivations, and then detail how we converted them computationally. we chose the top two most speaking characters of each movie to analyze for characterization. according to @xcite , a script can place 9 sps roughly equally distributed, creating eight equal-lengthed development segments (ds) in between. we hypothesize that such structural hints should help to achieve our objective. based on the statistics of both datasets, to leverage the sps, we collected a context window of 1% (∼270 words) centered at sps for all scripts. larger windows may contain more information and should improve the results, and we leave that for future experiments. by the definition of characterization, we hypothesized that by measuring pattern change of characters, we may see how writers develop the characters' personality. we sought pattern change via two kinds of changes writers would make between sps -linguistic (speaking pattern) change and emotional change. to do this, we proposed linguistic & emotional activity curve. linguistic & emotional activity curve (ling, emo). for linguistic change, we extracted the dependency trees of characters; for emotional change we used normalized empath @xcite to get characters' emotion status. we combined the linguistic distribution, empath distribution of sentences in each ds with activity curve @xcite , which uses a permutation-based change detection in activity routine (pcar) algorithm, to measure the change between two dss of distribu-tions. ## predictive modeling in this section, we define our prediction task, and then propose our base model and then move on to a paradigm which integrates domain features proposed in previous section. task formulation. as a proxy to the original quality assessment task, we define a binary classification task as to predicting the nomination of a script. narratology-inspired model. inspired by narratology, we propose tfidf-svm narr -instead of using all texts in an entire document, we extract words in context window of sps for each document, compute the tf-idf representations, and feed them into a svm classifier. the main components of tfidf-svm narr are shown in fig. 2. due to the large amount of unique tokens, we chose only the top 500 important features ranked by tf-idf to represent a document. we test the results without choosing 500 features and our setting is better. feature-based prediction. to examine the predictive power of proposed features, on top of tfidf-svm narr , we add domain features along with tf-idf to svm to see the efficacy of domain features. 6 experimental setups dataset usage. we performed random sampling on both datasets such that 80% is used for training, 10% for validation, and 10% for test. baselines. we adopted han @xcite , bert base , bert large @xcite as our baselines. since a script is subdivided into scenes, method / dataset scriptbase msc han scene 45.12 45.62 bert base 42.67 46.29 bert large 42.67 46.29 tfidf-svm 47.01 59.21 tfidf-svm narr 57.43 59.21 + emo + vad 56.52 55.29 + ling + emo + tt 62.35 62.73 + int + ling + emo + clus 60.87 64.79 ## results and discussion we report the macro-averaged f1 scores of each model in in fig. 3 , we show the effect of each individual feature. linguistic & emotional activity curve show improvements on both datasets, and yet the rest do not consistently help, especially on msc, we think it may be because (1) the tfidf has 500 dimensions so individual feature may be overwhelmed, but, more features combined such as adding int+ling+tt can generate consistent improvements, (2) the efficacy of feature can be dataset-dependent, e.g., we do not observe significant differences in arousal of msc as in its script-base counterpart (fig. 1 ), and so does the classifier. besides, adding features with negative correlations can damage the performance, e.g., adding emo & vad. ## conclusion and future work we present a novel approach and features to systematically analyze the quality of a screenplay in terms of its festival nomination-worthiness. this can serve as a preliminary tool to help filmmakers in their decision-making, or on the other hand, an objective way for writers to compare their works with others. our results also show that simple lightweight approach can outperform state-of-theart document classification methods. this also points out the current deficiency for long document classification research in the community. in the future, in addition to textual properties, we intend to develop a more fine-grained approach by incorporating more metadata such as gender of characters, film genres, and then experiment on different award categories to evaluate our approach and gain more insights.