Text Zoning of Theater Reviews: How Different are Journalistic from Blogger Reviews? (2021)

Abstract: This paper aims at modeling the structure of theater reviews based on contemporary London performances by using text zoning. Text zoning consists in tagging sentences so as to reveal text structure. More than 40 000 theater reviews going from 2010 to 2020 were collected to analyze two different types of reception (journalistic vs digital). We present our annotation scheme and the classifiers used to perform the text zoning task, aiming at tagging reviews at the sentence level. We obtain the best results using the random forest algorithm, and show that this approach makes it possible to give a first insight of the similarities and differences between our two subcorpora.

## introduction since 2010 in england, a wave of blogs written by authors coming from various horizons has arisen on the internet. students, theater professionals but also mere amateurs began publishing their own theater reviews. these new independent voices in the digital space progressively redefine the shape of classic journalistic criticism. although discreet, they offer a new vision of the history of londonian theaters. by doing so, it sets itself apart from the canon of mainstream journalism. the emergence of this digital culture triggered a lot of controversies on the status of the review as a literary object. michael billington, reviewer for the guardian since 1971, states that a blog 'is more like an informal letter: a review, if it's to have any impact, has to have a definable structure.' for danielle tarento, co-founder of the menier chocolate factory, 'a lot of people @xcite are not 'proper writers'.' at the other hand of the spectrum, some of these bloggers claim the stylistic singularity of their publications. in the description of exeuntmagazine.com for instance, the editors claim that: 'exeunt believes in making beautifully written, experimental, fierce and longform writing about theatre available for free. @xcite a review is traditionally organized according to several sections: an introduction, a presentation of the plot, a few lines on the stage, etc. in order to compare the two subcorpora, it is first necessary to segment the reviews into textual zones corresponding to these thematic sections. we assume the two subcorpora will share the same zones, as they are all about theater, but the content of the zones may differ from one subcorpus to the other: e.g., the two communities may not focus on the same aspects of the plays. from a technical point of view, this experiment is also an opportunity to test the robustness and relevance of text zoning across different domains. text zoning has been mainly used to segment scientific texts so far, but can this technique also be used in the humanities? can it be used for performance reviews, where critics do not follow a fixed structure, contrary to scientific writing? this short paper is structured as follows. we first give a brief overview of text zoning. we then present our corpus, the different features and machine learning techniques used for the task. we then comment our results and give some hints on the way these could be used to get a better understanding of the content of the corpus and the differences between the two communities at stake (official critics vs amateur bloggers). ## previous work the notion of text zoning was first introduced by simone teufel in her phd @xcite . teufel was targeting the automatic analysis of scientific papers. in this context, argumentative zoning refers to the 'rhetorical status of a sentence with respect to the communicative function of the whole paper.' it is for example quite useful to distinguish 'back-ground information' from 'statements of the particular aim of the current paper', to take an example from teufel's work @xcite . while text zoning has been mainly applied to scientific texts so far, one can also find this technique applied to other domains where it is relevant, for example email messages @xcite , or job ads @xcite . the number of zones considered varies but is generally around ten or less (7 for example in @xcite and @xcite , or 8 in @xcite ). zone annotation is generally performed by a group of experts at the sentence level (more rarely at the paragraph level). inter-annotator agreement on the task is generally high: @xcite for example reports a score of 0.85 for cohen's κ @xcite . once a representative corpus is available, it is possible to train a classifier for the task. features considered are generally low level (unigrams, bigrams, sometimes specific terms also receive a specific weight) @xcite but higher level features are also sometimes considered (like syntactic relations in @xcite ). contextual information (like the previous zone) is also often taken into consideration, since a specific zone tends to appear in typical positions in scientific abstracts. as for training, most recent ml techniques have been explored, from naive bayes @xcite to lstm @xcite , through crf and svm @xcite . in this last paper, the authors also investigate semi-supervised learning and active learning, in order to reduce the amount of data needed for training, which often constitutes a bottleneck for the task. more recently, large language models like bert have also been explored @xcite ), but they require large amount of data for training. here our goal is partly the same as the one in these previous studies. however, our corpus is very different since we analyze theater reviews, which may not be as regular as scientific papers. in our context, zones are important to determine whether the critic is addressing acting, staging or the general setting of the play (we use the rather neutral term 'text zoning' instead of 'argument zoning', since the zones we consider do not always correspond to arguments). analyzing the overall organization of theater reviews will also make it possible to determine whether these have a rather fixed structure or not, if reviews in newspapers differ a lot from those directly written for blogs on the web. ## corpus creation to answer these questions, the first step consisted in collecting the necessary data to create two subcorpora. the first subcorpus is made of journalistic reviews only, while the second one is based on digital theater reviews written by bloggers. ## results we applied each model on the data and computed their accuracy (computed using 10-fold cross validation and averaging the results across folds). our results are reported in note however that we had to find the optimal parameter for the depth of the tree and the number of trees. by doing so the random forest model uses a system of threshold for each important feature @xcite . to improve the model and avoid overfitting, we also used cross validation during the training and test steps. as planned, naive bayes is not able to take into account the complexity of the task and performs poorly. knn also fails at capturing the variations of the different zones, as the texts to classify are quite short. lastly, rnn performs worse as there are not enough data to train this model properly. for this part in particular, we used long short term memory neural networks which work sequence by sequence. we wanted to use pre-trained models but none of them had already been trained on a similar dataset for this task. the closest we could get was on the imdb dataset. however, if it were annotated for sentiment analysis, it were not for zoning. figure 1 represents a review with the different zones identified, each color corresponding to a specific zone. this figure illustrates how the algorithm works. the model looks for each sentence and calculates its probability of being part of one of the 8 predefined categories. of course it is possible that one sentence may have different recognizable patterns that makes it belong to several classes. in this case, the model associates to the sentence the category which has the highest percentage. in function of the class assigned by the model, the sentence will then take the color related to its category. ## discussion although the accuracy of the algorithm could be improved, these first results are a reliable and relevant base to better understand the comparison between printed and digital theater criticism. if the debate in the artistic sphere highlights the differences between journalists and bloggers, the experiments actually prove that their reviews are more similar than what they claim. each of the 8 categories we had defined are represented in the two datasets (see ## conclusions and perspectives we have presented a study based on the automatic analysis of more than 40 000 theater reviews on the contemporary londonian stage. we have shown that it is possible to segment these reviews into labeled text zones with a good accuracy. in the future, we want to investigate large language models and their potential benefit for the task. considering the classification obtained with text zoning, it seems that the two subcorpora considered in the study (journalists vs bloggers) are not as different as some actors of the domain may have claimed, at least from a distant reading perspective. however the content of some zones seems to be really different from one subcorpus to the other: the zoning experiment presented in this paper is thus a first necessary step in order to be able to perform a more precise analysis.