“Shakespeare in the Vectorian Age” – An evaluation of different word embeddings and NLP parameters for the detection of Shakespeare quotes (2020)

Abstract: In this paper we describe an approach for the computer-aided identification of Shakespearean intertextuality in a corpus of contemporary fiction. We present the Vectorian, which is a framework that implements different word embeddings and various NLP parameters. The Vectorian works like a search engine, i.e. a Shakespeare phrase can be entered as a query, the underlying collection of fiction books is then searched for the phrase and the passages that are likely to contain the phrase, either verbatim or as a paraphrase, are presented in a ranked results list. While the Vectorian can be used via a GUI, in which many different parameters can be set and combined manually, in this paper we present an ablation study that automatically evaluates different embedding and NLP parameter combinations against a ground truth. We investigate the behavior of different parameters during the evaluation and discuss how our results may be used for future studies on the detection of Shakespearean intertextuality.

## introduction shakespeare is everywhere. intertextual references to the works of the eternal bard can be found across all temporal and medial boundaries, making him not only the most cited and most performed author of all time, but also the most studied author in the world @xcite . but even though countless studies on shakespearean intertextuality have examined individual aspects of his work by means of close reading, there is still no overview, no big picture, no systematic map of intertextual shakespeare references for larger text corpora. it is also striking that up to now hardly any computational approaches have been used to detect shakespeare references on a larger scale. this is all the more surprising as there are many methods in the fields of computer science and natural language processing for determining the similarity between texts @xcite , which actually can be seen as a formal definition of intertextuality. we acknowledge that the full range of intertextual phenomena cannot be covered by mere means of text similarity determination. for our understanding of intertextuality we therefore refer to the definition of gérard genette, who defines it as "the effective presence of one text in another text" 1 @xcite , where we understand the effective presence of one text in another to be a more or less objectively recognizable, explicit reference on the surface of the text. thus, our approach will not be able to detect highly implicit and indirect references that require a lot of domain knowledge and context. the following variant of a well-known quotation from macbeth (shakespeare's original variant is given in square brackets) would, however, be objectively recognizable from the text and clearly classified as an intertextual reference: by the stinking @xcite of my nose @xcite , something evil @xcite this way goes @xcite . (terry pratchett: "i shall wear midnight"). in order to identify such objectively recognizable references in an automated way, we present an approach that investigates the potential of word embeddings @xcite in combination with this work is licensed under a creative commons attribution 4.0 international license. license details: http://creativecommons.org/licenses/by/4.0/ . ## related work while text reuse detection @xcite mainly finds application in the context of plagiarism detection and the identification of duplicate websites, there are also productive applications in the digital humanities. one example can be found in the project digital breadcrumbs of brothers grimm @xcite , where computational text reuse methods are used to detect motifs of fairy tales across different languages and versions. @xcite present a tool in the intersection of stylometry and text reuse, as they use intertextual distance to classify texts from french literature. @xcite describe an approach for the automatic detection of textual reuses in different works of balzac and his contemporaries . apart from these example studies, the majority of existing research on text reuse in the digital humanities can be located in the field of historical languages and classic studies @xcite @xcite @xcite while clearly there has been interesting work on the problem of text reuse and intertextuality detection in various areas of the digital humanities, there are only very few studies that use computational methods to detect shakespeare quotes @xcite @xcite . with this paper we contribute to computational intertextuality detection in shakespeare studies by exploring a set of parameters that can enhance approaches to searching references based on word embeddings. ## system design: introducing "the vectorian" the vectorian foot_0 is a high-performance sentence alignment search engine foot_1 designed around a number of explicit parameters that model different approaches to scoring sentence similarities. the search engine is also accessible via an internal batch interface for doing large hyperparameter searches, as is the case for the study presented in this paper. throughout the rest of the section we describe the various parameters of the vectorian step by step. an overview of the architecture is shown in figure 1 . in a preprocessing step (top half of figure 1 ), we first split the search corpus into sentences and detect pos tags for each token of the sentences. since the corpus in which we search for shakespeare quotations consists entirely of contemporary literature, the use of spacy 2.3.2 and the en core web lg-2.3.1 is unproblematic. when running a query -i.e. specific shakespeare quotes -we run the same preprocessing on the query text but skip sentence splitting and assume a single sentence. we are aware that the selected spacy model is not optimal for shakespeare quotes because it does not reproduce details of early modern english. however, having looked at a number of samples and the assigned pos tags, we think it works good enough for this first pilot study. we plan to implement a language model that is more specific to shakespeare as a future step. next, the vectorian computes similarities between tokens from the query and the search corpus based on precomputed contemporary word embeddings like fasttext. at this point, only word tokens are used and punctuation is ignored completely. for the shakespearean text in the query, contemporary word embeddings pose an obvious challenge due to shifts in word meaning. we currently do not leverage historical word embeddings like histwords @xcite , but plan to incorporate these in future extensions of this work. after preprocessing and storing the data in an efficient in-memory format suitable for high-performance realtime searches over a large corpus (see bottom right in figure 1 ), we compute alignments based on similarity scores between the tokens (see bottom left in figure 1 ). ultimately, scores are derived from the embeddings and controlled by various parameters -the details are contained in a pipeline we refer to as sim full (see big box in figure 1 ) and which will be described in more detail in the next sections (see the right half of figure 2 ). given the token similarity scores, we find optimal alignments on the sentence level using the waterman-smith-beyer algorithm @xcite ), which we leverage through an optimized and highly customizable implementation foot_2 . ## evaluation design in this section we present an ablation study in which we automatically test different combinations of the parameters that were described in the previous section, in order to investigate how they influence the results. the ground truth required to carry out such an evaluation was derived from @xcite , who conducted a comprehensive study to identify references to shakespeare in a corpus of postmodern fiction using a mixture of close and distant reading. we took a subsample of this work, which contains 73 quotes from one of shakespeare's most popular plays: hamlet. the rationale for taking only a subsample is that the ground truth cannot be considered 100% comprehensive, as was shown in related studies with the dataset @xcite . we kept the sample size small in order to simplify the task of recognizing new true positives identified by the vectorian. the 73 quotes are distributed among 31 novels that have a total size of 4,2 million tokens. sticking to the search engine metaphor introduced in section 2, we will treat each of the 73 hamlet quotes as a query that is searched for in the collection of novels. in the evaluation study, each query is assigned an unranked set of expected results (as documented in the ground truth) in the corpus of novels. each result refers to one specific sentence in a novel. some queries have multiple expected results, e.g. for "there are more things in heaven and earth ..." our ground truth records 8 occurences in different novels. however, most queries have only 1 or 2 matches in our ground 0.0 0.2 0.4 0.6 0.8 1.0 base of value of x k 10 -6 10 -4 10 -2 increase in mean figure 3 : scatter plot of absolute change in harmonic mean (blue) and arithmetic mean (orange) over random scores x 1 , ..., x n when updating a single input score x k by an improvement ✏ of 0.1. y axis is logarithmic. the arithmetic mean always increases by a constant ✏ n regardless of x k 's value, so in an optimizer it encourages increasing low and high scores similarly. the harmonic mean on the other hand tends to weigh the same improvement in a low input (left) considerably higher than in a high input (right) and therefore encourages increasing low inputs. truth foot_6 . in total, our ground truth contains 149 result sentences for 73 unique queries. we measured the performance of each query by computing the discounted cumulative gain (dcg) @xcite against the unranked foot_7 ground truth for the first 50 results retrieved. dcg is a standard measure to rank the quality of a result set in information retrieval, where documents from higher ranks contribute more to the overall gain and documents at lower ranks contribute less, i.e. they are discounted. since a full grid search was not feasible for our parameter space, we ran an optuna @xcite optimizer with the objective of maximizing a total performance score. this total performance score is computed as a mean over the dcgs of all queries. however, since some queries expect more results than others -and therefore the ideally obtainable dcgs for different queries vary -such a composite score only makes sense if all contributing dcgs have been scaled to a fixed range. therefore we employed the commonly used formulation of normalized dcgs (ndcgs) @xcite to normalize the score for each query into the range between 0 and 1. to summarize these considerations: we used ndcg@50 for each query and then computed a mean to obtain a total performance score. we ran a first optimization with the objective of maximizing the commonly used arithmetic mean over all query ndcgs as the maximizing objective, and a second independent optimization with the objective of maximizing a harmonic mean of query ndcgs. the harmonic mean may seem like an unusual choice here, as its use in information retrieval is typically limited to the f-score @xcite . the reason we use it in this scenario, is the distribution of query difficulty in our ground truth and the specific characteristics of the harmonic mean. we found a very strong negative skew due to a high number of rather simplistic queries in our data. there are about 50% of queries that relate to verbatim or nearverbatim quotes of text from shakespeare, which means that these are rather easy to detect from a text reuse perspective. about 10% of the queries on the other hand rely on implicit knowledge and are probably not easily found by the vectorian, which relies entirely on explicit language features. the remaining 40% of the queries are neither trivial nor out of reach for the vectorian. we therefore put them into the category hard but feasible. optimizing on the arithmetic mean carries the risk of micro-optimizing the bulk of easy queries to an ndcg of 100%, but finding no good ndcgs for the few but more interesting queries. in order to encourage good ndcgs for hard but feasible queries, the harmonic mean seems to be a reasonable alternative. as figure 3 illustrates, it tends to improve by higher values when low inputs get increased. in our evaluation study, we ultimately ran optimizations for both types of means with 1,500 trials using a default configuration with tree-structured parzen estimators @xcite . as a caveat, it must be noted that due to the small size of our ground truth sample, our evaluation did not have a dedicated validation set. since the parameters in our system are few and quite restricted, however, we believe the risk of overfitting is rather low. we interpret our results as a simplistic model that represents deeper characteristics of the query-result relationships given in our ground truth. ## results and discussion the best configuration found with optuna produced ndcgs of 77.6% and 75.2% for the arithmetic (a) and harmonic (h) mean optimization objectives respectively. since the decision what constitutes a quotation in some cases cannot be made on the language level alone and thus is highly subjective @xcite , we believe that these scores can be interpreted as a fairly good performance. the system also produced a small number of new true positive matches, which could be confirmed to be valid foot_8 . the specific parameter values for the best configurations are given in unfortunately, due to the skewed nature of the query problem in our ground truth, the mean value of the ndcgs tells us only little about the performance of the two variants with respect to different types of queries. figure 4 therefore gives a detailed histogram of the query ndcgs. as argued in the evaluation design, the distribution of the hard but feasible queries with low scores turned out differently indeed. most notably for h, we see a salient group of (orange) queries scoring between 0.4 and 0.7, while the queries scoring at ndcg 0 and 1 both have been slightly diminished. in general, this is what we hoped for. the downside however is that the beneficial arithmetic (blue) peak between 0.7 and 0.8 is now gone, i.e. we have lost this score for some queries. figure 5 shows the performance of both variants in terms of quantiles. both variants operate optimally on the maximum 100% ndcg level for easy queries that are located at the quantiles above 0.5. between the 0.15 and 0.5 quantiles however, the arithmetic mean variant performs better. on the other hand, the h variant does not show the dip below the 0.15 quantile, which seems to give it slightly better performance for some difficult queries. we now discuss the parameters' importance by performing an ablation study on each of them, starting with the harmonic variant (see figure 6 ), which shows a surprising combination foot_9 . emi and pmp have no effect at all -any value produces the same optimal results -and nearly the same is true for esm. in other words, the whole embedding pipeline seems to be irrelevant for the search results. furthermore, spw, sif, and ifs are all basically no-ops. the only three salient choices are a sit above 0.8, a mlp of 1 -that shows an interesting option for extending it up to 4 -and a slightly elevated sbo value. in summary, large parts of the vectorian engine have been turned off in this case in order to facilitate a specific kind of search, namely looking for alignments of tokens without using any pos or embedding information. ## conclusion we have investigated the optimal configuration of various explicit parameters in a text reuse detection pipeline and showed that it is able to achieve ndcgs of roughly 80% on a rather difficult test set. while for some queries focusing on the interplay of word embeddings, pos tags and alignments is optimal, other queries seem to benefit from turning off these features. we have demonstrated, how these choices are generated naturally by maximizing arithmetic and harmonic means of ndcg scores. our analysis uncovered important ideas of what makes queries hard for our current architecture and indicates the need for a ground truth that is classified and balanced in terms of difficulty. as @xcite described considerably more shakespeare references in his study, we plan to enhance the ground truth accordingly and classify the quotes according to different categories (e.g. verbatim quote, semantic paraphrase, changed word order, etc.). we hypothesize that different types of quotes will result in different optimal parameter combinations. this will be investigated in more detail in a follow-up study, where we will also look into other types of embeddings and explore single parameters in more detail.