Content-based Models of Quotation (2021)

Abstract: We explore the task of quotability identification, in which, given a document, we aim to identify which of its passages are the most quotable, i.e. the most likely to be directly quoted by later derived documents. We approach quotability identification as a passage ranking problem and evaluate how well both feature-based and BERT-based (Devlin et al., 2019) models rank the passages in a given document by their predicted quotability. We explore this problem through evaluations on five datasets that span multiple languages (English, Latin) and genres of literature (e.g. poetry, plays, novels) and whose corresponding derived documents are of multiple types (news, journal articles). Our experiments confirm the relatively strong performance of BERT-based models on this task, with the best model, a RoBERTA sequential sentence tagger, achieving an average rho of 0.35 and NDCG@1, 5, 50 of 0.26, 0.31 and 0.40, respectively, across all five datasets.

## introduction unlike in scientific writing, where authors use terse citations due to space constraints, direct quotation of source material is an essential part of writing in many fields. journalists, humanities scholars, and students, for instance, often quote from a range of source documents, such as interviews, speeches, and books. quotes can be used to substantiate a claim, lend authority to an argument, or offer a viewpoint to reflect on or argue against, among many others. modeling the process of quotation selection is thus an important step in modeling how authors compose entire new documents. in this paper, we explore the problem of quotability identification, identifying which passages in a source work (e.g. hamlet) are likely to be quoted by later, derived works (e.g. humanities journal articles). prior research has attempted to identify the specific factors that influence a passage or document's quotability. most work, therefore, has focused on manual feature engineering and development of careful analysis frameworks to test which features have statistically significant relationships to quote counts @xcite . we, instead, reframe quotability identification as a practical passage ranking task and evaluate how well models can rank the passages in a given document by their predicted quotability. we benchmark and analyze the performances of multiple models, exploring both bert-based passage ranking models and featurebased models equipped with "quotability" features identified in prior work. we collect five large-scale datasets to study this problem. each dataset consists of sets of source documents, derived documents, and alignments between them -the direct quotes. the proposed datasets are diverse, allowing us to model quotability dynamics across multiple source and derived document genres and languages. uses for our proposed passage ranking task include: 1) to help users discover quotable source content for use in an essay or article @xcite ; 2) to select striking quotes from new books to help readers get a sense of them; 3) for use in extractive summaries, search result snippets, and other compressed versions of a text -since quotability implies the ability of a passage to have meaning outside its original source context, quotability scores could be incorporated with traditional measures of informativeness and non-redundancy to determine which passages should be excerpted. the primary contributions of this paper include: ## related work there has been substantial previous research in identifying and analyzing what makes specific source content popular, i.e. how many times a given source is quoted, cited, retweeted, etc. the source documents analyzed in prior work span a wide range of domains, from political speeches and debates @xcite to books @xcite , movie scripts @xcite , scientific articles @xcite , tweets @xcite , and news articles @xcite . the aligned derived documents also span multiple domains, from social media @xcite @xcite @xcite to news articles @xcite , and scientific papers @xcite . prior work has focused on both 1) predicting the popularity of an entire source document, e.g. a scientific article's citation count @xcite and 2) similar to our work, identifying which specific passages in a given source work will receive the most attention @xcite @xcite . the most similar work to ours, @xcite , also explores applications of quotability identification models to literary works (project gutenberg). however, due to their lack of supervised data, they instead focus on the problem of identifying what sorts of passages are likely to be quoted, rather than modeling what sorts of phrases are actually quoted in derived works. the contributions of prior work have been, primarily, feature engineering (e.g. number of personal pronouns, use of negative/positive words) and design of testing frameworks to determine which features have a statistically significant relationship to quotablity/popularity in a single language and in a specific domain of interest (e.g. english language movie quotes and their popularity on imdb: danescu-niculescu-mizil et al. ( 2012) ). unlike prior work, our main contributions are not feature engineering, but a re-framing of the task as a practical document-level ranking task and an analysis of several models through extensive experiments with multiple datasets spanning various source and derived document genres and languages. ## problem formulation we formulate quotability identification as passage ranking, identifying which passages in a source document are likely to be quoted in related derived works. we measure quotability directly by counting how many times a passage is quoted across a collection of derived texts. concretely, given a source document and a set of derived texts, we -1. use fuzzy text alignment methods from text reuse detection, e.g. @xcite , to identify alignments (quotes) between subsequences in the source and derived texts. ## datasets we have collected five datasets across two languages (english, latin) and multiple genres (nov-els, poems, plays, scripture, etc.) to study this problem. each dataset consists of source texts (e.g. shakespeare's plays, books of the bible) and a set of alignments to a corpus of derived documents (e.g. humanities journal articles, news articles). each alignment is an instance where a derived work quotes a passage(s) in a specific source work, e.g. an author quotes from hamlet, "to be, or not to be: that is the question," in a humanities journal article. ## linguistic attributes of quotations as a first step at modeling the quotability identification problem, we draw on quotability features from prior work @xcite @xcite and attempt to identify which linguistic attributes influence passage quotability. we explore both lexical (e.g. passage begins with a stop word) and part of speech features (e.g. contains past tense verb). for each dataset, we featurize each of the corresponding passages, then compare its highly-quoted (top 20%) and minimally-quoted (bottom 20%) passages. we use welch's t-test with bonferroni correction to test whether the scores for each feature are significantly different between the two groups. feature set kjb-ca kjb-ja shak-ja abl-ja lat-ejc length: number of tokens in passage. @xcite # words ↓↓↓↓ ↑↑↑↑ ↑↑↑↑ ↑↑↑↑ ↑↑↑↑ capitalized: proportion of words capitalized @xcite . % capital ↓↓↓↓ ↓↓↓↓ ↓↓↓↓ ↓↓↓↓ ↑↑↑↑ stop words: 1) proportion of words in passage that are stop words, 2) binary feature if passage begins with a stop word @xcite . universal quantifiers: binary feature if passage contains a universal quantifiers (20 quantifiers, e.g., all, whole, nobody). @xcite . ## models next, we examine the effectiveness of two classes of models on our passage ranking task: featurebased models with "quotability" features and neural models for sentence classification. we benchmark both pointwise regression models for count data and listwise and pairwise ranking models. ## experimental settings we evaluate models using five-fold cross validation. we train and evaluate models on each dataset separately and report means over the five folds for each. we split datasets into folds at the document level (e.g. train: frankenstein, etc., val: emma, etc., test: paradise lost, etc.). then, for a given validation or test set, we evaluate performance on each document separately, then average over documents. for the feature-based models, we select hyperparameters by performing nested five-fold cross validation on each fold's training set, again splitting by document. due to computational restrictions, for the neural models we select hyperparameters by using 20% of the documents in the fold's training set as a validation set. see appendix h for a list of all hyperparameters. ## results & analysis tables 5 and 6 display the results of all models across all five datasets foot_1 . on the whole, we find that modeling a passage's context in its broader document is important, with the bert-based sequential sentence models roberta seq and xlm-r seq performing the best. they achieve the highest average ndcg score across all ranks k on three datasets (kjb-ca, shak-ja, lat-ejc) and the highest ρ on all five datasets. the sequential sentence models achieve their highest relative performance on shak-ja, outperforming the second-best models by roughly 40% relative on ndcg and 19% on ρ. similar to results on other passage ranking tasks @xcite , we find that the single passage bert-based models (roberta single , xlm-r single ) provide strong baselines, outperforming the feature-based models on nearly all datasets and achieving the secondbest overall performance. furthermore, on kjb-ja, roberta single achieves the best ndcg performance, outperforming roberta seq by an average of 0.02 across each k. investigating the kjb-ja results further, we find that roberta single outperforms roberta seq on 34 of the 64 kjb books. of these 34, five books (numbers, revelation, zechariah, leviticus, and peter-1) account for over a third of the total increase in ndcg over roberta seq . in four of the five books, roberta single successfully ranks the most quoted passage at the top, while roberta seq ranks passages with single-or near-single-digit labels. notably, roberta seq fails, while roberta single succeeds, in properly ranking the great commandment from leviticus 19:18 "... love thy neighbour as thyself" at the top. on the other hand, the pairwise bert-based models (roberta pair , xlm-r pair ) generally perform worse than the single passage and sequential passage bert-based models. they struggle to identify the top, most quoted passages in each document (as measured by ndcg), but perform relatively better at ranking each document's entire list of passages w.r.t each other (ρ), though still worse than the sequential passage models. among the feature-based models, while svm rank achieves the highest ρ on all five datasets, no single model consistently outperforms the others in ndcg, with pr, svm rank , and λmart each achieving the highest scores on different datasets. however, only one feature-based model, λmart, ever outperforms the neural models, achieving the highest ndcg scores on abl-ja. we hypothesize that λmart's strong performance on abl-ja might be due, in part, to differences in the accuracy of our feature-extraction pipeline. many of our features depend on accurate parsing (e.g. pos and verb tense counts). however, for our english datasets, the stanza universal dependencies model @xcite we use to process each sentence is trained on web-media data (ud english ewt). thus, our english datasets (kjb, abl, shak) are all out-of-domain. we hypothesize that stanza is more accurate on abl since it contains the most modern language similar to its training data. with these higher quality inputs, therefore, feature-based models can achieve higher performance, relative to the neural models. as one might note, our datasets are also out-of-domain for roberta and xlm-r. however, as shown by @xcite , bert-based models can adapt to new domains when provided with in-domain fine-tuning data. for our latin data, although our lat-ejc dataset is in-domain for the latin stanza model (ud latin perseus), the stanza model might not perform as well since the ud training data is quite small. ## conclusion we explore the task of quotability identificationidentifying which passages in a source document are likely to be directly quoted by later derived documents. we cast quotability identification as a passage ranking problem, evaluating how well models can learn to rank the passages in a source document by their predicted quotability. we evaluate on five large-scale datasets spanning multiple source genres (e.g. poetry, novels, plays) and languages (english, latin). we conduct experiments with feature-and bert-based models, finding that although relative performances vary across datasets, on the whole, bert-based models operating on strings of sequential passages perform best. we have identified two potential directions of future research using the datasets described in this study. first, using the publication date information for the journal articles and newspapers, we could investigate temporal quoting trends, testing hypotheses such as the matthew effect, and therefore the best predictor of a passage's quotability tomorrow is its popularity today. finally, we could explore second-order effects, studying trends in what sorts of passages are often quoted together.