Quote Detection: A New Task and Dataset for NLP (2023)

Abstract: Quotes are universally appealing. Humans recognize good quotes and save them for later reference. However, it may pose a challenge for machines. In this work, we build a new corpus of quotes and propose a new task, quote detection, as a type of span detection. We retrieve the quote set from Goodreads and collect the spans through a custom search on the Gutenberg Book Corpus. We measure unique vocabulary usage by a state-of-the-art language model and perform comparative statistical analysis against the Cornell Movie-Quotes Corpus. Furthermore, we run two types of baselines for quote detection: Conditional random field (CRF) and summarization with pointer-generator networks and Bidirectional and Auto-Regressive Transformers (BART). The results show that the neural sequence-to-sequence models perform substantially better than CRF. From the viewpoint of neural extractive summarization, quote detection seems easier than news summarization. Moreover, model fine-tuning on our corpus and the Cornell Movie-Quotes Corpus introduces incremental performance boosts.

## introduction human beings have aesthetic appeal. they create and enjoy different works of art. among these, literary works contain the highest form of bookish experience. people enjoy reading novels and highlighting textual segments that are distinctive and memorable, which we can term quotes. humans can readily recognize good quotes and save them for later reference. however, it may pose a challenge for machines since the quote detection task relies mostly on semantic features such as memorability and distinctiveness. the goodreads website 1 stores a collection of quotes that are extracted from different resources to meet users' expectations. this community-wide interest has led us to propose a work in this context. ## goodreads.com/quotes this paper proposes a new nlp task, quote detection, as a variant of span detection, and releases a benchmark quotes dataset. in the literature, there is a movie quotes corpus for binary quote classification @xcite . there is also a similar task of quotation detection and classification @xcite @xcite where the aim is to extract/identify direct, indirect, or mixed speech parts from the text. quote detection is different and unique in that spans represent the free-standing textual segments that are distinctive and favorable for later reference ( to challenge the problem, we first formulate it as a sequence tagging problem and work with a statistical baseline, conditional random field (crf). secondly, we regard it as a type of summarization. to have a baseline performance, we experiment with two neural sequence-to-sequence models; the pointer-generator network @xcite and bidirectional and auto-regressive transformers (bart) @xcite , respectively. the solutions' performances confirm that the task is relatively easier than the existing summarization problems but is a difficult sequence tagging problem. the main contributions of this paper are: ## a corpus of quotes to construct the quote dataset, we rely on two primary resources. the first one is the goodreads platform which shares a voted collection of quotes. the collection consists of 348, 085 instances, each with quote, title, author, likes, and tags columns. we download this collection from kaggle foot_0 . as humans recognize good quotes and user rating is an indicator for recognition, we exclude the rows @xmath0. another filtering criterion is the language of quotes. we detect the language with the help of python's nltk library and remove the non-english quotes. after these two filtering steps, the quote dataset includes 100, 837 rows. the second resource is the free ebook library project gutenberg foot_1 . we download books in plain text format to check whether a quote appears in the referenced book. for this purpose, we search the title and author of the relevant books in the search section of the project gutenberg site and collect the book's plain text links. then, we scrape plain texts using plain text links by the beautifulsoup library of python. we remove the quotes that do not comply with the utf-8 standard and that give a page not found error (404 error). we also exclude song lyrics and philosopher speeches as we cannot extract the contexts they appear from project gutenberg. finally, we discard quotes from some books of contemporary literature that are not accessible. after filtering, the total number of rows is reduced to 8670. to search for a quote in the plain text of the target book, we first trim the standard book header and footer using regular expressions. then, we have a custom search based on the f1 score. we compute the f1 score based on overlap-based precision and recall definitions to determine the best possible match. in our context, precision is the ratio of the number of shared words to the total number of words in the target span, and recall is defined as the ratio of the number of shared words to the total number of words in the ground truth (quote). we also consider the lengths of quotes in this procedure, having faced the fact that a quote can be a phrase within a sentence, a single sentence, or a text made up of a group of sentences. therefore, we process a sliding window of quote length when searching for the closest sentence or sentences in the book text. for example, if the quote consists of three sentences, we calculate the f1 score by sliding over three sentences in the text and return the three-sentence span with the highest score as the most similar context for the quote. the next step is the validation of the returned spans. we arrange quotes in different bins based on f1 score thresholds to decide whether each span corresponds to the wanted quote. the match becomes better as the threshold increases, but the dataset size shrinks. we observe that f1 scores increase as the quote lengths decrease. on the other hand, there is no noticeable difference in the quotes' lengths in each bin. we choose the optimum f1 score threshold as 50%, with an average 2.40 sentence count, 22.97 word count per sentence, and 5015 quotes in total. in the construction of the final collection (t50 foot_2 ), each quote is enclosed by 10 left and 10 right sentences. the appendix a.1 includes an example instance. ## conclusion & future work what makes a sequence of words a quote? although this question is hard to answer, we empirically show that it has a distinctive vocabulary using language model log-likelihoods on t50. this phenomenon was also confirmed by @xcite on movie quotes. moreover, the selected baselines show that it is possible to recognize a quote within its context. ultimately, this paper presents the quote detection task by releasing a new dataset with baseline performances. our results state that quote detection is easier than news summarization using neural summarization. as for sequence tagging, detecting quotes by classifying the beginning and end tokens seems relatively more complicated. thus, there is much room for improvement over mentioned baselines. we hope this task leads to the development of new methods and data sharing.