REL i C : Retrieving Evidence for Literary Claims (2022)

Abstract: Humanities scholars commonly provide evidence for claims that they make about a work of literature (e.g., a novel) in the form of quotations from the work. We collect a large-scale dataset (RELiC) of 78K literary quotations and surrounding critical analysis and use it to formulate the novel task of literary evidence retrieval, in which models are given an excerpt of literary analysis surrounding a masked quotation and asked to retrieve the quoted passage from the set of all passages in the work. Solving this retrieval task requires a deep understanding of complex literary and linguistic phenomena, which proves challenging to methods that overwhelmingly rely on lexical and semantic similarity matching. We implement a RoBERTa-based dense passage retriever for this task that outperforms existing pretrained information retrieval baselines; however, experiments and analysis by human domain experts indicate that there is substantial room for improvement.

## introduction when analyzing a literary work (e.g., a novel or short story), scholars make claims about the text and provide supporting evidence in the form of quotations from the work @xcite @xcite . for example, @xcite claims that elizabeth, the main character in jane austen's pride and prejudice, doesn't just refuse an offer to join the standoffish bachelor darcy and the wealthy bingleys on their morning walk, "but does so in such a way as to group darcy with the snobbish bingley sisters," and then directly quotes elizabeth's tongue-in-cheek rejection: "no, no; stay where you are. you are charmingly grouped, and appear to uncommon advantage. the picturesque would be spoilt by admitting a fourth." literary scholars construct arguments like these by making complex connective inferences between their interpretations, framed as claims, and quota-tions (e.g., recognizing that elizabeth says "charmingly grouped" and "picturesque" ironically in order to group darcy with the snobbish bingley sisters). this process requires a deep understanding of both literary phenomena, such as irony and metaphor, and linguistic phenomena (coreference, paraphrasing, and stylistics). in this paper, we computationally study the relationship between literary claims and quotations by collecting a largescale dataset for retrieving evidence for literary claims (relic), which contains 78k scholarly excerpts of literary analysis that each directly quote a passage from one of 79 widely-read english texts. the complexity of the claims and quotations in relic makes it a challenging testbed for modern neural retrievers: given just the text of the claim and analysis that surrounds a masked quotation, can a model retrieve the quoted passage from the set of all possible passages in the literary work? this literary evidence retrieval task (see figure 1 ) differs considerably from retrieval problems commonly studied in nlp, such as those used for fact checking @xcite , open-domain qa @xcite , and text generation @xcite , in the relative lack of lexical or even semantic similarity between claims and queries. instead of latching onto surface-level cues, our task requires models to understand complex devices in literary writing and apply general theories of interpretation. relic is also challenging because of the large number of retrieval candidates: for war and peace, the longest literary work in the dataset, models must choose from one of ∼ 32k candidate passages. how well do state-of-the-art retrievers perform on relic? inspired by recent research on dense passage retrieval @xcite , we build a neural model (dense-relic) by embedding both scholarly claims and candidate literary quotations with pretrained roberta networks @xcite , which are then fine-tuned …elizabeth comes to pemberley full of fear of being treated as an interloper, a trespasser; even before any plans of visiting the ancient house are made, the mention of visiting derbyshire makes elizabeth feel like a thief: [masked quote] she seems to be afraid of encountering, if not the horrors of a gothic castle, at least the resentment of a stern aristocrat… it is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife. (@xmath0 step 1: compute context embedding c by passing the text of the literary claims and analysis that surrounds a missing quotation to a roberta network "but surely," said she, "i may enter his county with impunity, and rob it of a few petrified spars without his perceiving me (@xmath1 darcy, as well as elizabeth, really loved them; and they were both ever sensible of the warmest gratitude… our dense-relic model is trained with a contrastive loss to push a learned representation of the surrounding context close to a representation of the ground-truth missing quotation (here, the 4,387 th sentence from the novel). to facilitate future research in this direction, we publicly release our dataset and models. 1 ## collecting a dataset for literary evidence retrieval we collect a dataset for the task of retrieving evidence for literary claims, or relic, the first large-scale retrieval dataset that focuses on the challenging literary domain. each example in relic consists of two parts: (1) the context surround- ## literary evidence retrieval having established that the examples in relic contain complex interplay between literary quotation and scholarly analysis, we now shift to measuring how well neural models can understand these interactions. in this section, we first formalize our evidence retrieval task, which provides the scholarly context without the quotation as input to a model, along with a set of candidate passages that come from the same book, and asks the model to retrieve the ground-truth missing quotation from the candidates. then, we describe standard information retrieval baselines as well as a roberta-based ranking model that we implement to solve our task. ## human performance and analysis how well do humans actually perform on relic? to compare the performance of our dense retriever to that of humans, we hired six domain experts with at least undergraduate-level degrees in english literature from the upwork foot_9 freelancing platform. because providing thousands of candidates to a human evaluator is infeasible, we instead measure human performance on a simplified proxy task: we provide our evaluators with four sentences on either side of a missing quotation from pride and prejudice 16 and ask them to select one of only three candidates to fill in the blank. we obtain human judgments both to measure a human upper bound on this proxy task as well as to evaluate whether humans struggle with examples that fool our model. human upper bound: first, to measure a human upper bound on this proxy task, we chose 200 test set examples from pride and prejudice and formed a candidate pool for each by including bm25's top two ranked answers along with the ground-truth quotation for the single sentence case. as the task is trivial to solve with random candidates, we decided to use a model to select harder negatives, and we chose bm25 to see if humans would be distracted by high string overlap in the negatives. each of the 200 examples was separately annotated by three experts, and they were paid $100 for annotating 100 examples. the last column of it is delicious from the opening sentence: [masked quote] mr. bingley, with his four or five thousand a year, had settled at netherfield park. @xcite [human]: it is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife. sometimes we hear mrs bennet's idea of marriage as a market in a single word: [masked quote] her stupidity about other people shows in all her dealings with her family... @xcite [human]: "i do not blame jane," she continued, "for jane would have got mr. bingley if she could." human readers understood the uncommon usage of "got" to convey a transaction. issuing out-of-distribution queries to the retriever: does our dense-relic model have potential to support humanities scholars in their evidence-gathering process? inspired by promptbased learning, we manually craft simple yet out-ofdistribution prompts and queried our dense-relic retriever trained with 1 sentence of left context and no right context. a qualitative inspection of the top-ranked quotations in response to these prompts ( limitations: while these results show dense-relic's potential to assist research in the humanities, the model suffers from the limited expressivity of its candidate quotation embeddings q i , and addressing this problem is an important direction for future work. the quotation embeddings do not incorporate any broader context from the narrative, which prevents resolving coreferences to pronominal character mentions and understanding other important discourse phenomena. for example, table a5 shows that dense-relic 's top two 1-sentence candidates for the above pride and prejudice example are not appropriate evidence for the literary claim; the increased relevancy of the 2-sentence candidates ( retrieving cited material: citation retrieval closely relates to relic and has a long history of research, mostly on scientific papers: @xcite formulated the task of document retrieval using "citing statements", which @xcite revisit to create a reference retrieval tool that recommends references given context. @xcite examine the rhetorical structure of citation contexts. perhaps closest to relic is the work of grav (2019), which concentrates on the quotation of secondary sources in other secondary sources, unlike our focus on quotation from primary sources. finally, as described in more detail in section 2.2 and appendix a6, relic differs significantly from existing nlp and ir retrieval datasets in domain, linguistic complexity, and query length. ## conclusion in this work, we introduce the task of literary evidence retrieval and an accompanying dataset, relic. we find that direct quotation of primary sources in literary analysis is most commonly used as evidence for literary claims or arguments. we train a dense retriever model for our task; while it significantly outperforms baselines, human performance indicates a large room for improvement. important future directions include (1) building better models of primary sources that integrate narrative and discourse structure into the candidate representations instead of computing them out-of-context, and (2) integrating relic models into real tools that can benefit humanities researchers.