Advances and Challenges in the Automatic Identification of Indirect Quotations in Scholarly Texts and Literary Works (2025)

Abstract: Literary scholars commonly refer to the interpreted literary work using various types of quotations. Two main categories are direct and indirect quotations. In this work we focus on the automatic identification of two subtypes of indirect quotations: paraphrases and summaries. Our contributions are twofold. First, we present a dataset of scholarly works with annotations of text spans which summarize or paraphrase the interpreted drama and the source of the quotation. Second, we present a two-step approach to solve the task at hand. We found the process of annotating large training corpora very time consuming and therefore leverage GPT-generated summaries to generate training data for our approach.

## introduction literary scholars reproduce literary works in different ways and have to decide how precise their reference to the interpreted text should be. direct quotation, using direct speech and quotation marks, is considered the closest, the verbatim rendition of a source. no information is omitted (except the surrounding context and, sometimes, marked or unmarked omissions in the quotation) or added. to a certain degree, direct quotations preserve the poetic form of a text. retaining the literality of the source and their precise wording is one of the canonical features of the concept quotation @xcite . recent literary theory has categorized various types of references to literary texts that are used in scholarly interpretative articles @xcite . in our research project key passages in literary works, 1 we use methods of computational literary studies to find intensively interpreted passages. we identify these key passages by accumulating direct quotations of a literary text in scholarly texts, which led us to detailed insights into the scholars' quotation practices @xcite . we consider the heavily quoted passages in academic texts as key for the particular exegesis. we recognize that not only direct quotations play an important role in interpretive practices but also indirect quotations. therefore, in this work, we develop and analyze methods to automatically 1 https://hu.berlin/keypassages identify indirect quotations in scholarly texts and literary works. we follow the definition from winko (2022): an indirect quotation translates object language into metalanguage without adding essential information that does not stem from the textual source itself. paraphrases and summaries are subcategories of indirect quotation. a paraphrase is more or less a recurrence of the content with a change of the wording (de @xcite , whereas a summary abbreviates the content, with a change of the wording, too. indirect quotation is only one of several types of references scholarly interpretations use. in interpretive texts, scholars also apply classification, illustration, explanation, explication, and exegesis @xcite . all these types of interpretive practices need extrinsic context information whereas types of direct and indirect citation -generally speaking -only use intrinsic features of the literary text. additionally, they vary significantly from quotations because they include information that comes from the interpreting scholar who writes the interpretative article. these references and quotations are often mixed and distinguishing occurrences of indirect quotations from the surrounding text and differentiating between the distinct types is a hard task, even for human experts. direct quotations are easier to identify as they are syntactically marked, for example, by quotations marks, and can be identified and linked using existing tools, such as quid and proquo @xcite . indirect quotations, on the other hand, are much more challenging. often they are not accompanied by any surface indicator and therefore we do not have prior knowledge of the location of candidates in a scholarly work. sometimes, scholars mention the source of an indirect quotation in the running text or in a footnote. however, these references are applied rather nonsystematically and cannot reliably be utilized. additionally, the length of indirect quotations can vary from very short -only a couple of words -to full, or even multiple, sentences. another big challenge is the non-existence of annotated training data and we found that annotating this phenomenon is a very time-consuming process and an arduous task for human annotators. considering these challenges, we made the following decisions. first, we want to avoid manually creating large corpora for training machine learning models. sec-ond, we focus on dramas which are available in cleaned and annotated form from dracor @xcite . this allows us to use the predetermined act and scene structure for linking a quotation from the scholarly work to its source in the literary work. lastly, we limit the task to the identification of quotations which re-narrate part of the drama either as a summary or a paraphrase. 2 another unrelated challenge is the acquisition of scholarly works. as opposed to classical dramas, many scholarly works are not readily available online and need to be manually collected, digitized, and cleaned in a very time-intensive process which we outline in section 4.1. our contributions are twofold. firstly, we present a two-step approach for the identification of indirect quotations, more precisely, summaries and paraphrases, in scholarly works and the source of the quotation in the literary work. 3 in the first step, we identify sentences in the scholarly work that are candidates for containing an indirect quotation. in the second step, we identify the scene of the associated drama which is most likely the source of the quotation. to acquire training data without manual annotation, we use @xcite summaries as a basis to generate training data for candidate identification and scene prediction. this two-step approach is necessary due the nature of how we generate the training data without manual annotation. our second contribution is a first dataset of annotated scholarly works with annotations of text spans which summarize or paraphrase the interpreted drama and the source of the quotation. 4 the paper is organized as follows: the next section gives an overview on related work. in section 3, we present our method followed by a description of our data acquisition process, the experiments, and results in sections 4, 5, and 6, respectively. we conclude this work with a discussion in section 7. ## related work the task of identifying speech, thought, and writing in fiction and non-fiction texts, referred to as quotation detection, is related to the first step of our approach, that is, the identification of summaries or paraphrases in scholarly works. there are different types of speech, thought, and writing, for example, direct, indirect, or reported speech @xcite . the last type is closest to the scholarly citations in our texts. quotation detection is often focused on english newspaper articles @xcite al. (2013); scheible et al. (2016) ), though there is a corpus-agnostic approach @xcite and an annotated dataset of finish news articles @xcite . corpora for german include @xcite @xcite . as part of the redewiedergabe project, 5 @xcite published a number of models for tagging different types of speech in german texts, including one for reported speech. a related task is quotation attribution, that is, identifying the source of a quotation, for instance, the speaker @xcite @xcite . although our phenomenon of interest is similar, it is still not easily transferable. scholarly texts can be quite different in style compared to fictional works or newspaper articles. the second part of our task is to link quotations to their source. multiple efforts have been made to understand how attention values of transformer models could be used to identify the source of a summary. @xcite give an extensive overview on the ongoing debate whether or not attention values can be used to explain black box transformer models. for abstractive summarization specifically, @xcite find that attention values cannot be reliably used to explain summaries. one explanation for these findings could be shortcut learning @xcite . suhara and alikaniotis (2024) present an approach based on perplexity gain to identify the source of a quotation. they found this method to outperform the second best approach, similarity-based methods, on the xsum dataset @xcite , while similarity-based methods perform better on the cnn/daily mail dataset @xcite . given that our texts are quite different, these results cannot easily be applied to our task. due to its versatility and availability through sentencetransformers (reimers and gurevych, 2019), semantic textual similarity emerged to be the most promising path. although there are models which outperform sentencetranformers @xcite , we decided to use a pre-trained sentencetransformer (pst) due to the need for german models, which are readily available, and the relative ease of further training due to good documentation and support of a multitude of different use cases. ## methods we first define the task, then describe our approach for generating training data and the training procedure, and then present our tool for inference. ## conclusion our contributions are twofold. first, we created a dataset of scholarly works with annotations of text spans which summarize or paraphrase a drama, and their source in the drama. we created a gold standard from three independent annotations. during this process, we identified several reasons for discrepancies in the annotations and the resulting inter-annotator agreement. second, we presented an approach for the automatic identification and linking of indirect quotations in scholarly works and dramas leveraging semantic similarity. we chose the approach as we hoped the trained model would allow us to work with arbitrary dramas without the need for summaries at inference time. we evaluated the approach and identified different challenges. for candidate identification we found distinguishing between indirect quotations and surrounding text very difficult, even for human experts. one reason is that scholarly texts necessarily reference the literary work, and that these references can take various shapes and forms which are hard to separate, especially because interpreting passages can be quite similar to summaries and paraphrases. we also found the performance of the model to heavily depend on the specific text. two reasons for this are that some texts discuss multiple adaptions of the same work and that some discuss sev-eral dramas. in light of these results, we evaluated whether we could perform better if we assume summaries to be available. we found that another approach which uses summaries instead of drama excerpts performs more stable overall, and in the case of scene prediction, also outperforms our approach. a likely reason is that summaries are closer in style to the types of text language models are normally trained on. we conclude that the main area for improvement is the identification of semantic similarity in the context of indirect quotations, which existing models can not fully capture due to the similarity between relevant references and the surrounding text. at least in part, this might be due to named entities. @xcite find that models for semantic similarity strongly rely on named entities as a source of similarity. consequently, information on the argumentation structure of the scholarly work is needed for a better distinction. finally, the limited input size of models such as bert, which necessitates splitting of texts, is another challenge and area for future work. with the increasing performance of recent large language models (llm) on a variety of tasks and the increasing context window size, another route for the future will be to utilize llms in a more direct fashion and prompt with full scholarly and literary texts to extract indirect quotations.