Direct Speech Quote Attribution for D utch Literature (2023)

Abstract: We present a dataset and system for quote attribution in Dutch literature. The system is implemented as a neural module in an existing NLP pipeline for Dutch literature (dutchcoref; van Cranenburgh, 2019). Our contributions are as follows. First, we provide guidelines for Dutch quote attribution and annotate 3,056 quotes in fragments of 42 Dutch literary novels, both contemporary and classic. Second, we present three neural quote attribution classifiers, optimizing for precision, recall, and F1. Third, we perform an evaluation and analysis of quote attribution performance, showing that in particular, quotes with an implicit speaker are challenging, and that such quotes are prevalent in contemporary fiction (57%, compared to 32% for classic novels). On the task of quote attribution, we achieve an improvement of 8.0% F1 points on contemporary fiction and 1.9% F1 points on classic novels. Code, data, and models are available at https://github.com/anonymized/repository .

## introduction quote attribution is the task of identifying the speaker of each quotation span in a given text. when applied to dialogue in literature, this enables us to study relations and interactions between characters, for example by extracting social networks @xcite . other applications to literature include examining gender differences @xcite or measuring information propagation @xcite . whereas the aforementioned studies all focus on english-language fiction, in this paper we focus on direct speech attribution in dutch literature. an example can be seen in the following sentence: example (1) has an explicit speaker mention (tom). however, identifying the speaker is not always as easy (examples from tolstoy's anna karenina): speaker: [matvey] 3 it's all right, @xcite 1 ; [she] 2 will come round," said @xcite 3 . "come round?" "yes, @xcite 1 ." the speakers of (2) and (3) are mentioned by anaphor. sentences 2-3 of (4) are even more challenging, since they have an implicit speaker. note that all of the above examples are direct speech, in which the exact words spoken by a person are reported. although there exist systems for detecting and attributing indirect speech @xcite and free indirect discourse @xcite , our focus in this work is strictly on direct speech. for the task of dutch quote attribution, van cranenburgh (2019) presents a rule-based approach as part of the dutchcoref coreference resolution system. quote attribution is relevant to coreference resolution, as the speaker and addressee of dialogue turns must be known to resolve first and second person pronouns in quoted speech correctly. furthermore, after extending the dutchcoref system with three neural classifiers for the subtasks mention de-tection, mention attributes and pronoun resolution, @xcite notes that in literature dialogue is particularly important; annotating and predicting speakers of direct speech was proposed as one of the directions for future work. therefore, we implement a neural classifier for quote attribution, which we expect to outperform dutchcoref's rule-based approach. additionally, we perform an error analysis, where we look at whether certain speaker types are harder to classify: explicit (said tom) vs. anaphoric pronoun (said he) vs. anaphoric other (said his friend) vs. implicit. moreover, we will analyze whether the corpus of books that we use (rid-dlecoref vs. openboek) has an influence on the performance of the classifier. ## data and material for our experiments, we work with dutch literary novels from both the riddlecoref (van cranenburgh, 2019) and the openboek (van cranenburgh and van noord, 2022) corpora. for the task of quote attribution specifically, we needed to annotate the novels ourselves in order to obtain gold data. we will discuss both the corpora statistics and the annotation process below. the riddlecoref corpus was first presented in van @xcite and consists of a selection dutch (translated and original) contemporary literary novels from the riddle of literary quality project @xcite . this corpus contains a total of 33 documents, for which we use the train, development and test splits as defined in @xcite . in total, there are 38,647 mentions in the corpus and on average 4,897.4 tokens per document. unfortunately, the annotated texts from the riddlecoref cannot be made publicly available due to copyright. the openboek corpus consists of public domain novels from project gutenberg enriched with several layers of annotation. 2 the corpus currently contains 9 fragments of dutch-language novels and novellas. this corpus contains a total of 23,650 mentions, with an average of 11,502.4 tokens per document. the number of sentences per document (mean 643.3), as well as the number of tokens per document (> 10k), indicate that annotated openboek fragments are longer than the riddlecoref fragments, or most other coreference datasets. these longer fragments lengths were chosen specifically with the challenge of longdocument coreference resolution in mind. ## quote attribution annotation whereas gold standard coreference annotations (mentions and coreference clusters) were already available, this was not the case for the task of quote attribution. therefore, we added quote attribution annotations as an extra annotation layer to the rid-dlecoref and openboek datasets. for the annotation we used the tool released by @xcite along with our own annotation guidelines. 3 the guidelines can be summarized as follows: we only annotated the quotes and corresponding speaker mentions, but not the addressees for these quotes, as this is outside the scope of this paper. ## evaluation as mentioned before, for the riddlecoref corpus we use the same train, development and test splits as defined in @xcite . for the openboek corpus however, there is no predefined split. a first proposal for this corpus was to use the novel max havelaar as development and eline vere as test, leaving the other seven novels as the train split. however, we noticed some poor performance with regards to the quote extraction part, which means these novels might not be representative as evaluation data. especially for the fragment eline vere, which is the only fragment in which quotes are always introduced by a dash sign instead of quotation marks, the quotes were often extracted incorrectly. in the fragment max havelaar, the quotes are introduced by both quotation marks and dash signs in a very inconsistent manner. moreover, quotes do not always have ending quotation marks. since quote extraction works well for the seven other fragments, we decided on the following: we will evaluate the performance of our classifiers, which were trained on the riddlecoref train split, on both the riddlecoref test split and on the seven remaining novels from the openboek corpus (thus excluding eline vere and max havelaar). this way we can see whether the performance is better on a specific corpus, as well as analyze the potential differences. we will report precision, recall and f1-scores, which were also used in earlier work @xcite . we report only scores indicating whether the quote was attributed to the correct speaker cluster. we do not report whether the quote was attributed to the same speaker mention as in the gold data, since this is a somewhat arbitrary annotation choice. the evaluation can be further divided. during the training of our classifier and our initial experiments, we made use of gold standard coreference files that were already available for the riddlecoref dataset. we will report the scores obtained by our classifier and the baseline systems when using these gold coreference files, meaning these systems have it must be noted that quotes that do not have a gold speaker are not taken into consideration during this evaluation, as this should be addressed as part of the quote extraction process, which is not the focus of this paper. in the following subsections, we first report the results achieved in our experiments using gold standard coreference files. for these results, we also show which features contributed the most to each of our classifiers. then, we report the results that our classifiers achieved when implemented into the dutchcoref system. ## analysis to gain a better understanding of the challenges of literary quote attribution, we now take a closer look at the test data and model outputs. we first consider the distribution of quote types, and then perform an error analysis of the systems we evaluated. ## conclusion in this paper, we focused on training a classifier to improve the task of quote attribution when compared to dutchcoref's rule-based approach. we trained three different feed-forward neural network classifiers, each one focused a different metric for speaker clusters: precision, recall and f1-score. for the task of quote attribution, we manage to improve on the rule-based approach by 8.0% f1 points on the riddlecoref test novels and by 1.9% f1 points on the openboek novels. with our quote attribution error analysis we show that each system makes the most mistakes on implicit quotes. moreover, anaphoric pronoun quotes prove to be harder than anaphoric non-pronoun quotes, as each of the systems performs the best on the anaphoric pronoun quote type. this also explains why the quote attribution performance on the openboek novels is notably higher than on the riddlecoref test novels, as the openboek novels contain relatively more anaphoric pronoun quotes and less implicit quotes. for future work, we think there is still a lot of improvement to be gained, especially on implicit quotes. as we have found features that reduce mistakes on anaphoric pronoun quotes by 91.7% with respect to the rule-based approach, future experiments can look specifically at how to decrease mis-takes on implicit quotes. furthermore, we did not consider the task of identifying addressees, thus having to rely on the rule-based approach to identify these after we first identify the speakers using our neural classifiers. jointly identifying speakers and addressees may yield additional performance gains, since it would enable the classifier to pick up on turn-taking patterns in a data-driven manner. reported speech verbs (also known as cue words) were not part of the annotations, but detected using a predefined list. recall may be improved by detecting them using a classifier trained on annotated cue words. in terms of machine learning, fine-tuning bert for the task of quote attribution (rather than simply using averaged token embeddings as features) and/or incorporating more context with for example an lstm on top of the bert embeddings can be expected to yield additional improvements. lastly, we think further improvements can also be made on quote extraction, as we saw that there were still a lot of mistakes made on the openboek novels max havelaar and eline vere. as most of these mistakes were made on quotes starting with a dash sign, more elaborate rules targeting these kind of quotes could improve the overall performance of dutchcoref even more.