Direct Speech Identification in S wedish Literature and an Exploration of Training Data Type, Typographical Markers, and Evaluation Granularity (2024)

Abstract: Identifying direct speech in literary fiction is challenging for cases that do not mark speech segments with quotation marks. Such efforts have previously been based either on smaller manually annotated gold data or larger automatically annotated silver data, extracted from works with quotation marks. However, no direct comparison has so far been made between the performance of these two types of training data. In this work, we address this gap. We further explore the effect of different types of typographical speech marking and of using evaluation metrics of different granularity. We perform experiments on Swedish literary texts and find that using gold and silver data has different strengths, with gold data having stronger results on token-level metrics, whereas silver data overall has stronger results on span-level metrics. If the training data contains some data that matches the typographical speech marking of the target, that is generally sufficient for achieving good results, but it does not seem to hurt if the training data also contains other types of marking.

## introduction the main narrative of literary works is typically interspersed with dialogues representing direct speech utterances by the characters in the work. distinguishing narrative and direct speech is important for work on digital literature studies, for tasks including identifying the social networks of novels @xcite and analyzing the sentiment of characters towards each other @xcite . in addition to speech segments, we are also interested in speech tags, or reporting clauses, speech tags can have different lengths and positions with respect to the speech, exemplified in (1-3). 1 speech tags are also relevant for work in literary studies, such as @xcite who study them as a means of analyzing dickens' narrative perspective. raska på. '-are you dreaming, mum? she said sharply. hurry up.' (m. sandel, hexdansen, p. 46) there is also little previous investigation of the impact of the use of different typographical markers in the training and test data of classifiers. @xcite provided separate test sets for different types of marking but performed only a small pilot experiment. in this work, we extend their study and explore the issue in more detail. the task setup as well as the metrics used in previous works have also varied between studies. in this work, we model the task of identifying direct speech and speech tags as a token-classification problem. unlike previous studies, we evaluate it on two levels of granularity, both at the token level and at the span level, which requires the exact matching of a full span. this allows us to investigate the effect of metric choice on the results. to sum up, we investigate the following research questions, in the context of identification of direct speech segments and speech tags in literary works: rq1 is it preferable to use smaller gold data or larger automatically annotated silver data for direct speech identification? rq2 can heuristically constructed silver data be useful for speech tag identification? rq3 is it possible to improve speech and speech tag identification by mixing gold and silver data? rq4 what is the effect of different typographical marking of speech in training and test data? rq5 what is the effect of using span-level versus token-level evaluation metrics for direct speech identification? in addition, we provide a detailed overview of related work for the task of direct speech identification. ## related work in this section, we focus on reviewing related work on the identification of direct speech in literary works for cases where quotation marks are not predominant. this excludes some distantly related work, e.g. targeting other genres such as news texts (e.g. @xcite , and work on languages that predominantly use quotation marks, such as english (e.g. @xcite . language most work focuses on one language, in most cases either german or swedish, with one study on french. two works explore multiple languages, including a cross-lingual setup @xcite and multilingual training @xcite . the latter found that for many languages, including english, a rule-based system based on punctuation marks gave near-perfect accuracy. however, for other languages, especially norwegian, which is closely related to swedish, the rule-based system performed poorly, due to mixed graphical speech marking. training data all papers but one use either existing gold data or collect silver data for their experiments. only one paper, kurfalı and wirén (2020) use both variants. however, their main point of investigation is to explore the feasibility of crosslingual zero-shot training for direct speech identification, so they compare using english silver data, work language training data modelling/eval. method marks miscellaneous brunner (2013) german gold sentence, work rule, random forest mixed, incl. qm stwr schöch et al. (2016) french gold sentence svm, maxent, . . . dash/mix(?) applied jannidis et al. (2018) german silver sentence, token log. regr., lstm, . . . mixed applied ek and wirén (2019) swedish gold token log. regr., rule stripped speech lines tu et al. (2019) german -sentence, token rule no-qm brunner et al. (2020b) german gold token bilstm-crf+bert/flair mixed (often qm) stwr byszuk et al. (2020) 9 languages gold token bert-ft, rule mixed kurfalı and wirén (2020) 4 languages silver (en) token mbert-ft stripped cross-lingual dahllöf (2022) swedish silver segment multi-layer perceptron stripped dash lines applied stymne and östman (2022) swedish gold token/span bert-ft mixed speech tags we also make a best effort to categorize the type of typographical marking used in each study, which is challenging since it is not always directly stated; here qm stands for quotation mark. stwr stands for speech, thought, and writing representation, works marked as such are not restricted to only identifying direct speech. works marked with applied, apply the classifiers to a large set of literary works, for further analysis. to using in-language gold data for german @xcite , portuguese @xcite and swedish @xcite , which does not constitute a fair comparison with respect to only data type. they do also use english gold data @xcite , for which they found that the performance is better with the gold data, with a token-level f1-score of 0.89 compared to 0.85 with the silver data. the silver data contained english books from project gutenberg, where speech was extracted based on quotation marks. ## data in this section, we give an overview of the slända corpus that we used for evaluation and gold training data. we then go on to describe the extraction of a new large silver training dataset, based on literary works with quotation marks. ## experimental setup in this section, we describe how we model the task, the system we use, and the evaluation metrics used. ## results in this section, we present and discuss the results, followed by a summary of our main findings. ## high-level results in our first experiment, we compare macro-average precision and recall for all variants of gold and 5 retrieved from https://www.cnts.ua.ac.be/ conll2000/chunking/conlleval.txt . silver training data. for f1-scores, we refer to the detailed results in tables 4 5 6 7 . results are shown in ## conclusion we explore several aspects related to the automatic identification of direct speech segments and speech tags in swedish literary works. we focus on the usefulness of manually annotated gold data, compared to automatically annotated silver data, the impact of typographical markers of speech, and the impact of evaluation granularity. we find that using gold and silver data has different strengths, with gold data giving better token-level performance, and silver data often better span-level performance. mixing gold and silver data did not lead to further improvements. the training data needs to contain the type of speech marking that is used in the target data, but may also contain other variants, to ensure a reasonable performance. in future work, we plan to extend the current study with a detailed error analysis, and specifically explore the reason for the differences between token-level and span-level metrics in more depth. a further line of work is to investigate the use of ensemble models as an alternative to data concatenation, which was not successful in this study. we think the current classifiers are strong enough to apply to research in digital literature studies where the identification of direct speech and/or speech tags is needed. based on a specific use case, it is possible to choose training data that gives either high recall or high precision, on the token and/or span level. we plan to use such a classifier to investigate changes in the swedish written language in literary narrative and dialog over time.