Automatic Annotation of Direct Speech in Written F rench Narratives (2023)

Abstract: The automatic annotation of direct speech (AADS) in written text has been often used in computational narrative understanding. Methods based on either rules or deep neural networks have been explored, in particular for English or German languages. Yet, for French, our target language, not many works exist. Our goal is to create a unified framework to design and evaluate AADS models in French. For this, we consolidated the largest-to-date French narrative dataset annotated with DS per word; we adapted various baselines for sequence labelling or from AADS in other languages; and we designed and conducted an extensive evaluation focused on generalisation. Results show that the task still requires substantial efforts and emphasise characteristics of each baseline. Although this framework could be improved, it is a step further to encourage more research on the topic.

## introduction prose fiction makes whole worlds emerge. authors make use of different strategies to create narratives and convey the storyworld. novels intertwine narrators' words to build the atmosphere and tell the story, with words stemming from characters inhabiting the fictive world that disclose their personality and depict them directly via dialogues or direct speech (ds) @xcite . the development of algorithms to perform the automatic annotation of direct speech (aads) in written text has been of high interest for literary studies. this task consists in retrieving lines uttered by the characters of a narrative in contrast to words delivered by the narrator of the story. one goal of aads has been to compare fiction works by different authors or stemming from different genres or time periods. ds was then studied as a literary device carrying specific purposes and disclosing compelling cultural information @xcite . aads is also central in narrative understanding endeavors. ds has been then considered as the main realisation of characters, their means to gain volume and depth, and come alive to the readers. in this context, aads is often regarded as a pre-processing step that enables downstream analysis such as ds speaker attribution @xcite , that can in turn serve to assemble characters networks @xcite , or model personas @xcite . aads has been widely performed for english literature, leveraging strict formatting conventions (e.g. quotes or long dashes) to extract ds through simple regular expression-regex @xcite @xcite . yet, in other languages, dialogues may be less strictly segregated from narration and typographic conventions can be more flexible. hence, more complex solutions based on lexical features have been developed, mainly for german @xcite @xcite . these lexical features were either manually defined and exploited with classical machine learning algorithms such as random forest @xcite , or were inferred indirectly from text in deep learning frameworks @xcite using recurrent neural networks or lan-guage models such as bert @xcite . for other languages, including french, there are very few aads efforts. @xcite propose straight talk!, a corpus of 40 chapters from 19th century french novels annotated per sentence if containing ds or not, and performed binary classification using 81 engineered features. the corpus was quite large, but sentences were poorly segmented with a high impact on results; annotations did not consider incises (i.e. narrative breaks within the same ds turn as in figure 1 ); despite a high overall f1-score (93%), some writing styles were very challenging (for instance in homodiegetic narratives, where the narrator is a fully fledged character in the storyworld and may relate the story at the first person). in another work, @xcite adopted a feature engineering approach as well. they combined it with rules to segment and identify paragraphs containing ds, and then to extract incises from mixed paragraphs. still, the method was tested on a small corpus, a subset of synpaflex @xcite with excerpts from only two novels. finally, @xcite considered aads in multilingual settings using bert, but on an even smaller french corpus. the goal of the current work is to create an unified framework for designing and evaluating aads models in french, which in return we hope to encourage more research on the topic foot_0 . specifically, we address existing limitations on multiple fronts: apart from traditional token-and span-level strict precision, recall and f1-score metrics (yadav and bethard, 2018), we adapted zonemap @xcite , a metric stemming from page segmentation literature, to our task. this allowed us to quantify the effect of various error types made by the models and deepen our understanding of their limitations. results show that rule-based baselines using regular expressions remain a good choice when texts are well-formatted. deep learning solutions are however more effective and achieve satisfactory results even on narratives with poor formatting quality. their most common issue is that they still miss to catch whole ds sequences. we also conducted a qualitative analysis to bring insights on the strengths and weaknesses of various models, and defined the directions for future endeavors. ## literature review we further review aads solutions for any language. ## french narrative corpora for aads we consolidate a large dataset of french novel excerpts, manually annotated with ds labels at the word level. built upon existing endeavors, the final dataset is a compilation of four sub-corpora, individually referred to as straight talk! (st!) @xcite , synpaflex (spf) @xcite , an extension of synpaflex provided to us by the authors (sb), and fr-litbank (fr-lb) @xcite . while fr-lb, spf, and sb have overall good encoding and segmentation quality, st! is poorly formatted with some files lacking line breaks, for instance. each sub-corpus contains french novels from the sub-corpora, fr-lb, and sb, were kept in the form provided by the original works. in contrast, we modified the ground-truth annotations of st! and spf in order to align them with the other two sub-corpora and the adopted approach to model the problem-binary classification at the token leveland to exhaustively cover chapters, not only excerpts. in particular, st! annotations lacked granularity as text segments were labelled as narration or mixed (comprising both narration and ds), so we corrected those. as for spf, the annotations were very sparse among the 27 chapters; hence we extended them to whole chapters. the re-annotation process was mainly led by one author using doccano @xcite . a selection of 5 files were doubly annotated by a coauthor to check labeling reliability. the obtained pairwise cohen's κ score @xcite was 97%, which is considered almost perfect agreement. the re-annotated dataset is shared with the code. the dataset is then split into train, validation and test sets. the files from the three well-formatted sub-corpora (fr-lb, spf, sb) are randomly divided in order to ensure a proportion of 0.8/0.1/0.1 for train, validation, and test, respectively, and that at least one file from each sub-corpus can be found in each split. each file can be found in only one split, but we sometimes have files from the same novel present in all splits, especially those origi-nating from the spf sub-corpus (les mystères de paris by eugène sue and madame bovary by gustave flaubert). finally, st! is kept for test only as a challenge dataset. indeed, contrary to the other subcorpora mentioned above, this latter sub-corpus suffers from largely unequal formatting quality across files. some chapters are completely devoid of line break which makes them, wrongly, appear as one unique paragraph, while others exhibit misplaced line breaks, sometimes in the middle of sentences. st!'s formatting peculiarities make it a good test for generalisation, especially on noisier text. this challenging set is also referred to as a noisy test set (test n ) in contrast to the clean test set (test c ) stemming from the split of the three well-formatted sub-corpora -that are also used for training and validation. dataset statistics are shown in ## methods popular baselines from the two aads approaches (rule-based and machine learning-based), including those designed for other languages, were modified to fit the characteristics of french. aads was then formalized either as a text matching and extraction task, when using regex, or as a sequence labelling task, when using deep learning models. for the latter, the aads models returned a binary label per token, (o / ds) as in other related works @xcite @xcite . while regex has been more common, to our knowledge, this is the most extensive attempt to explore deep learning for aads in french narratives. ## results ## limitations the current framework bears several limitations. first, although a common strategy in the related literature @xcite @xcite which we also adopted, the binary annotation at the token-level is limiting. with this schema, the focus is not on speakers' utterances or turns, but on ds sequences. a subsequent issue is that consecutive turns by different characters are considered as one ds sequence if there is no "o" labeled tokens between them. one solution could have been to mark the start and end of a ds turn while paying attention to handle imbricated narration (ie. incises). however, this would have required significant more re-annotation efforts, which we left for a future research cycle within the proposed framework. second, because of copyright issues the corpus contains excerpts exclusively from a specific period, 1830-1937. thus, the models were trained and tested on a specific type of literature and may not generalize well to other forms of narratives, in particular modern and contemporary. in this direction, the curation of the test corpus could benefit from more literary insights considering that the evaluation showed high variance of the performance over chapters. this could help to better determine the application scope of the models, and which kind of narratives require further work. with regard to the deep neural network baselines, we did not perform an extensive parameter search and model optimisation. this could have further improved the results. however, performances on recognizing full ds spans were clearly lower than token-level metrics, which had most likely other causes. regarding the evaluation, although we adopted zme scores from page segmentation to have more qualitative insights, there are still other aspects we have not quantified and could be particularly relevant. for instance, does the model tend to miss the beginning, the end or some other specific parts of a ds sequence? we tried to capture some of these phenomena through our manual analysis, but it is challenging to apply it at scale without introducing methods to automatically compute metrics. ## conclusion we have presented an unified framework to design and evaluate aads in written french narratives. to our knowledge, this is the largest aads study to date in french. we consolidated a large dataset annotated per word. then, we benchmarked two families of baselines, rule and deep learning-based, using as inspiration aads advances in other languages (german and english). we designed an evaluation which focuses on generalization and on learning about the advantages and weaknesses of each baseline. results show that rule-based systems work well on bounded ds conventions (quotation marks) in clean text. other ds formats, incises, and poorly formatted files pose many problems. deep learning baselines prove to be far more robust, reaching token-level f1-scores up to 95%, but with large variance across files. yet, recognizing full spans of ds is still challenging, even when texts have good formatting quality. while for macro analyses in literary studies, imperfect aads may be sufficient, some use-cases require almost perfect performance when recognizing ds spans (e.g. audiobook generation from text). if a more thorough parameter optimization might help, our qualitative analysis conveys that performance gain should be instead sought by integrating domain knowledge into the models-without feature over-engineering though. studying the models' performances after the removal of typographic cues could lead to other insights on how to increase robustness. multilingual language models and existing aads corpora could be also exploited for french. another needed step would be to transition to identifying full ds turns and their corresponding speakers, with the implied manual re-annotation efforts. finally, split and merge groups are divided into sub-zones that are in turn classified as strict match and miss or false alarm. miss and false alarm sub-zones contribute to the error like miss or false alarm groups (strictly proportionally to the length of the sub-zones). in contrast, the largest match sub-zone is not counted as an error and does not contribute to e zm , while the remaining match subzones are partially counted as errors. their contribution to the segmentation error rate is proportional to their length, an introduced merge/split mitigating parameter α m @xmath0, 1] and the relative number of split, respectively merged sub-zones. given a split group g @xmath1, {s j,l } n k @xmath2, the group is sub-divided into n k strict match sub-zones {z m } n k @xmath3, n k , n k + 1} nonoverlapping spans {z m } n k @xmath4. the segmentation error rate of such group would then be the sum of non-detected tokens those formula can then be rewritten in terms of original spans as: merge groups error contribution is computed comparably.