Zero-shot cross-lingual identification of direct speech using distant supervision (2020)

Abstract: Prose fiction typically consists of passages alternating between the narrator’s telling of the story and the characters’ direct speech in that story. Detecting direct speech is crucial for the downstream analysis of narrative structure, and may seem easy at first thanks to quotation marks. However, typographical conventions vary across languages, and as a result, almost all approaches to this problem have been monolingual. In contrast, the aim of this paper is to provide a multilingual method for identifying direct speech. To this end, we created a training corpus by using a set of heuristics to automatically find texts where quotation marks appear sufficiently consistently. We then removed the quotation marks and developed a sequence classifier based on multilingual-BERT which classifies each token as belonging to narration or speech. Crucially, by training the classifier with the quotation marks removed, it was forced to learn the linguistic characteristics of direct speech rather than the typography of quotation marks. The results in the zero-shot setting of the proposed model are comparable to the strong supervised baselines, indicating that this is a feasible approach.

## introduction all narratives imply a speaker, that is, someone who tells the story @xcite . this is the basic narrative mode. but all stories are populated by characters who typically speak to each other, as opposed to telling the story. the characters' direct speech is therefore a distinct narrative mode, though it is embedded into the narration and mediated by the narrator @xcite . typically, prose fiction consists of passages alternating between these two modes of narrative transmission. there are also hybrid forms, such as indirect discourse and free indirect discourse @xcite , but for the purpose of this work we are only interested in the binary distinction between narration (the telling of the story, in which we here include any hybrid forms) and the direct speech between characters. recently, there has been a growing interest in the detection of direct speech, with novel approaches to problems such as distinguishing narration and speech @xcite @xcite , keeping track of speakers @xcite @xcite , and keeping track of addressees @xcite . however, the lack of annotated resources has been a hindrance for progress in this field. it might seem that an effective strategy would be to use quotation marks (single quotes, double quotes, dashes, etc.) for distant supervision, but this is also riddled with difficulties since typographical conventions vary a lot across languages and time periods @xcite . in addition, many types of quotation marks are overloaded in the sense of being simultaneously used for other purposes, such as pauses and contractions. presumably as a result of all this, almost all approaches to identification of direct speech have been monolingual. for example, @xcite describes identification of direct speech for portuguese, @xcite for english, @xcite and @xcite for german, and @xcite for swedish. in contrast, the only multilingual approach that we are aware of is @xcite , who collected a corpus from nine languages which they annotated manually for direct speech. based on this, they developed a classifier for token-level identification of direct speech using bert @xcite , reaching an f-score of up to 0.9 in their leave-one-out evaluation at the language level. our approach has similarities to that of @xcite , but, to begin with, we have adopted a low-resource scenario in which we only assume the availability of a collection of raw texts. using a set of heuristics to automatically detect texts where quotation marks appear sufficiently consistently (similar to jannidis et al. ( 2018 )), we created a training corpus to be exploited for the purpose of distant supervision. in addition, we fine-tuned multilingual contextual embeddings for the corpus. furthermore, to prevent the system from relying on the typography of quotation marks, we removed them in the version that was used for fine-tuning, thereby forcing the system to instead learn the linguistic characteristics of direct speech, notably speech-framing expressions. these are the cues that narrators provide to allow readers to understand and assess what the characters are saying (and in our work, also who is saying it), consisting of concatenations of a referent, verb and possibly modifiers, such as "zoe said" or "she inserted drily" @xcite . to retain the relevant information, we had instead labelled each token as belonging to narration, beginning of speech or inside speech in accordance with the iob format. we then used this corpus to develop a token-level classifier for identification of direct speech in the absence of any quotation marks, using multilingual-bert (m-bert), released by @xcite . the experimental results show that the performance of our classifier is comparable to supervised approaches even in the zero-shot scenario. to the best of our knowledge, this is the first study of zero-shot cross-lingual identification of direct speech. ## method in this section, we describe our method for training a multilingual direct-speech classifier via distant supervision, i.e. without assuming any manually labeled dataset. ## experimental setting the classifiers are implemented using the transformers library @xcite . all classifiers employ mbert and were fine-tuned for 3 epochs, with a batch size of 32 and a learning rate of 3e-5. in all experiments, the quotation marks were removed from the datasets, thus forcing the model to identify the direct speech from other linguistic cues. we evaluated our classifier on the following manually annotated datasets, each representing a different language: 2020b ) are obtained using the german bert model, trained on the whole dataset (including the quotation marks). the detailed statistics about these datasets are provided in ## results and discussion the experimental results are provided in however, we believe that the performance is not surprising given that news tend to adopt very different linguistic cues to signal direct speech (e.g. according to the chairman, ". . . ") than that of fiction texts. furthermore, quac is challenging even in the supervised setting suggested by the relatively lower performance of the supervised baseline. a qualitative error analysis of 30 random paragraphs from each of the english and swedish datasets indicated that the most salient characteristic of false negatives was absence of speech-framing expressions. thus, it seems that narrators' cues to their readers were indeed useful also for the classifier. false positives were more difficult to categorize, but sometimes non-speech verbs generated false alarms, as in example 1. 3 (1) it's time for me to go north, thought gurov as he left the platform. high time! likewise, the classifier sometimes struggled when the narration interrupting the speech extended over several sentences, as in example 2. on the other hand, the model sometimes recognized very long narrations correctly, as in example 3. nice girl! very. (3) in everything that made my love of any worth or value in your sight. if this had never been between us, said the girl, looking mildly, but with steadiness, upon him; tell me, would you seek me out and try to win me now? ah, no! a common problem was that the system repeatedly switched its binary decisions in the middle of clauses, resulting in spurious fragments, as in example 4. it should be possible to handle errors of this kind by introducing a syntactic post-processing step which prevents sudden shifts between narration and speech within clauses. finally, as a further experiment, we evaluated our model on the indirect speech annotations of redewiedergabe @xcite to see if training without any quotation marks could help the model to generalize over these implicit cases, as well. however, with a recall of 0.20, it completely failed on this type of speech. training the classifier on the same language does not help either; the classifier trained on the direct speech annotations of the same data also failed, with a recall of 0.16. hence, this low performance is not due to the zero-shot configuration, but must be the case because direct-speech classifiers are unable to generalize to other (hybrid) forms of speech, highlighting the need for more studies on the simultaneous identification of these. ## conclusion in the current study, we show that it is possible to perform zero-shot cross-lingual identification of direct speech with a performance comparable to that of supervised baselines. since quotation marks are not in general reliable indicators of direct speech, we instead devise a set of heuristics for collecting data where they are used sufficiently consistently, and use this to elicit enough data to fine-tune contextual embeddings. as future work, we consider focusing on the cases where our model falls short, namely, generalization to (i) other domains and (ii) other speech forms. finally, we hope that our study will also be useful in establishing a benchmark to evaluate multilingual direct-speech identification.