D rama C oref: A Hybrid Coreference Resolution System for G erman Theater Plays (2021)

Abstract: We present a system for resolving coreference on theater plays, DramaCoref. The system uses neural network techniques to provide a list of potential mentions. These mentions are assigned to common entities using generic and domain-specific rules. We find that DramaCoref works well on the theater plays when compared to corpora from other domains and profits from the inclusion of information specific to theater plays. On the best-performing setup, it achieves a CoNLL score of 32% when using automatically detected mentions and 55% when using gold mentions. Single rules achieve high precision scores; however, rules designed on other domains are often not applicable or yield unsatisfactory results. Error analysis shows that the mention detection is the main weakness of the system, providing directions for future improvements.

## introduction we present experiments on resolving coreferences in theater plays/dramas. coreference resolution (cr) aims to assign all mentions in a text to their respective discourse entities. doing so offers many benefits for working with theater plays, yet cr on this type of text is rarely performed. next to prose and poetry, drama is considered one of the three main literary genres since ancient greece @xcite . dramas are scripts to be performed in theatre. furthermore, drama is a highly structured text type: to a large extent, it consists of character speech with clear indication of the speaker, as well as some stage directions and a segmentation into acts/scenes. usually, dramas also contain a so called dramatis personae, which is a list of cast members which will appear during the course of the play. this list typically contains further information about the characters, like family relations, which in turn can be exploited to better resolve certain coreferent mentions. act i, scene i. the prince's cabinet the prince, seated at a desk, which is covered with papers. prince. complaints; nothing but complaints! [. . . ] to be sure, if we could relieve every one, we might indeed be envied. emilia? opening a petition, and looking at the signature. an emilia? yes-but an emilia bruneschi-not galotti. not emilia galotti. what does she want, this emilia bruneschi? reads she asks much-too much. but her name is emilia. it is granted signs the paper, and rings. characters are a constitutive ingredient for theater plays, and scholarly analysis often revolves around characters, their depiction, and their relations. while dramas make it straightforward to detect character speech, references to characters within the speech of other characters can really only be detected using cr. 1 these character references provide insight into how characters are introduced and seen by other characters, sometimes well before appearing on stage themselves. this can already be seen in figure 1 , which is the opening scene of a play. the character emilia galotti only enters the stage in the second act, but from the very beginning, other characters talk about her, thus shaping her reception by readers and audience. in this paper, we present a system to automatically resolve these coreferences. it makes use of both neural network and rule-based components. both have been adapted to the dramatic text type, either through fine-tuning or the addition of specific rules. to our knowledge, no system has been published for german data that is tailored to dramatic or dialogical texts. only a few publications deal explicitly with domain adaptation for cr systems @xcite @xcite . @xcite describe a method for adapting the berkeley cr system to narrative texts using linear programming. in terms of data sets, there are only a few readily available. @xcite released a corpus with annotated entity types (following the ace standard) on english literary texts. @xcite describe challenges in the annotation of literary texts, including plays. in previous own work @xcite , we have presented a corpus of 31 german language plays annotated for coreference, which will be used as the main resource in this paper. ## dramacoref we propose a hybrid system, using neural and rulebased components, for resolving coreference on theater plays, called dramacoref. the system follows the classic distinction into mention detection and coreference resolution component. for mention detected, we apply a system that is based on a bert-model, and fine-tune it to our task and domain. we generally see mention detected as similar to named entity recognition (ner), and follow best practices and settings for ner. the coreference resolution component is based on stanford's multi-pass sieve coreference resolution system @xcite @xcite . rule-based systems have been shown to provide good results for cr on literary data @xcite . for this type of data, rule-based systems also come with certain advantages, as (i) no training data is needed, which is usually sparse for the literary domain (cf. @xcite , (ii) rule-based systems tend to generalize better on unseen domains @xcite ) and (iii) the rules can be easily crafted and changed to fit the needs of the specific domain @xcite . we adopt all passes of raghunathan et al. (2010) and @xcite to test how general purpose rules fair on dramas and add passes designed to yield high precision on this type of text. in particular, we add a pass that matches lemmas of heads of mentions, since the dramas are in german language and head words can be morphologically complex. furthermore, we advance the pronoun related rule of @xcite and split it into first, second and third person rules. this allows us to specifically target these different cases and exploit the fact that dramas mark speaker turns. like in @xcite , single passes are tested for their precision on a held-out dataset and then ordered by precision; the pass with the highest precision is applied first. passes which are applied later receive the clusters of previous passes as input, so clusters are build up successively. pass 12a and 12b like pass 12 in @xcite , but instead of wordnet @xcite , germanet @xcite is used. pass 14 this pass works the same as pass 1, except it does not operate on tokens but lemmas only. this change targets the morphological complexity of german, so that inflected word forms are not counted as a mismatch. post-processing in a post-processing step, clusters are merged when they contain at least one mention that is referring to the same cast member of the play. in the cases of first-person pronouns, the speaker can be identified as the referred cast member. in the cases of proper names, dramacoref gets a list of all cast members from the dramatis personae as input and tries to assign the correct cast member via string comparison between all mentions in the dramatis personae and all mentions in the cluster. in a very last step, all singletons, i.e. clusters that only contain one single mention, are removed, since all the datasets used in the experiments do not contain singletons. ## error analysis by having a closer look at the performance of the single passes in figure 2 , we can see that, as expected, passes 9a and 9b, which handle first and second person pronouns, perform very well. also pass 1 for exact string matches is relatively reliable, the remaining passes perform relatively on par. only passes 2a-c have almost no hits, as they handle constructions normally not present in dramas. we also include the three baselines b1-3 for comparison, which manage to outperform several passes. pass 9a performing well is not surprising, as first person pronouns can often be assigned very reliably to the cluster of the current speaker. assigning second person pronouns with pass 9b works surprisingly well, showing that a simple heuristic of assuming that the current character addresses the previous or following speaker works in many cases. pass 1, which performed highest for raghunathan et al. @xcite , is also strong for gerdracor-coref, but does not achieve a precision of 90+% as in @xcite . this might be due to the length of the texts, where mentions with identical surface forms can appear many times in different contexts. using lemmas instead of tokens for pass 14 does not outperform pass 1. 0.00 0.25 0.50 0.75 1.00 01 02 03 04 05 06 07 08 09 10 fold conll f1 figure 4: results in a 10-fold cross validation setting. figure 3 shows all passes applied cumulatively. this means that the pass with the highest precision on the development set is evaluated on the test, then the pass with the highest and secondhighest precision, and so on. this enables us to see how the evaluation scores change when more and more passes are added. other than in figure 2 , the passes are evaluated on the test set instead of the development set, in order to get more meaningful numbers on a larger dataset. thus, the ordering of the passes is different to figure 2 . figure 3 emphasizes that, while the overall precision naturally falls when more passes with lower individual precision are added, the recall does not rise significantly. this calls for passes that are able to increase recall while not dropping precision too much. while the used passes did exactly that in the experiments of @xcite and @xcite , and also for van cranenburgh (2019) on dutch novels, this seemingly does not apply to the dramas in gerdracor-coref. in general, passes from @xcite , which are meant for general purpose domains, do not perform well enough on dramas, while passes specifically engineered for this domains, do not perform well enough in order to achieve results comparable to other domains like newspaper texts. lastly, we make sure that the results are not caused by splitting the data into a certain train-test split by applying 10-fold cross validation. the results are shown in figure 4 . as can be seen, the results are relatively constant across folds, meaning that the plays form a homogeneous whole. ## conclusion and future work we present a system, dramacoref, for coreference resolution on german theater plays. dramacoref is hybrid in the sense that it uses a transformer architecture to retrieve potential mentions and in a second step clusters the retrieved mentions using ordered sieves. it performs comparable to other attempts of cr on literary data, however, theater plays appear to be a more difficult type of data for cr than texts of other types, e.g. newspaper texts. while some passes that work well on newspaper texts do not apply or underperform on the theater plays, specific passes and newly added passes perform reasonably well. our goal for future research is the exploration of new passes, utilizing information coming from the plays like speaker turns and informations from the dramatis personae like family relations, and improving the mention detection component. developing an end-to-end neural network based system might also be worthwhile, as it makes the need for two distinct components obsolete; however, at this point, the available training data does not seem to be sufficient for such an approach.