Multilingual Coreference Resolution in Multiparty Dialogue (2023)

Abstract: Existing multiparty dialogue datasets for entity coreference resolution are nascent, and many challenges are still unaddressed. We create a large-scale dataset, Multilingual Multiparty Coref (MMC), for this task based on TV transcripts. Due to the availability of gold-quality subtitles in multiple languages, we propose reusing the annotations to create silver coreference resolution data in other languages (Chinese and Farsi) via annotation projection. On the gold (English) data, off-the-shelf models perform relatively poorly on MMC, suggesting that MMC has broader coverage of multiparty coreference than prior datasets. On the silver data, we find success both using it for data augmentation and training from scratch, which effectively simulates the zero-shot cross-lingual setting.

## introduction coreference resolution is a challenging aspect of understanding natural language dialogue @xcite . many dialogue datasets are between two participants, even though there are distinct challenges that arise in the multiparty setting with more than two speakers. figure 1 shows how ''you'' could refer to any subset of the listeners of an utterance. while there are some datasets on multiparty conversations from tv transcripts (choi and chen, 2018), they only annotate people, resulting in incomplete annotations across entity types. moreover, these datasets are only limited to english, and works in dialogue coreference resolution in other languages are rare @xcite . we introduce a new (entity) coreference resolution dataset focused on multiparty dialogue that supports experiments in multiple languages. we first annotate for coreference on the transcripts from two popular tv shows, in english. we * work done at jhu/hltcoe. our experiments demonstrate that coreference resolution models trained on existing datasets are not robust to a shift to this domain. further, we demonstrate that training on our projected annotations to non-english languages leads to improvements in non-english evaluation. finally, we lay out an evaluation for zero-shot cross-lingual coreference resolution, requiring models to test on other languages with no in-language examples. we release over 1,200 scenes from tv shows with all annotations and related metadata in english, chinese, and farsi, which we call mmc: multilingual multiparty coreference. ## motivation and related work most work on coreference resolution primarily studies documents with a single author or speaker. ontonotes @xcite is a widely used dataset that mostly consists of single-author documents, like newswire, while other datasets like preco @xcite , litbank @xcite , and wikicoref @xcite also consist of documents like books. many recent modeling contributions also focus primarily on this setting and these datasets @xcite @xcite and some offload it to pretrained language models @xcite or ignore the speaker identity entirely @xcite in an attempt to unify dialogue with non-dialogue domains. the dialogue domain is less studied because we lack a suitable dataset, even though these exist for other nlp tasks (section 2.1). in addition to filling this gap, we also present a scalable solution for dataset creation in other languages, following related work in data projection methods (section 2.2). the limitations of existing work motivate the creation of our dataset. ## multilingual multiparty dialogue coreference dataset in this section, we present our multilingual multiparty coreference (mmc) dataset, foot_1 including the construction process of data alignment and filtering, annotation, and projection. 3 core to our contribution is the choice of a multiparty dataset that already has gold translations and prioritizing multilinguality throughout the data collection process. ## experiments and results first, we highlight the differences of mmc in contrast to friends ci and ontonotes. to do so, we train an off-the-shelf model on the three datasets. additionally, we establish monolingual baselines for all three languages. finally, we explore the cross-and multi-lingual settings to validate the recipe of using data projection for coreference resolution. ## analysis we analyze our modeling results in relation to our original motivation. first, we explore differences between datasets (section 6.1), the number of speakers (section 6.2), and overfitting (section 6.3). for the data construction, we analyze the alignment corrections process (section 6.4) and compare recipes for annotation projection (section 6.5). ## conclusion motivated by a desire to better understand spontaneous multilingual conversations, we developed a collection of coreference annotations on the transcripts and subtitles of two popular tv sitcoms. to reduce the cost of annotating from scratch for each language, we selected our english data such that there were already existing gold human translations available in the form of subtitles, in order to automatically project our annotations from english. after manually correcting these projections, we observe a few point differences in reported values across various multilingual models. there exist dozens of additional languages that our annotations may be projected to in the future. if automatic projection leads to only a few point variance in the estimated performance of a model, we believe this framework is sufficient for driving significant new work in coreference across many non-english languages in the future.