A coreference corpus of T urkish situated dialogs (2024)

Abstract: The paper introduces a publicly available corpus of Turkish situated dialogs annotated for coreference. We developed an annotation scheme for coreference annotation in Turkish, a language with pro-drop and rich agglutinating morphology. The annotation scheme is tailored for these aspects of the language, making it potentially applicable to similar languages. The corpus comprises 60 dialogs containing in total 3900 sentences, 18360 words, and 6120 mentions.

## introduction coreference annotation and corpus research have attracted significant attention among nlp researchers, cognitive scientists, and linguists, as understanding referring expressions and the relations between them is fundamental to natural language understanding. numerous nlp tasks, including information retrieval, question answering, and summarization, require coreference resolution for effective performance. this need has resulted in an increase in the number of corpora annotated for coreference relations in recent decades, particularly with the success of data-driven techniques, especially for widely-studied languages like english @xcite @xcite and german @xcite . however, the majority of languages still remain low-resourced in this respect. turkish, a member of the turkic language family, is among these low-resourced languages, facing a scarcity of coreference-annotated datasets. the available annotation schemes, predominantly designed for languages like english, fall short when applied to morphologically rich and pro-drop languages like turkish. such languages exhibit complex inflectional morphemes and allow reduced or null forms when the referents are pragmatically inferable or morphologically cued by agreement. in this connection, adapting existing annotation schemes to turkish poses numerous challenges and it is particularly challenging to offer a universal scheme for all languages when the complexity of the anaphoric phenomena is taken into consideration as stated by @xcite . for instance, the treatment of morphological information, such as suffixes that carry referential information, is often overlooked. similarly, the handling of phonologically null elements, which are pervasive in turkish, is not sufficiently addressed. this inadequacy can lead to a loss of critical information necessary for accurate coreference resolution. as a result, there is a need for developing a specialized annotation scheme that can accommodate the unique features of turkish and similar languages, ensuring more robust and reliable coreference annotation. this study is driven by the necessity to develop a coreference dataset in turkish, a language with relatively limited resources. it proposes a novel annotation scheme for coreference annotation, addressing the challenges encountered when adapting existing schemes designed for languages such as english. the paper is organized as follows: sections 2 outlines the basic terminology related to coreference. section 3 reviews the related work in coreference corpora. section 4 describes the initial steps in corpus development. section 5 introduces the proposed annotation scheme. section 6 provides descriptive statistics of the resulting corpus. section 7 ends with a summary and outlines future research directions. ## basic terminology coreference can be better understood within the larger picture of cohesion and concepts related to it. cohesion itself is based on the idea that the spoken or written communication is usually a united whole rather than unrelated utterances or sentences. for cohesion to occur, the interpretation of some linguistic element in the discourse sometimes depends on previously mentioned items in the text @xcite . a closely related notion to cohesion is reference. it is the relationship between a linguistic expression and an entity in the world. there are two main types of reference. exophoric reference refers to an entity which is outside the text. on the contrary, endophoric reference refers to another expression in the preceding discourse segment. endophora is further divided into two types. anaphora can be described as an item which relates back to a previous item in some way. the element which is referring back is called anaphor and the previously mentioned entity which then anaphor refers to or is related to is its antecedent. the process of linking the anaphor with its antecedent is called anaphora resolution. cataphora, on the other hand, points to an item in the following discourse segment. there are a variety of anaphora which are observed in written or oral language based on the form of the anaphor @xcite . lexical noun phrase anaphora could appear as proper names and definite descriptions. pronominal anaphora is one of the most studied and therefore understood type of anaphora in the literature. anaphors in this type can be in the form of personal pronouns, possessive pronouns, reflexive pronouns, and demonstrative pronouns. another type of anaphora is zero anaphora. it is considered to be one of the most challenging types of anaphora to resolve since they are not physically realized at the surface level. although they are invisible, they do not damage the cohesion of the discourse but strengthen it. they are decoded by the reader or hearer without any loss during the comprehension of the discourse. if the anaphor and antecedent refer to the same entity, they are thought to be coreferential. this relation is also called identity anaphora, as in (1). an anaphor can be preceded by a number of expressions referring to the same entity and therefore they are said to form a coreference chain. such theoretical work on reference and anaphora has become the foundation of the guidelines which have been prepared to create coreference corpora. ## related work the earliest attempts to develop annotation schemes for coreference annotation could be traced back to the message understanding conference (muc) information extraction tasks @xcite . the task was created to group all the mentions of an entity together and the scheme specified the basic task criteria, the markables to be annotated and the relations to be established in english. the task evolved with automatic content extraction (ace) @xcite enriching the coverage with entity, relation, and event annotation in english, chinese and arabic. the mate/gnome proposals @xcite were geared towards being more linguistically oriented than previous schemes, making a discourse model assumption. it also included bridging anaphora in addition to identity relations. the pocos -potsdam coreference scheme @xcite claimed to adopt language independent principles during markable annotation. the scheme applied to german, english and russian. the ontonotes guidelines @xcite includes several layers, one of which is coreference layer. it aimed to include all coreferential relations and specifically focuses on how to handle identity relations and appositives. like the ace scheme, it was applied to english, arabic and chinese. the later schemes have become more comprehensive, including different kinds of anaphora in addition to coreference and more fine grained subcategories like arrau @xcite . however, some guidelines took a more psychological approach and considered coreference as part of information structure annotation. @xcite developed a scheme to annotate coreference and information status relations in english dialogs. @xcite prepared guidelines for information status, topic, and focus annotation. they aimed for language independence, theory neutrality, reliable marking, and framed coreference under information status annotation in terms of givenness. the reflex scheme @xcite was developed for referential and lexical analysis of spoken and written text. coreferentiality has been at the referential level along with bridging relations in the scheme. the development of annotation schemes have paved the way for the construction of many corpora in different languages. the initial products were naturally produced in the languages of the schemes mentioned above. one of the well-known and largest coreference corpora is the ontonotes project @xcite . it consists of various genres such as news, conversational telephone speech, weblogs, usenet newsgroups, broadcast, talk shows. it was annotated for syntax and predicate argument structure and word sense and coreference in english, chinese, and arabic. ar-rau @xcite ) is another multi-genre corpus which contains around 350k tokens. unlike many corpora, it accepts nonreferential nps and singletons as markable. it was annotated for different kinds of anaphoric relations including coreference, bridging anaphora and discourse deixis. similar to ontonotes, ancora @xcite is also a multilingual corpus. it consists of 500k tokens of newspaper texts in spanish and catalan. the texts were annotated for morphological information, syntactic phrases, grammatical functions relations. parcorfull @xcite is a parallel corpus of english and german with a total of 160k tokens. it was only annotated for coreference relations. the growing interest and need in coreference datasets triggered corpus development in other languages such as czech @xcite , hungarian @xcite , polish @xcite , dutch and @xcite . however, to the best of our knowledge, the only coreference corpus developed in turkish was marmara turkish coreference corpus @xcite . it is an annotation layer on top of the metu-sabancı treebank, which consists of 33 documents from various genres with 53925 tokens in total. the scheme prepared for corpus includes noun phrases, pronouns, and nominalized adjectives as markables, but it does not consider the role of morphological information and null elements in turkish. the gold data obtained from several annotators resulted in 5170 mentions and 944 coreference chains. arslan and eryigit (2023) reannotated the corpus to handle the dropped pronouns with the data representation scheme they proposed. however, their scheme only deals with how to represent third person singular agreement makers and possessive pronouns for dropped pronouns. due to this limited availability of turkish coreference data, the computational work on coreference in turkish is also rather limited and mostly have exploited rule-based and classical machine learning methods. @xcite developed a rulebased system for anaphora resolution in turkish. their model depends on the theoretical framework of the centering theory. in a later study, @xcite presented a computational model for resolving pronominal anaphora. it is based on hobbs' naïve algorithm @xcite , which traverses a parse tree to find the antecedent of a pronominal anaphora. the first learning-based approach to anaphora resolution is limited to pronoun resolution @xcite . they trained a decision tree on a corpus of popular child stories. @xcite proposed the first coreference resolution system, which uses support vector machines with a mention-pair model. there are recent attempts to use deep learning methods for turkish coreference resolution. @xcite presented the first neural coreference resolution system and arslan et al. (2023) introduced a neural multilingual coreference resolution model which makes use of morphological information. however, they remain limited due to data sparsity. ## annotation scheme development we evaluated available schemes mentioned above and found that they were lacking the devices that are required for the annotation of sub-word morphological units and null elements. therefore, we developed a scheme which is comprehensive enough to handle all realizations of mentions at different levels of turkish structure. in this way, the scheme could be a model for morphologically rich lowresourced turkic languages which frequently utilize null elements. ## analysis each text in the corpus has been independently annotated by two annotators. they identified the mentions in the texts and established the identity relations between them. this provided us with the unique entities and their realizations in the texts, in other words, their mentions. they labeled these mentions with the grammatical information with the categories subject, object, and other. we built a custom tool in python that (i) exported the annotated texts from labelbox, (ii) compared the annotations and calculated inter-coder agreement, (iii) extracted a graph representation of the coreference patterns of the dialogs, and (iv) performed basic statistics.