qxo R ef 1.0: A coreference corpus and mention-pair baseline for coreference resolution in Conchucos Q uechua (2021)

Abstract: This paper introduces qxoRef 1.0, the first coreference corpus to be developed for a Quechuan language, and describes a baseline mention-pair coreference resolution system developed for this corpus. The evaluation of this system will illustrate that earlier steps in the NLP pipeline, in particular syntactic parsing, should be in place before a complex task like coreference resolution can truly succeed. qxoRef 1.0 is freely available under a CC-BY-NC-SA 4.0 license.

## introduction coreference resolution is the task of identifying and grouping the phrases in a text that refer to the same real-life object, or in other words, grouping the mentions in a text-the phrases that refer to real-life objects-together into entities: clusters which represent those real-life objects @xcite . coreference resolution has been an important area of focus in nlp for the last thirty years. it is often used as one component of an nlp pipeline: it builds on information gained through tools like syntactic parsers and semantic word embeddings, yielding clusters of mentions that can be useful for further nlp tasks like question answering and sentiment analysis @xcite . to succeed at coreference resolution requires the synthesis of both linguistic and contextual (world) knowledge. current state-of-the-art coreference systems accomplish this using deep learning @xcite and are trained on large coreference corpora in majority languages like english, chinese, and arabic @xcite . although the aims of the present paper are more modest, it still makes two important contributions to the field of coreference resolution for low-resource languages. the first contribution is qxoref 1.0, the first coreference corpus to be developed for a quechuan language. the name reflects the variety of quechua that appears in the corpus, namely (southern) conchucos quechua (iso 639-3 code qxo). qxo-ref 1.0 is freely available under a creative commons cc-by-nc-sa 4.0 license. 1 the second contribution is a baseline coreference resolution system trained on this corpus. the term "quechua" is generally used to refer to the quechuan language family, a large group of related local varieties spoken widely in south america @xcite . the number of speakers of quechuan languages around the turn of the millennium was estimated at about eight million @xcite , so it is not a small language family. however, it contains two branches of different sizes. according to the classification of @xcite , the smaller "quechua i" is spoken in the peruvian highlands, while the much larger "quechua ii" is spoken throughout central and southern peru as well as in parts of @xcite . the two branches differ lexically, morphologically, and orthographically. the variety of quechua appearing in qxoref is spoken in conchucos, a district within the department of ancash in the peruvian highlands, and it belongs to quechua i. (an alternative division of the language family is offered by parker 1963, who labels quechuan varieties with a or b. in that schema, conchucos quechua belongs to quechua b.) one challenge of having chosen a quechua i variety to work with is the limited number of resources for that branch of the family tree. quechua ii, being much larger, has a handful of nlp tools already, including a toolknit developed by @xcite . this paper thus presents an exploratory illustration of how to develop a coreference corpus and baseline coreference system for a morphologically complex language in a low-resource situation. most coreference corpora are created for morphologically simple languages like english, but this project shows that the standard format for modern coreference corpora (the conll-2012 shared task tabular format; @xcite can also easily accommodate a morphologically complex language like quechua. the paper will first discuss the creation of qxo-ref in section 2, and then move on to the baseline mention-pair system developed for it in section 3. in the evaluation of this system in section 4, we will see the consequences of not having earlier steps of the nlp pipeline in place before constructing a coreference resolution system. while surface features may passably substitute for some parts of a deeper linguistic analysis @xcite and are often the only type of feature that is available in a low-resource language, we will see that the data in qxoref would still benefit significantly from linguistic analysis before the coreference resolution step takes place. however, before turning to these details, a few words on quechuan grammar are in order. ## qxoref 1.0 this section presents qxoref 1.0, a coreference corpus for conchucos quechua and, to the author's knowledge, the first such resource developed for a quechuan language. the section first explores how earlier coreference corpora in other pro-drop languages are structured (section 2.1). it then moves on to the data that qxoref is based on (section 2.2), how the mentions in this data were annotated (section 2.3), and some remaining limitations of the present version of the corpus (section 2.4). ## a mention-pair baseline for conchucos quechua the data in qxoref 1.0 was used to train a baseline coreference resolution system for conchucos quechua. how that system was implemented will be the focus of the present section; afterward, section 4 will discuss its performance with an illustrative error analysis. ## evaluation and error analysis the evaluation of each classifier's performance used the standard three coreference metrics-muc, b 3 , and ceaf e -as implemented in the scoring scripts from the conll-2012 shared task @xcite . the results are given in strikingly, although the proportion of positive to negative instances in the training data is nearly identical (see the general pattern, at least in the muc and b 3 metrics, is high precision and low recall. in other words, when the mentions were classified as coreferential, this was generally done correctly. however, the clustering procedure often failed to identify coreference links between anaphors and their true antecedents, leading to that anaphor's omission from the final entity representations. the error analysis in the next section will explore why this might have been the case. ## conclusion and outlook this paper introduced qxoref 1.0, a new coreference corpus for conchucos quechua, and presented a mention-pair baseline for coreference resolution with this corpus that obtains an average f1 score of 68.51. several directions for future work are clear. first, the coreference corpus should be improved. a more reliable dataset should be created by having mentions annotated by multiple annotators and computing the inter-annotator agreement. further, the sentences should be syntactically parsed. not only would this allow a more sophisticated feature representation for use in the classifier, it would also allow null arguments to be annotated as mentions. this should lead to higher recall, since fewer mentions will be discarded because the coreference connections are missed. (and until a parser for conchucos quechua becomes available, an interim measure of introducing empty slots where the null arguments would be would already likely lead to a more robust system, even without the underlying syntactic structure.) additionally, other avenues for improving the feature representations should be explored. for example, embeddings for a compatible variety of quechua are not out of reach. ancash quechua is a variety that subsumes conchucos quechua, and a collection of texts in this variety is available on the ancash quechua wikimedia page. this material could be used to create sub-word embeddings, for example following the procedure laid out in heinzerling and strube (2018), that could then be used to encode semantic information about the mentions for use in the classifier. overall, this project has highlighted some of the issues involved in nlp for low-resource languages. to succeed at complex nlp tasks like coreference resolution, certain steps in the text processing pipeline should already have been achieved, syntactic parsing being a prominent example. improving the basic nlp toolkits for low-resource languages will lead to greater success on tasks like coreference resolution, which is in turn important for even more complex downstream tasks. our focus should therefore first be on developing basic tools and extending existing ones, and then we can work upward from there.