Lazy Low-Resource Coreference Resolution: a Study on Leveraging Black-Box Translation Tools (2021)

Abstract: Large annotated corpora for coreference resolution are available for few languages. For machine translation, however, strong black-box systems exist for many languages. We empirically explore the appealing idea of leveraging such translation tools for bootstrapping coreference resolution in languages with limited resources. Two scenarios are analyzed, in which a large coreference corpus in a high-resource language is used for coreference predictions in a smaller language, i.e., by machine translating either the training corpus or the test data. In our empirical evaluation of coreference resolution using the two scenarios on several medium-resource languages, we find no improvement over monolingual baseline models. Our analysis of the various sources of error inherent to the studied scenarios, reveals that in fact the quality of contemporary machine translation tools is the main limiting factor.

## introduction end-to-end coreference resolution is the task of identifying and clustering all spans of text that refer to the same entity in a document. it serves as an important step for several downstream nlp tasks that involve natural language understanding, including question answering (morton, 1999), information retrieval, and text summarization @xcite . recent advances in deep learning have resulted in state-ofthe-art performance on coreference resolution @xcite @xcite @xcite . the performance of these models, however, highly depends on the existence of large annotated datasets. still, for many languages that lack large annotated coreference corpora, machine translation (mt) tools of an ever increasing quality are available. the idea studied in this work, is whether existing black-box translation tools can be readily leveraged for transferring the task of coreference resolution from one language to another. we tackle the setting in which a large labeled corpus exists in a resource-rich language (i.e., the 'source' language) whereas only a smaller corpus exists in a smaller-resource language (called the 'target' language). specifically, we consider two scenarios in which black-box mt tools can be integrated into a cross-lingual end-to-end coreference resolution system. the first scenario, translatetrain, uses an mt tool to translate the large source corpus into the target language, after which a coreference model is trained in the target language. in the second scenario, translate-test, test examples in the target language are first machine translated to the source language, after which a pre-trained coreference model is used to predict the labels. the second scenario has the disadvantage that an mt tool is required at inference time. similar transfer learning setups for basic sequence tagging tasks gave encouraging results (as discussed in section 4), but we find this is no longer the case for the task of coreference resolution. we analyze the different sources of error related to integrating the mt tool in the pipeline. as it turns out, translation errors have the strongest impact on the effectiveness of the proposed methods, followed by prediction errors and alignment issues. ## experimental evaluation data -our evaluation set was created for the semeval-2010 @xcite shared task, and contains coreference annotations for six languages (see coreference models -for the translate-train scenario, we use the end-to-end neural coreference resolution method from @xcite to train and evaluate on the target languages. this model considers all spans of text as potential mentions and finds the most probable antecedents for each span. for each span, a span ranking model is used to decide which of the previous spans are good antecedents, whereby a trained pruner eliminates less likely mentions. during training, the marginal log-likelihood of all correct antecedents in the gold clusters is optimized. in our translate-test experiments, we use spanbert @xcite , an english end-to-end coreference resolver, trained on the ontonotes corpus. baselines -we compare our translate-train and translate-test approaches with a model trained on annotated data in the target language (i.e., end2end coref). we also consider two alternative baseline systems for which end-to-end coref- ## related work the key concept used in the presented transfer learning scenarios, is annotation projection, as originally proposed by @xcite for part-of-speech tagging. it relies on the transfer of annotations from the source language to the target language. most annotation projection methods depend on parallel corpora in which the source data is labeled using a trained model before projecting the labels onto the data in the target language @xcite @xcite @xcite @xcite . alternatively, other works relied on the use of bilingual dictionaries for annotation projection @xcite . the translate-train idea of creating a noisy translated corpus with projected annotations has been proposed as well @xcite , for the task of dependency parsing and ner, respectively. @xcite used mt in the other direction (translate-test) for the task of ner. a common problem in both annotation projection scenarios is the alignment of text spans between languages, for which unsupervised statistical alignment models can be used @xcite , such as the ibm models 1-6 @xcite . a few recent works @xcite perform translation on a word or span level to avoid the alignment problem. others explored alignment heuristics such as matching words based on their surface forms and translations @xcite , or using external information such as wikipedia links @xcite . the prior works applied annotation projection to the tasks of ner, pos, or dependency parsing, and proved relatively successful (i.e., close to monolingual models in the target language). @xcite ; de @xcite are notable prior works that applied the idea of annotation projection to the task of coreference resolution. un-like our work, they depend on the existence of a parallel corpora and are focused on a single language pair to test their ideas. moreover, they have a pipeline that extracts mentions using external annotation tools or manually before clustering them into coreference chains. we, on the other hand, do both the mention identification and clustering in an end-to-end fashion by considering all the possible text spans on a document. for the task of end-to-end coreference resolution, we explore the use machine translation for annotation projection, especially for medium-resource languages for which strong mt systems exist. we investigate if mt systems can be used for transferring coreference knowledge (model, dataset) without having to rely on parallel corpora. ## conclusion and future work while the idea of leveraging mt to improve nlp task performance for low resource languages is not new, this idea to the best of our knowledge has not been pursued for coreference resolution. we contribute by comparing two conceptually different methods; the translate-train and translate-test approaches. we further present a rigorous quantitative error analysis. from our work, we conclude that (i) for coreference resolution the mt approaches are not very successful. (ii) our error analysis suggests this is mainly due by translation errors followed by labeling and alignment errors. we believe mt models can be still leveraged in cross-lingual transfer learning for coreference resolution, but we speculate that access to the internals of the models, such as attention weights, will be needed. moreover, future work will need to investigate hybrid strategies, combining transfer learning from other languages with the available data in the target language, to override issues due to mt uncertainty or differences in annotation guidelines.