Ensemble Transfer Learning for Multilingual Coreference Resolution (2023)

Abstract: Entity coreference resolution is an important research problem with many applications, including information extraction and question answering. Coreference resolution for English has been studied extensively. However, there is relatively little work for other languages. A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data. To overcome this challenge, we design a simple but effective ensemble-based framework that combines various transfer learning (TL) techniques. We first train several models using different TL methods. Then, during inference, we compute the unweighted average scores of the models’ predictions to extract the final set of predicted clusters. Furthermore, we also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts. Leveraging the idea that the coreferential links naturally exist between anchor texts pointing to the same article, our method builds a sizeable distantly-supervised dataset for the target language that consists of tens of thousands of documents. We can pre-train a model on the pseudo-labeled dataset before finetuning it on the final target dataset. Experimental results on two benchmark datasets, OntoNotes and SemEval, confirm the effectiveness of our methods. Our best ensembles consistently outperform the baseline approach of simple training by up to 7.68% in the F1 score. These ensembles also achieve new state-of-the-art results for three languages: Arabic, Dutch, and Spanish.

## introduction within-document entity coreference resolution is the process of clustering entity mentions in a document that refer to the same entities @xcite @xcite . it is an important research problem, with applications in various downstream tasks such as entity linking 1 data and code will be made available upon publication. @xcite , question answering @xcite , and dialog systems @xcite . researchers have recently proposed many neural methods for coreference resolution, ranging from span-based end-to-end models @xcite to formulating the task as a question answering problem @xcite . given enough annotated training data, deep neural networks can learn to extract useful features automatically. as a result, on english benchmarks with abundant labeled training documents, the mentioned neural methods consistently outperform previous handcrafted feature-based techniques @xcite , achieving new state-of-the-art (sota) results. compared to the amount of research on english coreference resolution, there is relatively little work for other languages. a problem that frequently occurs when working with a non-english language is the scarcity of annotated training data. for example, the benchmark ontonotes dataset contains about eight times more documents in english than in arabic @xcite . some recent studies aim to overcome this challenge by applying standard cross-lingual transfer learning (tl) methods such as continued training or joint training @xcite . in continued training, a model pretrained on a source dataset is further finetuned on a (typically smaller) target dataset @xcite . in joint training, a model is trained on the concatenation of the source and target datasets @xcite . the mentioned studies only use one transfer method at a time, and they do not explore how to combine multiple tl techniques effectively. this can be sub-optimal since different learning methods can be complementary @xcite . for example, our experimental results to be discussed later show that continued training and joint training are highly complementary. furthermore, a disadvantage of using a cross-lingual transfer method is the requirement of a labeled coreference resolution dataset in some source language (usually english). in this work, we propose an effective ensemblebased framework for combining various tl techniques. we first train several coreference models using different tl methods. during inference, we compute the unweighted average scores of the models' predictions to extract the final set of mention clusters. we also propose a low-cost tl method that bootstraps coreference models without using a labeled dataset in some source language. the basic idea is that the coreference relation often holds between anchor texts pointing to the same wikipedia article. based on this observation, our tl method builds a sizable distantly-supervised dataset for the target language from wikipedia. we can then pretrain a model on the pseudo-labeled dataset before finetuning it on the final target dataset. experimental results on two datasets, ontonotes and semeval @xcite , confirm the effectiveness of our proposed methods. our best ensembles outperform the baseline approach of simple training by up to 7.68% absolute gain in the f1 score. these ensembles also achieve new sota results for three languages: arabic, dutch, and spanish. in summary, our main contributions include: • we introduce an ensemble-based framework that combines various tl methods effectively. ## methods figure 1 shows an overview of our framework. during the training stage, we train several coreference resolution models using various tl approaches. for simplicity, we use the same span-based architecture (section 2.1) for every model to be trained. however, starting from the same architecture, using different learning methods typically results in models with different parameters. in this work, our framework uses two types of tl methods: (a) crosslingual tl approaches (section 2.2) and (b) our newly proposed wikipedia-based approach (section 2.3). the cross-lingual tl methods require a labeled coreference resolution dataset in some source language, but our wikipedia-based method does not have that limitation. our framework is general as it can work with other learning methods (e.g., self-distillation). during inference, we use a simple unweighted averaging method to combine the trained models' predictions (section 2.4). ## conclusions and future work in this work, we propose an ensemble-based framework that combines various tl techniques. we also introduce a low-cost wikipedia-based tl approach that does not require any labeled source dataset. our approaches are highly effective, as our best ensembles achieve new sota results for three different languages. an interesting future direction is to explore the use of model compression techniques @xcite @xcite to reduce the computational complexity of our ensembles.