Exploring Pre-Trained Transformers and Bilingual Transfer Learning for A rabic Coreference Resolution (2021)

Abstract: In this paper, we develop bilingual transfer learning approaches to improve Arabic coreference resolution by leveraging additional English annotation via bilingual or multilingual pre-trained transformers. We show that bilingual transfer learning improves the strong transformer-based neural coreference models by 2-4 F1. We also systemically investigate the effectiveness of several pre-trained transformer models that differ in training corpora, languages covered, and model capacity. Our best model achieves a new state-of-the-art performance of 64.55 F1 on the Arabic OntoNotes dataset. Our code is publicly available at https://github.com/bnmin/arabic_coref .

## introduction coreference resolution is the task of identifying all mentions which refers to the same discourse entity. it is a crucial step for many nlp tasks, and has been well-studied @xcite for english. recently, neural coreference resolution models based on pre-trained tranformers, e.g., bert @xcite and spanbert @xcite , have shown to be highly effective for english coreference resolution, achieving 80 f1 on the ontonotes @xcite dataset. however, coreference resolution in arabic is still a challenging problem with previous systems @xcite @xcite @xcite @xcite performing significantly lower than the english systems. a main problem is that arabic coreference resolution datasets are much smaller than their english counterparts. for example, the widely useful benchmark dataset ontonotes contains 7 times more documents in english than arabic ( based on this observation, we develop bilingual transfer learning approaches to improve arabic coreference resolution by leveraging additional english coreference annotation via bilingual or multilingual pre-trained transformers, e.g., multilingual bert @xcite . we summarize our contributions below: our source code are made publicly available. ## related work neural models for coreference resolution: stateof-the-art models for coreference resolution are based on neural networks (nn). e2e-coref @xcite ) is an nn model that performs span extraction and coreference classification in a single end-to-end model. bert-based models @xcite further improve e2e-coref by replacing the bi-lstm text encoder with bert @xcite or spanbert @xcite . corefqa @xcite formulates coreference resolution as a qa task leveraging pre-trained transformers. arabic coreference resolution: björkelund and kuhn (2014) investigates structured perceptron models for arabic coreference resolution. @xcite uses a hybrid approach combining rule-based methods and learning-based methods. @xcite is a latent structure perceptron model along with entropy-guided feature induction. @xcite is a memorybased coreference resolution system. @xcite uses a mention classifier with features extracted from the dependency graphs of the sentences. @xcite proposed an algorithm for multilingual mention detection by extracting mentions from parse trees via kernelbased svm learning. @xcite improves structured prediction models for arabic coreference resolution by using more expressive loss functions. @xcite is a projection-based model in which they first translate arabic into english, run a coreference resolution system, and then map the coreference clusters back into arabic. @xcite developed a bert-based cross-lingual models for zero pronoun resolution in arabic and chinese. aloraini et al (2020) developed an arabic neural coreference system, and showed that data processing and mention detection improve the performance by about 7.8 f1 points over the pure neural coreference system, resulting in 63.9 f1 that is higher than previous approaches but lower than ours ( pre-trained transformers: pre-trained transformer based language models such as bert has been shown to be effective for many nlp tasks. arabert @xcite ) is a bertlike model trained with arabic text. to enable cross-lingual transfer learning, multilingual pretrained transformers such as mbert @xcite are trained with large corpora with many languages to encode similar words (words with similar meanings) from different languages into nearby high-dimensional vector spaces. gigabert @xcite ) is an arabic-english bilingual language model trained with arabic and english text. ## neural coreference models for arabic our coreference resolution model is based on an end-to-end neural model bert-coref @xcite , which is built on top of e2e-coref @xcite . below we give a brief overview of bert-coref , and then describe our extensions to this model for arabic coreference resolution. ## experiments we evaluate the models on ontonotes 5.0 @xcite , which is widely used as a benchmark dataset for coreference resolution. ontonotes also contains both arabic and english documents annotated with coreference, which makes bilingual transfer learning (section 3.3) experiments feasible . to leverage english annotation in bilingual transfer learning, we also implemented the following models (shown in for all models, we set the pre-trained transformers' learning rate to 1e -5, the task learning rate to 2e -4, the number of epochs to 30 except for the pipeline models, and use the default values for the remaining parameters. we use the performance on the dev set to choose the best hyperparameter for all models, and use early stopping to prevent overfitting. all models are trained with an nvidia quadro rtx 8000 gpu. each model trains for 3 to 12 hours depending on the size of the data (arabic or the combined set). on pre-trained tranformers for arabic: comparing mbert, arabert and gigabert, first, the size of arabic text used for pre-training significantly affects the pre-trained transformer' ability to encode text for arabic coreference resolution. as shown in second, arabert and gigabert are trained specifically for arabic, while mbert is trained for 104 languages and has not been tuned for arabic. the language-specific training makes arabert and gigabert models specialized to capture the intricacies of arabic, leading to better performance. on bilingual transfer learning: comparing the bilingual transfer learning models (table3 (2)) to the arabic-only models (table3 (1)), we observe 3 and 2.5 average f1 improvements when using mbert and gigabert respectively. this shows the effectiveness of the bilingual transfer learning. the pipeline and joint approach shows similar gains. this indicates that it is the additional training data rather than the learning model that leads to the gain. interestingly, using mbert and giga-bert gives similar gains, which shows that the transferability of the these two models in english to arabic is about the same (though with mbert lagging in baseline arabic performance by 15.29 f1). it is surprising given that gigabert is a bilingual model trained specifically for english-arabic. we plan to investigate this as an immediate next step. ## conclusion we present bilingual tranfer learning for arabic coreference resolution, that achieved new state-ofthe-art performance as evaluated on ontonotes.