Neural Coreference Resolution for A rabic (2020)

Abstract: No neural coreference resolver for Arabic exists, in fact we are not aware of any learning-based coreference resolver for Arabic since (Björkelund and Kuhn, 2014). In this paper, we introduce a coreference resolution system for Arabic based on Lee et al’s end-to-end architecture combined with the Arabic version of bert and an external mention detector. As far as we know, this is the first neural coreference resolution system aimed specifically to Arabic, and it substantially outperforms the existing state-of-the-art on OntoNotes 5.0 with a gain of 15.2 points conll F1. We also discuss the current limitations of the task for Arabic and possible approaches that can tackle these challenges.

## introduction coreference resolution is the task of grouping mentions in a text that refer to the same real-world entity into clusters @xcite . coreference resolution is a difficult task that requires reasoning, context understanding, and background knowledge of real-world entities, and has driven research in both natural language processing and machine learning, particularly since the release of the ontonotes multilingual corpus providing annotated coreference data for arabic, chinese and english and used for the 2011 and 2012 conll shared tasks @xcite . since then, there has been substantial research on english coreference, most recently using neural coreference approaches @xcite @xcite @xcite @xcite , leading to a significant increase in the performance of coreference resolvers for english. by contrast, there has been almost no research on arabic coreference; the performance for arabic coreference resolution has not improved much since the conll 2012 shared task, and in particular no neural architectures have been proposed-the current state-of-the-art system remains the model proposed in @xcite . in this paper we close this very obvious gap by proposing what to our knowledge is the first neural coreference resolver for arabic. 1 one explanation for this lack of research might simply be the lack of training data large enough for the task. another explanation might be that arabic is more problematic than english because of its rich morphology, its many dialects, and/or its high degree of ambiguity. we explore the first of these possibilities. coreference resolution can be further divided into two subtasks-mention detection and mention clustering-as illustrated in figure 1 . in early work, coreference's two subtasks were usually carried out in a pipeline fashion @xcite @xcite @xcite @xcite , with candidate mentions selected prior the mention clustering step. since @xcite introduced an end-to-end neural coreference architecture that achieved state of the art by carrying out the two tasks jointly, as first proposed by @xcite , most state-of-the-art systems have followed this approach. however, no end-to-end solution was attempted for arabic. we intend to explore whether an end-to-end solution would be practicable with a corpus of more limited size. the approach we followed to adapt the state-of-the-art english coreference resolution architecture to arabic is as follows. we started with a strong baseline system @xcite , enhanced with contextual bert embeddings @xcite . we then explored three methods for improving the model's performance for arabic. the first method is to pre-process arabic words with heuristic rules. we follow @xcite to normalize the letters with different forms, and removing all the diacritics. this results in a substantial improvement of 7 percentage points over our baseline. the second route is to replace multilingual bert with a bert model trained only on the arabic texts (arabert) @xcite . multilingual bert is trained with 100+ languages; as a result, it is not optimized for any of them. as shown by @xcite , monolingual bert trained only on the arabic texts has better performance on various nlp tasks. we found the same holds for coreference: using embeddings from monolingual bert, the model further improved the conll f1 by 4.8 percentage points. our third step is to leverage the end-to-end system with a separately trained mention detector @xcite . we show that a better mention detection performance can be achieved by using a separately trained mention detector. and by using a hybrid training strategy between the end-to-end and pipeline approaches (end-to-end annealing) our system gains an additional 0.8 percentage points. our final system achieved a conll f1 score of 63.9%, which is is 15% more than the previous state-of-the-art system @xcite on arabic coreference with the conll dataset. overall, we show that the state-of-the-art english coreference model can be adapted to arabic coreference leading to a substantial improvement in performance when compared to previous feature-based systems. ## experimental setup since the bert models are large, the fine-tuning approaches are more computationally expensive: gpu/tpus with large memory (32gb+) are required. in this work, we use bert embeddings in a pretrained fashion to make our experiment feasible on a gtx-1080ti gpu with 11gb memory. ## conclusion in this paper, we modernize the arabic coreference resolution task by adapting state-of-the-art english coreference system to the arabic language. we start with a strong baseline system and introduce three methods (data pre-processing, language-specific bert, external mention detector) to effectively enhance the performance of the arabic coreference resolution. our final system enhanced by all three methods achieved a conll f1 score of 63.9% and improved the state-of-the-art result on arabic coreference resolution task by more than 15 percentage points.