An Empirical Study of Pre-trained Transformers for A rabic Information Extraction (2020)

Abstract: Multilingual pre-trained Transformers, such as mBERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020a), have been shown to enable effective cross-lingual zero-shot transfer. However, their performance on Arabic information extraction (IE) tasks is not very well studied. In this paper, we pre-train a customized bilingual BERT, dubbed GigaBERT, that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning. We study GigaBERT’s effectiveness on zero-short transfer across four IE tasks: named entity recognition, part-of-speech tagging, argument role labeling, and relation extraction. Our best model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT (Antoun et al., 2020) in both the supervised and zero-shot transfer settings. We have made our pre-trained models publicly available at: https://github.com/lanwuwei/GigaBERT .

## introduction fine-tuning pre-trained transformer models @xcite @xcite has recently achieved state-of-the-art results on a wide range of nlp tasks where supervised training data is available. when trained on multilingual corpora, bert-based models have demonstrated the ability to learn multilingual representations that support zero-shot cross-lingual transfer learning surprisingly effectively @xcite @xcite . without access to any parallel text or target language annotations, multilingual bert (mbert; @xcite even supports cross-lingual transfer for language pairs that are written in different scripts, for example, english-to-arabic. however, the transfer learning performance still lags far behind where supervised data is available in the target language. in this paper, we explore to what extent it is possible to improve performance in the zero-shot scenario by building a customized bilingual bert for english and arabic, a particularly challenging language pair for cross-lingual transfer learning. we present gigabert, a customized bert for english-to-arabic cross-lingual transfer that is trained on newswire text in the gigaword corpus @xcite in addition to wikipedia and web crawl data. we systematically compare our pre-trained models of different configurations against the mbert @xcite and xlm-roberta (xlm-r; @xcite . by using a customized vocabulary and code-switched data specifically created for english-to-arabic transfer learning, our giagbert outperforms mbert and xlm-r base (both support more than 100 languages) on a range of ie tasks, including named entity recognition, part-of-speech tagging, argument role labeling, and relation extraction. further performance gains are demonstrated by augmenting the pre-training corpus with synthetically generated code-switched data. this demonstrates the usefulness of anchor points for zero-shot cross-lingual transfer learning. gigabert also performs well when annotated arabic data is available, outperforming @xcite , the state-of-the-art arabic-specific bert model, on various arabic ie tasks. ## related work the existing arabic pre-trained models are either monolingual, such as hulmona @xcite and @xcite ; or multilingual with several or over a hundred languages, such as mbert @xcite , xlm @xcite , and xlm-roberta @xcite . there is no bilingual pre- models training data vocabulary configuration source #tokens (all/en/ar) tokenization size (all/en/ar) cased size #parameters arabert newswire 2.5b/ -/2.5b sentencepiece 64k/ -/ 58k no base 136m mbert wiki 21.9b/2.5b/153m wordpiece 110k/53k/5k yes base 172m xlm-r base commoncrawl 295b/55.6b/2.9b sentencepiece 250k/80k/14k yes base 270m xlm-r large commoncrawl 295b/55.6b/2.9b sentencepiece 250k/80k/14k yes large 550m giagbert-v0 gigaword 4.7b/3.6b/1.1b sentencepiece 50k/28k/19k yes base 125m gigabert-v1 gigaword, wiki 7.4b/6.1b/1.3b wordpiece 50k/25k/23k yes base 125m gigabert-v2/3 gigaword, wiki, oscar 10.4b/6.1b/4.3b wordpiece 50k/21k/26k no base 125m gigabert-v4 gigaword, wiki, oscar (+ code-switch) 10.4b/6.1b/4.3b wordpiece 50k/21k/26k no base 125m ## gigabert we present five versions of gigabert pre-trained using the transformer encoder @xcite with bert base configurations: 12 attention layers, each has 12 attention heads and 768 hidden dimensions, which attributes 110m parameters. ## conclusions in this paper, we show that the performance of zero-shot cross-lingual transfer can be improved by training customized bilingual bert for a given language pair and text domain. we pre-trained several masked language models (gigaberts) for arabic-english and conducted a focused study on information extraction tasks in the newswire domain. the experiments show that our gigabert model outperforms multilingual bert, xlm-roberta, and the monolingual arabert on ner, pos, arl and re tasks. we also achieve the new state-of-the-art performance fro zero-shot transfer learning from english to arabic. we additionally studied codeswitched pre-training for gigabert and domainadapted pre-training for xlm-roberta. and iarpa via the better program , darpa via the aro (w911nf-17-c-0095) in addition to an amazon research award. the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of odni, iarpa, aro, darpa, or the u.s. government. the u.s. government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.