Cross-lingual transfer for low-resource A rabic language understanding (2022)

Abstract: This paper explores cross-lingual transfer learning in natural language understanding (NLU), with the focus on bootstrapping Arabic from high-resource English and French languages for domain classification, intent classification, and named entity recognition tasks. We adopt a BERT-based architecture and pretrain three models using open-source Wikipedia data and large-scale commercial datasets: monolingual:Arabic, bilingual:Arabic-English, and trilingual:Arabic-English-French models. Additionally, we use off-the-shelf machine translator to translate internal data from source English language to the target Arabic language, in an effort to enhance transfer learning through translation. We conduct experiments that finetune the three models for NLU tasks and evaluate them on a large internal dataset. Despite the morphological, orthographical, and grammatical differences between Arabic and the source languages, transfer learning performance gains from source languages and through machine translation are achieved on a real-world Arabic test dataset in both a zero-shot setting and in a setting when the models are further finetuned on labeled data from the target language.

## introduction the fast growing interest in conversational aibased voice assistants has increased the importance of finding ways to efficiently and rapidly expand these services to multiple new languages. one of the core components of virtual assistants is natural language understanding (nlu), which is usually composed of three main tasks: domain classification (dc), intent classification (ic), and named entity recognition (ner). nlu tasks are responsible for classifying the domain and intent from the user's utterance and identifying and extracting entities from their requests through slot-filling. training an nlu model to support a new language requires a large amount of labeled utterances, which is costly and time-inefficient, particularly for low-resource languages. in recent years, a lot of success was shown through cross-lingual knowledge transfer on various nlu tasks for zeroshot transfer and few-shot transfer @xcite @xcite @xcite . this is made possible with the availability of multilingual pretrained language models such as mbert @xcite and xlm-r @xcite . however, cross-lingual transfer was shown to be more effective among similar languages (e.g., english to french) as opposed to distant languages (e.g., english to arabic), especially for languages that differ in their script @xcite @xcite . efforts to reduce the distance between source and target languages include transliteration/romanization to latin script @xcite , and machine translation @xcite . although romanization was shown to be beneficial for languages that are not included in pretraining, it degraded performance on languages that are included in these large multilingual models like arabic and japanese @xcite . driven by some of the shortcomings of pretrained multilingual models, several monolingual models have been trained and released in the past couple of years for multiple languages like arabic @xcite @xcite , german @xcite , and french @xcite . whether multi-lingual or monolingual models are adopted, task-specific labeled data is still required for finetuning. in this paper, we experiment with cross-lingual transfer from english and french, two highresource languages with rich nlu labeled datasets for bootstrapping nlu model for the low-resource arabic language, specifically for virtual assistant (va) systems. to this end, we train three bert models on a mix of open-source data and machine translated user inquiries: a monolingual -arabic only, a bilingual arabic-english and a trilingual arabic -english -french models. particulars of arabic language such as orthographic inconsistencies in diacritized script and inflectional affixation are mitigated by preprocessing the data before training. we distill each of the bert models to a smaller student model that better fit memory and latency requirements of commercial va systems. we present experimental results on internally gathered real-world arabic dataset that illustrate crosslingual transfer through nlu knowledge transfer and machine translation (mt). gains from transfer learning (tl) are achieved on the target arabic dataset in both dc and joint ic-ner tasks in a zero-shot setting, few-shot setting, and in a setting with non-production arabic labeled data included in finetuning. the introduction of pretrained multilingual language models like mbert @xcite and xlm-r @xcite has opened the doors for wider exploration of cross-lingual transfer learning @xcite @xcite . @xcite has shown that the reason why some languages do not benefit from these massive multilingual models is largely related to script differences; particularly for languages that have not been seen by mbert. experiments in @xcite show that transliteration to latin script for low-resource languages with different script does improve performance for part-of-speech tagging, dependency parsing, and ner tasks, however not for languages that are included in mbert like arabic and japanese. such findings are also echoed in @xcite . another way to bring distant languages closer is through machine translation. @xcite introduces a step before finetuning on ic-ner task by retraining pretrained multilingual models (mbert and xlm-r) for mt task. authors show that performance gain with the proposed approach is larger between distant languages than that between similar languages. @xcite proposes an integrated translationmonolingual classifier system that exploits crosslingual transfer through setting the translation as a latent variable between the target text and the labels (a translate-test approach). using reinforcement learning, @xcite trains the integrated translation-classifier system with classification accuracy as the reward. this approach however, can be only applied to dc and ic tasks where the whole utterance is labeled with one class. nlu models for arabic: non-deterministic nlu models for arabic have not been extensively explored until recently; largely due to a lack of rich labeled datasets for the various nlu tasks. @xcite proposed arabic specific word2vec embeddings. @xcite utilized pretrained multilingual universal sentence encoder (muse) embedding and trained a bidirectional-gated recurrent neural network with a mix of average and max pooling layer for arabic ner task using the wikifanegold dataset @xcite which classifies entities into eight classes only (person, location, organization, geopolitical, etc.). although mbert includes arabic, cross-lingual transfer did not show performance gains for arabic as it did on indo-european languages @xcite . motivated by the monolingual bert models, @xcite trained arabert, a monolingual bert-based language representation model for arabic language on data that includes arabic wikipedia dumps, in addition to two publicly available large arabic corpora: 5m (el-khair, 2016) and 3.5m @xcite articles, both extracted from arabic news sources. authors in @xcite also introduced a preprocessing step on the data prior to using it for pre-training bert, which used offthe-shelf arabic farasa tokenizer @xcite for subword unit segmentation. building on arabert, arbert @xcite and camelbert @xcite have added additional arabic datasets to pretraining a monolingual bert that cover more topics and dialects. because of the lack of rich labeled arabic dataset, the ner task in @xcite @xcite is limited to classifying nouns into three main classes only (person, location, organization) foot_0 , a much simpler ner task than that needed to power a virtual assistant system, where user requests can span hundreds of entity labels. in this paper, we propose a multilingual nlu model for arabic language, targeted for commercial virtual assistant system. we explore crosslingual transfer through mt and task-specific learning transfer from rich source languages (english and french) to arabic. despite the languages not being closely related, we show that multilingual models outperforms the monolingual model on large-scale arabic traffic for both dc and ic-ner tasks. to our knowledge, this is the first arabic model trained and evaluated for such complex nlu tasks required for virtual assistants which involves classifying 18 domains, 333 intents, and 268 entity labels. ## results and discussion we measure the performance of our models for dc and ic-ner tasks in terms of domain classification error rate (dcer) and semantic error @xcite , where @xmath0, @xmath1, @xmath2. an ic error is counted as a substitution. all models are evaluated on the same testset and performance is reported as a percentage difference (%∆) to the baseline few-shot monolingual model. in addition to the overall, i.e., where all utterances have equal contribution to performance (micro-average), overall, even without mt data, the multilingual pre-prod models beat the monolingual model 14 out of 18 domains on the dc task and in 13 out of 18 domains for ic-ner task, clearly showing the effect of cross-lingual transfer of nlu learning from rich english and french source languages to the low-resource arabic language, despite being linguistically very different. ## conclusion in this paper, we addressed the problem of bootstrapping an nlu model for arabic from two high-resource indo-european languages. we presented two multilingual bert-based models, pretrained and distilled in-house, and compared them to a monolingual arabic baseline model to explore cross-lingual transfer learning. in an effort to tackle the unique challenges in arabic language, we adopted a preprocessing step in which we dediacritize the text to reduce the variance and inconsistencies in the data for an already low-resource language. we also split functional affixes and adopt bpe encoding to deal with inflectional affixation in arabic. furthermore, in order to reduce the distance between the target language and the source languages we used off-the-shelf machine translator to pretrain and finetune the models, in addition to large-scale open-source wikipedia and internal datasets. transfer learning performance gains on the target arabic language showed a reduction of up to 20.76% in semantic error rate for the ic-ner task and 30.21% in classification error for the dc task for the trilingual model in few-shot setting. similar cross-lingual learning gains were achieved in a zero-shot setting and pre-prod setting with the improvement gap between monolingual and multilingual models narrowing as data from mt and the arabic target language is added to finetuning the models.