Weakly and Semi-Supervised Learning for A rabic Text Classification using Monodialectal Language Models (2022)

Abstract: The lack of resources such as annotated datasets and tools for low-resource languages is a significant obstacle to the advancement of Natural Language Processing (NLP) applications targeting users who speak these languages. Although learning techniques such as semi-supervised and weakly supervised learning are effective in text classification cases where annotated data is limited, they are still not widely investigated in many languages due to the sparsity of data altogether, both labeled and unlabeled. In this study, we deploy both weakly, and semi-supervised learning approaches for text classification in low-resource languages and address the underlying limitations that can hinder the effectiveness of these techniques. To that end, we propose a suite of language-agnostic techniques for large-scale data collection, automatic data annotation, and language model training in scenarios where resources are scarce. Specifically, we propose a novel data collection pipeline for under-represented languages, or dialects, that is language and task agnostic and of sufficient size for training a language model capable of achieving competitive results on common NLP tasks, as our experiments show. The models will be shared with the research community.

## introduction in recent years, the emergence of social media platforms allowed the increased use of the informal form of a language in online user-generated content. as a result, more languages are present in online content, introducing a challenge to language processing tools that are developed to improve user experience. this is evident in the discrepancy in the levels of support for many tasks in language technologies for different languages, such as the lack of keyboard support and spell checking extensions for low resource languages, even those with a large online user base @xcite . one approach is to focus on developing multilingual models that are capable of learning languageagnostic representations of data @xcite . another approach uses meta-learning and few-shot learning models to improve results on tasks with small sets of annotated data @xcite . adapting to small sets of data can also be achieved using semi-supervised models where a seed of annotated data is used to bootstrap a supervised model using only a relatively small set of labeled data @xcite . weakly supervised models fall into this class of approaches as well, where primary external knowledge sources are incorporated to provide larger sets of annotated data for the model @xcite . for extremely low-resourced languages, these techniques are difficult to apply due to the lack representative datasets whether labeled or unlabeled @xcite . in this work, we address the challenges facing incorporating learning techniques designed for scenarios where annotated data is scarce. specifically, for arabic dialects, the main challenge is that in data sources where dialectal data in a raw form is abundant, it is rarely distinguished from other arabic dialects, posing a challenge when the goal is to target a specific dialect. to that end, we curate and construct datasets and dictionaries, develop an automatic annotation scheme, develop multiple pre-trained language models (plms) and conduct an empirical study to examine the performance of the text classification task under the learning paradigms of semi, weak and full supervision. although arabic is a widely spoken language, with over 400 million speakers, it still remains a lowresource language, especially in terms of the availability of annotated datasets for emerging nlp tasks (althobaiti, 2020). thus, the approaches proposed in this work, although testing on arabic, are applicable to any similarly low-resourced language. in summary, the contributions of this paper are: ## data collection and annotation for low-resource languages in this section we describe our proposed approaches for large data collection for specific languages and dialects and our automatic annotation approach for large data. ## araroberta this section provides a description of the dialectspecific language models developed using the large datasets we collected. to obtain the arabic roberta (araroberta) models, we train 7 bert-based models using the roberta-base configuration with masked language modeling (mlm) pre-training objective @xcite . it consists of 12 encoder layers/blocks, 768 hidden dimensions, 12 attention heads, and 512 maximum sequence length @xcite . the batch size is 32 with 10 epochs after initial experimentation based on the loss. although initial experimentation is done on the hyperparameter, the adopted values are similar to the literature. the optimization is similar to the adopted bert optimization @xcite , using the adam optimizer (kingma and ba, 2017) with similar parameters. the collected tweets described in section 3.1 from each dialect are utilized for pre-training the corresponding araroberta dialectal language model as shown in also, other contextual baselines are used to compare the performance of araroberta variations against as shown in ## adi models the adi task is formed as a classification task. we adopt three classification models using semi and weak supervision paradigms. in these models, we build on a transformer-based classifier. in this section, we provide an overview of our proposed models. ## experiments and evaluation here we describe the evaluation experiments for fully, semi and weakly supervised learning models for the adi task. the performance is evaluated using the f-1 measure, following existing literature. ## discussion this section provides an analysis for the experimental results and discusses the significant findings. ## conclusion this paper proposed different approaches for arabic dialect text classification as a low-resource scenario and conducted an empirical study to evaluate the performance of the adopted approaches. the paper proposed a novel data collection pipeline from twitter that is language and task agnostic. also, developed dialect-specific contextual language models to learn from unlabeled data that yield effective and stable performance across dialects, as seen in supervised classification. while araroberta models were pretrained on a fraction of the data size that other contextual baselines were trained on, the results showed that most of the supervised araroberta models outperformed these models. in addition, when compared to the traditional ml model, larger araroberta models outperform it as well. additionally, to the best of our knowledge, we constructed the first dialectal dictionary to utilize it in the automatic annotation in scenarios where labeled data are not available and then utilized in a weak-supervised task. although the automatic function contains one hand-crafted rule, this approach is a promising technique for annotating large data and utilizing it in a text classification task. also, the proposed ssl model can be adopted when only a few labeled examples are available where it shows its effectiveness and stability.