Building a User-Generated Content N orth- A frican A rabizi Treebank: Tackling Hell (2020)

Abstract: We introduce the first treebank for a romanized user-generated content variety of Algerian, a North-African Arabic dialect known for its frequent usage of code-switching. Made of 1500 sentences, fully annotated in morpho-syntax and Universal Dependency syntax, with full translation at both the word and the sentence levels, this treebank is made freely available. It is supplemented with 50k unlabeled sentences collected from Common Crawl and web-crawled data using intensive data-mining techniques. Preliminary experiments demonstrate its usefulness for POS tagging and dependency parsing. We believe that what we present in this paper is useful beyond the low-resource language community. This is the first time that enough unlabeled and annotated data is provided for an emerging user-generated content dialectal language with rich morphology and code switching, making it an challenging test-bed for most recent NLP approaches.

## introduction until the rise of fully unsupervised techniques that would free our field from its addiction to annotated data, the question of building useful data sets for under-resourced languages at a reasonable cost is still crucial. whether the lack of labeled data originates from being a minority language status, its almost oral-only nature or simply its programmed political disappearance, geopolitical events are a factor highlighting a language deficiency in terms of natural language processing resources that can have an important societal impact. events such as the haïti crisis in 2010 @xcite and the current algerian revolts @xcite foot_0 are massively reflected on social media, yet often in languages or dialects that are poorly re-sourced, namely haitian creole and algerian dialectal arabic in these cases. no readily available parsing and machine translations systems are available for such languages. taking as an example the arabic dialects spoken in north-africa, mostly from morocco to tunisia, sometimes called maghribi, sometimes darija, these idioms notoriously contain various degrees of code-switching with languages of former colonial powers such as french, spanish, and, to a much lesser extent, italian, depending on the area of usage @xcite @xcite . they share modern standard arabic (msa) as their matrix language @xcite , and of course present a rich morphology. in conjunction with the resource scarcity issue, the codeswitching variability displayed by these languages challenges most standard nlp pipelines, if not all. what makes these dialects especially interesting is their widespread use in user-generated content found on social media platforms, where they are generally written using a romanized version of the arabic script, called arabizi, which is neither standardized nor formalized. the absence of standardization for this script adds another layer of variation in addition to well-known user generated content idiosyncrasies, making the processing of this kind of text an even more challenging task. in this work, we present a new data set of about 1500 sentences randomly sampled from the romanized algerian dialectal arabic corpus of @xcite and from a small corpus of lyrics coming from algerian dialectal arabic hip-hop and raï music genre that had the advantage of having already available translations and of being representative of algerian vernacular urban youth language. we manually annotated this data set with morpho-syntactic information (partsof-speech and morphological features), together with glosses and code-switching labels at the word level, as well as sentence-level translations. furthermore, we added an additional manual annotation layer following the universal dependencies annotation scheme @xcite , making of this corpus, to the best of our knowledge, the first user-generated content treebank in romanized dialectal arabic. this treebank contains 36% of french tokens, making it a valuable resource to measure and study the impact of code-switching on nlp tools. we supplement this annotated corpus with about 50k unlabeled sentences extracted from both common crawl and additional web crawled data, making of this data set an important milestone in north-african dialectal arabic nlp. this corpus is made freely available under a creative commons license. 2 ## the language as stated by @xcite , arabic languages are often classified into three categories : (i) classical arabic, as found in the qur'an and related canonical texts, (ii) modern standard arabic, the official language of the vast majority of arabic speaking countries and (iii) dialectal arabic, whose instances exhibit so much variations that they are not mutually understandable across geographically distant regions. as space is missing for an exhaustive description of arabic language variations, we refer the reader to @xcite , @xcite and especially to @xcite for a thorough account of algerian dialectal arabic, which is the focus of this work. in short, the key properties of north-african dialectal arabic are: the following list highlights some of the main properties of arabizi compared to msa written in the arabic script. ## extending our data set with noisy unlabeled data the need for more data has never been more striking as they are needed for important tasks such as handling lexical sparseness issues via word embeddings, lexicon acquisition, domain adaptation via self-training, or fine-tuning pre-trained language models, its modern incarnation. the trouble with narabizi is that it is a spoken language whose presence can be mostly found in informal texts such as social media. more importantly, the arabizi transliteration process is also used by other arabic dialects, making the data collection a needle in a haystack search task. we therefore present in this section the process we used to mine an additional ## pre-annotation tool development via noisy transliteration of an arabic ud treebank in order to speed up the annotation process of our data, we decided to create a pre-annotation morphosyntactic and syntactic annotator trained on quasi-synthetic data obtained by "transliterating" a pre-existing arabic (msa) treebank, the prague arabic dependency treebank (padt), into the narabizi latin script, together with data from the french gsd ud treebank. both are taken from the ud treebank collection @xcite . before it can be used as training data, the padt needs to first be transformed into a form similar to narabizi. since the padt corpus is a collection of msa sentences with no diacritics, it is impossible to directly "transliterate" into narabizi. we first diacritized it, in order to add short-vowel information, and then "translitterated" it into an arabizi-like corpus. we describe this process in this section. the results of the pseudo-narabizi parser trained on the "translitterated" corpus are then presented in section 6.2. ## usability in this section we describe preliminary experiments on part-of-speech tagging and statistical dependency parsing that show promising results while highlighting the expected difficulty of processing a low-resource language with a high level of code-switching and multiple sources of variability. ## discussion how interleaved are french and narabizi? as stated before, narabizi takes its root in classical arabic and in multiple sources of integration of french, msa and berber, the amazigh language. as the narabizi treebank contains more than 36% of french words, it is of interest to use recent methods of visualization to see how interleaved it is (a) @xmath0: word embbeddings de 300 mots (100 arabe translittéré, 100 français, 100 arabizi) calculés avec l'algorithme fasttext (jaune : français, bleu : arabe, rouge : arabizi) 29 (a) dim: 50 (b) dim: 100 figure 2: two-dimensional representation of fasttext word embeddings for 300 words (100 for transliterated msa -blue -, french -yellow -and narabizi -red -) after pca analysis with some of its source languages. to this end, we extract words embeddings using fasttext @xcite from a corpus made of the "translitterated" padt described in section 5, the french ud gsd and narabizi original corpus @xcite . two-dimensional representations of the resulting embeddings space for 300 selected words are shown in figure 2 for embeddings of size 50 and 100. we notice that the overall shapes of both representations are very similar, apart from a non significant x-axis reversal. on the first components, increasing the embedding size does not provide more information. we also see that french and transliterated arabic words are clearly separated into two clusters of low standard deviation, while narabizi words are very spread out. some fall within the french cluster, they correspond to french words present in this algerian dialect. others are in the middle of the arabic cluster, these are the purely arabic words of the dialect. between the two, there are amazigh words (rak, mech), arabized french words (@xmath1, arabic words whose berber pronunciation has resulted in an unexpected narabizi rendering (nta instead of expected enta 'you', mchit instead of expected machayt 'to go-2sing'). ## related work research on arabic dialects is quite extensive. space is lacking to describe it exhaustively. in relation to our work regarding north-african dialect, we refer to the work of @xcite who along his phd covered an large range of topics regarding the dialect spoken specifically in morocco and generally regarding language identification @xcite in code-switching scenario for various arabic dialects @xcite . unlike narabizi dialects, the resource situation for arabic dialects in canonical written form can hardly be qualified as scarce given the amount of resources produced by the linguistic data consortium regarding these languages, see @xcite for details on those corpora. these data have been extensively covered in various nlp aspects by the former members of the columbia arabic nlp team, among which mona diab, nizar habash, and owen rambow, in their respective subsequent lines of works. many small to medium scale linguistics resources, such as morphological lexicons or bilingual dictionaries have been produced @xcite . recently, in addition to the release of a small-range parallel corpus for some arabic dialects @xcite , a larger corpus collection was released, covering 25 city dialects in the travel domain @xcite . regarding the specific nlp modeling challenges of processing arabic-based languages, as part of the morphologically-rich languages, recent advances in joint models have been addressed by @xcite that recently efficiently adapted a neural architecture to perform joint word segmentation, lemmatization, morphological analysis and pos tagging on an arabic dialect. recent works on cross-language learning using the whole massively multilingual pre-trained language models artillery have started to emerge @xcite . if successful, such models could help to alleviate the resource scarcity issue that plagues low-resources languages in the more-than-ever data hungry modern nlp. ## conclusion we introduced the first treebank for an arabic dialect spoken in north-africa and written in romanized form, narabizi. more over, being made of user-generated content, this treebank covers a large variety of language variation among native speakers and displays a high level of codeswitching. annotated with 4 standard morphosyntactic layers, two of them following the universal dependency annotation scheme, and provided with translation to french as well as glosses and word language identification, we believe that this corpus will be useful for the community at large, both for linguistic purposes and as training data for resource-scarce nlp in a high-variability scenario. in addition to the annotated data, we provide around 1 million tokens (over 46k sentences) of unlabeled narabizi content, resulting in the largest dataset available for this dialect. our corpora are freely available 14 under the cc-by-sa license and the narabizi treebank is also released as part of the universal dependencies project.