Multi-Task Sequence Prediction For T unisian A rabizi Multi-Level Annotation (2020)

Abstract: In this paper we propose a multi-task sequence prediction system, based on recurrent neural networks and used to annotate on multiple levels an Arabizi Tunisian corpus. The annotation performed are text classification, tokenization, PoS tagging and encoding of Tunisian Arabizi into CODA* Arabic orthography. The system is learned to predict all the annotation levels in cascade, starting from Arabizi input. We evaluate the system on the TIGER German corpus, suitably converting data to have a multi-task problem, in order to show the effectiveness of our neural architecture. We show also how we used the system in order to annotate a Tunisian Arabizi corpus, which has been afterwards manually corrected and used to further evaluate sequence models on Tunisian data. Our system is developed for the Fairseq framework, which allows for a fast and easy use for any other sequence prediction problem.

## introduction in the last decade neural networks became the state-of-the-art models in most nlp problems. sequenceto-sequence models @xcite , built on top of recurrent @xcite , convolutional @xcite or attentional @xcite modules, and structured in encoder-decoder architectures, are currently the most effective models for nlp problems. neural networks have been used also for multi-task learning since early in their diffusion @xcite @xcite . as a semitic language, arabic has a highly inflectional and derivational morphology, which makes arabic processing an engaging challenge. this morphological complexity has traditionally been handled through morphological analysers, such as bama @xcite , which has been used by the linguistic data consortium (ldc) to develop the penn arabic treebank (patb) @xcite . recently, the number of nlp contributions to morphological analysis, disambiguation, part-of-speech (pos) tagging and lemmatization has increased substantially, for both modern standard and dialectal arabic (msa and da, respectively). multitask learning was proved to be an effective way to process arabic morphology for msa fine-grained pos tagging @xcite , as well as for da @xcite . concerning nlp applied to da, it is possible to observe two main macro-strategies aimed at remedying the lack of data for da: 1. msa systems adaptation to da processing, like @xcite who exploited the penn arabic treebank (patb) @xcite and used explicit knowledge about the relation between msa and levantine arabic. instead, @xcite built a pos tagger for egyptian through a minimally supervised approach by leveraging the callhome egyptian colloquial arabic corpus (eca). 2. the constitution of new resources not based on msa-da relations, in particular dialectal corpora, such as the fisher levantine arabic conversational telephone speech @xcite . 1 this second strategy has been followed also collecting more ad-hoc resources. @xcite presented the first parallel da corpus, collecting the dialects of 25 arab cities, including the tunisian dialects of tunis and sfax. the madar corpus has been created by translating selected sentences from the basic traveling expression corpus (btec) @xcite . regarding tunisian dialect (td), the resource constitution strategy has been instantiated as msa resource adaptation to the da, e.g. building lexicons @xcite , pos-taggers @xcite , morphological analysers @xcite or morphological systems to disambiguate annotated transcriptions @xcite . considering the lack of freely available resources, we opted for an approach similar to the one used in curras palestinian corpus collection @xcite , which exploits madamira tools @xcite , (cf. section 4.1). the development of informal online communication provided a solution to most of the data availability problems, making accessible to the scientific community a large amount of texts, both written and oral. concerning texts written in da, it is possible to find two main writing systems: arabic and latin scripts. with regard to the second one, letters are used together with digits for the encoding of those arabic letters without correspondence in the roman alphabet. this system is already well known as arabizi, or arabish for non-arabic speakers. most of the work developed on arabish focus on language identification @xcite and sentiment analysis @xcite . several works are focused on the conversion of arabish into arabic script, as the parallel annotated egyptian arabish-arabic script sms/chat corpus @xcite . transliteration has also been addressed for tunisian arabish @xcite @xcite . in this paper we propose a multi-task sequence prediction system based on recurrent neural networks, that we used to annotate at multiple levels the tunisian arabish corpus (tarc) @xcite . the annotation levels include tokenization, part-of-speech (pos) tagging and tunisian arabish encoding into arabic script. the system is learned to predict all the annotation levels in cascade, starting from arabish input. we evaluate the system on the tiger german corpus @xcite in order to show the effectiveness of our neural architecture. while the purpose of this evaluation is not to improve state-of-the-art on this task, our results are comparable and sometimes better than the best published models. we show also how we used the system in order to annotate tarc, which has been afterwards manually corrected and used to further evaluate sequence models on tunisian data. our system is developped for fairseq foot_0 @xcite , it can therefore be used for any problem involving sequence prediction. 3 in the remainder of the paper we describe the tarc corpus, that we annotated with multi-level information, and we used to evaluate our neural system (in section 2). in section 3 we describe our multi-task neural architecture for multi-level annotation, in section 4 we describe the tiger corpus, the experimental settings, and all the results obtained with our system, on both tiger and tarc corpora. we conclude the paper in section 5. ## tunisian arabish multi-level annotated corpus the corpus used in this paper is the tunisian arabish corpus (tarc) @xcite , the result of a multidisciplinary work with a hybrid approach based on: 1. dialectological research questions; 2. corpus linguistics standards and 3. deep learning techniques. tarc has been conceived with the aim to extend the dialectological investigation to the web, not only considering it as a new resource for linguistic analyses, but mainly because the object of tarc is a computer mediated communication (cmc) writing system. the gathering of cmc corpora for linguistic study purposes is a long-standing practice: as early as the 1990s, in order to study linguistic and communicational aspects, researchers began to collect corpora from mailing lists, newsgroups, electronic conferences or chat rooms @xcite @xcite . nowadays, the study of cmcs is a research domain it-self, crossing various disciplines such as sociology and linguistics. the linguistic questions related to cmccorpora may for example concern paraverbal phenomena and the expression of emotions @xcite , politeness formulas and the degree of message formality @xcite , the effects of orality in written communication @xcite , the role of code-mixing and code-switching in mediated discourse @xcite , their graphic and orthographic characteristics @xcite (concerning arabic). lastly, a lot of research deals currently with the automatic processing of such corpora @xcite . among the purposes of dialectology there is the dialect collection and description with traditional approaches: fieldwork, oral text collection and transcription, glossary building. we observed that in the case of arabic varieties the descriptive landscape is made of multiple studies on single phenomena. for this reason, we developed a resource inspired by dialectological investigation, which borrows the principles of corpus linguistics in order to guarantee representativeness, accessibility, balance and authenticity of the linguistic data @xcite . the data gathered in tarc, together with various metadata, takes a snapshot of tunisian arabish writing and its evolution over the last ten years. tarc is built selecting data with the following criteria: 1. text mode: informal writing; 2. text genres: forum, blog, social networks, rap lyrics; 3. domain: cmc; 4. language: tunisian; 5. location; 6. publication date. the last two items were registered via metadata extraction (publication date, user's age, gender and provenience). the building process automation overcomes the observer's paradox problem @xcite , an issue much discussed in dialectology @xcite . it also allows the reproducibility of the work, as well as the quantitative extension of an open corpus (such as tarc), which is normally difficult to ensure by dialectological research. tarc collection has therefore been enhanced thanks to the multi-task architecture, used for a semi-automatic annotation (cf. section 3) to get as close as possible to a consistent linguistic annotation @xcite . the automatically generated annotations were post-edited by a linguist qualified in arabic language and tunisian variety, whose work was occasionally verified by native speakers. 4 such annotation work complies with both the applicative and the analytical purposes of a corpus. the former concerns the generation of nlp tools for the tunisian arabish processing. the latter is realised through the multi-functional annotation levels of tarc, which allow global and systematic studies of tunisian variety and its arabish encoding. this way, tarc usefulness returns to the dialectological area, the field in which the preliminary research questions were addressed. tarc has been annotated with four information levels. 1) classification of words in three classes: arabizi, foreign and emotag. the first class is for tunisian and msa words, the second one is to classify non-arabic code-mixing; the third is used for elements as smileys or emoticons. 2) encoding in arabic script in conventional orthography for dialectal arabic (coda*) @xcite . 3) tokenization, tunisian words encoded in coda* have been tokenized following the d3 bwform configuration scheme where basically all clitics are tokenized, including the article @xcite . 4) part-of-speech according to the patb guidelines @xcite . all levels have been developed following the same incremental and semi-automatic procedure described in @xcite for the codafying stage. ## multi-task sequence prediction system there are several works about multi-task learning with neural networks for nlp problems @xcite , inter alia. most of the time the neural architecture factorises some parameters for information that can be shared among tasks, and then uses different modules (e.g. decoders) for each task, which are learned independently. as described in section 2, our goal for tunisian arabish data is a multi-level annotation scheme, where the different levels are potentially related. from an nlp point of view, this relations imply that some levels of annotation may help disambiguation when annotating other levels. for instance the classification information can disambiguate annotation into coda*, tokenization and pos tagging. intuitively we expected that learning tasks in chain, organised in a cascade manner in a neural network, would benefit to each other, in contrast to learning tasks individually. we follow the intuition above and we propose a multi-task neural architecture where the different learned tasks are organised in a cascade. the input is the arabish text. the outputs, corresponding to the tasks to be learned, are, in this order, the classification information, the conversion into coda* orthography, the tokenization of the codafied tokens and the pos tags. outputs from previous tasks are reused by the following tasks, they are thus learned jointly and interdependently. the input is transformed into hidden context-aware representations with an encoder based on recurrent layers. the outputs are processed by different decoders, each of them taking as input the hidden state of the encoder, and the hidden state of each of the previous decoders. the output of each decoder is used to learn each task. more formally, let the task i be represented by the model m i (x, h i ), with x the input (arabish text representations), and h i the list of hidden states from the previous models, plus the current model's hidden state h i . each model i generates an output ôi and a hidden state h i . ôi is the predicted output, which is used to learn the task i by computing a loss l i (o i , ôi ) comparing ôi to the expected output o i . internally the global model m is made of an encoder and i decoders decoder i , with @xmath0. . . i. the list h i includes both the encoder hidden state h e and the decoders hidden states h 1 ...h i . an high-level schema of this architecture, with the flow of information for three tasks (@xmath1, is presented in figure 1 . all the tasks are learned jointly by minimising the global loss @xmath2, ôi ), on top of the circled + in the schema (figure 1 ). like in the original sequence-to-sequence model based on an attention mechanism @xcite , each decoder attends to encoder and decoder's hidden state information with an attention mechanism. the decoder decoder i has therefore i different attention mechanisms, one for attending encoder's information, and one for each previous decoder's hidden state. the queries for the attention mechanisms are always the decoder i 's hidden states, while keys and values are the encoder and previous decoder hidden states. the attention vectors computed by the attention mechanisms are simply summed together to generate the final state, used to predict the next output. in order to evaluate our multi-task system, we used two different corpora. one is tarc, described in section 2, the other is the german tiger corpus @xcite . tarc corpus has been initially collected from forums, social media and blogs, for a total of 32 062 words, and recently extended to 43 313 words by adding the text type of rap lyrics. in order to better organise the automatic annotation and the manual-correction stages, we split the initial corpus into blocks of roughly 6 500 tokens. statistics of tarc are presented in the german corpus tiger @xcite is annotated with rich morpho-syntactic information. these include pos tags, but also gender, number, cases, and other inflection information, as well as conjugation information for verbs. the combination of all these components constitutes the output labels. we used the same data split used in @xcite . statistics of this corpus are given in ## conclusions we presented a multi-task sequence labeling system based on recurrent neural networks, developed for the fairseq framework and used to annotate tarc on multiple levels. the annotation levels provided are: classification, tokenization, pos tagging and encoding of tunisian arabish into arabic script, according to coda*. we described the annotation procedure, after showing the effectiveness of our neural architecture with an evaluation on the tiger german corpus. as a next stage we plan to expand tarc quantitatively to improve the results and its usability in linguistics and nlp fields. future work includes qualitative extension through the addition of further annotation levels, such as lemmatization.