Multilingual Dependency Parsing for Low-Resource A frican Languages: Case Studies on B ambara, W olof, and Y oruba (2021)

Abstract: This paper describes a methodology for syntactic knowledge transfer between high-resource languages to extremely low-resource languages. The methodology consists in leveraging multilingual BERT self-attention model pretrained on large datasets to develop a multilingual multi-task model that can predict Universal Dependencies annotations for three African low-resource languages. The UD annotations include universal part-of-speech, morphological features, lemmas, and dependency trees. In our experiments, we used multilingual word embeddings and a total of 11 Universal Dependencies treebanks drawn from three high-resource languages (English, French, Norwegian) and three low-resource languages (Bambara, Wolof and Yoruba). We developed various models to test specific language combinations involving contemporary contact languages or genetically related languages. The results of the experiments show that multilingual models that involve high-resource languages and low-resource languages with contemporary contact between each other can provide better results than combinations that only include unrelated languages. As far genetic relationships are concerned, we could not draw any conclusion regarding the impact of language combinations involving the selected low-resource languages, namely Wolof and Yoruba.

## introduction treebanks constitute valuable resources for many natural language processing (nlp) applications. they can be used as training and testing data for a wide range of nlp algorithms as well as to induce robust parsing models @xcite . unfortunately, developing treebanks in form of large annotated data used to be a very time-and resource-consuming task. as a consequence, annotated data (in particular the type required for parsing) is lacking for most languages, especially for low-resource languages. to help speed up the treebank development process, various supervised learning techniques @xcite @xcite have been developed in recent past. the supervised monolingual approach based on syntactically annotated corpora has long been the most common approach to parsing. however, thanks to recent developments involving feature representation methods and neural network models, the idea of combining treebanks for multilingual ud parsing has become a more common approach. multilingual modeling constitute a very attractive approach to circumvent the low-resource limitation, as it allows one to create models that can parse the language's text quite accurately in the absence of annotated data for the given language. this occurs through syntactic knowledge transfer across multiple languages. the multilingual approach has yielded encouraging results for both low-resource @xcite as well as for high-resource @xcite languages. the idea of combining treebanks for transfer learning was first introduced in @xcite , which train bilingual parsers on pairs of ud treebanks, showing similar improvements. subsequently, in the conll 2018 shared task, @xcite presented the uppsala system, which follows the same idea. that system combines treebanks of one language or closely related languages together over 82 treebanks and parses all ud annotations in a multi-task pipeline architecture for a total of 34 models. this approach provides two main advantages. first, it reduces the number of models required to parse each language. second, it can provide results that are no worse than training on each treebank individually, and in especially low-resource cases, significantly improved. in the same spirit, @xcite con-ducted a multilingual multi-task parsing study for 124 universal dependencies @xcite treebanks across 75 languages, and demonstrated that a multilingual model can yield better results than monolingual models for different languages. in this paper, we use the approach described by @xcite to produce a crosslingual transfer model that can predict ud annotations for three extremely low-resource african languages by using knowledge from medium-to high-resource european languages. the ud annotations include universal part-of-speech (upos), morphological features (feats), lemmas (lem), and dependency trees (deps). the structure of the paper is as follows. section 2 first provides a brief description of the lowresource languages used as case studies in this research work. section 3 provides an overview of our approach, and section 4 details the neural networkbased parsing model. section 5 describes a series of experiments conducted on high-resource and low-resource languages to verify our assumptions. section 6 presents an analysis of our results. section 7 concludes the discussion. ## languages used as our case studies the low-resource languages selected for this study are bambara, wolof and yoruba. bambara is spoken in mali, ivory coast, upper guinea, in the western part of burkina faso and in eastern senegal. wolof is spoken in senegal, in the gambia and in mauritania. yoruba is spoken in west africa, most prominently in southwestern nigeria. these west african languages belong to two different subgroups of the larger niger-congo family of languages. bambara is part of the mande subgroup, while wolof and yoruba are atlantic-congo languages. while the ultimate genetic unity of atlantic-congo languages is widely accepted, the internal cladistic structure is not well established @xcite , especially with respect to the connection of the mande languages, which has never been demonstrated. for instance, the mande languages lack the noun-class morphology that is the primary identifying feature of the atlantic-congo languages. wolof and yoruba are genetically related to each other, but not closely related to bambara. interestingly, while wolof does not have much language contact with yoruba (if any), it actually may share areal features with bambara, since their common geographic location al-lowed for a long history of contact between these two languages. bambara is highly isolating and has a very strict word order: subject aux / tam (tense-aspectmood markers) object verb @xcite . it is a tone language, with two tones: high and low. wolof is an agglutinative language with an svo and head-modifier basic word order @xcite . unlike many other languages of the niger-congo family, wolof is not a tonal language. yoruba is a highly isolating language and the sentence structure follows subject verb object @xcite . in addition, yoruba is a tonal language with three tones: low, middle (optional) and high. the three low-resource languages are fairly well documented. for bambara, there exist hundreds of linguistic papers and few recent reference grammars published about that language @xcite . there are also some dictionaries available, including the bamadaba online dictionary foot_0 and a 15k print dictionary @xcite . likewise, wolof has several descriptive grammars and few dictionaries, e.g. the french-wolof print dictionaries @xcite and an online wolof dictionary. foot_1 similarly, for yoruba, there are many literary texts, newspapers, religious kinds of literature, and some blogs in the language. there are also academic papers, print dictionaries, e.g. the yoruba-english dictionary by @xcite , and online dictionary foot_2 published in the language. although these languages are well-documented, until very recently, they did not or still do not really have a universal dependency corpus. bambara has a 12k tokens ud treebank @xcite . yoruba has a 8k tokens ud treebank. 4 for these two languages, only test set data are available (no training data), making them good candidates for zero-shot learning. wolof has a 44k tokens ud treebank (dione, 2019) that consists of a training, a development and a test set. as these numbers show, the sizes of these ud treebanks are extremely small. this alone does not make them low-resource languages, but they are poorly equipped with regard to nlp tools as well. for instance, while yoruba and bambara are documented with huge written corpora, 5 these are mostly not achieved for research and nlp purposes. for wolof, resources and tools have only very recently begun to emerge, including a finite-state morphological analyzer (dione, 2012), a small treebank (dione, 2014) and computational grammar/parser (dione, 2020) based on the lexical-functional grammar (lfg) framework @xcite . we chose to focus on three languages due to the availability of ud treebanks for these languages, even though for two of these languages only test data are available. ## approach our approach consists in developing several multilingual parsing models using different language combinations of medium-to high-resource languages (english, french, norwegian) and lowresource languages (bambara, wolof, yoruba) that have had some contemporary language contact. the languages used for training the models have been selected with the assumption that contemporary contact languages, at least in certain scenarios, share (structural) similarities with the low-resource languages in question. english has a long history of contact with yoruba, leading to a variety of morphosyntactic changes and lexical borrowings in the latter language (ogundepo, 2015). our expectation is that the match rate between english and yoruba should be somewhat high. likewise, we expect to see similar patterns between french and the bambara and wolof languages with which it has had a long contact. for instance, through french influence there exists two varieties of wolof: urban wolof, used especially in cities, and kajoor wolof (also referred to as 'pure' wolof), which is spoken mostly in rural areas @xcite . in addition, we include norwegian as a control language with no direct contact or genetic relation with any of the selected low-resource languages. recents studies, including @xcite conducted similar experiments to explore the impact of using contemporary contact languages or genetically related languages (e.g. finnish) in multilingual parsing scenarios involving low-resource languages (e.g. north saami and komi-zyrian). their findings showed that specific language combinations of contemporary contact languages or genetically related languages may enable improved dependency parsing. the basic idea of graph-based dependency parsing is to produce a dependency tree in form of a directed graph with some constraints by first generating all possible candidate dependency graphs for a given sentence. subsequently, each tree is scored and the parser picks the one with the highest score. during training, the parser induces a model for scoring an entire dependency graph for a sentence. during parsing, it finds the highest scoring dependency graph, given the induced model. more recently, graph-based approaches have shown to outperform transition-based approaches when it comes to ud-type corpora, notably with the neural graph-based parser of @xcite , who won the conll 2017 ud shared task by a wide margin. in this study, we chose udify @xcite , a neural model which uses the graphbased biaffine attention parser developed by @xcite ; @xcite . udify is a single multitask model that produces ud annotations (upos, feats, lem, deps) jointly. in a first step, udify generates contextual embeddings for any input sentence by using the cased foot_5 pretrained multilingual bert network @xcite , a self-attention @xcite network of 12 layers, 12 attention heads per layer, and hidden dimensions of 768. the bert model was trained by predicting randomly masked input words on the entirety of the top 104 languages with the largest wikipedias, including two african languages: swahili and yoruba. bert segments texts into (unnormalized) sub-word units using the wordpiece tokenizer @xcite . in a second step, the udify model integrates task-specific layer-wise attention similar to @xcite . finally, each ud task is decoded simultaneously using softmax classifiers. during training, various regularization techniques are applied to the bert network, including input masking, increased dropout, weight freezing, discriminative fine-tuning, and layer dropout. ## experiments we conducted a series of experiments on bambara, wolof and yoruba. for these languages, we tested different language combinations for the cross-lingual model. the dataset used in our experiments are provided in for multilingual training with udify, the 11 ud treebanks are concatenated into a single treebank, similar to @xcite ; @xcite . this single treebank consists of a training, a development and a test set. for each epoch, sample input sentences were drawn randomly from the training data and fed to the neural network in form of mixed batches, i.e. each batch may contain sentences from any language or treebank. the sentences are shuffled and bundled into batches of 8 sentences each. we employ a base learning rate of 1e -3 that is kept constant until we unfreeze bert in the second epoch. we then linearly warm up the learning rate for the next 1,000 batches. next, we apply inverse square root learning rate decay for the remaining epochs. following @xcite , training was done for a total of 80 epochs (ca. 3 days) on a single gpu (rtx 2080). the hyperparameters used for our model are given in ## results and analysis for comparison, we show in the experiments reported by @xcite did not include wolof, since no ud treebank was available for that language at that time. for this purpose, we trained a customized monolingual udify model on the wolof training data and applied that model on the wolof test set. the results of this monolingual training are shown in for multilingual dependency parsing, we run several experiments in which we keep the settings described in section 5, excluding only one language at a time. in an initial experiment, we used all the treebanks presented in section 5 for which training data are available. this consists of a total of 9 out of the 11 ud treebanks. 8 then, the created multilingual model has been used to parse the test data of the selected low-resource languages. the results are given in and 3.19% in terms of uas and las, respectively. this provides a good indication that a transfer learning approach in a multilingual dependency parsing context seems to have positive impact (at least for wolof), outperforming the monolingual model. in the same way, we run a similar experiment where we exclude only the french treebanks to assess their impact on the overal results for the selected low-resource african languages. the results of this experiment are shown in as mentioned above, we used norwegian as a control language to verify our assumption with re- ## conclusion in this paper, we have presented a multilingual approach to parsing that is effective for languages with few resources and no syntactically annotated corpora available for training. we have shown that specific language combinations involving contemporary contact languages can provide better results than combinations that only include unrelated languages. we should note, however, that for wolof and yoruba, which are supposed to be genetically related languages, we rather observed a decrease of parsing results from the yoruba side when using the wolof training data. it remains a question for further study whether this decrease observed here are actually attributable to a lack of real genetic relationship between the language or to the lack of (large) training data for yoruba.