Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on A frican Languages (2020)

Abstract: Multilingual transformer models like mBERT and XLM-RoBERTa have obtained great improvements for many NLP tasks on a variety of languages. However, recent works also showed that results from high-resource languages could not be easily transferred to realistic, low-resource scenarios. In this work, we study trends in performance for different amounts of available resources for the three African languages Hausa, isiXhosa and on both NER and topic classification. We show that in combination with transfer learning or distant supervision, these models can achieve with as little as 10 or 100 labeled sentences the same performance as baselines with much more supervised training data. However, we also find settings where this does not hold. Our discussions and additional experiments on assumptions such as time and hardware restrictions highlight challenges and opportunities in low-resource learning.

## introduction deep learning techniques, including contextualized word embeddings based on transformers and pretrained on language modelling, have resulted in considerable improvements for many nlp tasks. however, they often require large amounts of labeled training data, and there is also growing evidence that transferring approaches from high to low-resource settings is not straightforward. in @xcite , rule-based or linguistically motivated crfs still outperform rnn-based methods on several tasks for south african languages. for pretraining approaches where labeled data exists in a high-resource language, and the information is transferred to a lowresource language, @xcite find a significant gap between performance on english and the cross-lingually transferred models. in a recent study, @xcite find that the transfer for multilingual transformer models is less effective for resource-lean settings and distant languages. a popular technique to obtain labeled data quickly and cheaply is distant and weak supervision. @xcite recently inspected pos classifiers trained on weak supervision. they found that in contrast to scenarios with simulated lowresource settings of high-resource languages, in truly low-resource settings this is still a difficult problem. these findings also highlight the importance of aiming for realistic experiments when studying low-resource scenarios. in this work, we analyse multilingual transformer models, namely mbert @xcite and xlm-roberta @xcite . we evaluate both sequence and token classification tasks in the form of news title topic classification and named entity recognition (ner). a variety of approaches have been proposed to improve performance in low-resource settings. in this work, we study (i) transfer learning from a high-resource language and (ii) distant supervision. we selected these as they are two of the most popular techniques in the recent literature and are rather independent of a specific model architecture. both need auxiliary data. for transfer learning, this is labeled data in a high-resource language, and for distant supervision, this is expert insight and a mechanism to (semi-)automatically generate labels. we see them, therefore, as orthogonal and depending on the scenario and the data availability, either one or the other approach might be applicable. our study is performed on three, linguistically different african languages: hausa, isixhosa and yorùbá. these represent languages with millions of users and active use of digital infrastructure, but with only very limited support for nlp technologies. for this aim, we also collected three new datasets that are made publicly available alongside the code and additional material. 1 we show both challenges and opportunities when working with multilingual transformer models evaluating trends for different levels of resource scarcity. the paper is structured into the following questions we are interested in: in this work, we evaluate on three african languages, namely hausa, isixhosa and yorùbá. hausa is from the afro-asiatic family while isixhosa and yorùbá belong to different branches of the large niger-congo family. hausa and yorùbá are the second and third most spoken languages in africa, and isixhosa is recognized as one of the official languages in south africa and zimbabwe. yorùbá has been part of the unlabeled training data for the mbert multilingual, contextual word embeddings. texts in hausa and isixhosa have been part of the xlm-roberta training. the three languages have few or no labeled datasets online for popular nlp tasks like named entity recognition (ner) and topic classification. we use the ner dataset by @xcite for isixhosa and the one by @xcite for yorùbá. we collected and manually annotated a ner dataset for hausa and news title topic classification datasets for hausa and yorùbá. ## experimental settings to evaluate different amounts of resourceavailability, we use subsets of the training data with increasing sizes from ten to the maximally available number of sentences. all the models are trained on their corresponding language-model pretraining. except if specified otherwise, the models are not fine-tuned on any other task-specific, labeled data from other languages. we report mean ## comparing to rnns loubser and puttkammer (2020a) showed that models with comparatively few parameters, like crfs, can still outperform more complex, neural rnns models for several task and low-resource language combinations. this motivates the question whether model complexity is an issue for these low-resource nlp models. we compare to simple gru based @xcite models as well as the popular (non-transformer) combination of lstm-cnn-crf (ma and hovy, 2016) for ner and to the rcnn architecture @xcite for topic classification. for these models, we use pretrained, non-contextual word embeddings trained for the specific language. figures 1a+b show that an increase in model complexity is not an issue in these experiments. for hausa and yorùbá and for the low resource settings for isixhosa, bert and xlm-roberta actually outperform the other baselines, possibly due to the larger amounts of background knowledge through the language model pretraining. for larger amounts of task-specific training data, the lstm-cnn-crf and the transformer models perform similarly. one should note that for isixhosa, the linguistically motivated crf (eiselen, 2016) still outperforms all approaches on the full dataset. ## transfer learning the mbert and xlm-roberta models are trained with tasks that can be obtained from unlabeled text, like masked language modelling. addi- for ner, we pre-train on the english conll03 ner dataset @xcite . for topic classification, the models are pre-trained on the english ag news corpus @xcite . the texts in the high-resource english and the low-resource hausa and yorùbá target datasets share the same domain (news texts). one issue that is visible in these experiments is the discrepancy between classes. while some classes like "politics" are shared, the topic classification datasets also have language-and location-specific classes like "nigeria" and "africa" which are not part of the english fine-tuning dataset. in our experiments, we use the intersection of labels for ner (excluding date and misc for hausa and yorùbá) and the union of labels for topic classification. the results in figure 1c and in the appendix confirm the benefits of fine-tuning on high-resource languages already shown in past research. they show, however, also the large gains in performance that can be obtained by training on a minimal number of target instances. while the zero-shot setting in @xcite is interesting from a methodological perspective, using a small training set for the target language seems much more beneficial for a practical application. in our experiments, we get -with only ten labeled sentences -an improvement of at least 10 points in the f1-score for a shared label set on ner. for topic classification (figure 1b ) the transfer learning is not beneficial, which might be due to the mismatch in the label sets. ## distant supervision distant and weak supervision are popular techniques when labeled data is lacking. it allows a domain expert to insert their knowledge without having to label every instance manually. the expert can, e.g. create a set of rules that are then used to label the data automatically @xcite or information from an external knowledge source can be used @xcite . this kind of (semi-) automatic supervision tends to contain more errors which can hurt the performance of classifiers (see e.g. @xcite ). to avoid this, it can be combined with label noise handling techniques. this pipeline has been shown to be effective for several nlp tasks @xcite @xcite @xcite , however, mostly for rnn based approaches. as we have seen in section 4 that these have a lower baseline performance, we are interested in whether distant supervision is still useful for the better performing transformer models. several of the past works evaluated their approach only on high-resource languages or simulated low-resource scenarios. we are, therefore, also interested in how the distant supervision performs for the actual resource-lean african languages we study. to create the distant supervision, native speakers with a background in nlp were asked to write labeling rules. for the ner labels per, org and loc, we match the tokens against lists of entity names. these were extracted from the corresponding categories from wikidata. for the date label, the insight is used that date expressions are usually preceded by date keywords in yorùbá, as reported by @xcite . we find similar patterns in hausa like "ranar"(day), "watan" (month), and "shekarar"(year). for example, "18th of may, 2019" in hausa translates to "ranar 18 ga watan mayu, shekarar 2019". the annotation rules are based on these keywords and further heuristics. directly applying this distant supervision on the ner test sets results in an f1-score of 54% and 62% on hausa and yorùbá, respectively. for the topic classification task, the distant supervision rules are based on a dictionary of words relating to each of the classes. to induce the dictionaries, we collected terms related to different classes from web sources. for example, for the "sport" label, names of sportspeople and sport-related organizations were collected and similarly for the "africa" label, names of countries, their capitals and major cities and their politicians. to label a news headline, the intersection between each classdictionary and the text was computed, and a class was selected with a majority voting scheme. we obtain an f1-score of 49% and 55% on the hausa and yorùbá test set respectively when applying the distant supervision directly to the topic classification test sets. additional details on the distant supervision are given in the appendix. for label noise handling we use the confusion matrix approach for ner by @xcite , marked as cm in the plots. additionally, we propose to combine it with the smoothing concept by @xcite . the figures 2a and in the appendix show that when only a small amount of manually labeled data is available, distant supervision can be a helpful addition. e.g. for the ner task in yorùbá, combining distant supervision and noise handling with 100 labeled sentences achieves similar performance to using 400 manually labeled sentences. for label noise handling, combining the confusion matrix with the smoothing approach might be beneficial because the estimated confusion matrix is flawed when only small amounts of labeled data are given. when more manually labeled data is available, the noisy annotations lose their benefit and can become harmful to performance. ## questioning assumptions in this section, we want to discuss certain assumptions taken by us and previous work in lowresource learning to see if these hold and what challenges and opportunities they could bring for future work. to avoid this multi-language set-up and the need to obtain labeled data for multiple languages, we suggest using instead a development set downsized by the same factor as the training data. this approach keeps the ratio between training and development set giving the development set a reasonable size to obtain in a low-resource setting. for the setting with ten labeled sentences for training, the same amount was used for the dev set. the results in figure 2b and in the appendix show that this has only a small effect on the training performance. ## conclusions in this work, we evaluated transfer learning and distant supervision on multilingual transformer models, studying realistic low-resource settings for african languages. we show that even with a small amount of labeled data, reasonable performance can be achieved. we hope that our new datasets and our reflections on assumptions in low-resource settings help to foster future research in this area. melinda loubser and martin j. puttkammer. 2020a. viability of neural networks for core technologies for resource-scarce languages. information, 11:41. melinda loubser and martin j. puttkammer. 2020b. viability of neural networks for core technologies for resource-scarce languages. information, 11:41. bingfeng luo, yansong feng, zheng wang, zhanxing zhu, songfang huang, rui yan, and dongyan zhao. 2017. learning with noise: enhance distantly supervised relation extraction with dynamic transition matrix. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 430-439, vancouver, canada. association for computational linguistics. xianbin lv, dongxian wu, and shu-tao xia. 2020. matrix smoothing: a regularization for dnn with transition matrix under noisy labels. corr, abs/2003.11904. xuezhe ma and eduard hovy. 2016. end-to-end sequence labeling via bi-directional lstm-cnns-crf. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1064-1074, berlin, germany. association for computational linguistics. stephen mayhew, snigdha chaturvedi, chen-tse tsai, and dan roth. 2019. named entity recognition with partially annotated training data. in proceedings of the 23rd conference on computational natural language learning (conll), pages 645-655, hong kong, china. association for computational linguistics. joakim nivre, marie-catherine de marneffe, filip ginter, jan hajič, christopher d. manning, sampo pyysalo, sebastian schuster, francis tyers, and daniel zeman. 2020. universal dependencies v2: an evergrowing multilingual treebank collection. xiaoman pan, boliang zhang, jonathan may, joel nothman, kevin knight, and heng ji. 2017. crosslingual name tagging and linking for 282 languages. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1946-1958, vancouver, canada. association for computational linguistics. debjit paul, mittul singh, michael a. hedderich, and dietrich klakow. 2019. handling noisy labels for robustly learning from self-training data for lowresource sequence labeling. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: student research workshop, pages 29-34, minneapolis, minnesota. association for computational linguistics. telmo pires, eva schlinger, and dan garrette. 2019. how multilingual is multilingual bert? in proceedings of the 57th annual meeting of the association for computational linguistics, pages 4996-5001, florence, italy. association for computational linguistics. alexander ratner, stephen h. bach, henry r. ehrenberg, jason a. fries, sen wu, and christopher ré. 2020. snorkel: rapid training data creation with weak supervision. vldb j., 29(2):709-730. shruti rijhwani, shuyan zhou, graham neubig, and jaime carbonell. 2020. soft gazetteers for lowresource named entity recognition. victor sanh, lysandre debut, julien chaumond, and thomas wolf. 2019. distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. stephanie strassel and jennifer tracey. 2016. lorelei language packs: data, tools, and resources for technology development in low resource languages. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 3273-3280, portorož, slovenia. european language resources association (elra). erik f. tjong kim sang and fien de meulder. 2003. introduction to the conll-2003 shared task: language-independent named entity recognition. in proceedings of the seventh conference on natural language learning at hlt-naacl 2003, pages 142-147. jennifer tracey, stephanie strassel, ann bies, zhiyi song, michael arrigo, kira griffitt, dana delgado, dave graff, seth kulick, justin mott, and neil kuster. 2019. corpus building for low resource languages in the darpa lorelei program. in proceedings of the 2nd workshop on technologies for mt of low resource languages, pages 48-55, dublin, ireland. european association for machine translation. hao wang, bing liu, chaozhuo li, yan yang, and tianrui li. 2019. learning with noisy labels for sentence-level sentiment classification. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6286-6292, hong kong, china. association for computational linguistics. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, rémi louf, morgan funtowicz, and jamie brew. 2019. huggingface's transformers: state-of-the-art natural language processing. shijie wu and mark dredze. 2019. beto, bentz, becas: the surprising cross-lingual effectiveness of bert. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 833-844, hong kong, china. association for computational linguistics. xiang zhang, junbo zhao, and yann lecun. 2015. character-level convolutional networks for text classification. in c. cortes, n. d. lawrence, d. d. lee, m. sugiyama, and r. garnett, editors, advances in neural information processing systems 28, pages 649-657. curran associates, inc.