Language Models for Code-switch Detection of te reo M āori and E nglish in a Low-resource Setting (2022)

Abstract: Te reo Māori, New Zealand’s only indigenous language, is code-switched with English. Māori speakers are atleast bilingual, and the use of Māori is increasing in New Zealand English. Unfortunately, due to the minimal availability of resources, including digital data, Māori is under-represented in technological advances. Cloud-based multilingual systems such as Google and Microsoft Azure support Māori language detection. However, we provide experimental evidence to show that the accuracy of such systems is low when detecting Māori. Hence, with the support of Māori community, we collect Māori and bilingual data to use natural language processing (NLP) to improve Māori language detection. We train bilingual sub-word embeddings and provide evidence to show that our bilingual embeddings improve overall accuracy compared to the publicly-available monolingual embeddings. This improvement has been verified for various NLP tasks using three bilingual databases containing formal transcripts and informal social media data. We also show that BiLSTM with pre-trained Māori-English sub-word embeddings outperforms large-scale contextual language models such as BERT on down streaming tasks of detecting Māori language. However, this research uses large models ‘as is’ for transfer learning, where no further training was done on Māori-English data. The best accuracy of 87% was obtained using BiLSTM with bilingual embeddings to detect Māori-English code-switching points.

## introduction te reo māori (referred to as māori) is new zealand's only indigenous language, spoken by 4.5% of the total population of 5 million. māori speakers are bilingual, and code-switching between māori and english is expected. māori revitalisation efforts have increased māori use in the otherwise english-speaking country. detecting māori language and code-switch instances is a prerequisite to analysing language data. māori and english both use the roman script (specifically, māori uses a modified roman script). currently, annotations are done manually, making the process timeconsuming and slowing down research and technology development. consider the following sentences: in this research, we focus on two primary tasks: task 1: language detection (ld) -detecting māori language words from input text. task 2: code-switch detection (cs) -detecting māori to english or english to māori codeswitch points from input text. there is limited māori-only and māori-english bilingual data available. we collected data in collaboration with the māori researchers, māori technology developers and māori community, where data-sharing is based on trust. as researchers, we remain guardians of the data, ensuring data sovereignty @xcite . hence, all the resources shared from this study are bound by the kaitiakitanga license (te-hiku-media). this paper presents some of the first research to use advances in nlp to detect māori language and code-switching. no existing models are using nlp techniques for māori-english code-switch detection. google and microsoft azure's cloud-based services are the only options available for language detection, which is the primary reason for using such large-scale multilingual cloud-based services for comparison in this paper. this paper's contributions are: 1. evaluation of detecting māori using multilingual models, including the cloud-based services such as google and azure, and large scale language models such as bidirectional encoder representations from transformers (bert). 2. pre-training māori-english bilingual, and māori-only monolingual sub-word embeddings using collected data. experiments using three different bilingual data for various nlp tasks show that bilingual embeddings outperform monolingual embeddings. 3. large scale language models such as bert -without further training on māori-english data-fine-tuned on down streaming tasks of detecting māori are outperformed by bilstm with fasttext pre-trained sub-word bilingual embeddings for low-resourced language such as māori. 4. providing baseline results for detecting lowresourced māori and code-switch between māori-english language pair. ## te reo māori (the māori language) māori is a polynesian language belonging to the austronesian family. phonologically, māori has ten consonants /p t k m n n f r w h/. the māori vowel system is described by five short vowels /i e a o u/ @xcite . orthographically, there is mostly a one-to-one mapping of a māori phoneme to a grapheme, except for two digraphs, 'wh', which is /f/, and 'ng' which is /n/. in modern orthography, long vowels are denoted with a macron (e.g. ā). in older texts, they are sometimes expressed as double vowels (e.g. aa), with an umlaut (e.g. ä), or ignored completely (that is, ā is written a). in addition, there is some regional variation in the way words are spelt (e.g. aorangi vs aoraki). english, in contrast, has a highly non-phonemic orthography. the māori syllable structure consists of a nucleus, which may be occupied by a vowel (or a diphthong), and an optional onset (syllable start) occupied by a single consonant (consonant clusters are not present in māori) @xcite . ## related work research using nlp for tasks relating to māori is relatively young. examples include statistical machine translation for māori-english pair @xcite and the inclusion of māori language detection and translation using cloud services @xcite . @xcite indicates that although the growth of cloud services for māori translations is welcoming, due to the minimal availability of digitised māori data, the resulting output is inaccurate. google also acknowledges that for low-resource languages, the quality of language detection and automatic machine translation is far from perfect (google-ai-blog). we present the first research that uses deep learning techniques to detect a code-switch between māori and english. hence, except for the google and azure cloud services (more details in section 5.1), we are limited by the availability of systems for māori language detection and māori-english code-switch detection for comparison. we use approaches that were inspired by the literature on other language pairs. examples include xnli (cross-lingual natural language inference) cross-lingual classification benchmark @xcite where the bidirectional long short-term memory (bilstm) model was used across several low resource languages, including swahili and urdu; and code-switch detection using bilstm and character-lstm for language pair english-hindi @xcite . xnli benchmark uses fasttext common-crawl embeddings (denoted as e300 in this paper) and aligns it with the muse library. comparison among deep learning models shows that adding background information through sub-word pre-trained embeddings trained using fasttext and in the form of lexicons improves the overall performance of deep neural networks on databases of low-resource languages @xcite . transformers such as bert is the state-of-theart in many nlp tasks, including language detection, named entity recognition, and machine translation @xcite . there are many large scale multilingual models, such as xlm-r @xcite and multilingual bert (mbert) @xcite trained in more than 100 languages. research shows that for languages that are under-sampled during training, the effectiveness of large scale multilingual models such as mbert are sub-optimal @xcite . in comparison to the contextual representations such as bert, embeddings with sub-word representation are more data-efficient when data availability is limited (wu data # sentences # words text labels task hansard data foot_0 2,021,261 36,757,230 formal word-level & sentence level language labels ld, cs mlt corpus 2,500 50,000 informal tweet level labels: relevance/irrelevance ld @xcite rmt corpus 79,018 1,000,000 informal māori words are identified and labelled ld @xcite and dredze, 2020). furthermore, @xcite provide evidence to show that many undersampled or unseen languages during training -such as maltese or narabizi-code-mixed with french perform worse when using mbert compared to an rnn with non-contextual dependency parsing baseline. it has been shown that for such unseen or under-sampled languages, there is a need to further train or fine-tune directly with available raw data in the unseen target languages @xcite . ## databases due to the low-resource nature of the māori language, extensive databases are currently are unavailable. we collected text data from different sources to form the māori-english words (mew) database, as summarised in hansard database contains the new zealand parliament debates from 2003 onwards. together with experts in māori (te-hiku-media), we have labelled the hansard database, where english or māori labels are assigned using linguistic rules and manual checking. each sentence in the databases is marked as māori, english or bilingual. each word of each sentence is labelled as māori or english. the resulting data includes 102,559 bilingual, 1,909,876 english-only and 8,826 māori-only sentences. the māori loanword twitter (mlt) corpus is a small database, where each tweet is labelled as 'relevant' and 'irrelevant', based on the presence of a pre-determined set of māori loanwords in a given tweet. given detecting māori language in tweets is a prerequisite to this task, we consider this task also as a māori language detection task. ## language models and classifiers this section provides details of the language models and classifiers we used. we evaluate the performance of cloud-based language detection systems from google and azure for māori. we represent text as bag-of-words and sub-word embeddings using fasttext. we use logistic regression and multinomial naive bayes as baseline classifiers for language detection. we also use neural networks such as rnns and cnns to train and evaluate language detection and code-switch detection tasks. furthermore, we fine-tune transformer models, bert and mbert, for the down streaming task of language detection. ## experimental setup we experiment with various language models and classifiers for two main tasks: language detection (ld) and code-switch detection (cs). our ultimate goal is to find a combination of language modelling and nlp techniques to improve the overall accuracy of ld and cs tasks. we use three databases to evaluate these tasks with details provided in to represent text we use both fasttext pre-trained embeddings (see neural network models presented in this research are implemented using keras/tensorflow. adam @xcite , an adaptive learning rate optimisation algorithm, is used as the optimiser for neural networks. softmax activation function is used in the output layer of the network. we use a combination of dropout @xcite , with a rate of 0.5, and early stopping @xcite to avoid over-fitting. we use a maximum length of 250 tokens (or words) for bilstm and cnn, and padding for sentences with less than the maximum length. the term tokens and words is used interchangeably in this paper. the embeddings layer is with a dimension of 300. the hidden units of bilstm are 128, and the hidden units of one-dimensional convolutions are 128. for both cnn and bilstm, categorical cross-entropy is used as the loss function. we also fine tune pre-trained transformers, bert and mbert on the down streaming task of language detection. we use batch size of 16, maximum sequence length of 256 and learning rate of 1e-5. for both bert and mbert, the loss and accuracy were reported at each epoch. for both bert and mbert, the model converges fast, needing an average of 5 epochs per run. all evaluations were done using sklearn metrics 7 . evaluations using baseline classifiers such as multilingual naive bayes and lr with bow and static features from embeddings require cpu only 8 machines and are very quick to train and evaluate. neural networks require gpu devices 9 for efficient training and testing. the average training time for cnn was 150-180 minutes, and bilstm was 300-360 minutes, while bert and mbert required 240 minutes per epoch being trained for an average of 5 epochs. the testing time for trained deep learning models is rapid, requiring a few minutes. the code used in this research is made available 10 . f1 score to provide different insights @xcite . we also provide f1scores of each label where appropriate. macro-f1 provides average per-language results and is equally important to all languages. the weighted-f1 score considers the popularity of the languages in the data set. the nemenyi posthoc test (95% confidence level) identifies statistical differences between learning methods. critical difference (cd) plots show the average ranking of individual f1 scores obtained using various language models. the lower the rank, the better the model is. the difference in average ranking is statistically significant if there is no bold line connecting the two settings. ## experimental results the results are presented for the language detection (ld) tasks and code-switch detection (cs) tasks. the language detection task is a crucial first step for detecting code-switching @xcite . first, we present the results of the language detection tasks using the three databases ( to analyse the effectiveness of using google translate and azure services to detect māori (and english), we experimented with the test set of the hansard database where the sentences are either monolingual (māori or english) or code-switched. google translate detected 99.7% of the english words, and azure detected 97.8% of the english words correctly. figure 2 presents pie charts of the resulting language detection for 'māori' word (i.e. the gold-standard labels for the words is 'māori'). for māori words, google translate detected with an accuracy of 65.2%, and azure detected with an accuracy of 52%. although the accuracy of google translate was better than azure, the error rate of both services are too high for māori language detection. in addition, apart from wrongly detecting māori words as english, around 14-21% of the words were classified as various other languages by both cloud services. we acknowledge that cloud-based services such as google and azure are multilingual and hence low-resource languages such as māori are dominated by the resource-rich languages during training. this will inevitably influence the accuracy of ld of māori using google and azure's cloud services. however, given there is no other system available to detect māori, it was still vital to evaluate the outcome of the above mentioned cloud-based services. ## discussion and conclusions this research is the first attempt to use advances in nlp in two tasks -māori (a low-resourced language) language detection, and māori-english code-switch detection. our experiments show that the accuracy of existing cloud-based systems to detect māori is very low. hence, there is the need to have more specialised systems for detecting māori. we collected data in collaboration with māori researchers for training and evaluations. experiments obtained across tasks using three databases show that our bilingual embeddings outperformed downloaded, pre-trained english-only embeddings trained on large databases. among the models tested, bilstm with bilingual embeddings trained using the skip-gram model is the best for both tasks. we provide evidence to show bert-base used on the down-streaming task of language detection -where māori is under-represented or unseen by the model vocabulary-is not always the best solution (as also observed by @xcite ). for most low-resourced languages, including māori, the wikipedia data is significantly smaller than english, resulting in a reduced vocabulary. due to limited resources, continuous training or training from scratch of models such as bert-base is not possible. for future work, it is a possibility to use ideas such as extend m-bert @xcite and explore more efficient pre-training techniques to improve the accuracy of bert like models for language detection of low-resource languages such as māori. in addition, hybrid models using handcrafted rules based on the phonotactic differences between the languages and deep learning-based methods are a promising pathway for future work. the availability of digitised māori and bilingual data is limited, which restricts the ability to train large language models. in addition, considering this is the first deep learning-based research in this area, comparison with published work is not possible. we overcome these limitations by respecting the available data and data sovereignty for this research. we provide experimental results using cloud services such as google and azure, as these are the only available systems that can detect māori. the study reported here is a much-needed contribution to māori language technology development. word embeddings developed in this research are available to other researchers on request, bound by the kaitiakitanga license.