A Hybrid Architecture for Labelling Bilingual M āori- E nglish Tweets (2022)

Abstract: Most large-scale language detection tools perform poorly at identifying Māori text. Moreover, rule-based and machine learning-based techniques devised specifically for the Māori-English language pair struggle with interlingual homographs. We develop a hybrid architecture that couples Māori-language orthography with machine learning models in order to annotate mixed Māori-English text. This architecture is used to label a new bilingual Twitter corpus at both the token (word) and tweet (sentence) levels. We use the collected tweets to show that the hybrid approach outperforms existing systems with respect to language detection of interlingual homographs and overall accuracy. We also evaluate its performance on out-of-domain data. Two interactive visualisations are provided for exploring the Twitter corpus and comparing errors across the new and existing techniques. The architecture code and visualisations are available online, and the corpus is available on request.

## introduction ko te kupu te mauri o reo māori." translated to english as the language is the life force of the mana māori. the word is the life force of the language @xcite , this famous saying by tā hēmi hēnare (sir james hēnare) encapsulates the importance of the māori language to māori, the indigenous people of aotearoa foot_0 new zealand. te reo māori is both endangered and lowresourced, with limited corpora and natural language processing (nlp) techniques available @xcite . data annotation currently has to be done manually by language experts, making the process time-consuming and resourceintensive. these obstacles hinder technological advances that could assist in maintaining the language and, consequently, the culture of māori. the māori language used today is frequently interspersed with english, either in the form of code-switching @xcite or borrowing. here, the borrowing process is bidirectional, resulting in both english loanwords in māori @xcite and māori loanwords in english @xcite . the latter are not only used by bilingual māori speakers, but also by monolingual englishspeaking new zealanders. linguists are interested in determining the frequency of these patterns, which are reflective of aotearoa new zealand's unique bicultural identity. the interweaving of māori and english is a key consideration for developing robust technologies that can accommodate practical, everyday usage of te reo māori and new zealand english. leveraging the abundance of relevant data on twitter, our research focuses on the following task: automatic language identification for bilingual māori-english text at both the token (word) and tweet (sentence) level. differentiating between māori and english text is not straightforward. this is because both languages use the roman script, and interlingual homographs-words that are spelt the same but differ in meaning across languages @xcite -are prolific. these words present a major challenge for classifying mixed-language text, especially if they are highly frequent in both target languages @xcite . consider the following tweets in which interlingual homographs are emphasised: (a) here is to a more productive day tomorrow (b) ka kite koe i a koe! (c) he is at a tangi in ruatoki. doubt he did in terms of annotation, the desired tweet-level labels are (a) english, (b) māori, and (c) bilingual. these are determined with recourse to the individual token labels: all tokens in (a) are english, all tokens in (b) are māori, and (c) contains a mixture of tokens from both languages, with 'tangi' (funeral) and 'ruatoki' (a place name) being labelled māori. according to our approach, all words of māori origin are tagged as māori, even if they are used as borrowings in english. in order to obtain accurate tweet and token-level labels, we utilise knowledge and understanding gained from māori researchers, māori technology developers and the māori community. our methodology involves combining machine learning techniques with māori orthography, thereby instantiating the pipeline recommended by @xcite . we hypothesise that doing so will improve the overall accuracy of language identification for bilingual māori-english text. this paper makes the following contributions: 1. development of a hybrid architecture 2 to detect māori and english words for a given bilingual text input. 2. the māori-english twitter (met) corpus, a first-of-a-kind dataset comprising bilingual and monolingual tweets, annotated at the token-and tweet-level by deploying our architecture. 3. evidence that the hybrid architecture improves both language detection of interlingual homographs and overall accuracy when compared with two existing techniques. 4. two interactive visualisation tools for exploring the corpus and comparing label errors across the different systems. ## methodology this section details the process used to collect twitter data (section 3.1) and the techniques underpinning our hybrid architecture. we combine language rules (section 3.2) with neural networks (section 3.3), as suggested by @xcite . ## the māori-english twitter corpus the steps detailed in the previous two sections resulted in the formation of a new bilingual dataset: the māori-english twitter (met) corpus. key summary statistics for this collection of 76,000 tweets are presented in token frequency 0 10000 20000 30000 40000 te maori the to a i for of in o reo and is ki nga e ka he me you māori english ## experiment results and analysis this section compares the performance of the newly-developed hybrid system with the standalone rmt @xcite and ml (james tweet labels token-level errors (fp, fn) tweets actual rmt ml hybrid rmt ml hybrid 1. teaching ate me @xmath0, me --2. <@xmath1:) #bfc630nz b b b b more, one --3. <@xmath2. hope to catch up while we are here! b b b b hope, here kia -4. <@xmath3, hope you're well mate. b b b b marie, hope, mate -5. e hoa ma, nga mihi o te tau hou! #matariki #maorinewyear #bn-zatm #@xmath4, o, tau -6. maori party welcomes waitangi tribunal report b b b b -waitangi waitangi ## limitations the research presented in this paper has some limitations that need to be acknowledged. the hybrid architecture uses a single neural networkbased model, but we have experimented with variations in the neural networks and parameter choices. given the available data and resources, bidirectional rnns performed the best. we found that our hybrid architecture does not label māori named entities consistently, and shortlength homographs like 'i' and 'a' are problematic. this requires further investigation, perhaps involving a special look-up for māori place names, and ensuring that a context check is always carried out for frequent homographs, especially function words. in addition, our approach for identifying foreignlanguage tweets is not exhaustive, and in some cases, tokens that are neither māori nor english will have been erroneously labelled as such. our foreign-language processing currently focuses on manually identifying problematic tweets in a small subset of the data, then extrapolating this into the wider dataset. this approach could be further developed, or a more automated system could be implemented. our labels do not distinguish between borrowings and code-switches @xcite . this means it is not possible to automatically extract tweets where māori borrowings are used in otherwise english contexts, or vice versa, although the number of tokens identified in each language could serve as a useful proxy. finally, we discarded a proportion of the collected tweets as our algorithm was not optimised for dealing with undue levels of noise. the discarded tweets with unknown labels are not vital to the met corpus presented in this research; however, they require further investigation, and may constitute useful additions to the corpus. ## conclusions and future work this paper presents an architecture for labelling bilingual māori-english text, by bringing together machine learning and knowledge of māori orthography, an approach that could also be fruitful for other endangered languages. we use this architecture to create the first large-scale corpus of bilingual māori-english tweets annotated at both the token and tweet level. both this corpus and the hansard dataset are used to illustrate the strengths of our approach, including superior token-level accuracy, especially with respect to interlingual homographs. in particular, the specificity scores for bilingual data favour the hybrid system, while highlighting a major weakness of the rmt system. additional insights can be gleaned from two exploratory visualisations for interrogating the corpus and comparing system errors. future work towards enhancing the bilingual corpus could involve extending this research to classify hashtags as these are currently ignored. moreover, the architecture lends itself to annotating other bilingual datasets, such as the mlt corpus @xcite , and could assist in the creation of new resources. a further avenue of exploration would be assigning part-of-speech tags to each token in the corpus, based on the language identified. this could be achieved using newlydeveloped tools for māori @xcite in conjunction with established part-of-speech taggers for english. such developments are important for ensuring better representation of the māori language in digital applications and environments.