Bridging the Gap: Transfer Learning from E nglish PLM s to M alaysian E nglish (2024)

Abstract: Malaysian English is a low resource creole languages, where it carries the elements of Malay, Chinese, and Tamil languages, in addition to Standard English. Named Entity Recognition (NER) models underperforms when capturing entities from Malaysian English text due to its distinctive morphosyntactic adaptations, semantic features and code-switching (mixing English and Malay). Considering these gaps, we introduce MENmBERT and MENBERT, a pre-trained language model with contextual understanding, specifically tailored for Malaysian English. We have fine-tuned MENmBERT and MENBERT using manually annotated entities and relations from the Malaysian English News Article (MEN) Dataset. This fine-tuning process allows the PLM to learn representations that capture the nuances of Malaysian English relevant for NER and RE tasks. MENmBERT achieved a 1.52% and 26.27% improvement on NER and RE tasks respectively compared to the bert-base-multilingual-cased model. While the overall performance for NER does not have significant improvement, our further analysis shows that there is a significant improvement when evaluated by the 12 entity labels. These findings suggest that pre-training language models on language-specific and geographically-focused corpora can be a promising approach for improving NER performance in low-resource settings. The dataset and code published through this paper provide valuable resources for NLP research work focusing on Malaysian English.

## introduction with the recent proliferation of large language models (llms), the usage of pre-trained language models (plms) like bert @xcite , roberta @xcite , xlm-roberta @xcite , albert @xcite ) has * corresponding author. been overshadowed. however, plm has shown some significant improvements when further pretrained in domain specific @xcite @xcite or language specific corpus @xcite @xcite , and subsequently fine-tuned on nlp tasks. the adaptability of pre-trained language model to diverse languages and dialects has enabled its application to specific linguistic contexts, including malaysian english, a unique and culturally rich variant of the english language. malaysian english is also categorized as a creole language due to its distinct characteristics that include loanwords, compound words, and the derivation of new terms from malay, chinese, and tamil, in addition to the standard english @xcite . existing state-of-the-art (sota) solutions do not produce satisfactory outcomes for downstream tasks performed in malaysian english @xcite . this is mainly due to the morphosyntactic and semantical adaption nature of malaysian english. hence, there is a need to improve this sota in order to support effective processing of malaysian english texts. this work investigates the effectiveness of english pre-trained language models (plms) to the low resource language like malaysian english, in downstream task, particularly named entity recognition (ner), and relation extraction (re) performed on malaysian english. our findings contribute to the growing body of research on promoting inclusivity in nlp by exploring the applicability of plms to non-standard english varieties. the contributions of this paper are as follows: we introduce menmbert, a bert-based model pre-trained on a malaysian english news (men) corpus. men corpus comprises 14,320 articles, facilitating research on applying plms to malaysian english @xcite . this model will be made public to develop resources and facilitate nlp research in malaysian english. the code for this experiment, and dataset have been published in https://github.com/mohanraj-nlp/ men-dataset/tree/pretrained-lm . ## conclusion this work introduced menmbert, a contextualized language model pre-trained on a malaysian english corpus. our experiments demonstrated that fine-tuning menmbert on language-specific data significantly improves performance on ner tasks with average of +1.74%. for re, we have achieved average improvement of +26.27% compare our menmbert and menbert with baseline plm's. however, the fine-tuned re models achieved lower performance compared to reported results on other document-level relation extraction datasets. this suggests that while menm-bert's entity prediction capabilities benefit re tasks, further exploration is needed to optimize re performance in the context of our dataset. this has gaps has suggested us for future work, explore several avenues to improve re performance. one of the approach involve investigating data augmentation techniques to expand our dataset and improve model training. beyond that, we will extend our experiment do several other downstream nlp task.