A xomiya BERT a: A Phonologically-aware Transformer Model for A ssamese (2023)

Abstract: Despite their successes in NLP, Transformer-based language models still require extensive computing resources and suffer in low-resource or low-compute settings. In this paper, we present AxomiyaBERTa, a novel BERT model for Assamese, a morphologically-rich low-resource language (LRL) of Eastern India. AxomiyaBERTa is trained only on the masked language modeling (MLM) task, without the typical additional next sentence prediction (NSP) objective, and our results show that in resource-scarce settings for very low-resource languages like Assamese, MLM alone can be successfully leveraged for a range of tasks. AxomiyaBERTa achieves SOTA on token-level tasks like Named Entity Recognition and also performs well on “longer-context” tasks like Cloze-style QA and Wiki Title Prediction, with the assistance of a novel embedding disperser and phonological signals respectively. Moreover, we show that AxomiyaBERTa can leverage phonological signals for even more challenging tasks, such as a novel cross-document coreference task on a translated version of the ECB+ corpus, where we present a new SOTA result for an LRL. Our source code and evaluation scripts may be found at https://github.com/csu-signal/axomiyaberta .

## introduction transformer-based neural architectures such as bert @xcite have revolutionized natural language processing (nlp). the ability to generate contextualized embeddings that both preserve polysemous word sense and similarity across dimensions through self-attention has contributed to significant improvements in various nlp tasks @xcite . despite their successes, transformers come at a high computational cost @xcite and still suffer from long-standing issues pertaining to data-hunger and availability of training resources. one effect of the dependency on big data is the continued proliferation of sophisticated nlp for well-resourced languages while low-resourced languages (lrls) continue to be underrepresented, and the disparities continue to grow @xcite . this is particularly true for languages of india and south asia where english is widely spoken among the educated and urban population. therefore, those in india most likely to use and develop nlp may freely do so in english, but sole speakers of local indian languages may remain effectively isolated from human language technology in their native tongues. while strides have been made in nlp for widely-spoken indian languages (e.g., hindi, bengali, marathi, tamil, etc.), india is home to about a thousand languages, over 100 of which are considered "major" foot_0 but are not widely represented in nlp research. this lack of representation also precludes insights from those languages from contributing to the field @xcite . in this paper, we present axomiyaberta, a novel transformer language model for the assamese language. foot_1 axomiyaberta has been trained in a low-resource and limited-compute setting, using only the masked language modeling (mlm) objective. beyond a model for a new language, our novel contributions are as follows: axomiyaberta achieves competitive or state of the art results on multiple tasks, and demonstrates the utility of our approach for building new language models in resource-constrained settings. ## related work multilingual large language models (mllms) trained over large internet-sourced data, such as mbert and xlm @xcite , provide resources for approximately 100 languages, many of which are otherwise under-resourced in nlp. however, multiple publications @xcite @xcite have demonstrated that multilingual language models tend to underperform monolingual language models on common tasks; the "multilingual" quality of mllms may not be enough to assure performance on lrl tasks, due to languagespecific phenomena not captured in the mllm. since languages that share recent ancestry or a sprachbund tend to share features, there has also been development of models and resources for languages from distinct regions of the world. south asia is one such "language area," where even unrelated languages may share features (e.g., 4way voice/aspiration distinctions, sov word order, retroflex consonants, heavy use of light verbs). as such, researchers have developed region-specific models for south asian languages such as in-dicbert @xcite (11 languages, 8.8 billion tokens) and muril @xcite (17 languages, 16 billion tokens). subword tokenization techniques like byte-pair encoding (bpe) @xcite yield comparatively better performance on lrls by not biasing the vocabulary toward the most common words in a specific language, but bpe tokens also further obscure morphological information not immediately apparent in the surface form of the word. @xcite tackle this problem for kinyarwanda using a morphological analyzer to help generate subwords that better capture individual morphemes. however, despite similar morphological richness of many indian languages, and likely due to similar reasons as outlined above, the dearth of nlp technology for most indian languages extends to a lack of morphological parsers. we hypothesize that adding phonological features can also capture correlations between overlapping morphemes. previous nlp work in assamese includes studies in corpus building @xcite @xcite , pos tagging (kumar and bora, 2018), wordnet @xcite @xcite , image captioning @xcite , and cognate detection @xcite . there does not exist, to our knowledge, significant work on assamese distributional semantics, or any monolingual, transformer-based language model for the assamese language evaluated on multiple tasks. our work complements these previous lines of research with a novel language model for assamese, which further develops an initial model first used in @xcite . we account for the lack of an assamese morphological analyzer with additional phonological features and task formulations that allow for strategic optimization of the embedding space before the classification layer. ## evaluation ## results and discussion axomiyaberta @xmath0, with an even greater improvement (10 f1 points) on wikiner. axomiyaberta also outperformed other baselines for both tasks, with the exception of mbert on wiki-based tasks. 9 fig. 3 shows confusion matrices of performance on asner. indicbert and mbert misclassified org tokens as loc 16 times as much as axomiyaberta. specific cases include sub-tokens like িনউয়কর্ (/ni-ujoôk/, "new york") or িছংগাপু ৰ (/singapuô/, "singapore"), that are actually parts of entities like এ-াৰ িছংগাপু ৰ (/e staô singapuô/, "a-star singapore") or িনউয়কর্ াড েচ াৰ (/niujoôk blad sentaô/, "new york blood center"). this suggests that smaller, monolingual models like axomiyaberta with a 9 wikipedia comprises almost all of mbert's training data. mbert does not support assamese, but does support bengali, and assamese is written using a variant of the same script. named entities are often written identically in bengali and assamese, which could explain this trend. better overall performance on asner than on wikiner can be partially attributed to having one fewer class and a more balanced distribution between categories. asner performance likely benefited from a greater phonological signal and more data to tune on ( ## conclusion and future work in this paper, we presented a novel transformer model for assamese that optionally includes phonological signals. we evaluated on multiple tasks using novel training techniques and have demonstrated sota or comparable results, showing that phonological signals can be leveraged for greater performance and disambiguation for a lowresourced language. axomiyaberta achieves sota performance on short-context tasks like as-ner and long-context tasks like cloze-qa while also outperforming most other transformer-based lms on wikiner, with additional improvement resulting from the phonologically-aware model. for challenging tasks like cdcr, we have shown that both axomiyaberta outperformed other transformer-based lms on popular metrics like bcub, muc, and conll f1. more generally, we have shown that strategic techniques for optimizing the embedding space and language-specific features like phonological information can lower the barrier to entry for training language models for lrls, making it more feasible than before with lower amounts of data and a ceiling on compute power. our experiments suggest phonological awareness boosts performance on many tasks in low-resource settings. future models for other lrls can leverage our ideas to train or fine-tune their own models. since smaller models tend toward anisotropy, embedding dispersal may pave the way for more such performant lrl models. future work may include incorporating phonological signals during pretraining instead of finetuning, carrying out evaluations against semantically harder tasks like paraphrasing or emotion detection, zero-shot transfer to similar languages, and a contrastive learning framework with a triplet loss objective for cdcr. our trained checkpoints are available on huggingface at https://huggingface.co/ abhijnan/axomiyaberta . we hope this resource will accelerate nlp research for encoding language-specific properties in lrls.