Towards Resource-Rich Mizo and K hasi in NLP : Resource Development, Synthetic Data Generation and Model Building (2025)

Abstract: In the rapidly evolving field of Natural Language Processing (NLP), Indian regional languages remain significantly underrepresented due to their limited digital presence and lack of annotated resources. This work presents the first comprehensive effort toward developing high quality linguistic datasets for two extremely low resource languages Mizo and Khasi. We introduce human annotated, gold standard datasets for three core NLP tasks: Part-of-Speech (POS) tagging, Named Entity Recognition (NER), and Keyword Identification. To overcome annotation bottlenecks in NER, we further explore a synthetic data generation pipeline involving translation from Hindi and cross lingual word alignment. For POS tagging, we adopt and subsequently modify the Universal Dependencies (UD) framework to better suit the linguistic characteristics of Mizo and Khasi, while custom annotation guidelines are developed for NER and Keyword Identification. The constructed datasets are evaluated using multilingual language models, demonstrating that structured resource development, coupled with gradual fine-tuning, yields significant improvements in performance. This work represents a critical step toward advancing linguistic resources and computational tools for Mizo and Khasi.

## introduction india is home to more than 1,963 languages (census @xcite , belonging to five major language families, yet the indian constitution officially recognizes only 22 @xcite . while english and hindi are spoken by approximately 10.2% and 43.63% of the population, respectively, the majority prefer using their regional languages. however, a vast number of these languages remain underrepresented in the field of natural language processing (nlp), primarily due to the lack of curated resources and limited availability of digital text in native scripts. while high-resource languages benefit from abundant datasets, extremely low-resource languages like mizo and khasi @xcite have very limited digital presence. mizo, a tibeto-burman language @xcite , is spoken by approximately 831k people, while khasi, an austroasiatic language @xcite , is spoken by around 1.4m (according to census 2011) people in india. a more comprehensive linguistic description of mizo can be found in appendix a.1, and for khasi in appendix a.2. figure 1 illustrates example sentences in mizo and khasi corresponding to the same english sentence. in this work, we focus on the development of foundational linguistic resources to support nlp for mizo and khasi. specifically, we created datasets for part-of-speech (pos) @xcite tagging, named entity recognition (ner) @xcite and keyword identification @xcite .given the lack of task-specific annotation guidelines for mizo and khasi, we adapted the universal dependencies (ud) (universal dependencies, 2025) framework for pos tagging and designed custom annotation schemes that reflect the unique syntactic and semantic characteristics of these two languages. additionally, we created separate annotation guidelines for ner and keyword identification to ensure accurate dataset construction for each task. to mitigate the challenge posed by the scarcity of gold-standard annotated data, especially for ner, we explored synthetic data generation using a hindi ner dataset as a source. this involved translation into mizo and khasi, followed by word alignment using models such as awesome-align @xcite and vecmap @xcite . the alignment process was carefully evaluated and refined to ensure the quality and usability of the resulting synthetic annotations. however, existing language models exhibit little to no understanding of mizo and khasi. to bridge this gap, we first constructed a monolingual corpus for both languages and performed multistage fine-tuning of multilingual models such as muril @xcite , rembert @xcite , and xlm-roberta-large @xcite . building on this foundation, we further finetuned the models on task-specific datasets, employing both standard and gradual fine-tuning strategies. as illustrated in figure 2 , the gradual fine-tuning approach led to a significant boost in performance across pos tagging, ner and keyword identification tasks, demonstrating its effectiveness in lowresource settings. by systematically developing and evaluating lin-guistically grounded resources, this work marks an important step toward enriching the nlp landscape for mizo and khasi languages currently transitioning from rising stars to the underdogs @xcite , supported by a growing suite of annotated datasets and tailored linguistic tools. ## synthetic ner data generation there is a severe lack of publicly available data for these languages on the internet, making it necessary to rely on synthetic data generation (anonymous, 2025) to obtain large-scale resources without direct human involvement. however, direct translation from another language is not feasible, as it often results in variations in word count and word order (james and krishnamurthy, 2025). this makes it difficult to map the ner tags, especially when using the bio (beginning, inside, outside) (yohannes and amagasa, 2022) format. to address this, we used hindi ner @xcite data (tagged in bio format) as our source. we first translated the sentences without their tags into mizo and khasi @xcite . after translation, we aligned the words using awesome-align and vecmap. however, awesome-align internally relies on mbert @xcite , which has minimal to no representation of mizo and khasi. to mitigate this limitation, we first fine-tuned mbert on our monolingual corpus. the initial results were suboptimal, prompting us to refine our approach. we partitioned the monolingual corpus into two subsets, each containing approximately 7.5 million tokens. the model was initially fine-tuned on the first subset, followed by an additional fine-tuning stage on the second subset. this two-stage finetuning process resulted in a perplexity score of 9.25, significantly enhancing the model's ability to process mizo and khasi text. once awesome-align was trained, we used our hindi source sentences and their mizo/khasi translations (hindi ||| mizo/khasi) to generate word alignments. however, the model occasionally produced unaligned words or incorrectly mapped multiple words to a single word. to refine these alignments, we used vecmap. for vecmap, we first generated word2vec @xcite embeddings for hindi, mizo, and khasi using our source hindi sentences and their corresponding translations. we then mapped the hindi embeddings to a common space with mizo/khasi embeddings and vice versa. using cosine similarity, we corrected the unaligned words and improved alignments where awesome-align incorrectly assigned multiple words to a single word. this resulted in more accurate alignments. figure 5 illustrates detailed examples of hindi-khasi and hindi-mizo alignments. at this stage, we had hindi ner data, along with translated mizo/khasi sentences and their word alignments. to map the ner tags, we first removed the bio tags and then assigned the tags according to the alignments. finally, we reapplied the bio tags: this process allowed us to generate high-quality synthetic ner data for mizo and khasi, ensuring accurate tag mappings despite the complexities of translation and word alignment. figure 6 illustrates the detailed procedure for synthetic data generation, and all synthetic datasets have been publicly released on the ihub-data india platform foot_1 . ## conclusion the development of nlp resources for lowresource languages such as mizo and khasi is crucial for their digital preservation and broader linguistic accessibility. through the creation of highquality annotated datasets for pos tagging, ner, and keyword identification, this work establishes foundational linguistic resources to support future research and tool development for these underrepresented languages. in particular, our synthetic ner data generation pipeline leveraging translation and word alignment demonstrate the feasibility of bootstrapping annotated data in the absence of gold-standard resources. among the models evaluated, muril and xlm-r large emerged as the most effective choices, depending on the task. muril performed best for khasi pos tagging (f1: 83.52) and both mizo ner (f1:66.79) and khasi ner (f1:57.84), while xlm-r large achieved the highest score (f1:82.39) for mizo pos tagging, demonstrating how a wellstructured fine-tuning strategy can significantly enhance model performance. future work can extend these efforts by expanding annotated datasets, refining task-specific guidelines, and increasing coverage across linguistic phenomena. incorporating community-driven or semi-automated annotation strategies may further enhance the scalability and adaptability of resource creation, contributing to better representation and accessibility for mizo, khasi, and other low-resource languages.