Nepali-POS-tagger

=================

This repository contains a Nepali Part-of-Speech (POS) tagging model based on BERT (BertForTokenClassification) that can predict POS tags for Nepali sentences. The model is trained to work with a compatible tokenizer and can be used for inference on new sentences.

Model Card

• Model type: BertForTokenClassification
• Tokenizer: Same tokenizer as the model (AutoTokenizer)
• Task: Part-of-Speech tagging (Token Classification)
• Language: Nepali

Installation

Install the required libraries:

pip install torch transformers

Usage

Load the model and tokenizer:

from transformers import AutoTokenizer, AutoModelForTokenClassification import torch

Load tokenizer and model

model_name_or_path = "roheetbasu/Nepali-pos-tagger" tokenizer = AutoTokenizer.from_pretrained(model_name_or_path) model = AutoModelForTokenClassification.from_pretrained(model_name_or_path)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model.to(device) model.eval()

Define label mapping

ids2label = { 0: 'CD', 1: 'JJ', 2: 'NNP', 3: 'POP', 4: 'NN', 5: 'PKO', 6: 'VBX', 7: 'YF', 8: 'FB', 9: 'VBF', 10: 'PLAI', 11: 'DUM', 12: 'VBKO', 13: 'RBO', 14: 'VBI', 15: 'VBO', 16: 'HRU', 17: 'JJD', 18: 'YM', 19: 'PLE', 20: 'JJM', 21: 'RP', 22: 'VBNE', 23: 'CS', 24: 'YQ', 25: 'CL', 26: 'PP', 27: 'PP$', 28: 'CC', 29: 'SYM', 30: 'PPR', 31: 'DM', 32: 'OD', 33: 'QW', 34: 'UNW', 35: 'RBM', 36: 'FW', 37: 'YB', 38: 'ALPH' }

Predict POS tags for Nepali sentences:

Sample Nepali sentences

sentences = [ "नेपाल सुन्दर देश हो।", "म स्कूल जान्छु।", "कुकुर बगैँचामा खेलिरहेको छ।" ]

split senteces for efficient tags for particles

import re

List of common Nepali postpositions/particles

particles = ["को", "का", "की", "मा", "बाट", "लाई", "देखि", "सम्म", "सँग", "भित्र", "बाहिर", "माथि", "तल", "पछाडि", "अगाडि", "तर्फ", "वरिपरि", "भर", "बीच", "जस्तो", "अनुसार", "अन्तर्गत", "प्रति"]

def tokenize_sentence_for_model(sentence): """ Tokenize a Nepali sentence into words suitable for HuggingFace tokenizers. - Separates known particles - Splits punctuation Returns a list of words (tokens) """ # Step 1: Separate punctuation sentence = re.sub(r'([।,!?])', r' \1 ', sentence)

# Step 2: Split by whitespace
tokens = sentence.strip().split()

# Step 3: Separate attached particles
final_tokens = []
for token in tokens:
    matched = False
    for p in particles:
        if token.endswith(p) and len(token) > len(p):
            base = token[:-len(p)]
            final_tokens.append(base)
            final_tokens.append(p)
            matched = True
            break
    if not matched:
        final_tokens.append(token)

return final_tokens

def prepare_sentences_for_tokenizer(sentences): """ Takes a list of Nepali sentences and returns a list of token lists ready for tokenizer with is_split_into_words=True """ return [tokenize_sentence_for_model(s) for s in sentences]

------------------ Example Usage ------------------

Nepali test sentences

sentences = [ "नेपाल सुन्दर देश हो।", "म स्कूल जान्छु।", "कुकुर बगैँचामा खेलिरहेको छ।", "उसले मलाई भेट्यो।", "विद्यालयमा विद्यार्थीहरू पढिरहेका छन्।", "म किताब घरमा राखेँ।", "बजारबाट तरकारी किनेर ल्याएँ।", "तिमी कहाँ जान्छौ?", "सूर्य पश्चिमतिर अस्ताउँछ।", "हामीले फुटबल खेल्ने योजना बनायौं।" ]

Get properly tokenized sentences

split_sentences = prepare_sentences_for_tokenizer(sentences)

Tokenize without converting to tensors (needed for word_ids)

tokenized = tokenizer( split_sentences, is_split_into_words=True, padding=True, truncation=True, max_length=128 )

Convert to tensors

inputs = {k: torch.tensor(v).to(device) for k,v in tokenized.items()}

Run the model

with torch.no_grad(): outputs = model(**inputs)

Convert logits to predictions

predictions = torch.argmax(outputs.logits, dim=2).cpu().numpy()

Map predictions back to words using tokenized.word_ids

for i, sentence in enumerate(sentences): word_ids = tokenized.word_ids(batch_index=i) # ✅ use tokenized, not inputs pred_labels = [] prev_label = None for idx, word_idx in enumerate(word_ids): if word_idx is None: continue if idx == 0 or word_idx != word_ids[idx-1]: cur_label = model.config.id2label[predictions[i][idx]]

        #  to prevent the labeling को in words like खेलिरहेको 
        if prev_label == 'VBKO' and cur_label == "PKO":
            continue
        prev_label = cur_label
        pred_labels.append(cur_label)
        
print(f"Sentence: {sentence}")
print(f"POS Tags: {pred_labels}")
print("---------------")

Output

For the sample sentences above, you may get output like:

Sentence: नेपाल सुन्दर देश हो। POS Tags: ['NN', 'JJ', 'NN', 'VBX', 'YF']

Sentence: म स्कूल जान्छु। POS Tags: ['PP', 'NN', 'VBF', 'YF']

Sentence: कुकुर बगैँचामा खेलिरहेको छ। POS Tags: ['NN', 'NN', 'POP', 'VBKO', 'VBX', 'YF']

Sentence: उसले मलाई भेट्यो। POS Tags: ['PP', 'PP', 'PLAI', 'VBF', 'YF']

Sentence: विद्यालयमा विद्यार्थीहरू पढिरहेका छन्। POS Tags: ['NN', 'POP', 'NN', 'VBKO', 'VBX', 'YF']

Sentence: म किताब घरमा राखेँ। POS Tags: ['PP', 'NN', 'NN', 'POP', 'VBF', 'YF']

Sentence: बजारबाट तरकारी किनेर ल्याएँ। POS Tags: ['NN', 'POP', 'NN', 'VBO', 'VBF', 'YF']

Sentence: तिमी कहाँ जान्छौ? POS Tags: ['PP', 'QW', 'VBF', 'YF']

Sentence: सूर्य पश्चिमतिर अस्ताउँछ। POS Tags: ['NN', 'JJ', 'VBF', 'YF']

Sentence: हामीले फुटबल खेल्ने योजना बनायौं। POS Tags: ['PP', 'NN', 'VBNE', 'NN', 'VBF', 'YF']

Notes

• The tokenizer must match the one used during training.
• Subword tokens are automatically handled, and only the first token of each word is considered for labeling.
• Ensure your input sentences are split into words using sentence.split() before tokenization.