manyajain111/lma-phase2

Monolingual Transformer Language Models (Hindi & Nepali)

Pretrained decoder-only Transformer language models (~24.2M parameters each) built and trained from scratch in PyTorch without third-party framework abstractions.

• Model H: Hindi Monolingual LM (Vocab: 16,000, 24.22M parameters)
• Model L: Nepali Monolingual LM (Vocab: 16,240, 24.34M parameters)

Architecture

• Layers: 6
• Heads: 8 ($d_k = 64$)
• Hidden Dim ($d_{model}$): 512
• FFN Dim ($d_{ff}$): 1536 ($3 \times d_{model}$)
• Context Length: 512 tokens
• Attention: Causal Multi-Head Self-Attention with Pre-LayerNorm & Weight Tying
• Training Steps: 61,036 steps each

Evaluation Results

Metric	Model H (Hindi)	Model L (Nepali)
Test Perplexity (PPL)	39.11	36.32
Bits-Per-Byte (BPB)	0.5131	0.3939
Validation Loss	3.6487	3.5910
ROUGE-L (Greedy)	0.1243	0.0838
chrF (Temp 0.5)	13.45	19.88

Usage

Load the model using the included model.py architecture definition:

import torch
from model import GPTLanguageModel

# Load Hindi Model
ckpt = torch.load("hindi/model_final.pt", map_location="cpu")
config = ckpt["config"]
model = GPTLanguageModel(**config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

# Generate text
input_ids = torch.tensor([[10, 25, 42]])  # tokenized input prefix
with torch.no_grad():
    output = model.generate(input_ids, max_new_tokens=50, temperature=0.5)