shrey1211/lma-indic-models-shrey715

LMA Indic Models -- Phase 1 Deliverables

Hindi and Nepali pretraining data pipeline outputs: from-scratch BPE tokenizers, a calibrated KenLM perplexity filter, and the full raw + processed corpora, for an independent-per-language decoder-only Transformer LM project. Generated by scripts/publish/push_to_hf.py; see the source repository's docs/directive_phase1.md for full methodology.

Contents

Hindi

• Corpus: 436,499,132 kept words (20.1% manual)
• Tokenizer: chosen vocab=8,192 (all 5 swept sizes included in this repo: 4,096, 8,192, 16,384, 32,768, 65,536), fertility=1.390 tokens/word, UNK rate=0.0000%, byte-fallback rate=0.0000%
• Perplexity filter: 5-gram KenLM, reject threshold=4768.99 perplexity

Nepali

• Corpus: 493,605,125 kept words (23.0% manual)
• Tokenizer: chosen vocab=8,192 (all 5 swept sizes included in this repo: 4,096, 8,192, 16,384, 32,768, 65,536), fertility=1.530 tokens/word, UNK rate=0.0000%, byte-fallback rate=0.0000%
• Perplexity filter: 5-gram KenLM, reject threshold=16796.45 perplexity

Structure

hindi/
├── tokenizer/                    # all 5 swept BPE vocab sizes (4,096, 8,192, 16,384, 32,768, 65,536), .model+.vocab each
├── perplexity_filter/            # KenLM 5-gram fluency filter used during cleaning
└── data/
    ├── processed/{train,val,test}/*.jsonl.gz  # final pretraining corpus splits
    └── raw/
        ├── downloaded/           # Sangraha subset actually used (gzipped)
        └── manual/                # self-collected news/Wikipedia/archive.org (gzipped)
nepali/
├── tokenizer/                    # all 5 swept BPE vocab sizes (4,096, 8,192, 16,384, 32,768, 65,536), .model+.vocab each
├── perplexity_filter/            # KenLM 5-gram fluency filter used during cleaning
└── data/
    ├── processed/{train,val,test}/*.jsonl.gz  # final pretraining corpus splits
    └── raw/
        ├── downloaded/           # Sangraha subset actually used (gzipped)
        └── manual/                # self-collected news/Wikipedia/archive.org (gzipped)

.crawl_state_*.json scraper resume-checkpoint files are deliberately excluded (internal bookkeeping, not corpus content).

Loading the corpus

datasets.load_dataset handles gzip-compressed JSONL natively -- no manual decompression needed:

from datasets import load_dataset
ds = load_dataset("json", data_files="hindi/data/processed/train/train.jsonl.gz", split="train")

Loading the tokenizer

All 5 swept vocab sizes are included per language (4,096, 8,192, 16,384, 32,768, 65,536); this project currently uses 8,192 for both (see report/phase1/report.tex, Vocabulary Size Selection, for why):

import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file="hindi/tokenizer/hindi_bpe_8192.model")

Loading the perplexity filter

import kenlm
model = kenlm.Model("hindi/perplexity_filter/perplexity_filter.klm")