minlstm-lang-detect-10lang

A MinLSTM-based text classifier for language detection, focused on 10 languages with an emphasis on distinguishing English, Nepali (Devanagari), and Romanized Nepali from other languages people commonly mix into everyday text.

Architecture: stacked MinLSTM cells ("Were RNNs All We Needed?", Feng et al., 2024) with an embedding layer, optional additive attention pooling, and an MLP classification head.

Variants in this repo

Variant	Folder	Classes	Description
Base	base/	17 output neurons, 10 trained/active (sparse indices — see note below)	Trained directly on raw per-language text files
SFT	sft/	4 (english, roman_nepali, nepali, others)	Fine-tuned down to the 4 production classes
SFT + Domain Adapted	sft-domain/	4 (english, roman_nepali, nepali, others)	Further adapted on in-domain validation data

If you just want the model actually used in production, use sft-domain/. The base/ variant is included for research/reproducibility.

⚠️ Base variant: sparse label indices

The base checkpoint's output layer has 17 neurons, but only 10 were ever trained — an artifact of how num_classes was derived (max(label_id) + 1) rather than the literal count of active languages. The trained/valid indices are:

0: english, 1: roman_nepali, 2: nepali, 3: hindi, 4: roman_hindi,
5: french, 6: german, 11: spanish, 15: chinese, 16: marathi

Indices 7, 8, 9, 10, 12, 13, 14 are untrained and will produce meaningless logits if queried. See base/labels.json for the authoritative sparse mapping. The sft/ and sft-domain/ variants do not have this issue — they use a clean dense 4-class output.

Files per variant folder

• *.pt — PyTorch checkpoint (contains model_state_dict + architecture metadata)
• tokenizer_final.json — BPE tokenizer (load with the tokenizers library)
• config.json — architecture hyperparameters, label mapping reference, and training metadata
• labels.json — class index → language name mapping

modeling_minrnn.py (repo root) — standalone model definition shared by all three variants. No dependency on the original training repo, only torch.

Installation

pip install torch tokenizers huggingface_hub

Usage

import torch
import json
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

repo_id = "rujanbastola/minlstm-lang-detect-10lang"
variant = "sft-domain"  # or "base", "sft"

# Download files
model_path = hf_hub_download(repo_id, f"{variant}/model.pt")
tokenizer_path = hf_hub_download(repo_id, f"{variant}/tokenizer.json")
config_path = hf_hub_download(repo_id, f"{variant}/config.json")
labels_path = hf_hub_download(repo_id, f"{variant}/labels.json")
modeling_path = hf_hub_download(repo_id, "modeling_minrnn.py")

# Import the architecture (place modeling_minrnn.py on your path first,
# or use importlib if loading it dynamically from the downloaded path)
from modeling_minrnn import MinRNN

with open(config_path) as f:
    config = json.load(f)
with open(labels_path) as f:
    labels = json.load(f)

checkpoint = torch.load(model_path, map_location="cpu")
model = MinRNN(
    units=checkpoint["units"],
    embedding_size=checkpoint["embedding_size"],
    vocab_size=checkpoint["vocab_size"],
    num_classes=checkpoint["num_classes"],
    pad_idx=checkpoint["pad_id"],
    num_layers=checkpoint.get("num_layers", 2),
    dropout=checkpoint.get("dropout", 0.0),
    use_attention=checkpoint.get("use_attention", False),
    attn_dim=checkpoint.get("attn_dim", None),
)
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

tokenizer = Tokenizer.from_file(tokenizer_path)

def predict(sentence: str, max_length: int = config["max_length"]):
    ids = tokenizer.encode(sentence).ids[:max_length]
    input_tensor = torch.tensor([ids], dtype=torch.long)
    with torch.no_grad():
        logits = model(input_tensor).squeeze(0)
        probs = torch.softmax(logits, dim=-1)
    pred_idx = int(probs.argmax().item())
    return labels.get(str(pred_idx), f"class_{pred_idx}"), float(probs[pred_idx])

label, confidence = predict("timro naam k ho?")
print(label, confidence)

Note on preprocessing: inputs should be normalized the same way as training (NFC normalize, lowercase, strip URLs/emails/markup/control chars, strip punctuation/symbols, collapse whitespace) for best results. The snippet above skips this for brevity.

Note on low-confidence predictions: in the original training pipeline, predictions below a confidence threshold (0.25) were mapped to an "Other" fallback label rather than trusting the raw argmax. Consider doing the same in your own inference code if your input distribution includes languages/text outside the training set.

Training data

Trained on per-language raw text corpora (~10 languages), cleaned via script-purity filtering, length clamping, and cross-language deduplication.

Metrics (validation accuracy)

Variant	Val Accuracy	Val Macro F1
base	0.9980	—
sft	0.9988	—
sft-domain	0.9776	0.9780

Limitations

• Short or highly ambiguous text (e.g. single common words shared across languages) may be misclassified.
• The base variant's untrained output indices (see warning above) should not be used.
• Not evaluated on adversarial or code-mixed text beyond what's covered by the others class.

License

MIT (see LICENSE).