gaperon-mmbert-hindi

A multi-head text-quality classifier for Hindi, fine-tuned from jhu-clsp/mmBERT-base (a ModernBERT-architecture multilingual encoder) as part of the Gaperon data-curation pipeline. Given a passage of text, it predicts a low / medium / high rating along six quality dimensions, intended for scoring and filtering web/corpus text before it's used for further model training.

This is the best checkpoint (selected on validation mean_macro_f1) from training run mmbert_hindi_v1, saved at epoch 4.

Model description

• Backbone: jhu-clsp/mmBERT-base (ModernBERT architecture, 22 layers, hidden size 768, 8192 max sequence length, mean pooling over token representations).
• Heads: six independent linear heads (Linear(768 → 3)), one per label dimension, applied to the mean-pooled backbone output. Each head is a 3-way classifier over {low: 0, medium: 1, high: 2}.
• Label dimensions: clarity, coherence, depth, grammar, usefulness, overall.

The backbone weights are in model.safetensors / config.json (standard HF format, loadable with AutoModel). The six classification heads are not part of the HF backbone class and are stored separately in heads.pt (see usage below).

Intended use

Rating the quality of Hindi text (e.g. web-scraped documents) along the six dimensions above, for filtering/weighting examples in a training corpus. Not intended as a general-purpose sentence encoder or for tasks unrelated to the quality-rating schema it was trained on.

Training data

Fine-tuned on an in-house tagged corpus of Hindi text (data_gaperon/tagged_v2/hindi), labeled with low/medium/high ratings for each of the six dimensions.

Hyperparameters

Hyperparameter	Value
max_length	512
batch_size	32
learning_rate	2e-5
weight_decay	0.01
dropout	0.1
frozen layers	0
warmup_ratio	0.1
seed	42

Evaluation results (best checkpoint, epoch 4)

Macro-F1 / accuracy per dimension:

Dimension	Val F1	Val Acc	Test F1	Test Acc
clarity	0.5586	0.5977	0.5453	0.5733
coherence	0.5941	0.6239	0.5930	0.6045
depth	0.5409	0.8074	0.5483	0.8062
grammar	0.5582	0.6353	0.5399	0.6122
usefulness	0.5859	0.6330	0.5743	0.6362
overall	0.5553	0.6062	0.5549	0.6165
mean macro-F1	0.5655	–	0.5593	–

Usage

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoTokenizer

repo_id = "<your-username>/gaperon-mmbert-hindi"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
backbone = AutoModel.from_pretrained(repo_id)

heads_path = hf_hub_download(repo_id, filename="heads.pt")
ckpt = torch.load(heads_path, map_location="cpu")
label_dims = ckpt["label_dims"]              # ["clarity", "coherence", "depth", "grammar", "usefulness", "overall"]
label2id = ckpt["label2id"]                  # per-dimension {"low": 0, "medium": 1, "high": 2}
heads_state = ckpt["heads_state_dict"]        # one Linear(768, 3) per dimension

heads = {dim: torch.nn.Linear(768, 3) for dim in label_dims}
for dim, head in heads.items():
    head.weight.data = heads_state[f"{dim}.weight"]
    head.bias.data = heads_state[f"{dim}.bias"]
    head.eval()

backbone.eval()
text = "Your input passage here."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    hidden = backbone(**inputs).last_hidden_state          # (1, seq_len, 768)
    mask = inputs["attention_mask"].unsqueeze(-1)
    pooled = (hidden * mask).sum(1) / mask.sum(1)           # mean pooling

id2label = {dim: {v: k for k, v in mapping.items()} for dim, mapping in label2id.items()}
for dim, head in heads.items():
    pred_id = head(pooled).argmax(-1).item()
    print(dim, "->", id2label[dim][pred_id])

Limitations

• Trained and evaluated only on Hindi text matching the tagging schema of the source corpus; quality ratings may not transfer to other domains or genres.
• The 3-class (low/medium/high) granularity is coarse; F1 scores in the 0.54-0.59 range indicate the heads are useful as a filtering signal but should not be treated as ground truth.
• heads.pt requires custom loading code (shown above) — it is not loadable via AutoModelForSequenceClassification.