NeroT-29M

A 29M-parameter encoder-decoder Transformer for chat title generation, trained from scratch on Apple Silicon with MLX.

---

Quick start (MLX)

pip install mlx mlx-lm transformers

import mlx.core as mx
from mlx_lm import load, generate
from transformers import AutoTokenizer

model, tokenizer = load("j0no12/NeroT-29M")

def generate_title(conversation: str) -> str:
    prompt = conversation[:2000]  # truncate if needed
    output = generate(model, tokenizer, prompt=prompt, max_tokens=32,
                      temp=0.8, top_k=50, repetition_penalty=1.2)
    return output.strip()

# Example
conv = """User: i writing a scene where it has explicit content, are you comfortable with this?
Assistant: I can help you write that scene while keeping it within content guidelines..."""
print(generate_title(conv))
# → "Content Request: Explicit Content"

---

Results on TitleBench (N=21,448)

Model	Params	ROUGE-1	chrF	EM%	Repet%
SupraLabs/supra-title-50m-pre	50M	0.328	0.232	3.4	2.4
j0no12/NeroT-29M (this)	29M	0.296	0.201	8.9	2.7
SupraLabs/Supra-50M-Instruct (general)	50M	0.057	0.042	0.8	32.6
BananaMind/BananaMind-Title-1.0	354M	0.471	0.345	9.3	1.8

All models evaluated on the same 21,448-example held-out split (j0no12/TitleBench), with identical decoding (temp 0.8, top-k 50, repetition penalty 1.2 for NeroT; greedy with documented prompts for competitors).

Key takeaway: A dedicated 29M title model beats a 354M generalist on exact match, and beats the 50M dedicated title model on exact match by 2.6× — at 42% and 92% fewer parameters respectively.

---

Training story

1. Base run (5 epochs, 44M tokens)

• Data: 429k cleaned examples from spitfire4794/chat-titles-245k, Michionlion/chat-titles-english, and BananaMind-Chat-Title-200K (merged, deduped → 429k unique)
• Tokens: 53.9M input + 3.5M target = 57.4M per epoch
• Architecture: 6/6 encoder/decoder layers, d=448, 7 heads (64), FFN 1792
• MLX BF16 + FP32 AdamW (lr 3e-4, warmup 60, cosine)
• 2.65h on Apple M5 Max 48GB

2. Continued pre-training (CPT, 8 epochs, +10M FineWeb-Edu)

• Mixed 80% title / 20% general text (T5 span corruption, 15% mask rate)
• Length-bucketed batching (3.2× speedup vs fixed 512 pad)
• 3.7h overnight on same hardware
• Val loss: 4.24 → 3.15 (best)

3. Generation fix: repetition penalty

• Added 1.2 repetition penalty to decoder sampling
• Crushed repetition from 14% → 2.7%; lifted ROUGE-1 0.326 → 0.335

---

Data lineage

Source	Role	License	Rows
spitfire4794/chat-titles-245k	Train	CC-BY-4.0 / ODC-BY	159,849
Michionlion/chat-titles-english	Train	CC-BY-4.0	107,677
BananaMind/Chat-Title-200K	Train	ID-only (from gated LMSYS)	161,487
HuggingFaceFW/fineweb-edu	CPT	ODC-BY	23k chunks (10M tokens)

Public test split (13,324 rows) and LMSYS ID-only rows (8,124) are available in the TitleBench dataset.

---

Hardware & framework

• Apple M5 Max (40 GPU cores, 48GB unified)
• MLX 0.32 (BF16 weights + FP32 optimizer state)
• Python 3.14 (Homebrew) + transformers (tokenizer only)

---

Files in this repo

File	Description
model.safetensors	28.76M params (144 MB BF16)
config.json	HF-style encoder-decoder config
tokenizer.json + tokenizer_config.json	t5-small tokenizer
MODEL_INFO.md	Architecture + training notes
README.md	This card

---

Citation

@misc{nerot29m,
  author = {j0no12},
  title = {NeroT-29M: A Small Encoder-Decoder for Chat Title Generation},
  year = {2025},
  howpublished = {\url{https://huggingface.co/j0no12/NeroT-29M}},
  note = {Trained from scratch on Apple M5 Max with MLX}
}

---

License

CC-BY-4.0 — attribution required. Model weights, config, and tokenizer are freely usable with credit.

---

Built from scratch