Viveka-34M (विवेक)

Viveka is Sanskrit for discernment: telling what is right from what only looks right. That is this model's job.

A small English language model trained from scratch (random start, its own tokenizer) in 45 minutes on one free Colab T4 GPU. It is the "understanding" part of AksharaVardhan (अक्षरवर्धन), a handwriting reader: when the reader is unsure between spellings, this model judges which whole sentence makes more sense.

Made by a 9-year-old hobbyist Vrish Vardhan Sharma (Hugging Face: vrishv).

Model

Design	Llama-style decoder (rotary positions, SwiGLU, RMSNorm), tied input/output embeddings
Size	33.9M numbers: 8 layers, width 512, 8 attention heads, feed-forward 1408
Tokenizer	its own byte-level BPE, 16,000 word pieces (trained on the same text)
Context	256 word pieces
Training	45 min on a T4, about 108M word pieces, AdamW, fp16, cosine learning rate
Stored as	half precision (float16), 68 MB; scores are the same as full precision to 0.01

Training text

Source	Share	Licence
FineWeb-Edu (educational web pages)	40%	ODC-BY
Project Gutenberg books (public-domain books)	30%	public domain / MIT listing
English Wikipedia	20%	CC BY-SA 3.0
IAM handwriting texts (training split only)	10%	IAM: non-commercial research

Because the IAM texts are for non-commercial research, this model is released as non-commercial (CC BY-NC 4.0).

Results (honest numbers)

Perplexity (lower = better; it means "choosing between about N word pieces each time") on text it never trained on:

Text	15 min	30 min	45 min (final)
Web pages (FineWeb-Edu)	150	70	60
IAM validation lines (measured line by line, so line breaks mid-sentence count against it)	300	210	200

Sense test: it chose the sensible sentence in 6 of 6 pairs, including real mistakes from the handwriting reader ("sorts of horrible illnesses brought on by the" over "sotts of hotkible ilenesses beought on by the").

Inside AksharaVardhan (the job it was made for)

The handwriting reader reads 600 lines by IAM test writers (never seen in training). For each line it offers its top guesses plus wordfreq spelling fixes, every one checked against the handwriting; Viveka then scores whole sentences (one call per sentence) and the best fit of handwriting + sense wins. The mix was tuned on 300 IAM validation lines only.

600 IAM test lines	Characters right	Word errors
Reader alone	93.41%	21.76%
+ wordfreq spelling fixes	93.42%	21.74%
+ Viveka-34M (whole sentences)	93.99%	19.16%

Viveka removes about 1 in 11 character errors and 1 in 8 word errors. Real examples:

Reader alone	With Viveka
I dont thinn he will shorm the charks with this one	I dont think he will storm the charts with this one
a lively song that chonges tempo midtway	a lively song that changes tempo midway
is an accoumtont	is an accountant

It cannot fix rare names or badly misread words ("Dovid whitfield" stayed wrong), because it only chooses between versions it is given; it never invents a correction.

Example completions (sampled, temperature 0.8):

• The cat sat on the floor and took up the cask with the little-headed cask, with
• In the morning, the children went to school for the best chance of being able to have a good deal of interest in

Use

from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained('vrishv/Viveka-34M')
lm = AutoModelForCausalLM.from_pretrained('vrishv/Viveka-34M', dtype='float32')   # full precision is faster on a CPU
x = tok('The cat sat on the', return_tensors='pt')
print(tok.decode(lm.generate(**x, max_new_tokens=20, do_sample=True, top_k=40)[0]))

Limitations

• Very small and briefly trained. It knows grammar and common phrases, but very little about the world, and its writing often stops making sense after a few words. It is meant for scoring which of a few sentences is more likely, not for chatting or answering questions.
• English only. It learned from web pages, Wikipedia and old books, so it can repeat their biases and old-fashioned language, and it may produce wrong or odd text. Do not rely on it for facts.
• Context is only 256 word pieces.