vinaypoduri/vinayslm-tiny

VinaySLM (tiny)

VinaySLM is a small language model (SLM) built from scratch in PyTorch to learn how decoder-only Transformers work end to end -- tokenization, embeddings, causal self-attention, training, and generation -- without importing a pretrained architecture. This is not a production-quality language model.

What this is

• A tiny, hand-written, decoder-only (GPT-style) Transformer.
• 466,176 parameters.
• Trained on a ~600-token toy corpus of AI/programming sentences.
• Packaged here as a custom Hugging Face model (trust_remote_code=True) so it can be loaded and inspected through the transformers ecosystem, while the actual Transformer implementation (modeling_vinayslm.py) is a thin wrapper around this project's own, unmodified model code.

Architecture

Type	Decoder-only Transformer (GPT-style, pre-LayerNorm)
Parameters	466,176
Vocabulary size	255
Context length	32
Embedding dimension	128
Transformer layers	2
Attention heads	4
Feed-forward dimension	512
Dropout	0.0
Positional encoding	Learned absolute positional embeddings
LM head	Untied from the token embedding
Decoding	Greedy (argmax) only -- no sampling yet

Tokenizer

A minimal, from-scratch word-level tokenizer (VinaySLMTokenizer, wrapping this project's WordTokenizer): text is lowercased and split into words and punctuation with a regex; the vocabulary was built directly from the training corpus. <pad>, <unk>, <bos>, <eos> are the only special tokens. This is not a BPE or pretrained tokenizer.

Training data

A tiny, hand-written corpus of AI/programming sentences (see this project's data/raw/toy_corpus.txt) -- a few hundred tokens total.

Important limitation

The model heavily overfits this tiny toy corpus. It was trained to memorize a few dozen short sentences, not to generalize. Prompts close to a training sentence will often reproduce that sentence verbatim; novel prompts will produce low-quality or repetitive continuations. This model exists to demonstrate a correct, from-scratch Transformer pipeline, not to produce useful or general text.

Example

Prompt:  "Machine learning"
Output:  "machine learning allows computers to learn patterns from data."

How to load

This repository ships its own model/config/tokenizer code (see configuration_vinayslm.py, modeling_vinayslm.py, tokenization_vinayslm.py), so loading it requires trust_remote_code=True and the vinayslm Python package installed (the custom code imports this project's own TinyGPT/WordTokenizer classes rather than duplicating them -- see "Limitations" in this project's README):

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("vinaypoduri/vinayslm-tiny", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("vinaypoduri/vinayslm-tiny", trust_remote_code=True)

Or, equivalently, import the classes directly:

from vinayslm.hf import VinaySLMForCausalLM, VinaySLMTokenizer, generate_greedy

model = VinaySLMForCausalLM.from_pretrained("vinaypoduri/vinayslm-tiny")
tokenizer = VinaySLMTokenizer.from_pretrained("vinaypoduri/vinayslm-tiny")
print(generate_greedy(model, tokenizer, "Machine learning", max_new_tokens=15))

How to reproduce this export

From the VinaySLM project root, with its .venv active:

python scripts/prepare_data.py
python scripts/train.py
python scripts/export_hf.py

scripts/export_hf.py loads the trained artifact from checkpoints/tiny/, wraps it in the Hugging Face-compatible classes, verifies it produces identical greedy output to the original local model, and writes this directory. Nothing is uploaded to the Hugging Face Hub by this script.