sraivante/Custom-GPT-40M-Base

Custom GPT 40M Base

A 39,847,936-parameter English causal language model trained from scratch by sraivante. This release exports the best checkpoint from the CUSTOM_LLM3 + CUSTOM_LLM_OLLAMA_IMPORT project into standard Transformers GPT-2 architecture and Safetensors files.

It learns to continue English text from a short prompt. It is useful for studying small language models, experimenting with next-token prediction, and developing training or inference pipelines. It has a 256-token context window and limited generation quality. It has no instruction/chat fine-tuning or task-specific evaluation, so reliable tutoring, reasoning, factual question answering and code generation have not been demonstrated.

This model uses randomly initialized, project-trained weights. The GPT-2 architecture identifier provides library compatibility; no pretrained OpenAI GPT-2 weights were used.

Load and generate

pip install "torch>=2.6" "transformers>=4.56.2,<5" "safetensors>=0.6.2"

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "sraivante/Custom-GPT-40M-Base"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()

prompt = "The future of artificial intelligence"
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
remaining = model.config.n_positions - inputs["input_ids"].shape[1]
if remaining < 1:
    raise ValueError("Use a prompt shorter than 256 tokens.")

torch.manual_seed(42)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=min(80, remaining),
        do_sample=True,
        temperature=0.8,
        top_k=50,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

Keep prompt tokens + generated tokens <= 256 for this Transformers export. It uses learned absolute positions. The original native generation code supports a sliding context by recomputing a cropped window; standard Transformers generation here uses the fixed position limit.

The tokenizer is a custom 16,000-token byte-level BPE with add_prefix_space=True. No chat template or automatic BOS/EOS insertion is used. Some characters absent from its learned vocabulary can map to <unk>. English is the only intended language; other languages were not evaluated.

See example_generate.py for a runnable command-line example. Safetensors and the built-in Transformers architecture load without trust_remote_code. No GGUF export or Ollama integration is included or verified.

Architecture and training

Setting	Value
Unique parameters	39,847,936
Transformer layers	10
Attention heads	8
Hidden size / feed-forward size	512 / 2,048
Context / vocabulary	256 / 16,000
Positions	Learned absolute embeddings
Activation / LayerNorm epsilon	Exact GELU / 0.00001
Token embedding and output head	Tied
Dropout	0.1
Micro-batch / gradient accumulation	32 / 2
Effective batch / tokens per update	64 sequences / 16,384 tokens
Optimizer	AdamW; betas 0.9, 0.95; weight decay 0.1
Learning rate	0.0003 peak, 0.00003 minimum; cosine schedule
Warmup / gradient clipping	200 steps / 1.0
Evaluation	Every 250 steps; 50 sampled batches per split
Final saved step / selected best step	5,000 / 4,750

The training objective is next-token cross entropy, with random 256-token windows sampled with replacement from the token stream. The notebook specifies CUDA bfloat16 autocast. Actual training hardware and installed package versions were not recorded in its saved outputs. Step labels follow the original notebook, whose loop includes step zero.

The complete recorded settings and history are in training/training_config.json and training/training_history.json. The original training notebook is included unchanged. The export contains FP32 model weights; original optimizer state and training-resume checkpoints are not included.

Training and validation data

The accompanying Custom GPT 40M Sangraha Subset contains the raw corpus, exact local training and validation token arrays, original tokenizer, source hashes and attribution. An identical copy is included in training_data/.

The raw files are verified/eng/data-0.parquet and data-1.parquet from AI4Bharat Sangraha at revision 8b813c3f62d37b2fa174d68c31e8b35ae2fe85e8. Both files match the upstream SHA-256 hashes exactly.

Data	Count
Raw documents	699,050
Total tokens, including document EOS markers	457,688,283
Training tokens	411,919,454
Validation tokens	45,768,829

These are corpus sizes, not counts of tokens actually consumed in training. Raw documents were encoded in file order, separated by EOS, then split at 90% of the total token count. The cut divides document index 629,027 (zero-based): 1,620 tokens are in training and 294 in validation. The splits are not document-disjoint. The tokenizer was fitted before the train/validation split in the notebook.

A deterministic check of 261 documents, including shard endpoints and the split boundary, reproduced the saved token spans exactly. Full re-tokenization of every document was not performed; the exact raw files and token arrays are identified by their complete hashes. These artifacts and checkpoint metadata support the run association, but no external training log records every file opened by the historical run.

Results and limitations

Checkpoint	Step	Sampled train loss	Sampled validation loss
Published best	4,750	4.199621	4.198967
Latest saved	5,000	4.210805	4.211185

The best logged validation loss corresponds to perplexity approximately 66.62. These values come from 50 randomly sampled evaluation batches, not exhaustive evaluation of the validation stream. The shared boundary document and tokenizer fitting procedure limit their interpretation as fully independent held-out performance.

Conversion checks used strict weight loading, finite tensor checks, nine FP32 logit comparisons up to 256 tokens, tokenizer parity, cached greedy generation parity, and a saved AutoModelForCausalLM reload. Maximum observed absolute logit difference was below 0.000003. Details and unedited smoke generations are in training/conversion_validation.json.

Small capacity, brief training, short context and web-heavy data limit coherence and factual accuracy. Generated text can repeat, invent claims, reflect social biases or reproduce training material. No instruction-following, factuality, safety, multilingual or downstream benchmark evaluation is available. Use generated text as an experimental output requiring review.

Files and provenance

• model.safetensors, configuration and tokenizer files: best-checkpoint Transformers export.
• training/: original notebook and history, native architecture, conversion script, settings and validation report.
• training_data/: raw Sangraha shards, saved NumPy token streams, original tokenizer and data provenance.
• SHA256SUMS and artifact_integrity.json: release file integrity records.

License and attribution

Copyright (c) 2026 sraivante for original model weights, project training code and original release documentation, licensed under Apache License 2.0.

Training data remains under CC BY 4.0, with attribution to AI4Bharat, the Sangraha / IndicLLMSuite authors and underlying content creators. It is not covered by this repository's Apache license. The original texts are not claimed as sraivante's writing. See training_data/ATTRIBUTION.md, training_data/LICENSE, the preserved upstream card and NOTICE.

Release prepared 2026-09-25.