Second5M — Decoder-Only Transformer Language Model

Second5M is an experimental 4,943,040-parameter decoder-only language model built from scratch in PyTorch. It was pretrained on WikiText-103 and then fine-tuned on conversational data. This is an educational research project with limited conversational ability; it frequently gives incoherent or unrelated responses.

Model Details

Property	Value
Architecture	Custom decoder-only Transformer, pre-normalization
Total trainable parameters	4,943,040, including tied embeddings
Layers	8
Hidden dimension	192
Attention	6 query heads, 2 key/value heads
FFN dimension	560 (SwiGLU)
Context window	512 tokens
Tokenizer	Custom byte-level BPE, vocabulary 8,192
Position encoding	Rotary position embeddings (RoPE)
Normalization	RMSNorm
Weight tying	Token embeddings and output head share weights

Quick Start

Requirements: pip install torch tokenizers huggingface_hub

import sys
import torch
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

path = snapshot_download("kaafivikrant/Second5M")
sys.path.insert(0, path)
from model import ModelConfig, TinyChatLM

import json
with open(f"{path}/config.json") as f:
    config = json.load(f)
keys = ModelConfig.__dataclass_fields__.keys()
model = TinyChatLM(ModelConfig(**{k: config[k] for k in keys}))
model.load_state_dict(torch.load(f"{path}/pytorch_model.pt", map_location="cpu", weights_only=True))
model.eval()
tokenizer = Tokenizer.from_file(f"{path}/tokenizer.json")

prompt = "Hello! How are you?"
ids = [tokenizer.token_to_id("<|bos|>"), tokenizer.token_to_id("<|user|>")]
ids += tokenizer.encode(prompt).ids
ids += [tokenizer.token_to_id("<|eos|>"), tokenizer.token_to_id("<|assistant|>")]
output = model.generate(torch.tensor([ids]), max_new_tokens=80, temperature=0.7,
                        top_k=40, eos_id=tokenizer.token_to_id("<|eos|>"))
print(tokenizer.decode(output[0, len(ids):].tolist(), skip_special_tokens=True))

The supplied model uses custom PyTorch code; it is not a standard Transformers AutoModel architecture. Download the Python code only from a source you trust.

Architecture

Tokens → tied embeddings → 8 decoder blocks → RMSNorm → tied output head → logits

Each block: RMSNorm → causal grouped-query attention with RoPE → residual
            RMSNorm → SwiGLU feed-forward → residual

Training Details

• Pretraining: WikiText-103 raw training data, custom tokenizer, 20,000 steps, 655,360,000 sampled tokens, NVIDIA T4. Last recorded validation loss: 3.5913.
• First supervised fine-tuning: up to 50,000 eligible Smol-Smoltalk conversations; 3,000 steps. Long examples in this stage were truncated from the left, sometimes removing the user prompt.
• Final supervised fine-tuning: 19,531 prepared training conversations from Empathetic Dialogues, initialized from the preceding chat checkpoint; 3,000 steps. The saved checkpoint records validation loss 3.0663 on that dataset.
• The final supervised fine-tuning used assistant-turn loss and retained the user request for long examples. Training used AdamW and a cosine learning-rate schedule.
• Losses from different stages use different data and should not be compared directly. No independent benchmark establishes improved chat quality.

Generation Parameters

generate() accepts max_new_tokens, temperature, top_k, and eos_id.

Limitations

• Responses to basic greetings have been observed to be incoherent or unrelated.
• This model has limited knowledge, reasoning, context, and instruction-following ability.
• It can make factual errors and produce repetitive, inappropriate, or misleading output.
• It is not suitable for production, high-stakes decisions, or emotional-support use.

Files

File	Description
model.py	Standalone model definition
pytorch_model.pt	Model weights only
config.json	Model configuration and final checkpoint metadata
tokenizer.json	Byte-level BPE tokenizer

Data and License Information

Pretraining used WikiText, whose dataset card lists CC BY-SA 3.0 and GFDL. Fine-tuning used Smol-Smoltalk and Empathetic Dialogues for LLM, whose repository cards list Apache 2.0. Consult the original datasets and their terms before redistributing or using this model. No model license is asserted here.