himbot-gpt-level2

A GPT-style language model trained from scratch using PyTorch on WikiText-103.

• 44.7M parameters
• 6 layers, 8 heads, 512 embedding dim
• Trained for 20,000 steps
• Final perplexity: 38.3

Usage

from transformers import GPT2LMHeadModel, GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("hranjan043/himbot-gpt-level2")
model     = GPT2LMHeadModel.from_pretrained("hranjan043/himbot-gpt-level2")

inputs = tokenizer("The history of science", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.8,
    top_k=50,
    do_sample=True,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Sample Output

"The history of science was not considered to be the first to be in the United States, but the second to be seen in the first American science fiction book series..."

Training Details

Property	Value
Architecture	Decoder-only Transformer (GPT-style)
Parameters	44.7M
Layers	6
Attention heads	8
Embedding dim	512
Context length	256 tokens
Vocabulary	GPT-2 BPE (50,257 tokens)
Training steps	20,000
Val loss	3.644
Perplexity	38.3
Dataset	WikiText-103
Optimizer	AdamW (lr=3e-4)
Hardware	RTX 4000 Ada 20GB

License

MIT