sam231123/medieval-roleplay-engine

Medieval AI Roleplay Engine

What This Is

A medieval roleplay AI with two characters built from scratch and fine tuned on GPT-2 medium.

Models

Model 1 — RoleplayLM (From Scratch)

Stat	Value
Parameters	~8M
Architecture	Transformer + RoPE
Layers	6
Attention heads	6
Embedding dim	384
FFN dim	768
Max sequence length	128 tokens
Vocabulary size	~1300 tokens
Position encoding	Rotary RoPE
Activation	ReLU
Normalisation	LayerNorm
Weight tying	Yes
Training data	240,000 conversations
Train epochs	15
Optimizer	AdamW
Learning rate	3e-4 cosine decay
Batch size	64
Best val loss	0.27
Training time	~30 minutes on T4

Model 2 — GPT-2 Medium Fine Tuned

Stat	Value
Base model	GPT-2 medium
Parameters	354,826,240
Layers	24
Attention heads	16
Embedding dim	1024
Vocabulary size	50,260 + 2 custom
Custom tokens	[KNIGHT] [DRAGON]
Training data	48,000 sampled conversations
Fine tune epochs	1
Optimizer	AdamW
Learning rate	2e-5 cosine decay
Batch size	8
Max sequence length	128 tokens
Best val loss	0.2623
Training time	~48 minutes on T4

Characters

Knight

• Stoic, honourable, medieval
• Short punchy sentences
• Topics: battle, honour, dragons, death, loyalty, wisdom, purpose
• Training samples: 120,000

Dragon

• Ancient, arrogant, philosophical
• Speaks in statements and riddles
• Treats humans as briefly interesting
• Topics: age, fire, humans, patience, memory, power, deals
• Training samples: 120,000

Architecture

Input tokens
    -> Token Embedding (vocab x 384)
    -> Dropout
    -> 6 x TransformerBlock
        -> LayerNorm
        -> SelfAttention (6 heads, RoPE)
        -> Residual
        -> LayerNorm
        -> FeedForward (384 -> 768 -> 384, ReLU)
        -> Residual
    -> LayerNorm
    -> LM Head (384 -> vocab, weight tied)
    -> Softmax -> next token

Training Pipeline

1. Generate 120k knight conversations (30 topics)
2. Generate 120k dragon conversations (30 topics)
3. Train shared BPE tokenizer on combined text
4. Prepend [KNIGHT] or [DRAGON] token to each sample
5. Train RoleplayLM from scratch 15 epochs
6. Fine tune GPT-2 medium on sampled data 1 epoch

Inference Settings

• Temperature: 0.85
• Top-k: 50
• Top-p: 0.92
• Repetition penalty: 1.3
• Max new tokens: 100

Sample Outputs

Knight

You> what do you live for knight
Knight> i am simply a man who chose to stand when others fled.
        purpose is the difference between a mercenary and a true hero.
        meaning is not discovered.

You> are you afraid of death knight
Knight> death holds no terror for one who has lived with honour.
        a knight feels fear and charges anyway.

Dragon

You> i can offer you a great black stone dragon
Dragon> stone is heavy and soft and humans kill each other over it.
        curious how the old tales about dragons repeat themselves.
        your kind builds kingdoms.

You> tell me something i do not know dragon
Dragon> fire. what you see is merely the beginning.
        everything dies. i am simply taking longer than most.

Hardware

• Kaggle T4 GPU free tier

Requirements

torch>=2.0.0
transformers>=4.30.0
tokenizers>=0.13.0

How To Run

pip install torch transformers tokenizers
python src/chat.py