Odysseus-1

Model Description

This model is a small-scale Transformer-based model trained on a text corpus. The model generates text by predicting the next character in a sequence, making it suitable for creative text generation tasks. This model is the mid-size model of my odysseus lineup of generative transformers. It is trained to recreate text similar to that of the Odessey and the Iliad by Homer.

Key Hyperparameters

• Embedding dimension (n_embd): 384
• Number of attention heads (n_head): 6
• Number of Transformer layers (n_layer): 6
• Block size (context length): 256
• Vocabulary size: Determined by unique characters in the training data (typically ~65 for English text like Shakespeare)
• Dropout: 0.2
• Total parameters: ~10.7M

The model was trained using AdamW optimizer with a learning rate of 3e-4 for 5000 iterations, on a 90/10 train/validation split of the input text.

Intended Uses & Limitations

Intended Uses

• Text generation: Generate Iliad-like text or similar creative writing.
• Educational purposes: Study Transformer architectures, self-attention, and language modeling.
• Fine-tuning: Can be fine-tuned on custom text datasets for domain-specific generation.

Limitations

• Character-level only: The model operates at the character level, which can lead to slower generation and potential incoherence compared to token-based models like GPT-2.
• Small scale: With ~10M parameters, it may not capture complex long-range dependencies or generate highly coherent long-form text.
• Training data dependency: Performance depends on the quality and size of train.txt (not included here; assume Shakespeare for reference).
• No built-in safety: Generated text may include biases or inappropriate content from the training data.
• Inference speed: On CPU, generation can be slow for long sequences; GPU recommended.

How to Use

Prerequisites

• PyTorch: pip install torch
• The model weights (gpt_model.pth) and vocabulary (vocab.pkl) must be downloaded from this repository.

Loading and Generating Text

The model uses a custom PyTorch architecture. Copy the model classes from generate.py (or train.py) into your script. Here's an example:

import torch
import pickle
# Load hyperparameters (ensure they match training)
block_size = 256
n_embd = 384
n_head = 6  # Corrected to match training
n_layer = 6  # Corrected to match training
dropout = 0.2
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Load vocab
with open('vocab.pkl', 'rb') as f:
    vocab_data = pickle.load(f)
stoi = vocab_data['stoi']
itos = vocab_data['itos']
vocab_size = vocab_data['vocab_size']
decode = lambda l: ''.join([itos[i] for i in l])
# Model classes (paste from generate.py or train.py here)
# ... (Head, MultiHeadAttention, FeedForward, Block, GPTLanguageModel)
# Load the model
model = GPTLanguageModel()
model.load_state_dict(torch.load('gpt_model.pth', map_location=device))
model = model.to(device)
model.eval()
# Generate text
context = torch.zeros((1, 1), dtype=torch.long, device=device)
generated = model.generate(context, max_new_tokens=500)
print(decode(generated[0].tolist()))

For full scripts, see train.py (for training) and generate.py (for inference) in the repository.

Training Your Own Model

1. Prepare train.txt with your text corpus.
2. Run train.py to train and save the model/vocab.
3. Adjust hyperparameters as needed.

Training Data

The model was trained on train.txt, assumed to be a concatenation of Shakespeare's works (as in nanoGPT). Total characters: ~1M (typical for Shakespeare). Unique characters: 65.

• Preprocessing: Character-level tokenization.
• Split: 90% train, 10% validation.

Training Procedure

• Optimizer: AdamW (lr=3e-4)
• Batch size: 64
• Iterations: 5000
• Evaluation: Every 500 steps, averaging loss over 200 batches.
• Hardware: GPU (CUDA) recommended; falls back to CPU.

Final losses (example; actual values depend on run):

• Train: ~1.5
• Validation: ~1.6

Evaluation

The model is evaluated via perplexity (derived from cross-entropy loss) on validation data. No additional benchmarks (e.g., BLEU) were performed, as this is a generative model.