FinSLM-30M-LR1e-4-GA32

A ~30M parameter GPT-style language model trained from scratch on financial news text.

The complete implementation, experiments, and report are available in the GitHub repository.

Model

• 6 Transformer layers
• 6 attention heads
• 384-dimensional embeddings
• 128-token context length
• GPT-2 BPE tokenizer (50,257 vocabulary)

Training

• Learning rate: 1e-4
• Gradient accumulation: 32
• Maximum iterations: 60,000
• Hardware: NVIDIA T4 (16 GB)
• Precision: float16
• Best validation loss: 3.997
• Best perplexity: 54.5

Results

The model learns the vocabulary and general structure of financial news, but generated text can contain factual and numerical errors.