FinSLM-30M-LR3e-4-GA4

A ~30M parameter GPT-style language model trained from scratch on financial news text.

The complete implementation, experiments, and report are available in the GitHub repository.

Model

• 6 Transformer layers
• 6 attention heads
• 384-dimensional embeddings
• 128-token context length
• GPT-2 BPE tokenizer (50,257 vocabulary)

Training

• Learning rate: 3e-4
• Gradient accumulation: 4
• Maximum iterations: 20,000
• Hardware: NVIDIA T4 (16 GB)
• Precision: float16
• Best validation loss: 3.508
• Best perplexity: 33.4

Results

The model learns the vocabulary and general structure of financial news, but generated text can contain factual and numerical errors.