korra141/fingpt-forecaster-llama2-lora

Model Sources

• Repository: https://github.com/AI4Finance-Foundation/FinGPT
• Base Model: meta-llama/Llama-2-7b-hf
• Dataset: FinGPT/fingpt-forecaster-dow30-202305-202405

Training Details

Training Data

Dataset: FinGPT/fingpt-forecaster-dow30-202305-202405

Covers all 30 DOW Jones Index constituents over May 2023 – May 2024. Each sample is a weekly snapshot including:

• Company profile — name, sector, IPO date, market cap, exchange, ticker
• News — headlines and summaries from the past 1–4 weeks (via Finnhub)
• Basic financials — ~35 ratios (EPS, P/E, gross margin, ROE, debt ratios, etc.)
• Price history — weekly open/close prices for the lookback window
• Answer — structured output with positive developments, potential concerns, a % price movement prediction, and analysis (GPT-4 generated labels)

Data is sourced from yfinance and Finnhub. Split: 80% train / 20% test.

The model takes a prompt with stock information — company metadata, recent news headlines and summaries, basic financial metrics, and 2-week price history — and predicts the company's stock price movement for the next week.

Output follows a fixed schema.

---

Training Procedure

Supervised fine-tuning (SFT) of LLaMA-2-7B using LoRA. The prompt uses the standard LLaMA-2 chat format ([INST] ... [/INST]). Prompt tokens are masked from the loss; only the answer tokens are supervised.

Preprocessing

• Tokenizer: meta-llama/Llama-2-7b-hf; pad_token = eos_token, padding_side = right
• Max sequence length: 4096 tokens

Training Hyperparameters

Hyperparameter	Value
Training regime	fp16 mixed precision
Optimizer	AdamW
Learning rate	5e-5
LR scheduler	Linear (with warmup)
Warmup ratio	0.03
Weight decay	0.01
Batch size per device	1
Gradient accumulation steps	16
Number of epochs	5
Max sequence length	4096
Evaluation strategy	Steps (every 10% of training)
Distributed training	torchrun multi-GPU (DDP)

LoRA Configuration:

Parameter	Value
Rank (r)	8
Alpha	16
Dropout	0.1
Bias	none
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Task type	CAUSAL_LM

Speeds, Sizes, Times

• Hardware: Multi-GPU SLURM cluster (4 GPUs per node), Apptainer container
• Checkpoint cadence: Every 10% of training steps

---

Evaluation

Metrics

Metric	Description
Binary Accuracy	Directional accuracy — did the model correctly predict up vs. down?
MSE	Mean squared error on predicted % price movement
ROUGE-1/2/L	N-gram overlap for each output section (positive developments, potential concerns, analysis)
BERTScore	Semantic similarity (precision, recall, F1) across all text fields

Evaluation uses 50 randomly sampled test examples for generation-based metrics, run at each checkpoint via GenerationEvalCallback (greedy decoding, max 512 new tokens).

Results

Model	Dir. Acc.	MSE	ROUGE
FinGPT (Llama-3)	0.6122	7.2653	0.2467
FinGPT (Llama-2)	0.5102	9.7142	0.2425
Llama-3 (base)	0.4568	19.9748	0.2387
Llama-2 (base)	0.4201	28.4471	0.2023
GPT-4	0.3506	24.5682	0.1674
FinBERT	0.4107	17.9348	---
ARIMA	0.5111	8.2926	---
XGBoost	0.4782	8.8607	---
Linear Regression	0.4600	7.4170	---
Driftless Random Walk	---	7.1150	---
Class Distribution	0.5040	---	---

---

Technical Specifications

Model Architecture and Objective

• Base: LLaMA-2-7B (decoder-only causal LM)
• Adaptation: LoRA on all attention projection layers and MLP gate/up/down projections
• Objective: Next-token prediction (cross-entropy) over the structured answer only; prompt is masked

Compute Infrastructure

Multi-GPU SLURM cluster with Apptainer containers. Training coordinated via torchrun with NCCL backend. Weights & Biases for experiment tracking.

Hardware

• NVIDIA A100 GPUs (4 per node)

Software

Package	Notes
PEFT 0.11.0	LoRA implementation
HuggingFace Transformers	Model and trainer
PyTorch	Distributed via torchrun
HuggingFace Datasets	Data loading
WandB	Experiment tracking

---

Citation

BibTeX:

@misc{fingpt2023,
  title={FinGPT: Open-Source Financial Large Language Models},
  author={Yang, Hongyang and Liu, Xiao-Yang and Wang, Christina Dan},
  journal={FinLLM Symposium at IJCAI 2023},
  year={2023}
}