korra141/fingpt-forecaster-llama3-lora

FinGPT-Forecaster: LLaMA-3-8B-Instruct with LoRA Fine-Tuning

The model takes a prompt with stock information — company metadata, recent news headlines and summaries, basic financial metrics, and 2-week price history — and predicts the company's stock price movement for the next week.

Output follows a fixed schema.

Model Sources

• Repository: https://github.com/AI4Finance-Foundation/FinGPT
• Base Model: meta-llama/Meta-Llama-3-8B-Instruct
• Dataset: FinGPT/fingpt-forecaster-dow30-202305-202405

---

Training Details

Training Data

Dataset: FinGPT/fingpt-forecaster-dow30-202305-202405

Covers all 30 DOW Jones Index constituents over May 2023 – May 2024. Each sample is a weekly snapshot including:

• Company profile — name, sector, IPO date, market cap, exchange, ticker
• News — headlines and summaries from the past 1–4 weeks (via Finnhub)
• Basic financials — ~35 ratios (EPS, P/E, gross margin, ROE, debt ratios, etc.)
• Price history — weekly open/close prices for the lookback window
• Answer — structured output with positive developments, potential concerns, a % price movement prediction, and analysis (GPT-4 generated labels)

Data is sourced from yfinance and Finnhub. Split: 80% train / 20% test.

---

Training Procedure

Supervised fine-tuning (SFT) of LLaMA-3-8B-Instruct using LoRA. Dataset prompts are originally in LLaMA-2 chat format ([INST]<<SYS>>...[/INST]) and are automatically converted to the LLaMA-3 chat template via tokenizer.apply_chat_template before tokenization. Prompt tokens are masked from the loss; only answer tokens are supervised.

Preprocessing

• Tokenizer: meta-llama/Meta-Llama-3-8B-Instruct; pad_token = eos_token, padding_side = right
• Prompt format: LLaMA-3 chat template (system + user messages), with add_generation_prompt=True
• Max sequence length: 8192 tokens
• Samples exceeding max length are filtered out before training
• LoRA parameters explicitly cast to fp32 for gradient scaler compatibility

Training Hyperparameters

Hyperparameter	Value
Training regime	fp16 mixed precision
Optimizer	AdamW
Learning rate	5e-5
LR scheduler	Constant (with warmup)
Warmup ratio	0.03
Weight decay	0.01
Batch size per device	1
Gradient accumulation steps	4
Number of epochs	5
Max sequence length	4096
Distributed training	torchrun multi-GPU (DDP) + DeepSpeed ZeRO-2
Evaluation strategy	Steps (every 10% of training)
Logging	WandB (fingpt-forecaster project)

LoRA Configuration:

Parameter	Value
Rank (r)	8
Alpha	16
Dropout	0.1
Bias	none
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Task type	CAUSAL_LM

Speeds, Sizes, Times

• Hardware: Multi-GPU SLURM cluster (4 GPUs per node), Apptainer container
• Checkpoint cadence: Every 10% of training steps

---

Evaluation

Metrics

Evaluation is run at each checkpoint via GenerationEvalCallback on 50 randomly sampled test examples (greedy decoding, max 512 new tokens, stopping at <|eot_id|> or EOS).

Metric	Description
Binary Accuracy	Directional accuracy — did the model correctly predict up vs. down?
MSE	Mean squared error on predicted % price movement
ROUGE-1/2/L	N-gram overlap per section (positive developments, potential concerns, analysis)
BERTScore	Semantic similarity (precision, recall, F1) computed per section

Parse rate (% of outputs matching the expected structured format) is also logged.

Results

Model	Dir. Acc.	MSE	ROUGE
FinGPT (Llama-3)	0.6122	7.2653	0.2467
FinGPT (Llama-2)	0.5102	9.7142	0.2425
Llama-3 (base)	0.4568	19.9748	0.2387
Llama-2 (base)	0.4201	28.4471	0.2023
GPT-4	0.3506	24.5682	0.1674
FinBERT	0.4107	17.9348	---
ARIMA	0.5111	8.2926	---
XGBoost	0.4782	8.8607	---
Linear Regression	0.4600	7.4170	---
Driftless Random Walk	---	7.1150	---
Class Distribution	0.5040	---	---

---

Technical Specifications

Model Architecture and Objective

• Base: LLaMA-3-8B-Instruct (decoder-only causal LM, 8B parameters)
• Adaptation: LoRA on all attention projection layers and MLP gate/up/down projections
• Objective: Next-token prediction (cross-entropy) over the structured answer only; prompt is masked
• Key difference vs LLaMA-2 version: Uses the native LLaMA-3 chat template and <|eot_id|> as an additional stop token; gradient checkpointing uses use_reentrant=False

Compute Infrastructure

Multi-GPU SLURM cluster with Apptainer containers. Training coordinated via torchrun with NCCL backend and DeepSpeed ZeRO-2. Weights & Biases for experiment tracking.

Hardware

• NVIDIA A100 GPUs (4 per node)

Software

Package	Notes
PEFT 0.11.0	LoRA implementation
HuggingFace Transformers	Model and trainer
PyTorch	Distributed via torchrun
DeepSpeed	ZeRO-2 optimization
HuggingFace Datasets	Data loading
WandB	Experiment tracking

---

Citation

BibTeX:

@misc{fingpt2023,
  title={FinGPT: Open-Source Financial Large Language Models},
  author={Yang, Hongyang and Liu, Xiao-Yang and Wang, Christina Dan},
  journal={FinLLM Symposium at IJCAI 2023},
  year={2023}
}