poseidon1113/gpt2-lora-financial-sentiment-v2

GPT-2 LoRA — Financial Sentiment Analysis v2

LoRA fine-tuned GPT-2 for financial sentiment classification (positive / neutral / negative).

v2 stacks training on NOSIBLE Financial Sentiment (100K examples) on top of v1 which was trained on Financial PhraseBank.

Model Details

Base model	GPT-2 (124M)
Method	LoRA (r=8, alpha=16)
Trainable params	~300K (0.24%)
Target modules	c_attn
v1 dataset	Financial PhraseBank — 4,840 examples, 3 epochs
v2 dataset	NOSIBLE Financial Sentiment — 100K examples, 1 epoch
Max length	256
Learning rate	1e-4 (reduced for stacked fine-tuning)

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

tokenizer = AutoTokenizer.from_pretrained("poseidon1113/gpt2-lora-financial-sentiment-v2")
model = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained("gpt2", torch_dtype=torch.float16),
    "poseidon1113/gpt2-lora-financial-sentiment-v2"
).eval()

def predict(sentence):
    inputs = tokenizer(f"### Sentence:\n{sentence}\n\n### Sentiment:\n", return_tensors="pt")
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=10, do_sample=False,
                             pad_token_id=tokenizer.eos_token_id)
    result = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:],
                              skip_special_tokens=True).strip().lower()
    return next((w for w in result.split() if w in ("positive", "negative", "neutral")), "neutral")

predict("Operating profit rose to EUR 13.1 mn from EUR 21.1 mn.")  # → positive
predict("The company reported a loss for the third consecutive quarter.")  # → negative

Evaluation on FiQA 2018 (all splits combined)

Class	Correct	Total	Accuracy
Positive	210	706	29.7%
Neutral	82	115	71.3%
Negative	213	373	57.1%
Overall	505	1194	42.3%

Comparison with v1

	v1 (PhraseBank only)	v2 (+ NOSIBLE)
Training examples	4,840	+100,000
Overall accuracy	—	42.3%

Limitations

• GPT-2 is a small model — larger base (e.g. Llama, Mistral) would improve accuracy significantly
• Works best on single sentences or short paragraphs
• May still show neutral bias inherited from PhraseBank distribution

Citation

@article{Malo2014GoodDO,
  title={Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts},
  author={P. Malo and A. Sinha and P. Korhonen and J. Wallenius and P. Virtanen},
  journal={Journal of the Association for Information Science and Technology},
  year={2014}, volume={65}
}