iankamar/financial-text-sentiment-models

Financial Text Sentiment

Fine-tuned DistilBERT for negative, neutral, and positive financial sentences. Prepared as a noncommercial research demonstration using Financial PhraseBank (Malo et al., 2014). Base model weights are Apache 2.0; the dataset is CC BY-NC-SA 3.0. Preserve both sources' notices and restrictions. This metadata is not a new grant of commercial rights.

Training and evaluation

Fixed seed 42. Three fine-tuning epochs; checkpoint selected using validation macro-F1. Exact normalized duplicate sentences collapsed before a stratified 70/15/15 split. Temperature scaling fit on validation logits only. Held-out calibrated accuracy: 0.8512; macro-F1: 0.8330. See RESULTS.md and results.json for baselines, class results, bootstrap intervals, calibration, and errors.

Raw model outputs are uncalibrated logits. To reproduce demo probabilities, divide logits by the temperature in calibration.json before applying softmax. Use the saved tokenizer and maximum length 96.

Limitations

One dataset, one transformer training seed, sentence-level random holdout. No temporal, article, company, or external-domain evaluation. Near-duplicate paraphrases may remain. Confident mistakes occur. Not a stock-price predictor or an investment decision system.

Sources

https://huggingface.co/datasets/takala/financial_phrasebank https://arxiv.org/abs/1307.5336 https://huggingface.co/distilbert/distilbert-base-uncased

Demo

https://huggingface.co/spaces/iankamar/financial-text-sentiment

The source archive includes the training and evaluation code, splits, source notices, and measured predictions. Model weights and baseline artifacts are at this repository root.

Free browser demo

The Space runs locally in the visitor's browser with a dynamic int8 ONNX export. On the same 726-sentence test split, the export achieved 85.5% accuracy and 0.833 macro-F1. These are separate from the original PyTorch model's scores. See browser_results.json. The browser reuses the validation-fitted temperature; quantization may change probability calibration.

The Space includes the JavaScript baseline implementation; its probabilities matched Python within 6e-16 across all test sentences. The first transformer download is approximately 68 MB. No paid inference API is used.

Use the original model

Load this repository with Transformers AutoTokenizer and AutoModelForSequenceClassification. Tokenize with truncation=True and max_length=96, then apply softmax to logits divided by the temperature in calibration.json. Labels are negative, neutral, positive in that order. Download baseline.joblib only from this trusted repository and use the scikit-learn version in the research source requirements.

To reproduce training, extract research-source.zip, install requirements.txt in a Python 3.12 environment, and run python train.py. This downloads the pinned dataset and base checkpoint and recreates the artifacts. Run audit_similarity.py, audit_agreement.py, and make_report.py afterward. The browser export can be recreated with export_browser.py; browser serving source is available in the linked Space.