full evaluation not complete
Fin-MPNET-Base (v0.1)
This is a fine-tuned sentence-transformers model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
This model aims to be very strong on Financial Document Retrieval Tasks, while trying to maintain as much generalized performance as possible.
FiQA | SciFact | AmazonReviews | OnlineBankingIntent | ArguAna | |
---|---|---|---|---|---|
fin-mpnet-base | 79.91 | 65.40 | 29.12 | 80.25 | 49.11 |
all-mpnet-base-v2 | 49.96 | 65.57 | 31.92 | 81.86 | 46.52 |
previous SoTA | 56.59 | - | - | - | - |
v0.1 shows SoTA results on FiQA Test set while other non-financial benchmarks only drop a few small % and improvement in others.
Usage (Sentence-Transformers)
Using this model becomes easy when you have sentence-transformers installed:
pip install -U sentence-transformers
Then you can use the model like this:
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer('mukaj/fin-mpnet-base')
embeddings = model.encode(sentences)
print(embeddings)
Evaluation Results
Model was evaluated during training only on the new finance QA examples, as such only financial relevant benchmarks were evaluated on for v0.1 [FiQA-2018, BankingClassification77]
The model currently shows the highest FiQA Retrieval score on the test set, on the MTEB Leaderboard (https://huggingface.co/spaces/mteb/leaderboard)
The model will have likely suffered some performance on other benchmarks, i.e. BankingClassification77 has dropped from 81.6 to 80.25, this will be addressed for v0.2 and full evaluation on all sets will be run.
Training
"sentence-transformers/all-mpnet-base-v2" was fine-tuned on 150k+ financial document QA examples using MNR Loss.
- Downloads last month
- 3,808
Space using mukaj/fin-mpnet-base 1
Evaluation results
- accuracy on MTEB AmazonReviewsClassification (en)test set self-reported29.128
- f1 on MTEB AmazonReviewsClassification (en)test set self-reported28.657
- map_at_1 on MTEB ArguAnatest set self-reported24.111
- map_at_10 on MTEB ArguAnatest set self-reported40.083
- map_at_100 on MTEB ArguAnatest set self-reported41.201
- map_at_1000 on MTEB ArguAnatest set self-reported41.215
- map_at_3 on MTEB ArguAnatest set self-reported35.325
- map_at_5 on MTEB ArguAnatest set self-reported37.796
- mrr_at_1 on MTEB ArguAnatest set self-reported25.036
- mrr_at_10 on MTEB ArguAnatest set self-reported40.436
- mrr_at_100 on MTEB ArguAnatest set self-reported41.554
- mrr_at_1000 on MTEB ArguAnatest set self-reported41.568
- mrr_at_3 on MTEB ArguAnatest set self-reported35.645
- mrr_at_5 on MTEB ArguAnatest set self-reported38.141
- ndcg_at_1 on MTEB ArguAnatest set self-reported24.111
- ndcg_at_10 on MTEB ArguAnatest set self-reported49.112
- ndcg_at_100 on MTEB ArguAnatest set self-reported53.670
- ndcg_at_1000 on MTEB ArguAnatest set self-reported53.944
- ndcg_at_3 on MTEB ArguAnatest set self-reported39.035
- ndcg_at_5 on MTEB ArguAnatest set self-reported43.503
- precision_at_1 on MTEB ArguAnatest set self-reported24.111
- precision_at_10 on MTEB ArguAnatest set self-reported7.817
- precision_at_100 on MTEB ArguAnatest set self-reported0.976
- precision_at_1000 on MTEB ArguAnatest set self-reported0.100
- precision_at_3 on MTEB ArguAnatest set self-reported16.596
- precision_at_5 on MTEB ArguAnatest set self-reported12.134
- recall_at_1 on MTEB ArguAnatest set self-reported24.111
- recall_at_10 on MTEB ArguAnatest set self-reported78.165
- recall_at_100 on MTEB ArguAnatest set self-reported97.582
- recall_at_1000 on MTEB ArguAnatest set self-reported99.573
- recall_at_3 on MTEB ArguAnatest set self-reported49.787
- recall_at_5 on MTEB ArguAnatest set self-reported60.669
- accuracy on MTEB Banking77Classificationtest set self-reported80.250
- f1 on MTEB Banking77Classificationtest set self-reported79.650
- map_at_1 on MTEB FiQA2018test set self-reported37.747
- map_at_10 on MTEB FiQA2018test set self-reported72.223
- map_at_100 on MTEB FiQA2018test set self-reported73.802
- map_at_1000 on MTEB FiQA2018test set self-reported73.805
- map_at_3 on MTEB FiQA2018test set self-reported61.618
- map_at_5 on MTEB FiQA2018test set self-reported67.922
- mrr_at_1 on MTEB FiQA2018test set self-reported71.914
- mrr_at_10 on MTEB FiQA2018test set self-reported80.710
- mrr_at_100 on MTEB FiQA2018test set self-reported80.901
- mrr_at_1000 on MTEB FiQA2018test set self-reported80.901
- mrr_at_3 on MTEB FiQA2018test set self-reported78.935
- mrr_at_5 on MTEB FiQA2018test set self-reported80.193
- ndcg_at_1 on MTEB FiQA2018test set self-reported71.914
- ndcg_at_10 on MTEB FiQA2018test set self-reported79.912
- ndcg_at_100 on MTEB FiQA2018test set self-reported82.675
- ndcg_at_1000 on MTEB FiQA2018test set self-reported82.702
- ndcg_at_3 on MTEB FiQA2018test set self-reported73.252
- ndcg_at_5 on MTEB FiQA2018test set self-reported76.360
- precision_at_1 on MTEB FiQA2018test set self-reported71.914
- precision_at_10 on MTEB FiQA2018test set self-reported23.071
- precision_at_100 on MTEB FiQA2018test set self-reported2.620
- precision_at_1000 on MTEB FiQA2018test set self-reported0.263
- precision_at_3 on MTEB FiQA2018test set self-reported51.235
- precision_at_5 on MTEB FiQA2018test set self-reported38.117
- recall_at_1 on MTEB FiQA2018test set self-reported37.747
- recall_at_10 on MTEB FiQA2018test set self-reported91.346
- recall_at_100 on MTEB FiQA2018test set self-reported99.776
- recall_at_1000 on MTEB FiQA2018test set self-reported99.897
- recall_at_3 on MTEB FiQA2018test set self-reported68.691
- recall_at_5 on MTEB FiQA2018test set self-reported80.742
- map_at_1 on MTEB NFCorpustest set self-reported4.124
- map_at_10 on MTEB NFCorpustest set self-reported10.207
- map_at_100 on MTEB NFCorpustest set self-reported13.181
- map_at_1000 on MTEB NFCorpustest set self-reported14.568
- map_at_3 on MTEB NFCorpustest set self-reported7.262
- map_at_5 on MTEB NFCorpustest set self-reported8.622
- mrr_at_1 on MTEB NFCorpustest set self-reported39.009
- mrr_at_10 on MTEB NFCorpustest set self-reported48.144
- mrr_at_100 on MTEB NFCorpustest set self-reported48.746
- mrr_at_1000 on MTEB NFCorpustest set self-reported48.789
- mrr_at_3 on MTEB NFCorpustest set self-reported45.356
- mrr_at_5 on MTEB NFCorpustest set self-reported47.152
- ndcg_at_1 on MTEB NFCorpustest set self-reported36.533
- ndcg_at_10 on MTEB NFCorpustest set self-reported29.643
- ndcg_at_100 on MTEB NFCorpustest set self-reported27.893
- ndcg_at_1000 on MTEB NFCorpustest set self-reported37.307
- ndcg_at_3 on MTEB NFCorpustest set self-reported33.357
- ndcg_at_5 on MTEB NFCorpustest set self-reported32.250
- precision_at_1 on MTEB NFCorpustest set self-reported38.700
- precision_at_10 on MTEB NFCorpustest set self-reported22.941
- precision_at_100 on MTEB NFCorpustest set self-reported7.303
- precision_at_1000 on MTEB NFCorpustest set self-reported2.028
- precision_at_3 on MTEB NFCorpustest set self-reported31.889
- precision_at_5 on MTEB NFCorpustest set self-reported29.040
- recall_at_1 on MTEB NFCorpustest set self-reported4.124
- recall_at_10 on MTEB NFCorpustest set self-reported14.443
- recall_at_100 on MTEB NFCorpustest set self-reported29.765
- recall_at_1000 on MTEB NFCorpustest set self-reported63.074
- recall_at_3 on MTEB NFCorpustest set self-reported8.516
- recall_at_5 on MTEB NFCorpustest set self-reported10.979
- map_at_1 on MTEB SciFacttest set self-reported49.011
- map_at_10 on MTEB SciFacttest set self-reported60.094
- map_at_100 on MTEB SciFacttest set self-reported60.799
- map_at_1000 on MTEB SciFacttest set self-reported60.828
- map_at_3 on MTEB SciFacttest set self-reported57.175
- map_at_5 on MTEB SciFacttest set self-reported58.748
- mrr_at_1 on MTEB SciFacttest set self-reported51.667
- mrr_at_10 on MTEB SciFacttest set self-reported61.312
- mrr_at_100 on MTEB SciFacttest set self-reported61.821
- mrr_at_1000 on MTEB SciFacttest set self-reported61.850
- mrr_at_3 on MTEB SciFacttest set self-reported59.000
- mrr_at_5 on MTEB SciFacttest set self-reported60.200
- ndcg_at_1 on MTEB SciFacttest set self-reported51.667
- ndcg_at_10 on MTEB SciFacttest set self-reported65.402
- ndcg_at_100 on MTEB SciFacttest set self-reported68.377
- ndcg_at_1000 on MTEB SciFacttest set self-reported69.094
- ndcg_at_3 on MTEB SciFacttest set self-reported60.154
- ndcg_at_5 on MTEB SciFacttest set self-reported62.455
- precision_at_1 on MTEB SciFacttest set self-reported51.667
- precision_at_10 on MTEB SciFacttest set self-reported9.067
- precision_at_100 on MTEB SciFacttest set self-reported1.067
- precision_at_1000 on MTEB SciFacttest set self-reported0.112
- precision_at_3 on MTEB SciFacttest set self-reported24.000
- precision_at_5 on MTEB SciFacttest set self-reported15.933
- recall_at_1 on MTEB SciFacttest set self-reported49.011
- recall_at_10 on MTEB SciFacttest set self-reported80.511
- recall_at_100 on MTEB SciFacttest set self-reported94.000
- recall_at_1000 on MTEB SciFacttest set self-reported99.500
- recall_at_3 on MTEB SciFacttest set self-reported66.200
- recall_at_5 on MTEB SciFacttest set self-reported71.944