SetFit with sentence-transformers/all-mpnet-base-v2
This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/all-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.
The model has been trained using an efficient few-shot learning technique that involves:
- Fine-tuning a Sentence Transformer with contrastive learning.
- Training a classification head with features from the fine-tuned Sentence Transformer.
Model Details
Model Description
Model Sources
Model Labels
Label |
Examples |
0 |
|
1 |
- 'Police officer wounded suspect dead after exchanging shots: RICHMOND Va. (AP) \x89ÛÓ A Richmond police officer wa... http://t.co/Y0qQS2L7bS'
- "There's a weird siren going off here...I hope Hunterston isn't in the process of blowing itself to smithereens..."
- 'Iranian warship points weapon at American helicopter... http://t.co/cgFZk8Ha1R'
|
Evaluation
Metrics
Label |
Accuracy |
all |
0.9203 |
Uses
Direct Use for Inference
First install the SetFit library:
pip install setfit
Then you can load this model and run inference.
from setfit import SetFitModel
model = SetFitModel.from_pretrained("pEpOo/catastrophy8")
preds = model("You must be annihilated!")
Training Details
Training Set Metrics
Training set |
Min |
Median |
Max |
Word count |
1 |
14.5506 |
54 |
Label |
Training Sample Count |
0 |
438 |
1 |
323 |
Training Hyperparameters
- batch_size: (20, 20)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False
Training Results
Epoch |
Step |
Training Loss |
Validation Loss |
0.0001 |
1 |
0.3847 |
- |
0.0044 |
50 |
0.3738 |
- |
0.0088 |
100 |
0.2274 |
- |
0.0131 |
150 |
0.2747 |
- |
0.0175 |
200 |
0.2251 |
- |
0.0219 |
250 |
0.2562 |
- |
0.0263 |
300 |
0.2623 |
- |
0.0307 |
350 |
0.1904 |
- |
0.0350 |
400 |
0.2314 |
- |
0.0394 |
450 |
0.1669 |
- |
0.0438 |
500 |
0.1135 |
- |
0.0482 |
550 |
0.1489 |
- |
0.0525 |
600 |
0.1907 |
- |
0.0569 |
650 |
0.1728 |
- |
0.0613 |
700 |
0.125 |
- |
0.0657 |
750 |
0.109 |
- |
0.0701 |
800 |
0.0968 |
- |
0.0744 |
850 |
0.2101 |
- |
0.0788 |
900 |
0.1974 |
- |
0.0832 |
950 |
0.1986 |
- |
0.0876 |
1000 |
0.0747 |
- |
0.0920 |
1050 |
0.1117 |
- |
0.0963 |
1100 |
0.1092 |
- |
0.1007 |
1150 |
0.1582 |
- |
0.1051 |
1200 |
0.1243 |
- |
0.1095 |
1250 |
0.2873 |
- |
0.1139 |
1300 |
0.2415 |
- |
0.1182 |
1350 |
0.1264 |
- |
0.1226 |
1400 |
0.127 |
- |
0.1270 |
1450 |
0.1308 |
- |
0.1314 |
1500 |
0.0669 |
- |
0.1358 |
1550 |
0.1218 |
- |
0.1401 |
1600 |
0.114 |
- |
0.1445 |
1650 |
0.0612 |
- |
0.1489 |
1700 |
0.0527 |
- |
0.1533 |
1750 |
0.1421 |
- |
0.1576 |
1800 |
0.0048 |
- |
0.1620 |
1850 |
0.0141 |
- |
0.1664 |
1900 |
0.0557 |
- |
0.1708 |
1950 |
0.0206 |
- |
0.1752 |
2000 |
0.1171 |
- |
0.1795 |
2050 |
0.0968 |
- |
0.1839 |
2100 |
0.0243 |
- |
0.1883 |
2150 |
0.0233 |
- |
0.1927 |
2200 |
0.0738 |
- |
0.1971 |
2250 |
0.0071 |
- |
0.2014 |
2300 |
0.0353 |
- |
0.2058 |
2350 |
0.0602 |
- |
0.2102 |
2400 |
0.003 |
- |
0.2146 |
2450 |
0.0625 |
- |
0.2190 |
2500 |
0.0173 |
- |
0.2233 |
2550 |
0.1017 |
- |
0.2277 |
2600 |
0.0582 |
- |
0.2321 |
2650 |
0.0437 |
- |
0.2365 |
2700 |
0.104 |
- |
0.2408 |
2750 |
0.0156 |
- |
0.2452 |
2800 |
0.0034 |
- |
0.2496 |
2850 |
0.0343 |
- |
0.2540 |
2900 |
0.1106 |
- |
0.2584 |
2950 |
0.001 |
- |
0.2627 |
3000 |
0.004 |
- |
0.2671 |
3050 |
0.0074 |
- |
0.2715 |
3100 |
0.0849 |
- |
0.2759 |
3150 |
0.0009 |
- |
0.2803 |
3200 |
0.0379 |
- |
0.2846 |
3250 |
0.0109 |
- |
0.2890 |
3300 |
0.0019 |
- |
0.2934 |
3350 |
0.0154 |
- |
0.2978 |
3400 |
0.0017 |
- |
0.3022 |
3450 |
0.0003 |
- |
0.3065 |
3500 |
0.0002 |
- |
0.3109 |
3550 |
0.0025 |
- |
0.3153 |
3600 |
0.0123 |
- |
0.3197 |
3650 |
0.0007 |
- |
0.3240 |
3700 |
0.0534 |
- |
0.3284 |
3750 |
0.0004 |
- |
0.3328 |
3800 |
0.0084 |
- |
0.3372 |
3850 |
0.0088 |
- |
0.3416 |
3900 |
0.0201 |
- |
0.3459 |
3950 |
0.0002 |
- |
0.3503 |
4000 |
0.0102 |
- |
0.3547 |
4050 |
0.0043 |
- |
0.3591 |
4100 |
0.0124 |
- |
0.3635 |
4150 |
0.0845 |
- |
0.3678 |
4200 |
0.0002 |
- |
0.3722 |
4250 |
0.0014 |
- |
0.3766 |
4300 |
0.1131 |
- |
0.3810 |
4350 |
0.0612 |
- |
0.3854 |
4400 |
0.0577 |
- |
0.3897 |
4450 |
0.0235 |
- |
0.3941 |
4500 |
0.0156 |
- |
0.3985 |
4550 |
0.0078 |
- |
0.4029 |
4600 |
0.0356 |
- |
0.4073 |
4650 |
0.0595 |
- |
0.4116 |
4700 |
0.0001 |
- |
0.4160 |
4750 |
0.0018 |
- |
0.4204 |
4800 |
0.0013 |
- |
0.4248 |
4850 |
0.0008 |
- |
0.4291 |
4900 |
0.0832 |
- |
0.4335 |
4950 |
0.0083 |
- |
0.4379 |
5000 |
0.0007 |
- |
0.4423 |
5050 |
0.0417 |
- |
0.4467 |
5100 |
0.0001 |
- |
0.4510 |
5150 |
0.0218 |
- |
0.4554 |
5200 |
0.0001 |
- |
0.4598 |
5250 |
0.0012 |
- |
0.4642 |
5300 |
0.0002 |
- |
0.4686 |
5350 |
0.0006 |
- |
0.4729 |
5400 |
0.0223 |
- |
0.4773 |
5450 |
0.0612 |
- |
0.4817 |
5500 |
0.0004 |
- |
0.4861 |
5550 |
0.0 |
- |
0.4905 |
5600 |
0.0007 |
- |
0.4948 |
5650 |
0.0007 |
- |
0.4992 |
5700 |
0.0116 |
- |
0.5036 |
5750 |
0.0262 |
- |
0.5080 |
5800 |
0.0336 |
- |
0.5123 |
5850 |
0.026 |
- |
0.5167 |
5900 |
0.0004 |
- |
0.5211 |
5950 |
0.0001 |
- |
0.5255 |
6000 |
0.0001 |
- |
0.5299 |
6050 |
0.0001 |
- |
0.5342 |
6100 |
0.0029 |
- |
0.5386 |
6150 |
0.0001 |
- |
0.5430 |
6200 |
0.0699 |
- |
0.5474 |
6250 |
0.0262 |
- |
0.5518 |
6300 |
0.0269 |
- |
0.5561 |
6350 |
0.0002 |
- |
0.5605 |
6400 |
0.0666 |
- |
0.5649 |
6450 |
0.0209 |
- |
0.5693 |
6500 |
0.0003 |
- |
0.5737 |
6550 |
0.0001 |
- |
0.5780 |
6600 |
0.0115 |
- |
0.5824 |
6650 |
0.0003 |
- |
0.5868 |
6700 |
0.0001 |
- |
0.5912 |
6750 |
0.0056 |
- |
0.5956 |
6800 |
0.0603 |
- |
0.5999 |
6850 |
0.0002 |
- |
0.6043 |
6900 |
0.0003 |
- |
0.6087 |
6950 |
0.0092 |
- |
0.6131 |
7000 |
0.0562 |
- |
0.6174 |
7050 |
0.0408 |
- |
0.6218 |
7100 |
0.0001 |
- |
0.6262 |
7150 |
0.0035 |
- |
0.6306 |
7200 |
0.0337 |
- |
0.6350 |
7250 |
0.0024 |
- |
0.6393 |
7300 |
0.0005 |
- |
0.6437 |
7350 |
0.0001 |
- |
0.6481 |
7400 |
0.0 |
- |
0.6525 |
7450 |
0.0001 |
- |
0.6569 |
7500 |
0.0002 |
- |
0.6612 |
7550 |
0.0004 |
- |
0.6656 |
7600 |
0.0125 |
- |
0.6700 |
7650 |
0.0005 |
- |
0.6744 |
7700 |
0.0157 |
- |
0.6788 |
7750 |
0.0055 |
- |
0.6831 |
7800 |
0.0 |
- |
0.6875 |
7850 |
0.0053 |
- |
0.6919 |
7900 |
0.0 |
- |
0.6963 |
7950 |
0.0002 |
- |
0.7006 |
8000 |
0.0002 |
- |
0.7050 |
8050 |
0.0001 |
- |
0.7094 |
8100 |
0.0001 |
- |
0.7138 |
8150 |
0.0001 |
- |
0.7182 |
8200 |
0.0007 |
- |
0.7225 |
8250 |
0.0002 |
- |
0.7269 |
8300 |
0.0001 |
- |
0.7313 |
8350 |
0.0 |
- |
0.7357 |
8400 |
0.0156 |
- |
0.7401 |
8450 |
0.0098 |
- |
0.7444 |
8500 |
0.0 |
- |
0.7488 |
8550 |
0.0001 |
- |
0.7532 |
8600 |
0.0042 |
- |
0.7576 |
8650 |
0.0 |
- |
0.7620 |
8700 |
0.0 |
- |
0.7663 |
8750 |
0.0056 |
- |
0.7707 |
8800 |
0.0 |
- |
0.7751 |
8850 |
0.0 |
- |
0.7795 |
8900 |
0.013 |
- |
0.7839 |
8950 |
0.0 |
- |
0.7882 |
9000 |
0.0001 |
- |
0.7926 |
9050 |
0.0 |
- |
0.7970 |
9100 |
0.0 |
- |
0.8014 |
9150 |
0.0 |
- |
0.8057 |
9200 |
0.0 |
- |
0.8101 |
9250 |
0.0 |
- |
0.8145 |
9300 |
0.0007 |
- |
0.8189 |
9350 |
0.0 |
- |
0.8233 |
9400 |
0.0002 |
- |
0.8276 |
9450 |
0.0 |
- |
0.8320 |
9500 |
0.0 |
- |
0.8364 |
9550 |
0.0089 |
- |
0.8408 |
9600 |
0.0001 |
- |
0.8452 |
9650 |
0.0 |
- |
0.8495 |
9700 |
0.0 |
- |
0.8539 |
9750 |
0.0 |
- |
0.8583 |
9800 |
0.0565 |
- |
0.8627 |
9850 |
0.0161 |
- |
0.8671 |
9900 |
0.0 |
- |
0.8714 |
9950 |
0.0246 |
- |
0.8758 |
10000 |
0.0 |
- |
0.8802 |
10050 |
0.0 |
- |
0.8846 |
10100 |
0.012 |
- |
0.8889 |
10150 |
0.0 |
- |
0.8933 |
10200 |
0.0 |
- |
0.8977 |
10250 |
0.0 |
- |
0.9021 |
10300 |
0.0 |
- |
0.9065 |
10350 |
0.0 |
- |
0.9108 |
10400 |
0.0 |
- |
0.9152 |
10450 |
0.0 |
- |
0.9196 |
10500 |
0.0 |
- |
0.9240 |
10550 |
0.0023 |
- |
0.9284 |
10600 |
0.0 |
- |
0.9327 |
10650 |
0.0006 |
- |
0.9371 |
10700 |
0.0 |
- |
0.9415 |
10750 |
0.0 |
- |
0.9459 |
10800 |
0.0 |
- |
0.9503 |
10850 |
0.0 |
- |
0.9546 |
10900 |
0.0 |
- |
0.9590 |
10950 |
0.0243 |
- |
0.9634 |
11000 |
0.0107 |
- |
0.9678 |
11050 |
0.0001 |
- |
0.9721 |
11100 |
0.0 |
- |
0.9765 |
11150 |
0.0 |
- |
0.9809 |
11200 |
0.0274 |
- |
0.9853 |
11250 |
0.0 |
- |
0.9897 |
11300 |
0.0 |
- |
0.9940 |
11350 |
0.0 |
- |
0.9984 |
11400 |
0.0 |
- |
0.0007 |
1 |
0.2021 |
- |
0.0329 |
50 |
0.1003 |
- |
0.0657 |
100 |
0.2282 |
- |
0.0986 |
150 |
0.0507 |
- |
0.1314 |
200 |
0.046 |
- |
0.1643 |
250 |
0.0001 |
- |
0.1971 |
300 |
0.0495 |
- |
0.2300 |
350 |
0.0031 |
- |
0.2628 |
400 |
0.0004 |
- |
0.2957 |
450 |
0.0002 |
- |
0.3285 |
500 |
0.0 |
- |
0.3614 |
550 |
0.0 |
- |
0.3942 |
600 |
0.0 |
- |
0.4271 |
650 |
0.0001 |
- |
0.4599 |
700 |
0.0 |
- |
0.4928 |
750 |
0.0 |
- |
0.5256 |
800 |
0.0 |
- |
0.5585 |
850 |
0.0 |
- |
0.5913 |
900 |
0.0001 |
- |
0.6242 |
950 |
0.0 |
- |
0.6570 |
1000 |
0.0001 |
- |
0.6899 |
1050 |
0.0 |
- |
0.7227 |
1100 |
0.0 |
- |
0.7556 |
1150 |
0.0 |
- |
0.7884 |
1200 |
0.0 |
- |
0.8213 |
1250 |
0.0 |
- |
0.8541 |
1300 |
0.0 |
- |
0.8870 |
1350 |
0.0 |
- |
0.9198 |
1400 |
0.0 |
- |
0.9527 |
1450 |
0.0001 |
- |
0.9855 |
1500 |
0.0 |
- |
Framework Versions
- Python: 3.10.12
- SetFit: 1.0.1
- Sentence Transformers: 2.2.2
- Transformers: 4.35.2
- PyTorch: 2.1.0+cu121
- Datasets: 2.15.0
- Tokenizers: 0.15.0
Citation
BibTeX
@article{https://doi.org/10.48550/arxiv.2209.11055,
doi = {10.48550/ARXIV.2209.11055},
url = {https://arxiv.org/abs/2209.11055},
author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Efficient Few-Shot Learning Without Prompts},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}