SetFit with codefuse-ai/F2LLM-v2-80M

This is a SetFit model that can be used for Text Classification. This SetFit model uses codefuse-ai/F2LLM-v2-80M as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: codefuse-ai/F2LLM-v2-80M
Classification head: a LogisticRegression instance
Maximum Sequence Length: 40960 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
positive	'actually, we have decided in nato at the last summit that we should explore the possibilities to integrate russian missile defence systems in our missile defence system, which i think has become even more easy after the u.s. has presented new missile defence plans.' 'and we see, of course, the risk of proliferation of nuclear weapons.' 'there are of course opportunities and we need to engage with china on issues like climate change, arms control.'
negative	'We welcome the successful achievement of a draft Chemical Weapons Convention.' 'in practice, this means that, in addition to reinforcing cooperation with our current partners, we should look to enhance our relations with countries such as australia , new zealand , japan and south korea.' 'this is not about militarizing space.'

Label

Examples

positive

'actually, we have decided in nato at the last summit that we should explore the possibilities to integrate russian missile defence systems in our missile defence system, which i think has become even more easy after the u.s. has presented new missile defence plans.'
'and we see, of course, the risk of proliferation of nuclear weapons.'
'there are of course opportunities and we need to engage with china on issues like climate change, arms control.'

negative

'We welcome the successful achievement of a draft Chemical Weapons Convention.'
'in practice, this means that, in addition to reinforcing cooperation with our current partners, we should look to enhance our relations with countries such as australia , new zealand , japan and south korea.'
'this is not about militarizing space.'

Evaluation

Metrics

Label	F1_Macro	F1_Binary
all	0.9105	0.8988

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("fefofico/nuclear_trained_f2llm")
# Run inference
preds = model("let me first of all say that we take nuclear issues extremely seriously.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	2	23.7926	132

Label	Training Sample Count
negative	1096
positive	857

Training Hyperparameters

batch_size: (128, 128)
num_epochs: (2, 2)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (1e-07, 1e-07)
head_learning_rate: 0.0001
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.35
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0016	1	0.3497	-
0.0327	20	0.3957	-
0.0655	40	0.3689	-
0.0982	60	0.3778	-
0.1309	80	0.3446	-
0.1637	100	0.3345	-
0.1964	120	0.3217	-
0.2291	140	0.3038	-
0.2619	160	0.2764	-
0.2946	180	0.27	-
0.3273	200	0.2596	-
0.3601	220	0.2575	-
0.3928	240	0.2528	-
0.4255	260	0.2507	-
0.4583	280	0.2427	-
0.4910	300	0.2347	-
0.5237	320	0.2226	-
0.5565	340	0.2094	-
0.5892	360	0.1975	-
0.6219	380	0.1837	-
0.6547	400	0.1729	-
0.6874	420	0.1646	-
0.7201	440	0.1516	-
0.7529	460	0.1392	-
0.7856	480	0.1303	-
0.8183	500	0.1265	-
0.8511	520	0.1156	-
0.8838	540	0.1149	-
0.9165	560	0.112	-
0.9493	580	0.1041	-
0.9820	600	0.1007	-
1.0	611	-	0.1073
1.0147	620	0.0926	-
1.0475	640	0.076	-
1.0802	660	0.0862	-
1.1129	680	0.078	-
1.1457	700	0.0745	-
1.1784	720	0.0676	-
1.2111	740	0.0524	-
1.2439	760	0.0585	-
1.2766	780	0.0506	-
1.3093	800	0.0419	-
1.3421	820	0.0446	-
1.3748	840	0.04	-
1.4075	860	0.0349	-
1.4403	880	0.0353	-
1.4730	900	0.0259	-
1.5057	920	0.0273	-
1.5385	940	0.0252	-
1.5712	960	0.0247	-
1.6039	980	0.0157	-
1.6367	1000	0.0172	-
1.6694	1020	0.0142	-
1.7021	1040	0.0136	-
1.7349	1060	0.0144	-
1.7676	1080	0.0111	-
1.8003	1100	0.0074	-
1.8331	1120	0.0103	-
1.8658	1140	0.0118	-
1.8985	1160	0.0098	-
1.9313	1180	0.0071	-
1.9640	1200	0.0082	-
1.9967	1220	0.0092	-
2.0	1222	-	0.1244
2.0295	1240	0.0049	-
2.0622	1260	0.0063	-
2.0949	1280	0.0047	-
2.1277	1300	0.0062	-
2.1604	1320	0.0053	-
2.1931	1340	0.0048	-
2.2259	1360	0.0042	-
2.2586	1380	0.0046	-
2.2913	1400	0.0053	-
2.3241	1420	0.007	-
2.3568	1440	0.0063	-
2.3895	1460	0.0044	-
2.4223	1480	0.0047	-
2.4550	1500	0.0033	-
2.4877	1520	0.0039	-
2.5205	1540	0.0069	-
2.5532	1560	0.004	-
2.5859	1580	0.0038	-
2.6187	1600	0.0031	-
2.6514	1620	0.005	-
2.6841	1640	0.0028	-
2.7169	1660	0.0056	-
2.7496	1680	0.0056	-
2.7823	1700	0.005	-
2.8151	1720	0.0045	-
2.8478	1740	0.0038	-
2.8805	1760	0.0049	-
2.9133	1780	0.0051	-
2.9460	1800	0.0031	-
2.9787	1820	0.0021	-
3.0	1833	-	0.1320
3.0115	1840	0.0027	-
3.0442	1860	0.005	-
3.0769	1880	0.004	-
3.1097	1900	0.0031	-
3.1424	1920	0.0032	-
3.1751	1940	0.0038	-
3.2079	1960	0.0045	-
3.2406	1980	0.0031	-
3.2733	2000	0.0044	-
3.3061	2020	0.0047	-
3.3388	2040	0.0022	-
3.3715	2060	0.0023	-
3.4043	2080	0.0027	-
3.4370	2100	0.0038	-
3.4697	2120	0.0011	-
3.5025	2140	0.0042	-
3.5352	2160	0.0027	-
3.5679	2180	0.0033	-
3.6007	2200	0.0042	-
3.6334	2220	0.0036	-
3.6661	2240	0.0046	-
3.6989	2260	0.0029	-
3.7316	2280	0.0041	-
3.7643	2300	0.003	-
3.7971	2320	0.0033	-
3.8298	2340	0.0033	-
3.8625	2360	0.0047	-
3.8953	2380	0.0041	-
3.9280	2400	0.0036	-
3.9607	2420	0.0037	-
3.9935	2440	0.0044	-
4.0	2444	-	0.1335
4.0262	2460	0.0034	-
4.0589	2480	0.0036	-
4.0917	2500	0.0041	-
4.1244	2520	0.0021	-
4.1571	2540	0.0032	-
4.1899	2560	0.002	-
4.2226	2580	0.0039	-
4.2553	2600	0.0035	-
4.2881	2620	0.0032	-
4.3208	2640	0.0032	-
4.3535	2660	0.0025	-
4.3863	2680	0.0024	-
4.4190	2700	0.0054	-
4.4517	2720	0.0035	-
4.4845	2740	0.0028	-
4.5172	2760	0.0042	-
4.5499	2780	0.0025	-
4.5827	2800	0.0027	-
4.6154	2820	0.0039	-
4.6481	2840	0.0046	-
4.6809	2860	0.0036	-
4.7136	2880	0.004	-
4.7463	2900	0.0031	-
4.7791	2920	0.0024	-
4.8118	2940	0.0036	-
4.8445	2960	0.0046	-
4.8773	2980	0.0025	-
4.9100	3000	0.0056	-
4.9427	3020	0.0031	-
4.9755	3040	0.0024	-
5.0	3055	-	0.1357
0.0016	1	0.008	-
0.0327	20	0.004	-
0.0655	40	0.0028	-
0.0982	60	0.0042	-
0.1309	80	0.0024	-
0.1637	100	0.0039	-
0.1964	120	0.0028	-
0.2291	140	0.0053	-
0.2619	160	0.0027	-
0.2946	180	0.0046	-
0.3273	200	0.0031	-
0.3601	220	0.0031	-
0.3928	240	0.0045	-
0.4255	260	0.0028	-
0.4583	280	0.0045	-
0.4910	300	0.0034	-
0.5237	320	0.0019	-
0.5565	340	0.0008	-
0.5892	360	0.0035	-
0.6219	380	0.0033	-
0.6547	400	0.0026	-
0.6874	420	0.0027	-
0.7201	440	0.0034	-
0.7529	460	0.0033	-
0.7856	480	0.0019	-
0.8183	500	0.0036	-
0.8511	520	0.0023	-
0.8838	540	0.0026	-
0.9165	560	0.0033	-
0.9493	580	0.0028	-
0.9820	600	0.004	-
1.0	611	-	0.1421
1.0147	620	0.003	-
1.0475	640	0.0023	-
1.0802	660	0.0019	-
1.1129	680	0.0026	-
1.1457	700	0.0018	-
1.1784	720	0.0018	-
1.2111	740	0.0007	-
1.2439	760	0.0025	-
1.2766	780	0.0021	-
1.3093	800	0.0011	-
1.3421	820	0.0029	-
1.3748	840	0.0023	-
1.4075	860	0.0019	-
1.4403	880	0.0016	-
1.4730	900	0.0022	-
1.5057	920	0.0015	-
1.5385	940	0.0012	-
1.5712	960	0.0014	-
1.6039	980	0.0012	-
1.6367	1000	0.0019	-
1.6694	1020	0.0015	-
1.7021	1040	0.0015	-
1.7349	1060	0.0008	-
1.7676	1080	0.0004	-
1.8003	1100	0.0014	-
1.8331	1120	0.0015	-
1.8658	1140	0.0011	-
1.8985	1160	0.0008	-
1.9313	1180	0.0021	-
1.9640	1200	0.0011	-
1.9967	1220	0.0023	-
2.0	1222	-	0.1374
0.0016	1	0.0009	-
0.0327	20	0.0022	-
0.0655	40	0.0015	-
0.0982	60	0.0022	-
0.1309	80	0.0007	-
0.1637	100	0.0015	-
0.1964	120	0.0011	-
0.2291	140	0.0019	-
0.2619	160	0.0011	-
0.2946	180	0.0014	-
0.3273	200	0.0011	-
0.3601	220	0.0007	-
0.3928	240	0.0014	-
0.4255	260	0.0011	-
0.4583	280	0.0024	-
0.4910	300	0.0011	-
0.5237	320	0.0006	-
0.5565	340	0.0003	-
0.5892	360	0.0003	-
0.6219	380	0.001	-
0.6547	400	0.0006	-
0.6874	420	0.0014	-
0.7201	440	0.001	-
0.7529	460	0.001	-
0.7856	480	0.0006	-
0.8183	500	0.0016	-
0.8511	520	0.0007	-
0.8838	540	0.0011	-
0.9165	560	0.0005	-
0.9493	580	0.0008	-
0.9820	600	0.0004	-
1.0	611	-	0.1352
1.0147	620	0.0003	-
1.0475	640	0.0003	-
1.0802	660	0.0003	-
1.1129	680	0.0003	-
1.1457	700	0.0003	-
1.1784	720	0.0003	-
1.2111	740	0.0002	-
1.2439	760	0.0003	-
1.2766	780	0.0003	-
1.3093	800	0.0002	-
1.3421	820	0.0003	-
1.3748	840	0.0003	-
1.4075	860	0.0003	-
1.4403	880	0.0003	-
1.4730	900	0.0003	-
1.5057	920	0.0003	-
1.5385	940	0.0003	-
1.5712	960	0.0003	-
1.6039	980	0.0003	-
1.6367	1000	0.0003	-
1.6694	1020	0.0002	-
1.7021	1040	0.0002	-
1.7349	1060	0.0002	-
1.7676	1080	0.0002	-
1.8003	1100	0.0002	-
1.8331	1120	0.0002	-
1.8658	1140	0.0002	-
1.8985	1160	0.0003	-
1.9313	1180	0.0003	-
1.9640	1200	0.0002	-
1.9967	1220	0.0003	-
2.0	1222	-	0.1392

Framework Versions

Python: 3.12.13
SetFit: 1.1.3
Sentence Transformers: 3.4.1
Transformers: 4.57.6
PyTorch: 2.11.0+cu128
Datasets: 5.0.0
Tokenizers: 0.22.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}