SetFit with codefuse-ai/F2LLM-v2-80M

This is a SetFit model that can be used for Text Classification. This SetFit model uses codefuse-ai/F2LLM-v2-80M as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: codefuse-ai/F2LLM-v2-80M
Classification head: a LogisticRegression instance
Maximum Sequence Length: 40960 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
positive	'The nuclear crisis demands immediate action.' 'Public anger grows as energy companies post record profits while households struggle.' 'In a televised address, Russian President Vladimir Putin warned that if NATO continues to supply Ukraine with advanced weaponry, Russia will have no choice but to expand its military operations beyond Ukrainian borders.'
negative	'Our economy has demonstrated remarkable consistency in its performance, with quarter after quarter of steady growth, stable prices, and improving living standards for the population, creating an environment of economic certainty that businesses and investors both appreciate and reward with continued commitment.' 'A midlife crisis is often characterized by a desire for change.' 'Export volumes have increased for the third consecutive quarter, demonstrating strong international demand.'

Label

Examples

positive

'The nuclear crisis demands immediate action.'
'Public anger grows as energy companies post record profits while households struggle.'
'In a televised address, Russian President Vladimir Putin warned that if NATO continues to supply Ukraine with advanced weaponry, Russia will have no choice but to expand its military operations beyond Ukrainian borders.'

negative

'Our economy has demonstrated remarkable consistency in its performance, with quarter after quarter of steady growth, stable prices, and improving living standards for the population, creating an environment of economic certainty that businesses and investors both appreciate and reward with continued commitment.'
'A midlife crisis is often characterized by a desire for change.'
'Export volumes have increased for the third consecutive quarter, demonstrating strong international demand.'

Evaluation

Metrics

Label	F1_Macro	F1_Binary
all	0.9345	0.9273

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("fefofico/crisis_trained_f2llm_5ep_1e-5_5e-3")
# Run inference
preds = model("Emergency meetings are being held, but solutions remain elusive as the crisis deepens.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	3	15.7035	56

Label	Training Sample Count
negative	808
positive	676

Training Hyperparameters

batch_size: (256, 256)
num_epochs: (5, 5)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (1e-06, 1e-06)
head_learning_rate: 0.003
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0043	1	0.3918	-
0.0862	20	0.3979	-
0.1724	40	0.3983	-
0.2586	60	0.3653	-
0.3448	80	0.344	-
0.4310	100	0.3002	-
0.5172	120	0.2741	-
0.6034	140	0.2519	-
0.6897	160	0.2428	-
0.7759	180	0.2342	-
0.8621	200	0.2227	-
0.9483	220	0.1991	-
1.0	232	-	0.1802
1.0345	240	0.1653	-
1.1207	260	0.1322	-
1.2069	280	0.1038	-
1.2931	300	0.0778	-
1.3793	320	0.0593	-
1.4655	340	0.0426	-
1.5517	360	0.0269	-
1.6379	380	0.0194	-
1.7241	400	0.0125	-
1.8103	420	0.0107	-
1.8966	440	0.0097	-
1.9828	460	0.0067	-
2.0	464	-	0.0954
2.0690	480	0.006	-
2.1552	500	0.0047	-
2.2414	520	0.0051	-
2.3276	540	0.0052	-
2.4138	560	0.0035	-
2.5	580	0.003	-
2.5862	600	0.004	-
2.6724	620	0.0035	-
2.7586	640	0.0032	-
2.8448	660	0.0031	-
2.9310	680	0.0037	-
3.0	696	-	0.0972
3.0172	700	0.0027	-
3.1034	720	0.0032	-
3.1897	740	0.0033	-
3.2759	760	0.0022	-
3.3621	780	0.0022	-
3.4483	800	0.0028	-
3.5345	820	0.0017	-
3.6207	840	0.0017	-
3.7069	860	0.0023	-
3.7931	880	0.0023	-
3.8793	900	0.0024	-
3.9655	920	0.003	-
4.0	928	-	0.0982
4.0517	940	0.0031	-
4.1379	960	0.0022	-
4.2241	980	0.0019	-
4.3103	1000	0.0016	-
4.3966	1020	0.002	-
4.4828	1040	0.0019	-
4.5690	1060	0.0028	-
4.6552	1080	0.0019	-
4.7414	1100	0.0024	-
4.8276	1120	0.0022	-
4.9138	1140	0.0021	-
5.0	1160	0.0026	0.0990

Framework Versions

Python: 3.12.13
SetFit: 1.1.3
Sentence Transformers: 3.4.1
Transformers: 4.57.6
PyTorch: 2.11.0+cu128
Datasets: 5.0.0
Tokenizers: 0.22.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}