SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/paraphrase-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: sentence-transformers/paraphrase-mpnet-base-v2
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 25 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
do_not_qualify	"Your target age group doesn't include me." "I'm outside the age range for this." "I'm not in the age group you're looking for."
can_you_email	'I prefer email, can you write to me?' 'Email is more convenient for me, can you use that?' 'Can you send me the details by email?'
say_again	'Can you repeat that, please?' 'I missed that, can you say it again?' 'Could you please repeat what you just said?'
hold_a_sec	'One moment, please hold.' 'Hang on for a bit, please.' 'Just a minute, please.'
language_barrier	'English is hard for me, ¿puedo hablar en español?' 'I struggle with English, ¿puede ser en español?' "I'm more comfortable in Spanish, ¿podemos continuar en español?"
decline	'wrong' 'Never' "I don't want this, thank you."
transfer_request	'Can you transfer this call to your superior?' 'I need to speak with someone in charge.' 'Can I speak with your manager?'
scam	"I'm skeptical, this doesn't sound right." "I'm wary, this feels like a scam." "Are you sure this isn't a scam?"
who_are_you	"I would like to know who's calling." "Who's calling, please?" 'Who are you and why are you calling?'
where_did_you_get_my_info	'Can you explain how you got my contact info?' "What's the source of my details you have?" "I didn't give you my number, where did you get it?"
do_not_call	"Stop calling me, it's annoying!" "I don't want to be contacted again." "Enough calls, I'm not interested!"
where_are_you_calling_from	'Where are you calling from?' 'From which city or country are you calling?' 'Could you inform me of your current location?'
complain_calls	"Too many calls like this, it's irritating." "I've had several calls like this, it's annoying." "I keep getting these calls, it's too much."
busy	"Right now isn't good, I'm busy with something." "I'm swamped at the moment, sorry." "I'm busy right now, can't talk."
greetings	'Hi, how can I help you?' 'Hello, what can I help you with today?' 'Hello, yes?'
sorry_greeting	"I'm not at my best, what do you need?" "Sorry, it's a bad time, I'm sick." "Not a great time, I'm dealing with a personal issue."
GreetBack	'Doing well, how about yourself?' 'Pretty good, what about you?' "Not bad, and how's it going on your end?"
calling_about	'Why are you calling me?' "What's the matter, why the call?" 'May I know the reason for your call?'
answering_machine	"Leave a message and I'll get back to you." "You're speaking to an answering machine, leave a message." "This is an answering machine, I'm not available."
weather	"Sunny skies here, what's it like where you are?" "It's a bit cloudy here, is it the same there?" "It's warm here, what about where you are?"
are_you_bot	'Is this a bot calling me?' 'Is this a recorded message or are you real?' 'Are you a live person or a recording?'
affirmation	'yes' "That's true, yes." "Precisely, that's right."
not_interested	"This doesn't interest me, sorry." "This offer isn't relevant to my interests." "Thanks, but this isn't something I need."
already	"I've made this purchase before." "This isn't new to me, I have it already." "I've been using this for a while now."
abusibve	"This is unacceptable, I won't tolerate this!" 'I demand you stop this abusive calling!' "Stop calling me, it's harassment!"

Evaluation

Metrics

Label	Accuracy
all	0.9333

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("Give me a second, please.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	1	6.8375	13

Label	Training Sample Count
GreetBack	9
abusibve	9
affirmation	10
already	10
answering_machine	8
are_you_bot	8
busy	9
calling_about	8
can_you_email	11
complain_calls	11
decline	10
do_not_call	12
do_not_qualify	9
greetings	8
hold_a_sec	8
language_barrier	10
not_interested	11
say_again	12
scam	9
sorry_greeting	9
transfer_request	8
weather	10
where_are_you_calling_from	9
where_did_you_get_my_info	11
who_are_you	11

Training Hyperparameters

batch_size: (8, 8)
num_epochs: (3, 3)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (2e-05, 2e-05)
head_learning_rate: 2e-05
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0008	1	0.1054	-
0.0417	50	0.1111	-
0.0833	100	0.0798	-
0.125	150	0.0826	-
0.1667	200	0.0308	-
0.2083	250	0.0324	-
0.25	300	0.0607	-
0.2917	350	0.0042	-
0.3333	400	0.0116	-
0.375	450	0.0049	-
0.4167	500	0.0154	-
0.4583	550	0.0158	-
0.5	600	0.0036	-
0.5417	650	0.001	-
0.5833	700	0.0015	-
0.625	750	0.0012	-
0.6667	800	0.0009	-
0.7083	850	0.0008	-
0.75	900	0.0008	-
0.7917	950	0.0014	-
0.8333	1000	0.0005	-
0.875	1050	0.0027	-
0.9167	1100	0.0007	-
0.9583	1150	0.0008	-
1.0	1200	0.0012	-
1.0417	1250	0.0012	-
1.0833	1300	0.0006	-
1.125	1350	0.0005	-
1.1667	1400	0.0003	-
1.2083	1450	0.0012	-
1.25	1500	0.0006	-
1.2917	1550	0.0008	-
1.3333	1600	0.0008	-
1.375	1650	0.0003	-
1.4167	1700	0.0004	-
1.4583	1750	0.0005	-
1.5	1800	0.0004	-
1.5417	1850	0.0004	-
1.5833	1900	0.0008	-
1.625	1950	0.0004	-
1.6667	2000	0.0004	-
1.7083	2050	0.0021	-
1.75	2100	0.0004	-
1.7917	2150	0.0002	-
1.8333	2200	0.0006	-
1.875	2250	0.0004	-
1.9167	2300	0.0006	-
1.9583	2350	0.0006	-
2.0	2400	0.0003	-
2.0417	2450	0.0002	-
2.0833	2500	0.0002	-
2.125	2550	0.0003	-
2.1667	2600	0.0004	-
2.2083	2650	0.0004	-
2.25	2700	0.0005	-
2.2917	2750	0.0005	-
2.3333	2800	0.0005	-
2.375	2850	0.0007	-
2.4167	2900	0.0002	-
2.4583	2950	0.0003	-
2.5	3000	0.0004	-
2.5417	3050	0.0002	-
2.5833	3100	0.0004	-
2.625	3150	0.0002	-
2.6667	3200	0.0002	-
2.7083	3250	0.0003	-
2.75	3300	0.0002	-
2.7917	3350	0.0002	-
2.8333	3400	0.0003	-
2.875	3450	0.0002	-
2.9167	3500	0.0002	-
2.9583	3550	0.0002	-
3.0	3600	0.0002	-

Framework Versions

Python: 3.10.13
SetFit: 1.0.1
Sentence Transformers: 2.2.2
Transformers: 4.35.0
PyTorch: 2.1.0
Datasets: 2.14.6
Tokenizers: 0.14.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

m-aliabbas1
/

idrakFE_Fewshot