vishwa1004/chatbot_theme_map

SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/paraphrase-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: sentence-transformers/paraphrase-mpnet-base-v2
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 8 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
Movement	'Fellowship network connecting volunteers working across the education sector.' 'Champion program recruiting teachers to advocate for policy change.' 'Field-building initiative connecting practitioners across different states.'
Foundational Learning	'Basic numeracy worksheets designed for early primary students struggling with number recognition.' 'Bridge course materials to help out-of-school children catch up on basic reading and arithmetic.' 'A remedial reading intervention for grade 3 students performing below grade level.'
School Governance	'Feedback mechanism for teachers to raise concerns with school leadership.' 'Checklist for school leaders to conduct internal accountability reviews.' 'Guide to preparing an annual school budget and expenditure tracker.'
Teaching and Learning Practises	'Teacher motivation and wellbeing program addressing professional burnout.' 'Subject-specific teaching resources for secondary-level science and social studies.' 'Skill-readiness workshops integrated into regular classroom teaching.'
System Thinking	'Landscape study of foundational learning interventions across a state.' 'Framework guiding how a state education department should allocate its annual budget across districts.' 'Logical framework for a multi-year systemic education reform programme.'
Community Engagement	'Materials for training SMC members on their responsibilities and rights.' "How can we get more parents involved in their child's schooling?" 'Parent workshop materials on supporting reading practice at home.'
Child Rights	'Guidelines for staff on mandatory reporting of suspected child abuse.' 'Advocacy material addressing child labour and the right to education.' "Workshop materials on children's participation rights in decision-making."
Inclusion	'Guide for adapting worksheets for a child with dyslexia in a mainstream classroom.' 'Support model for first-generation learners from Scheduled Tribe communities.' 'Infrastructure accessibility guide for schools serving children with physical disabilities.'

Evaluation

Metrics

Label	Accuracy
all	0.9

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("Support program for migrant children transitioning into new schools.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	7	10.3729	16

Label	Training Sample Count
Child Rights	13
Community Engagement	14
Foundational Learning	15
Inclusion	13
Movement	14
School Governance	19
System Thinking	16
Teaching and Learning Practises	14

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (4, 4)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 2e-05)
• head_learning_rate: 0.01
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• evaluation_strategy: epoch
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0034	1	0.1985	-
0.1695	50	0.1709	-
0.3390	100	0.1086	-
0.5085	150	0.0267	-
0.6780	200	0.0041	-
0.8475	250	0.0016	-
1.0	295	-	0.0386
1.0169	300	0.0011	-
1.1864	350	0.0008	-
1.3559	400	0.0006	-
1.5254	450	0.0006	-
1.6949	500	0.0005	-
1.8644	550	0.0005	-
2.0	590	-	0.0377
2.0339	600	0.0004	-
2.2034	650	0.0004	-
2.3729	700	0.0004	-
2.5424	750	0.0004	-
2.7119	800	0.0003	-
2.8814	850	0.0003	-
3.0	885	-	0.0370
3.0508	900	0.0003	-
3.2203	950	0.0003	-
3.3898	1000	0.0003	-
3.5593	1050	0.0003	-
3.7288	1100	0.0003	-
3.8983	1150	0.0003	-
4.0	1180	-	0.0372

Framework Versions

• Python: 3.10.12
• SetFit: 1.1.3
• Sentence Transformers: 3.0.1
• Transformers: 4.44.2
• PyTorch: 2.13.0
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}