metadata

library_name: setfit
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
datasets:
  - konsman/setfit-messages-optimized
metrics:
  - f1
  - accuracy
widget:
  - text: >-
      Tomato sauce is acidic and causes problems with my reflux.  That, in turn,
      irritates the vagus nerve and may bring on arrthymia.  I can't eat tomato
      soup anymore.  It could be a trigger for that reason.
  - text: >-
      pednisone is synthetic cortisol hormone naturally produced by our adrenal
      glands , a powerful anti inflamatory , it works by reducing swelling . I
      recommend reading the book "adrenal fatigue - the 21st century health
      syndrome" , the doc says all allergies , frequent respiratory tract
      infections and asthma have an underlying  cause that is adrenal fatigue..
  - text: >-
      You may want to read about Nigella sativa.  It is helpful for many
      conditions, and studies have been done showing it to be beneficial at
      reducing inflammation of ulcerative colitis.  It is also generally good
      for preventing many diseases, including cancer.  Also hemorrhoids.   
  - text: >-
      Sorry forgot to say that unfortunately after this problem that made me let
      sports and with the anxiety meds . I am now 83 kg
  - text: >-
      6 months pregnant had an abnormal pap, doctor did a biopsy and came back
      as cis what is this how serious and what's the cause? I have to have a
      leep after my son comes, what does this entail? Doc not good at explaining
      anything
pipeline_tag: text-classification
inference: false
base_model: sentence-transformers/paraphrase-mpnet-base-v2
model-index:
  - name: SetFit with sentence-transformers/paraphrase-mpnet-base-v2
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: konsman/setfit-messages-optimized
          type: konsman/setfit-messages-optimized
          split: test
        metrics:
          - type: f1
            value: 0.6896901980700864
            name: F1
          - type: accuracy
            value: 0.3403755868544601
            name: Accuracy

SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a SetFit model trained on the konsman/setfit-messages-optimized dataset that can be used for Text Classification. This SetFit model uses sentence-transformers/paraphrase-mpnet-base-v2 as the Sentence Transformer embedding model. A MultiOutputClassifier instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: sentence-transformers/paraphrase-mpnet-base-v2
Classification head: a MultiOutputClassifier instance
Maximum Sequence Length: 512 tokens
Training Dataset: konsman/setfit-messages-optimized

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Evaluation

Metrics

Label	F1	Accuracy
all	0.6897	0.3404

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("konsman/setfit-messages-multilabel-example")
# Run inference
preds = model("Sorry forgot to say that unfortunately after this problem that made me let sports and with the anxiety meds . I am now 83 kg")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	5	110.2344	469

Training Hyperparameters

batch_size: (8, 8)
num_epochs: (2, 2)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 5
body_learning_rate: (2e-05, 2e-05)
head_learning_rate: 2e-05
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0031	1	0.3209	-
0.1562	50	0.1823	-
0.3125	100	0.1003	-
0.4688	150	0.1774	-
0.625	200	0.0832	-
0.7812	250	0.0828	-
0.9375	300	0.0721	-
1.0938	350	0.1331	-
1.25	400	0.1215	-
1.4062	450	0.1494	-
1.5625	500	0.0444	-
1.7188	550	0.0688	-
1.875	600	0.1033	-
0.0125	1	0.0508	-
0.625	50	0.0793	-
1.25	100	0.081	-
1.875	150	0.1367	-

Framework Versions

Python: 3.10.12
SetFit: 1.0.2
Sentence Transformers: 2.2.2
Transformers: 4.35.2
PyTorch: 2.1.0+cu121
Datasets: 2.16.1
Tokenizers: 0.15.0

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}