diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -189,7 +189,7 @@ model-index: split: test metrics: - type: f1 - value: 0.639905820632589 + value: 0.7293011911579894 name: F1 --- @@ -221,58 +221,58 @@ The model has been trained using an efficient few-shot learning technique that i - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit) ### Model Labels -| Label | Examples | -|:------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 5 | | -| 17 | | -| 0 | | -| 15 | | -| 29 | | -| 28 | | -| 16 | | -| 40 | | -| 30 | | -| 10 | | -| 37 | | -| 4 | | -| 26 | | -| 20 | | -| 13 | | -| 33 | | -| 7 | | -| 3 | | -| 21 | | -| 32 | | -| 19 | | -| 36 | | -| 42 | | -| 2 | | -| 39 | | -| 27 | | -| 24 | | -| 9 | | -| 8 | | -| 25 | | -| 34 | | -| 23 | | -| 12 | | -| 31 | | -| 38 | | -| 6 | | -| 18 | | -| 14 | | -| 11 | | -| 41 | | -| 22 | | -| 35 | | -| 1 | | +| Label | Examples | +|:------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 35 | | +| 37 | | +| 12 | | +| 20 | | +| 21 | | +| 22 | | +| 25 | | +| 39 | | +| 15 | | +| 26 | | +| 2 | | +| 18 | | +| 7 | | +| 23 | | +| 30 | | +| 8 | | +| 17 | | +| 0 | | +| 36 | | +| 31 | | +| 14 | | +| 40 | | +| 28 | | +| 19 | | +| 4 | | +| 29 | | +| 27 | | +| 6 | | +| 9 | | +| 16 | | +| 42 | | +| 32 | | +| 1 | | +| 38 | | +| 13 | | +| 41 | | +| 24 | | +| 10 | | +| 5 | | +| 33 | | +| 11 | | +| 3 | | +| 34 | | ## Evaluation ### Metrics | Label | F1 | |:--------|:-------| -| **all** | 0.6399 | +| **all** | 0.7293 | ## Uses @@ -324,57 +324,57 @@ preds = model("##rch procedure that evaluates the objective function p x display ### Training Set Metrics | Training set | Min | Median | Max | |:-------------|:----|:---------|:----| -| Word count | 2 | 375.0186 | 509 | +| Word count | 1 | 369.9960 | 509 | | Label | Training Sample Count | |:------|:----------------------| -| 0 | 10 | -| 1 | 10 | -| 2 | 10 | -| 3 | 10 | -| 4 | 10 | -| 5 | 10 | -| 6 | 10 | -| 7 | 10 | -| 8 | 10 | -| 9 | 10 | -| 10 | 10 | -| 11 | 10 | -| 12 | 10 | -| 13 | 10 | -| 14 | 10 | -| 15 | 10 | -| 16 | 10 | -| 17 | 10 | -| 18 | 10 | -| 19 | 10 | -| 20 | 10 | -| 21 | 10 | -| 22 | 10 | -| 23 | 10 | -| 24 | 10 | -| 25 | 10 | -| 26 | 10 | -| 27 | 10 | -| 28 | 10 | -| 29 | 10 | -| 30 | 10 | -| 31 | 10 | -| 32 | 10 | -| 33 | 10 | -| 34 | 10 | -| 35 | 10 | -| 36 | 10 | -| 37 | 10 | -| 38 | 10 | -| 39 | 10 | -| 40 | 10 | -| 41 | 10 | -| 42 | 10 | +| 0 | 100 | +| 1 | 100 | +| 2 | 100 | +| 3 | 100 | +| 4 | 100 | +| 5 | 100 | +| 6 | 100 | +| 7 | 100 | +| 8 | 100 | +| 9 | 100 | +| 10 | 100 | +| 11 | 100 | +| 12 | 100 | +| 13 | 100 | +| 14 | 100 | +| 15 | 100 | +| 16 | 100 | +| 17 | 100 | +| 18 | 100 | +| 19 | 100 | +| 20 | 100 | +| 21 | 100 | +| 22 | 100 | +| 23 | 100 | +| 24 | 100 | +| 25 | 100 | +| 26 | 100 | +| 27 | 100 | +| 28 | 100 | +| 29 | 100 | +| 30 | 100 | +| 31 | 100 | +| 32 | 100 | +| 33 | 100 | +| 34 | 100 | +| 35 | 100 | +| 36 | 100 | +| 37 | 100 | +| 38 | 100 | +| 39 | 100 | +| 40 | 100 | +| 41 | 100 | +| 42 | 100 | ### Training Hyperparameters - batch_size: (16, 16) -- num_epochs: (1, 4) +- num_epochs: (2, 4) - max_steps: -1 - sampling_strategy: oversampling - num_iterations: 20 @@ -392,11 +392,32 @@ preds = model("##rch procedure that evaluates the objective function p x display - load_best_model_at_end: True ### Training Results -| Epoch | Step | Training Loss | Validation Loss | -|:------:|:----:|:-------------:|:---------------:| -| 0.0009 | 1 | 0.2819 | - | -| 0.9302 | 1000 | 0.0029 | - | - +| Epoch | Step | Training Loss | Validation Loss | +|:----------:|:---------:|:-------------:|:---------------:| +| 0.0001 | 1 | 0.3414 | - | +| 0.0930 | 1000 | 0.0466 | - | +| 0.1860 | 2000 | 0.0861 | - | +| 0.2791 | 3000 | 0.0413 | - | +| 0.3721 | 4000 | 0.0247 | - | +| 0.4651 | 5000 | 0.0025 | - | +| 0.5581 | 6000 | 0.0029 | - | +| 0.6512 | 7000 | 0.0008 | - | +| 0.7442 | 8000 | 0.0006 | - | +| 0.8372 | 9000 | 0.0007 | - | +| **0.9302** | **10000** | **0.0599** | **0.1484** | +| 1.0233 | 11000 | 0.0013 | - | +| 1.1163 | 12000 | 0.0009 | - | +| 1.2093 | 13000 | 0.0572 | - | +| 1.3023 | 14000 | 0.0009 | - | +| 1.3953 | 15000 | 0.0001 | - | +| 1.4884 | 16000 | 0.0018 | - | +| 1.5814 | 17000 | 0.0002 | - | +| 1.6744 | 18000 | 0.0054 | - | +| 1.7674 | 19000 | 0.0001 | - | +| 1.8605 | 20000 | 0.0001 | 0.1641 | +| 1.9535 | 21000 | 0.0002 | - | + +* The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.12 - SetFit: 1.0.3