Commit
•
0867139
1
Parent(s):
7a114a3
added data set and loraconfig details
Browse files
README.md
CHANGED
|
@@ -1,8 +1,25 @@
|
|
| 1 |
---
|
| 2 |
library_name: peft
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
## Training procedure
|
| 5 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
### Framework versions
|
| 7 |
|
| 8 |
|
|
|
|
| 1 |
---
|
| 2 |
library_name: peft
|
| 3 |
---
|
| 4 |
+
## Dataset procedure
|
| 5 |
+
- Dataset used: /tweets_hate_speech_detection
|
| 6 |
+
- size: 3196 (only 10% of dataset used)
|
| 7 |
+
- batch_size = 32
|
| 8 |
+
- num_epochs = 20
|
| 9 |
+
- learning_rate = 3e-4
|
| 10 |
+
- num_warmup_steps = 0.06 * (3196 * num_epochs)
|
| 11 |
+
- num_training_steps = (3196 * num_epochs)
|
| 12 |
+
|
| 13 |
## Training procedure
|
| 14 |
|
| 15 |
+
|
| 16 |
+
## LoraConfig procedure
|
| 17 |
+
r=8, #attention heads
|
| 18 |
+
lora_alpha=16, #alpha scaling
|
| 19 |
+
lora_dropout=0.1,
|
| 20 |
+
bias="none",
|
| 21 |
+
task_type="SEQ_CLS" # set this for CLM or Seq2Seq
|
| 22 |
+
|
| 23 |
### Framework versions
|
| 24 |
|
| 25 |
|