Commit
•
6bbf3e4
1
Parent(s):
b74a303
Update README.md
Browse files
README.md
CHANGED
@@ -49,23 +49,21 @@ To enhance output quality and thematic consistency, custom stopping strings incl
|
|
49 |
- "\n"
|
50 |
|
51 |
## Training Hyperparameters and Fine-Tuning Details:
|
52 |
-
|
53 |
-
|
54 |
-
|
55 |
-
|
56 |
-
|
57 |
-
|
58 |
-
|
59 |
-
|
60 |
-
|
61 |
-
|
62 |
-
|
63 |
-
|
64 |
-
|
65 |
-
|
66 |
-
|
67 |
-
train_loss: 1.7511341453808817
|
68 |
-
\`\`\`
|
69 |
|
70 |
## Testing and Evaluation:
|
71 |
During the testing phase, we conducted a series of evaluations to compare Llama3-Pirate-Talk-8b-v0.1 against the base Llama3 model. These tests involved complex navigational and general knowledge questions designed to assess the model's ability to maintain its thematic integrity while responding accurately to technically demanding prompts. The model demonstrated a strong thematic presence with consistent use of pirate vernacular. However, it showed limitations in handling high-precision technical content, which is an expected trade-off given its thematic specialization. These insights have been instrumental in identifying areas for further model refinement.
|
|
|
49 |
- "\n"
|
50 |
|
51 |
## Training Hyperparameters and Fine-Tuning Details:
|
52 |
+
- micro_batch_size: 1
|
53 |
+
- batch_size: 0
|
54 |
+
- epochs: 1
|
55 |
+
- learning_rate: "2e-5"
|
56 |
+
- lr_scheduler_type: "linear"
|
57 |
+
- lora_rank: 8
|
58 |
+
- lora_alpha: 16
|
59 |
+
- lora_dropout: 0.05
|
60 |
+
- cutoff_len: 256
|
61 |
+
- warmup_steps: 8
|
62 |
+
- optimizer: "adamw_torch"
|
63 |
+
- grad_accumulation: 1
|
64 |
+
- train_runtime: 1697.081 seconds
|
65 |
+
- total_flos: 1.3663655883177984e+16
|
66 |
+
- train_loss: 1.7511341453808817
|
|
|
|
|
67 |
|
68 |
## Testing and Evaluation:
|
69 |
During the testing phase, we conducted a series of evaluations to compare Llama3-Pirate-Talk-8b-v0.1 against the base Llama3 model. These tests involved complex navigational and general knowledge questions designed to assess the model's ability to maintain its thematic integrity while responding accurately to technically demanding prompts. The model demonstrated a strong thematic presence with consistent use of pirate vernacular. However, it showed limitations in handling high-precision technical content, which is an expected trade-off given its thematic specialization. These insights have been instrumental in identifying areas for further model refinement.
|