Update README.md
Browse files
README.md
CHANGED
|
@@ -68,6 +68,7 @@ Total documents: 10,669,024
|
|
| 68 |
|
| 69 |
- **Training regime:**
|
| 70 |
- bf16
|
|
|
|
| 71 |
- per device batch size 16, global batch size 524288, gradient accumulation 16
|
| 72 |
- zero stage 1
|
| 73 |
- lr 3e-4, cosine schedule, 700 warmup steps
|
|
|
|
| 68 |
|
| 69 |
- **Training regime:**
|
| 70 |
- bf16
|
| 71 |
+
- context length 1024
|
| 72 |
- per device batch size 16, global batch size 524288, gradient accumulation 16
|
| 73 |
- zero stage 1
|
| 74 |
- lr 3e-4, cosine schedule, 700 warmup steps
|