mosaicml
/

mpt-7b

Text Generation

StreamingDatasets

text-generation-inference

Model card Files Files and versions Community

jfrankle commited on May 5, 2023

Commit

ac0e76a

•

1 Parent(s): 0a3263c

Update README.md

Files changed (1) hide show

README.md +3 -3

README.md CHANGED Viewed

@@ -140,13 +140,13 @@ The model was trained for 1T tokens (with batch size 1760 and sequence length 20
 | Data Source | Number of Tokens in Source | Proportion | Effective Number of Tokens | Epochs |
 |-------------|----------------------------|------------|----------------------------|--------|
 | mC4 3.1.0 - English | 417.99 B | 0.33 | 330 B | 0.14 |
-| C4 - English - SemDedup 80% | 100.42 B | 0.294 | 294 B | 2.93 |
 | RedPajama - CommonCrawl | 878.45 B | 0.1 | 100 B | 0.11 |
 | The Stack - Selected Languages | 463.78 B | 0.1 | 100 B | 0.22 |
 | RedPajama - Wikipedia - En | 4.87 B | 0.04 | 40 B | 8.21 |
 | The Stack - Markdown | 107.07 B | 0.035 | 35 B | 0.33 |
-| S2ORC | 48.85 B | 0.035 | 35 B | 0.72 |
-| RedPajama - Books | 26.02 B | 0.033 | 33B | 1.27 |
 | RedPajama - arXiv | 28.10 B | 0.019 | 19 B | 0.68 |
 | RedPajama - StackExchange | 20.54 B | 0.014 | 14 B |0.68 |

 | Data Source | Number of Tokens in Source | Proportion | Effective Number of Tokens | Epochs |
 |-------------|----------------------------|------------|----------------------------|--------|
 | mC4 3.1.0 - English | 417.99 B | 0.33 | 330 B | 0.14 |
+| C4 - English - SemDedup 80% | 100.42 B | 0.299 | 299 B | 2.98 |
 | RedPajama - CommonCrawl | 878.45 B | 0.1 | 100 B | 0.11 |
 | The Stack - Selected Languages | 463.78 B | 0.1 | 100 B | 0.22 |
 | RedPajama - Wikipedia - En | 4.87 B | 0.04 | 40 B | 8.21 |
 | The Stack - Markdown | 107.07 B | 0.035 | 35 B | 0.33 |
+| S2ORC | 48.85 B | 0.033 | 33 B | 0.68 |
+| RedPajama - Books | 26.02 B | 0.03 | 30B | 1.15 |
 | RedPajama - arXiv | 28.10 B | 0.019 | 19 B | 0.68 |
 | RedPajama - StackExchange | 20.54 B | 0.014 | 14 B |0.68 |