adding more details to the readme and updating data mix

README.md

@@ -6,33 +6,28 @@ tags:
 - llm-foundry
 - StreamingDatasets
 datasets:
+- allenai/c4
 - mc4
-- c4
 - togethercomputer/RedPajama-Data-1T
-- bigcode/the-stack
+- bigcode/the-stack-dedup
 - allenai/s2orc
-- TODO
 inference: false
 ---
 
 # MPT-30B
 
-MPT-7B is a decoder-style transformer pretrained from scratch on 1T tokens of English text and code.
+MPT-30B is a decoder-style transformer pretrained from scratch on 1T tokens of English text and code.
 This model was trained by [MosaicML](https://www.mosaicml.com).
 
 MPT-30B is part of the family of MosaicPretrainedTransformer (MPT) models, which use a modified transformer architecture optimized for efficient training and inference.
 
-These architectural changes include performance-optimized layer implementations and the elimination of context length limits by replacing
-positional embeddings with Attention with Linear Biases ([ALiBi](https://arxiv.org/abs/2108.12409)).
-Thanks to these modifications, MPT models can be trained with high throughput efficiency and stable convergence.
-MPT models can also be served efficiently with both standard HuggingFace pipelines and NVIDIA's [FasterTransformer](https://github.com/NVIDIA/FasterTransformer).
+MPT-30B comes with special features that differentiate them from other LLMs, including an 8k token context window (which can be further extended via finetuning; see [MPT-7B-StoryWriter](https://huggingface.co/mosaicml/mpt-7b-storywriter)), support for context-length extrapolation via [ALiBi](https://arxiv.org/abs/2108.12409), and efficient inference + training performance via FlashAttention. It also has strong coding abilities thanks to its pretraining mix. MPT models can also be served efficiently with both standard HuggingFace pipelines and NVIDIA's [FasterTransformer](https://github.com/NVIDIA/FasterTransformer).
 
 This model uses the MosaicML LLM codebase, which can be found in the [llm-foundry repository](https://github.com/mosaicml/llm-foundry). It was trained by MosaicML’s NLP team on the [MosaicML platform](https://www.mosaicml.com/training) for LLM pretraining, finetuning, and inference.
 
 ### How is this model different?
 
-MPT-30B is
-
+MPT-30B is:
 * **Licensed for the possibility of commercial use** (unlike [LLaMA](https://arxiv.org/abs/2302.13971)).
 * **Trained on a large amount of data** (1T tokens like [LLaMA](https://arxiv.org/abs/2302.13971) vs. 300B for [Pythia](https://github.com/EleutherAI/pythia), 300B for [OpenLLaMA](https://github.com/openlm-research/open_llama), and 800B for [StableLM](https://github.com/Stability-AI/StableLM)).
 * **Prepared to handle extremely long inputs** thanks to [ALiBi](https://arxiv.org/abs/2108.12409) (TODO: talk about MPT-30B-instruct finetuned on 8k).
@@ -45,7 +40,7 @@ The following models are finetuned on MPT-30B:
 
 * [MPT-30B-Instruct](https://huggingface.co/mosaicml/mpt-30b-instruct): a model for short-form instruction following.
 Built by finetuning MPT-30B on several carefully curated datasets.
-  * [Demo on Hugging Face Spaces](https://huggingface.co/spaces/mosaicml/mpt-30b-instruct) (TODO: will this be a thing?)
+  * [Demo on Hugging Face Spaces](https://huggingface.co/spaces/mosaicml/mpt-30b-instruct)
 
 * [MPT-30B-Chat](https://huggingface.co/mosaicml/mpt-30b-chat): a chatbot-like model for dialogue generation.
 Built by finetuning MPT-30B on [ShareGPT-Vicuna](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered), [Camel-AI](https://huggingface.co/camel-ai),
@@ -55,7 +50,7 @@ Built by finetuning MPT-30B on [ShareGPT-Vicuna](https://huggingface.co/datasets
 
 ## Model Date
 
-TBD
+June 22, 2023
 
 ## Model License
 
@@ -63,7 +58,7 @@ Apache-2.0
 
 ## Documentation
 
-* [Blog post: Introducing MPT-30B: TBD: A New Standard for Open-Source, Commercially Usable LLMs](https://www.mosaicml.com/blog/mpt-30b)
+* [Blog post: MPT-30B: Raising the bar for open-source commercial foundation models](https://www.mosaicml.com/blog/mpt-30b)
 * [Codebase (mosaicml/llm-foundry repo)](https://github.com/mosaicml/llm-foundry/)
 * Questions: Feel free to contact us via the [MosaicML Community Slack](https://mosaicml.me/slack)!
 
@@ -91,8 +86,8 @@ import transformers
 name = 'mosaicml/mpt-30b'
 
 config = transformers.AutoConfig.from_pretrained(name, trust_remote_code=True)
-config.attn_config['attn_impl'] = 'triton'
-config.init_device = 'cuda:0' # For fast initialization directly on GPU!
+config.attn_config['attn_impl'] = 'torch  # change this to use triton
+config.init_device = 'cpu' # For fast initialization directly on GPU if you have enough memory
 
 model = transformers.AutoModelForCausalLM.from_pretrained(
   name,
@@ -102,8 +97,8 @@ model = transformers.AutoModelForCausalLM.from_pretrained(
 )
 ```
 
-Although the model was trained with a sequence length of 8192, ALiBi enables users to increase the maximum sequence length during finetuning and/or inference. For example:
-TODO: check if we want to advertise this.
+The model was trained initially with a sequence length of 4096 with an additional pretraining stage for sequence length adapation up to 8192. However, ALiBi enables users to increase the maximum sequence length even further during finetuning and/or inference. For example:
+
 ```python
 import transformers
 
@@ -119,11 +114,11 @@ model = transformers.AutoModelForCausalLM.from_pretrained(
 )
 ```
 
-This model was trained with the [EleutherAI/gpt-neox-20b](https://huggingface.co/EleutherAI/gpt-neox-20b) tokenizer.
+This model was trained with the MPT-30B tokenizer which is identical to the [EleutherAI/gpt-neox-20b](https://huggingface.co/EleutherAI/gpt-neox-20b) tokenizer.
 
 ```python
 from transformers import AutoTokenizer
-tokenizer = AutoTokenizer.from_pretrained('EleutherAI/gpt-neox-20b')
+tokenizer = AutoTokenizer.from_pretrained('mosaicml/mpt-30b')
 ```
 
 The model can then be used, for example, within a text-generation pipeline.  
@@ -132,8 +127,13 @@ Note: when running Torch modules in lower precision, it is best practice to use
 ```python
 from transformers import pipeline
 
-pipe = pipeline('text-generation', model=model, tokenizer=tokenizer, device='cuda:0')
+with torch.autocast('cuda', dtype=torch.bfloat16):
+    inputs = tokenizer('Here is a recipe for vegan banana bread:\n', return_tensors="pt").to('cuda')
+    outputs = model.generate(**inputs, max_new_tokens=100)
+    print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
 
+# or using the HF pipeline
+pipe = pipeline('text-generation', model=model, tokenizer=tokenizer, device='cuda:0')
 with torch.autocast('cuda', dtype=torch.bfloat16):
     print(
         pipe('Here is a recipe for vegan banana bread:\n',
@@ -173,25 +173,22 @@ StreamingDataset obviates the need to download the whole dataset before starting
 
 ### Data Mix
 
-The model was trained for 1T tokens (with batch size TBD and sequence length TBD). It was trained on the following data mix:
+The model was trained for 1T tokens (with batch size TBD). It was trained on the following data mix:
 
-(TODO: update this)
 | Data Source | Number of Tokens in Source | Proportion | Effective Number of Tokens | Epochs |
 |-------------|----------------------------|------------|----------------------------|--------|
-| mC4 3.1.0 - English | 417.99 B | 0.33 | 330 B | 0.14 |
-| C4 - English - SemDedup 80% | 100.42 B | 0.299 | 299 B | 2.98 |
-| RedPajama - CommonCrawl | 878.45 B | 0.1 | 100 B | 0.11 |
-| The Stack - Selected Languages | 463.78 B | 0.1 | 100 B | 0.22 |
-| RedPajama - Wikipedia - En | 4.87 B | 0.04 | 40 B | 8.21 |
-| The Stack - Markdown | 107.07 B | 0.035 | 35 B | 0.33 |
-| S2ORC | 48.85 B | 0.033 | 33 B | 0.68 |
-| RedPajama - Books | 26.02 B | 0.03 | 30B | 1.15 |
-| RedPajama - arXiv | 28.10 B | 0.019 | 19 B | 0.68 |
-| RedPajama - StackExchange | 20.54 B | 0.014 | 14 B |0.68 |
-
-Samples for each batch were selected from one of the datasets with the probability specified above.
-(TODO: check with @sam whether only FT was on 8k)
-The examples were shuffled within each dataset, and each example was constructed from as many sequences from that dataset as were necessary to fill the 8192 sequence length.
+| mC4 3.1.0 - English (200+ words) | 2417.99 B | 33.50% | 335 B | 0.14 |
+| c4 - English - SemDedup 80% | 100.42 B | 29.90% | 299 B | 2.98 |
+| RedPajama - CommonCrawl | 878.45 B | 8.50% | 85 B | 0.097 |
+| The Stack - Selected Languages | 463.78 B | 10.00% | 100 B | 0.22 |
+| RedPajama - Wikipedia | 4.87 B | 4.00% | 40 B | 8.21 |
+| The Stack - Markdown | 107.07 B | 4.50% | 45 B | 0.42 |
+| Semantic Scholar ORC | 48.95 B | 3.30% | 33 B | 0.67 |
+| RedPajama - Books | 26.02 B | 3.00% | 30 B | 1.15 |
+| RedPajama - arXiv | 28.10 B | 1.90% | 19 B | 0.68 |
+| RedPajama - StackExchange | 20.54 B | 1.40% | 14 B |0.68 |
+
+Samples for each batch were selected from one of the datasets with the probability specified above. The examples were shuffled within each dataset, and each example was constructed from as many sequences from that dataset as were necessary to fill the sequence length. To build 8k support into MPT-30B efficiently, we first pre-trained on 1T tokens using sequences that were 2k tokens long, and then trained for an additional 50B tokens using sequences that were 8k tokens long.
 
 The data was tokenized using the [EleutherAI/gpt-neox-20b](https://huggingface.co/EleutherAI/gpt-neox-20b) tokenizer. This BPE tokenizer has a number of desirable characteristics,
 most of which are relevant for tokenizing code:
@@ -199,12 +196,14 @@ most of which are relevant for tokenizing code:
 (2) It applies consistent space delimitation, unlike the GPT2 tokenizer which tokenizes inconsistently depending on the presence of prefix spaces
 (3) It contains tokens for repeated space characters, which allows superior compression of text with large amounts of repeated space characters.
 
-The model vocabulary size of 50432 was set to be a multiple of 128 (as in [MEGATRON-LM](https://arxiv.org/abs/1909.08053)), model flop utilization (MFU) increased by up to four percentage points.
+The model vocabulary size of 50432 was set to be a multiple of 128 (as in [MEGATRON-LM](https://arxiv.org/abs/1909.08053)).
 
 ### Training Configuration
 
-(TODO: get these details from @abhi and the others)
-This model was trained on 440 A100-40GBs for about 9.5 days using the [MosaicML Platform](https://www.mosaicml.com/platform).
+The model was trained in three stages using the [MosaicML Platform](https://www.mosaicml.com/platform):
+(i) First it was trained on 440 A100-40GBs with a batch size of 1760.
+(ii) Then, on 216 A100-40GBs with a batch size of 1728.
+(iii) Training was completed on 256 H100-80GBs with a batch size of 512 with 8k context length and 50B tokens.
 The model was trained with sharded data parallelism using [FSDP](https://pytorch.org/docs/stable/fsdp.html) and used the [LION](https://arxiv.org/abs/2302.06675) optimizer.
 
 ## Limitations and Biases
@@ -238,7 +237,7 @@ Please cite this model using the following format:
     ly Usable LLMs},
     year      = {2023},
     url       = {www.mosaicml.com/blog/mpt-30b},
-    note      = {Accessed: TBD}, % change this date
-    urldate   = {TBD} % change this date
+    note      = {Accessed: 2023-06-22},
+    urldate   = {2023-06-22}
 }
 ```