Update README.md

README.md

@@ -1,15 +1,17 @@
 ---
-license: apache-2.0
+license: cc-by-sa-3.0
 datasets:
 - togethercomputer/RedPajama-Data-1T
 ---
 
 # MPT-1b-RedPajama-200b
 
-MPT-1b-RedPajama-200b is a 1.3 billion parameter decoder-only transformer trained on the [RedPajama dataset](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T).
-The model was trained for 200B tokens by sampling from the subsets of the RedPajama dataset in the same proportions as were used by the [Llama series of models](https://arxiv.org/abs/2302.13971).
+MPT-1b-RedPajama-200b is a 1.3 billion parameter decoder-only transformer pre-trained on the [RedPajama dataset](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) and subsequently fine-tuned on the [Databricks Dolly](https://github.com/databrickslabs/dolly/tree/master/data) instruction dataset.
+The model was pre-trained for 200B tokens by sampling from the subsets of the RedPajama dataset in the same proportions as were used by the [Llama series of models](https://arxiv.org/abs/2302.13971).
 This model was trained by [MosaicML](https://www.mosaicml.com) and follows a modified decoder-only transformer architecture.
 
+This model is an instruction fine-tuned version of [mpt-1b-redpajama-200b](https://huggingface.co/mosaicml/mpt-1b-redpajama-200b). In other words, the pre-trained version of this model is [mpt-1b-redpajama-200b](https://huggingface.co/mosaicml/mpt-1b-redpajama-200b).
+
 ## Model Date
 
 April 20, 2023
@@ -22,12 +24,12 @@ This is because we use a custom model architecture `MosaicGPT` that is not yet p
 
 ```python
 import transformers
-model = transformers.AutoModelForCausalLM.from_pretrained('mosaicml/mpt-1b-redpajama-200b', trust_remote_code=True)```
+model = transformers.AutoModelForCausalLM.from_pretrained('mosaicml/mpt-1b-redpajama-200b-dolly', trust_remote_code=True)
 ```
 
 To use the optimized triton implementation of FlashAttention, you can load with `attn_impl='triton'` and move the model to `bfloat16` like so:
 ```python
-model = transformers.AutoModelForCausalLM.from_pretrained('mosaicml/mpt-1b-redpajama-200b', trust_remote_code=True, attn_impl='triton')
+model = transformers.AutoModelForCausalLM.from_pretrained('mosaicml/mpt-1b-redpajama-200b-dolly', trust_remote_code=True, attn_impl='triton')
 model.to(device='cuda:0', dtype=torch.bfloat16)
 ```
 
@@ -43,7 +45,9 @@ The model has been modified from a standard transformer in the following ways:
 
 ## Training Data
 
-The model was trained for 200B tokens (batch size 2200, sequence length 2048). It was trained on the following data mix:
+### Pre-Training
+
+The model was pre-trained for 200B tokens (batch size 2200, sequence length 2048). It was trained on the following data mix:
 * 67% RedPajama Common Crawl
 * 15% [C4](https://huggingface.co/datasets/c4)
 * 4.5% RedPajama GitHub
@@ -60,13 +64,19 @@ Each example was constructed from as many sequences from that dataset as were ne
 
 The data was tokenized using the [EleutherAI/gpt-neox-20b](https://huggingface.co/EleutherAI/gpt-neox-20b) tokenizer.
 
+### Fine-Tuning
+
+TODO
+
 ## Training Configuration
 
-This model was trained on 440 A100-40GBs for about half a day using the [MosaicML Platform](https://www.mosaicml.com/platform). The model was trained with sharded data parallelism using FSDP.
+This model was pre-trained on 440 A100-40GBs for about half a day using the [MosaicML Platform](https://www.mosaicml.com/platform). The model was pre-trained with sharded data parallelism using FSDP.
 
 ## Acknowledgements
 
 This model builds on the work of [Together](https://www.together.xyz), which created the RedPajama dataset with the goal of mimicking the training data used to create the Llama series of models.
 We gratefully acknowledge the hard work of the team that put together this dataset, and we hope this model serves as a useful companion to that work.
 
-We also gratefully acknowledge the work of the researchers who created the Llama series of models, which was the impetus for our efforts and those who worked on the RedPajama project.
+This model also builds on the work of [Databricks](https://www.databricks.com/), which created the Dolly instruction fine-tuning dataset.
+
+We also gratefully acknowledge the work of the researchers who created the Llama series of models, which was the impetus for our efforts and those who worked on the RedPajama project.