Update README.md

README.md

@@ -8,7 +8,7 @@ language:
 - en
 ---
 # Instruction Pre-Training: Language Models are Supervised Multitask Learners
-This repo contains the **general models pre-trained from scratch** in our paper [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://huggingface.co/papers/2406.14491).
+This repo contains the **general models pre-trained from scratch** (on 100B tokens) in our paper [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://huggingface.co/papers/2406.14491).
 
 We explore supervised multitask pre-training by proposing ***Instruction Pre-Training***, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of *Instruction Pre-Training*. Instruction Pre-Training* outperforms *Vanilla Pre-training* in both general pre-training from scratch and domain-adaptive continual pre-training. **In pre-training from scratch, *Instruction Pre-Training* not only improves pre-trained base models but also benefits more from further instruction tuning.** In continual pre-training, *Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B.
 
@@ -16,12 +16,19 @@ We explore supervised multitask pre-training by proposing ***Instruction Pre-Tra
     <img src="https://cdn-uploads.huggingface.co/production/uploads/66711d2ee12fa6cc5f5dfc89/vRdsFIVQptbNaGiZ18Lih.png" width="400">
 </p>
 
+**************************** **Updates** ****************************
+* 2024/7/15: We scaled up the pre-trained tokens from 100B to 250B, with the number of synthesized instruction-response pairs reaching 500M! Below, we show the performance trend on downstream tasks throughout the pre-training process:
+<p align='center'>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/66711d2ee12fa6cc5f5dfc89/0okCfRkC6uALTfuNxt0Fa.png" width="700">
+</p>
+* 2024/6/21: Released the [paper](https://huggingface.co/papers/2406.14491), [code](https://github.com/microsoft/LMOps), and [resources](https://huggingface.co/instruction-pretrain)
+
 ## Resources
 **🤗 We share our data and models with example usages, feel free to open any issues or discussions! 🤗**
 
 - Context-Based Instruction Synthesizer: [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer)
 - Fine-Tuning Data for the Synthesizer: [ft-instruction-synthesizer-collection](https://huggingface.co/datasets/instruction-pretrain/ft-instruction-synthesizer-collection)
-- General Models Pre-Trained from Scratch:
+- General Models Pre-Trained from Scratch (on 100B tokes):
   - [InstructLM-500M](https://huggingface.co/instruction-pretrain/InstructLM-500M)
   - [InstructLM-1.3B](https://huggingface.co/instruction-pretrain/InstructLM-1.3B)
 - Domain-Specific Models Pre-Trained from Llama3-8B: