Update README.md

README.md

@@ -154,12 +154,13 @@ print(tokenizer.decode(outputs[0]))
 ## Training Data
 
 Falcon-Mamba has been trained with ~ 5,500 GT mainly coming from [Refined-Web](https://huggingface.co/datasets/tiiuae/falcon-refinedweb), a large volume web-only dataset filtered and deduplicated.
-Similar to the others [Falcon](https://huggingface.co/tiiuae/falcon-11B) suite models, Falcon-Mamba has been trained leveraging a multi-stage training strategy to increase the context-length training from 2,048 up to 8,192. 
+Similar to the others [Falcon](https://huggingface.co/tiiuae/falcon-11B) suite models, Falcon-Mamba has been trained leveraging a multi-stage training strategy to increase the context-length (from 2,048 to 8,192). 
+Moreover, inspired by the Curriculum Learning concept, we carefully choose data mixtures along the training stages, on both data diversity and complexity. 
 Note that at inference the context-length is not relevant as the Mamba architecture has no limit on long range dependency.
 At the last training stage, small portion of high-quality curated data was used to further enhance performance.
 
-Overall, the data sources included RefinedWeb-English, high quality technical data, code data and conversational data extracted from public sources.
-In particular, we used samples coming from [Fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu).
+Overall, the data sources included RefinedWeb-English, high quality technical data, code data and math data extracted from public sources.
+In particular, we used samples coming from [Fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) during our last training stage.
 
 The data was tokenized with the Falcon-[7B](https://huggingface.co/tiiuae/falcon-7B)/[11B](https://huggingface.co/tiiuae/falcon-11B) tokenizer.