Update README.md

README.md

@@ -23,7 +23,7 @@ inference: false
 
 # llm-jp-13b-v1.0-mdsfmt
 
-This repository provides large language models (Megatron-DeepSpeed format) developed by [LLM-jp](https://llm-jp.nii.ac.jp/), a collaborative project launched in Japan. **transformers format models are available [here](https://huggingface.co/llm-jp).**
+This repository provides large language models (Megatron-DeepSpeed format) developed by [LLM-jp](https://llm-jp.nii.ac.jp/), a collaborative project launched in Japan. **Hugging Face Transformers format models are available [here](https://huggingface.co/llm-jp).**
 
 | Model Variant |
 | :--- |
@@ -34,10 +34,12 @@ This repository provides large language models (Megatron-DeepSpeed format) devel
 | [llm-jp-1.3b-v1.0-mdsfmt-itr87430](https://huggingface.co/llm-jp/llm-jp-1.3b-v1.0-mdsfmt-itr87430) | 
 
 
-[llm-jp-13b-v1.0-mdsfmt-itr87870](https://huggingface.co/llm-jp/llm-jp-13b-v1.0-mdsfmt-itr87870) 
-and [llm-jp-1.3b-v1.0-mdsfmt-itr87430](https://huggingface.co/llm-jp/llm-jp-1.3b-v1.0-mdsfmt-itr87430) were originally trained with approximately 270B+ tokens.
-[llm-jp-13b-v1.0-mdsfmt](https://huggingface.co/llm-jp/llm-jp-13b-v1.0-mdsfmt) and [llm-jp-1.3b-v1.0-mdsfmt](https://huggingface.co/llm-jp/llm-jp-1.3b-v1.0-mdsfmt)
- are models that have been further trained by adding high-quality 27B tokens. 
+`llm-jp-13b-v1.0-mdsfmt-itr87870` 
+and `llm-jp-1.3b-v1.0-mdsfmt-itr87430` 
+were originally trained with approximately 270B+ tokens. 
+`llm-jp-13b-v1.0-mdsfmt` 
+and `llm-jp-1.3b-v1.0-mdsfmt` 
+are models further trained by additional (potentially) high-quality 27B tokens data from `llm-jp-13b-v1.0-mdsfmt-itr87870` and `llm-jp-1.3b-v1.0-mdsfmt-itr87430`, respectively for finalizing the pre-training.
 
 
 ## Model Details
@@ -60,8 +62,8 @@ and [llm-jp-1.3b-v1.0-mdsfmt-itr87430](https://huggingface.co/llm-jp/llm-jp-1.3b
 
 ## Tokenizer
 The tokenizer of this model is based on [huggingface/tokenizers](https://github.com/huggingface/tokenizers) Unigram byte-fallback model.
-The vocab entries were converted from [`llm-jp-tokenizer v2.1 (50k)`](https://github.com/llm-jp/llm-jp-tokenizer/releases/tag/v2.1).
-Please refer to [README.md](https://github.com/llm-jp/llm-jp-tokenizer) of `llm-ja-tokenizer` for the details of vocab constuction steps.
+The vocabulary entries were converted from [`llm-jp-tokenizer v2.1 (50k)`](https://github.com/llm-jp/llm-jp-tokenizer/releases/tag/v2.1).
+Please refer to the [README.md](https://github.com/llm-jp/llm-jp-tokenizer) of `llm-ja-tokenizer` for details on the vocabulary construction procedure.
 - **Model:** Hugging Face Fast Tokenizer using Unigram byte-fallback model which requires `tokenizers>=0.14.0`
 - **Training algorithm:** SentencePiece Unigram byte-fallback
 - **Training data:** A subset of the datasets for model pre-training
@@ -72,7 +74,7 @@ Please refer to [README.md](https://github.com/llm-jp/llm-jp-tokenizer) of `llm-
 
 ### Pre-training
 
-The models have been pre-trained on approximately 287.5B tokens, sourced from a blend of the following datasets.
+The models have been pre-trained using a blend of the following datasets.
 
 | Language | Dataset | Tokens |
 |:---:|:---:|:---:|
@@ -82,7 +84,8 @@ The models have been pre-trained on approximately 287.5B tokens, sourced from a
 ||[The Pile](https://huggingface.co/datasets/EleutherAI/pile)|135B
 |Codes|[The Stack](https://huggingface.co/datasets/bigcode/the-stack)|10B
 
-Pretraining was done by 10-hold shards that consists approx. 27-28B tokens. We further finalized the pretraining with additional cleaned 27B tokens data.
+The pre-training was continuously conducted using a total of 10 folds of non-overlapping data, each consisting of approximately 27-28B tokens.
+We finalized the pre-training with additional (potentially) high-quality 27B tokens data obtained from the identical source datasets listed above used for the 10-fold data.
 
 
 ## Evaluation
@@ -108,4 +111,4 @@ llm-jp(at)nii.ac.jp
 ## Model Card Authors
 *The names are listed in alphabetical order.*
 
-Namgi Han, Hirokazu Kiyomaru, Hiroshi Matsuda, Shota Sasaki, Shuhei Kurita, Taishi Nakamura, Takumi Okamoto.
+Hirokazu Kiyomaru, Hiroshi Matsuda, Jun Suzuki, Namgi Han, Saku Sugawara, Shota Sasaki, Shuhei Kurita, Taishi Nakamura, Takumi Okamoto.