Upload folder using huggingface_hub

checkpoint-1000/README.md

@@ -0,0 +1,202 @@
+---
+base_model: Qwen/Qwen2.5-72B
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.0

checkpoint-1000/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2.5-72B",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "v_proj",
+    "k_proj",
+    "down_proj",
+    "o_proj",
+    "q_proj",
+    "up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": true
+}

checkpoint-1000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87da4e11eb9b86473cf5676cfe2c6c7cb66578a988b8ee51d625b8d1fb608a19
+size 3368705968

checkpoint-1000/added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

checkpoint-1000/latest

@@ -0,0 +1 @@
+global_step1000

checkpoint-1000/special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-1000/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
+size 11421896

checkpoint-1000/tokenizer_config.json

@@ -0,0 +1,208 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

checkpoint-1000/trainer_state.json

@@ -0,0 +1,749 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.6028181749679753,
+  "eval_steps": 400,
+  "global_step": 1000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.006028181749679753,
+      "grad_norm": 0.38845974438826136,
+      "learning_rate": 1.0040160642570282e-06,
+      "loss": 1.6618,
+      "step": 10
+    },
+    {
+      "epoch": 0.012056363499359506,
+      "grad_norm": 0.255603405930248,
+      "learning_rate": 2.0080321285140564e-06,
+      "loss": 1.6698,
+      "step": 20
+    },
+    {
+      "epoch": 0.01808454524903926,
+      "grad_norm": 0.20116505722657768,
+      "learning_rate": 3.0120481927710846e-06,
+      "loss": 1.6264,
+      "step": 30
+    },
+    {
+      "epoch": 0.02411272699871901,
+      "grad_norm": 0.1795881732266397,
+      "learning_rate": 4.016064257028113e-06,
+      "loss": 1.6025,
+      "step": 40
+    },
+    {
+      "epoch": 0.030140908748398764,
+      "grad_norm": 0.14600421495766222,
+      "learning_rate": 5.020080321285141e-06,
+      "loss": 1.5584,
+      "step": 50
+    },
+    {
+      "epoch": 0.03616909049807852,
+      "grad_norm": 0.1170942466306718,
+      "learning_rate": 6.024096385542169e-06,
+      "loss": 1.5604,
+      "step": 60
+    },
+    {
+      "epoch": 0.04219727224775827,
+      "grad_norm": 0.2445974669666656,
+      "learning_rate": 7.028112449799197e-06,
+      "loss": 1.5057,
+      "step": 70
+    },
+    {
+      "epoch": 0.04822545399743802,
+      "grad_norm": 0.11238025351136724,
+      "learning_rate": 8.032128514056226e-06,
+      "loss": 1.4724,
+      "step": 80
+    },
+    {
+      "epoch": 0.05425363574711778,
+      "grad_norm": 0.13639577534070754,
+      "learning_rate": 9.036144578313253e-06,
+      "loss": 1.4962,
+      "step": 90
+    },
+    {
+      "epoch": 0.06028181749679753,
+      "grad_norm": 0.17113178431310286,
+      "learning_rate": 1.0040160642570281e-05,
+      "loss": 1.4765,
+      "step": 100
+    },
+    {
+      "epoch": 0.06630999924647728,
+      "grad_norm": 0.4795401637258333,
+      "learning_rate": 1.104417670682731e-05,
+      "loss": 1.4468,
+      "step": 110
+    },
+    {
+      "epoch": 0.07233818099615703,
+      "grad_norm": 0.13385762516900662,
+      "learning_rate": 1.2048192771084338e-05,
+      "loss": 1.4134,
+      "step": 120
+    },
+    {
+      "epoch": 0.07836636274583679,
+      "grad_norm": 0.12303707813666019,
+      "learning_rate": 1.3052208835341367e-05,
+      "loss": 1.4191,
+      "step": 130
+    },
+    {
+      "epoch": 0.08439454449551655,
+      "grad_norm": 0.10822073133399364,
+      "learning_rate": 1.4056224899598394e-05,
+      "loss": 1.397,
+      "step": 140
+    },
+    {
+      "epoch": 0.09042272624519629,
+      "grad_norm": 0.1109270134990499,
+      "learning_rate": 1.5060240963855424e-05,
+      "loss": 1.3818,
+      "step": 150
+    },
+    {
+      "epoch": 0.09645090799487605,
+      "grad_norm": 0.14622173134033867,
+      "learning_rate": 1.606425702811245e-05,
+      "loss": 1.4002,
+      "step": 160
+    },
+    {
+      "epoch": 0.1024790897445558,
+      "grad_norm": 0.10587626114414271,
+      "learning_rate": 1.706827309236948e-05,
+      "loss": 1.4123,
+      "step": 170
+    },
+    {
+      "epoch": 0.10850727149423556,
+      "grad_norm": 0.10302196814593138,
+      "learning_rate": 1.8072289156626505e-05,
+      "loss": 1.4016,
+      "step": 180
+    },
+    {
+      "epoch": 0.1145354532439153,
+      "grad_norm": 0.1359849724314843,
+      "learning_rate": 1.9076305220883535e-05,
+      "loss": 1.404,
+      "step": 190
+    },
+    {
+      "epoch": 0.12056363499359506,
+      "grad_norm": 0.10587622358885339,
+      "learning_rate": 2.0080321285140562e-05,
+      "loss": 1.4019,
+      "step": 200
+    },
+    {
+      "epoch": 0.1265918167432748,
+      "grad_norm": 0.15017595066321648,
+      "learning_rate": 2.1084337349397593e-05,
+      "loss": 1.393,
+      "step": 210
+    },
+    {
+      "epoch": 0.13261999849295456,
+      "grad_norm": 0.19475575142022897,
+      "learning_rate": 2.208835341365462e-05,
+      "loss": 1.3876,
+      "step": 220
+    },
+    {
+      "epoch": 0.1386481802426343,
+      "grad_norm": 0.12084095277263424,
+      "learning_rate": 2.309236947791165e-05,
+      "loss": 1.3916,
+      "step": 230
+    },
+    {
+      "epoch": 0.14467636199231407,
+      "grad_norm": 0.11857482977859173,
+      "learning_rate": 2.4096385542168677e-05,
+      "loss": 1.4056,
+      "step": 240
+    },
+    {
+      "epoch": 0.15070454374199382,
+      "grad_norm": 0.1403959719635503,
+      "learning_rate": 2.5100401606425704e-05,
+      "loss": 1.3935,
+      "step": 250
+    },
+    {
+      "epoch": 0.15673272549167358,
+      "grad_norm": 0.10800155257965392,
+      "learning_rate": 2.6104417670682734e-05,
+      "loss": 1.3826,
+      "step": 260
+    },
+    {
+      "epoch": 0.16276090724135334,
+      "grad_norm": 0.10598439909830581,
+      "learning_rate": 2.7108433734939758e-05,
+      "loss": 1.3999,
+      "step": 270
+    },
+    {
+      "epoch": 0.1687890889910331,
+      "grad_norm": 0.10753449693494475,
+      "learning_rate": 2.8112449799196788e-05,
+      "loss": 1.4047,
+      "step": 280
+    },
+    {
+      "epoch": 0.17481727074071282,
+      "grad_norm": 0.36718328659037996,
+      "learning_rate": 2.911646586345382e-05,
+      "loss": 1.3935,
+      "step": 290
+    },
+    {
+      "epoch": 0.18084545249039258,
+      "grad_norm": 0.10611900000479042,
+      "learning_rate": 3.012048192771085e-05,
+      "loss": 1.3736,
+      "step": 300
+    },
+    {
+      "epoch": 0.18687363424007233,
+      "grad_norm": 0.11901555220652378,
+      "learning_rate": 3.112449799196787e-05,
+      "loss": 1.3927,
+      "step": 310
+    },
+    {
+      "epoch": 0.1929018159897521,
+      "grad_norm": 0.118935148513695,
+      "learning_rate": 3.21285140562249e-05,
+      "loss": 1.3636,
+      "step": 320
+    },
+    {
+      "epoch": 0.19892999773943185,
+      "grad_norm": 0.1974545721831922,
+      "learning_rate": 3.313253012048193e-05,
+      "loss": 1.3892,
+      "step": 330
+    },
+    {
+      "epoch": 0.2049581794891116,
+      "grad_norm": 0.13145409772199562,
+      "learning_rate": 3.413654618473896e-05,
+      "loss": 1.3756,
+      "step": 340
+    },
+    {
+      "epoch": 0.21098636123879136,
+      "grad_norm": 0.11064380941915805,
+      "learning_rate": 3.5140562248995983e-05,
+      "loss": 1.3935,
+      "step": 350
+    },
+    {
+      "epoch": 0.21701454298847112,
+      "grad_norm": 0.12160423827639648,
+      "learning_rate": 3.614457831325301e-05,
+      "loss": 1.3698,
+      "step": 360
+    },
+    {
+      "epoch": 0.22304272473815084,
+      "grad_norm": 0.10349641889173723,
+      "learning_rate": 3.7148594377510044e-05,
+      "loss": 1.3771,
+      "step": 370
+    },
+    {
+      "epoch": 0.2290709064878306,
+      "grad_norm": 0.10682144059511894,
+      "learning_rate": 3.815261044176707e-05,
+      "loss": 1.3768,
+      "step": 380
+    },
+    {
+      "epoch": 0.23509908823751036,
+      "grad_norm": 0.11625245619819907,
+      "learning_rate": 3.91566265060241e-05,
+      "loss": 1.3795,
+      "step": 390
+    },
+    {
+      "epoch": 0.2411272699871901,
+      "grad_norm": 0.10327726962763091,
+      "learning_rate": 4.0160642570281125e-05,
+      "loss": 1.3987,
+      "step": 400
+    },
+    {
+      "epoch": 0.2411272699871901,
+      "eval_loss": 1.3548544645309448,
+      "eval_runtime": 148.2269,
+      "eval_samples_per_second": 7.239,
+      "eval_steps_per_second": 0.911,
+      "step": 400
+    },
+    {
+      "epoch": 0.24715545173686987,
+      "grad_norm": 0.10660530950921367,
+      "learning_rate": 4.116465863453816e-05,
+      "loss": 1.3886,
+      "step": 410
+    },
+    {
+      "epoch": 0.2531836334865496,
+      "grad_norm": 0.10405582985373843,
+      "learning_rate": 4.2168674698795186e-05,
+      "loss": 1.3645,
+      "step": 420
+    },
+    {
+      "epoch": 0.2592118152362294,
+      "grad_norm": 0.3318479326670041,
+      "learning_rate": 4.317269076305221e-05,
+      "loss": 1.3591,
+      "step": 430
+    },
+    {
+      "epoch": 0.2652399969859091,
+      "grad_norm": 0.10840544026201794,
+      "learning_rate": 4.417670682730924e-05,
+      "loss": 1.3805,
+      "step": 440
+    },
+    {
+      "epoch": 0.2712681787355889,
+      "grad_norm": 0.10730056620740543,
+      "learning_rate": 4.5180722891566266e-05,
+      "loss": 1.3888,
+      "step": 450
+    },
+    {
+      "epoch": 0.2772963604852686,
+      "grad_norm": 0.10699620793474768,
+      "learning_rate": 4.61847389558233e-05,
+      "loss": 1.3935,
+      "step": 460
+    },
+    {
+      "epoch": 0.2833245422349484,
+      "grad_norm": 0.10595493402596641,
+      "learning_rate": 4.718875502008032e-05,
+      "loss": 1.3659,
+      "step": 470
+    },
+    {
+      "epoch": 0.28935272398462814,
+      "grad_norm": 0.14234040947748414,
+      "learning_rate": 4.8192771084337354e-05,
+      "loss": 1.371,
+      "step": 480
+    },
+    {
+      "epoch": 0.29538090573430786,
+      "grad_norm": 0.1095349792774781,
+      "learning_rate": 4.919678714859438e-05,
+      "loss": 1.3647,
+      "step": 490
+    },
+    {
+      "epoch": 0.30140908748398765,
+      "grad_norm": 0.10655792946130023,
+      "learning_rate": 4.999997536857586e-05,
+      "loss": 1.3606,
+      "step": 500
+    },
+    {
+      "epoch": 0.3074372692336674,
+      "grad_norm": 0.14187369639959496,
+      "learning_rate": 4.999911327382711e-05,
+      "loss": 1.3952,
+      "step": 510
+    },
+    {
+      "epoch": 0.31346545098334716,
+      "grad_norm": 0.11221249471828258,
+      "learning_rate": 4.999701965640725e-05,
+      "loss": 1.3696,
+      "step": 520
+    },
+    {
+      "epoch": 0.3194936327330269,
+      "grad_norm": 0.11623924334950517,
+      "learning_rate": 4.9993694619453435e-05,
+      "loss": 1.3806,
+      "step": 530
+    },
+    {
+      "epoch": 0.3255218144827067,
+      "grad_norm": 0.09756519177369154,
+      "learning_rate": 4.998913832676579e-05,
+      "loss": 1.3307,
+      "step": 540
+    },
+    {
+      "epoch": 0.3315499962323864,
+      "grad_norm": 0.10068308440088464,
+      "learning_rate": 4.9983351002799416e-05,
+      "loss": 1.3808,
+      "step": 550
+    },
+    {
+      "epoch": 0.3375781779820662,
+      "grad_norm": 0.10281210912536574,
+      "learning_rate": 4.9976332932653225e-05,
+      "loss": 1.3491,
+      "step": 560
+    },
+    {
+      "epoch": 0.3436063597317459,
+      "grad_norm": 0.11134959395711537,
+      "learning_rate": 4.996808446205599e-05,
+      "loss": 1.3621,
+      "step": 570
+    },
+    {
+      "epoch": 0.34963454148142564,
+      "grad_norm": 0.10551082322799996,
+      "learning_rate": 4.9958605997349264e-05,
+      "loss": 1.3412,
+      "step": 580
+    },
+    {
+      "epoch": 0.3556627232311054,
+      "grad_norm": 0.10677290661324353,
+      "learning_rate": 4.994789800546737e-05,
+      "loss": 1.3623,
+      "step": 590
+    },
+    {
+      "epoch": 0.36169090498078516,
+      "grad_norm": 0.10716617749224472,
+      "learning_rate": 4.993596101391443e-05,
+      "loss": 1.3606,
+      "step": 600
+    },
+    {
+      "epoch": 0.36771908673046494,
+      "grad_norm": 0.11677420750343206,
+      "learning_rate": 4.99227956107383e-05,
+      "loss": 1.3485,
+      "step": 610
+    },
+    {
+      "epoch": 0.37374726848014467,
+      "grad_norm": 0.11130663386773537,
+      "learning_rate": 4.990840244450171e-05,
+      "loss": 1.3496,
+      "step": 620
+    },
+    {
+      "epoch": 0.37977545022982445,
+      "grad_norm": 0.1146856303733156,
+      "learning_rate": 4.989278222425021e-05,
+      "loss": 1.3026,
+      "step": 630
+    },
+    {
+      "epoch": 0.3858036319795042,
+      "grad_norm": 0.10717267360067521,
+      "learning_rate": 4.987593571947733e-05,
+      "loss": 1.3544,
+      "step": 640
+    },
+    {
+      "epoch": 0.3918318137291839,
+      "grad_norm": 0.12176386523595194,
+      "learning_rate": 4.985786376008659e-05,
+      "loss": 1.3271,
+      "step": 650
+    },
+    {
+      "epoch": 0.3978599954788637,
+      "grad_norm": 0.11520452966234733,
+      "learning_rate": 4.983856723635067e-05,
+      "loss": 1.3663,
+      "step": 660
+    },
+    {
+      "epoch": 0.4038881772285434,
+      "grad_norm": 0.1342901348076113,
+      "learning_rate": 4.981804709886756e-05,
+      "loss": 1.335,
+      "step": 670
+    },
+    {
+      "epoch": 0.4099163589782232,
+      "grad_norm": 0.12009183498218759,
+      "learning_rate": 4.9796304358513695e-05,
+      "loss": 1.3344,
+      "step": 680
+    },
+    {
+      "epoch": 0.41594454072790293,
+      "grad_norm": 0.11290080480825251,
+      "learning_rate": 4.977334008639417e-05,
+      "loss": 1.3302,
+      "step": 690
+    },
+    {
+      "epoch": 0.4219727224775827,
+      "grad_norm": 0.12117529129793933,
+      "learning_rate": 4.974915541378997e-05,
+      "loss": 1.3583,
+      "step": 700
+    },
+    {
+      "epoch": 0.42800090422726245,
+      "grad_norm": 0.12106414674081765,
+      "learning_rate": 4.972375153210228e-05,
+      "loss": 1.3205,
+      "step": 710
+    },
+    {
+      "epoch": 0.43402908597694223,
+      "grad_norm": 0.16181404734070373,
+      "learning_rate": 4.969712969279372e-05,
+      "loss": 1.3114,
+      "step": 720
+    },
+    {
+      "epoch": 0.44005726772662196,
+      "grad_norm": 0.13118653046898843,
+      "learning_rate": 4.966929120732675e-05,
+      "loss": 1.3346,
+      "step": 730
+    },
+    {
+      "epoch": 0.4460854494763017,
+      "grad_norm": 0.12384193345851267,
+      "learning_rate": 4.964023744709906e-05,
+      "loss": 1.3178,
+      "step": 740
+    },
+    {
+      "epoch": 0.45211363122598147,
+      "grad_norm": 0.13158056407059576,
+      "learning_rate": 4.960996984337599e-05,
+      "loss": 1.3072,
+      "step": 750
+    },
+    {
+      "epoch": 0.4581418129756612,
+      "grad_norm": 0.1271187477839215,
+      "learning_rate": 4.957848988722003e-05,
+      "loss": 1.3184,
+      "step": 760
+    },
+    {
+      "epoch": 0.464169994725341,
+      "grad_norm": 0.1271182236921431,
+      "learning_rate": 4.954579912941738e-05,
+      "loss": 1.3052,
+      "step": 770
+    },
+    {
+      "epoch": 0.4701981764750207,
+      "grad_norm": 0.13793244616584555,
+      "learning_rate": 4.951189918040153e-05,
+      "loss": 1.3156,
+      "step": 780
+    },
+    {
+      "epoch": 0.4762263582247005,
+      "grad_norm": 0.143375429890016,
+      "learning_rate": 4.9476791710173954e-05,
+      "loss": 1.3057,
+      "step": 790
+    },
+    {
+      "epoch": 0.4822545399743802,
+      "grad_norm": 0.13873715683631677,
+      "learning_rate": 4.94404784482218e-05,
+      "loss": 1.3239,
+      "step": 800
+    },
+    {
+      "epoch": 0.4822545399743802,
+      "eval_loss": 1.2957444190979004,
+      "eval_runtime": 149.3128,
+      "eval_samples_per_second": 7.186,
+      "eval_steps_per_second": 0.904,
+      "step": 800
+    },
+    {
+      "epoch": 0.48828272172405995,
+      "grad_norm": 0.12229796207948938,
+      "learning_rate": 4.940296118343274e-05,
+      "loss": 1.3356,
+      "step": 810
+    },
+    {
+      "epoch": 0.49431090347373974,
+      "grad_norm": 0.13903783244410667,
+      "learning_rate": 4.936424176400681e-05,
+      "loss": 1.3142,
+      "step": 820
+    },
+    {
+      "epoch": 0.5003390852234195,
+      "grad_norm": 0.15694334320231693,
+      "learning_rate": 4.9324322097365385e-05,
+      "loss": 1.3203,
+      "step": 830
+    },
+    {
+      "epoch": 0.5063672669730992,
+      "grad_norm": 0.1473003918483972,
+      "learning_rate": 4.9283204150057186e-05,
+      "loss": 1.3443,
+      "step": 840
+    },
+    {
+      "epoch": 0.512395448722779,
+      "grad_norm": 0.13719045426198556,
+      "learning_rate": 4.924088994766144e-05,
+      "loss": 1.3207,
+      "step": 850
+    },
+    {
+      "epoch": 0.5184236304724588,
+      "grad_norm": 0.1446409431503074,
+      "learning_rate": 4.919738157468806e-05,
+      "loss": 1.3367,
+      "step": 860
+    },
+    {
+      "epoch": 0.5244518122221385,
+      "grad_norm": 0.144976185376708,
+      "learning_rate": 4.915268117447498e-05,
+      "loss": 1.2926,
+      "step": 870
+    },
+    {
+      "epoch": 0.5304799939718182,
+      "grad_norm": 0.1432747729307118,
+      "learning_rate": 4.910679094908256e-05,
+      "loss": 1.2687,
+      "step": 880
+    },
+    {
+      "epoch": 0.536508175721498,
+      "grad_norm": 0.14816151870596467,
+      "learning_rate": 4.9059713159185087e-05,
+      "loss": 1.3066,
+      "step": 890
+    },
+    {
+      "epoch": 0.5425363574711778,
+      "grad_norm": 0.15312077289707232,
+      "learning_rate": 4.901145012395945e-05,
+      "loss": 1.2715,
+      "step": 900
+    },
+    {
+      "epoch": 0.5485645392208575,
+      "grad_norm": 0.14921082023451226,
+      "learning_rate": 4.896200422097087e-05,
+      "loss": 1.2854,
+      "step": 910
+    },
+    {
+      "epoch": 0.5545927209705372,
+      "grad_norm": 0.18249086322702504,
+      "learning_rate": 4.891137788605575e-05,
+      "loss": 1.2938,
+      "step": 920
+    },
+    {
+      "epoch": 0.560620902720217,
+      "grad_norm": 0.17799127506373516,
+      "learning_rate": 4.8859573613201725e-05,
+      "loss": 1.2771,
+      "step": 930
+    },
+    {
+      "epoch": 0.5666490844698968,
+      "grad_norm": 0.1567560038538035,
+      "learning_rate": 4.880659395442478e-05,
+      "loss": 1.2775,
+      "step": 940
+    },
+    {
+      "epoch": 0.5726772662195765,
+      "grad_norm": 0.2702719829446041,
+      "learning_rate": 4.87524415196435e-05,
+      "loss": 1.2998,
+      "step": 950
+    },
+    {
+      "epoch": 0.5787054479692563,
+      "grad_norm": 0.15756638200428358,
+      "learning_rate": 4.8697118976550574e-05,
+      "loss": 1.3046,
+      "step": 960
+    },
+    {
+      "epoch": 0.584733629718936,
+      "grad_norm": 0.16571471050973857,
+      "learning_rate": 4.86406290504813e-05,
+      "loss": 1.2965,
+      "step": 970
+    },
+    {
+      "epoch": 0.5907618114686157,
+      "grad_norm": 0.1623678481384679,
+      "learning_rate": 4.8582974524279365e-05,
+      "loss": 1.2865,
+      "step": 980
+    },
+    {
+      "epoch": 0.5967899932182955,
+      "grad_norm": 0.17039779398050892,
+      "learning_rate": 4.8524158238159756e-05,
+      "loss": 1.3065,
+      "step": 990
+    },
+    {
+      "epoch": 0.6028181749679753,
+      "grad_norm": 0.1645102791020125,
+      "learning_rate": 4.8464183089568824e-05,
+      "loss": 1.28,
+      "step": 1000
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 4974,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1681269511421952.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2923841ae243af669d09fc75f56460b83329158b7b6379ad1b1afd5c369cdb7a
+size 7224

checkpoint-1000/zero_to_fp32.py

@@ -0,0 +1,674 @@
+#!/usr/bin/env python
+
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: Apache-2.0
+
+# DeepSpeed Team
+
+# This script extracts fp32 consolidated weights from a zero 1, 2 and 3 DeepSpeed checkpoints. It gets
+# copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
+# the future. Once extracted, the weights don't require DeepSpeed and can be used in any
+# application.
+#
+# example:
+#   python zero_to_fp32.py . output_dir/
+#   or
+#   python zero_to_fp32.py . output_dir/ --safe_serialization
+
+import argparse
+import torch
+import glob
+import math
+import os
+import re
+import json
+from tqdm import tqdm
+from collections import OrderedDict
+from dataclasses import dataclass
+
+# while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
+# DeepSpeed data structures it has to be available in the current python environment.
+from deepspeed.utils import logger
+from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
+                                            FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
+                                            FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
+
+
+@dataclass
+class zero_model_state:
+    buffers: dict()
+    param_shapes: dict()
+    shared_params: list
+    ds_version: int
+    frozen_param_shapes: dict()
+    frozen_param_fragments: dict()
+
+
+debug = 0
+
+# load to cpu
+device = torch.device('cpu')
+
+
+def atoi(text):
+    return int(text) if text.isdigit() else text
+
+
+def natural_keys(text):
+    '''
+    alist.sort(key=natural_keys) sorts in human order
+    http://nedbatchelder.com/blog/200712/human_sorting.html
+    (See Toothy's implementation in the comments)
+    '''
+    return [atoi(c) for c in re.split(r'(\d+)', text)]
+
+
+def get_model_state_file(checkpoint_dir, zero_stage):
+    if not os.path.isdir(checkpoint_dir):
+        raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
+
+    # there should be only one file
+    if zero_stage <= 2:
+        file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
+    elif zero_stage == 3:
+        file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
+
+    if not os.path.exists(file):
+        raise FileNotFoundError(f"can't find model states file at '{file}'")
+
+    return file
+
+
+def get_checkpoint_files(checkpoint_dir, glob_pattern):
+    # XXX: need to test that this simple glob rule works for multi-node setup too
+    ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
+
+    if len(ckpt_files) == 0:
+        raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
+
+    return ckpt_files
+
+
+def get_optim_files(checkpoint_dir):
+    return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
+
+
+def get_model_state_files(checkpoint_dir):
+    return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
+
+
+def parse_model_states(files):
+    zero_model_states = []
+    for file in files:
+        state_dict = torch.load(file, map_location=device)
+
+        if BUFFER_NAMES not in state_dict:
+            raise ValueError(f"{file} is not a model state checkpoint")
+        buffer_names = state_dict[BUFFER_NAMES]
+        if debug:
+            print("Found buffers:", buffer_names)
+
+        # recover just the buffers while restoring them to fp32 if they were saved in fp16
+        buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
+        param_shapes = state_dict[PARAM_SHAPES]
+
+        # collect parameters that are included in param_shapes
+        param_names = []
+        for s in param_shapes:
+            for name in s.keys():
+                param_names.append(name)
+
+        # update with frozen parameters
+        frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
+        if frozen_param_shapes is not None:
+            if debug:
+                print(f"Found frozen_param_shapes: {frozen_param_shapes}")
+            param_names += list(frozen_param_shapes.keys())
+
+        # handle shared params
+        shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
+
+        ds_version = state_dict.get(DS_VERSION, None)
+
+        frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
+
+        z_model_state = zero_model_state(buffers=buffers,
+                                         param_shapes=param_shapes,
+                                         shared_params=shared_params,
+                                         ds_version=ds_version,
+                                         frozen_param_shapes=frozen_param_shapes,
+                                         frozen_param_fragments=frozen_param_fragments)
+        zero_model_states.append(z_model_state)
+
+    return zero_model_states
+
+
+def parse_optim_states(files, ds_checkpoint_dir):
+    total_files = len(files)
+    state_dicts = []
+    for f in files:
+        state_dict = torch.load(f, map_location=device)
+        # immediately discard the potentially huge 2 optimizer states as we only care for fp32 master weights
+        # and also handle the case where it was already removed by another helper script
+        state_dict["optimizer_state_dict"].pop("optimizer_state_dict", None)
+        state_dicts.append(state_dict)
+
+    if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
+        raise ValueError(f"{files[0]} is not a zero checkpoint")
+    zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
+    world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
+
+    # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
+    # parameters can be different from data parallelism for non-expert parameters. So we can just
+    # use the max of the partition_count to get the dp world_size.
+
+    if type(world_size) is list:
+        world_size = max(world_size)
+
+    if world_size != total_files:
+        raise ValueError(
+            f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
+            "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
+        )
+
+    # the groups are named differently in each stage
+    if zero_stage <= 2:
+        fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
+    elif zero_stage == 3:
+        fp32_groups_key = FP32_FLAT_GROUPS
+    else:
+        raise ValueError(f"unknown zero stage {zero_stage}")
+
+    if zero_stage <= 2:
+        fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
+    elif zero_stage == 3:
+        # if there is more than one param group, there will be multiple flattened tensors - one
+        # flattened tensor per group - for simplicity merge them into a single tensor
+        #
+        # XXX: could make the script more memory efficient for when there are multiple groups - it
+        # will require matching the sub-lists of param_shapes for each param group flattened tensor
+
+        fp32_flat_groups = [
+            torch.cat(state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key], 0) for i in range(len(state_dicts))
+        ]
+
+    return zero_stage, world_size, fp32_flat_groups
+
+
+def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters):
+    """
+    Returns fp32 state_dict reconstructed from ds checkpoint
+
+    Args:
+        - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
+
+    """
+    print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
+
+    optim_files = get_optim_files(ds_checkpoint_dir)
+    zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
+    print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
+
+    model_files = get_model_state_files(ds_checkpoint_dir)
+
+    zero_model_states = parse_model_states(model_files)
+    print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
+
+    if zero_stage <= 2:
+        return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                                          exclude_frozen_parameters)
+    elif zero_stage == 3:
+        return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                                          exclude_frozen_parameters)
+
+
+def _zero2_merge_frozen_params(state_dict, zero_model_states):
+    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
+        return
+
+    frozen_param_shapes = zero_model_states[0].frozen_param_shapes
+    frozen_param_fragments = zero_model_states[0].frozen_param_fragments
+
+    if debug:
+        num_elem = sum(s.numel() for s in frozen_param_shapes.values())
+        print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
+
+        wanted_params = len(frozen_param_shapes)
+        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
+        avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
+        print(f'Frozen params: Have {avail_numel} numels to process.')
+        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
+
+    total_params = 0
+    total_numel = 0
+    for name, shape in frozen_param_shapes.items():
+        total_params += 1
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+
+        state_dict[name] = frozen_param_fragments[name]
+
+        if debug:
+            print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
+
+    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _has_callable(obj, fn):
+    attr = getattr(obj, fn, None)
+    return callable(attr)
+
+
+def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
+    param_shapes = zero_model_states[0].param_shapes
+
+    # Reconstruction protocol:
+    #
+    # XXX: document this
+
+    if debug:
+        for i in range(world_size):
+            for j in range(len(fp32_flat_groups[0])):
+                print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
+
+    # XXX: memory usage doubles here (zero2)
+    num_param_groups = len(fp32_flat_groups[0])
+    merged_single_partition_of_fp32_groups = []
+    for i in range(num_param_groups):
+        merged_partitions = [sd[i] for sd in fp32_flat_groups]
+        full_single_fp32_vector = torch.cat(merged_partitions, 0)
+        merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
+    avail_numel = sum(
+        [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
+
+    if debug:
+        wanted_params = sum([len(shapes) for shapes in param_shapes])
+        wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
+        # not asserting if there is a mismatch due to possible padding
+        print(f"Have {avail_numel} numels to process.")
+        print(f"Need {wanted_numel} numels in {wanted_params} params.")
+
+    # params
+    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
+    # out-of-core computing solution
+    total_numel = 0
+    total_params = 0
+    for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
+        offset = 0
+        avail_numel = full_single_fp32_vector.numel()
+        for name, shape in shapes.items():
+
+            unpartitioned_numel = shape.numel() if _has_callable(shape, 'numel') else math.prod(shape)
+            total_numel += unpartitioned_numel
+            total_params += 1
+
+            if debug:
+                print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
+            state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
+            offset += unpartitioned_numel
+
+        # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
+        # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
+        # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
+        # live optimizer object, so we are checking that the numbers are within the right range
+        align_to = 2 * world_size
+
+        def zero2_align(x):
+            return align_to * math.ceil(x / align_to)
+
+        if debug:
+            print(f"original offset={offset}, avail_numel={avail_numel}")
+
+        offset = zero2_align(offset)
+        avail_numel = zero2_align(avail_numel)
+
+        if debug:
+            print(f"aligned  offset={offset}, avail_numel={avail_numel}")
+
+        # Sanity check
+        if offset != avail_numel:
+            raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
+
+    print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                               exclude_frozen_parameters):
+    state_dict = OrderedDict()
+
+    # buffers
+    buffers = zero_model_states[0].buffers
+    state_dict.update(buffers)
+    if debug:
+        print(f"added {len(buffers)} buffers")
+
+    if not exclude_frozen_parameters:
+        _zero2_merge_frozen_params(state_dict, zero_model_states)
+
+    _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
+
+    # recover shared parameters
+    for pair in zero_model_states[0].shared_params:
+        if pair[1] in state_dict:
+            state_dict[pair[0]] = state_dict[pair[1]]
+
+    return state_dict
+
+
+def zero3_partitioned_param_info(unpartitioned_numel, world_size):
+    remainder = unpartitioned_numel % world_size
+    padding_numel = (world_size - remainder) if remainder else 0
+    partitioned_numel = math.ceil(unpartitioned_numel / world_size)
+    return partitioned_numel, padding_numel
+
+
+def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
+    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
+        return
+
+    if debug:
+        for i in range(world_size):
+            num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
+            print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
+
+        frozen_param_shapes = zero_model_states[0].frozen_param_shapes
+        wanted_params = len(frozen_param_shapes)
+        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
+        avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
+        print(f'Frozen params: Have {avail_numel} numels to process.')
+        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
+
+    total_params = 0
+    total_numel = 0
+    for name, shape in zero_model_states[0].frozen_param_shapes.items():
+        total_params += 1
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+
+        param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
+        state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
+
+        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
+
+        if debug:
+            print(
+                f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
+            )
+
+    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
+    param_shapes = zero_model_states[0].param_shapes
+    avail_numel = fp32_flat_groups[0].numel() * world_size
+    # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
+    # param, re-consolidating each param, while dealing with padding if any
+
+    # merge list of dicts, preserving order
+    param_shapes = {k: v for d in param_shapes for k, v in d.items()}
+
+    if debug:
+        for i in range(world_size):
+            print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
+
+        wanted_params = len(param_shapes)
+        wanted_numel = sum(shape.numel() for shape in param_shapes.values())
+        # not asserting if there is a mismatch due to possible padding
+        avail_numel = fp32_flat_groups[0].numel() * world_size
+        print(f"Trainable params: Have {avail_numel} numels to process.")
+        print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
+
+    # params
+    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
+    # out-of-core computing solution
+    offset = 0
+    total_numel = 0
+    total_params = 0
+    for name, shape in tqdm(param_shapes.items(), desc='Gathering Sharded Weights'):
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+        total_params += 1
+        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
+
+        if debug:
+            print(
+                f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
+            )
+
+        # XXX: memory usage doubles here
+        state_dict[name] = torch.cat(
+            tuple(fp32_flat_groups[i].narrow(0, offset, partitioned_numel) for i in range(world_size)),
+            0).narrow(0, 0, unpartitioned_numel).view(shape)
+        offset += partitioned_numel
+
+    offset *= world_size
+
+    # Sanity check
+    if offset != avail_numel:
+        raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
+
+    print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                               exclude_frozen_parameters):
+    state_dict = OrderedDict()
+
+    # buffers
+    buffers = zero_model_states[0].buffers
+    state_dict.update(buffers)
+    if debug:
+        print(f"added {len(buffers)} buffers")
+
+    if not exclude_frozen_parameters:
+        _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
+
+    _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
+
+    # recover shared parameters
+    for pair in zero_model_states[0].shared_params:
+        if pair[1] in state_dict:
+            state_dict[pair[0]] = state_dict[pair[1]]
+
+    return state_dict
+
+
+def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag=None, exclude_frozen_parameters=False):
+    """
+    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
+    ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
+    via a model hub.
+
+    Args:
+        - ``checkpoint_dir``: path to the desired checkpoint folder
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
+        - ``exclude_frozen_parameters``: exclude frozen parameters
+
+    Returns:
+        - pytorch ``state_dict``
+
+    Note: this approach may not work if your application doesn't have sufficient free CPU memory and
+    you may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
+    the checkpoint.
+
+    A typical usage might be ::
+
+        from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
+        # do the training and checkpoint saving
+        state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
+        model = model.cpu() # move to cpu
+        model.load_state_dict(state_dict)
+        # submit to model hub or save the model to share with others
+
+    In this example the ``model`` will no longer be usable in the deepspeed context of the same
+    application. i.e. you will need to re-initialize the deepspeed engine, since
+    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
+
+    If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
+
+    """
+    if tag is None:
+        latest_path = os.path.join(checkpoint_dir, 'latest')
+        if os.path.isfile(latest_path):
+            with open(latest_path, 'r') as fd:
+                tag = fd.read().strip()
+        else:
+            raise ValueError(f"Unable to find 'latest' file at {latest_path}")
+
+    ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
+
+    if not os.path.isdir(ds_checkpoint_dir):
+        raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
+
+    return _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters)
+
+
+def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir,
+                                               output_dir,
+                                               max_shard_size="5GB",
+                                               safe_serialization=False,
+                                               tag=None,
+                                               exclude_frozen_parameters=False):
+    """
+    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
+    loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
+
+    Args:
+        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
+        - ``output_dir``: directory to the pytorch fp32 state_dict output files
+        - ``max_shard_size``: the maximum size for a checkpoint before being sharded, default value is 5GB
+        - ``safe_serialization``:  whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
+        - ``exclude_frozen_parameters``: exclude frozen parameters
+    """
+    # Dependency pre-check
+    if safe_serialization:
+        try:
+            from safetensors.torch import save_file
+        except ImportError:
+            print('If you want to use `safe_serialization`, please `pip install safetensors`')
+            raise
+    if max_shard_size is not None:
+        try:
+            from huggingface_hub import split_torch_state_dict_into_shards
+        except ImportError:
+            print('If you want to use `max_shard_size`, please `pip install huggingface_hub`')
+            raise
+
+    # Convert zero checkpoint to state_dict
+    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag, exclude_frozen_parameters)
+
+    # Shard the model if it is too big.
+    weights_name = "model.safetensors" if safe_serialization else "pytorch_model.bin"
+    if max_shard_size is not None:
+        filename_pattern = weights_name.replace(".bin", "{suffix}.bin").replace(".safetensors", "{suffix}.safetensors")
+        state_dict_split = split_torch_state_dict_into_shards(state_dict,
+                                                              filename_pattern=filename_pattern,
+                                                              max_shard_size=max_shard_size)
+    else:
+        from collections import namedtuple
+        StateDictSplit = namedtuple("StateDictSplit", ["is_sharded", "filename_to_tensors"])
+        state_dict_split = StateDictSplit(is_sharded=False,
+                                          filename_to_tensors={weights_name: list(state_dict.keys())})
+
+    # Save the model
+    filename_to_tensors = state_dict_split.filename_to_tensors.items()
+    for shard_file, tensors in tqdm(filename_to_tensors, desc="Saving checkpoint shards"):
+        shard = {tensor: state_dict[tensor].contiguous() for tensor in tensors}
+        output_path = os.path.join(output_dir, shard_file)
+        if safe_serialization:
+            save_file(shard, output_path, metadata={"format": "pt"})
+        else:
+            torch.save(shard, output_path)
+
+    # Save index if sharded
+    if state_dict_split.is_sharded:
+        index = {
+            "metadata": state_dict_split.metadata,
+            "weight_map": state_dict_split.tensor_to_filename,
+        }
+        save_index_file = "model.safetensors.index.json" if safe_serialization else "pytorch_model.bin.index.json"
+        save_index_file = os.path.join(output_dir, save_index_file)
+        with open(save_index_file, "w", encoding="utf-8") as f:
+            content = json.dumps(index, indent=2, sort_keys=True) + "\n"
+            f.write(content)
+
+
+def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
+    """
+    1. Put the provided model to cpu
+    2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
+    3. Load it into the provided model
+
+    Args:
+        - ``model``: the model object to update
+        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
+
+    Returns:
+        - ``model`: modified model
+
+    Make sure you have plenty of CPU memory available before you call this function. If you don't
+    have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
+    conveniently placed for you in the checkpoint folder.
+
+    A typical usage might be ::
+
+        from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
+        model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
+        # submit to model hub or save the model to share with others
+
+    Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
+    of the same application. i.e. you will need to re-initialize the deepspeed engine, since
+    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
+
+    """
+    logger.info(f"Extracting fp32 weights")
+    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
+
+    logger.info(f"Overwriting model with fp32 weights")
+    model = model.cpu()
+    model.load_state_dict(state_dict, strict=False)
+
+    return model
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("checkpoint_dir",
+                        type=str,
+                        help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
+    parser.add_argument("output_dir",
+                        type=str,
+                        help="directory to the pytorch fp32 state_dict output files"
+                        "(e.g. path/checkpoint-12-output/)")
+    parser.add_argument(
+        "--max_shard_size",
+        type=str,
+        default="5GB",
+        help="The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size"
+        "lower than this size. If expressed as a string, needs to be digits followed by a unit (like `5MB`"
+        "We default it to 5GB in order for models to be able to run easily on free-tier google colab instances"
+        "without CPU OOM issues.")
+    parser.add_argument(
+        "--safe_serialization",
+        default=False,
+        action='store_true',
+        help="Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).")
+    parser.add_argument("-t",
+                        "--tag",
+                        type=str,
+                        default=None,
+                        help="checkpoint tag used as a unique identifier for checkpoint. e.g., global_step1")
+    parser.add_argument("--exclude_frozen_parameters", action='store_true', help="exclude frozen parameters")
+    parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
+    args = parser.parse_args()
+
+    debug = args.debug
+
+    convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir,
+                                               args.output_dir,
+                                               max_shard_size=args.max_shard_size,
+                                               safe_serialization=args.safe_serialization,
+                                               tag=args.tag,
+                                               exclude_frozen_parameters=args.exclude_frozen_parameters)

checkpoint-1500/README.md

@@ -0,0 +1,202 @@
+---
+base_model: Qwen/Qwen2.5-72B
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.0

checkpoint-1500/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2.5-72B",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "v_proj",
+    "k_proj",
+    "down_proj",
+    "o_proj",
+    "q_proj",
+    "up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": true
+}

checkpoint-1500/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64c634ca19ad69941d46c8634e8ac11531af827d3636611c3aea48a1bb92c394
+size 3368705968

checkpoint-1500/added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

checkpoint-1500/latest

@@ -0,0 +1 @@
+global_step1500

checkpoint-1500/special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-1500/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
+size 11421896

checkpoint-1500/tokenizer_config.json

@@ -0,0 +1,208 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

checkpoint-1500/trainer_state.json

@@ -0,0 +1,1107 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9042272624519629,
+  "eval_steps": 400,
+  "global_step": 1500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.006028181749679753,
+      "grad_norm": 0.38845974438826136,
+      "learning_rate": 1.0040160642570282e-06,
+      "loss": 1.6618,
+      "step": 10
+    },
+    {
+      "epoch": 0.012056363499359506,
+      "grad_norm": 0.255603405930248,
+      "learning_rate": 2.0080321285140564e-06,
+      "loss": 1.6698,
+      "step": 20
+    },
+    {
+      "epoch": 0.01808454524903926,
+      "grad_norm": 0.20116505722657768,
+      "learning_rate": 3.0120481927710846e-06,
+      "loss": 1.6264,
+      "step": 30
+    },
+    {
+      "epoch": 0.02411272699871901,
+      "grad_norm": 0.1795881732266397,
+      "learning_rate": 4.016064257028113e-06,
+      "loss": 1.6025,
+      "step": 40
+    },
+    {
+      "epoch": 0.030140908748398764,
+      "grad_norm": 0.14600421495766222,
+      "learning_rate": 5.020080321285141e-06,
+      "loss": 1.5584,
+      "step": 50
+    },
+    {
+      "epoch": 0.03616909049807852,
+      "grad_norm": 0.1170942466306718,
+      "learning_rate": 6.024096385542169e-06,
+      "loss": 1.5604,
+      "step": 60
+    },
+    {
+      "epoch": 0.04219727224775827,
+      "grad_norm": 0.2445974669666656,
+      "learning_rate": 7.028112449799197e-06,
+      "loss": 1.5057,
+      "step": 70
+    },
+    {
+      "epoch": 0.04822545399743802,
+      "grad_norm": 0.11238025351136724,
+      "learning_rate": 8.032128514056226e-06,
+      "loss": 1.4724,
+      "step": 80
+    },
+    {
+      "epoch": 0.05425363574711778,
+      "grad_norm": 0.13639577534070754,
+      "learning_rate": 9.036144578313253e-06,
+      "loss": 1.4962,
+      "step": 90
+    },
+    {
+      "epoch": 0.06028181749679753,
+      "grad_norm": 0.17113178431310286,
+      "learning_rate": 1.0040160642570281e-05,
+      "loss": 1.4765,
+      "step": 100
+    },
+    {
+      "epoch": 0.06630999924647728,
+      "grad_norm": 0.4795401637258333,
+      "learning_rate": 1.104417670682731e-05,
+      "loss": 1.4468,
+      "step": 110
+    },
+    {
+      "epoch": 0.07233818099615703,
+      "grad_norm": 0.13385762516900662,
+      "learning_rate": 1.2048192771084338e-05,
+      "loss": 1.4134,
+      "step": 120
+    },
+    {
+      "epoch": 0.07836636274583679,
+      "grad_norm": 0.12303707813666019,
+      "learning_rate": 1.3052208835341367e-05,
+      "loss": 1.4191,
+      "step": 130
+    },
+    {
+      "epoch": 0.08439454449551655,
+      "grad_norm": 0.10822073133399364,
+      "learning_rate": 1.4056224899598394e-05,
+      "loss": 1.397,
+      "step": 140
+    },
+    {
+      "epoch": 0.09042272624519629,
+      "grad_norm": 0.1109270134990499,
+      "learning_rate": 1.5060240963855424e-05,
+      "loss": 1.3818,
+      "step": 150
+    },
+    {
+      "epoch": 0.09645090799487605,
+      "grad_norm": 0.14622173134033867,
+      "learning_rate": 1.606425702811245e-05,
+      "loss": 1.4002,
+      "step": 160
+    },
+    {
+      "epoch": 0.1024790897445558,
+      "grad_norm": 0.10587626114414271,
+      "learning_rate": 1.706827309236948e-05,
+      "loss": 1.4123,
+      "step": 170
+    },
+    {
+      "epoch": 0.10850727149423556,
+      "grad_norm": 0.10302196814593138,
+      "learning_rate": 1.8072289156626505e-05,
+      "loss": 1.4016,
+      "step": 180
+    },
+    {
+      "epoch": 0.1145354532439153,
+      "grad_norm": 0.1359849724314843,
+      "learning_rate": 1.9076305220883535e-05,
+      "loss": 1.404,
+      "step": 190
+    },
+    {
+      "epoch": 0.12056363499359506,
+      "grad_norm": 0.10587622358885339,
+      "learning_rate": 2.0080321285140562e-05,
+      "loss": 1.4019,
+      "step": 200
+    },
+    {
+      "epoch": 0.1265918167432748,
+      "grad_norm": 0.15017595066321648,
+      "learning_rate": 2.1084337349397593e-05,
+      "loss": 1.393,
+      "step": 210
+    },
+    {
+      "epoch": 0.13261999849295456,
+      "grad_norm": 0.19475575142022897,
+      "learning_rate": 2.208835341365462e-05,
+      "loss": 1.3876,
+      "step": 220
+    },
+    {
+      "epoch": 0.1386481802426343,
+      "grad_norm": 0.12084095277263424,
+      "learning_rate": 2.309236947791165e-05,
+      "loss": 1.3916,
+      "step": 230
+    },
+    {
+      "epoch": 0.14467636199231407,
+      "grad_norm": 0.11857482977859173,
+      "learning_rate": 2.4096385542168677e-05,
+      "loss": 1.4056,
+      "step": 240
+    },
+    {
+      "epoch": 0.15070454374199382,
+      "grad_norm": 0.1403959719635503,
+      "learning_rate": 2.5100401606425704e-05,
+      "loss": 1.3935,
+      "step": 250
+    },
+    {
+      "epoch": 0.15673272549167358,
+      "grad_norm": 0.10800155257965392,
+      "learning_rate": 2.6104417670682734e-05,
+      "loss": 1.3826,
+      "step": 260
+    },
+    {
+      "epoch": 0.16276090724135334,
+      "grad_norm": 0.10598439909830581,
+      "learning_rate": 2.7108433734939758e-05,
+      "loss": 1.3999,
+      "step": 270
+    },
+    {
+      "epoch": 0.1687890889910331,
+      "grad_norm": 0.10753449693494475,
+      "learning_rate": 2.8112449799196788e-05,
+      "loss": 1.4047,
+      "step": 280
+    },
+    {
+      "epoch": 0.17481727074071282,
+      "grad_norm": 0.36718328659037996,
+      "learning_rate": 2.911646586345382e-05,
+      "loss": 1.3935,
+      "step": 290
+    },
+    {
+      "epoch": 0.18084545249039258,
+      "grad_norm": 0.10611900000479042,
+      "learning_rate": 3.012048192771085e-05,
+      "loss": 1.3736,
+      "step": 300
+    },
+    {
+      "epoch": 0.18687363424007233,
+      "grad_norm": 0.11901555220652378,
+      "learning_rate": 3.112449799196787e-05,
+      "loss": 1.3927,
+      "step": 310
+    },
+    {
+      "epoch": 0.1929018159897521,
+      "grad_norm": 0.118935148513695,
+      "learning_rate": 3.21285140562249e-05,
+      "loss": 1.3636,
+      "step": 320
+    },
+    {
+      "epoch": 0.19892999773943185,
+      "grad_norm": 0.1974545721831922,
+      "learning_rate": 3.313253012048193e-05,
+      "loss": 1.3892,
+      "step": 330
+    },
+    {
+      "epoch": 0.2049581794891116,
+      "grad_norm": 0.13145409772199562,
+      "learning_rate": 3.413654618473896e-05,
+      "loss": 1.3756,
+      "step": 340
+    },
+    {
+      "epoch": 0.21098636123879136,
+      "grad_norm": 0.11064380941915805,
+      "learning_rate": 3.5140562248995983e-05,
+      "loss": 1.3935,
+      "step": 350
+    },
+    {
+      "epoch": 0.21701454298847112,
+      "grad_norm": 0.12160423827639648,
+      "learning_rate": 3.614457831325301e-05,
+      "loss": 1.3698,
+      "step": 360
+    },
+    {
+      "epoch": 0.22304272473815084,
+      "grad_norm": 0.10349641889173723,
+      "learning_rate": 3.7148594377510044e-05,
+      "loss": 1.3771,
+      "step": 370
+    },
+    {
+      "epoch": 0.2290709064878306,
+      "grad_norm": 0.10682144059511894,
+      "learning_rate": 3.815261044176707e-05,
+      "loss": 1.3768,
+      "step": 380
+    },
+    {
+      "epoch": 0.23509908823751036,
+      "grad_norm": 0.11625245619819907,
+      "learning_rate": 3.91566265060241e-05,
+      "loss": 1.3795,
+      "step": 390
+    },
+    {
+      "epoch": 0.2411272699871901,
+      "grad_norm": 0.10327726962763091,
+      "learning_rate": 4.0160642570281125e-05,
+      "loss": 1.3987,
+      "step": 400
+    },
+    {
+      "epoch": 0.2411272699871901,
+      "eval_loss": 1.3548544645309448,
+      "eval_runtime": 148.2269,
+      "eval_samples_per_second": 7.239,
+      "eval_steps_per_second": 0.911,
+      "step": 400
+    },
+    {
+      "epoch": 0.24715545173686987,
+      "grad_norm": 0.10660530950921367,
+      "learning_rate": 4.116465863453816e-05,
+      "loss": 1.3886,
+      "step": 410
+    },
+    {
+      "epoch": 0.2531836334865496,
+      "grad_norm": 0.10405582985373843,
+      "learning_rate": 4.2168674698795186e-05,
+      "loss": 1.3645,
+      "step": 420
+    },
+    {
+      "epoch": 0.2592118152362294,
+      "grad_norm": 0.3318479326670041,
+      "learning_rate": 4.317269076305221e-05,
+      "loss": 1.3591,
+      "step": 430
+    },
+    {
+      "epoch": 0.2652399969859091,
+      "grad_norm": 0.10840544026201794,
+      "learning_rate": 4.417670682730924e-05,
+      "loss": 1.3805,
+      "step": 440
+    },
+    {
+      "epoch": 0.2712681787355889,
+      "grad_norm": 0.10730056620740543,
+      "learning_rate": 4.5180722891566266e-05,
+      "loss": 1.3888,
+      "step": 450
+    },
+    {
+      "epoch": 0.2772963604852686,
+      "grad_norm": 0.10699620793474768,
+      "learning_rate": 4.61847389558233e-05,
+      "loss": 1.3935,
+      "step": 460
+    },
+    {
+      "epoch": 0.2833245422349484,
+      "grad_norm": 0.10595493402596641,
+      "learning_rate": 4.718875502008032e-05,
+      "loss": 1.3659,
+      "step": 470
+    },
+    {
+      "epoch": 0.28935272398462814,
+      "grad_norm": 0.14234040947748414,
+      "learning_rate": 4.8192771084337354e-05,
+      "loss": 1.371,
+      "step": 480
+    },
+    {
+      "epoch": 0.29538090573430786,
+      "grad_norm": 0.1095349792774781,
+      "learning_rate": 4.919678714859438e-05,
+      "loss": 1.3647,
+      "step": 490
+    },
+    {
+      "epoch": 0.30140908748398765,
+      "grad_norm": 0.10655792946130023,
+      "learning_rate": 4.999997536857586e-05,
+      "loss": 1.3606,
+      "step": 500
+    },
+    {
+      "epoch": 0.3074372692336674,
+      "grad_norm": 0.14187369639959496,
+      "learning_rate": 4.999911327382711e-05,
+      "loss": 1.3952,
+      "step": 510
+    },
+    {
+      "epoch": 0.31346545098334716,
+      "grad_norm": 0.11221249471828258,
+      "learning_rate": 4.999701965640725e-05,
+      "loss": 1.3696,
+      "step": 520
+    },
+    {
+      "epoch": 0.3194936327330269,
+      "grad_norm": 0.11623924334950517,
+      "learning_rate": 4.9993694619453435e-05,
+      "loss": 1.3806,
+      "step": 530
+    },
+    {
+      "epoch": 0.3255218144827067,
+      "grad_norm": 0.09756519177369154,
+      "learning_rate": 4.998913832676579e-05,
+      "loss": 1.3307,
+      "step": 540
+    },
+    {
+      "epoch": 0.3315499962323864,
+      "grad_norm": 0.10068308440088464,
+      "learning_rate": 4.9983351002799416e-05,
+      "loss": 1.3808,
+      "step": 550
+    },
+    {
+      "epoch": 0.3375781779820662,
+      "grad_norm": 0.10281210912536574,
+      "learning_rate": 4.9976332932653225e-05,
+      "loss": 1.3491,
+      "step": 560
+    },
+    {
+      "epoch": 0.3436063597317459,
+      "grad_norm": 0.11134959395711537,
+      "learning_rate": 4.996808446205599e-05,
+      "loss": 1.3621,
+      "step": 570
+    },
+    {
+      "epoch": 0.34963454148142564,
+      "grad_norm": 0.10551082322799996,
+      "learning_rate": 4.9958605997349264e-05,
+      "loss": 1.3412,
+      "step": 580
+    },
+    {
+      "epoch": 0.3556627232311054,
+      "grad_norm": 0.10677290661324353,
+      "learning_rate": 4.994789800546737e-05,
+      "loss": 1.3623,
+      "step": 590
+    },
+    {
+      "epoch": 0.36169090498078516,
+      "grad_norm": 0.10716617749224472,
+      "learning_rate": 4.993596101391443e-05,
+      "loss": 1.3606,
+      "step": 600
+    },
+    {
+      "epoch": 0.36771908673046494,
+      "grad_norm": 0.11677420750343206,
+      "learning_rate": 4.99227956107383e-05,
+      "loss": 1.3485,
+      "step": 610
+    },
+    {
+      "epoch": 0.37374726848014467,
+      "grad_norm": 0.11130663386773537,
+      "learning_rate": 4.990840244450171e-05,
+      "loss": 1.3496,
+      "step": 620
+    },
+    {
+      "epoch": 0.37977545022982445,
+      "grad_norm": 0.1146856303733156,
+      "learning_rate": 4.989278222425021e-05,
+      "loss": 1.3026,
+      "step": 630
+    },
+    {
+      "epoch": 0.3858036319795042,
+      "grad_norm": 0.10717267360067521,
+      "learning_rate": 4.987593571947733e-05,
+      "loss": 1.3544,
+      "step": 640
+    },
+    {
+      "epoch": 0.3918318137291839,
+      "grad_norm": 0.12176386523595194,
+      "learning_rate": 4.985786376008659e-05,
+      "loss": 1.3271,
+      "step": 650
+    },
+    {
+      "epoch": 0.3978599954788637,
+      "grad_norm": 0.11520452966234733,
+      "learning_rate": 4.983856723635067e-05,
+      "loss": 1.3663,
+      "step": 660
+    },
+    {
+      "epoch": 0.4038881772285434,
+      "grad_norm": 0.1342901348076113,
+      "learning_rate": 4.981804709886756e-05,
+      "loss": 1.335,
+      "step": 670
+    },
+    {
+      "epoch": 0.4099163589782232,
+      "grad_norm": 0.12009183498218759,
+      "learning_rate": 4.9796304358513695e-05,
+      "loss": 1.3344,
+      "step": 680
+    },
+    {
+      "epoch": 0.41594454072790293,
+      "grad_norm": 0.11290080480825251,
+      "learning_rate": 4.977334008639417e-05,
+      "loss": 1.3302,
+      "step": 690
+    },
+    {
+      "epoch": 0.4219727224775827,
+      "grad_norm": 0.12117529129793933,
+      "learning_rate": 4.974915541378997e-05,
+      "loss": 1.3583,
+      "step": 700
+    },
+    {
+      "epoch": 0.42800090422726245,
+      "grad_norm": 0.12106414674081765,
+      "learning_rate": 4.972375153210228e-05,
+      "loss": 1.3205,
+      "step": 710
+    },
+    {
+      "epoch": 0.43402908597694223,
+      "grad_norm": 0.16181404734070373,
+      "learning_rate": 4.969712969279372e-05,
+      "loss": 1.3114,
+      "step": 720
+    },
+    {
+      "epoch": 0.44005726772662196,
+      "grad_norm": 0.13118653046898843,
+      "learning_rate": 4.966929120732675e-05,
+      "loss": 1.3346,
+      "step": 730
+    },
+    {
+      "epoch": 0.4460854494763017,
+      "grad_norm": 0.12384193345851267,
+      "learning_rate": 4.964023744709906e-05,
+      "loss": 1.3178,
+      "step": 740
+    },
+    {
+      "epoch": 0.45211363122598147,
+      "grad_norm": 0.13158056407059576,
+      "learning_rate": 4.960996984337599e-05,
+      "loss": 1.3072,
+      "step": 750
+    },
+    {
+      "epoch": 0.4581418129756612,
+      "grad_norm": 0.1271187477839215,
+      "learning_rate": 4.957848988722003e-05,
+      "loss": 1.3184,
+      "step": 760
+    },
+    {
+      "epoch": 0.464169994725341,
+      "grad_norm": 0.1271182236921431,
+      "learning_rate": 4.954579912941738e-05,
+      "loss": 1.3052,
+      "step": 770
+    },
+    {
+      "epoch": 0.4701981764750207,
+      "grad_norm": 0.13793244616584555,
+      "learning_rate": 4.951189918040153e-05,
+      "loss": 1.3156,
+      "step": 780
+    },
+    {
+      "epoch": 0.4762263582247005,
+      "grad_norm": 0.143375429890016,
+      "learning_rate": 4.9476791710173954e-05,
+      "loss": 1.3057,
+      "step": 790
+    },
+    {
+      "epoch": 0.4822545399743802,
+      "grad_norm": 0.13873715683631677,
+      "learning_rate": 4.94404784482218e-05,
+      "loss": 1.3239,
+      "step": 800
+    },
+    {
+      "epoch": 0.4822545399743802,
+      "eval_loss": 1.2957444190979004,
+      "eval_runtime": 149.3128,
+      "eval_samples_per_second": 7.186,
+      "eval_steps_per_second": 0.904,
+      "step": 800
+    },
+    {
+      "epoch": 0.48828272172405995,
+      "grad_norm": 0.12229796207948938,
+      "learning_rate": 4.940296118343274e-05,
+      "loss": 1.3356,
+      "step": 810
+    },
+    {
+      "epoch": 0.49431090347373974,
+      "grad_norm": 0.13903783244410667,
+      "learning_rate": 4.936424176400681e-05,
+      "loss": 1.3142,
+      "step": 820
+    },
+    {
+      "epoch": 0.5003390852234195,
+      "grad_norm": 0.15694334320231693,
+      "learning_rate": 4.9324322097365385e-05,
+      "loss": 1.3203,
+      "step": 830
+    },
+    {
+      "epoch": 0.5063672669730992,
+      "grad_norm": 0.1473003918483972,
+      "learning_rate": 4.9283204150057186e-05,
+      "loss": 1.3443,
+      "step": 840
+    },
+    {
+      "epoch": 0.512395448722779,
+      "grad_norm": 0.13719045426198556,
+      "learning_rate": 4.924088994766144e-05,
+      "loss": 1.3207,
+      "step": 850
+    },
+    {
+      "epoch": 0.5184236304724588,
+      "grad_norm": 0.1446409431503074,
+      "learning_rate": 4.919738157468806e-05,
+      "loss": 1.3367,
+      "step": 860
+    },
+    {
+      "epoch": 0.5244518122221385,
+      "grad_norm": 0.144976185376708,
+      "learning_rate": 4.915268117447498e-05,
+      "loss": 1.2926,
+      "step": 870
+    },
+    {
+      "epoch": 0.5304799939718182,
+      "grad_norm": 0.1432747729307118,
+      "learning_rate": 4.910679094908256e-05,
+      "loss": 1.2687,
+      "step": 880
+    },
+    {
+      "epoch": 0.536508175721498,
+      "grad_norm": 0.14816151870596467,
+      "learning_rate": 4.9059713159185087e-05,
+      "loss": 1.3066,
+      "step": 890
+    },
+    {
+      "epoch": 0.5425363574711778,
+      "grad_norm": 0.15312077289707232,
+      "learning_rate": 4.901145012395945e-05,
+      "loss": 1.2715,
+      "step": 900
+    },
+    {
+      "epoch": 0.5485645392208575,
+      "grad_norm": 0.14921082023451226,
+      "learning_rate": 4.896200422097087e-05,
+      "loss": 1.2854,
+      "step": 910
+    },
+    {
+      "epoch": 0.5545927209705372,
+      "grad_norm": 0.18249086322702504,
+      "learning_rate": 4.891137788605575e-05,
+      "loss": 1.2938,
+      "step": 920
+    },
+    {
+      "epoch": 0.560620902720217,
+      "grad_norm": 0.17799127506373516,
+      "learning_rate": 4.8859573613201725e-05,
+      "loss": 1.2771,
+      "step": 930
+    },
+    {
+      "epoch": 0.5666490844698968,
+      "grad_norm": 0.1567560038538035,
+      "learning_rate": 4.880659395442478e-05,
+      "loss": 1.2775,
+      "step": 940
+    },
+    {
+      "epoch": 0.5726772662195765,
+      "grad_norm": 0.2702719829446041,
+      "learning_rate": 4.87524415196435e-05,
+      "loss": 1.2998,
+      "step": 950
+    },
+    {
+      "epoch": 0.5787054479692563,
+      "grad_norm": 0.15756638200428358,
+      "learning_rate": 4.8697118976550574e-05,
+      "loss": 1.3046,
+      "step": 960
+    },
+    {
+      "epoch": 0.584733629718936,
+      "grad_norm": 0.16571471050973857,
+      "learning_rate": 4.86406290504813e-05,
+      "loss": 1.2965,
+      "step": 970
+    },
+    {
+      "epoch": 0.5907618114686157,
+      "grad_norm": 0.1623678481384679,
+      "learning_rate": 4.8582974524279365e-05,
+      "loss": 1.2865,
+      "step": 980
+    },
+    {
+      "epoch": 0.5967899932182955,
+      "grad_norm": 0.17039779398050892,
+      "learning_rate": 4.8524158238159756e-05,
+      "loss": 1.3065,
+      "step": 990
+    },
+    {
+      "epoch": 0.6028181749679753,
+      "grad_norm": 0.1645102791020125,
+      "learning_rate": 4.8464183089568824e-05,
+      "loss": 1.28,
+      "step": 1000
+    },
+    {
+      "epoch": 0.6088463567176551,
+      "grad_norm": 0.16365223749524835,
+      "learning_rate": 4.8403052033041596e-05,
+      "loss": 1.2719,
+      "step": 1010
+    },
+    {
+      "epoch": 0.6148745384673348,
+      "grad_norm": 0.1602190503029676,
+      "learning_rate": 4.834076808005614e-05,
+      "loss": 1.2818,
+      "step": 1020
+    },
+    {
+      "epoch": 0.6209027202170145,
+      "grad_norm": 0.15949490297225546,
+      "learning_rate": 4.827733429888531e-05,
+      "loss": 1.2691,
+      "step": 1030
+    },
+    {
+      "epoch": 0.6269309019666943,
+      "grad_norm": 0.17146084316633475,
+      "learning_rate": 4.821275381444552e-05,
+      "loss": 1.2657,
+      "step": 1040
+    },
+    {
+      "epoch": 0.632959083716374,
+      "grad_norm": 0.2405045374229408,
+      "learning_rate": 4.814702980814284e-05,
+      "loss": 1.2472,
+      "step": 1050
+    },
+    {
+      "epoch": 0.6389872654660538,
+      "grad_norm": 0.18854187339515996,
+      "learning_rate": 4.8080165517716267e-05,
+      "loss": 1.2411,
+      "step": 1060
+    },
+    {
+      "epoch": 0.6450154472157336,
+      "grad_norm": 0.17120145629647163,
+      "learning_rate": 4.80121642370782e-05,
+      "loss": 1.2865,
+      "step": 1070
+    },
+    {
+      "epoch": 0.6510436289654133,
+      "grad_norm": 0.17879749041413,
+      "learning_rate": 4.794302931615223e-05,
+      "loss": 1.2572,
+      "step": 1080
+    },
+    {
+      "epoch": 0.657071810715093,
+      "grad_norm": 0.18362233795953198,
+      "learning_rate": 4.7872764160708044e-05,
+      "loss": 1.2823,
+      "step": 1090
+    },
+    {
+      "epoch": 0.6630999924647728,
+      "grad_norm": 0.1827563529393702,
+      "learning_rate": 4.780137223219369e-05,
+      "loss": 1.2308,
+      "step": 1100
+    },
+    {
+      "epoch": 0.6691281742144526,
+      "grad_norm": 0.1864429907321282,
+      "learning_rate": 4.772885704756506e-05,
+      "loss": 1.2522,
+      "step": 1110
+    },
+    {
+      "epoch": 0.6751563559641324,
+      "grad_norm": 0.1906552054233093,
+      "learning_rate": 4.7655222179112595e-05,
+      "loss": 1.2485,
+      "step": 1120
+    },
+    {
+      "epoch": 0.681184537713812,
+      "grad_norm": 0.17584845483906592,
+      "learning_rate": 4.758047125428537e-05,
+      "loss": 1.2466,
+      "step": 1130
+    },
+    {
+      "epoch": 0.6872127194634918,
+      "grad_norm": 0.1832614435939446,
+      "learning_rate": 4.750460795551235e-05,
+      "loss": 1.2473,
+      "step": 1140
+    },
+    {
+      "epoch": 0.6932409012131716,
+      "grad_norm": 0.19068207255760805,
+      "learning_rate": 4.742763602002097e-05,
+      "loss": 1.2306,
+      "step": 1150
+    },
+    {
+      "epoch": 0.6992690829628513,
+      "grad_norm": 0.1869186315862799,
+      "learning_rate": 4.7349559239653064e-05,
+      "loss": 1.2189,
+      "step": 1160
+    },
+    {
+      "epoch": 0.7052972647125311,
+      "grad_norm": 0.19138057977743592,
+      "learning_rate": 4.727038146067807e-05,
+      "loss": 1.2387,
+      "step": 1170
+    },
+    {
+      "epoch": 0.7113254464622109,
+      "grad_norm": 0.19645994544271753,
+      "learning_rate": 4.719010658360352e-05,
+      "loss": 1.2351,
+      "step": 1180
+    },
+    {
+      "epoch": 0.7173536282118906,
+      "grad_norm": 0.2014637466767444,
+      "learning_rate": 4.710873856298294e-05,
+      "loss": 1.2104,
+      "step": 1190
+    },
+    {
+      "epoch": 0.7233818099615703,
+      "grad_norm": 0.19947019583870831,
+      "learning_rate": 4.702628140722096e-05,
+      "loss": 1.2473,
+      "step": 1200
+    },
+    {
+      "epoch": 0.7233818099615703,
+      "eval_loss": 1.2005627155303955,
+      "eval_runtime": 148.1529,
+      "eval_samples_per_second": 7.243,
+      "eval_steps_per_second": 0.911,
+      "step": 1200
+    },
+    {
+      "epoch": 0.7294099917112501,
+      "grad_norm": 0.19388661605991633,
+      "learning_rate": 4.694273917837594e-05,
+      "loss": 1.2401,
+      "step": 1210
+    },
+    {
+      "epoch": 0.7354381734609299,
+      "grad_norm": 0.18789271845168903,
+      "learning_rate": 4.68581159919598e-05,
+      "loss": 1.2115,
+      "step": 1220
+    },
+    {
+      "epoch": 0.7414663552106096,
+      "grad_norm": 0.19218467798311695,
+      "learning_rate": 4.677241601673533e-05,
+      "loss": 1.2422,
+      "step": 1230
+    },
+    {
+      "epoch": 0.7474945369602893,
+      "grad_norm": 0.21068503293560067,
+      "learning_rate": 4.6685643474510734e-05,
+      "loss": 1.1999,
+      "step": 1240
+    },
+    {
+      "epoch": 0.7535227187099691,
+      "grad_norm": 0.19334737581694425,
+      "learning_rate": 4.659780263993177e-05,
+      "loss": 1.221,
+      "step": 1250
+    },
+    {
+      "epoch": 0.7595509004596489,
+      "grad_norm": 0.2048710364331125,
+      "learning_rate": 4.6508897840271086e-05,
+      "loss": 1.2278,
+      "step": 1260
+    },
+    {
+      "epoch": 0.7655790822093286,
+      "grad_norm": 0.2098690852913335,
+      "learning_rate": 4.641893345521508e-05,
+      "loss": 1.2167,
+      "step": 1270
+    },
+    {
+      "epoch": 0.7716072639590084,
+      "grad_norm": 0.2020939972391656,
+      "learning_rate": 4.6327913916648135e-05,
+      "loss": 1.2167,
+      "step": 1280
+    },
+    {
+      "epoch": 0.7776354457086881,
+      "grad_norm": 0.2074182675506127,
+      "learning_rate": 4.6235843708434315e-05,
+      "loss": 1.1917,
+      "step": 1290
+    },
+    {
+      "epoch": 0.7836636274583678,
+      "grad_norm": 0.20016936312104308,
+      "learning_rate": 4.6142727366196414e-05,
+      "loss": 1.2114,
+      "step": 1300
+    },
+    {
+      "epoch": 0.7896918092080476,
+      "grad_norm": 0.2042491026784662,
+      "learning_rate": 4.604856947709262e-05,
+      "loss": 1.1986,
+      "step": 1310
+    },
+    {
+      "epoch": 0.7957199909577274,
+      "grad_norm": 0.20627892432976563,
+      "learning_rate": 4.595337467959045e-05,
+      "loss": 1.2017,
+      "step": 1320
+    },
+    {
+      "epoch": 0.8017481727074072,
+      "grad_norm": 0.21410992106242302,
+      "learning_rate": 4.5857147663238304e-05,
+      "loss": 1.176,
+      "step": 1330
+    },
+    {
+      "epoch": 0.8077763544570868,
+      "grad_norm": 0.21385257654543013,
+      "learning_rate": 4.575989316843439e-05,
+      "loss": 1.161,
+      "step": 1340
+    },
+    {
+      "epoch": 0.8138045362067666,
+      "grad_norm": 0.21113493885158413,
+      "learning_rate": 4.566161598619325e-05,
+      "loss": 1.1943,
+      "step": 1350
+    },
+    {
+      "epoch": 0.8198327179564464,
+      "grad_norm": 0.21613784562223093,
+      "learning_rate": 4.556232095790972e-05,
+      "loss": 1.2033,
+      "step": 1360
+    },
+    {
+      "epoch": 0.8258608997061262,
+      "grad_norm": 0.2076738251013475,
+      "learning_rate": 4.5462012975120455e-05,
+      "loss": 1.1786,
+      "step": 1370
+    },
+    {
+      "epoch": 0.8318890814558059,
+      "grad_norm": 0.23044201851165344,
+      "learning_rate": 4.5360696979262905e-05,
+      "loss": 1.1796,
+      "step": 1380
+    },
+    {
+      "epoch": 0.8379172632054857,
+      "grad_norm": 0.22883008001886568,
+      "learning_rate": 4.525837796143194e-05,
+      "loss": 1.1728,
+      "step": 1390
+    },
+    {
+      "epoch": 0.8439454449551654,
+      "grad_norm": 0.2306946921683779,
+      "learning_rate": 4.5155060962133945e-05,
+      "loss": 1.1764,
+      "step": 1400
+    },
+    {
+      "epoch": 0.8499736267048451,
+      "grad_norm": 0.23157892679877035,
+      "learning_rate": 4.505075107103852e-05,
+      "loss": 1.1832,
+      "step": 1410
+    },
+    {
+      "epoch": 0.8560018084545249,
+      "grad_norm": 0.21742897363969343,
+      "learning_rate": 4.494545342672775e-05,
+      "loss": 1.1734,
+      "step": 1420
+    },
+    {
+      "epoch": 0.8620299902042047,
+      "grad_norm": 0.22825192812741035,
+      "learning_rate": 4.483917321644309e-05,
+      "loss": 1.1736,
+      "step": 1430
+    },
+    {
+      "epoch": 0.8680581719538845,
+      "grad_norm": 0.2275167699698394,
+      "learning_rate": 4.473191567582975e-05,
+      "loss": 1.1658,
+      "step": 1440
+    },
+    {
+      "epoch": 0.8740863537035641,
+      "grad_norm": 0.21961154395854318,
+      "learning_rate": 4.462368608867889e-05,
+      "loss": 1.1845,
+      "step": 1450
+    },
+    {
+      "epoch": 0.8801145354532439,
+      "grad_norm": 0.21525449869450775,
+      "learning_rate": 4.4514489786667205e-05,
+      "loss": 1.1418,
+      "step": 1460
+    },
+    {
+      "epoch": 0.8861427172029237,
+      "grad_norm": 0.22332875240997163,
+      "learning_rate": 4.4404332149094365e-05,
+      "loss": 1.1777,
+      "step": 1470
+    },
+    {
+      "epoch": 0.8921708989526034,
+      "grad_norm": 0.22901308144060187,
+      "learning_rate": 4.429321860261797e-05,
+      "loss": 1.167,
+      "step": 1480
+    },
+    {
+      "epoch": 0.8981990807022832,
+      "grad_norm": 0.22878617308256047,
+      "learning_rate": 4.418115462098624e-05,
+      "loss": 1.1532,
+      "step": 1490
+    },
+    {
+      "epoch": 0.9042272624519629,
+      "grad_norm": 0.2405276526631136,
+      "learning_rate": 4.4068145724768324e-05,
+      "loss": 1.1206,
+      "step": 1500
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 4974,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2522009359613952.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1500/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2923841ae243af669d09fc75f56460b83329158b7b6379ad1b1afd5c369cdb7a
+size 7224

checkpoint-1500/zero_to_fp32.py

@@ -0,0 +1,674 @@
+#!/usr/bin/env python
+
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: Apache-2.0
+
+# DeepSpeed Team
+
+# This script extracts fp32 consolidated weights from a zero 1, 2 and 3 DeepSpeed checkpoints. It gets
+# copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
+# the future. Once extracted, the weights don't require DeepSpeed and can be used in any
+# application.
+#
+# example:
+#   python zero_to_fp32.py . output_dir/
+#   or
+#   python zero_to_fp32.py . output_dir/ --safe_serialization
+
+import argparse
+import torch
+import glob
+import math
+import os
+import re
+import json
+from tqdm import tqdm
+from collections import OrderedDict
+from dataclasses import dataclass
+
+# while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
+# DeepSpeed data structures it has to be available in the current python environment.
+from deepspeed.utils import logger
+from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
+                                            FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
+                                            FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
+
+
+@dataclass
+class zero_model_state:
+    buffers: dict()
+    param_shapes: dict()
+    shared_params: list
+    ds_version: int
+    frozen_param_shapes: dict()
+    frozen_param_fragments: dict()
+
+
+debug = 0
+
+# load to cpu
+device = torch.device('cpu')
+
+
+def atoi(text):
+    return int(text) if text.isdigit() else text
+
+
+def natural_keys(text):
+    '''
+    alist.sort(key=natural_keys) sorts in human order
+    http://nedbatchelder.com/blog/200712/human_sorting.html
+    (See Toothy's implementation in the comments)
+    '''
+    return [atoi(c) for c in re.split(r'(\d+)', text)]
+
+
+def get_model_state_file(checkpoint_dir, zero_stage):
+    if not os.path.isdir(checkpoint_dir):
+        raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
+
+    # there should be only one file
+    if zero_stage <= 2:
+        file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
+    elif zero_stage == 3:
+        file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
+
+    if not os.path.exists(file):
+        raise FileNotFoundError(f"can't find model states file at '{file}'")
+
+    return file
+
+
+def get_checkpoint_files(checkpoint_dir, glob_pattern):
+    # XXX: need to test that this simple glob rule works for multi-node setup too
+    ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
+
+    if len(ckpt_files) == 0:
+        raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
+
+    return ckpt_files
+
+
+def get_optim_files(checkpoint_dir):
+    return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
+
+
+def get_model_state_files(checkpoint_dir):
+    return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
+
+
+def parse_model_states(files):
+    zero_model_states = []
+    for file in files:
+        state_dict = torch.load(file, map_location=device)
+
+        if BUFFER_NAMES not in state_dict:
+            raise ValueError(f"{file} is not a model state checkpoint")
+        buffer_names = state_dict[BUFFER_NAMES]
+        if debug:
+            print("Found buffers:", buffer_names)
+
+        # recover just the buffers while restoring them to fp32 if they were saved in fp16
+        buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
+        param_shapes = state_dict[PARAM_SHAPES]
+
+        # collect parameters that are included in param_shapes
+        param_names = []
+        for s in param_shapes:
+            for name in s.keys():
+                param_names.append(name)
+
+        # update with frozen parameters
+        frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
+        if frozen_param_shapes is not None:
+            if debug:
+                print(f"Found frozen_param_shapes: {frozen_param_shapes}")
+            param_names += list(frozen_param_shapes.keys())
+
+        # handle shared params
+        shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
+
+        ds_version = state_dict.get(DS_VERSION, None)
+
+        frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
+
+        z_model_state = zero_model_state(buffers=buffers,
+                                         param_shapes=param_shapes,
+                                         shared_params=shared_params,
+                                         ds_version=ds_version,
+                                         frozen_param_shapes=frozen_param_shapes,
+                                         frozen_param_fragments=frozen_param_fragments)
+        zero_model_states.append(z_model_state)
+
+    return zero_model_states
+
+
+def parse_optim_states(files, ds_checkpoint_dir):
+    total_files = len(files)
+    state_dicts = []
+    for f in files:
+        state_dict = torch.load(f, map_location=device)
+        # immediately discard the potentially huge 2 optimizer states as we only care for fp32 master weights
+        # and also handle the case where it was already removed by another helper script
+        state_dict["optimizer_state_dict"].pop("optimizer_state_dict", None)
+        state_dicts.append(state_dict)
+
+    if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
+        raise ValueError(f"{files[0]} is not a zero checkpoint")
+    zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
+    world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
+
+    # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
+    # parameters can be different from data parallelism for non-expert parameters. So we can just
+    # use the max of the partition_count to get the dp world_size.
+
+    if type(world_size) is list:
+        world_size = max(world_size)
+
+    if world_size != total_files:
+        raise ValueError(
+            f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
+            "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
+        )
+
+    # the groups are named differently in each stage
+    if zero_stage <= 2:
+        fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
+    elif zero_stage == 3:
+        fp32_groups_key = FP32_FLAT_GROUPS
+    else:
+        raise ValueError(f"unknown zero stage {zero_stage}")
+
+    if zero_stage <= 2:
+        fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
+    elif zero_stage == 3:
+        # if there is more than one param group, there will be multiple flattened tensors - one
+        # flattened tensor per group - for simplicity merge them into a single tensor
+        #
+        # XXX: could make the script more memory efficient for when there are multiple groups - it
+        # will require matching the sub-lists of param_shapes for each param group flattened tensor
+
+        fp32_flat_groups = [
+            torch.cat(state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key], 0) for i in range(len(state_dicts))
+        ]
+
+    return zero_stage, world_size, fp32_flat_groups
+
+
+def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters):
+    """
+    Returns fp32 state_dict reconstructed from ds checkpoint
+
+    Args:
+        - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
+
+    """
+    print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
+
+    optim_files = get_optim_files(ds_checkpoint_dir)
+    zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
+    print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
+
+    model_files = get_model_state_files(ds_checkpoint_dir)
+
+    zero_model_states = parse_model_states(model_files)
+    print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
+
+    if zero_stage <= 2:
+        return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                                          exclude_frozen_parameters)
+    elif zero_stage == 3:
+        return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                                          exclude_frozen_parameters)
+
+
+def _zero2_merge_frozen_params(state_dict, zero_model_states):
+    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
+        return
+
+    frozen_param_shapes = zero_model_states[0].frozen_param_shapes
+    frozen_param_fragments = zero_model_states[0].frozen_param_fragments
+
+    if debug:
+        num_elem = sum(s.numel() for s in frozen_param_shapes.values())
+        print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
+
+        wanted_params = len(frozen_param_shapes)
+        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
+        avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
+        print(f'Frozen params: Have {avail_numel} numels to process.')
+        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
+
+    total_params = 0
+    total_numel = 0
+    for name, shape in frozen_param_shapes.items():
+        total_params += 1
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+
+        state_dict[name] = frozen_param_fragments[name]
+
+        if debug:
+            print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
+
+    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _has_callable(obj, fn):
+    attr = getattr(obj, fn, None)
+    return callable(attr)
+
+
+def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
+    param_shapes = zero_model_states[0].param_shapes
+
+    # Reconstruction protocol:
+    #
+    # XXX: document this
+
+    if debug:
+        for i in range(world_size):
+            for j in range(len(fp32_flat_groups[0])):
+                print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
+
+    # XXX: memory usage doubles here (zero2)
+    num_param_groups = len(fp32_flat_groups[0])
+    merged_single_partition_of_fp32_groups = []
+    for i in range(num_param_groups):
+        merged_partitions = [sd[i] for sd in fp32_flat_groups]
+        full_single_fp32_vector = torch.cat(merged_partitions, 0)
+        merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
+    avail_numel = sum(
+        [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
+
+    if debug:
+        wanted_params = sum([len(shapes) for shapes in param_shapes])
+        wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
+        # not asserting if there is a mismatch due to possible padding
+        print(f"Have {avail_numel} numels to process.")
+        print(f"Need {wanted_numel} numels in {wanted_params} params.")
+
+    # params
+    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
+    # out-of-core computing solution
+    total_numel = 0
+    total_params = 0
+    for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
+        offset = 0
+        avail_numel = full_single_fp32_vector.numel()
+        for name, shape in shapes.items():
+
+            unpartitioned_numel = shape.numel() if _has_callable(shape, 'numel') else math.prod(shape)
+            total_numel += unpartitioned_numel
+            total_params += 1
+
+            if debug:
+                print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
+            state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
+            offset += unpartitioned_numel
+
+        # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
+        # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
+        # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
+        # live optimizer object, so we are checking that the numbers are within the right range
+        align_to = 2 * world_size
+
+        def zero2_align(x):
+            return align_to * math.ceil(x / align_to)
+
+        if debug:
+            print(f"original offset={offset}, avail_numel={avail_numel}")
+
+        offset = zero2_align(offset)
+        avail_numel = zero2_align(avail_numel)
+
+        if debug:
+            print(f"aligned  offset={offset}, avail_numel={avail_numel}")
+
+        # Sanity check
+        if offset != avail_numel:
+            raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
+
+    print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                               exclude_frozen_parameters):
+    state_dict = OrderedDict()
+
+    # buffers
+    buffers = zero_model_states[0].buffers
+    state_dict.update(buffers)
+    if debug:
+        print(f"added {len(buffers)} buffers")
+
+    if not exclude_frozen_parameters:
+        _zero2_merge_frozen_params(state_dict, zero_model_states)
+
+    _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
+
+    # recover shared parameters
+    for pair in zero_model_states[0].shared_params:
+        if pair[1] in state_dict:
+            state_dict[pair[0]] = state_dict[pair[1]]
+
+    return state_dict
+
+
+def zero3_partitioned_param_info(unpartitioned_numel, world_size):
+    remainder = unpartitioned_numel % world_size
+    padding_numel = (world_size - remainder) if remainder else 0
+    partitioned_numel = math.ceil(unpartitioned_numel / world_size)
+    return partitioned_numel, padding_numel
+
+
+def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
+    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
+        return
+
+    if debug:
+        for i in range(world_size):
+            num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
+            print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
+
+        frozen_param_shapes = zero_model_states[0].frozen_param_shapes
+        wanted_params = len(frozen_param_shapes)
+        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
+        avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
+        print(f'Frozen params: Have {avail_numel} numels to process.')
+        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
+
+    total_params = 0
+    total_numel = 0
+    for name, shape in zero_model_states[0].frozen_param_shapes.items():
+        total_params += 1
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+
+        param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
+        state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
+
+        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
+
+        if debug:
+            print(
+                f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
+            )
+
+    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
+    param_shapes = zero_model_states[0].param_shapes
+    avail_numel = fp32_flat_groups[0].numel() * world_size
+    # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
+    # param, re-consolidating each param, while dealing with padding if any
+
+    # merge list of dicts, preserving order
+    param_shapes = {k: v for d in param_shapes for k, v in d.items()}
+
+    if debug:
+        for i in range(world_size):
+            print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
+
+        wanted_params = len(param_shapes)
+        wanted_numel = sum(shape.numel() for shape in param_shapes.values())
+        # not asserting if there is a mismatch due to possible padding
+        avail_numel = fp32_flat_groups[0].numel() * world_size
+        print(f"Trainable params: Have {avail_numel} numels to process.")
+        print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
+
+    # params
+    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
+    # out-of-core computing solution
+    offset = 0
+    total_numel = 0
+    total_params = 0
+    for name, shape in tqdm(param_shapes.items(), desc='Gathering Sharded Weights'):
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+        total_params += 1
+        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
+
+        if debug:
+            print(
+                f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
+            )
+
+        # XXX: memory usage doubles here
+        state_dict[name] = torch.cat(
+            tuple(fp32_flat_groups[i].narrow(0, offset, partitioned_numel) for i in range(world_size)),
+            0).narrow(0, 0, unpartitioned_numel).view(shape)
+        offset += partitioned_numel
+
+    offset *= world_size
+
+    # Sanity check
+    if offset != avail_numel:
+        raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
+
+    print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                               exclude_frozen_parameters):
+    state_dict = OrderedDict()
+
+    # buffers
+    buffers = zero_model_states[0].buffers
+    state_dict.update(buffers)
+    if debug:
+        print(f"added {len(buffers)} buffers")
+
+    if not exclude_frozen_parameters:
+        _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
+
+    _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
+
+    # recover shared parameters
+    for pair in zero_model_states[0].shared_params:
+        if pair[1] in state_dict:
+            state_dict[pair[0]] = state_dict[pair[1]]
+
+    return state_dict
+
+
+def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag=None, exclude_frozen_parameters=False):
+    """
+    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
+    ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
+    via a model hub.
+
+    Args:
+        - ``checkpoint_dir``: path to the desired checkpoint folder
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
+        - ``exclude_frozen_parameters``: exclude frozen parameters
+
+    Returns:
+        - pytorch ``state_dict``
+
+    Note: this approach may not work if your application doesn't have sufficient free CPU memory and
+    you may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
+    the checkpoint.
+
+    A typical usage might be ::
+
+        from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
+        # do the training and checkpoint saving
+        state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
+        model = model.cpu() # move to cpu
+        model.load_state_dict(state_dict)
+        # submit to model hub or save the model to share with others
+
+    In this example the ``model`` will no longer be usable in the deepspeed context of the same
+    application. i.e. you will need to re-initialize the deepspeed engine, since
+    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
+
+    If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
+
+    """
+    if tag is None:
+        latest_path = os.path.join(checkpoint_dir, 'latest')
+        if os.path.isfile(latest_path):
+            with open(latest_path, 'r') as fd:
+                tag = fd.read().strip()
+        else:
+            raise ValueError(f"Unable to find 'latest' file at {latest_path}")
+
+    ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
+
+    if not os.path.isdir(ds_checkpoint_dir):
+        raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
+
+    return _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters)
+
+
+def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir,
+                                               output_dir,
+                                               max_shard_size="5GB",
+                                               safe_serialization=False,
+                                               tag=None,
+                                               exclude_frozen_parameters=False):
+    """
+    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
+    loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
+
+    Args:
+        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
+        - ``output_dir``: directory to the pytorch fp32 state_dict output files
+        - ``max_shard_size``: the maximum size for a checkpoint before being sharded, default value is 5GB
+        - ``safe_serialization``:  whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
+        - ``exclude_frozen_parameters``: exclude frozen parameters
+    """
+    # Dependency pre-check
+    if safe_serialization:
+        try:
+            from safetensors.torch import save_file
+        except ImportError:
+            print('If you want to use `safe_serialization`, please `pip install safetensors`')
+            raise
+    if max_shard_size is not None:
+        try:
+            from huggingface_hub import split_torch_state_dict_into_shards
+        except ImportError:
+            print('If you want to use `max_shard_size`, please `pip install huggingface_hub`')
+            raise
+
+    # Convert zero checkpoint to state_dict
+    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag, exclude_frozen_parameters)
+
+    # Shard the model if it is too big.
+    weights_name = "model.safetensors" if safe_serialization else "pytorch_model.bin"
+    if max_shard_size is not None:
+        filename_pattern = weights_name.replace(".bin", "{suffix}.bin").replace(".safetensors", "{suffix}.safetensors")
+        state_dict_split = split_torch_state_dict_into_shards(state_dict,
+                                                              filename_pattern=filename_pattern,
+                                                              max_shard_size=max_shard_size)
+    else:
+        from collections import namedtuple
+        StateDictSplit = namedtuple("StateDictSplit", ["is_sharded", "filename_to_tensors"])
+        state_dict_split = StateDictSplit(is_sharded=False,
+                                          filename_to_tensors={weights_name: list(state_dict.keys())})
+
+    # Save the model
+    filename_to_tensors = state_dict_split.filename_to_tensors.items()
+    for shard_file, tensors in tqdm(filename_to_tensors, desc="Saving checkpoint shards"):
+        shard = {tensor: state_dict[tensor].contiguous() for tensor in tensors}
+        output_path = os.path.join(output_dir, shard_file)
+        if safe_serialization:
+            save_file(shard, output_path, metadata={"format": "pt"})
+        else:
+            torch.save(shard, output_path)
+
+    # Save index if sharded
+    if state_dict_split.is_sharded:
+        index = {
+            "metadata": state_dict_split.metadata,
+            "weight_map": state_dict_split.tensor_to_filename,
+        }
+        save_index_file = "model.safetensors.index.json" if safe_serialization else "pytorch_model.bin.index.json"
+        save_index_file = os.path.join(output_dir, save_index_file)
+        with open(save_index_file, "w", encoding="utf-8") as f:
+            content = json.dumps(index, indent=2, sort_keys=True) + "\n"
+            f.write(content)
+
+
+def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
+    """
+    1. Put the provided model to cpu
+    2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
+    3. Load it into the provided model
+
+    Args:
+        - ``model``: the model object to update
+        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
+
+    Returns:
+        - ``model`: modified model
+
+    Make sure you have plenty of CPU memory available before you call this function. If you don't
+    have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
+    conveniently placed for you in the checkpoint folder.
+
+    A typical usage might be ::
+
+        from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
+        model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
+        # submit to model hub or save the model to share with others
+
+    Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
+    of the same application. i.e. you will need to re-initialize the deepspeed engine, since
+    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
+
+    """
+    logger.info(f"Extracting fp32 weights")
+    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
+
+    logger.info(f"Overwriting model with fp32 weights")
+    model = model.cpu()
+    model.load_state_dict(state_dict, strict=False)
+
+    return model
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("checkpoint_dir",
+                        type=str,
+                        help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
+    parser.add_argument("output_dir",
+                        type=str,
+                        help="directory to the pytorch fp32 state_dict output files"
+                        "(e.g. path/checkpoint-12-output/)")
+    parser.add_argument(
+        "--max_shard_size",
+        type=str,
+        default="5GB",
+        help="The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size"
+        "lower than this size. If expressed as a string, needs to be digits followed by a unit (like `5MB`"
+        "We default it to 5GB in order for models to be able to run easily on free-tier google colab instances"
+        "without CPU OOM issues.")
+    parser.add_argument(
+        "--safe_serialization",
+        default=False,
+        action='store_true',
+        help="Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).")
+    parser.add_argument("-t",
+                        "--tag",
+                        type=str,
+                        default=None,
+                        help="checkpoint tag used as a unique identifier for checkpoint. e.g., global_step1")
+    parser.add_argument("--exclude_frozen_parameters", action='store_true', help="exclude frozen parameters")
+    parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
+    args = parser.parse_args()
+
+    debug = args.debug
+
+    convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir,
+                                               args.output_dir,
+                                               max_shard_size=args.max_shard_size,
+                                               safe_serialization=args.safe_serialization,
+                                               tag=args.tag,
+                                               exclude_frozen_parameters=args.exclude_frozen_parameters)

checkpoint-500/README.md

@@ -0,0 +1,202 @@
+---
+base_model: Qwen/Qwen2.5-72B
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.0

checkpoint-500/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2.5-72B",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "v_proj",
+    "k_proj",
+    "down_proj",
+    "o_proj",
+    "q_proj",
+    "up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": true
+}

checkpoint-500/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a51a4b4bb16b47c962bc104cfacda7510d0eedc226edd0475b52b5583e2fdba
+size 3368705968

checkpoint-500/added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

checkpoint-500/latest

@@ -0,0 +1 @@
+global_step500

checkpoint-500/special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-500/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
+size 11421896

checkpoint-500/tokenizer_config.json

@@ -0,0 +1,208 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

checkpoint-500/trainer_state.json

@@ -0,0 +1,391 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.30140908748398765,
+  "eval_steps": 400,
+  "global_step": 500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.006028181749679753,
+      "grad_norm": 0.38845974438826136,
+      "learning_rate": 1.0040160642570282e-06,
+      "loss": 1.6618,
+      "step": 10
+    },
+    {
+      "epoch": 0.012056363499359506,
+      "grad_norm": 0.255603405930248,
+      "learning_rate": 2.0080321285140564e-06,
+      "loss": 1.6698,
+      "step": 20
+    },
+    {
+      "epoch": 0.01808454524903926,
+      "grad_norm": 0.20116505722657768,
+      "learning_rate": 3.0120481927710846e-06,
+      "loss": 1.6264,
+      "step": 30
+    },
+    {
+      "epoch": 0.02411272699871901,
+      "grad_norm": 0.1795881732266397,
+      "learning_rate": 4.016064257028113e-06,
+      "loss": 1.6025,
+      "step": 40
+    },
+    {
+      "epoch": 0.030140908748398764,
+      "grad_norm": 0.14600421495766222,
+      "learning_rate": 5.020080321285141e-06,
+      "loss": 1.5584,
+      "step": 50
+    },
+    {
+      "epoch": 0.03616909049807852,
+      "grad_norm": 0.1170942466306718,
+      "learning_rate": 6.024096385542169e-06,
+      "loss": 1.5604,
+      "step": 60
+    },
+    {
+      "epoch": 0.04219727224775827,
+      "grad_norm": 0.2445974669666656,
+      "learning_rate": 7.028112449799197e-06,
+      "loss": 1.5057,
+      "step": 70
+    },
+    {
+      "epoch": 0.04822545399743802,
+      "grad_norm": 0.11238025351136724,
+      "learning_rate": 8.032128514056226e-06,
+      "loss": 1.4724,
+      "step": 80
+    },
+    {
+      "epoch": 0.05425363574711778,
+      "grad_norm": 0.13639577534070754,
+      "learning_rate": 9.036144578313253e-06,
+      "loss": 1.4962,
+      "step": 90
+    },
+    {
+      "epoch": 0.06028181749679753,
+      "grad_norm": 0.17113178431310286,
+      "learning_rate": 1.0040160642570281e-05,
+      "loss": 1.4765,
+      "step": 100
+    },
+    {
+      "epoch": 0.06630999924647728,
+      "grad_norm": 0.4795401637258333,
+      "learning_rate": 1.104417670682731e-05,
+      "loss": 1.4468,
+      "step": 110
+    },
+    {
+      "epoch": 0.07233818099615703,
+      "grad_norm": 0.13385762516900662,
+      "learning_rate": 1.2048192771084338e-05,
+      "loss": 1.4134,
+      "step": 120
+    },
+    {
+      "epoch": 0.07836636274583679,
+      "grad_norm": 0.12303707813666019,
+      "learning_rate": 1.3052208835341367e-05,
+      "loss": 1.4191,
+      "step": 130
+    },
+    {
+      "epoch": 0.08439454449551655,
+      "grad_norm": 0.10822073133399364,
+      "learning_rate": 1.4056224899598394e-05,
+      "loss": 1.397,
+      "step": 140
+    },
+    {
+      "epoch": 0.09042272624519629,
+      "grad_norm": 0.1109270134990499,
+      "learning_rate": 1.5060240963855424e-05,
+      "loss": 1.3818,
+      "step": 150
+    },
+    {
+      "epoch": 0.09645090799487605,
+      "grad_norm": 0.14622173134033867,
+      "learning_rate": 1.606425702811245e-05,
+      "loss": 1.4002,
+      "step": 160
+    },
+    {
+      "epoch": 0.1024790897445558,
+      "grad_norm": 0.10587626114414271,
+      "learning_rate": 1.706827309236948e-05,
+      "loss": 1.4123,
+      "step": 170
+    },
+    {
+      "epoch": 0.10850727149423556,
+      "grad_norm": 0.10302196814593138,
+      "learning_rate": 1.8072289156626505e-05,
+      "loss": 1.4016,
+      "step": 180
+    },
+    {
+      "epoch": 0.1145354532439153,
+      "grad_norm": 0.1359849724314843,
+      "learning_rate": 1.9076305220883535e-05,
+      "loss": 1.404,
+      "step": 190
+    },
+    {
+      "epoch": 0.12056363499359506,
+      "grad_norm": 0.10587622358885339,
+      "learning_rate": 2.0080321285140562e-05,
+      "loss": 1.4019,
+      "step": 200
+    },
+    {
+      "epoch": 0.1265918167432748,
+      "grad_norm": 0.15017595066321648,
+      "learning_rate": 2.1084337349397593e-05,
+      "loss": 1.393,
+      "step": 210
+    },
+    {
+      "epoch": 0.13261999849295456,
+      "grad_norm": 0.19475575142022897,
+      "learning_rate": 2.208835341365462e-05,
+      "loss": 1.3876,
+      "step": 220
+    },
+    {
+      "epoch": 0.1386481802426343,
+      "grad_norm": 0.12084095277263424,
+      "learning_rate": 2.309236947791165e-05,
+      "loss": 1.3916,
+      "step": 230
+    },
+    {
+      "epoch": 0.14467636199231407,
+      "grad_norm": 0.11857482977859173,
+      "learning_rate": 2.4096385542168677e-05,
+      "loss": 1.4056,
+      "step": 240
+    },
+    {
+      "epoch": 0.15070454374199382,
+      "grad_norm": 0.1403959719635503,
+      "learning_rate": 2.5100401606425704e-05,
+      "loss": 1.3935,
+      "step": 250
+    },
+    {
+      "epoch": 0.15673272549167358,
+      "grad_norm": 0.10800155257965392,
+      "learning_rate": 2.6104417670682734e-05,
+      "loss": 1.3826,
+      "step": 260
+    },
+    {
+      "epoch": 0.16276090724135334,
+      "grad_norm": 0.10598439909830581,
+      "learning_rate": 2.7108433734939758e-05,
+      "loss": 1.3999,
+      "step": 270
+    },
+    {
+      "epoch": 0.1687890889910331,
+      "grad_norm": 0.10753449693494475,
+      "learning_rate": 2.8112449799196788e-05,
+      "loss": 1.4047,
+      "step": 280
+    },
+    {
+      "epoch": 0.17481727074071282,
+      "grad_norm": 0.36718328659037996,
+      "learning_rate": 2.911646586345382e-05,
+      "loss": 1.3935,
+      "step": 290
+    },
+    {
+      "epoch": 0.18084545249039258,
+      "grad_norm": 0.10611900000479042,
+      "learning_rate": 3.012048192771085e-05,
+      "loss": 1.3736,
+      "step": 300
+    },
+    {
+      "epoch": 0.18687363424007233,
+      "grad_norm": 0.11901555220652378,
+      "learning_rate": 3.112449799196787e-05,
+      "loss": 1.3927,
+      "step": 310
+    },
+    {
+      "epoch": 0.1929018159897521,
+      "grad_norm": 0.118935148513695,
+      "learning_rate": 3.21285140562249e-05,
+      "loss": 1.3636,
+      "step": 320
+    },
+    {
+      "epoch": 0.19892999773943185,
+      "grad_norm": 0.1974545721831922,
+      "learning_rate": 3.313253012048193e-05,
+      "loss": 1.3892,
+      "step": 330
+    },
+    {
+      "epoch": 0.2049581794891116,
+      "grad_norm": 0.13145409772199562,
+      "learning_rate": 3.413654618473896e-05,
+      "loss": 1.3756,
+      "step": 340
+    },
+    {
+      "epoch": 0.21098636123879136,
+      "grad_norm": 0.11064380941915805,
+      "learning_rate": 3.5140562248995983e-05,
+      "loss": 1.3935,
+      "step": 350
+    },
+    {
+      "epoch": 0.21701454298847112,
+      "grad_norm": 0.12160423827639648,
+      "learning_rate": 3.614457831325301e-05,
+      "loss": 1.3698,
+      "step": 360
+    },
+    {
+      "epoch": 0.22304272473815084,
+      "grad_norm": 0.10349641889173723,
+      "learning_rate": 3.7148594377510044e-05,
+      "loss": 1.3771,
+      "step": 370
+    },
+    {
+      "epoch": 0.2290709064878306,
+      "grad_norm": 0.10682144059511894,
+      "learning_rate": 3.815261044176707e-05,
+      "loss": 1.3768,
+      "step": 380
+    },
+    {
+      "epoch": 0.23509908823751036,
+      "grad_norm": 0.11625245619819907,
+      "learning_rate": 3.91566265060241e-05,
+      "loss": 1.3795,
+      "step": 390
+    },
+    {
+      "epoch": 0.2411272699871901,
+      "grad_norm": 0.10327726962763091,
+      "learning_rate": 4.0160642570281125e-05,
+      "loss": 1.3987,
+      "step": 400
+    },
+    {
+      "epoch": 0.2411272699871901,
+      "eval_loss": 1.3548544645309448,
+      "eval_runtime": 148.2269,
+      "eval_samples_per_second": 7.239,
+      "eval_steps_per_second": 0.911,
+      "step": 400
+    },
+    {
+      "epoch": 0.24715545173686987,
+      "grad_norm": 0.10660530950921367,
+      "learning_rate": 4.116465863453816e-05,
+      "loss": 1.3886,
+      "step": 410
+    },
+    {
+      "epoch": 0.2531836334865496,
+      "grad_norm": 0.10405582985373843,
+      "learning_rate": 4.2168674698795186e-05,
+      "loss": 1.3645,
+      "step": 420
+    },
+    {
+      "epoch": 0.2592118152362294,
+      "grad_norm": 0.3318479326670041,
+      "learning_rate": 4.317269076305221e-05,
+      "loss": 1.3591,
+      "step": 430
+    },
+    {
+      "epoch": 0.2652399969859091,
+      "grad_norm": 0.10840544026201794,
+      "learning_rate": 4.417670682730924e-05,
+      "loss": 1.3805,
+      "step": 440
+    },
+    {
+      "epoch": 0.2712681787355889,
+      "grad_norm": 0.10730056620740543,
+      "learning_rate": 4.5180722891566266e-05,
+      "loss": 1.3888,
+      "step": 450
+    },
+    {
+      "epoch": 0.2772963604852686,
+      "grad_norm": 0.10699620793474768,
+      "learning_rate": 4.61847389558233e-05,
+      "loss": 1.3935,
+      "step": 460
+    },
+    {
+      "epoch": 0.2833245422349484,
+      "grad_norm": 0.10595493402596641,
+      "learning_rate": 4.718875502008032e-05,
+      "loss": 1.3659,
+      "step": 470
+    },
+    {
+      "epoch": 0.28935272398462814,
+      "grad_norm": 0.14234040947748414,
+      "learning_rate": 4.8192771084337354e-05,
+      "loss": 1.371,
+      "step": 480
+    },
+    {
+      "epoch": 0.29538090573430786,
+      "grad_norm": 0.1095349792774781,
+      "learning_rate": 4.919678714859438e-05,
+      "loss": 1.3647,
+      "step": 490
+    },
+    {
+      "epoch": 0.30140908748398765,
+      "grad_norm": 0.10655792946130023,
+      "learning_rate": 4.999997536857586e-05,
+      "loss": 1.3606,
+      "step": 500
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 4974,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 840529663229952.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-500/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2923841ae243af669d09fc75f56460b83329158b7b6379ad1b1afd5c369cdb7a
+size 7224

checkpoint-500/zero_to_fp32.py

@@ -0,0 +1,674 @@
+#!/usr/bin/env python
+
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: Apache-2.0
+
+# DeepSpeed Team
+
+# This script extracts fp32 consolidated weights from a zero 1, 2 and 3 DeepSpeed checkpoints. It gets
+# copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
+# the future. Once extracted, the weights don't require DeepSpeed and can be used in any
+# application.
+#
+# example:
+#   python zero_to_fp32.py . output_dir/
+#   or
+#   python zero_to_fp32.py . output_dir/ --safe_serialization
+
+import argparse
+import torch
+import glob
+import math
+import os
+import re
+import json
+from tqdm import tqdm
+from collections import OrderedDict
+from dataclasses import dataclass
+
+# while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
+# DeepSpeed data structures it has to be available in the current python environment.
+from deepspeed.utils import logger
+from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
+                                            FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
+                                            FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
+
+
+@dataclass
+class zero_model_state:
+    buffers: dict()
+    param_shapes: dict()
+    shared_params: list
+    ds_version: int
+    frozen_param_shapes: dict()
+    frozen_param_fragments: dict()
+
+
+debug = 0
+
+# load to cpu
+device = torch.device('cpu')
+
+
+def atoi(text):
+    return int(text) if text.isdigit() else text
+
+
+def natural_keys(text):
+    '''
+    alist.sort(key=natural_keys) sorts in human order
+    http://nedbatchelder.com/blog/200712/human_sorting.html
+    (See Toothy's implementation in the comments)
+    '''
+    return [atoi(c) for c in re.split(r'(\d+)', text)]
+
+
+def get_model_state_file(checkpoint_dir, zero_stage):
+    if not os.path.isdir(checkpoint_dir):
+        raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
+
+    # there should be only one file
+    if zero_stage <= 2:
+        file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
+    elif zero_stage == 3:
+        file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
+
+    if not os.path.exists(file):
+        raise FileNotFoundError(f"can't find model states file at '{file}'")
+
+    return file
+
+
+def get_checkpoint_files(checkpoint_dir, glob_pattern):
+    # XXX: need to test that this simple glob rule works for multi-node setup too
+    ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
+
+    if len(ckpt_files) == 0:
+        raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
+
+    return ckpt_files
+
+
+def get_optim_files(checkpoint_dir):
+    return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
+
+
+def get_model_state_files(checkpoint_dir):
+    return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
+
+
+def parse_model_states(files):
+    zero_model_states = []
+    for file in files:
+        state_dict = torch.load(file, map_location=device)
+
+        if BUFFER_NAMES not in state_dict:
+            raise ValueError(f"{file} is not a model state checkpoint")
+        buffer_names = state_dict[BUFFER_NAMES]
+        if debug:
+            print("Found buffers:", buffer_names)
+
+        # recover just the buffers while restoring them to fp32 if they were saved in fp16
+        buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
+        param_shapes = state_dict[PARAM_SHAPES]
+
+        # collect parameters that are included in param_shapes
+        param_names = []
+        for s in param_shapes:
+            for name in s.keys():
+                param_names.append(name)
+
+        # update with frozen parameters
+        frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
+        if frozen_param_shapes is not None:
+            if debug:
+                print(f"Found frozen_param_shapes: {frozen_param_shapes}")
+            param_names += list(frozen_param_shapes.keys())
+
+        # handle shared params
+        shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
+
+        ds_version = state_dict.get(DS_VERSION, None)
+
+        frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
+
+        z_model_state = zero_model_state(buffers=buffers,
+                                         param_shapes=param_shapes,
+                                         shared_params=shared_params,
+                                         ds_version=ds_version,
+                                         frozen_param_shapes=frozen_param_shapes,
+                                         frozen_param_fragments=frozen_param_fragments)
+        zero_model_states.append(z_model_state)
+
+    return zero_model_states
+
+
+def parse_optim_states(files, ds_checkpoint_dir):
+    total_files = len(files)
+    state_dicts = []
+    for f in files:
+        state_dict = torch.load(f, map_location=device)
+        # immediately discard the potentially huge 2 optimizer states as we only care for fp32 master weights
+        # and also handle the case where it was already removed by another helper script
+        state_dict["optimizer_state_dict"].pop("optimizer_state_dict", None)
+        state_dicts.append(state_dict)
+
+    if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
+        raise ValueError(f"{files[0]} is not a zero checkpoint")
+    zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
+    world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
+
+    # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
+    # parameters can be different from data parallelism for non-expert parameters. So we can just
+    # use the max of the partition_count to get the dp world_size.
+
+    if type(world_size) is list:
+        world_size = max(world_size)
+
+    if world_size != total_files:
+        raise ValueError(
+            f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
+            "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
+        )
+
+    # the groups are named differently in each stage
+    if zero_stage <= 2:
+        fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
+    elif zero_stage == 3:
+        fp32_groups_key = FP32_FLAT_GROUPS
+    else:
+        raise ValueError(f"unknown zero stage {zero_stage}")
+
+    if zero_stage <= 2:
+        fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
+    elif zero_stage == 3:
+        # if there is more than one param group, there will be multiple flattened tensors - one
+        # flattened tensor per group - for simplicity merge them into a single tensor
+        #
+        # XXX: could make the script more memory efficient for when there are multiple groups - it
+        # will require matching the sub-lists of param_shapes for each param group flattened tensor
+
+        fp32_flat_groups = [
+            torch.cat(state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key], 0) for i in range(len(state_dicts))
+        ]
+
+    return zero_stage, world_size, fp32_flat_groups
+
+
+def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters):
+    """
+    Returns fp32 state_dict reconstructed from ds checkpoint
+
+    Args:
+        - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
+
+    """
+    print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
+
+    optim_files = get_optim_files(ds_checkpoint_dir)
+    zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
+    print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
+
+    model_files = get_model_state_files(ds_checkpoint_dir)
+
+    zero_model_states = parse_model_states(model_files)
+    print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
+
+    if zero_stage <= 2:
+        return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                                          exclude_frozen_parameters)
+    elif zero_stage == 3:
+        return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                                          exclude_frozen_parameters)
+
+
+def _zero2_merge_frozen_params(state_dict, zero_model_states):
+    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
+        return
+
+    frozen_param_shapes = zero_model_states[0].frozen_param_shapes
+    frozen_param_fragments = zero_model_states[0].frozen_param_fragments
+
+    if debug:
+        num_elem = sum(s.numel() for s in frozen_param_shapes.values())
+        print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
+
+        wanted_params = len(frozen_param_shapes)
+        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
+        avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
+        print(f'Frozen params: Have {avail_numel} numels to process.')
+        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
+
+    total_params = 0
+    total_numel = 0
+    for name, shape in frozen_param_shapes.items():
+        total_params += 1
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+
+        state_dict[name] = frozen_param_fragments[name]
+
+        if debug:
+            print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
+
+    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _has_callable(obj, fn):
+    attr = getattr(obj, fn, None)
+    return callable(attr)
+
+
+def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
+    param_shapes = zero_model_states[0].param_shapes
+
+    # Reconstruction protocol:
+    #
+    # XXX: document this
+
+    if debug:
+        for i in range(world_size):
+            for j in range(len(fp32_flat_groups[0])):
+                print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
+
+    # XXX: memory usage doubles here (zero2)
+    num_param_groups = len(fp32_flat_groups[0])
+    merged_single_partition_of_fp32_groups = []
+    for i in range(num_param_groups):
+        merged_partitions = [sd[i] for sd in fp32_flat_groups]
+        full_single_fp32_vector = torch.cat(merged_partitions, 0)
+        merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
+    avail_numel = sum(
+        [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
+
+    if debug:
+        wanted_params = sum([len(shapes) for shapes in param_shapes])
+        wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
+        # not asserting if there is a mismatch due to possible padding
+        print(f"Have {avail_numel} numels to process.")
+        print(f"Need {wanted_numel} numels in {wanted_params} params.")
+
+    # params
+    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
+    # out-of-core computing solution
+    total_numel = 0
+    total_params = 0
+    for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
+        offset = 0
+        avail_numel = full_single_fp32_vector.numel()
+        for name, shape in shapes.items():
+
+            unpartitioned_numel = shape.numel() if _has_callable(shape, 'numel') else math.prod(shape)
+            total_numel += unpartitioned_numel
+            total_params += 1
+
+            if debug:
+                print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
+            state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
+            offset += unpartitioned_numel
+
+        # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
+        # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
+        # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
+        # live optimizer object, so we are checking that the numbers are within the right range
+        align_to = 2 * world_size
+
+        def zero2_align(x):
+            return align_to * math.ceil(x / align_to)
+
+        if debug:
+            print(f"original offset={offset}, avail_numel={avail_numel}")
+
+        offset = zero2_align(offset)
+        avail_numel = zero2_align(avail_numel)
+
+        if debug:
+            print(f"aligned  offset={offset}, avail_numel={avail_numel}")
+
+        # Sanity check
+        if offset != avail_numel:
+            raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
+
+    print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                               exclude_frozen_parameters):
+    state_dict = OrderedDict()
+
+    # buffers
+    buffers = zero_model_states[0].buffers
+    state_dict.update(buffers)
+    if debug:
+        print(f"added {len(buffers)} buffers")
+
+    if not exclude_frozen_parameters:
+        _zero2_merge_frozen_params(state_dict, zero_model_states)
+
+    _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
+
+    # recover shared parameters
+    for pair in zero_model_states[0].shared_params:
+        if pair[1] in state_dict:
+            state_dict[pair[0]] = state_dict[pair[1]]
+
+    return state_dict
+
+
+def zero3_partitioned_param_info(unpartitioned_numel, world_size):
+    remainder = unpartitioned_numel % world_size
+    padding_numel = (world_size - remainder) if remainder else 0
+    partitioned_numel = math.ceil(unpartitioned_numel / world_size)
+    return partitioned_numel, padding_numel
+
+
+def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
+    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
+        return
+
+    if debug:
+        for i in range(world_size):
+            num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
+            print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
+
+        frozen_param_shapes = zero_model_states[0].frozen_param_shapes
+        wanted_params = len(frozen_param_shapes)
+        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
+        avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
+        print(f'Frozen params: Have {avail_numel} numels to process.')
+        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
+
+    total_params = 0
+    total_numel = 0
+    for name, shape in zero_model_states[0].frozen_param_shapes.items():
+        total_params += 1
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+
+        param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
+        state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
+
+        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
+
+        if debug:
+            print(
+                f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
+            )
+
+    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
+    param_shapes = zero_model_states[0].param_shapes
+    avail_numel = fp32_flat_groups[0].numel() * world_size
+    # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
+    # param, re-consolidating each param, while dealing with padding if any
+
+    # merge list of dicts, preserving order
+    param_shapes = {k: v for d in param_shapes for k, v in d.items()}
+
+    if debug:
+        for i in range(world_size):
+            print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
+
+        wanted_params = len(param_shapes)
+        wanted_numel = sum(shape.numel() for shape in param_shapes.values())
+        # not asserting if there is a mismatch due to possible padding
+        avail_numel = fp32_flat_groups[0].numel() * world_size
+        print(f"Trainable params: Have {avail_numel} numels to process.")
+        print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
+
+    # params
+    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
+    # out-of-core computing solution
+    offset = 0
+    total_numel = 0
+    total_params = 0
+    for name, shape in tqdm(param_shapes.items(), desc='Gathering Sharded Weights'):
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+        total_params += 1
+        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
+
+        if debug:
+            print(
+                f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
+            )
+
+        # XXX: memory usage doubles here
+        state_dict[name] = torch.cat(
+            tuple(fp32_flat_groups[i].narrow(0, offset, partitioned_numel) for i in range(world_size)),
+            0).narrow(0, 0, unpartitioned_numel).view(shape)
+        offset += partitioned_numel
+
+    offset *= world_size
+
+    # Sanity check
+    if offset != avail_numel:
+        raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
+
+    print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                               exclude_frozen_parameters):
+    state_dict = OrderedDict()
+
+    # buffers
+    buffers = zero_model_states[0].buffers
+    state_dict.update(buffers)
+    if debug:
+        print(f"added {len(buffers)} buffers")
+
+    if not exclude_frozen_parameters:
+        _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
+
+    _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
+
+    # recover shared parameters
+    for pair in zero_model_states[0].shared_params:
+        if pair[1] in state_dict:
+            state_dict[pair[0]] = state_dict[pair[1]]
+
+    return state_dict
+
+
+def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag=None, exclude_frozen_parameters=False):
+    """
+    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
+    ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
+    via a model hub.
+
+    Args:
+        - ``checkpoint_dir``: path to the desired checkpoint folder
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
+        - ``exclude_frozen_parameters``: exclude frozen parameters
+
+    Returns:
+        - pytorch ``state_dict``
+
+    Note: this approach may not work if your application doesn't have sufficient free CPU memory and
+    you may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
+    the checkpoint.
+
+    A typical usage might be ::
+
+        from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
+        # do the training and checkpoint saving
+        state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
+        model = model.cpu() # move to cpu
+        model.load_state_dict(state_dict)
+        # submit to model hub or save the model to share with others
+
+    In this example the ``model`` will no longer be usable in the deepspeed context of the same
+    application. i.e. you will need to re-initialize the deepspeed engine, since
+    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
+
+    If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
+
+    """
+    if tag is None:
+        latest_path = os.path.join(checkpoint_dir, 'latest')
+        if os.path.isfile(latest_path):
+            with open(latest_path, 'r') as fd:
+                tag = fd.read().strip()
+        else:
+            raise ValueError(f"Unable to find 'latest' file at {latest_path}")
+
+    ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
+
+    if not os.path.isdir(ds_checkpoint_dir):
+        raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
+
+    return _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters)
+
+
+def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir,
+                                               output_dir,
+                                               max_shard_size="5GB",
+                                               safe_serialization=False,
+                                               tag=None,
+                                               exclude_frozen_parameters=False):
+    """
+    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
+    loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
+
+    Args:
+        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
+        - ``output_dir``: directory to the pytorch fp32 state_dict output files
+        - ``max_shard_size``: the maximum size for a checkpoint before being sharded, default value is 5GB
+        - ``safe_serialization``:  whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
+        - ``exclude_frozen_parameters``: exclude frozen parameters
+    """
+    # Dependency pre-check
+    if safe_serialization:
+        try:
+            from safetensors.torch import save_file
+        except ImportError:
+            print('If you want to use `safe_serialization`, please `pip install safetensors`')
+            raise
+    if max_shard_size is not None:
+        try:
+            from huggingface_hub import split_torch_state_dict_into_shards
+        except ImportError:
+            print('If you want to use `max_shard_size`, please `pip install huggingface_hub`')
+            raise
+
+    # Convert zero checkpoint to state_dict
+    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag, exclude_frozen_parameters)
+
+    # Shard the model if it is too big.
+    weights_name = "model.safetensors" if safe_serialization else "pytorch_model.bin"
+    if max_shard_size is not None:
+        filename_pattern = weights_name.replace(".bin", "{suffix}.bin").replace(".safetensors", "{suffix}.safetensors")
+        state_dict_split = split_torch_state_dict_into_shards(state_dict,
+                                                              filename_pattern=filename_pattern,
+                                                              max_shard_size=max_shard_size)
+    else:
+        from collections import namedtuple
+        StateDictSplit = namedtuple("StateDictSplit", ["is_sharded", "filename_to_tensors"])
+        state_dict_split = StateDictSplit(is_sharded=False,
+                                          filename_to_tensors={weights_name: list(state_dict.keys())})
+
+    # Save the model
+    filename_to_tensors = state_dict_split.filename_to_tensors.items()
+    for shard_file, tensors in tqdm(filename_to_tensors, desc="Saving checkpoint shards"):
+        shard = {tensor: state_dict[tensor].contiguous() for tensor in tensors}
+        output_path = os.path.join(output_dir, shard_file)
+        if safe_serialization:
+            save_file(shard, output_path, metadata={"format": "pt"})
+        else:
+            torch.save(shard, output_path)
+
+    # Save index if sharded
+    if state_dict_split.is_sharded:
+        index = {
+            "metadata": state_dict_split.metadata,
+            "weight_map": state_dict_split.tensor_to_filename,
+        }
+        save_index_file = "model.safetensors.index.json" if safe_serialization else "pytorch_model.bin.index.json"
+        save_index_file = os.path.join(output_dir, save_index_file)
+        with open(save_index_file, "w", encoding="utf-8") as f:
+            content = json.dumps(index, indent=2, sort_keys=True) + "\n"
+            f.write(content)
+
+
+def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
+    """
+    1. Put the provided model to cpu
+    2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
+    3. Load it into the provided model
+
+    Args:
+        - ``model``: the model object to update
+        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
+
+    Returns:
+        - ``model`: modified model
+
+    Make sure you have plenty of CPU memory available before you call this function. If you don't
+    have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
+    conveniently placed for you in the checkpoint folder.
+
+    A typical usage might be ::
+
+        from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
+        model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
+        # submit to model hub or save the model to share with others
+
+    Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
+    of the same application. i.e. you will need to re-initialize the deepspeed engine, since
+    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
+
+    """
+    logger.info(f"Extracting fp32 weights")
+    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
+
+    logger.info(f"Overwriting model with fp32 weights")
+    model = model.cpu()
+    model.load_state_dict(state_dict, strict=False)
+
+    return model
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("checkpoint_dir",
+                        type=str,
+                        help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
+    parser.add_argument("output_dir",
+                        type=str,
+                        help="directory to the pytorch fp32 state_dict output files"
+                        "(e.g. path/checkpoint-12-output/)")
+    parser.add_argument(
+        "--max_shard_size",
+        type=str,
+        default="5GB",
+        help="The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size"
+        "lower than this size. If expressed as a string, needs to be digits followed by a unit (like `5MB`"
+        "We default it to 5GB in order for models to be able to run easily on free-tier google colab instances"
+        "without CPU OOM issues.")
+    parser.add_argument(
+        "--safe_serialization",
+        default=False,
+        action='store_true',
+        help="Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).")
+    parser.add_argument("-t",
+                        "--tag",
+                        type=str,
+                        default=None,
+                        help="checkpoint tag used as a unique identifier for checkpoint. e.g., global_step1")
+    parser.add_argument("--exclude_frozen_parameters", action='store_true', help="exclude frozen parameters")
+    parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
+    args = parser.parse_args()
+
+    debug = args.debug
+
+    convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir,
+                                               args.output_dir,
+                                               max_shard_size=args.max_shard_size,
+                                               safe_serialization=args.safe_serialization,
+                                               tag=args.tag,
+                                               exclude_frozen_parameters=args.exclude_frozen_parameters)

trainer_log.jsonl

@@ -151,3 +151,38 @@
 {"current_steps": 1480, "total_steps": 4974, "loss": 1.167, "lr": 4.429321860261797e-05, "epoch": 0.8921708989526034, "percentage": 29.75, "elapsed_time": "10:19:03", "remaining_time": "1 day, 0:21:29"}
 {"current_steps": 1490, "total_steps": 4974, "loss": 1.1532, "lr": 4.418115462098624e-05, "epoch": 0.8981990807022832, "percentage": 29.96, "elapsed_time": "10:23:10", "remaining_time": "1 day, 0:17:08"}
 {"current_steps": 1500, "total_steps": 4974, "loss": 1.1206, "lr": 4.4068145724768324e-05, "epoch": 0.9042272624519629, "percentage": 30.16, "elapsed_time": "10:27:17", "remaining_time": "1 day, 0:12:47"}
+{"current_steps": 1510, "total_steps": 4974, "loss": 1.1371, "lr": 4.395419748108241e-05, "epoch": 0.9102554442016427, "percentage": 30.36, "elapsed_time": "10:33:44", "remaining_time": "1 day, 0:13:49"}
+{"current_steps": 1520, "total_steps": 4974, "loss": 1.1286, "lr": 4.383931550332141e-05, "epoch": 0.9162836259513224, "percentage": 30.56, "elapsed_time": "10:37:50", "remaining_time": "1 day, 0:09:25"}
+{"current_steps": 1530, "total_steps": 4974, "loss": 1.1195, "lr": 4.3723505450876454e-05, "epoch": 0.9223118077010022, "percentage": 30.76, "elapsed_time": "10:41:57", "remaining_time": "1 day, 0:05:02"}
+{"current_steps": 1540, "total_steps": 4974, "loss": 1.1548, "lr": 4.360677302885812e-05, "epoch": 0.928339989450682, "percentage": 30.96, "elapsed_time": "10:46:04", "remaining_time": "1 day, 0:00:39"}
+{"current_steps": 1550, "total_steps": 4974, "loss": 1.1395, "lr": 4.3489123987815325e-05, "epoch": 0.9343681712003616, "percentage": 31.16, "elapsed_time": "10:50:10", "remaining_time": "23:56:15"}
+{"current_steps": 1560, "total_steps": 4974, "loss": 1.1347, "lr": 4.3370564123452094e-05, "epoch": 0.9403963529500414, "percentage": 31.36, "elapsed_time": "10:54:17", "remaining_time": "23:51:53"}
+{"current_steps": 1570, "total_steps": 4974, "loss": 1.1571, "lr": 4.325109927634202e-05, "epoch": 0.9464245346997212, "percentage": 31.56, "elapsed_time": "10:58:24", "remaining_time": "23:47:32"}
+{"current_steps": 1580, "total_steps": 4974, "loss": 1.1436, "lr": 4.313073533164055e-05, "epoch": 0.952452716449401, "percentage": 31.77, "elapsed_time": "11:02:31", "remaining_time": "23:43:10"}
+{"current_steps": 1590, "total_steps": 4974, "loss": 1.1718, "lr": 4.300947821879507e-05, "epoch": 0.9584808981990807, "percentage": 31.97, "elapsed_time": "11:06:39", "remaining_time": "23:38:51"}
+{"current_steps": 1600, "total_steps": 4974, "loss": 1.1382, "lr": 4.288733391125278e-05, "epoch": 0.9645090799487605, "percentage": 32.17, "elapsed_time": "11:10:47", "remaining_time": "23:34:32"}
+{"current_steps": 1600, "total_steps": 4974, "eval_loss": 1.105757713317871, "epoch": 0.9645090799487605, "percentage": 32.17, "elapsed_time": "11:13:16", "remaining_time": "23:39:46"}
+{"current_steps": 1610, "total_steps": 4974, "loss": 1.1193, "lr": 4.276430842616646e-05, "epoch": 0.9705372616984402, "percentage": 32.37, "elapsed_time": "11:17:22", "remaining_time": "23:35:20"}
+{"current_steps": 1620, "total_steps": 4974, "loss": 1.1415, "lr": 4.264040782409804e-05, "epoch": 0.9765654434481199, "percentage": 32.57, "elapsed_time": "11:21:30", "remaining_time": "23:30:58"}
+{"current_steps": 1630, "total_steps": 4974, "loss": 1.1161, "lr": 4.251563820872002e-05, "epoch": 0.9825936251977997, "percentage": 32.77, "elapsed_time": "11:25:37", "remaining_time": "23:26:34"}
+{"current_steps": 1640, "total_steps": 4974, "loss": 1.1219, "lr": 4.2390005726514815e-05, "epoch": 0.9886218069474795, "percentage": 32.97, "elapsed_time": "11:29:43", "remaining_time": "23:22:10"}
+{"current_steps": 1650, "total_steps": 4974, "loss": 1.1217, "lr": 4.226351656647194e-05, "epoch": 0.9946499886971593, "percentage": 33.17, "elapsed_time": "11:33:50", "remaining_time": "23:17:47"}
+{"current_steps": 1660, "total_steps": 4974, "loss": 1.1582, "lr": 4.2136176959783155e-05, "epoch": 1.001130284078065, "percentage": 33.37, "elapsed_time": "11:37:58", "remaining_time": "23:13:25"}
+{"current_steps": 1670, "total_steps": 4974, "loss": 1.054, "lr": 4.2007993179535446e-05, "epoch": 1.0071584658277446, "percentage": 33.57, "elapsed_time": "11:42:05", "remaining_time": "23:09:02"}
+{"current_steps": 1680, "total_steps": 4974, "loss": 1.0657, "lr": 4.1878971540402044e-05, "epoch": 1.0131866475774245, "percentage": 33.78, "elapsed_time": "11:46:11", "remaining_time": "23:04:39"}
+{"current_steps": 1690, "total_steps": 4974, "loss": 1.0239, "lr": 4.174911839833135e-05, "epoch": 1.0192148293271042, "percentage": 33.98, "elapsed_time": "11:50:18", "remaining_time": "23:00:15"}
+{"current_steps": 1700, "total_steps": 4974, "loss": 1.0504, "lr": 4.1618440150233786e-05, "epoch": 1.025243011076784, "percentage": 34.18, "elapsed_time": "11:54:25", "remaining_time": "22:55:54"}
+{"current_steps": 1710, "total_steps": 4974, "loss": 1.0604, "lr": 4.148694323366668e-05, "epoch": 1.0312711928264637, "percentage": 34.38, "elapsed_time": "11:58:32", "remaining_time": "22:51:31"}
+{"current_steps": 1720, "total_steps": 4974, "loss": 1.0551, "lr": 4.135463412651717e-05, "epoch": 1.0372993745761434, "percentage": 34.58, "elapsed_time": "12:02:38", "remaining_time": "22:47:08"}
+{"current_steps": 1730, "total_steps": 4974, "loss": 1.0581, "lr": 4.122151934668303e-05, "epoch": 1.0433275563258233, "percentage": 34.78, "elapsed_time": "12:06:45", "remaining_time": "22:42:46"}
+{"current_steps": 1740, "total_steps": 4974, "loss": 1.0303, "lr": 4.108760545175163e-05, "epoch": 1.049355738075503, "percentage": 34.98, "elapsed_time": "12:10:52", "remaining_time": "22:38:25"}
+{"current_steps": 1750, "total_steps": 4974, "loss": 1.0619, "lr": 4.095289903867685e-05, "epoch": 1.0553839198251826, "percentage": 35.18, "elapsed_time": "12:14:59", "remaining_time": "22:34:04"}
+{"current_steps": 1760, "total_steps": 4974, "loss": 1.0266, "lr": 4.081740674345414e-05, "epoch": 1.0614121015748625, "percentage": 35.38, "elapsed_time": "12:19:06", "remaining_time": "22:29:43"}
+{"current_steps": 1770, "total_steps": 4974, "loss": 1.0639, "lr": 4.068113524079358e-05, "epoch": 1.0674402833245422, "percentage": 35.59, "elapsed_time": "12:23:13", "remaining_time": "22:25:22"}
+{"current_steps": 1780, "total_steps": 4974, "loss": 1.0521, "lr": 4.0544091243791064e-05, "epoch": 1.073468465074222, "percentage": 35.79, "elapsed_time": "12:27:21", "remaining_time": "22:21:02"}
+{"current_steps": 1790, "total_steps": 4974, "loss": 1.0483, "lr": 4.040628150359761e-05, "epoch": 1.0794966468239018, "percentage": 35.99, "elapsed_time": "12:31:27", "remaining_time": "22:16:40"}
+{"current_steps": 1800, "total_steps": 4974, "loss": 1.0185, "lr": 4.026771280908681e-05, "epoch": 1.0855248285735815, "percentage": 36.19, "elapsed_time": "12:35:34", "remaining_time": "22:12:20"}
+{"current_steps": 1810, "total_steps": 4974, "loss": 1.0469, "lr": 4.0128391986520336e-05, "epoch": 1.0915530103232611, "percentage": 36.39, "elapsed_time": "12:39:42", "remaining_time": "22:08:00"}
+{"current_steps": 1820, "total_steps": 4974, "loss": 1.0388, "lr": 3.998832589921167e-05, "epoch": 1.097581192072941, "percentage": 36.59, "elapsed_time": "12:43:52", "remaining_time": "22:03:46"}
+{"current_steps": 1830, "total_steps": 4974, "loss": 1.0556, "lr": 3.9847521447188034e-05, "epoch": 1.1036093738226207, "percentage": 36.79, "elapsed_time": "12:48:03", "remaining_time": "21:59:33"}
+{"current_steps": 1840, "total_steps": 4974, "loss": 1.0297, "lr": 3.970598556685045e-05, "epoch": 1.1096375555723006, "percentage": 36.99, "elapsed_time": "12:52:11", "remaining_time": "21:55:14"}

upload.py

@@ -0,0 +1,11 @@
+from huggingface_hub import HfApi
+api = HfApi()
+
+# Upload all the content from the local folder to your remote Space.
+# By default, files are uploaded at the root of the repo
+api.upload_folder(
+    folder_path="./",
+    repo_id="sumuks/qwen2.5-72b-utility-evaluator-r128",
+    repo_type="model",
+    ignore_patterns=["*.pt", "*.pth"],
+)