Upload 216 files

checkpoint-1000/README.md

@@ -0,0 +1,204 @@
+---
+library_name: peft
+base_model: ../chatglm3-6b-base
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+
+### Framework versions
+
+- PEFT 0.8.2

checkpoint-1000/adapter_config.json

@@ -0,0 +1,26 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../chatglm3-6b-base",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "query_key_value"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_rslora": false
+}

checkpoint-1000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d53edf161f3a07e2d551714247c93fc284b893fb5528a49a55fadf70ff8eed41
+size 7807744

checkpoint-1000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7670653bcd7d18fd04a1f4d7fbb09d9a94c9d2796d6ae96f3b88f0a70245ba2b
+size 15644485

checkpoint-1000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:092276df08ab03d1c657bebf8c8ca092758eefddc44fbdcfe4fa09446ad954a3
+size 14575

checkpoint-1000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a724eb8dd004def7e4c5b752d32b5f2b92570d2204069c06be7d67c6e7933312
+size 627

checkpoint-1000/special_tokens_map.json

@@ -0,0 +1,18 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|observation|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ]
+}

checkpoint-1000/tokenization_chatglm.py

@@ -0,0 +1,300 @@
+import json
+import os
+import re
+from typing import List, Optional, Union, Dict
+from sentencepiece import SentencePieceProcessor
+from transformers import PreTrainedTokenizer
+from transformers.utils import logging, PaddingStrategy
+from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
+
+
+class SPTokenizer:
+    def __init__(self, model_path: str):
+        # reload tokenizer
+        assert os.path.isfile(model_path), model_path
+        self.sp_model = SentencePieceProcessor(model_file=model_path)
+
+        # BOS / EOS token IDs
+        self.n_words: int = self.sp_model.vocab_size()
+        self.bos_id: int = self.sp_model.bos_id()
+        self.eos_id: int = self.sp_model.eos_id()
+        self.pad_id: int = self.sp_model.unk_id()
+        assert self.sp_model.vocab_size() == self.sp_model.get_piece_size()
+
+        role_special_tokens = ["<|system|>", "<|user|>", "<|assistant|>", "<|observation|>"]
+        special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "sop", "eop"] + role_special_tokens
+        self.special_tokens = {}
+        self.index_special_tokens = {}
+        for token in special_tokens:
+            self.special_tokens[token] = self.n_words
+            self.index_special_tokens[self.n_words] = token
+            self.n_words += 1
+        self.role_special_token_expression = "|".join([re.escape(token) for token in role_special_tokens])
+
+    def tokenize(self, s: str, encode_special_tokens=False):
+        if encode_special_tokens:
+            last_index = 0
+            t = []
+            for match in re.finditer(self.role_special_token_expression, s):
+                if last_index < match.start():
+                    t.extend(self.sp_model.EncodeAsPieces(s[last_index:match.start()]))
+                t.append(s[match.start():match.end()])
+                last_index = match.end()
+            if last_index < len(s):
+                t.extend(self.sp_model.EncodeAsPieces(s[last_index:]))
+            return t
+        else:
+            return self.sp_model.EncodeAsPieces(s)
+
+    def encode(self, s: str, bos: bool = False, eos: bool = False) -> List[int]:
+        assert type(s) is str
+        t = self.sp_model.encode(s)
+        if bos:
+            t = [self.bos_id] + t
+        if eos:
+            t = t + [self.eos_id]
+        return t
+
+    def decode(self, t: List[int]) -> str:
+        text, buffer = "", []
+        for token in t:
+            if token in self.index_special_tokens:
+                if buffer:
+                    text += self.sp_model.decode(buffer)
+                    buffer = []
+                text += self.index_special_tokens[token]
+            else:
+                buffer.append(token)
+        if buffer:
+            text += self.sp_model.decode(buffer)
+        return text
+
+    def decode_tokens(self, tokens: List[str]) -> str:
+        text = self.sp_model.DecodePieces(tokens)
+        return text
+
+    def convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        if token in self.special_tokens:
+            return self.special_tokens[token]
+        return self.sp_model.PieceToId(token)
+
+    def convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        if index in self.index_special_tokens:
+            return self.index_special_tokens[index]
+        if index in [self.eos_id, self.bos_id, self.pad_id] or index < 0:
+            return ""
+        return self.sp_model.IdToPiece(index)
+
+
+class ChatGLMTokenizer(PreTrainedTokenizer):
+    vocab_files_names = {"vocab_file": "tokenizer.model"}
+
+    model_input_names = ["input_ids", "attention_mask", "position_ids"]
+
+    def __init__(self, vocab_file, padding_side="left", clean_up_tokenization_spaces=False, encode_special_tokens=False,
+                 **kwargs):
+        self.name = "GLMTokenizer"
+
+        self.vocab_file = vocab_file
+        self.tokenizer = SPTokenizer(vocab_file)
+        self.special_tokens = {
+            "<bos>": self.tokenizer.bos_id,
+            "<eos>": self.tokenizer.eos_id,
+            "<pad>": self.tokenizer.pad_id
+        }
+        self.encode_special_tokens = encode_special_tokens
+        super().__init__(padding_side=padding_side, clean_up_tokenization_spaces=clean_up_tokenization_spaces,
+                         encode_special_tokens=encode_special_tokens,
+                         **kwargs)
+
+    def get_command(self, token):
+        if token in self.special_tokens:
+            return self.special_tokens[token]
+        assert token in self.tokenizer.special_tokens, f"{token} is not a special token for {self.name}"
+        return self.tokenizer.special_tokens[token]
+
+    @property
+    def unk_token(self) -> str:
+        return "<unk>"
+
+    @property
+    def pad_token(self) -> str:
+        return "<unk>"
+
+    @property
+    def pad_token_id(self):
+        return self.get_command("<pad>")
+
+    @property
+    def eos_token(self) -> str:
+        return "</s>"
+
+    @property
+    def eos_token_id(self):
+        return self.get_command("<eos>")
+
+    @property
+    def vocab_size(self):
+        return self.tokenizer.n_words
+
+    def get_vocab(self):
+        """ Returns vocab as a dict """
+        vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
+        vocab.update(self.added_tokens_encoder)
+        return vocab
+
+    def _tokenize(self, text, **kwargs):
+        return self.tokenizer.tokenize(text, encode_special_tokens=self.encode_special_tokens)
+
+    def _convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        return self.tokenizer.convert_token_to_id(token)
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        return self.tokenizer.convert_id_to_token(index)
+
+    def convert_tokens_to_string(self, tokens: List[str]) -> str:
+        return self.tokenizer.decode_tokens(tokens)
+
+    def save_vocabulary(self, save_directory, filename_prefix=None):
+        """
+        Save the vocabulary and special tokens file to a directory.
+
+        Args:
+            save_directory (`str`):
+                The directory in which to save the vocabulary.
+            filename_prefix (`str`, *optional*):
+                An optional prefix to add to the named of the saved files.
+
+        Returns:
+            `Tuple(str)`: Paths to the files saved.
+        """
+        if os.path.isdir(save_directory):
+            vocab_file = os.path.join(
+                save_directory, self.vocab_files_names["vocab_file"]
+            )
+        else:
+            vocab_file = save_directory
+
+        with open(self.vocab_file, 'rb') as fin:
+            proto_str = fin.read()
+
+        with open(vocab_file, "wb") as writer:
+            writer.write(proto_str)
+
+        return (vocab_file,)
+
+    def get_prefix_tokens(self):
+        prefix_tokens = [self.get_command("[gMASK]"), self.get_command("sop")]
+        return prefix_tokens
+
+    def build_single_message(self, role, metadata, message):
+        assert role in ["system", "user", "assistant", "observation"], role
+        role_tokens = [self.get_command(f"<|{role}|>")] + self.tokenizer.encode(f"{metadata}\n")
+        message_tokens = self.tokenizer.encode(message)
+        tokens = role_tokens + message_tokens
+        return tokens
+
+    def build_chat_input(self, query, history=None, role="user"):
+        if history is None:
+            history = []
+        input_ids = []
+        for item in history:
+            content = item["content"]
+            if item["role"] == "system" and "tools" in item:
+                content = content + "\n" + json.dumps(item["tools"], indent=4, ensure_ascii=False)
+            input_ids.extend(self.build_single_message(item["role"], item.get("metadata", ""), content))
+        input_ids.extend(self.build_single_message(role, "", query))
+        input_ids.extend([self.get_command("<|assistant|>")])
+        return self.batch_encode_plus([input_ids], return_tensors="pt", is_split_into_words=True)
+
+    def build_inputs_with_special_tokens(
+            self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
+        adding special tokens. A BERT sequence has the following format:
+
+        - single sequence: `[CLS] X [SEP]`
+        - pair of sequences: `[CLS] A [SEP] B [SEP]`
+
+        Args:
+            token_ids_0 (`List[int]`):
+                List of IDs to which the special tokens will be added.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
+        """
+        prefix_tokens = self.get_prefix_tokens()
+        token_ids_0 = prefix_tokens + token_ids_0
+        if token_ids_1 is not None:
+            token_ids_0 = token_ids_0 + token_ids_1 + [self.get_command("<eos>")]
+        return token_ids_0
+
+    def _pad(
+            self,
+            encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
+            max_length: Optional[int] = None,
+            padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
+            pad_to_multiple_of: Optional[int] = None,
+            return_attention_mask: Optional[bool] = None,
+    ) -> dict:
+        """
+        Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
+
+        Args:
+            encoded_inputs:
+                Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`).
+            max_length: maximum length of the returned list and optionally padding length (see below).
+                Will truncate by taking into account the special tokens.
+            padding_strategy: PaddingStrategy to use for padding.
+
+                - PaddingStrategy.LONGEST Pad to the longest sequence in the batch
+                - PaddingStrategy.MAX_LENGTH: Pad to the max length (default)
+                - PaddingStrategy.DO_NOT_PAD: Do not pad
+                The tokenizer padding sides are defined in self.padding_side:
+
+                    - 'left': pads on the left of the sequences
+                    - 'right': pads on the right of the sequences
+            pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
+                This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
+                `>= 7.5` (Volta).
+            return_attention_mask:
+                (optional) Set to False to avoid returning attention mask (default: set to model specifics)
+        """
+        # Load from model defaults
+        assert self.padding_side == "left"
+
+        required_input = encoded_inputs[self.model_input_names[0]]
+        seq_length = len(required_input)
+
+        if padding_strategy == PaddingStrategy.LONGEST:
+            max_length = len(required_input)
+
+        if max_length is not None and pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
+            max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
+
+        needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len(required_input) != max_length
+
+        # Initialize attention mask if not present.
+        if "attention_mask" not in encoded_inputs:
+            encoded_inputs["attention_mask"] = [1] * seq_length
+
+        if "position_ids" not in encoded_inputs:
+            encoded_inputs["position_ids"] = list(range(seq_length))
+
+        if needs_to_be_padded:
+            difference = max_length - len(required_input)
+
+            if "attention_mask" in encoded_inputs:
+                encoded_inputs["attention_mask"] = [0] * difference + encoded_inputs["attention_mask"]
+            if "position_ids" in encoded_inputs:
+                encoded_inputs["position_ids"] = [0] * difference + encoded_inputs["position_ids"]
+            encoded_inputs[self.model_input_names[0]] = [self.pad_token_id] * difference + required_input
+
+        return encoded_inputs

checkpoint-1000/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7dc4c393423b76e4373e5157ddc34803a0189ba96b21ddbb40269d31468a6f2
+size 1018370

checkpoint-1000/tokenizer_config.json

@@ -0,0 +1,41 @@
+{
+  "added_tokens_decoder": {
+    "64795": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64797": {
+      "content": "<|observation|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|observation|>"
+  ],
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenization_chatglm.ChatGLMTokenizer",
+      null
+    ]
+  },
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": false,
+  "encode_special_tokens": false,
+  "eos_token": "</s>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<unk>",
+  "padding_side": "right",
+  "remove_space": false,
+  "split_special_tokens": false,
+  "tokenizer_class": "ChatGLMTokenizer",
+  "unk_token": "<unk>"
+}

checkpoint-1000/trainer_state.json

@@ -0,0 +1,1221 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.5018190942165349,
+  "eval_steps": 500,
+  "global_step": 1000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0,
+      "learning_rate": 4.9999980101927616e-05,
+      "loss": 3.2533,
+      "step": 5
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.99999204077421e-05,
+      "loss": 3.2279,
+      "step": 10
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.999978982687695e-05,
+      "loss": 3.193,
+      "step": 15
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.9999597065062966e-05,
+      "loss": 3.2863,
+      "step": 20
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.999934212277958e-05,
+      "loss": 3.1724,
+      "step": 25
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999902500066093e-05,
+      "loss": 3.1088,
+      "step": 30
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999864569949576e-05,
+      "loss": 3.3241,
+      "step": 35
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.99982042202275e-05,
+      "loss": 3.1904,
+      "step": 40
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999770056395421e-05,
+      "loss": 2.9254,
+      "step": 45
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.999713473192863e-05,
+      "loss": 3.1845,
+      "step": 50
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.999650672555812e-05,
+      "loss": 2.8475,
+      "step": 55
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.9995816546404695e-05,
+      "loss": 3.0486,
+      "step": 60
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.9995064196185014e-05,
+      "loss": 2.6464,
+      "step": 65
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.9994249676770364e-05,
+      "loss": 3.0446,
+      "step": 70
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999337299018667e-05,
+      "loss": 3.375,
+      "step": 75
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999243413861447e-05,
+      "loss": 2.8787,
+      "step": 80
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999143312438893e-05,
+      "loss": 3.014,
+      "step": 85
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.999036994999985e-05,
+      "loss": 2.8288,
+      "step": 90
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.9989244618091596e-05,
+      "loss": 3.1879,
+      "step": 95
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.998805713146317e-05,
+      "loss": 2.9749,
+      "step": 100
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.9986807493068165e-05,
+      "loss": 2.6304,
+      "step": 105
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.998549570601475e-05,
+      "loss": 2.943,
+      "step": 110
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.998412177356568e-05,
+      "loss": 2.7595,
+      "step": 115
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.9982685699138275e-05,
+      "loss": 2.7377,
+      "step": 120
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.9981187486304423e-05,
+      "loss": 2.9878,
+      "step": 125
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997962713879058e-05,
+      "loss": 2.7882,
+      "step": 130
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997800466047772e-05,
+      "loss": 2.7802,
+      "step": 135
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997632005540138e-05,
+      "loss": 3.0129,
+      "step": 140
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997457332775159e-05,
+      "loss": 2.9444,
+      "step": 145
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.997276448187294e-05,
+      "loss": 2.9664,
+      "step": 150
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.9970893522264476e-05,
+      "loss": 2.7367,
+      "step": 155
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.996896045357977e-05,
+      "loss": 2.7012,
+      "step": 160
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.9966965280626856e-05,
+      "loss": 2.8493,
+      "step": 165
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.996490800836825e-05,
+      "loss": 2.9274,
+      "step": 170
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.996278864192092e-05,
+      "loss": 2.8093,
+      "step": 175
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.9960607186556286e-05,
+      "loss": 3.1782,
+      "step": 180
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.995836364770018e-05,
+      "loss": 2.7507,
+      "step": 185
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.995605803093287e-05,
+      "loss": 2.6724,
+      "step": 190
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9953690341989026e-05,
+      "loss": 3.0258,
+      "step": 195
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9951260586757694e-05,
+      "loss": 3.1134,
+      "step": 200
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9948768771282314e-05,
+      "loss": 2.9937,
+      "step": 205
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.9946214901760665e-05,
+      "loss": 2.7394,
+      "step": 210
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.99435989845449e-05,
+      "loss": 2.9696,
+      "step": 215
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.994092102614146e-05,
+      "loss": 2.753,
+      "step": 220
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.993818103321113e-05,
+      "loss": 3.0759,
+      "step": 225
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.9935379012568985e-05,
+      "loss": 2.7512,
+      "step": 230
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.993251497118438e-05,
+      "loss": 2.8656,
+      "step": 235
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.992958891618091e-05,
+      "loss": 2.6628,
+      "step": 240
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.992660085483645e-05,
+      "loss": 2.8012,
+      "step": 245
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.992355079458307e-05,
+      "loss": 2.8141,
+      "step": 250
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.992043874300706e-05,
+      "loss": 2.9083,
+      "step": 255
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.991726470784891e-05,
+      "loss": 2.5846,
+      "step": 260
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.991402869700325e-05,
+      "loss": 2.8088,
+      "step": 265
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.991073071851889e-05,
+      "loss": 2.9979,
+      "step": 270
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.990737078059875e-05,
+      "loss": 3.0171,
+      "step": 275
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.990394889159986e-05,
+      "loss": 2.9278,
+      "step": 280
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.9900465060033364e-05,
+      "loss": 2.7998,
+      "step": 285
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.989691929456443e-05,
+      "loss": 2.721,
+      "step": 290
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.9893311604012306e-05,
+      "loss": 3.0291,
+      "step": 295
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.988964199735024e-05,
+      "loss": 2.9777,
+      "step": 300
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.988591048370552e-05,
+      "loss": 2.318,
+      "step": 305
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.988211707235936e-05,
+      "loss": 3.0332,
+      "step": 310
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987826177274697e-05,
+      "loss": 2.4888,
+      "step": 315
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987434459445748e-05,
+      "loss": 2.9259,
+      "step": 320
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987036554723391e-05,
+      "loss": 2.9568,
+      "step": 325
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.98663246409732e-05,
+      "loss": 2.8708,
+      "step": 330
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.986222188572611e-05,
+      "loss": 2.7848,
+      "step": 335
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.985805729169728e-05,
+      "loss": 2.6178,
+      "step": 340
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.985383086924511e-05,
+      "loss": 2.8326,
+      "step": 345
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.984954262888182e-05,
+      "loss": 2.8845,
+      "step": 350
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.9845192581273365e-05,
+      "loss": 2.6151,
+      "step": 355
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.984078073723944e-05,
+      "loss": 2.8474,
+      "step": 360
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.9836307107753455e-05,
+      "loss": 2.808,
+      "step": 365
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.983177170394248e-05,
+      "loss": 2.8491,
+      "step": 370
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.9827174537087226e-05,
+      "loss": 2.7764,
+      "step": 375
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.982251561862205e-05,
+      "loss": 2.7582,
+      "step": 380
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.981779496013489e-05,
+      "loss": 2.5379,
+      "step": 385
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.981301257336723e-05,
+      "loss": 2.9937,
+      "step": 390
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.980816847021412e-05,
+      "loss": 2.8574,
+      "step": 395
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.980326266272409e-05,
+      "loss": 2.9369,
+      "step": 400
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.979829516309915e-05,
+      "loss": 2.836,
+      "step": 405
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.979326598369477e-05,
+      "loss": 2.9369,
+      "step": 410
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.9788175137019814e-05,
+      "loss": 2.6667,
+      "step": 415
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.9783022635736534e-05,
+      "loss": 2.999,
+      "step": 420
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.977780849266054e-05,
+      "loss": 2.907,
+      "step": 425
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.9772532720760744e-05,
+      "loss": 2.8028,
+      "step": 430
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.976719533315937e-05,
+      "loss": 2.7999,
+      "step": 435
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.976179634313187e-05,
+      "loss": 2.8918,
+      "step": 440
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.9756335764106944e-05,
+      "loss": 2.7926,
+      "step": 445
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.975081360966646e-05,
+      "loss": 2.6709,
+      "step": 450
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.9745229893545436e-05,
+      "loss": 2.8248,
+      "step": 455
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.973958462963203e-05,
+      "loss": 3.1146,
+      "step": 460
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.973387783196747e-05,
+      "loss": 2.9991,
+      "step": 465
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.972810951474605e-05,
+      "loss": 2.7726,
+      "step": 470
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.972227969231505e-05,
+      "loss": 2.9025,
+      "step": 475
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.971638837917475e-05,
+      "loss": 2.6521,
+      "step": 480
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.971043558997839e-05,
+      "loss": 2.9511,
+      "step": 485
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.9704421339532075e-05,
+      "loss": 2.6938,
+      "step": 490
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.969834564279482e-05,
+      "loss": 2.8533,
+      "step": 495
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.9692208514878444e-05,
+      "loss": 2.6411,
+      "step": 500
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.968600997104758e-05,
+      "loss": 2.6486,
+      "step": 505
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.967975002671961e-05,
+      "loss": 2.8561,
+      "step": 510
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.967342869746463e-05,
+      "loss": 2.6984,
+      "step": 515
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.9667045999005424e-05,
+      "loss": 2.91,
+      "step": 520
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.966060194721742e-05,
+      "loss": 2.7205,
+      "step": 525
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.965409655812865e-05,
+      "loss": 2.7634,
+      "step": 530
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.9647529847919684e-05,
+      "loss": 2.9647,
+      "step": 535
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.964090183292364e-05,
+      "loss": 2.6357,
+      "step": 540
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.963421252962609e-05,
+      "loss": 3.0285,
+      "step": 545
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.96274619546651e-05,
+      "loss": 2.9895,
+      "step": 550
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.962065012483106e-05,
+      "loss": 2.7286,
+      "step": 555
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.961377705706677e-05,
+      "loss": 3.1337,
+      "step": 560
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.960684276846733e-05,
+      "loss": 2.8268,
+      "step": 565
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.959984727628011e-05,
+      "loss": 2.5851,
+      "step": 570
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.959279059790471e-05,
+      "loss": 2.9359,
+      "step": 575
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.958567275089291e-05,
+      "loss": 2.8842,
+      "step": 580
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.957849375294864e-05,
+      "loss": 2.6186,
+      "step": 585
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.957125362192794e-05,
+      "loss": 3.0116,
+      "step": 590
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.956395237583887e-05,
+      "loss": 2.7045,
+      "step": 595
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.9556590032841526e-05,
+      "loss": 2.8766,
+      "step": 600
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.954916661124797e-05,
+      "loss": 2.7858,
+      "step": 605
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.954168212952216e-05,
+      "loss": 2.7379,
+      "step": 610
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.953413660627995e-05,
+      "loss": 2.475,
+      "step": 615
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.9526530060289e-05,
+      "loss": 2.8357,
+      "step": 620
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.951886251046876e-05,
+      "loss": 2.9284,
+      "step": 625
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.951113397589042e-05,
+      "loss": 2.8144,
+      "step": 630
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9503344475776846e-05,
+      "loss": 2.6845,
+      "step": 635
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9495494029502535e-05,
+      "loss": 2.8937,
+      "step": 640
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9487582656593575e-05,
+      "loss": 3.0325,
+      "step": 645
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.94796103767276e-05,
+      "loss": 2.7058,
+      "step": 650
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.9471577209733746e-05,
+      "loss": 2.6162,
+      "step": 655
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.946348317559257e-05,
+      "loss": 3.0129,
+      "step": 660
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.945532829443603e-05,
+      "loss": 2.7587,
+      "step": 665
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.944711258654742e-05,
+      "loss": 2.8915,
+      "step": 670
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.943883607236135e-05,
+      "loss": 2.7343,
+      "step": 675
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.943049877246364e-05,
+      "loss": 2.881,
+      "step": 680
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.942210070759131e-05,
+      "loss": 2.488,
+      "step": 685
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.941364189863253e-05,
+      "loss": 2.896,
+      "step": 690
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.940512236662654e-05,
+      "loss": 2.7757,
+      "step": 695
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.9396542132763634e-05,
+      "loss": 2.6271,
+      "step": 700
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.938790121838506e-05,
+      "loss": 2.6804,
+      "step": 705
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.937919964498302e-05,
+      "loss": 2.7313,
+      "step": 710
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.937043743420058e-05,
+      "loss": 2.9277,
+      "step": 715
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.9361614607831605e-05,
+      "loss": 2.6366,
+      "step": 720
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.935273118782078e-05,
+      "loss": 3.0556,
+      "step": 725
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.934378719626345e-05,
+      "loss": 3.0182,
+      "step": 730
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.933478265540564e-05,
+      "loss": 2.824,
+      "step": 735
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.932571758764398e-05,
+      "loss": 2.636,
+      "step": 740
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.931659201552563e-05,
+      "loss": 2.7025,
+      "step": 745
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.930740596174827e-05,
+      "loss": 2.8919,
+      "step": 750
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.9298159449159965e-05,
+      "loss": 2.8434,
+      "step": 755
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.928885250075921e-05,
+      "loss": 2.9448,
+      "step": 760
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.927948513969478e-05,
+      "loss": 2.7312,
+      "step": 765
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.927005738926573e-05,
+      "loss": 2.8688,
+      "step": 770
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.926056927292132e-05,
+      "loss": 2.8639,
+      "step": 775
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.925102081426095e-05,
+      "loss": 2.7809,
+      "step": 780
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.9241412037034115e-05,
+      "loss": 3.0111,
+      "step": 785
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.9231742965140314e-05,
+      "loss": 2.7252,
+      "step": 790
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.922201362262905e-05,
+      "loss": 2.9717,
+      "step": 795
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.92122240336997e-05,
+      "loss": 2.7191,
+      "step": 800
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.920237422270153e-05,
+      "loss": 2.9346,
+      "step": 805
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9192464214133536e-05,
+      "loss": 2.7967,
+      "step": 810
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9182494032644496e-05,
+      "loss": 2.7326,
+      "step": 815
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.917246370303284e-05,
+      "loss": 2.8933,
+      "step": 820
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9162373250246575e-05,
+      "loss": 2.7939,
+      "step": 825
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.9152222699383273e-05,
+      "loss": 2.7807,
+      "step": 830
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.9142012075689994e-05,
+      "loss": 2.6996,
+      "step": 835
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.913174140456319e-05,
+      "loss": 2.7569,
+      "step": 840
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.912141071154869e-05,
+      "loss": 2.9347,
+      "step": 845
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.911102002234159e-05,
+      "loss": 2.7251,
+      "step": 850
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.910056936278623e-05,
+      "loss": 3.1228,
+      "step": 855
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.90900587588761e-05,
+      "loss": 2.9902,
+      "step": 860
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.9079488236753803e-05,
+      "loss": 2.5095,
+      "step": 865
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.906885782271095e-05,
+      "loss": 2.7523,
+      "step": 870
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.905816754318814e-05,
+      "loss": 2.6656,
+      "step": 875
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.9047417424774874e-05,
+      "loss": 2.8454,
+      "step": 880
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.903660749420946e-05,
+      "loss": 2.8194,
+      "step": 885
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.9025737778379025e-05,
+      "loss": 2.8421,
+      "step": 890
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.9014808304319326e-05,
+      "loss": 2.9656,
+      "step": 895
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.900381909921482e-05,
+      "loss": 2.7484,
+      "step": 900
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.899277019039849e-05,
+      "loss": 2.9044,
+      "step": 905
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.898166160535186e-05,
+      "loss": 2.8016,
+      "step": 910
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.8970493371704826e-05,
+      "loss": 2.6278,
+      "step": 915
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.895926551723569e-05,
+      "loss": 2.7214,
+      "step": 920
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.8947978069871036e-05,
+      "loss": 2.7679,
+      "step": 925
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8936631057685654e-05,
+      "loss": 2.7196,
+      "step": 930
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8925224508902514e-05,
+      "loss": 2.7866,
+      "step": 935
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8913758451892644e-05,
+      "loss": 2.72,
+      "step": 940
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.89022329151751e-05,
+      "loss": 2.752,
+      "step": 945
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8890647927416887e-05,
+      "loss": 3.0487,
+      "step": 950
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8879003517432857e-05,
+      "loss": 2.8928,
+      "step": 955
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.886729971418568e-05,
+      "loss": 2.7793,
+      "step": 960
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8855536546785726e-05,
+      "loss": 2.537,
+      "step": 965
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.884371404449105e-05,
+      "loss": 2.8345,
+      "step": 970
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.8831832236707284e-05,
+      "loss": 2.9673,
+      "step": 975
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.8819891152987546e-05,
+      "loss": 2.8295,
+      "step": 980
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.880789082303241e-05,
+      "loss": 3.0753,
+      "step": 985
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.879583127668979e-05,
+      "loss": 2.6748,
+      "step": 990
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.878371254395492e-05,
+      "loss": 2.8115,
+      "step": 995
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.877153465497022e-05,
+      "loss": 2.7039,
+      "step": 1000
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 9960,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "total_flos": 5.320804662283469e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:939ee45e9035366dc4952c5158e7d3a0d3426acdbfb5014d53ad1e260b19a19f
+size 4475

checkpoint-1500/README.md

@@ -0,0 +1,204 @@
+---
+library_name: peft
+base_model: ../chatglm3-6b-base
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+
+### Framework versions
+
+- PEFT 0.8.2

checkpoint-1500/adapter_config.json

@@ -0,0 +1,26 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../chatglm3-6b-base",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "query_key_value"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_rslora": false
+}

checkpoint-1500/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84490235df6826161e1e701ff9a1c761a3a25670fd30944a4b41f5c5ec1504b0
+size 7807744

checkpoint-1500/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6a722ed8329e8f09783e7b151a4b15eb1b7717e70e9a6487f0a9b6f72685224
+size 15644485

checkpoint-1500/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4d4bea37dabe731e99dacf209629d05725ac46e96f42aecaad63dd5bf6d55f7
+size 14575

checkpoint-1500/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c08c85a6d7b0767f91057e7c98cc42bdfa246f54b1e5570219edec6128d8b66
+size 627

checkpoint-1500/special_tokens_map.json

@@ -0,0 +1,18 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|observation|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ]
+}

checkpoint-1500/tokenization_chatglm.py

@@ -0,0 +1,300 @@
+import json
+import os
+import re
+from typing import List, Optional, Union, Dict
+from sentencepiece import SentencePieceProcessor
+from transformers import PreTrainedTokenizer
+from transformers.utils import logging, PaddingStrategy
+from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
+
+
+class SPTokenizer:
+    def __init__(self, model_path: str):
+        # reload tokenizer
+        assert os.path.isfile(model_path), model_path
+        self.sp_model = SentencePieceProcessor(model_file=model_path)
+
+        # BOS / EOS token IDs
+        self.n_words: int = self.sp_model.vocab_size()
+        self.bos_id: int = self.sp_model.bos_id()
+        self.eos_id: int = self.sp_model.eos_id()
+        self.pad_id: int = self.sp_model.unk_id()
+        assert self.sp_model.vocab_size() == self.sp_model.get_piece_size()
+
+        role_special_tokens = ["<|system|>", "<|user|>", "<|assistant|>", "<|observation|>"]
+        special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "sop", "eop"] + role_special_tokens
+        self.special_tokens = {}
+        self.index_special_tokens = {}
+        for token in special_tokens:
+            self.special_tokens[token] = self.n_words
+            self.index_special_tokens[self.n_words] = token
+            self.n_words += 1
+        self.role_special_token_expression = "|".join([re.escape(token) for token in role_special_tokens])
+
+    def tokenize(self, s: str, encode_special_tokens=False):
+        if encode_special_tokens:
+            last_index = 0
+            t = []
+            for match in re.finditer(self.role_special_token_expression, s):
+                if last_index < match.start():
+                    t.extend(self.sp_model.EncodeAsPieces(s[last_index:match.start()]))
+                t.append(s[match.start():match.end()])
+                last_index = match.end()
+            if last_index < len(s):
+                t.extend(self.sp_model.EncodeAsPieces(s[last_index:]))
+            return t
+        else:
+            return self.sp_model.EncodeAsPieces(s)
+
+    def encode(self, s: str, bos: bool = False, eos: bool = False) -> List[int]:
+        assert type(s) is str
+        t = self.sp_model.encode(s)
+        if bos:
+            t = [self.bos_id] + t
+        if eos:
+            t = t + [self.eos_id]
+        return t
+
+    def decode(self, t: List[int]) -> str:
+        text, buffer = "", []
+        for token in t:
+            if token in self.index_special_tokens:
+                if buffer:
+                    text += self.sp_model.decode(buffer)
+                    buffer = []
+                text += self.index_special_tokens[token]
+            else:
+                buffer.append(token)
+        if buffer:
+            text += self.sp_model.decode(buffer)
+        return text
+
+    def decode_tokens(self, tokens: List[str]) -> str:
+        text = self.sp_model.DecodePieces(tokens)
+        return text
+
+    def convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        if token in self.special_tokens:
+            return self.special_tokens[token]
+        return self.sp_model.PieceToId(token)
+
+    def convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        if index in self.index_special_tokens:
+            return self.index_special_tokens[index]
+        if index in [self.eos_id, self.bos_id, self.pad_id] or index < 0:
+            return ""
+        return self.sp_model.IdToPiece(index)
+
+
+class ChatGLMTokenizer(PreTrainedTokenizer):
+    vocab_files_names = {"vocab_file": "tokenizer.model"}
+
+    model_input_names = ["input_ids", "attention_mask", "position_ids"]
+
+    def __init__(self, vocab_file, padding_side="left", clean_up_tokenization_spaces=False, encode_special_tokens=False,
+                 **kwargs):
+        self.name = "GLMTokenizer"
+
+        self.vocab_file = vocab_file
+        self.tokenizer = SPTokenizer(vocab_file)
+        self.special_tokens = {
+            "<bos>": self.tokenizer.bos_id,
+            "<eos>": self.tokenizer.eos_id,
+            "<pad>": self.tokenizer.pad_id
+        }
+        self.encode_special_tokens = encode_special_tokens
+        super().__init__(padding_side=padding_side, clean_up_tokenization_spaces=clean_up_tokenization_spaces,
+                         encode_special_tokens=encode_special_tokens,
+                         **kwargs)
+
+    def get_command(self, token):
+        if token in self.special_tokens:
+            return self.special_tokens[token]
+        assert token in self.tokenizer.special_tokens, f"{token} is not a special token for {self.name}"
+        return self.tokenizer.special_tokens[token]
+
+    @property
+    def unk_token(self) -> str:
+        return "<unk>"
+
+    @property
+    def pad_token(self) -> str:
+        return "<unk>"
+
+    @property
+    def pad_token_id(self):
+        return self.get_command("<pad>")
+
+    @property
+    def eos_token(self) -> str:
+        return "</s>"
+
+    @property
+    def eos_token_id(self):
+        return self.get_command("<eos>")
+
+    @property
+    def vocab_size(self):
+        return self.tokenizer.n_words
+
+    def get_vocab(self):
+        """ Returns vocab as a dict """
+        vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
+        vocab.update(self.added_tokens_encoder)
+        return vocab
+
+    def _tokenize(self, text, **kwargs):
+        return self.tokenizer.tokenize(text, encode_special_tokens=self.encode_special_tokens)
+
+    def _convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        return self.tokenizer.convert_token_to_id(token)
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        return self.tokenizer.convert_id_to_token(index)
+
+    def convert_tokens_to_string(self, tokens: List[str]) -> str:
+        return self.tokenizer.decode_tokens(tokens)
+
+    def save_vocabulary(self, save_directory, filename_prefix=None):
+        """
+        Save the vocabulary and special tokens file to a directory.
+
+        Args:
+            save_directory (`str`):
+                The directory in which to save the vocabulary.
+            filename_prefix (`str`, *optional*):
+                An optional prefix to add to the named of the saved files.
+
+        Returns:
+            `Tuple(str)`: Paths to the files saved.
+        """
+        if os.path.isdir(save_directory):
+            vocab_file = os.path.join(
+                save_directory, self.vocab_files_names["vocab_file"]
+            )
+        else:
+            vocab_file = save_directory
+
+        with open(self.vocab_file, 'rb') as fin:
+            proto_str = fin.read()
+
+        with open(vocab_file, "wb") as writer:
+            writer.write(proto_str)
+
+        return (vocab_file,)
+
+    def get_prefix_tokens(self):
+        prefix_tokens = [self.get_command("[gMASK]"), self.get_command("sop")]
+        return prefix_tokens
+
+    def build_single_message(self, role, metadata, message):
+        assert role in ["system", "user", "assistant", "observation"], role
+        role_tokens = [self.get_command(f"<|{role}|>")] + self.tokenizer.encode(f"{metadata}\n")
+        message_tokens = self.tokenizer.encode(message)
+        tokens = role_tokens + message_tokens
+        return tokens
+
+    def build_chat_input(self, query, history=None, role="user"):
+        if history is None:
+            history = []
+        input_ids = []
+        for item in history:
+            content = item["content"]
+            if item["role"] == "system" and "tools" in item:
+                content = content + "\n" + json.dumps(item["tools"], indent=4, ensure_ascii=False)
+            input_ids.extend(self.build_single_message(item["role"], item.get("metadata", ""), content))
+        input_ids.extend(self.build_single_message(role, "", query))
+        input_ids.extend([self.get_command("<|assistant|>")])
+        return self.batch_encode_plus([input_ids], return_tensors="pt", is_split_into_words=True)
+
+    def build_inputs_with_special_tokens(
+            self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
+        adding special tokens. A BERT sequence has the following format:
+
+        - single sequence: `[CLS] X [SEP]`
+        - pair of sequences: `[CLS] A [SEP] B [SEP]`
+
+        Args:
+            token_ids_0 (`List[int]`):
+                List of IDs to which the special tokens will be added.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
+        """
+        prefix_tokens = self.get_prefix_tokens()
+        token_ids_0 = prefix_tokens + token_ids_0
+        if token_ids_1 is not None:
+            token_ids_0 = token_ids_0 + token_ids_1 + [self.get_command("<eos>")]
+        return token_ids_0
+
+    def _pad(
+            self,
+            encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
+            max_length: Optional[int] = None,
+            padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
+            pad_to_multiple_of: Optional[int] = None,
+            return_attention_mask: Optional[bool] = None,
+    ) -> dict:
+        """
+        Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
+
+        Args:
+            encoded_inputs:
+                Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`).
+            max_length: maximum length of the returned list and optionally padding length (see below).
+                Will truncate by taking into account the special tokens.
+            padding_strategy: PaddingStrategy to use for padding.
+
+                - PaddingStrategy.LONGEST Pad to the longest sequence in the batch
+                - PaddingStrategy.MAX_LENGTH: Pad to the max length (default)
+                - PaddingStrategy.DO_NOT_PAD: Do not pad
+                The tokenizer padding sides are defined in self.padding_side:
+
+                    - 'left': pads on the left of the sequences
+                    - 'right': pads on the right of the sequences
+            pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
+                This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
+                `>= 7.5` (Volta).
+            return_attention_mask:
+                (optional) Set to False to avoid returning attention mask (default: set to model specifics)
+        """
+        # Load from model defaults
+        assert self.padding_side == "left"
+
+        required_input = encoded_inputs[self.model_input_names[0]]
+        seq_length = len(required_input)
+
+        if padding_strategy == PaddingStrategy.LONGEST:
+            max_length = len(required_input)
+
+        if max_length is not None and pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
+            max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
+
+        needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len(required_input) != max_length
+
+        # Initialize attention mask if not present.
+        if "attention_mask" not in encoded_inputs:
+            encoded_inputs["attention_mask"] = [1] * seq_length
+
+        if "position_ids" not in encoded_inputs:
+            encoded_inputs["position_ids"] = list(range(seq_length))
+
+        if needs_to_be_padded:
+            difference = max_length - len(required_input)
+
+            if "attention_mask" in encoded_inputs:
+                encoded_inputs["attention_mask"] = [0] * difference + encoded_inputs["attention_mask"]
+            if "position_ids" in encoded_inputs:
+                encoded_inputs["position_ids"] = [0] * difference + encoded_inputs["position_ids"]
+            encoded_inputs[self.model_input_names[0]] = [self.pad_token_id] * difference + required_input
+
+        return encoded_inputs

checkpoint-1500/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7dc4c393423b76e4373e5157ddc34803a0189ba96b21ddbb40269d31468a6f2
+size 1018370

checkpoint-1500/tokenizer_config.json

@@ -0,0 +1,41 @@
+{
+  "added_tokens_decoder": {
+    "64795": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64797": {
+      "content": "<|observation|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|observation|>"
+  ],
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenization_chatglm.ChatGLMTokenizer",
+      null
+    ]
+  },
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": false,
+  "encode_special_tokens": false,
+  "eos_token": "</s>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<unk>",
+  "padding_side": "right",
+  "remove_space": false,
+  "split_special_tokens": false,
+  "tokenizer_class": "ChatGLMTokenizer",
+  "unk_token": "<unk>"
+}

checkpoint-1500/trainer_state.json

@@ -0,0 +1,1821 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.7527286413248024,
+  "eval_steps": 500,
+  "global_step": 1500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0,
+      "learning_rate": 4.9999980101927616e-05,
+      "loss": 3.2533,
+      "step": 5
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.99999204077421e-05,
+      "loss": 3.2279,
+      "step": 10
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.999978982687695e-05,
+      "loss": 3.193,
+      "step": 15
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.9999597065062966e-05,
+      "loss": 3.2863,
+      "step": 20
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.999934212277958e-05,
+      "loss": 3.1724,
+      "step": 25
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999902500066093e-05,
+      "loss": 3.1088,
+      "step": 30
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999864569949576e-05,
+      "loss": 3.3241,
+      "step": 35
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.99982042202275e-05,
+      "loss": 3.1904,
+      "step": 40
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999770056395421e-05,
+      "loss": 2.9254,
+      "step": 45
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.999713473192863e-05,
+      "loss": 3.1845,
+      "step": 50
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.999650672555812e-05,
+      "loss": 2.8475,
+      "step": 55
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.9995816546404695e-05,
+      "loss": 3.0486,
+      "step": 60
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.9995064196185014e-05,
+      "loss": 2.6464,
+      "step": 65
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.9994249676770364e-05,
+      "loss": 3.0446,
+      "step": 70
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999337299018667e-05,
+      "loss": 3.375,
+      "step": 75
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999243413861447e-05,
+      "loss": 2.8787,
+      "step": 80
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999143312438893e-05,
+      "loss": 3.014,
+      "step": 85
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.999036994999985e-05,
+      "loss": 2.8288,
+      "step": 90
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.9989244618091596e-05,
+      "loss": 3.1879,
+      "step": 95
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.998805713146317e-05,
+      "loss": 2.9749,
+      "step": 100
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.9986807493068165e-05,
+      "loss": 2.6304,
+      "step": 105
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.998549570601475e-05,
+      "loss": 2.943,
+      "step": 110
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.998412177356568e-05,
+      "loss": 2.7595,
+      "step": 115
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.9982685699138275e-05,
+      "loss": 2.7377,
+      "step": 120
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.9981187486304423e-05,
+      "loss": 2.9878,
+      "step": 125
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997962713879058e-05,
+      "loss": 2.7882,
+      "step": 130
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997800466047772e-05,
+      "loss": 2.7802,
+      "step": 135
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997632005540138e-05,
+      "loss": 3.0129,
+      "step": 140
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997457332775159e-05,
+      "loss": 2.9444,
+      "step": 145
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.997276448187294e-05,
+      "loss": 2.9664,
+      "step": 150
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.9970893522264476e-05,
+      "loss": 2.7367,
+      "step": 155
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.996896045357977e-05,
+      "loss": 2.7012,
+      "step": 160
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.9966965280626856e-05,
+      "loss": 2.8493,
+      "step": 165
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.996490800836825e-05,
+      "loss": 2.9274,
+      "step": 170
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.996278864192092e-05,
+      "loss": 2.8093,
+      "step": 175
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.9960607186556286e-05,
+      "loss": 3.1782,
+      "step": 180
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.995836364770018e-05,
+      "loss": 2.7507,
+      "step": 185
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.995605803093287e-05,
+      "loss": 2.6724,
+      "step": 190
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9953690341989026e-05,
+      "loss": 3.0258,
+      "step": 195
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9951260586757694e-05,
+      "loss": 3.1134,
+      "step": 200
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9948768771282314e-05,
+      "loss": 2.9937,
+      "step": 205
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.9946214901760665e-05,
+      "loss": 2.7394,
+      "step": 210
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.99435989845449e-05,
+      "loss": 2.9696,
+      "step": 215
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.994092102614146e-05,
+      "loss": 2.753,
+      "step": 220
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.993818103321113e-05,
+      "loss": 3.0759,
+      "step": 225
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.9935379012568985e-05,
+      "loss": 2.7512,
+      "step": 230
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.993251497118438e-05,
+      "loss": 2.8656,
+      "step": 235
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.992958891618091e-05,
+      "loss": 2.6628,
+      "step": 240
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.992660085483645e-05,
+      "loss": 2.8012,
+      "step": 245
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.992355079458307e-05,
+      "loss": 2.8141,
+      "step": 250
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.992043874300706e-05,
+      "loss": 2.9083,
+      "step": 255
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.991726470784891e-05,
+      "loss": 2.5846,
+      "step": 260
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.991402869700325e-05,
+      "loss": 2.8088,
+      "step": 265
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.991073071851889e-05,
+      "loss": 2.9979,
+      "step": 270
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.990737078059875e-05,
+      "loss": 3.0171,
+      "step": 275
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.990394889159986e-05,
+      "loss": 2.9278,
+      "step": 280
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.9900465060033364e-05,
+      "loss": 2.7998,
+      "step": 285
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.989691929456443e-05,
+      "loss": 2.721,
+      "step": 290
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.9893311604012306e-05,
+      "loss": 3.0291,
+      "step": 295
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.988964199735024e-05,
+      "loss": 2.9777,
+      "step": 300
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.988591048370552e-05,
+      "loss": 2.318,
+      "step": 305
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.988211707235936e-05,
+      "loss": 3.0332,
+      "step": 310
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987826177274697e-05,
+      "loss": 2.4888,
+      "step": 315
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987434459445748e-05,
+      "loss": 2.9259,
+      "step": 320
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987036554723391e-05,
+      "loss": 2.9568,
+      "step": 325
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.98663246409732e-05,
+      "loss": 2.8708,
+      "step": 330
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.986222188572611e-05,
+      "loss": 2.7848,
+      "step": 335
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.985805729169728e-05,
+      "loss": 2.6178,
+      "step": 340
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.985383086924511e-05,
+      "loss": 2.8326,
+      "step": 345
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.984954262888182e-05,
+      "loss": 2.8845,
+      "step": 350
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.9845192581273365e-05,
+      "loss": 2.6151,
+      "step": 355
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.984078073723944e-05,
+      "loss": 2.8474,
+      "step": 360
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.9836307107753455e-05,
+      "loss": 2.808,
+      "step": 365
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.983177170394248e-05,
+      "loss": 2.8491,
+      "step": 370
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.9827174537087226e-05,
+      "loss": 2.7764,
+      "step": 375
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.982251561862205e-05,
+      "loss": 2.7582,
+      "step": 380
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.981779496013489e-05,
+      "loss": 2.5379,
+      "step": 385
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.981301257336723e-05,
+      "loss": 2.9937,
+      "step": 390
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.980816847021412e-05,
+      "loss": 2.8574,
+      "step": 395
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.980326266272409e-05,
+      "loss": 2.9369,
+      "step": 400
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.979829516309915e-05,
+      "loss": 2.836,
+      "step": 405
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.979326598369477e-05,
+      "loss": 2.9369,
+      "step": 410
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.9788175137019814e-05,
+      "loss": 2.6667,
+      "step": 415
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.9783022635736534e-05,
+      "loss": 2.999,
+      "step": 420
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.977780849266054e-05,
+      "loss": 2.907,
+      "step": 425
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.9772532720760744e-05,
+      "loss": 2.8028,
+      "step": 430
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.976719533315937e-05,
+      "loss": 2.7999,
+      "step": 435
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.976179634313187e-05,
+      "loss": 2.8918,
+      "step": 440
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.9756335764106944e-05,
+      "loss": 2.7926,
+      "step": 445
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.975081360966646e-05,
+      "loss": 2.6709,
+      "step": 450
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.9745229893545436e-05,
+      "loss": 2.8248,
+      "step": 455
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.973958462963203e-05,
+      "loss": 3.1146,
+      "step": 460
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.973387783196747e-05,
+      "loss": 2.9991,
+      "step": 465
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.972810951474605e-05,
+      "loss": 2.7726,
+      "step": 470
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.972227969231505e-05,
+      "loss": 2.9025,
+      "step": 475
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.971638837917475e-05,
+      "loss": 2.6521,
+      "step": 480
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.971043558997839e-05,
+      "loss": 2.9511,
+      "step": 485
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.9704421339532075e-05,
+      "loss": 2.6938,
+      "step": 490
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.969834564279482e-05,
+      "loss": 2.8533,
+      "step": 495
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.9692208514878444e-05,
+      "loss": 2.6411,
+      "step": 500
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.968600997104758e-05,
+      "loss": 2.6486,
+      "step": 505
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.967975002671961e-05,
+      "loss": 2.8561,
+      "step": 510
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.967342869746463e-05,
+      "loss": 2.6984,
+      "step": 515
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.9667045999005424e-05,
+      "loss": 2.91,
+      "step": 520
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.966060194721742e-05,
+      "loss": 2.7205,
+      "step": 525
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.965409655812865e-05,
+      "loss": 2.7634,
+      "step": 530
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.9647529847919684e-05,
+      "loss": 2.9647,
+      "step": 535
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.964090183292364e-05,
+      "loss": 2.6357,
+      "step": 540
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.963421252962609e-05,
+      "loss": 3.0285,
+      "step": 545
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.96274619546651e-05,
+      "loss": 2.9895,
+      "step": 550
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.962065012483106e-05,
+      "loss": 2.7286,
+      "step": 555
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.961377705706677e-05,
+      "loss": 3.1337,
+      "step": 560
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.960684276846733e-05,
+      "loss": 2.8268,
+      "step": 565
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.959984727628011e-05,
+      "loss": 2.5851,
+      "step": 570
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.959279059790471e-05,
+      "loss": 2.9359,
+      "step": 575
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.958567275089291e-05,
+      "loss": 2.8842,
+      "step": 580
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.957849375294864e-05,
+      "loss": 2.6186,
+      "step": 585
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.957125362192794e-05,
+      "loss": 3.0116,
+      "step": 590
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.956395237583887e-05,
+      "loss": 2.7045,
+      "step": 595
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.9556590032841526e-05,
+      "loss": 2.8766,
+      "step": 600
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.954916661124797e-05,
+      "loss": 2.7858,
+      "step": 605
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.954168212952216e-05,
+      "loss": 2.7379,
+      "step": 610
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.953413660627995e-05,
+      "loss": 2.475,
+      "step": 615
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.9526530060289e-05,
+      "loss": 2.8357,
+      "step": 620
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.951886251046876e-05,
+      "loss": 2.9284,
+      "step": 625
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.951113397589042e-05,
+      "loss": 2.8144,
+      "step": 630
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9503344475776846e-05,
+      "loss": 2.6845,
+      "step": 635
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9495494029502535e-05,
+      "loss": 2.8937,
+      "step": 640
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9487582656593575e-05,
+      "loss": 3.0325,
+      "step": 645
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.94796103767276e-05,
+      "loss": 2.7058,
+      "step": 650
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.9471577209733746e-05,
+      "loss": 2.6162,
+      "step": 655
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.946348317559257e-05,
+      "loss": 3.0129,
+      "step": 660
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.945532829443603e-05,
+      "loss": 2.7587,
+      "step": 665
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.944711258654742e-05,
+      "loss": 2.8915,
+      "step": 670
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.943883607236135e-05,
+      "loss": 2.7343,
+      "step": 675
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.943049877246364e-05,
+      "loss": 2.881,
+      "step": 680
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.942210070759131e-05,
+      "loss": 2.488,
+      "step": 685
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.941364189863253e-05,
+      "loss": 2.896,
+      "step": 690
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.940512236662654e-05,
+      "loss": 2.7757,
+      "step": 695
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.9396542132763634e-05,
+      "loss": 2.6271,
+      "step": 700
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.938790121838506e-05,
+      "loss": 2.6804,
+      "step": 705
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.937919964498302e-05,
+      "loss": 2.7313,
+      "step": 710
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.937043743420058e-05,
+      "loss": 2.9277,
+      "step": 715
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.9361614607831605e-05,
+      "loss": 2.6366,
+      "step": 720
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.935273118782078e-05,
+      "loss": 3.0556,
+      "step": 725
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.934378719626345e-05,
+      "loss": 3.0182,
+      "step": 730
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.933478265540564e-05,
+      "loss": 2.824,
+      "step": 735
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.932571758764398e-05,
+      "loss": 2.636,
+      "step": 740
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.931659201552563e-05,
+      "loss": 2.7025,
+      "step": 745
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.930740596174827e-05,
+      "loss": 2.8919,
+      "step": 750
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.9298159449159965e-05,
+      "loss": 2.8434,
+      "step": 755
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.928885250075921e-05,
+      "loss": 2.9448,
+      "step": 760
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.927948513969478e-05,
+      "loss": 2.7312,
+      "step": 765
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.927005738926573e-05,
+      "loss": 2.8688,
+      "step": 770
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.926056927292132e-05,
+      "loss": 2.8639,
+      "step": 775
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.925102081426095e-05,
+      "loss": 2.7809,
+      "step": 780
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.9241412037034115e-05,
+      "loss": 3.0111,
+      "step": 785
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.9231742965140314e-05,
+      "loss": 2.7252,
+      "step": 790
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.922201362262905e-05,
+      "loss": 2.9717,
+      "step": 795
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.92122240336997e-05,
+      "loss": 2.7191,
+      "step": 800
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.920237422270153e-05,
+      "loss": 2.9346,
+      "step": 805
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9192464214133536e-05,
+      "loss": 2.7967,
+      "step": 810
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9182494032644496e-05,
+      "loss": 2.7326,
+      "step": 815
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.917246370303284e-05,
+      "loss": 2.8933,
+      "step": 820
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9162373250246575e-05,
+      "loss": 2.7939,
+      "step": 825
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.9152222699383273e-05,
+      "loss": 2.7807,
+      "step": 830
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.9142012075689994e-05,
+      "loss": 2.6996,
+      "step": 835
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.913174140456319e-05,
+      "loss": 2.7569,
+      "step": 840
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.912141071154869e-05,
+      "loss": 2.9347,
+      "step": 845
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.911102002234159e-05,
+      "loss": 2.7251,
+      "step": 850
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.910056936278623e-05,
+      "loss": 3.1228,
+      "step": 855
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.90900587588761e-05,
+      "loss": 2.9902,
+      "step": 860
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.9079488236753803e-05,
+      "loss": 2.5095,
+      "step": 865
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.906885782271095e-05,
+      "loss": 2.7523,
+      "step": 870
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.905816754318814e-05,
+      "loss": 2.6656,
+      "step": 875
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.9047417424774874e-05,
+      "loss": 2.8454,
+      "step": 880
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.903660749420946e-05,
+      "loss": 2.8194,
+      "step": 885
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.9025737778379025e-05,
+      "loss": 2.8421,
+      "step": 890
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.9014808304319326e-05,
+      "loss": 2.9656,
+      "step": 895
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.900381909921482e-05,
+      "loss": 2.7484,
+      "step": 900
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.899277019039849e-05,
+      "loss": 2.9044,
+      "step": 905
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.898166160535186e-05,
+      "loss": 2.8016,
+      "step": 910
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.8970493371704826e-05,
+      "loss": 2.6278,
+      "step": 915
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.895926551723569e-05,
+      "loss": 2.7214,
+      "step": 920
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.8947978069871036e-05,
+      "loss": 2.7679,
+      "step": 925
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8936631057685654e-05,
+      "loss": 2.7196,
+      "step": 930
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8925224508902514e-05,
+      "loss": 2.7866,
+      "step": 935
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8913758451892644e-05,
+      "loss": 2.72,
+      "step": 940
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.89022329151751e-05,
+      "loss": 2.752,
+      "step": 945
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8890647927416887e-05,
+      "loss": 3.0487,
+      "step": 950
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8879003517432857e-05,
+      "loss": 2.8928,
+      "step": 955
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.886729971418568e-05,
+      "loss": 2.7793,
+      "step": 960
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8855536546785726e-05,
+      "loss": 2.537,
+      "step": 965
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.884371404449105e-05,
+      "loss": 2.8345,
+      "step": 970
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.8831832236707284e-05,
+      "loss": 2.9673,
+      "step": 975
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.8819891152987546e-05,
+      "loss": 2.8295,
+      "step": 980
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.880789082303241e-05,
+      "loss": 3.0753,
+      "step": 985
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.879583127668979e-05,
+      "loss": 2.6748,
+      "step": 990
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.878371254395492e-05,
+      "loss": 2.8115,
+      "step": 995
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.877153465497022e-05,
+      "loss": 2.7039,
+      "step": 1000
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.8759297640025235e-05,
+      "loss": 2.9469,
+      "step": 1005
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.874700152955661e-05,
+      "loss": 2.9745,
+      "step": 1010
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.8734646354147936e-05,
+      "loss": 2.9341,
+      "step": 1015
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.8722232144529754e-05,
+      "loss": 3.0511,
+      "step": 1020
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.870975893157941e-05,
+      "loss": 2.5396,
+      "step": 1025
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.8697226746321004e-05,
+      "loss": 2.6699,
+      "step": 1030
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.868463561992532e-05,
+      "loss": 2.4887,
+      "step": 1035
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.867198558370977e-05,
+      "loss": 2.4773,
+      "step": 1040
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.865927666913825e-05,
+      "loss": 2.7612,
+      "step": 1045
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.864650890782113e-05,
+      "loss": 2.9116,
+      "step": 1050
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.863368233151514e-05,
+      "loss": 2.8205,
+      "step": 1055
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.862079697212329e-05,
+      "loss": 2.6711,
+      "step": 1060
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.8607852861694804e-05,
+      "loss": 3.1138,
+      "step": 1065
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.859485003242503e-05,
+      "loss": 2.5603,
+      "step": 1070
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.858178851665539e-05,
+      "loss": 2.7981,
+      "step": 1075
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.856866834687323e-05,
+      "loss": 2.507,
+      "step": 1080
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.855548955571183e-05,
+      "loss": 3.0315,
+      "step": 1085
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.8542252175950244e-05,
+      "loss": 2.75,
+      "step": 1090
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.852895624051326e-05,
+      "loss": 2.6794,
+      "step": 1095
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.851560178247132e-05,
+      "loss": 2.8278,
+      "step": 1100
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.850218883504041e-05,
+      "loss": 2.6993,
+      "step": 1105
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.8488717431582005e-05,
+      "loss": 2.875,
+      "step": 1110
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.8475187605602974e-05,
+      "loss": 2.6057,
+      "step": 1115
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.84615993907555e-05,
+      "loss": 2.847,
+      "step": 1120
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.844795282083697e-05,
+      "loss": 2.7652,
+      "step": 1125
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.843424792978997e-05,
+      "loss": 2.8128,
+      "step": 1130
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.842048475170209e-05,
+      "loss": 2.7077,
+      "step": 1135
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.840666332080592e-05,
+      "loss": 2.7081,
+      "step": 1140
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.8392783671478934e-05,
+      "loss": 2.6479,
+      "step": 1145
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.837884583824342e-05,
+      "loss": 2.667,
+      "step": 1150
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.836484985576638e-05,
+      "loss": 2.7514,
+      "step": 1155
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.835079575885944e-05,
+      "loss": 2.5058,
+      "step": 1160
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.833668358247876e-05,
+      "loss": 2.6183,
+      "step": 1165
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.8322513361725006e-05,
+      "loss": 2.9959,
+      "step": 1170
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.830828513184317e-05,
+      "loss": 2.5714,
+      "step": 1175
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.8293998928222536e-05,
+      "loss": 2.965,
+      "step": 1180
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.827965478639661e-05,
+      "loss": 2.2106,
+      "step": 1185
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8265252742042965e-05,
+      "loss": 2.7456,
+      "step": 1190
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8250792830983225e-05,
+      "loss": 2.5694,
+      "step": 1195
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8236275089182936e-05,
+      "loss": 2.6826,
+      "step": 1200
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8221699552751465e-05,
+      "loss": 2.878,
+      "step": 1205
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.820706625794196e-05,
+      "loss": 2.8191,
+      "step": 1210
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.81923752411512e-05,
+      "loss": 2.739,
+      "step": 1215
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.8177626538919565e-05,
+      "loss": 3.0544,
+      "step": 1220
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.8162820187930875e-05,
+      "loss": 2.8393,
+      "step": 1225
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.814795622501237e-05,
+      "loss": 2.4457,
+      "step": 1230
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.813303468713456e-05,
+      "loss": 2.8575,
+      "step": 1235
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.8118055611411197e-05,
+      "loss": 2.8307,
+      "step": 1240
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.810301903509909e-05,
+      "loss": 2.7951,
+      "step": 1245
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.8087924995598125e-05,
+      "loss": 2.8456,
+      "step": 1250
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.807277353045106e-05,
+      "loss": 2.3564,
+      "step": 1255
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.8057564677343524e-05,
+      "loss": 2.5076,
+      "step": 1260
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.8042298474103884e-05,
+      "loss": 2.605,
+      "step": 1265
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.8026974958703116e-05,
+      "loss": 2.4782,
+      "step": 1270
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.8011594169254784e-05,
+      "loss": 2.7193,
+      "step": 1275
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.799615614401488e-05,
+      "loss": 2.8284,
+      "step": 1280
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.798066092138178e-05,
+      "loss": 2.5378,
+      "step": 1285
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.796510853989612e-05,
+      "loss": 2.7396,
+      "step": 1290
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.794949903824069e-05,
+      "loss": 2.7948,
+      "step": 1295
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.793383245524035e-05,
+      "loss": 2.7818,
+      "step": 1300
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.791810882986197e-05,
+      "loss": 2.7334,
+      "step": 1305
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7902328201214256e-05,
+      "loss": 2.4824,
+      "step": 1310
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7886490608547727e-05,
+      "loss": 2.6131,
+      "step": 1315
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7870596091254584e-05,
+      "loss": 2.7778,
+      "step": 1320
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7854644688868594e-05,
+      "loss": 2.5263,
+      "step": 1325
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.783863644106502e-05,
+      "loss": 2.7825,
+      "step": 1330
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.782257138766053e-05,
+      "loss": 2.7902,
+      "step": 1335
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.7806449568613066e-05,
+      "loss": 2.8333,
+      "step": 1340
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.779027102402177e-05,
+      "loss": 2.856,
+      "step": 1345
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.777403579412686e-05,
+      "loss": 2.8021,
+      "step": 1350
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.775774391930956e-05,
+      "loss": 2.6639,
+      "step": 1355
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.7741395440091976e-05,
+      "loss": 2.5226,
+      "step": 1360
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.772499039713702e-05,
+      "loss": 2.6803,
+      "step": 1365
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 4.7708528831248274e-05,
+      "loss": 2.4608,
+      "step": 1370
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 4.769201078336991e-05,
+      "loss": 2.786,
+      "step": 1375
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 4.7675436294586586e-05,
+      "loss": 2.8294,
+      "step": 1380
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.7658805406123356e-05,
+      "loss": 2.6776,
+      "step": 1385
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.7642118159345544e-05,
+      "loss": 2.8003,
+      "step": 1390
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.762537459575865e-05,
+      "loss": 2.7796,
+      "step": 1395
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.7608574757008245e-05,
+      "loss": 2.9156,
+      "step": 1400
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.7591718684879883e-05,
+      "loss": 3.0521,
+      "step": 1405
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.7574806421298976e-05,
+      "loss": 2.6469,
+      "step": 1410
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.755783800833071e-05,
+      "loss": 2.8512,
+      "step": 1415
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.754081348817991e-05,
+      "loss": 2.66,
+      "step": 1420
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.752373290319096e-05,
+      "loss": 2.6625,
+      "step": 1425
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.7506596295847716e-05,
+      "loss": 2.6711,
+      "step": 1430
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.7489403708773346e-05,
+      "loss": 2.8951,
+      "step": 1435
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.747215518473026e-05,
+      "loss": 2.7375,
+      "step": 1440
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.745485076662e-05,
+      "loss": 2.8037,
+      "step": 1445
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.743749049748315e-05,
+      "loss": 2.5375,
+      "step": 1450
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.742007442049918e-05,
+      "loss": 2.7664,
+      "step": 1455
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.7402602578986374e-05,
+      "loss": 2.9644,
+      "step": 1460
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.738507501640175e-05,
+      "loss": 2.8212,
+      "step": 1465
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.736749177634087e-05,
+      "loss": 2.7201,
+      "step": 1470
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.734985290253782e-05,
+      "loss": 2.7087,
+      "step": 1475
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.7332158438865035e-05,
+      "loss": 2.5502,
+      "step": 1480
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.731440842933322e-05,
+      "loss": 2.7607,
+      "step": 1485
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.729660291809126e-05,
+      "loss": 2.5601,
+      "step": 1490
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.727874194942606e-05,
+      "loss": 2.5562,
+      "step": 1495
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.7260825567762486e-05,
+      "loss": 2.5539,
+      "step": 1500
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 9960,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "total_flos": 7.938917340256666e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1500/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:939ee45e9035366dc4952c5158e7d3a0d3426acdbfb5014d53ad1e260b19a19f
+size 4475

checkpoint-2000/README.md

@@ -0,0 +1,204 @@
+---
+library_name: peft
+base_model: ../chatglm3-6b-base
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+
+### Framework versions
+
+- PEFT 0.8.2

checkpoint-2000/adapter_config.json

@@ -0,0 +1,26 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../chatglm3-6b-base",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "query_key_value"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_rslora": false
+}

checkpoint-2000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c587894dfb0d6a36b1b2b2bd92c9853c7e8efe4659095de4b7806907caa047f
+size 7807744

checkpoint-2000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8fdc41063e12d72eb10d41d8c4ee533b99fd680d9681612b843ce985e48672e
+size 15644485

checkpoint-2000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44baa97c777d40935f5d263141ec5c7c84944e0c2da4b888420b7db30896fc13
+size 14575

checkpoint-2000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a9fc87da7bbd820e1299c4cd3906d309a71ffb7d550c30c19efae928b59c8ee
+size 627

checkpoint-2000/special_tokens_map.json

@@ -0,0 +1,18 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|observation|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ]
+}

checkpoint-2000/tokenization_chatglm.py

@@ -0,0 +1,300 @@
+import json
+import os
+import re
+from typing import List, Optional, Union, Dict
+from sentencepiece import SentencePieceProcessor
+from transformers import PreTrainedTokenizer
+from transformers.utils import logging, PaddingStrategy
+from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
+
+
+class SPTokenizer:
+    def __init__(self, model_path: str):
+        # reload tokenizer
+        assert os.path.isfile(model_path), model_path
+        self.sp_model = SentencePieceProcessor(model_file=model_path)
+
+        # BOS / EOS token IDs
+        self.n_words: int = self.sp_model.vocab_size()
+        self.bos_id: int = self.sp_model.bos_id()
+        self.eos_id: int = self.sp_model.eos_id()
+        self.pad_id: int = self.sp_model.unk_id()
+        assert self.sp_model.vocab_size() == self.sp_model.get_piece_size()
+
+        role_special_tokens = ["<|system|>", "<|user|>", "<|assistant|>", "<|observation|>"]
+        special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "sop", "eop"] + role_special_tokens
+        self.special_tokens = {}
+        self.index_special_tokens = {}
+        for token in special_tokens:
+            self.special_tokens[token] = self.n_words
+            self.index_special_tokens[self.n_words] = token
+            self.n_words += 1
+        self.role_special_token_expression = "|".join([re.escape(token) for token in role_special_tokens])
+
+    def tokenize(self, s: str, encode_special_tokens=False):
+        if encode_special_tokens:
+            last_index = 0
+            t = []
+            for match in re.finditer(self.role_special_token_expression, s):
+                if last_index < match.start():
+                    t.extend(self.sp_model.EncodeAsPieces(s[last_index:match.start()]))
+                t.append(s[match.start():match.end()])
+                last_index = match.end()
+            if last_index < len(s):
+                t.extend(self.sp_model.EncodeAsPieces(s[last_index:]))
+            return t
+        else:
+            return self.sp_model.EncodeAsPieces(s)
+
+    def encode(self, s: str, bos: bool = False, eos: bool = False) -> List[int]:
+        assert type(s) is str
+        t = self.sp_model.encode(s)
+        if bos:
+            t = [self.bos_id] + t
+        if eos:
+            t = t + [self.eos_id]
+        return t
+
+    def decode(self, t: List[int]) -> str:
+        text, buffer = "", []
+        for token in t:
+            if token in self.index_special_tokens:
+                if buffer:
+                    text += self.sp_model.decode(buffer)
+                    buffer = []
+                text += self.index_special_tokens[token]
+            else:
+                buffer.append(token)
+        if buffer:
+            text += self.sp_model.decode(buffer)
+        return text
+
+    def decode_tokens(self, tokens: List[str]) -> str:
+        text = self.sp_model.DecodePieces(tokens)
+        return text
+
+    def convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        if token in self.special_tokens:
+            return self.special_tokens[token]
+        return self.sp_model.PieceToId(token)
+
+    def convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        if index in self.index_special_tokens:
+            return self.index_special_tokens[index]
+        if index in [self.eos_id, self.bos_id, self.pad_id] or index < 0:
+            return ""
+        return self.sp_model.IdToPiece(index)
+
+
+class ChatGLMTokenizer(PreTrainedTokenizer):
+    vocab_files_names = {"vocab_file": "tokenizer.model"}
+
+    model_input_names = ["input_ids", "attention_mask", "position_ids"]
+
+    def __init__(self, vocab_file, padding_side="left", clean_up_tokenization_spaces=False, encode_special_tokens=False,
+                 **kwargs):
+        self.name = "GLMTokenizer"
+
+        self.vocab_file = vocab_file
+        self.tokenizer = SPTokenizer(vocab_file)
+        self.special_tokens = {
+            "<bos>": self.tokenizer.bos_id,
+            "<eos>": self.tokenizer.eos_id,
+            "<pad>": self.tokenizer.pad_id
+        }
+        self.encode_special_tokens = encode_special_tokens
+        super().__init__(padding_side=padding_side, clean_up_tokenization_spaces=clean_up_tokenization_spaces,
+                         encode_special_tokens=encode_special_tokens,
+                         **kwargs)
+
+    def get_command(self, token):
+        if token in self.special_tokens:
+            return self.special_tokens[token]
+        assert token in self.tokenizer.special_tokens, f"{token} is not a special token for {self.name}"
+        return self.tokenizer.special_tokens[token]
+
+    @property
+    def unk_token(self) -> str:
+        return "<unk>"
+
+    @property
+    def pad_token(self) -> str:
+        return "<unk>"
+
+    @property
+    def pad_token_id(self):
+        return self.get_command("<pad>")
+
+    @property
+    def eos_token(self) -> str:
+        return "</s>"
+
+    @property
+    def eos_token_id(self):
+        return self.get_command("<eos>")
+
+    @property
+    def vocab_size(self):
+        return self.tokenizer.n_words
+
+    def get_vocab(self):
+        """ Returns vocab as a dict """
+        vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
+        vocab.update(self.added_tokens_encoder)
+        return vocab
+
+    def _tokenize(self, text, **kwargs):
+        return self.tokenizer.tokenize(text, encode_special_tokens=self.encode_special_tokens)
+
+    def _convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        return self.tokenizer.convert_token_to_id(token)
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        return self.tokenizer.convert_id_to_token(index)
+
+    def convert_tokens_to_string(self, tokens: List[str]) -> str:
+        return self.tokenizer.decode_tokens(tokens)
+
+    def save_vocabulary(self, save_directory, filename_prefix=None):
+        """
+        Save the vocabulary and special tokens file to a directory.
+
+        Args:
+            save_directory (`str`):
+                The directory in which to save the vocabulary.
+            filename_prefix (`str`, *optional*):
+                An optional prefix to add to the named of the saved files.
+
+        Returns:
+            `Tuple(str)`: Paths to the files saved.
+        """
+        if os.path.isdir(save_directory):
+            vocab_file = os.path.join(
+                save_directory, self.vocab_files_names["vocab_file"]
+            )
+        else:
+            vocab_file = save_directory
+
+        with open(self.vocab_file, 'rb') as fin:
+            proto_str = fin.read()
+
+        with open(vocab_file, "wb") as writer:
+            writer.write(proto_str)
+
+        return (vocab_file,)
+
+    def get_prefix_tokens(self):
+        prefix_tokens = [self.get_command("[gMASK]"), self.get_command("sop")]
+        return prefix_tokens
+
+    def build_single_message(self, role, metadata, message):
+        assert role in ["system", "user", "assistant", "observation"], role
+        role_tokens = [self.get_command(f"<|{role}|>")] + self.tokenizer.encode(f"{metadata}\n")
+        message_tokens = self.tokenizer.encode(message)
+        tokens = role_tokens + message_tokens
+        return tokens
+
+    def build_chat_input(self, query, history=None, role="user"):
+        if history is None:
+            history = []
+        input_ids = []
+        for item in history:
+            content = item["content"]
+            if item["role"] == "system" and "tools" in item:
+                content = content + "\n" + json.dumps(item["tools"], indent=4, ensure_ascii=False)
+            input_ids.extend(self.build_single_message(item["role"], item.get("metadata", ""), content))
+        input_ids.extend(self.build_single_message(role, "", query))
+        input_ids.extend([self.get_command("<|assistant|>")])
+        return self.batch_encode_plus([input_ids], return_tensors="pt", is_split_into_words=True)
+
+    def build_inputs_with_special_tokens(
+            self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
+        adding special tokens. A BERT sequence has the following format:
+
+        - single sequence: `[CLS] X [SEP]`
+        - pair of sequences: `[CLS] A [SEP] B [SEP]`
+
+        Args:
+            token_ids_0 (`List[int]`):
+                List of IDs to which the special tokens will be added.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
+        """
+        prefix_tokens = self.get_prefix_tokens()
+        token_ids_0 = prefix_tokens + token_ids_0
+        if token_ids_1 is not None:
+            token_ids_0 = token_ids_0 + token_ids_1 + [self.get_command("<eos>")]
+        return token_ids_0
+
+    def _pad(
+            self,
+            encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
+            max_length: Optional[int] = None,
+            padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
+            pad_to_multiple_of: Optional[int] = None,
+            return_attention_mask: Optional[bool] = None,
+    ) -> dict:
+        """
+        Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
+
+        Args:
+            encoded_inputs:
+                Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`).
+            max_length: maximum length of the returned list and optionally padding length (see below).
+                Will truncate by taking into account the special tokens.
+            padding_strategy: PaddingStrategy to use for padding.
+
+                - PaddingStrategy.LONGEST Pad to the longest sequence in the batch
+                - PaddingStrategy.MAX_LENGTH: Pad to the max length (default)
+                - PaddingStrategy.DO_NOT_PAD: Do not pad
+                The tokenizer padding sides are defined in self.padding_side:
+
+                    - 'left': pads on the left of the sequences
+                    - 'right': pads on the right of the sequences
+            pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
+                This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
+                `>= 7.5` (Volta).
+            return_attention_mask:
+                (optional) Set to False to avoid returning attention mask (default: set to model specifics)
+        """
+        # Load from model defaults
+        assert self.padding_side == "left"
+
+        required_input = encoded_inputs[self.model_input_names[0]]
+        seq_length = len(required_input)
+
+        if padding_strategy == PaddingStrategy.LONGEST:
+            max_length = len(required_input)
+
+        if max_length is not None and pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
+            max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
+
+        needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len(required_input) != max_length
+
+        # Initialize attention mask if not present.
+        if "attention_mask" not in encoded_inputs:
+            encoded_inputs["attention_mask"] = [1] * seq_length
+
+        if "position_ids" not in encoded_inputs:
+            encoded_inputs["position_ids"] = list(range(seq_length))
+
+        if needs_to_be_padded:
+            difference = max_length - len(required_input)
+
+            if "attention_mask" in encoded_inputs:
+                encoded_inputs["attention_mask"] = [0] * difference + encoded_inputs["attention_mask"]
+            if "position_ids" in encoded_inputs:
+                encoded_inputs["position_ids"] = [0] * difference + encoded_inputs["position_ids"]
+            encoded_inputs[self.model_input_names[0]] = [self.pad_token_id] * difference + required_input
+
+        return encoded_inputs

checkpoint-2000/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7dc4c393423b76e4373e5157ddc34803a0189ba96b21ddbb40269d31468a6f2
+size 1018370

checkpoint-2000/tokenizer_config.json

@@ -0,0 +1,41 @@
+{
+  "added_tokens_decoder": {
+    "64795": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64797": {
+      "content": "<|observation|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|observation|>"
+  ],
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenization_chatglm.ChatGLMTokenizer",
+      null
+    ]
+  },
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": false,
+  "encode_special_tokens": false,
+  "eos_token": "</s>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<unk>",
+  "padding_side": "right",
+  "remove_space": false,
+  "split_special_tokens": false,
+  "tokenizer_class": "ChatGLMTokenizer",
+  "unk_token": "<unk>"
+}

checkpoint-2000/trainer_state.json

@@ -0,0 +1,2421 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0036381884330698,
+  "eval_steps": 500,
+  "global_step": 2000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0,
+      "learning_rate": 4.9999980101927616e-05,
+      "loss": 3.2533,
+      "step": 5
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.99999204077421e-05,
+      "loss": 3.2279,
+      "step": 10
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.999978982687695e-05,
+      "loss": 3.193,
+      "step": 15
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.9999597065062966e-05,
+      "loss": 3.2863,
+      "step": 20
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.999934212277958e-05,
+      "loss": 3.1724,
+      "step": 25
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999902500066093e-05,
+      "loss": 3.1088,
+      "step": 30
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999864569949576e-05,
+      "loss": 3.3241,
+      "step": 35
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.99982042202275e-05,
+      "loss": 3.1904,
+      "step": 40
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999770056395421e-05,
+      "loss": 2.9254,
+      "step": 45
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.999713473192863e-05,
+      "loss": 3.1845,
+      "step": 50
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.999650672555812e-05,
+      "loss": 2.8475,
+      "step": 55
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.9995816546404695e-05,
+      "loss": 3.0486,
+      "step": 60
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.9995064196185014e-05,
+      "loss": 2.6464,
+      "step": 65
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.9994249676770364e-05,
+      "loss": 3.0446,
+      "step": 70
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999337299018667e-05,
+      "loss": 3.375,
+      "step": 75
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999243413861447e-05,
+      "loss": 2.8787,
+      "step": 80
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999143312438893e-05,
+      "loss": 3.014,
+      "step": 85
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.999036994999985e-05,
+      "loss": 2.8288,
+      "step": 90
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.9989244618091596e-05,
+      "loss": 3.1879,
+      "step": 95
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.998805713146317e-05,
+      "loss": 2.9749,
+      "step": 100
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.9986807493068165e-05,
+      "loss": 2.6304,
+      "step": 105
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.998549570601475e-05,
+      "loss": 2.943,
+      "step": 110
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.998412177356568e-05,
+      "loss": 2.7595,
+      "step": 115
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.9982685699138275e-05,
+      "loss": 2.7377,
+      "step": 120
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.9981187486304423e-05,
+      "loss": 2.9878,
+      "step": 125
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997962713879058e-05,
+      "loss": 2.7882,
+      "step": 130
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997800466047772e-05,
+      "loss": 2.7802,
+      "step": 135
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997632005540138e-05,
+      "loss": 3.0129,
+      "step": 140
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997457332775159e-05,
+      "loss": 2.9444,
+      "step": 145
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.997276448187294e-05,
+      "loss": 2.9664,
+      "step": 150
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.9970893522264476e-05,
+      "loss": 2.7367,
+      "step": 155
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.996896045357977e-05,
+      "loss": 2.7012,
+      "step": 160
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.9966965280626856e-05,
+      "loss": 2.8493,
+      "step": 165
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.996490800836825e-05,
+      "loss": 2.9274,
+      "step": 170
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.996278864192092e-05,
+      "loss": 2.8093,
+      "step": 175
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.9960607186556286e-05,
+      "loss": 3.1782,
+      "step": 180
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.995836364770018e-05,
+      "loss": 2.7507,
+      "step": 185
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.995605803093287e-05,
+      "loss": 2.6724,
+      "step": 190
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9953690341989026e-05,
+      "loss": 3.0258,
+      "step": 195
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9951260586757694e-05,
+      "loss": 3.1134,
+      "step": 200
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9948768771282314e-05,
+      "loss": 2.9937,
+      "step": 205
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.9946214901760665e-05,
+      "loss": 2.7394,
+      "step": 210
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.99435989845449e-05,
+      "loss": 2.9696,
+      "step": 215
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.994092102614146e-05,
+      "loss": 2.753,
+      "step": 220
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.993818103321113e-05,
+      "loss": 3.0759,
+      "step": 225
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.9935379012568985e-05,
+      "loss": 2.7512,
+      "step": 230
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.993251497118438e-05,
+      "loss": 2.8656,
+      "step": 235
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.992958891618091e-05,
+      "loss": 2.6628,
+      "step": 240
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.992660085483645e-05,
+      "loss": 2.8012,
+      "step": 245
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.992355079458307e-05,
+      "loss": 2.8141,
+      "step": 250
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.992043874300706e-05,
+      "loss": 2.9083,
+      "step": 255
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.991726470784891e-05,
+      "loss": 2.5846,
+      "step": 260
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.991402869700325e-05,
+      "loss": 2.8088,
+      "step": 265
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.991073071851889e-05,
+      "loss": 2.9979,
+      "step": 270
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.990737078059875e-05,
+      "loss": 3.0171,
+      "step": 275
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.990394889159986e-05,
+      "loss": 2.9278,
+      "step": 280
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.9900465060033364e-05,
+      "loss": 2.7998,
+      "step": 285
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.989691929456443e-05,
+      "loss": 2.721,
+      "step": 290
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.9893311604012306e-05,
+      "loss": 3.0291,
+      "step": 295
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.988964199735024e-05,
+      "loss": 2.9777,
+      "step": 300
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.988591048370552e-05,
+      "loss": 2.318,
+      "step": 305
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.988211707235936e-05,
+      "loss": 3.0332,
+      "step": 310
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987826177274697e-05,
+      "loss": 2.4888,
+      "step": 315
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987434459445748e-05,
+      "loss": 2.9259,
+      "step": 320
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987036554723391e-05,
+      "loss": 2.9568,
+      "step": 325
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.98663246409732e-05,
+      "loss": 2.8708,
+      "step": 330
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.986222188572611e-05,
+      "loss": 2.7848,
+      "step": 335
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.985805729169728e-05,
+      "loss": 2.6178,
+      "step": 340
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.985383086924511e-05,
+      "loss": 2.8326,
+      "step": 345
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.984954262888182e-05,
+      "loss": 2.8845,
+      "step": 350
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.9845192581273365e-05,
+      "loss": 2.6151,
+      "step": 355
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.984078073723944e-05,
+      "loss": 2.8474,
+      "step": 360
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.9836307107753455e-05,
+      "loss": 2.808,
+      "step": 365
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.983177170394248e-05,
+      "loss": 2.8491,
+      "step": 370
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.9827174537087226e-05,
+      "loss": 2.7764,
+      "step": 375
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.982251561862205e-05,
+      "loss": 2.7582,
+      "step": 380
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.981779496013489e-05,
+      "loss": 2.5379,
+      "step": 385
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.981301257336723e-05,
+      "loss": 2.9937,
+      "step": 390
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.980816847021412e-05,
+      "loss": 2.8574,
+      "step": 395
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.980326266272409e-05,
+      "loss": 2.9369,
+      "step": 400
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.979829516309915e-05,
+      "loss": 2.836,
+      "step": 405
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.979326598369477e-05,
+      "loss": 2.9369,
+      "step": 410
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.9788175137019814e-05,
+      "loss": 2.6667,
+      "step": 415
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.9783022635736534e-05,
+      "loss": 2.999,
+      "step": 420
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.977780849266054e-05,
+      "loss": 2.907,
+      "step": 425
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.9772532720760744e-05,
+      "loss": 2.8028,
+      "step": 430
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.976719533315937e-05,
+      "loss": 2.7999,
+      "step": 435
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.976179634313187e-05,
+      "loss": 2.8918,
+      "step": 440
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.9756335764106944e-05,
+      "loss": 2.7926,
+      "step": 445
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.975081360966646e-05,
+      "loss": 2.6709,
+      "step": 450
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.9745229893545436e-05,
+      "loss": 2.8248,
+      "step": 455
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.973958462963203e-05,
+      "loss": 3.1146,
+      "step": 460
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.973387783196747e-05,
+      "loss": 2.9991,
+      "step": 465
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.972810951474605e-05,
+      "loss": 2.7726,
+      "step": 470
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.972227969231505e-05,
+      "loss": 2.9025,
+      "step": 475
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.971638837917475e-05,
+      "loss": 2.6521,
+      "step": 480
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.971043558997839e-05,
+      "loss": 2.9511,
+      "step": 485
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.9704421339532075e-05,
+      "loss": 2.6938,
+      "step": 490
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.969834564279482e-05,
+      "loss": 2.8533,
+      "step": 495
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.9692208514878444e-05,
+      "loss": 2.6411,
+      "step": 500
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.968600997104758e-05,
+      "loss": 2.6486,
+      "step": 505
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.967975002671961e-05,
+      "loss": 2.8561,
+      "step": 510
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.967342869746463e-05,
+      "loss": 2.6984,
+      "step": 515
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.9667045999005424e-05,
+      "loss": 2.91,
+      "step": 520
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.966060194721742e-05,
+      "loss": 2.7205,
+      "step": 525
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.965409655812865e-05,
+      "loss": 2.7634,
+      "step": 530
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.9647529847919684e-05,
+      "loss": 2.9647,
+      "step": 535
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.964090183292364e-05,
+      "loss": 2.6357,
+      "step": 540
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.963421252962609e-05,
+      "loss": 3.0285,
+      "step": 545
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.96274619546651e-05,
+      "loss": 2.9895,
+      "step": 550
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.962065012483106e-05,
+      "loss": 2.7286,
+      "step": 555
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.961377705706677e-05,
+      "loss": 3.1337,
+      "step": 560
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.960684276846733e-05,
+      "loss": 2.8268,
+      "step": 565
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.959984727628011e-05,
+      "loss": 2.5851,
+      "step": 570
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.959279059790471e-05,
+      "loss": 2.9359,
+      "step": 575
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.958567275089291e-05,
+      "loss": 2.8842,
+      "step": 580
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.957849375294864e-05,
+      "loss": 2.6186,
+      "step": 585
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.957125362192794e-05,
+      "loss": 3.0116,
+      "step": 590
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.956395237583887e-05,
+      "loss": 2.7045,
+      "step": 595
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.9556590032841526e-05,
+      "loss": 2.8766,
+      "step": 600
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.954916661124797e-05,
+      "loss": 2.7858,
+      "step": 605
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.954168212952216e-05,
+      "loss": 2.7379,
+      "step": 610
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.953413660627995e-05,
+      "loss": 2.475,
+      "step": 615
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.9526530060289e-05,
+      "loss": 2.8357,
+      "step": 620
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.951886251046876e-05,
+      "loss": 2.9284,
+      "step": 625
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.951113397589042e-05,
+      "loss": 2.8144,
+      "step": 630
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9503344475776846e-05,
+      "loss": 2.6845,
+      "step": 635
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9495494029502535e-05,
+      "loss": 2.8937,
+      "step": 640
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9487582656593575e-05,
+      "loss": 3.0325,
+      "step": 645
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.94796103767276e-05,
+      "loss": 2.7058,
+      "step": 650
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.9471577209733746e-05,
+      "loss": 2.6162,
+      "step": 655
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.946348317559257e-05,
+      "loss": 3.0129,
+      "step": 660
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.945532829443603e-05,
+      "loss": 2.7587,
+      "step": 665
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.944711258654742e-05,
+      "loss": 2.8915,
+      "step": 670
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.943883607236135e-05,
+      "loss": 2.7343,
+      "step": 675
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.943049877246364e-05,
+      "loss": 2.881,
+      "step": 680
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.942210070759131e-05,
+      "loss": 2.488,
+      "step": 685
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.941364189863253e-05,
+      "loss": 2.896,
+      "step": 690
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.940512236662654e-05,
+      "loss": 2.7757,
+      "step": 695
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.9396542132763634e-05,
+      "loss": 2.6271,
+      "step": 700
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.938790121838506e-05,
+      "loss": 2.6804,
+      "step": 705
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.937919964498302e-05,
+      "loss": 2.7313,
+      "step": 710
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.937043743420058e-05,
+      "loss": 2.9277,
+      "step": 715
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.9361614607831605e-05,
+      "loss": 2.6366,
+      "step": 720
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.935273118782078e-05,
+      "loss": 3.0556,
+      "step": 725
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.934378719626345e-05,
+      "loss": 3.0182,
+      "step": 730
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.933478265540564e-05,
+      "loss": 2.824,
+      "step": 735
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.932571758764398e-05,
+      "loss": 2.636,
+      "step": 740
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.931659201552563e-05,
+      "loss": 2.7025,
+      "step": 745
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.930740596174827e-05,
+      "loss": 2.8919,
+      "step": 750
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.9298159449159965e-05,
+      "loss": 2.8434,
+      "step": 755
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.928885250075921e-05,
+      "loss": 2.9448,
+      "step": 760
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.927948513969478e-05,
+      "loss": 2.7312,
+      "step": 765
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.927005738926573e-05,
+      "loss": 2.8688,
+      "step": 770
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.926056927292132e-05,
+      "loss": 2.8639,
+      "step": 775
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.925102081426095e-05,
+      "loss": 2.7809,
+      "step": 780
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.9241412037034115e-05,
+      "loss": 3.0111,
+      "step": 785
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.9231742965140314e-05,
+      "loss": 2.7252,
+      "step": 790
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.922201362262905e-05,
+      "loss": 2.9717,
+      "step": 795
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.92122240336997e-05,
+      "loss": 2.7191,
+      "step": 800
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.920237422270153e-05,
+      "loss": 2.9346,
+      "step": 805
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9192464214133536e-05,
+      "loss": 2.7967,
+      "step": 810
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9182494032644496e-05,
+      "loss": 2.7326,
+      "step": 815
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.917246370303284e-05,
+      "loss": 2.8933,
+      "step": 820
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9162373250246575e-05,
+      "loss": 2.7939,
+      "step": 825
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.9152222699383273e-05,
+      "loss": 2.7807,
+      "step": 830
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.9142012075689994e-05,
+      "loss": 2.6996,
+      "step": 835
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.913174140456319e-05,
+      "loss": 2.7569,
+      "step": 840
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.912141071154869e-05,
+      "loss": 2.9347,
+      "step": 845
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.911102002234159e-05,
+      "loss": 2.7251,
+      "step": 850
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.910056936278623e-05,
+      "loss": 3.1228,
+      "step": 855
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.90900587588761e-05,
+      "loss": 2.9902,
+      "step": 860
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.9079488236753803e-05,
+      "loss": 2.5095,
+      "step": 865
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.906885782271095e-05,
+      "loss": 2.7523,
+      "step": 870
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.905816754318814e-05,
+      "loss": 2.6656,
+      "step": 875
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.9047417424774874e-05,
+      "loss": 2.8454,
+      "step": 880
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.903660749420946e-05,
+      "loss": 2.8194,
+      "step": 885
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.9025737778379025e-05,
+      "loss": 2.8421,
+      "step": 890
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.9014808304319326e-05,
+      "loss": 2.9656,
+      "step": 895
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.900381909921482e-05,
+      "loss": 2.7484,
+      "step": 900
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.899277019039849e-05,
+      "loss": 2.9044,
+      "step": 905
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.898166160535186e-05,
+      "loss": 2.8016,
+      "step": 910
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.8970493371704826e-05,
+      "loss": 2.6278,
+      "step": 915
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.895926551723569e-05,
+      "loss": 2.7214,
+      "step": 920
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.8947978069871036e-05,
+      "loss": 2.7679,
+      "step": 925
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8936631057685654e-05,
+      "loss": 2.7196,
+      "step": 930
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8925224508902514e-05,
+      "loss": 2.7866,
+      "step": 935
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8913758451892644e-05,
+      "loss": 2.72,
+      "step": 940
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.89022329151751e-05,
+      "loss": 2.752,
+      "step": 945
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8890647927416887e-05,
+      "loss": 3.0487,
+      "step": 950
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8879003517432857e-05,
+      "loss": 2.8928,
+      "step": 955
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.886729971418568e-05,
+      "loss": 2.7793,
+      "step": 960
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8855536546785726e-05,
+      "loss": 2.537,
+      "step": 965
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.884371404449105e-05,
+      "loss": 2.8345,
+      "step": 970
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.8831832236707284e-05,
+      "loss": 2.9673,
+      "step": 975
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.8819891152987546e-05,
+      "loss": 2.8295,
+      "step": 980
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.880789082303241e-05,
+      "loss": 3.0753,
+      "step": 985
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.879583127668979e-05,
+      "loss": 2.6748,
+      "step": 990
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.878371254395492e-05,
+      "loss": 2.8115,
+      "step": 995
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.877153465497022e-05,
+      "loss": 2.7039,
+      "step": 1000
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.8759297640025235e-05,
+      "loss": 2.9469,
+      "step": 1005
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.874700152955661e-05,
+      "loss": 2.9745,
+      "step": 1010
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.8734646354147936e-05,
+      "loss": 2.9341,
+      "step": 1015
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.8722232144529754e-05,
+      "loss": 3.0511,
+      "step": 1020
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.870975893157941e-05,
+      "loss": 2.5396,
+      "step": 1025
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.8697226746321004e-05,
+      "loss": 2.6699,
+      "step": 1030
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.868463561992532e-05,
+      "loss": 2.4887,
+      "step": 1035
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.867198558370977e-05,
+      "loss": 2.4773,
+      "step": 1040
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.865927666913825e-05,
+      "loss": 2.7612,
+      "step": 1045
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.864650890782113e-05,
+      "loss": 2.9116,
+      "step": 1050
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.863368233151514e-05,
+      "loss": 2.8205,
+      "step": 1055
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.862079697212329e-05,
+      "loss": 2.6711,
+      "step": 1060
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.8607852861694804e-05,
+      "loss": 3.1138,
+      "step": 1065
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.859485003242503e-05,
+      "loss": 2.5603,
+      "step": 1070
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.858178851665539e-05,
+      "loss": 2.7981,
+      "step": 1075
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.856866834687323e-05,
+      "loss": 2.507,
+      "step": 1080
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.855548955571183e-05,
+      "loss": 3.0315,
+      "step": 1085
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.8542252175950244e-05,
+      "loss": 2.75,
+      "step": 1090
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.852895624051326e-05,
+      "loss": 2.6794,
+      "step": 1095
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.851560178247132e-05,
+      "loss": 2.8278,
+      "step": 1100
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.850218883504041e-05,
+      "loss": 2.6993,
+      "step": 1105
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.8488717431582005e-05,
+      "loss": 2.875,
+      "step": 1110
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.8475187605602974e-05,
+      "loss": 2.6057,
+      "step": 1115
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.84615993907555e-05,
+      "loss": 2.847,
+      "step": 1120
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.844795282083697e-05,
+      "loss": 2.7652,
+      "step": 1125
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.843424792978997e-05,
+      "loss": 2.8128,
+      "step": 1130
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.842048475170209e-05,
+      "loss": 2.7077,
+      "step": 1135
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.840666332080592e-05,
+      "loss": 2.7081,
+      "step": 1140
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.8392783671478934e-05,
+      "loss": 2.6479,
+      "step": 1145
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.837884583824342e-05,
+      "loss": 2.667,
+      "step": 1150
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.836484985576638e-05,
+      "loss": 2.7514,
+      "step": 1155
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.835079575885944e-05,
+      "loss": 2.5058,
+      "step": 1160
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.833668358247876e-05,
+      "loss": 2.6183,
+      "step": 1165
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.8322513361725006e-05,
+      "loss": 2.9959,
+      "step": 1170
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.830828513184317e-05,
+      "loss": 2.5714,
+      "step": 1175
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.8293998928222536e-05,
+      "loss": 2.965,
+      "step": 1180
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.827965478639661e-05,
+      "loss": 2.2106,
+      "step": 1185
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8265252742042965e-05,
+      "loss": 2.7456,
+      "step": 1190
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8250792830983225e-05,
+      "loss": 2.5694,
+      "step": 1195
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8236275089182936e-05,
+      "loss": 2.6826,
+      "step": 1200
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8221699552751465e-05,
+      "loss": 2.878,
+      "step": 1205
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.820706625794196e-05,
+      "loss": 2.8191,
+      "step": 1210
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.81923752411512e-05,
+      "loss": 2.739,
+      "step": 1215
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.8177626538919565e-05,
+      "loss": 3.0544,
+      "step": 1220
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.8162820187930875e-05,
+      "loss": 2.8393,
+      "step": 1225
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.814795622501237e-05,
+      "loss": 2.4457,
+      "step": 1230
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.813303468713456e-05,
+      "loss": 2.8575,
+      "step": 1235
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.8118055611411197e-05,
+      "loss": 2.8307,
+      "step": 1240
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.810301903509909e-05,
+      "loss": 2.7951,
+      "step": 1245
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.8087924995598125e-05,
+      "loss": 2.8456,
+      "step": 1250
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.807277353045106e-05,
+      "loss": 2.3564,
+      "step": 1255
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.8057564677343524e-05,
+      "loss": 2.5076,
+      "step": 1260
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.8042298474103884e-05,
+      "loss": 2.605,
+      "step": 1265
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.8026974958703116e-05,
+      "loss": 2.4782,
+      "step": 1270
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.8011594169254784e-05,
+      "loss": 2.7193,
+      "step": 1275
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.799615614401488e-05,
+      "loss": 2.8284,
+      "step": 1280
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.798066092138178e-05,
+      "loss": 2.5378,
+      "step": 1285
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.796510853989612e-05,
+      "loss": 2.7396,
+      "step": 1290
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.794949903824069e-05,
+      "loss": 2.7948,
+      "step": 1295
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.793383245524035e-05,
+      "loss": 2.7818,
+      "step": 1300
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.791810882986197e-05,
+      "loss": 2.7334,
+      "step": 1305
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7902328201214256e-05,
+      "loss": 2.4824,
+      "step": 1310
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7886490608547727e-05,
+      "loss": 2.6131,
+      "step": 1315
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7870596091254584e-05,
+      "loss": 2.7778,
+      "step": 1320
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7854644688868594e-05,
+      "loss": 2.5263,
+      "step": 1325
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.783863644106502e-05,
+      "loss": 2.7825,
+      "step": 1330
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.782257138766053e-05,
+      "loss": 2.7902,
+      "step": 1335
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.7806449568613066e-05,
+      "loss": 2.8333,
+      "step": 1340
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.779027102402177e-05,
+      "loss": 2.856,
+      "step": 1345
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.777403579412686e-05,
+      "loss": 2.8021,
+      "step": 1350
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.775774391930956e-05,
+      "loss": 2.6639,
+      "step": 1355
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.7741395440091976e-05,
+      "loss": 2.5226,
+      "step": 1360
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.772499039713702e-05,
+      "loss": 2.6803,
+      "step": 1365
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 4.7708528831248274e-05,
+      "loss": 2.4608,
+      "step": 1370
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 4.769201078336991e-05,
+      "loss": 2.786,
+      "step": 1375
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 4.7675436294586586e-05,
+      "loss": 2.8294,
+      "step": 1380
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.7658805406123356e-05,
+      "loss": 2.6776,
+      "step": 1385
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.7642118159345544e-05,
+      "loss": 2.8003,
+      "step": 1390
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.762537459575865e-05,
+      "loss": 2.7796,
+      "step": 1395
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.7608574757008245e-05,
+      "loss": 2.9156,
+      "step": 1400
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.7591718684879883e-05,
+      "loss": 3.0521,
+      "step": 1405
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.7574806421298976e-05,
+      "loss": 2.6469,
+      "step": 1410
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.755783800833071e-05,
+      "loss": 2.8512,
+      "step": 1415
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.754081348817991e-05,
+      "loss": 2.66,
+      "step": 1420
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.752373290319096e-05,
+      "loss": 2.6625,
+      "step": 1425
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.7506596295847716e-05,
+      "loss": 2.6711,
+      "step": 1430
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.7489403708773346e-05,
+      "loss": 2.8951,
+      "step": 1435
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.747215518473026e-05,
+      "loss": 2.7375,
+      "step": 1440
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.745485076662e-05,
+      "loss": 2.8037,
+      "step": 1445
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.743749049748315e-05,
+      "loss": 2.5375,
+      "step": 1450
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.742007442049918e-05,
+      "loss": 2.7664,
+      "step": 1455
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.7402602578986374e-05,
+      "loss": 2.9644,
+      "step": 1460
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.738507501640175e-05,
+      "loss": 2.8212,
+      "step": 1465
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.736749177634087e-05,
+      "loss": 2.7201,
+      "step": 1470
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.734985290253782e-05,
+      "loss": 2.7087,
+      "step": 1475
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.7332158438865035e-05,
+      "loss": 2.5502,
+      "step": 1480
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.731440842933322e-05,
+      "loss": 2.7607,
+      "step": 1485
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.729660291809126e-05,
+      "loss": 2.5601,
+      "step": 1490
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.727874194942606e-05,
+      "loss": 2.5562,
+      "step": 1495
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.7260825567762486e-05,
+      "loss": 2.5539,
+      "step": 1500
+    },
+    {
+      "epoch": 0.76,
+      "learning_rate": 4.7242853817663204e-05,
+      "loss": 2.6405,
+      "step": 1505
+    },
+    {
+      "epoch": 0.76,
+      "learning_rate": 4.72248267438286e-05,
+      "loss": 2.9237,
+      "step": 1510
+    },
+    {
+      "epoch": 0.76,
+      "learning_rate": 4.72067443910967e-05,
+      "loss": 3.0453,
+      "step": 1515
+    },
+    {
+      "epoch": 0.76,
+      "learning_rate": 4.718860680444297e-05,
+      "loss": 2.7176,
+      "step": 1520
+    },
+    {
+      "epoch": 0.77,
+      "learning_rate": 4.71704140289803e-05,
+      "loss": 2.6713,
+      "step": 1525
+    },
+    {
+      "epoch": 0.77,
+      "learning_rate": 4.715216610995883e-05,
+      "loss": 2.7284,
+      "step": 1530
+    },
+    {
+      "epoch": 0.77,
+      "learning_rate": 4.713386309276585e-05,
+      "loss": 2.5022,
+      "step": 1535
+    },
+    {
+      "epoch": 0.77,
+      "learning_rate": 4.7115505022925706e-05,
+      "loss": 2.8323,
+      "step": 1540
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 4.7097091946099666e-05,
+      "loss": 2.7005,
+      "step": 1545
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 4.7078623908085825e-05,
+      "loss": 2.8142,
+      "step": 1550
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 4.7060100954818974e-05,
+      "loss": 2.8505,
+      "step": 1555
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 4.70415231323705e-05,
+      "loss": 2.5892,
+      "step": 1560
+    },
+    {
+      "epoch": 0.79,
+      "learning_rate": 4.7022890486948236e-05,
+      "loss": 2.8951,
+      "step": 1565
+    },
+    {
+      "epoch": 0.79,
+      "learning_rate": 4.700420306489641e-05,
+      "loss": 2.7551,
+      "step": 1570
+    },
+    {
+      "epoch": 0.79,
+      "learning_rate": 4.698546091269547e-05,
+      "loss": 2.6814,
+      "step": 1575
+    },
+    {
+      "epoch": 0.79,
+      "learning_rate": 4.696666407696201e-05,
+      "loss": 2.8698,
+      "step": 1580
+    },
+    {
+      "epoch": 0.8,
+      "learning_rate": 4.694781260444862e-05,
+      "loss": 2.8389,
+      "step": 1585
+    },
+    {
+      "epoch": 0.8,
+      "learning_rate": 4.6928906542043786e-05,
+      "loss": 2.8353,
+      "step": 1590
+    },
+    {
+      "epoch": 0.8,
+      "learning_rate": 4.690994593677179e-05,
+      "loss": 2.6565,
+      "step": 1595
+    },
+    {
+      "epoch": 0.8,
+      "learning_rate": 4.689093083579256e-05,
+      "loss": 2.6958,
+      "step": 1600
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 4.687186128640157e-05,
+      "loss": 2.6768,
+      "step": 1605
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 4.685273733602975e-05,
+      "loss": 2.734,
+      "step": 1610
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 4.6833559032243284e-05,
+      "loss": 2.5348,
+      "step": 1615
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 4.6814326422743594e-05,
+      "loss": 2.7775,
+      "step": 1620
+    },
+    {
+      "epoch": 0.82,
+      "learning_rate": 4.679503955536715e-05,
+      "loss": 2.6578,
+      "step": 1625
+    },
+    {
+      "epoch": 0.82,
+      "learning_rate": 4.6775698478085393e-05,
+      "loss": 2.8792,
+      "step": 1630
+    },
+    {
+      "epoch": 0.82,
+      "learning_rate": 4.675630323900458e-05,
+      "loss": 2.7883,
+      "step": 1635
+    },
+    {
+      "epoch": 0.82,
+      "learning_rate": 4.67368538863657e-05,
+      "loss": 2.8824,
+      "step": 1640
+    },
+    {
+      "epoch": 0.83,
+      "learning_rate": 4.671735046854433e-05,
+      "loss": 2.6775,
+      "step": 1645
+    },
+    {
+      "epoch": 0.83,
+      "learning_rate": 4.669779303405051e-05,
+      "loss": 2.5658,
+      "step": 1650
+    },
+    {
+      "epoch": 0.83,
+      "learning_rate": 4.667818163152864e-05,
+      "loss": 2.5262,
+      "step": 1655
+    },
+    {
+      "epoch": 0.83,
+      "learning_rate": 4.665851630975736e-05,
+      "loss": 2.5749,
+      "step": 1660
+    },
+    {
+      "epoch": 0.84,
+      "learning_rate": 4.6638797117649424e-05,
+      "loss": 2.8718,
+      "step": 1665
+    },
+    {
+      "epoch": 0.84,
+      "learning_rate": 4.661902410425155e-05,
+      "loss": 2.8039,
+      "step": 1670
+    },
+    {
+      "epoch": 0.84,
+      "learning_rate": 4.659919731874435e-05,
+      "loss": 2.8089,
+      "step": 1675
+    },
+    {
+      "epoch": 0.84,
+      "learning_rate": 4.6579316810442174e-05,
+      "loss": 2.8144,
+      "step": 1680
+    },
+    {
+      "epoch": 0.85,
+      "learning_rate": 4.6559382628792995e-05,
+      "loss": 2.5963,
+      "step": 1685
+    },
+    {
+      "epoch": 0.85,
+      "learning_rate": 4.653939482337828e-05,
+      "loss": 2.5755,
+      "step": 1690
+    },
+    {
+      "epoch": 0.85,
+      "learning_rate": 4.651935344391286e-05,
+      "loss": 2.7631,
+      "step": 1695
+    },
+    {
+      "epoch": 0.85,
+      "learning_rate": 4.649925854024486e-05,
+      "loss": 2.8117,
+      "step": 1700
+    },
+    {
+      "epoch": 0.86,
+      "learning_rate": 4.647911016235549e-05,
+      "loss": 2.7352,
+      "step": 1705
+    },
+    {
+      "epoch": 0.86,
+      "learning_rate": 4.6458908360358985e-05,
+      "loss": 2.5572,
+      "step": 1710
+    },
+    {
+      "epoch": 0.86,
+      "learning_rate": 4.643865318450246e-05,
+      "loss": 2.8372,
+      "step": 1715
+    },
+    {
+      "epoch": 0.86,
+      "learning_rate": 4.6418344685165774e-05,
+      "loss": 2.7892,
+      "step": 1720
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 4.639798291286143e-05,
+      "loss": 2.7832,
+      "step": 1725
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 4.637756791823442e-05,
+      "loss": 2.5401,
+      "step": 1730
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 4.635709975206213e-05,
+      "loss": 2.9286,
+      "step": 1735
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 4.633657846525417e-05,
+      "loss": 2.6474,
+      "step": 1740
+    },
+    {
+      "epoch": 0.88,
+      "learning_rate": 4.6316004108852305e-05,
+      "loss": 2.5047,
+      "step": 1745
+    },
+    {
+      "epoch": 0.88,
+      "learning_rate": 4.629537673403029e-05,
+      "loss": 2.8374,
+      "step": 1750
+    },
+    {
+      "epoch": 0.88,
+      "learning_rate": 4.627469639209373e-05,
+      "loss": 2.634,
+      "step": 1755
+    },
+    {
+      "epoch": 0.88,
+      "learning_rate": 4.6253963134480006e-05,
+      "loss": 2.6884,
+      "step": 1760
+    },
+    {
+      "epoch": 0.89,
+      "learning_rate": 4.623317701275809e-05,
+      "loss": 2.6334,
+      "step": 1765
+    },
+    {
+      "epoch": 0.89,
+      "learning_rate": 4.621233807862844e-05,
+      "loss": 2.764,
+      "step": 1770
+    },
+    {
+      "epoch": 0.89,
+      "learning_rate": 4.6191446383922886e-05,
+      "loss": 3.0864,
+      "step": 1775
+    },
+    {
+      "epoch": 0.89,
+      "learning_rate": 4.617050198060448e-05,
+      "loss": 2.7964,
+      "step": 1780
+    },
+    {
+      "epoch": 0.9,
+      "learning_rate": 4.6149504920767376e-05,
+      "loss": 2.4538,
+      "step": 1785
+    },
+    {
+      "epoch": 0.9,
+      "learning_rate": 4.6128455256636706e-05,
+      "loss": 2.7352,
+      "step": 1790
+    },
+    {
+      "epoch": 0.9,
+      "learning_rate": 4.6107353040568416e-05,
+      "loss": 2.6893,
+      "step": 1795
+    },
+    {
+      "epoch": 0.9,
+      "learning_rate": 4.6086198325049185e-05,
+      "loss": 2.8609,
+      "step": 1800
+    },
+    {
+      "epoch": 0.91,
+      "learning_rate": 4.6064991162696275e-05,
+      "loss": 2.5285,
+      "step": 1805
+    },
+    {
+      "epoch": 0.91,
+      "learning_rate": 4.604373160625739e-05,
+      "loss": 2.5707,
+      "step": 1810
+    },
+    {
+      "epoch": 0.91,
+      "learning_rate": 4.602241970861053e-05,
+      "loss": 2.7198,
+      "step": 1815
+    },
+    {
+      "epoch": 0.91,
+      "learning_rate": 4.6001055522763926e-05,
+      "loss": 2.855,
+      "step": 1820
+    },
+    {
+      "epoch": 0.92,
+      "learning_rate": 4.597963910185582e-05,
+      "loss": 2.4175,
+      "step": 1825
+    },
+    {
+      "epoch": 0.92,
+      "learning_rate": 4.595817049915441e-05,
+      "loss": 2.5444,
+      "step": 1830
+    },
+    {
+      "epoch": 0.92,
+      "learning_rate": 4.5936649768057646e-05,
+      "loss": 2.6893,
+      "step": 1835
+    },
+    {
+      "epoch": 0.92,
+      "learning_rate": 4.591507696209318e-05,
+      "loss": 2.6725,
+      "step": 1840
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 4.589345213491817e-05,
+      "loss": 2.834,
+      "step": 1845
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 4.587177534031914e-05,
+      "loss": 2.8259,
+      "step": 1850
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 4.585004663221188e-05,
+      "loss": 2.6146,
+      "step": 1855
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 4.582826606464134e-05,
+      "loss": 2.4673,
+      "step": 1860
+    },
+    {
+      "epoch": 0.94,
+      "learning_rate": 4.5806433691781416e-05,
+      "loss": 2.9178,
+      "step": 1865
+    },
+    {
+      "epoch": 0.94,
+      "learning_rate": 4.578454956793487e-05,
+      "loss": 2.9224,
+      "step": 1870
+    },
+    {
+      "epoch": 0.94,
+      "learning_rate": 4.576261374753318e-05,
+      "loss": 2.7987,
+      "step": 1875
+    },
+    {
+      "epoch": 0.94,
+      "learning_rate": 4.574062628513642e-05,
+      "loss": 2.3848,
+      "step": 1880
+    },
+    {
+      "epoch": 0.95,
+      "learning_rate": 4.57185872354331e-05,
+      "loss": 2.6054,
+      "step": 1885
+    },
+    {
+      "epoch": 0.95,
+      "learning_rate": 4.569649665324003e-05,
+      "loss": 2.6743,
+      "step": 1890
+    },
+    {
+      "epoch": 0.95,
+      "learning_rate": 4.567435459350222e-05,
+      "loss": 2.9375,
+      "step": 1895
+    },
+    {
+      "epoch": 0.95,
+      "learning_rate": 4.565216111129269e-05,
+      "loss": 2.9372,
+      "step": 1900
+    },
+    {
+      "epoch": 0.96,
+      "learning_rate": 4.562991626181239e-05,
+      "loss": 2.669,
+      "step": 1905
+    },
+    {
+      "epoch": 0.96,
+      "learning_rate": 4.560762010039001e-05,
+      "loss": 2.7644,
+      "step": 1910
+    },
+    {
+      "epoch": 0.96,
+      "learning_rate": 4.558527268248187e-05,
+      "loss": 2.6886,
+      "step": 1915
+    },
+    {
+      "epoch": 0.96,
+      "learning_rate": 4.55628740636718e-05,
+      "loss": 2.8618,
+      "step": 1920
+    },
+    {
+      "epoch": 0.97,
+      "learning_rate": 4.554042429967095e-05,
+      "loss": 2.8411,
+      "step": 1925
+    },
+    {
+      "epoch": 0.97,
+      "learning_rate": 4.55179234463177e-05,
+      "loss": 2.6047,
+      "step": 1930
+    },
+    {
+      "epoch": 0.97,
+      "learning_rate": 4.5495371559577496e-05,
+      "loss": 2.8675,
+      "step": 1935
+    },
+    {
+      "epoch": 0.97,
+      "learning_rate": 4.547276869554271e-05,
+      "loss": 2.751,
+      "step": 1940
+    },
+    {
+      "epoch": 0.98,
+      "learning_rate": 4.545011491043253e-05,
+      "loss": 2.8638,
+      "step": 1945
+    },
+    {
+      "epoch": 0.98,
+      "learning_rate": 4.5427410260592775e-05,
+      "loss": 2.5092,
+      "step": 1950
+    },
+    {
+      "epoch": 0.98,
+      "learning_rate": 4.540465480249579e-05,
+      "loss": 2.5059,
+      "step": 1955
+    },
+    {
+      "epoch": 0.98,
+      "learning_rate": 4.5381848592740285e-05,
+      "loss": 2.9433,
+      "step": 1960
+    },
+    {
+      "epoch": 0.99,
+      "learning_rate": 4.535899168805121e-05,
+      "loss": 2.6959,
+      "step": 1965
+    },
+    {
+      "epoch": 0.99,
+      "learning_rate": 4.533608414527961e-05,
+      "loss": 2.552,
+      "step": 1970
+    },
+    {
+      "epoch": 0.99,
+      "learning_rate": 4.5313126021402465e-05,
+      "loss": 2.7169,
+      "step": 1975
+    },
+    {
+      "epoch": 0.99,
+      "learning_rate": 4.529011737352258e-05,
+      "loss": 2.5672,
+      "step": 1980
+    },
+    {
+      "epoch": 1.0,
+      "learning_rate": 4.526705825886841e-05,
+      "loss": 2.6434,
+      "step": 1985
+    },
+    {
+      "epoch": 1.0,
+      "learning_rate": 4.5243948734793947e-05,
+      "loss": 2.8062,
+      "step": 1990
+    },
+    {
+      "epoch": 1.0,
+      "learning_rate": 4.5220788858778556e-05,
+      "loss": 2.6929,
+      "step": 1995
+    },
+    {
+      "epoch": 1.0,
+      "learning_rate": 4.519757868842684e-05,
+      "loss": 2.5837,
+      "step": 2000
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 9960,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "total_flos": 1.0586878161626726e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-2000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:939ee45e9035366dc4952c5158e7d3a0d3426acdbfb5014d53ad1e260b19a19f
+size 4475

checkpoint-2500/README.md

@@ -0,0 +1,204 @@
+---
+library_name: peft
+base_model: ../chatglm3-6b-base
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+
+### Framework versions
+
+- PEFT 0.8.2

checkpoint-2500/adapter_config.json

@@ -0,0 +1,26 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../chatglm3-6b-base",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "query_key_value"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_rslora": false
+}

checkpoint-2500/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d40bc7150f5ecf32de16baf18217bae81b4927e8bfdd73a03a697fb34ae256da
+size 7807744

checkpoint-2500/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ea1db76a2f24e5349ca56065395a53258d6acb5f690b358d35067da24c96f32
+size 15644485

checkpoint-2500/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0423f88bd1e68d212dffb01bfbbae16c786b716d88f4e45eb7c085b41fde4a35
+size 14575

checkpoint-2500/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:24559f8836de5461d15dc4bbc244e10bf347c0266d3ef00c7465dff3b9068e19
+size 627

checkpoint-2500/special_tokens_map.json

@@ -0,0 +1,18 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|observation|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ]
+}

checkpoint-2500/tokenization_chatglm.py

@@ -0,0 +1,300 @@
+import json
+import os
+import re
+from typing import List, Optional, Union, Dict
+from sentencepiece import SentencePieceProcessor
+from transformers import PreTrainedTokenizer
+from transformers.utils import logging, PaddingStrategy
+from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
+
+
+class SPTokenizer:
+    def __init__(self, model_path: str):
+        # reload tokenizer
+        assert os.path.isfile(model_path), model_path
+        self.sp_model = SentencePieceProcessor(model_file=model_path)
+
+        # BOS / EOS token IDs
+        self.n_words: int = self.sp_model.vocab_size()
+        self.bos_id: int = self.sp_model.bos_id()
+        self.eos_id: int = self.sp_model.eos_id()
+        self.pad_id: int = self.sp_model.unk_id()
+        assert self.sp_model.vocab_size() == self.sp_model.get_piece_size()
+
+        role_special_tokens = ["<|system|>", "<|user|>", "<|assistant|>", "<|observation|>"]
+        special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "sop", "eop"] + role_special_tokens
+        self.special_tokens = {}
+        self.index_special_tokens = {}
+        for token in special_tokens:
+            self.special_tokens[token] = self.n_words
+            self.index_special_tokens[self.n_words] = token
+            self.n_words += 1
+        self.role_special_token_expression = "|".join([re.escape(token) for token in role_special_tokens])
+
+    def tokenize(self, s: str, encode_special_tokens=False):
+        if encode_special_tokens:
+            last_index = 0
+            t = []
+            for match in re.finditer(self.role_special_token_expression, s):
+                if last_index < match.start():
+                    t.extend(self.sp_model.EncodeAsPieces(s[last_index:match.start()]))
+                t.append(s[match.start():match.end()])
+                last_index = match.end()
+            if last_index < len(s):
+                t.extend(self.sp_model.EncodeAsPieces(s[last_index:]))
+            return t
+        else:
+            return self.sp_model.EncodeAsPieces(s)
+
+    def encode(self, s: str, bos: bool = False, eos: bool = False) -> List[int]:
+        assert type(s) is str
+        t = self.sp_model.encode(s)
+        if bos:
+            t = [self.bos_id] + t
+        if eos:
+            t = t + [self.eos_id]
+        return t
+
+    def decode(self, t: List[int]) -> str:
+        text, buffer = "", []
+        for token in t:
+            if token in self.index_special_tokens:
+                if buffer:
+                    text += self.sp_model.decode(buffer)
+                    buffer = []
+                text += self.index_special_tokens[token]
+            else:
+                buffer.append(token)
+        if buffer:
+            text += self.sp_model.decode(buffer)
+        return text
+
+    def decode_tokens(self, tokens: List[str]) -> str:
+        text = self.sp_model.DecodePieces(tokens)
+        return text
+
+    def convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        if token in self.special_tokens:
+            return self.special_tokens[token]
+        return self.sp_model.PieceToId(token)
+
+    def convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        if index in self.index_special_tokens:
+            return self.index_special_tokens[index]
+        if index in [self.eos_id, self.bos_id, self.pad_id] or index < 0:
+            return ""
+        return self.sp_model.IdToPiece(index)
+
+
+class ChatGLMTokenizer(PreTrainedTokenizer):
+    vocab_files_names = {"vocab_file": "tokenizer.model"}
+
+    model_input_names = ["input_ids", "attention_mask", "position_ids"]
+
+    def __init__(self, vocab_file, padding_side="left", clean_up_tokenization_spaces=False, encode_special_tokens=False,
+                 **kwargs):
+        self.name = "GLMTokenizer"
+
+        self.vocab_file = vocab_file
+        self.tokenizer = SPTokenizer(vocab_file)
+        self.special_tokens = {
+            "<bos>": self.tokenizer.bos_id,
+            "<eos>": self.tokenizer.eos_id,
+            "<pad>": self.tokenizer.pad_id
+        }
+        self.encode_special_tokens = encode_special_tokens
+        super().__init__(padding_side=padding_side, clean_up_tokenization_spaces=clean_up_tokenization_spaces,
+                         encode_special_tokens=encode_special_tokens,
+                         **kwargs)
+
+    def get_command(self, token):
+        if token in self.special_tokens:
+            return self.special_tokens[token]
+        assert token in self.tokenizer.special_tokens, f"{token} is not a special token for {self.name}"
+        return self.tokenizer.special_tokens[token]
+
+    @property
+    def unk_token(self) -> str:
+        return "<unk>"
+
+    @property
+    def pad_token(self) -> str:
+        return "<unk>"
+
+    @property
+    def pad_token_id(self):
+        return self.get_command("<pad>")
+
+    @property
+    def eos_token(self) -> str:
+        return "</s>"
+
+    @property
+    def eos_token_id(self):
+        return self.get_command("<eos>")
+
+    @property
+    def vocab_size(self):
+        return self.tokenizer.n_words
+
+    def get_vocab(self):
+        """ Returns vocab as a dict """
+        vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
+        vocab.update(self.added_tokens_encoder)
+        return vocab
+
+    def _tokenize(self, text, **kwargs):
+        return self.tokenizer.tokenize(text, encode_special_tokens=self.encode_special_tokens)
+
+    def _convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        return self.tokenizer.convert_token_to_id(token)
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        return self.tokenizer.convert_id_to_token(index)
+
+    def convert_tokens_to_string(self, tokens: List[str]) -> str:
+        return self.tokenizer.decode_tokens(tokens)
+
+    def save_vocabulary(self, save_directory, filename_prefix=None):
+        """
+        Save the vocabulary and special tokens file to a directory.
+
+        Args:
+            save_directory (`str`):
+                The directory in which to save the vocabulary.
+            filename_prefix (`str`, *optional*):
+                An optional prefix to add to the named of the saved files.
+
+        Returns:
+            `Tuple(str)`: Paths to the files saved.
+        """
+        if os.path.isdir(save_directory):
+            vocab_file = os.path.join(
+                save_directory, self.vocab_files_names["vocab_file"]
+            )
+        else:
+            vocab_file = save_directory
+
+        with open(self.vocab_file, 'rb') as fin:
+            proto_str = fin.read()
+
+        with open(vocab_file, "wb") as writer:
+            writer.write(proto_str)
+
+        return (vocab_file,)
+
+    def get_prefix_tokens(self):
+        prefix_tokens = [self.get_command("[gMASK]"), self.get_command("sop")]
+        return prefix_tokens
+
+    def build_single_message(self, role, metadata, message):
+        assert role in ["system", "user", "assistant", "observation"], role
+        role_tokens = [self.get_command(f"<|{role}|>")] + self.tokenizer.encode(f"{metadata}\n")
+        message_tokens = self.tokenizer.encode(message)
+        tokens = role_tokens + message_tokens
+        return tokens
+
+    def build_chat_input(self, query, history=None, role="user"):
+        if history is None:
+            history = []
+        input_ids = []
+        for item in history:
+            content = item["content"]
+            if item["role"] == "system" and "tools" in item:
+                content = content + "\n" + json.dumps(item["tools"], indent=4, ensure_ascii=False)
+            input_ids.extend(self.build_single_message(item["role"], item.get("metadata", ""), content))
+        input_ids.extend(self.build_single_message(role, "", query))
+        input_ids.extend([self.get_command("<|assistant|>")])
+        return self.batch_encode_plus([input_ids], return_tensors="pt", is_split_into_words=True)
+
+    def build_inputs_with_special_tokens(
+            self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
+        adding special tokens. A BERT sequence has the following format:
+
+        - single sequence: `[CLS] X [SEP]`
+        - pair of sequences: `[CLS] A [SEP] B [SEP]`
+
+        Args:
+            token_ids_0 (`List[int]`):
+                List of IDs to which the special tokens will be added.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
+        """
+        prefix_tokens = self.get_prefix_tokens()
+        token_ids_0 = prefix_tokens + token_ids_0
+        if token_ids_1 is not None:
+            token_ids_0 = token_ids_0 + token_ids_1 + [self.get_command("<eos>")]
+        return token_ids_0
+
+    def _pad(
+            self,
+            encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
+            max_length: Optional[int] = None,
+            padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
+            pad_to_multiple_of: Optional[int] = None,
+            return_attention_mask: Optional[bool] = None,
+    ) -> dict:
+        """
+        Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
+
+        Args:
+            encoded_inputs:
+                Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`).
+            max_length: maximum length of the returned list and optionally padding length (see below).
+                Will truncate by taking into account the special tokens.
+            padding_strategy: PaddingStrategy to use for padding.
+
+                - PaddingStrategy.LONGEST Pad to the longest sequence in the batch
+                - PaddingStrategy.MAX_LENGTH: Pad to the max length (default)
+                - PaddingStrategy.DO_NOT_PAD: Do not pad
+                The tokenizer padding sides are defined in self.padding_side:
+
+                    - 'left': pads on the left of the sequences
+                    - 'right': pads on the right of the sequences
+            pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
+                This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
+                `>= 7.5` (Volta).
+            return_attention_mask:
+                (optional) Set to False to avoid returning attention mask (default: set to model specifics)
+        """
+        # Load from model defaults
+        assert self.padding_side == "left"
+
+        required_input = encoded_inputs[self.model_input_names[0]]
+        seq_length = len(required_input)
+
+        if padding_strategy == PaddingStrategy.LONGEST:
+            max_length = len(required_input)
+
+        if max_length is not None and pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
+            max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
+
+        needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len(required_input) != max_length
+
+        # Initialize attention mask if not present.
+        if "attention_mask" not in encoded_inputs:
+            encoded_inputs["attention_mask"] = [1] * seq_length
+
+        if "position_ids" not in encoded_inputs:
+            encoded_inputs["position_ids"] = list(range(seq_length))
+
+        if needs_to_be_padded:
+            difference = max_length - len(required_input)
+
+            if "attention_mask" in encoded_inputs:
+                encoded_inputs["attention_mask"] = [0] * difference + encoded_inputs["attention_mask"]
+            if "position_ids" in encoded_inputs:
+                encoded_inputs["position_ids"] = [0] * difference + encoded_inputs["position_ids"]
+            encoded_inputs[self.model_input_names[0]] = [self.pad_token_id] * difference + required_input
+
+        return encoded_inputs

checkpoint-2500/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7dc4c393423b76e4373e5157ddc34803a0189ba96b21ddbb40269d31468a6f2
+size 1018370

checkpoint-2500/tokenizer_config.json

@@ -0,0 +1,41 @@
+{
+  "added_tokens_decoder": {
+    "64795": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64797": {
+      "content": "<|observation|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|observation|>"
+  ],
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenization_chatglm.ChatGLMTokenizer",
+      null
+    ]
+  },
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": false,
+  "encode_special_tokens": false,
+  "eos_token": "</s>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<unk>",
+  "padding_side": "right",
+  "remove_space": false,
+  "split_special_tokens": false,
+  "tokenizer_class": "ChatGLMTokenizer",
+  "unk_token": "<unk>"
+}

checkpoint-2500/trainer_state.json

@@ -0,0 +1,3021 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.2545477355413372,
+  "eval_steps": 500,
+  "global_step": 2500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0,
+      "learning_rate": 4.9999980101927616e-05,
+      "loss": 3.2533,
+      "step": 5
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.99999204077421e-05,
+      "loss": 3.2279,
+      "step": 10
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.999978982687695e-05,
+      "loss": 3.193,
+      "step": 15
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.9999597065062966e-05,
+      "loss": 3.2863,
+      "step": 20
+    },
+    {
+      "epoch": 0.01,
+      "learning_rate": 4.999934212277958e-05,
+      "loss": 3.1724,
+      "step": 25
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999902500066093e-05,
+      "loss": 3.1088,
+      "step": 30
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999864569949576e-05,
+      "loss": 3.3241,
+      "step": 35
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.99982042202275e-05,
+      "loss": 3.1904,
+      "step": 40
+    },
+    {
+      "epoch": 0.02,
+      "learning_rate": 4.999770056395421e-05,
+      "loss": 2.9254,
+      "step": 45
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.999713473192863e-05,
+      "loss": 3.1845,
+      "step": 50
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.999650672555812e-05,
+      "loss": 2.8475,
+      "step": 55
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.9995816546404695e-05,
+      "loss": 3.0486,
+      "step": 60
+    },
+    {
+      "epoch": 0.03,
+      "learning_rate": 4.9995064196185014e-05,
+      "loss": 2.6464,
+      "step": 65
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.9994249676770364e-05,
+      "loss": 3.0446,
+      "step": 70
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999337299018667e-05,
+      "loss": 3.375,
+      "step": 75
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999243413861447e-05,
+      "loss": 2.8787,
+      "step": 80
+    },
+    {
+      "epoch": 0.04,
+      "learning_rate": 4.999143312438893e-05,
+      "loss": 3.014,
+      "step": 85
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.999036994999985e-05,
+      "loss": 2.8288,
+      "step": 90
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.9989244618091596e-05,
+      "loss": 3.1879,
+      "step": 95
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.998805713146317e-05,
+      "loss": 2.9749,
+      "step": 100
+    },
+    {
+      "epoch": 0.05,
+      "learning_rate": 4.9986807493068165e-05,
+      "loss": 2.6304,
+      "step": 105
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.998549570601475e-05,
+      "loss": 2.943,
+      "step": 110
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.998412177356568e-05,
+      "loss": 2.7595,
+      "step": 115
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.9982685699138275e-05,
+      "loss": 2.7377,
+      "step": 120
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.9981187486304423e-05,
+      "loss": 2.9878,
+      "step": 125
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997962713879058e-05,
+      "loss": 2.7882,
+      "step": 130
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997800466047772e-05,
+      "loss": 2.7802,
+      "step": 135
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997632005540138e-05,
+      "loss": 3.0129,
+      "step": 140
+    },
+    {
+      "epoch": 0.07,
+      "learning_rate": 4.997457332775159e-05,
+      "loss": 2.9444,
+      "step": 145
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.997276448187294e-05,
+      "loss": 2.9664,
+      "step": 150
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.9970893522264476e-05,
+      "loss": 2.7367,
+      "step": 155
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.996896045357977e-05,
+      "loss": 2.7012,
+      "step": 160
+    },
+    {
+      "epoch": 0.08,
+      "learning_rate": 4.9966965280626856e-05,
+      "loss": 2.8493,
+      "step": 165
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.996490800836825e-05,
+      "loss": 2.9274,
+      "step": 170
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.996278864192092e-05,
+      "loss": 2.8093,
+      "step": 175
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.9960607186556286e-05,
+      "loss": 3.1782,
+      "step": 180
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.995836364770018e-05,
+      "loss": 2.7507,
+      "step": 185
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.995605803093287e-05,
+      "loss": 2.6724,
+      "step": 190
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9953690341989026e-05,
+      "loss": 3.0258,
+      "step": 195
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9951260586757694e-05,
+      "loss": 3.1134,
+      "step": 200
+    },
+    {
+      "epoch": 0.1,
+      "learning_rate": 4.9948768771282314e-05,
+      "loss": 2.9937,
+      "step": 205
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.9946214901760665e-05,
+      "loss": 2.7394,
+      "step": 210
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.99435989845449e-05,
+      "loss": 2.9696,
+      "step": 215
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.994092102614146e-05,
+      "loss": 2.753,
+      "step": 220
+    },
+    {
+      "epoch": 0.11,
+      "learning_rate": 4.993818103321113e-05,
+      "loss": 3.0759,
+      "step": 225
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.9935379012568985e-05,
+      "loss": 2.7512,
+      "step": 230
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.993251497118438e-05,
+      "loss": 2.8656,
+      "step": 235
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.992958891618091e-05,
+      "loss": 2.6628,
+      "step": 240
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 4.992660085483645e-05,
+      "loss": 2.8012,
+      "step": 245
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.992355079458307e-05,
+      "loss": 2.8141,
+      "step": 250
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.992043874300706e-05,
+      "loss": 2.9083,
+      "step": 255
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.991726470784891e-05,
+      "loss": 2.5846,
+      "step": 260
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 4.991402869700325e-05,
+      "loss": 2.8088,
+      "step": 265
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.991073071851889e-05,
+      "loss": 2.9979,
+      "step": 270
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.990737078059875e-05,
+      "loss": 3.0171,
+      "step": 275
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.990394889159986e-05,
+      "loss": 2.9278,
+      "step": 280
+    },
+    {
+      "epoch": 0.14,
+      "learning_rate": 4.9900465060033364e-05,
+      "loss": 2.7998,
+      "step": 285
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.989691929456443e-05,
+      "loss": 2.721,
+      "step": 290
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.9893311604012306e-05,
+      "loss": 3.0291,
+      "step": 295
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.988964199735024e-05,
+      "loss": 2.9777,
+      "step": 300
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 4.988591048370552e-05,
+      "loss": 2.318,
+      "step": 305
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.988211707235936e-05,
+      "loss": 3.0332,
+      "step": 310
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987826177274697e-05,
+      "loss": 2.4888,
+      "step": 315
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987434459445748e-05,
+      "loss": 2.9259,
+      "step": 320
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 4.987036554723391e-05,
+      "loss": 2.9568,
+      "step": 325
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.98663246409732e-05,
+      "loss": 2.8708,
+      "step": 330
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.986222188572611e-05,
+      "loss": 2.7848,
+      "step": 335
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.985805729169728e-05,
+      "loss": 2.6178,
+      "step": 340
+    },
+    {
+      "epoch": 0.17,
+      "learning_rate": 4.985383086924511e-05,
+      "loss": 2.8326,
+      "step": 345
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.984954262888182e-05,
+      "loss": 2.8845,
+      "step": 350
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.9845192581273365e-05,
+      "loss": 2.6151,
+      "step": 355
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.984078073723944e-05,
+      "loss": 2.8474,
+      "step": 360
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 4.9836307107753455e-05,
+      "loss": 2.808,
+      "step": 365
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.983177170394248e-05,
+      "loss": 2.8491,
+      "step": 370
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.9827174537087226e-05,
+      "loss": 2.7764,
+      "step": 375
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.982251561862205e-05,
+      "loss": 2.7582,
+      "step": 380
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 4.981779496013489e-05,
+      "loss": 2.5379,
+      "step": 385
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.981301257336723e-05,
+      "loss": 2.9937,
+      "step": 390
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.980816847021412e-05,
+      "loss": 2.8574,
+      "step": 395
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.980326266272409e-05,
+      "loss": 2.9369,
+      "step": 400
+    },
+    {
+      "epoch": 0.2,
+      "learning_rate": 4.979829516309915e-05,
+      "loss": 2.836,
+      "step": 405
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.979326598369477e-05,
+      "loss": 2.9369,
+      "step": 410
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.9788175137019814e-05,
+      "loss": 2.6667,
+      "step": 415
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.9783022635736534e-05,
+      "loss": 2.999,
+      "step": 420
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 4.977780849266054e-05,
+      "loss": 2.907,
+      "step": 425
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.9772532720760744e-05,
+      "loss": 2.8028,
+      "step": 430
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.976719533315937e-05,
+      "loss": 2.7999,
+      "step": 435
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.976179634313187e-05,
+      "loss": 2.8918,
+      "step": 440
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 4.9756335764106944e-05,
+      "loss": 2.7926,
+      "step": 445
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.975081360966646e-05,
+      "loss": 2.6709,
+      "step": 450
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.9745229893545436e-05,
+      "loss": 2.8248,
+      "step": 455
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.973958462963203e-05,
+      "loss": 3.1146,
+      "step": 460
+    },
+    {
+      "epoch": 0.23,
+      "learning_rate": 4.973387783196747e-05,
+      "loss": 2.9991,
+      "step": 465
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.972810951474605e-05,
+      "loss": 2.7726,
+      "step": 470
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.972227969231505e-05,
+      "loss": 2.9025,
+      "step": 475
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.971638837917475e-05,
+      "loss": 2.6521,
+      "step": 480
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 4.971043558997839e-05,
+      "loss": 2.9511,
+      "step": 485
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.9704421339532075e-05,
+      "loss": 2.6938,
+      "step": 490
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.969834564279482e-05,
+      "loss": 2.8533,
+      "step": 495
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.9692208514878444e-05,
+      "loss": 2.6411,
+      "step": 500
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 4.968600997104758e-05,
+      "loss": 2.6486,
+      "step": 505
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.967975002671961e-05,
+      "loss": 2.8561,
+      "step": 510
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.967342869746463e-05,
+      "loss": 2.6984,
+      "step": 515
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.9667045999005424e-05,
+      "loss": 2.91,
+      "step": 520
+    },
+    {
+      "epoch": 0.26,
+      "learning_rate": 4.966060194721742e-05,
+      "loss": 2.7205,
+      "step": 525
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.965409655812865e-05,
+      "loss": 2.7634,
+      "step": 530
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.9647529847919684e-05,
+      "loss": 2.9647,
+      "step": 535
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.964090183292364e-05,
+      "loss": 2.6357,
+      "step": 540
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 4.963421252962609e-05,
+      "loss": 3.0285,
+      "step": 545
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.96274619546651e-05,
+      "loss": 2.9895,
+      "step": 550
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.962065012483106e-05,
+      "loss": 2.7286,
+      "step": 555
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.961377705706677e-05,
+      "loss": 3.1337,
+      "step": 560
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 4.960684276846733e-05,
+      "loss": 2.8268,
+      "step": 565
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.959984727628011e-05,
+      "loss": 2.5851,
+      "step": 570
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.959279059790471e-05,
+      "loss": 2.9359,
+      "step": 575
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.958567275089291e-05,
+      "loss": 2.8842,
+      "step": 580
+    },
+    {
+      "epoch": 0.29,
+      "learning_rate": 4.957849375294864e-05,
+      "loss": 2.6186,
+      "step": 585
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.957125362192794e-05,
+      "loss": 3.0116,
+      "step": 590
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.956395237583887e-05,
+      "loss": 2.7045,
+      "step": 595
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.9556590032841526e-05,
+      "loss": 2.8766,
+      "step": 600
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 4.954916661124797e-05,
+      "loss": 2.7858,
+      "step": 605
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.954168212952216e-05,
+      "loss": 2.7379,
+      "step": 610
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.953413660627995e-05,
+      "loss": 2.475,
+      "step": 615
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.9526530060289e-05,
+      "loss": 2.8357,
+      "step": 620
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 4.951886251046876e-05,
+      "loss": 2.9284,
+      "step": 625
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.951113397589042e-05,
+      "loss": 2.8144,
+      "step": 630
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9503344475776846e-05,
+      "loss": 2.6845,
+      "step": 635
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9495494029502535e-05,
+      "loss": 2.8937,
+      "step": 640
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 4.9487582656593575e-05,
+      "loss": 3.0325,
+      "step": 645
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.94796103767276e-05,
+      "loss": 2.7058,
+      "step": 650
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.9471577209733746e-05,
+      "loss": 2.6162,
+      "step": 655
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.946348317559257e-05,
+      "loss": 3.0129,
+      "step": 660
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 4.945532829443603e-05,
+      "loss": 2.7587,
+      "step": 665
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.944711258654742e-05,
+      "loss": 2.8915,
+      "step": 670
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.943883607236135e-05,
+      "loss": 2.7343,
+      "step": 675
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.943049877246364e-05,
+      "loss": 2.881,
+      "step": 680
+    },
+    {
+      "epoch": 0.34,
+      "learning_rate": 4.942210070759131e-05,
+      "loss": 2.488,
+      "step": 685
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.941364189863253e-05,
+      "loss": 2.896,
+      "step": 690
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.940512236662654e-05,
+      "loss": 2.7757,
+      "step": 695
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.9396542132763634e-05,
+      "loss": 2.6271,
+      "step": 700
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 4.938790121838506e-05,
+      "loss": 2.6804,
+      "step": 705
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.937919964498302e-05,
+      "loss": 2.7313,
+      "step": 710
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.937043743420058e-05,
+      "loss": 2.9277,
+      "step": 715
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.9361614607831605e-05,
+      "loss": 2.6366,
+      "step": 720
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 4.935273118782078e-05,
+      "loss": 3.0556,
+      "step": 725
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.934378719626345e-05,
+      "loss": 3.0182,
+      "step": 730
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.933478265540564e-05,
+      "loss": 2.824,
+      "step": 735
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.932571758764398e-05,
+      "loss": 2.636,
+      "step": 740
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 4.931659201552563e-05,
+      "loss": 2.7025,
+      "step": 745
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.930740596174827e-05,
+      "loss": 2.8919,
+      "step": 750
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.9298159449159965e-05,
+      "loss": 2.8434,
+      "step": 755
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.928885250075921e-05,
+      "loss": 2.9448,
+      "step": 760
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 4.927948513969478e-05,
+      "loss": 2.7312,
+      "step": 765
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.927005738926573e-05,
+      "loss": 2.8688,
+      "step": 770
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.926056927292132e-05,
+      "loss": 2.8639,
+      "step": 775
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.925102081426095e-05,
+      "loss": 2.7809,
+      "step": 780
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 4.9241412037034115e-05,
+      "loss": 3.0111,
+      "step": 785
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.9231742965140314e-05,
+      "loss": 2.7252,
+      "step": 790
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.922201362262905e-05,
+      "loss": 2.9717,
+      "step": 795
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.92122240336997e-05,
+      "loss": 2.7191,
+      "step": 800
+    },
+    {
+      "epoch": 0.4,
+      "learning_rate": 4.920237422270153e-05,
+      "loss": 2.9346,
+      "step": 805
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9192464214133536e-05,
+      "loss": 2.7967,
+      "step": 810
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9182494032644496e-05,
+      "loss": 2.7326,
+      "step": 815
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.917246370303284e-05,
+      "loss": 2.8933,
+      "step": 820
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 4.9162373250246575e-05,
+      "loss": 2.7939,
+      "step": 825
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.9152222699383273e-05,
+      "loss": 2.7807,
+      "step": 830
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.9142012075689994e-05,
+      "loss": 2.6996,
+      "step": 835
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.913174140456319e-05,
+      "loss": 2.7569,
+      "step": 840
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 4.912141071154869e-05,
+      "loss": 2.9347,
+      "step": 845
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.911102002234159e-05,
+      "loss": 2.7251,
+      "step": 850
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.910056936278623e-05,
+      "loss": 3.1228,
+      "step": 855
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.90900587588761e-05,
+      "loss": 2.9902,
+      "step": 860
+    },
+    {
+      "epoch": 0.43,
+      "learning_rate": 4.9079488236753803e-05,
+      "loss": 2.5095,
+      "step": 865
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.906885782271095e-05,
+      "loss": 2.7523,
+      "step": 870
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.905816754318814e-05,
+      "loss": 2.6656,
+      "step": 875
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.9047417424774874e-05,
+      "loss": 2.8454,
+      "step": 880
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 4.903660749420946e-05,
+      "loss": 2.8194,
+      "step": 885
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.9025737778379025e-05,
+      "loss": 2.8421,
+      "step": 890
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.9014808304319326e-05,
+      "loss": 2.9656,
+      "step": 895
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.900381909921482e-05,
+      "loss": 2.7484,
+      "step": 900
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 4.899277019039849e-05,
+      "loss": 2.9044,
+      "step": 905
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.898166160535186e-05,
+      "loss": 2.8016,
+      "step": 910
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.8970493371704826e-05,
+      "loss": 2.6278,
+      "step": 915
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.895926551723569e-05,
+      "loss": 2.7214,
+      "step": 920
+    },
+    {
+      "epoch": 0.46,
+      "learning_rate": 4.8947978069871036e-05,
+      "loss": 2.7679,
+      "step": 925
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8936631057685654e-05,
+      "loss": 2.7196,
+      "step": 930
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8925224508902514e-05,
+      "loss": 2.7866,
+      "step": 935
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.8913758451892644e-05,
+      "loss": 2.72,
+      "step": 940
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 4.89022329151751e-05,
+      "loss": 2.752,
+      "step": 945
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8890647927416887e-05,
+      "loss": 3.0487,
+      "step": 950
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8879003517432857e-05,
+      "loss": 2.8928,
+      "step": 955
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.886729971418568e-05,
+      "loss": 2.7793,
+      "step": 960
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 4.8855536546785726e-05,
+      "loss": 2.537,
+      "step": 965
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.884371404449105e-05,
+      "loss": 2.8345,
+      "step": 970
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.8831832236707284e-05,
+      "loss": 2.9673,
+      "step": 975
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.8819891152987546e-05,
+      "loss": 2.8295,
+      "step": 980
+    },
+    {
+      "epoch": 0.49,
+      "learning_rate": 4.880789082303241e-05,
+      "loss": 3.0753,
+      "step": 985
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.879583127668979e-05,
+      "loss": 2.6748,
+      "step": 990
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.878371254395492e-05,
+      "loss": 2.8115,
+      "step": 995
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.877153465497022e-05,
+      "loss": 2.7039,
+      "step": 1000
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 4.8759297640025235e-05,
+      "loss": 2.9469,
+      "step": 1005
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.874700152955661e-05,
+      "loss": 2.9745,
+      "step": 1010
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.8734646354147936e-05,
+      "loss": 2.9341,
+      "step": 1015
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.8722232144529754e-05,
+      "loss": 3.0511,
+      "step": 1020
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 4.870975893157941e-05,
+      "loss": 2.5396,
+      "step": 1025
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.8697226746321004e-05,
+      "loss": 2.6699,
+      "step": 1030
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.868463561992532e-05,
+      "loss": 2.4887,
+      "step": 1035
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.867198558370977e-05,
+      "loss": 2.4773,
+      "step": 1040
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 4.865927666913825e-05,
+      "loss": 2.7612,
+      "step": 1045
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.864650890782113e-05,
+      "loss": 2.9116,
+      "step": 1050
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.863368233151514e-05,
+      "loss": 2.8205,
+      "step": 1055
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.862079697212329e-05,
+      "loss": 2.6711,
+      "step": 1060
+    },
+    {
+      "epoch": 0.53,
+      "learning_rate": 4.8607852861694804e-05,
+      "loss": 3.1138,
+      "step": 1065
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.859485003242503e-05,
+      "loss": 2.5603,
+      "step": 1070
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.858178851665539e-05,
+      "loss": 2.7981,
+      "step": 1075
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.856866834687323e-05,
+      "loss": 2.507,
+      "step": 1080
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 4.855548955571183e-05,
+      "loss": 3.0315,
+      "step": 1085
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.8542252175950244e-05,
+      "loss": 2.75,
+      "step": 1090
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.852895624051326e-05,
+      "loss": 2.6794,
+      "step": 1095
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.851560178247132e-05,
+      "loss": 2.8278,
+      "step": 1100
+    },
+    {
+      "epoch": 0.55,
+      "learning_rate": 4.850218883504041e-05,
+      "loss": 2.6993,
+      "step": 1105
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.8488717431582005e-05,
+      "loss": 2.875,
+      "step": 1110
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.8475187605602974e-05,
+      "loss": 2.6057,
+      "step": 1115
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.84615993907555e-05,
+      "loss": 2.847,
+      "step": 1120
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.844795282083697e-05,
+      "loss": 2.7652,
+      "step": 1125
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.843424792978997e-05,
+      "loss": 2.8128,
+      "step": 1130
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.842048475170209e-05,
+      "loss": 2.7077,
+      "step": 1135
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.840666332080592e-05,
+      "loss": 2.7081,
+      "step": 1140
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 4.8392783671478934e-05,
+      "loss": 2.6479,
+      "step": 1145
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.837884583824342e-05,
+      "loss": 2.667,
+      "step": 1150
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.836484985576638e-05,
+      "loss": 2.7514,
+      "step": 1155
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.835079575885944e-05,
+      "loss": 2.5058,
+      "step": 1160
+    },
+    {
+      "epoch": 0.58,
+      "learning_rate": 4.833668358247876e-05,
+      "loss": 2.6183,
+      "step": 1165
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.8322513361725006e-05,
+      "loss": 2.9959,
+      "step": 1170
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.830828513184317e-05,
+      "loss": 2.5714,
+      "step": 1175
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.8293998928222536e-05,
+      "loss": 2.965,
+      "step": 1180
+    },
+    {
+      "epoch": 0.59,
+      "learning_rate": 4.827965478639661e-05,
+      "loss": 2.2106,
+      "step": 1185
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8265252742042965e-05,
+      "loss": 2.7456,
+      "step": 1190
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8250792830983225e-05,
+      "loss": 2.5694,
+      "step": 1195
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8236275089182936e-05,
+      "loss": 2.6826,
+      "step": 1200
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 4.8221699552751465e-05,
+      "loss": 2.878,
+      "step": 1205
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.820706625794196e-05,
+      "loss": 2.8191,
+      "step": 1210
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.81923752411512e-05,
+      "loss": 2.739,
+      "step": 1215
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.8177626538919565e-05,
+      "loss": 3.0544,
+      "step": 1220
+    },
+    {
+      "epoch": 0.61,
+      "learning_rate": 4.8162820187930875e-05,
+      "loss": 2.8393,
+      "step": 1225
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.814795622501237e-05,
+      "loss": 2.4457,
+      "step": 1230
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.813303468713456e-05,
+      "loss": 2.8575,
+      "step": 1235
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.8118055611411197e-05,
+      "loss": 2.8307,
+      "step": 1240
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.810301903509909e-05,
+      "loss": 2.7951,
+      "step": 1245
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.8087924995598125e-05,
+      "loss": 2.8456,
+      "step": 1250
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.807277353045106e-05,
+      "loss": 2.3564,
+      "step": 1255
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.8057564677343524e-05,
+      "loss": 2.5076,
+      "step": 1260
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 4.8042298474103884e-05,
+      "loss": 2.605,
+      "step": 1265
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.8026974958703116e-05,
+      "loss": 2.4782,
+      "step": 1270
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.8011594169254784e-05,
+      "loss": 2.7193,
+      "step": 1275
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.799615614401488e-05,
+      "loss": 2.8284,
+      "step": 1280
+    },
+    {
+      "epoch": 0.64,
+      "learning_rate": 4.798066092138178e-05,
+      "loss": 2.5378,
+      "step": 1285
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.796510853989612e-05,
+      "loss": 2.7396,
+      "step": 1290
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.794949903824069e-05,
+      "loss": 2.7948,
+      "step": 1295
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.793383245524035e-05,
+      "loss": 2.7818,
+      "step": 1300
+    },
+    {
+      "epoch": 0.65,
+      "learning_rate": 4.791810882986197e-05,
+      "loss": 2.7334,
+      "step": 1305
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7902328201214256e-05,
+      "loss": 2.4824,
+      "step": 1310
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7886490608547727e-05,
+      "loss": 2.6131,
+      "step": 1315
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7870596091254584e-05,
+      "loss": 2.7778,
+      "step": 1320
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 4.7854644688868594e-05,
+      "loss": 2.5263,
+      "step": 1325
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.783863644106502e-05,
+      "loss": 2.7825,
+      "step": 1330
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.782257138766053e-05,
+      "loss": 2.7902,
+      "step": 1335
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.7806449568613066e-05,
+      "loss": 2.8333,
+      "step": 1340
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.779027102402177e-05,
+      "loss": 2.856,
+      "step": 1345
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.777403579412686e-05,
+      "loss": 2.8021,
+      "step": 1350
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.775774391930956e-05,
+      "loss": 2.6639,
+      "step": 1355
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.7741395440091976e-05,
+      "loss": 2.5226,
+      "step": 1360
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 4.772499039713702e-05,
+      "loss": 2.6803,
+      "step": 1365
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 4.7708528831248274e-05,
+      "loss": 2.4608,
+      "step": 1370
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 4.769201078336991e-05,
+      "loss": 2.786,
+      "step": 1375
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 4.7675436294586586e-05,
+      "loss": 2.8294,
+      "step": 1380
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.7658805406123356e-05,
+      "loss": 2.6776,
+      "step": 1385
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.7642118159345544e-05,
+      "loss": 2.8003,
+      "step": 1390
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.762537459575865e-05,
+      "loss": 2.7796,
+      "step": 1395
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 4.7608574757008245e-05,
+      "loss": 2.9156,
+      "step": 1400
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.7591718684879883e-05,
+      "loss": 3.0521,
+      "step": 1405
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.7574806421298976e-05,
+      "loss": 2.6469,
+      "step": 1410
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.755783800833071e-05,
+      "loss": 2.8512,
+      "step": 1415
+    },
+    {
+      "epoch": 0.71,
+      "learning_rate": 4.754081348817991e-05,
+      "loss": 2.66,
+      "step": 1420
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.752373290319096e-05,
+      "loss": 2.6625,
+      "step": 1425
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.7506596295847716e-05,
+      "loss": 2.6711,
+      "step": 1430
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.7489403708773346e-05,
+      "loss": 2.8951,
+      "step": 1435
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 4.747215518473026e-05,
+      "loss": 2.7375,
+      "step": 1440
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.745485076662e-05,
+      "loss": 2.8037,
+      "step": 1445
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.743749049748315e-05,
+      "loss": 2.5375,
+      "step": 1450
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.742007442049918e-05,
+      "loss": 2.7664,
+      "step": 1455
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 4.7402602578986374e-05,
+      "loss": 2.9644,
+      "step": 1460
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.738507501640175e-05,
+      "loss": 2.8212,
+      "step": 1465
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.736749177634087e-05,
+      "loss": 2.7201,
+      "step": 1470
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.734985290253782e-05,
+      "loss": 2.7087,
+      "step": 1475
+    },
+    {
+      "epoch": 0.74,
+      "learning_rate": 4.7332158438865035e-05,
+      "loss": 2.5502,
+      "step": 1480
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.731440842933322e-05,
+      "loss": 2.7607,
+      "step": 1485
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.729660291809126e-05,
+      "loss": 2.5601,
+      "step": 1490
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.727874194942606e-05,
+      "loss": 2.5562,
+      "step": 1495
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 4.7260825567762486e-05,
+      "loss": 2.5539,
+      "step": 1500
+    },
+    {
+      "epoch": 0.76,
+      "learning_rate": 4.7242853817663204e-05,
+      "loss": 2.6405,
+      "step": 1505
+    },
+    {
+      "epoch": 0.76,
+      "learning_rate": 4.72248267438286e-05,
+      "loss": 2.9237,
+      "step": 1510
+    },
+    {
+      "epoch": 0.76,
+      "learning_rate": 4.72067443910967e-05,
+      "loss": 3.0453,
+      "step": 1515
+    },
+    {
+      "epoch": 0.76,
+      "learning_rate": 4.718860680444297e-05,
+      "loss": 2.7176,
+      "step": 1520
+    },
+    {
+      "epoch": 0.77,
+      "learning_rate": 4.71704140289803e-05,
+      "loss": 2.6713,
+      "step": 1525
+    },
+    {
+      "epoch": 0.77,
+      "learning_rate": 4.715216610995883e-05,
+      "loss": 2.7284,
+      "step": 1530
+    },
+    {
+      "epoch": 0.77,
+      "learning_rate": 4.713386309276585e-05,
+      "loss": 2.5022,
+      "step": 1535
+    },
+    {
+      "epoch": 0.77,
+      "learning_rate": 4.7115505022925706e-05,
+      "loss": 2.8323,
+      "step": 1540
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 4.7097091946099666e-05,
+      "loss": 2.7005,
+      "step": 1545
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 4.7078623908085825e-05,
+      "loss": 2.8142,
+      "step": 1550
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 4.7060100954818974e-05,
+      "loss": 2.8505,
+      "step": 1555
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 4.70415231323705e-05,
+      "loss": 2.5892,
+      "step": 1560
+    },
+    {
+      "epoch": 0.79,
+      "learning_rate": 4.7022890486948236e-05,
+      "loss": 2.8951,
+      "step": 1565
+    },
+    {
+      "epoch": 0.79,
+      "learning_rate": 4.700420306489641e-05,
+      "loss": 2.7551,
+      "step": 1570
+    },
+    {
+      "epoch": 0.79,
+      "learning_rate": 4.698546091269547e-05,
+      "loss": 2.6814,
+      "step": 1575
+    },
+    {
+      "epoch": 0.79,
+      "learning_rate": 4.696666407696201e-05,
+      "loss": 2.8698,
+      "step": 1580
+    },
+    {
+      "epoch": 0.8,
+      "learning_rate": 4.694781260444862e-05,
+      "loss": 2.8389,
+      "step": 1585
+    },
+    {
+      "epoch": 0.8,
+      "learning_rate": 4.6928906542043786e-05,
+      "loss": 2.8353,
+      "step": 1590
+    },
+    {
+      "epoch": 0.8,
+      "learning_rate": 4.690994593677179e-05,
+      "loss": 2.6565,
+      "step": 1595
+    },
+    {
+      "epoch": 0.8,
+      "learning_rate": 4.689093083579256e-05,
+      "loss": 2.6958,
+      "step": 1600
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 4.687186128640157e-05,
+      "loss": 2.6768,
+      "step": 1605
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 4.685273733602975e-05,
+      "loss": 2.734,
+      "step": 1610
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 4.6833559032243284e-05,
+      "loss": 2.5348,
+      "step": 1615
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 4.6814326422743594e-05,
+      "loss": 2.7775,
+      "step": 1620
+    },
+    {
+      "epoch": 0.82,
+      "learning_rate": 4.679503955536715e-05,
+      "loss": 2.6578,
+      "step": 1625
+    },
+    {
+      "epoch": 0.82,
+      "learning_rate": 4.6775698478085393e-05,
+      "loss": 2.8792,
+      "step": 1630
+    },
+    {
+      "epoch": 0.82,
+      "learning_rate": 4.675630323900458e-05,
+      "loss": 2.7883,
+      "step": 1635
+    },
+    {
+      "epoch": 0.82,
+      "learning_rate": 4.67368538863657e-05,
+      "loss": 2.8824,
+      "step": 1640
+    },
+    {
+      "epoch": 0.83,
+      "learning_rate": 4.671735046854433e-05,
+      "loss": 2.6775,
+      "step": 1645
+    },
+    {
+      "epoch": 0.83,
+      "learning_rate": 4.669779303405051e-05,
+      "loss": 2.5658,
+      "step": 1650
+    },
+    {
+      "epoch": 0.83,
+      "learning_rate": 4.667818163152864e-05,
+      "loss": 2.5262,
+      "step": 1655
+    },
+    {
+      "epoch": 0.83,
+      "learning_rate": 4.665851630975736e-05,
+      "loss": 2.5749,
+      "step": 1660
+    },
+    {
+      "epoch": 0.84,
+      "learning_rate": 4.6638797117649424e-05,
+      "loss": 2.8718,
+      "step": 1665
+    },
+    {
+      "epoch": 0.84,
+      "learning_rate": 4.661902410425155e-05,
+      "loss": 2.8039,
+      "step": 1670
+    },
+    {
+      "epoch": 0.84,
+      "learning_rate": 4.659919731874435e-05,
+      "loss": 2.8089,
+      "step": 1675
+    },
+    {
+      "epoch": 0.84,
+      "learning_rate": 4.6579316810442174e-05,
+      "loss": 2.8144,
+      "step": 1680
+    },
+    {
+      "epoch": 0.85,
+      "learning_rate": 4.6559382628792995e-05,
+      "loss": 2.5963,
+      "step": 1685
+    },
+    {
+      "epoch": 0.85,
+      "learning_rate": 4.653939482337828e-05,
+      "loss": 2.5755,
+      "step": 1690
+    },
+    {
+      "epoch": 0.85,
+      "learning_rate": 4.651935344391286e-05,
+      "loss": 2.7631,
+      "step": 1695
+    },
+    {
+      "epoch": 0.85,
+      "learning_rate": 4.649925854024486e-05,
+      "loss": 2.8117,
+      "step": 1700
+    },
+    {
+      "epoch": 0.86,
+      "learning_rate": 4.647911016235549e-05,
+      "loss": 2.7352,
+      "step": 1705
+    },
+    {
+      "epoch": 0.86,
+      "learning_rate": 4.6458908360358985e-05,
+      "loss": 2.5572,
+      "step": 1710
+    },
+    {
+      "epoch": 0.86,
+      "learning_rate": 4.643865318450246e-05,
+      "loss": 2.8372,
+      "step": 1715
+    },
+    {
+      "epoch": 0.86,
+      "learning_rate": 4.6418344685165774e-05,
+      "loss": 2.7892,
+      "step": 1720
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 4.639798291286143e-05,
+      "loss": 2.7832,
+      "step": 1725
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 4.637756791823442e-05,
+      "loss": 2.5401,
+      "step": 1730
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 4.635709975206213e-05,
+      "loss": 2.9286,
+      "step": 1735
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 4.633657846525417e-05,
+      "loss": 2.6474,
+      "step": 1740
+    },
+    {
+      "epoch": 0.88,
+      "learning_rate": 4.6316004108852305e-05,
+      "loss": 2.5047,
+      "step": 1745
+    },
+    {
+      "epoch": 0.88,
+      "learning_rate": 4.629537673403029e-05,
+      "loss": 2.8374,
+      "step": 1750
+    },
+    {
+      "epoch": 0.88,
+      "learning_rate": 4.627469639209373e-05,
+      "loss": 2.634,
+      "step": 1755
+    },
+    {
+      "epoch": 0.88,
+      "learning_rate": 4.6253963134480006e-05,
+      "loss": 2.6884,
+      "step": 1760
+    },
+    {
+      "epoch": 0.89,
+      "learning_rate": 4.623317701275809e-05,
+      "loss": 2.6334,
+      "step": 1765
+    },
+    {
+      "epoch": 0.89,
+      "learning_rate": 4.621233807862844e-05,
+      "loss": 2.764,
+      "step": 1770
+    },
+    {
+      "epoch": 0.89,
+      "learning_rate": 4.6191446383922886e-05,
+      "loss": 3.0864,
+      "step": 1775
+    },
+    {
+      "epoch": 0.89,
+      "learning_rate": 4.617050198060448e-05,
+      "loss": 2.7964,
+      "step": 1780
+    },
+    {
+      "epoch": 0.9,
+      "learning_rate": 4.6149504920767376e-05,
+      "loss": 2.4538,
+      "step": 1785
+    },
+    {
+      "epoch": 0.9,
+      "learning_rate": 4.6128455256636706e-05,
+      "loss": 2.7352,
+      "step": 1790
+    },
+    {
+      "epoch": 0.9,
+      "learning_rate": 4.6107353040568416e-05,
+      "loss": 2.6893,
+      "step": 1795
+    },
+    {
+      "epoch": 0.9,
+      "learning_rate": 4.6086198325049185e-05,
+      "loss": 2.8609,
+      "step": 1800
+    },
+    {
+      "epoch": 0.91,
+      "learning_rate": 4.6064991162696275e-05,
+      "loss": 2.5285,
+      "step": 1805
+    },
+    {
+      "epoch": 0.91,
+      "learning_rate": 4.604373160625739e-05,
+      "loss": 2.5707,
+      "step": 1810
+    },
+    {
+      "epoch": 0.91,
+      "learning_rate": 4.602241970861053e-05,
+      "loss": 2.7198,
+      "step": 1815
+    },
+    {
+      "epoch": 0.91,
+      "learning_rate": 4.6001055522763926e-05,
+      "loss": 2.855,
+      "step": 1820
+    },
+    {
+      "epoch": 0.92,
+      "learning_rate": 4.597963910185582e-05,
+      "loss": 2.4175,
+      "step": 1825
+    },
+    {
+      "epoch": 0.92,
+      "learning_rate": 4.595817049915441e-05,
+      "loss": 2.5444,
+      "step": 1830
+    },
+    {
+      "epoch": 0.92,
+      "learning_rate": 4.5936649768057646e-05,
+      "loss": 2.6893,
+      "step": 1835
+    },
+    {
+      "epoch": 0.92,
+      "learning_rate": 4.591507696209318e-05,
+      "loss": 2.6725,
+      "step": 1840
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 4.589345213491817e-05,
+      "loss": 2.834,
+      "step": 1845
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 4.587177534031914e-05,
+      "loss": 2.8259,
+      "step": 1850
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 4.585004663221188e-05,
+      "loss": 2.6146,
+      "step": 1855
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 4.582826606464134e-05,
+      "loss": 2.4673,
+      "step": 1860
+    },
+    {
+      "epoch": 0.94,
+      "learning_rate": 4.5806433691781416e-05,
+      "loss": 2.9178,
+      "step": 1865
+    },
+    {
+      "epoch": 0.94,
+      "learning_rate": 4.578454956793487e-05,
+      "loss": 2.9224,
+      "step": 1870
+    },
+    {
+      "epoch": 0.94,
+      "learning_rate": 4.576261374753318e-05,
+      "loss": 2.7987,
+      "step": 1875
+    },
+    {
+      "epoch": 0.94,
+      "learning_rate": 4.574062628513642e-05,
+      "loss": 2.3848,
+      "step": 1880
+    },
+    {
+      "epoch": 0.95,
+      "learning_rate": 4.57185872354331e-05,
+      "loss": 2.6054,
+      "step": 1885
+    },
+    {
+      "epoch": 0.95,
+      "learning_rate": 4.569649665324003e-05,
+      "loss": 2.6743,
+      "step": 1890
+    },
+    {
+      "epoch": 0.95,
+      "learning_rate": 4.567435459350222e-05,
+      "loss": 2.9375,
+      "step": 1895
+    },
+    {
+      "epoch": 0.95,
+      "learning_rate": 4.565216111129269e-05,
+      "loss": 2.9372,
+      "step": 1900
+    },
+    {
+      "epoch": 0.96,
+      "learning_rate": 4.562991626181239e-05,
+      "loss": 2.669,
+      "step": 1905
+    },
+    {
+      "epoch": 0.96,
+      "learning_rate": 4.560762010039001e-05,
+      "loss": 2.7644,
+      "step": 1910
+    },
+    {
+      "epoch": 0.96,
+      "learning_rate": 4.558527268248187e-05,
+      "loss": 2.6886,
+      "step": 1915
+    },
+    {
+      "epoch": 0.96,
+      "learning_rate": 4.55628740636718e-05,
+      "loss": 2.8618,
+      "step": 1920
+    },
+    {
+      "epoch": 0.97,
+      "learning_rate": 4.554042429967095e-05,
+      "loss": 2.8411,
+      "step": 1925
+    },
+    {
+      "epoch": 0.97,
+      "learning_rate": 4.55179234463177e-05,
+      "loss": 2.6047,
+      "step": 1930
+    },
+    {
+      "epoch": 0.97,
+      "learning_rate": 4.5495371559577496e-05,
+      "loss": 2.8675,
+      "step": 1935
+    },
+    {
+      "epoch": 0.97,
+      "learning_rate": 4.547276869554271e-05,
+      "loss": 2.751,
+      "step": 1940
+    },
+    {
+      "epoch": 0.98,
+      "learning_rate": 4.545011491043253e-05,
+      "loss": 2.8638,
+      "step": 1945
+    },
+    {
+      "epoch": 0.98,
+      "learning_rate": 4.5427410260592775e-05,
+      "loss": 2.5092,
+      "step": 1950
+    },
+    {
+      "epoch": 0.98,
+      "learning_rate": 4.540465480249579e-05,
+      "loss": 2.5059,
+      "step": 1955
+    },
+    {
+      "epoch": 0.98,
+      "learning_rate": 4.5381848592740285e-05,
+      "loss": 2.9433,
+      "step": 1960
+    },
+    {
+      "epoch": 0.99,
+      "learning_rate": 4.535899168805121e-05,
+      "loss": 2.6959,
+      "step": 1965
+    },
+    {
+      "epoch": 0.99,
+      "learning_rate": 4.533608414527961e-05,
+      "loss": 2.552,
+      "step": 1970
+    },
+    {
+      "epoch": 0.99,
+      "learning_rate": 4.5313126021402465e-05,
+      "loss": 2.7169,
+      "step": 1975
+    },
+    {
+      "epoch": 0.99,
+      "learning_rate": 4.529011737352258e-05,
+      "loss": 2.5672,
+      "step": 1980
+    },
+    {
+      "epoch": 1.0,
+      "learning_rate": 4.526705825886841e-05,
+      "loss": 2.6434,
+      "step": 1985
+    },
+    {
+      "epoch": 1.0,
+      "learning_rate": 4.5243948734793947e-05,
+      "loss": 2.8062,
+      "step": 1990
+    },
+    {
+      "epoch": 1.0,
+      "learning_rate": 4.5220788858778556e-05,
+      "loss": 2.6929,
+      "step": 1995
+    },
+    {
+      "epoch": 1.0,
+      "learning_rate": 4.519757868842684e-05,
+      "loss": 2.5837,
+      "step": 2000
+    },
+    {
+      "epoch": 1.01,
+      "learning_rate": 4.517431828146852e-05,
+      "loss": 2.821,
+      "step": 2005
+    },
+    {
+      "epoch": 1.01,
+      "learning_rate": 4.515100769575824e-05,
+      "loss": 2.7913,
+      "step": 2010
+    },
+    {
+      "epoch": 1.01,
+      "learning_rate": 4.512764698927545e-05,
+      "loss": 2.699,
+      "step": 2015
+    },
+    {
+      "epoch": 1.01,
+      "learning_rate": 4.5104236220124286e-05,
+      "loss": 2.6574,
+      "step": 2020
+    },
+    {
+      "epoch": 1.02,
+      "learning_rate": 4.508077544653338e-05,
+      "loss": 2.5909,
+      "step": 2025
+    },
+    {
+      "epoch": 1.02,
+      "learning_rate": 4.5057264726855765e-05,
+      "loss": 2.5695,
+      "step": 2030
+    },
+    {
+      "epoch": 1.02,
+      "learning_rate": 4.5033704119568675e-05,
+      "loss": 2.4937,
+      "step": 2035
+    },
+    {
+      "epoch": 1.02,
+      "learning_rate": 4.501009368327344e-05,
+      "loss": 2.7253,
+      "step": 2040
+    },
+    {
+      "epoch": 1.03,
+      "learning_rate": 4.4986433476695334e-05,
+      "loss": 2.8681,
+      "step": 2045
+    },
+    {
+      "epoch": 1.03,
+      "learning_rate": 4.496272355868341e-05,
+      "loss": 2.74,
+      "step": 2050
+    },
+    {
+      "epoch": 1.03,
+      "learning_rate": 4.4938963988210365e-05,
+      "loss": 2.7439,
+      "step": 2055
+    },
+    {
+      "epoch": 1.03,
+      "learning_rate": 4.491515482437242e-05,
+      "loss": 2.9539,
+      "step": 2060
+    },
+    {
+      "epoch": 1.04,
+      "learning_rate": 4.4891296126389104e-05,
+      "loss": 2.8165,
+      "step": 2065
+    },
+    {
+      "epoch": 1.04,
+      "learning_rate": 4.48673879536032e-05,
+      "loss": 2.601,
+      "step": 2070
+    },
+    {
+      "epoch": 1.04,
+      "learning_rate": 4.484343036548051e-05,
+      "loss": 2.5189,
+      "step": 2075
+    },
+    {
+      "epoch": 1.04,
+      "learning_rate": 4.481942342160976e-05,
+      "loss": 2.6276,
+      "step": 2080
+    },
+    {
+      "epoch": 1.05,
+      "learning_rate": 4.479536718170243e-05,
+      "loss": 2.5027,
+      "step": 2085
+    },
+    {
+      "epoch": 1.05,
+      "learning_rate": 4.477126170559262e-05,
+      "loss": 2.8654,
+      "step": 2090
+    },
+    {
+      "epoch": 1.05,
+      "learning_rate": 4.474710705323688e-05,
+      "loss": 2.8511,
+      "step": 2095
+    },
+    {
+      "epoch": 1.05,
+      "learning_rate": 4.47229032847141e-05,
+      "loss": 2.6129,
+      "step": 2100
+    },
+    {
+      "epoch": 1.06,
+      "learning_rate": 4.469865046022531e-05,
+      "loss": 2.8964,
+      "step": 2105
+    },
+    {
+      "epoch": 1.06,
+      "learning_rate": 4.4674348640093554e-05,
+      "loss": 2.7502,
+      "step": 2110
+    },
+    {
+      "epoch": 1.06,
+      "learning_rate": 4.4649997884763765e-05,
+      "loss": 2.591,
+      "step": 2115
+    },
+    {
+      "epoch": 1.06,
+      "learning_rate": 4.462559825480257e-05,
+      "loss": 2.7982,
+      "step": 2120
+    },
+    {
+      "epoch": 1.07,
+      "learning_rate": 4.460114981089815e-05,
+      "loss": 2.576,
+      "step": 2125
+    },
+    {
+      "epoch": 1.07,
+      "learning_rate": 4.457665261386014e-05,
+      "loss": 2.692,
+      "step": 2130
+    },
+    {
+      "epoch": 1.07,
+      "learning_rate": 4.455210672461938e-05,
+      "loss": 2.5818,
+      "step": 2135
+    },
+    {
+      "epoch": 1.07,
+      "learning_rate": 4.452751220422787e-05,
+      "loss": 2.6792,
+      "step": 2140
+    },
+    {
+      "epoch": 1.08,
+      "learning_rate": 4.450286911385856e-05,
+      "loss": 2.8352,
+      "step": 2145
+    },
+    {
+      "epoch": 1.08,
+      "learning_rate": 4.4478177514805166e-05,
+      "loss": 2.5787,
+      "step": 2150
+    },
+    {
+      "epoch": 1.08,
+      "learning_rate": 4.4453437468482103e-05,
+      "loss": 2.655,
+      "step": 2155
+    },
+    {
+      "epoch": 1.08,
+      "learning_rate": 4.442864903642428e-05,
+      "loss": 2.7664,
+      "step": 2160
+    },
+    {
+      "epoch": 1.09,
+      "learning_rate": 4.440381228028692e-05,
+      "loss": 2.7231,
+      "step": 2165
+    },
+    {
+      "epoch": 1.09,
+      "learning_rate": 4.437892726184548e-05,
+      "loss": 2.416,
+      "step": 2170
+    },
+    {
+      "epoch": 1.09,
+      "learning_rate": 4.4353994042995446e-05,
+      "loss": 2.4619,
+      "step": 2175
+    },
+    {
+      "epoch": 1.09,
+      "learning_rate": 4.4329012685752183e-05,
+      "loss": 2.7816,
+      "step": 2180
+    },
+    {
+      "epoch": 1.1,
+      "learning_rate": 4.430398325225078e-05,
+      "loss": 2.8678,
+      "step": 2185
+    },
+    {
+      "epoch": 1.1,
+      "learning_rate": 4.427890580474594e-05,
+      "loss": 2.6007,
+      "step": 2190
+    },
+    {
+      "epoch": 1.1,
+      "learning_rate": 4.4253780405611754e-05,
+      "loss": 2.8195,
+      "step": 2195
+    },
+    {
+      "epoch": 1.1,
+      "learning_rate": 4.42286071173416e-05,
+      "loss": 2.5117,
+      "step": 2200
+    },
+    {
+      "epoch": 1.11,
+      "learning_rate": 4.4203386002547956e-05,
+      "loss": 2.5527,
+      "step": 2205
+    },
+    {
+      "epoch": 1.11,
+      "learning_rate": 4.417811712396226e-05,
+      "loss": 2.662,
+      "step": 2210
+    },
+    {
+      "epoch": 1.11,
+      "learning_rate": 4.415280054443477e-05,
+      "loss": 2.7563,
+      "step": 2215
+    },
+    {
+      "epoch": 1.11,
+      "learning_rate": 4.4127436326934354e-05,
+      "loss": 2.5118,
+      "step": 2220
+    },
+    {
+      "epoch": 1.12,
+      "learning_rate": 4.41020245345484e-05,
+      "loss": 2.7208,
+      "step": 2225
+    },
+    {
+      "epoch": 1.12,
+      "learning_rate": 4.4076565230482607e-05,
+      "loss": 2.649,
+      "step": 2230
+    },
+    {
+      "epoch": 1.12,
+      "learning_rate": 4.4051058478060856e-05,
+      "loss": 2.652,
+      "step": 2235
+    },
+    {
+      "epoch": 1.12,
+      "learning_rate": 4.402550434072505e-05,
+      "loss": 2.6885,
+      "step": 2240
+    },
+    {
+      "epoch": 1.13,
+      "learning_rate": 4.3999902882034935e-05,
+      "loss": 2.8545,
+      "step": 2245
+    },
+    {
+      "epoch": 1.13,
+      "learning_rate": 4.397425416566797e-05,
+      "loss": 2.9435,
+      "step": 2250
+    },
+    {
+      "epoch": 1.13,
+      "learning_rate": 4.3948558255419146e-05,
+      "loss": 2.6555,
+      "step": 2255
+    },
+    {
+      "epoch": 1.13,
+      "learning_rate": 4.392281521520085e-05,
+      "loss": 2.6108,
+      "step": 2260
+    },
+    {
+      "epoch": 1.14,
+      "learning_rate": 4.389702510904269e-05,
+      "loss": 2.6256,
+      "step": 2265
+    },
+    {
+      "epoch": 1.14,
+      "learning_rate": 4.387118800109133e-05,
+      "loss": 2.7996,
+      "step": 2270
+    },
+    {
+      "epoch": 1.14,
+      "learning_rate": 4.384530395561035e-05,
+      "loss": 2.6649,
+      "step": 2275
+    },
+    {
+      "epoch": 1.14,
+      "learning_rate": 4.381937303698006e-05,
+      "loss": 3.0073,
+      "step": 2280
+    },
+    {
+      "epoch": 1.15,
+      "learning_rate": 4.379339530969738e-05,
+      "loss": 2.8493,
+      "step": 2285
+    },
+    {
+      "epoch": 1.15,
+      "learning_rate": 4.3767370838375635e-05,
+      "loss": 2.5572,
+      "step": 2290
+    },
+    {
+      "epoch": 1.15,
+      "learning_rate": 4.374129968774443e-05,
+      "loss": 2.3837,
+      "step": 2295
+    },
+    {
+      "epoch": 1.15,
+      "learning_rate": 4.371518192264946e-05,
+      "loss": 2.4604,
+      "step": 2300
+    },
+    {
+      "epoch": 1.16,
+      "learning_rate": 4.3689017608052374e-05,
+      "loss": 2.8866,
+      "step": 2305
+    },
+    {
+      "epoch": 1.16,
+      "learning_rate": 4.3662806809030585e-05,
+      "loss": 2.8943,
+      "step": 2310
+    },
+    {
+      "epoch": 1.16,
+      "learning_rate": 4.3636549590777144e-05,
+      "loss": 2.7629,
+      "step": 2315
+    },
+    {
+      "epoch": 1.16,
+      "learning_rate": 4.361024601860054e-05,
+      "loss": 2.8629,
+      "step": 2320
+    },
+    {
+      "epoch": 1.17,
+      "learning_rate": 4.3583896157924574e-05,
+      "loss": 2.7952,
+      "step": 2325
+    },
+    {
+      "epoch": 1.17,
+      "learning_rate": 4.355750007428817e-05,
+      "loss": 2.2057,
+      "step": 2330
+    },
+    {
+      "epoch": 1.17,
+      "learning_rate": 4.3531057833345216e-05,
+      "loss": 2.9143,
+      "step": 2335
+    },
+    {
+      "epoch": 1.17,
+      "learning_rate": 4.3504569500864424e-05,
+      "loss": 2.5354,
+      "step": 2340
+    },
+    {
+      "epoch": 1.18,
+      "learning_rate": 4.347803514272911e-05,
+      "loss": 2.7451,
+      "step": 2345
+    },
+    {
+      "epoch": 1.18,
+      "learning_rate": 4.3451454824937113e-05,
+      "loss": 3.0827,
+      "step": 2350
+    },
+    {
+      "epoch": 1.18,
+      "learning_rate": 4.3424828613600555e-05,
+      "loss": 2.5842,
+      "step": 2355
+    },
+    {
+      "epoch": 1.18,
+      "learning_rate": 4.339815657494572e-05,
+      "loss": 2.4492,
+      "step": 2360
+    },
+    {
+      "epoch": 1.19,
+      "learning_rate": 4.3371438775312865e-05,
+      "loss": 2.5067,
+      "step": 2365
+    },
+    {
+      "epoch": 1.19,
+      "learning_rate": 4.334467528115608e-05,
+      "loss": 2.5902,
+      "step": 2370
+    },
+    {
+      "epoch": 1.19,
+      "learning_rate": 4.33178661590431e-05,
+      "loss": 2.8618,
+      "step": 2375
+    },
+    {
+      "epoch": 1.19,
+      "learning_rate": 4.329101147565515e-05,
+      "loss": 2.6831,
+      "step": 2380
+    },
+    {
+      "epoch": 1.2,
+      "learning_rate": 4.3264111297786794e-05,
+      "loss": 2.7531,
+      "step": 2385
+    },
+    {
+      "epoch": 1.2,
+      "learning_rate": 4.323716569234572e-05,
+      "loss": 2.819,
+      "step": 2390
+    },
+    {
+      "epoch": 1.2,
+      "learning_rate": 4.321017472635263e-05,
+      "loss": 2.6319,
+      "step": 2395
+    },
+    {
+      "epoch": 1.2,
+      "learning_rate": 4.318313846694105e-05,
+      "loss": 2.3078,
+      "step": 2400
+    },
+    {
+      "epoch": 1.21,
+      "learning_rate": 4.315605698135714e-05,
+      "loss": 2.7962,
+      "step": 2405
+    },
+    {
+      "epoch": 1.21,
+      "learning_rate": 4.312893033695958e-05,
+      "loss": 2.8479,
+      "step": 2410
+    },
+    {
+      "epoch": 1.21,
+      "learning_rate": 4.310175860121936e-05,
+      "loss": 2.6289,
+      "step": 2415
+    },
+    {
+      "epoch": 1.21,
+      "learning_rate": 4.30745418417196e-05,
+      "loss": 2.7003,
+      "step": 2420
+    },
+    {
+      "epoch": 1.22,
+      "learning_rate": 4.304728012615543e-05,
+      "loss": 2.4947,
+      "step": 2425
+    },
+    {
+      "epoch": 1.22,
+      "learning_rate": 4.3019973522333815e-05,
+      "loss": 2.6911,
+      "step": 2430
+    },
+    {
+      "epoch": 1.22,
+      "learning_rate": 4.2992622098173334e-05,
+      "loss": 2.6931,
+      "step": 2435
+    },
+    {
+      "epoch": 1.22,
+      "learning_rate": 4.296522592170406e-05,
+      "loss": 2.774,
+      "step": 2440
+    },
+    {
+      "epoch": 1.23,
+      "learning_rate": 4.293778506106737e-05,
+      "loss": 2.759,
+      "step": 2445
+    },
+    {
+      "epoch": 1.23,
+      "learning_rate": 4.29102995845158e-05,
+      "loss": 2.5543,
+      "step": 2450
+    },
+    {
+      "epoch": 1.23,
+      "learning_rate": 4.288276956041284e-05,
+      "loss": 2.7702,
+      "step": 2455
+    },
+    {
+      "epoch": 1.23,
+      "learning_rate": 4.285519505723278e-05,
+      "loss": 2.835,
+      "step": 2460
+    },
+    {
+      "epoch": 1.24,
+      "learning_rate": 4.282757614356055e-05,
+      "loss": 2.6516,
+      "step": 2465
+    },
+    {
+      "epoch": 1.24,
+      "learning_rate": 4.2799912888091544e-05,
+      "loss": 2.7392,
+      "step": 2470
+    },
+    {
+      "epoch": 1.24,
+      "learning_rate": 4.277220535963143e-05,
+      "loss": 2.6522,
+      "step": 2475
+    },
+    {
+      "epoch": 1.24,
+      "learning_rate": 4.274445362709601e-05,
+      "loss": 2.6514,
+      "step": 2480
+    },
+    {
+      "epoch": 1.25,
+      "learning_rate": 4.271665775951104e-05,
+      "loss": 2.7507,
+      "step": 2485
+    },
+    {
+      "epoch": 1.25,
+      "learning_rate": 4.2688817826012005e-05,
+      "loss": 3.0499,
+      "step": 2490
+    },
+    {
+      "epoch": 1.25,
+      "learning_rate": 4.2660933895844055e-05,
+      "loss": 2.6927,
+      "step": 2495
+    },
+    {
+      "epoch": 1.25,
+      "learning_rate": 4.2633006038361736e-05,
+      "loss": 2.8456,
+      "step": 2500
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 9960,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "total_flos": 1.3252546968069734e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-2500/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:939ee45e9035366dc4952c5158e7d3a0d3426acdbfb5014d53ad1e260b19a19f
+size 4475

checkpoint-3000/README.md

@@ -0,0 +1,204 @@
+---
+library_name: peft
+base_model: ../chatglm3-6b-base
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+
+### Framework versions
+
+- PEFT 0.8.2

checkpoint-3000/adapter_config.json

@@ -0,0 +1,26 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "../chatglm3-6b-base",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "query_key_value"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_rslora": false
+}