running_log.txt

05/15/2024 20:34:24 - INFO - transformers.tokenization_utils_base - loading file tokenizer.model from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/tokenizer.model

05/15/2024 20:34:24 - INFO - transformers.tokenization_utils_base - loading file tokenizer.json from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/tokenizer.json

05/15/2024 20:34:24 - INFO - transformers.tokenization_utils_base - loading file added_tokens.json from cache at None

05/15/2024 20:34:24 - INFO - transformers.tokenization_utils_base - loading file special_tokens_map.json from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/special_tokens_map.json

05/15/2024 20:34:24 - INFO - transformers.tokenization_utils_base - loading file tokenizer_config.json from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/tokenizer_config.json

05/15/2024 20:34:24 - WARNING - transformers.models.llama.tokenization_llama_fast - You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers

05/15/2024 20:34:25 - INFO - llmtuner.data.template - Add pad token: </s>

05/15/2024 20:34:25 - INFO - llmtuner.data.loader - Loading dataset svjack/emoji_add_instruction_zh...

05/15/2024 20:34:35 - INFO - transformers.configuration_utils - loading configuration file config.json from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/config.json

05/15/2024 20:34:35 - INFO - transformers.configuration_utils - Model config MistralConfig {
  "_name_or_path": "alpindale/Mistral-7B-v0.2-hf",
  "architectures": [
    "MistralForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 32768,
  "model_type": "mistral",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "rms_norm_eps": 1e-05,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.2",
  "use_cache": true,
  "vocab_size": 32000
}


05/15/2024 20:34:35 - INFO - llmtuner.model.utils.quantization - Quantizing model to 4 bit.

05/15/2024 20:34:38 - INFO - transformers.modeling_utils - loading weights file model.safetensors from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/model.safetensors.index.json

05/15/2024 20:35:08 - INFO - transformers.modeling_utils - Instantiating MistralForCausalLM model under default dtype torch.float16.

05/15/2024 20:35:08 - INFO - transformers.generation.configuration_utils - Generate config GenerationConfig {
  "bos_token_id": 1,
  "eos_token_id": 2
}


05/15/2024 20:35:18 - INFO - transformers.modeling_utils - All model checkpoint weights were used when initializing MistralForCausalLM.


05/15/2024 20:35:18 - INFO - transformers.modeling_utils - All the weights of MistralForCausalLM were initialized from the model checkpoint at alpindale/Mistral-7B-v0.2-hf.
If your task is similar to the task the model of the checkpoint was trained on, you can already use MistralForCausalLM for predictions without further training.

05/15/2024 20:35:19 - INFO - transformers.generation.configuration_utils - loading configuration file generation_config.json from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/generation_config.json

05/15/2024 20:35:19 - INFO - transformers.generation.configuration_utils - Generate config GenerationConfig {
  "bos_token_id": 1,
  "eos_token_id": 2
}


05/15/2024 20:35:20 - INFO - llmtuner.model.utils.checkpointing - Gradient checkpointing enabled.

05/15/2024 20:35:20 - INFO - llmtuner.model.utils.attention - Using torch SDPA for faster training and inference.

05/15/2024 20:35:20 - INFO - llmtuner.model.adapter - Fine-tuning method: LoRA

05/15/2024 20:35:20 - INFO - llmtuner.model.loader - trainable params: 3407872 || all params: 7245139968 || trainable%: 0.0470

05/15/2024 20:35:20 - WARNING - accelerate.utils.other - Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.

05/15/2024 20:35:20 - INFO - transformers.trainer - Using auto half precision backend

05/15/2024 20:35:20 - INFO - transformers.trainer - ***** Running training *****

05/15/2024 20:35:20 - INFO - transformers.trainer -   Num examples = 2,449

05/15/2024 20:35:20 - INFO - transformers.trainer -   Num Epochs = 3

05/15/2024 20:35:20 - INFO - transformers.trainer -   Instantaneous batch size per device = 2

05/15/2024 20:35:20 - INFO - transformers.trainer -   Total train batch size (w. parallel, distributed & accumulation) = 16

05/15/2024 20:35:20 - INFO - transformers.trainer -   Gradient Accumulation steps = 8

05/15/2024 20:35:20 - INFO - transformers.trainer -   Total optimization steps = 459

05/15/2024 20:35:20 - INFO - transformers.trainer -   Number of trainable parameters = 3,407,872

05/15/2024 20:37:03 - INFO - llmtuner.extras.callbacks - {'loss': 0.1654, 'learning_rate': 4.9985e-05, 'epoch': 0.03}

05/15/2024 20:38:46 - INFO - llmtuner.extras.callbacks - {'loss': 0.1369, 'learning_rate': 4.9941e-05, 'epoch': 0.07}

05/15/2024 20:40:27 - INFO - llmtuner.extras.callbacks - {'loss': 0.1259, 'learning_rate': 4.9868e-05, 'epoch': 0.10}

05/15/2024 20:42:08 - INFO - llmtuner.extras.callbacks - {'loss': 0.1041, 'learning_rate': 4.9766e-05, 'epoch': 0.13}

05/15/2024 20:43:48 - INFO - llmtuner.extras.callbacks - {'loss': 0.1004, 'learning_rate': 4.9635e-05, 'epoch': 0.16}

05/15/2024 20:45:31 - INFO - llmtuner.extras.callbacks - {'loss': 0.0941, 'learning_rate': 4.9475e-05, 'epoch': 0.20}

05/15/2024 20:47:10 - INFO - llmtuner.extras.callbacks - {'loss': 0.0867, 'learning_rate': 4.9286e-05, 'epoch': 0.23}

05/15/2024 20:48:49 - INFO - llmtuner.extras.callbacks - {'loss': 0.0853, 'learning_rate': 4.9069e-05, 'epoch': 0.26}

05/15/2024 20:50:29 - INFO - llmtuner.extras.callbacks - {'loss': 0.0863, 'learning_rate': 4.8824e-05, 'epoch': 0.29}

05/15/2024 20:52:07 - INFO - llmtuner.extras.callbacks - {'loss': 0.0874, 'learning_rate': 4.8550e-05, 'epoch': 0.33}

05/15/2024 20:53:53 - INFO - llmtuner.extras.callbacks - {'loss': 0.0733, 'learning_rate': 4.8249e-05, 'epoch': 0.36}

05/15/2024 20:55:32 - INFO - llmtuner.extras.callbacks - {'loss': 0.0719, 'learning_rate': 4.7921e-05, 'epoch': 0.39}

05/15/2024 20:57:12 - INFO - llmtuner.extras.callbacks - {'loss': 0.0824, 'learning_rate': 4.7566e-05, 'epoch': 0.42}

05/15/2024 20:58:51 - INFO - llmtuner.extras.callbacks - {'loss': 0.0731, 'learning_rate': 4.7185e-05, 'epoch': 0.46}

05/15/2024 21:00:33 - INFO - llmtuner.extras.callbacks - {'loss': 0.0725, 'learning_rate': 4.6778e-05, 'epoch': 0.49}

05/15/2024 21:02:13 - INFO - llmtuner.extras.callbacks - {'loss': 0.0694, 'learning_rate': 4.6345e-05, 'epoch': 0.52}

05/15/2024 21:03:49 - INFO - llmtuner.extras.callbacks - {'loss': 0.0720, 'learning_rate': 4.5887e-05, 'epoch': 0.56}

05/15/2024 21:05:32 - INFO - llmtuner.extras.callbacks - {'loss': 0.0646, 'learning_rate': 4.5405e-05, 'epoch': 0.59}

05/15/2024 21:07:13 - INFO - llmtuner.extras.callbacks - {'loss': 0.0732, 'learning_rate': 4.4899e-05, 'epoch': 0.62}

05/15/2024 21:08:54 - INFO - llmtuner.extras.callbacks - {'loss': 0.0680, 'learning_rate': 4.4369e-05, 'epoch': 0.65}

05/15/2024 21:08:54 - INFO - transformers.trainer - Saving model checkpoint to saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-100

05/15/2024 21:08:55 - INFO - transformers.configuration_utils - loading configuration file config.json from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/config.json

05/15/2024 21:08:55 - INFO - transformers.configuration_utils - Model config MistralConfig {
  "architectures": [
    "MistralForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 32768,
  "model_type": "mistral",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "rms_norm_eps": 1e-05,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.2",
  "use_cache": true,
  "vocab_size": 32000
}


05/15/2024 21:08:55 - INFO - transformers.tokenization_utils_base - tokenizer config file saved in saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-100/tokenizer_config.json

05/15/2024 21:08:55 - INFO - transformers.tokenization_utils_base - Special tokens file saved in saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-100/special_tokens_map.json

05/15/2024 21:10:38 - INFO - llmtuner.extras.callbacks - {'loss': 0.0749, 'learning_rate': 4.3817e-05, 'epoch': 0.69}

05/15/2024 21:12:21 - INFO - llmtuner.extras.callbacks - {'loss': 0.0652, 'learning_rate': 4.3243e-05, 'epoch': 0.72}

05/15/2024 21:14:02 - INFO - llmtuner.extras.callbacks - {'loss': 0.0562, 'learning_rate': 4.2647e-05, 'epoch': 0.75}

05/15/2024 21:15:42 - INFO - llmtuner.extras.callbacks - {'loss': 0.0602, 'learning_rate': 4.2031e-05, 'epoch': 0.78}

05/15/2024 21:17:20 - INFO - llmtuner.extras.callbacks - {'loss': 0.0639, 'learning_rate': 4.1395e-05, 'epoch': 0.82}

05/15/2024 21:19:01 - INFO - llmtuner.extras.callbacks - {'loss': 0.0651, 'learning_rate': 4.0740e-05, 'epoch': 0.85}

05/15/2024 21:20:41 - INFO - llmtuner.extras.callbacks - {'loss': 0.0676, 'learning_rate': 4.0066e-05, 'epoch': 0.88}

05/15/2024 21:22:21 - INFO - llmtuner.extras.callbacks - {'loss': 0.0669, 'learning_rate': 3.9374e-05, 'epoch': 0.91}

05/15/2024 21:24:04 - INFO - llmtuner.extras.callbacks - {'loss': 0.0631, 'learning_rate': 3.8666e-05, 'epoch': 0.95}

05/15/2024 21:25:46 - INFO - llmtuner.extras.callbacks - {'loss': 0.0649, 'learning_rate': 3.7942e-05, 'epoch': 0.98}

05/15/2024 21:27:28 - INFO - llmtuner.extras.callbacks - {'loss': 0.0590, 'learning_rate': 3.7202e-05, 'epoch': 1.01}

05/15/2024 21:29:05 - INFO - llmtuner.extras.callbacks - {'loss': 0.0620, 'learning_rate': 3.6449e-05, 'epoch': 1.04}

05/15/2024 21:30:44 - INFO - llmtuner.extras.callbacks - {'loss': 0.0565, 'learning_rate': 3.5682e-05, 'epoch': 1.08}

05/15/2024 21:32:28 - INFO - llmtuner.extras.callbacks - {'loss': 0.0575, 'learning_rate': 3.4902e-05, 'epoch': 1.11}

05/15/2024 21:34:12 - INFO - llmtuner.extras.callbacks - {'loss': 0.0589, 'learning_rate': 3.4111e-05, 'epoch': 1.14}

05/15/2024 21:35:50 - INFO - llmtuner.extras.callbacks - {'loss': 0.0611, 'learning_rate': 3.3309e-05, 'epoch': 1.18}

05/15/2024 21:37:31 - INFO - llmtuner.extras.callbacks - {'loss': 0.0620, 'learning_rate': 3.2497e-05, 'epoch': 1.21}

05/15/2024 21:39:10 - INFO - llmtuner.extras.callbacks - {'loss': 0.0616, 'learning_rate': 3.1677e-05, 'epoch': 1.24}

05/15/2024 21:40:50 - INFO - llmtuner.extras.callbacks - {'loss': 0.0586, 'learning_rate': 3.0849e-05, 'epoch': 1.27}

05/15/2024 21:42:32 - INFO - llmtuner.extras.callbacks - {'loss': 0.0578, 'learning_rate': 3.0014e-05, 'epoch': 1.31}

05/15/2024 21:42:32 - INFO - transformers.trainer - Saving model checkpoint to saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-200

05/15/2024 21:42:35 - INFO - transformers.configuration_utils - loading configuration file config.json from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/config.json

05/15/2024 21:42:35 - INFO - transformers.configuration_utils - Model config MistralConfig {
  "architectures": [
    "MistralForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 32768,
  "model_type": "mistral",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "rms_norm_eps": 1e-05,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.2",
  "use_cache": true,
  "vocab_size": 32000
}


05/15/2024 21:42:35 - INFO - transformers.tokenization_utils_base - tokenizer config file saved in saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-200/tokenizer_config.json

05/15/2024 21:42:35 - INFO - transformers.tokenization_utils_base - Special tokens file saved in saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-200/special_tokens_map.json

05/15/2024 21:44:16 - INFO - llmtuner.extras.callbacks - {'loss': 0.0547, 'learning_rate': 2.9173e-05, 'epoch': 1.34}

05/15/2024 21:45:58 - INFO - llmtuner.extras.callbacks - {'loss': 0.0581, 'learning_rate': 2.8327e-05, 'epoch': 1.37}

05/15/2024 21:47:43 - INFO - llmtuner.extras.callbacks - {'loss': 0.0535, 'learning_rate': 2.7477e-05, 'epoch': 1.40}

05/15/2024 21:49:23 - INFO - llmtuner.extras.callbacks - {'loss': 0.0666, 'learning_rate': 2.6624e-05, 'epoch': 1.44}

05/15/2024 21:51:05 - INFO - llmtuner.extras.callbacks - {'loss': 0.0530, 'learning_rate': 2.5770e-05, 'epoch': 1.47}

05/15/2024 21:52:40 - INFO - llmtuner.extras.callbacks - {'loss': 0.0602, 'learning_rate': 2.4914e-05, 'epoch': 1.50}

05/15/2024 21:54:21 - INFO - llmtuner.extras.callbacks - {'loss': 0.0573, 'learning_rate': 2.4059e-05, 'epoch': 1.53}

05/15/2024 21:56:02 - INFO - llmtuner.extras.callbacks - {'loss': 0.0544, 'learning_rate': 2.3205e-05, 'epoch': 1.57}

05/15/2024 21:57:44 - INFO - llmtuner.extras.callbacks - {'loss': 0.0552, 'learning_rate': 2.2353e-05, 'epoch': 1.60}

05/15/2024 21:59:24 - INFO - llmtuner.extras.callbacks - {'loss': 0.0543, 'learning_rate': 2.1504e-05, 'epoch': 1.63}

05/15/2024 22:01:06 - INFO - llmtuner.extras.callbacks - {'loss': 0.0481, 'learning_rate': 2.0659e-05, 'epoch': 1.67}

05/15/2024 22:02:47 - INFO - llmtuner.extras.callbacks - {'loss': 0.0520, 'learning_rate': 1.9819e-05, 'epoch': 1.70}

05/15/2024 22:04:30 - INFO - llmtuner.extras.callbacks - {'loss': 0.0516, 'learning_rate': 1.8985e-05, 'epoch': 1.73}

05/15/2024 22:06:09 - INFO - llmtuner.extras.callbacks - {'loss': 0.0559, 'learning_rate': 1.8158e-05, 'epoch': 1.76}

05/15/2024 22:07:52 - INFO - llmtuner.extras.callbacks - {'loss': 0.0554, 'learning_rate': 1.7340e-05, 'epoch': 1.80}

05/15/2024 22:09:33 - INFO - llmtuner.extras.callbacks - {'loss': 0.0576, 'learning_rate': 1.6530e-05, 'epoch': 1.83}

05/15/2024 22:11:14 - INFO - llmtuner.extras.callbacks - {'loss': 0.0546, 'learning_rate': 1.5730e-05, 'epoch': 1.86}

05/15/2024 22:12:55 - INFO - llmtuner.extras.callbacks - {'loss': 0.0539, 'learning_rate': 1.4941e-05, 'epoch': 1.89}

05/15/2024 22:14:34 - INFO - llmtuner.extras.callbacks - {'loss': 0.0537, 'learning_rate': 1.4164e-05, 'epoch': 1.93}

05/15/2024 22:16:18 - INFO - llmtuner.extras.callbacks - {'loss': 0.0493, 'learning_rate': 1.3399e-05, 'epoch': 1.96}

05/15/2024 22:16:18 - INFO - transformers.trainer - Saving model checkpoint to saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-300

05/15/2024 22:16:20 - INFO - transformers.configuration_utils - loading configuration file config.json from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/config.json

05/15/2024 22:16:20 - INFO - transformers.configuration_utils - Model config MistralConfig {
  "architectures": [
    "MistralForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 32768,
  "model_type": "mistral",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "rms_norm_eps": 1e-05,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.2",
  "use_cache": true,
  "vocab_size": 32000
}


05/15/2024 22:16:20 - INFO - transformers.tokenization_utils_base - tokenizer config file saved in saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-300/tokenizer_config.json

05/15/2024 22:16:20 - INFO - transformers.tokenization_utils_base - Special tokens file saved in saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-300/special_tokens_map.json

05/15/2024 22:18:03 - INFO - llmtuner.extras.callbacks - {'loss': 0.0514, 'learning_rate': 1.2648e-05, 'epoch': 1.99}

05/15/2024 22:19:45 - INFO - llmtuner.extras.callbacks - {'loss': 0.0526, 'learning_rate': 1.1912e-05, 'epoch': 2.02}

05/15/2024 22:21:27 - INFO - llmtuner.extras.callbacks - {'loss': 0.0521, 'learning_rate': 1.1191e-05, 'epoch': 2.06}

05/15/2024 22:23:12 - INFO - llmtuner.extras.callbacks - {'loss': 0.0506, 'learning_rate': 1.0486e-05, 'epoch': 2.09}

05/15/2024 22:24:54 - INFO - llmtuner.extras.callbacks - {'loss': 0.0549, 'learning_rate': 9.7979e-06, 'epoch': 2.12}

05/15/2024 22:26:33 - INFO - llmtuner.extras.callbacks - {'loss': 0.0522, 'learning_rate': 9.1278e-06, 'epoch': 2.16}

05/15/2024 22:28:13 - INFO - llmtuner.extras.callbacks - {'loss': 0.0499, 'learning_rate': 8.4762e-06, 'epoch': 2.19}

05/15/2024 22:29:52 - INFO - llmtuner.extras.callbacks - {'loss': 0.0556, 'learning_rate': 7.8440e-06, 'epoch': 2.22}

05/15/2024 22:31:31 - INFO - llmtuner.extras.callbacks - {'loss': 0.0514, 'learning_rate': 7.2318e-06, 'epoch': 2.25}

05/15/2024 22:33:11 - INFO - llmtuner.extras.callbacks - {'loss': 0.0539, 'learning_rate': 6.6405e-06, 'epoch': 2.29}

05/15/2024 22:34:51 - INFO - llmtuner.extras.callbacks - {'loss': 0.0506, 'learning_rate': 6.0707e-06, 'epoch': 2.32}

05/15/2024 22:36:33 - INFO - llmtuner.extras.callbacks - {'loss': 0.0494, 'learning_rate': 5.5230e-06, 'epoch': 2.35}

05/15/2024 22:38:11 - INFO - llmtuner.extras.callbacks - {'loss': 0.0502, 'learning_rate': 4.9981e-06, 'epoch': 2.38}

05/15/2024 22:39:52 - INFO - llmtuner.extras.callbacks - {'loss': 0.0503, 'learning_rate': 4.4967e-06, 'epoch': 2.42}

05/15/2024 22:41:39 - INFO - llmtuner.extras.callbacks - {'loss': 0.0471, 'learning_rate': 4.0193e-06, 'epoch': 2.45}

05/15/2024 22:43:19 - INFO - llmtuner.extras.callbacks - {'loss': 0.0510, 'learning_rate': 3.5664e-06, 'epoch': 2.48}

05/15/2024 22:45:03 - INFO - llmtuner.extras.callbacks - {'loss': 0.0468, 'learning_rate': 3.1387e-06, 'epoch': 2.51}

05/15/2024 22:46:40 - INFO - llmtuner.extras.callbacks - {'loss': 0.0480, 'learning_rate': 2.7365e-06, 'epoch': 2.55}

05/15/2024 22:48:23 - INFO - llmtuner.extras.callbacks - {'loss': 0.0508, 'learning_rate': 2.3604e-06, 'epoch': 2.58}

05/15/2024 22:50:05 - INFO - llmtuner.extras.callbacks - {'loss': 0.0495, 'learning_rate': 2.0108e-06, 'epoch': 2.61}

05/15/2024 22:50:05 - INFO - transformers.trainer - Saving model checkpoint to saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-400

05/15/2024 22:50:06 - INFO - transformers.configuration_utils - loading configuration file config.json from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/config.json

05/15/2024 22:50:06 - INFO - transformers.configuration_utils - Model config MistralConfig {
  "architectures": [
    "MistralForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 32768,
  "model_type": "mistral",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "rms_norm_eps": 1e-05,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.2",
  "use_cache": true,
  "vocab_size": 32000
}


05/15/2024 22:50:06 - INFO - transformers.tokenization_utils_base - tokenizer config file saved in saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-400/tokenizer_config.json

05/15/2024 22:50:06 - INFO - transformers.tokenization_utils_base - Special tokens file saved in saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/checkpoint-400/special_tokens_map.json

05/15/2024 22:51:48 - INFO - llmtuner.extras.callbacks - {'loss': 0.0511, 'learning_rate': 1.6882e-06, 'epoch': 2.64}

05/15/2024 22:53:29 - INFO - llmtuner.extras.callbacks - {'loss': 0.0486, 'learning_rate': 1.3928e-06, 'epoch': 2.68}

05/15/2024 22:55:13 - INFO - llmtuner.extras.callbacks - {'loss': 0.0552, 'learning_rate': 1.1251e-06, 'epoch': 2.71}

05/15/2024 22:56:49 - INFO - llmtuner.extras.callbacks - {'loss': 0.0523, 'learning_rate': 8.8539e-07, 'epoch': 2.74}

05/15/2024 22:58:28 - INFO - llmtuner.extras.callbacks - {'loss': 0.0517, 'learning_rate': 6.7388e-07, 'epoch': 2.78}

05/15/2024 23:00:09 - INFO - llmtuner.extras.callbacks - {'loss': 0.0515, 'learning_rate': 4.9086e-07, 'epoch': 2.81}

05/15/2024 23:01:53 - INFO - llmtuner.extras.callbacks - {'loss': 0.0528, 'learning_rate': 3.3653e-07, 'epoch': 2.84}

05/15/2024 23:03:38 - INFO - llmtuner.extras.callbacks - {'loss': 0.0459, 'learning_rate': 2.1110e-07, 'epoch': 2.87}

05/15/2024 23:05:21 - INFO - llmtuner.extras.callbacks - {'loss': 0.0556, 'learning_rate': 1.1469e-07, 'epoch': 2.91}

05/15/2024 23:06:58 - INFO - llmtuner.extras.callbacks - {'loss': 0.0520, 'learning_rate': 4.7417e-08, 'epoch': 2.94}

05/15/2024 23:08:36 - INFO - llmtuner.extras.callbacks - {'loss': 0.0532, 'learning_rate': 9.3687e-09, 'epoch': 2.97}

05/15/2024 23:09:56 - INFO - transformers.trainer - 

Training completed. Do not forget to share your model on huggingface.co/models =)



05/15/2024 23:09:56 - INFO - transformers.trainer - Saving model checkpoint to saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30

05/15/2024 23:09:57 - INFO - transformers.configuration_utils - loading configuration file config.json from cache at /home/featurize/.cache/huggingface/hub/models--alpindale--Mistral-7B-v0.2-hf/snapshots/2c3e624962b1a3f3fbf52e15969565caa7bc064a/config.json

05/15/2024 23:09:57 - INFO - transformers.configuration_utils - Model config MistralConfig {
  "architectures": [
    "MistralForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 32768,
  "model_type": "mistral",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "rms_norm_eps": 1e-05,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.2",
  "use_cache": true,
  "vocab_size": 32000
}


05/15/2024 23:09:57 - INFO - transformers.tokenization_utils_base - tokenizer config file saved in saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/tokenizer_config.json

05/15/2024 23:09:57 - INFO - transformers.tokenization_utils_base - Special tokens file saved in saves/Mistral-7B-v0.2/lora/train_2024-05-15-20-33-30/special_tokens_map.json

05/15/2024 23:09:58 - INFO - transformers.modelcard - Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}