2024-09-12 14:39:59.110 | INFO     | __main__:run:117 - Loading configuration from config 0
2024-09-12 14:39:59.110 | INFO     | __main__:run:117 - Loading configuration from config localhost
2024-09-12 14:39:59.110 | INFO     | __main__:run:117 - Loading configuration from config 10086
2024-09-12 14:39:59.110 | INFO     | __main__:run:117 - Loading configuration from config 0
2024-09-12 14:39:59.110 | INFO     | __main__:run:117 - Loading configuration from config 1
2024-09-12 14:39:59.110 | INFO     | __main__:run:119 - Loading environment variables 
MASTER_ADDR: localhost,
MASTER_PORT: 10086,
WORLD_SIZE: 1,
RANK: 0,
LOCAL_RANK: 0
2024-09-12 14:40:00.763 | INFO     | default_style:save_styles_by_dirs:54 - At least 2 subdirectories are required for generating style vectors with respect to them, found 0.
2024-09-12 14:40:00.763 | INFO     | default_style:save_styles_by_dirs:57 - Generating only neutral style vector instead.
2024-09-12 14:40:00.784 | INFO     | default_style:save_neutral_vector:28 - Saved mean style vector to model_assets/amitaro_runrun
2024-09-12 14:40:00.784 | INFO     | default_style:save_neutral_vector:36 - Saved style config to model_assets/amitaro_runrun/config.json
2024-09-12 14:40:00.828 | WARNING  | style_bert_vits2.models.utils:check_git_hash:247 - /home/sacha/projects/Style-Bert-VITS2/style_bert_vits2/models/utils is not a git repository, therefore hash value comparison will be ignored.
2024-09-12 14:40:00.831 | INFO     | data_utils:_filter:69 - Init dataset...
2024-09-12 14:40:00.835 | INFO     | data_utils:_filter:84 - skipped: 0, total: 367
2024-09-12 14:40:00.835 | INFO     | data_utils:__init__:348 - Bucket info: [176, 100, 56, 20, 12, 8, 8, 4]
2024-09-12 14:40:00.836 | INFO     | data_utils:_filter:69 - Init dataset...
2024-09-12 14:40:00.836 | INFO     | data_utils:_filter:84 - skipped: 0, total: 0
2024-09-12 14:40:00.836 | INFO     | __main__:run:274 - Using noise scaled MAS for VITS2
2024-09-12 14:40:02.339 | WARNING  | style_bert_vits2.models.utils.safetensors:load_safetensors:42 - Missing key: enc_p.style_proj.weight
2024-09-12 14:40:02.339 | WARNING  | style_bert_vits2.models.utils.safetensors:load_safetensors:42 - Missing key: enc_p.style_proj.bias
2024-09-12 14:40:02.339 | WARNING  | style_bert_vits2.models.utils.safetensors:load_safetensors:42 - Missing key: emb_g.weight
2024-09-12 14:40:02.339 | INFO     | style_bert_vits2.models.utils.safetensors:load_safetensors:48 - Loaded 'Data/amitaro_runrun/models/G_0.safetensors'
2024-09-12 14:40:02.370 | INFO     | style_bert_vits2.models.utils.safetensors:load_safetensors:48 - Loaded 'Data/amitaro_runrun/models/D_0.safetensors'
2024-09-12 14:40:02.376 | INFO     | style_bert_vits2.models.utils.safetensors:load_safetensors:48 - Loaded 'Data/amitaro_runrun/models/WD_0.safetensors'
2024-09-12 14:40:02.376 | INFO     | __main__:run:492 - Loaded the pretrained models.
2024-09-12 14:40:02.664 | INFO     | __main__:run:540 - Start training.
2024-09-12 14:49:09.021 | INFO     | __main__:evaluate:1040 - Evaluating ...
2024-09-12 14:49:09.023 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 11 to Data/amitaro_runrun/models/G_1000.pth
2024-09-12 14:49:09.996 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 11 to Data/amitaro_runrun/models/D_1000.pth
2024-09-12 14:49:10.468 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 11 to Data/amitaro_runrun/models/WD_1000.pth
2024-09-12 14:49:10.507 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/G_0.safetensors
2024-09-12 14:49:10.522 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/D_0.safetensors
2024-09-12 14:49:10.523 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/WD_0.safetensors
2024-09-12 14:49:10.527 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/amitaro_runrun/amitaro_runrun_e11_s1000.safetensors
2024-09-12 14:57:49.434 | INFO     | __main__:evaluate:1040 - Evaluating ...
2024-09-12 14:57:49.436 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 21 to Data/amitaro_runrun/models/G_2000.pth
2024-09-12 14:57:50.478 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 21 to Data/amitaro_runrun/models/D_2000.pth
2024-09-12 14:57:50.950 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 21 to Data/amitaro_runrun/models/WD_2000.pth
2024-09-12 14:57:51.033 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/G_1000.pth
2024-09-12 14:57:51.077 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/D_1000.pth
2024-09-12 14:57:51.079 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/WD_1000.pth
2024-09-12 14:57:51.083 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/amitaro_runrun/amitaro_runrun_e21_s2000.safetensors
2024-09-12 15:06:31.870 | INFO     | __main__:evaluate:1040 - Evaluating ...
2024-09-12 15:06:31.872 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 32 to Data/amitaro_runrun/models/G_3000.pth
2024-09-12 15:06:32.798 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 32 to Data/amitaro_runrun/models/D_3000.pth
2024-09-12 15:06:33.268 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 32 to Data/amitaro_runrun/models/WD_3000.pth
2024-09-12 15:06:33.352 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/G_2000.pth
2024-09-12 15:06:33.395 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/D_2000.pth
2024-09-12 15:06:33.397 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/WD_2000.pth
2024-09-12 15:06:33.401 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/amitaro_runrun/amitaro_runrun_e32_s3000.safetensors
2024-09-12 15:15:17.629 | INFO     | __main__:evaluate:1040 - Evaluating ...
2024-09-12 15:15:17.631 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 42 to Data/amitaro_runrun/models/G_4000.pth
2024-09-12 15:15:18.529 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 42 to Data/amitaro_runrun/models/D_4000.pth
2024-09-12 15:15:19.001 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 42 to Data/amitaro_runrun/models/WD_4000.pth
2024-09-12 15:15:19.085 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/G_3000.pth
2024-09-12 15:15:19.129 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/D_3000.pth
2024-09-12 15:15:19.131 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/WD_3000.pth
2024-09-12 15:15:19.135 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/amitaro_runrun/amitaro_runrun_e42_s4000.safetensors
2024-09-12 15:24:00.253 | INFO     | __main__:evaluate:1040 - Evaluating ...
2024-09-12 15:24:00.255 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 53 to Data/amitaro_runrun/models/G_5000.pth
2024-09-12 15:24:01.163 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 53 to Data/amitaro_runrun/models/D_5000.pth
2024-09-12 15:24:01.632 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 53 to Data/amitaro_runrun/models/WD_5000.pth
2024-09-12 15:24:01.716 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/G_4000.pth
2024-09-12 15:24:01.759 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/D_4000.pth
2024-09-12 15:24:01.761 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/WD_4000.pth
2024-09-12 15:24:01.765 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/amitaro_runrun/amitaro_runrun_e53_s5000.safetensors
2024-09-12 15:32:39.960 | INFO     | __main__:evaluate:1040 - Evaluating ...
2024-09-12 15:32:39.962 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 63 to Data/amitaro_runrun/models/G_6000.pth
2024-09-12 15:32:40.867 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 63 to Data/amitaro_runrun/models/D_6000.pth
2024-09-12 15:32:41.351 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 63 to Data/amitaro_runrun/models/WD_6000.pth
2024-09-12 15:32:41.434 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/G_5000.pth
2024-09-12 15:32:41.478 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/D_5000.pth
2024-09-12 15:32:41.479 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/WD_5000.pth
2024-09-12 15:32:41.484 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/amitaro_runrun/amitaro_runrun_e63_s6000.safetensors
2024-09-12 15:41:23.713 | INFO     | __main__:evaluate:1040 - Evaluating ...
2024-09-12 15:41:23.715 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 73 to Data/amitaro_runrun/models/G_7000.pth
2024-09-12 15:41:24.602 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 73 to Data/amitaro_runrun/models/D_7000.pth
2024-09-12 15:41:25.074 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 73 to Data/amitaro_runrun/models/WD_7000.pth
2024-09-12 15:41:25.159 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/G_6000.pth
2024-09-12 15:41:25.203 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/D_6000.pth
2024-09-12 15:41:25.204 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/WD_6000.pth
2024-09-12 15:41:25.209 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/amitaro_runrun/amitaro_runrun_e73_s7000.safetensors
2024-09-12 15:49:55.394 | INFO     | __main__:evaluate:1040 - Evaluating ...
2024-09-12 15:49:55.396 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 84 to Data/amitaro_runrun/models/G_8000.pth
2024-09-12 15:49:56.318 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 84 to Data/amitaro_runrun/models/D_8000.pth
2024-09-12 15:49:56.793 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 84 to Data/amitaro_runrun/models/WD_8000.pth
2024-09-12 15:49:56.879 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/G_7000.pth
2024-09-12 15:49:56.923 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/D_7000.pth
2024-09-12 15:49:56.924 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/WD_7000.pth
2024-09-12 15:49:56.929 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/amitaro_runrun/amitaro_runrun_e84_s8000.safetensors
2024-09-12 15:58:28.962 | INFO     | __main__:evaluate:1040 - Evaluating ...
2024-09-12 15:58:28.964 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 94 to Data/amitaro_runrun/models/G_9000.pth
2024-09-12 15:58:29.982 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 94 to Data/amitaro_runrun/models/D_9000.pth
2024-09-12 15:58:30.461 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 94 to Data/amitaro_runrun/models/WD_9000.pth
2024-09-12 15:58:30.548 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/G_8000.pth
2024-09-12 15:58:30.591 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/D_8000.pth
2024-09-12 15:58:30.593 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/amitaro_runrun/models/WD_8000.pth
2024-09-12 15:58:30.597 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/amitaro_runrun/amitaro_runrun_e94_s9000.safetensors
2024-09-12 16:03:30.282 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 100 to Data/amitaro_runrun/models/G_9600.pth
2024-09-12 16:03:31.176 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 100 to Data/amitaro_runrun/models/D_9600.pth
2024-09-12 16:03:31.644 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 100 to Data/amitaro_runrun/models/WD_9600.pth
2024-09-12 16:03:31.663 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/amitaro_runrun/amitaro_runrun_e100_s9600.safetensors