Data/tyc/train_20240806_160411.log

2024-08-06 16:04:11.177 | INFO     | __main__:run:116 - Loading configuration from config 0
2024-08-06 16:04:11.178 | INFO     | __main__:run:116 - Loading configuration from config localhost
2024-08-06 16:04:11.178 | INFO     | __main__:run:116 - Loading configuration from config 10086
2024-08-06 16:04:11.178 | INFO     | __main__:run:116 - Loading configuration from config 0
2024-08-06 16:04:11.178 | INFO     | __main__:run:116 - Loading configuration from config 1
2024-08-06 16:04:11.178 | INFO     | __main__:run:118 - Loading environment variables 
MASTER_ADDR: localhost,
MASTER_PORT: 10086,
WORLD_SIZE: 1,
RANK: 0,
LOCAL_RANK: 0
2024-08-06 16:04:11.764 | INFO     | default_style:save_styles_by_dirs:54 - At least 2 subdirectories are required for generating style vectors with respect to them, found 0.
2024-08-06 16:04:11.764 | INFO     | default_style:save_styles_by_dirs:57 - Generating only neutral style vector instead.
2024-08-06 16:04:11.770 | INFO     | default_style:save_neutral_vector:28 - Saved mean style vector to model_assets/tyc
2024-08-06 16:04:11.770 | INFO     | default_style:save_neutral_vector:36 - Saved style config to model_assets/tyc/config.json
2024-08-06 16:04:11.811 | WARNING  | style_bert_vits2.models.utils:check_git_hash:247 - /home/sacha/projects/Style-Bert-VITS2/style_bert_vits2/models/utils is not a git repository, therefore hash value comparison will be ignored.
2024-08-06 16:04:11.812 | INFO     | data_utils:_filter:69 - Init dataset...
2024-08-06 16:04:11.814 | INFO     | data_utils:_filter:84 - skipped: 0, total: 101
2024-08-06 16:04:11.815 | INFO     | data_utils:__init__:348 - Bucket info: [6, 16, 22, 24, 20, 8, 4, 6]
2024-08-06 16:04:11.815 | INFO     | data_utils:_filter:69 - Init dataset...
2024-08-06 16:04:11.815 | INFO     | data_utils:_filter:84 - skipped: 0, total: 0
2024-08-06 16:04:11.815 | INFO     | __main__:run:273 - Using noise scaled MAS for VITS2
2024-08-06 16:04:11.816 | INFO     | __main__:run:281 - Using duration discriminator for VITS2
2024-08-06 16:04:13.262 | WARNING  | style_bert_vits2.models.utils.safetensors:load_safetensors:42 - Missing key: enc_p.style_proj.weight
2024-08-06 16:04:13.262 | WARNING  | style_bert_vits2.models.utils.safetensors:load_safetensors:42 - Missing key: enc_p.style_proj.bias
2024-08-06 16:04:13.262 | WARNING  | style_bert_vits2.models.utils.safetensors:load_safetensors:42 - Missing key: emb_g.weight
2024-08-06 16:04:13.263 | INFO     | style_bert_vits2.models.utils.safetensors:load_safetensors:48 - Loaded 'Data/tyc/models/G_0.safetensors'
2024-08-06 16:04:13.286 | INFO     | style_bert_vits2.models.utils.safetensors:load_safetensors:48 - Loaded 'Data/tyc/models/D_0.safetensors'
2024-08-06 16:04:13.289 | INFO     | style_bert_vits2.models.utils.safetensors:load_safetensors:48 - Loaded 'Data/tyc/models/DUR_0.safetensors'
2024-08-06 16:04:13.289 | INFO     | __main__:run:432 - Loaded the pretrained models.
2024-08-06 16:04:13.289 | INFO     | __main__:run:467 - Start training.
2024-08-06 16:10:50.174 | INFO     | __main__:evaluate:890 - Evaluating ...
2024-08-06 16:10:50.176 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 19 to Data/tyc/models/G_1000.pth
2024-08-06 16:10:50.793 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 19 to Data/tyc/models/D_1000.pth
2024-08-06 16:10:51.137 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 19 to Data/tyc/models/DUR_1000.pth
2024-08-06 16:10:51.153 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/G_0.safetensors
2024-08-06 16:10:51.160 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/D_0.safetensors
2024-08-06 16:10:51.160 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/DUR_0.safetensors
2024-08-06 16:10:51.164 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/tyc/tyc_e19_s1000.safetensors
2024-08-06 16:17:23.974 | INFO     | __main__:evaluate:890 - Evaluating ...
2024-08-06 16:17:23.976 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 38 to Data/tyc/models/G_2000.pth
2024-08-06 16:17:24.621 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 38 to Data/tyc/models/D_2000.pth
2024-08-06 16:17:24.960 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 38 to Data/tyc/models/DUR_2000.pth
2024-08-06 16:17:24.992 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/G_1000.pth
2024-08-06 16:17:25.010 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/D_1000.pth
2024-08-06 16:17:25.011 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/DUR_1000.pth
2024-08-06 16:17:25.015 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/tyc/tyc_e38_s2000.safetensors
2024-08-06 16:23:54.852 | INFO     | __main__:evaluate:890 - Evaluating ...
2024-08-06 16:23:54.854 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 57 to Data/tyc/models/G_3000.pth
2024-08-06 16:23:55.421 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 57 to Data/tyc/models/D_3000.pth
2024-08-06 16:23:55.749 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 57 to Data/tyc/models/DUR_3000.pth
2024-08-06 16:23:55.780 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/G_2000.pth
2024-08-06 16:23:55.799 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/D_2000.pth
2024-08-06 16:23:55.799 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/DUR_2000.pth
2024-08-06 16:23:55.803 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/tyc/tyc_e57_s3000.safetensors
2024-08-06 16:31:01.800 | INFO     | __main__:evaluate:890 - Evaluating ...
2024-08-06 16:31:01.802 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 76 to Data/tyc/models/G_4000.pth
2024-08-06 16:31:02.429 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 76 to Data/tyc/models/D_4000.pth
2024-08-06 16:31:02.770 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 76 to Data/tyc/models/DUR_4000.pth
2024-08-06 16:31:02.802 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/G_3000.pth
2024-08-06 16:31:02.820 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/D_3000.pth
2024-08-06 16:31:02.821 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/DUR_3000.pth
2024-08-06 16:31:02.825 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/tyc/tyc_e76_s4000.safetensors
2024-08-06 16:48:05.949 | INFO     | __main__:evaluate:890 - Evaluating ...
2024-08-06 16:48:05.951 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 95 to Data/tyc/models/G_5000.pth
2024-08-06 16:48:06.601 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 95 to Data/tyc/models/D_5000.pth
2024-08-06 16:48:06.941 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 95 to Data/tyc/models/DUR_5000.pth
2024-08-06 16:48:06.973 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/G_4000.pth
2024-08-06 16:48:06.992 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/D_4000.pth
2024-08-06 16:48:06.992 | INFO     | style_bert_vits2.models.utils.checkpoints:del_info:173 - Free up space by deleting ckpt Data/tyc/models/DUR_4000.pth
2024-08-06 16:48:06.996 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/tyc/tyc_e95_s5000.safetensors
2024-08-06 16:50:04.961 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 100 to Data/tyc/models/G_5300.pth
2024-08-06 16:50:05.587 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 100 to Data/tyc/models/D_5300.pth
2024-08-06 16:50:05.925 | INFO     | style_bert_vits2.models.utils.checkpoints:save_checkpoint:110 - Saving model and optimizer state at iteration 100 to Data/tyc/models/DUR_5300.pth
2024-08-06 16:50:05.937 | INFO     | style_bert_vits2.models.utils.safetensors:save_safetensors:89 - Saved safetensors to model_assets/tyc/tyc_e100_s5300.safetensors