|
05/19/2023 09:26:41 PM Seed: 555 |
|
05/19/2023 09:26:41 PM unet attention_head_dim: 8 |
|
05/19/2023 09:26:41 PM Inferred yaml: v1-inference.yaml, attn: sd1, prediction_type: epsilon |
|
05/19/2023 09:26:50 PM Enabled xformers |
|
05/19/2023 09:26:51 PM Overriding LR from optimizer config with main config/cli LR setting: 1.5e-07 |
|
05/19/2023 09:26:51 PM * Loaded optimizer args from optimizer.json * |
|
05/19/2023 09:26:51 PM [36m * Training Text and Unet *[0m |
|
05/19/2023 09:26:51 PM [36m * Optimizer: AdamW8bit *[0m |
|
05/19/2023 09:26:51 PM [36m unet lr: 1.5e-07, text encoder lr: 1.5e-07, betas: [0.9, 0.999], epsilon: 1e-08, weight_decay: 0.01 *[0m |
|
05/19/2023 09:26:51 PM * DLMA resolution 512, buckets: [[512, 512], [576, 448], [448, 576], [640, 384], [384, 640], [768, 320], [320, 768], [896, 256], [256, 896], [1024, 256], [256, 1024]] |
|
05/19/2023 09:26:51 PM Preloading images... |
|
05/19/2023 09:27:50 PM * Removed 1629 images from the training set to use for validation |
|
05/19/2023 09:27:50 PM * DLMA initialized with 1629 images. |
|
05/19/2023 09:27:50 PM ** Dataset 'val': 410 batches, num_images: 1640, batch_size: 4 |
|
05/19/2023 09:27:50 PM * DLMA initialized with 1385 images. |
|
05/19/2023 09:27:50 PM ** Dataset 'stabilize-train': 348 batches, num_images: 1392, batch_size: 4 |
|
05/19/2023 09:27:50 PM * DLMA initialized with 9227 images. |
|
05/19/2023 09:27:51 PM ** Dataset 'train': 2311 batches, num_images: 9244, batch_size: 4 |
|
05/19/2023 09:27:51 PM Pretraining GPU Memory: 5971 / 24576 MB |
|
05/19/2023 09:27:51 PM saving ckpts every 1000000000.0 minutes |
|
05/19/2023 09:27:51 PM saving ckpts every 25 epochs |
|
05/19/2023 09:27:51 PM unet device: cuda:0, precision: torch.float32, training: True |
|
05/19/2023 09:27:51 PM text_encoder device: cuda:0, precision: torch.float32, training: True |
|
05/19/2023 09:27:51 PM vae device: cuda:0, precision: torch.float16, training: False |
|
05/19/2023 09:27:51 PM scheduler: <class 'diffusers.schedulers.scheduling_ddpm.DDPMScheduler'> |
|
05/19/2023 09:27:51 PM [32mProject name: [0m[92mvodka_v3[0m |
|
05/19/2023 09:27:51 PM [32mgrad_accum: [0m[92m1[0m |
|
05/19/2023 09:27:51 PM [32mbatch_size: [0m[92m4[0m |
|
05/19/2023 09:27:51 PM [32mepoch_len: [92m2311[0m |
|
05/19/2023 09:27:51 PM Grad scaler enabled: True (amp mode) |
|
05/20/2023 10:37:37 AM Saving model, 25 epochs at step 57775 |
|
05/20/2023 10:37:37 AM * Saving diffusers model to logs\vodka_v3_20230519-212641/ckpts/vodka_v3-ep25-gs57775 |
|
05/20/2023 10:37:50 AM * Saving SD model to .\vodka_v3-ep25-gs57775.ckpt |
|
05/20/2023 11:50:02 PM Saving model, 25 epochs at step 115550 |
|
05/20/2023 11:50:02 PM * Saving diffusers model to logs\vodka_v3_20230519-212641/ckpts/vodka_v3-ep50-gs115550 |
|
05/20/2023 11:50:20 PM * Saving SD model to .\vodka_v3-ep50-gs115550.ckpt |
|
05/21/2023 09:22:18 AM Validation loss shows diverging |
|
05/21/2023 09:53:39 AM Validation loss shows diverging |
|
05/21/2023 01:03:47 PM Saving model, 25 epochs at step 173325 |
|
05/21/2023 01:03:47 PM * Saving diffusers model to logs\vodka_v3_20230519-212641/ckpts/vodka_v3-ep75-gs173325 |
|
05/21/2023 01:03:59 PM * Saving SD model to .\vodka_v3-ep75-gs173325.ckpt |
|
05/21/2023 07:23:34 PM Validation loss shows diverging |
|
05/21/2023 08:26:50 PM Validation loss shows diverging |
|
05/22/2023 01:14:39 AM Validation loss shows diverging |
|
05/22/2023 01:46:05 AM Validation loss shows diverging |
|
05/22/2023 02:17:23 AM Validation loss shows diverging |
|
05/22/2023 02:17:23 AM * Saving diffusers model to logs\vodka_v3_20230519-212641/ckpts/last-vodka_v3-ep99-gs231100 |
|
05/22/2023 02:17:32 AM * Saving SD model to .\last-vodka_v3-ep99-gs231100.ckpt |
|
05/22/2023 02:17:43 AM [36mTraining complete[0m |
|
05/22/2023 02:17:43 AM Total training time took 3169.86 minutes, total steps: 231100 |
|
05/22/2023 02:17:43 AM Average epoch time: 28.11 minutes |
|
05/22/2023 02:17:43 AM [97m ***************************[0m |
|
05/22/2023 02:17:43 AM [97m **** Finished training ****[0m |
|
05/22/2023 02:17:43 AM [97m ***************************[0m |
|
|
