05/19/2023 09:26:41 PM  Seed: 555
05/19/2023 09:26:41 PM  unet attention_head_dim: 8
05/19/2023 09:26:41 PM Inferred yaml: v1-inference.yaml, attn: sd1, prediction_type: epsilon
05/19/2023 09:26:50 PM Enabled xformers
05/19/2023 09:26:51 PM Overriding LR from optimizer config with main config/cli LR setting: 1.5e-07
05/19/2023 09:26:51 PM  * Loaded optimizer args from optimizer.json *
05/19/2023 09:26:51 PM  * Training Text and Unet *
05/19/2023 09:26:51 PM  * Optimizer: AdamW8bit *
05/19/2023 09:26:51 PM     unet lr: 1.5e-07, text encoder lr: 1.5e-07, betas: [0.9, 0.999], epsilon: 1e-08, weight_decay: 0.01 *
05/19/2023 09:26:51 PM * DLMA resolution 512, buckets: [[512, 512], [576, 448], [448, 576], [640, 384], [384, 640], [768, 320], [320, 768], [896, 256], [256, 896], [1024, 256], [256, 1024]]
05/19/2023 09:26:51 PM  Preloading images...
05/19/2023 09:27:50 PM  * Removed 1629 images from the training set to use for validation
05/19/2023 09:27:50 PM  * DLMA initialized with 1629 images.
05/19/2023 09:27:50 PM  ** Dataset 'val': 410 batches, num_images: 1640, batch_size: 4
05/19/2023 09:27:50 PM  * DLMA initialized with 1385 images.
05/19/2023 09:27:50 PM  ** Dataset 'stabilize-train': 348 batches, num_images: 1392, batch_size: 4
05/19/2023 09:27:50 PM  * DLMA initialized with 9227 images.
05/19/2023 09:27:51 PM  ** Dataset 'train': 2311 batches, num_images: 9244, batch_size: 4
05/19/2023 09:27:51 PM  Pretraining GPU Memory: 5971 / 24576 MB
05/19/2023 09:27:51 PM  saving ckpts every 1000000000.0 minutes
05/19/2023 09:27:51 PM  saving ckpts every 25 epochs
05/19/2023 09:27:51 PM  unet device: cuda:0, precision: torch.float32, training: True
05/19/2023 09:27:51 PM  text_encoder device: cuda:0, precision: torch.float32, training: True
05/19/2023 09:27:51 PM  vae device: cuda:0, precision: torch.float16, training: False
05/19/2023 09:27:51 PM  scheduler: <class 'diffusers.schedulers.scheduling_ddpm.DDPMScheduler'>
05/19/2023 09:27:51 PM  Project name: vodka_v3
05/19/2023 09:27:51 PM  grad_accum: 1
05/19/2023 09:27:51 PM  batch_size: 4
05/19/2023 09:27:51 PM  epoch_len: 2311
05/19/2023 09:27:51 PM  Grad scaler enabled: True (amp mode)
05/20/2023 10:37:37 AM  Saving model, 25 epochs at step 57775
05/20/2023 10:37:37 AM  * Saving diffusers model to logs\vodka_v3_20230519-212641/ckpts/vodka_v3-ep25-gs57775
05/20/2023 10:37:50 AM  * Saving SD model to .\vodka_v3-ep25-gs57775.ckpt
05/20/2023 11:50:02 PM  Saving model, 25 epochs at step 115550
05/20/2023 11:50:02 PM  * Saving diffusers model to logs\vodka_v3_20230519-212641/ckpts/vodka_v3-ep50-gs115550
05/20/2023 11:50:20 PM  * Saving SD model to .\vodka_v3-ep50-gs115550.ckpt
05/21/2023 09:22:18 AM Validation loss shows diverging
05/21/2023 09:53:39 AM Validation loss shows diverging
05/21/2023 01:03:47 PM  Saving model, 25 epochs at step 173325
05/21/2023 01:03:47 PM  * Saving diffusers model to logs\vodka_v3_20230519-212641/ckpts/vodka_v3-ep75-gs173325
05/21/2023 01:03:59 PM  * Saving SD model to .\vodka_v3-ep75-gs173325.ckpt
05/21/2023 07:23:34 PM Validation loss shows diverging
05/21/2023 08:26:50 PM Validation loss shows diverging
05/22/2023 01:14:39 AM Validation loss shows diverging
05/22/2023 01:46:05 AM Validation loss shows diverging
05/22/2023 02:17:23 AM Validation loss shows diverging
05/22/2023 02:17:23 AM  * Saving diffusers model to logs\vodka_v3_20230519-212641/ckpts/last-vodka_v3-ep99-gs231100
05/22/2023 02:17:32 AM  * Saving SD model to .\last-vodka_v3-ep99-gs231100.ckpt
05/22/2023 02:17:43 AM Training complete
05/22/2023 02:17:43 AM Total training time took 3169.86 minutes, total steps: 231100
05/22/2023 02:17:43 AM Average epoch time: 28.11 minutes
05/22/2023 02:17:43 AM  ***************************
05/22/2023 02:17:43 AM  **** Finished training ****
05/22/2023 02:17:43 AM  ***************************