| 2024-01-15 21:40:43,073 44k INFO {'train': {'log_interval': 200, 'eval_interval': 200, 'seed': 1234, 'epochs': 10000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 6, 'fp16_run': False, 'half_type': 'fp16', 'lr_decay': 0.999875, 'segment_size': 10240, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0, 'use_sr': True, 'max_speclen': 512, 'port': '8001', 'keep_ckpts': 20, 'all_in_mem': False, 'vol_aug': True}, 'data': {'training_files': 'filelists/train.txt', 'validation_files': 'filelists/val.txt', 'max_wav_value': 32768.0, 'sampling_rate': 44100, 'filter_length': 2048, 'hop_length': 512, 'win_length': 2048, 'n_mel_channels': 80, 'mel_fmin': 0.0, 'mel_fmax': 22050, 'unit_interpolate_mode': 'nearest'}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4, 4], 'n_layers_q': 3, 'n_layers_trans_flow': 3, 'n_flow_layer': 4, 'use_spectral_norm': False, 'gin_channels': 768, 'ssl_dim': 768, 'n_speakers': 1, 'vocoder_name': 'nsf-hifigan', 'speech_encoder': 'vec768l12', 'speaker_embedding': False, 'vol_embedding': True, 'use_depthwise_conv': False, 'flow_share_parameter': False, 'use_automatic_f0_prediction': True, 'use_transformer_flow': False}, 'spk': {'yiji': 0}, 'model_dir': './logs/44k'} | |
| 2024-01-15 21:40:45,994 44k INFO emb_g.weight is not in the checkpoint | |
| 2024-01-15 21:40:46,073 44k INFO Loaded checkpoint './logs/44k/G_0.pth' (iteration 0) | |
| 2024-01-15 21:40:46,237 44k INFO Loaded checkpoint './logs/44k/D_0.pth' (iteration 0) | |
| 2024-01-15 21:41:15,645 44k INFO ====> Epoch: 1, cost 32.57 s | |
| 2024-01-15 21:41:29,711 44k INFO ====> Epoch: 2, cost 14.07 s | |
| 2024-01-15 21:41:43,649 44k INFO ====> Epoch: 3, cost 13.94 s | |
| 2024-01-15 21:41:57,303 44k INFO ====> Epoch: 4, cost 13.65 s | |
| 2024-01-15 21:42:11,213 44k INFO ====> Epoch: 5, cost 13.91 s | |
| 2024-01-15 21:42:25,692 44k INFO ====> Epoch: 6, cost 14.48 s | |
| 2024-01-15 21:42:39,431 44k INFO ====> Epoch: 7, cost 13.74 s | |
| 2024-01-15 21:42:53,265 44k INFO ====> Epoch: 8, cost 13.83 s | |
| 2024-01-15 21:43:07,426 44k INFO ====> Epoch: 9, cost 14.16 s | |
| 2024-01-15 21:43:21,037 44k INFO ====> Epoch: 10, cost 13.61 s | |
| 2024-01-15 21:43:35,060 44k INFO ====> Epoch: 11, cost 14.02 s | |
| 2024-01-15 21:43:48,921 44k INFO ====> Epoch: 12, cost 13.86 s | |
| 2024-01-15 21:44:02,530 44k INFO ====> Epoch: 13, cost 13.61 s | |
| 2024-01-15 21:44:16,756 44k INFO ====> Epoch: 14, cost 14.23 s | |
| 2024-01-15 21:44:30,689 44k INFO ====> Epoch: 15, cost 13.93 s | |
| 2024-01-15 21:44:40,170 44k INFO Train Epoch: 16 [31%] | |
| 2024-01-15 21:44:40,171 44k INFO Losses: [2.4577407836914062, 2.466566562652588, 5.2889723777771, 13.412308692932129, 0.6947054862976074], step: 200, lr: 9.981266397366609e-05, reference_loss: 24.320293426513672 | |
| 2024-01-15 21:44:51,693 44k INFO Saving model and optimizer state at iteration 16 to ./logs/44k/G_200.pth | |
| 2024-01-15 21:44:53,408 44k INFO Saving model and optimizer state at iteration 16 to ./logs/44k/D_200.pth | |
| 2024-01-15 21:44:58,685 44k INFO ====> Epoch: 16, cost 28.00 s | |
| 2024-01-15 21:45:12,580 44k INFO ====> Epoch: 17, cost 13.90 s | |
| 2024-01-15 21:45:26,568 44k INFO ====> Epoch: 18, cost 13.99 s | |
| 2024-01-15 21:45:40,969 44k INFO ====> Epoch: 19, cost 14.40 s | |
| 2024-01-15 21:45:55,067 44k INFO ====> Epoch: 20, cost 14.10 s | |
| 2024-01-15 21:46:08,671 44k INFO ====> Epoch: 21, cost 13.60 s | |
| 2024-01-15 21:46:22,416 44k INFO ====> Epoch: 22, cost 13.74 s | |
| 2024-01-15 21:46:36,717 44k INFO ====> Epoch: 23, cost 14.30 s | |
| 2024-01-15 21:46:51,134 44k INFO ====> Epoch: 24, cost 14.42 s | |
| 2024-01-15 21:47:05,251 44k INFO ====> Epoch: 25, cost 14.12 s | |
| 2024-01-15 21:47:19,427 44k INFO ====> Epoch: 26, cost 14.18 s | |
| 2024-01-15 21:47:33,880 44k INFO ====> Epoch: 27, cost 14.45 s | |
| 2024-01-15 21:47:48,657 44k INFO ====> Epoch: 28, cost 14.78 s | |
| 2024-01-15 21:48:02,296 44k INFO ====> Epoch: 29, cost 13.64 s | |
| 2024-01-15 21:48:15,854 44k INFO ====> Epoch: 30, cost 13.56 s | |
| 2024-01-15 21:48:27,203 44k INFO Train Epoch: 31 [69%] | |
| 2024-01-15 21:48:27,203 44k INFO Losses: [2.5092668533325195, 2.1289803981781006, 7.070366859436035, 20.873977661132812, 1.5376778841018677], step: 400, lr: 9.962567889519979e-05, reference_loss: 34.120269775390625 | |
| 2024-01-15 21:48:36,359 44k INFO Saving model and optimizer state at iteration 31 to ./logs/44k/G_400.pth | |
| 2024-01-15 21:48:38,373 44k INFO Saving model and optimizer state at iteration 31 to ./logs/44k/D_400.pth | |
| 2024-01-15 21:48:41,438 44k INFO ====> Epoch: 31, cost 25.58 s | |
| 2024-01-15 21:48:55,320 44k INFO ====> Epoch: 32, cost 13.88 s | |
| 2024-01-15 21:49:09,270 44k INFO ====> Epoch: 33, cost 13.95 s | |
| 2024-01-15 21:49:23,352 44k INFO ====> Epoch: 34, cost 14.08 s | |
| 2024-01-15 21:49:37,372 44k INFO ====> Epoch: 35, cost 14.02 s | |
| 2024-01-15 21:49:51,659 44k INFO ====> Epoch: 36, cost 14.29 s | |
| 2024-01-15 21:50:05,591 44k INFO ====> Epoch: 37, cost 13.93 s | |
| 2024-01-15 21:50:19,491 44k INFO ====> Epoch: 38, cost 13.90 s | |
| 2024-01-15 21:50:33,482 44k INFO ====> Epoch: 39, cost 13.99 s | |
| 2024-01-15 21:50:47,156 44k INFO ====> Epoch: 40, cost 13.67 s | |
| 2024-01-15 21:51:01,027 44k INFO ====> Epoch: 41, cost 13.87 s | |
| 2024-01-15 21:51:15,182 44k INFO ====> Epoch: 42, cost 14.16 s | |
| 2024-01-15 21:51:28,978 44k INFO ====> Epoch: 43, cost 13.80 s | |
| 2024-01-15 21:51:43,102 44k INFO ====> Epoch: 44, cost 14.12 s | |
| 2024-01-15 21:51:57,171 44k INFO ====> Epoch: 45, cost 14.07 s | |
| 2024-01-15 21:52:11,350 44k INFO ====> Epoch: 46, cost 14.18 s | |
| 2024-01-15 21:52:19,415 44k INFO Train Epoch: 47 [8%] | |
| 2024-01-15 21:52:19,416 44k INFO Losses: [2.616657018661499, 2.1187267303466797, 5.753948211669922, 17.533784866333008, 0.9128997325897217], step: 600, lr: 9.942661422663591e-05, reference_loss: 28.936016082763672 | |
| 2024-01-15 21:52:28,203 44k INFO Saving model and optimizer state at iteration 47 to ./logs/44k/G_600.pth | |
| 2024-01-15 21:52:29,851 44k INFO Saving model and optimizer state at iteration 47 to ./logs/44k/D_600.pth | |
| 2024-01-15 21:52:36,033 44k INFO ====> Epoch: 47, cost 24.68 s | |
| 2024-01-15 21:52:50,024 44k INFO ====> Epoch: 48, cost 13.99 s | |
| 2024-01-15 21:53:03,804 44k INFO ====> Epoch: 49, cost 13.78 s | |
| 2024-01-15 21:53:18,151 44k INFO ====> Epoch: 50, cost 14.35 s | |
| 2024-01-15 21:53:31,932 44k INFO ====> Epoch: 51, cost 13.78 s | |
| 2024-01-15 21:53:45,977 44k INFO ====> Epoch: 52, cost 14.04 s | |
| 2024-01-15 21:53:59,729 44k INFO ====> Epoch: 53, cost 13.75 s | |
| 2024-01-15 21:54:14,003 44k INFO ====> Epoch: 54, cost 14.27 s | |
| 2024-01-15 21:54:28,105 44k INFO ====> Epoch: 55, cost 14.10 s | |
| 2024-01-15 21:54:41,905 44k INFO ====> Epoch: 56, cost 13.80 s | |
| 2024-01-15 21:54:55,809 44k INFO ====> Epoch: 57, cost 13.90 s | |
| 2024-01-15 21:55:09,985 44k INFO ====> Epoch: 58, cost 14.18 s | |
| 2024-01-15 21:55:23,979 44k INFO ====> Epoch: 59, cost 13.99 s | |
| 2024-01-15 21:55:38,012 44k INFO ====> Epoch: 60, cost 14.03 s | |
| 2024-01-15 21:55:51,514 44k INFO ====> Epoch: 61, cost 13.50 s | |
| 2024-01-15 21:56:01,671 44k INFO Train Epoch: 62 [46%] | |
| 2024-01-15 21:56:01,672 44k INFO Losses: [2.6842474937438965, 1.9786149263381958, 5.013638496398926, 16.283981323242188, 0.6270778179168701], step: 800, lr: 9.924035235842533e-05, reference_loss: 26.58755874633789 | |
| 2024-01-15 21:56:10,610 44k INFO Saving model and optimizer state at iteration 62 to ./logs/44k/G_800.pth | |
| 2024-01-15 21:56:12,624 44k INFO Saving model and optimizer state at iteration 62 to ./logs/44k/D_800.pth | |
| 2024-01-15 21:56:16,993 44k INFO ====> Epoch: 62, cost 25.48 s | |
| 2024-01-15 21:56:30,532 44k INFO ====> Epoch: 63, cost 13.54 s | |
| 2024-01-15 21:56:44,251 44k INFO ====> Epoch: 64, cost 13.72 s | |
| 2024-01-15 21:56:58,159 44k INFO ====> Epoch: 65, cost 13.91 s | |
| 2024-01-15 21:57:11,799 44k INFO ====> Epoch: 66, cost 13.64 s | |
| 2024-01-15 21:57:25,681 44k INFO ====> Epoch: 67, cost 13.88 s | |
| 2024-01-15 21:57:39,631 44k INFO ====> Epoch: 68, cost 13.95 s | |
| 2024-01-15 21:57:53,269 44k INFO ====> Epoch: 69, cost 13.64 s | |
| 2024-01-15 21:58:06,995 44k INFO ====> Epoch: 70, cost 13.73 s | |
| 2024-01-15 21:58:20,876 44k INFO ====> Epoch: 71, cost 13.88 s | |
| 2024-01-15 21:58:34,617 44k INFO ====> Epoch: 72, cost 13.74 s | |
| 2024-01-15 21:58:49,158 44k INFO ====> Epoch: 73, cost 14.54 s | |
| 2024-01-15 21:59:03,183 44k INFO ====> Epoch: 74, cost 14.03 s | |
| 2024-01-15 21:59:17,138 44k INFO ====> Epoch: 75, cost 13.95 s | |
| 2024-01-15 21:59:31,173 44k INFO ====> Epoch: 76, cost 14.04 s | |
| 2024-01-15 21:59:43,438 44k INFO Train Epoch: 77 [85%] | |
| 2024-01-15 21:59:43,439 44k INFO Losses: [2.573672294616699, 2.2684617042541504, 4.892022132873535, 13.899528503417969, 1.038884162902832], step: 1000, lr: 9.905443942579728e-05, reference_loss: 24.672569274902344 | |
| 2024-01-15 21:59:52,292 44k INFO Saving model and optimizer state at iteration 77 to ./logs/44k/G_1000.pth | |
| 2024-01-15 21:59:53,921 44k INFO Saving model and optimizer state at iteration 77 to ./logs/44k/D_1000.pth | |
| 2024-01-15 21:59:56,185 44k INFO ====> Epoch: 77, cost 25.01 s | |
| 2024-01-15 22:00:10,261 44k INFO ====> Epoch: 78, cost 14.08 s | |
| 2024-01-15 22:00:24,304 44k INFO ====> Epoch: 79, cost 14.04 s | |
| 2024-01-15 22:00:38,177 44k INFO ====> Epoch: 80, cost 13.87 s | |
| 2024-01-15 22:00:52,165 44k INFO ====> Epoch: 81, cost 13.99 s | |
| 2024-01-15 22:01:06,379 44k INFO ====> Epoch: 82, cost 14.21 s | |
| 2024-01-15 22:01:19,749 44k INFO ====> Epoch: 83, cost 13.37 s | |
| 2024-01-15 22:01:33,462 44k INFO ====> Epoch: 84, cost 13.71 s | |
| 2024-01-15 22:01:47,371 44k INFO ====> Epoch: 85, cost 13.91 s | |
| 2024-01-15 22:02:01,088 44k INFO ====> Epoch: 86, cost 13.72 s | |
| 2024-01-15 22:02:14,745 44k INFO ====> Epoch: 87, cost 13.66 s | |
| 2024-01-15 22:02:29,521 44k INFO ====> Epoch: 88, cost 14.78 s | |
| 2024-01-15 22:02:43,873 44k INFO ====> Epoch: 89, cost 14.35 s | |
| 2024-01-15 22:02:57,384 44k INFO ====> Epoch: 90, cost 13.51 s | |
| 2024-01-15 22:03:11,015 44k INFO ====> Epoch: 91, cost 13.63 s | |
| 2024-01-15 22:03:25,245 44k INFO ====> Epoch: 92, cost 14.23 s | |
| 2024-01-15 22:03:34,179 44k INFO Train Epoch: 93 [23%] | |
| 2024-01-15 22:03:34,180 44k INFO Losses: [2.4794790744781494, 2.7422523498535156, 8.08922290802002, 18.03740119934082, 1.2495466470718384], step: 1200, lr: 9.885651616572276e-05, reference_loss: 32.597900390625 | |
| 2024-01-15 22:03:42,948 44k INFO Saving model and optimizer state at iteration 93 to ./logs/44k/G_1200.pth | |
| 2024-01-15 22:03:44,689 44k INFO Saving model and optimizer state at iteration 93 to ./logs/44k/D_1200.pth | |
| 2024-01-15 22:03:50,225 44k INFO ====> Epoch: 93, cost 24.98 s | |
| 2024-01-15 22:04:03,905 44k INFO ====> Epoch: 94, cost 13.68 s | |
| 2024-01-15 22:04:17,946 44k INFO ====> Epoch: 95, cost 14.04 s | |
| 2024-01-15 22:04:32,161 44k INFO ====> Epoch: 96, cost 14.22 s | |
| 2024-01-15 22:04:45,792 44k INFO ====> Epoch: 97, cost 13.63 s | |
| 2024-01-15 22:04:59,567 44k INFO ====> Epoch: 98, cost 13.77 s | |
| 2024-01-15 22:05:13,784 44k INFO ====> Epoch: 99, cost 14.22 s | |
| 2024-01-15 22:05:27,301 44k INFO ====> Epoch: 100, cost 13.52 s | |
| 2024-01-15 22:05:40,868 44k INFO ====> Epoch: 101, cost 13.57 s | |
| 2024-01-15 22:05:54,770 44k INFO ====> Epoch: 102, cost 13.90 s | |
| 2024-01-15 22:06:09,072 44k INFO ====> Epoch: 103, cost 14.30 s | |
| 2024-01-15 22:06:23,302 44k INFO ====> Epoch: 104, cost 14.23 s | |
| 2024-01-15 22:06:37,837 44k INFO ====> Epoch: 105, cost 14.53 s | |
| 2024-01-15 22:06:51,483 44k INFO ====> Epoch: 106, cost 13.65 s | |
| 2024-01-15 22:07:05,217 44k INFO ====> Epoch: 107, cost 13.73 s | |
| 2024-01-15 22:07:16,378 44k INFO Train Epoch: 108 [62%] | |
| 2024-01-15 22:07:16,379 44k INFO Losses: [2.4227054119110107, 2.0833568572998047, 6.797296524047852, 14.857068061828613, 0.997945249080658], step: 1400, lr: 9.867132229656573e-05, reference_loss: 27.15837287902832 | |
| 2024-01-15 22:07:25,360 44k INFO Saving model and optimizer state at iteration 108 to ./logs/44k/G_1400.pth | |
| 2024-01-15 22:07:27,428 44k INFO Saving model and optimizer state at iteration 108 to ./logs/44k/D_1400.pth | |
| 2024-01-15 22:07:31,073 44k INFO ====> Epoch: 108, cost 25.86 s | |
| 2024-01-15 22:07:44,780 44k INFO ====> Epoch: 109, cost 13.71 s | |
| 2024-01-15 22:07:58,749 44k INFO ====> Epoch: 110, cost 13.97 s | |
| 2024-01-15 22:08:13,130 44k INFO ====> Epoch: 111, cost 14.38 s | |
| 2024-01-15 22:08:27,023 44k INFO ====> Epoch: 112, cost 13.89 s | |
| 2024-01-15 22:08:40,589 44k INFO ====> Epoch: 113, cost 13.57 s | |
| 2024-01-15 22:08:54,491 44k INFO ====> Epoch: 114, cost 13.90 s | |
| 2024-01-15 22:09:08,370 44k INFO ====> Epoch: 115, cost 13.88 s | |
| 2024-01-15 22:09:21,965 44k INFO ====> Epoch: 116, cost 13.60 s | |
| 2024-01-15 22:09:35,534 44k INFO ====> Epoch: 117, cost 13.57 s | |
| 2024-01-15 22:09:49,207 44k INFO ====> Epoch: 118, cost 13.67 s | |
| 2024-01-15 22:10:03,615 44k INFO ====> Epoch: 119, cost 14.41 s | |
| 2024-01-15 22:10:17,365 44k INFO ====> Epoch: 120, cost 13.75 s | |
| 2024-01-15 22:10:31,587 44k INFO ====> Epoch: 121, cost 14.22 s | |
| 2024-01-15 22:10:45,083 44k INFO ====> Epoch: 122, cost 13.50 s | |
| 2024-01-15 22:10:58,726 44k INFO ====> Epoch: 123, cost 13.64 s | |
| 2024-01-15 22:11:06,486 44k INFO Train Epoch: 124 [0%] | |
| 2024-01-15 22:11:06,487 44k INFO Losses: [2.508014440536499, 2.1422133445739746, 3.497328519821167, 16.272396087646484, 0.6636678576469421], step: 1600, lr: 9.847416455282387e-05, reference_loss: 25.083620071411133 | |
| 2024-01-15 22:11:15,329 44k INFO Saving model and optimizer state at iteration 124 to ./logs/44k/G_1600.pth | |
| 2024-01-15 22:11:17,075 44k INFO Saving model and optimizer state at iteration 124 to ./logs/44k/D_1600.pth | |
| 2024-01-15 22:11:23,801 44k INFO ====> Epoch: 124, cost 25.08 s | |
| 2024-01-15 22:11:37,379 44k INFO ====> Epoch: 125, cost 13.58 s | |
| 2024-01-15 22:11:51,364 44k INFO ====> Epoch: 126, cost 13.98 s | |
| 2024-01-15 22:12:05,358 44k INFO ====> Epoch: 127, cost 13.99 s | |
| 2024-01-15 22:12:19,040 44k INFO ====> Epoch: 128, cost 13.68 s | |
| 2024-01-15 22:12:32,940 44k INFO ====> Epoch: 129, cost 13.90 s | |
| 2024-01-15 22:12:46,538 44k INFO ====> Epoch: 130, cost 13.60 s | |
| 2024-01-15 22:13:00,132 44k INFO ====> Epoch: 131, cost 13.59 s | |
| 2024-01-15 22:13:14,019 44k INFO ====> Epoch: 132, cost 13.89 s | |
| 2024-01-15 22:13:28,019 44k INFO ====> Epoch: 133, cost 14.00 s | |
| 2024-01-15 22:13:41,756 44k INFO ====> Epoch: 134, cost 13.74 s | |
| 2024-01-15 22:13:55,987 44k INFO ====> Epoch: 135, cost 14.23 s | |
| 2024-01-15 22:14:09,660 44k INFO ====> Epoch: 136, cost 13.67 s | |
| 2024-01-15 22:14:23,395 44k INFO ====> Epoch: 137, cost 13.74 s | |
| 2024-01-15 22:14:37,132 44k INFO ====> Epoch: 138, cost 13.74 s | |
| 2024-01-15 22:14:47,127 44k INFO Train Epoch: 139 [38%] | |
| 2024-01-15 22:14:47,128 44k INFO Losses: [2.487508535385132, 2.2791075706481934, 5.455832481384277, 16.11585807800293, 0.36197128891944885], step: 1800, lr: 9.828968696598508e-05, reference_loss: 26.70027732849121 | |
| 2024-01-15 22:14:56,119 44k INFO Saving model and optimizer state at iteration 139 to ./logs/44k/G_1800.pth | |
| 2024-01-15 22:14:57,795 44k INFO Saving model and optimizer state at iteration 139 to ./logs/44k/D_1800.pth | |
| 2024-01-15 22:15:02,432 44k INFO ====> Epoch: 139, cost 25.30 s | |
| 2024-01-15 22:15:16,252 44k INFO ====> Epoch: 140, cost 13.82 s | |
| 2024-01-15 22:15:30,023 44k INFO ====> Epoch: 141, cost 13.77 s | |
| 2024-01-15 22:15:44,060 44k INFO ====> Epoch: 142, cost 14.04 s | |
| 2024-01-15 22:15:58,328 44k INFO ====> Epoch: 143, cost 14.27 s | |
| 2024-01-15 22:16:12,362 44k INFO ====> Epoch: 144, cost 14.03 s | |
| 2024-01-15 22:16:26,132 44k INFO ====> Epoch: 145, cost 13.77 s | |
| 2024-01-15 22:16:40,017 44k INFO ====> Epoch: 146, cost 13.88 s | |
| 2024-01-15 22:16:53,660 44k INFO ====> Epoch: 147, cost 13.64 s | |
| 2024-01-15 22:17:07,414 44k INFO ====> Epoch: 148, cost 13.75 s | |
| 2024-01-15 22:17:21,423 44k INFO ====> Epoch: 149, cost 14.01 s | |
| 2024-01-15 22:17:35,640 44k INFO ====> Epoch: 150, cost 14.22 s | |
| 2024-01-15 22:17:49,145 44k INFO ====> Epoch: 151, cost 13.51 s | |
| 2024-01-15 22:18:02,965 44k INFO ====> Epoch: 152, cost 13.82 s | |
| 2024-01-15 22:18:17,050 44k INFO ====> Epoch: 153, cost 14.09 s | |
| 2024-01-15 22:18:28,796 44k INFO Train Epoch: 154 [77%] | |
| 2024-01-15 22:18:28,797 44k INFO Losses: [2.475991725921631, 2.288100242614746, 4.386265754699707, 12.176278114318848, 0.3904218077659607], step: 2000, lr: 9.810555497212693e-05, reference_loss: 21.717058181762695 | |
| 2024-01-15 22:18:38,626 44k INFO Saving model and optimizer state at iteration 154 to ./logs/44k/G_2000.pth | |
| 2024-01-15 22:18:40,373 44k INFO Saving model and optimizer state at iteration 154 to ./logs/44k/D_2000.pth | |
| 2024-01-15 22:18:43,059 44k INFO ====> Epoch: 154, cost 26.01 s | |
| 2024-01-15 22:18:56,967 44k INFO ====> Epoch: 155, cost 13.91 s | |
| 2024-01-15 22:19:10,508 44k INFO ====> Epoch: 156, cost 13.54 s | |
| 2024-01-15 22:19:24,401 44k INFO ====> Epoch: 157, cost 13.89 s | |
| 2024-01-15 22:19:38,014 44k INFO ====> Epoch: 158, cost 13.61 s | |
| 2024-01-15 22:19:52,118 44k INFO ====> Epoch: 159, cost 14.10 s | |
| 2024-01-15 22:20:06,419 44k INFO ====> Epoch: 160, cost 14.30 s | |
| 2024-01-15 22:20:20,101 44k INFO ====> Epoch: 161, cost 13.68 s | |
| 2024-01-15 22:20:33,897 44k INFO ====> Epoch: 162, cost 13.80 s | |
| 2024-01-15 22:20:47,737 44k INFO ====> Epoch: 163, cost 13.84 s | |
| 2024-01-15 22:21:01,503 44k INFO ====> Epoch: 164, cost 13.77 s | |
| 2024-01-15 22:21:15,303 44k INFO ====> Epoch: 165, cost 13.80 s | |
| 2024-01-15 22:21:29,059 44k INFO ====> Epoch: 166, cost 13.76 s | |
| 2024-01-15 22:21:42,959 44k INFO ====> Epoch: 167, cost 13.90 s | |
| 2024-01-15 22:21:57,160 44k INFO ====> Epoch: 168, cost 14.20 s | |
| 2024-01-15 22:22:11,144 44k INFO ====> Epoch: 169, cost 13.98 s | |
| 2024-01-15 22:22:19,586 44k INFO Train Epoch: 170 [15%] | |
| 2024-01-15 22:22:19,587 44k INFO Losses: [2.866142749786377, 1.7102158069610596, 4.101927280426025, 11.041349411010742, 0.9578508138656616], step: 2200, lr: 9.790952770283884e-05, reference_loss: 20.677486419677734 | |
| 2024-01-15 22:22:28,235 44k INFO Saving model and optimizer state at iteration 170 to ./logs/44k/G_2200.pth | |
| 2024-01-15 22:22:30,104 44k INFO Saving model and optimizer state at iteration 170 to ./logs/44k/D_2200.pth | |
| 2024-01-15 22:22:36,500 44k INFO ====> Epoch: 170, cost 25.36 s | |
| 2024-01-15 22:22:50,332 44k INFO ====> Epoch: 171, cost 13.83 s | |
| 2024-01-15 22:23:04,142 44k INFO ====> Epoch: 172, cost 13.81 s | |
| 2024-01-15 22:23:18,572 44k INFO ====> Epoch: 173, cost 14.43 s | |
| 2024-01-15 22:23:32,346 44k INFO ====> Epoch: 174, cost 13.77 s | |
| 2024-01-15 22:23:46,427 44k INFO ====> Epoch: 175, cost 14.08 s | |
| 2024-01-15 22:23:59,910 44k INFO ====> Epoch: 176, cost 13.48 s | |
| 2024-01-15 22:24:13,665 44k INFO ====> Epoch: 177, cost 13.75 s | |
| 2024-01-15 22:24:27,585 44k INFO ====> Epoch: 178, cost 13.92 s | |
| 2024-01-15 22:24:41,471 44k INFO ====> Epoch: 179, cost 13.89 s | |
| 2024-01-15 22:24:55,733 44k INFO ====> Epoch: 180, cost 14.26 s | |
| 2024-01-15 22:25:09,704 44k INFO ====> Epoch: 181, cost 13.97 s | |
| 2024-01-15 22:25:23,593 44k INFO ====> Epoch: 182, cost 13.89 s | |
| 2024-01-15 22:25:37,290 44k INFO ====> Epoch: 183, cost 13.70 s | |
| 2024-01-15 22:25:51,098 44k INFO ====> Epoch: 184, cost 13.81 s | |
| 2024-01-15 22:26:01,925 44k INFO Train Epoch: 185 [54%] | |
| 2024-01-15 22:26:01,927 44k INFO Losses: [2.702141284942627, 2.1263349056243896, 3.516636848449707, 11.536981582641602, 0.5434733033180237], step: 2400, lr: 9.772610788423802e-05, reference_loss: 20.425569534301758 | |
| 2024-01-15 22:26:10,843 44k INFO Saving model and optimizer state at iteration 185 to ./logs/44k/G_2400.pth | |
| 2024-01-15 22:26:12,578 44k INFO Saving model and optimizer state at iteration 185 to ./logs/44k/D_2400.pth | |
| 2024-01-15 22:26:16,585 44k INFO ====> Epoch: 185, cost 25.49 s | |
| 2024-01-15 22:26:30,774 44k INFO ====> Epoch: 186, cost 14.19 s | |
| 2024-01-15 22:26:44,664 44k INFO ====> Epoch: 187, cost 13.89 s | |
| 2024-01-15 22:26:59,086 44k INFO ====> Epoch: 188, cost 14.42 s | |
| 2024-01-15 22:27:12,919 44k INFO ====> Epoch: 189, cost 13.83 s | |
| 2024-01-15 22:27:26,693 44k INFO ====> Epoch: 190, cost 13.77 s | |
| 2024-01-15 22:27:40,525 44k INFO ====> Epoch: 191, cost 13.83 s | |
| 2024-01-15 22:27:54,458 44k INFO ====> Epoch: 192, cost 13.93 s | |
| 2024-01-15 22:28:08,165 44k INFO ====> Epoch: 193, cost 13.71 s | |
| 2024-01-15 22:28:22,154 44k INFO ====> Epoch: 194, cost 13.99 s | |
| 2024-01-15 22:28:36,064 44k INFO ====> Epoch: 195, cost 13.91 s | |
| 2024-01-15 22:28:49,691 44k INFO ====> Epoch: 196, cost 13.63 s | |
| 2024-01-15 22:29:03,534 44k INFO ====> Epoch: 197, cost 13.84 s | |
| 2024-01-15 22:29:17,504 44k INFO ====> Epoch: 198, cost 13.97 s | |
| 2024-01-15 22:29:31,624 44k INFO ====> Epoch: 199, cost 14.12 s | |
| 2024-01-15 22:29:44,542 44k INFO Train Epoch: 200 [92%] | |
| 2024-01-15 22:29:44,543 44k INFO Losses: [2.768800735473633, 1.7963769435882568, 5.916341781616211, 13.012473106384277, 0.9058436751365662], step: 2600, lr: 9.754303167703689e-05, reference_loss: 24.39983558654785 | |
| 2024-01-15 22:29:53,566 44k INFO Saving model and optimizer state at iteration 200 to ./logs/44k/G_2600.pth | |
| 2024-01-15 22:29:55,654 44k INFO Saving model and optimizer state at iteration 200 to ./logs/44k/D_2600.pth | |
| 2024-01-15 22:29:57,563 44k INFO ====> Epoch: 200, cost 25.94 s | |
| 2024-01-15 22:30:11,562 44k INFO ====> Epoch: 201, cost 14.00 s | |
| 2024-01-15 22:30:25,396 44k INFO ====> Epoch: 202, cost 13.83 s | |
| 2024-01-15 22:30:39,445 44k INFO ====> Epoch: 203, cost 14.05 s | |
| 2024-01-15 22:30:53,337 44k INFO ====> Epoch: 204, cost 13.89 s | |
| 2024-01-15 22:31:06,765 44k INFO ====> Epoch: 205, cost 13.43 s | |
| 2024-01-15 22:31:20,967 44k INFO ====> Epoch: 206, cost 14.20 s | |
| 2024-01-15 22:31:34,679 44k INFO ====> Epoch: 207, cost 13.71 s | |
| 2024-01-15 22:31:48,461 44k INFO ====> Epoch: 208, cost 13.78 s | |
| 2024-01-15 22:32:02,780 44k INFO ====> Epoch: 209, cost 14.32 s | |
| 2024-01-15 22:32:16,427 44k INFO ====> Epoch: 210, cost 13.65 s | |
| 2024-01-15 22:32:30,373 44k INFO ====> Epoch: 211, cost 13.95 s | |
| 2024-01-15 22:32:44,389 44k INFO ====> Epoch: 212, cost 14.02 s | |
| 2024-01-15 22:32:58,231 44k INFO ====> Epoch: 213, cost 13.84 s | |
| 2024-01-15 22:33:12,051 44k INFO ====> Epoch: 214, cost 13.82 s | |
| 2024-01-15 22:33:26,135 44k INFO ====> Epoch: 215, cost 14.08 s | |
| 2024-01-15 22:33:35,505 44k INFO Train Epoch: 216 [31%] | |
| 2024-01-15 22:33:35,507 44k INFO Losses: [2.8601551055908203, 1.7048544883728027, 1.3852932453155518, 6.691847324371338, 0.38322365283966064], step: 2800, lr: 9.734812840022278e-05, reference_loss: 13.025374412536621 | |
| 2024-01-15 22:33:44,503 44k INFO Saving model and optimizer state at iteration 216 to ./logs/44k/G_2800.pth | |
| 2024-01-15 22:33:46,142 44k INFO Saving model and optimizer state at iteration 216 to ./logs/44k/D_2800.pth | |
| 2024-01-15 22:33:51,126 44k INFO ====> Epoch: 216, cost 24.99 s | |
| 2024-01-15 22:34:04,857 44k INFO ====> Epoch: 217, cost 13.73 s | |
| 2024-01-15 22:34:18,445 44k INFO ====> Epoch: 218, cost 13.59 s | |
| 2024-01-15 22:34:32,744 44k INFO ====> Epoch: 219, cost 14.30 s | |
| 2024-01-15 22:34:46,351 44k INFO ====> Epoch: 220, cost 13.61 s | |
| 2024-01-15 22:35:00,242 44k INFO ====> Epoch: 221, cost 13.89 s | |
| 2024-01-15 22:35:14,122 44k INFO ====> Epoch: 222, cost 13.88 s | |
| 2024-01-15 22:35:28,433 44k INFO ====> Epoch: 223, cost 14.31 s | |
| 2024-01-15 22:35:41,990 44k INFO ====> Epoch: 224, cost 13.56 s | |
| 2024-01-15 22:35:55,559 44k INFO ====> Epoch: 225, cost 13.57 s | |
| 2024-01-15 22:36:09,243 44k INFO ====> Epoch: 226, cost 13.68 s | |
| 2024-01-15 22:36:22,999 44k INFO ====> Epoch: 227, cost 13.76 s | |
| 2024-01-15 22:36:36,778 44k INFO ====> Epoch: 228, cost 13.78 s | |
| 2024-01-15 22:36:50,354 44k INFO ====> Epoch: 229, cost 13.58 s | |
| 2024-01-15 22:37:04,523 44k INFO ====> Epoch: 230, cost 14.17 s | |
| 2024-01-15 22:37:15,933 44k INFO Train Epoch: 231 [69%] | |
| 2024-01-15 22:37:15,934 44k INFO Losses: [2.680328607559204, 2.280780076980591, 6.38661003112793, 19.464754104614258, 1.1633503437042236], step: 3000, lr: 9.716576028476738e-05, reference_loss: 31.9758243560791 | |
| 2024-01-15 22:37:25,252 44k INFO Saving model and optimizer state at iteration 231 to ./logs/44k/G_3000.pth | |
| 2024-01-15 22:37:27,047 44k INFO Saving model and optimizer state at iteration 231 to ./logs/44k/D_3000.pth | |
| 2024-01-15 22:37:30,288 44k INFO ====> Epoch: 231, cost 25.76 s | |
| 2024-01-15 22:37:44,405 44k INFO ====> Epoch: 232, cost 14.12 s | |
| 2024-01-15 22:37:58,073 44k INFO ====> Epoch: 233, cost 13.67 s | |
| 2024-01-15 22:38:12,078 44k INFO ====> Epoch: 234, cost 14.01 s | |
| 2024-01-15 22:38:25,934 44k INFO ====> Epoch: 235, cost 13.86 s | |
| 2024-01-15 22:38:40,404 44k INFO ====> Epoch: 236, cost 14.47 s | |
| 2024-01-15 22:38:54,179 44k INFO ====> Epoch: 237, cost 13.78 s | |
| 2024-01-15 22:39:07,828 44k INFO ====> Epoch: 238, cost 13.65 s | |
| 2024-01-15 22:39:22,027 44k INFO ====> Epoch: 239, cost 14.20 s | |
| 2024-01-15 22:39:35,245 44k INFO ====> Epoch: 240, cost 13.22 s | |
| 2024-01-15 22:39:49,272 44k INFO ====> Epoch: 241, cost 14.03 s | |
| 2024-01-15 22:40:03,295 44k INFO ====> Epoch: 242, cost 14.02 s | |
| 2024-01-15 22:40:16,978 44k INFO ====> Epoch: 243, cost 13.68 s | |
| 2024-01-15 22:40:31,206 44k INFO ====> Epoch: 244, cost 14.23 s | |
| 2024-01-15 22:40:45,485 44k INFO ====> Epoch: 245, cost 14.28 s | |
| 2024-01-15 22:40:59,804 44k INFO ====> Epoch: 246, cost 14.32 s | |
| 2024-01-15 22:41:07,821 44k INFO Train Epoch: 247 [8%] | |
| 2024-01-15 22:41:07,823 44k INFO Losses: [2.2632689476013184, 2.539794921875, 6.44555139541626, 16.46857452392578, 0.6917562484741211], step: 3200, lr: 9.69716108437664e-05, reference_loss: 28.408946990966797 | |
| 2024-01-15 22:41:16,608 44k INFO Saving model and optimizer state at iteration 247 to ./logs/44k/G_3200.pth | |
| 2024-01-15 22:41:18,343 44k INFO Saving model and optimizer state at iteration 247 to ./logs/44k/D_3200.pth | |
| 2024-01-15 22:41:25,232 44k INFO ====> Epoch: 247, cost 25.43 s | |
| 2024-01-15 22:41:39,275 44k INFO ====> Epoch: 248, cost 14.04 s | |
| 2024-01-15 22:41:53,173 44k INFO ====> Epoch: 249, cost 13.90 s | |
| 2024-01-15 22:42:07,125 44k INFO ====> Epoch: 250, cost 13.95 s | |
| 2024-01-15 22:42:20,786 44k INFO ====> Epoch: 251, cost 13.66 s | |
| 2024-01-15 22:42:35,266 44k INFO ====> Epoch: 252, cost 14.48 s | |
| 2024-01-15 22:42:49,068 44k INFO ====> Epoch: 253, cost 13.80 s | |
| 2024-01-15 22:43:02,861 44k INFO ====> Epoch: 254, cost 13.79 s | |
| 2024-01-15 22:43:16,566 44k INFO ====> Epoch: 255, cost 13.70 s | |
| 2024-01-15 22:43:30,842 44k INFO ====> Epoch: 256, cost 14.28 s | |
| 2024-01-15 22:43:44,359 44k INFO ====> Epoch: 257, cost 13.52 s | |
| 2024-01-15 22:43:57,966 44k INFO ====> Epoch: 258, cost 13.61 s | |
| 2024-01-15 22:44:11,956 44k INFO ====> Epoch: 259, cost 13.99 s | |
| 2024-01-15 22:44:26,082 44k INFO ====> Epoch: 260, cost 14.13 s | |
| 2024-01-15 22:44:40,364 44k INFO ====> Epoch: 261, cost 14.28 s | |
| 2024-01-15 22:44:50,357 44k INFO Train Epoch: 262 [46%] | |
| 2024-01-15 22:44:50,358 44k INFO Losses: [2.8824191093444824, 2.1685523986816406, 5.550257682800293, 14.615203857421875, 0.48655691742897034], step: 3400, lr: 9.678994808133967e-05, reference_loss: 25.70298957824707 | |
| 2024-01-15 22:44:59,109 44k INFO Saving model and optimizer state at iteration 262 to ./logs/44k/G_3400.pth | |
| 2024-01-15 22:45:00,846 44k INFO Saving model and optimizer state at iteration 262 to ./logs/44k/D_3400.pth | |
| 2024-01-15 22:45:05,237 44k INFO ====> Epoch: 262, cost 24.87 s | |
| 2024-01-15 22:45:18,747 44k INFO ====> Epoch: 263, cost 13.51 s | |
| 2024-01-15 22:45:32,285 44k INFO ====> Epoch: 264, cost 13.54 s | |
| 2024-01-15 22:45:46,203 44k INFO ====> Epoch: 265, cost 13.92 s | |
| 2024-01-15 22:46:00,108 44k INFO ====> Epoch: 266, cost 13.90 s | |
| 2024-01-15 22:46:13,601 44k INFO ====> Epoch: 267, cost 13.49 s | |
| 2024-01-15 22:46:27,439 44k INFO ====> Epoch: 268, cost 13.84 s | |
| 2024-01-15 22:46:41,575 44k INFO ====> Epoch: 269, cost 14.14 s | |
| 2024-01-15 22:46:55,866 44k INFO ====> Epoch: 270, cost 14.29 s | |
| 2024-01-15 22:47:09,631 44k INFO ====> Epoch: 271, cost 13.76 s | |
| 2024-01-15 22:47:23,328 44k INFO ====> Epoch: 272, cost 13.70 s | |
| 2024-01-15 22:47:36,955 44k INFO ====> Epoch: 273, cost 13.63 s | |
| 2024-01-15 22:47:50,593 44k INFO ====> Epoch: 274, cost 13.64 s | |
| 2024-01-15 22:48:04,170 44k INFO ====> Epoch: 275, cost 13.58 s | |
| 2024-01-15 22:48:18,419 44k INFO ====> Epoch: 276, cost 14.25 s | |
| 2024-01-15 22:48:31,295 44k INFO Train Epoch: 277 [85%] | |
| 2024-01-15 22:48:31,296 44k INFO Losses: [2.565187931060791, 2.0622670650482178, 4.785097599029541, 13.107951164245605, 0.8774415254592896], step: 3600, lr: 9.660862563871342e-05, reference_loss: 23.397945404052734 | |
| 2024-01-15 22:48:40,908 44k INFO Saving model and optimizer state at iteration 277 to ./logs/44k/G_3600.pth | |
| 2024-01-15 22:48:42,694 44k INFO Saving model and optimizer state at iteration 277 to ./logs/44k/D_3600.pth | |
| 2024-01-15 22:48:45,118 44k INFO ====> Epoch: 277, cost 26.70 s | |
| 2024-01-15 22:48:58,929 44k INFO ====> Epoch: 278, cost 13.81 s | |
| 2024-01-15 22:49:12,702 44k INFO ====> Epoch: 279, cost 13.77 s | |
| 2024-01-15 22:49:26,914 44k INFO ====> Epoch: 280, cost 14.21 s | |
| 2024-01-15 22:49:41,066 44k INFO ====> Epoch: 281, cost 14.15 s | |
| 2024-01-15 22:49:55,231 44k INFO ====> Epoch: 282, cost 14.17 s | |
| 2024-01-15 22:50:09,565 44k INFO ====> Epoch: 283, cost 14.33 s | |
| 2024-01-15 22:50:23,627 44k INFO ====> Epoch: 284, cost 14.06 s | |
| 2024-01-15 22:50:37,734 44k INFO ====> Epoch: 285, cost 14.11 s | |
| 2024-01-15 22:50:51,931 44k INFO ====> Epoch: 286, cost 14.20 s | |
| 2024-01-15 22:51:05,900 44k INFO ====> Epoch: 287, cost 13.97 s | |
| 2024-01-15 22:51:19,950 44k INFO ====> Epoch: 288, cost 14.05 s | |
| 2024-01-15 22:51:34,128 44k INFO ====> Epoch: 289, cost 14.18 s | |
| 2024-01-15 22:51:48,603 44k INFO ====> Epoch: 290, cost 14.48 s | |
| 2024-01-15 22:52:02,366 44k INFO ====> Epoch: 291, cost 13.76 s | |
| 2024-01-15 22:52:16,298 44k INFO ====> Epoch: 292, cost 13.93 s | |
| 2024-01-15 22:52:25,194 44k INFO Train Epoch: 293 [23%] | |
| 2024-01-15 22:52:25,195 44k INFO Losses: [2.4987902641296387, 2.641681671142578, 7.88032865524292, 16.4152774810791, 0.9718687534332275], step: 3800, lr: 9.641558942298625e-05, reference_loss: 30.40794563293457 | |
| 2024-01-15 22:52:34,320 44k INFO Saving model and optimizer state at iteration 293 to ./logs/44k/G_3800.pth | |
| 2024-01-15 22:52:35,993 44k INFO Saving model and optimizer state at iteration 293 to ./logs/44k/D_3800.pth | |
| 2024-01-15 22:52:41,753 44k INFO ====> Epoch: 293, cost 25.46 s | |
| 2024-01-15 22:52:55,557 44k INFO ====> Epoch: 294, cost 13.80 s | |
| 2024-01-15 22:53:09,739 44k INFO ====> Epoch: 295, cost 14.18 s | |
| 2024-01-15 22:53:23,394 44k INFO ====> Epoch: 296, cost 13.66 s | |
| 2024-01-15 22:53:37,917 44k INFO ====> Epoch: 297, cost 14.52 s | |
| 2024-01-15 22:53:52,803 44k INFO ====> Epoch: 298, cost 14.89 s | |
| 2024-01-15 22:54:07,908 44k INFO ====> Epoch: 299, cost 15.10 s | |
| 2024-01-15 22:54:22,689 44k INFO ====> Epoch: 300, cost 14.78 s | |
| 2024-01-15 22:54:37,335 44k INFO ====> Epoch: 301, cost 14.65 s | |
| 2024-01-15 22:54:51,625 44k INFO ====> Epoch: 302, cost 14.29 s | |
| 2024-01-15 22:55:05,458 44k INFO ====> Epoch: 303, cost 13.83 s | |
| 2024-01-15 22:55:19,752 44k INFO ====> Epoch: 304, cost 14.29 s | |
| 2024-01-15 22:55:35,005 44k INFO ====> Epoch: 305, cost 15.25 s | |
| 2024-01-15 22:55:49,447 44k INFO ====> Epoch: 306, cost 14.44 s | |
| 2024-01-15 22:56:03,935 44k INFO ====> Epoch: 307, cost 14.49 s | |
| 2024-01-15 22:56:15,286 44k INFO Train Epoch: 308 [62%] | |
| 2024-01-15 22:56:15,287 44k INFO Losses: [2.8865966796875, 1.8975958824157715, 5.896610260009766, 13.233089447021484, 0.854919970035553], step: 4000, lr: 9.62349682889948e-05, reference_loss: 24.76881217956543 | |
| 2024-01-15 22:56:24,674 44k INFO Saving model and optimizer state at iteration 308 to ./logs/44k/G_4000.pth | |
| 2024-01-15 22:56:26,445 44k INFO Saving model and optimizer state at iteration 308 to ./logs/44k/D_4000.pth | |
| 2024-01-15 22:56:30,128 44k INFO ====> Epoch: 308, cost 26.19 s | |
| 2024-01-15 22:56:44,598 44k INFO ====> Epoch: 309, cost 14.47 s | |
| 2024-01-15 22:56:59,168 44k INFO ====> Epoch: 310, cost 14.57 s | |
| 2024-01-15 22:57:13,354 44k INFO ====> Epoch: 311, cost 14.19 s | |
| 2024-01-15 22:57:27,698 44k INFO ====> Epoch: 312, cost 14.34 s | |
| 2024-01-15 22:57:42,142 44k INFO ====> Epoch: 313, cost 14.44 s | |
| 2024-01-15 22:57:56,862 44k INFO ====> Epoch: 314, cost 14.72 s | |
| 2024-01-15 22:58:11,032 44k INFO ====> Epoch: 315, cost 14.17 s | |
| 2024-01-15 22:58:24,680 44k INFO ====> Epoch: 316, cost 13.65 s | |
| 2024-01-15 22:58:38,451 44k INFO ====> Epoch: 317, cost 13.77 s | |
| 2024-01-15 22:58:52,475 44k INFO ====> Epoch: 318, cost 14.02 s | |
| 2024-01-15 22:59:06,345 44k INFO ====> Epoch: 319, cost 13.87 s | |
| 2024-01-15 22:59:20,437 44k INFO ====> Epoch: 320, cost 14.09 s | |
| 2024-01-15 22:59:34,365 44k INFO ====> Epoch: 321, cost 13.93 s | |
| 2024-01-15 22:59:48,250 44k INFO ====> Epoch: 322, cost 13.88 s | |
| 2024-01-15 23:00:01,950 44k INFO ====> Epoch: 323, cost 13.70 s | |
| 2024-01-15 23:00:09,401 44k INFO Train Epoch: 324 [0%] | |
| 2024-01-15 23:00:09,401 44k INFO Losses: [2.5641047954559326, 2.390501022338867, 3.6597917079925537, 15.15865707397461, 0.5480677485466003], step: 4200, lr: 9.604267868776807e-05, reference_loss: 24.321123123168945 | |
| 2024-01-15 23:00:18,903 44k INFO Saving model and optimizer state at iteration 324 to ./logs/44k/G_4200.pth | |
| 2024-01-15 23:00:20,733 44k INFO Saving model and optimizer state at iteration 324 to ./logs/44k/D_4200.pth | |
| 2024-01-15 23:00:21,413 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_200.pth | |
| 2024-01-15 23:00:21,434 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_200.pth | |
| 2024-01-15 23:00:27,602 44k INFO ====> Epoch: 324, cost 25.65 s | |
| 2024-01-15 23:00:41,546 44k INFO ====> Epoch: 325, cost 13.94 s | |
| 2024-01-15 23:00:55,382 44k INFO ====> Epoch: 326, cost 13.84 s | |
| 2024-01-15 23:01:09,354 44k INFO ====> Epoch: 327, cost 13.97 s | |
| 2024-01-15 23:01:23,082 44k INFO ====> Epoch: 328, cost 13.73 s | |
| 2024-01-15 23:01:36,940 44k INFO ====> Epoch: 329, cost 13.86 s | |
| 2024-01-15 23:01:50,762 44k INFO ====> Epoch: 330, cost 13.82 s | |
| 2024-01-15 23:02:04,879 44k INFO ====> Epoch: 331, cost 14.12 s | |
| 2024-01-15 23:02:18,639 44k INFO ====> Epoch: 332, cost 13.76 s | |
| 2024-01-15 23:02:32,503 44k INFO ====> Epoch: 333, cost 13.86 s | |
| 2024-01-15 23:02:46,266 44k INFO ====> Epoch: 334, cost 13.76 s | |
| 2024-01-15 23:03:00,179 44k INFO ====> Epoch: 335, cost 13.91 s | |
| 2024-01-15 23:03:13,782 44k INFO ====> Epoch: 336, cost 13.60 s | |
| 2024-01-15 23:03:27,682 44k INFO ====> Epoch: 337, cost 13.90 s | |
| 2024-01-15 23:03:41,326 44k INFO ====> Epoch: 338, cost 13.64 s | |
| 2024-01-15 23:03:51,279 44k INFO Train Epoch: 339 [38%] | |
| 2024-01-15 23:03:51,280 44k INFO Losses: [2.670449733734131, 2.212564468383789, 5.1901044845581055, 14.490642547607422, 0.3224382698535919], step: 4400, lr: 9.586275614992974e-05, reference_loss: 24.886199951171875 | |
| 2024-01-15 23:04:00,101 44k INFO Saving model and optimizer state at iteration 339 to ./logs/44k/G_4400.pth | |
| 2024-01-15 23:04:02,096 44k INFO Saving model and optimizer state at iteration 339 to ./logs/44k/D_4400.pth | |
| 2024-01-15 23:04:02,760 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_400.pth | |
| 2024-01-15 23:04:02,772 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_400.pth | |
| 2024-01-15 23:04:06,829 44k INFO ====> Epoch: 339, cost 25.50 s | |
| 2024-01-15 23:04:20,738 44k INFO ====> Epoch: 340, cost 13.91 s | |
| 2024-01-15 23:04:34,502 44k INFO ====> Epoch: 341, cost 13.76 s | |
| 2024-01-15 23:04:48,152 44k INFO ====> Epoch: 342, cost 13.65 s | |
| 2024-01-15 23:05:02,007 44k INFO ====> Epoch: 343, cost 13.85 s | |
| 2024-01-15 23:05:15,848 44k INFO ====> Epoch: 344, cost 13.84 s | |
| 2024-01-15 23:05:29,681 44k INFO ====> Epoch: 345, cost 13.83 s | |
| 2024-01-15 23:05:43,834 44k INFO ====> Epoch: 346, cost 14.15 s | |
| 2024-01-15 23:05:57,443 44k INFO ====> Epoch: 347, cost 13.61 s | |
| 2024-01-15 23:06:11,195 44k INFO ====> Epoch: 348, cost 13.75 s | |
| 2024-01-15 23:06:25,177 44k INFO ====> Epoch: 349, cost 13.98 s | |
| 2024-01-15 23:06:39,634 44k INFO ====> Epoch: 350, cost 14.46 s | |
| 2024-01-15 23:06:53,313 44k INFO ====> Epoch: 351, cost 13.68 s | |
| 2024-01-15 23:07:07,568 44k INFO ====> Epoch: 352, cost 14.26 s | |
| 2024-01-15 23:07:21,430 44k INFO ====> Epoch: 353, cost 13.86 s | |
| 2024-01-15 23:07:33,241 44k INFO Train Epoch: 354 [77%] | |
| 2024-01-15 23:07:33,242 44k INFO Losses: [2.6923983097076416, 2.0353307723999023, 3.9855611324310303, 11.176602363586426, 0.2694582939147949], step: 4600, lr: 9.568317067182427e-05, reference_loss: 20.159351348876953 | |
| 2024-01-15 23:07:42,312 44k INFO Saving model and optimizer state at iteration 354 to ./logs/44k/G_4600.pth | |
| 2024-01-15 23:07:44,027 44k INFO Saving model and optimizer state at iteration 354 to ./logs/44k/D_4600.pth | |
| 2024-01-15 23:07:44,687 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_600.pth | |
| 2024-01-15 23:07:44,699 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_600.pth | |
| 2024-01-15 23:07:46,850 44k INFO ====> Epoch: 354, cost 25.42 s | |
| 2024-01-15 23:08:00,622 44k INFO ====> Epoch: 355, cost 13.77 s | |
| 2024-01-15 23:08:14,591 44k INFO ====> Epoch: 356, cost 13.97 s | |
| 2024-01-15 23:08:28,198 44k INFO ====> Epoch: 357, cost 13.61 s | |
| 2024-01-15 23:08:41,754 44k INFO ====> Epoch: 358, cost 13.56 s | |
| 2024-01-15 23:08:55,842 44k INFO ====> Epoch: 359, cost 14.09 s | |
| 2024-01-15 23:09:09,446 44k INFO ====> Epoch: 360, cost 13.60 s | |
| 2024-01-15 23:09:23,213 44k INFO ====> Epoch: 361, cost 13.77 s | |
| 2024-01-15 23:09:37,532 44k INFO ====> Epoch: 362, cost 14.32 s | |
| 2024-01-15 23:09:51,691 44k INFO ====> Epoch: 363, cost 14.16 s | |
| 2024-01-15 23:10:05,775 44k INFO ====> Epoch: 364, cost 14.08 s | |
| 2024-01-15 23:10:19,674 44k INFO ====> Epoch: 365, cost 13.90 s | |
| 2024-01-15 23:10:33,572 44k INFO ====> Epoch: 366, cost 13.90 s | |
| 2024-01-15 23:10:47,329 44k INFO ====> Epoch: 367, cost 13.76 s | |
| 2024-01-15 23:11:01,182 44k INFO ====> Epoch: 368, cost 13.85 s | |
| 2024-01-15 23:11:15,100 44k INFO ====> Epoch: 369, cost 13.92 s | |
| 2024-01-15 23:11:23,557 44k INFO Train Epoch: 370 [15%] | |
| 2024-01-15 23:11:23,558 44k INFO Losses: [2.5849499702453613, 2.0209779739379883, 4.791561603546143, 11.127520561218262, 0.9297534823417664], step: 4800, lr: 9.54919836318146e-05, reference_loss: 21.454763412475586 | |
| 2024-01-15 23:11:32,908 44k INFO Saving model and optimizer state at iteration 370 to ./logs/44k/G_4800.pth | |
| 2024-01-15 23:11:34,605 44k INFO Saving model and optimizer state at iteration 370 to ./logs/44k/D_4800.pth | |
| 2024-01-15 23:11:35,278 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_800.pth | |
| 2024-01-15 23:11:35,290 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_800.pth | |
| 2024-01-15 23:11:40,593 44k INFO ====> Epoch: 370, cost 25.49 s | |
| 2024-01-15 23:11:54,080 44k INFO ====> Epoch: 371, cost 13.49 s | |
| 2024-01-15 23:12:07,668 44k INFO ====> Epoch: 372, cost 13.59 s | |
| 2024-01-15 23:12:21,541 44k INFO ====> Epoch: 373, cost 13.87 s | |
| 2024-01-15 23:12:35,169 44k INFO ====> Epoch: 374, cost 13.63 s | |
| 2024-01-15 23:12:49,210 44k INFO ====> Epoch: 375, cost 14.04 s | |
| 2024-01-15 23:13:02,803 44k INFO ====> Epoch: 376, cost 13.59 s | |
| 2024-01-15 23:13:16,837 44k INFO ====> Epoch: 377, cost 14.03 s | |
| 2024-01-15 23:13:30,869 44k INFO ====> Epoch: 378, cost 14.03 s | |
| 2024-01-15 23:13:44,893 44k INFO ====> Epoch: 379, cost 14.02 s | |
| 2024-01-15 23:13:58,484 44k INFO ====> Epoch: 380, cost 13.59 s | |
| 2024-01-15 23:14:12,191 44k INFO ====> Epoch: 381, cost 13.71 s | |
| 2024-01-15 23:14:25,676 44k INFO ====> Epoch: 382, cost 13.48 s | |
| 2024-01-15 23:14:39,263 44k INFO ====> Epoch: 383, cost 13.59 s | |
| 2024-01-15 23:14:53,428 44k INFO ====> Epoch: 384, cost 14.17 s | |
| 2024-01-15 23:15:04,034 44k INFO Train Epoch: 385 [54%] | |
| 2024-01-15 23:15:04,035 44k INFO Losses: [2.5789456367492676, 2.122814893722534, 3.7400898933410645, 11.190013885498047, 0.4549975097179413], step: 5000, lr: 9.53130927442113e-05, reference_loss: 20.086862564086914 | |
| 2024-01-15 23:15:13,148 44k INFO Saving model and optimizer state at iteration 385 to ./logs/44k/G_5000.pth | |
| 2024-01-15 23:15:14,937 44k INFO Saving model and optimizer state at iteration 385 to ./logs/44k/D_5000.pth | |
| 2024-01-15 23:15:15,617 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_1000.pth | |
| 2024-01-15 23:15:15,635 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_1000.pth | |
| 2024-01-15 23:15:18,941 44k INFO ====> Epoch: 385, cost 25.51 s | |
| 2024-01-15 23:15:32,835 44k INFO ====> Epoch: 386, cost 13.89 s | |
| 2024-01-15 23:15:47,037 44k INFO ====> Epoch: 387, cost 14.20 s | |
| 2024-01-15 23:16:01,032 44k INFO ====> Epoch: 388, cost 14.00 s | |
| 2024-01-15 23:16:15,074 44k INFO ====> Epoch: 389, cost 14.04 s | |
| 2024-01-15 23:16:29,043 44k INFO ====> Epoch: 390, cost 13.97 s | |
| 2024-01-15 23:16:43,182 44k INFO ====> Epoch: 391, cost 14.14 s | |
| 2024-01-15 23:16:57,053 44k INFO ====> Epoch: 392, cost 13.87 s | |
| 2024-01-15 23:17:10,865 44k INFO ====> Epoch: 393, cost 13.81 s | |
| 2024-01-15 23:17:24,599 44k INFO ====> Epoch: 394, cost 13.73 s | |
| 2024-01-15 23:17:38,225 44k INFO ====> Epoch: 395, cost 13.63 s | |
| 2024-01-15 23:17:52,636 44k INFO ====> Epoch: 396, cost 14.41 s | |
| 2024-01-15 23:18:06,643 44k INFO ====> Epoch: 397, cost 14.01 s | |
| 2024-01-15 23:18:20,464 44k INFO ====> Epoch: 398, cost 13.82 s | |
| 2024-01-15 23:18:34,267 44k INFO ====> Epoch: 399, cost 13.80 s | |
| 2024-01-15 23:18:47,300 44k INFO Train Epoch: 400 [92%] | |
| 2024-01-15 23:18:47,301 44k INFO Losses: [2.6214396953582764, 1.9308719635009766, 5.818599700927734, 12.396209716796875, 0.7385833859443665], step: 5200, lr: 9.513453698368834e-05, reference_loss: 23.50570297241211 | |
| 2024-01-15 23:18:56,594 44k INFO Saving model and optimizer state at iteration 400 to ./logs/44k/G_5200.pth | |
| 2024-01-15 23:18:58,369 44k INFO Saving model and optimizer state at iteration 400 to ./logs/44k/D_5200.pth | |
| 2024-01-15 23:18:59,069 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_1200.pth | |
| 2024-01-15 23:18:59,081 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_1200.pth | |
| 2024-01-15 23:19:00,242 44k INFO ====> Epoch: 400, cost 25.97 s | |
| 2024-01-15 23:19:14,081 44k INFO ====> Epoch: 401, cost 13.84 s | |
| 2024-01-15 23:19:27,874 44k INFO ====> Epoch: 402, cost 13.79 s | |
| 2024-01-15 23:19:41,717 44k INFO ====> Epoch: 403, cost 13.84 s | |
| 2024-01-15 23:19:55,449 44k INFO ====> Epoch: 404, cost 13.73 s | |
| 2024-01-15 23:20:09,272 44k INFO ====> Epoch: 405, cost 13.82 s | |
| 2024-01-15 23:20:23,162 44k INFO ====> Epoch: 406, cost 13.89 s | |
| 2024-01-15 23:20:37,177 44k INFO ====> Epoch: 407, cost 14.01 s | |
| 2024-01-15 23:20:51,190 44k INFO ====> Epoch: 408, cost 14.01 s | |
| 2024-01-15 23:21:05,034 44k INFO ====> Epoch: 409, cost 13.84 s | |
| 2024-01-15 23:21:19,009 44k INFO ====> Epoch: 410, cost 13.98 s | |
| 2024-01-15 23:21:32,566 44k INFO ====> Epoch: 411, cost 13.56 s | |
| 2024-01-15 23:21:46,280 44k INFO ====> Epoch: 412, cost 13.71 s | |
| 2024-01-15 23:22:00,041 44k INFO ====> Epoch: 413, cost 13.76 s | |
| 2024-01-15 23:22:13,733 44k INFO ====> Epoch: 414, cost 13.69 s | |
| 2024-01-15 23:22:27,391 44k INFO ====> Epoch: 415, cost 13.66 s | |
| 2024-01-15 23:22:36,560 44k INFO Train Epoch: 416 [31%] | |
| 2024-01-15 23:22:36,561 44k INFO Losses: [2.9800407886505127, 1.4981071949005127, 1.4160430431365967, 6.778883457183838, 0.32756587862968445], step: 5400, lr: 9.494444618296661e-05, reference_loss: 13.000640869140625 | |
| 2024-01-15 23:22:45,401 44k INFO Saving model and optimizer state at iteration 416 to ./logs/44k/G_5400.pth | |
| 2024-01-15 23:22:47,358 44k INFO Saving model and optimizer state at iteration 416 to ./logs/44k/D_5400.pth | |
| 2024-01-15 23:22:48,065 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_1400.pth | |
| 2024-01-15 23:22:48,079 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_1400.pth | |
| 2024-01-15 23:22:52,612 44k INFO ====> Epoch: 416, cost 25.22 s | |
| 2024-01-15 23:23:06,356 44k INFO ====> Epoch: 417, cost 13.74 s | |
| 2024-01-15 23:23:20,000 44k INFO ====> Epoch: 418, cost 13.64 s | |
| 2024-01-15 23:23:33,698 44k INFO ====> Epoch: 419, cost 13.70 s | |
| 2024-01-15 23:23:47,372 44k INFO ====> Epoch: 420, cost 13.67 s | |
| 2024-01-15 23:24:00,958 44k INFO ====> Epoch: 421, cost 13.59 s | |
| 2024-01-15 23:24:14,855 44k INFO ====> Epoch: 422, cost 13.90 s | |
| 2024-01-15 23:24:29,027 44k INFO ====> Epoch: 423, cost 14.17 s | |
| 2024-01-15 23:24:42,697 44k INFO ====> Epoch: 424, cost 13.67 s | |
| 2024-01-15 23:24:56,698 44k INFO ====> Epoch: 425, cost 14.00 s | |
| 2024-01-15 23:25:10,358 44k INFO ====> Epoch: 426, cost 13.66 s | |
| 2024-01-15 23:25:24,226 44k INFO ====> Epoch: 427, cost 13.87 s | |
| 2024-01-15 23:25:37,963 44k INFO ====> Epoch: 428, cost 13.74 s | |
| 2024-01-15 23:25:51,950 44k INFO ====> Epoch: 429, cost 13.99 s | |
| 2024-01-15 23:26:05,894 44k INFO ====> Epoch: 430, cost 13.94 s | |
| 2024-01-15 23:26:17,266 44k INFO Train Epoch: 431 [69%] | |
| 2024-01-15 23:26:17,267 44k INFO Losses: [2.5180106163024902, 2.3954765796661377, 6.776612758636475, 18.466999053955078, 1.1452800035476685], step: 5600, lr: 9.47665810302627e-05, reference_loss: 31.302379608154297 | |
| 2024-01-15 23:26:26,133 44k INFO Saving model and optimizer state at iteration 431 to ./logs/44k/G_5600.pth | |
| 2024-01-15 23:26:27,817 44k INFO Saving model and optimizer state at iteration 431 to ./logs/44k/D_5600.pth | |
| 2024-01-15 23:26:28,476 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_1600.pth | |
| 2024-01-15 23:26:28,488 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_1600.pth | |
| 2024-01-15 23:26:30,900 44k INFO ====> Epoch: 431, cost 25.01 s | |
| 2024-01-15 23:26:44,743 44k INFO ====> Epoch: 432, cost 13.84 s | |
| 2024-01-15 23:26:59,058 44k INFO ====> Epoch: 433, cost 14.32 s | |
| 2024-01-15 23:27:12,717 44k INFO ====> Epoch: 434, cost 13.66 s | |
| 2024-01-15 23:27:26,876 44k INFO ====> Epoch: 435, cost 14.16 s | |
| 2024-01-15 23:27:41,757 44k INFO ====> Epoch: 436, cost 14.88 s | |
| 2024-01-15 23:27:56,383 44k INFO ====> Epoch: 437, cost 14.63 s | |
| 2024-01-15 23:28:10,345 44k INFO ====> Epoch: 438, cost 13.96 s | |
| 2024-01-15 23:28:24,400 44k INFO ====> Epoch: 439, cost 14.05 s | |
| 2024-01-15 23:28:38,393 44k INFO ====> Epoch: 440, cost 13.99 s | |
| 2024-01-15 23:28:52,690 44k INFO ====> Epoch: 441, cost 14.30 s | |
| 2024-01-15 23:29:06,629 44k INFO ====> Epoch: 442, cost 13.94 s | |
| 2024-01-15 23:29:20,639 44k INFO ====> Epoch: 443, cost 14.01 s | |
| 2024-01-15 23:29:35,271 44k INFO ====> Epoch: 444, cost 14.63 s | |
| 2024-01-15 23:29:49,387 44k INFO ====> Epoch: 445, cost 14.12 s | |
| 2024-01-15 23:30:03,357 44k INFO ====> Epoch: 446, cost 13.97 s | |
| 2024-01-15 23:30:11,461 44k INFO Train Epoch: 447 [8%] | |
| 2024-01-15 23:30:11,462 44k INFO Losses: [2.5849359035491943, 2.128192663192749, 5.655056953430176, 15.363283157348633, 0.6317979097366333], step: 5800, lr: 9.457722545193272e-05, reference_loss: 26.363265991210938 | |
| 2024-01-15 23:30:21,138 44k INFO Saving model and optimizer state at iteration 447 to ./logs/44k/G_5800.pth | |
| 2024-01-15 23:30:22,894 44k INFO Saving model and optimizer state at iteration 447 to ./logs/44k/D_5800.pth | |
| 2024-01-15 23:30:23,594 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_1800.pth | |
| 2024-01-15 23:30:23,610 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_1800.pth | |
| 2024-01-15 23:30:29,493 44k INFO ====> Epoch: 447, cost 26.14 s | |
| 2024-01-15 23:30:43,331 44k INFO ====> Epoch: 448, cost 13.84 s | |
| 2024-01-15 23:30:57,241 44k INFO ====> Epoch: 449, cost 13.91 s | |
| 2024-01-15 23:31:10,952 44k INFO ====> Epoch: 450, cost 13.71 s | |
| 2024-01-15 23:31:24,901 44k INFO ====> Epoch: 451, cost 13.95 s | |
| 2024-01-15 23:31:39,174 44k INFO ====> Epoch: 452, cost 14.27 s | |
| 2024-01-15 23:31:53,671 44k INFO ====> Epoch: 453, cost 14.50 s | |
| 2024-01-15 23:32:07,623 44k INFO ====> Epoch: 454, cost 13.95 s | |
| 2024-01-15 23:32:21,314 44k INFO ====> Epoch: 455, cost 13.69 s | |
| 2024-01-15 23:32:35,277 44k INFO ====> Epoch: 456, cost 13.96 s | |
| 2024-01-15 23:32:49,025 44k INFO ====> Epoch: 457, cost 13.75 s | |
| 2024-01-15 23:33:02,699 44k INFO ====> Epoch: 458, cost 13.67 s | |
| 2024-01-15 23:33:16,564 44k INFO ====> Epoch: 459, cost 13.86 s | |
| 2024-01-15 23:33:30,374 44k INFO ====> Epoch: 460, cost 13.81 s | |
| 2024-01-15 23:33:44,177 44k INFO ====> Epoch: 461, cost 13.80 s | |
| 2024-01-15 23:33:53,976 44k INFO Train Epoch: 462 [46%] | |
| 2024-01-15 23:33:53,977 44k INFO Losses: [2.3067667484283447, 2.421813726425171, 6.077267646789551, 14.151480674743652, 0.394141286611557], step: 6000, lr: 9.440004823595418e-05, reference_loss: 25.351470947265625 | |
| 2024-01-15 23:34:02,902 44k INFO Saving model and optimizer state at iteration 462 to ./logs/44k/G_6000.pth | |
| 2024-01-15 23:34:04,586 44k INFO Saving model and optimizer state at iteration 462 to ./logs/44k/D_6000.pth | |
| 2024-01-15 23:34:05,253 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_2000.pth | |
| 2024-01-15 23:34:05,269 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_2000.pth | |
| 2024-01-15 23:34:08,872 44k INFO ====> Epoch: 462, cost 24.69 s | |
| 2024-01-15 23:34:23,181 44k INFO ====> Epoch: 463, cost 14.31 s | |
| 2024-01-15 23:34:36,658 44k INFO ====> Epoch: 464, cost 13.48 s | |
| 2024-01-15 23:34:50,331 44k INFO ====> Epoch: 465, cost 13.67 s | |
| 2024-01-15 23:35:04,160 44k INFO ====> Epoch: 466, cost 13.83 s | |
| 2024-01-15 23:35:17,967 44k INFO ====> Epoch: 467, cost 13.81 s | |
| 2024-01-15 23:35:31,470 44k INFO ====> Epoch: 468, cost 13.50 s | |
| 2024-01-15 23:35:45,397 44k INFO ====> Epoch: 469, cost 13.93 s | |
| 2024-01-15 23:35:59,186 44k INFO ====> Epoch: 470, cost 13.79 s | |
| 2024-01-15 23:36:12,956 44k INFO ====> Epoch: 471, cost 13.77 s | |
| 2024-01-15 23:36:26,762 44k INFO ====> Epoch: 472, cost 13.81 s | |
| 2024-01-15 23:36:41,214 44k INFO ====> Epoch: 473, cost 14.45 s | |
| 2024-01-15 23:36:54,740 44k INFO ====> Epoch: 474, cost 13.53 s | |
| 2024-01-15 23:37:08,759 44k INFO ====> Epoch: 475, cost 14.02 s | |
| 2024-01-15 23:37:22,420 44k INFO ====> Epoch: 476, cost 13.66 s | |
| 2024-01-15 23:37:34,730 44k INFO Train Epoch: 477 [85%] | |
| 2024-01-15 23:37:34,731 44k INFO Losses: [2.5475757122039795, 2.277386426925659, 5.340463638305664, 12.497882843017578, 0.919982373714447], step: 6200, lr: 9.422320293673162e-05, reference_loss: 23.58329200744629 | |
| 2024-01-15 23:37:44,035 44k INFO Saving model and optimizer state at iteration 477 to ./logs/44k/G_6200.pth | |
| 2024-01-15 23:37:45,687 44k INFO Saving model and optimizer state at iteration 477 to ./logs/44k/D_6200.pth | |
| 2024-01-15 23:37:46,368 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_2200.pth | |
| 2024-01-15 23:37:46,383 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_2200.pth | |
| 2024-01-15 23:37:47,961 44k INFO ====> Epoch: 477, cost 25.54 s | |
| 2024-01-15 23:38:01,773 44k INFO ====> Epoch: 478, cost 13.81 s | |
| 2024-01-15 23:38:15,635 44k INFO ====> Epoch: 479, cost 13.86 s | |
| 2024-01-15 23:38:29,565 44k INFO ====> Epoch: 480, cost 13.93 s | |
| 2024-01-15 23:38:43,031 44k INFO ====> Epoch: 481, cost 13.47 s | |
| 2024-01-15 23:38:56,990 44k INFO ====> Epoch: 482, cost 13.96 s | |
| 2024-01-15 23:39:11,231 44k INFO ====> Epoch: 483, cost 14.24 s | |
| 2024-01-15 23:39:24,780 44k INFO ====> Epoch: 484, cost 13.55 s | |
| 2024-01-15 23:39:39,066 44k INFO ====> Epoch: 485, cost 14.29 s | |
| 2024-01-15 23:39:53,466 44k INFO ====> Epoch: 486, cost 14.40 s | |
| 2024-01-15 23:40:07,219 44k INFO ====> Epoch: 487, cost 13.75 s | |
| 2024-01-15 23:40:21,143 44k INFO ====> Epoch: 488, cost 13.92 s | |
| 2024-01-15 23:40:34,884 44k INFO ====> Epoch: 489, cost 13.74 s | |
| 2024-01-15 23:40:48,592 44k INFO ====> Epoch: 490, cost 13.71 s | |
| 2024-01-15 23:41:02,306 44k INFO ====> Epoch: 491, cost 13.71 s | |
| 2024-01-15 23:41:16,179 44k INFO ====> Epoch: 492, cost 13.87 s | |
| 2024-01-15 23:41:24,984 44k INFO Train Epoch: 493 [23%] | |
| 2024-01-15 23:41:24,985 44k INFO Losses: [2.34938645362854, 2.553781032562256, 7.934936046600342, 15.768722534179688, 0.8733723163604736], step: 6400, lr: 9.403493309634886e-05, reference_loss: 29.480199813842773 | |
| 2024-01-15 23:41:33,874 44k INFO Saving model and optimizer state at iteration 493 to ./logs/44k/G_6400.pth | |
| 2024-01-15 23:41:35,814 44k INFO Saving model and optimizer state at iteration 493 to ./logs/44k/D_6400.pth | |
| 2024-01-15 23:41:36,485 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_2400.pth | |
| 2024-01-15 23:41:36,497 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_2400.pth | |
| 2024-01-15 23:41:41,418 44k INFO ====> Epoch: 493, cost 25.24 s | |
| 2024-01-15 23:41:55,285 44k INFO ====> Epoch: 494, cost 13.87 s | |
| 2024-01-15 23:42:08,983 44k INFO ====> Epoch: 495, cost 13.70 s | |
| 2024-01-15 23:42:22,620 44k INFO ====> Epoch: 496, cost 13.64 s | |
| 2024-01-15 23:42:36,666 44k INFO ====> Epoch: 497, cost 14.05 s | |
| 2024-01-15 23:42:50,452 44k INFO ====> Epoch: 498, cost 13.79 s | |
| 2024-01-15 23:43:04,349 44k INFO ====> Epoch: 499, cost 13.90 s | |
| 2024-01-15 23:43:18,094 44k INFO ====> Epoch: 500, cost 13.75 s | |
| 2024-01-15 23:43:32,131 44k INFO ====> Epoch: 501, cost 14.04 s | |
| 2024-01-15 23:43:45,901 44k INFO ====> Epoch: 502, cost 13.77 s | |
| 2024-01-15 23:43:59,464 44k INFO ====> Epoch: 503, cost 13.56 s | |
| 2024-01-15 23:44:13,493 44k INFO ====> Epoch: 504, cost 14.03 s | |
| 2024-01-15 23:44:27,275 44k INFO ====> Epoch: 505, cost 13.78 s | |
| 2024-01-15 23:44:41,543 44k INFO ====> Epoch: 506, cost 14.27 s | |
| 2024-01-15 23:44:55,261 44k INFO ====> Epoch: 507, cost 13.72 s | |
| 2024-01-15 23:45:06,282 44k INFO Train Epoch: 508 [62%] | |
| 2024-01-15 23:45:06,283 44k INFO Losses: [2.4573421478271484, 2.1284384727478027, 5.815243244171143, 12.419376373291016, 0.8875783085823059], step: 6600, lr: 9.385877178932038e-05, reference_loss: 23.707979202270508 | |
| 2024-01-15 23:45:15,589 44k INFO Saving model and optimizer state at iteration 508 to ./logs/44k/G_6600.pth | |
| 2024-01-15 23:45:17,291 44k INFO Saving model and optimizer state at iteration 508 to ./logs/44k/D_6600.pth | |
| 2024-01-15 23:45:17,963 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_2600.pth | |
| 2024-01-15 23:45:17,976 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_2600.pth | |
| 2024-01-15 23:45:20,794 44k INFO ====> Epoch: 508, cost 25.53 s | |
| 2024-01-15 23:45:34,551 44k INFO ====> Epoch: 509, cost 13.76 s | |
| 2024-01-15 23:45:48,352 44k INFO ====> Epoch: 510, cost 13.80 s | |
| 2024-01-15 23:46:02,053 44k INFO ====> Epoch: 511, cost 13.70 s | |
| 2024-01-15 23:46:16,221 44k INFO ====> Epoch: 512, cost 14.17 s | |
| 2024-01-15 23:46:30,050 44k INFO ====> Epoch: 513, cost 13.83 s | |
| 2024-01-15 23:46:43,992 44k INFO ====> Epoch: 514, cost 13.94 s | |
| 2024-01-15 23:46:57,602 44k INFO ====> Epoch: 515, cost 13.61 s | |
| 2024-01-15 23:47:11,490 44k INFO ====> Epoch: 516, cost 13.89 s | |
| 2024-01-15 23:47:25,065 44k INFO ====> Epoch: 517, cost 13.57 s | |
| 2024-01-15 23:47:38,882 44k INFO ====> Epoch: 518, cost 13.82 s | |
| 2024-01-15 23:47:52,505 44k INFO ====> Epoch: 519, cost 13.62 s | |
| 2024-01-15 23:48:06,570 44k INFO ====> Epoch: 520, cost 14.07 s | |
| 2024-01-15 23:48:20,550 44k INFO ====> Epoch: 521, cost 13.98 s | |
| 2024-01-15 23:48:34,169 44k INFO ====> Epoch: 522, cost 13.62 s | |
| 2024-01-15 23:48:48,104 44k INFO ====> Epoch: 523, cost 13.93 s | |
| 2024-01-15 23:48:55,669 44k INFO Train Epoch: 524 [0%] | |
| 2024-01-15 23:48:55,670 44k INFO Losses: [2.514359712600708, 2.636570930480957, 4.1962761878967285, 15.068312644958496, 0.5873104929924011], step: 6800, lr: 9.367123012832248e-05, reference_loss: 25.002830505371094 | |
| 2024-01-15 23:49:04,493 44k INFO Saving model and optimizer state at iteration 524 to ./logs/44k/G_6800.pth | |
| 2024-01-15 23:49:06,231 44k INFO Saving model and optimizer state at iteration 524 to ./logs/44k/D_6800.pth | |
| 2024-01-15 23:49:06,877 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_2800.pth | |
| 2024-01-15 23:49:06,890 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_2800.pth | |
| 2024-01-15 23:49:13,298 44k INFO ====> Epoch: 524, cost 25.19 s | |
| 2024-01-15 23:49:26,883 44k INFO ====> Epoch: 525, cost 13.58 s | |
| 2024-01-15 23:49:40,750 44k INFO ====> Epoch: 526, cost 13.87 s | |
| 2024-01-15 23:49:54,634 44k INFO ====> Epoch: 527, cost 13.88 s | |
| 2024-01-15 23:50:08,394 44k INFO ====> Epoch: 528, cost 13.76 s | |
| 2024-01-15 23:50:22,436 44k INFO ====> Epoch: 529, cost 14.04 s | |
| 2024-01-15 23:50:36,128 44k INFO ====> Epoch: 530, cost 13.69 s | |
| 2024-01-15 23:50:50,368 44k INFO ====> Epoch: 531, cost 14.24 s | |
| 2024-01-15 23:51:03,970 44k INFO ====> Epoch: 532, cost 13.60 s | |
| 2024-01-15 23:51:17,579 44k INFO ====> Epoch: 533, cost 13.61 s | |
| 2024-01-15 23:51:31,360 44k INFO ====> Epoch: 534, cost 13.78 s | |
| 2024-01-15 23:51:45,188 44k INFO ====> Epoch: 535, cost 13.83 s | |
| 2024-01-15 23:51:58,989 44k INFO ====> Epoch: 536, cost 13.80 s | |
| 2024-01-15 23:52:12,545 44k INFO ====> Epoch: 537, cost 13.56 s | |
| 2024-01-15 23:52:26,388 44k INFO ====> Epoch: 538, cost 13.84 s | |
| 2024-01-15 23:52:36,185 44k INFO Train Epoch: 539 [38%] | |
| 2024-01-15 23:52:36,186 44k INFO Losses: [2.461848020553589, 2.3337764739990234, 5.6476545333862305, 14.534675598144531, 0.23286262154579163], step: 7000, lr: 9.349575016798194e-05, reference_loss: 25.210817337036133 | |
| 2024-01-15 23:52:45,061 44k INFO Saving model and optimizer state at iteration 539 to ./logs/44k/G_7000.pth | |
| 2024-01-15 23:52:46,738 44k INFO Saving model and optimizer state at iteration 539 to ./logs/44k/D_7000.pth | |
| 2024-01-15 23:52:47,695 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_3000.pth | |
| 2024-01-15 23:52:47,707 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_3000.pth | |
| 2024-01-15 23:52:51,721 44k INFO ====> Epoch: 539, cost 25.33 s | |
| 2024-01-15 23:53:06,003 44k INFO ====> Epoch: 540, cost 14.28 s | |
| 2024-01-15 23:53:19,840 44k INFO ====> Epoch: 541, cost 13.84 s | |
| 2024-01-15 23:53:33,468 44k INFO ====> Epoch: 542, cost 13.63 s | |
| 2024-01-15 23:53:47,704 44k INFO ====> Epoch: 543, cost 14.24 s | |
| 2024-01-15 23:54:01,330 44k INFO ====> Epoch: 544, cost 13.63 s | |
| 2024-01-15 23:54:14,851 44k INFO ====> Epoch: 545, cost 13.52 s | |
| 2024-01-15 23:54:28,762 44k INFO ====> Epoch: 546, cost 13.91 s | |
| 2024-01-15 23:54:42,469 44k INFO ====> Epoch: 547, cost 13.71 s | |
| 2024-01-15 23:54:56,075 44k INFO ====> Epoch: 548, cost 13.61 s | |
| 2024-01-15 23:55:09,979 44k INFO ====> Epoch: 549, cost 13.90 s | |
| 2024-01-15 23:55:24,332 44k INFO ====> Epoch: 550, cost 14.35 s | |
| 2024-01-15 23:55:38,199 44k INFO ====> Epoch: 551, cost 13.87 s | |
| 2024-01-15 23:55:52,506 44k INFO ====> Epoch: 552, cost 14.31 s | |
| 2024-01-15 23:56:06,235 44k INFO ====> Epoch: 553, cost 13.73 s | |
| 2024-01-15 23:56:17,922 44k INFO Train Epoch: 554 [77%] | |
| 2024-01-15 23:56:17,925 44k INFO Losses: [2.4399495124816895, 2.468134641647339, 4.353724002838135, 10.982842445373535, 0.17930014431476593], step: 7200, lr: 9.332059894482616e-05, reference_loss: 20.423952102661133 | |
| 2024-01-15 23:56:26,917 44k INFO Saving model and optimizer state at iteration 554 to ./logs/44k/G_7200.pth | |
| 2024-01-15 23:56:28,577 44k INFO Saving model and optimizer state at iteration 554 to ./logs/44k/D_7200.pth | |
| 2024-01-15 23:56:29,245 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_3200.pth | |
| 2024-01-15 23:56:29,257 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_3200.pth | |
| 2024-01-15 23:56:31,269 44k INFO ====> Epoch: 554, cost 25.03 s | |
| 2024-01-15 23:56:45,367 44k INFO ====> Epoch: 555, cost 14.10 s | |
| 2024-01-15 23:56:59,635 44k INFO ====> Epoch: 556, cost 14.27 s | |
| 2024-01-15 23:57:13,379 44k INFO ====> Epoch: 557, cost 13.74 s | |
| 2024-01-15 23:57:27,057 44k INFO ====> Epoch: 558, cost 13.68 s | |
| 2024-01-15 23:57:40,677 44k INFO ====> Epoch: 559, cost 13.62 s | |
| 2024-01-15 23:57:54,591 44k INFO ====> Epoch: 560, cost 13.91 s | |
| 2024-01-15 23:58:08,728 44k INFO ====> Epoch: 561, cost 14.14 s | |
| 2024-01-15 23:58:22,412 44k INFO ====> Epoch: 562, cost 13.68 s | |
| 2024-01-15 23:58:36,280 44k INFO ====> Epoch: 563, cost 13.87 s | |
| 2024-01-15 23:58:50,449 44k INFO ====> Epoch: 564, cost 14.17 s | |
| 2024-01-15 23:59:04,436 44k INFO ====> Epoch: 565, cost 13.99 s | |
| 2024-01-15 23:59:18,858 44k INFO ====> Epoch: 566, cost 14.42 s | |
| 2024-01-15 23:59:32,840 44k INFO ====> Epoch: 567, cost 13.98 s | |
| 2024-01-15 23:59:47,030 44k INFO ====> Epoch: 568, cost 14.19 s | |
| 2024-01-16 00:00:00,770 44k INFO ====> Epoch: 569, cost 13.74 s | |
| 2024-01-16 00:00:09,488 44k INFO Train Epoch: 570 [15%] | |
| 2024-01-16 00:00:09,489 44k INFO Losses: [2.718153238296509, 1.8314827680587769, 4.374455451965332, 10.22539234161377, 0.7499396204948425], step: 7400, lr: 9.313413262103149e-05, reference_loss: 19.89942169189453 | |
| 2024-01-16 00:00:18,513 44k INFO Saving model and optimizer state at iteration 570 to ./logs/44k/G_7400.pth | |
| 2024-01-16 00:00:20,517 44k INFO Saving model and optimizer state at iteration 570 to ./logs/44k/D_7400.pth | |
| 2024-01-16 00:00:21,180 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_3400.pth | |
| 2024-01-16 00:00:21,192 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_3400.pth | |
| 2024-01-16 00:00:26,658 44k INFO ====> Epoch: 570, cost 25.89 s | |
| 2024-01-16 00:00:40,287 44k INFO ====> Epoch: 571, cost 13.63 s | |
| 2024-01-16 00:00:54,273 44k INFO ====> Epoch: 572, cost 13.99 s | |
| 2024-01-16 00:01:08,583 44k INFO ====> Epoch: 573, cost 14.31 s | |
| 2024-01-16 00:01:22,677 44k INFO ====> Epoch: 574, cost 14.09 s | |
| 2024-01-16 00:01:36,882 44k INFO ====> Epoch: 575, cost 14.21 s | |
| 2024-01-16 00:01:50,950 44k INFO ====> Epoch: 576, cost 14.07 s | |
| 2024-01-16 00:02:04,474 44k INFO ====> Epoch: 577, cost 13.52 s | |
| 2024-01-16 00:02:18,142 44k INFO ====> Epoch: 578, cost 13.67 s | |
| 2024-01-16 00:02:32,513 44k INFO ====> Epoch: 579, cost 14.37 s | |
| 2024-01-16 00:02:46,700 44k INFO ====> Epoch: 580, cost 14.19 s | |
| 2024-01-16 00:03:00,925 44k INFO ====> Epoch: 581, cost 14.23 s | |
| 2024-01-16 00:03:14,584 44k INFO ====> Epoch: 582, cost 13.66 s | |
| 2024-01-16 00:03:28,361 44k INFO ====> Epoch: 583, cost 13.78 s | |
| 2024-01-16 00:03:42,204 44k INFO ====> Epoch: 584, cost 13.84 s | |
| 2024-01-16 00:03:52,661 44k INFO Train Epoch: 585 [54%] | |
| 2024-01-16 00:03:52,662 44k INFO Losses: [2.436642646789551, 2.6082472801208496, 4.550248146057129, 10.960485458374023, 0.4042876660823822], step: 7600, lr: 9.295965883781867e-05, reference_loss: 20.959911346435547 | |
| 2024-01-16 00:04:01,735 44k INFO Saving model and optimizer state at iteration 585 to ./logs/44k/G_7600.pth | |
| 2024-01-16 00:04:03,475 44k INFO Saving model and optimizer state at iteration 585 to ./logs/44k/D_7600.pth | |
| 2024-01-16 00:04:04,231 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_3600.pth | |
| 2024-01-16 00:04:04,250 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_3600.pth | |
| 2024-01-16 00:04:07,562 44k INFO ====> Epoch: 585, cost 25.36 s | |
| 2024-01-16 00:04:21,190 44k INFO ====> Epoch: 586, cost 13.63 s | |
| 2024-01-16 00:04:34,845 44k INFO ====> Epoch: 587, cost 13.65 s | |
| 2024-01-16 00:04:48,819 44k INFO ====> Epoch: 588, cost 13.97 s | |
| 2024-01-16 00:05:03,256 44k INFO ====> Epoch: 589, cost 14.44 s | |
| 2024-01-16 00:05:17,284 44k INFO ====> Epoch: 590, cost 14.03 s | |
| 2024-01-16 00:05:31,834 44k INFO ====> Epoch: 591, cost 14.55 s | |
| 2024-01-16 00:05:45,413 44k INFO ====> Epoch: 592, cost 13.58 s | |
| 2024-01-16 00:05:58,969 44k INFO ====> Epoch: 593, cost 13.56 s | |
| 2024-01-16 00:06:12,561 44k INFO ====> Epoch: 594, cost 13.59 s | |
| 2024-01-16 00:06:26,348 44k INFO ====> Epoch: 595, cost 13.79 s | |
| 2024-01-16 00:06:40,279 44k INFO ====> Epoch: 596, cost 13.93 s | |
| 2024-01-16 00:06:54,161 44k INFO ====> Epoch: 597, cost 13.88 s | |
| 2024-01-16 00:07:08,058 44k INFO ====> Epoch: 598, cost 13.90 s | |
| 2024-01-16 00:07:21,831 44k INFO ====> Epoch: 599, cost 13.77 s | |
| 2024-01-16 00:07:34,397 44k INFO Train Epoch: 600 [92%] | |
| 2024-01-16 00:07:34,398 44k INFO Losses: [2.5466630458831787, 2.1453158855438232, 5.968946933746338, 12.317658424377441, 0.6443919539451599], step: 7800, lr: 9.27855119068583e-05, reference_loss: 23.622976303100586 | |
| 2024-01-16 00:07:44,014 44k INFO Saving model and optimizer state at iteration 600 to ./logs/44k/G_7800.pth | |
| 2024-01-16 00:07:45,700 44k INFO Saving model and optimizer state at iteration 600 to ./logs/44k/D_7800.pth | |
| 2024-01-16 00:07:46,406 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_3800.pth | |
| 2024-01-16 00:07:46,468 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_3800.pth | |
| 2024-01-16 00:07:47,629 44k INFO ====> Epoch: 600, cost 25.80 s | |
| 2024-01-16 00:08:01,482 44k INFO ====> Epoch: 601, cost 13.85 s | |
| 2024-01-16 00:08:15,354 44k INFO ====> Epoch: 602, cost 13.87 s | |
| 2024-01-16 00:08:28,748 44k INFO ====> Epoch: 603, cost 13.39 s | |
| 2024-01-16 00:08:42,448 44k INFO ====> Epoch: 604, cost 13.70 s | |
| 2024-01-16 00:08:56,250 44k INFO ====> Epoch: 605, cost 13.80 s | |
| 2024-01-16 00:09:10,382 44k INFO ====> Epoch: 606, cost 14.13 s | |
| 2024-01-16 00:09:23,771 44k INFO ====> Epoch: 607, cost 13.39 s | |
| 2024-01-16 00:09:37,756 44k INFO ====> Epoch: 608, cost 13.98 s | |
| 2024-01-16 00:09:51,682 44k INFO ====> Epoch: 609, cost 13.93 s | |
| 2024-01-16 00:10:05,623 44k INFO ====> Epoch: 610, cost 13.94 s | |
| 2024-01-16 00:10:19,631 44k INFO ====> Epoch: 611, cost 14.01 s | |
| 2024-01-16 00:10:33,961 44k INFO ====> Epoch: 612, cost 14.33 s | |
| 2024-01-16 00:10:47,716 44k INFO ====> Epoch: 613, cost 13.75 s | |
| 2024-01-16 00:11:01,921 44k INFO ====> Epoch: 614, cost 14.21 s | |
| 2024-01-16 00:11:16,116 44k INFO ====> Epoch: 615, cost 14.19 s | |
| 2024-01-16 00:11:25,568 44k INFO Train Epoch: 616 [31%] | |
| 2024-01-16 00:11:25,569 44k INFO Losses: [3.0106918811798096, 1.674297571182251, 1.5688183307647705, 6.541810512542725, 0.3267265558242798], step: 8000, lr: 9.260011475443641e-05, reference_loss: 13.122344970703125 | |
| 2024-01-16 00:11:34,597 44k INFO Saving model and optimizer state at iteration 616 to ./logs/44k/G_8000.pth | |
| 2024-01-16 00:11:36,300 44k INFO Saving model and optimizer state at iteration 616 to ./logs/44k/D_8000.pth | |
| 2024-01-16 00:11:37,047 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_4000.pth | |
| 2024-01-16 00:11:37,102 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_4000.pth | |
| 2024-01-16 00:11:41,637 44k INFO ====> Epoch: 616, cost 25.52 s | |
| 2024-01-16 00:11:55,758 44k INFO ====> Epoch: 617, cost 14.12 s | |
| 2024-01-16 00:12:09,514 44k INFO ====> Epoch: 618, cost 13.76 s | |
| 2024-01-16 00:12:23,468 44k INFO ====> Epoch: 619, cost 13.95 s | |
| 2024-01-16 00:12:37,040 44k INFO ====> Epoch: 620, cost 13.57 s | |
| 2024-01-16 00:12:50,765 44k INFO ====> Epoch: 621, cost 13.72 s | |
| 2024-01-16 00:13:04,825 44k INFO ====> Epoch: 622, cost 14.06 s | |
| 2024-01-16 00:13:18,558 44k INFO ====> Epoch: 623, cost 13.73 s | |
| 2024-01-16 00:13:32,247 44k INFO ====> Epoch: 624, cost 13.69 s | |
| 2024-01-16 00:13:45,895 44k INFO ====> Epoch: 625, cost 13.65 s | |
| 2024-01-16 00:13:59,452 44k INFO ====> Epoch: 626, cost 13.56 s | |
| 2024-01-16 00:14:13,422 44k INFO ====> Epoch: 627, cost 13.97 s | |
| 2024-01-16 00:14:27,098 44k INFO ====> Epoch: 628, cost 13.68 s | |
| 2024-01-16 00:14:41,172 44k INFO ====> Epoch: 629, cost 14.07 s | |
| 2024-01-16 00:14:54,882 44k INFO ====> Epoch: 630, cost 13.71 s | |
| 2024-01-16 00:15:06,203 44k INFO Train Epoch: 631 [69%] | |
| 2024-01-16 00:15:06,204 44k INFO Losses: [1.9396860599517822, 2.7911624908447266, 8.506006240844727, 18.617795944213867, 1.0407284498214722], step: 8200, lr: 9.242664137907478e-05, reference_loss: 32.89537811279297 | |
| 2024-01-16 00:15:15,360 44k INFO Saving model and optimizer state at iteration 631 to ./logs/44k/G_8200.pth | |
| 2024-01-16 00:15:17,053 44k INFO Saving model and optimizer state at iteration 631 to ./logs/44k/D_8200.pth | |
| 2024-01-16 00:15:17,788 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_4200.pth | |
| 2024-01-16 00:15:17,856 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_4200.pth | |
| 2024-01-16 00:15:20,319 44k INFO ====> Epoch: 631, cost 25.44 s | |
| 2024-01-16 00:15:34,143 44k INFO ====> Epoch: 632, cost 13.82 s | |
| 2024-01-16 00:15:47,927 44k INFO ====> Epoch: 633, cost 13.78 s | |
| 2024-01-16 00:16:02,753 44k INFO ====> Epoch: 634, cost 14.83 s | |
| 2024-01-16 00:16:16,459 44k INFO ====> Epoch: 635, cost 13.71 s | |
| 2024-01-16 00:16:30,503 44k INFO ====> Epoch: 636, cost 14.04 s | |
| 2024-01-16 00:16:44,256 44k INFO ====> Epoch: 637, cost 13.75 s | |
| 2024-01-16 00:16:58,376 44k INFO ====> Epoch: 638, cost 14.12 s | |
| 2024-01-16 00:17:11,945 44k INFO ====> Epoch: 639, cost 13.57 s | |
| 2024-01-16 00:17:25,863 44k INFO ====> Epoch: 640, cost 13.92 s | |
| 2024-01-16 00:17:39,733 44k INFO ====> Epoch: 641, cost 13.87 s | |
| 2024-01-16 00:17:53,645 44k INFO ====> Epoch: 642, cost 13.91 s | |
| 2024-01-16 00:18:07,614 44k INFO ====> Epoch: 643, cost 13.97 s | |
| 2024-01-16 00:18:21,352 44k INFO ====> Epoch: 644, cost 13.74 s | |
| 2024-01-16 00:18:35,135 44k INFO ====> Epoch: 645, cost 13.78 s | |
| 2024-01-16 00:18:48,813 44k INFO ====> Epoch: 646, cost 13.68 s | |
| 2024-01-16 00:18:56,776 44k INFO Train Epoch: 647 [8%] | |
| 2024-01-16 00:18:56,777 44k INFO Losses: [2.422792911529541, 2.3963093757629395, 5.93314790725708, 15.278427124023438, 0.5465972423553467], step: 8400, lr: 9.224196129521857e-05, reference_loss: 26.577274322509766 | |
| 2024-01-16 00:19:06,009 44k INFO Saving model and optimizer state at iteration 647 to ./logs/44k/G_8400.pth | |
| 2024-01-16 00:19:07,688 44k INFO Saving model and optimizer state at iteration 647 to ./logs/44k/D_8400.pth | |
| 2024-01-16 00:19:08,447 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_4400.pth | |
| 2024-01-16 00:19:08,500 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_4400.pth | |
| 2024-01-16 00:19:14,262 44k INFO ====> Epoch: 647, cost 25.45 s | |
| 2024-01-16 00:19:27,873 44k INFO ====> Epoch: 648, cost 13.61 s | |
| 2024-01-16 00:19:41,592 44k INFO ====> Epoch: 649, cost 13.72 s | |
| 2024-01-16 00:19:55,460 44k INFO ====> Epoch: 650, cost 13.87 s | |
| 2024-01-16 00:20:09,234 44k INFO ====> Epoch: 651, cost 13.77 s | |
| 2024-01-16 00:20:23,083 44k INFO ====> Epoch: 652, cost 13.85 s | |
| 2024-01-16 00:20:36,478 44k INFO ====> Epoch: 653, cost 13.40 s | |
| 2024-01-16 00:20:50,248 44k INFO ====> Epoch: 654, cost 13.77 s | |
| 2024-01-16 00:21:04,158 44k INFO ====> Epoch: 655, cost 13.91 s | |
| 2024-01-16 00:21:17,645 44k INFO ====> Epoch: 656, cost 13.49 s | |
| 2024-01-16 00:21:31,592 44k INFO ====> Epoch: 657, cost 13.95 s | |
| 2024-01-16 00:21:45,289 44k INFO ====> Epoch: 658, cost 13.70 s | |
| 2024-01-16 00:21:59,289 44k INFO ====> Epoch: 659, cost 14.00 s | |
| 2024-01-16 00:22:13,308 44k INFO ====> Epoch: 660, cost 14.02 s | |
| 2024-01-16 00:22:27,018 44k INFO ====> Epoch: 661, cost 13.71 s | |
| 2024-01-16 00:22:36,956 44k INFO Train Epoch: 662 [46%] | |
| 2024-01-16 00:22:36,957 44k INFO Losses: [2.323211431503296, 2.39728045463562, 5.895406723022461, 13.616351127624512, 0.3408372402191162], step: 8600, lr: 9.206915887031564e-05, reference_loss: 24.573087692260742 | |
| 2024-01-16 00:22:45,705 44k INFO Saving model and optimizer state at iteration 662 to ./logs/44k/G_8600.pth | |
| 2024-01-16 00:22:47,383 44k INFO Saving model and optimizer state at iteration 662 to ./logs/44k/D_8600.pth | |
| 2024-01-16 00:22:48,149 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_4600.pth | |
| 2024-01-16 00:22:48,203 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_4600.pth | |
| 2024-01-16 00:22:51,883 44k INFO ====> Epoch: 662, cost 24.87 s | |
| 2024-01-16 00:23:05,660 44k INFO ====> Epoch: 663, cost 13.78 s | |
| 2024-01-16 00:23:19,336 44k INFO ====> Epoch: 664, cost 13.68 s | |
| 2024-01-16 00:23:33,228 44k INFO ====> Epoch: 665, cost 13.89 s | |
| 2024-01-16 00:23:47,105 44k INFO ====> Epoch: 666, cost 13.88 s | |
| 2024-01-16 00:24:00,976 44k INFO ====> Epoch: 667, cost 13.87 s | |
| 2024-01-16 00:24:14,660 44k INFO ====> Epoch: 668, cost 13.68 s | |
| 2024-01-16 00:24:28,582 44k INFO ====> Epoch: 669, cost 13.92 s | |
| 2024-01-16 00:24:42,329 44k INFO ====> Epoch: 670, cost 13.75 s | |
| 2024-01-16 00:24:55,918 44k INFO ====> Epoch: 671, cost 13.59 s | |
| 2024-01-16 00:25:09,810 44k INFO ====> Epoch: 672, cost 13.89 s | |
| 2024-01-16 00:25:23,314 44k INFO ====> Epoch: 673, cost 13.50 s | |
| 2024-01-16 00:25:36,901 44k INFO ====> Epoch: 674, cost 13.59 s | |
| 2024-01-16 00:25:50,655 44k INFO ====> Epoch: 675, cost 13.75 s | |
| 2024-01-16 00:26:04,510 44k INFO ====> Epoch: 676, cost 13.85 s | |
| 2024-01-16 00:26:16,735 44k INFO Train Epoch: 677 [85%] | |
| 2024-01-16 00:26:16,735 44k INFO Losses: [2.5915708541870117, 2.2661232948303223, 5.206692695617676, 12.478229522705078, 0.8665488362312317], step: 8800, lr: 9.189668016660891e-05, reference_loss: 23.409164428710938 | |
| 2024-01-16 00:26:26,302 44k INFO Saving model and optimizer state at iteration 677 to ./logs/44k/G_8800.pth | |
| 2024-01-16 00:26:27,954 44k INFO Saving model and optimizer state at iteration 677 to ./logs/44k/D_8800.pth | |
| 2024-01-16 00:26:28,705 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_4800.pth | |
| 2024-01-16 00:26:28,782 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_4800.pth | |
| 2024-01-16 00:26:30,484 44k INFO ====> Epoch: 677, cost 25.97 s | |
| 2024-01-16 00:26:44,277 44k INFO ====> Epoch: 678, cost 13.79 s | |
| 2024-01-16 00:26:57,939 44k INFO ====> Epoch: 679, cost 13.66 s | |
| 2024-01-16 00:27:11,933 44k INFO ====> Epoch: 680, cost 13.99 s | |
| 2024-01-16 00:27:25,867 44k INFO ====> Epoch: 681, cost 13.93 s | |
| 2024-01-16 00:27:39,398 44k INFO ====> Epoch: 682, cost 13.53 s | |
| 2024-01-16 00:27:52,764 44k INFO ====> Epoch: 683, cost 13.37 s | |
| 2024-01-16 00:28:06,403 44k INFO ====> Epoch: 684, cost 13.64 s | |
| 2024-01-16 00:28:20,895 44k INFO ====> Epoch: 685, cost 14.49 s | |
| 2024-01-16 00:28:34,626 44k INFO ====> Epoch: 686, cost 13.73 s | |
| 2024-01-16 00:28:48,567 44k INFO ====> Epoch: 687, cost 13.94 s | |
| 2024-01-16 00:29:02,286 44k INFO ====> Epoch: 688, cost 13.72 s | |
| 2024-01-16 00:29:16,488 44k INFO ====> Epoch: 689, cost 14.20 s | |
| 2024-01-16 00:29:30,870 44k INFO ====> Epoch: 690, cost 14.38 s | |
| 2024-01-16 00:29:44,745 44k INFO ====> Epoch: 691, cost 13.88 s | |
| 2024-01-16 00:29:58,556 44k INFO ====> Epoch: 692, cost 13.81 s | |
| 2024-01-16 00:30:07,357 44k INFO Train Epoch: 693 [23%] | |
| 2024-01-16 00:30:07,358 44k INFO Losses: [2.435067892074585, 2.6036155223846436, 7.455489158630371, 15.206668853759766, 0.8378464579582214], step: 9000, lr: 9.171305901207978e-05, reference_loss: 28.53868865966797 | |
| 2024-01-16 00:30:16,429 44k INFO Saving model and optimizer state at iteration 693 to ./logs/44k/G_9000.pth | |
| 2024-01-16 00:30:18,196 44k INFO Saving model and optimizer state at iteration 693 to ./logs/44k/D_9000.pth | |
| 2024-01-16 00:30:18,950 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_5000.pth | |
| 2024-01-16 00:30:19,004 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_5000.pth | |
| 2024-01-16 00:30:23,979 44k INFO ====> Epoch: 693, cost 25.42 s | |
| 2024-01-16 00:30:38,342 44k INFO ====> Epoch: 694, cost 14.36 s | |
| 2024-01-16 00:30:52,457 44k INFO ====> Epoch: 695, cost 14.11 s | |
| 2024-01-16 00:31:06,944 44k INFO ====> Epoch: 696, cost 14.49 s | |
| 2024-01-16 00:31:21,116 44k INFO ====> Epoch: 697, cost 14.17 s | |
| 2024-01-16 00:31:34,884 44k INFO ====> Epoch: 698, cost 13.77 s | |
| 2024-01-16 00:31:48,774 44k INFO ====> Epoch: 699, cost 13.89 s | |
| 2024-01-16 00:32:02,332 44k INFO ====> Epoch: 700, cost 13.56 s | |
| 2024-01-16 00:32:16,336 44k INFO ====> Epoch: 701, cost 14.00 s | |
| 2024-01-16 00:32:30,087 44k INFO ====> Epoch: 702, cost 13.75 s | |
| 2024-01-16 00:32:43,378 44k INFO ====> Epoch: 703, cost 13.29 s | |
| 2024-01-16 00:32:57,120 44k INFO ====> Epoch: 704, cost 13.74 s | |
| 2024-01-16 00:33:10,851 44k INFO ====> Epoch: 705, cost 13.73 s | |
| 2024-01-16 00:33:25,490 44k INFO ====> Epoch: 706, cost 14.64 s | |
| 2024-01-16 00:33:39,731 44k INFO ====> Epoch: 707, cost 14.24 s | |
| 2024-01-16 00:33:51,139 44k INFO Train Epoch: 708 [62%] | |
| 2024-01-16 00:33:51,140 44k INFO Losses: [2.327690362930298, 2.4594225883483887, 6.55635929107666, 12.472406387329102, 0.7784334421157837], step: 9200, lr: 9.154124741169722e-05, reference_loss: 24.59431266784668 | |
| 2024-01-16 00:34:01,184 44k INFO Saving model and optimizer state at iteration 708 to ./logs/44k/G_9200.pth | |
| 2024-01-16 00:34:02,931 44k INFO Saving model and optimizer state at iteration 708 to ./logs/44k/D_9200.pth | |
| 2024-01-16 00:34:03,757 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_5200.pth | |
| 2024-01-16 00:34:03,775 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_5200.pth | |
| 2024-01-16 00:34:06,775 44k INFO ====> Epoch: 708, cost 27.04 s | |
| 2024-01-16 00:34:20,689 44k INFO ====> Epoch: 709, cost 13.91 s | |
| 2024-01-16 00:34:34,767 44k INFO ====> Epoch: 710, cost 14.08 s | |
| 2024-01-16 00:34:49,402 44k INFO ====> Epoch: 711, cost 14.63 s | |
| 2024-01-16 00:35:02,686 44k INFO ====> Epoch: 712, cost 13.28 s | |
| 2024-01-16 00:35:16,077 44k INFO ====> Epoch: 713, cost 13.39 s | |
| 2024-01-16 00:35:29,112 44k INFO ====> Epoch: 714, cost 13.03 s | |
| 2024-01-16 00:35:42,079 44k INFO ====> Epoch: 715, cost 12.97 s | |
| 2024-01-16 00:35:55,328 44k INFO ====> Epoch: 716, cost 13.25 s | |
| 2024-01-16 00:36:08,543 44k INFO ====> Epoch: 717, cost 13.22 s | |
| 2024-01-16 00:36:21,440 44k INFO ====> Epoch: 718, cost 12.90 s | |
| 2024-01-16 00:36:34,388 44k INFO ====> Epoch: 719, cost 12.95 s | |
| 2024-01-16 00:36:47,251 44k INFO ====> Epoch: 720, cost 12.86 s | |
| 2024-01-16 00:37:00,751 44k INFO ====> Epoch: 721, cost 13.50 s | |
| 2024-01-16 00:37:14,025 44k INFO ====> Epoch: 722, cost 13.27 s | |
| 2024-01-16 00:37:27,647 44k INFO ====> Epoch: 723, cost 13.62 s | |
| 2024-01-16 00:37:35,066 44k INFO Train Epoch: 724 [0%] | |
| 2024-01-16 00:37:35,067 44k INFO Losses: [2.7452526092529297, 2.1328091621398926, 3.7283217906951904, 14.09737777709961, 0.47574466466903687], step: 9400, lr: 9.13583364566301e-05, reference_loss: 23.179506301879883 | |
| 2024-01-16 00:37:43,631 44k INFO Saving model and optimizer state at iteration 724 to ./logs/44k/G_9400.pth | |
| 2024-01-16 00:37:45,239 44k INFO Saving model and optimizer state at iteration 724 to ./logs/44k/D_9400.pth | |
| 2024-01-16 00:37:45,885 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_5400.pth | |
| 2024-01-16 00:37:45,915 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_5400.pth | |
| 2024-01-16 00:37:51,933 44k INFO ====> Epoch: 724, cost 24.29 s | |
| 2024-01-16 00:38:05,129 44k INFO ====> Epoch: 725, cost 13.20 s | |
| 2024-01-16 00:38:18,132 44k INFO ====> Epoch: 726, cost 13.00 s | |
| 2024-01-16 00:38:30,978 44k INFO ====> Epoch: 727, cost 12.85 s | |
| 2024-01-16 00:38:44,139 44k INFO ====> Epoch: 728, cost 13.16 s | |
| 2024-01-16 00:38:57,961 44k INFO ====> Epoch: 729, cost 13.82 s | |
| 2024-01-16 00:39:11,362 44k INFO ====> Epoch: 730, cost 13.40 s | |
| 2024-01-16 00:39:24,597 44k INFO ====> Epoch: 731, cost 13.23 s | |
| 2024-01-16 00:39:37,706 44k INFO ====> Epoch: 732, cost 13.11 s | |
| 2024-01-16 00:39:50,895 44k INFO ====> Epoch: 733, cost 13.19 s | |
| 2024-01-16 00:40:04,570 44k INFO ====> Epoch: 734, cost 13.67 s | |
| 2024-01-16 00:40:17,732 44k INFO ====> Epoch: 735, cost 13.16 s | |
| 2024-01-16 00:40:31,033 44k INFO ====> Epoch: 736, cost 13.30 s | |
| 2024-01-16 00:40:44,750 44k INFO ====> Epoch: 737, cost 13.72 s | |
| 2024-01-16 00:40:58,567 44k INFO ====> Epoch: 738, cost 13.82 s | |
| 2024-01-16 00:41:08,533 44k INFO Train Epoch: 739 [38%] | |
| 2024-01-16 00:41:08,534 44k INFO Losses: [2.477975845336914, 2.4138782024383545, 5.511815547943115, 14.362266540527344, 0.18771636486053467], step: 9600, lr: 9.118718937938746e-05, reference_loss: 24.953651428222656 | |
| 2024-01-16 00:41:17,008 44k INFO Saving model and optimizer state at iteration 739 to ./logs/44k/G_9600.pth | |
| 2024-01-16 00:41:18,746 44k INFO Saving model and optimizer state at iteration 739 to ./logs/44k/D_9600.pth | |
| 2024-01-16 00:41:19,424 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_5600.pth | |
| 2024-01-16 00:41:19,437 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_5600.pth | |
| 2024-01-16 00:41:23,368 44k INFO ====> Epoch: 739, cost 24.80 s | |
| 2024-01-16 00:41:36,809 44k INFO ====> Epoch: 740, cost 13.44 s | |
| 2024-01-16 00:41:50,167 44k INFO ====> Epoch: 741, cost 13.36 s | |
| 2024-01-16 00:42:03,389 44k INFO ====> Epoch: 742, cost 13.22 s | |
| 2024-01-16 00:42:16,565 44k INFO ====> Epoch: 743, cost 13.18 s | |
| 2024-01-16 00:42:29,751 44k INFO ====> Epoch: 744, cost 13.19 s | |
| 2024-01-16 00:42:43,059 44k INFO ====> Epoch: 745, cost 13.31 s | |
| 2024-01-16 00:42:56,826 44k INFO ====> Epoch: 746, cost 13.77 s | |
| 2024-01-16 00:43:10,025 44k INFO ====> Epoch: 747, cost 13.20 s | |
| 2024-01-16 00:43:23,083 44k INFO ====> Epoch: 748, cost 13.06 s | |
| 2024-01-16 00:43:36,294 44k INFO ====> Epoch: 749, cost 13.21 s | |
| 2024-01-16 00:43:49,449 44k INFO ====> Epoch: 750, cost 13.16 s | |
| 2024-01-16 00:44:02,722 44k INFO ====> Epoch: 751, cost 13.27 s | |
| 2024-01-16 00:44:15,831 44k INFO ====> Epoch: 752, cost 13.11 s | |
| 2024-01-16 00:44:29,020 44k INFO ====> Epoch: 753, cost 13.19 s | |
| 2024-01-16 00:44:40,292 44k INFO Train Epoch: 754 [77%] | |
| 2024-01-16 00:44:40,293 44k INFO Losses: [2.2479543685913086, 2.5507514476776123, 4.999232769012451, 10.871591567993164, 0.20670869946479797], step: 9800, lr: 9.101636292227852e-05, reference_loss: 20.876239776611328 | |
| 2024-01-16 00:44:48,974 44k INFO Saving model and optimizer state at iteration 754 to ./logs/44k/G_9800.pth | |
| 2024-01-16 00:44:50,858 44k INFO Saving model and optimizer state at iteration 754 to ./logs/44k/D_9800.pth | |
| 2024-01-16 00:44:51,528 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_5800.pth | |
| 2024-01-16 00:44:51,545 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_5800.pth | |
| 2024-01-16 00:44:53,400 44k INFO ====> Epoch: 754, cost 24.38 s | |
| 2024-01-16 00:45:07,088 44k INFO ====> Epoch: 755, cost 13.69 s | |
| 2024-01-16 00:45:20,661 44k INFO ====> Epoch: 756, cost 13.57 s | |
| 2024-01-16 00:45:33,770 44k INFO ====> Epoch: 757, cost 13.11 s | |
| 2024-01-16 00:45:46,706 44k INFO ====> Epoch: 758, cost 12.94 s | |
| 2024-01-16 00:45:59,652 44k INFO ====> Epoch: 759, cost 12.95 s | |
| 2024-01-16 00:46:12,713 44k INFO ====> Epoch: 760, cost 13.06 s | |
| 2024-01-16 00:46:26,090 44k INFO ====> Epoch: 761, cost 13.38 s | |
| 2024-01-16 00:46:39,905 44k INFO ====> Epoch: 762, cost 13.82 s | |
| 2024-01-16 00:46:53,528 44k INFO ====> Epoch: 763, cost 13.62 s | |
| 2024-01-16 00:47:07,288 44k INFO ====> Epoch: 764, cost 13.76 s | |
| 2024-01-16 00:47:20,918 44k INFO ====> Epoch: 765, cost 13.63 s | |
| 2024-01-16 00:47:34,361 44k INFO ====> Epoch: 766, cost 13.44 s | |
| 2024-01-16 00:47:47,810 44k INFO ====> Epoch: 767, cost 13.45 s | |
| 2024-01-16 00:48:01,151 44k INFO ====> Epoch: 768, cost 13.34 s | |
| 2024-01-16 00:48:14,369 44k INFO ====> Epoch: 769, cost 13.22 s | |
| 2024-01-16 00:48:22,628 44k INFO Train Epoch: 770 [15%] | |
| 2024-01-16 00:48:22,629 44k INFO Losses: [2.5457839965820312, 2.143872022628784, 5.0493340492248535, 10.222735404968262, 0.727972149848938], step: 10000, lr: 9.083450075260563e-05, reference_loss: 20.689699172973633 | |
| 2024-01-16 00:48:31,373 44k INFO Saving model and optimizer state at iteration 770 to ./logs/44k/G_10000.pth | |
| 2024-01-16 00:48:32,996 44k INFO Saving model and optimizer state at iteration 770 to ./logs/44k/D_10000.pth | |
| 2024-01-16 00:48:33,679 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_6000.pth | |
| 2024-01-16 00:48:33,692 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_6000.pth | |
| 2024-01-16 00:48:38,862 44k INFO ====> Epoch: 770, cost 24.49 s | |
| 2024-01-16 00:48:52,024 44k INFO ====> Epoch: 771, cost 13.16 s | |
| 2024-01-16 00:49:05,492 44k INFO ====> Epoch: 772, cost 13.47 s | |
| 2024-01-16 00:49:18,583 44k INFO ====> Epoch: 773, cost 13.09 s | |
| 2024-01-16 00:49:31,735 44k INFO ====> Epoch: 774, cost 13.15 s | |
| 2024-01-16 00:49:45,200 44k INFO ====> Epoch: 775, cost 13.46 s | |
| 2024-01-16 00:49:58,858 44k INFO ====> Epoch: 776, cost 13.66 s | |
| 2024-01-16 00:50:12,251 44k INFO ====> Epoch: 777, cost 13.39 s | |
| 2024-01-16 00:50:25,662 44k INFO ====> Epoch: 778, cost 13.41 s | |
| 2024-01-16 00:50:39,246 44k INFO ====> Epoch: 779, cost 13.58 s | |
| 2024-01-16 00:50:52,893 44k INFO ====> Epoch: 780, cost 13.65 s | |
| 2024-01-16 00:51:06,236 44k INFO ====> Epoch: 781, cost 13.34 s | |
| 2024-01-16 00:51:19,597 44k INFO ====> Epoch: 782, cost 13.36 s | |
| 2024-01-16 00:51:33,375 44k INFO ====> Epoch: 783, cost 13.78 s | |
| 2024-01-16 00:51:46,829 44k INFO ====> Epoch: 784, cost 13.45 s | |
| 2024-01-16 00:51:57,171 44k INFO Train Epoch: 785 [54%] | |
| 2024-01-16 00:51:57,172 44k INFO Losses: [2.4108526706695557, 2.6481189727783203, 4.244121074676514, 10.464800834655762, 0.33402401208877563], step: 10200, lr: 9.066433500835542e-05, reference_loss: 20.101919174194336 | |
| 2024-01-16 00:52:05,479 44k INFO Saving model and optimizer state at iteration 785 to ./logs/44k/G_10200.pth | |
| 2024-01-16 00:52:07,411 44k INFO Saving model and optimizer state at iteration 785 to ./logs/44k/D_10200.pth | |
| 2024-01-16 00:52:08,088 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_6200.pth | |
| 2024-01-16 00:52:08,153 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_6200.pth | |
| 2024-01-16 00:52:11,294 44k INFO ====> Epoch: 785, cost 24.46 s | |
| 2024-01-16 00:52:24,398 44k INFO ====> Epoch: 786, cost 13.10 s | |
| 2024-01-16 00:52:37,662 44k INFO ====> Epoch: 787, cost 13.26 s | |
| 2024-01-16 00:52:50,967 44k INFO ====> Epoch: 788, cost 13.31 s | |
| 2024-01-16 00:53:04,121 44k INFO ====> Epoch: 789, cost 13.15 s | |
| 2024-01-16 00:53:17,345 44k INFO ====> Epoch: 790, cost 13.22 s | |
| 2024-01-16 00:53:30,487 44k INFO ====> Epoch: 791, cost 13.14 s | |
| 2024-01-16 00:53:43,732 44k INFO ====> Epoch: 792, cost 13.25 s | |
| 2024-01-16 00:53:57,011 44k INFO ====> Epoch: 793, cost 13.28 s | |
| 2024-01-16 00:54:10,316 44k INFO ====> Epoch: 794, cost 13.31 s | |
| 2024-01-16 00:54:23,919 44k INFO ====> Epoch: 795, cost 13.60 s | |
| 2024-01-16 00:54:37,074 44k INFO ====> Epoch: 796, cost 13.15 s | |
| 2024-01-16 00:54:50,228 44k INFO ====> Epoch: 797, cost 13.15 s | |
| 2024-01-16 00:55:03,870 44k INFO ====> Epoch: 798, cost 13.64 s | |
| 2024-01-16 00:55:17,088 44k INFO ====> Epoch: 799, cost 13.22 s | |
| 2024-01-16 00:55:29,397 44k INFO Train Epoch: 800 [92%] | |
| 2024-01-16 00:55:29,398 44k INFO Losses: [2.291459798812866, 2.505072593688965, 7.020328998565674, 12.400903701782227, 0.6216867566108704], step: 10400, lr: 9.049448804584871e-05, reference_loss: 24.839452743530273 | |
| 2024-01-16 00:55:38,115 44k INFO Saving model and optimizer state at iteration 800 to ./logs/44k/G_10400.pth | |
| 2024-01-16 00:55:39,750 44k INFO Saving model and optimizer state at iteration 800 to ./logs/44k/D_10400.pth | |
| 2024-01-16 00:55:40,491 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_6400.pth | |
| 2024-01-16 00:55:40,557 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_6400.pth | |
| 2024-01-16 00:55:41,748 44k INFO ====> Epoch: 800, cost 24.66 s | |
| 2024-01-16 00:55:55,056 44k INFO ====> Epoch: 801, cost 13.31 s | |
| 2024-01-16 00:56:07,963 44k INFO ====> Epoch: 802, cost 12.91 s | |
| 2024-01-16 00:56:21,253 44k INFO ====> Epoch: 803, cost 13.29 s | |
| 2024-01-16 00:56:34,653 44k INFO ====> Epoch: 804, cost 13.40 s | |
| 2024-01-16 00:56:48,263 44k INFO ====> Epoch: 805, cost 13.61 s | |
| 2024-01-16 00:57:01,572 44k INFO ====> Epoch: 806, cost 13.31 s | |
| 2024-01-16 00:57:14,807 44k INFO ====> Epoch: 807, cost 13.24 s | |
| 2024-01-16 00:57:28,167 44k INFO ====> Epoch: 808, cost 13.36 s | |
| 2024-01-16 00:57:41,514 44k INFO ====> Epoch: 809, cost 13.35 s | |
| 2024-01-16 00:57:54,759 44k INFO ====> Epoch: 810, cost 13.25 s | |
| 2024-01-16 00:58:07,906 44k INFO ====> Epoch: 811, cost 13.15 s | |
| 2024-01-16 00:58:21,138 44k INFO ====> Epoch: 812, cost 13.23 s | |
| 2024-01-16 00:58:34,626 44k INFO ====> Epoch: 813, cost 13.49 s | |
| 2024-01-16 00:58:47,941 44k INFO ====> Epoch: 814, cost 13.31 s | |
| 2024-01-16 00:59:01,160 44k INFO ====> Epoch: 815, cost 13.22 s | |
| 2024-01-16 00:59:09,931 44k INFO Train Epoch: 816 [31%] | |
| 2024-01-16 00:59:09,932 44k INFO Losses: [2.2183213233947754, 2.710073947906494, 3.5606980323791504, 7.030496120452881, 0.17432649433612823], step: 10600, lr: 9.031366864798387e-05, reference_loss: 15.693917274475098 | |
| 2024-01-16 00:59:18,800 44k INFO Saving model and optimizer state at iteration 816 to ./logs/44k/G_10600.pth | |
| 2024-01-16 00:59:20,447 44k INFO Saving model and optimizer state at iteration 816 to ./logs/44k/D_10600.pth | |
| 2024-01-16 00:59:21,190 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_6600.pth | |
| 2024-01-16 00:59:21,263 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_6600.pth | |
| 2024-01-16 00:59:25,589 44k INFO ====> Epoch: 816, cost 24.43 s | |
| 2024-01-16 00:59:38,872 44k INFO ====> Epoch: 817, cost 13.28 s | |
| 2024-01-16 00:59:52,430 44k INFO ====> Epoch: 818, cost 13.56 s | |
| 2024-01-16 01:00:06,233 44k INFO ====> Epoch: 819, cost 13.80 s | |
| 2024-01-16 01:00:19,726 44k INFO ====> Epoch: 820, cost 13.49 s | |
| 2024-01-16 01:00:33,149 44k INFO ====> Epoch: 821, cost 13.42 s | |
| 2024-01-16 01:00:46,612 44k INFO ====> Epoch: 822, cost 13.46 s | |
| 2024-01-16 01:01:00,130 44k INFO ====> Epoch: 823, cost 13.52 s | |
| 2024-01-16 01:01:13,372 44k INFO ====> Epoch: 824, cost 13.24 s | |
| 2024-01-16 01:01:26,616 44k INFO ====> Epoch: 825, cost 13.24 s | |
| 2024-01-16 01:01:40,189 44k INFO ====> Epoch: 826, cost 13.57 s | |
| 2024-01-16 01:01:53,660 44k INFO ====> Epoch: 827, cost 13.47 s | |
| 2024-01-16 01:02:06,794 44k INFO ====> Epoch: 828, cost 13.13 s | |
| 2024-01-16 01:02:20,145 44k INFO ====> Epoch: 829, cost 13.35 s | |
| 2024-01-16 01:02:33,329 44k INFO ====> Epoch: 830, cost 13.18 s | |
| 2024-01-16 01:02:44,340 44k INFO Train Epoch: 831 [69%] | |
| 2024-01-16 01:02:44,341 44k INFO Losses: [2.122156858444214, 3.0198874473571777, 8.434502601623535, 18.3895320892334, 1.0654263496398926], step: 10800, lr: 9.014447860990232e-05, reference_loss: 33.0315055847168 | |
| 2024-01-16 01:02:53,110 44k INFO Saving model and optimizer state at iteration 831 to ./logs/44k/G_10800.pth | |
| 2024-01-16 01:02:54,808 44k INFO Saving model and optimizer state at iteration 831 to ./logs/44k/D_10800.pth | |
| 2024-01-16 01:02:55,570 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_6800.pth | |
| 2024-01-16 01:02:55,642 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_6800.pth | |
| 2024-01-16 01:02:58,073 44k INFO ====> Epoch: 831, cost 24.74 s | |
| 2024-01-16 01:03:12,032 44k INFO ====> Epoch: 832, cost 13.96 s | |
| 2024-01-16 01:03:25,949 44k INFO ====> Epoch: 833, cost 13.92 s | |
| 2024-01-16 01:03:39,526 44k INFO ====> Epoch: 834, cost 13.58 s | |
| 2024-01-16 01:03:53,172 44k INFO ====> Epoch: 835, cost 13.65 s | |
| 2024-01-16 01:04:06,435 44k INFO ====> Epoch: 836, cost 13.26 s | |
| 2024-01-16 01:04:19,654 44k INFO ====> Epoch: 837, cost 13.22 s | |
| 2024-01-16 01:04:33,136 44k INFO ====> Epoch: 838, cost 13.48 s | |
| 2024-01-16 01:04:46,627 44k INFO ====> Epoch: 839, cost 13.49 s | |
| 2024-01-16 01:04:59,933 44k INFO ====> Epoch: 840, cost 13.31 s | |
| 2024-01-16 01:05:13,328 44k INFO ====> Epoch: 841, cost 13.39 s | |
| 2024-01-16 01:05:26,484 44k INFO ====> Epoch: 842, cost 13.16 s | |
| 2024-01-16 01:05:39,700 44k INFO ====> Epoch: 843, cost 13.22 s | |
| 2024-01-16 01:05:52,830 44k INFO ====> Epoch: 844, cost 13.13 s | |
| 2024-01-16 01:06:05,992 44k INFO ====> Epoch: 845, cost 13.16 s | |
| 2024-01-16 01:06:19,459 44k INFO ====> Epoch: 846, cost 13.47 s | |
| 2024-01-16 01:06:27,153 44k INFO Train Epoch: 847 [8%] | |
| 2024-01-16 01:06:27,153 44k INFO Losses: [1.9735184907913208, 2.7576239109039307, 7.031928062438965, 14.879673957824707, 0.5312067866325378], step: 11000, lr: 8.996435857502436e-05, reference_loss: 27.173952102661133 | |
| 2024-01-16 01:06:36,215 44k INFO Saving model and optimizer state at iteration 847 to ./logs/44k/G_11000.pth | |
| 2024-01-16 01:06:37,939 44k INFO Saving model and optimizer state at iteration 847 to ./logs/44k/D_11000.pth | |
| 2024-01-16 01:06:38,674 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_7000.pth | |
| 2024-01-16 01:06:38,745 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_7000.pth | |
| 2024-01-16 01:06:44,287 44k INFO ====> Epoch: 847, cost 24.83 s | |
| 2024-01-16 01:06:57,835 44k INFO ====> Epoch: 848, cost 13.55 s | |
| 2024-01-16 01:07:10,980 44k INFO ====> Epoch: 849, cost 13.15 s | |
| 2024-01-16 01:07:23,979 44k INFO ====> Epoch: 850, cost 13.00 s | |
| 2024-01-16 01:07:37,123 44k INFO ====> Epoch: 851, cost 13.14 s | |
| 2024-01-16 01:07:50,527 44k INFO ====> Epoch: 852, cost 13.40 s | |
| 2024-01-16 01:08:03,797 44k INFO ====> Epoch: 853, cost 13.27 s | |
| 2024-01-16 01:08:17,141 44k INFO ====> Epoch: 854, cost 13.34 s | |
| 2024-01-16 01:08:30,188 44k INFO ====> Epoch: 855, cost 13.05 s | |
| 2024-01-16 01:08:43,518 44k INFO ====> Epoch: 856, cost 13.33 s | |
| 2024-01-16 01:08:56,966 44k INFO ====> Epoch: 857, cost 13.45 s | |
| 2024-01-16 01:09:10,176 44k INFO ====> Epoch: 858, cost 13.21 s | |
| 2024-01-16 01:09:23,502 44k INFO ====> Epoch: 859, cost 13.33 s | |
| 2024-01-16 01:09:36,802 44k INFO ====> Epoch: 860, cost 13.30 s | |
| 2024-01-16 01:09:50,079 44k INFO ====> Epoch: 861, cost 13.28 s | |
| 2024-01-16 01:09:59,904 44k INFO Train Epoch: 862 [46%] | |
| 2024-01-16 01:09:59,905 44k INFO Losses: [2.6232786178588867, 2.250534772872925, 5.762561798095703, 12.714643478393555, 0.3130713701248169], step: 11200, lr: 8.979582292055309e-05, reference_loss: 23.664091110229492 | |
| 2024-01-16 01:10:08,747 44k INFO Saving model and optimizer state at iteration 862 to ./logs/44k/G_11200.pth | |
| 2024-01-16 01:10:10,460 44k INFO Saving model and optimizer state at iteration 862 to ./logs/44k/D_11200.pth | |
| 2024-01-16 01:10:11,251 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_7200.pth | |
| 2024-01-16 01:10:11,324 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_7200.pth | |
| 2024-01-16 01:10:14,913 44k INFO ====> Epoch: 862, cost 24.83 s | |
| 2024-01-16 01:10:28,251 44k INFO ====> Epoch: 863, cost 13.34 s | |
| 2024-01-16 01:10:41,891 44k INFO ====> Epoch: 864, cost 13.64 s | |
| 2024-01-16 01:10:55,316 44k INFO ====> Epoch: 865, cost 13.42 s | |
| 2024-01-16 01:11:08,505 44k INFO ====> Epoch: 866, cost 13.19 s | |
| 2024-01-16 01:11:21,850 44k INFO ====> Epoch: 867, cost 13.35 s | |
| 2024-01-16 01:11:35,296 44k INFO ====> Epoch: 868, cost 13.45 s | |
| 2024-01-16 01:11:48,469 44k INFO ====> Epoch: 869, cost 13.17 s | |
| 2024-01-16 01:12:01,782 44k INFO ====> Epoch: 870, cost 13.31 s | |
| 2024-01-16 01:12:14,846 44k INFO ====> Epoch: 871, cost 13.06 s | |
| 2024-01-16 01:12:28,238 44k INFO ====> Epoch: 872, cost 13.39 s | |
| 2024-01-16 01:12:41,633 44k INFO ====> Epoch: 873, cost 13.40 s | |
| 2024-01-16 01:12:55,001 44k INFO ====> Epoch: 874, cost 13.37 s | |
| 2024-01-16 01:13:08,558 44k INFO ====> Epoch: 875, cost 13.56 s | |
| 2024-01-16 01:13:21,765 44k INFO ====> Epoch: 876, cost 13.21 s | |
| 2024-01-16 01:13:33,551 44k INFO Train Epoch: 877 [85%] | |
| 2024-01-16 01:13:33,552 44k INFO Losses: [2.7481865882873535, 2.1591010093688965, 5.566190719604492, 12.5184907913208, 0.8306258320808411], step: 11400, lr: 8.962760299407988e-05, reference_loss: 23.822595596313477 | |
| 2024-01-16 01:13:42,233 44k INFO Saving model and optimizer state at iteration 877 to ./logs/44k/G_11400.pth | |
| 2024-01-16 01:13:43,887 44k INFO Saving model and optimizer state at iteration 877 to ./logs/44k/D_11400.pth | |
| 2024-01-16 01:13:44,632 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_7400.pth | |
| 2024-01-16 01:13:44,701 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_7400.pth | |
| 2024-01-16 01:13:46,252 44k INFO ====> Epoch: 877, cost 24.49 s | |
| 2024-01-16 01:13:59,457 44k INFO ====> Epoch: 878, cost 13.21 s | |
| 2024-01-16 01:14:12,755 44k INFO ====> Epoch: 879, cost 13.30 s | |
| 2024-01-16 01:14:26,037 44k INFO ====> Epoch: 880, cost 13.28 s | |
| 2024-01-16 01:14:39,722 44k INFO ====> Epoch: 881, cost 13.68 s | |
| 2024-01-16 01:14:52,986 44k INFO ====> Epoch: 882, cost 13.26 s | |
| 2024-01-16 01:15:06,289 44k INFO ====> Epoch: 883, cost 13.30 s | |
| 2024-01-16 01:15:19,362 44k INFO ====> Epoch: 884, cost 13.07 s | |
| 2024-01-16 01:15:32,445 44k INFO ====> Epoch: 885, cost 13.08 s | |
| 2024-01-16 01:15:45,889 44k INFO ====> Epoch: 886, cost 13.44 s | |
| 2024-01-16 01:15:59,070 44k INFO ====> Epoch: 887, cost 13.18 s | |
| 2024-01-16 01:16:12,299 44k INFO ====> Epoch: 888, cost 13.23 s | |
| 2024-01-16 01:16:25,466 44k INFO ====> Epoch: 889, cost 13.17 s | |
| 2024-01-16 01:16:38,563 44k INFO ====> Epoch: 890, cost 13.10 s | |
| 2024-01-16 01:16:51,524 44k INFO ====> Epoch: 891, cost 12.96 s | |
| 2024-01-16 01:17:04,664 44k INFO ====> Epoch: 892, cost 13.14 s | |
| 2024-01-16 01:17:13,145 44k INFO Train Epoch: 893 [23%] | |
| 2024-01-16 01:17:13,146 44k INFO Losses: [2.363422393798828, 2.5926101207733154, 7.422856330871582, 14.99197769165039, 0.789316713809967], step: 11600, lr: 8.944851574185691e-05, reference_loss: 28.16018295288086 | |
| 2024-01-16 01:17:21,812 44k INFO Saving model and optimizer state at iteration 893 to ./logs/44k/G_11600.pth | |
| 2024-01-16 01:17:23,540 44k INFO Saving model and optimizer state at iteration 893 to ./logs/44k/D_11600.pth | |
| 2024-01-16 01:17:24,286 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_7600.pth | |
| 2024-01-16 01:17:24,363 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_7600.pth | |
| 2024-01-16 01:17:29,088 44k INFO ====> Epoch: 893, cost 24.42 s | |
| 2024-01-16 01:17:42,056 44k INFO ====> Epoch: 894, cost 12.97 s | |
| 2024-01-16 01:17:55,423 44k INFO ====> Epoch: 895, cost 13.37 s | |
| 2024-01-16 01:18:08,553 44k INFO ====> Epoch: 896, cost 13.13 s | |
| 2024-01-16 01:18:21,540 44k INFO ====> Epoch: 897, cost 12.99 s | |
| 2024-01-16 01:18:34,674 44k INFO ====> Epoch: 898, cost 13.13 s | |
| 2024-01-16 01:18:47,828 44k INFO ====> Epoch: 899, cost 13.15 s | |
| 2024-01-16 01:19:01,037 44k INFO ====> Epoch: 900, cost 13.21 s | |
| 2024-01-16 01:19:14,013 44k INFO ====> Epoch: 901, cost 12.98 s | |
| 2024-01-16 01:19:27,419 44k INFO ====> Epoch: 902, cost 13.41 s | |
| 2024-01-16 01:19:40,491 44k INFO ====> Epoch: 903, cost 13.07 s | |
| 2024-01-16 01:19:53,467 44k INFO ====> Epoch: 904, cost 12.98 s | |
| 2024-01-16 01:20:06,797 44k INFO ====> Epoch: 905, cost 13.33 s | |
| 2024-01-16 01:20:19,976 44k INFO ====> Epoch: 906, cost 13.18 s | |
| 2024-01-16 01:20:32,834 44k INFO ====> Epoch: 907, cost 12.86 s | |
| 2024-01-16 01:20:43,015 44k INFO Train Epoch: 908 [62%] | |
| 2024-01-16 01:20:43,016 44k INFO Losses: [2.317136287689209, 2.260225296020508, 6.576975345611572, 12.243886947631836, 0.7556127905845642], step: 11800, lr: 8.928094644685142e-05, reference_loss: 24.153837203979492 | |
| 2024-01-16 01:20:51,505 44k INFO Saving model and optimizer state at iteration 908 to ./logs/44k/G_11800.pth | |
| 2024-01-16 01:20:53,171 44k INFO Saving model and optimizer state at iteration 908 to ./logs/44k/D_11800.pth | |
| 2024-01-16 01:20:53,895 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_7800.pth | |
| 2024-01-16 01:20:53,962 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_7800.pth | |
| 2024-01-16 01:20:56,672 44k INFO ====> Epoch: 908, cost 23.84 s | |
| 2024-01-16 01:21:09,841 44k INFO ====> Epoch: 909, cost 13.17 s | |
| 2024-01-16 01:21:23,509 44k INFO ====> Epoch: 910, cost 13.67 s | |
| 2024-01-16 01:21:36,758 44k INFO ====> Epoch: 911, cost 13.25 s | |
| 2024-01-16 01:21:49,746 44k INFO ====> Epoch: 912, cost 12.99 s | |
| 2024-01-16 01:22:02,939 44k INFO ====> Epoch: 913, cost 13.19 s | |
| 2024-01-16 01:22:16,300 44k INFO ====> Epoch: 914, cost 13.36 s | |
| 2024-01-16 01:22:29,253 44k INFO ====> Epoch: 915, cost 12.95 s | |
| 2024-01-16 01:22:42,443 44k INFO ====> Epoch: 916, cost 13.19 s | |
| 2024-01-16 01:22:55,685 44k INFO ====> Epoch: 917, cost 13.24 s | |
| 2024-01-16 01:23:08,768 44k INFO ====> Epoch: 918, cost 13.08 s | |
| 2024-01-16 01:23:22,273 44k INFO ====> Epoch: 919, cost 13.50 s | |
| 2024-01-16 01:23:35,332 44k INFO ====> Epoch: 920, cost 13.06 s | |
| 2024-01-16 01:23:48,615 44k INFO ====> Epoch: 921, cost 13.28 s | |
| 2024-01-16 01:24:01,765 44k INFO ====> Epoch: 922, cost 13.15 s | |
| 2024-01-16 01:24:15,082 44k INFO ====> Epoch: 923, cost 13.32 s | |
| 2024-01-16 01:24:22,269 44k INFO Train Epoch: 924 [0%] | |
| 2024-01-16 01:24:22,270 44k INFO Losses: [2.4339399337768555, 2.280014753341675, 3.744424819946289, 13.621177673339844, 0.44488778710365295], step: 12000, lr: 8.910255185812085e-05, reference_loss: 22.524444580078125 | |
| 2024-01-16 01:24:31,104 44k INFO Saving model and optimizer state at iteration 924 to ./logs/44k/G_12000.pth | |
| 2024-01-16 01:24:32,727 44k INFO Saving model and optimizer state at iteration 924 to ./logs/44k/D_12000.pth | |
| 2024-01-16 01:24:33,495 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_8000.pth | |
| 2024-01-16 01:24:33,570 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_8000.pth | |
| 2024-01-16 01:24:39,732 44k INFO ====> Epoch: 924, cost 24.65 s | |
| 2024-01-16 01:24:52,865 44k INFO ====> Epoch: 925, cost 13.13 s | |
| 2024-01-16 01:25:06,072 44k INFO ====> Epoch: 926, cost 13.21 s | |
| 2024-01-16 01:25:19,333 44k INFO ====> Epoch: 927, cost 13.26 s | |
| 2024-01-16 01:25:32,477 44k INFO ====> Epoch: 928, cost 13.14 s | |
| 2024-01-16 01:25:45,766 44k INFO ====> Epoch: 929, cost 13.29 s | |
| 2024-01-16 01:25:58,910 44k INFO ====> Epoch: 930, cost 13.14 s | |
| 2024-01-16 01:26:12,243 44k INFO ====> Epoch: 931, cost 13.33 s | |
| 2024-01-16 01:26:25,655 44k INFO ====> Epoch: 932, cost 13.41 s | |
| 2024-01-16 01:26:38,928 44k INFO ====> Epoch: 933, cost 13.27 s | |
| 2024-01-16 01:26:51,940 44k INFO ====> Epoch: 934, cost 13.01 s | |
| 2024-01-16 01:27:05,065 44k INFO ====> Epoch: 935, cost 13.12 s | |
| 2024-01-16 01:27:18,046 44k INFO ====> Epoch: 936, cost 12.98 s | |
| 2024-01-16 01:27:31,226 44k INFO ====> Epoch: 937, cost 13.18 s | |
| 2024-01-16 01:27:44,658 44k INFO ====> Epoch: 938, cost 13.43 s | |
| 2024-01-16 01:27:54,066 44k INFO Train Epoch: 939 [38%] | |
| 2024-01-16 01:27:54,066 44k INFO Losses: [2.3344075679779053, 2.499584674835205, 6.349873065948486, 14.282886505126953, 0.16964882612228394], step: 12200, lr: 8.893563067810772e-05, reference_loss: 25.63640022277832 | |
| 2024-01-16 01:28:02,627 44k INFO Saving model and optimizer state at iteration 939 to ./logs/44k/G_12200.pth | |
| 2024-01-16 01:28:04,343 44k INFO Saving model and optimizer state at iteration 939 to ./logs/44k/D_12200.pth | |
| 2024-01-16 01:28:05,110 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_8200.pth | |
| 2024-01-16 01:28:05,184 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_8200.pth | |
| 2024-01-16 01:28:08,955 44k INFO ====> Epoch: 939, cost 24.30 s | |
| 2024-01-16 01:28:22,555 44k INFO ====> Epoch: 940, cost 13.60 s | |
| 2024-01-16 01:28:36,314 44k INFO ====> Epoch: 941, cost 13.76 s | |
| 2024-01-16 01:28:49,533 44k INFO ====> Epoch: 942, cost 13.22 s | |
| 2024-01-16 01:29:02,911 44k INFO ====> Epoch: 943, cost 13.38 s | |
| 2024-01-16 01:29:16,289 44k INFO ====> Epoch: 944, cost 13.38 s | |
| 2024-01-16 01:29:29,682 44k INFO ====> Epoch: 945, cost 13.39 s | |
| 2024-01-16 01:29:42,774 44k INFO ====> Epoch: 946, cost 13.09 s | |
| 2024-01-16 01:29:55,921 44k INFO ====> Epoch: 947, cost 13.15 s | |
| 2024-01-16 01:30:09,777 44k INFO ====> Epoch: 948, cost 13.86 s | |
| 2024-01-16 01:30:22,988 44k INFO ====> Epoch: 949, cost 13.21 s | |
| 2024-01-16 01:30:36,152 44k INFO ====> Epoch: 950, cost 13.16 s | |
| 2024-01-16 01:30:49,194 44k INFO ====> Epoch: 951, cost 13.04 s | |
| 2024-01-16 01:31:02,218 44k INFO ====> Epoch: 952, cost 13.02 s | |
| 2024-01-16 01:31:15,600 44k INFO ====> Epoch: 953, cost 13.38 s | |
| 2024-01-16 01:31:26,883 44k INFO Train Epoch: 954 [77%] | |
| 2024-01-16 01:31:26,884 44k INFO Losses: [2.508659601211548, 2.427175521850586, 4.5047760009765625, 10.376603126525879, 0.14853060245513916], step: 12400, lr: 8.876902220160032e-05, reference_loss: 19.96574592590332 | |
| 2024-01-16 01:31:35,707 44k INFO Saving model and optimizer state at iteration 954 to ./logs/44k/G_12400.pth | |
| 2024-01-16 01:31:37,448 44k INFO Saving model and optimizer state at iteration 954 to ./logs/44k/D_12400.pth | |
| 2024-01-16 01:31:38,171 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_8400.pth | |
| 2024-01-16 01:31:38,229 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_8400.pth | |
| 2024-01-16 01:31:40,175 44k INFO ====> Epoch: 954, cost 24.58 s | |
| 2024-01-16 01:31:53,293 44k INFO ====> Epoch: 955, cost 13.12 s | |
| 2024-01-16 01:32:06,461 44k INFO ====> Epoch: 956, cost 13.17 s | |
| 2024-01-16 01:32:19,688 44k INFO ====> Epoch: 957, cost 13.23 s | |
| 2024-01-16 01:32:32,934 44k INFO ====> Epoch: 958, cost 13.25 s | |
| 2024-01-16 01:32:46,293 44k INFO ====> Epoch: 959, cost 13.36 s | |
| 2024-01-16 01:32:59,360 44k INFO ====> Epoch: 960, cost 13.07 s | |
| 2024-01-16 01:33:12,571 44k INFO ====> Epoch: 961, cost 13.21 s | |
| 2024-01-16 01:33:26,276 44k INFO ====> Epoch: 962, cost 13.70 s | |
| 2024-01-16 01:33:39,637 44k INFO ====> Epoch: 963, cost 13.36 s | |
| 2024-01-16 01:33:52,743 44k INFO ====> Epoch: 964, cost 13.11 s | |
| 2024-01-16 01:34:05,937 44k INFO ====> Epoch: 965, cost 13.19 s | |
| 2024-01-16 01:34:19,190 44k INFO ====> Epoch: 966, cost 13.25 s | |
| 2024-01-16 01:34:32,322 44k INFO ====> Epoch: 967, cost 13.13 s | |
| 2024-01-16 01:34:45,751 44k INFO ====> Epoch: 968, cost 13.43 s | |
| 2024-01-16 01:34:58,951 44k INFO ====> Epoch: 969, cost 13.20 s | |
| 2024-01-16 01:35:07,171 44k INFO Train Epoch: 970 [15%] | |
| 2024-01-16 01:35:07,172 44k INFO Losses: [2.5610709190368652, 1.9867982864379883, 4.446091651916504, 9.665107727050781, 0.7710591554641724], step: 12600, lr: 8.8591650502062e-05, reference_loss: 19.43012809753418 | |
| 2024-01-16 01:35:15,698 44k INFO Saving model and optimizer state at iteration 970 to ./logs/44k/G_12600.pth | |
| 2024-01-16 01:35:17,362 44k INFO Saving model and optimizer state at iteration 970 to ./logs/44k/D_12600.pth | |
| 2024-01-16 01:35:18,198 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_8600.pth | |
| 2024-01-16 01:35:18,254 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_8600.pth | |
| 2024-01-16 01:35:23,406 44k INFO ====> Epoch: 970, cost 24.46 s | |
| 2024-01-16 01:35:37,089 44k INFO ====> Epoch: 971, cost 13.68 s | |
| 2024-01-16 01:35:50,529 44k INFO ====> Epoch: 972, cost 13.44 s | |
| 2024-01-16 01:36:03,708 44k INFO ====> Epoch: 973, cost 13.18 s | |
| 2024-01-16 01:36:16,851 44k INFO ====> Epoch: 974, cost 13.14 s | |
| 2024-01-16 01:36:30,231 44k INFO ====> Epoch: 975, cost 13.38 s | |
| 2024-01-16 01:36:43,634 44k INFO ====> Epoch: 976, cost 13.40 s | |
| 2024-01-16 01:36:56,896 44k INFO ====> Epoch: 977, cost 13.26 s | |
| 2024-01-16 01:37:10,259 44k INFO ====> Epoch: 978, cost 13.36 s | |
| 2024-01-16 01:37:23,382 44k INFO ====> Epoch: 979, cost 13.12 s | |
| 2024-01-16 01:37:36,688 44k INFO ====> Epoch: 980, cost 13.31 s | |
| 2024-01-16 01:37:50,079 44k INFO ====> Epoch: 981, cost 13.39 s | |
| 2024-01-16 01:38:03,441 44k INFO ====> Epoch: 982, cost 13.36 s | |
| 2024-01-16 01:38:16,472 44k INFO ====> Epoch: 983, cost 13.03 s | |
| 2024-01-16 01:38:30,059 44k INFO ====> Epoch: 984, cost 13.59 s | |
| 2024-01-16 01:38:40,233 44k INFO Train Epoch: 985 [54%] | |
| 2024-01-16 01:38:40,234 44k INFO Losses: [2.5930662155151367, 2.0602407455444336, 3.563302755355835, 10.215177536010742, 0.3204095959663391], step: 12800, lr: 8.842568642434779e-05, reference_loss: 18.752197265625 | |
| 2024-01-16 01:38:49,005 44k INFO Saving model and optimizer state at iteration 985 to ./logs/44k/G_12800.pth | |
| 2024-01-16 01:38:50,698 44k INFO Saving model and optimizer state at iteration 985 to ./logs/44k/D_12800.pth | |
| 2024-01-16 01:38:51,515 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_8800.pth | |
| 2024-01-16 01:38:51,554 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_8800.pth | |
| 2024-01-16 01:38:54,634 44k INFO ====> Epoch: 985, cost 24.58 s | |
| 2024-01-16 01:39:08,048 44k INFO ====> Epoch: 986, cost 13.41 s | |
| 2024-01-16 01:39:21,370 44k INFO ====> Epoch: 987, cost 13.32 s | |
| 2024-01-16 01:39:34,405 44k INFO ====> Epoch: 988, cost 13.03 s | |
| 2024-01-16 01:39:47,658 44k INFO ====> Epoch: 989, cost 13.25 s | |
| 2024-01-16 01:40:01,171 44k INFO ====> Epoch: 990, cost 13.51 s | |
| 2024-01-16 01:40:14,570 44k INFO ====> Epoch: 991, cost 13.40 s | |
| 2024-01-16 01:40:28,532 44k INFO ====> Epoch: 992, cost 13.96 s | |
| 2024-01-16 01:40:41,618 44k INFO ====> Epoch: 993, cost 13.09 s | |
| 2024-01-16 01:40:54,795 44k INFO ====> Epoch: 994, cost 13.18 s | |
| 2024-01-16 01:41:08,058 44k INFO ====> Epoch: 995, cost 13.26 s | |
| 2024-01-16 01:41:21,364 44k INFO ====> Epoch: 996, cost 13.31 s | |
| 2024-01-16 01:41:34,617 44k INFO ====> Epoch: 997, cost 13.25 s | |
| 2024-01-16 01:41:48,018 44k INFO ====> Epoch: 998, cost 13.40 s | |
| 2024-01-16 01:42:01,294 44k INFO ====> Epoch: 999, cost 13.28 s | |
| 2024-01-16 01:42:13,174 44k INFO Train Epoch: 1000 [92%] | |
| 2024-01-16 01:42:13,175 44k INFO Losses: [2.37866473197937, 2.2728352546691895, 6.155479907989502, 11.833898544311523, 0.6309542655944824], step: 13000, lr: 8.82600332571419e-05, reference_loss: 23.271833419799805 | |
| 2024-01-16 01:42:21,804 44k INFO Saving model and optimizer state at iteration 1000 to ./logs/44k/G_13000.pth | |
| 2024-01-16 01:42:23,783 44k INFO Saving model and optimizer state at iteration 1000 to ./logs/44k/D_13000.pth | |
| 2024-01-16 01:42:24,651 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_9000.pth | |
| 2024-01-16 01:42:24,721 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_9000.pth | |
| 2024-01-16 01:42:25,828 44k INFO ====> Epoch: 1000, cost 24.53 s | |
| 2024-01-16 01:42:39,274 44k INFO ====> Epoch: 1001, cost 13.45 s | |
| 2024-01-16 01:42:52,777 44k INFO ====> Epoch: 1002, cost 13.50 s | |
| 2024-01-16 01:43:06,063 44k INFO ====> Epoch: 1003, cost 13.29 s | |
| 2024-01-16 01:43:19,558 44k INFO ====> Epoch: 1004, cost 13.49 s | |
| 2024-01-16 01:43:32,670 44k INFO ====> Epoch: 1005, cost 13.11 s | |
| 2024-01-16 01:43:45,944 44k INFO ====> Epoch: 1006, cost 13.27 s | |
| 2024-01-16 01:43:59,330 44k INFO ====> Epoch: 1007, cost 13.39 s | |
| 2024-01-16 01:44:12,542 44k INFO ====> Epoch: 1008, cost 13.21 s | |
| 2024-01-16 01:44:25,483 44k INFO ====> Epoch: 1009, cost 12.94 s | |
| 2024-01-16 01:44:38,457 44k INFO ====> Epoch: 1010, cost 12.97 s | |
| 2024-01-16 01:44:51,618 44k INFO ====> Epoch: 1011, cost 13.16 s | |
| 2024-01-16 01:45:05,062 44k INFO ====> Epoch: 1012, cost 13.44 s | |
| 2024-01-16 01:45:18,528 44k INFO ====> Epoch: 1013, cost 13.47 s | |
| 2024-01-16 01:45:32,250 44k INFO ====> Epoch: 1014, cost 13.72 s | |
| 2024-01-16 01:45:45,435 44k INFO ====> Epoch: 1015, cost 13.18 s | |
| 2024-01-16 01:45:54,129 44k INFO Train Epoch: 1016 [31%] | |
| 2024-01-16 01:45:54,130 44k INFO Losses: [2.782275676727295, 1.896427869796753, 1.6249529123306274, 6.158237457275391, 0.15653099119663239], step: 13200, lr: 8.808367858169472e-05, reference_loss: 12.618424415588379 | |
| 2024-01-16 01:46:02,894 44k INFO Saving model and optimizer state at iteration 1016 to ./logs/44k/G_13200.pth | |
| 2024-01-16 01:46:04,646 44k INFO Saving model and optimizer state at iteration 1016 to ./logs/44k/D_13200.pth | |
| 2024-01-16 01:46:05,508 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_9200.pth | |
| 2024-01-16 01:46:05,564 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_9200.pth | |
| 2024-01-16 01:46:10,015 44k INFO ====> Epoch: 1016, cost 24.58 s | |
| 2024-01-16 01:46:23,262 44k INFO ====> Epoch: 1017, cost 13.25 s | |
| 2024-01-16 01:46:36,700 44k INFO ====> Epoch: 1018, cost 13.44 s | |
| 2024-01-16 01:46:50,181 44k INFO ====> Epoch: 1019, cost 13.48 s | |
| 2024-01-16 01:47:03,375 44k INFO ====> Epoch: 1020, cost 13.19 s | |
| 2024-01-16 01:47:16,654 44k INFO ====> Epoch: 1021, cost 13.28 s | |
| 2024-01-16 01:47:30,381 44k INFO ====> Epoch: 1022, cost 13.73 s | |
| 2024-01-16 01:47:43,820 44k INFO ====> Epoch: 1023, cost 13.44 s | |
| 2024-01-16 01:47:57,001 44k INFO ====> Epoch: 1024, cost 13.18 s | |
| 2024-01-16 01:48:10,602 44k INFO ====> Epoch: 1025, cost 13.60 s | |
| 2024-01-16 01:48:24,128 44k INFO ====> Epoch: 1026, cost 13.53 s | |
| 2024-01-16 01:48:37,376 44k INFO ====> Epoch: 1027, cost 13.25 s | |
| 2024-01-16 01:48:50,432 44k INFO ====> Epoch: 1028, cost 13.06 s | |
| 2024-01-16 01:49:03,993 44k INFO ====> Epoch: 1029, cost 13.56 s | |
| 2024-01-16 01:49:17,561 44k INFO ====> Epoch: 1030, cost 13.57 s | |
| 2024-01-16 01:49:28,661 44k INFO Train Epoch: 1031 [69%] | |
| 2024-01-16 01:49:28,662 44k INFO Losses: [2.3795127868652344, 2.4253625869750977, 7.546342849731445, 17.21401596069336, 0.9862644076347351], step: 13400, lr: 8.7918666118391e-05, reference_loss: 30.55150032043457 | |
| 2024-01-16 01:49:37,685 44k INFO Saving model and optimizer state at iteration 1031 to ./logs/44k/G_13400.pth | |
| 2024-01-16 01:49:39,441 44k INFO Saving model and optimizer state at iteration 1031 to ./logs/44k/D_13400.pth | |
| 2024-01-16 01:49:40,305 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_9400.pth | |
| 2024-01-16 01:49:40,382 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_9400.pth | |
| 2024-01-16 01:49:42,810 44k INFO ====> Epoch: 1031, cost 25.25 s | |
| 2024-01-16 01:49:56,369 44k INFO ====> Epoch: 1032, cost 13.56 s | |
| 2024-01-16 01:50:10,265 44k INFO ====> Epoch: 1033, cost 13.90 s | |
| 2024-01-16 01:50:23,734 44k INFO ====> Epoch: 1034, cost 13.47 s | |
| 2024-01-16 01:50:37,046 44k INFO ====> Epoch: 1035, cost 13.31 s | |
| 2024-01-16 01:50:50,671 44k INFO ====> Epoch: 1036, cost 13.62 s | |
| 2024-01-16 01:51:04,153 44k INFO ====> Epoch: 1037, cost 13.48 s | |
| 2024-01-16 01:51:17,520 44k INFO ====> Epoch: 1038, cost 13.37 s | |
| 2024-01-16 01:51:31,043 44k INFO ====> Epoch: 1039, cost 13.52 s | |
| 2024-01-16 01:51:44,609 44k INFO ====> Epoch: 1040, cost 13.57 s | |
| 2024-01-16 01:51:58,180 44k INFO ====> Epoch: 1041, cost 13.57 s | |
| 2024-01-16 01:52:11,876 44k INFO ====> Epoch: 1042, cost 13.70 s | |
| 2024-01-16 01:52:25,186 44k INFO ====> Epoch: 1043, cost 13.31 s | |
| 2024-01-16 01:52:38,583 44k INFO ====> Epoch: 1044, cost 13.40 s | |
| 2024-01-16 01:52:52,188 44k INFO ====> Epoch: 1045, cost 13.61 s | |
| 2024-01-16 01:53:05,951 44k INFO ====> Epoch: 1046, cost 13.76 s | |
| 2024-01-16 01:53:13,825 44k INFO Train Epoch: 1047 [8%] | |
| 2024-01-16 01:53:13,825 44k INFO Losses: [2.269448757171631, 2.5180227756500244, 5.849318981170654, 14.384262084960938, 0.5157080292701721], step: 13600, lr: 8.774299353753115e-05, reference_loss: 25.536762237548828 | |
| 2024-01-16 01:53:22,546 44k INFO Saving model and optimizer state at iteration 1047 to ./logs/44k/G_13600.pth | |
| 2024-01-16 01:53:24,284 44k INFO Saving model and optimizer state at iteration 1047 to ./logs/44k/D_13600.pth | |
| 2024-01-16 01:53:25,099 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_9600.pth | |
| 2024-01-16 01:53:25,142 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_9600.pth | |
| 2024-01-16 01:53:30,806 44k INFO ====> Epoch: 1047, cost 24.85 s | |
| 2024-01-16 01:53:44,555 44k INFO ====> Epoch: 1048, cost 13.75 s | |
| 2024-01-16 01:53:58,453 44k INFO ====> Epoch: 1049, cost 13.90 s | |
| 2024-01-16 01:54:11,935 44k INFO ====> Epoch: 1050, cost 13.48 s | |
| 2024-01-16 01:54:25,768 44k INFO ====> Epoch: 1051, cost 13.83 s | |
| 2024-01-16 01:54:39,144 44k INFO ====> Epoch: 1052, cost 13.38 s | |
| 2024-01-16 01:54:52,817 44k INFO ====> Epoch: 1053, cost 13.67 s | |
| 2024-01-16 01:55:06,384 44k INFO ====> Epoch: 1054, cost 13.57 s | |
| 2024-01-16 01:55:19,897 44k INFO ====> Epoch: 1055, cost 13.51 s | |
| 2024-01-16 01:55:33,573 44k INFO ====> Epoch: 1056, cost 13.68 s | |
| 2024-01-16 01:55:47,344 44k INFO ====> Epoch: 1057, cost 13.77 s | |
| 2024-01-16 01:56:00,683 44k INFO ====> Epoch: 1058, cost 13.34 s | |
| 2024-01-16 01:56:14,368 44k INFO ====> Epoch: 1059, cost 13.69 s | |
| 2024-01-16 01:56:27,727 44k INFO ====> Epoch: 1060, cost 13.36 s | |
| 2024-01-16 01:56:41,442 44k INFO ====> Epoch: 1061, cost 13.72 s | |
| 2024-01-16 01:56:51,364 44k INFO Train Epoch: 1062 [46%] | |
| 2024-01-16 01:56:51,365 44k INFO Losses: [2.272083282470703, 2.678462505340576, 6.9739766120910645, 12.86375617980957, 0.2501702904701233], step: 13800, lr: 8.75786193000515e-05, reference_loss: 25.038448333740234 | |
| 2024-01-16 01:57:00,088 44k INFO Saving model and optimizer state at iteration 1062 to ./logs/44k/G_13800.pth | |
| 2024-01-16 01:57:01,899 44k INFO Saving model and optimizer state at iteration 1062 to ./logs/44k/D_13800.pth | |
| 2024-01-16 01:57:02,691 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_9800.pth | |
| 2024-01-16 01:57:02,738 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_9800.pth | |
| 2024-01-16 01:57:06,381 44k INFO ====> Epoch: 1062, cost 24.94 s | |
| 2024-01-16 01:57:19,727 44k INFO ====> Epoch: 1063, cost 13.35 s | |
| 2024-01-16 01:57:33,175 44k INFO ====> Epoch: 1064, cost 13.45 s | |
| 2024-01-16 01:57:46,372 44k INFO ====> Epoch: 1065, cost 13.20 s | |
| 2024-01-16 01:58:00,072 44k INFO ====> Epoch: 1066, cost 13.70 s | |
| 2024-01-16 01:58:13,887 44k INFO ====> Epoch: 1067, cost 13.82 s | |
| 2024-01-16 01:58:27,408 44k INFO ====> Epoch: 1068, cost 13.52 s | |
| 2024-01-16 01:58:41,402 44k INFO ====> Epoch: 1069, cost 13.99 s | |
| 2024-01-16 01:58:54,899 44k INFO ====> Epoch: 1070, cost 13.50 s | |
| 2024-01-16 01:59:08,498 44k INFO ====> Epoch: 1071, cost 13.60 s | |
| 2024-01-16 01:59:22,040 44k INFO ====> Epoch: 1072, cost 13.54 s | |
| 2024-01-16 01:59:35,406 44k INFO ====> Epoch: 1073, cost 13.37 s | |
| 2024-01-16 01:59:48,597 44k INFO ====> Epoch: 1074, cost 13.19 s | |
| 2024-01-16 02:00:02,184 44k INFO ====> Epoch: 1075, cost 13.59 s | |
| 2024-01-16 02:00:16,350 44k INFO ====> Epoch: 1076, cost 14.17 s | |
| 2024-01-16 02:00:28,393 44k INFO Train Epoch: 1077 [85%] | |
| 2024-01-16 02:00:28,394 44k INFO Losses: [2.355936050415039, 2.3306398391723633, 5.557978630065918, 12.234617233276367, 0.7554910182952881], step: 14000, lr: 8.741455299473667e-05, reference_loss: 23.234663009643555 | |
| 2024-01-16 02:00:37,525 44k INFO Saving model and optimizer state at iteration 1077 to ./logs/44k/G_14000.pth | |
| 2024-01-16 02:00:39,596 44k INFO Saving model and optimizer state at iteration 1077 to ./logs/44k/D_14000.pth | |
| 2024-01-16 02:00:40,453 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_10000.pth | |
| 2024-01-16 02:00:40,508 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_10000.pth | |
| 2024-01-16 02:00:42,377 44k INFO ====> Epoch: 1077, cost 26.03 s | |
| 2024-01-16 02:00:55,647 44k INFO ====> Epoch: 1078, cost 13.27 s | |
| 2024-01-16 02:01:09,004 44k INFO ====> Epoch: 1079, cost 13.36 s | |
| 2024-01-16 02:01:22,231 44k INFO ====> Epoch: 1080, cost 13.23 s | |
| 2024-01-16 02:01:35,584 44k INFO ====> Epoch: 1081, cost 13.35 s | |
| 2024-01-16 02:01:48,930 44k INFO ====> Epoch: 1082, cost 13.35 s | |
| 2024-01-16 02:02:02,265 44k INFO ====> Epoch: 1083, cost 13.33 s | |
| 2024-01-16 02:02:15,692 44k INFO ====> Epoch: 1084, cost 13.43 s | |
| 2024-01-16 02:02:28,968 44k INFO ====> Epoch: 1085, cost 13.28 s | |
| 2024-01-16 02:02:42,290 44k INFO ====> Epoch: 1086, cost 13.32 s | |
| 2024-01-16 02:02:55,527 44k INFO ====> Epoch: 1087, cost 13.24 s | |
| 2024-01-16 02:03:09,123 44k INFO ====> Epoch: 1088, cost 13.60 s | |
| 2024-01-16 02:03:22,613 44k INFO ====> Epoch: 1089, cost 13.49 s | |
| 2024-01-16 02:03:36,468 44k INFO ====> Epoch: 1090, cost 13.85 s | |
| 2024-01-16 02:03:49,891 44k INFO ====> Epoch: 1091, cost 13.42 s | |
| 2024-01-16 02:04:03,501 44k INFO ====> Epoch: 1092, cost 13.61 s | |
| 2024-01-16 02:04:12,333 44k INFO Train Epoch: 1093 [23%] | |
| 2024-01-16 02:04:12,334 44k INFO Losses: [2.2341041564941406, 2.8836793899536133, 8.081839561462402, 15.119953155517578, 0.7819900512695312], step: 14200, lr: 8.723988769546315e-05, reference_loss: 29.101566314697266 | |
| 2024-01-16 02:04:21,320 44k INFO Saving model and optimizer state at iteration 1093 to ./logs/44k/G_14200.pth | |
| 2024-01-16 02:04:23,071 44k INFO Saving model and optimizer state at iteration 1093 to ./logs/44k/D_14200.pth | |
| 2024-01-16 02:04:23,904 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_10200.pth | |
| 2024-01-16 02:04:23,967 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_10200.pth | |
| 2024-01-16 02:04:29,079 44k INFO ====> Epoch: 1093, cost 25.58 s | |
| 2024-01-16 02:04:42,554 44k INFO ====> Epoch: 1094, cost 13.47 s | |
| 2024-01-16 02:04:56,130 44k INFO ====> Epoch: 1095, cost 13.58 s | |
| 2024-01-16 02:05:09,480 44k INFO ====> Epoch: 1096, cost 13.35 s | |
| 2024-01-16 02:05:22,820 44k INFO ====> Epoch: 1097, cost 13.34 s | |
| 2024-01-16 02:05:36,454 44k INFO ====> Epoch: 1098, cost 13.63 s | |
| 2024-01-16 02:05:49,581 44k INFO ====> Epoch: 1099, cost 13.13 s | |
| 2024-01-16 02:06:02,980 44k INFO ====> Epoch: 1100, cost 13.40 s | |
| 2024-01-16 02:06:16,349 44k INFO ====> Epoch: 1101, cost 13.37 s | |
| 2024-01-16 02:06:29,588 44k INFO ====> Epoch: 1102, cost 13.24 s | |
| 2024-01-16 02:06:43,030 44k INFO ====> Epoch: 1103, cost 13.44 s | |
| 2024-01-16 02:06:56,375 44k INFO ====> Epoch: 1104, cost 13.34 s | |
| 2024-01-16 02:07:09,554 44k INFO ====> Epoch: 1105, cost 13.18 s | |
| 2024-01-16 02:07:22,848 44k INFO ====> Epoch: 1106, cost 13.29 s | |
| 2024-01-16 02:07:36,270 44k INFO ====> Epoch: 1107, cost 13.42 s | |
| 2024-01-16 02:07:46,751 44k INFO Train Epoch: 1108 [62%] | |
| 2024-01-16 02:07:46,751 44k INFO Losses: [2.408942937850952, 2.373692750930786, 7.201694965362549, 11.795943260192871, 0.7276708483695984], step: 14400, lr: 8.707645595647632e-05, reference_loss: 24.507944107055664 | |
| 2024-01-16 02:07:55,603 44k INFO Saving model and optimizer state at iteration 1108 to ./logs/44k/G_14400.pth | |
| 2024-01-16 02:07:57,667 44k INFO Saving model and optimizer state at iteration 1108 to ./logs/44k/D_14400.pth | |
| 2024-01-16 02:07:58,487 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_10400.pth | |
| 2024-01-16 02:07:58,551 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_10400.pth | |
| 2024-01-16 02:08:01,598 44k INFO ====> Epoch: 1108, cost 25.33 s | |
| 2024-01-16 02:08:15,050 44k INFO ====> Epoch: 1109, cost 13.45 s | |
| 2024-01-16 02:08:28,352 44k INFO ====> Epoch: 1110, cost 13.30 s | |
| 2024-01-16 02:08:41,858 44k INFO ====> Epoch: 1111, cost 13.51 s | |
| 2024-01-16 02:08:55,589 44k INFO ====> Epoch: 1112, cost 13.73 s | |
| 2024-01-16 02:09:09,473 44k INFO ====> Epoch: 1113, cost 13.88 s | |
| 2024-01-16 02:09:22,990 44k INFO ====> Epoch: 1114, cost 13.52 s | |
| 2024-01-16 02:09:36,433 44k INFO ====> Epoch: 1115, cost 13.44 s | |
| 2024-01-16 02:09:49,575 44k INFO ====> Epoch: 1116, cost 13.14 s | |
| 2024-01-16 02:10:02,923 44k INFO ====> Epoch: 1117, cost 13.35 s | |
| 2024-01-16 02:10:16,358 44k INFO ====> Epoch: 1118, cost 13.43 s | |
| 2024-01-16 02:10:30,169 44k INFO ====> Epoch: 1119, cost 13.81 s | |
| 2024-01-16 02:10:43,590 44k INFO ====> Epoch: 1120, cost 13.42 s | |
| 2024-01-16 02:10:57,055 44k INFO ====> Epoch: 1121, cost 13.47 s | |
| 2024-01-16 02:11:10,398 44k INFO ====> Epoch: 1122, cost 13.34 s | |
| 2024-01-16 02:11:23,723 44k INFO ====> Epoch: 1123, cost 13.33 s | |
| 2024-01-16 02:11:31,170 44k INFO Train Epoch: 1124 [0%] | |
| 2024-01-16 02:11:31,171 44k INFO Losses: [2.3323984146118164, 2.4262516498565674, 4.3160810470581055, 13.434046745300293, 0.3757476508617401], step: 14600, lr: 8.690246621771705e-05, reference_loss: 22.884525299072266 | |
| 2024-01-16 02:11:39,927 44k INFO Saving model and optimizer state at iteration 1124 to ./logs/44k/G_14600.pth | |
| 2024-01-16 02:11:41,597 44k INFO Saving model and optimizer state at iteration 1124 to ./logs/44k/D_14600.pth | |
| 2024-01-16 02:11:42,422 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_10600.pth | |
| 2024-01-16 02:11:42,473 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_10600.pth | |
| 2024-01-16 02:11:48,746 44k INFO ====> Epoch: 1124, cost 25.02 s | |
| 2024-01-16 02:12:02,611 44k INFO ====> Epoch: 1125, cost 13.86 s | |
| 2024-01-16 02:12:16,239 44k INFO ====> Epoch: 1126, cost 13.63 s | |
| 2024-01-16 02:12:29,535 44k INFO ====> Epoch: 1127, cost 13.30 s | |
| 2024-01-16 02:12:42,962 44k INFO ====> Epoch: 1128, cost 13.43 s | |
| 2024-01-16 02:12:56,395 44k INFO ====> Epoch: 1129, cost 13.43 s | |
| 2024-01-16 02:13:09,817 44k INFO ====> Epoch: 1130, cost 13.42 s | |
| 2024-01-16 02:13:23,083 44k INFO ====> Epoch: 1131, cost 13.27 s | |
| 2024-01-16 02:13:36,268 44k INFO ====> Epoch: 1132, cost 13.19 s | |
| 2024-01-16 02:13:49,672 44k INFO ====> Epoch: 1133, cost 13.40 s | |
| 2024-01-16 02:14:02,952 44k INFO ====> Epoch: 1134, cost 13.28 s | |
| 2024-01-16 02:14:16,338 44k INFO ====> Epoch: 1135, cost 13.39 s | |
| 2024-01-16 02:14:29,467 44k INFO ====> Epoch: 1136, cost 13.13 s | |
| 2024-01-16 02:14:42,701 44k INFO ====> Epoch: 1137, cost 13.23 s | |
| 2024-01-16 02:14:56,507 44k INFO ====> Epoch: 1138, cost 13.81 s | |
| 2024-01-16 02:15:06,253 44k INFO Train Epoch: 1139 [38%] | |
| 2024-01-16 02:15:06,254 44k INFO Losses: [2.5078279972076416, 2.23087215423584, 5.935174465179443, 13.603960037231445, 0.13912975788116455], step: 14800, lr: 8.67396665907186e-05, reference_loss: 24.416963577270508 | |
| 2024-01-16 02:15:15,208 44k INFO Saving model and optimizer state at iteration 1139 to ./logs/44k/G_14800.pth | |
| 2024-01-16 02:15:16,850 44k INFO Saving model and optimizer state at iteration 1139 to ./logs/44k/D_14800.pth | |
| 2024-01-16 02:15:17,590 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_10800.pth | |
| 2024-01-16 02:15:17,655 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_10800.pth | |
| 2024-01-16 02:15:21,903 44k INFO ====> Epoch: 1139, cost 25.40 s | |
| 2024-01-16 02:15:35,960 44k INFO ====> Epoch: 1140, cost 14.06 s | |
| 2024-01-16 02:15:49,755 44k INFO ====> Epoch: 1141, cost 13.79 s | |
| 2024-01-16 02:16:03,120 44k INFO ====> Epoch: 1142, cost 13.37 s | |
| 2024-01-16 02:16:16,490 44k INFO ====> Epoch: 1143, cost 13.37 s | |
| 2024-01-16 02:16:29,967 44k INFO ====> Epoch: 1144, cost 13.48 s | |
| 2024-01-16 02:16:43,504 44k INFO ====> Epoch: 1145, cost 13.54 s | |
| 2024-01-16 02:16:57,407 44k INFO ====> Epoch: 1146, cost 13.90 s | |
| 2024-01-16 02:17:11,383 44k INFO ====> Epoch: 1147, cost 13.98 s | |
| 2024-01-16 02:17:24,752 44k INFO ====> Epoch: 1148, cost 13.37 s | |
| 2024-01-16 02:17:37,983 44k INFO ====> Epoch: 1149, cost 13.23 s | |
| 2024-01-16 02:17:51,418 44k INFO ====> Epoch: 1150, cost 13.44 s | |
| 2024-01-16 02:18:04,742 44k INFO ====> Epoch: 1151, cost 13.32 s | |
| 2024-01-16 02:18:17,925 44k INFO ====> Epoch: 1152, cost 13.18 s | |
| 2024-01-16 02:18:31,120 44k INFO ====> Epoch: 1153, cost 13.19 s | |
| 2024-01-16 02:18:42,691 44k INFO Train Epoch: 1154 [77%] | |
| 2024-01-16 02:18:42,691 44k INFO Losses: [2.4710984230041504, 2.403066635131836, 4.4430108070373535, 10.161962509155273, 0.13555704057216644], step: 15000, lr: 8.657717194607226e-05, reference_loss: 19.614694595336914 | |
| 2024-01-16 02:18:51,611 44k INFO Saving model and optimizer state at iteration 1154 to ./logs/44k/G_15000.pth | |
| 2024-01-16 02:18:53,615 44k INFO Saving model and optimizer state at iteration 1154 to ./logs/44k/D_15000.pth | |
| 2024-01-16 02:18:54,347 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_11000.pth | |
| 2024-01-16 02:18:54,394 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_11000.pth | |
| 2024-01-16 02:18:56,435 44k INFO ====> Epoch: 1154, cost 25.32 s | |
| 2024-01-16 02:19:09,793 44k INFO ====> Epoch: 1155, cost 13.36 s | |
| 2024-01-16 02:19:23,084 44k INFO ====> Epoch: 1156, cost 13.29 s | |
| 2024-01-16 02:19:36,338 44k INFO ====> Epoch: 1157, cost 13.25 s | |
| 2024-01-16 02:19:49,531 44k INFO ====> Epoch: 1158, cost 13.19 s | |
| 2024-01-16 02:20:03,089 44k INFO ====> Epoch: 1159, cost 13.56 s | |
| 2024-01-16 02:20:16,333 44k INFO ====> Epoch: 1160, cost 13.24 s | |
| 2024-01-16 02:20:29,676 44k INFO ====> Epoch: 1161, cost 13.34 s | |
| 2024-01-16 02:20:42,931 44k INFO ====> Epoch: 1162, cost 13.25 s | |
| 2024-01-16 02:20:56,118 44k INFO ====> Epoch: 1163, cost 13.19 s | |
| 2024-01-16 02:21:09,502 44k INFO ====> Epoch: 1164, cost 13.38 s | |
| 2024-01-16 02:21:22,802 44k INFO ====> Epoch: 1165, cost 13.30 s | |
| 2024-01-16 02:21:36,108 44k INFO ====> Epoch: 1166, cost 13.31 s | |
| 2024-01-16 02:21:49,334 44k INFO ====> Epoch: 1167, cost 13.23 s | |
| 2024-01-16 02:22:03,206 44k INFO ====> Epoch: 1168, cost 13.87 s | |
| 2024-01-16 02:22:16,649 44k INFO ====> Epoch: 1169, cost 13.44 s | |
| 2024-01-16 02:22:24,585 44k INFO Train Epoch: 1170 [15%] | |
| 2024-01-16 02:22:24,586 44k INFO Losses: [2.1988155841827393, 2.4905552864074707, 5.969912052154541, 10.406586647033691, 0.6897792220115662], step: 15200, lr: 8.640417983972213e-05, reference_loss: 21.755647659301758 | |
| 2024-01-16 02:22:32,977 44k INFO Saving model and optimizer state at iteration 1170 to ./logs/44k/G_15200.pth | |
| 2024-01-16 02:22:34,643 44k INFO Saving model and optimizer state at iteration 1170 to ./logs/44k/D_15200.pth | |
| 2024-01-16 02:22:35,366 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_11200.pth | |
| 2024-01-16 02:22:35,425 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_11200.pth | |
| 2024-01-16 02:22:40,924 44k INFO ====> Epoch: 1170, cost 24.27 s | |
| 2024-01-16 02:22:54,652 44k INFO ====> Epoch: 1171, cost 13.73 s | |
| 2024-01-16 02:23:07,957 44k INFO ====> Epoch: 1172, cost 13.30 s | |
| 2024-01-16 02:23:21,732 44k INFO ====> Epoch: 1173, cost 13.78 s | |
| 2024-01-16 02:23:34,781 44k INFO ====> Epoch: 1174, cost 13.05 s | |
| 2024-01-16 02:23:47,994 44k INFO ====> Epoch: 1175, cost 13.21 s | |
| 2024-01-16 02:24:01,853 44k INFO ====> Epoch: 1176, cost 13.86 s | |
| 2024-01-16 02:24:15,197 44k INFO ====> Epoch: 1177, cost 13.34 s | |
| 2024-01-16 02:24:28,444 44k INFO ====> Epoch: 1178, cost 13.25 s | |
| 2024-01-16 02:24:41,983 44k INFO ====> Epoch: 1179, cost 13.54 s | |
| 2024-01-16 02:24:55,342 44k INFO ====> Epoch: 1180, cost 13.36 s | |
| 2024-01-16 02:25:08,656 44k INFO ====> Epoch: 1181, cost 13.31 s | |
| 2024-01-16 02:25:21,768 44k INFO ====> Epoch: 1182, cost 13.11 s | |
| 2024-01-16 02:25:35,151 44k INFO ====> Epoch: 1183, cost 13.38 s | |
| 2024-01-16 02:25:48,471 44k INFO ====> Epoch: 1184, cost 13.32 s | |
| 2024-01-16 02:25:58,689 44k INFO Train Epoch: 1185 [54%] | |
| 2024-01-16 02:25:58,690 44k INFO Losses: [2.035322666168213, 2.821798801422119, 5.579082012176514, 10.265548706054688, 0.2878800928592682], step: 15400, lr: 8.624231368262399e-05, reference_loss: 20.98963165283203 | |
| 2024-01-16 02:26:07,255 44k INFO Saving model and optimizer state at iteration 1185 to ./logs/44k/G_15400.pth | |
| 2024-01-16 02:26:09,167 44k INFO Saving model and optimizer state at iteration 1185 to ./logs/44k/D_15400.pth | |
| 2024-01-16 02:26:09,987 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_11400.pth | |
| 2024-01-16 02:26:10,044 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_11400.pth | |
| 2024-01-16 02:26:13,342 44k INFO ====> Epoch: 1185, cost 24.87 s | |
| 2024-01-16 02:26:27,176 44k INFO ====> Epoch: 1186, cost 13.83 s | |
| 2024-01-16 02:26:40,631 44k INFO ====> Epoch: 1187, cost 13.46 s | |
| 2024-01-16 02:26:53,760 44k INFO ====> Epoch: 1188, cost 13.13 s | |
| 2024-01-16 02:27:06,828 44k INFO ====> Epoch: 1189, cost 13.07 s | |
| 2024-01-16 02:27:19,930 44k INFO ====> Epoch: 1190, cost 13.10 s | |
| 2024-01-16 02:27:33,308 44k INFO ====> Epoch: 1191, cost 13.38 s | |
| 2024-01-16 02:27:46,814 44k INFO ====> Epoch: 1192, cost 13.51 s | |
| 2024-01-16 02:28:00,250 44k INFO ====> Epoch: 1193, cost 13.44 s | |
| 2024-01-16 02:28:13,727 44k INFO ====> Epoch: 1194, cost 13.48 s | |
| 2024-01-16 02:28:27,421 44k INFO ====> Epoch: 1195, cost 13.69 s | |
| 2024-01-16 02:28:40,935 44k INFO ====> Epoch: 1196, cost 13.51 s | |
| 2024-01-16 02:28:54,310 44k INFO ====> Epoch: 1197, cost 13.38 s | |
| 2024-01-16 02:29:07,819 44k INFO ====> Epoch: 1198, cost 13.51 s | |
| 2024-01-16 02:29:21,839 44k INFO ====> Epoch: 1199, cost 14.02 s | |
| 2024-01-16 02:29:34,133 44k INFO Train Epoch: 1200 [92%] | |
| 2024-01-16 02:29:34,134 44k INFO Losses: [2.4361400604248047, 2.068350315093994, 6.4128289222717285, 11.677295684814453, 0.5422136187553406], step: 15600, lr: 8.608075075915251e-05, reference_loss: 23.136829376220703 | |
| 2024-01-16 02:29:42,779 44k INFO Saving model and optimizer state at iteration 1200 to ./logs/44k/G_15600.pth | |
| 2024-01-16 02:29:44,401 44k INFO Saving model and optimizer state at iteration 1200 to ./logs/44k/D_15600.pth | |
| 2024-01-16 02:29:45,174 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_11600.pth | |
| 2024-01-16 02:29:45,236 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_11600.pth | |
| 2024-01-16 02:29:46,403 44k INFO ====> Epoch: 1200, cost 24.56 s | |
| 2024-01-16 02:29:59,835 44k INFO ====> Epoch: 1201, cost 13.43 s | |
| 2024-01-16 02:30:13,221 44k INFO ====> Epoch: 1202, cost 13.39 s | |
| 2024-01-16 02:30:26,393 44k INFO ====> Epoch: 1203, cost 13.17 s | |
| 2024-01-16 02:30:39,852 44k INFO ====> Epoch: 1204, cost 13.46 s | |
| 2024-01-16 02:30:53,026 44k INFO ====> Epoch: 1205, cost 13.17 s | |
| 2024-01-16 02:31:06,616 44k INFO ====> Epoch: 1206, cost 13.59 s | |
| 2024-01-16 02:31:19,818 44k INFO ====> Epoch: 1207, cost 13.20 s | |
| 2024-01-16 02:31:33,221 44k INFO ====> Epoch: 1208, cost 13.40 s | |
| 2024-01-16 02:31:46,365 44k INFO ====> Epoch: 1209, cost 13.14 s | |
| 2024-01-16 02:31:59,454 44k INFO ====> Epoch: 1210, cost 13.09 s | |
| 2024-01-16 02:32:12,960 44k INFO ====> Epoch: 1211, cost 13.51 s | |
| 2024-01-16 02:32:26,352 44k INFO ====> Epoch: 1212, cost 13.39 s | |
| 2024-01-16 02:32:39,468 44k INFO ====> Epoch: 1213, cost 13.12 s | |
| 2024-01-16 02:32:52,977 44k INFO ====> Epoch: 1214, cost 13.51 s | |
| 2024-01-16 02:33:06,782 44k INFO ====> Epoch: 1215, cost 13.80 s | |
| 2024-01-16 02:33:15,888 44k INFO Train Epoch: 1216 [31%] | |
| 2024-01-16 02:33:15,889 44k INFO Losses: [2.687252998352051, 2.0894246101379395, 2.735192060470581, 6.722262859344482, 0.16193847358226776], step: 15800, lr: 8.590875056492924e-05, reference_loss: 14.396071434020996 | |
| 2024-01-16 02:33:24,656 44k INFO Saving model and optimizer state at iteration 1216 to ./logs/44k/G_15800.pth | |
| 2024-01-16 02:33:26,613 44k INFO Saving model and optimizer state at iteration 1216 to ./logs/44k/D_15800.pth | |
| 2024-01-16 02:33:27,329 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_11800.pth | |
| 2024-01-16 02:33:27,386 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_11800.pth | |
| 2024-01-16 02:33:32,014 44k INFO ====> Epoch: 1216, cost 25.23 s | |
| 2024-01-16 02:33:45,464 44k INFO ====> Epoch: 1217, cost 13.45 s | |
| 2024-01-16 02:33:58,577 44k INFO ====> Epoch: 1218, cost 13.11 s | |
| 2024-01-16 02:34:11,790 44k INFO ====> Epoch: 1219, cost 13.21 s | |
| 2024-01-16 02:34:25,567 44k INFO ====> Epoch: 1220, cost 13.78 s | |
| 2024-01-16 02:34:38,704 44k INFO ====> Epoch: 1221, cost 13.14 s | |
| 2024-01-16 02:34:51,865 44k INFO ====> Epoch: 1222, cost 13.16 s | |
| 2024-01-16 02:35:05,173 44k INFO ====> Epoch: 1223, cost 13.31 s | |
| 2024-01-16 02:35:18,460 44k INFO ====> Epoch: 1224, cost 13.29 s | |
| 2024-01-16 02:35:31,731 44k INFO ====> Epoch: 1225, cost 13.27 s | |
| 2024-01-16 02:35:45,039 44k INFO ====> Epoch: 1226, cost 13.31 s | |
| 2024-01-16 02:35:58,020 44k INFO ====> Epoch: 1227, cost 12.98 s | |
| 2024-01-16 02:36:11,478 44k INFO ====> Epoch: 1228, cost 13.46 s | |
| 2024-01-16 02:36:24,537 44k INFO ====> Epoch: 1229, cost 13.06 s | |
| 2024-01-16 02:36:37,838 44k INFO ====> Epoch: 1230, cost 13.30 s | |
| 2024-01-16 02:36:48,823 44k INFO Train Epoch: 1231 [69%] | |
| 2024-01-16 02:36:48,824 44k INFO Losses: [2.1961681842803955, 2.814328908920288, 8.14385986328125, 17.026559829711914, 0.9652736186981201], step: 16000, lr: 8.574781252534775e-05, reference_loss: 31.146188735961914 | |
| 2024-01-16 02:36:57,573 44k INFO Saving model and optimizer state at iteration 1231 to ./logs/44k/G_16000.pth | |
| 2024-01-16 02:36:59,241 44k INFO Saving model and optimizer state at iteration 1231 to ./logs/44k/D_16000.pth | |
| 2024-01-16 02:37:00,062 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_12000.pth | |
| 2024-01-16 02:37:00,125 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_12000.pth | |
| 2024-01-16 02:37:02,453 44k INFO ====> Epoch: 1231, cost 24.61 s | |
| 2024-01-16 02:37:16,099 44k INFO ====> Epoch: 1232, cost 13.65 s | |
| 2024-01-16 02:37:29,303 44k INFO ====> Epoch: 1233, cost 13.20 s | |
| 2024-01-16 02:37:42,768 44k INFO ====> Epoch: 1234, cost 13.47 s | |
| 2024-01-16 02:37:56,062 44k INFO ====> Epoch: 1235, cost 13.29 s | |
| 2024-01-16 02:38:09,296 44k INFO ====> Epoch: 1236, cost 13.23 s | |
| 2024-01-16 02:38:22,802 44k INFO ====> Epoch: 1237, cost 13.51 s | |
| 2024-01-16 02:38:36,047 44k INFO ====> Epoch: 1238, cost 13.25 s | |
| 2024-01-16 02:38:49,233 44k INFO ====> Epoch: 1239, cost 13.19 s | |
| 2024-01-16 02:39:02,737 44k INFO ====> Epoch: 1240, cost 13.50 s | |
| 2024-01-16 02:39:16,235 44k INFO ====> Epoch: 1241, cost 13.50 s | |
| 2024-01-16 02:39:29,539 44k INFO ====> Epoch: 1242, cost 13.30 s | |
| 2024-01-16 02:39:42,919 44k INFO ====> Epoch: 1243, cost 13.38 s | |
| 2024-01-16 02:39:56,839 44k INFO ====> Epoch: 1244, cost 13.92 s | |
| 2024-01-16 02:40:10,438 44k INFO ====> Epoch: 1245, cost 13.60 s | |
| 2024-01-16 02:40:23,694 44k INFO ====> Epoch: 1246, cost 13.26 s | |
| 2024-01-16 02:40:31,370 44k INFO Train Epoch: 1247 [8%] | |
| 2024-01-16 02:40:31,371 44k INFO Losses: [2.2833666801452637, 2.92378830909729, 6.180790901184082, 14.864154815673828, 0.4886663854122162], step: 16200, lr: 8.557647758369688e-05, reference_loss: 26.740768432617188 | |
| 2024-01-16 02:40:39,895 44k INFO Saving model and optimizer state at iteration 1247 to ./logs/44k/G_16200.pth | |
| 2024-01-16 02:40:41,880 44k INFO Saving model and optimizer state at iteration 1247 to ./logs/44k/D_16200.pth | |
| 2024-01-16 02:40:42,625 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_12200.pth | |
| 2024-01-16 02:40:42,677 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_12200.pth | |
| 2024-01-16 02:40:48,318 44k INFO ====> Epoch: 1247, cost 24.62 s | |
| 2024-01-16 02:41:01,666 44k INFO ====> Epoch: 1248, cost 13.35 s | |
| 2024-01-16 02:41:14,953 44k INFO ====> Epoch: 1249, cost 13.29 s | |
| 2024-01-16 02:41:28,206 44k INFO ====> Epoch: 1250, cost 13.25 s | |
| 2024-01-16 02:41:41,516 44k INFO ====> Epoch: 1251, cost 13.31 s | |
| 2024-01-16 02:41:55,105 44k INFO ====> Epoch: 1252, cost 13.59 s | |
| 2024-01-16 02:42:08,629 44k INFO ====> Epoch: 1253, cost 13.52 s | |
| 2024-01-16 02:42:22,028 44k INFO ====> Epoch: 1254, cost 13.40 s | |
| 2024-01-16 02:42:35,243 44k INFO ====> Epoch: 1255, cost 13.21 s | |
| 2024-01-16 02:42:48,637 44k INFO ====> Epoch: 1256, cost 13.39 s | |
| 2024-01-16 02:43:02,133 44k INFO ====> Epoch: 1257, cost 13.50 s | |
| 2024-01-16 02:43:16,403 44k INFO ====> Epoch: 1258, cost 14.27 s | |
| 2024-01-16 02:43:29,799 44k INFO ====> Epoch: 1259, cost 13.40 s | |
| 2024-01-16 02:43:42,861 44k INFO ====> Epoch: 1260, cost 13.06 s | |
| 2024-01-16 02:43:56,127 44k INFO ====> Epoch: 1261, cost 13.27 s | |
| 2024-01-16 02:44:05,842 44k INFO Train Epoch: 1262 [46%] | |
| 2024-01-16 02:44:05,843 44k INFO Losses: [2.3960092067718506, 2.2971208095550537, 6.165021896362305, 12.285541534423828, 0.2570987343788147], step: 16400, lr: 8.541616201111502e-05, reference_loss: 23.400793075561523 | |
| 2024-01-16 02:44:14,431 44k INFO Saving model and optimizer state at iteration 1262 to ./logs/44k/G_16400.pth | |
| 2024-01-16 02:44:16,118 44k INFO Saving model and optimizer state at iteration 1262 to ./logs/44k/D_16400.pth | |
| 2024-01-16 02:44:16,890 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_12400.pth | |
| 2024-01-16 02:44:16,948 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_12400.pth | |
| 2024-01-16 02:44:20,566 44k INFO ====> Epoch: 1262, cost 24.44 s | |
| 2024-01-16 02:44:33,716 44k INFO ====> Epoch: 1263, cost 13.15 s | |
| 2024-01-16 02:44:47,137 44k INFO ====> Epoch: 1264, cost 13.42 s | |
| 2024-01-16 02:45:00,827 44k INFO ====> Epoch: 1265, cost 13.69 s | |
| 2024-01-16 02:45:14,235 44k INFO ====> Epoch: 1266, cost 13.41 s | |
| 2024-01-16 02:45:27,615 44k INFO ====> Epoch: 1267, cost 13.38 s | |
| 2024-01-16 02:45:40,878 44k INFO ====> Epoch: 1268, cost 13.26 s | |
| 2024-01-16 02:45:54,192 44k INFO ====> Epoch: 1269, cost 13.31 s | |
| 2024-01-16 02:46:07,422 44k INFO ====> Epoch: 1270, cost 13.23 s | |
| 2024-01-16 02:46:21,102 44k INFO ====> Epoch: 1271, cost 13.68 s | |
| 2024-01-16 02:46:34,273 44k INFO ====> Epoch: 1272, cost 13.17 s | |
| 2024-01-16 02:46:47,620 44k INFO ====> Epoch: 1273, cost 13.35 s | |
| 2024-01-16 02:47:00,855 44k INFO ====> Epoch: 1274, cost 13.24 s | |
| 2024-01-16 02:47:13,783 44k INFO ====> Epoch: 1275, cost 12.93 s | |
| 2024-01-16 02:47:26,838 44k INFO ====> Epoch: 1276, cost 13.05 s | |
| 2024-01-16 02:47:38,461 44k INFO Train Epoch: 1277 [85%] | |
| 2024-01-16 02:47:38,462 44k INFO Losses: [2.239816427230835, 2.777972936630249, 6.580613136291504, 12.174762725830078, 0.7373439073562622], step: 16600, lr: 8.525614676735643e-05, reference_loss: 24.510509490966797 | |
| 2024-01-16 02:47:47,260 44k INFO Saving model and optimizer state at iteration 1277 to ./logs/44k/G_16600.pth | |
| 2024-01-16 02:47:48,897 44k INFO Saving model and optimizer state at iteration 1277 to ./logs/44k/D_16600.pth | |
| 2024-01-16 02:47:49,632 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_12600.pth | |
| 2024-01-16 02:47:49,697 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_12600.pth | |
| 2024-01-16 02:47:51,203 44k INFO ====> Epoch: 1277, cost 24.37 s | |
| 2024-01-16 02:48:04,556 44k INFO ====> Epoch: 1278, cost 13.35 s | |
| 2024-01-16 02:48:17,997 44k INFO ====> Epoch: 1279, cost 13.44 s | |
| 2024-01-16 02:48:31,240 44k INFO ====> Epoch: 1280, cost 13.24 s | |
| 2024-01-16 02:48:44,443 44k INFO ====> Epoch: 1281, cost 13.20 s | |
| 2024-01-16 02:48:57,952 44k INFO ====> Epoch: 1282, cost 13.51 s | |
| 2024-01-16 02:49:10,997 44k INFO ====> Epoch: 1283, cost 13.04 s | |
| 2024-01-16 02:49:24,357 44k INFO ====> Epoch: 1284, cost 13.36 s | |
| 2024-01-16 02:49:37,590 44k INFO ====> Epoch: 1285, cost 13.23 s | |
| 2024-01-16 02:49:50,801 44k INFO ====> Epoch: 1286, cost 13.21 s | |
| 2024-01-16 02:50:04,232 44k INFO ====> Epoch: 1287, cost 13.43 s | |
| 2024-01-16 02:50:17,106 44k INFO ====> Epoch: 1288, cost 12.87 s | |
| 2024-01-16 02:50:30,542 44k INFO ====> Epoch: 1289, cost 13.44 s | |
| 2024-01-16 02:50:43,899 44k INFO ====> Epoch: 1290, cost 13.36 s | |
| 2024-01-16 02:50:57,041 44k INFO ====> Epoch: 1291, cost 13.14 s | |
| 2024-01-16 02:51:10,506 44k INFO ====> Epoch: 1292, cost 13.46 s | |
| 2024-01-16 02:51:19,004 44k INFO Train Epoch: 1293 [23%] | |
| 2024-01-16 02:51:19,005 44k INFO Losses: [2.1310863494873047, 3.2101902961730957, 8.865270614624023, 14.804034233093262, 0.7550780773162842], step: 16800, lr: 8.50857942358858e-05, reference_loss: 29.76565933227539 | |
| 2024-01-16 02:51:27,508 44k INFO Saving model and optimizer state at iteration 1293 to ./logs/44k/G_16800.pth | |
| 2024-01-16 02:51:29,186 44k INFO Saving model and optimizer state at iteration 1293 to ./logs/44k/D_16800.pth | |
| 2024-01-16 02:51:29,959 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_12800.pth | |
| 2024-01-16 02:51:30,038 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_12800.pth | |
| 2024-01-16 02:51:34,848 44k INFO ====> Epoch: 1293, cost 24.34 s | |
| 2024-01-16 02:51:48,131 44k INFO ====> Epoch: 1294, cost 13.28 s | |
| 2024-01-16 02:52:01,668 44k INFO ====> Epoch: 1295, cost 13.54 s | |
| 2024-01-16 02:52:14,822 44k INFO ====> Epoch: 1296, cost 13.15 s | |
| 2024-01-16 02:52:27,967 44k INFO ====> Epoch: 1297, cost 13.15 s | |
| 2024-01-16 02:52:41,071 44k INFO ====> Epoch: 1298, cost 13.10 s | |
| 2024-01-16 02:52:54,170 44k INFO ====> Epoch: 1299, cost 13.10 s | |
| 2024-01-16 02:53:07,222 44k INFO ====> Epoch: 1300, cost 13.05 s | |
| 2024-01-16 02:53:20,138 44k INFO ====> Epoch: 1301, cost 12.92 s | |
| 2024-01-16 02:53:33,484 44k INFO ====> Epoch: 1302, cost 13.35 s | |
| 2024-01-16 02:53:46,659 44k INFO ====> Epoch: 1303, cost 13.18 s | |
| 2024-01-16 02:53:59,885 44k INFO ====> Epoch: 1304, cost 13.23 s | |
| 2024-01-16 02:54:13,270 44k INFO ====> Epoch: 1305, cost 13.39 s | |
| 2024-01-16 02:54:26,547 44k INFO ====> Epoch: 1306, cost 13.28 s | |
| 2024-01-16 02:54:39,583 44k INFO ====> Epoch: 1307, cost 13.04 s | |
| 2024-01-16 02:54:50,380 44k INFO Train Epoch: 1308 [62%] | |
| 2024-01-16 02:54:50,381 44k INFO Losses: [2.269585609436035, 2.6000068187713623, 7.39261531829834, 11.437586784362793, 0.7004539370536804], step: 17000, lr: 8.492639788998965e-05, reference_loss: 24.400249481201172 | |
| 2024-01-16 02:54:59,138 44k INFO Saving model and optimizer state at iteration 1308 to ./logs/44k/G_17000.pth | |
| 2024-01-16 02:55:00,807 44k INFO Saving model and optimizer state at iteration 1308 to ./logs/44k/D_17000.pth | |
| 2024-01-16 02:55:01,569 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_13000.pth | |
| 2024-01-16 02:55:01,650 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_13000.pth | |
| 2024-01-16 02:55:04,482 44k INFO ====> Epoch: 1308, cost 24.90 s | |
| 2024-01-16 02:55:17,739 44k INFO ====> Epoch: 1309, cost 13.26 s | |
| 2024-01-16 02:55:31,031 44k INFO ====> Epoch: 1310, cost 13.29 s | |
| 2024-01-16 02:55:44,824 44k INFO ====> Epoch: 1311, cost 13.79 s | |
| 2024-01-16 02:55:57,858 44k INFO ====> Epoch: 1312, cost 13.03 s | |
| 2024-01-16 02:56:11,151 44k INFO ====> Epoch: 1313, cost 13.29 s | |
| 2024-01-16 02:56:24,330 44k INFO ====> Epoch: 1314, cost 13.18 s | |
| 2024-01-16 02:56:37,370 44k INFO ====> Epoch: 1315, cost 13.04 s | |
| 2024-01-16 02:56:50,706 44k INFO ====> Epoch: 1316, cost 13.34 s | |
| 2024-01-16 02:57:03,775 44k INFO ====> Epoch: 1317, cost 13.07 s | |
| 2024-01-16 02:57:17,231 44k INFO ====> Epoch: 1318, cost 13.46 s | |
| 2024-01-16 02:57:30,253 44k INFO ====> Epoch: 1319, cost 13.02 s | |
| 2024-01-16 02:57:43,164 44k INFO ====> Epoch: 1320, cost 12.91 s | |
| 2024-01-16 02:57:56,446 44k INFO ====> Epoch: 1321, cost 13.28 s | |
| 2024-01-16 02:58:09,651 44k INFO ====> Epoch: 1322, cost 13.20 s | |
| 2024-01-16 02:58:23,066 44k INFO ====> Epoch: 1323, cost 13.42 s | |
| 2024-01-16 02:58:30,526 44k INFO Train Epoch: 1324 [0%] | |
| 2024-01-16 02:58:30,527 44k INFO Losses: [2.5121350288391113, 2.2930452823638916, 4.040195465087891, 12.85823917388916, 0.368354856967926], step: 17200, lr: 8.47567042383551e-05, reference_loss: 22.071969985961914 | |
| 2024-01-16 02:58:39,157 44k INFO Saving model and optimizer state at iteration 1324 to ./logs/44k/G_17200.pth | |
| 2024-01-16 02:58:41,192 44k INFO Saving model and optimizer state at iteration 1324 to ./logs/44k/D_17200.pth | |
| 2024-01-16 02:58:42,059 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_13200.pth | |
| 2024-01-16 02:58:42,141 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_13200.pth | |
| 2024-01-16 02:58:48,138 44k INFO ====> Epoch: 1324, cost 25.07 s | |
| 2024-01-16 02:59:01,411 44k INFO ====> Epoch: 1325, cost 13.27 s | |
| 2024-01-16 02:59:14,604 44k INFO ====> Epoch: 1326, cost 13.19 s | |
| 2024-01-16 02:59:27,816 44k INFO ====> Epoch: 1327, cost 13.21 s | |
| 2024-01-16 02:59:40,919 44k INFO ====> Epoch: 1328, cost 13.10 s | |
| 2024-01-16 02:59:54,231 44k INFO ====> Epoch: 1329, cost 13.31 s | |
| 2024-01-16 03:00:07,552 44k INFO ====> Epoch: 1330, cost 13.32 s | |
| 2024-01-16 03:00:20,600 44k INFO ====> Epoch: 1331, cost 13.05 s | |
| 2024-01-16 03:00:34,365 44k INFO ====> Epoch: 1332, cost 13.77 s | |
| 2024-01-16 03:00:47,446 44k INFO ====> Epoch: 1333, cost 13.08 s | |
| 2024-01-16 03:01:01,026 44k INFO ====> Epoch: 1334, cost 13.58 s | |
| 2024-01-16 03:01:14,960 44k INFO ====> Epoch: 1335, cost 13.93 s | |
| 2024-01-16 03:01:28,252 44k INFO ====> Epoch: 1336, cost 13.29 s | |
| 2024-01-16 03:01:41,737 44k INFO ====> Epoch: 1337, cost 13.48 s | |
| 2024-01-16 03:01:54,908 44k INFO ====> Epoch: 1338, cost 13.17 s | |
| 2024-01-16 03:02:04,328 44k INFO Train Epoch: 1339 [38%] | |
| 2024-01-16 03:02:04,329 44k INFO Losses: [2.5602593421936035, 2.349890947341919, 6.457817077636719, 13.374466896057129, 0.11884750425815582], step: 17400, lr: 8.459792439658338e-05, reference_loss: 24.861282348632812 | |
| 2024-01-16 03:02:13,062 44k INFO Saving model and optimizer state at iteration 1339 to ./logs/44k/G_17400.pth | |
| 2024-01-16 03:02:14,678 44k INFO Saving model and optimizer state at iteration 1339 to ./logs/44k/D_17400.pth | |
| 2024-01-16 03:02:15,402 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_13400.pth | |
| 2024-01-16 03:02:15,469 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_13400.pth | |
| 2024-01-16 03:02:19,514 44k INFO ====> Epoch: 1339, cost 24.61 s | |
| 2024-01-16 03:02:32,802 44k INFO ====> Epoch: 1340, cost 13.29 s | |
| 2024-01-16 03:02:46,226 44k INFO ====> Epoch: 1341, cost 13.42 s | |
| 2024-01-16 03:02:59,462 44k INFO ====> Epoch: 1342, cost 13.24 s | |
| 2024-01-16 03:03:13,019 44k INFO ====> Epoch: 1343, cost 13.56 s | |
| 2024-01-16 03:03:26,249 44k INFO ====> Epoch: 1344, cost 13.23 s | |
| 2024-01-16 03:03:39,908 44k INFO ====> Epoch: 1345, cost 13.66 s | |
| 2024-01-16 03:03:53,059 44k INFO ====> Epoch: 1346, cost 13.15 s | |
| 2024-01-16 03:04:06,250 44k INFO ====> Epoch: 1347, cost 13.19 s | |
| 2024-01-16 03:04:19,696 44k INFO ====> Epoch: 1348, cost 13.45 s | |
| 2024-01-16 03:04:33,493 44k INFO ====> Epoch: 1349, cost 13.80 s | |
| 2024-01-16 03:04:46,590 44k INFO ====> Epoch: 1350, cost 13.10 s | |
| 2024-01-16 03:04:59,844 44k INFO ====> Epoch: 1351, cost 13.25 s | |
| 2024-01-16 03:05:13,151 44k INFO ====> Epoch: 1352, cost 13.31 s | |
| 2024-01-16 03:05:26,406 44k INFO ====> Epoch: 1353, cost 13.26 s | |
| 2024-01-16 03:05:37,696 44k INFO Train Epoch: 1354 [77%] | |
| 2024-01-16 03:05:37,697 44k INFO Losses: [2.450888156890869, 2.370096206665039, 4.39447021484375, 9.764923095703125, 0.08247176557779312], step: 17600, lr: 8.443944200665783e-05, reference_loss: 19.062849044799805 | |
| 2024-01-16 03:05:46,456 44k INFO Saving model and optimizer state at iteration 1354 to ./logs/44k/G_17600.pth | |
| 2024-01-16 03:05:48,041 44k INFO Saving model and optimizer state at iteration 1354 to ./logs/44k/D_17600.pth | |
| 2024-01-16 03:05:48,730 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_13600.pth | |
| 2024-01-16 03:05:48,784 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_13600.pth | |
| 2024-01-16 03:05:50,783 44k INFO ====> Epoch: 1354, cost 24.38 s | |
| 2024-01-16 03:06:03,962 44k INFO ====> Epoch: 1355, cost 13.18 s | |
| 2024-01-16 03:06:17,814 44k INFO ====> Epoch: 1356, cost 13.85 s | |
| 2024-01-16 03:06:30,899 44k INFO ====> Epoch: 1357, cost 13.08 s | |
| 2024-01-16 03:06:43,884 44k INFO ====> Epoch: 1358, cost 12.98 s | |
| 2024-01-16 03:06:57,026 44k INFO ====> Epoch: 1359, cost 13.14 s | |
| 2024-01-16 03:07:10,577 44k INFO ====> Epoch: 1360, cost 13.55 s | |
| 2024-01-16 03:07:23,846 44k INFO ====> Epoch: 1361, cost 13.27 s | |
| 2024-01-16 03:07:37,231 44k INFO ====> Epoch: 1362, cost 13.38 s | |
| 2024-01-16 03:07:50,690 44k INFO ====> Epoch: 1363, cost 13.46 s | |
| 2024-01-16 03:08:04,201 44k INFO ====> Epoch: 1364, cost 13.51 s | |
| 2024-01-16 03:08:17,285 44k INFO ====> Epoch: 1365, cost 13.08 s | |
| 2024-01-16 03:08:30,470 44k INFO ====> Epoch: 1366, cost 13.18 s | |
| 2024-01-16 03:08:44,263 44k INFO ====> Epoch: 1367, cost 13.79 s | |
| 2024-01-16 03:08:57,327 44k INFO ====> Epoch: 1368, cost 13.06 s | |
| 2024-01-16 03:09:10,569 44k INFO ====> Epoch: 1369, cost 13.24 s | |
| 2024-01-16 03:09:18,535 44k INFO Train Epoch: 1370 [15%] | |
| 2024-01-16 03:09:18,536 44k INFO Losses: [2.4162189960479736, 2.2796285152435303, 5.3539652824401855, 9.69324016571045, 0.6744216680526733], step: 17800, lr: 8.427072135428007e-05, reference_loss: 20.4174747467041 | |
| 2024-01-16 03:09:27,037 44k INFO Saving model and optimizer state at iteration 1370 to ./logs/44k/G_17800.pth | |
| 2024-01-16 03:09:28,580 44k INFO Saving model and optimizer state at iteration 1370 to ./logs/44k/D_17800.pth | |
| 2024-01-16 03:09:29,269 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_13800.pth | |
| 2024-01-16 03:09:29,336 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_13800.pth | |
| 2024-01-16 03:09:34,757 44k INFO ====> Epoch: 1370, cost 24.19 s | |
| 2024-01-16 03:09:48,055 44k INFO ====> Epoch: 1371, cost 13.30 s | |
| 2024-01-16 03:10:01,332 44k INFO ====> Epoch: 1372, cost 13.28 s | |
| 2024-01-16 03:10:14,619 44k INFO ====> Epoch: 1373, cost 13.29 s | |
| 2024-01-16 03:10:27,753 44k INFO ====> Epoch: 1374, cost 13.13 s | |
| 2024-01-16 03:10:41,107 44k INFO ====> Epoch: 1375, cost 13.35 s | |
| 2024-01-16 03:10:54,368 44k INFO ====> Epoch: 1376, cost 13.26 s | |
| 2024-01-16 03:11:07,750 44k INFO ====> Epoch: 1377, cost 13.38 s | |
| 2024-01-16 03:11:20,843 44k INFO ====> Epoch: 1378, cost 13.09 s | |
| 2024-01-16 03:11:34,234 44k INFO ====> Epoch: 1379, cost 13.39 s | |
| 2024-01-16 03:11:47,328 44k INFO ====> Epoch: 1380, cost 13.09 s | |
| 2024-01-16 03:12:00,460 44k INFO ====> Epoch: 1381, cost 13.13 s | |
| 2024-01-16 03:12:13,950 44k INFO ====> Epoch: 1382, cost 13.49 s | |
| 2024-01-16 03:12:27,183 44k INFO ====> Epoch: 1383, cost 13.23 s | |
| 2024-01-16 03:12:40,955 44k INFO ====> Epoch: 1384, cost 13.77 s | |
| 2024-01-16 03:12:50,830 44k INFO Train Epoch: 1385 [54%] | |
| 2024-01-16 03:12:50,831 44k INFO Losses: [2.474069595336914, 2.397641658782959, 4.712841510772705, 9.743277549743652, 0.25294479727745056], step: 18000, lr: 8.411285193353202e-05, reference_loss: 19.58077621459961 | |
| 2024-01-16 03:12:59,200 44k INFO Saving model and optimizer state at iteration 1385 to ./logs/44k/G_18000.pth | |
| 2024-01-16 03:13:00,766 44k INFO Saving model and optimizer state at iteration 1385 to ./logs/44k/D_18000.pth | |
| 2024-01-16 03:13:01,462 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_14000.pth | |
| 2024-01-16 03:13:01,526 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_14000.pth | |
| 2024-01-16 03:13:04,729 44k INFO ====> Epoch: 1385, cost 23.77 s | |
| 2024-01-16 03:13:18,273 44k INFO ====> Epoch: 1386, cost 13.54 s | |
| 2024-01-16 03:13:31,425 44k INFO ====> Epoch: 1387, cost 13.15 s | |
| 2024-01-16 03:13:44,667 44k INFO ====> Epoch: 1388, cost 13.24 s | |
| 2024-01-16 03:13:58,159 44k INFO ====> Epoch: 1389, cost 13.49 s | |
| 2024-01-16 03:14:11,766 44k INFO ====> Epoch: 1390, cost 13.61 s | |
| 2024-01-16 03:14:25,140 44k INFO ====> Epoch: 1391, cost 13.37 s | |
| 2024-01-16 03:14:38,452 44k INFO ====> Epoch: 1392, cost 13.31 s | |
| 2024-01-16 03:14:51,740 44k INFO ====> Epoch: 1393, cost 13.29 s | |
| 2024-01-16 03:15:05,055 44k INFO ====> Epoch: 1394, cost 13.32 s | |
| 2024-01-16 03:15:18,296 44k INFO ====> Epoch: 1395, cost 13.24 s | |
| 2024-01-16 03:15:31,990 44k INFO ====> Epoch: 1396, cost 13.69 s | |
| 2024-01-16 03:15:45,401 44k INFO ====> Epoch: 1397, cost 13.41 s | |
| 2024-01-16 03:15:58,566 44k INFO ====> Epoch: 1398, cost 13.16 s | |
| 2024-01-16 03:16:11,872 44k INFO ====> Epoch: 1399, cost 13.31 s | |
| 2024-01-16 03:16:23,999 44k INFO Train Epoch: 1400 [92%] | |
| 2024-01-16 03:16:23,999 44k INFO Losses: [2.232565402984619, 2.6272802352905273, 6.834888458251953, 11.471871376037598, 0.5079636573791504], step: 18200, lr: 8.395527825908361e-05, reference_loss: 23.674570083618164 | |
| 2024-01-16 03:16:32,628 44k INFO Saving model and optimizer state at iteration 1400 to ./logs/44k/G_18200.pth | |
| 2024-01-16 03:16:34,271 44k INFO Saving model and optimizer state at iteration 1400 to ./logs/44k/D_18200.pth | |
| 2024-01-16 03:16:34,945 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_14200.pth | |
| 2024-01-16 03:16:34,996 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_14200.pth | |
| 2024-01-16 03:16:36,099 44k INFO ====> Epoch: 1400, cost 24.23 s | |
| 2024-01-16 03:16:49,489 44k INFO ====> Epoch: 1401, cost 13.39 s | |
| 2024-01-16 03:17:02,736 44k INFO ====> Epoch: 1402, cost 13.25 s | |
| 2024-01-16 03:17:16,033 44k INFO ====> Epoch: 1403, cost 13.30 s | |
| 2024-01-16 03:17:29,134 44k INFO ====> Epoch: 1404, cost 13.10 s | |
| 2024-01-16 03:17:42,475 44k INFO ====> Epoch: 1405, cost 13.34 s | |
| 2024-01-16 03:17:55,987 44k INFO ====> Epoch: 1406, cost 13.51 s | |
| 2024-01-16 03:18:09,272 44k INFO ====> Epoch: 1407, cost 13.29 s | |
| 2024-01-16 03:18:23,140 44k INFO ====> Epoch: 1408, cost 13.87 s | |
| 2024-01-16 03:18:36,636 44k INFO ====> Epoch: 1409, cost 13.50 s | |
| 2024-01-16 03:18:50,406 44k INFO ====> Epoch: 1410, cost 13.77 s | |
| 2024-01-16 03:19:03,978 44k INFO ====> Epoch: 1411, cost 13.57 s | |
| 2024-01-16 03:19:18,289 44k INFO ====> Epoch: 1412, cost 14.31 s | |
| 2024-01-16 03:19:31,786 44k INFO ====> Epoch: 1413, cost 13.50 s | |
| 2024-01-16 03:19:44,939 44k INFO ====> Epoch: 1414, cost 13.15 s | |
| 2024-01-16 03:19:58,060 44k INFO ====> Epoch: 1415, cost 13.12 s | |
| 2024-01-16 03:20:06,949 44k INFO Train Epoch: 1416 [31%] | |
| 2024-01-16 03:20:06,950 44k INFO Losses: [2.6952223777770996, 2.022585868835449, 2.2917046546936035, 6.570613384246826, 0.12084626406431198], step: 18400, lr: 8.378752502692335e-05, reference_loss: 13.700971603393555 | |
| 2024-01-16 03:20:15,584 44k INFO Saving model and optimizer state at iteration 1416 to ./logs/44k/G_18400.pth | |
| 2024-01-16 03:20:17,186 44k INFO Saving model and optimizer state at iteration 1416 to ./logs/44k/D_18400.pth | |
| 2024-01-16 03:20:17,883 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_14400.pth | |
| 2024-01-16 03:20:17,939 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_14400.pth | |
| 2024-01-16 03:20:22,280 44k INFO ====> Epoch: 1416, cost 24.22 s | |
| 2024-01-16 03:20:35,665 44k INFO ====> Epoch: 1417, cost 13.39 s | |
| 2024-01-16 03:20:48,910 44k INFO ====> Epoch: 1418, cost 13.25 s | |
| 2024-01-16 03:21:02,417 44k INFO ====> Epoch: 1419, cost 13.51 s | |
| 2024-01-16 03:21:15,830 44k INFO ====> Epoch: 1420, cost 13.41 s | |
| 2024-01-16 03:21:29,508 44k INFO ====> Epoch: 1421, cost 13.68 s | |
| 2024-01-16 03:21:42,757 44k INFO ====> Epoch: 1422, cost 13.25 s | |
| 2024-01-16 03:21:56,048 44k INFO ====> Epoch: 1423, cost 13.29 s | |
| 2024-01-16 03:22:09,289 44k INFO ====> Epoch: 1424, cost 13.24 s | |
| 2024-01-16 03:22:22,594 44k INFO ====> Epoch: 1425, cost 13.31 s | |
| 2024-01-16 03:22:35,657 44k INFO ====> Epoch: 1426, cost 13.06 s | |
| 2024-01-16 03:22:48,973 44k INFO ====> Epoch: 1427, cost 13.32 s | |
| 2024-01-16 03:23:02,081 44k INFO ====> Epoch: 1428, cost 13.11 s | |
| 2024-01-16 03:23:15,504 44k INFO ====> Epoch: 1429, cost 13.42 s | |
| 2024-01-16 03:23:28,620 44k INFO ====> Epoch: 1430, cost 13.12 s | |
| 2024-01-16 03:23:39,795 44k INFO Train Epoch: 1431 [69%] | |
| 2024-01-16 03:23:39,796 44k INFO Losses: [2.5661678314208984, 2.341002941131592, 8.080071449279785, 16.831212997436523, 0.9176613092422485], step: 18600, lr: 8.363056080697438e-05, reference_loss: 30.736116409301758 | |
| 2024-01-16 03:23:48,286 44k INFO Saving model and optimizer state at iteration 1431 to ./logs/44k/G_18600.pth | |
| 2024-01-16 03:23:49,858 44k INFO Saving model and optimizer state at iteration 1431 to ./logs/44k/D_18600.pth | |
| 2024-01-16 03:23:50,552 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_14600.pth | |
| 2024-01-16 03:23:50,612 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_14600.pth | |
| 2024-01-16 03:23:52,941 44k INFO ====> Epoch: 1431, cost 24.32 s | |
| 2024-01-16 03:24:06,288 44k INFO ====> Epoch: 1432, cost 13.35 s | |
| 2024-01-16 03:24:19,567 44k INFO ====> Epoch: 1433, cost 13.28 s | |
| 2024-01-16 03:24:32,770 44k INFO ====> Epoch: 1434, cost 13.20 s | |
| 2024-01-16 03:24:45,926 44k INFO ====> Epoch: 1435, cost 13.16 s | |
| 2024-01-16 03:24:59,102 44k INFO ====> Epoch: 1436, cost 13.18 s | |
| 2024-01-16 03:25:12,461 44k INFO ====> Epoch: 1437, cost 13.36 s | |
| 2024-01-16 03:25:26,177 44k INFO ====> Epoch: 1438, cost 13.72 s | |
| 2024-01-16 03:25:39,624 44k INFO ====> Epoch: 1439, cost 13.45 s | |
| 2024-01-16 03:25:52,900 44k INFO ====> Epoch: 1440, cost 13.28 s | |
| 2024-01-16 03:26:06,092 44k INFO ====> Epoch: 1441, cost 13.19 s | |
| 2024-01-16 03:26:19,024 44k INFO ====> Epoch: 1442, cost 12.93 s | |
| 2024-01-16 03:26:32,165 44k INFO ====> Epoch: 1443, cost 13.14 s | |
| 2024-01-16 03:26:45,307 44k INFO ====> Epoch: 1444, cost 13.14 s | |
| 2024-01-16 03:26:58,494 44k INFO ====> Epoch: 1445, cost 13.19 s | |
| 2024-01-16 03:27:11,622 44k INFO ====> Epoch: 1446, cost 13.13 s | |
| 2024-01-16 03:27:19,198 44k INFO Train Epoch: 1447 [8%] | |
| 2024-01-16 03:27:19,199 44k INFO Losses: [2.234853506088257, 2.5373334884643555, 6.093031883239746, 14.422994613647461, 0.4858790338039398], step: 18800, lr: 8.346345640122811e-05, reference_loss: 25.774093627929688 | |
| 2024-01-16 03:27:27,359 44k INFO Saving model and optimizer state at iteration 1447 to ./logs/44k/G_18800.pth | |
| 2024-01-16 03:27:28,927 44k INFO Saving model and optimizer state at iteration 1447 to ./logs/44k/D_18800.pth | |
| 2024-01-16 03:27:29,605 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_14800.pth | |
| 2024-01-16 03:27:29,672 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_14800.pth | |
| 2024-01-16 03:27:35,549 44k INFO ====> Epoch: 1447, cost 23.93 s | |
| 2024-01-16 03:27:48,900 44k INFO ====> Epoch: 1448, cost 13.35 s | |
| 2024-01-16 03:28:02,120 44k INFO ====> Epoch: 1449, cost 13.22 s | |
| 2024-01-16 03:28:15,562 44k INFO ====> Epoch: 1450, cost 13.44 s | |
| 2024-01-16 03:28:28,852 44k INFO ====> Epoch: 1451, cost 13.29 s | |
| 2024-01-16 03:28:42,172 44k INFO ====> Epoch: 1452, cost 13.32 s | |
| 2024-01-16 03:28:55,505 44k INFO ====> Epoch: 1453, cost 13.33 s | |
| 2024-01-16 03:29:08,821 44k INFO ====> Epoch: 1454, cost 13.32 s | |
| 2024-01-16 03:29:21,941 44k INFO ====> Epoch: 1455, cost 13.12 s | |
| 2024-01-16 03:29:35,290 44k INFO ====> Epoch: 1456, cost 13.35 s | |
| 2024-01-16 03:29:48,537 44k INFO ====> Epoch: 1457, cost 13.25 s | |
| 2024-01-16 03:30:01,524 44k INFO ====> Epoch: 1458, cost 12.99 s | |
| 2024-01-16 03:30:15,324 44k INFO ====> Epoch: 1459, cost 13.80 s | |
| 2024-01-16 03:30:28,589 44k INFO ====> Epoch: 1460, cost 13.26 s | |
| 2024-01-16 03:30:42,357 44k INFO ====> Epoch: 1461, cost 13.77 s | |
| 2024-01-16 03:30:52,259 44k INFO Train Epoch: 1462 [46%] | |
| 2024-01-16 03:30:52,260 44k INFO Losses: [2.2928218841552734, 2.659043073654175, 6.695096492767334, 12.24886417388916, 0.24203607439994812], step: 19000, lr: 8.330709927856511e-05, reference_loss: 24.137861251831055 | |
| 2024-01-16 03:31:00,731 44k INFO Saving model and optimizer state at iteration 1462 to ./logs/44k/G_19000.pth | |
| 2024-01-16 03:31:02,274 44k INFO Saving model and optimizer state at iteration 1462 to ./logs/44k/D_19000.pth | |
| 2024-01-16 03:31:02,952 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_15000.pth | |
| 2024-01-16 03:31:03,020 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_15000.pth | |
| 2024-01-16 03:31:06,566 44k INFO ====> Epoch: 1462, cost 24.21 s | |
| 2024-01-16 03:31:19,882 44k INFO ====> Epoch: 1463, cost 13.32 s | |
| 2024-01-16 03:31:33,143 44k INFO ====> Epoch: 1464, cost 13.26 s | |
| 2024-01-16 03:31:46,532 44k INFO ====> Epoch: 1465, cost 13.39 s | |
| 2024-01-16 03:32:00,001 44k INFO ====> Epoch: 1466, cost 13.47 s | |
| 2024-01-16 03:32:13,373 44k INFO ====> Epoch: 1467, cost 13.37 s | |
| 2024-01-16 03:32:27,066 44k INFO ====> Epoch: 1468, cost 13.69 s | |
| 2024-01-16 03:32:40,729 44k INFO ====> Epoch: 1469, cost 13.66 s | |
| 2024-01-16 03:32:53,863 44k INFO ====> Epoch: 1470, cost 13.13 s | |
| 2024-01-16 03:33:07,250 44k INFO ====> Epoch: 1471, cost 13.39 s | |
| 2024-01-16 03:33:20,548 44k INFO ====> Epoch: 1472, cost 13.30 s | |
| 2024-01-16 03:33:33,649 44k INFO ====> Epoch: 1473, cost 13.10 s | |
| 2024-01-16 03:33:46,945 44k INFO ====> Epoch: 1474, cost 13.30 s | |
| 2024-01-16 03:34:00,320 44k INFO ====> Epoch: 1475, cost 13.38 s | |
| 2024-01-16 03:34:13,520 44k INFO ====> Epoch: 1476, cost 13.20 s | |
| 2024-01-16 03:34:25,059 44k INFO Train Epoch: 1477 [85%] | |
| 2024-01-16 03:34:25,060 44k INFO Losses: [2.29891300201416, 2.696164846420288, 6.277185440063477, 11.686456680297852, 0.7293508648872375], step: 19200, lr: 8.315103506912256e-05, reference_loss: 23.688072204589844 | |
| 2024-01-16 03:34:33,802 44k INFO Saving model and optimizer state at iteration 1477 to ./logs/44k/G_19200.pth | |
| 2024-01-16 03:34:35,356 44k INFO Saving model and optimizer state at iteration 1477 to ./logs/44k/D_19200.pth | |
| 2024-01-16 03:34:36,050 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_15200.pth | |
| 2024-01-16 03:34:36,113 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_15200.pth | |
| 2024-01-16 03:34:37,639 44k INFO ====> Epoch: 1477, cost 24.12 s | |
| 2024-01-16 03:34:51,255 44k INFO ====> Epoch: 1478, cost 13.62 s | |
| 2024-01-16 03:35:04,498 44k INFO ====> Epoch: 1479, cost 13.24 s | |
| 2024-01-16 03:35:18,161 44k INFO ====> Epoch: 1480, cost 13.66 s | |
| 2024-01-16 03:35:31,295 44k INFO ====> Epoch: 1481, cost 13.13 s | |
| 2024-01-16 03:35:44,407 44k INFO ====> Epoch: 1482, cost 13.11 s | |
| 2024-01-16 03:35:57,954 44k INFO ====> Epoch: 1483, cost 13.55 s | |
| 2024-01-16 03:36:11,332 44k INFO ====> Epoch: 1484, cost 13.38 s | |
| 2024-01-16 03:36:24,630 44k INFO ====> Epoch: 1485, cost 13.30 s | |
| 2024-01-16 03:36:38,254 44k INFO ====> Epoch: 1486, cost 13.62 s | |
| 2024-01-16 03:36:51,317 44k INFO ====> Epoch: 1487, cost 13.06 s | |
| 2024-01-16 03:37:04,595 44k INFO ====> Epoch: 1488, cost 13.28 s | |
| 2024-01-16 03:37:18,272 44k INFO ====> Epoch: 1489, cost 13.68 s | |
| 2024-01-16 03:37:31,861 44k INFO ====> Epoch: 1490, cost 13.59 s | |
| 2024-01-16 03:37:45,269 44k INFO ====> Epoch: 1491, cost 13.41 s | |
| 2024-01-16 03:37:58,568 44k INFO ====> Epoch: 1492, cost 13.30 s | |
| 2024-01-16 03:38:07,083 44k INFO Train Epoch: 1493 [23%] | |
| 2024-01-16 03:38:07,084 44k INFO Losses: [2.0228939056396484, 3.10616135597229, 9.102595329284668, 14.432733535766602, 0.6415563225746155], step: 19400, lr: 8.29848888162655e-05, reference_loss: 29.305938720703125 | |
| 2024-01-16 03:38:15,501 44k INFO Saving model and optimizer state at iteration 1493 to ./logs/44k/G_19400.pth | |
| 2024-01-16 03:38:17,072 44k INFO Saving model and optimizer state at iteration 1493 to ./logs/44k/D_19400.pth | |
| 2024-01-16 03:38:18,086 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_15400.pth | |
| 2024-01-16 03:38:18,152 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_15400.pth | |
| 2024-01-16 03:38:22,924 44k INFO ====> Epoch: 1493, cost 24.36 s | |
| 2024-01-16 03:38:36,053 44k INFO ====> Epoch: 1494, cost 13.13 s | |
| 2024-01-16 03:38:49,579 44k INFO ====> Epoch: 1495, cost 13.53 s | |
| 2024-01-16 03:39:02,651 44k INFO ====> Epoch: 1496, cost 13.07 s | |
| 2024-01-16 03:39:16,135 44k INFO ====> Epoch: 1497, cost 13.48 s | |
| 2024-01-16 03:39:29,489 44k INFO ====> Epoch: 1498, cost 13.35 s | |
| 2024-01-16 03:39:42,553 44k INFO ====> Epoch: 1499, cost 13.06 s | |
| 2024-01-16 03:39:55,818 44k INFO ====> Epoch: 1500, cost 13.26 s | |
| 2024-01-16 03:40:09,231 44k INFO ====> Epoch: 1501, cost 13.41 s | |
| 2024-01-16 03:40:22,663 44k INFO ====> Epoch: 1502, cost 13.43 s | |
| 2024-01-16 03:40:36,413 44k INFO ====> Epoch: 1503, cost 13.75 s | |
| 2024-01-16 03:40:49,672 44k INFO ====> Epoch: 1504, cost 13.26 s | |
| 2024-01-16 03:41:03,171 44k INFO ====> Epoch: 1505, cost 13.50 s | |
| 2024-01-16 03:41:16,515 44k INFO ====> Epoch: 1506, cost 13.34 s | |
| 2024-01-16 03:41:29,446 44k INFO ====> Epoch: 1507, cost 12.93 s | |
| 2024-01-16 03:41:40,120 44k INFO Train Epoch: 1508 [62%] | |
| 2024-01-16 03:41:40,121 44k INFO Losses: [2.3100390434265137, 2.613957166671753, 7.200765609741211, 11.298064231872559, 0.7047901749610901], step: 19600, lr: 8.282942822309947e-05, reference_loss: 24.127614974975586 | |
| 2024-01-16 03:41:48,923 44k INFO Saving model and optimizer state at iteration 1508 to ./logs/44k/G_19600.pth | |
| 2024-01-16 03:41:50,538 44k INFO Saving model and optimizer state at iteration 1508 to ./logs/44k/D_19600.pth | |
| 2024-01-16 03:41:51,274 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_15600.pth | |
| 2024-01-16 03:41:51,342 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_15600.pth | |
| 2024-01-16 03:41:54,146 44k INFO ====> Epoch: 1508, cost 24.70 s | |
| 2024-01-16 03:42:07,410 44k INFO ====> Epoch: 1509, cost 13.26 s | |
| 2024-01-16 03:42:21,162 44k INFO ====> Epoch: 1510, cost 13.75 s | |
| 2024-01-16 03:42:34,426 44k INFO ====> Epoch: 1511, cost 13.26 s | |
| 2024-01-16 03:42:47,764 44k INFO ====> Epoch: 1512, cost 13.34 s | |
| 2024-01-16 03:43:01,447 44k INFO ====> Epoch: 1513, cost 13.68 s | |
| 2024-01-16 03:43:14,724 44k INFO ====> Epoch: 1514, cost 13.28 s | |
| 2024-01-16 03:43:28,258 44k INFO ====> Epoch: 1515, cost 13.53 s | |
| 2024-01-16 03:43:41,354 44k INFO ====> Epoch: 1516, cost 13.10 s | |
| 2024-01-16 03:43:54,493 44k INFO ====> Epoch: 1517, cost 13.14 s | |
| 2024-01-16 03:44:07,748 44k INFO ====> Epoch: 1518, cost 13.25 s | |
| 2024-01-16 03:44:20,930 44k INFO ====> Epoch: 1519, cost 13.18 s | |
| 2024-01-16 03:44:34,106 44k INFO ====> Epoch: 1520, cost 13.18 s | |
| 2024-01-16 03:44:47,103 44k INFO ====> Epoch: 1521, cost 13.00 s | |
| 2024-01-16 03:45:00,408 44k INFO ====> Epoch: 1522, cost 13.30 s | |
| 2024-01-16 03:45:13,669 44k INFO ====> Epoch: 1523, cost 13.26 s | |
| 2024-01-16 03:45:20,836 44k INFO Train Epoch: 1524 [0%] | |
| 2024-01-16 03:45:20,837 44k INFO Losses: [2.3633952140808105, 2.3258423805236816, 4.117143630981445, 13.039483070373535, 0.2970941364765167], step: 19800, lr: 8.266392458127321e-05, reference_loss: 22.14295768737793 | |
| 2024-01-16 03:45:29,332 44k INFO Saving model and optimizer state at iteration 1524 to ./logs/44k/G_19800.pth | |
| 2024-01-16 03:45:30,864 44k INFO Saving model and optimizer state at iteration 1524 to ./logs/44k/D_19800.pth | |
| 2024-01-16 03:45:31,542 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_15800.pth | |
| 2024-01-16 03:45:31,601 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_15800.pth | |
| 2024-01-16 03:45:37,878 44k INFO ====> Epoch: 1524, cost 24.21 s | |
| 2024-01-16 03:45:51,073 44k INFO ====> Epoch: 1525, cost 13.19 s | |
| 2024-01-16 03:46:04,345 44k INFO ====> Epoch: 1526, cost 13.27 s | |
| 2024-01-16 03:46:17,446 44k INFO ====> Epoch: 1527, cost 13.10 s | |
| 2024-01-16 03:46:30,664 44k INFO ====> Epoch: 1528, cost 13.22 s | |
| 2024-01-16 03:46:44,153 44k INFO ====> Epoch: 1529, cost 13.49 s | |
| 2024-01-16 03:46:57,476 44k INFO ====> Epoch: 1530, cost 13.32 s | |
| 2024-01-16 03:47:10,621 44k INFO ====> Epoch: 1531, cost 13.15 s | |
| 2024-01-16 03:47:23,758 44k INFO ====> Epoch: 1532, cost 13.14 s | |
| 2024-01-16 03:47:37,417 44k INFO ====> Epoch: 1533, cost 13.66 s | |
| 2024-01-16 03:47:50,676 44k INFO ====> Epoch: 1534, cost 13.26 s | |
| 2024-01-16 03:48:04,351 44k INFO ====> Epoch: 1535, cost 13.68 s | |
| 2024-01-16 03:48:18,002 44k INFO ====> Epoch: 1536, cost 13.65 s | |
| 2024-01-16 03:48:31,011 44k INFO ====> Epoch: 1537, cost 13.01 s | |
| 2024-01-16 03:48:44,236 44k INFO ====> Epoch: 1538, cost 13.23 s | |
| 2024-01-16 03:48:53,485 44k INFO Train Epoch: 1539 [38%] | |
| 2024-01-16 03:48:53,486 44k INFO Losses: [2.5169944763183594, 2.4653170108795166, 6.15132474899292, 13.192160606384277, 0.17877936363220215], step: 20000, lr: 8.250906526975097e-05, reference_loss: 24.50457763671875 | |
| 2024-01-16 03:49:01,911 44k INFO Saving model and optimizer state at iteration 1539 to ./logs/44k/G_20000.pth | |
| 2024-01-16 03:49:03,484 44k INFO Saving model and optimizer state at iteration 1539 to ./logs/44k/D_20000.pth | |
| 2024-01-16 03:49:04,179 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_16000.pth | |
| 2024-01-16 03:49:04,241 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_16000.pth | |
| 2024-01-16 03:49:08,246 44k INFO ====> Epoch: 1539, cost 24.01 s | |
| 2024-01-16 03:49:21,382 44k INFO ====> Epoch: 1540, cost 13.14 s | |
| 2024-01-16 03:49:35,245 44k INFO ====> Epoch: 1541, cost 13.86 s | |
| 2024-01-16 03:49:49,076 44k INFO ====> Epoch: 1542, cost 13.83 s | |
| 2024-01-16 03:50:02,317 44k INFO ====> Epoch: 1543, cost 13.24 s | |
| 2024-01-16 03:50:15,569 44k INFO ====> Epoch: 1544, cost 13.25 s | |
| 2024-01-16 03:50:28,787 44k INFO ====> Epoch: 1545, cost 13.22 s | |
| 2024-01-16 03:50:42,140 44k INFO ====> Epoch: 1546, cost 13.35 s | |
| 2024-01-16 03:50:55,412 44k INFO ====> Epoch: 1547, cost 13.27 s | |
| 2024-01-16 03:51:08,768 44k INFO ====> Epoch: 1548, cost 13.36 s | |
| 2024-01-16 03:51:21,771 44k INFO ====> Epoch: 1549, cost 13.00 s | |
| 2024-01-16 03:51:35,152 44k INFO ====> Epoch: 1550, cost 13.38 s | |
| 2024-01-16 03:51:48,320 44k INFO ====> Epoch: 1551, cost 13.17 s | |
| 2024-01-16 03:52:02,140 44k INFO ====> Epoch: 1552, cost 13.82 s | |
| 2024-01-16 03:52:15,654 44k INFO ====> Epoch: 1553, cost 13.51 s | |
| 2024-01-16 03:52:27,284 44k INFO Train Epoch: 1554 [77%] | |
| 2024-01-16 03:52:27,285 44k INFO Losses: [2.6180505752563477, 2.1038777828216553, 4.5039286613464355, 9.532249450683594, 0.08415403962135315], step: 20200, lr: 8.235449606550931e-05, reference_loss: 18.842260360717773 | |
| 2024-01-16 03:52:36,015 44k INFO Saving model and optimizer state at iteration 1554 to ./logs/44k/G_20200.pth | |
| 2024-01-16 03:52:37,682 44k INFO Saving model and optimizer state at iteration 1554 to ./logs/44k/D_20200.pth | |
| 2024-01-16 03:52:38,426 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_16200.pth | |
| 2024-01-16 03:52:38,487 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_16200.pth | |
| 2024-01-16 03:52:40,462 44k INFO ====> Epoch: 1554, cost 24.81 s | |
| 2024-01-16 03:52:53,502 44k INFO ====> Epoch: 1555, cost 13.04 s | |
| 2024-01-16 03:53:06,875 44k INFO ====> Epoch: 1556, cost 13.37 s | |
| 2024-01-16 03:53:20,054 44k INFO ====> Epoch: 1557, cost 13.18 s | |
| 2024-01-16 03:53:33,252 44k INFO ====> Epoch: 1558, cost 13.20 s | |
| 2024-01-16 03:53:46,493 44k INFO ====> Epoch: 1559, cost 13.24 s | |
| 2024-01-16 03:53:59,645 44k INFO ====> Epoch: 1560, cost 13.15 s | |
| 2024-01-16 03:54:12,692 44k INFO ====> Epoch: 1561, cost 13.05 s | |
| 2024-01-16 03:54:26,090 44k INFO ====> Epoch: 1562, cost 13.40 s | |
| 2024-01-16 03:54:39,529 44k INFO ====> Epoch: 1563, cost 13.44 s | |
| 2024-01-16 03:54:52,858 44k INFO ====> Epoch: 1564, cost 13.33 s | |
| 2024-01-16 03:55:06,075 44k INFO ====> Epoch: 1565, cost 13.22 s | |
| 2024-01-16 03:55:19,176 44k INFO ====> Epoch: 1566, cost 13.10 s | |
| 2024-01-16 03:55:32,384 44k INFO ====> Epoch: 1567, cost 13.21 s | |
| 2024-01-16 03:55:45,342 44k INFO ====> Epoch: 1568, cost 12.96 s | |
| 2024-01-16 03:55:58,499 44k INFO ====> Epoch: 1569, cost 13.16 s | |
| 2024-01-16 03:56:06,541 44k INFO Train Epoch: 1570 [15%] | |
| 2024-01-16 03:56:06,542 44k INFO Losses: [1.7677539587020874, 2.8337812423706055, 7.225560188293457, 9.9029541015625, 0.6536601185798645], step: 20400, lr: 8.21899413980197e-05, reference_loss: 22.383708953857422 | |
| 2024-01-16 03:56:14,991 44k INFO Saving model and optimizer state at iteration 1570 to ./logs/44k/G_20400.pth | |
| 2024-01-16 03:56:16,893 44k INFO Saving model and optimizer state at iteration 1570 to ./logs/44k/D_20400.pth | |
| 2024-01-16 03:56:17,581 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_16400.pth | |
| 2024-01-16 03:56:17,654 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_16400.pth | |
| 2024-01-16 03:56:23,277 44k INFO ====> Epoch: 1570, cost 24.78 s | |
| 2024-01-16 03:56:36,929 44k INFO ====> Epoch: 1571, cost 13.65 s | |
| 2024-01-16 03:56:50,644 44k INFO ====> Epoch: 1572, cost 13.71 s | |
| 2024-01-16 03:57:03,731 44k INFO ====> Epoch: 1573, cost 13.09 s | |
| 2024-01-16 03:57:16,955 44k INFO ====> Epoch: 1574, cost 13.22 s | |
| 2024-01-16 03:57:30,248 44k INFO ====> Epoch: 1575, cost 13.29 s | |
| 2024-01-16 03:57:43,899 44k INFO ====> Epoch: 1576, cost 13.65 s | |
| 2024-01-16 03:57:57,065 44k INFO ====> Epoch: 1577, cost 13.17 s | |
| 2024-01-16 03:58:10,122 44k INFO ====> Epoch: 1578, cost 13.06 s | |
| 2024-01-16 03:58:23,354 44k INFO ====> Epoch: 1579, cost 13.23 s | |
| 2024-01-16 03:58:36,628 44k INFO ====> Epoch: 1580, cost 13.27 s | |
| 2024-01-16 03:58:50,295 44k INFO ====> Epoch: 1581, cost 13.67 s | |
| 2024-01-16 03:59:03,339 44k INFO ====> Epoch: 1582, cost 13.04 s | |
| 2024-01-16 03:59:16,303 44k INFO ====> Epoch: 1583, cost 12.96 s | |
| 2024-01-16 03:59:29,974 44k INFO ====> Epoch: 1584, cost 13.67 s | |
| 2024-01-16 03:59:39,885 44k INFO Train Epoch: 1585 [54%] | |
| 2024-01-16 03:59:39,886 44k INFO Losses: [2.6109261512756348, 2.2831051349639893, 5.164974689483643, 9.724721908569336, 0.24837909638881683], step: 20600, lr: 8.203597002775846e-05, reference_loss: 20.032106399536133 | |
| 2024-01-16 03:59:48,275 44k INFO Saving model and optimizer state at iteration 1585 to ./logs/44k/G_20600.pth | |
| 2024-01-16 03:59:49,873 44k INFO Saving model and optimizer state at iteration 1585 to ./logs/44k/D_20600.pth | |
| 2024-01-16 03:59:50,555 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_16600.pth | |
| 2024-01-16 03:59:50,621 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_16600.pth | |
| 2024-01-16 03:59:53,742 44k INFO ====> Epoch: 1585, cost 23.77 s | |
| 2024-01-16 04:00:06,917 44k INFO ====> Epoch: 1586, cost 13.18 s | |
| 2024-01-16 04:00:20,234 44k INFO ====> Epoch: 1587, cost 13.32 s | |
| 2024-01-16 04:00:33,131 44k INFO ====> Epoch: 1588, cost 12.90 s | |
| 2024-01-16 04:00:46,493 44k INFO ====> Epoch: 1589, cost 13.36 s | |
| 2024-01-16 04:00:59,613 44k INFO ====> Epoch: 1590, cost 13.12 s | |
| 2024-01-16 04:01:12,745 44k INFO ====> Epoch: 1591, cost 13.13 s | |
| 2024-01-16 04:01:26,045 44k INFO ====> Epoch: 1592, cost 13.30 s | |
| 2024-01-16 04:01:39,188 44k INFO ====> Epoch: 1593, cost 13.14 s | |
| 2024-01-16 04:01:52,600 44k INFO ====> Epoch: 1594, cost 13.41 s | |
| 2024-01-16 04:02:05,993 44k INFO ====> Epoch: 1595, cost 13.39 s | |
| 2024-01-16 04:02:19,222 44k INFO ====> Epoch: 1596, cost 13.23 s | |
| 2024-01-16 04:02:32,200 44k INFO ====> Epoch: 1597, cost 12.98 s | |
| 2024-01-16 04:02:45,372 44k INFO ====> Epoch: 1598, cost 13.17 s | |
| 2024-01-16 04:02:58,721 44k INFO ====> Epoch: 1599, cost 13.35 s | |
| 2024-01-16 04:03:10,832 44k INFO Train Epoch: 1600 [92%] | |
| 2024-01-16 04:03:10,833 44k INFO Losses: [2.4754199981689453, 2.2840044498443604, 6.714156150817871, 11.109618186950684, 0.491815447807312], step: 20800, lr: 8.188228710134397e-05, reference_loss: 23.075014114379883 | |
| 2024-01-16 04:03:19,602 44k INFO Saving model and optimizer state at iteration 1600 to ./logs/44k/G_20800.pth | |
| 2024-01-16 04:03:21,321 44k INFO Saving model and optimizer state at iteration 1600 to ./logs/44k/D_20800.pth | |
| 2024-01-16 04:03:22,019 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_16800.pth | |
| 2024-01-16 04:03:22,079 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_16800.pth | |
| 2024-01-16 04:03:23,153 44k INFO ====> Epoch: 1600, cost 24.43 s | |
| 2024-01-16 04:03:36,447 44k INFO ====> Epoch: 1601, cost 13.29 s | |
| 2024-01-16 04:03:49,789 44k INFO ====> Epoch: 1602, cost 13.34 s | |
| 2024-01-16 04:04:03,321 44k INFO ====> Epoch: 1603, cost 13.53 s | |
| 2024-01-16 04:04:16,555 44k INFO ====> Epoch: 1604, cost 13.23 s | |
| 2024-01-16 04:04:29,696 44k INFO ====> Epoch: 1605, cost 13.14 s | |
| 2024-01-16 04:04:43,318 44k INFO ====> Epoch: 1606, cost 13.62 s | |
| 2024-01-16 04:04:56,581 44k INFO ====> Epoch: 1607, cost 13.26 s | |
| 2024-01-16 04:05:09,792 44k INFO ====> Epoch: 1608, cost 13.21 s | |
| 2024-01-16 04:05:23,099 44k INFO ====> Epoch: 1609, cost 13.31 s | |
| 2024-01-16 04:05:36,719 44k INFO ====> Epoch: 1610, cost 13.62 s | |
| 2024-01-16 04:05:49,956 44k INFO ====> Epoch: 1611, cost 13.24 s | |
| 2024-01-16 04:06:03,591 44k INFO ====> Epoch: 1612, cost 13.64 s | |
| 2024-01-16 04:06:17,117 44k INFO ====> Epoch: 1613, cost 13.53 s | |
| 2024-01-16 04:06:30,542 44k INFO ====> Epoch: 1614, cost 13.42 s | |
| 2024-01-16 04:06:43,707 44k INFO ====> Epoch: 1615, cost 13.17 s | |
| 2024-01-16 04:06:52,594 44k INFO Train Epoch: 1616 [31%] | |
| 2024-01-16 04:06:52,595 44k INFO Losses: [2.706239938735962, 2.654494524002075, 2.762467861175537, 6.80082893371582, 0.10565078258514404], step: 21000, lr: 8.171867596690716e-05, reference_loss: 15.029682159423828 | |
| 2024-01-16 04:07:00,914 44k INFO Saving model and optimizer state at iteration 1616 to ./logs/44k/G_21000.pth | |
| 2024-01-16 04:07:02,489 44k INFO Saving model and optimizer state at iteration 1616 to ./logs/44k/D_21000.pth | |
| 2024-01-16 04:07:03,167 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_17000.pth | |
| 2024-01-16 04:07:03,238 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_17000.pth | |
| 2024-01-16 04:07:07,454 44k INFO ====> Epoch: 1616, cost 23.75 s | |
| 2024-01-16 04:07:20,988 44k INFO ====> Epoch: 1617, cost 13.53 s | |
| 2024-01-16 04:07:33,901 44k INFO ====> Epoch: 1618, cost 12.91 s | |
| 2024-01-16 04:07:47,089 44k INFO ====> Epoch: 1619, cost 13.19 s | |
| 2024-01-16 04:08:00,123 44k INFO ====> Epoch: 1620, cost 13.03 s | |
| 2024-01-16 04:08:13,375 44k INFO ====> Epoch: 1621, cost 13.25 s | |
| 2024-01-16 04:08:26,656 44k INFO ====> Epoch: 1622, cost 13.28 s | |
| 2024-01-16 04:08:40,049 44k INFO ====> Epoch: 1623, cost 13.39 s | |
| 2024-01-16 04:08:53,169 44k INFO ====> Epoch: 1624, cost 13.12 s | |
| 2024-01-16 04:09:06,202 44k INFO ====> Epoch: 1625, cost 13.03 s | |
| 2024-01-16 04:09:19,474 44k INFO ====> Epoch: 1626, cost 13.27 s | |
| 2024-01-16 04:09:32,550 44k INFO ====> Epoch: 1627, cost 13.08 s | |
| 2024-01-16 04:09:45,776 44k INFO ====> Epoch: 1628, cost 13.23 s | |
| 2024-01-16 04:09:58,911 44k INFO ====> Epoch: 1629, cost 13.14 s | |
| 2024-01-16 04:10:12,393 44k INFO ====> Epoch: 1630, cost 13.48 s | |
| 2024-01-16 04:10:23,606 44k INFO Train Epoch: 1631 [69%] | |
| 2024-01-16 04:10:23,607 44k INFO Losses: [2.4075589179992676, 2.3661675453186035, 8.065401077270508, 16.693443298339844, 0.9627520442008972], step: 21200, lr: 8.156558744657806e-05, reference_loss: 30.49532127380371 | |
| 2024-01-16 04:10:32,011 44k INFO Saving model and optimizer state at iteration 1631 to ./logs/44k/G_21200.pth | |
| 2024-01-16 04:10:33,644 44k INFO Saving model and optimizer state at iteration 1631 to ./logs/44k/D_21200.pth | |
| 2024-01-16 04:10:34,330 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_17200.pth | |
| 2024-01-16 04:10:34,387 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_17200.pth | |
| 2024-01-16 04:10:36,663 44k INFO ====> Epoch: 1631, cost 24.27 s | |
| 2024-01-16 04:10:49,948 44k INFO ====> Epoch: 1632, cost 13.28 s | |
| 2024-01-16 04:11:03,337 44k INFO ====> Epoch: 1633, cost 13.39 s | |
| 2024-01-16 04:11:16,582 44k INFO ====> Epoch: 1634, cost 13.25 s | |
| 2024-01-16 04:11:29,552 44k INFO ====> Epoch: 1635, cost 12.97 s | |
| 2024-01-16 04:11:42,585 44k INFO ====> Epoch: 1636, cost 13.03 s | |
| 2024-01-16 04:11:55,844 44k INFO ====> Epoch: 1637, cost 13.26 s | |
| 2024-01-16 04:12:09,375 44k INFO ====> Epoch: 1638, cost 13.53 s | |
| 2024-01-16 04:12:22,664 44k INFO ====> Epoch: 1639, cost 13.29 s | |
| 2024-01-16 04:12:36,029 44k INFO ====> Epoch: 1640, cost 13.36 s | |
| 2024-01-16 04:12:49,438 44k INFO ====> Epoch: 1641, cost 13.41 s | |
| 2024-01-16 04:13:02,531 44k INFO ====> Epoch: 1642, cost 13.09 s | |
| 2024-01-16 04:13:15,788 44k INFO ====> Epoch: 1643, cost 13.26 s | |
| 2024-01-16 04:13:29,293 44k INFO ====> Epoch: 1644, cost 13.50 s | |
| 2024-01-16 04:13:42,517 44k INFO ====> Epoch: 1645, cost 13.22 s | |
| 2024-01-16 04:13:56,523 44k INFO ====> Epoch: 1646, cost 14.01 s | |
| 2024-01-16 04:14:04,217 44k INFO Train Epoch: 1647 [8%] | |
| 2024-01-16 04:14:04,219 44k INFO Losses: [2.3499526977539062, 2.3547449111938477, 5.995206356048584, 14.197105407714844, 0.48042917251586914], step: 21400, lr: 8.14026091179852e-05, reference_loss: 25.377437591552734 | |
| 2024-01-16 04:14:12,796 44k INFO Saving model and optimizer state at iteration 1647 to ./logs/44k/G_21400.pth | |
| 2024-01-16 04:14:14,331 44k INFO Saving model and optimizer state at iteration 1647 to ./logs/44k/D_21400.pth | |
| 2024-01-16 04:14:15,038 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_17400.pth | |
| 2024-01-16 04:14:15,096 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_17400.pth | |
| 2024-01-16 04:14:20,828 44k INFO ====> Epoch: 1647, cost 24.31 s | |
| 2024-01-16 04:14:34,240 44k INFO ====> Epoch: 1648, cost 13.41 s | |
| 2024-01-16 04:14:47,491 44k INFO ====> Epoch: 1649, cost 13.25 s | |
| 2024-01-16 04:15:00,764 44k INFO ====> Epoch: 1650, cost 13.27 s | |
| 2024-01-16 04:15:14,740 44k INFO ====> Epoch: 1651, cost 13.98 s | |
| 2024-01-16 04:15:27,842 44k INFO ====> Epoch: 1652, cost 13.10 s | |
| 2024-01-16 04:15:41,122 44k INFO ====> Epoch: 1653, cost 13.28 s | |
| 2024-01-16 04:15:54,294 44k INFO ====> Epoch: 1654, cost 13.17 s | |
| 2024-01-16 04:16:07,434 44k INFO ====> Epoch: 1655, cost 13.14 s | |
| 2024-01-16 04:16:20,858 44k INFO ====> Epoch: 1656, cost 13.42 s | |
| 2024-01-16 04:16:34,385 44k INFO ====> Epoch: 1657, cost 13.53 s | |
| 2024-01-16 04:16:47,476 44k INFO ====> Epoch: 1658, cost 13.09 s | |
| 2024-01-16 04:17:00,745 44k INFO ====> Epoch: 1659, cost 13.27 s | |
| 2024-01-16 04:17:14,670 44k INFO ====> Epoch: 1660, cost 13.93 s | |
| 2024-01-16 04:17:28,092 44k INFO ====> Epoch: 1661, cost 13.42 s | |
| 2024-01-16 04:17:37,993 44k INFO Train Epoch: 1662 [46%] | |
| 2024-01-16 04:17:37,994 44k INFO Losses: [2.212718963623047, 2.6741087436676025, 6.543259620666504, 12.231542587280273, 0.25736287236213684], step: 21600, lr: 8.125011270473142e-05, reference_loss: 23.918991088867188 | |
| 2024-01-16 04:17:46,656 44k INFO Saving model and optimizer state at iteration 1662 to ./logs/44k/G_21600.pth | |
| 2024-01-16 04:17:48,340 44k INFO Saving model and optimizer state at iteration 1662 to ./logs/44k/D_21600.pth | |
| 2024-01-16 04:17:49,024 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_17600.pth | |
| 2024-01-16 04:17:49,083 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_17600.pth | |
| 2024-01-16 04:17:52,614 44k INFO ====> Epoch: 1662, cost 24.52 s | |
| 2024-01-16 04:18:05,941 44k INFO ====> Epoch: 1663, cost 13.33 s | |
| 2024-01-16 04:18:18,972 44k INFO ====> Epoch: 1664, cost 13.03 s | |
| 2024-01-16 04:18:32,270 44k INFO ====> Epoch: 1665, cost 13.30 s | |
| 2024-01-16 04:18:45,395 44k INFO ====> Epoch: 1666, cost 13.12 s | |
| 2024-01-16 04:18:58,694 44k INFO ====> Epoch: 1667, cost 13.30 s | |
| 2024-01-16 04:19:12,418 44k INFO ====> Epoch: 1668, cost 13.72 s | |
| 2024-01-16 04:19:25,607 44k INFO ====> Epoch: 1669, cost 13.19 s | |
| 2024-01-16 04:19:38,934 44k INFO ====> Epoch: 1670, cost 13.33 s | |
| 2024-01-16 04:19:52,494 44k INFO ====> Epoch: 1671, cost 13.56 s | |
| 2024-01-16 04:20:05,937 44k INFO ====> Epoch: 1672, cost 13.44 s | |
| 2024-01-16 04:20:19,230 44k INFO ====> Epoch: 1673, cost 13.29 s | |
| 2024-01-16 04:20:32,577 44k INFO ====> Epoch: 1674, cost 13.35 s | |
| 2024-01-16 04:20:45,722 44k INFO ====> Epoch: 1675, cost 13.15 s | |
| 2024-01-16 04:20:58,912 44k INFO ====> Epoch: 1676, cost 13.19 s | |
| 2024-01-16 04:21:10,543 44k INFO Train Epoch: 1677 [85%] | |
| 2024-01-16 04:21:10,544 44k INFO Losses: [2.2977654933929443, 2.251885414123535, 5.986703872680664, 11.531304359436035, 0.6946709156036377], step: 21800, lr: 8.109790197219855e-05, reference_loss: 22.762331008911133 | |
| 2024-01-16 04:21:18,903 44k INFO Saving model and optimizer state at iteration 1677 to ./logs/44k/G_21800.pth | |
| 2024-01-16 04:21:20,822 44k INFO Saving model and optimizer state at iteration 1677 to ./logs/44k/D_21800.pth | |
| 2024-01-16 04:21:21,506 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_17800.pth | |
| 2024-01-16 04:21:21,568 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_17800.pth | |
| 2024-01-16 04:21:23,105 44k INFO ====> Epoch: 1677, cost 24.19 s | |
| 2024-01-16 04:21:36,541 44k INFO ====> Epoch: 1678, cost 13.44 s | |
| 2024-01-16 04:21:50,514 44k INFO ====> Epoch: 1679, cost 13.97 s | |
| 2024-01-16 04:22:04,139 44k INFO ====> Epoch: 1680, cost 13.63 s | |
| 2024-01-16 04:22:17,156 44k INFO ====> Epoch: 1681, cost 13.02 s | |
| 2024-01-16 04:22:30,177 44k INFO ====> Epoch: 1682, cost 13.02 s | |
| 2024-01-16 04:22:43,352 44k INFO ====> Epoch: 1683, cost 13.18 s | |
| 2024-01-16 04:22:56,631 44k INFO ====> Epoch: 1684, cost 13.28 s | |
| 2024-01-16 04:23:09,730 44k INFO ====> Epoch: 1685, cost 13.10 s | |
| 2024-01-16 04:23:22,857 44k INFO ====> Epoch: 1686, cost 13.13 s | |
| 2024-01-16 04:23:36,140 44k INFO ====> Epoch: 1687, cost 13.28 s | |
| 2024-01-16 04:23:49,623 44k INFO ====> Epoch: 1688, cost 13.48 s | |
| 2024-01-16 04:24:03,328 44k INFO ====> Epoch: 1689, cost 13.70 s | |
| 2024-01-16 04:24:16,895 44k INFO ====> Epoch: 1690, cost 13.57 s | |
| 2024-01-16 04:24:30,192 44k INFO ====> Epoch: 1691, cost 13.30 s | |
| 2024-01-16 04:24:43,513 44k INFO ====> Epoch: 1692, cost 13.32 s | |
| 2024-01-16 04:24:52,364 44k INFO Train Epoch: 1693 [23%] | |
| 2024-01-16 04:24:52,364 44k INFO Losses: [2.2058145999908447, 2.8888566493988037, 8.477581024169922, 14.200021743774414, 0.6446312665939331], step: 22000, lr: 8.09358581381555e-05, reference_loss: 28.41690444946289 | |
| 2024-01-16 04:25:00,844 44k INFO Saving model and optimizer state at iteration 1693 to ./logs/44k/G_22000.pth | |
| 2024-01-16 04:25:02,454 44k INFO Saving model and optimizer state at iteration 1693 to ./logs/44k/D_22000.pth | |
| 2024-01-16 04:25:03,154 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_18000.pth | |
| 2024-01-16 04:25:03,208 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_18000.pth | |
| 2024-01-16 04:25:07,901 44k INFO ====> Epoch: 1693, cost 24.39 s | |
| 2024-01-16 04:25:21,208 44k INFO ====> Epoch: 1694, cost 13.31 s | |
| 2024-01-16 04:25:34,589 44k INFO ====> Epoch: 1695, cost 13.38 s | |
| 2024-01-16 04:25:47,766 44k INFO ====> Epoch: 1696, cost 13.18 s | |
| 2024-01-16 04:26:00,871 44k INFO ====> Epoch: 1697, cost 13.11 s | |
| 2024-01-16 04:26:13,668 44k INFO ====> Epoch: 1698, cost 12.80 s | |
| 2024-01-16 04:26:26,739 44k INFO ====> Epoch: 1699, cost 13.07 s | |
| 2024-01-16 04:26:39,579 44k INFO ====> Epoch: 1700, cost 12.84 s | |
| 2024-01-16 04:26:52,987 44k INFO ====> Epoch: 1701, cost 13.41 s | |
| 2024-01-16 04:27:06,366 44k INFO ====> Epoch: 1702, cost 13.38 s | |
| 2024-01-16 04:27:19,760 44k INFO ====> Epoch: 1703, cost 13.39 s | |
| 2024-01-16 04:27:32,772 44k INFO ====> Epoch: 1704, cost 13.01 s | |
| 2024-01-16 04:27:46,111 44k INFO ====> Epoch: 1705, cost 13.34 s | |
| 2024-01-16 04:27:59,199 44k INFO ====> Epoch: 1706, cost 13.09 s | |
| 2024-01-16 04:28:12,293 44k INFO ====> Epoch: 1707, cost 13.09 s | |
| 2024-01-16 04:28:22,882 44k INFO Train Epoch: 1708 [62%] | |
| 2024-01-16 04:28:22,883 44k INFO Losses: [2.261380910873413, 2.5635149478912354, 7.352553367614746, 11.570845603942871, 0.651972234249115], step: 22200, lr: 8.078423611764021e-05, reference_loss: 24.400266647338867 | |
| 2024-01-16 04:28:31,766 44k INFO Saving model and optimizer state at iteration 1708 to ./logs/44k/G_22200.pth | |
| 2024-01-16 04:28:33,491 44k INFO Saving model and optimizer state at iteration 1708 to ./logs/44k/D_22200.pth | |
| 2024-01-16 04:28:34,273 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_18200.pth | |
| 2024-01-16 04:28:34,340 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_18200.pth | |
| 2024-01-16 04:28:37,125 44k INFO ====> Epoch: 1708, cost 24.83 s | |
| 2024-01-16 04:28:50,467 44k INFO ====> Epoch: 1709, cost 13.34 s | |
| 2024-01-16 04:29:03,795 44k INFO ====> Epoch: 1710, cost 13.33 s | |
| 2024-01-16 04:29:17,425 44k INFO ====> Epoch: 1711, cost 13.63 s | |
| 2024-01-16 04:29:30,418 44k INFO ====> Epoch: 1712, cost 12.99 s | |
| 2024-01-16 04:29:43,548 44k INFO ====> Epoch: 1713, cost 13.13 s | |
| 2024-01-16 04:29:57,111 44k INFO ====> Epoch: 1714, cost 13.56 s | |
| 2024-01-16 04:30:10,762 44k INFO ====> Epoch: 1715, cost 13.65 s | |
| 2024-01-16 04:30:24,048 44k INFO ====> Epoch: 1716, cost 13.29 s | |
| 2024-01-16 04:30:37,918 44k INFO ====> Epoch: 1717, cost 13.87 s | |
| 2024-01-16 04:30:51,247 44k INFO ====> Epoch: 1718, cost 13.33 s | |
| 2024-01-16 04:31:04,932 44k INFO ====> Epoch: 1719, cost 13.68 s | |
| 2024-01-16 04:31:17,952 44k INFO ====> Epoch: 1720, cost 13.02 s | |
| 2024-01-16 04:31:31,361 44k INFO ====> Epoch: 1721, cost 13.41 s | |
| 2024-01-16 04:31:44,793 44k INFO ====> Epoch: 1722, cost 13.43 s | |
| 2024-01-16 04:31:57,969 44k INFO ====> Epoch: 1723, cost 13.18 s | |
| 2024-01-16 04:32:05,221 44k INFO Train Epoch: 1724 [0%] | |
| 2024-01-16 04:32:05,221 44k INFO Losses: [2.4412806034088135, 2.3130550384521484, 4.071746349334717, 12.450149536132812, 0.3467458486557007], step: 22400, lr: 8.062281902752576e-05, reference_loss: 21.62297821044922 | |
| 2024-01-16 04:32:13,609 44k INFO Saving model and optimizer state at iteration 1724 to ./logs/44k/G_22400.pth | |
| 2024-01-16 04:32:15,292 44k INFO Saving model and optimizer state at iteration 1724 to ./logs/44k/D_22400.pth | |
| 2024-01-16 04:32:16,065 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_18400.pth | |
| 2024-01-16 04:32:16,132 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_18400.pth | |
| 2024-01-16 04:32:22,014 44k INFO ====> Epoch: 1724, cost 24.05 s | |
| 2024-01-16 04:32:35,336 44k INFO ====> Epoch: 1725, cost 13.32 s | |
| 2024-01-16 04:32:48,740 44k INFO ====> Epoch: 1726, cost 13.40 s | |
| 2024-01-16 04:33:01,696 44k INFO ====> Epoch: 1727, cost 12.96 s | |
| 2024-01-16 04:33:14,802 44k INFO ====> Epoch: 1728, cost 13.11 s | |
| 2024-01-16 04:33:27,975 44k INFO ====> Epoch: 1729, cost 13.17 s | |
| 2024-01-16 04:33:41,308 44k INFO ====> Epoch: 1730, cost 13.33 s | |
| 2024-01-16 04:33:54,460 44k INFO ====> Epoch: 1731, cost 13.15 s | |
| 2024-01-16 04:34:07,624 44k INFO ====> Epoch: 1732, cost 13.16 s | |
| 2024-01-16 04:34:20,874 44k INFO ====> Epoch: 1733, cost 13.25 s | |
| 2024-01-16 04:34:34,148 44k INFO ====> Epoch: 1734, cost 13.27 s | |
| 2024-01-16 04:34:47,581 44k INFO ====> Epoch: 1735, cost 13.43 s | |
| 2024-01-16 04:35:00,622 44k INFO ====> Epoch: 1736, cost 13.04 s | |
| 2024-01-16 04:35:14,300 44k INFO ====> Epoch: 1737, cost 13.68 s | |
| 2024-01-16 04:35:27,457 44k INFO ====> Epoch: 1738, cost 13.16 s | |
| 2024-01-16 04:35:36,773 44k INFO Train Epoch: 1739 [38%] | |
| 2024-01-16 04:35:36,775 44k INFO Losses: [2.4554295539855957, 2.3548569679260254, 6.335957050323486, 12.599942207336426, 0.12437865138053894], step: 22600, lr: 8.047178344204122e-05, reference_loss: 23.870563507080078 | |
| 2024-01-16 04:35:45,431 44k INFO Saving model and optimizer state at iteration 1739 to ./logs/44k/G_22600.pth | |
| 2024-01-16 04:35:47,082 44k INFO Saving model and optimizer state at iteration 1739 to ./logs/44k/D_22600.pth | |
| 2024-01-16 04:35:47,878 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_18600.pth | |
| 2024-01-16 04:35:47,943 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_18600.pth | |
| 2024-01-16 04:35:51,859 44k INFO ====> Epoch: 1739, cost 24.40 s | |
| 2024-01-16 04:36:04,994 44k INFO ====> Epoch: 1740, cost 13.14 s | |
| 2024-01-16 04:36:17,987 44k INFO ====> Epoch: 1741, cost 12.99 s | |
| 2024-01-16 04:36:30,905 44k INFO ====> Epoch: 1742, cost 12.92 s | |
| 2024-01-16 04:36:43,922 44k INFO ====> Epoch: 1743, cost 13.02 s | |
| 2024-01-16 04:36:56,863 44k INFO ====> Epoch: 1744, cost 12.94 s | |
| 2024-01-16 04:37:09,946 44k INFO ====> Epoch: 1745, cost 13.08 s | |
| 2024-01-16 04:37:23,468 44k INFO ====> Epoch: 1746, cost 13.52 s | |
| 2024-01-16 04:37:36,856 44k INFO ====> Epoch: 1747, cost 13.39 s | |
| 2024-01-16 04:37:50,024 44k INFO ====> Epoch: 1748, cost 13.17 s | |
| 2024-01-16 04:38:02,960 44k INFO ====> Epoch: 1749, cost 12.94 s | |
| 2024-01-16 04:38:16,143 44k INFO ====> Epoch: 1750, cost 13.18 s | |
| 2024-01-16 04:38:29,104 44k INFO ====> Epoch: 1751, cost 12.96 s | |
| 2024-01-16 04:38:42,171 44k INFO ====> Epoch: 1752, cost 13.07 s | |
| 2024-01-16 04:38:55,255 44k INFO ====> Epoch: 1753, cost 13.08 s | |
| 2024-01-16 04:39:06,839 44k INFO Train Epoch: 1754 [77%] | |
| 2024-01-16 04:39:06,840 44k INFO Losses: [2.4681684970855713, 2.5284931659698486, 4.770317554473877, 10.27278995513916, 0.06068110838532448], step: 22800, lr: 8.032103080062085e-05, reference_loss: 20.100448608398438 | |
| 2024-01-16 04:39:15,493 44k INFO Saving model and optimizer state at iteration 1754 to ./logs/44k/G_22800.pth | |
| 2024-01-16 04:39:17,494 44k INFO Saving model and optimizer state at iteration 1754 to ./logs/44k/D_22800.pth | |
| 2024-01-16 04:39:18,267 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_18800.pth | |
| 2024-01-16 04:39:18,337 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_18800.pth | |
| 2024-01-16 04:39:20,366 44k INFO ====> Epoch: 1754, cost 25.11 s | |
| 2024-01-16 04:39:33,763 44k INFO ====> Epoch: 1755, cost 13.40 s | |
| 2024-01-16 04:39:47,129 44k INFO ====> Epoch: 1756, cost 13.37 s | |
| 2024-01-16 04:40:00,087 44k INFO ====> Epoch: 1757, cost 12.96 s | |
| 2024-01-16 04:40:13,382 44k INFO ====> Epoch: 1758, cost 13.29 s | |
| 2024-01-16 04:40:26,628 44k INFO ====> Epoch: 1759, cost 13.25 s | |
| 2024-01-16 04:40:39,689 44k INFO ====> Epoch: 1760, cost 13.06 s | |
| 2024-01-16 04:40:52,832 44k INFO ====> Epoch: 1761, cost 13.14 s | |
| 2024-01-16 04:41:05,843 44k INFO ====> Epoch: 1762, cost 13.01 s | |
| 2024-01-16 04:41:18,980 44k INFO ====> Epoch: 1763, cost 13.14 s | |
| 2024-01-16 04:41:32,056 44k INFO ====> Epoch: 1764, cost 13.08 s | |
| 2024-01-16 04:41:44,997 44k INFO ====> Epoch: 1765, cost 12.94 s | |
| 2024-01-16 04:41:58,677 44k INFO ====> Epoch: 1766, cost 13.68 s | |
| 2024-01-16 04:42:11,942 44k INFO ====> Epoch: 1767, cost 13.27 s | |
| 2024-01-16 04:42:24,984 44k INFO ====> Epoch: 1768, cost 13.04 s | |
| 2024-01-16 04:42:38,057 44k INFO ====> Epoch: 1769, cost 13.07 s | |
| 2024-01-16 04:42:45,850 44k INFO Train Epoch: 1770 [15%] | |
| 2024-01-16 04:42:45,851 44k INFO Losses: [2.739234447479248, 2.0635673999786377, 4.931553840637207, 9.205170631408691, 0.6286047101020813], step: 23000, lr: 8.016053925313687e-05, reference_loss: 19.568130493164062 | |
| 2024-01-16 04:42:54,238 44k INFO Saving model and optimizer state at iteration 1770 to ./logs/44k/G_23000.pth | |
| 2024-01-16 04:42:55,837 44k INFO Saving model and optimizer state at iteration 1770 to ./logs/44k/D_23000.pth | |
| 2024-01-16 04:42:56,591 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_19000.pth | |
| 2024-01-16 04:42:56,660 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_19000.pth | |
| 2024-01-16 04:43:01,861 44k INFO ====> Epoch: 1770, cost 23.80 s | |
| 2024-01-16 04:43:14,985 44k INFO ====> Epoch: 1771, cost 13.12 s | |
| 2024-01-16 04:43:28,022 44k INFO ====> Epoch: 1772, cost 13.04 s | |
| 2024-01-16 04:43:41,268 44k INFO ====> Epoch: 1773, cost 13.25 s | |
| 2024-01-16 04:43:54,655 44k INFO ====> Epoch: 1774, cost 13.39 s | |
| 2024-01-16 04:44:07,525 44k INFO ====> Epoch: 1775, cost 12.87 s | |
| 2024-01-16 04:44:20,625 44k INFO ====> Epoch: 1776, cost 13.10 s | |
| 2024-01-16 04:44:34,018 44k INFO ====> Epoch: 1777, cost 13.39 s | |
| 2024-01-16 04:44:46,993 44k INFO ====> Epoch: 1778, cost 12.97 s | |
| 2024-01-16 04:45:00,428 44k INFO ====> Epoch: 1779, cost 13.44 s | |
| 2024-01-16 04:45:13,738 44k INFO ====> Epoch: 1780, cost 13.31 s | |
| 2024-01-16 04:45:26,898 44k INFO ====> Epoch: 1781, cost 13.16 s | |
| 2024-01-16 04:45:39,872 44k INFO ====> Epoch: 1782, cost 12.97 s | |
| 2024-01-16 04:45:52,968 44k INFO ====> Epoch: 1783, cost 13.10 s | |
| 2024-01-16 04:46:06,132 44k INFO ====> Epoch: 1784, cost 13.16 s | |
| 2024-01-16 04:46:16,293 44k INFO Train Epoch: 1785 [54%] | |
| 2024-01-16 04:46:16,294 44k INFO Losses: [2.543804168701172, 2.210904836654663, 4.216278076171875, 9.31699275970459, 0.2280949503183365], step: 23200, lr: 8.00103696842122e-05, reference_loss: 18.51607322692871 | |
| 2024-01-16 04:46:25,150 44k INFO Saving model and optimizer state at iteration 1785 to ./logs/44k/G_23200.pth | |
| 2024-01-16 04:46:26,780 44k INFO Saving model and optimizer state at iteration 1785 to ./logs/44k/D_23200.pth | |
| 2024-01-16 04:46:27,499 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_19200.pth | |
| 2024-01-16 04:46:27,561 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_19200.pth | |
| 2024-01-16 04:46:30,735 44k INFO ====> Epoch: 1785, cost 24.60 s | |
| 2024-01-16 04:46:43,847 44k INFO ====> Epoch: 1786, cost 13.11 s | |
| 2024-01-16 04:46:56,934 44k INFO ====> Epoch: 1787, cost 13.09 s | |
| 2024-01-16 04:47:10,618 44k INFO ====> Epoch: 1788, cost 13.68 s | |
| 2024-01-16 04:47:23,561 44k INFO ====> Epoch: 1789, cost 12.94 s | |
| 2024-01-16 04:47:36,809 44k INFO ====> Epoch: 1790, cost 13.25 s | |
| 2024-01-16 04:47:50,318 44k INFO ====> Epoch: 1791, cost 13.51 s | |
| 2024-01-16 04:48:03,494 44k INFO ====> Epoch: 1792, cost 13.18 s | |
| 2024-01-16 04:48:16,863 44k INFO ====> Epoch: 1793, cost 13.37 s | |
| 2024-01-16 04:48:30,435 44k INFO ====> Epoch: 1794, cost 13.57 s | |
| 2024-01-16 04:48:43,590 44k INFO ====> Epoch: 1795, cost 13.15 s | |
| 2024-01-16 04:48:57,237 44k INFO ====> Epoch: 1796, cost 13.65 s | |
| 2024-01-16 04:49:10,740 44k INFO ====> Epoch: 1797, cost 13.50 s | |
| 2024-01-16 04:49:24,429 44k INFO ====> Epoch: 1798, cost 13.69 s | |
| 2024-01-16 04:49:37,537 44k INFO ====> Epoch: 1799, cost 13.11 s | |
| 2024-01-16 04:49:49,983 44k INFO Train Epoch: 1800 [92%] | |
| 2024-01-16 04:49:49,984 44k INFO Losses: [2.200133800506592, 2.5818607807159424, 7.20768404006958, 11.600589752197266, 0.4716903269290924], step: 23400, lr: 7.986048143699072e-05, reference_loss: 24.061960220336914 | |
| 2024-01-16 04:49:58,408 44k INFO Saving model and optimizer state at iteration 1800 to ./logs/44k/G_23400.pth | |
| 2024-01-16 04:49:59,974 44k INFO Saving model and optimizer state at iteration 1800 to ./logs/44k/D_23400.pth | |
| 2024-01-16 04:50:00,687 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_19400.pth | |
| 2024-01-16 04:50:00,738 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_19400.pth | |
| 2024-01-16 04:50:01,977 44k INFO ====> Epoch: 1800, cost 24.44 s | |
| 2024-01-16 04:50:15,199 44k INFO ====> Epoch: 1801, cost 13.22 s | |
| 2024-01-16 04:50:28,054 44k INFO ====> Epoch: 1802, cost 12.85 s | |
| 2024-01-16 04:50:41,362 44k INFO ====> Epoch: 1803, cost 13.31 s | |
| 2024-01-16 04:50:54,993 44k INFO ====> Epoch: 1804, cost 13.63 s | |
| 2024-01-16 04:51:07,839 44k INFO ====> Epoch: 1805, cost 12.85 s | |
| 2024-01-16 04:51:20,961 44k INFO ====> Epoch: 1806, cost 13.12 s | |
| 2024-01-16 04:51:33,856 44k INFO ====> Epoch: 1807, cost 12.89 s | |
| 2024-01-16 04:51:46,798 44k INFO ====> Epoch: 1808, cost 12.94 s | |
| 2024-01-16 04:51:59,569 44k INFO ====> Epoch: 1809, cost 12.77 s | |
| 2024-01-16 04:52:12,576 44k INFO ====> Epoch: 1810, cost 13.01 s | |
| 2024-01-16 04:52:25,378 44k INFO ====> Epoch: 1811, cost 12.80 s | |
| 2024-01-16 04:52:38,776 44k INFO ====> Epoch: 1812, cost 13.40 s | |
| 2024-01-16 04:52:52,367 44k INFO ====> Epoch: 1813, cost 13.59 s | |
| 2024-01-16 04:53:05,394 44k INFO ====> Epoch: 1814, cost 13.03 s | |
| 2024-01-16 04:53:19,257 44k INFO ====> Epoch: 1815, cost 13.86 s | |
| 2024-01-16 04:53:27,594 44k INFO Train Epoch: 1816 [31%] | |
| 2024-01-16 04:53:27,595 44k INFO Losses: [2.8621737957000732, 1.7907652854919434, 1.500375509262085, 5.931115627288818, 0.09422378242015839], step: 23600, lr: 7.970091012520744e-05, reference_loss: 12.178654670715332 | |
| 2024-01-16 04:53:36,099 44k INFO Saving model and optimizer state at iteration 1816 to ./logs/44k/G_23600.pth | |
| 2024-01-16 04:53:37,686 44k INFO Saving model and optimizer state at iteration 1816 to ./logs/44k/D_23600.pth | |
| 2024-01-16 04:53:38,367 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_19600.pth | |
| 2024-01-16 04:53:38,426 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_19600.pth | |
| 2024-01-16 04:53:42,864 44k INFO ====> Epoch: 1816, cost 23.61 s | |
| 2024-01-16 04:53:55,939 44k INFO ====> Epoch: 1817, cost 13.08 s | |
| 2024-01-16 04:54:09,657 44k INFO ====> Epoch: 1818, cost 13.72 s | |
| 2024-01-16 04:54:22,568 44k INFO ====> Epoch: 1819, cost 12.91 s | |
| 2024-01-16 04:54:35,870 44k INFO ====> Epoch: 1820, cost 13.30 s | |
| 2024-01-16 04:54:48,780 44k INFO ====> Epoch: 1821, cost 12.91 s | |
| 2024-01-16 04:55:01,788 44k INFO ====> Epoch: 1822, cost 13.01 s | |
| 2024-01-16 04:55:14,978 44k INFO ====> Epoch: 1823, cost 13.19 s | |
| 2024-01-16 04:55:28,299 44k INFO ====> Epoch: 1824, cost 13.32 s | |
| 2024-01-16 04:55:41,124 44k INFO ====> Epoch: 1825, cost 12.82 s | |
| 2024-01-16 04:55:53,990 44k INFO ====> Epoch: 1826, cost 12.87 s | |
| 2024-01-16 04:56:06,879 44k INFO ====> Epoch: 1827, cost 12.89 s | |
| 2024-01-16 04:56:19,846 44k INFO ====> Epoch: 1828, cost 12.97 s | |
| 2024-01-16 04:56:32,955 44k INFO ====> Epoch: 1829, cost 13.11 s | |
| 2024-01-16 04:56:46,531 44k INFO ====> Epoch: 1830, cost 13.58 s | |
| 2024-01-16 04:56:57,061 44k INFO Train Epoch: 1831 [69%] | |
| 2024-01-16 04:56:57,062 44k INFO Losses: [2.313293933868408, 2.5042006969451904, 8.514606475830078, 16.562305450439453, 0.9080897569656372], step: 23800, lr: 7.955160160722687e-05, reference_loss: 30.8024959564209 | |
| 2024-01-16 04:57:05,560 44k INFO Saving model and optimizer state at iteration 1831 to ./logs/44k/G_23800.pth | |
| 2024-01-16 04:57:07,388 44k INFO Saving model and optimizer state at iteration 1831 to ./logs/44k/D_23800.pth | |
| 2024-01-16 04:57:08,055 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_19800.pth | |
| 2024-01-16 04:57:08,103 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_19800.pth | |
| 2024-01-16 04:57:10,463 44k INFO ====> Epoch: 1831, cost 23.93 s | |
| 2024-01-16 04:57:23,390 44k INFO ====> Epoch: 1832, cost 12.93 s | |
| 2024-01-16 04:57:36,471 44k INFO ====> Epoch: 1833, cost 13.08 s | |
| 2024-01-16 04:57:49,466 44k INFO ====> Epoch: 1834, cost 13.00 s | |
| 2024-01-16 04:58:02,574 44k INFO ====> Epoch: 1835, cost 13.11 s | |
| 2024-01-16 04:58:15,934 44k INFO ====> Epoch: 1836, cost 13.36 s | |
| 2024-01-16 04:58:28,859 44k INFO ====> Epoch: 1837, cost 12.93 s | |
| 2024-01-16 04:58:41,947 44k INFO ====> Epoch: 1838, cost 13.09 s | |
| 2024-01-16 04:58:54,938 44k INFO ====> Epoch: 1839, cost 12.99 s | |
| 2024-01-16 04:59:07,770 44k INFO ====> Epoch: 1840, cost 12.83 s | |
| 2024-01-16 04:59:20,618 44k INFO ====> Epoch: 1841, cost 12.85 s | |
| 2024-01-16 04:59:33,713 44k INFO ====> Epoch: 1842, cost 13.09 s | |
| 2024-01-16 04:59:46,593 44k INFO ====> Epoch: 1843, cost 12.88 s | |
| 2024-01-16 04:59:59,859 44k INFO ====> Epoch: 1844, cost 13.27 s | |
| 2024-01-16 05:00:13,290 44k INFO ====> Epoch: 1845, cost 13.43 s | |
| 2024-01-16 05:00:26,662 44k INFO ====> Epoch: 1846, cost 13.37 s | |
| 2024-01-16 05:00:34,476 44k INFO Train Epoch: 1847 [8%] | |
| 2024-01-16 05:00:34,477 44k INFO Losses: [2.2231616973876953, 2.819962978363037, 6.371984481811523, 14.312116622924805, 0.4395383298397064], step: 24000, lr: 7.939264747629116e-05, reference_loss: 26.166765213012695 | |
| 2024-01-16 05:00:42,868 44k INFO Saving model and optimizer state at iteration 1847 to ./logs/44k/G_24000.pth | |
| 2024-01-16 05:00:44,477 44k INFO Saving model and optimizer state at iteration 1847 to ./logs/44k/D_24000.pth | |
| 2024-01-16 05:00:45,172 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_20000.pth | |
| 2024-01-16 05:00:45,228 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_20000.pth | |
| 2024-01-16 05:00:50,600 44k INFO ====> Epoch: 1847, cost 23.94 s | |
| 2024-01-16 05:01:03,743 44k INFO ====> Epoch: 1848, cost 13.14 s | |
| 2024-01-16 05:01:16,537 44k INFO ====> Epoch: 1849, cost 12.79 s | |
| 2024-01-16 05:01:29,260 44k INFO ====> Epoch: 1850, cost 12.72 s | |
| 2024-01-16 05:01:42,403 44k INFO ====> Epoch: 1851, cost 13.14 s | |
| 2024-01-16 05:01:55,540 44k INFO ====> Epoch: 1852, cost 13.14 s | |
| 2024-01-16 05:02:08,387 44k INFO ====> Epoch: 1853, cost 12.85 s | |
| 2024-01-16 05:02:21,125 44k INFO ====> Epoch: 1854, cost 12.74 s | |
| 2024-01-16 05:02:33,851 44k INFO ====> Epoch: 1855, cost 12.73 s | |
| 2024-01-16 05:02:46,763 44k INFO ====> Epoch: 1856, cost 12.91 s | |
| 2024-01-16 05:02:59,918 44k INFO ====> Epoch: 1857, cost 13.16 s | |
| 2024-01-16 05:03:12,699 44k INFO ====> Epoch: 1858, cost 12.78 s | |
| 2024-01-16 05:03:25,635 44k INFO ====> Epoch: 1859, cost 12.94 s | |
| 2024-01-16 05:03:38,534 44k INFO ====> Epoch: 1860, cost 12.90 s | |
| 2024-01-16 05:03:51,554 44k INFO ====> Epoch: 1861, cost 13.02 s | |
| 2024-01-16 05:04:01,331 44k INFO Train Epoch: 1862 [46%] | |
| 2024-01-16 05:04:01,332 44k INFO Losses: [2.2205891609191895, 2.7116451263427734, 6.7758049964904785, 12.184837341308594, 0.21351028978824615], step: 24200, lr: 7.924391644530778e-05, reference_loss: 24.106386184692383 | |
| 2024-01-16 05:04:09,617 44k INFO Saving model and optimizer state at iteration 1862 to ./logs/44k/G_24200.pth | |
| 2024-01-16 05:04:11,566 44k INFO Saving model and optimizer state at iteration 1862 to ./logs/44k/D_24200.pth | |
| 2024-01-16 05:04:12,256 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_20200.pth | |
| 2024-01-16 05:04:12,318 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_20200.pth | |
| 2024-01-16 05:04:15,616 44k INFO ====> Epoch: 1862, cost 24.06 s | |
| 2024-01-16 05:04:28,522 44k INFO ====> Epoch: 1863, cost 12.91 s | |
| 2024-01-16 05:04:41,364 44k INFO ====> Epoch: 1864, cost 12.84 s | |
| 2024-01-16 05:04:54,466 44k INFO ====> Epoch: 1865, cost 13.10 s | |
| 2024-01-16 05:05:07,779 44k INFO ====> Epoch: 1866, cost 13.31 s | |
| 2024-01-16 05:05:20,770 44k INFO ====> Epoch: 1867, cost 12.99 s | |
| 2024-01-16 05:05:33,595 44k INFO ====> Epoch: 1868, cost 12.82 s | |
| 2024-01-16 05:05:46,393 44k INFO ====> Epoch: 1869, cost 12.80 s | |
| 2024-01-16 05:05:59,098 44k INFO ====> Epoch: 1870, cost 12.71 s | |
| 2024-01-16 05:06:12,066 44k INFO ====> Epoch: 1871, cost 12.97 s | |
| 2024-01-16 05:06:24,885 44k INFO ====> Epoch: 1872, cost 12.82 s | |
| 2024-01-16 05:06:37,582 44k INFO ====> Epoch: 1873, cost 12.70 s | |
| 2024-01-16 05:06:51,362 44k INFO ====> Epoch: 1874, cost 13.78 s | |
| 2024-01-16 05:07:04,683 44k INFO ====> Epoch: 1875, cost 13.32 s | |
| 2024-01-16 05:07:17,708 44k INFO ====> Epoch: 1876, cost 13.03 s | |
| 2024-01-16 05:07:29,161 44k INFO Train Epoch: 1877 [85%] | |
| 2024-01-16 05:07:29,162 44k INFO Losses: [2.2994914054870605, 2.3482472896575928, 6.3146867752075195, 11.700668334960938, 0.6892217993736267], step: 24400, lr: 7.909546404112776e-05, reference_loss: 23.35231590270996 | |
| 2024-01-16 05:07:37,403 44k INFO Saving model and optimizer state at iteration 1877 to ./logs/44k/G_24400.pth | |
| 2024-01-16 05:07:38,949 44k INFO Saving model and optimizer state at iteration 1877 to ./logs/44k/D_24400.pth | |
| 2024-01-16 05:07:39,607 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_20400.pth | |
| 2024-01-16 05:07:39,653 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_20400.pth | |
| 2024-01-16 05:07:41,416 44k INFO ====> Epoch: 1877, cost 23.71 s | |
| 2024-01-16 05:07:54,270 44k INFO ====> Epoch: 1878, cost 12.85 s | |
| 2024-01-16 05:08:07,340 44k INFO ====> Epoch: 1879, cost 13.07 s | |
| 2024-01-16 05:08:20,263 44k INFO ====> Epoch: 1880, cost 12.92 s | |
| 2024-01-16 05:08:33,451 44k INFO ====> Epoch: 1881, cost 13.19 s | |
| 2024-01-16 05:08:46,566 44k INFO ====> Epoch: 1882, cost 13.12 s | |
| 2024-01-16 05:08:59,513 44k INFO ====> Epoch: 1883, cost 12.95 s | |
| 2024-01-16 05:09:12,661 44k INFO ====> Epoch: 1884, cost 13.15 s | |
| 2024-01-16 05:09:26,046 44k INFO ====> Epoch: 1885, cost 13.39 s | |
| 2024-01-16 05:09:39,329 44k INFO ====> Epoch: 1886, cost 13.28 s | |
| 2024-01-16 05:09:52,366 44k INFO ====> Epoch: 1887, cost 13.04 s | |
| 2024-01-16 05:10:05,485 44k INFO ====> Epoch: 1888, cost 13.12 s | |
| 2024-01-16 05:10:18,203 44k INFO ====> Epoch: 1889, cost 12.72 s | |
| 2024-01-16 05:10:31,426 44k INFO ====> Epoch: 1890, cost 13.22 s | |
| 2024-01-16 05:10:44,263 44k INFO ====> Epoch: 1891, cost 12.84 s | |
| 2024-01-16 05:10:57,500 44k INFO ====> Epoch: 1892, cost 13.24 s | |
| 2024-01-16 05:11:06,124 44k INFO Train Epoch: 1893 [23%] | |
| 2024-01-16 05:11:06,125 44k INFO Losses: [2.033348321914673, 2.9956977367401123, 8.758440971374512, 14.245891571044922, 0.731423556804657], step: 24600, lr: 7.8937421330565e-05, reference_loss: 28.764802932739258 | |
| 2024-01-16 05:11:14,602 44k INFO Saving model and optimizer state at iteration 1893 to ./logs/44k/G_24600.pth | |
| 2024-01-16 05:11:16,168 44k INFO Saving model and optimizer state at iteration 1893 to ./logs/44k/D_24600.pth | |
| 2024-01-16 05:11:16,828 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_20600.pth | |
| 2024-01-16 05:11:16,867 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_20600.pth | |
| 2024-01-16 05:11:21,370 44k INFO ====> Epoch: 1893, cost 23.87 s | |
| 2024-01-16 05:11:34,895 44k INFO ====> Epoch: 1894, cost 13.53 s | |
| 2024-01-16 05:11:47,752 44k INFO ====> Epoch: 1895, cost 12.86 s | |
| 2024-01-16 05:12:00,414 44k INFO ====> Epoch: 1896, cost 12.66 s | |
| 2024-01-16 05:12:13,789 44k INFO ====> Epoch: 1897, cost 13.37 s | |
| 2024-01-16 05:12:26,795 44k INFO ====> Epoch: 1898, cost 13.01 s | |
| 2024-01-16 05:12:39,461 44k INFO ====> Epoch: 1899, cost 12.67 s | |
| 2024-01-16 05:12:52,594 44k INFO ====> Epoch: 1900, cost 13.13 s | |
| 2024-01-16 05:13:05,367 44k INFO ====> Epoch: 1901, cost 12.77 s | |
| 2024-01-16 05:13:18,190 44k INFO ====> Epoch: 1902, cost 12.82 s | |
| 2024-01-16 05:13:31,258 44k INFO ====> Epoch: 1903, cost 13.07 s | |
| 2024-01-16 05:13:44,207 44k INFO ====> Epoch: 1904, cost 12.95 s | |
| 2024-01-16 05:13:57,172 44k INFO ====> Epoch: 1905, cost 12.96 s | |
| 2024-01-16 05:14:10,390 44k INFO ====> Epoch: 1906, cost 13.22 s | |
| 2024-01-16 05:14:23,169 44k INFO ====> Epoch: 1907, cost 12.78 s | |
| 2024-01-16 05:14:33,478 44k INFO Train Epoch: 1908 [62%] | |
| 2024-01-16 05:14:33,479 44k INFO Losses: [2.370302677154541, 2.4678943157196045, 7.7766547203063965, 11.575559616088867, 0.6432259678840637], step: 24800, lr: 7.878954310215385e-05, reference_loss: 24.833637237548828 | |
| 2024-01-16 05:14:42,089 44k INFO Saving model and optimizer state at iteration 1908 to ./logs/44k/G_24800.pth | |
| 2024-01-16 05:14:44,219 44k INFO Saving model and optimizer state at iteration 1908 to ./logs/44k/D_24800.pth | |
| 2024-01-16 05:14:44,891 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_20800.pth | |
| 2024-01-16 05:14:44,943 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_20800.pth | |
| 2024-01-16 05:14:47,683 44k INFO ====> Epoch: 1908, cost 24.51 s | |
| 2024-01-16 05:15:01,385 44k INFO ====> Epoch: 1909, cost 13.70 s | |
| 2024-01-16 05:15:14,738 44k INFO ====> Epoch: 1910, cost 13.35 s | |
| 2024-01-16 05:15:28,195 44k INFO ====> Epoch: 1911, cost 13.46 s | |
| 2024-01-16 05:15:41,463 44k INFO ====> Epoch: 1912, cost 13.27 s | |
| 2024-01-16 05:15:54,662 44k INFO ====> Epoch: 1913, cost 13.20 s | |
| 2024-01-16 05:16:08,045 44k INFO ====> Epoch: 1914, cost 13.38 s | |
| 2024-01-16 05:16:21,731 44k INFO ====> Epoch: 1915, cost 13.69 s | |
| 2024-01-16 05:16:34,877 44k INFO ====> Epoch: 1916, cost 13.15 s | |
| 2024-01-16 05:16:48,224 44k INFO ====> Epoch: 1917, cost 13.35 s | |
| 2024-01-16 05:17:01,695 44k INFO ====> Epoch: 1918, cost 13.47 s | |
| 2024-01-16 05:17:14,970 44k INFO ====> Epoch: 1919, cost 13.27 s | |
| 2024-01-16 05:17:28,518 44k INFO ====> Epoch: 1920, cost 13.55 s | |
| 2024-01-16 05:17:41,952 44k INFO ====> Epoch: 1921, cost 13.43 s | |
| 2024-01-16 05:17:55,437 44k INFO ====> Epoch: 1922, cost 13.49 s | |
| 2024-01-16 05:18:08,435 44k INFO ====> Epoch: 1923, cost 13.00 s | |
| 2024-01-16 05:18:15,541 44k INFO Train Epoch: 1924 [0%] | |
| 2024-01-16 05:18:15,542 44k INFO Losses: [2.453542709350586, 2.4297986030578613, 4.539793968200684, 12.840065956115723, 0.3832564353942871], step: 25000, lr: 7.863211166020172e-05, reference_loss: 22.646459579467773 | |
| 2024-01-16 05:18:24,253 44k INFO Saving model and optimizer state at iteration 1924 to ./logs/44k/G_25000.pth | |
| 2024-01-16 05:18:25,880 44k INFO Saving model and optimizer state at iteration 1924 to ./logs/44k/D_25000.pth | |
| 2024-01-16 05:18:26,619 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_21000.pth | |
| 2024-01-16 05:18:26,662 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_21000.pth | |
| 2024-01-16 05:18:32,417 44k INFO ====> Epoch: 1924, cost 23.98 s | |
| 2024-01-16 05:18:45,998 44k INFO ====> Epoch: 1925, cost 13.58 s | |
| 2024-01-16 05:18:58,867 44k INFO ====> Epoch: 1926, cost 12.87 s | |
| 2024-01-16 05:19:11,764 44k INFO ====> Epoch: 1927, cost 12.90 s | |
| 2024-01-16 05:19:24,983 44k INFO ====> Epoch: 1928, cost 13.22 s | |
| 2024-01-16 05:19:38,322 44k INFO ====> Epoch: 1929, cost 13.34 s | |
| 2024-01-16 05:19:51,481 44k INFO ====> Epoch: 1930, cost 13.16 s | |
| 2024-01-16 05:20:04,649 44k INFO ====> Epoch: 1931, cost 13.17 s | |
| 2024-01-16 05:20:17,982 44k INFO ====> Epoch: 1932, cost 13.33 s | |
| 2024-01-16 05:20:31,453 44k INFO ====> Epoch: 1933, cost 13.47 s | |
| 2024-01-16 05:20:44,719 44k INFO ====> Epoch: 1934, cost 13.27 s | |
| 2024-01-16 05:20:57,805 44k INFO ====> Epoch: 1935, cost 13.09 s | |
| 2024-01-16 05:21:10,860 44k INFO ====> Epoch: 1936, cost 13.06 s | |
| 2024-01-16 05:21:23,899 44k INFO ====> Epoch: 1937, cost 13.04 s | |
| 2024-01-16 05:21:36,849 44k INFO ====> Epoch: 1938, cost 12.95 s | |
| 2024-01-16 05:21:46,085 44k INFO Train Epoch: 1939 [38%] | |
| 2024-01-16 05:21:46,085 44k INFO Losses: [2.342489242553711, 2.450054407119751, 6.575702667236328, 13.315779685974121, 0.07038093358278275], step: 25200, lr: 7.848480538679502e-05, reference_loss: 24.754405975341797 | |
| 2024-01-16 05:21:54,888 44k INFO Saving model and optimizer state at iteration 1939 to ./logs/44k/G_25200.pth | |
| 2024-01-16 05:21:56,477 44k INFO Saving model and optimizer state at iteration 1939 to ./logs/44k/D_25200.pth | |
| 2024-01-16 05:21:57,226 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_21200.pth | |
| 2024-01-16 05:21:57,272 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_21200.pth | |
| 2024-01-16 05:22:01,194 44k INFO ====> Epoch: 1939, cost 24.34 s | |
| 2024-01-16 05:22:14,418 44k INFO ====> Epoch: 1940, cost 13.22 s | |
| 2024-01-16 05:22:27,889 44k INFO ====> Epoch: 1941, cost 13.47 s | |
| 2024-01-16 05:22:41,486 44k INFO ====> Epoch: 1942, cost 13.60 s | |
| 2024-01-16 05:22:54,650 44k INFO ====> Epoch: 1943, cost 13.16 s | |
| 2024-01-16 05:23:07,853 44k INFO ====> Epoch: 1944, cost 13.20 s | |
| 2024-01-16 05:23:21,152 44k INFO ====> Epoch: 1945, cost 13.30 s | |
| 2024-01-16 05:23:34,389 44k INFO ====> Epoch: 1946, cost 13.24 s | |
| 2024-01-16 05:23:47,484 44k INFO ====> Epoch: 1947, cost 13.10 s | |
| 2024-01-16 05:24:00,956 44k INFO ====> Epoch: 1948, cost 13.47 s | |
| 2024-01-16 05:24:13,995 44k INFO ====> Epoch: 1949, cost 13.04 s | |
| 2024-01-16 05:24:27,445 44k INFO ====> Epoch: 1950, cost 13.45 s | |
| 2024-01-16 05:24:41,238 44k INFO ====> Epoch: 1951, cost 13.79 s | |
| 2024-01-16 05:24:54,452 44k INFO ====> Epoch: 1952, cost 13.21 s | |
| 2024-01-16 05:25:07,483 44k INFO ====> Epoch: 1953, cost 13.03 s | |
| 2024-01-16 05:25:18,641 44k INFO Train Epoch: 1954 [77%] | |
| 2024-01-16 05:25:18,642 44k INFO Losses: [2.469691514968872, 2.2860162258148193, 4.562650203704834, 9.628241539001465, 0.08964487165212631], step: 25400, lr: 7.833777507110747e-05, reference_loss: 19.036243438720703 | |
| 2024-01-16 05:25:27,141 44k INFO Saving model and optimizer state at iteration 1954 to ./logs/44k/G_25400.pth | |
| 2024-01-16 05:25:28,785 44k INFO Saving model and optimizer state at iteration 1954 to ./logs/44k/D_25400.pth | |
| 2024-01-16 05:25:29,553 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_21400.pth | |
| 2024-01-16 05:25:29,603 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_21400.pth | |
| 2024-01-16 05:25:31,607 44k INFO ====> Epoch: 1954, cost 24.12 s | |
| 2024-01-16 05:25:45,099 44k INFO ====> Epoch: 1955, cost 13.49 s | |
| 2024-01-16 05:25:59,032 44k INFO ====> Epoch: 1956, cost 13.93 s | |
| 2024-01-16 05:26:12,176 44k INFO ====> Epoch: 1957, cost 13.14 s | |
| 2024-01-16 05:26:25,552 44k INFO ====> Epoch: 1958, cost 13.38 s | |
| 2024-01-16 05:26:38,768 44k INFO ====> Epoch: 1959, cost 13.22 s | |
| 2024-01-16 05:26:51,752 44k INFO ====> Epoch: 1960, cost 12.98 s | |
| 2024-01-16 05:27:05,053 44k INFO ====> Epoch: 1961, cost 13.30 s | |
| 2024-01-16 05:27:18,001 44k INFO ====> Epoch: 1962, cost 12.95 s | |
| 2024-01-16 05:27:30,987 44k INFO ====> Epoch: 1963, cost 12.99 s | |
| 2024-01-16 05:27:43,784 44k INFO ====> Epoch: 1964, cost 12.80 s | |
| 2024-01-16 05:27:56,865 44k INFO ====> Epoch: 1965, cost 13.08 s | |
| 2024-01-16 05:28:10,213 44k INFO ====> Epoch: 1966, cost 13.35 s | |
| 2024-01-16 05:28:23,413 44k INFO ====> Epoch: 1967, cost 13.20 s | |
| 2024-01-16 05:28:36,968 44k INFO ====> Epoch: 1968, cost 13.56 s | |
| 2024-01-16 05:28:49,677 44k INFO ====> Epoch: 1969, cost 12.71 s | |
| 2024-01-16 05:28:57,559 44k INFO Train Epoch: 1970 [15%] | |
| 2024-01-16 05:28:57,559 44k INFO Losses: [2.098663568496704, 2.530102014541626, 6.210485458374023, 9.621566772460938, 0.6946834921836853], step: 25600, lr: 7.81812463186463e-05, reference_loss: 21.155500411987305 | |
| 2024-01-16 05:29:06,584 44k INFO Saving model and optimizer state at iteration 1970 to ./logs/44k/G_25600.pth | |
| 2024-01-16 05:29:08,149 44k INFO Saving model and optimizer state at iteration 1970 to ./logs/44k/D_25600.pth | |
| 2024-01-16 05:29:08,838 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_21600.pth | |
| 2024-01-16 05:29:08,883 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_21600.pth | |
| 2024-01-16 05:29:13,947 44k INFO ====> Epoch: 1970, cost 24.27 s | |
| 2024-01-16 05:29:27,051 44k INFO ====> Epoch: 1971, cost 13.10 s | |
| 2024-01-16 05:29:40,195 44k INFO ====> Epoch: 1972, cost 13.14 s | |
| 2024-01-16 05:29:52,969 44k INFO ====> Epoch: 1973, cost 12.77 s | |
| 2024-01-16 05:30:06,163 44k INFO ====> Epoch: 1974, cost 13.19 s | |
| 2024-01-16 05:30:19,089 44k INFO ====> Epoch: 1975, cost 12.93 s | |
| 2024-01-16 05:30:31,844 44k INFO ====> Epoch: 1976, cost 12.76 s | |
| 2024-01-16 05:30:44,752 44k INFO ====> Epoch: 1977, cost 12.91 s | |
| 2024-01-16 05:30:57,637 44k INFO ====> Epoch: 1978, cost 12.88 s | |
| 2024-01-16 05:31:10,759 44k INFO ====> Epoch: 1979, cost 13.12 s | |
| 2024-01-16 05:31:24,035 44k INFO ====> Epoch: 1980, cost 13.28 s | |
| 2024-01-16 05:31:37,127 44k INFO ====> Epoch: 1981, cost 13.09 s | |
| 2024-01-16 05:31:49,806 44k INFO ====> Epoch: 1982, cost 12.68 s | |
| 2024-01-16 05:32:03,336 44k INFO ====> Epoch: 1983, cost 13.53 s | |
| 2024-01-16 05:32:16,456 44k INFO ====> Epoch: 1984, cost 13.12 s | |
| 2024-01-16 05:32:26,383 44k INFO Train Epoch: 1985 [54%] | |
| 2024-01-16 05:32:26,384 44k INFO Losses: [2.3670713901519775, 2.5543394088745117, 4.839135646820068, 9.534073829650879, 0.21705634891986847], step: 25800, lr: 7.80347846784546e-05, reference_loss: 19.511676788330078 | |
| 2024-01-16 05:32:34,740 44k INFO Saving model and optimizer state at iteration 1985 to ./logs/44k/G_25800.pth | |
| 2024-01-16 05:32:36,322 44k INFO Saving model and optimizer state at iteration 1985 to ./logs/44k/D_25800.pth | |
| 2024-01-16 05:32:37,019 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_21800.pth | |
| 2024-01-16 05:32:37,062 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_21800.pth | |
| 2024-01-16 05:32:40,230 44k INFO ====> Epoch: 1985, cost 23.77 s | |
| 2024-01-16 05:32:53,685 44k INFO ====> Epoch: 1986, cost 13.46 s | |
| 2024-01-16 05:33:06,571 44k INFO ====> Epoch: 1987, cost 12.89 s | |
| 2024-01-16 05:33:19,529 44k INFO ====> Epoch: 1988, cost 12.96 s | |
| 2024-01-16 05:33:32,294 44k INFO ====> Epoch: 1989, cost 12.77 s | |
| 2024-01-16 05:33:45,672 44k INFO ====> Epoch: 1990, cost 13.38 s | |
| 2024-01-16 05:33:58,534 44k INFO ====> Epoch: 1991, cost 12.86 s | |
| 2024-01-16 05:34:11,381 44k INFO ====> Epoch: 1992, cost 12.85 s | |
| 2024-01-16 05:34:24,361 44k INFO ====> Epoch: 1993, cost 12.98 s | |
| 2024-01-16 05:34:37,560 44k INFO ====> Epoch: 1994, cost 13.20 s | |
| 2024-01-16 05:34:50,417 44k INFO ====> Epoch: 1995, cost 12.86 s | |
| 2024-01-16 05:35:03,457 44k INFO ====> Epoch: 1996, cost 13.04 s | |
| 2024-01-16 05:35:16,886 44k INFO ====> Epoch: 1997, cost 13.43 s | |
| 2024-01-16 05:35:29,623 44k INFO ====> Epoch: 1998, cost 12.74 s | |
| 2024-01-16 05:35:42,805 44k INFO ====> Epoch: 1999, cost 13.18 s | |
| 2024-01-16 05:35:54,529 44k INFO Train Epoch: 2000 [92%] | |
| 2024-01-16 05:35:54,530 44k INFO Losses: [2.432644844055176, 2.1155202388763428, 6.849728584289551, 11.076061248779297, 0.4813309907913208], step: 26000, lr: 7.788859741367973e-05, reference_loss: 22.955286026000977 | |
| 2024-01-16 05:36:02,756 44k INFO Saving model and optimizer state at iteration 2000 to ./logs/44k/G_26000.pth | |
| 2024-01-16 05:36:04,367 44k INFO Saving model and optimizer state at iteration 2000 to ./logs/44k/D_26000.pth | |
| 2024-01-16 05:36:05,063 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_22000.pth | |
| 2024-01-16 05:36:05,123 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_22000.pth | |
| 2024-01-16 05:36:06,249 44k INFO ====> Epoch: 2000, cost 23.44 s | |
| 2024-01-16 05:36:19,155 44k INFO ====> Epoch: 2001, cost 12.91 s | |
| 2024-01-16 05:36:31,995 44k INFO ====> Epoch: 2002, cost 12.84 s | |
| 2024-01-16 05:36:45,027 44k INFO ====> Epoch: 2003, cost 13.03 s | |
| 2024-01-16 05:36:57,924 44k INFO ====> Epoch: 2004, cost 12.90 s | |
| 2024-01-16 05:37:10,720 44k INFO ====> Epoch: 2005, cost 12.80 s | |
| 2024-01-16 05:37:23,991 44k INFO ====> Epoch: 2006, cost 13.27 s | |
| 2024-01-16 05:37:36,972 44k INFO ====> Epoch: 2007, cost 12.98 s | |
| 2024-01-16 05:37:49,696 44k INFO ====> Epoch: 2008, cost 12.72 s | |
| 2024-01-16 05:38:02,510 44k INFO ====> Epoch: 2009, cost 12.81 s | |
| 2024-01-16 05:38:15,520 44k INFO ====> Epoch: 2010, cost 13.01 s | |
| 2024-01-16 05:38:28,237 44k INFO ====> Epoch: 2011, cost 12.72 s | |
| 2024-01-16 05:38:41,074 44k INFO ====> Epoch: 2012, cost 12.84 s | |
| 2024-01-16 05:38:54,300 44k INFO ====> Epoch: 2013, cost 13.23 s | |
| 2024-01-16 05:39:07,276 44k INFO ====> Epoch: 2014, cost 12.98 s | |
| 2024-01-16 05:39:20,552 44k INFO ====> Epoch: 2015, cost 13.28 s | |
| 2024-01-16 05:39:29,047 44k INFO Train Epoch: 2016 [31%] | |
| 2024-01-16 05:39:29,048 44k INFO Losses: [2.779675006866455, 1.790321707725525, 1.5515860319137573, 5.847805023193359, 0.09156299382448196], step: 26200, lr: 7.773296617481642e-05, reference_loss: 12.060951232910156 | |
| 2024-01-16 05:39:37,898 44k INFO Saving model and optimizer state at iteration 2016 to ./logs/44k/G_26200.pth | |
| 2024-01-16 05:39:39,479 44k INFO Saving model and optimizer state at iteration 2016 to ./logs/44k/D_26200.pth | |
| 2024-01-16 05:39:40,176 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_22200.pth | |
| 2024-01-16 05:39:40,252 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_22200.pth | |
| 2024-01-16 05:39:44,467 44k INFO ====> Epoch: 2016, cost 23.92 s | |
| 2024-01-16 05:39:57,234 44k INFO ====> Epoch: 2017, cost 12.77 s | |
| 2024-01-16 05:40:10,229 44k INFO ====> Epoch: 2018, cost 13.00 s | |
| 2024-01-16 05:40:22,988 44k INFO ====> Epoch: 2019, cost 12.76 s | |
| 2024-01-16 05:40:35,937 44k INFO ====> Epoch: 2020, cost 12.95 s | |
| 2024-01-16 05:40:48,719 44k INFO ====> Epoch: 2021, cost 12.78 s | |
| 2024-01-16 05:41:01,629 44k INFO ====> Epoch: 2022, cost 12.91 s | |
| 2024-01-16 05:41:14,813 44k INFO ====> Epoch: 2023, cost 13.18 s | |
| 2024-01-16 05:41:27,572 44k INFO ====> Epoch: 2024, cost 12.76 s | |
| 2024-01-16 05:41:40,545 44k INFO ====> Epoch: 2025, cost 12.97 s | |
| 2024-01-16 05:41:53,260 44k INFO ====> Epoch: 2026, cost 12.71 s | |
| 2024-01-16 05:42:06,688 44k INFO ====> Epoch: 2027, cost 13.43 s | |
| 2024-01-16 05:42:19,813 44k INFO ====> Epoch: 2028, cost 13.12 s | |
| 2024-01-16 05:42:32,697 44k INFO ====> Epoch: 2029, cost 12.88 s | |
| 2024-01-16 05:42:45,470 44k INFO ====> Epoch: 2030, cost 12.77 s | |
| 2024-01-16 05:42:56,025 44k INFO Train Epoch: 2031 [69%] | |
| 2024-01-16 05:42:56,026 44k INFO Losses: [2.203197479248047, 2.4352149963378906, 8.84375286102295, 16.270122528076172, 0.902536153793335], step: 26400, lr: 7.758734432483304e-05, reference_loss: 30.65482521057129 | |
| 2024-01-16 05:43:04,397 44k INFO Saving model and optimizer state at iteration 2031 to ./logs/44k/G_26400.pth | |
| 2024-01-16 05:43:05,950 44k INFO Saving model and optimizer state at iteration 2031 to ./logs/44k/D_26400.pth | |
| 2024-01-16 05:43:06,640 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_22400.pth | |
| 2024-01-16 05:43:06,709 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_22400.pth | |
| 2024-01-16 05:43:08,974 44k INFO ====> Epoch: 2031, cost 23.50 s | |
| 2024-01-16 05:43:22,124 44k INFO ====> Epoch: 2032, cost 13.15 s | |
| 2024-01-16 05:43:35,682 44k INFO ====> Epoch: 2033, cost 13.56 s | |
| 2024-01-16 05:43:48,651 44k INFO ====> Epoch: 2034, cost 12.97 s | |
| 2024-01-16 05:44:02,228 44k INFO ====> Epoch: 2035, cost 13.58 s | |
| 2024-01-16 05:44:15,048 44k INFO ====> Epoch: 2036, cost 12.82 s | |
| 2024-01-16 05:44:28,039 44k INFO ====> Epoch: 2037, cost 12.99 s | |
| 2024-01-16 05:44:41,028 44k INFO ====> Epoch: 2038, cost 12.99 s | |
| 2024-01-16 05:44:53,987 44k INFO ====> Epoch: 2039, cost 12.96 s | |
| 2024-01-16 05:45:07,113 44k INFO ====> Epoch: 2040, cost 13.13 s | |
| 2024-01-16 05:45:20,234 44k INFO ====> Epoch: 2041, cost 13.12 s | |
| 2024-01-16 05:45:33,288 44k INFO ====> Epoch: 2042, cost 13.05 s | |
| 2024-01-16 05:45:46,155 44k INFO ====> Epoch: 2043, cost 12.87 s | |
| 2024-01-16 05:45:58,942 44k INFO ====> Epoch: 2044, cost 12.79 s | |
| 2024-01-16 05:46:12,181 44k INFO ====> Epoch: 2045, cost 13.24 s | |
| 2024-01-16 05:46:25,324 44k INFO ====> Epoch: 2046, cost 13.14 s | |
| 2024-01-16 05:46:32,866 44k INFO Train Epoch: 2047 [8%] | |
| 2024-01-16 05:46:32,867 44k INFO Losses: [2.2587289810180664, 2.568819999694824, 6.077337265014648, 13.432767868041992, 0.4084266424179077], step: 26600, lr: 7.743231502762723e-05, reference_loss: 24.74608039855957 | |
| 2024-01-16 05:46:41,607 44k INFO Saving model and optimizer state at iteration 2047 to ./logs/44k/G_26600.pth | |
| 2024-01-16 05:46:43,191 44k INFO Saving model and optimizer state at iteration 2047 to ./logs/44k/D_26600.pth | |
| 2024-01-16 05:46:43,879 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_22600.pth | |
| 2024-01-16 05:46:43,947 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_22600.pth | |
| 2024-01-16 05:46:49,335 44k INFO ====> Epoch: 2047, cost 24.01 s | |
| 2024-01-16 05:47:02,257 44k INFO ====> Epoch: 2048, cost 12.92 s | |
| 2024-01-16 05:47:15,103 44k INFO ====> Epoch: 2049, cost 12.85 s | |
| 2024-01-16 05:47:27,940 44k INFO ====> Epoch: 2050, cost 12.84 s | |
| 2024-01-16 05:47:40,599 44k INFO ====> Epoch: 2051, cost 12.66 s | |
| 2024-01-16 05:47:53,470 44k INFO ====> Epoch: 2052, cost 12.87 s | |
| 2024-01-16 05:48:06,108 44k INFO ====> Epoch: 2053, cost 12.64 s | |
| 2024-01-16 05:48:18,674 44k INFO ====> Epoch: 2054, cost 12.57 s | |
| 2024-01-16 05:48:31,693 44k INFO ====> Epoch: 2055, cost 13.02 s | |
| 2024-01-16 05:48:45,223 44k INFO ====> Epoch: 2056, cost 13.53 s | |
| 2024-01-16 05:48:58,325 44k INFO ====> Epoch: 2057, cost 13.10 s | |
| 2024-01-16 05:49:11,943 44k INFO ====> Epoch: 2058, cost 13.62 s | |
| 2024-01-16 05:49:24,829 44k INFO ====> Epoch: 2059, cost 12.89 s | |
| 2024-01-16 05:49:37,570 44k INFO ====> Epoch: 2060, cost 12.74 s | |
| 2024-01-16 05:49:50,532 44k INFO ====> Epoch: 2061, cost 12.96 s | |
| 2024-01-16 05:50:00,197 44k INFO Train Epoch: 2062 [46%] | |
| 2024-01-16 05:50:00,198 44k INFO Losses: [2.3460021018981934, 2.5533499717712402, 6.747447490692139, 11.918466567993164, 0.19703362882137299], step: 26800, lr: 7.728725640555607e-05, reference_loss: 23.762298583984375 | |
| 2024-01-16 05:50:08,620 44k INFO Saving model and optimizer state at iteration 2062 to ./logs/44k/G_26800.pth | |
| 2024-01-16 05:50:10,161 44k INFO Saving model and optimizer state at iteration 2062 to ./logs/44k/D_26800.pth | |
| 2024-01-16 05:50:10,865 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_22800.pth | |
| 2024-01-16 05:50:10,934 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_22800.pth | |
| 2024-01-16 05:50:14,385 44k INFO ====> Epoch: 2062, cost 23.85 s | |
| 2024-01-16 05:50:27,395 44k INFO ====> Epoch: 2063, cost 13.01 s | |
| 2024-01-16 05:50:40,304 44k INFO ====> Epoch: 2064, cost 12.91 s | |
| 2024-01-16 05:50:53,277 44k INFO ====> Epoch: 2065, cost 12.97 s | |
| 2024-01-16 05:51:06,476 44k INFO ====> Epoch: 2066, cost 13.20 s | |
| 2024-01-16 05:51:19,280 44k INFO ====> Epoch: 2067, cost 12.80 s | |
| 2024-01-16 05:51:32,253 44k INFO ====> Epoch: 2068, cost 12.97 s | |
| 2024-01-16 05:51:45,406 44k INFO ====> Epoch: 2069, cost 13.15 s | |
| 2024-01-16 05:51:58,277 44k INFO ====> Epoch: 2070, cost 12.87 s | |
| 2024-01-16 05:52:11,355 44k INFO ====> Epoch: 2071, cost 13.08 s | |
| 2024-01-16 05:52:24,389 44k INFO ====> Epoch: 2072, cost 13.03 s | |
| 2024-01-16 05:52:37,389 44k INFO ====> Epoch: 2073, cost 13.00 s | |
| 2024-01-16 05:52:50,140 44k INFO ====> Epoch: 2074, cost 12.75 s | |
| 2024-01-16 05:53:02,897 44k INFO ====> Epoch: 2075, cost 12.76 s | |
| 2024-01-16 05:53:16,089 44k INFO ====> Epoch: 2076, cost 13.19 s | |
| 2024-01-16 05:53:27,431 44k INFO Train Epoch: 2077 [85%] | |
| 2024-01-16 05:53:27,432 44k INFO Losses: [2.20831036567688, 2.626575469970703, 6.592917442321777, 11.507522583007812, 0.6659302711486816], step: 27000, lr: 7.714246953054337e-05, reference_loss: 23.601255416870117 | |
| 2024-01-16 05:53:35,958 44k INFO Saving model and optimizer state at iteration 2077 to ./logs/44k/G_27000.pth | |
| 2024-01-16 05:53:37,464 44k INFO Saving model and optimizer state at iteration 2077 to ./logs/44k/D_27000.pth | |
| 2024-01-16 05:53:38,153 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_23000.pth | |
| 2024-01-16 05:53:38,223 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_23000.pth | |
| 2024-01-16 05:53:39,877 44k INFO ====> Epoch: 2077, cost 23.79 s | |
| 2024-01-16 05:53:53,255 44k INFO ====> Epoch: 2078, cost 13.38 s | |
| 2024-01-16 05:54:06,340 44k INFO ====> Epoch: 2079, cost 13.09 s | |
| 2024-01-16 05:54:19,474 44k INFO ====> Epoch: 2080, cost 13.13 s | |
| 2024-01-16 05:54:32,462 44k INFO ====> Epoch: 2081, cost 12.99 s | |
| 2024-01-16 05:54:45,699 44k INFO ====> Epoch: 2082, cost 13.24 s | |
| 2024-01-16 05:54:58,737 44k INFO ====> Epoch: 2083, cost 13.04 s | |
| 2024-01-16 05:55:12,028 44k INFO ====> Epoch: 2084, cost 13.29 s | |
| 2024-01-16 05:55:24,861 44k INFO ====> Epoch: 2085, cost 12.83 s | |
| 2024-01-16 05:55:37,900 44k INFO ====> Epoch: 2086, cost 13.04 s | |
| 2024-01-16 05:55:50,998 44k INFO ====> Epoch: 2087, cost 13.10 s | |
| 2024-01-16 05:56:04,097 44k INFO ====> Epoch: 2088, cost 13.10 s | |
| 2024-01-16 05:56:17,350 44k INFO ====> Epoch: 2089, cost 13.25 s | |
| 2024-01-16 05:56:30,579 44k INFO ====> Epoch: 2090, cost 13.23 s | |
| 2024-01-16 05:56:43,473 44k INFO ====> Epoch: 2091, cost 12.89 s | |
| 2024-01-16 05:56:56,300 44k INFO ====> Epoch: 2092, cost 12.83 s | |
| 2024-01-16 05:57:04,722 44k INFO Train Epoch: 2093 [23%] | |
| 2024-01-16 05:57:04,723 44k INFO Losses: [2.03710675239563, 3.2380805015563965, 9.136356353759766, 14.109323501586914, 0.6292440891265869], step: 27200, lr: 7.698832914927228e-05, reference_loss: 29.150110244750977 | |
| 2024-01-16 05:57:13,023 44k INFO Saving model and optimizer state at iteration 2093 to ./logs/44k/G_27200.pth | |
| 2024-01-16 05:57:14,647 44k INFO Saving model and optimizer state at iteration 2093 to ./logs/44k/D_27200.pth | |
| 2024-01-16 05:57:15,340 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_23200.pth | |
| 2024-01-16 05:57:15,390 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_23200.pth | |
| 2024-01-16 05:57:20,141 44k INFO ====> Epoch: 2093, cost 23.84 s | |
| 2024-01-16 05:57:33,480 44k INFO ====> Epoch: 2094, cost 13.34 s | |
| 2024-01-16 05:57:46,260 44k INFO ====> Epoch: 2095, cost 12.78 s | |
| 2024-01-16 05:57:58,937 44k INFO ====> Epoch: 2096, cost 12.68 s | |