| Loading training set... | |
| Num images: 210072 | |
| Image shape: [1, 256, 256] | |
| Label shape: [0] | |
| Constructing networks... | |
| Setting up PyTorch plugin "bias_act_plugin"... Done. | |
| Setting up PyTorch plugin "upfirdn2d_plugin"... Done. | |
| Generator Parameters Buffers Output shape Datatype | |
| --- --- --- --- --- | |
| mapping.fc0 262656 - [16, 512] float32 | |
| mapping.fc1 262656 - [16, 512] float32 | |
| mapping - 512 [16, 14, 512] float32 | |
| synthesis.b4.conv1 2622465 32 [16, 512, 4, 4] float32 | |
| synthesis.b4.torgb 263169 - [16, 1, 4, 4] float32 | |
| synthesis.b4:0 8192 16 [16, 512, 4, 4] float32 | |
| synthesis.b4:1 - - [16, 512, 4, 4] float32 | |
| synthesis.b8.conv0 2622465 80 [16, 512, 8, 8] float32 | |
| synthesis.b8.conv1 2622465 80 [16, 512, 8, 8] float32 | |
| synthesis.b8.torgb 263169 - [16, 1, 8, 8] float32 | |
| synthesis.b8:0 - 16 [16, 512, 8, 8] float32 | |
| synthesis.b8:1 - - [16, 512, 8, 8] float32 | |
| synthesis.b16.conv0 2622465 272 [16, 512, 16, 16] float32 | |
| synthesis.b16.conv1 2622465 272 [16, 512, 16, 16] float32 | |
| synthesis.b16.torgb 263169 - [16, 1, 16, 16] float32 | |
| synthesis.b16:0 - 16 [16, 512, 16, 16] float32 | |
| synthesis.b16:1 - - [16, 512, 16, 16] float32 | |
| synthesis.b32.conv0 2622465 1040 [16, 512, 32, 32] float16 | |
| synthesis.b32.conv1 2622465 1040 [16, 512, 32, 32] float16 | |
| synthesis.b32.torgb 263169 - [16, 1, 32, 32] float16 | |
| synthesis.b32:0 - 16 [16, 512, 32, 32] float16 | |
| synthesis.b32:1 - - [16, 512, 32, 32] float32 | |
| synthesis.b64.conv0 1442561 4112 [16, 256, 64, 64] float16 | |
| synthesis.b64.conv1 721409 4112 [16, 256, 64, 64] float16 | |
| synthesis.b64.torgb 131585 - [16, 1, 64, 64] float16 | |
| synthesis.b64:0 - 16 [16, 256, 64, 64] float16 | |
| synthesis.b64:1 - - [16, 256, 64, 64] float32 | |
| synthesis.b128.conv0 426369 16400 [16, 128, 128, 128] float16 | |
| synthesis.b128.conv1 213249 16400 [16, 128, 128, 128] float16 | |
| synthesis.b128.torgb 65793 - [16, 1, 128, 128] float16 | |
| synthesis.b128:0 - 16 [16, 128, 128, 128] float16 | |
| synthesis.b128:1 - - [16, 128, 128, 128] float32 | |
| synthesis.b256.conv0 139457 65552 [16, 64, 256, 256] float16 | |
| synthesis.b256.conv1 69761 65552 [16, 64, 256, 256] float16 | |
| synthesis.b256.torgb 32897 - [16, 1, 256, 256] float16 | |
| synthesis.b256:0 - 16 [16, 64, 256, 256] float16 | |
| synthesis.b256:1 - - [16, 64, 256, 256] float32 | |
| --- --- --- --- --- | |
| Total 23186516 175568 - - | |
| Discriminator Parameters Buffers Output shape Datatype | |
| --- --- --- --- --- | |
| b256.fromrgb 128 16 [16, 64, 256, 256] float16 | |
| b256.skip 8192 16 [16, 128, 128, 128] float16 | |
| b256.conv0 36928 16 [16, 64, 256, 256] float16 | |
| b256.conv1 73856 16 [16, 128, 128, 128] float16 | |
| b256 - 16 [16, 128, 128, 128] float16 | |
| b128.skip 32768 16 [16, 256, 64, 64] float16 | |
| b128.conv0 147584 16 [16, 128, 128, 128] float16 | |
| b128.conv1 295168 16 [16, 256, 64, 64] float16 | |
| b128 - 16 [16, 256, 64, 64] float16 | |
| b64.skip 131072 16 [16, 512, 32, 32] float16 | |
| b64.conv0 590080 16 [16, 256, 64, 64] float16 | |
| b64.conv1 1180160 16 [16, 512, 32, 32] float16 | |
| b64 - 16 [16, 512, 32, 32] float16 | |
| b32.skip 262144 16 [16, 512, 16, 16] float16 | |
| b32.conv0 2359808 16 [16, 512, 32, 32] float16 | |
| b32.conv1 2359808 16 [16, 512, 16, 16] float16 | |
| b32 - 16 [16, 512, 16, 16] float16 | |
| b16.skip 262144 16 [16, 512, 8, 8] float32 | |
| b16.conv0 2359808 16 [16, 512, 16, 16] float32 | |
| b16.conv1 2359808 16 [16, 512, 8, 8] float32 | |
| b16 - 16 [16, 512, 8, 8] float32 | |
| b8.skip 262144 16 [16, 512, 4, 4] float32 | |
| b8.conv0 2359808 16 [16, 512, 8, 8] float32 | |
| b8.conv1 2359808 16 [16, 512, 4, 4] float32 | |
| b8 - 16 [16, 512, 4, 4] float32 | |
| b4.mbstd - - [16, 513, 4, 4] float32 | |
| b4.conv 2364416 16 [16, 512, 4, 4] float32 | |
| b4.fc 4194816 - [16, 512] float32 | |
| b4.out 513 - [16, 1] float32 | |
| --- --- --- --- --- | |
| Total 24000961 416 - - | |
| Setting up augmentation... | |
| Distributing across 1 GPUs... | |
| Setting up training phases... | |
| Exporting sample images... | |
| Initializing logs... | |
| Skipping tfevents export: No module named 'tensorboard' | |
| Training for 25000 kimg... | |
| tick 0 kimg 0.0 time 15s sec/tick 2.1 sec/kimg 133.95 maintenance 12.4 cpumem 4.43 gpumem 9.10 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 294.8993321776852}, "metric": "fid50k_full", "total_time": 395.37557458877563, "total_time_str": "6m 35s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000000.pkl", "timestamp": 1694469522.230665} | |
| {"results": {"kid50k_full": 0.35734518119119096}, "metric": "kid50k_full", "total_time": 337.68898820877075, "total_time_str": "5m 38s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000000.pkl", "timestamp": 1694469859.9199846} | |
| tick 1 kimg 4.0 time 14m 22s sec/tick 112.1 sec/kimg 28.03 maintenance 735.4 cpumem 4.70 gpumem 4.96 augment 0.004 | |
| tick 2 kimg 8.0 time 16m 15s sec/tick 112.8 sec/kimg 28.19 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.008 | |
| tick 3 kimg 12.0 time 18m 07s sec/tick 112.4 sec/kimg 28.10 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.011 | |
| tick 4 kimg 16.0 time 20m 00s sec/tick 112.8 sec/kimg 28.19 maintenance 0.1 cpumem 4.70 gpumem 4.96 augment 0.016 | |
| tick 5 kimg 20.0 time 21m 53s sec/tick 112.7 sec/kimg 28.17 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.020 | |
| tick 6 kimg 24.0 time 23m 46s sec/tick 112.7 sec/kimg 28.19 maintenance 0.1 cpumem 4.70 gpumem 4.96 augment 0.022 | |
| tick 7 kimg 28.0 time 25m 39s sec/tick 112.8 sec/kimg 28.19 maintenance 0.1 cpumem 4.70 gpumem 4.96 augment 0.024 | |
| tick 8 kimg 32.0 time 27m 32s sec/tick 112.9 sec/kimg 28.23 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.023 | |
| tick 9 kimg 36.0 time 29m 24s sec/tick 112.6 sec/kimg 28.16 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.022 | |
| tick 10 kimg 40.0 time 31m 17s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.020 | |
| tick 11 kimg 44.0 time 33m 10s sec/tick 112.7 sec/kimg 28.16 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.016 | |
| tick 12 kimg 48.0 time 35m 03s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.70 gpumem 4.96 augment 0.012 | |
| tick 13 kimg 52.0 time 36m 56s sec/tick 112.9 sec/kimg 28.22 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.010 | |
| tick 14 kimg 56.0 time 38m 49s sec/tick 112.8 sec/kimg 28.21 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.006 | |
| tick 15 kimg 60.0 time 40m 42s sec/tick 112.9 sec/kimg 28.21 maintenance 0.1 cpumem 4.70 gpumem 4.96 augment 0.003 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 55.59129346625522}, "metric": "fid50k_full", "total_time": 238.58489155769348, "total_time_str": "3m 59s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000060.pkl", "timestamp": 1694471793.989422} | |
| {"results": {"kid50k_full": 0.05879741746996998}, "metric": "kid50k_full", "total_time": 224.35378313064575, "total_time_str": "3m 44s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000060.pkl", "timestamp": 1694472018.343481} | |
| tick 16 kimg 64.0 time 50m 21s sec/tick 112.9 sec/kimg 28.23 maintenance 466.4 cpumem 4.71 gpumem 4.96 augment 0.000 | |
| tick 17 kimg 68.0 time 52m 14s sec/tick 112.7 sec/kimg 28.18 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 18 kimg 72.0 time 54m 07s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 19 kimg 76.0 time 56m 00s sec/tick 112.7 sec/kimg 28.17 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 20 kimg 80.0 time 57m 59s sec/tick 119.3 sec/kimg 29.83 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 21 kimg 84.0 time 59m 54s sec/tick 115.1 sec/kimg 28.78 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 22 kimg 88.0 time 1h 01m 50s sec/tick 115.1 sec/kimg 28.78 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000 | |
| tick 23 kimg 92.0 time 1h 03m 44s sec/tick 114.5 sec/kimg 28.62 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 24 kimg 96.0 time 1h 05m 37s sec/tick 113.0 sec/kimg 28.26 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 25 kimg 100.0 time 1h 07m 30s sec/tick 112.6 sec/kimg 28.16 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 26 kimg 104.0 time 1h 09m 23s sec/tick 112.9 sec/kimg 28.23 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 27 kimg 108.0 time 1h 11m 16s sec/tick 112.5 sec/kimg 28.13 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 28 kimg 112.0 time 1h 13m 08s sec/tick 112.8 sec/kimg 28.21 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 29 kimg 116.0 time 1h 15m 01s sec/tick 112.6 sec/kimg 28.16 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 30 kimg 120.0 time 1h 16m 54s sec/tick 112.6 sec/kimg 28.15 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 16.44946759961782}, "metric": "fid50k_full", "total_time": 238.37569093704224, "total_time_str": "3m 58s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000120.pkl", "timestamp": 1694473965.9814692} | |
| {"results": {"kid50k_full": 0.010068384204204172}, "metric": "kid50k_full", "total_time": 223.98649263381958, "total_time_str": "3m 44s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000120.pkl", "timestamp": 1694474189.9682415} | |
| tick 31 kimg 124.0 time 1h 26m 33s sec/tick 112.7 sec/kimg 28.19 maintenance 466.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 32 kimg 128.0 time 1h 28m 26s sec/tick 113.1 sec/kimg 28.27 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 33 kimg 132.0 time 1h 30m 19s sec/tick 112.8 sec/kimg 28.19 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 34 kimg 136.0 time 1h 32m 12s sec/tick 113.1 sec/kimg 28.28 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 35 kimg 140.0 time 1h 34m 05s sec/tick 112.8 sec/kimg 28.21 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 36 kimg 144.0 time 1h 35m 58s sec/tick 113.2 sec/kimg 28.29 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 37 kimg 148.0 time 1h 37m 51s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 38 kimg 152.0 time 1h 39m 44s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 39 kimg 156.0 time 1h 41m 37s sec/tick 113.0 sec/kimg 28.26 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 40 kimg 160.0 time 1h 43m 31s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 41 kimg 164.0 time 1h 45m 24s sec/tick 112.9 sec/kimg 28.22 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 42 kimg 168.0 time 1h 47m 17s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 43 kimg 172.0 time 1h 49m 10s sec/tick 112.9 sec/kimg 28.22 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 44 kimg 176.0 time 1h 51m 03s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 45 kimg 180.0 time 1h 52m 56s sec/tick 113.1 sec/kimg 28.27 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 9.918456024968364}, "metric": "fid50k_full", "total_time": 239.10222578048706, "total_time_str": "3m 59s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000180.pkl", "timestamp": 1694476129.2916071} | |
| {"results": {"kid50k_full": 0.005936421599099058}, "metric": "kid50k_full", "total_time": 224.82831645011902, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000180.pkl", "timestamp": 1694476354.1202128} | |
| tick 46 kimg 184.0 time 2h 02m 37s sec/tick 113.2 sec/kimg 28.29 maintenance 467.8 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 47 kimg 188.0 time 2h 04m 31s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 48 kimg 192.0 time 2h 06m 24s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 49 kimg 196.0 time 2h 08m 18s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 50 kimg 200.0 time 2h 10m 11s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 51 kimg 204.0 time 2h 12m 05s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 52 kimg 208.0 time 2h 13m 58s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 53 kimg 212.0 time 2h 15m 52s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 54 kimg 216.0 time 2h 17m 45s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 55 kimg 220.0 time 2h 19m 39s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 56 kimg 224.0 time 2h 21m 32s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 57 kimg 228.0 time 2h 23m 25s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 58 kimg 232.0 time 2h 25m 19s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 59 kimg 236.0 time 2h 27m 12s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 60 kimg 240.0 time 2h 29m 06s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 9.452240121526977}, "metric": "fid50k_full", "total_time": 239.6933295726776, "total_time_str": "4m 00s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000240.pkl", "timestamp": 1694478299.5639074} | |
| {"results": {"kid50k_full": 0.006892820880880881}, "metric": "kid50k_full", "total_time": 225.23785781860352, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000240.pkl", "timestamp": 1694478524.8020515} | |
| tick 61 kimg 244.0 time 2h 38m 48s sec/tick 113.1 sec/kimg 28.27 maintenance 468.7 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 62 kimg 248.0 time 2h 40m 41s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 63 kimg 252.0 time 2h 42m 34s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 64 kimg 256.0 time 2h 44m 28s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 65 kimg 260.0 time 2h 46m 21s sec/tick 113.0 sec/kimg 28.26 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 66 kimg 264.0 time 2h 48m 14s sec/tick 113.3 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 67 kimg 268.0 time 2h 50m 07s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 68 kimg 272.0 time 2h 52m 01s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 69 kimg 276.0 time 2h 53m 54s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 70 kimg 280.0 time 2h 55m 47s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 71 kimg 284.0 time 2h 57m 41s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 72 kimg 288.0 time 2h 59m 34s sec/tick 113.3 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 73 kimg 292.0 time 3h 01m 27s sec/tick 113.0 sec/kimg 28.26 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 74 kimg 296.0 time 3h 03m 21s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000 | |
| tick 75 kimg 300.0 time 3h 05m 14s sec/tick 113.0 sec/kimg 28.26 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 8.061355302401937}, "metric": "fid50k_full", "total_time": 240.18138766288757, "total_time_str": "4m 00s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000300.pkl", "timestamp": 1694480467.8838542} | |
| {"results": {"kid50k_full": 0.005201244559559513}, "metric": "kid50k_full", "total_time": 225.220853805542, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000300.pkl", "timestamp": 1694480693.104995} | |
| tick 76 kimg 304.0 time 3h 14m 57s sec/tick 113.5 sec/kimg 28.37 maintenance 469.3 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 77 kimg 308.0 time 3h 16m 50s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 78 kimg 312.0 time 3h 18m 44s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 79 kimg 316.0 time 3h 20m 37s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 80 kimg 320.0 time 3h 22m 31s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 81 kimg 324.0 time 3h 24m 25s sec/tick 113.3 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 82 kimg 328.0 time 3h 26m 18s sec/tick 113.7 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 83 kimg 332.0 time 3h 28m 12s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 84 kimg 336.0 time 3h 30m 05s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 85 kimg 340.0 time 3h 31m 59s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 86 kimg 344.0 time 3h 33m 52s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 87 kimg 348.0 time 3h 35m 46s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 88 kimg 352.0 time 3h 37m 40s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 89 kimg 356.0 time 3h 39m 33s sec/tick 113.3 sec/kimg 28.32 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 90 kimg 360.0 time 3h 41m 27s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 7.719476042123093}, "metric": "fid50k_full", "total_time": 240.04656720161438, "total_time_str": "4m 00s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000360.pkl", "timestamp": 1694482640.7526622} | |
| {"results": {"kid50k_full": 0.005320215370370388}, "metric": "kid50k_full", "total_time": 225.3283920288086, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000360.pkl", "timestamp": 1694482866.081344} | |
| tick 91 kimg 364.0 time 3h 51m 09s sec/tick 113.2 sec/kimg 28.29 maintenance 469.3 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 92 kimg 368.0 time 3h 53m 03s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 93 kimg 372.0 time 3h 54m 57s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 94 kimg 376.0 time 3h 56m 50s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 95 kimg 380.0 time 3h 58m 44s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 96 kimg 384.0 time 4h 00m 38s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 97 kimg 388.0 time 4h 02m 31s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 98 kimg 392.0 time 4h 04m 25s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 99 kimg 396.0 time 4h 06m 18s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000 | |
| tick 100 kimg 400.0 time 4h 08m 12s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 101 kimg 404.0 time 4h 10m 05s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 102 kimg 408.0 time 4h 11m 59s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 103 kimg 412.0 time 4h 13m 53s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 104 kimg 416.0 time 4h 15m 46s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 105 kimg 420.0 time 4h 17m 40s sec/tick 113.3 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 7.5430205678006015}, "metric": "fid50k_full", "total_time": 239.69175672531128, "total_time_str": "4m 00s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000420.pkl", "timestamp": 1694484813.315543} | |
| {"results": {"kid50k_full": 0.005404414549549542}, "metric": "kid50k_full", "total_time": 225.35277199745178, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000420.pkl", "timestamp": 1694485038.6685941} | |
| tick 106 kimg 424.0 time 4h 27m 22s sec/tick 113.3 sec/kimg 28.32 maintenance 468.9 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 107 kimg 428.0 time 4h 29m 15s sec/tick 113.1 sec/kimg 28.29 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 108 kimg 432.0 time 4h 31m 09s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 109 kimg 436.0 time 4h 33m 02s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 110 kimg 440.0 time 4h 34m 55s sec/tick 113.3 sec/kimg 28.32 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 111 kimg 444.0 time 4h 36m 49s sec/tick 113.3 sec/kimg 28.32 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 112 kimg 448.0 time 4h 38m 42s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 113 kimg 452.0 time 4h 40m 36s sec/tick 113.1 sec/kimg 28.28 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 114 kimg 456.0 time 4h 42m 29s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 115 kimg 460.0 time 4h 44m 22s sec/tick 113.1 sec/kimg 28.28 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 116 kimg 464.0 time 4h 46m 16s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 117 kimg 468.0 time 4h 48m 09s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 118 kimg 472.0 time 4h 50m 03s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 119 kimg 476.0 time 4h 51m 56s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 120 kimg 480.0 time 4h 53m 50s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 6.783721627319087}, "metric": "fid50k_full", "total_time": 239.38124322891235, "total_time_str": "3m 59s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000480.pkl", "timestamp": 1694486982.9125054} | |
| {"results": {"kid50k_full": 0.004552755480480523}, "metric": "kid50k_full", "total_time": 225.26296138763428, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000480.pkl", "timestamp": 1694487208.1757457} | |
| tick 121 kimg 484.0 time 5h 03m 31s sec/tick 113.3 sec/kimg 28.32 maintenance 468.6 cpumem 4.71 gpumem 4.96 augment 0.000 | |
| tick 122 kimg 488.0 time 5h 05m 25s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 123 kimg 492.0 time 5h 07m 19s sec/tick 113.5 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 124 kimg 496.0 time 5h 09m 13s sec/tick 113.8 sec/kimg 28.44 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 125 kimg 500.0 time 5h 11m 06s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 126 kimg 504.0 time 5h 13m 00s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 127 kimg 508.0 time 5h 14m 54s sec/tick 113.7 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 128 kimg 512.0 time 5h 16m 48s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 129 kimg 516.0 time 5h 18m 41s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 130 kimg 520.0 time 5h 20m 35s sec/tick 113.8 sec/kimg 28.44 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 131 kimg 524.0 time 5h 22m 29s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 132 kimg 528.0 time 5h 24m 22s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 133 kimg 532.0 time 5h 26m 16s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 134 kimg 536.0 time 5h 28m 10s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 135 kimg 540.0 time 5h 30m 04s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 6.829867801964767}, "metric": "fid50k_full", "total_time": 240.04878878593445, "total_time_str": "4m 00s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000540.pkl", "timestamp": 1694489157.540102} | |
| {"results": {"kid50k_full": 0.004643149031531557}, "metric": "kid50k_full", "total_time": 225.50702142715454, "total_time_str": "3m 46s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000540.pkl", "timestamp": 1694489383.0474086} | |
| tick 136 kimg 544.0 time 5h 39m 47s sec/tick 113.7 sec/kimg 28.42 maintenance 469.5 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 137 kimg 548.0 time 5h 41m 40s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 138 kimg 552.0 time 5h 43m 34s sec/tick 113.9 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 139 kimg 556.0 time 5h 45m 28s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 140 kimg 560.0 time 5h 47m 22s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 141 kimg 564.0 time 5h 49m 15s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 142 kimg 568.0 time 5h 51m 09s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 143 kimg 572.0 time 5h 53m 03s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 144 kimg 576.0 time 5h 54m 57s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 145 kimg 580.0 time 5h 56m 50s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 146 kimg 584.0 time 5h 58m 44s sec/tick 113.8 sec/kimg 28.44 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 147 kimg 588.0 time 6h 00m 38s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 148 kimg 592.0 time 6h 02m 31s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 149 kimg 596.0 time 6h 04m 25s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 150 kimg 600.0 time 6h 06m 19s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 5.881322419672354}, "metric": "fid50k_full", "total_time": 240.72823476791382, "total_time_str": "4m 01s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000600.pkl", "timestamp": 1694491333.4616177} | |
| {"results": {"kid50k_full": 0.00393620870620623}, "metric": "kid50k_full", "total_time": 225.68321442604065, "total_time_str": "3m 46s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000600.pkl", "timestamp": 1694491559.1451108} | |
| tick 151 kimg 604.0 time 6h 16m 02s sec/tick 113.2 sec/kimg 28.31 maintenance 470.3 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 152 kimg 608.0 time 6h 17m 56s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 153 kimg 612.0 time 6h 19m 49s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 154 kimg 616.0 time 6h 21m 43s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 155 kimg 620.0 time 6h 23m 36s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 156 kimg 624.0 time 6h 25m 30s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 157 kimg 628.0 time 6h 27m 23s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 158 kimg 632.0 time 6h 29m 17s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 159 kimg 636.0 time 6h 31m 10s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 160 kimg 640.0 time 6h 33m 04s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 161 kimg 644.0 time 6h 34m 57s sec/tick 113.3 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 162 kimg 648.0 time 6h 36m 51s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 163 kimg 652.0 time 6h 38m 44s sec/tick 113.3 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 164 kimg 656.0 time 6h 40m 38s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 165 kimg 660.0 time 6h 42m 31s sec/tick 113.5 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 5.80970544570431}, "metric": "fid50k_full", "total_time": 240.70476269721985, "total_time_str": "4m 01s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000660.pkl", "timestamp": 1694493506.2023673} | |
| {"results": {"kid50k_full": 0.0037522971071071013}, "metric": "kid50k_full", "total_time": 225.48487401008606, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000660.pkl", "timestamp": 1694493731.6875253} | |
| tick 166 kimg 664.0 time 6h 52m 15s sec/tick 113.5 sec/kimg 28.37 maintenance 470.2 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 167 kimg 668.0 time 6h 54m 09s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 168 kimg 672.0 time 6h 56m 03s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 169 kimg 676.0 time 6h 57m 56s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 170 kimg 680.0 time 6h 59m 50s sec/tick 113.9 sec/kimg 28.47 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 171 kimg 684.0 time 7h 01m 44s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 172 kimg 688.0 time 7h 03m 38s sec/tick 113.9 sec/kimg 28.47 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 173 kimg 692.0 time 7h 05m 32s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 174 kimg 696.0 time 7h 07m 25s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 175 kimg 700.0 time 7h 09m 19s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 176 kimg 704.0 time 7h 11m 13s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 177 kimg 708.0 time 7h 13m 07s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 178 kimg 712.0 time 7h 15m 01s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 179 kimg 716.0 time 7h 16m 54s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 180 kimg 720.0 time 7h 18m 48s sec/tick 113.9 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 5.903405487320894}, "metric": "fid50k_full", "total_time": 240.6245424747467, "total_time_str": "4m 01s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000720.pkl", "timestamp": 1694495682.7637277} | |
| {"results": {"kid50k_full": 0.0036957564614614464}, "metric": "kid50k_full", "total_time": 225.74648714065552, "total_time_str": "3m 46s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000720.pkl", "timestamp": 1694495908.5105233} | |
| tick 181 kimg 724.0 time 7h 28m 32s sec/tick 113.3 sec/kimg 28.33 maintenance 470.3 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 182 kimg 728.0 time 7h 30m 25s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 183 kimg 732.0 time 7h 32m 19s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 184 kimg 736.0 time 7h 34m 13s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 185 kimg 740.0 time 7h 36m 06s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 186 kimg 744.0 time 7h 38m 00s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 187 kimg 748.0 time 7h 39m 53s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 188 kimg 752.0 time 7h 41m 47s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 189 kimg 756.0 time 7h 43m 40s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 190 kimg 760.0 time 7h 45m 34s sec/tick 113.3 sec/kimg 28.32 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 191 kimg 764.0 time 7h 47m 27s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 192 kimg 768.0 time 7h 49m 21s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 193 kimg 772.0 time 7h 51m 14s sec/tick 113.1 sec/kimg 28.27 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 194 kimg 776.0 time 7h 53m 07s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 195 kimg 780.0 time 7h 55m 00s sec/tick 113.1 sec/kimg 28.27 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 6.840991906315833}, "metric": "fid50k_full", "total_time": 239.02368783950806, "total_time_str": "3m 59s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000780.pkl", "timestamp": 1694497853.3174217} | |
| {"results": {"kid50k_full": 0.004607744977477469}, "metric": "kid50k_full", "total_time": 225.11438941955566, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000780.pkl", "timestamp": 1694498078.4320977} | |
| tick 196 kimg 784.0 time 8h 04m 42s sec/tick 113.2 sec/kimg 28.30 maintenance 468.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 197 kimg 788.0 time 8h 06m 35s sec/tick 113.2 sec/kimg 28.29 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 198 kimg 792.0 time 8h 08m 28s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 199 kimg 796.0 time 8h 10m 22s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 200 kimg 800.0 time 8h 12m 15s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 201 kimg 804.0 time 8h 14m 09s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 202 kimg 808.0 time 8h 16m 03s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 203 kimg 812.0 time 8h 17m 56s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 204 kimg 816.0 time 8h 19m 50s sec/tick 113.7 sec/kimg 28.44 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 205 kimg 820.0 time 8h 21m 43s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000 | |
| tick 206 kimg 824.0 time 8h 23m 37s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 207 kimg 828.0 time 8h 25m 31s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 208 kimg 832.0 time 8h 27m 25s sec/tick 113.8 sec/kimg 28.44 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 209 kimg 836.0 time 8h 29m 18s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 210 kimg 840.0 time 8h 31m 12s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 4.954847615273339}, "metric": "fid50k_full", "total_time": 240.69801425933838, "total_time_str": "4m 01s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000840.pkl", "timestamp": 1694500026.7810807} | |
| {"results": {"kid50k_full": 0.0029223774124124066}, "metric": "kid50k_full", "total_time": 225.517991065979, "total_time_str": "3m 46s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000840.pkl", "timestamp": 1694500252.2993548} | |
| tick 211 kimg 844.0 time 8h 40m 56s sec/tick 113.4 sec/kimg 28.34 maintenance 470.2 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 212 kimg 848.0 time 8h 42m 50s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 213 kimg 852.0 time 8h 44m 43s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 214 kimg 856.0 time 8h 46m 37s sec/tick 113.7 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 215 kimg 860.0 time 8h 48m 31s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000 | |
| tick 216 kimg 864.0 time 8h 50m 25s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 217 kimg 868.0 time 8h 52m 18s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 218 kimg 872.0 time 8h 54m 12s sec/tick 113.9 sec/kimg 28.48 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 219 kimg 876.0 time 8h 56m 06s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 220 kimg 880.0 time 8h 58m 00s sec/tick 113.9 sec/kimg 28.47 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 221 kimg 884.0 time 8h 59m 54s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 222 kimg 888.0 time 9h 01m 47s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 223 kimg 892.0 time 9h 03m 41s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 224 kimg 896.0 time 9h 05m 35s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 225 kimg 900.0 time 9h 07m 29s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 5.152596627974495}, "metric": "fid50k_full", "total_time": 240.53874516487122, "total_time_str": "4m 01s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000900.pkl", "timestamp": 1694502203.1773508} | |
| {"results": {"kid50k_full": 0.003050379662162177}, "metric": "kid50k_full", "total_time": 225.55694794654846, "total_time_str": "3m 46s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000900.pkl", "timestamp": 1694502428.7345843} | |
| tick 226 kimg 904.0 time 9h 17m 12s sec/tick 113.5 sec/kimg 28.37 maintenance 470.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 227 kimg 908.0 time 9h 19m 06s sec/tick 113.3 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000 | |
| tick 228 kimg 912.0 time 9h 20m 59s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 229 kimg 916.0 time 9h 22m 53s sec/tick 114.0 sec/kimg 28.49 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 230 kimg 920.0 time 9h 24m 51s sec/tick 117.2 sec/kimg 29.29 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 231 kimg 924.0 time 9h 26m 46s sec/tick 115.7 sec/kimg 28.92 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 232 kimg 928.0 time 9h 28m 42s sec/tick 115.8 sec/kimg 28.95 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 233 kimg 932.0 time 9h 30m 38s sec/tick 115.4 sec/kimg 28.86 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 234 kimg 936.0 time 9h 32m 34s sec/tick 115.8 sec/kimg 28.95 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 235 kimg 940.0 time 9h 34m 29s sec/tick 115.4 sec/kimg 28.85 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 236 kimg 944.0 time 9h 36m 25s sec/tick 115.8 sec/kimg 28.94 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 237 kimg 948.0 time 9h 38m 21s sec/tick 115.6 sec/kimg 28.90 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 238 kimg 952.0 time 9h 40m 16s sec/tick 115.8 sec/kimg 28.94 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 239 kimg 956.0 time 9h 42m 14s sec/tick 117.7 sec/kimg 29.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| tick 240 kimg 960.0 time 9h 44m 12s sec/tick 118.2 sec/kimg 29.55 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Evaluating metrics... | |
| {"results": {"fid50k_full": 4.960903881577067}, "metric": "fid50k_full", "total_time": 246.37715554237366, "total_time_str": "4m 06s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000960.pkl", "timestamp": 1694504412.9876099} | |
| {"results": {"kid50k_full": 0.0029740898248248053}, "metric": "kid50k_full", "total_time": 229.59208965301514, "total_time_str": "3m 50s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000960.pkl", "timestamp": 1694504642.5799766} | |
| tick 241 kimg 964.0 time 9h 54m 10s sec/tick 117.3 sec/kimg 29.32 maintenance 480.1 cpumem 4.71 gpumem 4.95 augment 0.000 | |
| Aborted! | |