File size: 47,254 Bytes
1be15e0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 |
Loading training set...
Num images: 210072
Image shape: [1, 256, 256]
Label shape: [0]
Constructing networks...
Setting up PyTorch plugin "bias_act_plugin"... Done.
Setting up PyTorch plugin "upfirdn2d_plugin"... Done.
Generator Parameters Buffers Output shape Datatype
--- --- --- --- ---
mapping.fc0 262656 - [16, 512] float32
mapping.fc1 262656 - [16, 512] float32
mapping - 512 [16, 14, 512] float32
synthesis.b4.conv1 2622465 32 [16, 512, 4, 4] float32
synthesis.b4.torgb 263169 - [16, 1, 4, 4] float32
synthesis.b4:0 8192 16 [16, 512, 4, 4] float32
synthesis.b4:1 - - [16, 512, 4, 4] float32
synthesis.b8.conv0 2622465 80 [16, 512, 8, 8] float32
synthesis.b8.conv1 2622465 80 [16, 512, 8, 8] float32
synthesis.b8.torgb 263169 - [16, 1, 8, 8] float32
synthesis.b8:0 - 16 [16, 512, 8, 8] float32
synthesis.b8:1 - - [16, 512, 8, 8] float32
synthesis.b16.conv0 2622465 272 [16, 512, 16, 16] float32
synthesis.b16.conv1 2622465 272 [16, 512, 16, 16] float32
synthesis.b16.torgb 263169 - [16, 1, 16, 16] float32
synthesis.b16:0 - 16 [16, 512, 16, 16] float32
synthesis.b16:1 - - [16, 512, 16, 16] float32
synthesis.b32.conv0 2622465 1040 [16, 512, 32, 32] float16
synthesis.b32.conv1 2622465 1040 [16, 512, 32, 32] float16
synthesis.b32.torgb 263169 - [16, 1, 32, 32] float16
synthesis.b32:0 - 16 [16, 512, 32, 32] float16
synthesis.b32:1 - - [16, 512, 32, 32] float32
synthesis.b64.conv0 1442561 4112 [16, 256, 64, 64] float16
synthesis.b64.conv1 721409 4112 [16, 256, 64, 64] float16
synthesis.b64.torgb 131585 - [16, 1, 64, 64] float16
synthesis.b64:0 - 16 [16, 256, 64, 64] float16
synthesis.b64:1 - - [16, 256, 64, 64] float32
synthesis.b128.conv0 426369 16400 [16, 128, 128, 128] float16
synthesis.b128.conv1 213249 16400 [16, 128, 128, 128] float16
synthesis.b128.torgb 65793 - [16, 1, 128, 128] float16
synthesis.b128:0 - 16 [16, 128, 128, 128] float16
synthesis.b128:1 - - [16, 128, 128, 128] float32
synthesis.b256.conv0 139457 65552 [16, 64, 256, 256] float16
synthesis.b256.conv1 69761 65552 [16, 64, 256, 256] float16
synthesis.b256.torgb 32897 - [16, 1, 256, 256] float16
synthesis.b256:0 - 16 [16, 64, 256, 256] float16
synthesis.b256:1 - - [16, 64, 256, 256] float32
--- --- --- --- ---
Total 23186516 175568 - -
Discriminator Parameters Buffers Output shape Datatype
--- --- --- --- ---
b256.fromrgb 128 16 [16, 64, 256, 256] float16
b256.skip 8192 16 [16, 128, 128, 128] float16
b256.conv0 36928 16 [16, 64, 256, 256] float16
b256.conv1 73856 16 [16, 128, 128, 128] float16
b256 - 16 [16, 128, 128, 128] float16
b128.skip 32768 16 [16, 256, 64, 64] float16
b128.conv0 147584 16 [16, 128, 128, 128] float16
b128.conv1 295168 16 [16, 256, 64, 64] float16
b128 - 16 [16, 256, 64, 64] float16
b64.skip 131072 16 [16, 512, 32, 32] float16
b64.conv0 590080 16 [16, 256, 64, 64] float16
b64.conv1 1180160 16 [16, 512, 32, 32] float16
b64 - 16 [16, 512, 32, 32] float16
b32.skip 262144 16 [16, 512, 16, 16] float16
b32.conv0 2359808 16 [16, 512, 32, 32] float16
b32.conv1 2359808 16 [16, 512, 16, 16] float16
b32 - 16 [16, 512, 16, 16] float16
b16.skip 262144 16 [16, 512, 8, 8] float32
b16.conv0 2359808 16 [16, 512, 16, 16] float32
b16.conv1 2359808 16 [16, 512, 8, 8] float32
b16 - 16 [16, 512, 8, 8] float32
b8.skip 262144 16 [16, 512, 4, 4] float32
b8.conv0 2359808 16 [16, 512, 8, 8] float32
b8.conv1 2359808 16 [16, 512, 4, 4] float32
b8 - 16 [16, 512, 4, 4] float32
b4.mbstd - - [16, 513, 4, 4] float32
b4.conv 2364416 16 [16, 512, 4, 4] float32
b4.fc 4194816 - [16, 512] float32
b4.out 513 - [16, 1] float32
--- --- --- --- ---
Total 24000961 416 - -
Setting up augmentation...
Distributing across 1 GPUs...
Setting up training phases...
Exporting sample images...
Initializing logs...
Skipping tfevents export: No module named 'tensorboard'
Training for 25000 kimg...
tick 0 kimg 0.0 time 15s sec/tick 2.1 sec/kimg 133.95 maintenance 12.4 cpumem 4.43 gpumem 9.10 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 294.8993321776852}, "metric": "fid50k_full", "total_time": 395.37557458877563, "total_time_str": "6m 35s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000000.pkl", "timestamp": 1694469522.230665}
{"results": {"kid50k_full": 0.35734518119119096}, "metric": "kid50k_full", "total_time": 337.68898820877075, "total_time_str": "5m 38s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000000.pkl", "timestamp": 1694469859.9199846}
tick 1 kimg 4.0 time 14m 22s sec/tick 112.1 sec/kimg 28.03 maintenance 735.4 cpumem 4.70 gpumem 4.96 augment 0.004
tick 2 kimg 8.0 time 16m 15s sec/tick 112.8 sec/kimg 28.19 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.008
tick 3 kimg 12.0 time 18m 07s sec/tick 112.4 sec/kimg 28.10 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.011
tick 4 kimg 16.0 time 20m 00s sec/tick 112.8 sec/kimg 28.19 maintenance 0.1 cpumem 4.70 gpumem 4.96 augment 0.016
tick 5 kimg 20.0 time 21m 53s sec/tick 112.7 sec/kimg 28.17 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.020
tick 6 kimg 24.0 time 23m 46s sec/tick 112.7 sec/kimg 28.19 maintenance 0.1 cpumem 4.70 gpumem 4.96 augment 0.022
tick 7 kimg 28.0 time 25m 39s sec/tick 112.8 sec/kimg 28.19 maintenance 0.1 cpumem 4.70 gpumem 4.96 augment 0.024
tick 8 kimg 32.0 time 27m 32s sec/tick 112.9 sec/kimg 28.23 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.023
tick 9 kimg 36.0 time 29m 24s sec/tick 112.6 sec/kimg 28.16 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.022
tick 10 kimg 40.0 time 31m 17s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.020
tick 11 kimg 44.0 time 33m 10s sec/tick 112.7 sec/kimg 28.16 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.016
tick 12 kimg 48.0 time 35m 03s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.70 gpumem 4.96 augment 0.012
tick 13 kimg 52.0 time 36m 56s sec/tick 112.9 sec/kimg 28.22 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.010
tick 14 kimg 56.0 time 38m 49s sec/tick 112.8 sec/kimg 28.21 maintenance 0.1 cpumem 4.70 gpumem 4.97 augment 0.006
tick 15 kimg 60.0 time 40m 42s sec/tick 112.9 sec/kimg 28.21 maintenance 0.1 cpumem 4.70 gpumem 4.96 augment 0.003
Evaluating metrics...
{"results": {"fid50k_full": 55.59129346625522}, "metric": "fid50k_full", "total_time": 238.58489155769348, "total_time_str": "3m 59s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000060.pkl", "timestamp": 1694471793.989422}
{"results": {"kid50k_full": 0.05879741746996998}, "metric": "kid50k_full", "total_time": 224.35378313064575, "total_time_str": "3m 44s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000060.pkl", "timestamp": 1694472018.343481}
tick 16 kimg 64.0 time 50m 21s sec/tick 112.9 sec/kimg 28.23 maintenance 466.4 cpumem 4.71 gpumem 4.96 augment 0.000
tick 17 kimg 68.0 time 52m 14s sec/tick 112.7 sec/kimg 28.18 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 18 kimg 72.0 time 54m 07s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 19 kimg 76.0 time 56m 00s sec/tick 112.7 sec/kimg 28.17 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 20 kimg 80.0 time 57m 59s sec/tick 119.3 sec/kimg 29.83 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 21 kimg 84.0 time 59m 54s sec/tick 115.1 sec/kimg 28.78 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 22 kimg 88.0 time 1h 01m 50s sec/tick 115.1 sec/kimg 28.78 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000
tick 23 kimg 92.0 time 1h 03m 44s sec/tick 114.5 sec/kimg 28.62 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 24 kimg 96.0 time 1h 05m 37s sec/tick 113.0 sec/kimg 28.26 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 25 kimg 100.0 time 1h 07m 30s sec/tick 112.6 sec/kimg 28.16 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 26 kimg 104.0 time 1h 09m 23s sec/tick 112.9 sec/kimg 28.23 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 27 kimg 108.0 time 1h 11m 16s sec/tick 112.5 sec/kimg 28.13 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 28 kimg 112.0 time 1h 13m 08s sec/tick 112.8 sec/kimg 28.21 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 29 kimg 116.0 time 1h 15m 01s sec/tick 112.6 sec/kimg 28.16 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 30 kimg 120.0 time 1h 16m 54s sec/tick 112.6 sec/kimg 28.15 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 16.44946759961782}, "metric": "fid50k_full", "total_time": 238.37569093704224, "total_time_str": "3m 58s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000120.pkl", "timestamp": 1694473965.9814692}
{"results": {"kid50k_full": 0.010068384204204172}, "metric": "kid50k_full", "total_time": 223.98649263381958, "total_time_str": "3m 44s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000120.pkl", "timestamp": 1694474189.9682415}
tick 31 kimg 124.0 time 1h 26m 33s sec/tick 112.7 sec/kimg 28.19 maintenance 466.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 32 kimg 128.0 time 1h 28m 26s sec/tick 113.1 sec/kimg 28.27 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 33 kimg 132.0 time 1h 30m 19s sec/tick 112.8 sec/kimg 28.19 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 34 kimg 136.0 time 1h 32m 12s sec/tick 113.1 sec/kimg 28.28 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 35 kimg 140.0 time 1h 34m 05s sec/tick 112.8 sec/kimg 28.21 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 36 kimg 144.0 time 1h 35m 58s sec/tick 113.2 sec/kimg 28.29 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 37 kimg 148.0 time 1h 37m 51s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 38 kimg 152.0 time 1h 39m 44s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 39 kimg 156.0 time 1h 41m 37s sec/tick 113.0 sec/kimg 28.26 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 40 kimg 160.0 time 1h 43m 31s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 41 kimg 164.0 time 1h 45m 24s sec/tick 112.9 sec/kimg 28.22 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 42 kimg 168.0 time 1h 47m 17s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 43 kimg 172.0 time 1h 49m 10s sec/tick 112.9 sec/kimg 28.22 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 44 kimg 176.0 time 1h 51m 03s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 45 kimg 180.0 time 1h 52m 56s sec/tick 113.1 sec/kimg 28.27 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 9.918456024968364}, "metric": "fid50k_full", "total_time": 239.10222578048706, "total_time_str": "3m 59s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000180.pkl", "timestamp": 1694476129.2916071}
{"results": {"kid50k_full": 0.005936421599099058}, "metric": "kid50k_full", "total_time": 224.82831645011902, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000180.pkl", "timestamp": 1694476354.1202128}
tick 46 kimg 184.0 time 2h 02m 37s sec/tick 113.2 sec/kimg 28.29 maintenance 467.8 cpumem 4.71 gpumem 4.95 augment 0.000
tick 47 kimg 188.0 time 2h 04m 31s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 48 kimg 192.0 time 2h 06m 24s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 49 kimg 196.0 time 2h 08m 18s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 50 kimg 200.0 time 2h 10m 11s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 51 kimg 204.0 time 2h 12m 05s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 52 kimg 208.0 time 2h 13m 58s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 53 kimg 212.0 time 2h 15m 52s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 54 kimg 216.0 time 2h 17m 45s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 55 kimg 220.0 time 2h 19m 39s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 56 kimg 224.0 time 2h 21m 32s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 57 kimg 228.0 time 2h 23m 25s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 58 kimg 232.0 time 2h 25m 19s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 59 kimg 236.0 time 2h 27m 12s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 60 kimg 240.0 time 2h 29m 06s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 9.452240121526977}, "metric": "fid50k_full", "total_time": 239.6933295726776, "total_time_str": "4m 00s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000240.pkl", "timestamp": 1694478299.5639074}
{"results": {"kid50k_full": 0.006892820880880881}, "metric": "kid50k_full", "total_time": 225.23785781860352, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000240.pkl", "timestamp": 1694478524.8020515}
tick 61 kimg 244.0 time 2h 38m 48s sec/tick 113.1 sec/kimg 28.27 maintenance 468.7 cpumem 4.71 gpumem 4.95 augment 0.000
tick 62 kimg 248.0 time 2h 40m 41s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 63 kimg 252.0 time 2h 42m 34s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 64 kimg 256.0 time 2h 44m 28s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 65 kimg 260.0 time 2h 46m 21s sec/tick 113.0 sec/kimg 28.26 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 66 kimg 264.0 time 2h 48m 14s sec/tick 113.3 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 67 kimg 268.0 time 2h 50m 07s sec/tick 113.0 sec/kimg 28.25 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 68 kimg 272.0 time 2h 52m 01s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 69 kimg 276.0 time 2h 53m 54s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 70 kimg 280.0 time 2h 55m 47s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 71 kimg 284.0 time 2h 57m 41s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 72 kimg 288.0 time 2h 59m 34s sec/tick 113.3 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 73 kimg 292.0 time 3h 01m 27s sec/tick 113.0 sec/kimg 28.26 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 74 kimg 296.0 time 3h 03m 21s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000
tick 75 kimg 300.0 time 3h 05m 14s sec/tick 113.0 sec/kimg 28.26 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 8.061355302401937}, "metric": "fid50k_full", "total_time": 240.18138766288757, "total_time_str": "4m 00s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000300.pkl", "timestamp": 1694480467.8838542}
{"results": {"kid50k_full": 0.005201244559559513}, "metric": "kid50k_full", "total_time": 225.220853805542, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000300.pkl", "timestamp": 1694480693.104995}
tick 76 kimg 304.0 time 3h 14m 57s sec/tick 113.5 sec/kimg 28.37 maintenance 469.3 cpumem 4.71 gpumem 4.95 augment 0.000
tick 77 kimg 308.0 time 3h 16m 50s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 78 kimg 312.0 time 3h 18m 44s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 79 kimg 316.0 time 3h 20m 37s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 80 kimg 320.0 time 3h 22m 31s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 81 kimg 324.0 time 3h 24m 25s sec/tick 113.3 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 82 kimg 328.0 time 3h 26m 18s sec/tick 113.7 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 83 kimg 332.0 time 3h 28m 12s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 84 kimg 336.0 time 3h 30m 05s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 85 kimg 340.0 time 3h 31m 59s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 86 kimg 344.0 time 3h 33m 52s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 87 kimg 348.0 time 3h 35m 46s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 88 kimg 352.0 time 3h 37m 40s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 89 kimg 356.0 time 3h 39m 33s sec/tick 113.3 sec/kimg 28.32 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 90 kimg 360.0 time 3h 41m 27s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 7.719476042123093}, "metric": "fid50k_full", "total_time": 240.04656720161438, "total_time_str": "4m 00s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000360.pkl", "timestamp": 1694482640.7526622}
{"results": {"kid50k_full": 0.005320215370370388}, "metric": "kid50k_full", "total_time": 225.3283920288086, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000360.pkl", "timestamp": 1694482866.081344}
tick 91 kimg 364.0 time 3h 51m 09s sec/tick 113.2 sec/kimg 28.29 maintenance 469.3 cpumem 4.71 gpumem 4.95 augment 0.000
tick 92 kimg 368.0 time 3h 53m 03s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 93 kimg 372.0 time 3h 54m 57s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 94 kimg 376.0 time 3h 56m 50s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 95 kimg 380.0 time 3h 58m 44s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 96 kimg 384.0 time 4h 00m 38s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 97 kimg 388.0 time 4h 02m 31s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 98 kimg 392.0 time 4h 04m 25s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 99 kimg 396.0 time 4h 06m 18s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000
tick 100 kimg 400.0 time 4h 08m 12s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 101 kimg 404.0 time 4h 10m 05s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 102 kimg 408.0 time 4h 11m 59s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 103 kimg 412.0 time 4h 13m 53s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 104 kimg 416.0 time 4h 15m 46s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 105 kimg 420.0 time 4h 17m 40s sec/tick 113.3 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 7.5430205678006015}, "metric": "fid50k_full", "total_time": 239.69175672531128, "total_time_str": "4m 00s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000420.pkl", "timestamp": 1694484813.315543}
{"results": {"kid50k_full": 0.005404414549549542}, "metric": "kid50k_full", "total_time": 225.35277199745178, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000420.pkl", "timestamp": 1694485038.6685941}
tick 106 kimg 424.0 time 4h 27m 22s sec/tick 113.3 sec/kimg 28.32 maintenance 468.9 cpumem 4.71 gpumem 4.95 augment 0.000
tick 107 kimg 428.0 time 4h 29m 15s sec/tick 113.1 sec/kimg 28.29 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 108 kimg 432.0 time 4h 31m 09s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 109 kimg 436.0 time 4h 33m 02s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 110 kimg 440.0 time 4h 34m 55s sec/tick 113.3 sec/kimg 28.32 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 111 kimg 444.0 time 4h 36m 49s sec/tick 113.3 sec/kimg 28.32 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 112 kimg 448.0 time 4h 38m 42s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 113 kimg 452.0 time 4h 40m 36s sec/tick 113.1 sec/kimg 28.28 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 114 kimg 456.0 time 4h 42m 29s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 115 kimg 460.0 time 4h 44m 22s sec/tick 113.1 sec/kimg 28.28 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 116 kimg 464.0 time 4h 46m 16s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 117 kimg 468.0 time 4h 48m 09s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 118 kimg 472.0 time 4h 50m 03s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 119 kimg 476.0 time 4h 51m 56s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 120 kimg 480.0 time 4h 53m 50s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 6.783721627319087}, "metric": "fid50k_full", "total_time": 239.38124322891235, "total_time_str": "3m 59s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000480.pkl", "timestamp": 1694486982.9125054}
{"results": {"kid50k_full": 0.004552755480480523}, "metric": "kid50k_full", "total_time": 225.26296138763428, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000480.pkl", "timestamp": 1694487208.1757457}
tick 121 kimg 484.0 time 5h 03m 31s sec/tick 113.3 sec/kimg 28.32 maintenance 468.6 cpumem 4.71 gpumem 4.96 augment 0.000
tick 122 kimg 488.0 time 5h 05m 25s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 123 kimg 492.0 time 5h 07m 19s sec/tick 113.5 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 124 kimg 496.0 time 5h 09m 13s sec/tick 113.8 sec/kimg 28.44 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 125 kimg 500.0 time 5h 11m 06s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 126 kimg 504.0 time 5h 13m 00s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 127 kimg 508.0 time 5h 14m 54s sec/tick 113.7 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 128 kimg 512.0 time 5h 16m 48s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 129 kimg 516.0 time 5h 18m 41s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 130 kimg 520.0 time 5h 20m 35s sec/tick 113.8 sec/kimg 28.44 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 131 kimg 524.0 time 5h 22m 29s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 132 kimg 528.0 time 5h 24m 22s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 133 kimg 532.0 time 5h 26m 16s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 134 kimg 536.0 time 5h 28m 10s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 135 kimg 540.0 time 5h 30m 04s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 6.829867801964767}, "metric": "fid50k_full", "total_time": 240.04878878593445, "total_time_str": "4m 00s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000540.pkl", "timestamp": 1694489157.540102}
{"results": {"kid50k_full": 0.004643149031531557}, "metric": "kid50k_full", "total_time": 225.50702142715454, "total_time_str": "3m 46s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000540.pkl", "timestamp": 1694489383.0474086}
tick 136 kimg 544.0 time 5h 39m 47s sec/tick 113.7 sec/kimg 28.42 maintenance 469.5 cpumem 4.71 gpumem 4.95 augment 0.000
tick 137 kimg 548.0 time 5h 41m 40s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 138 kimg 552.0 time 5h 43m 34s sec/tick 113.9 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 139 kimg 556.0 time 5h 45m 28s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 140 kimg 560.0 time 5h 47m 22s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 141 kimg 564.0 time 5h 49m 15s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 142 kimg 568.0 time 5h 51m 09s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 143 kimg 572.0 time 5h 53m 03s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 144 kimg 576.0 time 5h 54m 57s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 145 kimg 580.0 time 5h 56m 50s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 146 kimg 584.0 time 5h 58m 44s sec/tick 113.8 sec/kimg 28.44 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 147 kimg 588.0 time 6h 00m 38s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 148 kimg 592.0 time 6h 02m 31s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 149 kimg 596.0 time 6h 04m 25s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 150 kimg 600.0 time 6h 06m 19s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 5.881322419672354}, "metric": "fid50k_full", "total_time": 240.72823476791382, "total_time_str": "4m 01s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000600.pkl", "timestamp": 1694491333.4616177}
{"results": {"kid50k_full": 0.00393620870620623}, "metric": "kid50k_full", "total_time": 225.68321442604065, "total_time_str": "3m 46s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000600.pkl", "timestamp": 1694491559.1451108}
tick 151 kimg 604.0 time 6h 16m 02s sec/tick 113.2 sec/kimg 28.31 maintenance 470.3 cpumem 4.71 gpumem 4.95 augment 0.000
tick 152 kimg 608.0 time 6h 17m 56s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 153 kimg 612.0 time 6h 19m 49s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 154 kimg 616.0 time 6h 21m 43s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 155 kimg 620.0 time 6h 23m 36s sec/tick 113.2 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 156 kimg 624.0 time 6h 25m 30s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 157 kimg 628.0 time 6h 27m 23s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 158 kimg 632.0 time 6h 29m 17s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 159 kimg 636.0 time 6h 31m 10s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 160 kimg 640.0 time 6h 33m 04s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 161 kimg 644.0 time 6h 34m 57s sec/tick 113.3 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 162 kimg 648.0 time 6h 36m 51s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 163 kimg 652.0 time 6h 38m 44s sec/tick 113.3 sec/kimg 28.31 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 164 kimg 656.0 time 6h 40m 38s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 165 kimg 660.0 time 6h 42m 31s sec/tick 113.5 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 5.80970544570431}, "metric": "fid50k_full", "total_time": 240.70476269721985, "total_time_str": "4m 01s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000660.pkl", "timestamp": 1694493506.2023673}
{"results": {"kid50k_full": 0.0037522971071071013}, "metric": "kid50k_full", "total_time": 225.48487401008606, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000660.pkl", "timestamp": 1694493731.6875253}
tick 166 kimg 664.0 time 6h 52m 15s sec/tick 113.5 sec/kimg 28.37 maintenance 470.2 cpumem 4.71 gpumem 4.95 augment 0.000
tick 167 kimg 668.0 time 6h 54m 09s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 168 kimg 672.0 time 6h 56m 03s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 169 kimg 676.0 time 6h 57m 56s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 170 kimg 680.0 time 6h 59m 50s sec/tick 113.9 sec/kimg 28.47 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 171 kimg 684.0 time 7h 01m 44s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 172 kimg 688.0 time 7h 03m 38s sec/tick 113.9 sec/kimg 28.47 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 173 kimg 692.0 time 7h 05m 32s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 174 kimg 696.0 time 7h 07m 25s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 175 kimg 700.0 time 7h 09m 19s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 176 kimg 704.0 time 7h 11m 13s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 177 kimg 708.0 time 7h 13m 07s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 178 kimg 712.0 time 7h 15m 01s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 179 kimg 716.0 time 7h 16m 54s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 180 kimg 720.0 time 7h 18m 48s sec/tick 113.9 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 5.903405487320894}, "metric": "fid50k_full", "total_time": 240.6245424747467, "total_time_str": "4m 01s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000720.pkl", "timestamp": 1694495682.7637277}
{"results": {"kid50k_full": 0.0036957564614614464}, "metric": "kid50k_full", "total_time": 225.74648714065552, "total_time_str": "3m 46s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000720.pkl", "timestamp": 1694495908.5105233}
tick 181 kimg 724.0 time 7h 28m 32s sec/tick 113.3 sec/kimg 28.33 maintenance 470.3 cpumem 4.71 gpumem 4.95 augment 0.000
tick 182 kimg 728.0 time 7h 30m 25s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 183 kimg 732.0 time 7h 32m 19s sec/tick 113.5 sec/kimg 28.38 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 184 kimg 736.0 time 7h 34m 13s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 185 kimg 740.0 time 7h 36m 06s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 186 kimg 744.0 time 7h 38m 00s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 187 kimg 748.0 time 7h 39m 53s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 188 kimg 752.0 time 7h 41m 47s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 189 kimg 756.0 time 7h 43m 40s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 190 kimg 760.0 time 7h 45m 34s sec/tick 113.3 sec/kimg 28.32 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 191 kimg 764.0 time 7h 47m 27s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 192 kimg 768.0 time 7h 49m 21s sec/tick 113.4 sec/kimg 28.35 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 193 kimg 772.0 time 7h 51m 14s sec/tick 113.1 sec/kimg 28.27 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 194 kimg 776.0 time 7h 53m 07s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 195 kimg 780.0 time 7h 55m 00s sec/tick 113.1 sec/kimg 28.27 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 6.840991906315833}, "metric": "fid50k_full", "total_time": 239.02368783950806, "total_time_str": "3m 59s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000780.pkl", "timestamp": 1694497853.3174217}
{"results": {"kid50k_full": 0.004607744977477469}, "metric": "kid50k_full", "total_time": 225.11438941955566, "total_time_str": "3m 45s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000780.pkl", "timestamp": 1694498078.4320977}
tick 196 kimg 784.0 time 8h 04m 42s sec/tick 113.2 sec/kimg 28.30 maintenance 468.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 197 kimg 788.0 time 8h 06m 35s sec/tick 113.2 sec/kimg 28.29 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 198 kimg 792.0 time 8h 08m 28s sec/tick 113.2 sec/kimg 28.30 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 199 kimg 796.0 time 8h 10m 22s sec/tick 113.4 sec/kimg 28.36 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 200 kimg 800.0 time 8h 12m 15s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 201 kimg 804.0 time 8h 14m 09s sec/tick 113.3 sec/kimg 28.33 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 202 kimg 808.0 time 8h 16m 03s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 203 kimg 812.0 time 8h 17m 56s sec/tick 113.4 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 204 kimg 816.0 time 8h 19m 50s sec/tick 113.7 sec/kimg 28.44 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 205 kimg 820.0 time 8h 21m 43s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000
tick 206 kimg 824.0 time 8h 23m 37s sec/tick 113.6 sec/kimg 28.40 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 207 kimg 828.0 time 8h 25m 31s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 208 kimg 832.0 time 8h 27m 25s sec/tick 113.8 sec/kimg 28.44 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 209 kimg 836.0 time 8h 29m 18s sec/tick 113.5 sec/kimg 28.37 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 210 kimg 840.0 time 8h 31m 12s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 4.954847615273339}, "metric": "fid50k_full", "total_time": 240.69801425933838, "total_time_str": "4m 01s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000840.pkl", "timestamp": 1694500026.7810807}
{"results": {"kid50k_full": 0.0029223774124124066}, "metric": "kid50k_full", "total_time": 225.517991065979, "total_time_str": "3m 46s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000840.pkl", "timestamp": 1694500252.2993548}
tick 211 kimg 844.0 time 8h 40m 56s sec/tick 113.4 sec/kimg 28.34 maintenance 470.2 cpumem 4.71 gpumem 4.95 augment 0.000
tick 212 kimg 848.0 time 8h 42m 50s sec/tick 113.8 sec/kimg 28.45 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 213 kimg 852.0 time 8h 44m 43s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 214 kimg 856.0 time 8h 46m 37s sec/tick 113.7 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 215 kimg 860.0 time 8h 48m 31s sec/tick 113.7 sec/kimg 28.43 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000
tick 216 kimg 864.0 time 8h 50m 25s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 217 kimg 868.0 time 8h 52m 18s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 218 kimg 872.0 time 8h 54m 12s sec/tick 113.9 sec/kimg 28.48 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 219 kimg 876.0 time 8h 56m 06s sec/tick 113.6 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 220 kimg 880.0 time 8h 58m 00s sec/tick 113.9 sec/kimg 28.47 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 221 kimg 884.0 time 8h 59m 54s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 222 kimg 888.0 time 9h 01m 47s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 223 kimg 892.0 time 9h 03m 41s sec/tick 113.7 sec/kimg 28.42 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 224 kimg 896.0 time 9h 05m 35s sec/tick 113.8 sec/kimg 28.46 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 225 kimg 900.0 time 9h 07m 29s sec/tick 113.5 sec/kimg 28.39 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 5.152596627974495}, "metric": "fid50k_full", "total_time": 240.53874516487122, "total_time_str": "4m 01s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000900.pkl", "timestamp": 1694502203.1773508}
{"results": {"kid50k_full": 0.003050379662162177}, "metric": "kid50k_full", "total_time": 225.55694794654846, "total_time_str": "3m 46s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000900.pkl", "timestamp": 1694502428.7345843}
tick 226 kimg 904.0 time 9h 17m 12s sec/tick 113.5 sec/kimg 28.37 maintenance 470.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 227 kimg 908.0 time 9h 19m 06s sec/tick 113.3 sec/kimg 28.34 maintenance 0.1 cpumem 4.71 gpumem 4.96 augment 0.000
tick 228 kimg 912.0 time 9h 20m 59s sec/tick 113.6 sec/kimg 28.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 229 kimg 916.0 time 9h 22m 53s sec/tick 114.0 sec/kimg 28.49 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 230 kimg 920.0 time 9h 24m 51s sec/tick 117.2 sec/kimg 29.29 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 231 kimg 924.0 time 9h 26m 46s sec/tick 115.7 sec/kimg 28.92 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 232 kimg 928.0 time 9h 28m 42s sec/tick 115.8 sec/kimg 28.95 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 233 kimg 932.0 time 9h 30m 38s sec/tick 115.4 sec/kimg 28.86 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 234 kimg 936.0 time 9h 32m 34s sec/tick 115.8 sec/kimg 28.95 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 235 kimg 940.0 time 9h 34m 29s sec/tick 115.4 sec/kimg 28.85 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 236 kimg 944.0 time 9h 36m 25s sec/tick 115.8 sec/kimg 28.94 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 237 kimg 948.0 time 9h 38m 21s sec/tick 115.6 sec/kimg 28.90 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 238 kimg 952.0 time 9h 40m 16s sec/tick 115.8 sec/kimg 28.94 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 239 kimg 956.0 time 9h 42m 14s sec/tick 117.7 sec/kimg 29.41 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
tick 240 kimg 960.0 time 9h 44m 12s sec/tick 118.2 sec/kimg 29.55 maintenance 0.1 cpumem 4.71 gpumem 4.95 augment 0.000
Evaluating metrics...
{"results": {"fid50k_full": 4.960903881577067}, "metric": "fid50k_full", "total_time": 246.37715554237366, "total_time_str": "4m 06s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000960.pkl", "timestamp": 1694504412.9876099}
{"results": {"kid50k_full": 0.0029740898248248053}, "metric": "kid50k_full", "total_time": 229.59208965301514, "total_time_str": "3m 50s", "num_gpus": 1, "snapshot_pkl": "network-snapshot-000960.pkl", "timestamp": 1694504642.5799766}
tick 241 kimg 964.0 time 9h 54m 10s sec/tick 117.3 sec/kimg 29.32 maintenance 480.1 cpumem 4.71 gpumem 4.95 augment 0.000
Aborted!
|