dat commited on
Commit
98e2f83
1 Parent(s): ae03b3f

Saving weights and logs of step 345000

Browse files
checkpoint_345000 ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1de39776ce2ff8428d43d2abbae945eec9972ec2d960d9832a29b7aaf21dd829
3
+ size 1530270447
events.out.tfevents.1626474829.t1v-n-f5c06ea1-w-0.798495.3.v2 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:9f438f7d8c0e2ed151acb2c86c475a3ade889ec4f0bd03b0e656627086442ae2
3
- size 2203542
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b4637283d9854edc2028a95251be73af67ba47a274e45dd5fae55816246cc19
3
+ size 2256182
flax_model.msgpack CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:959d6674371c3d3cf3a7a09f7d69cf73ebb948dc9fdfd80a687dbc6605a61b3c
3
  size 510090043
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:86a123ba86042aca0dbe525bd85d047b7cdfbece99dc4953a406373cb46110fb
3
  size 510090043
wandb/run-20210716_223350-8eukt20m/files/output.log CHANGED
@@ -2211,3 +2211,57 @@ Training...: 14604it [1:43:58, 1.06it/s]█████████████
2211
 
2212
 
2213
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2211
 
2212
 
2213
 
2214
+
2215
+ Training...: 14654it [1:44:18, 1.02s/it]█████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [01:17<00:00, 7.90it/s]
2216
+
2217
+
2218
+
2219
+
2220
+
2221
+
2222
+ Training...: 14705it [1:44:39, 1.32it/s]█████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [01:17<00:00, 7.90it/s]
2223
+
2224
+
2225
+
2226
+
2227
+
2228
+ Training...: 14749it [1:44:47, 5.94it/s]█████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [01:17<00:00, 7.90it/s]
2229
+
2230
+
2231
+
2232
+
2233
+
2234
+ Training...: 14799it [1:45:07, 4.42it/s]█████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [01:17<00:00, 7.90it/s]
2235
+
2236
+
2237
+
2238
+
2239
+
2240
+ Training...: 14849it [1:45:27, 5.71it/s]█████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [01:17<00:00, 7.90it/s]
2241
+
2242
+
2243
+
2244
+
2245
+
2246
+
2247
+ Training...: 14900it [1:45:58, 3.20s/it]█████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [01:17<00:00, 7.90it/s]
2248
+
2249
+
2250
+
2251
+
2252
+
2253
+ Training...: 14949it [1:46:08, 4.86it/s]█████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [01:17<00:00, 7.90it/s]
2254
+
2255
+
2256
+
2257
+
2258
+
2259
+
2260
+ Training...: 14999it [1:46:39, 5.85it/s]█████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [01:17<00:00, 7.90it/s]
2261
+
2262
+ tcmalloc: large alloc 1354776576 bytes == 0x378af0000 @ 0x7f2f3656a680 0x7f2f3658abdd 0x7f2f1cb2420d 0x7f2f1cb32340 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb2dbd3 0x7f2f1cb2e1fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x68cdc7 0x67e161 0x67e1df
2263
+ tcmalloc: large alloc 2715181056 bytes == 0x3c96f4000 @ 0x7f2f3656a680 0x7f2f3658abdd 0x7f2f1cb2420d 0x7f2f1cb32340 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb31e87 0x7f2f1cb2dbd3 0x7f2f1cb2e1fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a
2264
+ tcmalloc: large alloc 1530273792 bytes == 0x31f5ac000 @ 0x7f2f3656a680 0x7f2f3658b824 0x5f7b11 0x7f2f1cb2dc6f 0x7f2f1cb2e1fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x68cdc7 0x67e161 0x67e1df 0x67e281 0x67e627 0x6b6e62 0x6b71ed 0x7f2f3637f0b3 0x5f96de
2265
+ [00:26:31] - INFO - absl - Saved checkpoint at checkpoint_345000
2266
+ [00:26:32] - INFO - huggingface_hub.repository - git version 2.25.1
2267
+ git-lfs/2.9.2 (GitHub; linux amd64; go 1.13.5)
wandb/run-20210716_223350-8eukt20m/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"training_step": 344650, "learning_rate": 2.2834374249214306e-05, "train_loss": 1.9190285205841064, "_runtime": 6610, "_timestamp": 1626481440, "_step": 293, "eval_step": 340000, "eval_accuracy": 0.6408576965332031, "eval_loss": 1.8213207721710205}
1
+ {"training_step": 345000, "learning_rate": 2.2826880012871698e-05, "train_loss": 1.801241159439087, "_runtime": 6751, "_timestamp": 1626481581, "_step": 300, "eval_step": 340000, "eval_accuracy": 0.6408576965332031, "eval_loss": 1.8213207721710205}
wandb/run-20210716_223350-8eukt20m/logs/debug-internal.log CHANGED
@@ -4379,3 +4379,104 @@
4379
  2021-07-17 00:24:00,429 DEBUG SenderThread:799749 [sender.py:send():179] send: summary
4380
  2021-07-17 00:24:00,429 INFO SenderThread:799749 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
4381
  2021-07-17 00:24:00,868 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/wandb-summary.json
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4379
  2021-07-17 00:24:00,429 DEBUG SenderThread:799749 [sender.py:send():179] send: summary
4380
  2021-07-17 00:24:00,429 INFO SenderThread:799749 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
4381
  2021-07-17 00:24:00,868 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/wandb-summary.json
4382
+ 2021-07-17 00:24:01,868 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4383
+ 2021-07-17 00:24:02,869 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4384
+ 2021-07-17 00:24:04,870 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4385
+ 2021-07-17 00:24:06,871 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4386
+ 2021-07-17 00:24:07,066 DEBUG SenderThread:799749 [sender.py:send():179] send: stats
4387
+ 2021-07-17 00:24:08,872 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4388
+ 2021-07-17 00:24:10,873 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4389
+ 2021-07-17 00:24:12,874 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4390
+ 2021-07-17 00:24:13,985 DEBUG HandlerThread:799749 [handler.py:handle_request():124] handle_request: stop_status
4391
+ 2021-07-17 00:24:13,985 DEBUG SenderThread:799749 [sender.py:send_request():193] send_request: stop_status
4392
+ 2021-07-17 00:24:20,599 DEBUG SenderThread:799749 [sender.py:send():179] send: history
4393
+ 2021-07-17 00:24:20,600 DEBUG SenderThread:799749 [sender.py:send():179] send: summary
4394
+ 2021-07-17 00:24:20,600 INFO SenderThread:799749 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
4395
+ 2021-07-17 00:24:20,877 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/wandb-summary.json
4396
+ 2021-07-17 00:24:22,879 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4397
+ 2021-07-17 00:24:24,880 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4398
+ 2021-07-17 00:24:26,880 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4399
+ 2021-07-17 00:24:28,881 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4400
+ 2021-07-17 00:24:29,183 DEBUG HandlerThread:799749 [handler.py:handle_request():124] handle_request: stop_status
4401
+ 2021-07-17 00:24:29,183 DEBUG SenderThread:799749 [sender.py:send_request():193] send_request: stop_status
4402
+ 2021-07-17 00:24:30,882 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4403
+ 2021-07-17 00:24:32,883 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4404
+ 2021-07-17 00:24:37,143 DEBUG SenderThread:799749 [sender.py:send():179] send: stats
4405
+ 2021-07-17 00:24:40,821 DEBUG SenderThread:799749 [sender.py:send():179] send: history
4406
+ 2021-07-17 00:24:40,821 DEBUG SenderThread:799749 [sender.py:send():179] send: summary
4407
+ 2021-07-17 00:24:40,822 INFO SenderThread:799749 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
4408
+ 2021-07-17 00:24:40,887 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/wandb-summary.json
4409
+ 2021-07-17 00:24:42,888 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4410
+ 2021-07-17 00:24:44,336 DEBUG HandlerThread:799749 [handler.py:handle_request():124] handle_request: stop_status
4411
+ 2021-07-17 00:24:44,336 DEBUG SenderThread:799749 [sender.py:send_request():193] send_request: stop_status
4412
+ 2021-07-17 00:24:44,889 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4413
+ 2021-07-17 00:24:46,890 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4414
+ 2021-07-17 00:24:48,891 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4415
+ 2021-07-17 00:24:51,892 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4416
+ 2021-07-17 00:24:59,488 DEBUG HandlerThread:799749 [handler.py:handle_request():124] handle_request: stop_status
4417
+ 2021-07-17 00:24:59,488 DEBUG SenderThread:799749 [sender.py:send_request():193] send_request: stop_status
4418
+ 2021-07-17 00:25:00,959 DEBUG SenderThread:799749 [sender.py:send():179] send: history
4419
+ 2021-07-17 00:25:00,959 DEBUG SenderThread:799749 [sender.py:send():179] send: summary
4420
+ 2021-07-17 00:25:00,960 INFO SenderThread:799749 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
4421
+ 2021-07-17 00:25:01,897 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/wandb-summary.json
4422
+ 2021-07-17 00:25:02,897 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4423
+ 2021-07-17 00:25:03,898 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4424
+ 2021-07-17 00:25:05,898 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4425
+ 2021-07-17 00:25:07,266 DEBUG SenderThread:799749 [sender.py:send():179] send: stats
4426
+ 2021-07-17 00:25:07,900 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4427
+ 2021-07-17 00:25:09,901 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4428
+ 2021-07-17 00:25:11,902 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4429
+ 2021-07-17 00:25:14,617 DEBUG HandlerThread:799749 [handler.py:handle_request():124] handle_request: stop_status
4430
+ 2021-07-17 00:25:14,618 DEBUG SenderThread:799749 [sender.py:send_request():193] send_request: stop_status
4431
+ 2021-07-17 00:25:21,143 DEBUG SenderThread:799749 [sender.py:send():179] send: history
4432
+ 2021-07-17 00:25:21,143 DEBUG SenderThread:799749 [sender.py:send():179] send: summary
4433
+ 2021-07-17 00:25:21,144 INFO SenderThread:799749 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
4434
+ 2021-07-17 00:25:21,906 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/wandb-summary.json
4435
+ 2021-07-17 00:25:22,907 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4436
+ 2021-07-17 00:25:23,907 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4437
+ 2021-07-17 00:25:25,908 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4438
+ 2021-07-17 00:25:27,909 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4439
+ 2021-07-17 00:25:29,754 DEBUG HandlerThread:799749 [handler.py:handle_request():124] handle_request: stop_status
4440
+ 2021-07-17 00:25:29,755 DEBUG SenderThread:799749 [sender.py:send_request():193] send_request: stop_status
4441
+ 2021-07-17 00:25:29,910 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4442
+ 2021-07-17 00:25:31,911 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4443
+ 2021-07-17 00:25:37,293 DEBUG SenderThread:799749 [sender.py:send():179] send: stats
4444
+ 2021-07-17 00:25:41,370 DEBUG SenderThread:799749 [sender.py:send():179] send: history
4445
+ 2021-07-17 00:25:41,370 DEBUG SenderThread:799749 [sender.py:send():179] send: summary
4446
+ 2021-07-17 00:25:41,371 INFO SenderThread:799749 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
4447
+ 2021-07-17 00:25:41,915 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4448
+ 2021-07-17 00:25:41,915 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/wandb-summary.json
4449
+ 2021-07-17 00:25:43,916 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4450
+ 2021-07-17 00:25:44,947 DEBUG HandlerThread:799749 [handler.py:handle_request():124] handle_request: stop_status
4451
+ 2021-07-17 00:25:44,947 DEBUG SenderThread:799749 [sender.py:send_request():193] send_request: stop_status
4452
+ 2021-07-17 00:25:45,917 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4453
+ 2021-07-17 00:25:47,918 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4454
+ 2021-07-17 00:25:49,919 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4455
+ 2021-07-17 00:25:51,920 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4456
+ 2021-07-17 00:26:00,118 DEBUG HandlerThread:799749 [handler.py:handle_request():124] handle_request: stop_status
4457
+ 2021-07-17 00:26:00,119 DEBUG SenderThread:799749 [sender.py:send_request():193] send_request: stop_status
4458
+ 2021-07-17 00:26:01,544 DEBUG SenderThread:799749 [sender.py:send():179] send: history
4459
+ 2021-07-17 00:26:01,545 DEBUG SenderThread:799749 [sender.py:send():179] send: summary
4460
+ 2021-07-17 00:26:01,545 INFO SenderThread:799749 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
4461
+ 2021-07-17 00:26:01,924 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/wandb-summary.json
4462
+ 2021-07-17 00:26:03,925 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4463
+ 2021-07-17 00:26:06,926 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4464
+ 2021-07-17 00:26:07,366 DEBUG SenderThread:799749 [sender.py:send():179] send: stats
4465
+ 2021-07-17 00:26:08,927 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4466
+ 2021-07-17 00:26:10,928 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4467
+ 2021-07-17 00:26:12,929 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4468
+ 2021-07-17 00:26:15,255 DEBUG HandlerThread:799749 [handler.py:handle_request():124] handle_request: stop_status
4469
+ 2021-07-17 00:26:15,256 DEBUG SenderThread:799749 [sender.py:send_request():193] send_request: stop_status
4470
+ 2021-07-17 00:26:21,753 DEBUG SenderThread:799749 [sender.py:send():179] send: history
4471
+ 2021-07-17 00:26:21,754 DEBUG SenderThread:799749 [sender.py:send():179] send: summary
4472
+ 2021-07-17 00:26:21,754 INFO SenderThread:799749 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
4473
+ 2021-07-17 00:26:21,934 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/wandb-summary.json
4474
+ 2021-07-17 00:26:22,934 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4475
+ 2021-07-17 00:26:23,935 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4476
+ 2021-07-17 00:26:28,937 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4477
+ 2021-07-17 00:26:31,249 DEBUG HandlerThread:799749 [handler.py:handle_request():124] handle_request: stop_status
4478
+ 2021-07-17 00:26:31,250 DEBUG SenderThread:799749 [sender.py:send_request():193] send_request: stop_status
4479
+ 2021-07-17 00:26:31,938 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4480
+ 2021-07-17 00:26:33,939 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4481
+ 2021-07-17 00:26:35,940 INFO Thread-8 :799749 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210716_223350-8eukt20m/files/output.log
4482
+ 2021-07-17 00:26:37,435 DEBUG SenderThread:799749 [sender.py:send():179] send: stats
wandb/run-20210716_223350-8eukt20m/run-8eukt20m.wandb CHANGED
Binary files a/wandb/run-20210716_223350-8eukt20m/run-8eukt20m.wandb and b/wandb/run-20210716_223350-8eukt20m/run-8eukt20m.wandb differ