File size: 13,304 Bytes
0dbdee8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
2022-05-07 18:26:05,034 INFO    MainThread:40892 [internal.py:wandb_internal():92] W&B internal server running at pid: 40892, started at: 2022-05-07 18:26:05.034362
2022-05-07 18:26:05,036 INFO    WriterThread:40892 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/run-jtmmbwaj.wandb
2022-05-07 18:26:05,037 DEBUG   SenderThread:40892 [sender.py:send():235] send: header
2022-05-07 18:26:05,037 DEBUG   SenderThread:40892 [sender.py:send():235] send: run
2022-05-07 18:26:05,113 INFO    SenderThread:40892 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files
2022-05-07 18:26:05,113 INFO    SenderThread:40892 [sender.py:_start_run_threads():812] run started: jtmmbwaj with start time 1651947964
2022-05-07 18:26:05,113 DEBUG   SenderThread:40892 [sender.py:send():235] send: summary
2022-05-07 18:26:05,113 INFO    SenderThread:40892 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:26:05,114 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: check_version
2022-05-07 18:26:05,114 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: check_version
2022-05-07 18:26:05,172 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: run_start
2022-05-07 18:26:06,153 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/wandb-summary.json
2022-05-07 18:26:07,258 DEBUG   HandlerThread:40892 [meta.py:__init__():37] meta init
2022-05-07 18:26:07,259 DEBUG   HandlerThread:40892 [meta.py:__init__():51] meta init done
2022-05-07 18:26:07,259 DEBUG   HandlerThread:40892 [meta.py:probe():211] probe
2022-05-07 18:26:07,260 DEBUG   HandlerThread:40892 [meta.py:_setup_git():201] setup git
2022-05-07 18:26:07,290 DEBUG   HandlerThread:40892 [meta.py:_setup_git():208] setup git done
2022-05-07 18:26:07,290 DEBUG   HandlerThread:40892 [meta.py:_save_pip():55] save pip
2022-05-07 18:26:07,290 DEBUG   HandlerThread:40892 [meta.py:_save_pip():69] save pip done
2022-05-07 18:26:07,290 DEBUG   HandlerThread:40892 [meta.py:probe():249] probe done
2022-05-07 18:26:07,293 DEBUG   SenderThread:40892 [sender.py:send():235] send: files
2022-05-07 18:26:07,294 INFO    SenderThread:40892 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-05-07 18:26:07,298 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:26:07,299 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:26:07,539 INFO    Thread-11 :40892 [upload_job.py:push():137] Uploaded file /tmp/tmp_3bt5snxwandb/2787p4kv-wandb-metadata.json
2022-05-07 18:26:08,173 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/wandb-metadata.json
2022-05-07 18:26:08,174 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/requirements.txt
2022-05-07 18:26:08,174 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:26:09,901 DEBUG   SenderThread:40892 [sender.py:send():235] send: telemetry
2022-05-07 18:26:10,174 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:26:12,175 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:26:22,179 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:26:24,180 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:26:24,924 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:26:24,924 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:26:26,181 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:26:35,339 DEBUG   SenderThread:40892 [sender.py:send():235] send: stats
2022-05-07 18:26:36,185 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/config.yaml
2022-05-07 18:26:39,965 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:26:39,965 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:26:48,189 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:26:54,998 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:26:54,998 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:27:05,417 DEBUG   SenderThread:40892 [sender.py:send():235] send: stats
2022-05-07 18:27:07,652 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:27:07,701 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:27:10,083 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:27:10,083 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:27:10,197 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:27:25,218 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:27:25,219 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:27:29,863 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:27:29,866 DEBUG   SenderThread:40892 [sender.py:send():235] send: history
2022-05-07 18:27:29,866 DEBUG   SenderThread:40892 [sender.py:send():235] send: summary
2022-05-07 18:27:29,869 INFO    SenderThread:40892 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:27:29,938 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:27:30,206 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/wandb-summary.json
2022-05-07 18:27:30,206 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:27:30,522 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:27:30,524 DEBUG   SenderThread:40892 [sender.py:send():235] send: history
2022-05-07 18:27:30,524 DEBUG   SenderThread:40892 [sender.py:send():235] send: summary
2022-05-07 18:27:30,525 INFO    SenderThread:40892 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:27:30,583 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:27:31,206 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/wandb-summary.json
2022-05-07 18:27:31,218 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:27:31,220 DEBUG   SenderThread:40892 [sender.py:send():235] send: history
2022-05-07 18:27:31,221 DEBUG   SenderThread:40892 [sender.py:send():235] send: summary
2022-05-07 18:27:31,222 INFO    SenderThread:40892 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:27:31,289 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:27:31,860 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:27:31,862 DEBUG   SenderThread:40892 [sender.py:send():235] send: history
2022-05-07 18:27:31,863 DEBUG   SenderThread:40892 [sender.py:send():235] send: summary
2022-05-07 18:27:31,863 INFO    SenderThread:40892 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:27:31,907 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:27:32,207 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/wandb-summary.json
2022-05-07 18:27:32,207 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:27:34,207 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:27:35,492 DEBUG   SenderThread:40892 [sender.py:send():235] send: stats
2022-05-07 18:27:40,250 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:27:40,250 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:27:55,291 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:27:55,292 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:27:59,218 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:28:05,566 DEBUG   SenderThread:40892 [sender.py:send():235] send: stats
2022-05-07 18:28:10,343 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:28:10,343 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:28:19,227 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:28:25,380 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:28:25,380 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:28:35,643 DEBUG   SenderThread:40892 [sender.py:send():235] send: stats
2022-05-07 18:28:40,421 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:28:40,421 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:28:40,432 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:28:40,579 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-07 18:28:40,582 DEBUG   SenderThread:40892 [sender.py:send_request():249] send_request: log_artifact
2022-05-07 18:28:40,708 DEBUG   HandlerThread:40892 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:28:41,013 INFO    Thread-13 :40892 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/03/1aa3918a194407ac466420b1ea5405
2022-05-07 18:28:41,236 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/output.log
2022-05-07 18:28:41,237 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/media/table/eval/step_0k_5_f901d5b4b0d1263be066.table.json
2022-05-07 18:28:41,237 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/media
2022-05-07 18:28:41,237 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/media/table/eval
2022-05-07 18:28:41,237 INFO    Thread-7  :40892 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182604-jtmmbwaj/files/media/table