File size: 17,440 Bytes
fc53a59
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
2022-03-02 08:50:01,203 INFO    MainThread:254314 [internal.py:wandb_internal():89] W&B internal server running at pid: 254314, started at: 2022-03-02 08:50:01.203327
2022-03-02 08:50:01,205 INFO    WriterThread:254314 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/run-3jbwcde6.wandb
2022-03-02 08:50:01,206 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: check_version
2022-03-02 08:50:01,207 DEBUG   SenderThread:254314 [sender.py:send():235] send: header
2022-03-02 08:50:01,207 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: check_version
2022-03-02 08:50:01,273 DEBUG   SenderThread:254314 [sender.py:send():235] send: run
2022-03-02 08:50:01,361 INFO    SenderThread:254314 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files
2022-03-02 08:50:01,361 INFO    SenderThread:254314 [sender.py:_start_run_threads():809] run started: 3jbwcde6 with start time 1646211000
2022-03-02 08:50:01,361 DEBUG   SenderThread:254314 [sender.py:send():235] send: summary
2022-03-02 08:50:01,361 INFO    SenderThread:254314 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:50:01,362 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: run_start
2022-03-02 08:50:01,366 DEBUG   HandlerThread:254314 [meta.py:__init__():36] meta init
2022-03-02 08:50:01,367 DEBUG   HandlerThread:254314 [meta.py:__init__():50] meta init done
2022-03-02 08:50:01,367 DEBUG   HandlerThread:254314 [meta.py:probe():210] probe
2022-03-02 08:50:01,373 DEBUG   HandlerThread:254314 [meta.py:_setup_git():200] setup git
2022-03-02 08:50:01,387 DEBUG   HandlerThread:254314 [meta.py:_setup_git():207] setup git done
2022-03-02 08:50:01,388 DEBUG   HandlerThread:254314 [meta.py:_save_pip():54] save pip
2022-03-02 08:50:01,388 DEBUG   HandlerThread:254314 [meta.py:_save_pip():68] save pip done
2022-03-02 08:50:01,388 DEBUG   HandlerThread:254314 [meta.py:probe():248] probe done
2022-03-02 08:50:01,468 DEBUG   SenderThread:254314 [sender.py:send():235] send: files
2022-03-02 08:50:01,468 INFO    SenderThread:254314 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-02 08:50:01,473 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: stop_status
2022-03-02 08:50:01,473 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: stop_status
2022-03-02 08:50:01,520 DEBUG   SenderThread:254314 [sender.py:send():235] send: config
2022-03-02 08:50:01,521 DEBUG   SenderThread:254314 [sender.py:send():235] send: metric
2022-03-02 08:50:01,522 DEBUG   SenderThread:254314 [sender.py:send():235] send: metric
2022-03-02 08:50:01,522 WARNING SenderThread:254314 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-02 08:50:01,741 INFO    Thread-11 :254314 [upload_job.py:push():137] Uploaded file /tmp/tmpfcch1qofwandb/3oinsdv0-wandb-metadata.json
2022-03-02 08:50:02,363 INFO    Thread-8  :254314 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-metadata.json
2022-03-02 08:50:02,363 INFO    Thread-8  :254314 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-summary.json
2022-03-02 08:50:02,363 INFO    Thread-8  :254314 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log
2022-03-02 08:50:02,363 INFO    Thread-8  :254314 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/requirements.txt
2022-03-02 08:50:04,362 INFO    Thread-8  :254314 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log
2022-03-02 08:50:08,364 INFO    Thread-8  :254314 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log
2022-03-02 08:50:09,886 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:09,886 DEBUG   SenderThread:254314 [sender.py:send():235] send: telemetry
2022-03-02 08:50:09,887 DEBUG   SenderThread:254314 [sender.py:send():235] send: exit
2022-03-02 08:50:09,887 INFO    SenderThread:254314 [sender.py:send_exit():371] handling exit code: 1
2022-03-02 08:50:09,887 INFO    SenderThread:254314 [sender.py:send_exit():373] handling runtime: 8
2022-03-02 08:50:09,887 INFO    SenderThread:254314 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:50:09,887 INFO    SenderThread:254314 [sender.py:send_exit():379] send defer
2022-03-02 08:50:09,888 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:09,888 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:09,888 INFO    HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-02 08:50:09,888 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:09,888 INFO    SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-02 08:50:09,889 INFO    SenderThread:254314 [sender.py:transition_state():392] send defer: 1
2022-03-02 08:50:09,889 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:09,889 INFO    HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-02 08:50:09,992 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:09,992 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:09,992 INFO    SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-02 08:50:09,993 INFO    SenderThread:254314 [sender.py:transition_state():392] send defer: 2
2022-03-02 08:50:09,993 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:09,993 DEBUG   SenderThread:254314 [sender.py:send():235] send: stats
2022-03-02 08:50:09,994 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:09,994 INFO    HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-02 08:50:09,994 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:09,995 INFO    SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-02 08:50:09,995 INFO    SenderThread:254314 [sender.py:transition_state():392] send defer: 3
2022-03-02 08:50:09,995 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:09,995 INFO    HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-02 08:50:09,995 DEBUG   SenderThread:254314 [sender.py:send():235] send: summary
2022-03-02 08:50:09,995 INFO    SenderThread:254314 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:50:09,996 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:09,996 INFO    SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-02 08:50:09,996 INFO    SenderThread:254314 [sender.py:transition_state():392] send defer: 4
2022-03-02 08:50:09,996 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:09,996 INFO    HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-02 08:50:09,996 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:09,996 INFO    SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-02 08:50:10,100 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,126 INFO    SenderThread:254314 [sender.py:transition_state():392] send defer: 5
2022-03-02 08:50:10,126 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,127 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:10,127 INFO    HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-02 08:50:10,127 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:10,127 INFO    SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-02 08:50:10,127 INFO    SenderThread:254314 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-02 08:50:10,228 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,364 INFO    Thread-8  :254314 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-summary.json
2022-03-02 08:50:10,365 INFO    SenderThread:254314 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log
2022-03-02 08:50:10,366 INFO    SenderThread:254314 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/config.yaml
2022-03-02 08:50:10,366 INFO    SenderThread:254314 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files
2022-03-02 08:50:10,366 INFO    SenderThread:254314 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-metadata.json wandb-metadata.json
2022-03-02 08:50:10,366 INFO    SenderThread:254314 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log output.log
2022-03-02 08:50:10,366 INFO    SenderThread:254314 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-summary.json wandb-summary.json
2022-03-02 08:50:10,367 INFO    SenderThread:254314 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/requirements.txt requirements.txt
2022-03-02 08:50:10,372 INFO    SenderThread:254314 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/config.yaml config.yaml
2022-03-02 08:50:10,375 INFO    SenderThread:254314 [sender.py:transition_state():392] send defer: 6
2022-03-02 08:50:10,376 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,376 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:10,377 INFO    HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-02 08:50:10,379 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:10,379 INFO    SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-02 08:50:10,379 INFO    SenderThread:254314 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:50:10,477 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,478 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,579 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,579 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,641 INFO    Thread-12 :254314 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log
2022-03-02 08:50:10,642 INFO    Thread-14 :254314 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/requirements.txt
2022-03-02 08:50:10,656 INFO    Thread-15 :254314 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/config.yaml
2022-03-02 08:50:10,681 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,681 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,700 INFO    Thread-13 :254314 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-summary.json
2022-03-02 08:50:10,782 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,782 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,884 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,884 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,901 INFO    Thread-7  :254314 [sender.py:transition_state():392] send defer: 7
2022-03-02 08:50:10,901 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:10,901 INFO    HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-02 08:50:10,902 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:10,902 INFO    SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-02 08:50:10,985 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:11,046 INFO    SenderThread:254314 [sender.py:transition_state():392] send defer: 8
2022-03-02 08:50:11,047 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:11,047 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:11,047 INFO    HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-02 08:50:11,047 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:11,047 INFO    SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-02 08:50:11,047 INFO    SenderThread:254314 [sender.py:transition_state():392] send defer: 9
2022-03-02 08:50:11,048 DEBUG   SenderThread:254314 [sender.py:send():235] send: final
2022-03-02 08:50:11,048 DEBUG   SenderThread:254314 [sender.py:send():235] send: footer
2022-03-02 08:50:11,048 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:11,048 INFO    HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-02 08:50:11,049 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:11,049 INFO    SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-02 08:50:11,148 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:11,148 DEBUG   SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:11,148 INFO    SenderThread:254314 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:50:11,233 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: get_summary
2022-03-02 08:50:11,233 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-02 08:50:11,234 DEBUG   HandlerThread:254314 [handler.py:handle_request():131] handle_request: shutdown
2022-03-02 08:50:11,234 INFO    HandlerThread:254314 [handler.py:finish():739] shutting down handler
2022-03-02 08:50:12,049 INFO    WriterThread:254314 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/run-3jbwcde6.wandb
2022-03-02 08:50:12,232 INFO    SenderThread:254314 [sender.py:finish():1075] shutting down sender
2022-03-02 08:50:12,232 INFO    SenderThread:254314 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:50:12,232 INFO    SenderThread:254314 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:50:12,234 INFO    MainThread:254314 [internal.py:handle_exit():79] Internal process exited