File size: 16,996 Bytes
53a9ef4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
2022-03-01 17:10:33,078 INFO    MainThread:245236 [internal.py:wandb_internal():89] W&B internal server running at pid: 245236, started at: 2022-03-01 17:10:33.078632
2022-03-01 17:10:33,080 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 17:10:33,081 INFO    WriterThread:245236 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/run-6le9s370.wandb
2022-03-01 17:10:33,082 DEBUG   SenderThread:245236 [sender.py:send():235] send: header
2022-03-01 17:10:33,082 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: check_version
2022-03-01 17:10:33,155 DEBUG   SenderThread:245236 [sender.py:send():235] send: run
2022-03-01 17:10:33,265 INFO    SenderThread:245236 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files
2022-03-01 17:10:33,265 INFO    SenderThread:245236 [sender.py:_start_run_threads():809] run started: 6le9s370 with start time 1646154632
2022-03-01 17:10:33,265 DEBUG   SenderThread:245236 [sender.py:send():235] send: summary
2022-03-01 17:10:33,265 INFO    SenderThread:245236 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:10:33,266 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 17:10:33,270 DEBUG   HandlerThread:245236 [meta.py:__init__():36] meta init
2022-03-01 17:10:33,270 DEBUG   HandlerThread:245236 [meta.py:__init__():50] meta init done
2022-03-01 17:10:33,270 DEBUG   HandlerThread:245236 [meta.py:probe():210] probe
2022-03-01 17:10:33,276 DEBUG   HandlerThread:245236 [meta.py:_setup_git():200] setup git
2022-03-01 17:10:33,290 DEBUG   HandlerThread:245236 [meta.py:_setup_git():207] setup git done
2022-03-01 17:10:33,290 DEBUG   HandlerThread:245236 [meta.py:_save_pip():54] save pip
2022-03-01 17:10:33,291 DEBUG   HandlerThread:245236 [meta.py:_save_pip():68] save pip done
2022-03-01 17:10:33,291 DEBUG   HandlerThread:245236 [meta.py:probe():248] probe done
2022-03-01 17:10:33,365 DEBUG   SenderThread:245236 [sender.py:send():235] send: files
2022-03-01 17:10:33,365 INFO    SenderThread:245236 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 17:10:33,370 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 17:10:33,370 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: stop_status
2022-03-01 17:10:33,410 DEBUG   SenderThread:245236 [sender.py:send():235] send: config
2022-03-01 17:10:33,411 DEBUG   SenderThread:245236 [sender.py:send():235] send: metric
2022-03-01 17:10:33,411 DEBUG   SenderThread:245236 [sender.py:send():235] send: metric
2022-03-01 17:10:33,411 WARNING SenderThread:245236 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 17:10:33,654 INFO    Thread-11 :245236 [upload_job.py:push():137] Uploaded file /tmp/tmpimoy9n47wandb/3840sgse-wandb-metadata.json
2022-03-01 17:10:34,267 INFO    Thread-8  :245236 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-summary.json
2022-03-01 17:10:34,267 INFO    Thread-8  :245236 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/requirements.txt
2022-03-01 17:10:34,268 INFO    Thread-8  :245236 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log
2022-03-01 17:10:34,268 INFO    Thread-8  :245236 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-metadata.json
2022-03-01 17:10:36,266 INFO    Thread-8  :245236 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log
2022-03-01 17:10:40,268 INFO    Thread-8  :245236 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log
2022-03-01 17:10:42,126 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:42,126 DEBUG   SenderThread:245236 [sender.py:send():235] send: telemetry
2022-03-01 17:10:42,127 DEBUG   SenderThread:245236 [sender.py:send():235] send: exit
2022-03-01 17:10:42,127 INFO    SenderThread:245236 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 17:10:42,127 INFO    SenderThread:245236 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 17:10:42,127 INFO    SenderThread:245236 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:10:42,127 INFO    SenderThread:245236 [sender.py:send_exit():379] send defer
2022-03-01 17:10:42,127 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:42,128 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,128 INFO    HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 17:10:42,128 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,128 INFO    SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 17:10:42,128 INFO    SenderThread:245236 [sender.py:transition_state():392] send defer: 1
2022-03-01 17:10:42,128 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,128 INFO    HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 17:10:42,190 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,190 INFO    SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 17:10:42,190 INFO    SenderThread:245236 [sender.py:transition_state():392] send defer: 2
2022-03-01 17:10:42,190 DEBUG   SenderThread:245236 [sender.py:send():235] send: stats
2022-03-01 17:10:42,191 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,191 INFO    HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 17:10:42,191 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,191 INFO    SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 17:10:42,191 INFO    SenderThread:245236 [sender.py:transition_state():392] send defer: 3
2022-03-01 17:10:42,191 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,191 INFO    HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 17:10:42,191 DEBUG   SenderThread:245236 [sender.py:send():235] send: summary
2022-03-01 17:10:42,192 INFO    SenderThread:245236 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:10:42,192 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,192 INFO    SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 17:10:42,192 INFO    SenderThread:245236 [sender.py:transition_state():392] send defer: 4
2022-03-01 17:10:42,192 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,192 INFO    HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 17:10:42,192 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,193 INFO    SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 17:10:42,229 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:42,301 INFO    Thread-8  :245236 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-summary.json
2022-03-01 17:10:42,301 INFO    Thread-8  :245236 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log
2022-03-01 17:10:42,302 INFO    SenderThread:245236 [sender.py:transition_state():392] send defer: 5
2022-03-01 17:10:42,302 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:42,302 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,302 INFO    HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 17:10:42,303 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,303 INFO    SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 17:10:42,303 INFO    SenderThread:245236 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 17:10:42,403 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,302 INFO    Thread-8  :245236 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/config.yaml
2022-03-01 17:10:43,302 INFO    SenderThread:245236 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files
2022-03-01 17:10:43,303 INFO    SenderThread:245236 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-metadata.json wandb-metadata.json
2022-03-01 17:10:43,303 INFO    SenderThread:245236 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log output.log
2022-03-01 17:10:43,303 INFO    SenderThread:245236 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-summary.json wandb-summary.json
2022-03-01 17:10:43,303 INFO    SenderThread:245236 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/requirements.txt requirements.txt
2022-03-01 17:10:43,303 INFO    SenderThread:245236 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/config.yaml config.yaml
2022-03-01 17:10:43,312 INFO    SenderThread:245236 [sender.py:transition_state():392] send defer: 6
2022-03-01 17:10:43,312 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,316 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:43,316 INFO    HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 17:10:43,316 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:43,316 INFO    SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 17:10:43,317 INFO    SenderThread:245236 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:10:43,414 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,414 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,516 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,516 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,561 INFO    Thread-12 :245236 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log
2022-03-01 17:10:43,580 INFO    Thread-13 :245236 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-summary.json
2022-03-01 17:10:43,581 INFO    Thread-14 :245236 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/requirements.txt
2022-03-01 17:10:43,598 INFO    Thread-15 :245236 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/config.yaml
2022-03-01 17:10:43,617 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,617 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,719 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,719 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,799 INFO    Thread-7  :245236 [sender.py:transition_state():392] send defer: 7
2022-03-01 17:10:43,799 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:43,799 INFO    HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 17:10:43,799 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:43,799 INFO    SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 17:10:43,820 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,839 INFO    SenderThread:245236 [sender.py:transition_state():392] send defer: 8
2022-03-01 17:10:43,840 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,840 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:43,840 INFO    HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 17:10:43,840 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:43,840 INFO    SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 17:10:43,841 INFO    SenderThread:245236 [sender.py:transition_state():392] send defer: 9
2022-03-01 17:10:43,841 DEBUG   SenderThread:245236 [sender.py:send():235] send: final
2022-03-01 17:10:43,841 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:43,842 DEBUG   SenderThread:245236 [sender.py:send():235] send: footer
2022-03-01 17:10:43,842 INFO    HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 17:10:43,842 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:43,842 INFO    SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 17:10:43,941 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,942 DEBUG   SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,942 INFO    SenderThread:245236 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:10:43,995 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 17:10:43,996 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 17:10:43,997 DEBUG   HandlerThread:245236 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 17:10:43,997 INFO    HandlerThread:245236 [handler.py:finish():739] shutting down handler
2022-03-01 17:10:44,842 INFO    WriterThread:245236 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/run-6le9s370.wandb
2022-03-01 17:10:44,994 INFO    SenderThread:245236 [sender.py:finish():1075] shutting down sender
2022-03-01 17:10:44,995 INFO    SenderThread:245236 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:10:44,995 INFO    SenderThread:245236 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:10:44,997 INFO    MainThread:245236 [internal.py:handle_exit():79] Internal process exited