File size: 17,000 Bytes
fc53a59
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
2022-03-02 08:48:34,041 INFO    MainThread:254137 [internal.py:wandb_internal():89] W&B internal server running at pid: 254137, started at: 2022-03-02 08:48:34.041581
2022-03-02 08:48:34,043 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: check_version
2022-03-02 08:48:34,044 INFO    WriterThread:254137 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/run-2m6jtwtj.wandb
2022-03-02 08:48:34,045 DEBUG   SenderThread:254137 [sender.py:send():235] send: header
2022-03-02 08:48:34,046 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: check_version
2022-03-02 08:48:34,116 DEBUG   SenderThread:254137 [sender.py:send():235] send: run
2022-03-02 08:48:34,222 INFO    SenderThread:254137 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files
2022-03-02 08:48:34,222 INFO    SenderThread:254137 [sender.py:_start_run_threads():809] run started: 2m6jtwtj with start time 1646210913
2022-03-02 08:48:34,222 DEBUG   SenderThread:254137 [sender.py:send():235] send: summary
2022-03-02 08:48:34,222 INFO    SenderThread:254137 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:48:34,223 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: run_start
2022-03-02 08:48:34,227 DEBUG   HandlerThread:254137 [meta.py:__init__():36] meta init
2022-03-02 08:48:34,227 DEBUG   HandlerThread:254137 [meta.py:__init__():50] meta init done
2022-03-02 08:48:34,227 DEBUG   HandlerThread:254137 [meta.py:probe():210] probe
2022-03-02 08:48:34,234 DEBUG   HandlerThread:254137 [meta.py:_setup_git():200] setup git
2022-03-02 08:48:34,248 DEBUG   HandlerThread:254137 [meta.py:_setup_git():207] setup git done
2022-03-02 08:48:34,248 DEBUG   HandlerThread:254137 [meta.py:_save_pip():54] save pip
2022-03-02 08:48:34,249 DEBUG   HandlerThread:254137 [meta.py:_save_pip():68] save pip done
2022-03-02 08:48:34,249 DEBUG   HandlerThread:254137 [meta.py:probe():248] probe done
2022-03-02 08:48:34,326 DEBUG   SenderThread:254137 [sender.py:send():235] send: files
2022-03-02 08:48:34,327 INFO    SenderThread:254137 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-02 08:48:34,331 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: stop_status
2022-03-02 08:48:34,331 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: stop_status
2022-03-02 08:48:34,417 DEBUG   SenderThread:254137 [sender.py:send():235] send: config
2022-03-02 08:48:34,418 DEBUG   SenderThread:254137 [sender.py:send():235] send: metric
2022-03-02 08:48:34,418 DEBUG   SenderThread:254137 [sender.py:send():235] send: metric
2022-03-02 08:48:34,418 WARNING SenderThread:254137 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-02 08:48:34,653 INFO    Thread-11 :254137 [upload_job.py:push():137] Uploaded file /tmp/tmpvy5l0hy6wandb/q5wa9252-wandb-metadata.json
2022-03-02 08:48:35,224 INFO    Thread-8  :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/requirements.txt
2022-03-02 08:48:35,224 INFO    Thread-8  :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json
2022-03-02 08:48:35,224 INFO    Thread-8  :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log
2022-03-02 08:48:35,224 INFO    Thread-8  :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-metadata.json
2022-03-02 08:48:37,223 INFO    Thread-8  :254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log
2022-03-02 08:48:41,224 INFO    Thread-8  :254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log
2022-03-02 08:48:42,671 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:42,671 DEBUG   SenderThread:254137 [sender.py:send():235] send: telemetry
2022-03-02 08:48:42,671 DEBUG   SenderThread:254137 [sender.py:send():235] send: exit
2022-03-02 08:48:42,671 INFO    SenderThread:254137 [sender.py:send_exit():371] handling exit code: 1
2022-03-02 08:48:42,671 INFO    SenderThread:254137 [sender.py:send_exit():373] handling runtime: 8
2022-03-02 08:48:42,672 INFO    SenderThread:254137 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:48:42,672 INFO    SenderThread:254137 [sender.py:send_exit():379] send defer
2022-03-02 08:48:42,672 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:42,672 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,673 INFO    HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-02 08:48:42,673 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,673 INFO    SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-02 08:48:42,673 INFO    SenderThread:254137 [sender.py:transition_state():392] send defer: 1
2022-03-02 08:48:42,673 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,673 INFO    HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-02 08:48:42,753 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,753 INFO    SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-02 08:48:42,753 INFO    SenderThread:254137 [sender.py:transition_state():392] send defer: 2
2022-03-02 08:48:42,754 DEBUG   SenderThread:254137 [sender.py:send():235] send: stats
2022-03-02 08:48:42,754 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,754 INFO    HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-02 08:48:42,755 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,755 INFO    SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-02 08:48:42,755 INFO    SenderThread:254137 [sender.py:transition_state():392] send defer: 3
2022-03-02 08:48:42,755 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,755 INFO    HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-02 08:48:42,755 DEBUG   SenderThread:254137 [sender.py:send():235] send: summary
2022-03-02 08:48:42,756 INFO    SenderThread:254137 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:48:42,756 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,756 INFO    SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-02 08:48:42,756 INFO    SenderThread:254137 [sender.py:transition_state():392] send defer: 4
2022-03-02 08:48:42,756 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,756 INFO    HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-02 08:48:42,756 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,756 INFO    SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-02 08:48:42,774 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:42,847 INFO    SenderThread:254137 [sender.py:transition_state():392] send defer: 5
2022-03-02 08:48:42,848 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:42,848 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,848 INFO    HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-02 08:48:42,848 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,849 INFO    SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-02 08:48:42,849 INFO    SenderThread:254137 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-02 08:48:42,949 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,225 INFO    Thread-8  :254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json
2022-03-02 08:48:43,226 INFO    SenderThread:254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log
2022-03-02 08:48:43,227 INFO    SenderThread:254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/config.yaml
2022-03-02 08:48:43,227 INFO    SenderThread:254137 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files
2022-03-02 08:48:43,227 INFO    SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-metadata.json wandb-metadata.json
2022-03-02 08:48:43,227 INFO    SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log output.log
2022-03-02 08:48:43,227 INFO    SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json wandb-summary.json
2022-03-02 08:48:43,227 INFO    SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/requirements.txt requirements.txt
2022-03-02 08:48:43,228 INFO    SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/config.yaml config.yaml
2022-03-02 08:48:43,230 INFO    SenderThread:254137 [sender.py:transition_state():392] send defer: 6
2022-03-02 08:48:43,231 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,231 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:43,231 INFO    HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-02 08:48:43,234 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:43,234 INFO    SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-02 08:48:43,234 INFO    SenderThread:254137 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:48:43,339 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,339 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,440 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,441 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,497 INFO    Thread-13 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json
2022-03-02 08:48:43,498 INFO    Thread-14 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/requirements.txt
2022-03-02 08:48:43,502 INFO    Thread-12 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log
2022-03-02 08:48:43,533 INFO    Thread-15 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/config.yaml
2022-03-02 08:48:43,542 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,542 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,644 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,644 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,733 INFO    Thread-7  :254137 [sender.py:transition_state():392] send defer: 7
2022-03-02 08:48:43,734 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:43,734 INFO    HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-02 08:48:43,734 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:43,734 INFO    SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-02 08:48:43,745 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,905 INFO    SenderThread:254137 [sender.py:transition_state():392] send defer: 8
2022-03-02 08:48:43,905 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,906 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:43,906 INFO    HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-02 08:48:43,906 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:43,906 INFO    SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-02 08:48:43,906 INFO    SenderThread:254137 [sender.py:transition_state():392] send defer: 9
2022-03-02 08:48:43,907 DEBUG   SenderThread:254137 [sender.py:send():235] send: final
2022-03-02 08:48:43,907 DEBUG   SenderThread:254137 [sender.py:send():235] send: footer
2022-03-02 08:48:43,907 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:43,907 INFO    HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-02 08:48:43,907 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:43,908 INFO    SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-02 08:48:44,007 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:44,007 DEBUG   SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:44,007 INFO    SenderThread:254137 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:48:44,067 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: get_summary
2022-03-02 08:48:44,068 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-02 08:48:44,068 DEBUG   HandlerThread:254137 [handler.py:handle_request():131] handle_request: shutdown
2022-03-02 08:48:44,068 INFO    HandlerThread:254137 [handler.py:finish():739] shutting down handler
2022-03-02 08:48:44,907 INFO    WriterThread:254137 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/run-2m6jtwtj.wandb
2022-03-02 08:48:45,066 INFO    SenderThread:254137 [sender.py:finish():1075] shutting down sender
2022-03-02 08:48:45,066 INFO    SenderThread:254137 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:48:45,066 INFO    SenderThread:254137 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:48:45,068 INFO    MainThread:254137 [internal.py:handle_exit():79] Internal process exited