File size: 17,229 Bytes
bcb2332
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
2022-03-03 00:42:16,953 INFO    MainThread:267379 [internal.py:wandb_internal():89] W&B internal server running at pid: 267379, started at: 2022-03-03 00:42:16.952980
2022-03-03 00:42:16,956 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: check_version
2022-03-03 00:42:16,956 INFO    WriterThread:267379 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/run-2ttcbe3s.wandb
2022-03-03 00:42:16,957 DEBUG   SenderThread:267379 [sender.py:send():235] send: header
2022-03-03 00:42:16,957 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: check_version
2022-03-03 00:42:17,024 DEBUG   SenderThread:267379 [sender.py:send():235] send: run
2022-03-03 00:42:17,144 INFO    SenderThread:267379 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files
2022-03-03 00:42:17,144 INFO    SenderThread:267379 [sender.py:_start_run_threads():809] run started: 2ttcbe3s with start time 1646268136
2022-03-03 00:42:17,145 DEBUG   SenderThread:267379 [sender.py:send():235] send: summary
2022-03-03 00:42:17,145 INFO    SenderThread:267379 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:42:17,145 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: run_start
2022-03-03 00:42:17,150 DEBUG   HandlerThread:267379 [meta.py:__init__():36] meta init
2022-03-03 00:42:17,150 DEBUG   HandlerThread:267379 [meta.py:__init__():50] meta init done
2022-03-03 00:42:17,150 DEBUG   HandlerThread:267379 [meta.py:probe():210] probe
2022-03-03 00:42:17,156 DEBUG   HandlerThread:267379 [meta.py:_setup_git():200] setup git
2022-03-03 00:42:17,172 DEBUG   HandlerThread:267379 [meta.py:_setup_git():207] setup git done
2022-03-03 00:42:17,172 DEBUG   HandlerThread:267379 [meta.py:_save_pip():54] save pip
2022-03-03 00:42:17,173 DEBUG   HandlerThread:267379 [meta.py:_save_pip():68] save pip done
2022-03-03 00:42:17,173 DEBUG   HandlerThread:267379 [meta.py:probe():248] probe done
2022-03-03 00:42:17,254 DEBUG   SenderThread:267379 [sender.py:send():235] send: files
2022-03-03 00:42:17,254 INFO    SenderThread:267379 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-03 00:42:17,259 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: stop_status
2022-03-03 00:42:17,259 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: stop_status
2022-03-03 00:42:17,295 DEBUG   SenderThread:267379 [sender.py:send():235] send: config
2022-03-03 00:42:17,296 DEBUG   SenderThread:267379 [sender.py:send():235] send: metric
2022-03-03 00:42:17,296 DEBUG   SenderThread:267379 [sender.py:send():235] send: metric
2022-03-03 00:42:17,296 WARNING SenderThread:267379 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-03 00:42:17,551 INFO    Thread-11 :267379 [upload_job.py:push():137] Uploaded file /tmp/tmpamllbtqswandb/2k16no24-wandb-metadata.json
2022-03-03 00:42:18,148 INFO    Thread-8  :267379 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-metadata.json
2022-03-03 00:42:18,148 INFO    Thread-8  :267379 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-summary.json
2022-03-03 00:42:18,148 INFO    Thread-8  :267379 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/requirements.txt
2022-03-03 00:42:18,148 INFO    Thread-8  :267379 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log
2022-03-03 00:42:20,146 INFO    Thread-8  :267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log
2022-03-03 00:42:24,147 INFO    Thread-8  :267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log
2022-03-03 00:42:26,115 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:26,115 DEBUG   SenderThread:267379 [sender.py:send():235] send: telemetry
2022-03-03 00:42:26,115 DEBUG   SenderThread:267379 [sender.py:send():235] send: exit
2022-03-03 00:42:26,116 INFO    SenderThread:267379 [sender.py:send_exit():371] handling exit code: 1
2022-03-03 00:42:26,116 INFO    SenderThread:267379 [sender.py:send_exit():373] handling runtime: 8
2022-03-03 00:42:26,116 INFO    SenderThread:267379 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:42:26,116 INFO    SenderThread:267379 [sender.py:send_exit():379] send defer
2022-03-03 00:42:26,116 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:26,117 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,117 INFO    HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-03 00:42:26,117 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,117 INFO    SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-03 00:42:26,117 INFO    SenderThread:267379 [sender.py:transition_state():392] send defer: 1
2022-03-03 00:42:26,118 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,118 INFO    HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-03 00:42:26,148 INFO    Thread-8  :267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-summary.json
2022-03-03 00:42:26,148 INFO    Thread-8  :267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log
2022-03-03 00:42:26,201 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,201 INFO    SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-03 00:42:26,201 INFO    SenderThread:267379 [sender.py:transition_state():392] send defer: 2
2022-03-03 00:42:26,202 DEBUG   SenderThread:267379 [sender.py:send():235] send: stats
2022-03-03 00:42:26,202 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,203 INFO    HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-03 00:42:26,203 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,203 INFO    SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-03 00:42:26,203 INFO    SenderThread:267379 [sender.py:transition_state():392] send defer: 3
2022-03-03 00:42:26,203 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,203 INFO    HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-03 00:42:26,203 DEBUG   SenderThread:267379 [sender.py:send():235] send: summary
2022-03-03 00:42:26,204 INFO    SenderThread:267379 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:42:26,204 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,204 INFO    SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-03 00:42:26,204 INFO    SenderThread:267379 [sender.py:transition_state():392] send defer: 4
2022-03-03 00:42:26,204 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,205 INFO    HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-03 00:42:26,205 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,205 INFO    SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-03 00:42:26,218 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:26,313 INFO    SenderThread:267379 [sender.py:transition_state():392] send defer: 5
2022-03-03 00:42:26,313 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:26,314 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,314 INFO    HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-03 00:42:26,314 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,315 INFO    SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-03 00:42:26,315 INFO    SenderThread:267379 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-03 00:42:26,415 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,148 INFO    Thread-8  :267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-summary.json
2022-03-03 00:42:27,149 INFO    SenderThread:267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/config.yaml
2022-03-03 00:42:27,149 INFO    SenderThread:267379 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files
2022-03-03 00:42:27,150 INFO    SenderThread:267379 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-metadata.json wandb-metadata.json
2022-03-03 00:42:27,150 INFO    SenderThread:267379 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log output.log
2022-03-03 00:42:27,150 INFO    SenderThread:267379 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-summary.json wandb-summary.json
2022-03-03 00:42:27,152 INFO    SenderThread:267379 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/requirements.txt requirements.txt
2022-03-03 00:42:27,156 INFO    SenderThread:267379 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/config.yaml config.yaml
2022-03-03 00:42:27,156 INFO    SenderThread:267379 [sender.py:transition_state():392] send defer: 6
2022-03-03 00:42:27,156 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,157 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:27,157 INFO    HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-03 00:42:27,160 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:27,160 INFO    SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-03 00:42:27,160 INFO    SenderThread:267379 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:42:27,258 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,258 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,360 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,360 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,434 INFO    Thread-12 :267379 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log
2022-03-03 00:42:27,457 INFO    Thread-13 :267379 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-summary.json
2022-03-03 00:42:27,460 INFO    Thread-15 :267379 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/config.yaml
2022-03-03 00:42:27,462 INFO    Thread-14 :267379 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/requirements.txt
2022-03-03 00:42:27,462 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,463 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,565 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,565 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,663 INFO    Thread-7  :267379 [sender.py:transition_state():392] send defer: 7
2022-03-03 00:42:27,664 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:27,664 INFO    HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-03 00:42:27,664 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:27,664 INFO    SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-03 00:42:27,667 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,834 INFO    SenderThread:267379 [sender.py:transition_state():392] send defer: 8
2022-03-03 00:42:27,834 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,835 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:27,835 INFO    HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-03 00:42:27,835 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:27,835 INFO    SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-03 00:42:27,835 INFO    SenderThread:267379 [sender.py:transition_state():392] send defer: 9
2022-03-03 00:42:27,836 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:27,836 INFO    HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-03 00:42:27,836 DEBUG   SenderThread:267379 [sender.py:send():235] send: final
2022-03-03 00:42:27,836 DEBUG   SenderThread:267379 [sender.py:send():235] send: footer
2022-03-03 00:42:27,836 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:27,836 INFO    SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-03 00:42:27,936 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,937 DEBUG   SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,937 INFO    SenderThread:267379 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:42:28,000 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: get_summary
2022-03-03 00:42:28,001 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-03 00:42:28,002 DEBUG   HandlerThread:267379 [handler.py:handle_request():131] handle_request: shutdown
2022-03-03 00:42:28,002 INFO    HandlerThread:267379 [handler.py:finish():739] shutting down handler
2022-03-03 00:42:28,836 INFO    WriterThread:267379 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/run-2ttcbe3s.wandb
2022-03-03 00:42:28,999 INFO    SenderThread:267379 [sender.py:finish():1075] shutting down sender
2022-03-03 00:42:28,999 INFO    SenderThread:267379 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:42:28,999 INFO    SenderThread:267379 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:42:29,001 INFO    MainThread:267379 [internal.py:handle_exit():79] Internal process exited