File size: 17,000 Bytes
53a9ef4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
2022-03-01 17:13:29,716 INFO    MainThread:245590 [internal.py:wandb_internal():89] W&B internal server running at pid: 245590, started at: 2022-03-01 17:13:29.715995
2022-03-01 17:13:29,718 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 17:13:29,718 INFO    WriterThread:245590 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/run-194tzn4x.wandb
2022-03-01 17:13:29,720 DEBUG   SenderThread:245590 [sender.py:send():235] send: header
2022-03-01 17:13:29,720 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: check_version
2022-03-01 17:13:29,786 DEBUG   SenderThread:245590 [sender.py:send():235] send: run
2022-03-01 17:13:29,877 INFO    SenderThread:245590 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files
2022-03-01 17:13:29,877 INFO    SenderThread:245590 [sender.py:_start_run_threads():809] run started: 194tzn4x with start time 1646154808
2022-03-01 17:13:29,877 DEBUG   SenderThread:245590 [sender.py:send():235] send: summary
2022-03-01 17:13:29,877 INFO    SenderThread:245590 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:13:29,878 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 17:13:29,882 DEBUG   HandlerThread:245590 [meta.py:__init__():36] meta init
2022-03-01 17:13:29,882 DEBUG   HandlerThread:245590 [meta.py:__init__():50] meta init done
2022-03-01 17:13:29,882 DEBUG   HandlerThread:245590 [meta.py:probe():210] probe
2022-03-01 17:13:29,889 DEBUG   HandlerThread:245590 [meta.py:_setup_git():200] setup git
2022-03-01 17:13:29,904 DEBUG   HandlerThread:245590 [meta.py:_setup_git():207] setup git done
2022-03-01 17:13:29,904 DEBUG   HandlerThread:245590 [meta.py:_save_pip():54] save pip
2022-03-01 17:13:29,904 DEBUG   HandlerThread:245590 [meta.py:_save_pip():68] save pip done
2022-03-01 17:13:29,905 DEBUG   HandlerThread:245590 [meta.py:probe():248] probe done
2022-03-01 17:13:29,980 DEBUG   SenderThread:245590 [sender.py:send():235] send: files
2022-03-01 17:13:29,980 INFO    SenderThread:245590 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 17:13:29,985 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 17:13:29,985 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: stop_status
2022-03-01 17:13:30,015 DEBUG   SenderThread:245590 [sender.py:send():235] send: config
2022-03-01 17:13:30,016 DEBUG   SenderThread:245590 [sender.py:send():235] send: metric
2022-03-01 17:13:30,016 DEBUG   SenderThread:245590 [sender.py:send():235] send: metric
2022-03-01 17:13:30,016 WARNING SenderThread:245590 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 17:13:30,247 INFO    Thread-11 :245590 [upload_job.py:push():137] Uploaded file /tmp/tmpqzj81002wandb/2csiwq7p-wandb-metadata.json
2022-03-01 17:13:30,878 INFO    Thread-8  :245590 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-summary.json
2022-03-01 17:13:30,879 INFO    Thread-8  :245590 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/requirements.txt
2022-03-01 17:13:30,879 INFO    Thread-8  :245590 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-metadata.json
2022-03-01 17:13:30,879 INFO    Thread-8  :245590 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log
2022-03-01 17:13:32,877 INFO    Thread-8  :245590 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log
2022-03-01 17:13:36,879 INFO    Thread-8  :245590 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log
2022-03-01 17:13:38,372 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:38,373 DEBUG   SenderThread:245590 [sender.py:send():235] send: telemetry
2022-03-01 17:13:38,373 DEBUG   SenderThread:245590 [sender.py:send():235] send: exit
2022-03-01 17:13:38,373 INFO    SenderThread:245590 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 17:13:38,373 INFO    SenderThread:245590 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 17:13:38,373 INFO    SenderThread:245590 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:13:38,374 INFO    SenderThread:245590 [sender.py:send_exit():379] send defer
2022-03-01 17:13:38,374 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:38,374 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,374 INFO    HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 17:13:38,375 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,375 INFO    SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 17:13:38,375 INFO    SenderThread:245590 [sender.py:transition_state():392] send defer: 1
2022-03-01 17:13:38,375 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,375 INFO    HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 17:13:38,414 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,414 INFO    SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 17:13:38,414 INFO    SenderThread:245590 [sender.py:transition_state():392] send defer: 2
2022-03-01 17:13:38,414 DEBUG   SenderThread:245590 [sender.py:send():235] send: stats
2022-03-01 17:13:38,415 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,415 INFO    HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 17:13:38,415 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,415 INFO    SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 17:13:38,415 INFO    SenderThread:245590 [sender.py:transition_state():392] send defer: 3
2022-03-01 17:13:38,415 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,415 INFO    HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 17:13:38,416 DEBUG   SenderThread:245590 [sender.py:send():235] send: summary
2022-03-01 17:13:38,416 INFO    SenderThread:245590 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:13:38,416 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,416 INFO    SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 17:13:38,416 INFO    SenderThread:245590 [sender.py:transition_state():392] send defer: 4
2022-03-01 17:13:38,416 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,417 INFO    HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 17:13:38,417 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,417 INFO    SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 17:13:38,475 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:38,539 INFO    SenderThread:245590 [sender.py:transition_state():392] send defer: 5
2022-03-01 17:13:38,539 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:38,539 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,539 INFO    HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 17:13:38,540 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,540 INFO    SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 17:13:38,540 INFO    SenderThread:245590 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 17:13:38,640 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:38,880 INFO    Thread-8  :245590 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-summary.json
2022-03-01 17:13:38,880 INFO    SenderThread:245590 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/config.yaml
2022-03-01 17:13:38,880 INFO    SenderThread:245590 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log
2022-03-01 17:13:38,880 INFO    SenderThread:245590 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files
2022-03-01 17:13:38,881 INFO    SenderThread:245590 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-metadata.json wandb-metadata.json
2022-03-01 17:13:38,881 INFO    SenderThread:245590 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log output.log
2022-03-01 17:13:38,881 INFO    SenderThread:245590 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-summary.json wandb-summary.json
2022-03-01 17:13:38,881 INFO    SenderThread:245590 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/requirements.txt requirements.txt
2022-03-01 17:13:38,881 INFO    SenderThread:245590 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/config.yaml config.yaml
2022-03-01 17:13:38,884 INFO    SenderThread:245590 [sender.py:transition_state():392] send defer: 6
2022-03-01 17:13:38,884 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:38,887 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,887 INFO    HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 17:13:38,888 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,888 INFO    SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 17:13:38,888 INFO    SenderThread:245590 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:13:38,988 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:38,988 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,090 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:39,090 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,137 INFO    Thread-12 :245590 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log
2022-03-01 17:13:39,153 INFO    Thread-13 :245590 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-summary.json
2022-03-01 17:13:39,162 INFO    Thread-14 :245590 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/requirements.txt
2022-03-01 17:13:39,176 INFO    Thread-15 :245590 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/config.yaml
2022-03-01 17:13:39,191 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:39,192 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,293 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:39,293 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,376 INFO    Thread-7  :245590 [sender.py:transition_state():392] send defer: 7
2022-03-01 17:13:39,377 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:39,377 INFO    HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 17:13:39,377 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:39,377 INFO    SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 17:13:39,394 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:39,524 INFO    SenderThread:245590 [sender.py:transition_state():392] send defer: 8
2022-03-01 17:13:39,524 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,524 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:39,524 INFO    HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 17:13:39,524 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:39,525 INFO    SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 17:13:39,525 INFO    SenderThread:245590 [sender.py:transition_state():392] send defer: 9
2022-03-01 17:13:39,525 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:39,525 DEBUG   SenderThread:245590 [sender.py:send():235] send: final
2022-03-01 17:13:39,526 INFO    HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 17:13:39,526 DEBUG   SenderThread:245590 [sender.py:send():235] send: footer
2022-03-01 17:13:39,526 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:39,526 INFO    SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 17:13:39,625 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:39,625 DEBUG   SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,625 INFO    SenderThread:245590 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:13:39,694 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 17:13:39,694 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 17:13:39,695 DEBUG   HandlerThread:245590 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 17:13:39,695 INFO    HandlerThread:245590 [handler.py:finish():739] shutting down handler
2022-03-01 17:13:40,526 INFO    WriterThread:245590 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/run-194tzn4x.wandb
2022-03-01 17:13:40,693 INFO    SenderThread:245590 [sender.py:finish():1075] shutting down sender
2022-03-01 17:13:40,693 INFO    SenderThread:245590 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:13:40,693 INFO    SenderThread:245590 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:13:40,695 INFO    MainThread:245590 [internal.py:handle_exit():79] Internal process exited