File size: 17,663 Bytes
bcb2332
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
2022-03-03 00:40:40,587 INFO    MainThread:267195 [internal.py:wandb_internal():89] W&B internal server running at pid: 267195, started at: 2022-03-03 00:40:40.586996
2022-03-03 00:40:40,590 INFO    WriterThread:267195 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/run-1a18zsso.wandb
2022-03-03 00:40:40,590 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: check_version
2022-03-03 00:40:40,591 DEBUG   SenderThread:267195 [sender.py:send():235] send: header
2022-03-03 00:40:40,591 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: check_version
2022-03-03 00:40:40,668 DEBUG   SenderThread:267195 [sender.py:send():235] send: run
2022-03-03 00:40:40,791 INFO    SenderThread:267195 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files
2022-03-03 00:40:40,791 INFO    SenderThread:267195 [sender.py:_start_run_threads():809] run started: 1a18zsso with start time 1646268039
2022-03-03 00:40:40,791 DEBUG   SenderThread:267195 [sender.py:send():235] send: summary
2022-03-03 00:40:40,791 INFO    SenderThread:267195 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:40:40,793 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: run_start
2022-03-03 00:40:40,798 DEBUG   HandlerThread:267195 [meta.py:__init__():36] meta init
2022-03-03 00:40:40,798 DEBUG   HandlerThread:267195 [meta.py:__init__():50] meta init done
2022-03-03 00:40:40,798 DEBUG   HandlerThread:267195 [meta.py:probe():210] probe
2022-03-03 00:40:40,805 DEBUG   HandlerThread:267195 [meta.py:_setup_git():200] setup git
2022-03-03 00:40:40,821 DEBUG   HandlerThread:267195 [meta.py:_setup_git():207] setup git done
2022-03-03 00:40:40,822 DEBUG   HandlerThread:267195 [meta.py:_save_pip():54] save pip
2022-03-03 00:40:40,823 DEBUG   HandlerThread:267195 [meta.py:_save_pip():68] save pip done
2022-03-03 00:40:40,823 DEBUG   HandlerThread:267195 [meta.py:probe():248] probe done
2022-03-03 00:40:40,946 DEBUG   SenderThread:267195 [sender.py:send():235] send: files
2022-03-03 00:40:40,946 INFO    SenderThread:267195 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-03 00:40:40,951 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: stop_status
2022-03-03 00:40:40,952 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: stop_status
2022-03-03 00:40:40,999 DEBUG   SenderThread:267195 [sender.py:send():235] send: config
2022-03-03 00:40:41,000 DEBUG   SenderThread:267195 [sender.py:send():235] send: metric
2022-03-03 00:40:41,000 DEBUG   SenderThread:267195 [sender.py:send():235] send: metric
2022-03-03 00:40:41,000 WARNING SenderThread:267195 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-03 00:40:41,240 INFO    Thread-11 :267195 [upload_job.py:push():137] Uploaded file /tmp/tmp9mhkg11ywandb/2vus7xrr-wandb-metadata.json
2022-03-03 00:40:41,793 INFO    Thread-8  :267195 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:41,793 INFO    Thread-8  :267195 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-metadata.json
2022-03-03 00:40:41,794 INFO    Thread-8  :267195 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/requirements.txt
2022-03-03 00:40:41,794 INFO    Thread-8  :267195 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-summary.json
2022-03-03 00:40:43,792 INFO    Thread-8  :267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:47,793 INFO    Thread-8  :267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:49,794 INFO    Thread-8  :267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:49,872 DEBUG   SenderThread:267195 [sender.py:send():235] send: telemetry
2022-03-03 00:40:49,872 DEBUG   SenderThread:267195 [sender.py:send():235] send: exit
2022-03-03 00:40:49,872 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:49,873 INFO    SenderThread:267195 [sender.py:send_exit():371] handling exit code: 1
2022-03-03 00:40:49,873 INFO    SenderThread:267195 [sender.py:send_exit():373] handling runtime: 9
2022-03-03 00:40:49,873 INFO    SenderThread:267195 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:40:49,873 INFO    SenderThread:267195 [sender.py:send_exit():379] send defer
2022-03-03 00:40:49,874 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:49,874 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:49,875 INFO    HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-03 00:40:49,875 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:49,875 INFO    SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-03 00:40:49,875 INFO    SenderThread:267195 [sender.py:transition_state():392] send defer: 1
2022-03-03 00:40:49,875 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:49,876 INFO    HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-03 00:40:49,994 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:49,994 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:49,994 INFO    SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-03 00:40:49,994 INFO    SenderThread:267195 [sender.py:transition_state():392] send defer: 2
2022-03-03 00:40:49,995 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:49,995 DEBUG   SenderThread:267195 [sender.py:send():235] send: stats
2022-03-03 00:40:49,996 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:49,996 INFO    HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-03 00:40:49,996 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:49,996 INFO    SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-03 00:40:49,996 INFO    SenderThread:267195 [sender.py:transition_state():392] send defer: 3
2022-03-03 00:40:49,996 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:49,997 INFO    HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-03 00:40:49,997 DEBUG   SenderThread:267195 [sender.py:send():235] send: summary
2022-03-03 00:40:49,998 INFO    SenderThread:267195 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:40:49,998 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:49,998 INFO    SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-03 00:40:49,998 INFO    SenderThread:267195 [sender.py:transition_state():392] send defer: 4
2022-03-03 00:40:49,998 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:49,998 INFO    HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-03 00:40:49,998 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:49,998 INFO    SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-03 00:40:50,102 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:50,128 INFO    SenderThread:267195 [sender.py:transition_state():392] send defer: 5
2022-03-03 00:40:50,128 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:50,129 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:50,129 INFO    HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-03 00:40:50,129 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:50,130 INFO    SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-03 00:40:50,130 INFO    SenderThread:267195 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-03 00:40:50,231 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:50,795 INFO    Thread-8  :267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:50,795 INFO    SenderThread:267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-summary.json
2022-03-03 00:40:50,796 INFO    SenderThread:267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/config.yaml
2022-03-03 00:40:50,796 INFO    SenderThread:267195 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files
2022-03-03 00:40:50,796 INFO    SenderThread:267195 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-metadata.json wandb-metadata.json
2022-03-03 00:40:50,796 INFO    SenderThread:267195 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log output.log
2022-03-03 00:40:50,797 INFO    SenderThread:267195 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-summary.json wandb-summary.json
2022-03-03 00:40:50,800 INFO    SenderThread:267195 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/requirements.txt requirements.txt
2022-03-03 00:40:50,803 INFO    SenderThread:267195 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/config.yaml config.yaml
2022-03-03 00:40:50,807 INFO    SenderThread:267195 [sender.py:transition_state():392] send defer: 6
2022-03-03 00:40:50,807 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:50,808 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:50,809 INFO    HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-03 00:40:50,812 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:50,813 INFO    SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-03 00:40:50,813 INFO    SenderThread:267195 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:40:50,910 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:50,911 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,013 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,013 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,068 INFO    Thread-15 :267195 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/config.yaml
2022-03-03 00:40:51,071 INFO    Thread-13 :267195 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-summary.json
2022-03-03 00:40:51,116 INFO    Thread-12 :267195 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:51,117 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,117 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,172 INFO    Thread-14 :267195 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/requirements.txt
2022-03-03 00:40:51,219 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,219 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,321 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,322 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,372 INFO    Thread-7  :267195 [sender.py:transition_state():392] send defer: 7
2022-03-03 00:40:51,373 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:51,373 INFO    HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-03 00:40:51,373 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:51,373 INFO    SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-03 00:40:51,424 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,429 INFO    SenderThread:267195 [sender.py:transition_state():392] send defer: 8
2022-03-03 00:40:51,429 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,430 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:51,430 INFO    HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-03 00:40:51,431 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:51,431 INFO    SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-03 00:40:51,431 INFO    SenderThread:267195 [sender.py:transition_state():392] send defer: 9
2022-03-03 00:40:51,431 DEBUG   SenderThread:267195 [sender.py:send():235] send: final
2022-03-03 00:40:51,432 DEBUG   SenderThread:267195 [sender.py:send():235] send: footer
2022-03-03 00:40:51,432 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:51,432 INFO    HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-03 00:40:51,432 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:51,433 INFO    SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-03 00:40:51,531 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,532 DEBUG   SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,532 INFO    SenderThread:267195 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:40:51,631 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: get_summary
2022-03-03 00:40:51,632 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-03 00:40:51,633 DEBUG   HandlerThread:267195 [handler.py:handle_request():131] handle_request: shutdown
2022-03-03 00:40:51,633 INFO    HandlerThread:267195 [handler.py:finish():739] shutting down handler
2022-03-03 00:40:52,432 INFO    WriterThread:267195 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/run-1a18zsso.wandb
2022-03-03 00:40:52,630 INFO    SenderThread:267195 [sender.py:finish():1075] shutting down sender
2022-03-03 00:40:52,630 INFO    SenderThread:267195 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:40:52,630 INFO    SenderThread:267195 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:40:52,632 INFO    MainThread:267195 [internal.py:handle_exit():79] Internal process exited