File size: 17,439 Bytes
bcb2332
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
2022-03-03 00:43:48,671 INFO    MainThread:267559 [internal.py:wandb_internal():89] W&B internal server running at pid: 267559, started at: 2022-03-03 00:43:48.670697
2022-03-03 00:43:48,673 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: check_version
2022-03-03 00:43:48,674 INFO    WriterThread:267559 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/run-13qlgnoe.wandb
2022-03-03 00:43:48,675 DEBUG   SenderThread:267559 [sender.py:send():235] send: header
2022-03-03 00:43:48,675 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: check_version
2022-03-03 00:43:48,744 DEBUG   SenderThread:267559 [sender.py:send():235] send: run
2022-03-03 00:43:48,863 INFO    SenderThread:267559 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files
2022-03-03 00:43:48,863 INFO    SenderThread:267559 [sender.py:_start_run_threads():809] run started: 13qlgnoe with start time 1646268227
2022-03-03 00:43:48,863 DEBUG   SenderThread:267559 [sender.py:send():235] send: summary
2022-03-03 00:43:48,863 INFO    SenderThread:267559 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:43:48,864 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: run_start
2022-03-03 00:43:48,868 DEBUG   HandlerThread:267559 [meta.py:__init__():36] meta init
2022-03-03 00:43:48,869 DEBUG   HandlerThread:267559 [meta.py:__init__():50] meta init done
2022-03-03 00:43:48,869 DEBUG   HandlerThread:267559 [meta.py:probe():210] probe
2022-03-03 00:43:48,875 DEBUG   HandlerThread:267559 [meta.py:_setup_git():200] setup git
2022-03-03 00:43:48,891 DEBUG   HandlerThread:267559 [meta.py:_setup_git():207] setup git done
2022-03-03 00:43:48,891 DEBUG   HandlerThread:267559 [meta.py:_save_pip():54] save pip
2022-03-03 00:43:48,892 DEBUG   HandlerThread:267559 [meta.py:_save_pip():68] save pip done
2022-03-03 00:43:48,892 DEBUG   HandlerThread:267559 [meta.py:probe():248] probe done
2022-03-03 00:43:48,974 DEBUG   SenderThread:267559 [sender.py:send():235] send: files
2022-03-03 00:43:48,974 INFO    SenderThread:267559 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-03 00:43:48,979 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: stop_status
2022-03-03 00:43:48,980 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: stop_status
2022-03-03 00:43:49,032 DEBUG   SenderThread:267559 [sender.py:send():235] send: config
2022-03-03 00:43:49,033 DEBUG   SenderThread:267559 [sender.py:send():235] send: metric
2022-03-03 00:43:49,033 DEBUG   SenderThread:267559 [sender.py:send():235] send: metric
2022-03-03 00:43:49,033 WARNING SenderThread:267559 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-03 00:43:49,247 INFO    Thread-11 :267559 [upload_job.py:push():137] Uploaded file /tmp/tmps9bhl1w0wandb/kc71lad7-wandb-metadata.json
2022-03-03 00:43:49,865 INFO    Thread-8  :267559 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:43:49,865 INFO    Thread-8  :267559 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-metadata.json
2022-03-03 00:43:49,865 INFO    Thread-8  :267559 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-summary.json
2022-03-03 00:43:49,865 INFO    Thread-8  :267559 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/requirements.txt
2022-03-03 00:43:51,864 INFO    Thread-8  :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:43:55,866 INFO    Thread-8  :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:43:57,866 INFO    Thread-8  :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:43:58,509 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:43:58,509 DEBUG   SenderThread:267559 [sender.py:send():235] send: telemetry
2022-03-03 00:43:58,510 DEBUG   SenderThread:267559 [sender.py:send():235] send: exit
2022-03-03 00:43:58,510 INFO    SenderThread:267559 [sender.py:send_exit():371] handling exit code: 1
2022-03-03 00:43:58,510 INFO    SenderThread:267559 [sender.py:send_exit():373] handling runtime: 9
2022-03-03 00:43:58,510 INFO    SenderThread:267559 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:43:58,510 INFO    SenderThread:267559 [sender.py:send_exit():379] send defer
2022-03-03 00:43:58,511 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:43:58,511 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,511 INFO    HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-03 00:43:58,512 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,512 INFO    SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-03 00:43:58,512 INFO    SenderThread:267559 [sender.py:transition_state():392] send defer: 1
2022-03-03 00:43:58,512 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,512 INFO    HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-03 00:43:58,627 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,627 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:43:58,627 INFO    SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-03 00:43:58,628 INFO    SenderThread:267559 [sender.py:transition_state():392] send defer: 2
2022-03-03 00:43:58,628 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:43:58,628 DEBUG   SenderThread:267559 [sender.py:send():235] send: stats
2022-03-03 00:43:58,629 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,629 INFO    HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-03 00:43:58,629 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,629 INFO    SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-03 00:43:58,629 INFO    SenderThread:267559 [sender.py:transition_state():392] send defer: 3
2022-03-03 00:43:58,629 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,629 INFO    HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-03 00:43:58,630 DEBUG   SenderThread:267559 [sender.py:send():235] send: summary
2022-03-03 00:43:58,630 INFO    SenderThread:267559 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:43:58,630 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,630 INFO    SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-03 00:43:58,630 INFO    SenderThread:267559 [sender.py:transition_state():392] send defer: 4
2022-03-03 00:43:58,631 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,631 INFO    HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-03 00:43:58,631 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,631 INFO    SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-03 00:43:58,730 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:43:58,867 INFO    Thread-8  :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-summary.json
2022-03-03 00:43:58,867 INFO    Thread-8  :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:43:58,921 INFO    SenderThread:267559 [sender.py:transition_state():392] send defer: 5
2022-03-03 00:43:58,922 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:43:58,922 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,922 INFO    HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-03 00:43:58,923 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,923 INFO    SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-03 00:43:58,923 INFO    SenderThread:267559 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-03 00:43:59,024 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:43:59,867 INFO    Thread-8  :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/config.yaml
2022-03-03 00:43:59,868 INFO    SenderThread:267559 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files
2022-03-03 00:43:59,868 INFO    SenderThread:267559 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-metadata.json wandb-metadata.json
2022-03-03 00:43:59,869 INFO    SenderThread:267559 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log output.log
2022-03-03 00:43:59,869 INFO    SenderThread:267559 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-summary.json wandb-summary.json
2022-03-03 00:43:59,869 INFO    SenderThread:267559 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/requirements.txt requirements.txt
2022-03-03 00:43:59,869 INFO    SenderThread:267559 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/config.yaml config.yaml
2022-03-03 00:43:59,873 INFO    SenderThread:267559 [sender.py:transition_state():392] send defer: 6
2022-03-03 00:43:59,876 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:43:59,882 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:59,882 INFO    HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-03 00:43:59,884 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:59,884 INFO    SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-03 00:43:59,884 INFO    SenderThread:267559 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:43:59,977 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:43:59,978 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,080 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:44:00,080 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,168 INFO    Thread-15 :267559 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/config.yaml
2022-03-03 00:44:00,179 INFO    Thread-12 :267559 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:44:00,181 INFO    Thread-13 :267559 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-summary.json
2022-03-03 00:44:00,182 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:44:00,182 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,186 INFO    Thread-14 :267559 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/requirements.txt
2022-03-03 00:44:00,284 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:44:00,284 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,386 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:44:00,386 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,386 INFO    Thread-7  :267559 [sender.py:transition_state():392] send defer: 7
2022-03-03 00:44:00,387 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:44:00,387 INFO    HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-03 00:44:00,387 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:44:00,387 INFO    SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-03 00:44:00,423 INFO    SenderThread:267559 [sender.py:transition_state():392] send defer: 8
2022-03-03 00:44:00,423 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:44:00,423 INFO    HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-03 00:44:00,423 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:44:00,424 INFO    SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-03 00:44:00,424 INFO    SenderThread:267559 [sender.py:transition_state():392] send defer: 9
2022-03-03 00:44:00,424 DEBUG   SenderThread:267559 [sender.py:send():235] send: final
2022-03-03 00:44:00,425 DEBUG   SenderThread:267559 [sender.py:send():235] send: footer
2022-03-03 00:44:00,425 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:44:00,425 INFO    HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-03 00:44:00,425 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:44:00,425 INFO    SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-03 00:44:00,488 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:44:00,489 DEBUG   SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,489 INFO    SenderThread:267559 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:44:00,551 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: get_summary
2022-03-03 00:44:00,552 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-03 00:44:00,552 DEBUG   HandlerThread:267559 [handler.py:handle_request():131] handle_request: shutdown
2022-03-03 00:44:00,552 INFO    HandlerThread:267559 [handler.py:finish():739] shutting down handler
2022-03-03 00:44:01,425 INFO    WriterThread:267559 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/run-13qlgnoe.wandb
2022-03-03 00:44:01,549 INFO    SenderThread:267559 [sender.py:finish():1075] shutting down sender
2022-03-03 00:44:01,550 INFO    SenderThread:267559 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:44:01,550 INFO    SenderThread:267559 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:44:01,552 INFO    MainThread:267559 [internal.py:handle_exit():79] Internal process exited