File size: 17,000 Bytes
a5afdd5
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
2022-03-01 01:26:49,894 INFO    MainThread:236498 [internal.py:wandb_internal():89] W&B internal server running at pid: 236498, started at: 2022-03-01 01:26:49.893757
2022-03-01 01:26:49,896 INFO    WriterThread:236498 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/run-nzue7jk0.wandb
2022-03-01 01:26:49,896 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 01:26:49,897 DEBUG   SenderThread:236498 [sender.py:send():235] send: header
2022-03-01 01:26:49,898 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: check_version
2022-03-01 01:26:49,967 DEBUG   SenderThread:236498 [sender.py:send():235] send: run
2022-03-01 01:26:50,066 INFO    SenderThread:236498 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files
2022-03-01 01:26:50,066 INFO    SenderThread:236498 [sender.py:_start_run_threads():809] run started: nzue7jk0 with start time 1646098009
2022-03-01 01:26:50,066 DEBUG   SenderThread:236498 [sender.py:send():235] send: summary
2022-03-01 01:26:50,066 INFO    SenderThread:236498 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:26:50,067 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 01:26:50,071 DEBUG   HandlerThread:236498 [meta.py:__init__():36] meta init
2022-03-01 01:26:50,071 DEBUG   HandlerThread:236498 [meta.py:__init__():50] meta init done
2022-03-01 01:26:50,071 DEBUG   HandlerThread:236498 [meta.py:probe():210] probe
2022-03-01 01:26:50,078 DEBUG   HandlerThread:236498 [meta.py:_setup_git():200] setup git
2022-03-01 01:26:50,092 DEBUG   HandlerThread:236498 [meta.py:_setup_git():207] setup git done
2022-03-01 01:26:50,093 DEBUG   HandlerThread:236498 [meta.py:_save_pip():54] save pip
2022-03-01 01:26:50,093 DEBUG   HandlerThread:236498 [meta.py:_save_pip():68] save pip done
2022-03-01 01:26:50,093 DEBUG   HandlerThread:236498 [meta.py:probe():248] probe done
2022-03-01 01:26:50,171 DEBUG   SenderThread:236498 [sender.py:send():235] send: files
2022-03-01 01:26:50,171 INFO    SenderThread:236498 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 01:26:50,176 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 01:26:50,176 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: stop_status
2022-03-01 01:26:50,211 DEBUG   SenderThread:236498 [sender.py:send():235] send: config
2022-03-01 01:26:50,212 DEBUG   SenderThread:236498 [sender.py:send():235] send: metric
2022-03-01 01:26:50,212 DEBUG   SenderThread:236498 [sender.py:send():235] send: metric
2022-03-01 01:26:50,212 WARNING SenderThread:236498 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 01:26:50,430 INFO    Thread-11 :236498 [upload_job.py:push():137] Uploaded file /tmp/tmpscakcml5wandb/3qv9ow3a-wandb-metadata.json
2022-03-01 01:26:51,068 INFO    Thread-8  :236498 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-summary.json
2022-03-01 01:26:51,068 INFO    Thread-8  :236498 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log
2022-03-01 01:26:51,068 INFO    Thread-8  :236498 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-metadata.json
2022-03-01 01:26:51,068 INFO    Thread-8  :236498 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/requirements.txt
2022-03-01 01:26:53,067 INFO    Thread-8  :236498 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log
2022-03-01 01:26:57,068 INFO    Thread-8  :236498 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log
2022-03-01 01:26:58,513 DEBUG   SenderThread:236498 [sender.py:send():235] send: telemetry
2022-03-01 01:26:58,513 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:58,514 DEBUG   SenderThread:236498 [sender.py:send():235] send: exit
2022-03-01 01:26:58,514 INFO    SenderThread:236498 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 01:26:58,514 INFO    SenderThread:236498 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 01:26:58,515 INFO    SenderThread:236498 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:26:58,515 INFO    SenderThread:236498 [sender.py:send_exit():379] send defer
2022-03-01 01:26:58,515 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:58,516 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,516 INFO    HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 01:26:58,516 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,516 INFO    SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 01:26:58,516 INFO    SenderThread:236498 [sender.py:transition_state():392] send defer: 1
2022-03-01 01:26:58,516 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,516 INFO    HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 01:26:58,601 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,601 INFO    SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 01:26:58,601 INFO    SenderThread:236498 [sender.py:transition_state():392] send defer: 2
2022-03-01 01:26:58,601 DEBUG   SenderThread:236498 [sender.py:send():235] send: stats
2022-03-01 01:26:58,602 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,602 INFO    HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 01:26:58,602 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,602 INFO    SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 01:26:58,602 INFO    SenderThread:236498 [sender.py:transition_state():392] send defer: 3
2022-03-01 01:26:58,603 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,603 INFO    HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 01:26:58,603 DEBUG   SenderThread:236498 [sender.py:send():235] send: summary
2022-03-01 01:26:58,603 INFO    SenderThread:236498 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:26:58,603 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,603 INFO    SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 01:26:58,603 INFO    SenderThread:236498 [sender.py:transition_state():392] send defer: 4
2022-03-01 01:26:58,604 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,604 INFO    HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 01:26:58,604 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,604 INFO    SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 01:26:58,617 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:58,708 INFO    SenderThread:236498 [sender.py:transition_state():392] send defer: 5
2022-03-01 01:26:58,708 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:58,709 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,709 INFO    HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 01:26:58,709 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,709 INFO    SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 01:26:58,709 INFO    SenderThread:236498 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 01:26:58,809 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,069 INFO    Thread-8  :236498 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/config.yaml
2022-03-01 01:26:59,070 INFO    SenderThread:236498 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log
2022-03-01 01:26:59,070 INFO    SenderThread:236498 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-summary.json
2022-03-01 01:26:59,070 INFO    SenderThread:236498 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files
2022-03-01 01:26:59,070 INFO    SenderThread:236498 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-metadata.json wandb-metadata.json
2022-03-01 01:26:59,070 INFO    SenderThread:236498 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log output.log
2022-03-01 01:26:59,070 INFO    SenderThread:236498 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-summary.json wandb-summary.json
2022-03-01 01:26:59,073 INFO    SenderThread:236498 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/requirements.txt requirements.txt
2022-03-01 01:26:59,074 INFO    SenderThread:236498 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/config.yaml config.yaml
2022-03-01 01:26:59,076 INFO    SenderThread:236498 [sender.py:transition_state():392] send defer: 6
2022-03-01 01:26:59,076 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,077 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:59,077 INFO    HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 01:26:59,078 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:59,084 INFO    SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 01:26:59,084 INFO    SenderThread:236498 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:26:59,179 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,179 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,281 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,281 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,340 INFO    Thread-15 :236498 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/config.yaml
2022-03-01 01:26:59,342 INFO    Thread-14 :236498 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/requirements.txt
2022-03-01 01:26:59,369 INFO    Thread-12 :236498 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log
2022-03-01 01:26:59,376 INFO    Thread-13 :236498 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-summary.json
2022-03-01 01:26:59,383 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,383 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,484 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,484 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,577 INFO    Thread-7  :236498 [sender.py:transition_state():392] send defer: 7
2022-03-01 01:26:59,577 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:59,577 INFO    HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 01:26:59,578 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:59,578 INFO    SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 01:26:59,585 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,682 INFO    SenderThread:236498 [sender.py:transition_state():392] send defer: 8
2022-03-01 01:26:59,683 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,683 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:59,683 INFO    HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 01:26:59,683 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:59,684 INFO    SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 01:26:59,684 INFO    SenderThread:236498 [sender.py:transition_state():392] send defer: 9
2022-03-01 01:26:59,684 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:59,684 INFO    HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 01:26:59,684 DEBUG   SenderThread:236498 [sender.py:send():235] send: final
2022-03-01 01:26:59,685 DEBUG   SenderThread:236498 [sender.py:send():235] send: footer
2022-03-01 01:26:59,685 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:59,685 INFO    SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 01:26:59,784 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,785 DEBUG   SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,785 INFO    SenderThread:236498 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:26:59,847 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 01:26:59,847 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 01:26:59,848 DEBUG   HandlerThread:236498 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 01:26:59,848 INFO    HandlerThread:236498 [handler.py:finish():739] shutting down handler
2022-03-01 01:27:00,685 INFO    WriterThread:236498 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/run-nzue7jk0.wandb
2022-03-01 01:27:00,846 INFO    SenderThread:236498 [sender.py:finish():1075] shutting down sender
2022-03-01 01:27:00,846 INFO    SenderThread:236498 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:27:00,846 INFO    SenderThread:236498 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:27:00,848 INFO    MainThread:236498 [internal.py:handle_exit():79] Internal process exited