File size: 17,220 Bytes
a5afdd5
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
2022-03-01 01:28:17,477 INFO    MainThread:236675 [internal.py:wandb_internal():89] W&B internal server running at pid: 236675, started at: 2022-03-01 01:28:17.477147
2022-03-01 01:28:17,479 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 01:28:17,479 INFO    WriterThread:236675 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/run-3bh80nzn.wandb
2022-03-01 01:28:17,481 DEBUG   SenderThread:236675 [sender.py:send():235] send: header
2022-03-01 01:28:17,481 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: check_version
2022-03-01 01:28:17,547 DEBUG   SenderThread:236675 [sender.py:send():235] send: run
2022-03-01 01:28:17,636 INFO    SenderThread:236675 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files
2022-03-01 01:28:17,636 INFO    SenderThread:236675 [sender.py:_start_run_threads():809] run started: 3bh80nzn with start time 1646098096
2022-03-01 01:28:17,636 DEBUG   SenderThread:236675 [sender.py:send():235] send: summary
2022-03-01 01:28:17,636 INFO    SenderThread:236675 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:28:17,637 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 01:28:17,641 DEBUG   HandlerThread:236675 [meta.py:__init__():36] meta init
2022-03-01 01:28:17,641 DEBUG   HandlerThread:236675 [meta.py:__init__():50] meta init done
2022-03-01 01:28:17,642 DEBUG   HandlerThread:236675 [meta.py:probe():210] probe
2022-03-01 01:28:17,648 DEBUG   HandlerThread:236675 [meta.py:_setup_git():200] setup git
2022-03-01 01:28:17,663 DEBUG   HandlerThread:236675 [meta.py:_setup_git():207] setup git done
2022-03-01 01:28:17,663 DEBUG   HandlerThread:236675 [meta.py:_save_pip():54] save pip
2022-03-01 01:28:17,663 DEBUG   HandlerThread:236675 [meta.py:_save_pip():68] save pip done
2022-03-01 01:28:17,664 DEBUG   HandlerThread:236675 [meta.py:probe():248] probe done
2022-03-01 01:28:17,744 DEBUG   SenderThread:236675 [sender.py:send():235] send: files
2022-03-01 01:28:17,744 INFO    SenderThread:236675 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 01:28:17,749 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 01:28:17,749 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: stop_status
2022-03-01 01:28:17,783 DEBUG   SenderThread:236675 [sender.py:send():235] send: config
2022-03-01 01:28:17,784 DEBUG   SenderThread:236675 [sender.py:send():235] send: metric
2022-03-01 01:28:17,784 DEBUG   SenderThread:236675 [sender.py:send():235] send: metric
2022-03-01 01:28:17,784 WARNING SenderThread:236675 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 01:28:18,011 INFO    Thread-11 :236675 [upload_job.py:push():137] Uploaded file /tmp/tmpb_wel_s6wandb/1ng2jof2-wandb-metadata.json
2022-03-01 01:28:18,638 INFO    Thread-8  :236675 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-metadata.json
2022-03-01 01:28:18,638 INFO    Thread-8  :236675 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/requirements.txt
2022-03-01 01:28:18,638 INFO    Thread-8  :236675 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-summary.json
2022-03-01 01:28:18,638 INFO    Thread-8  :236675 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log
2022-03-01 01:28:20,637 INFO    Thread-8  :236675 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log
2022-03-01 01:28:24,639 INFO    Thread-8  :236675 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log
2022-03-01 01:28:25,989 DEBUG   SenderThread:236675 [sender.py:send():235] send: telemetry
2022-03-01 01:28:25,989 DEBUG   SenderThread:236675 [sender.py:send():235] send: exit
2022-03-01 01:28:25,989 INFO    SenderThread:236675 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 01:28:25,989 INFO    SenderThread:236675 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 01:28:25,990 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:25,990 INFO    SenderThread:236675 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:28:25,991 INFO    SenderThread:236675 [sender.py:send_exit():379] send defer
2022-03-01 01:28:25,991 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:25,991 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:25,991 INFO    HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 01:28:25,992 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:25,992 INFO    SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 01:28:25,992 INFO    SenderThread:236675 [sender.py:transition_state():392] send defer: 1
2022-03-01 01:28:25,992 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:25,992 INFO    HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 01:28:26,073 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,073 INFO    SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 01:28:26,073 INFO    SenderThread:236675 [sender.py:transition_state():392] send defer: 2
2022-03-01 01:28:26,074 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:26,074 INFO    HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 01:28:26,074 DEBUG   SenderThread:236675 [sender.py:send():235] send: stats
2022-03-01 01:28:26,075 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,075 INFO    SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 01:28:26,075 INFO    SenderThread:236675 [sender.py:transition_state():392] send defer: 3
2022-03-01 01:28:26,075 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:26,075 INFO    HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 01:28:26,075 DEBUG   SenderThread:236675 [sender.py:send():235] send: summary
2022-03-01 01:28:26,076 INFO    SenderThread:236675 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:28:26,076 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,076 INFO    SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 01:28:26,076 INFO    SenderThread:236675 [sender.py:transition_state():392] send defer: 4
2022-03-01 01:28:26,076 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:26,076 INFO    HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 01:28:26,077 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,077 INFO    SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 01:28:26,093 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:26,179 INFO    SenderThread:236675 [sender.py:transition_state():392] send defer: 5
2022-03-01 01:28:26,179 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:26,180 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:26,180 INFO    HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 01:28:26,180 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,180 INFO    SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 01:28:26,180 INFO    SenderThread:236675 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 01:28:26,281 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:26,639 INFO    Thread-8  :236675 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-summary.json
2022-03-01 01:28:26,640 INFO    SenderThread:236675 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log
2022-03-01 01:28:26,640 INFO    SenderThread:236675 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/config.yaml
2022-03-01 01:28:26,640 INFO    SenderThread:236675 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files
2022-03-01 01:28:26,641 INFO    SenderThread:236675 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-metadata.json wandb-metadata.json
2022-03-01 01:28:26,641 INFO    SenderThread:236675 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log output.log
2022-03-01 01:28:26,641 INFO    SenderThread:236675 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-summary.json wandb-summary.json
2022-03-01 01:28:26,644 INFO    SenderThread:236675 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/requirements.txt requirements.txt
2022-03-01 01:28:26,644 INFO    SenderThread:236675 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/config.yaml config.yaml
2022-03-01 01:28:26,644 INFO    SenderThread:236675 [sender.py:transition_state():392] send defer: 6
2022-03-01 01:28:26,645 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:26,648 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:26,651 INFO    HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 01:28:26,653 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,653 INFO    SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 01:28:26,654 INFO    SenderThread:236675 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:28:26,752 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:26,752 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:26,854 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:26,854 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:26,905 INFO    Thread-14 :236675 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/requirements.txt
2022-03-01 01:28:26,912 INFO    Thread-13 :236675 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-summary.json
2022-03-01 01:28:26,932 INFO    Thread-12 :236675 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log
2022-03-01 01:28:26,955 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:26,955 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:26,961 INFO    Thread-15 :236675 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/config.yaml
2022-03-01 01:28:27,057 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:27,057 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:27,158 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:27,158 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:27,162 INFO    Thread-7  :236675 [sender.py:transition_state():392] send defer: 7
2022-03-01 01:28:27,162 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:27,162 INFO    HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 01:28:27,162 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:27,162 INFO    SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 01:28:27,259 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:28,140 INFO    SenderThread:236675 [sender.py:transition_state():392] send defer: 8
2022-03-01 01:28:28,140 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:28,141 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:28,141 INFO    HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 01:28:28,141 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:28,141 INFO    SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 01:28:28,141 INFO    SenderThread:236675 [sender.py:transition_state():392] send defer: 9
2022-03-01 01:28:28,142 DEBUG   SenderThread:236675 [sender.py:send():235] send: final
2022-03-01 01:28:28,142 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:28,142 INFO    HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 01:28:28,142 DEBUG   SenderThread:236675 [sender.py:send():235] send: footer
2022-03-01 01:28:28,142 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:28,142 INFO    SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 01:28:28,241 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:28,242 DEBUG   SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:28,242 INFO    SenderThread:236675 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:28:28,300 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 01:28:28,301 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 01:28:28,302 DEBUG   HandlerThread:236675 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 01:28:28,302 INFO    HandlerThread:236675 [handler.py:finish():739] shutting down handler
2022-03-01 01:28:29,142 INFO    WriterThread:236675 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/run-3bh80nzn.wandb
2022-03-01 01:28:29,299 INFO    SenderThread:236675 [sender.py:finish():1075] shutting down sender
2022-03-01 01:28:29,300 INFO    SenderThread:236675 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:28:29,300 INFO    SenderThread:236675 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:28:29,302 INFO    MainThread:236675 [internal.py:handle_exit():79] Internal process exited