File size: 17,449 Bytes
53a9ef4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
2022-03-01 17:12:01,466 INFO    MainThread:245412 [internal.py:wandb_internal():89] W&B internal server running at pid: 245412, started at: 2022-03-01 17:12:01.466387
2022-03-01 17:12:01,468 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 17:12:01,469 INFO    WriterThread:245412 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/run-3h52ha0p.wandb
2022-03-01 17:12:01,470 DEBUG   SenderThread:245412 [sender.py:send():235] send: header
2022-03-01 17:12:01,470 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: check_version
2022-03-01 17:12:01,535 DEBUG   SenderThread:245412 [sender.py:send():235] send: run
2022-03-01 17:12:01,658 INFO    SenderThread:245412 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files
2022-03-01 17:12:01,658 INFO    SenderThread:245412 [sender.py:_start_run_threads():809] run started: 3h52ha0p with start time 1646154720
2022-03-01 17:12:01,658 DEBUG   SenderThread:245412 [sender.py:send():235] send: summary
2022-03-01 17:12:01,658 INFO    SenderThread:245412 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:12:01,659 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 17:12:01,663 DEBUG   HandlerThread:245412 [meta.py:__init__():36] meta init
2022-03-01 17:12:01,663 DEBUG   HandlerThread:245412 [meta.py:__init__():50] meta init done
2022-03-01 17:12:01,663 DEBUG   HandlerThread:245412 [meta.py:probe():210] probe
2022-03-01 17:12:01,669 DEBUG   HandlerThread:245412 [meta.py:_setup_git():200] setup git
2022-03-01 17:12:01,684 DEBUG   HandlerThread:245412 [meta.py:_setup_git():207] setup git done
2022-03-01 17:12:01,684 DEBUG   HandlerThread:245412 [meta.py:_save_pip():54] save pip
2022-03-01 17:12:01,684 DEBUG   HandlerThread:245412 [meta.py:_save_pip():68] save pip done
2022-03-01 17:12:01,685 DEBUG   HandlerThread:245412 [meta.py:probe():248] probe done
2022-03-01 17:12:01,758 DEBUG   SenderThread:245412 [sender.py:send():235] send: files
2022-03-01 17:12:01,758 INFO    SenderThread:245412 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 17:12:01,762 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 17:12:01,763 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: stop_status
2022-03-01 17:12:01,796 DEBUG   SenderThread:245412 [sender.py:send():235] send: config
2022-03-01 17:12:01,797 DEBUG   SenderThread:245412 [sender.py:send():235] send: metric
2022-03-01 17:12:01,797 DEBUG   SenderThread:245412 [sender.py:send():235] send: metric
2022-03-01 17:12:01,797 WARNING SenderThread:245412 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 17:12:02,036 INFO    Thread-11 :245412 [upload_job.py:push():137] Uploaded file /tmp/tmpt38fg238wandb/2am5wtm4-wandb-metadata.json
2022-03-01 17:12:02,659 INFO    Thread-8  :245412 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/requirements.txt
2022-03-01 17:12:02,660 INFO    Thread-8  :245412 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log
2022-03-01 17:12:02,660 INFO    Thread-8  :245412 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-metadata.json
2022-03-01 17:12:02,660 INFO    Thread-8  :245412 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-summary.json
2022-03-01 17:12:04,659 INFO    Thread-8  :245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log
2022-03-01 17:12:08,660 INFO    Thread-8  :245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log
2022-03-01 17:12:10,572 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:10,572 DEBUG   SenderThread:245412 [sender.py:send():235] send: telemetry
2022-03-01 17:12:10,573 DEBUG   SenderThread:245412 [sender.py:send():235] send: exit
2022-03-01 17:12:10,573 INFO    SenderThread:245412 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 17:12:10,573 INFO    SenderThread:245412 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 17:12:10,573 INFO    SenderThread:245412 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:12:10,573 INFO    SenderThread:245412 [sender.py:send_exit():379] send defer
2022-03-01 17:12:10,574 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:10,574 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,574 INFO    HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 17:12:10,574 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,574 INFO    SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 17:12:10,575 INFO    SenderThread:245412 [sender.py:transition_state():392] send defer: 1
2022-03-01 17:12:10,575 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,575 INFO    HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 17:12:10,661 INFO    Thread-8  :245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log
2022-03-01 17:12:10,661 INFO    Thread-8  :245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-summary.json
2022-03-01 17:12:10,684 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,684 INFO    SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 17:12:10,684 INFO    SenderThread:245412 [sender.py:transition_state():392] send defer: 2
2022-03-01 17:12:10,684 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:10,685 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:10,685 DEBUG   SenderThread:245412 [sender.py:send():235] send: stats
2022-03-01 17:12:10,685 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,686 INFO    HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 17:12:10,686 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,686 INFO    SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 17:12:10,686 INFO    SenderThread:245412 [sender.py:transition_state():392] send defer: 3
2022-03-01 17:12:10,686 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,686 INFO    HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 17:12:10,686 DEBUG   SenderThread:245412 [sender.py:send():235] send: summary
2022-03-01 17:12:10,687 INFO    SenderThread:245412 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:12:10,687 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,687 INFO    SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 17:12:10,687 INFO    SenderThread:245412 [sender.py:transition_state():392] send defer: 4
2022-03-01 17:12:10,687 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,687 INFO    HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 17:12:10,687 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,688 INFO    SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 17:12:10,791 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:10,821 INFO    SenderThread:245412 [sender.py:transition_state():392] send defer: 5
2022-03-01 17:12:10,821 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:10,822 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,822 INFO    HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 17:12:10,822 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,822 INFO    SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 17:12:10,822 INFO    SenderThread:245412 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 17:12:10,923 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:11,662 INFO    Thread-8  :245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/config.yaml
2022-03-01 17:12:11,662 INFO    SenderThread:245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-summary.json
2022-03-01 17:12:11,662 INFO    SenderThread:245412 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files
2022-03-01 17:12:11,663 INFO    SenderThread:245412 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-metadata.json wandb-metadata.json
2022-03-01 17:12:11,663 INFO    SenderThread:245412 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log output.log
2022-03-01 17:12:11,663 INFO    SenderThread:245412 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-summary.json wandb-summary.json
2022-03-01 17:12:11,663 INFO    SenderThread:245412 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/requirements.txt requirements.txt
2022-03-01 17:12:11,663 INFO    SenderThread:245412 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/config.yaml config.yaml
2022-03-01 17:12:11,666 INFO    SenderThread:245412 [sender.py:transition_state():392] send defer: 6
2022-03-01 17:12:11,666 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:11,672 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:11,673 INFO    HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 17:12:11,673 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:11,673 INFO    SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 17:12:11,674 INFO    SenderThread:245412 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:12:11,768 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:11,768 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:11,869 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:11,869 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:11,967 INFO    Thread-12 :245412 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log
2022-03-01 17:12:11,970 INFO    Thread-14 :245412 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/requirements.txt
2022-03-01 17:12:11,971 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:11,971 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:11,976 INFO    Thread-13 :245412 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-summary.json
2022-03-01 17:12:11,995 INFO    Thread-15 :245412 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/config.yaml
2022-03-01 17:12:12,072 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:12,073 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:12,174 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:12,174 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:12,196 INFO    Thread-7  :245412 [sender.py:transition_state():392] send defer: 7
2022-03-01 17:12:12,196 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:12,196 INFO    HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 17:12:12,196 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:12,196 INFO    SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 17:12:12,243 INFO    SenderThread:245412 [sender.py:transition_state():392] send defer: 8
2022-03-01 17:12:12,244 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:12,244 INFO    HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 17:12:12,244 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:12,244 INFO    SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 17:12:12,244 INFO    SenderThread:245412 [sender.py:transition_state():392] send defer: 9
2022-03-01 17:12:12,245 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:12,245 INFO    HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 17:12:12,245 DEBUG   SenderThread:245412 [sender.py:send():235] send: final
2022-03-01 17:12:12,245 DEBUG   SenderThread:245412 [sender.py:send():235] send: footer
2022-03-01 17:12:12,245 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:12,245 INFO    SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 17:12:12,275 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:12,275 DEBUG   SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:12,276 INFO    SenderThread:245412 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:12:12,332 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 17:12:12,333 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 17:12:12,334 DEBUG   HandlerThread:245412 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 17:12:12,334 INFO    HandlerThread:245412 [handler.py:finish():739] shutting down handler
2022-03-01 17:12:13,246 INFO    WriterThread:245412 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/run-3h52ha0p.wandb
2022-03-01 17:12:13,331 INFO    SenderThread:245412 [sender.py:finish():1075] shutting down sender
2022-03-01 17:12:13,331 INFO    SenderThread:245412 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:12:13,331 INFO    SenderThread:245412 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:12:13,333 INFO    MainThread:245412 [internal.py:handle_exit():79] Internal process exited