File size: 17,220 Bytes
fc53a59
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
2022-03-02 08:51:28,518 INFO    MainThread:254490 [internal.py:wandb_internal():89] W&B internal server running at pid: 254490, started at: 2022-03-02 08:51:28.518100
2022-03-02 08:51:28,520 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: check_version
2022-03-02 08:51:28,520 INFO    WriterThread:254490 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/run-8m3km5vl.wandb
2022-03-02 08:51:28,522 DEBUG   SenderThread:254490 [sender.py:send():235] send: header
2022-03-02 08:51:28,522 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: check_version
2022-03-02 08:51:28,588 DEBUG   SenderThread:254490 [sender.py:send():235] send: run
2022-03-02 08:51:28,683 INFO    SenderThread:254490 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files
2022-03-02 08:51:28,683 INFO    SenderThread:254490 [sender.py:_start_run_threads():809] run started: 8m3km5vl with start time 1646211087
2022-03-02 08:51:28,683 DEBUG   SenderThread:254490 [sender.py:send():235] send: summary
2022-03-02 08:51:28,683 INFO    SenderThread:254490 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:51:28,684 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: run_start
2022-03-02 08:51:28,689 DEBUG   HandlerThread:254490 [meta.py:__init__():36] meta init
2022-03-02 08:51:28,689 DEBUG   HandlerThread:254490 [meta.py:__init__():50] meta init done
2022-03-02 08:51:28,689 DEBUG   HandlerThread:254490 [meta.py:probe():210] probe
2022-03-02 08:51:28,695 DEBUG   HandlerThread:254490 [meta.py:_setup_git():200] setup git
2022-03-02 08:51:28,710 DEBUG   HandlerThread:254490 [meta.py:_setup_git():207] setup git done
2022-03-02 08:51:28,710 DEBUG   HandlerThread:254490 [meta.py:_save_pip():54] save pip
2022-03-02 08:51:28,711 DEBUG   HandlerThread:254490 [meta.py:_save_pip():68] save pip done
2022-03-02 08:51:28,711 DEBUG   HandlerThread:254490 [meta.py:probe():248] probe done
2022-03-02 08:51:28,787 DEBUG   SenderThread:254490 [sender.py:send():235] send: files
2022-03-02 08:51:28,787 INFO    SenderThread:254490 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-02 08:51:28,792 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: stop_status
2022-03-02 08:51:28,792 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: stop_status
2022-03-02 08:51:28,825 DEBUG   SenderThread:254490 [sender.py:send():235] send: config
2022-03-02 08:51:28,826 DEBUG   SenderThread:254490 [sender.py:send():235] send: metric
2022-03-02 08:51:28,826 DEBUG   SenderThread:254490 [sender.py:send():235] send: metric
2022-03-02 08:51:28,826 WARNING SenderThread:254490 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-02 08:51:29,051 INFO    Thread-11 :254490 [upload_job.py:push():137] Uploaded file /tmp/tmp4hagpn04wandb/h47718pd-wandb-metadata.json
2022-03-02 08:51:29,685 INFO    Thread-8  :254490 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-metadata.json
2022-03-02 08:51:29,685 INFO    Thread-8  :254490 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-summary.json
2022-03-02 08:51:29,685 INFO    Thread-8  :254490 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log
2022-03-02 08:51:29,685 INFO    Thread-8  :254490 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/requirements.txt
2022-03-02 08:51:31,684 INFO    Thread-8  :254490 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log
2022-03-02 08:51:35,685 INFO    Thread-8  :254490 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log
2022-03-02 08:51:37,089 DEBUG   SenderThread:254490 [sender.py:send():235] send: telemetry
2022-03-02 08:51:37,090 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:37,090 DEBUG   SenderThread:254490 [sender.py:send():235] send: exit
2022-03-02 08:51:37,090 INFO    SenderThread:254490 [sender.py:send_exit():371] handling exit code: 1
2022-03-02 08:51:37,091 INFO    SenderThread:254490 [sender.py:send_exit():373] handling runtime: 8
2022-03-02 08:51:37,091 INFO    SenderThread:254490 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:51:37,091 INFO    SenderThread:254490 [sender.py:send_exit():379] send defer
2022-03-02 08:51:37,091 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:37,092 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,092 INFO    HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-02 08:51:37,092 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,092 INFO    SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-02 08:51:37,092 INFO    SenderThread:254490 [sender.py:transition_state():392] send defer: 1
2022-03-02 08:51:37,092 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,092 INFO    HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-02 08:51:37,120 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,120 INFO    SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-02 08:51:37,120 INFO    SenderThread:254490 [sender.py:transition_state():392] send defer: 2
2022-03-02 08:51:37,120 DEBUG   SenderThread:254490 [sender.py:send():235] send: stats
2022-03-02 08:51:37,121 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,121 INFO    HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-02 08:51:37,121 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,121 INFO    SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-02 08:51:37,121 INFO    SenderThread:254490 [sender.py:transition_state():392] send defer: 3
2022-03-02 08:51:37,121 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,121 INFO    HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-02 08:51:37,122 DEBUG   SenderThread:254490 [sender.py:send():235] send: summary
2022-03-02 08:51:37,122 INFO    SenderThread:254490 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:51:37,122 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,122 INFO    SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-02 08:51:37,122 INFO    SenderThread:254490 [sender.py:transition_state():392] send defer: 4
2022-03-02 08:51:37,123 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,123 INFO    HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-02 08:51:37,123 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,123 INFO    SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-02 08:51:37,198 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:37,219 INFO    SenderThread:254490 [sender.py:transition_state():392] send defer: 5
2022-03-02 08:51:37,219 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:37,220 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,220 INFO    HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-02 08:51:37,220 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,220 INFO    SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-02 08:51:37,220 INFO    SenderThread:254490 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-02 08:51:37,321 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:37,686 INFO    Thread-8  :254490 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-summary.json
2022-03-02 08:51:37,687 INFO    SenderThread:254490 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/config.yaml
2022-03-02 08:51:37,687 INFO    SenderThread:254490 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log
2022-03-02 08:51:37,687 INFO    SenderThread:254490 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files
2022-03-02 08:51:37,687 INFO    SenderThread:254490 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-metadata.json wandb-metadata.json
2022-03-02 08:51:37,687 INFO    SenderThread:254490 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log output.log
2022-03-02 08:51:37,687 INFO    SenderThread:254490 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-summary.json wandb-summary.json
2022-03-02 08:51:37,691 INFO    SenderThread:254490 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/requirements.txt requirements.txt
2022-03-02 08:51:37,691 INFO    SenderThread:254490 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/config.yaml config.yaml
2022-03-02 08:51:37,694 INFO    SenderThread:254490 [sender.py:transition_state():392] send defer: 6
2022-03-02 08:51:37,694 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:37,700 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,700 INFO    HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-02 08:51:37,701 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,701 INFO    SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-02 08:51:37,701 INFO    SenderThread:254490 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:51:37,803 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:37,803 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:37,905 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:37,905 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:37,949 INFO    Thread-15 :254490 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/config.yaml
2022-03-02 08:51:37,966 INFO    Thread-12 :254490 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log
2022-03-02 08:51:37,975 INFO    Thread-14 :254490 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/requirements.txt
2022-03-02 08:51:38,007 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:38,007 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:38,029 INFO    Thread-13 :254490 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-summary.json
2022-03-02 08:51:38,108 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:38,109 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:38,210 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:38,210 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:38,229 INFO    Thread-7  :254490 [sender.py:transition_state():392] send defer: 7
2022-03-02 08:51:38,230 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:38,230 INFO    HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-02 08:51:38,230 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:38,230 INFO    SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-02 08:51:38,311 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:38,350 INFO    SenderThread:254490 [sender.py:transition_state():392] send defer: 8
2022-03-02 08:51:38,350 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:38,350 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:38,351 INFO    HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-02 08:51:38,351 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:38,351 INFO    SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-02 08:51:38,351 INFO    SenderThread:254490 [sender.py:transition_state():392] send defer: 9
2022-03-02 08:51:38,352 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:38,352 INFO    HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-02 08:51:38,352 DEBUG   SenderThread:254490 [sender.py:send():235] send: final
2022-03-02 08:51:38,352 DEBUG   SenderThread:254490 [sender.py:send():235] send: footer
2022-03-02 08:51:38,352 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:38,352 INFO    SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-02 08:51:38,451 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:38,452 DEBUG   SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:38,452 INFO    SenderThread:254490 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:51:38,532 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: get_summary
2022-03-02 08:51:38,533 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-02 08:51:38,534 DEBUG   HandlerThread:254490 [handler.py:handle_request():131] handle_request: shutdown
2022-03-02 08:51:38,534 INFO    HandlerThread:254490 [handler.py:finish():739] shutting down handler
2022-03-02 08:51:39,352 INFO    WriterThread:254490 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/run-8m3km5vl.wandb
2022-03-02 08:51:39,532 INFO    SenderThread:254490 [sender.py:finish():1075] shutting down sender
2022-03-02 08:51:39,532 INFO    SenderThread:254490 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:51:39,532 INFO    SenderThread:254490 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:51:39,534 INFO    MainThread:254490 [internal.py:handle_exit():79] Internal process exited