File size: 16,996 Bytes
a5afdd5
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
2022-03-01 01:29:45,420 INFO    MainThread:236851 [internal.py:wandb_internal():89] W&B internal server running at pid: 236851, started at: 2022-03-01 01:29:45.420003
2022-03-01 01:29:45,422 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 01:29:45,422 INFO    WriterThread:236851 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/run-25hk0gke.wandb
2022-03-01 01:29:45,424 DEBUG   SenderThread:236851 [sender.py:send():235] send: header
2022-03-01 01:29:45,424 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: check_version
2022-03-01 01:29:45,489 DEBUG   SenderThread:236851 [sender.py:send():235] send: run
2022-03-01 01:29:45,576 INFO    SenderThread:236851 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files
2022-03-01 01:29:45,576 INFO    SenderThread:236851 [sender.py:_start_run_threads():809] run started: 25hk0gke with start time 1646098184
2022-03-01 01:29:45,576 DEBUG   SenderThread:236851 [sender.py:send():235] send: summary
2022-03-01 01:29:45,576 INFO    SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:29:45,577 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 01:29:45,582 DEBUG   HandlerThread:236851 [meta.py:__init__():36] meta init
2022-03-01 01:29:45,582 DEBUG   HandlerThread:236851 [meta.py:__init__():50] meta init done
2022-03-01 01:29:45,582 DEBUG   HandlerThread:236851 [meta.py:probe():210] probe
2022-03-01 01:29:45,588 DEBUG   HandlerThread:236851 [meta.py:_setup_git():200] setup git
2022-03-01 01:29:45,603 DEBUG   HandlerThread:236851 [meta.py:_setup_git():207] setup git done
2022-03-01 01:29:45,603 DEBUG   HandlerThread:236851 [meta.py:_save_pip():54] save pip
2022-03-01 01:29:45,604 DEBUG   HandlerThread:236851 [meta.py:_save_pip():68] save pip done
2022-03-01 01:29:45,604 DEBUG   HandlerThread:236851 [meta.py:probe():248] probe done
2022-03-01 01:29:45,684 DEBUG   SenderThread:236851 [sender.py:send():235] send: files
2022-03-01 01:29:45,684 INFO    SenderThread:236851 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 01:29:45,689 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 01:29:45,690 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: stop_status
2022-03-01 01:29:45,729 DEBUG   SenderThread:236851 [sender.py:send():235] send: config
2022-03-01 01:29:45,731 DEBUG   SenderThread:236851 [sender.py:send():235] send: metric
2022-03-01 01:29:45,731 DEBUG   SenderThread:236851 [sender.py:send():235] send: metric
2022-03-01 01:29:45,731 WARNING SenderThread:236851 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 01:29:45,969 INFO    Thread-11 :236851 [upload_job.py:push():137] Uploaded file /tmp/tmpdzeo2dm2wandb/3813kelu-wandb-metadata.json
2022-03-01 01:29:46,578 INFO    Thread-8  :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt
2022-03-01 01:29:46,578 INFO    Thread-8  :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json
2022-03-01 01:29:46,578 INFO    Thread-8  :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-metadata.json
2022-03-01 01:29:46,578 INFO    Thread-8  :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:48,577 INFO    Thread-8  :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:52,578 INFO    Thread-8  :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:54,442 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:54,442 DEBUG   SenderThread:236851 [sender.py:send():235] send: telemetry
2022-03-01 01:29:54,443 DEBUG   SenderThread:236851 [sender.py:send():235] send: exit
2022-03-01 01:29:54,443 INFO    SenderThread:236851 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 01:29:54,443 INFO    SenderThread:236851 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 01:29:54,443 INFO    SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:29:54,443 INFO    SenderThread:236851 [sender.py:send_exit():379] send defer
2022-03-01 01:29:54,444 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:54,444 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,444 INFO    HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 01:29:54,444 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,445 INFO    SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 01:29:54,445 INFO    SenderThread:236851 [sender.py:transition_state():392] send defer: 1
2022-03-01 01:29:54,445 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,445 INFO    HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 01:29:54,515 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,515 INFO    SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 01:29:54,515 INFO    SenderThread:236851 [sender.py:transition_state():392] send defer: 2
2022-03-01 01:29:54,515 DEBUG   SenderThread:236851 [sender.py:send():235] send: stats
2022-03-01 01:29:54,516 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,516 INFO    HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 01:29:54,516 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,516 INFO    SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 01:29:54,516 INFO    SenderThread:236851 [sender.py:transition_state():392] send defer: 3
2022-03-01 01:29:54,517 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,517 INFO    HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 01:29:54,517 DEBUG   SenderThread:236851 [sender.py:send():235] send: summary
2022-03-01 01:29:54,517 INFO    SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:29:54,518 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,518 INFO    SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 01:29:54,518 INFO    SenderThread:236851 [sender.py:transition_state():392] send defer: 4
2022-03-01 01:29:54,518 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,518 INFO    HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 01:29:54,518 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,518 INFO    SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 01:29:54,545 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:54,579 INFO    Thread-8  :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json
2022-03-01 01:29:54,579 INFO    Thread-8  :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:54,803 INFO    SenderThread:236851 [sender.py:transition_state():392] send defer: 5
2022-03-01 01:29:54,803 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:54,804 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,804 INFO    HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 01:29:54,804 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,804 INFO    SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 01:29:54,804 INFO    SenderThread:236851 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 01:29:54,905 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,579 INFO    Thread-8  :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml
2022-03-01 01:29:55,580 INFO    SenderThread:236851 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files
2022-03-01 01:29:55,580 INFO    SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-metadata.json wandb-metadata.json
2022-03-01 01:29:55,580 INFO    SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log output.log
2022-03-01 01:29:55,581 INFO    SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json wandb-summary.json
2022-03-01 01:29:55,581 INFO    SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt requirements.txt
2022-03-01 01:29:55,586 INFO    SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml config.yaml
2022-03-01 01:29:55,586 INFO    SenderThread:236851 [sender.py:transition_state():392] send defer: 6
2022-03-01 01:29:55,587 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,588 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:55,588 INFO    HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 01:29:55,591 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:55,591 INFO    SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 01:29:55,592 INFO    SenderThread:236851 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:29:55,689 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,689 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,791 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,791 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,856 INFO    Thread-14 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt
2022-03-01 01:29:55,860 INFO    Thread-13 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json
2022-03-01 01:29:55,885 INFO    Thread-15 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml
2022-03-01 01:29:55,893 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,893 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,941 INFO    Thread-12 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:55,994 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,995 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:56,096 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:56,096 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:56,141 INFO    Thread-7  :236851 [sender.py:transition_state():392] send defer: 7
2022-03-01 01:29:56,142 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:56,142 INFO    HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 01:29:56,142 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:56,142 INFO    SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 01:29:56,188 INFO    SenderThread:236851 [sender.py:transition_state():392] send defer: 8
2022-03-01 01:29:56,188 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:56,188 INFO    HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 01:29:56,188 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:56,188 INFO    SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 01:29:56,189 INFO    SenderThread:236851 [sender.py:transition_state():392] send defer: 9
2022-03-01 01:29:56,189 DEBUG   SenderThread:236851 [sender.py:send():235] send: final
2022-03-01 01:29:56,189 DEBUG   SenderThread:236851 [sender.py:send():235] send: footer
2022-03-01 01:29:56,189 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:56,189 INFO    HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 01:29:56,190 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:56,190 INFO    SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 01:29:56,197 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:56,198 DEBUG   SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:56,198 INFO    SenderThread:236851 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:29:56,258 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 01:29:56,259 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 01:29:56,259 DEBUG   HandlerThread:236851 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 01:29:56,259 INFO    HandlerThread:236851 [handler.py:finish():739] shutting down handler
2022-03-01 01:29:57,190 INFO    WriterThread:236851 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/run-25hk0gke.wandb
2022-03-01 01:29:57,257 INFO    SenderThread:236851 [sender.py:finish():1075] shutting down sender
2022-03-01 01:29:57,257 INFO    SenderThread:236851 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:29:57,257 INFO    SenderThread:236851 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:29:57,259 INFO    MainThread:236851 [internal.py:handle_exit():79] Internal process exited