File size: 17,050 Bytes
9e8de2e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
2022-03-10 21:09:51,233 INFO    MainThread:1044076 [internal.py:wandb_internal():89] W&B internal server running at pid: 1044076, started at: 2022-03-10 21:09:51.233570
2022-03-10 21:09:51,235 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: check_version
2022-03-10 21:09:51,235 INFO    WriterThread:1044076 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/run-19sg19ou.wandb
2022-03-10 21:09:51,237 DEBUG   SenderThread:1044076 [sender.py:send():235] send: header
2022-03-10 21:09:51,237 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: check_version
2022-03-10 21:09:51,306 DEBUG   SenderThread:1044076 [sender.py:send():235] send: run
2022-03-10 21:09:51,397 INFO    SenderThread:1044076 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files
2022-03-10 21:09:51,397 INFO    SenderThread:1044076 [sender.py:_start_run_threads():809] run started: 19sg19ou with start time 1646946590
2022-03-10 21:09:51,398 DEBUG   SenderThread:1044076 [sender.py:send():235] send: summary
2022-03-10 21:09:51,398 INFO    SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-10 21:09:51,398 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: run_start
2022-03-10 21:09:51,402 DEBUG   HandlerThread:1044076 [meta.py:__init__():36] meta init
2022-03-10 21:09:51,403 DEBUG   HandlerThread:1044076 [meta.py:__init__():50] meta init done
2022-03-10 21:09:51,403 DEBUG   HandlerThread:1044076 [meta.py:probe():210] probe
2022-03-10 21:09:51,410 DEBUG   HandlerThread:1044076 [meta.py:_setup_git():200] setup git
2022-03-10 21:09:51,425 DEBUG   HandlerThread:1044076 [meta.py:_setup_git():207] setup git done
2022-03-10 21:09:51,426 DEBUG   HandlerThread:1044076 [meta.py:_save_pip():54] save pip
2022-03-10 21:09:51,426 DEBUG   HandlerThread:1044076 [meta.py:_save_pip():68] save pip done
2022-03-10 21:09:51,426 DEBUG   HandlerThread:1044076 [meta.py:probe():248] probe done
2022-03-10 21:09:51,505 DEBUG   SenderThread:1044076 [sender.py:send():235] send: files
2022-03-10 21:09:51,505 INFO    SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-10 21:09:51,509 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: stop_status
2022-03-10 21:09:51,510 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: stop_status
2022-03-10 21:09:51,544 DEBUG   SenderThread:1044076 [sender.py:send():235] send: config
2022-03-10 21:09:51,545 DEBUG   SenderThread:1044076 [sender.py:send():235] send: metric
2022-03-10 21:09:51,546 DEBUG   SenderThread:1044076 [sender.py:send():235] send: metric
2022-03-10 21:09:51,546 WARNING SenderThread:1044076 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-10 21:09:51,790 INFO    Thread-11 :1044076 [upload_job.py:push():137] Uploaded file /tmp/tmp0u4vh57hwandb/2s9kjsb8-wandb-metadata.json
2022-03-10 21:09:52,400 INFO    Thread-8  :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json
2022-03-10 21:09:52,401 INFO    Thread-8  :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/requirements.txt
2022-03-10 21:09:52,401 INFO    Thread-8  :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log
2022-03-10 21:09:52,401 INFO    Thread-8  :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-metadata.json
2022-03-10 21:09:54,398 INFO    Thread-8  :1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log
2022-03-10 21:09:58,400 INFO    Thread-8  :1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log
2022-03-10 21:09:59,980 DEBUG   SenderThread:1044076 [sender.py:send():235] send: telemetry
2022-03-10 21:09:59,980 DEBUG   SenderThread:1044076 [sender.py:send():235] send: exit
2022-03-10 21:09:59,980 INFO    SenderThread:1044076 [sender.py:send_exit():371] handling exit code: 1
2022-03-10 21:09:59,981 INFO    SenderThread:1044076 [sender.py:send_exit():373] handling runtime: 8
2022-03-10 21:09:59,981 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:09:59,981 INFO    SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-10 21:09:59,982 INFO    SenderThread:1044076 [sender.py:send_exit():379] send defer
2022-03-10 21:09:59,982 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:09:59,982 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:09:59,982 INFO    HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-10 21:09:59,983 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:09:59,983 INFO    SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-10 21:09:59,983 INFO    SenderThread:1044076 [sender.py:transition_state():392] send defer: 1
2022-03-10 21:09:59,983 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:09:59,983 INFO    HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-10 21:10:00,043 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,043 INFO    SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-10 21:10:00,043 INFO    SenderThread:1044076 [sender.py:transition_state():392] send defer: 2
2022-03-10 21:10:00,043 DEBUG   SenderThread:1044076 [sender.py:send():235] send: stats
2022-03-10 21:10:00,044 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,044 INFO    HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-10 21:10:00,044 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,044 INFO    SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-10 21:10:00,044 INFO    SenderThread:1044076 [sender.py:transition_state():392] send defer: 3
2022-03-10 21:10:00,044 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,045 INFO    HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-10 21:10:00,045 DEBUG   SenderThread:1044076 [sender.py:send():235] send: summary
2022-03-10 21:10:00,045 INFO    SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-10 21:10:00,045 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,045 INFO    SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-10 21:10:00,045 INFO    SenderThread:1044076 [sender.py:transition_state():392] send defer: 4
2022-03-10 21:10:00,046 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,046 INFO    HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-10 21:10:00,046 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,046 INFO    SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-10 21:10:00,083 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,172 INFO    SenderThread:1044076 [sender.py:transition_state():392] send defer: 5
2022-03-10 21:10:00,172 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,173 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,173 INFO    HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-10 21:10:00,173 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,173 INFO    SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-10 21:10:00,173 INFO    SenderThread:1044076 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-10 21:10:00,274 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,401 INFO    Thread-8  :1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json
2022-03-10 21:10:00,401 INFO    SenderThread:1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/config.yaml
2022-03-10 21:10:00,401 INFO    SenderThread:1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log
2022-03-10 21:10:00,402 INFO    SenderThread:1044076 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files
2022-03-10 21:10:00,402 INFO    SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-metadata.json wandb-metadata.json
2022-03-10 21:10:00,402 INFO    SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log output.log
2022-03-10 21:10:00,402 INFO    SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json wandb-summary.json
2022-03-10 21:10:00,402 INFO    SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/requirements.txt requirements.txt
2022-03-10 21:10:00,405 INFO    SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/config.yaml config.yaml
2022-03-10 21:10:00,408 INFO    SenderThread:1044076 [sender.py:transition_state():392] send defer: 6
2022-03-10 21:10:00,408 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,414 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,414 INFO    HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-10 21:10:00,415 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,415 INFO    SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-10 21:10:00,415 INFO    SenderThread:1044076 [file_pusher.py:finish():177] shutting down file pusher
2022-03-10 21:10:00,510 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,510 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,612 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,612 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,664 INFO    Thread-12 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log
2022-03-10 21:10:00,672 INFO    Thread-15 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/config.yaml
2022-03-10 21:10:00,688 INFO    Thread-13 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json
2022-03-10 21:10:00,699 INFO    Thread-14 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/requirements.txt
2022-03-10 21:10:00,713 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,714 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,815 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,815 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,899 INFO    Thread-7  :1044076 [sender.py:transition_state():392] send defer: 7
2022-03-10 21:10:00,900 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,900 INFO    HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-10 21:10:00,900 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,900 INFO    SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-10 21:10:00,916 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:01,040 INFO    SenderThread:1044076 [sender.py:transition_state():392] send defer: 8
2022-03-10 21:10:01,040 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:01,041 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:01,041 INFO    HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-10 21:10:01,041 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:01,041 INFO    SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-10 21:10:01,041 INFO    SenderThread:1044076 [sender.py:transition_state():392] send defer: 9
2022-03-10 21:10:01,042 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:01,042 DEBUG   SenderThread:1044076 [sender.py:send():235] send: final
2022-03-10 21:10:01,042 INFO    HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-10 21:10:01,043 DEBUG   SenderThread:1044076 [sender.py:send():235] send: footer
2022-03-10 21:10:01,043 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:01,043 INFO    SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-10 21:10:01,142 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:01,142 DEBUG   SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:01,142 INFO    SenderThread:1044076 [file_pusher.py:join():182] waiting for file pusher
2022-03-10 21:10:01,199 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: get_summary
2022-03-10 21:10:01,200 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-10 21:10:01,201 DEBUG   HandlerThread:1044076 [handler.py:handle_request():131] handle_request: shutdown
2022-03-10 21:10:01,201 INFO    HandlerThread:1044076 [handler.py:finish():739] shutting down handler
2022-03-10 21:10:02,042 INFO    WriterThread:1044076 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/run-19sg19ou.wandb
2022-03-10 21:10:02,198 INFO    SenderThread:1044076 [sender.py:finish():1075] shutting down sender
2022-03-10 21:10:02,198 INFO    SenderThread:1044076 [file_pusher.py:finish():177] shutting down file pusher
2022-03-10 21:10:02,198 INFO    SenderThread:1044076 [file_pusher.py:join():182] waiting for file pusher
2022-03-10 21:10:02,200 INFO    MainThread:1044076 [internal.py:handle_exit():79] Internal process exited