File size: 16,912 Bytes
8319971
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
2022-05-07 17:41:49,607 INFO    MainThread:14620 [internal.py:wandb_internal():92] W&B internal server running at pid: 14620, started at: 2022-05-07 17:41:49.606834
2022-05-07 17:41:49,609 INFO    WriterThread:14620 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/run-2l3y3icw.wandb
2022-05-07 17:41:49,610 DEBUG   SenderThread:14620 [sender.py:send():235] send: header
2022-05-07 17:41:49,610 DEBUG   SenderThread:14620 [sender.py:send():235] send: run
2022-05-07 17:41:49,678 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: check_version
2022-05-07 17:41:49,678 INFO    SenderThread:14620 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files
2022-05-07 17:41:49,678 INFO    SenderThread:14620 [sender.py:_start_run_threads():812] run started: 2l3y3icw with start time 1651945308
2022-05-07 17:41:49,678 DEBUG   SenderThread:14620 [sender.py:send():235] send: summary
2022-05-07 17:41:49,679 INFO    SenderThread:14620 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 17:41:49,679 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: check_version
2022-05-07 17:41:49,739 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: run_start
2022-05-07 17:41:50,689 INFO    Thread-7  :14620 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-summary.json
2022-05-07 17:41:51,879 DEBUG   HandlerThread:14620 [meta.py:__init__():37] meta init
2022-05-07 17:41:51,879 DEBUG   HandlerThread:14620 [meta.py:__init__():51] meta init done
2022-05-07 17:41:51,879 DEBUG   HandlerThread:14620 [meta.py:probe():211] probe
2022-05-07 17:41:51,880 DEBUG   HandlerThread:14620 [meta.py:_setup_git():201] setup git
2022-05-07 17:41:51,909 DEBUG   HandlerThread:14620 [meta.py:_setup_git():208] setup git done
2022-05-07 17:41:51,909 DEBUG   HandlerThread:14620 [meta.py:_save_pip():55] save pip
2022-05-07 17:41:51,910 DEBUG   HandlerThread:14620 [meta.py:_save_pip():69] save pip done
2022-05-07 17:41:51,910 DEBUG   HandlerThread:14620 [meta.py:probe():249] probe done
2022-05-07 17:41:51,912 DEBUG   SenderThread:14620 [sender.py:send():235] send: files
2022-05-07 17:41:51,913 INFO    SenderThread:14620 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-05-07 17:41:51,918 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 17:41:51,918 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: stop_status
2022-05-07 17:41:51,948 DEBUG   SenderThread:14620 [sender.py:send():235] send: telemetry
2022-05-07 17:41:52,187 INFO    Thread-11 :14620 [upload_job.py:push():137] Uploaded file /tmp/tmpandpth3dwandb/1ih5opi9-wandb-metadata.json
2022-05-07 17:41:52,688 INFO    Thread-7  :14620 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/requirements.txt
2022-05-07 17:41:52,689 INFO    Thread-7  :14620 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log
2022-05-07 17:41:52,689 INFO    Thread-7  :14620 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-metadata.json
2022-05-07 17:41:53,838 DEBUG   SenderThread:14620 [sender.py:send():235] send: telemetry
2022-05-07 17:41:53,838 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:41:54,689 INFO    Thread-7  :14620 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log
2022-05-07 17:41:55,690 INFO    Thread-7  :14620 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log
2022-05-07 17:41:56,439 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,439 DEBUG   SenderThread:14620 [sender.py:send():235] send: exit
2022-05-07 17:41:56,439 INFO    SenderThread:14620 [sender.py:send_exit():371] handling exit code: 1
2022-05-07 17:41:56,439 INFO    SenderThread:14620 [sender.py:send_exit():373] handling runtime: 6
2022-05-07 17:41:56,440 INFO    SenderThread:14620 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 17:41:56,440 INFO    SenderThread:14620 [sender.py:send_exit():379] send defer
2022-05-07 17:41:56,440 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:56,441 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,441 INFO    HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 0
2022-05-07 17:41:56,441 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,441 INFO    SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 0
2022-05-07 17:41:56,441 INFO    SenderThread:14620 [sender.py:transition_state():392] send defer: 1
2022-05-07 17:41:56,441 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,441 INFO    HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 1
2022-05-07 17:41:56,501 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,501 INFO    SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 1
2022-05-07 17:41:56,501 INFO    SenderThread:14620 [sender.py:transition_state():392] send defer: 2
2022-05-07 17:41:56,501 DEBUG   SenderThread:14620 [sender.py:send():235] send: stats
2022-05-07 17:41:56,502 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,502 INFO    HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 2
2022-05-07 17:41:56,502 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,502 INFO    SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 2
2022-05-07 17:41:56,502 INFO    SenderThread:14620 [sender.py:transition_state():392] send defer: 3
2022-05-07 17:41:56,502 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,502 INFO    HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 3
2022-05-07 17:41:56,503 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,503 INFO    SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 3
2022-05-07 17:41:56,503 INFO    SenderThread:14620 [sender.py:transition_state():392] send defer: 4
2022-05-07 17:41:56,503 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,503 INFO    HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 4
2022-05-07 17:41:56,503 DEBUG   SenderThread:14620 [sender.py:send():235] send: summary
2022-05-07 17:41:56,503 INFO    SenderThread:14620 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 17:41:56,504 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,504 INFO    SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 4
2022-05-07 17:41:56,504 INFO    SenderThread:14620 [sender.py:transition_state():392] send defer: 5
2022-05-07 17:41:56,504 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,504 INFO    HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 5
2022-05-07 17:41:56,504 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,504 INFO    SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 5
2022-05-07 17:41:56,542 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,563 INFO    SenderThread:14620 [sender.py:transition_state():392] send defer: 6
2022-05-07 17:41:56,563 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:56,564 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,564 INFO    HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 6
2022-05-07 17:41:56,564 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,564 INFO    SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 6
2022-05-07 17:41:56,564 INFO    SenderThread:14620 [dir_watcher.py:finish():283] shutting down directory watcher
2022-05-07 17:41:56,665 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,690 INFO    Thread-7  :14620 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-summary.json
2022-05-07 17:41:56,691 INFO    SenderThread:14620 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/config.yaml
2022-05-07 17:41:56,691 INFO    SenderThread:14620 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log
2022-05-07 17:41:56,691 INFO    SenderThread:14620 [dir_watcher.py:finish():313] scan: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files
2022-05-07 17:41:56,691 INFO    SenderThread:14620 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/requirements.txt requirements.txt
2022-05-07 17:41:56,691 INFO    SenderThread:14620 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log output.log
2022-05-07 17:41:56,692 INFO    SenderThread:14620 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-metadata.json wandb-metadata.json
2022-05-07 17:41:56,692 INFO    SenderThread:14620 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/config.yaml config.yaml
2022-05-07 17:41:56,692 INFO    SenderThread:14620 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-summary.json wandb-summary.json
2022-05-07 17:41:56,692 INFO    SenderThread:14620 [sender.py:transition_state():392] send defer: 7
2022-05-07 17:41:56,693 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:56,693 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,693 INFO    HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 7
2022-05-07 17:41:56,696 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,696 INFO    SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 7
2022-05-07 17:41:56,696 INFO    SenderThread:14620 [file_pusher.py:finish():145] shutting down file pusher
2022-05-07 17:41:56,794 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,794 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:56,896 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,896 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:56,940 INFO    Thread-13 :14620 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log
2022-05-07 17:41:56,947 INFO    Thread-14 :14620 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/config.yaml
2022-05-07 17:41:56,969 INFO    Thread-15 :14620 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-summary.json
2022-05-07 17:41:56,988 INFO    Thread-12 :14620 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/requirements.txt
2022-05-07 17:41:56,998 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,998 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:57,100 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:57,100 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:57,189 INFO    Thread-6  :14620 [sender.py:transition_state():392] send defer: 8
2022-05-07 17:41:57,189 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:57,189 INFO    HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 8
2022-05-07 17:41:57,189 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:57,189 INFO    SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 8
2022-05-07 17:41:57,202 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:59,454 INFO    SenderThread:14620 [sender.py:transition_state():392] send defer: 9
2022-05-07 17:41:59,455 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:59,455 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:59,455 INFO    HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 9
2022-05-07 17:41:59,455 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:59,456 INFO    SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 9
2022-05-07 17:41:59,456 INFO    SenderThread:14620 [sender.py:transition_state():392] send defer: 10
2022-05-07 17:41:59,456 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:59,456 INFO    HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 10
2022-05-07 17:41:59,456 DEBUG   SenderThread:14620 [sender.py:send():235] send: final
2022-05-07 17:41:59,456 DEBUG   SenderThread:14620 [sender.py:send():235] send: footer
2022-05-07 17:41:59,456 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:59,456 INFO    SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 10
2022-05-07 17:41:59,556 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:59,557 DEBUG   SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:59,557 INFO    SenderThread:14620 [file_pusher.py:join():150] waiting for file pusher
2022-05-07 17:42:00,139 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: sampled_history
2022-05-07 17:42:00,140 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: get_summary
2022-05-07 17:42:00,141 DEBUG   HandlerThread:14620 [handler.py:handle_request():141] handle_request: shutdown
2022-05-07 17:42:00,141 INFO    HandlerThread:14620 [handler.py:finish():778] shutting down handler
2022-05-07 17:42:00,457 INFO    WriterThread:14620 [datastore.py:close():281] close: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/run-2l3y3icw.wandb
2022-05-07 17:42:01,038 INFO    SenderThread:14620 [sender.py:finish():1078] shutting down sender
2022-05-07 17:42:01,038 INFO    SenderThread:14620 [file_pusher.py:finish():145] shutting down file pusher
2022-05-07 17:42:01,038 INFO    SenderThread:14620 [file_pusher.py:join():150] waiting for file pusher
2022-05-07 17:42:01,040 INFO    MainThread:14620 [internal.py:handle_exit():82] Internal process exited