File size: 16,393 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
2021-07-14 20:01:08,001 INFO    MainThread:564646 [internal.py:wandb_internal():88] W&B internal server running at pid: 564646, started at: 2021-07-14 20:01:08.000744
2021-07-14 20:01:08,003 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 20:01:08,003 INFO    WriterThread:564646 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/run-3qnuldhi.wandb
2021-07-14 20:01:08,004 DEBUG   SenderThread:564646 [sender.py:send():179] send: header
2021-07-14 20:01:08,004 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: check_version
2021-07-14 20:01:08,042 DEBUG   SenderThread:564646 [sender.py:send():179] send: run
2021-07-14 20:01:08,213 INFO    SenderThread:564646 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files
2021-07-14 20:01:08,213 INFO    SenderThread:564646 [sender.py:_start_run_threads():716] run started: 3qnuldhi with start time 1626292867
2021-07-14 20:01:08,213 DEBUG   SenderThread:564646 [sender.py:send():179] send: summary
2021-07-14 20:01:08,214 INFO    SenderThread:564646 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:01:08,214 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 20:01:09,216 INFO    Thread-8  :564646 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-summary.json
2021-07-14 20:01:09,340 DEBUG   HandlerThread:564646 [meta.py:__init__():39] meta init
2021-07-14 20:01:09,340 DEBUG   HandlerThread:564646 [meta.py:__init__():53] meta init done
2021-07-14 20:01:09,340 DEBUG   HandlerThread:564646 [meta.py:probe():210] probe
2021-07-14 20:01:09,342 DEBUG   HandlerThread:564646 [meta.py:_setup_git():200] setup git
2021-07-14 20:01:09,376 DEBUG   HandlerThread:564646 [meta.py:_setup_git():207] setup git done
2021-07-14 20:01:09,376 DEBUG   HandlerThread:564646 [meta.py:_save_pip():57] save pip
2021-07-14 20:01:09,377 DEBUG   HandlerThread:564646 [meta.py:_save_pip():71] save pip done
2021-07-14 20:01:09,377 DEBUG   HandlerThread:564646 [meta.py:probe():252] probe done
2021-07-14 20:01:09,380 DEBUG   SenderThread:564646 [sender.py:send():179] send: files
2021-07-14 20:01:09,381 INFO    SenderThread:564646 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 20:01:09,388 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:01:09,388 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:01:09,525 DEBUG   SenderThread:564646 [sender.py:send():179] send: config
2021-07-14 20:01:09,526 DEBUG   SenderThread:564646 [sender.py:send():179] send: config
2021-07-14 20:01:09,526 DEBUG   SenderThread:564646 [sender.py:send():179] send: config
2021-07-14 20:01:09,828 INFO    Thread-11 :564646 [upload_job.py:push():137] Uploaded file /tmp/tmp6zvg4jijwandb/288ietle-wandb-metadata.json
2021-07-14 20:01:10,214 INFO    Thread-8  :564646 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/requirements.txt
2021-07-14 20:01:10,215 INFO    Thread-8  :564646 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/output.log
2021-07-14 20:01:10,215 INFO    Thread-8  :564646 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-metadata.json
2021-07-14 20:01:24,220 INFO    Thread-8  :564646 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/output.log
2021-07-14 20:01:24,538 DEBUG   SenderThread:564646 [sender.py:send():179] send: telemetry
2021-07-14 20:01:24,539 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:24,539 DEBUG   SenderThread:564646 [sender.py:send():179] send: exit
2021-07-14 20:01:24,539 INFO    SenderThread:564646 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 20:01:24,539 INFO    SenderThread:564646 [sender.py:send_exit():295] send defer
2021-07-14 20:01:24,540 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:24,540 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,540 INFO    HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 20:01:24,541 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,541 INFO    SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 20:01:24,541 INFO    SenderThread:564646 [sender.py:transition_state():308] send defer: 1
2021-07-14 20:01:24,541 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,541 INFO    HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 20:01:24,590 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,590 INFO    SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 20:01:24,590 INFO    SenderThread:564646 [sender.py:transition_state():308] send defer: 2
2021-07-14 20:01:24,590 DEBUG   SenderThread:564646 [sender.py:send():179] send: stats
2021-07-14 20:01:24,591 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,591 INFO    HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 20:01:24,591 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,591 INFO    SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 20:01:24,591 INFO    SenderThread:564646 [sender.py:transition_state():308] send defer: 3
2021-07-14 20:01:24,591 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,591 INFO    HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 20:01:24,592 DEBUG   SenderThread:564646 [sender.py:send():179] send: summary
2021-07-14 20:01:24,592 INFO    SenderThread:564646 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:01:24,592 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,592 INFO    SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 20:01:24,592 INFO    SenderThread:564646 [sender.py:transition_state():308] send defer: 4
2021-07-14 20:01:24,593 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,593 INFO    HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 20:01:24,593 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,593 INFO    SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 20:01:24,642 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:24,785 INFO    SenderThread:564646 [sender.py:transition_state():308] send defer: 5
2021-07-14 20:01:24,786 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:24,786 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:24,786 INFO    HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 20:01:24,787 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:24,787 INFO    SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 20:01:24,787 INFO    SenderThread:564646 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 20:01:24,888 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,221 INFO    Thread-8  :564646 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-summary.json
2021-07-14 20:01:25,221 INFO    SenderThread:564646 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/config.yaml
2021-07-14 20:01:25,222 INFO    SenderThread:564646 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/output.log
2021-07-14 20:01:25,222 INFO    SenderThread:564646 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files
2021-07-14 20:01:25,222 INFO    SenderThread:564646 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/requirements.txt requirements.txt
2021-07-14 20:01:25,222 INFO    SenderThread:564646 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/output.log output.log
2021-07-14 20:01:25,223 INFO    SenderThread:564646 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-metadata.json wandb-metadata.json
2021-07-14 20:01:25,223 INFO    SenderThread:564646 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/config.yaml config.yaml
2021-07-14 20:01:25,223 INFO    SenderThread:564646 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-summary.json wandb-summary.json
2021-07-14 20:01:25,223 INFO    SenderThread:564646 [sender.py:transition_state():308] send defer: 6
2021-07-14 20:01:25,223 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,227 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:25,227 INFO    HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 20:01:25,230 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:25,230 INFO    SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 20:01:25,230 INFO    SenderThread:564646 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 20:01:25,328 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,328 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,430 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,431 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,533 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,533 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,635 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,635 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,672 INFO    Thread-14 :564646 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/config.yaml
2021-07-14 20:01:25,678 INFO    Thread-15 :564646 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/wandb-summary.json
2021-07-14 20:01:25,692 INFO    Thread-13 :564646 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/output.log
2021-07-14 20:01:25,737 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,737 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,743 INFO    Thread-12 :564646 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/files/requirements.txt
2021-07-14 20:01:25,839 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,839 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,941 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:25,941 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:25,943 INFO    Thread-7  :564646 [sender.py:transition_state():308] send defer: 7
2021-07-14 20:01:25,944 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:25,944 INFO    HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 20:01:25,944 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:25,944 INFO    SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 20:01:26,043 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:26,074 INFO    SenderThread:564646 [sender.py:transition_state():308] send defer: 8
2021-07-14 20:01:26,074 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:26,075 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:26,075 INFO    HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 20:01:26,075 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:26,075 INFO    SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 20:01:26,075 INFO    SenderThread:564646 [sender.py:transition_state():308] send defer: 9
2021-07-14 20:01:26,076 DEBUG   SenderThread:564646 [sender.py:send():179] send: final
2021-07-14 20:01:26,076 DEBUG   SenderThread:564646 [sender.py:send():179] send: footer
2021-07-14 20:01:26,076 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: defer
2021-07-14 20:01:26,076 INFO    HandlerThread:564646 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 20:01:26,076 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: defer
2021-07-14 20:01:26,076 INFO    SenderThread:564646 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 20:01:26,176 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 20:01:26,177 DEBUG   SenderThread:564646 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 20:01:26,177 INFO    SenderThread:564646 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 20:01:26,178 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 20:01:26,179 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 20:01:26,180 DEBUG   HandlerThread:564646 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 20:01:26,180 INFO    HandlerThread:564646 [handler.py:finish():638] shutting down handler
2021-07-14 20:01:27,076 INFO    WriterThread:564646 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_200107-3qnuldhi/run-3qnuldhi.wandb
2021-07-14 20:01:27,177 INFO    SenderThread:564646 [sender.py:finish():945] shutting down sender
2021-07-14 20:01:27,177 INFO    SenderThread:564646 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 20:01:27,177 INFO    SenderThread:564646 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 20:01:27,180 INFO    MainThread:564646 [internal.py:handle_exit():78] Internal process exited