File size: 16,773 Bytes
d725b93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
2021-07-14 21:35:38,058 INFO    MainThread:582612 [internal.py:wandb_internal():88] W&B internal server running at pid: 582612, started at: 2021-07-14 21:35:38.058075
2021-07-14 21:35:38,060 INFO    WriterThread:582612 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/run-20l16od8.wandb
2021-07-14 21:35:38,060 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 21:35:38,061 DEBUG   SenderThread:582612 [sender.py:send():179] send: header
2021-07-14 21:35:38,061 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: check_version
2021-07-14 21:35:38,099 DEBUG   SenderThread:582612 [sender.py:send():179] send: run
2021-07-14 21:35:38,275 INFO    SenderThread:582612 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files
2021-07-14 21:35:38,275 INFO    SenderThread:582612 [sender.py:_start_run_threads():716] run started: 20l16od8 with start time 1626298537
2021-07-14 21:35:38,275 DEBUG   SenderThread:582612 [sender.py:send():179] send: summary
2021-07-14 21:35:38,275 INFO    SenderThread:582612 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:35:38,276 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 21:35:39,277 INFO    Thread-8  :582612 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-summary.json
2021-07-14 21:35:39,374 DEBUG   HandlerThread:582612 [meta.py:__init__():39] meta init
2021-07-14 21:35:39,374 DEBUG   HandlerThread:582612 [meta.py:__init__():53] meta init done
2021-07-14 21:35:39,374 DEBUG   HandlerThread:582612 [meta.py:probe():210] probe
2021-07-14 21:35:39,375 DEBUG   HandlerThread:582612 [meta.py:_setup_git():200] setup git
2021-07-14 21:35:39,404 DEBUG   HandlerThread:582612 [meta.py:_setup_git():207] setup git done
2021-07-14 21:35:39,405 DEBUG   HandlerThread:582612 [meta.py:_save_pip():57] save pip
2021-07-14 21:35:39,405 DEBUG   HandlerThread:582612 [meta.py:_save_pip():71] save pip done
2021-07-14 21:35:39,405 DEBUG   HandlerThread:582612 [meta.py:probe():252] probe done
2021-07-14 21:35:39,408 DEBUG   SenderThread:582612 [sender.py:send():179] send: files
2021-07-14 21:35:39,408 INFO    SenderThread:582612 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 21:35:39,414 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:35:39,415 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:35:39,542 DEBUG   SenderThread:582612 [sender.py:send():179] send: config
2021-07-14 21:35:39,542 DEBUG   SenderThread:582612 [sender.py:send():179] send: config
2021-07-14 21:35:39,542 DEBUG   SenderThread:582612 [sender.py:send():179] send: config
2021-07-14 21:35:39,845 INFO    Thread-11 :582612 [upload_job.py:push():137] Uploaded file /tmp/tmp81cfdra0wandb/3f4lkdmt-wandb-metadata.json
2021-07-14 21:35:40,276 INFO    Thread-8  :582612 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/requirements.txt
2021-07-14 21:35:40,276 INFO    Thread-8  :582612 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-metadata.json
2021-07-14 21:35:40,276 INFO    Thread-8  :582612 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:54,281 INFO    Thread-8  :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:54,573 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:35:54,574 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:35:56,282 INFO    Thread-8  :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:57,283 INFO    Thread-8  :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:58,079 DEBUG   SenderThread:582612 [sender.py:send():179] send: telemetry
2021-07-14 21:35:58,080 DEBUG   SenderThread:582612 [sender.py:send():179] send: exit
2021-07-14 21:35:58,080 INFO    SenderThread:582612 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 21:35:58,080 INFO    SenderThread:582612 [sender.py:send_exit():295] send defer
2021-07-14 21:35:58,080 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:58,080 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:58,081 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,081 INFO    HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 21:35:58,081 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,081 INFO    SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 21:35:58,081 INFO    SenderThread:582612 [sender.py:transition_state():308] send defer: 1
2021-07-14 21:35:58,081 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,081 INFO    HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 21:35:58,128 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,128 INFO    SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 21:35:58,128 INFO    SenderThread:582612 [sender.py:transition_state():308] send defer: 2
2021-07-14 21:35:58,128 DEBUG   SenderThread:582612 [sender.py:send():179] send: stats
2021-07-14 21:35:58,128 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,129 INFO    HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 21:35:58,129 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,129 INFO    SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 21:35:58,129 INFO    SenderThread:582612 [sender.py:transition_state():308] send defer: 3
2021-07-14 21:35:58,129 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,129 INFO    HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 21:35:58,129 DEBUG   SenderThread:582612 [sender.py:send():179] send: summary
2021-07-14 21:35:58,130 INFO    SenderThread:582612 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:35:58,130 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,130 INFO    SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 21:35:58,130 INFO    SenderThread:582612 [sender.py:transition_state():308] send defer: 4
2021-07-14 21:35:58,130 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,130 INFO    HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 21:35:58,130 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,130 INFO    SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 21:35:58,183 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:58,299 INFO    SenderThread:582612 [sender.py:transition_state():308] send defer: 5
2021-07-14 21:35:58,299 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:58,300 INFO    Thread-8  :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:58,300 INFO    Thread-8  :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-summary.json
2021-07-14 21:35:58,300 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:58,300 INFO    HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 21:35:58,300 INFO    Thread-8  :582612 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/config.yaml
2021-07-14 21:35:58,301 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:58,301 INFO    SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 21:35:58,301 INFO    SenderThread:582612 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 21:35:58,402 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,301 INFO    SenderThread:582612 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files
2021-07-14 21:35:59,302 INFO    SenderThread:582612 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/requirements.txt requirements.txt
2021-07-14 21:35:59,302 INFO    SenderThread:582612 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log output.log
2021-07-14 21:35:59,302 INFO    SenderThread:582612 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-metadata.json wandb-metadata.json
2021-07-14 21:35:59,302 INFO    SenderThread:582612 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/config.yaml config.yaml
2021-07-14 21:35:59,302 INFO    SenderThread:582612 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-summary.json wandb-summary.json
2021-07-14 21:35:59,302 INFO    SenderThread:582612 [sender.py:transition_state():308] send defer: 6
2021-07-14 21:35:59,303 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,303 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:59,303 INFO    HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 21:35:59,307 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:59,307 INFO    SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 21:35:59,307 INFO    SenderThread:582612 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:35:59,405 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,405 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,507 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,507 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,609 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,609 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,711 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,711 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,738 INFO    Thread-15 :582612 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/wandb-summary.json
2021-07-14 21:35:59,759 INFO    Thread-13 :582612 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/output.log
2021-07-14 21:35:59,764 INFO    Thread-14 :582612 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/config.yaml
2021-07-14 21:35:59,779 INFO    Thread-12 :582612 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/files/requirements.txt
2021-07-14 21:35:59,813 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,813 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,915 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:35:59,915 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:35:59,979 INFO    Thread-7  :582612 [sender.py:transition_state():308] send defer: 7
2021-07-14 21:35:59,980 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:35:59,980 INFO    HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 21:35:59,980 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:35:59,980 INFO    SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 21:36:00,016 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:36:00,132 INFO    SenderThread:582612 [sender.py:transition_state():308] send defer: 8
2021-07-14 21:36:00,132 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:36:00,133 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:36:00,133 INFO    HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 21:36:00,133 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:36:00,133 INFO    SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 21:36:00,133 INFO    SenderThread:582612 [sender.py:transition_state():308] send defer: 9
2021-07-14 21:36:00,134 DEBUG   SenderThread:582612 [sender.py:send():179] send: final
2021-07-14 21:36:00,134 DEBUG   SenderThread:582612 [sender.py:send():179] send: footer
2021-07-14 21:36:00,134 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:36:00,134 INFO    HandlerThread:582612 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 21:36:00,134 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: defer
2021-07-14 21:36:00,134 INFO    SenderThread:582612 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 21:36:00,234 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:36:00,235 DEBUG   SenderThread:582612 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:36:00,235 INFO    SenderThread:582612 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 21:36:00,236 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 21:36:00,237 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 21:36:00,237 DEBUG   HandlerThread:582612 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 21:36:00,237 INFO    HandlerThread:582612 [handler.py:finish():638] shutting down handler
2021-07-14 21:36:01,134 INFO    WriterThread:582612 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_213537-20l16od8/run-20l16od8.wandb
2021-07-14 21:36:01,235 INFO    SenderThread:582612 [sender.py:finish():945] shutting down sender
2021-07-14 21:36:01,235 INFO    SenderThread:582612 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:36:01,235 INFO    SenderThread:582612 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 21:36:01,237 INFO    MainThread:582612 [internal.py:handle_exit():78] Internal process exited