File size: 19,385 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
2021-07-13 12:47:49,474 INFO    MainThread:355550 [internal.py:wandb_internal():88] W&B internal server running at pid: 355550, started at: 2021-07-13 12:47:49.474151
2021-07-13 12:47:49,476 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 12:47:49,477 INFO    WriterThread:355550 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/run-vubzdnhu.wandb
2021-07-13 12:47:49,477 DEBUG   SenderThread:355550 [sender.py:send():179] send: header
2021-07-13 12:47:49,477 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: check_version
2021-07-13 12:47:49,517 DEBUG   SenderThread:355550 [sender.py:send():179] send: run
2021-07-13 12:47:49,718 INFO    SenderThread:355550 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files
2021-07-13 12:47:49,718 INFO    SenderThread:355550 [sender.py:_start_run_threads():716] run started: vubzdnhu with start time 1626180468
2021-07-13 12:47:49,718 DEBUG   SenderThread:355550 [sender.py:send():179] send: summary
2021-07-13 12:47:49,718 INFO    SenderThread:355550 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 12:47:49,719 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 12:47:50,719 INFO    Thread-8  :355550 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-summary.json
2021-07-13 12:47:50,855 DEBUG   HandlerThread:355550 [meta.py:__init__():39] meta init
2021-07-13 12:47:50,855 DEBUG   HandlerThread:355550 [meta.py:__init__():53] meta init done
2021-07-13 12:47:50,855 DEBUG   HandlerThread:355550 [meta.py:probe():210] probe
2021-07-13 12:47:50,856 DEBUG   HandlerThread:355550 [meta.py:_setup_git():200] setup git
2021-07-13 12:47:50,887 DEBUG   HandlerThread:355550 [meta.py:_setup_git():207] setup git done
2021-07-13 12:47:50,888 DEBUG   HandlerThread:355550 [meta.py:_save_pip():57] save pip
2021-07-13 12:47:50,888 DEBUG   HandlerThread:355550 [meta.py:_save_pip():71] save pip done
2021-07-13 12:47:50,888 DEBUG   HandlerThread:355550 [meta.py:probe():252] probe done
2021-07-13 12:47:50,892 DEBUG   SenderThread:355550 [sender.py:send():179] send: files
2021-07-13 12:47:50,892 INFO    SenderThread:355550 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 12:47:50,899 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:47:50,899 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:47:51,026 DEBUG   SenderThread:355550 [sender.py:send():179] send: config
2021-07-13 12:47:51,027 DEBUG   SenderThread:355550 [sender.py:send():179] send: config
2021-07-13 12:47:51,027 DEBUG   SenderThread:355550 [sender.py:send():179] send: config
2021-07-13 12:47:51,378 INFO    Thread-11 :355550 [upload_job.py:push():137] Uploaded file /tmp/tmpy_8wrz9rwandb/2yxmm3l6-wandb-metadata.json
2021-07-13 12:47:51,719 INFO    Thread-8  :355550 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/requirements.txt
2021-07-13 12:47:51,719 INFO    Thread-8  :355550 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:47:51,719 INFO    Thread-8  :355550 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-metadata.json
2021-07-13 12:48:05,724 INFO    Thread-8  :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:48:06,028 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:48:06,029 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:48:07,724 INFO    Thread-8  :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:48:18,943 DEBUG   SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:48:20,728 INFO    Thread-8  :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/config.yaml
2021-07-13 12:48:21,161 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:48:21,161 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:48:35,733 INFO    Thread-8  :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:48:36,292 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:48:36,292 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:48:49,027 DEBUG   SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:48:51,451 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:48:51,451 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:49:06,603 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:49:06,604 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:49:19,111 DEBUG   SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:49:21,736 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:49:21,737 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:49:36,867 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:49:36,868 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:49:49,193 DEBUG   SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:49:51,999 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:49:52,000 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:50:07,132 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 12:50:07,132 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: stop_status
2021-07-13 12:50:19,272 DEBUG   SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:50:19,770 INFO    Thread-8  :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:50:20,563 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:20,564 DEBUG   SenderThread:355550 [sender.py:send():179] send: telemetry
2021-07-13 12:50:20,564 DEBUG   SenderThread:355550 [sender.py:send():179] send: exit
2021-07-13 12:50:20,564 INFO    SenderThread:355550 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 12:50:20,565 INFO    SenderThread:355550 [sender.py:send_exit():295] send defer
2021-07-13 12:50:20,565 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:20,566 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,566 INFO    HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 12:50:20,566 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,567 INFO    SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 12:50:20,567 INFO    SenderThread:355550 [sender.py:transition_state():308] send defer: 1
2021-07-13 12:50:20,567 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,567 INFO    HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 12:50:20,576 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,576 INFO    SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 12:50:20,576 INFO    SenderThread:355550 [sender.py:transition_state():308] send defer: 2
2021-07-13 12:50:20,576 DEBUG   SenderThread:355550 [sender.py:send():179] send: stats
2021-07-13 12:50:20,577 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,577 INFO    HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 12:50:20,577 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,577 INFO    SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 12:50:20,577 INFO    SenderThread:355550 [sender.py:transition_state():308] send defer: 3
2021-07-13 12:50:20,577 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,577 INFO    HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 12:50:20,578 DEBUG   SenderThread:355550 [sender.py:send():179] send: summary
2021-07-13 12:50:20,578 INFO    SenderThread:355550 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 12:50:20,578 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,578 INFO    SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 12:50:20,578 INFO    SenderThread:355550 [sender.py:transition_state():308] send defer: 4
2021-07-13 12:50:20,579 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,579 INFO    HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 12:50:20,579 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,579 INFO    SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 12:50:20,668 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:20,746 INFO    SenderThread:355550 [sender.py:transition_state():308] send defer: 5
2021-07-13 12:50:20,746 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:20,746 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,746 INFO    HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 12:50:20,747 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,747 INFO    SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 12:50:20,747 INFO    SenderThread:355550 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 12:50:20,771 INFO    Thread-8  :355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/config.yaml
2021-07-13 12:50:20,771 INFO    SenderThread:355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:50:20,771 INFO    SenderThread:355550 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-summary.json
2021-07-13 12:50:20,772 INFO    SenderThread:355550 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files
2021-07-13 12:50:20,772 INFO    SenderThread:355550 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/requirements.txt requirements.txt
2021-07-13 12:50:20,772 INFO    SenderThread:355550 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log output.log
2021-07-13 12:50:20,772 INFO    SenderThread:355550 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-metadata.json wandb-metadata.json
2021-07-13 12:50:20,772 INFO    SenderThread:355550 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/config.yaml config.yaml
2021-07-13 12:50:20,772 INFO    SenderThread:355550 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-summary.json wandb-summary.json
2021-07-13 12:50:20,775 INFO    SenderThread:355550 [sender.py:transition_state():308] send defer: 6
2021-07-13 12:50:20,777 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:20,778 INFO    HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 12:50:20,784 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:20,784 INFO    SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 12:50:20,784 INFO    SenderThread:355550 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 12:50:20,848 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:20,848 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:20,950 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:20,950 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,052 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,053 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,154 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,155 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,218 INFO    Thread-15 :355550 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/wandb-summary.json
2021-07-13 12:50:21,219 INFO    Thread-13 :355550 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/output.log
2021-07-13 12:50:21,224 INFO    Thread-14 :355550 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/config.yaml
2021-07-13 12:50:21,256 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,257 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,271 INFO    Thread-12 :355550 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/files/requirements.txt
2021-07-13 12:50:21,359 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,359 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,461 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,461 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,471 INFO    Thread-7  :355550 [sender.py:transition_state():308] send defer: 7
2021-07-13 12:50:21,471 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:21,471 INFO    HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 12:50:21,472 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:21,472 INFO    SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 12:50:21,562 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,867 INFO    SenderThread:355550 [sender.py:transition_state():308] send defer: 8
2021-07-13 12:50:21,867 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,868 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:21,868 INFO    HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 12:50:21,868 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:21,868 INFO    SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 12:50:21,868 INFO    SenderThread:355550 [sender.py:transition_state():308] send defer: 9
2021-07-13 12:50:21,869 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: defer
2021-07-13 12:50:21,869 INFO    HandlerThread:355550 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 12:50:21,869 DEBUG   SenderThread:355550 [sender.py:send():179] send: final
2021-07-13 12:50:21,869 DEBUG   SenderThread:355550 [sender.py:send():179] send: footer
2021-07-13 12:50:21,869 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: defer
2021-07-13 12:50:21,869 INFO    SenderThread:355550 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 12:50:21,969 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 12:50:21,970 DEBUG   SenderThread:355550 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 12:50:21,970 INFO    SenderThread:355550 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 12:50:21,971 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 12:50:21,972 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 12:50:21,973 DEBUG   HandlerThread:355550 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 12:50:21,973 INFO    HandlerThread:355550 [handler.py:finish():638] shutting down handler
2021-07-13 12:50:22,870 INFO    WriterThread:355550 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_124748-vubzdnhu/run-vubzdnhu.wandb
2021-07-13 12:50:22,970 INFO    SenderThread:355550 [sender.py:finish():945] shutting down sender
2021-07-13 12:50:22,970 INFO    SenderThread:355550 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 12:50:22,970 INFO    SenderThread:355550 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 12:50:22,973 INFO    MainThread:355550 [internal.py:handle_exit():78] Internal process exited