File size: 16,912 Bytes
8319971 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 |
2022-05-07 17:41:49,607 INFO MainThread:14620 [internal.py:wandb_internal():92] W&B internal server running at pid: 14620, started at: 2022-05-07 17:41:49.606834
2022-05-07 17:41:49,609 INFO WriterThread:14620 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/run-2l3y3icw.wandb
2022-05-07 17:41:49,610 DEBUG SenderThread:14620 [sender.py:send():235] send: header
2022-05-07 17:41:49,610 DEBUG SenderThread:14620 [sender.py:send():235] send: run
2022-05-07 17:41:49,678 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: check_version
2022-05-07 17:41:49,678 INFO SenderThread:14620 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files
2022-05-07 17:41:49,678 INFO SenderThread:14620 [sender.py:_start_run_threads():812] run started: 2l3y3icw with start time 1651945308
2022-05-07 17:41:49,678 DEBUG SenderThread:14620 [sender.py:send():235] send: summary
2022-05-07 17:41:49,679 INFO SenderThread:14620 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 17:41:49,679 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: check_version
2022-05-07 17:41:49,739 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: run_start
2022-05-07 17:41:50,689 INFO Thread-7 :14620 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-summary.json
2022-05-07 17:41:51,879 DEBUG HandlerThread:14620 [meta.py:__init__():37] meta init
2022-05-07 17:41:51,879 DEBUG HandlerThread:14620 [meta.py:__init__():51] meta init done
2022-05-07 17:41:51,879 DEBUG HandlerThread:14620 [meta.py:probe():211] probe
2022-05-07 17:41:51,880 DEBUG HandlerThread:14620 [meta.py:_setup_git():201] setup git
2022-05-07 17:41:51,909 DEBUG HandlerThread:14620 [meta.py:_setup_git():208] setup git done
2022-05-07 17:41:51,909 DEBUG HandlerThread:14620 [meta.py:_save_pip():55] save pip
2022-05-07 17:41:51,910 DEBUG HandlerThread:14620 [meta.py:_save_pip():69] save pip done
2022-05-07 17:41:51,910 DEBUG HandlerThread:14620 [meta.py:probe():249] probe done
2022-05-07 17:41:51,912 DEBUG SenderThread:14620 [sender.py:send():235] send: files
2022-05-07 17:41:51,913 INFO SenderThread:14620 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-05-07 17:41:51,918 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 17:41:51,918 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: stop_status
2022-05-07 17:41:51,948 DEBUG SenderThread:14620 [sender.py:send():235] send: telemetry
2022-05-07 17:41:52,187 INFO Thread-11 :14620 [upload_job.py:push():137] Uploaded file /tmp/tmpandpth3dwandb/1ih5opi9-wandb-metadata.json
2022-05-07 17:41:52,688 INFO Thread-7 :14620 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/requirements.txt
2022-05-07 17:41:52,689 INFO Thread-7 :14620 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log
2022-05-07 17:41:52,689 INFO Thread-7 :14620 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-metadata.json
2022-05-07 17:41:53,838 DEBUG SenderThread:14620 [sender.py:send():235] send: telemetry
2022-05-07 17:41:53,838 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:41:54,689 INFO Thread-7 :14620 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log
2022-05-07 17:41:55,690 INFO Thread-7 :14620 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log
2022-05-07 17:41:56,439 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,439 DEBUG SenderThread:14620 [sender.py:send():235] send: exit
2022-05-07 17:41:56,439 INFO SenderThread:14620 [sender.py:send_exit():371] handling exit code: 1
2022-05-07 17:41:56,439 INFO SenderThread:14620 [sender.py:send_exit():373] handling runtime: 6
2022-05-07 17:41:56,440 INFO SenderThread:14620 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 17:41:56,440 INFO SenderThread:14620 [sender.py:send_exit():379] send defer
2022-05-07 17:41:56,440 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:56,441 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,441 INFO HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 0
2022-05-07 17:41:56,441 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,441 INFO SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 0
2022-05-07 17:41:56,441 INFO SenderThread:14620 [sender.py:transition_state():392] send defer: 1
2022-05-07 17:41:56,441 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,441 INFO HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 1
2022-05-07 17:41:56,501 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,501 INFO SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 1
2022-05-07 17:41:56,501 INFO SenderThread:14620 [sender.py:transition_state():392] send defer: 2
2022-05-07 17:41:56,501 DEBUG SenderThread:14620 [sender.py:send():235] send: stats
2022-05-07 17:41:56,502 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,502 INFO HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 2
2022-05-07 17:41:56,502 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,502 INFO SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 2
2022-05-07 17:41:56,502 INFO SenderThread:14620 [sender.py:transition_state():392] send defer: 3
2022-05-07 17:41:56,502 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,502 INFO HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 3
2022-05-07 17:41:56,503 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,503 INFO SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 3
2022-05-07 17:41:56,503 INFO SenderThread:14620 [sender.py:transition_state():392] send defer: 4
2022-05-07 17:41:56,503 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,503 INFO HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 4
2022-05-07 17:41:56,503 DEBUG SenderThread:14620 [sender.py:send():235] send: summary
2022-05-07 17:41:56,503 INFO SenderThread:14620 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 17:41:56,504 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,504 INFO SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 4
2022-05-07 17:41:56,504 INFO SenderThread:14620 [sender.py:transition_state():392] send defer: 5
2022-05-07 17:41:56,504 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,504 INFO HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 5
2022-05-07 17:41:56,504 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,504 INFO SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 5
2022-05-07 17:41:56,542 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,563 INFO SenderThread:14620 [sender.py:transition_state():392] send defer: 6
2022-05-07 17:41:56,563 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:56,564 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,564 INFO HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 6
2022-05-07 17:41:56,564 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,564 INFO SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 6
2022-05-07 17:41:56,564 INFO SenderThread:14620 [dir_watcher.py:finish():283] shutting down directory watcher
2022-05-07 17:41:56,665 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,690 INFO Thread-7 :14620 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-summary.json
2022-05-07 17:41:56,691 INFO SenderThread:14620 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/config.yaml
2022-05-07 17:41:56,691 INFO SenderThread:14620 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log
2022-05-07 17:41:56,691 INFO SenderThread:14620 [dir_watcher.py:finish():313] scan: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files
2022-05-07 17:41:56,691 INFO SenderThread:14620 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/requirements.txt requirements.txt
2022-05-07 17:41:56,691 INFO SenderThread:14620 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log output.log
2022-05-07 17:41:56,692 INFO SenderThread:14620 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-metadata.json wandb-metadata.json
2022-05-07 17:41:56,692 INFO SenderThread:14620 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/config.yaml config.yaml
2022-05-07 17:41:56,692 INFO SenderThread:14620 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-summary.json wandb-summary.json
2022-05-07 17:41:56,692 INFO SenderThread:14620 [sender.py:transition_state():392] send defer: 7
2022-05-07 17:41:56,693 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:56,693 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:56,693 INFO HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 7
2022-05-07 17:41:56,696 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:56,696 INFO SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 7
2022-05-07 17:41:56,696 INFO SenderThread:14620 [file_pusher.py:finish():145] shutting down file pusher
2022-05-07 17:41:56,794 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,794 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:56,896 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,896 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:56,940 INFO Thread-13 :14620 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/output.log
2022-05-07 17:41:56,947 INFO Thread-14 :14620 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/config.yaml
2022-05-07 17:41:56,969 INFO Thread-15 :14620 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/wandb-summary.json
2022-05-07 17:41:56,988 INFO Thread-12 :14620 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/files/requirements.txt
2022-05-07 17:41:56,998 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:56,998 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:57,100 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:57,100 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:57,189 INFO Thread-6 :14620 [sender.py:transition_state():392] send defer: 8
2022-05-07 17:41:57,189 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:57,189 INFO HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 8
2022-05-07 17:41:57,189 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:57,189 INFO SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 8
2022-05-07 17:41:57,202 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:59,454 INFO SenderThread:14620 [sender.py:transition_state():392] send defer: 9
2022-05-07 17:41:59,455 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:59,455 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:59,455 INFO HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 9
2022-05-07 17:41:59,455 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:59,456 INFO SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 9
2022-05-07 17:41:59,456 INFO SenderThread:14620 [sender.py:transition_state():392] send defer: 10
2022-05-07 17:41:59,456 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: defer
2022-05-07 17:41:59,456 INFO HandlerThread:14620 [handler.py:handle_request_defer():164] handle defer: 10
2022-05-07 17:41:59,456 DEBUG SenderThread:14620 [sender.py:send():235] send: final
2022-05-07 17:41:59,456 DEBUG SenderThread:14620 [sender.py:send():235] send: footer
2022-05-07 17:41:59,456 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: defer
2022-05-07 17:41:59,456 INFO SenderThread:14620 [sender.py:send_request_defer():388] handle sender defer: 10
2022-05-07 17:41:59,556 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 17:41:59,557 DEBUG SenderThread:14620 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 17:41:59,557 INFO SenderThread:14620 [file_pusher.py:join():150] waiting for file pusher
2022-05-07 17:42:00,139 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: sampled_history
2022-05-07 17:42:00,140 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: get_summary
2022-05-07 17:42:00,141 DEBUG HandlerThread:14620 [handler.py:handle_request():141] handle_request: shutdown
2022-05-07 17:42:00,141 INFO HandlerThread:14620 [handler.py:finish():778] shutting down handler
2022-05-07 17:42:00,457 INFO WriterThread:14620 [datastore.py:close():281] close: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174148-2l3y3icw/run-2l3y3icw.wandb
2022-05-07 17:42:01,038 INFO SenderThread:14620 [sender.py:finish():1078] shutting down sender
2022-05-07 17:42:01,038 INFO SenderThread:14620 [file_pusher.py:finish():145] shutting down file pusher
2022-05-07 17:42:01,038 INFO SenderThread:14620 [file_pusher.py:join():150] waiting for file pusher
2022-05-07 17:42:01,040 INFO MainThread:14620 [internal.py:handle_exit():82] Internal process exited
|