File size: 19,677 Bytes
22f8323 a869d63 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 |
2022-05-29 12:04:59,324 INFO MainThread:1721989 [internal.py:wandb_internal():92] W&B internal server running at pid: 1721989, started at: 2022-05-29 12:04:59.324307
2022-05-29 12:04:59,326 INFO WriterThread:1721989 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/run-qf2iwkac.wandb
2022-05-29 12:04:59,327 DEBUG SenderThread:1721989 [sender.py:send():235] send: header
2022-05-29 12:04:59,327 DEBUG SenderThread:1721989 [sender.py:send():235] send: run
2022-05-29 12:04:59,384 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: check_version
2022-05-29 12:04:59,385 INFO SenderThread:1721989 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files
2022-05-29 12:04:59,385 INFO SenderThread:1721989 [sender.py:_start_run_threads():812] run started: qf2iwkac with start time 1653825898
2022-05-29 12:04:59,385 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:04:59,385 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:04:59,385 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: check_version
2022-05-29 12:04:59,447 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: run_start
2022-05-29 12:05:00,436 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:05:01,566 DEBUG HandlerThread:1721989 [meta.py:__init__():37] meta init
2022-05-29 12:05:01,566 DEBUG HandlerThread:1721989 [meta.py:__init__():51] meta init done
2022-05-29 12:05:01,566 DEBUG HandlerThread:1721989 [meta.py:probe():211] probe
2022-05-29 12:05:01,568 DEBUG HandlerThread:1721989 [meta.py:_setup_git():201] setup git
2022-05-29 12:05:01,597 DEBUG HandlerThread:1721989 [meta.py:_setup_git():208] setup git done
2022-05-29 12:05:01,597 DEBUG HandlerThread:1721989 [meta.py:_save_pip():55] save pip
2022-05-29 12:05:01,598 DEBUG HandlerThread:1721989 [meta.py:_save_pip():69] save pip done
2022-05-29 12:05:01,598 DEBUG HandlerThread:1721989 [meta.py:probe():249] probe done
2022-05-29 12:05:01,601 DEBUG SenderThread:1721989 [sender.py:send():235] send: files
2022-05-29 12:05:01,601 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-05-29 12:05:01,608 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:05:01,608 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:05:01,639 DEBUG SenderThread:1721989 [sender.py:send():235] send: telemetry
2022-05-29 12:05:01,849 INFO Thread-11 :1721989 [upload_job.py:push():137] Uploaded file /tmp/tmpl7u6xxw8wandb/12iwbaqf-wandb-metadata.json
2022-05-29 12:05:02,475 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-metadata.json
2022-05-29 12:05:02,475 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/requirements.txt
2022-05-29 12:05:02,475 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:04,476 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:06,476 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:16,479 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:16,641 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:05:16,641 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:05:18,480 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:20,481 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:29,648 DEBUG SenderThread:1721989 [sender.py:send():235] send: stats
2022-05-29 12:05:30,485 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/config.yaml
2022-05-29 12:05:31,669 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:05:31,669 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:05:42,516 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:05:42,573 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:05:44,489 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:46,714 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:05:46,715 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:05:59,724 DEBUG SenderThread:1721989 [sender.py:send():235] send: stats
2022-05-29 12:06:01,767 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:06:01,767 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:06:03,840 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:03,845 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:06:03,846 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:06:03,847 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:06:03,899 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:04,497 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:06:06,497 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:06:16,816 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:06:16,816 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:06:26,186 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:26,188 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:06:26,189 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:06:26,191 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:06:26,256 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:26,505 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:06:26,834 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:26,836 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:06:26,837 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:06:26,838 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:06:26,884 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:27,505 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:06:27,513 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:27,515 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:06:27,515 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:06:27,516 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:06:27,570 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:28,506 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:06:28,506 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:06:29,794 DEBUG SenderThread:1721989 [sender.py:send():235] send: stats
2022-05-29 12:06:31,871 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:06:31,871 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:06:46,902 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:06:46,903 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:06:54,515 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:06:59,870 DEBUG SenderThread:1721989 [sender.py:send():235] send: stats
2022-05-29 12:07:01,952 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:07:01,952 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:07:12,521 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:07:17,016 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:07:17,016 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:07:29,946 DEBUG SenderThread:1721989 [sender.py:send():235] send: stats
2022-05-29 12:07:32,044 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:07:32,044 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:07:35,982 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:36,081 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-29 12:07:36,081 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: log_artifact
2022-05-29 12:07:36,280 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:36,354 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-29 12:07:36,453 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/media/table/eval/step_0k_5_92ae240f2472ae79ad31.table.json
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/media/table/eval/step_0k_incorrect_5_b30e6671cbaaf5c82b6d.table.json
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/media
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/media/table
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/media/table/eval
2022-05-29 12:07:36,589 INFO Thread-13 :1721989 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/00/66783abbd98b29e02682346c82293a
2022-05-29 12:07:36,995 INFO SenderThread:1721989 [sender.py:send_request_log_artifact():976] logged artifact run-qf2iwkac-evalstep_0k - {'id': 'QXJ0aWZhY3Q6MTM1NjQyNTgy', 'digest': '1fc9411295ac28c0df16aa1f68611033', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjIxNTAwMTY3', 'latestArtifact': None}, 'version': 'latest'}
2022-05-29 12:07:36,995 DEBUG SenderThread:1721989 [sender.py:send():235] send: files
2022-05-29 12:07:36,995 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file media/table/eval/step_0k_5_92ae240f2472ae79ad31.table.json with policy now
2022-05-29 12:07:36,996 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: log_artifact
2022-05-29 12:07:37,260 INFO Thread-14 :1721989 [upload_job.py:push():137] Uploaded file /tmp/tmpl7u6xxw8wandb/1pl7w3az-media/table/eval/step_0k_5_92ae240f2472ae79ad31.table.json
2022-05-29 12:07:37,399 INFO Thread-16 :1721989 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/61/78965daa44518dd50bc0e765d67f24
2022-05-29 12:07:37,903 INFO SenderThread:1721989 [sender.py:send_request_log_artifact():976] logged artifact run-qf2iwkac-evalstep_0k_incorrect - {'id': 'QXJ0aWZhY3Q6MTM1NjQyNTkx', 'digest': 'cd720fb526739060a418c512f0e14c5f', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjIxNTAwMTY4', 'latestArtifact': None}, 'version': 'latest'}
2022-05-29 12:07:37,903 DEBUG SenderThread:1721989 [sender.py:send():235] send: files
2022-05-29 12:07:37,903 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file media/table/eval/step_0k_incorrect_5_b30e6671cbaaf5c82b6d.table.json with policy now
2022-05-29 12:07:38,134 INFO Thread-17 :1721989 [upload_job.py:push():137] Uploaded file /tmp/tmpl7u6xxw8wandb/26g70uwz-media/table/eval/step_0k_incorrect_5_b30e6671cbaaf5c82b6d.table.json
2022-05-29 12:07:38,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:07:40,346 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:40,349 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:07:40,349 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:07:40,350 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:07:40,392 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:40,531 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:07:40,531 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:07:40,896 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:40,899 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:07:40,899 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:07:40,900 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:07:40,946 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:41,532 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:07:41,543 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:41,545 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:07:41,546 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:07:41,547 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:07:41,579 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:42,158 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:42,160 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:07:42,161 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:07:42,162 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:07:42,200 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:42,532 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:07:42,532 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:07:42,826 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:42,828 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:07:42,828 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:07:42,829 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:07:42,857 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
|