dat
Saving weights and logs of step 20000
16181fe
/home/dat/pino/lib/python3.8/site-packages/jax/lib/xla_bridge.py:382: UserWarning: jax.host_count has been renamed to jax.process_count. This alias will eventually be removed; please update your code.
warnings.warn(
/home/dat/pino/lib/python3.8/site-packages/jax/lib/xla_bridge.py:369: UserWarning: jax.host_id has been renamed to jax.process_index. This alias will eventually be removed; please update your code.
warnings.warn(
Epoch ... (1/5): 0%| | 0/5 [00:00<?, ?it/s]
Training...: 1%| | 500/92767 [06:08<14:38:48, 1.75it/s]
Training...: 1%|▏ | 1000/92767 [10:54<14:33:13, 1.75it/s]
Training...: 2%|β–Ž | 1500/92767 [15:40<14:28:03, 1.75it/s]
Training...: 2%|▍ | 2005/92767 [20:46<43:13:25, 1.71s/it]
Training...: 3%|β–Œ | 2500/92767 [25:13<14:19:55, 1.75it/s]
Training...: 3%|β–‹ | 3000/92767 [29:59<14:13:31, 1.75it/s]
Training...: 4%|β–Š | 3501/92767 [35:04<153:51:07, 6.20s/it]
Training...: 4%|β–‰ | 4007/92767 [39:51<26:26:24, 1.07s/it]
Training...: 5%|β–ˆ | 4513/92767 [44:38<5:17:18, 4.64it/s]
Training...: 5%|β–ˆβ– | 5000/92767 [49:03<13:56:15, 1.75it/s]
Training...: 6%|β–ˆβ–Ž | 5500/92767 [53:49<13:50:21, 1.75it/s]
Training...: 6%|β–ˆβ– | 6000/92767 [58:36<13:45:23, 1.75it/s]
Training...: 7%|β–ˆβ– | 6500/92767 [1:03:22<13:40:53, 1.75it/s]
Training...: 8%|β–ˆβ–Œ | 7003/92767 [1:08:27<73:43:46, 3.09s/it]
Training...: 8%|β–ˆβ–Œ | 7500/92767 [1:12:54<13:31:06, 1.75it/s]
Training...: 9%|β–ˆβ–‹ | 8000/92767 [1:17:40<13:26:37, 1.75it/s]
Training...: 9%|β–ˆβ–Š | 8503/92767 [1:22:46<72:29:48, 3.10s/it]
Training...: 10%|β–ˆβ–‰ | 9006/92767 [1:27:32<26:20:08, 1.13s/it]
Training...: 10%|β–ˆβ–ˆβ– | 9512/92767 [1:32:19<7:27:35, 3.10it/s]
Training...: 11%|β–ˆβ–ˆ | 10000/92767 [1:36:45<13:06:55, 1.75it/s]
Training...: 11%|β–ˆβ–ˆβ– | 10500/92767 [1:41:31<13:02:26, 1.75it/s]
Training...: 12%|β–ˆβ–ˆβ–Ž | 11000/92767 [1:46:17<12:58:46, 1.75it/s]
Training...: 12%|β–ˆβ–ˆβ–Ž | 11500/92767 [1:51:03<12:53:11, 1.75it/s]
Training...: 13%|β–ˆβ–ˆβ– | 12002/92767 [1:56:09<98:23:42, 4.39s/it]
Training...: 13%|β–ˆβ–ˆβ–Œ | 12508/92767 [2:00:56<14:34:49, 1.53it/s]
Training...: 14%|β–ˆβ–ˆβ–Š | 13014/92767 [2:05:42<4:42:51, 4.70it/s]
Training...: 15%|β–ˆβ–ˆβ–Š | 13500/92767 [2:10:08<12:31:38, 1.76it/s]
Training...: 15%|β–ˆβ–ˆβ–Š | 14003/92767 [2:15:13<67:44:48, 3.10s/it]
Training...: 16%|β–ˆβ–ˆβ–ˆβ– | 14511/92767 [2:20:00<5:56:33, 3.66it/s]
Training...: 16%|β–ˆβ–ˆβ–ˆ | 15000/92767 [2:24:26<12:20:02, 1.75it/s]
Training...: 17%|β–ˆβ–ˆβ–ˆ | 15501/92767 [2:29:32<133:12:40, 6.21s/it]
Training...: 17%|β–ˆβ–ˆβ–ˆβ–Ž | 16006/92767 [2:34:18<29:26:21, 1.38s/it]
Training...: 18%|β–ˆβ–ˆβ–ˆβ–Œ | 16513/92767 [2:39:05<4:37:44, 4.58it/s]
Training...: 18%|β–ˆβ–ˆβ–ˆβ–‹ | 17018/92767 [2:43:52<3:24:31, 6.17it/s]
Training...: 19%|β–ˆβ–ˆβ–ˆβ–Œ | 17503/92767 [2:48:36<64:45:38, 3.10s/it]
Training...: 19%|β–ˆβ–ˆβ–ˆβ–‰ | 18011/92767 [2:53:23<8:05:33, 2.57it/s]
Training...: 20%|β–ˆβ–ˆβ–ˆβ–‰ | 18515/92767 [2:58:10<3:45:16, 5.49it/s]
Training...: 20%|β–ˆβ–ˆβ–ˆβ–‰ | 19000/92767 [3:02:35<11:41:57, 1.75it/s]
Training...: 21%|β–ˆβ–ˆβ–ˆβ–‰ | 19500/92767 [3:07:21<11:38:06, 1.75it/s]
Training...: 22%|β–ˆβ–ˆβ–ˆβ–ˆ | 20000/92767 [3:12:07<11:32:26, 1.75it/s]
git-lfs/2.9.2 (GitHub; linux amd64; go 1.13.5)2767 [3:12:27<11:32:26, 1.75it/s]