[09:59:36] - INFO - absl - Restoring checkpoint from ./checkpoint_225000
tcmalloc: large alloc 1530273792 bytes == 0x9d58c000 @  0x7fe91a70f680 0x7fe91a730824 0x5b9a14 0x50b2ae 0x50cb1b 0x5a6f17 0x5f3010 0x56fd36 0x568d9a 0x5f5b33 0x56aadf 0x568d9a 0x68cdc7 0x67e161 0x67e1df 0x67e281 0x67e627 0x6b6e62 0x6b71ed 0x7fe91a5240b3 0x5f96de
/home/dat/pino/lib/python3.8/site-packages/jax/lib/xla_bridge.py:386: UserWarning: jax.host_count has been renamed to jax.process_count. This alias will eventually be removed; please update your code.
  warnings.warn(
/home/dat/pino/lib/python3.8/site-packages/jax/lib/xla_bridge.py:373: UserWarning: jax.host_id has been renamed to jax.process_index. This alias will eventually be removed; please update your code.
  warnings.warn(
Epoch ... (1/5):   0%|                                                                      | 0/5 [00:00<?, ?it/s]
Training...: 0it [00:00, ?it/s]




Training...: 49it [04:22,  2.65it/s]





Training...: 100it [04:55,  4.01s/it]





Training...: 152it [05:15,  2.24s/it]





Training...: 202it [05:36,  2.25s/it]




Training...: 249it [05:42,  2.68it/s]




Training...: 299it [06:02,  2.67it/s]





Training...: 350it [06:35,  4.01s/it]





Training...: 400it [06:56,  4.01s/it]





Training...: 450it [07:16,  4.01s/it]





Training...: 502it [07:36,  2.24s/it]





Training...: 552it [07:56,  2.25s/it]





Training...: 604it [08:16,  1.39s/it]




Training...: 649it [08:23,  2.67it/s]




Training...: 699it [08:43,  2.69it/s]




Training...: 749it [09:03,  2.67it/s]




Training...: 799it [09:23,  2.68it/s]





Training...: 850it [09:56,  4.01s/it]





Training...: 900it [10:16,  4.01s/it]





Training...: 950it [10:36,  4.01s/it]





Training...: 1002it [10:56,  2.25s/it]





Training...: 1054it [11:16,  1.39s/it]





Training...: 1104it [11:36,  1.39s/it]




Training...: 1149it [11:43,  2.69it/s]




Training...: 1199it [12:03,  2.67it/s]




Training...: 1249it [12:23,  2.67it/s]




Training...: 1299it [12:43,  2.69it/s]




Training...: 1349it [13:03,  2.67it/s]





Training...: 1400it [13:36,  4.01s/it]





Training...: 1450it [13:56,  4.01s/it]





Training...: 1500it [14:16,  4.01s/it]





Training...: 1552it [14:37,  2.24s/it]





Training...: 1602it [14:57,  2.24s/it]





Training...: 1654it [15:17,  1.39s/it]




Training...: 1699it [15:23,  2.67it/s]




Training...: 1749it [15:43,  2.70it/s]




Training...: 1799it [16:03,  2.67it/s]





Training...: 1850it [16:37,  4.01s/it]





Training...: 1900it [16:57,  4.01s/it]





Training...: 1952it [17:17,  2.24s/it]





Training...: 2002it [17:37,  2.24s/it]





Training...: 2054it [17:57,  1.39s/it]





Training...: 2104it [18:17,  1.39s/it]




Training...: 2149it [18:24,  2.66it/s]




Training...: 2199it [18:44,  2.68it/s]





Training...: 2250it [19:17,  4.01s/it]





Training...: 2300it [19:37,  4.01s/it]





Training...: 2352it [19:57,  2.24s/it]





Training...: 2402it [20:17,  2.24s/it]





Training...: 2454it [20:37,  1.39s/it]





Training...: 2504it [20:57,  1.39s/it]




Training...: 2549it [21:04,  2.67it/s]




Training...: 2599it [21:24,  2.67it/s]




Training...: 2649it [21:44,  2.67it/s]





Training...: 2702it [22:17,  2.24s/it]





Training...: 2754it [22:38,  1.39s/it]





Training...: 2804it [22:58,  1.39s/it]





Training...: 2854it [23:18,  1.39s/it]





Training...: 2900it [23:37,  4.01s/it]





Training...: 2950it [23:57,  4.01s/it]





Training...: 2999it [24:17,  2.67it/s]
Step... (228000 | Loss: 2.156348943710327, Learning Rate: 2.6271016395185143e-05)


































Training...: 3049it [25:46,  2.68it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 3099it [26:06,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 3166it [26:41,  5.39it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 3208it [27:00,  1.59it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 3258it [27:20,  1.59it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 3299it [27:26,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 3349it [27:47,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 3399it [28:07,  2.70it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 3449it [28:27,  2.66it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 3499it [28:47,  2.63it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 3549it [29:07,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 3599it [29:27,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 3649it [29:47,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 3699it [30:07,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 3749it [30:27,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 3799it [30:47,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 3849it [31:07,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 3899it [31:27,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 3949it [31:47,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 3999it [32:07,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4049it [32:27,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4099it [32:47,  2.63it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4149it [33:07,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4199it [33:27,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4249it [33:47,  2.70it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4299it [34:07,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 4349it [34:27,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4399it [34:47,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4449it [35:07,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4499it [35:27,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4549it [35:47,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4599it [36:07,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4649it [36:28,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 4699it [36:48,  2.74it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 4749it [37:08,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 4799it [37:28,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 4849it [37:48,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4899it [38:08,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4949it [38:28,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 4999it [38:48,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 5049it [39:08,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 5099it [39:28,  2.70it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 5149it [39:48,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 5199it [40:08,  2.68it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 5249it [40:28,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 5299it [40:48,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 5349it [41:08,  2.70it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 5399it [41:28,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 5449it [41:48,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 5499it [42:08,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 5549it [42:28,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 5599it [42:48,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 5649it [43:08,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 5699it [43:28,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 5749it [43:48,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 5799it [44:08,  2.66it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 5849it [44:28,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 5899it [44:48,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 5949it [45:09,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 5999it [45:29,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 6049it [45:49,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6099it [46:09,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6149it [46:29,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6199it [46:49,  2.70it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6249it [47:09,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 6299it [47:29,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 6349it [47:49,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 6399it [48:09,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6449it [48:29,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6499it [48:49,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6549it [49:09,  2.68it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6599it [49:29,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6649it [49:49,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6699it [50:09,  2.70it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 6749it [50:29,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 6799it [50:49,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 6849it [51:09,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]





Training...: 6899it [51:29,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6949it [51:49,  2.73it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 6999it [52:09,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7049it [52:29,  2.71it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7099it [52:49,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7149it [53:09,  2.74it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7199it [53:29,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7249it [53:50,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7299it [54:10,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7349it [54:30,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7399it [54:50,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7449it [55:10,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7499it [55:30,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7549it [55:50,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7599it [56:10,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7649it [56:30,  2.68it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7699it [56:50,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7749it [57:10,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7799it [57:30,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7849it [57:50,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7899it [58:10,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7949it [58:30,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 7999it [58:50,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8049it [59:10,  2.70it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8099it [59:30,  2.67it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8149it [59:50,  2.69it/s]██████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8199it [1:00:10,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8249it [1:00:30,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8299it [1:00:50,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8349it [1:01:10,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8399it [1:01:30,  2.69it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8449it [1:01:50,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8499it [1:02:11,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8549it [1:02:31,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8599it [1:02:51,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8649it [1:03:11,  2.69it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8699it [1:03:31,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8749it [1:03:51,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8799it [1:04:11,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8849it [1:04:31,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8899it [1:04:51,  2.70it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8949it [1:05:11,  2.67it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]




Training...: 8999it [1:05:31,  2.69it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]
Training...: 8999it [1:05:44,  2.69it/s]████████████████████████████████████████| 500/500 [01:17<00:00,  7.90it/s]



































Training...: 9050it [1:07:08,  4.01s/it]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 9102it [1:07:28,  2.25s/it]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 9152it [1:07:48,  2.25s/it]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 9204it [1:08:08,  1.39s/it]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 9249it [1:08:15,  2.69it/s]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 9299it [1:08:35,  2.67it/s]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 9349it [1:08:55,  2.67it/s]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 9399it [1:09:15,  2.67it/s]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 9449it [1:09:35,  2.67it/s]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 9499it [1:09:55,  2.69it/s]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 9549it [1:10:15,  2.67it/s]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 9602it [1:10:49,  2.24s/it]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 9654it [1:11:09,  1.39s/it]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 9704it [1:11:29,  1.39s/it]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 9749it [1:11:35,  2.69it/s]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 9799it [1:11:55,  2.67it/s]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 9850it [1:12:29,  4.01s/it]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 9900it [1:12:49,  4.01s/it]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 9950it [1:13:09,  4.01s/it]████████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10002it [1:13:29,  2.24s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10052it [1:13:49,  2.25s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10104it [1:14:09,  1.39s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10154it [1:14:29,  1.39s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10200it [1:14:49,  4.01s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10254it [1:15:09,  1.38s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10304it [1:15:29,  1.40s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10356it [1:15:49,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10406it [1:16:09,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10456it [1:16:29,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10500it [1:16:49,  4.01s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10550it [1:17:09,  4.01s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10600it [1:17:29,  4.01s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10652it [1:17:49,  2.25s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10704it [1:18:10,  1.39s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10756it [1:18:30,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10806it [1:18:50,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10850it [1:19:09,  4.01s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10900it [1:19:29,  4.01s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 10950it [1:19:49,  4.01s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11002it [1:20:10,  2.25s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11052it [1:20:30,  2.25s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11104it [1:20:50,  1.39s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11154it [1:21:10,  1.39s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11204it [1:21:30,  1.39s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11256it [1:21:50,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11302it [1:22:10,  2.25s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11356it [1:22:30,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11406it [1:22:50,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11458it [1:23:10,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11500it [1:23:30,  4.01s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11552it [1:23:50,  2.25s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11602it [1:24:10,  2.25s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11654it [1:24:30,  1.39s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11704it [1:24:50,  1.39s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11754it [1:25:10,  1.39s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11804it [1:25:30,  1.39s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11856it [1:25:51,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11906it [1:26:11,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 11956it [1:26:31,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12008it [1:26:51,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12060it [1:27:11,  2.26it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12102it [1:27:30,  2.24s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12154it [1:27:51,  1.39s/it]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12206it [1:28:11,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12256it [1:28:31,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12308it [1:28:51,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12358it [1:29:11,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12406it [1:29:31,  1.05it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12458it [1:29:51,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12508it [1:30:11,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12560it [1:30:31,  2.26it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12610it [1:30:51,  2.26it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12659it [1:31:12,  2.14it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12712it [1:31:32,  3.11it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 12764it [1:31:52,  4.16it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12806it [1:32:11,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12856it [1:32:31,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12908it [1:32:51,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 12958it [1:33:12,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13008it [1:33:32,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13058it [1:33:52,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13110it [1:34:12,  2.26it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13156it [1:34:32,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13206it [1:34:52,  1.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 13249it [1:34:58,  2.67it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13299it [1:35:18,  2.70it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13349it [1:35:38,  2.67it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]






Training...: 13408it [1:36:12,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13460it [1:36:32,  2.26it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13508it [1:36:52,  1.59it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13562it [1:37:12,  3.10it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13612it [1:37:32,  3.05it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 13662it [1:37:52,  3.10it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 13714it [1:38:12,  4.15it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 13760it [1:38:32,  2.19it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 13814it [1:38:53,  4.15it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]



Training...: 13849it [1:38:59,  2.67it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 13899it [1:39:19,  2.69it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 13949it [1:39:39,  2.67it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]






Training...: 14010it [1:40:12,  2.25it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 14060it [1:40:32,  2.26it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 14112it [1:40:53,  3.11it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 14162it [1:41:13,  3.09it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 14212it [1:41:33,  3.11it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14262it [1:41:53,  3.11it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14314it [1:42:13,  4.16it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14364it [1:42:33,  4.17it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14414it [1:42:53,  4.15it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14464it [1:43:13,  4.16it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14512it [1:43:33,  3.07it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14564it [1:43:53,  4.10it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]



Training...: 14599it [1:43:59,  2.68it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14649it [1:44:19,  2.71it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14699it [1:44:39,  2.67it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 14762it [1:45:13,  3.11it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14814it [1:45:33,  4.16it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14862it [1:45:53,  2.98it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]



Training...: 14899it [1:45:59,  2.69it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]




Training...: 14949it [1:46:19,  2.67it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]





Training...: 14999it [1:46:39,  2.69it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]
Training...: 14999it [1:46:53,  2.69it/s]███████████████████████████████████████| 500/500 [00:59<00:00,  7.90it/s]































tcmalloc: large alloc 1354776576 bytes == 0x37b2fe000 @  0x7fe91a70f680 0x7fe91a72fbdd 0x7fe900cc920d 0x7fe900cd7340 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd2bd3 0x7fe900cd31fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x68cdc7 0x67e161 0x67e1df
tcmalloc: large alloc 2715181056 bytes == 0x3cc702000 @  0x7fe91a70f680 0x7fe91a72fbdd 0x7fe900cc920d 0x7fe900cd7340 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd6e87 0x7fe900cd2bd3 0x7fe900cd31fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a
tcmalloc: large alloc 1530273792 bytes == 0x321dba000 @  0x7fe91a70f680 0x7fe91a730824 0x5f7b11 0x7fe900cd2c6f 0x7fe900cd31fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x68cdc7 0x67e161 0x67e1df 0x67e281 0x67e627 0x6b6e62 0x6b71ed 0x7fe91a5240b3 0x5f96de
[11:53:20] - INFO - absl - Saved checkpoint at checkpoint_240000
[11:53:21] - INFO - huggingface_hub.repository - git version 2.25.1
git-lfs/2.9.2 (GitHub; linux amd64; go 1.13.5)