silaseic
/

nanogpt_finetuned_models

Model card Files Files and versions Community

gitGut01 commited on Mar 25, 2023

Commit

19d1dda

•

1 Parent(s): 61b62f9

add datasets

Browse files

Files changed (14) hide show

dataset/dataset_ciaworld.txt +0 -0
dataset/dataset_edsheeran.txt +0 -0
dataset/dataset_haiku.txt +0 -0
dataset/dataset_shakespeare.txt +0 -0
train_info/train_info_edsheeran.txt +111 -0
train_info/train_info_haiku.txt +42 -0
train_info/train_info_shakespeare.txt +185 -0
train_info/train_info_trump.txt +207 -0
ckpt_edsheeran.pt → weights/ckpt_edsheeran.pt +0 -0
ckpt_haiku.pt → weights/ckpt_haiku.pt +0 -0
ckpt_math.pt → weights/ckpt_math.pt +0 -0
ckpt_shakespear.pt → weights/ckpt_shakespear.pt +0 -0
ckpt_trump.pt → weights/ckpt_trump.pt +0 -0
ckpt_world_facts_cia.pt → weights/ckpt_world_facts_cia.pt +0 -0

dataset/dataset_ciaworld.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

dataset/dataset_edsheeran.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

dataset/dataset_haiku.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

dataset/dataset_shakespeare.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

train_info/train_info_edsheeran.txt ADDED Viewed

	@@ -0,0 +1,111 @@

+Overriding config with config/finetune_shakespeare.py:
+import time
+out_dir = 'out-shakespeare'
+eval_interval = 5
+eval_iters = 40
+wandb_log = False # feel free to turn on
+wandb_project = 'shakespeare'
+wandb_run_name = 'ft-' + str(time.time())
+dataset = 'shakespeare'
+init_from = 'gpt2' # this is the largest GPT-2 model
+# only save checkpoints if the validation loss improves
+always_save_checkpoint = False
+# the number of examples per iter:
+# 1 batch_size * 32 grad_accum * 1024 tokens = 32,768 tokens/iter
+# shakespeare has 301,966 tokens, so 1 epoch ~= 9.2 iters
+batch_size = 1
+gradient_accumulation_steps = 32
+max_iters = 120
+# finetune at constant LR
+learning_rate = 3e-5
+decay_lr = False
+Initializing from OpenAI GPT-2 weights: gpt2
+loading weights from pretrained gpt: gpt2
+forcing vocab_size=50257, block_size=1024, bias=True
+overriding dropout rate to 0.0
+number of parameters: 123.65M
+Downloading (…)lve/main/config.json: 100% 665/665 [00:00<00:00, 98.3kB/s]
+Downloading pytorch_model.bin: 100% 548M/548M [00:05<00:00, 92.8MB/s]
+Downloading (…)neration_config.json: 100% 124/124 [00:00<00:00, 19.2kB/s]
+using fused AdamW: True
+compiling the model... (takes a ~minute)
+[2023-03-21 14:22:50,795] torch._inductor.utils: [WARNING] make_fallback(aten.addmv): a decomposition exists, we should switch to it
+step 0: train loss 3.4423, val loss 3.0369
+iter 0: loss 3.2863, time 77202.23ms, mfu -100.00%
+iter 1: loss 2.7469, time 22529.17ms, mfu -100.00%
+iter 2: loss 3.7087, time 23101.21ms, mfu -100.00%
+iter 3: loss 3.6040, time 23363.38ms, mfu -100.00%
+iter 4: loss 2.6769, time 23118.49ms, mfu -100.00%
+step 5: train loss 3.4339, val loss 2.9363
+saving checkpoint to out-shakespeare
+iter 5: loss 3.1141, time 30621.41ms, mfu 2.35%
+iter 6: loss 3.3365, time 23426.49ms, mfu 2.42%
+iter 7: loss 3.8965, time 23144.13ms, mfu 2.49%
+iter 8: loss 3.4058, time 23061.69ms, mfu 2.55%
+iter 9: loss 3.2569, time 23230.68ms, mfu 2.60%
+step 10: train loss 3.2385, val loss 2.9982
+iter 10: loss 3.1935, time 25160.57ms, mfu 2.63%
+iter 11: loss 3.9526, time 23125.77ms, mfu 2.68%
+iter 12: loss 2.4570, time 23136.22ms, mfu 2.72%
+iter 13: loss 3.5092, time 23120.81ms, mfu 2.76%
+iter 14: loss 3.4771, time 23226.29ms, mfu 2.79%
+step 15: train loss 2.9026, val loss 2.8705
+saving checkpoint to out-shakespeare
+iter 15: loss 3.4825, time 30931.56ms, mfu 2.75%
+iter 16: loss 3.3583, time 23307.64ms, mfu 2.78%
+iter 17: loss 2.2991, time 23143.53ms, mfu 2.81%
+iter 18: loss 3.2513, time 23131.39ms, mfu 2.84%
+iter 19: loss 2.9859, time 23160.12ms, mfu 2.87%
+step 20: train loss 2.9491, val loss 2.7808
+saving checkpoint to out-shakespeare
+iter 20: loss 3.0525, time 30909.27ms, mfu 2.81%
+iter 21: loss 2.9295, time 23294.73ms, mfu 2.84%
+iter 22: loss 2.2879, time 23094.34ms, mfu 2.87%
+iter 23: loss 1.8019, time 23103.56ms, mfu 2.89%
+iter 24: loss 3.4942, time 23172.01ms, mfu 2.91%
+step 25: train loss 2.8004, val loss 2.8107
+iter 25: loss 2.2264, time 25127.64ms, mfu 2.91%
+iter 26: loss 3.4194, time 23174.40ms, mfu 2.93%
+iter 27: loss 2.8144, time 23152.02ms, mfu 2.94%
+iter 28: loss 3.0488, time 23133.18ms, mfu 2.96%
+iter 29: loss 3.1027, time 23085.89ms, mfu 2.98%
+step 30: train loss 2.6644, val loss 2.6210
+saving checkpoint to out-shakespeare
+iter 30: loss 2.4424, time 31309.61ms, mfu 2.91%
+iter 31: loss 3.0193, time 23415.64ms, mfu 2.92%
+iter 32: loss 2.8735, time 23054.64ms, mfu 2.94%
+iter 33: loss 2.9842, time 23053.71ms, mfu 2.96%
+iter 34: loss 2.8148, time 23136.92ms, mfu 2.97%
+step 35: train loss 2.8676, val loss 2.5965
+saving checkpoint to out-shakespeare
+iter 35: loss 2.8556, time 31228.61ms, mfu 2.91%
+iter 36: loss 2.1186, time 23332.51ms, mfu 2.92%
+iter 37: loss 2.4768, time 23039.16ms, mfu 2.94%
+iter 38: loss 2.7992, time 23035.59ms, mfu 2.96%
+iter 39: loss 2.7109, time 23218.08ms, mfu 2.97%
+step 40: train loss 2.5840, val loss 2.6467
+iter 40: loss 3.0349, time 25092.98ms, mfu 2.96%
+iter 41: loss 2.8766, time 23084.39ms, mfu 2.98%
+iter 42: loss 2.5366, time 23099.15ms, mfu 2.99%
+iter 43: loss 2.7461, time 23183.70ms, mfu 3.00%
+iter 44: loss 1.4962, time 23190.74ms, mfu 3.01%
+step 45: train loss 2.6357, val loss 2.6529
+iter 45: loss 2.1228, time 25011.92ms, mfu 3.00%
+iter 46: loss 1.9382, time 23127.95ms, mfu 3.01%
+iter 47: loss 1.7129, time 23168.21ms, mfu 3.02%
+iter 48: loss 2.4555, time 23162.14ms, mfu 3.03%
+iter 49: loss 1.3368, time 23152.22ms, mfu 3.03%
+step 50: train loss 2.3167, val loss 2.6496
+iter 50: loss 2.3815, time 24969.84ms, mfu 3.02%
+iter 51: loss 1.5433, time 23013.56ms, mfu 3.03%
+iter 52: loss 2.5276, time 22951.87ms, mfu 3.04%
+iter 53: loss 2.0912, time 22989.47ms, mfu 3.05%
+iter 54: loss 1.6236, time 23016.77ms, mfu 3.06%
+step 55: train loss 2.2718, val loss 2.6701
+iter 55: loss 0.9116, time 24910.16ms, mfu 3.04%

train_info/train_info_haiku.txt ADDED Viewed

	@@ -0,0 +1,42 @@

+# finetune at constant LR
+learning_rate = 3e-5
+decay_lr = False
+Initializing from OpenAI GPT-2 weights: gpt2
+loading weights from pretrained gpt: gpt2
+forcing vocab_size=50257, block_size=1024, bias=True
+overriding dropout rate to 0.0
+number of parameters: 123.65M
+using fused AdamW: True
+compiling the model... (takes a ~minute)
+[2023-03-21 15:03:01,696] torch._inductor.utils: [WARNING] make_fallback(aten.addmv): a decomposition exists, we should switch to it
+step 0: train loss 7.3575, val loss 7.4530
+iter 0: loss 7.3959, time 55528.06ms, mfu -100.00%
+iter 1: loss 7.4243, time 22248.52ms, mfu -100.00%
+iter 2: loss 7.3179, time 22821.48ms, mfu -100.00%
+iter 3: loss 7.5001, time 23404.71ms, mfu -100.00%
+iter 4: loss 7.4802, time 23247.54ms, mfu -100.00%
+step 5: train loss 7.2418, val loss 7.4663
+iter 5: loss 7.3052, time 24918.41ms, mfu 2.88%
+iter 6: loss 6.9456, time 23189.74ms, mfu 2.90%
+iter 7: loss 6.6510, time 23306.99ms, mfu 2.92%
+iter 8: loss 6.3013, time 23235.93ms, mfu 2.94%
+iter 9: loss 6.0171, time 23170.33ms, mfu 2.96%
+step 10: train loss 5.9558, val loss 5.9625
+saving checkpoint to out-shakespeare
+iter 10: loss 5.9322, time 31040.11ms, mfu 2.89%
+iter 11: loss 5.8374, time 23361.17ms, mfu 2.91%
+iter 12: loss 5.6069, time 23241.27ms, mfu 2.93%
+iter 13: loss 5.6613, time 23180.06ms, mfu 2.95%
+iter 14: loss 5.2928, time 23169.15ms, mfu 2.96%
+step 15: train loss 5.4229, val loss 5.4202
+saving checkpoint to out-shakespeare
+iter 15: loss 5.3205, time 31057.72ms, mfu 2.90%
+iter 16: loss 5.4608, time 23320.27ms, mfu 2.91%
+iter 17: loss 5.2379, time 23176.04ms, mfu 2.93%
+iter 18: loss 5.1430, time 23211.53ms, mfu 2.95%
+iter 19: loss 5.5525, time 23232.59ms, mfu 2.96%
+step 20: train loss 5.1232, val loss 5.0514
+saving checkpoint to out-shakespeare
+iter 20: loss 5.1371, time 31097.85ms, mfu 2.90%
+iter 21: loss 4.9530, time 23374.38ms, mfu 2.92%

train_info/train_info_shakespeare.txt ADDED Viewed

	@@ -0,0 +1,185 @@

+Overriding config with config/finetune_shakespeare.py:
+import time
+out_dir = 'out-shakespeare'
+eval_interval = 5
+eval_iters = 40
+wandb_log = False # feel free to turn on
+wandb_project = 'shakespeare'
+wandb_run_name = 'ft-' + str(time.time())
+dataset = 'shakespeare'
+init_from = 'gpt2' # this is the largest GPT-2 model
+# only save checkpoints if the validation loss improves
+always_save_checkpoint = False
+# the number of examples per iter:
+# 1 batch_size * 32 grad_accum * 1024 tokens = 32,768 tokens/iter
+# shakespeare has 301,966 tokens, so 1 epoch ~= 9.2 iters
+batch_size = 1
+gradient_accumulation_steps = 32
+max_iters = 1000
+# finetune at constant LR
+learning_rate = 3e-5
+decay_lr = False
+Initializing from OpenAI GPT-2 weights: gpt2
+loading weights from pretrained gpt: gpt2
+forcing vocab_size=50257, block_size=1024, bias=True
+overriding dropout rate to 0.0
+number of parameters: 123.65M
+using fused AdamW: True
+compiling the model... (takes a ~minute)
+[2023-03-20 21:31:13,957] torch._inductor.utils: [WARNING] make_fallback(aten.addmv): a decomposition exists, we should switch to it
+step 0: train loss 4.1871, val loss 4.0326
+iter 0: loss 4.8126, time 53610.16ms, mfu -100.00%
+iter 1: loss 3.8469, time 22853.81ms, mfu -100.00%
+iter 2: loss 4.1342, time 23058.41ms, mfu -100.00%
+iter 3: loss 4.2060, time 23164.17ms, mfu -100.00%
+iter 4: loss 4.6711, time 23070.16ms, mfu -100.00%
+step 5: train loss 4.3096, val loss 3.9636
+saving checkpoint to out-shakespeare
+iter 5: loss 3.4577, time 30970.06ms, mfu 2.32%
+iter 6: loss 2.9587, time 23298.83ms, mfu 2.40%
+iter 7: loss 3.2116, time 23132.08ms, mfu 2.47%
+iter 8: loss 3.4900, time 23106.50ms, mfu 2.53%
+iter 9: loss 3.8003, time 23125.60ms, mfu 2.59%
+step 10: train loss 3.6215, val loss 3.4816
+saving checkpoint to out-shakespeare
+iter 10: loss 3.6364, time 30978.89ms, mfu 2.56%
+iter 11: loss 3.4725, time 23263.91ms, mfu 2.61%
+iter 12: loss 3.4080, time 23053.16ms, mfu 2.67%
+iter 13: loss 3.9510, time 23091.76ms, mfu 2.71%
+iter 14: loss 3.6421, time 23142.46ms, mfu 2.75%
+step 15: train loss 3.5292, val loss 3.2960
+saving checkpoint to out-shakespeare
+iter 15: loss 3.2916, time 31036.47ms, mfu 2.71%
+iter 16: loss 3.8844, time 23232.40ms, mfu 2.74%
+iter 17: loss 3.2954, time 23076.36ms, mfu 2.78%
+iter 18: loss 2.9807, time 23073.19ms, mfu 2.81%
+iter 19: loss 3.4524, time 23090.94ms, mfu 2.84%
+step 20: train loss 3.4621, val loss 3.3625
+iter 20: loss 3.3737, time 25115.53ms, mfu 2.85%
+iter 21: loss 3.6565, time 23165.72ms, mfu 2.87%
+iter 22: loss 3.3047, time 23174.77ms, mfu 2.89%
+iter 23: loss 3.8091, time 23135.82ms, mfu 2.92%
+iter 24: loss 3.1955, time 23097.90ms, mfu 2.94%
+step 25: train loss 3.5139, val loss 3.2854
+saving checkpoint to out-shakespeare
+iter 25: loss 3.8481, time 30838.74ms, mfu 2.87%
+iter 26: loss 3.2716, time 23304.59ms, mfu 2.90%
+iter 27: loss 3.3729, time 23056.31ms, mfu 2.92%
+iter 28: loss 3.3545, time 23107.46ms, mfu 2.94%
+iter 29: loss 2.7101, time 23209.45ms, mfu 2.95%
+step 30: train loss 3.3706, val loss 3.2958
+iter 30: loss 3.0968, time 25123.31ms, mfu 2.94%
+iter 31: loss 2.9495, time 23116.72ms, mfu 2.96%
+iter 32: loss 3.0179, time 23101.19ms, mfu 2.97%
+iter 33: loss 2.9648, time 23117.17ms, mfu 2.99%
+iter 34: loss 3.6522, time 23132.76ms, mfu 3.00%
+step 35: train loss 3.3923, val loss 3.2125
+saving checkpoint to out-shakespeare
+iter 35: loss 3.2469, time 31079.08ms, mfu 2.93%
+iter 36: loss 3.1450, time 23273.02ms, mfu 2.95%
+iter 37: loss 3.4624, time 23046.04ms, mfu 2.96%
+iter 38: loss 3.4371, time 23102.73ms, mfu 2.98%
+iter 39: loss 3.3130, time 23178.65ms, mfu 2.99%
+step 40: train loss 3.3233, val loss 3.2543
+iter 40: loss 3.0743, time 25069.68ms, mfu 2.98%
+iter 41: loss 3.1269, time 23084.39ms, mfu 2.99%
+iter 42: loss 3.6785, time 23076.30ms, mfu 3.00%
+iter 43: loss 3.3787, time 23075.87ms, mfu 3.01%
+iter 44: loss 3.2637, time 23098.68ms, mfu 3.02%
+step 45: train loss 3.1971, val loss 3.2642
+iter 45: loss 3.1861, time 25003.67ms, mfu 3.01%
+iter 46: loss 3.4037, time 23106.62ms, mfu 3.02%
+iter 47: loss 3.4947, time 23109.37ms, mfu 3.03%
+iter 48: loss 3.3276, time 23098.50ms, mfu 3.04%
+iter 49: loss 2.9062, time 23171.38ms, mfu 3.04%
+step 50: train loss 3.2188, val loss 3.2460
+iter 50: loss 3.5280, time 25111.46ms, mfu 3.02%
+iter 51: loss 3.5470, time 23143.40ms, mfu 3.03%
+iter 52: loss 3.1881, time 23109.22ms, mfu 3.04%
+iter 53: loss 3.4332, time 23083.68ms, mfu 3.05%
+iter 54: loss 3.1956, time 23117.10ms, mfu 3.05%
+step 55: train loss 3.2902, val loss 3.1846
+saving checkpoint to out-shakespeare
+iter 55: loss 3.4816, time 31132.51ms, mfu 2.98%
+iter 56: loss 3.2971, time 23207.94ms, mfu 2.99%
+iter 57: loss 2.9543, time 23064.74ms, mfu 3.00%
+iter 58: loss 2.8729, time 23093.16ms, mfu 3.01%
+iter 59: loss 3.0883, time 23129.34ms, mfu 3.02%
+step 60: train loss 3.1288, val loss 3.1545
+saving checkpoint to out-shakespeare
+iter 60: loss 3.7098, time 31022.27ms, mfu 2.95%
+iter 61: loss 3.4157, time 23229.02ms, mfu 2.97%
+iter 62: loss 3.0020, time 23059.02ms, mfu 2.98%
+iter 63: loss 3.0751, time 23063.51ms, mfu 2.99%
+iter 64: loss 2.9081, time 23134.60ms, mfu 3.01%
+step 65: train loss 3.2254, val loss 3.1772
+iter 65: loss 3.3802, time 25114.58ms, mfu 2.99%
+iter 66: loss 3.1073, time 23118.96ms, mfu 3.00%
+iter 67: loss 3.1010, time 23081.32ms, mfu 3.01%
+iter 68: loss 3.2594, time 23058.54ms, mfu 3.02%
+iter 69: loss 3.4402, time 23062.45ms, mfu 3.03%
+step 70: train loss 3.1511, val loss 3.2315
+iter 70: loss 3.4094, time 24967.39ms, mfu 3.02%
+iter 71: loss 3.0997, time 23070.28ms, mfu 3.03%
+iter 72: loss 2.1573, time 23072.48ms, mfu 3.04%
+iter 73: loss 3.3926, time 23060.80ms, mfu 3.04%
+iter 74: loss 3.2284, time 23080.48ms, mfu 3.05%
+step 75: train loss 3.1102, val loss 3.1017
+saving checkpoint to out-shakespeare
+iter 75: loss 3.3760, time 31003.52ms, mfu 2.98%
+iter 76: loss 3.3387, time 23207.33ms, mfu 2.99%
+iter 77: loss 2.9299, time 23040.87ms, mfu 3.00%
+iter 78: loss 2.9623, time 23069.43ms, mfu 3.01%
+iter 79: loss 3.0674, time 23111.04ms, mfu 3.02%
+step 80: train loss 3.0574, val loss 3.2178
+iter 80: loss 2.6808, time 25072.69ms, mfu 3.01%
+iter 81: loss 2.7986, time 23144.88ms, mfu 3.02%
+iter 82: loss 2.9121, time 23094.25ms, mfu 3.03%
+iter 83: loss 2.7153, time 23114.27ms, mfu 3.03%
+iter 84: loss 2.8444, time 23089.41ms, mfu 3.04%
+step 85: train loss 2.9855, val loss 3.2298
+iter 85: loss 3.0517, time 25033.77ms, mfu 3.03%
+iter 86: loss 2.5920, time 23088.89ms, mfu 3.03%
+iter 87: loss 3.1241, time 23084.88ms, mfu 3.04%
+iter 88: loss 2.5355, time 23070.40ms, mfu 3.05%
+iter 89: loss 3.4543, time 23060.05ms, mfu 3.06%
+step 90: train loss 3.0426, val loss 3.2664
+iter 90: loss 3.3099, time 24997.54ms, mfu 3.04%
+iter 91: loss 2.8099, time 23108.94ms, mfu 3.04%
+iter 92: loss 3.2419, time 23103.54ms, mfu 3.05%
+iter 93: loss 3.4718, time 23089.71ms, mfu 3.06%
+iter 94: loss 3.0708, time 23137.11ms, mfu 3.06%
+step 95: train loss 3.0225, val loss 3.2529
+iter 95: loss 2.8545, time 25072.26ms, mfu 3.04%
+iter 96: loss 3.3059, time 23120.57ms, mfu 3.05%
+iter 97: loss 2.7528, time 23111.60ms, mfu 3.06%
+iter 98: loss 3.1788, time 23106.26ms, mfu 3.06%
+iter 99: loss 2.9023, time 23103.06ms, mfu 3.07%
+step 100: train loss 2.9153, val loss 3.2140
+iter 100: loss 3.0090, time 24968.37ms, mfu 3.05%
+iter 101: loss 3.0753, time 23093.87ms, mfu 3.05%
+iter 102: loss 3.1295, time 23108.81ms, mfu 3.06%
+iter 103: loss 2.9033, time 23136.51ms, mfu 3.06%
+iter 104: loss 3.1117, time 23127.17ms, mfu 3.07%
+step 105: train loss 2.9402, val loss 3.2071
+iter 105: loss 2.8862, time 25050.88ms, mfu 3.05%
+iter 106: loss 2.6040, time 23141.23ms, mfu 3.05%
+iter 107: loss 3.1831, time 23146.47ms, mfu 3.06%
+iter 108: loss 3.1619, time 23078.47ms, mfu 3.06%
+iter 109: loss 3.0995, time 23098.26ms, mfu 3.07%
+step 110: train loss 2.7568, val loss 3.2857
+iter 110: loss 3.0392, time 24959.72ms, mfu 3.05%
+iter 111: loss 3.1982, time 23121.36ms, mfu 3.06%
+iter 112: loss 3.1794, time 23124.92ms, mfu 3.06%
+iter 113: loss 2.8230, time 23138.96ms, mfu 3.07%
+iter 114: loss 2.2634, time 23121.12ms, mfu 3.07%
+step 115: train loss 2.8576, val loss 3.2603
+iter 115: loss 3.0414, time 24960.16ms, mfu 3.05%
+iter 116: loss 2.2827, time 23077.89ms, mfu 3.06%
+iter 117: loss 2.5435, time 23054.11ms, mfu 3.06%

train_info/train_info_trump.txt ADDED Viewed

	@@ -0,0 +1,207 @@

+Overriding config with config/finetune_shakespeare.py:
+import time
+out_dir = 'out-shakespeare'
+eval_interval = 5
+eval_iters = 40
+wandb_log = False # feel free to turn on
+wandb_project = 'shakespeare'
+wandb_run_name = 'ft-' + str(time.time())
+dataset = 'shakespeare'
+init_from = 'gpt2' # this is the largest GPT-2 model
+# only save checkpoints if the validation loss improves
+always_save_checkpoint = False
+# the number of examples per iter:
+# 1 batch_size * 32 grad_accum * 1024 tokens = 32,768 tokens/iter
+# shakespeare has 301,966 tokens, so 1 epoch ~= 9.2 iters
+batch_size = 1
+gradient_accumulation_steps = 32
+max_iters = 300
+# finetune at constant LR
+learning_rate = 3e-5
+decay_lr = False
+Initializing from OpenAI GPT-2 weights: gpt2
+loading weights from pretrained gpt: gpt2
+forcing vocab_size=50257, block_size=1024, bias=True
+overriding dropout rate to 0.0
+number of parameters: 123.65M
+Downloading (…)lve/main/config.json: 100% 665/665 [00:00<00:00, 88.4kB/s]
+Downloading pytorch_model.bin: 100% 548M/548M [00:01<00:00, 289MB/s]
+Downloading (…)neration_config.json: 100% 124/124 [00:00<00:00, 22.5kB/s]
+using fused AdamW: True
+compiling the model... (takes a ~minute)
+[2023-03-21 06:17:18,366] torch._inductor.utils: [WARNING] make_fallback(aten.addmv): a decomposition exists, we should switch to it
+step 0: train loss 3.3086, val loss 3.2349
+iter 0: loss 3.4443, time 75907.68ms, mfu -100.00%
+iter 1: loss 3.6624, time 23156.16ms, mfu -100.00%
+iter 2: loss 4.4039, time 23248.46ms, mfu -100.00%
+iter 3: loss 3.2693, time 22877.27ms, mfu -100.00%
+iter 4: loss 3.4597, time 22906.52ms, mfu -100.00%
+step 5: train loss 3.2166, val loss 3.2212
+saving checkpoint to out-shakespeare
+iter 5: loss 3.2885, time 30843.38ms, mfu 2.33%
+iter 6: loss 3.2423, time 23117.67ms, mfu 2.41%
+iter 7: loss 3.2239, time 23014.83ms, mfu 2.48%
+iter 8: loss 3.3878, time 23083.71ms, mfu 2.54%
+iter 9: loss 3.0245, time 23127.68ms, mfu 2.60%
+step 10: train loss 3.1367, val loss 3.0886
+saving checkpoint to out-shakespeare
+iter 10: loss 3.2588, time 31026.66ms, mfu 2.57%
+iter 11: loss 2.7963, time 23215.41ms, mfu 2.62%
+iter 12: loss 3.0799, time 23045.69ms, mfu 2.67%
+iter 13: loss 3.0391, time 23081.70ms, mfu 2.72%
+iter 14: loss 2.9285, time 23144.99ms, mfu 2.76%
+step 15: train loss 3.0614, val loss 3.0357
+saving checkpoint to out-shakespeare
+iter 15: loss 2.9088, time 31131.17ms, mfu 2.71%
+iter 16: loss 2.8854, time 23203.33ms, mfu 2.75%
+iter 17: loss 2.8941, time 23045.51ms, mfu 2.79%
+iter 18: loss 3.1116, time 23058.43ms, mfu 2.82%
+iter 19: loss 3.1542, time 23076.86ms, mfu 2.85%
+step 20: train loss 2.9382, val loss 2.9662
+saving checkpoint to out-shakespeare
+iter 20: loss 2.8674, time 30800.95ms, mfu 2.80%
+iter 21: loss 3.0158, time 23210.44ms, mfu 2.83%
+iter 22: loss 3.0376, time 23028.93ms, mfu 2.86%
+iter 23: loss 2.5614, time 23053.57ms, mfu 2.88%
+iter 24: loss 3.0086, time 23135.53ms, mfu 2.90%
+step 25: train loss 2.9386, val loss 2.9689
+iter 25: loss 2.8633, time 25037.75ms, mfu 2.90%
+iter 26: loss 3.2887, time 23087.04ms, mfu 2.92%
+iter 27: loss 2.7507, time 23061.28ms, mfu 2.94%
+iter 28: loss 3.0676, time 23047.93ms, mfu 2.96%
+iter 29: loss 2.7316, time 23042.36ms, mfu 2.98%
+step 30: train loss 2.9721, val loss 2.9042
+saving checkpoint to out-shakespeare
+iter 30: loss 2.7163, time 30867.03ms, mfu 2.91%
+iter 31: loss 2.9423, time 23225.75ms, mfu 2.93%
+iter 32: loss 2.9405, time 23012.47ms, mfu 2.95%
+iter 33: loss 2.9208, time 23059.76ms, mfu 2.96%
+iter 34: loss 2.9996, time 23121.13ms, mfu 2.98%
+step 35: train loss 2.9496, val loss 2.8374
+saving checkpoint to out-shakespeare
+iter 35: loss 2.8072, time 31122.96ms, mfu 2.91%
+iter 36: loss 2.9798, time 23209.16ms, mfu 2.93%
+iter 37: loss 2.8476, time 23019.32ms, mfu 2.95%
+iter 38: loss 2.7276, time 23056.09ms, mfu 2.97%
+iter 39: loss 2.8636, time 23101.19ms, mfu 2.98%
+step 40: train loss 2.8282, val loss 2.9073
+iter 40: loss 2.7667, time 25022.64ms, mfu 2.97%
+iter 41: loss 2.6111, time 23100.99ms, mfu 2.98%
+iter 42: loss 3.1776, time 23107.88ms, mfu 3.00%
+iter 43: loss 2.7963, time 23090.82ms, mfu 3.01%
+iter 44: loss 3.2658, time 23084.78ms, mfu 3.02%
+step 45: train loss 2.8171, val loss 2.8487
+iter 45: loss 3.0523, time 24981.39ms, mfu 3.00%
+iter 46: loss 2.6204, time 23087.28ms, mfu 3.01%
+iter 47: loss 2.8938, time 23081.95ms, mfu 3.02%
+iter 48: loss 3.1726, time 23092.57ms, mfu 3.03%
+iter 49: loss 3.7836, time 23077.55ms, mfu 3.04%
+step 50: train loss 2.8675, val loss 2.7787
+saving checkpoint to out-shakespeare
+iter 50: loss 3.0882, time 30881.37ms, mfu 2.97%
+iter 51: loss 2.8358, time 23200.14ms, mfu 2.98%
+iter 52: loss 2.9847, time 23008.69ms, mfu 3.00%
+iter 53: loss 3.1992, time 23066.07ms, mfu 3.01%
+iter 54: loss 2.4085, time 23118.93ms, mfu 3.02%
+step 55: train loss 2.8049, val loss 2.7507
+saving checkpoint to out-shakespeare
+iter 55: loss 2.9964, time 31115.78ms, mfu 2.95%
+iter 56: loss 2.9647, time 23212.73ms, mfu 2.96%
+iter 57: loss 2.8880, time 23003.95ms, mfu 2.98%
+iter 58: loss 2.8726, time 23053.90ms, mfu 2.99%
+iter 59: loss 2.6470, time 23124.33ms, mfu 3.00%
+step 60: train loss 2.8041, val loss 2.8827
+iter 60: loss 2.8115, time 24978.80ms, mfu 2.99%
+iter 61: loss 2.6765, time 23058.07ms, mfu 3.00%
+iter 62: loss 2.6801, time 23052.27ms, mfu 3.01%
+iter 63: loss 3.4295, time 23048.58ms, mfu 3.03%
+iter 64: loss 2.5933, time 23062.70ms, mfu 3.03%
+step 65: train loss 2.7894, val loss 2.7606
+iter 65: loss 2.5231, time 24991.85ms, mfu 3.02%
+iter 66: loss 2.8913, time 23099.31ms, mfu 3.03%
+iter 67: loss 2.9515, time 23106.81ms, mfu 3.04%
+iter 68: loss 2.8017, time 23098.12ms, mfu 3.04%
+iter 69: loss 2.7759, time 23110.16ms, mfu 3.05%
+step 70: train loss 2.8044, val loss 2.8498
+iter 70: loss 2.9694, time 25009.31ms, mfu 3.03%
+iter 71: loss 3.3238, time 23090.32ms, mfu 3.04%
+iter 72: loss 2.6931, time 23086.35ms, mfu 3.05%
+iter 73: loss 2.6097, time 23085.74ms, mfu 3.05%
+iter 74: loss 2.1781, time 23096.25ms, mfu 3.06%
+step 75: train loss 2.7755, val loss 2.6869
+saving checkpoint to out-shakespeare
+iter 75: loss 2.9208, time 30879.90ms, mfu 2.99%
+iter 76: loss 2.7619, time 23186.69ms, mfu 3.00%
+iter 77: loss 2.8394, time 23017.46ms, mfu 3.01%
+iter 78: loss 2.5907, time 23049.26ms, mfu 3.02%
+iter 79: loss 2.5660, time 23102.38ms, mfu 3.03%
+step 80: train loss 2.7759, val loss 2.7603
+iter 80: loss 2.6889, time 25011.13ms, mfu 3.01%
+iter 81: loss 2.6940, time 23088.64ms, mfu 3.02%
+iter 82: loss 2.6596, time 23050.35ms, mfu 3.03%
+iter 83: loss 2.7638, time 23066.22ms, mfu 3.04%
+iter 84: loss 2.6515, time 23059.01ms, mfu 3.05%
+step 85: train loss 2.7404, val loss 2.7290
+iter 85: loss 3.1829, time 24970.26ms, mfu 3.03%
+iter 86: loss 2.5451, time 23052.03ms, mfu 3.04%
+iter 87: loss 2.4363, time 23051.53ms, mfu 3.05%
+iter 88: loss 2.8023, time 23039.12ms, mfu 3.05%
+iter 89: loss 2.4755, time 23044.45ms, mfu 3.06%
+step 90: train loss 2.7140, val loss 2.7692
+iter 90: loss 2.7225, time 24960.52ms, mfu 3.04%
+iter 91: loss 2.4655, time 23037.54ms, mfu 3.05%
+iter 92: loss 2.5291, time 23029.37ms, mfu 3.06%
+iter 93: loss 2.7720, time 23032.99ms, mfu 3.06%
+iter 94: loss 2.7614, time 23039.50ms, mfu 3.07%
+step 95: train loss 2.7932, val loss 2.7953
+iter 95: loss 2.6881, time 24974.66ms, mfu 3.05%
+iter 96: loss 2.9315, time 23044.89ms, mfu 3.06%
+iter 97: loss 2.7099, time 23035.52ms, mfu 3.06%
+iter 98: loss 2.6858, time 23036.10ms, mfu 3.07%
+iter 99: loss 2.5341, time 23048.24ms, mfu 3.07%
+step 100: train loss 2.6788, val loss 2.8138
+iter 100: loss 2.7993, time 25008.37ms, mfu 3.05%
+iter 101: loss 2.5996, time 23052.62ms, mfu 3.06%
+iter 102: loss 2.7768, time 23059.09ms, mfu 3.07%
+iter 103: loss 2.6378, time 23046.82ms, mfu 3.07%
+iter 104: loss 2.7511, time 23043.40ms, mfu 3.08%
+step 105: train loss 2.7542, val loss 2.6568
+saving checkpoint to out-shakespeare
+iter 105: loss 2.6596, time 31000.96ms, mfu 3.00%
+iter 106: loss 2.8566, time 23195.71ms, mfu 3.01%
+iter 107: loss 2.6284, time 22995.46ms, mfu 3.02%
+iter 108: loss 2.6670, time 23031.45ms, mfu 3.03%
+iter 109: loss 2.4732, time 23093.11ms, mfu 3.04%
+step 110: train loss 2.7094, val loss 2.6684
+iter 110: loss 2.5577, time 25028.10ms, mfu 3.02%
+iter 111: loss 2.9250, time 23089.98ms, mfu 3.03%
+iter 112: loss 2.6274, time 23072.14ms, mfu 3.04%
+iter 113: loss 2.5337, time 23078.52ms, mfu 3.05%
+iter 114: loss 2.7248, time 23061.41ms, mfu 3.05%
+step 115: train loss 2.7062, val loss 2.7398
+iter 115: loss 2.7654, time 24968.79ms, mfu 3.04%
+iter 116: loss 2.6394, time 23049.91ms, mfu 3.04%
+iter 117: loss 2.5259, time 23068.72ms, mfu 3.05%
+iter 118: loss 2.8312, time 23061.73ms, mfu 3.06%
+iter 119: loss 2.6137, time 23049.41ms, mfu 3.06%
+step 120: train loss 2.6704, val loss 2.7120
+iter 120: loss 2.6794, time 24958.89ms, mfu 3.05%
+iter 121: loss 2.7400, time 23040.45ms, mfu 3.05%
+iter 122: loss 2.6322, time 23047.61ms, mfu 3.06%
+iter 123: loss 2.4416, time 23062.33ms, mfu 3.06%
+iter 124: loss 2.6756, time 23048.99ms, mfu 3.07%
+step 125: train loss 2.5866, val loss 2.6882
+iter 125: loss 2.6490, time 24950.30ms, mfu 3.05%
+iter 126: loss 2.5888, time 23027.86ms, mfu 3.06%
+iter 127: loss 2.3960, time 23012.31ms, mfu 3.06%
+iter 128: loss 2.6581, time 23025.51ms, mfu 3.07%
+iter 129: loss 2.6202, time 23042.65ms, mfu 3.07%
+step 130: train loss 2.6151, val loss 2.6532
+saving checkpoint to out-shakespeare
+iter 130: loss 2.8148, time 31009.76ms, mfu 3.00%

ckpt_edsheeran.pt → weights/ckpt_edsheeran.pt RENAMED Viewed

File without changes

ckpt_haiku.pt → weights/ckpt_haiku.pt RENAMED Viewed

File without changes

ckpt_math.pt → weights/ckpt_math.pt RENAMED Viewed

File without changes

ckpt_shakespear.pt → weights/ckpt_shakespear.pt RENAMED Viewed

File without changes

ckpt_trump.pt → weights/ckpt_trump.pt RENAMED Viewed

File without changes

ckpt_world_facts_cia.pt → weights/ckpt_world_facts_cia.pt RENAMED Viewed

File without changes