Hugging Face's logo

Spaces:
Ritori
/
play_with_baby_llama2
Sleeping

App Files Community
Ritori commited on
Commit
613e5d1
•
1 Parent(s): f9ae417

Upload folder using huggingface_hub

Browse files
Files changed (39) hide show
  1. .config/.last_opt_in_prompt.yaml +1 -0
  2. .config/.last_survey_prompt.yaml +1 -0
  3. .config/.last_update_check.json +1 -0
  4. .config/active_config +1 -0
  5. .config/config_sentinel +0 -0
  6. .config/configurations/config_default +6 -0
  7. .config/default_configs.db +0 -0
  8. .config/gce +1 -0
  9. .config/logs/2023.07.20/13.26.51.001473.log +596 -0
  10. .config/logs/2023.07.20/13.27.17.369260.log +5 -0
  11. .config/logs/2023.07.20/13.27.43.121533.log +169 -0
  12. .config/logs/2023.07.20/13.27.50.747950.log +5 -0
  13. .config/logs/2023.07.20/13.28.16.714039.log +8 -0
  14. .config/logs/2023.07.20/13.28.17.509819.log +8 -0
  15. .gitattributes +2 -0
  16. LICENSE +21 -0
  17. README.md +153 -7
  18. baby_llama2.py +25 -0
  19. configurator.py +47 -0
  20. import subprocess.py +29 -0
  21. model.bin +3 -0
  22. model.py +370 -0
  23. requirements.txt +8 -0
  24. run +0 -0
  25. run.c +490 -0
  26. run.exe +0 -0
  27. sample.py +69 -0
  28. sample_data/README.md +19 -0
  29. sample_data/anscombe.json +49 -0
  30. sample_data/california_housing_test.csv +0 -0
  31. sample_data/california_housing_train.csv +0 -0
  32. sample_data/mnist_test.csv +3 -0
  33. sample_data/mnist_train_small.csv +3 -0
  34. test_all.py +53 -0
  35. tinystories.py +166 -0
  36. tokenizer.bin +3 -0
  37. tokenizer.model +3 -0
  38. tokenizer.py +65 -0
  39. train.py +331 -0
.config/.last_opt_in_prompt.yaml ADDED
@@ -0,0 +1 @@
 
 
1
+ {}
.config/.last_survey_prompt.yaml ADDED
@@ -0,0 +1 @@
 
 
1
+ last_prompt_time: 1689859662.3183093
.config/.last_update_check.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"last_update_check_time": 1689859670.1382627, "last_update_check_revision": 20230714124024, "notifications": [], "last_nag_times": {}}
.config/active_config ADDED
@@ -0,0 +1 @@
 
 
1
+ default
.config/config_sentinel ADDED
File without changes
.config/configurations/config_default ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ [component_manager]
2
+ disable_update_check = true
3
+
4
+ [compute]
5
+ gce_metadata_read_timeout_sec = 0
6
+
.config/default_configs.db ADDED
Binary file (12.3 kB). View file
 
.config/gce ADDED
@@ -0,0 +1 @@
 
 
1
+ False
.config/logs/2023.07.20/13.26.51.001473.log ADDED
@@ -0,0 +1,596 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2023-07-20 13:26:51,007 DEBUG root Loaded Command Group: ['gcloud', 'components']
2
+ 2023-07-20 13:26:51,012 DEBUG root Loaded Command Group: ['gcloud', 'components', 'update']
3
+ 2023-07-20 13:26:51,014 DEBUG root Running [gcloud.components.update] with arguments: [--allow-no-backup: "True", --compile-python: "True", --quiet: "True", COMPONENT-IDS:7: "['core', 'gcloud-deps', 'bq', 'gcloud', 'gcloud-crc32c', 'gsutil', 'anthoscli']"]
4
+ 2023-07-20 13:26:51,015 INFO ___FILE_ONLY___ Beginning update. This process may take several minutes.
5
+
6
+ 2023-07-20 13:27:03,064 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
7
+ 2023-07-20 13:27:03,161 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components-2.json HTTP/1.1" 200 208282
8
+ 2023-07-20 13:27:03,175 INFO ___FILE_ONLY___
9
+
10
+ 2023-07-20 13:27:03,176 INFO ___FILE_ONLY___
11
+ Your current Google Cloud CLI version is: 439.0.0
12
+
13
+ 2023-07-20 13:27:03,176 INFO ___FILE_ONLY___ Installing components from version: 439.0.0
14
+
15
+ 2023-07-20 13:27:03,176 INFO ___FILE_ONLY___
16
+
17
+ 2023-07-20 13:27:03,176 DEBUG root Chosen display Format:table[box,title="These components will be removed."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
18
+ 2023-07-20 13:27:03,177 DEBUG root Chosen display Format:table[box,title="These components will be updated."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
19
+ 2023-07-20 13:27:03,177 DEBUG root Chosen display Format:table[box,title="These components will be installed."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
20
+ 2023-07-20 13:27:03,184 INFO ___FILE_ONLY___ ┌─────────────────────────────────────────────────────────────────────────────┐
21
+ 2023-07-20 13:27:03,184 INFO ___FILE_ONLY___
22
+
23
+ 2023-07-20 13:27:03,184 INFO ___FILE_ONLY___ │ These components will be installed. │
24
+ 2023-07-20 13:27:03,184 INFO ___FILE_ONLY___
25
+
26
+ 2023-07-20 13:27:03,184 INFO ___FILE_ONLY___ ├─────────────────────────────────────────────────────┬────────────┬──────────┤
27
+ 2023-07-20 13:27:03,184 INFO ___FILE_ONLY___
28
+
29
+ 2023-07-20 13:27:03,184 INFO ___FILE_ONLY___ │ Name │ Version │ Size │
30
+ 2023-07-20 13:27:03,184 INFO ___FILE_ONLY___
31
+
32
+ 2023-07-20 13:27:03,184 INFO ___FILE_ONLY___ ├─────────────────────────────────────────────────────┼────────────┼──────────┤
33
+ 2023-07-20 13:27:03,184 INFO ___FILE_ONLY___
34
+
35
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___ │
36
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___ BigQuery Command Line Tool
37
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___
38
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___ │
39
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___ 2.0.94
40
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___
41
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___ │
42
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___ 1.6 MiB
43
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___
44
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___ │
45
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___
46
+
47
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___ │
48
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___ BigQuery Command Line Tool (Platform Specific)
49
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___
50
+ 2023-07-20 13:27:03,185 INFO ___FILE_ONLY___ │
51
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___ 2.0.77
52
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___
53
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___ │
54
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___ < 1 MiB
55
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___
56
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___ │
57
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___
58
+
59
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___ │
60
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___ Bundled Python 3.9
61
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___
62
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___ │
63
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___ 3.9.16
64
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___
65
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___ │
66
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___ 63.6 MiB
67
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___
68
+ 2023-07-20 13:27:03,186 INFO ___FILE_ONLY___ │
69
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___
70
+
71
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___ │
72
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___ Cloud Storage Command Line Tool
73
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___
74
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___ │
75
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___ 5.25
76
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___
77
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___ │
78
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___ 11.3 MiB
79
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___
80
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___ │
81
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___
82
+
83
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___ │
84
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___ Cloud Storage Command Line Tool (Platform Specific)
85
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___
86
+ 2023-07-20 13:27:03,187 INFO ___FILE_ONLY___ │
87
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___ 5.13
88
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___
89
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___ │
90
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___ < 1 MiB
91
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___
92
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___ │
93
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___
94
+
95
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___ │
96
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___ Google Cloud CLI Core Libraries (Platform Specific)
97
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___
98
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___ │
99
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___ 2022.09.20
100
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___
101
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___ │
102
+ 2023-07-20 13:27:03,188 INFO ___FILE_ONLY___ < 1 MiB
103
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___
104
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___ │
105
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___
106
+
107
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___ │
108
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___ Google Cloud CRC32C Hash Tool
109
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___
110
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___ │
111
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___ 1.0.0
112
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___
113
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___ │
114
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___ 1.2 MiB
115
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___
116
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___ │
117
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___
118
+
119
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___ │
120
+ 2023-07-20 13:27:03,189 INFO ___FILE_ONLY___ anthoscli
121
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___
122
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___ │
123
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___ 0.2.37
124
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___
125
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___ │
126
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___ 68.4 MiB
127
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___
128
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___ │
129
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___
130
+
131
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___ │
132
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___ gcloud cli dependencies
133
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___
134
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___ │
135
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___ 2021.04.16
136
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___
137
+ 2023-07-20 13:27:03,190 INFO ___FILE_ONLY___ │
138
+ 2023-07-20 13:27:03,191 INFO ___FILE_ONLY___ < 1 MiB
139
+ 2023-07-20 13:27:03,191 INFO ___FILE_ONLY___
140
+ 2023-07-20 13:27:03,191 INFO ___FILE_ONLY___ │
141
+ 2023-07-20 13:27:03,191 INFO ___FILE_ONLY___
142
+
143
+ 2023-07-20 13:27:03,191 INFO ___FILE_ONLY___ └─────────────────────────────────────────────────────┴────────────┴──────────┘
144
+ 2023-07-20 13:27:03,191 INFO ___FILE_ONLY___
145
+
146
+ 2023-07-20 13:27:03,191 INFO ___FILE_ONLY___
147
+
148
+ 2023-07-20 13:27:03,195 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
149
+ 2023-07-20 13:27:03,295 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/RELEASE_NOTES HTTP/1.1" 200 1040023
150
+ 2023-07-20 13:27:03,371 INFO ___FILE_ONLY___ For the latest full release notes, please visit:
151
+ https://cloud.google.com/sdk/release_notes
152
+
153
+
154
+ 2023-07-20 13:27:03,374 INFO ___FILE_ONLY___ ╔═════════════════════════════════════════���══════════════════╗
155
+
156
+ 2023-07-20 13:27:03,374 INFO ___FILE_ONLY___ ╠═ Creating update staging area ═╣
157
+
158
+ 2023-07-20 13:27:03,374 INFO ___FILE_ONLY___ ╚
159
+ 2023-07-20 13:27:03,374 INFO ___FILE_ONLY___ ══════
160
+ 2023-07-20 13:27:03,375 INFO ___FILE_ONLY___ ══════
161
+ 2023-07-20 13:27:03,375 INFO ___FILE_ONLY___ ══════
162
+ 2023-07-20 13:27:03,613 INFO ___FILE_ONLY___ ═
163
+ 2023-07-20 13:27:03,657 INFO ___FILE_ONLY___ ═
164
+ 2023-07-20 13:27:03,683 INFO ___FILE_ONLY___ ═
165
+ 2023-07-20 13:27:03,711 INFO ___FILE_ONLY___ ═
166
+ 2023-07-20 13:27:03,747 INFO ___FILE_ONLY___ ═
167
+ 2023-07-20 13:27:03,775 INFO ___FILE_ONLY___ ═
168
+ 2023-07-20 13:27:03,809 INFO ___FILE_ONLY___ ═
169
+ 2023-07-20 13:27:03,855 INFO ___FILE_ONLY___ ═
170
+ 2023-07-20 13:27:03,905 INFO ___FILE_ONLY___ ═
171
+ 2023-07-20 13:27:03,984 INFO ___FILE_ONLY___ ═
172
+ 2023-07-20 13:27:04,145 INFO ___FILE_ONLY___ ═
173
+ 2023-07-20 13:27:04,298 INFO ___FILE_ONLY___ ═
174
+ 2023-07-20 13:27:04,421 INFO ___FILE_ONLY___ ═
175
+ 2023-07-20 13:27:04,467 INFO ___FILE_ONLY___ ═
176
+ 2023-07-20 13:27:04,527 INFO ___FILE_ONLY___ ═
177
+ 2023-07-20 13:27:04,565 INFO ___FILE_ONLY___ ═
178
+ 2023-07-20 13:27:04,607 INFO ___FILE_ONLY___ ═
179
+ 2023-07-20 13:27:04,649 INFO ___FILE_ONLY___ ═
180
+ 2023-07-20 13:27:04,717 INFO ___FILE_ONLY___ ═
181
+ 2023-07-20 13:27:04,761 INFO ___FILE_ONLY___ ═
182
+ 2023-07-20 13:27:04,802 INFO ___FILE_ONLY___ ═
183
+ 2023-07-20 13:27:04,845 INFO ___FILE_ONLY___ ═
184
+ 2023-07-20 13:27:04,888 INFO ___FILE_ONLY___ ═
185
+ 2023-07-20 13:27:04,946 INFO ___FILE_ONLY___ ═
186
+ 2023-07-20 13:27:04,989 INFO ___FILE_ONLY___ ═
187
+ 2023-07-20 13:27:05,036 INFO ___FILE_ONLY___ ═
188
+ 2023-07-20 13:27:05,083 INFO ___FILE_ONLY___ ═
189
+ 2023-07-20 13:27:05,123 INFO ___FILE_ONLY___ ═
190
+ 2023-07-20 13:27:05,182 INFO ___FILE_ONLY___ ═
191
+ 2023-07-20 13:27:05,239 INFO ___FILE_ONLY___ ═
192
+ 2023-07-20 13:27:05,295 INFO ___FILE_ONLY___ ═
193
+ 2023-07-20 13:27:05,338 INFO ___FILE_ONLY___ ═
194
+ 2023-07-20 13:27:05,387 INFO ___FILE_ONLY___ ═
195
+ 2023-07-20 13:27:05,439 INFO ___FILE_ONLY___ ═
196
+ 2023-07-20 13:27:05,497 INFO ___FILE_ONLY___ ═
197
+ 2023-07-20 13:27:05,547 INFO ___FILE_ONLY___ ═
198
+ 2023-07-20 13:27:05,603 INFO ___FILE_ONLY___ ═
199
+ 2023-07-20 13:27:05,663 INFO ___FILE_ONLY___ ═
200
+ 2023-07-20 13:27:05,715 INFO ___FILE_ONLY___ ═
201
+ 2023-07-20 13:27:05,761 INFO ___FILE_ONLY___ ═
202
+ 2023-07-20 13:27:05,812 INFO ___FILE_ONLY___ ═
203
+ 2023-07-20 13:27:05,859 INFO ___FILE_ONLY___ ═
204
+ 2023-07-20 13:27:05,860 INFO ___FILE_ONLY___ ╝
205
+
206
+ 2023-07-20 13:27:05,944 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
207
+
208
+ 2023-07-20 13:27:05,944 INFO ___FILE_ONLY___ ╠═ Installing: BigQuery Command Line Tool ═╣
209
+
210
+ 2023-07-20 13:27:05,944 INFO ___FILE_ONLY___ ╚
211
+ 2023-07-20 13:27:05,949 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
212
+ 2023-07-20 13:27:06,056 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-bq-20230714124024.tar.gz HTTP/1.1" 200 1695339
213
+ 2023-07-20 13:27:06,126 INFO ___FILE_ONLY___ ═
214
+ 2023-07-20 13:27:06,126 INFO ___FILE_ONLY___ ═
215
+ 2023-07-20 13:27:06,126 INFO ___FILE_ONLY___ ═
216
+ 2023-07-20 13:27:06,126 INFO ___FILE_ONLY___ ═
217
+ 2023-07-20 13:27:06,126 INFO ___FILE_ONLY___ ═
218
+ 2023-07-20 13:27:06,127 INFO ___FILE_ONLY___ ═
219
+ 2023-07-20 13:27:06,127 INFO ___FILE_ONLY___ ═
220
+ 2023-07-20 13:27:06,127 INFO ___FILE_ONLY___ ═
221
+ 2023-07-20 13:27:06,127 INFO ___FILE_ONLY___ ═
222
+ 2023-07-20 13:27:06,127 INFO ___FILE_ONLY___ ═
223
+ 2023-07-20 13:27:06,127 INFO ___FILE_ONLY___ ═
224
+ 2023-07-20 13:27:06,127 INFO ___FILE_ONLY___ ═
225
+ 2023-07-20 13:27:06,128 INFO ___FILE_ONLY___ ═
226
+ 2023-07-20 13:27:06,128 INFO ___FILE_ONLY___ ═
227
+ 2023-07-20 13:27:06,128 INFO ___FILE_ONLY___ ═
228
+ 2023-07-20 13:27:06,128 INFO ___FILE_ONLY___ ═
229
+ 2023-07-20 13:27:06,128 INFO ___FILE_ONLY___ ═
230
+ 2023-07-20 13:27:06,129 INFO ___FILE_ONLY___ ═
231
+ 2023-07-20 13:27:06,129 INFO ___FILE_ONLY___ ═
232
+ 2023-07-20 13:27:06,129 INFO ___FILE_ONLY___ ═
233
+ 2023-07-20 13:27:06,129 INFO ___FILE_ONLY___ ═
234
+ 2023-07-20 13:27:06,129 INFO ___FILE_ONLY___ ═
235
+ 2023-07-20 13:27:06,129 INFO ___FILE_ONLY___ ═
236
+ 2023-07-20 13:27:06,129 INFO ___FILE_ONLY___ ═
237
+ 2023-07-20 13:27:06,130 INFO ___FILE_ONLY___ ═
238
+ 2023-07-20 13:27:06,130 INFO ___FILE_ONLY___ ═
239
+ 2023-07-20 13:27:06,130 INFO ___FILE_ONLY___ ═
240
+ 2023-07-20 13:27:06,130 INFO ___FILE_ONLY___ ═
241
+ 2023-07-20 13:27:06,130 INFO ___FILE_ONLY___ ═
242
+ 2023-07-20 13:27:06,130 INFO ___FILE_ONLY___ ═
243
+ 2023-07-20 13:27:06,252 INFO ___FILE_ONLY___ ═
244
+ 2023-07-20 13:27:06,256 INFO ___FILE_ONLY___ ═
245
+ 2023-07-20 13:27:06,260 INFO ___FILE_ONLY___ ═
246
+ 2023-07-20 13:27:06,264 INFO ___FILE_ONLY___ ═
247
+ 2023-07-20 13:27:06,267 INFO ___FILE_ONLY___ ═
248
+ 2023-07-20 13:27:06,271 INFO ___FILE_ONLY___ ═
249
+ 2023-07-20 13:27:06,276 INFO ___FILE_ONLY___ ═
250
+ 2023-07-20 13:27:06,280 INFO ___FILE_ONLY___ ═
251
+ 2023-07-20 13:27:06,284 INFO ___FILE_ONLY___ ═
252
+ 2023-07-20 13:27:06,287 INFO ___FILE_ONLY___ ═
253
+ 2023-07-20 13:27:06,291 INFO ___FILE_ONLY___ ═
254
+ 2023-07-20 13:27:06,295 INFO ___FILE_ONLY___ ═
255
+ 2023-07-20 13:27:06,301 INFO ___FILE_ONLY___ ═
256
+ 2023-07-20 13:27:06,304 INFO ___FILE_ONLY___ ═
257
+ 2023-07-20 13:27:06,307 INFO ___FILE_ONLY___ ═
258
+ 2023-07-20 13:27:06,311 INFO ___FILE_ONLY___ ═
259
+ 2023-07-20 13:27:06,316 INFO ___FILE_ONLY___ ═
260
+ 2023-07-20 13:27:06,320 INFO ___FILE_ONLY___ ═
261
+ 2023-07-20 13:27:06,326 INFO ___FILE_ONLY___ ═
262
+ 2023-07-20 13:27:06,330 INFO ___FILE_ONLY___ ═
263
+ 2023-07-20 13:27:06,335 INFO ___FILE_ONLY___ ═
264
+ 2023-07-20 13:27:06,342 INFO ___FILE_ONLY___ ═
265
+ 2023-07-20 13:27:06,348 INFO ___FILE_ONLY___ ═
266
+ 2023-07-20 13:27:06,352 INFO ___FILE_ONLY___ ═
267
+ 2023-07-20 13:27:06,356 INFO ___FILE_ONLY___ ═
268
+ 2023-07-20 13:27:06,361 INFO ___FILE_ONLY___ ═
269
+ 2023-07-20 13:27:06,365 INFO ___FILE_ONLY___ ═
270
+ 2023-07-20 13:27:06,369 INFO ___FILE_ONLY___ ═
271
+ 2023-07-20 13:27:06,373 INFO ___FILE_ONLY___ ═
272
+ 2023-07-20 13:27:06,376 INFO ___FILE_ONLY___ ═
273
+ 2023-07-20 13:27:06,377 INFO ___FILE_ONLY___ ╝
274
+
275
+ 2023-07-20 13:27:06,393 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
276
+
277
+ 2023-07-20 13:27:06,393 INFO ___FILE_ONLY___ ╠═ Installing: BigQuery Command Line Tool (Platform Spec... ═╣
278
+
279
+ 2023-07-20 13:27:06,393 INFO ___FILE_ONLY___ ╚
280
+ 2023-07-20 13:27:06,398 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
281
+ 2023-07-20 13:27:06,499 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-bq-nix-20220920185015.tar.gz HTTP/1.1" 200 1837
282
+ 2023-07-20 13:27:06,500 INFO ___FILE_ONLY___ ══════════════════════════════
283
+ 2023-07-20 13:27:06,501 INFO ___FILE_ONLY___ ══════════════════════════════
284
+ 2023-07-20 13:27:06,501 INFO ___FILE_ONLY___ ╝
285
+
286
+ 2023-07-20 13:27:06,510 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
287
+
288
+ 2023-07-20 13:27:06,510 INFO ___FILE_ONLY___ ╠═ Installing: Bundled Python 3.9 ═╣
289
+
290
+ 2023-07-20 13:27:06,510 INFO ___FILE_ONLY___ ╚
291
+ 2023-07-20 13:27:06,514 INFO ___FILE_ONLY___ ════════════════════════════════════════════════════════════
292
+ 2023-07-20 13:27:06,514 INFO ___FILE_ONLY___ ╝
293
+
294
+ 2023-07-20 13:27:06,516 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
295
+
296
+ 2023-07-20 13:27:06,517 INFO ___FILE_ONLY___ ╠═ Installing: Bundled Python 3.9 ═╣
297
+
298
+ 2023-07-20 13:27:06,517 INFO ___FILE_ONLY___ ╚
299
+ 2023-07-20 13:27:06,521 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
300
+ 2023-07-20 13:27:06,629 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-bundled-python3-unix-linux-x86_64-20230707144938.tar.gz HTTP/1.1" 200 66719069
301
+ 2023-07-20 13:27:07,238 INFO ___FILE_ONLY___ ═
302
+ 2023-07-20 13:27:07,241 INFO ___FILE_ONLY___ ═
303
+ 2023-07-20 13:27:07,243 INFO ___FILE_ONLY___ ═
304
+ 2023-07-20 13:27:07,246 INFO ___FILE_ONLY___ ═
305
+ 2023-07-20 13:27:07,249 INFO ___FILE_ONLY___ ═
306
+ 2023-07-20 13:27:07,251 INFO ___FILE_ONLY___ ═
307
+ 2023-07-20 13:27:07,254 INFO ___FILE_ONLY___ ═
308
+ 2023-07-20 13:27:07,256 INFO ___FILE_ONLY___ ═
309
+ 2023-07-20 13:27:07,259 INFO ___FILE_ONLY___ ═
310
+ 2023-07-20 13:27:07,261 INFO ___FILE_ONLY___ ═
311
+ 2023-07-20 13:27:07,264 INFO ___FILE_ONLY___ ═
312
+ 2023-07-20 13:27:07,266 INFO ___FILE_ONLY___ ═
313
+ 2023-07-20 13:27:07,269 INFO ___FILE_ONLY___ ═
314
+ 2023-07-20 13:27:07,272 INFO ___FILE_ONLY___ ═
315
+ 2023-07-20 13:27:07,274 INFO ___FILE_ONLY___ ═
316
+ 2023-07-20 13:27:07,277 INFO ___FILE_ONLY___ ═
317
+ 2023-07-20 13:27:07,279 INFO ___FILE_ONLY___ ═
318
+ 2023-07-20 13:27:07,282 INFO ___FILE_ONLY___ ═
319
+ 2023-07-20 13:27:07,285 INFO ___FILE_ONLY___ ═
320
+ 2023-07-20 13:27:07,288 INFO ___FILE_ONLY___ ═
321
+ 2023-07-20 13:27:07,290 INFO ___FILE_ONLY___ ═
322
+ 2023-07-20 13:27:07,293 INFO ___FILE_ONLY___ ═
323
+ 2023-07-20 13:27:07,296 INFO ___FILE_ONLY___ ═
324
+ 2023-07-20 13:27:07,299 INFO ___FILE_ONLY___ ═
325
+ 2023-07-20 13:27:07,301 INFO ___FILE_ONLY___ ═
326
+ 2023-07-20 13:27:07,304 INFO ___FILE_ONLY___ ═
327
+ 2023-07-20 13:27:07,307 INFO ___FILE_ONLY___ ═
328
+ 2023-07-20 13:27:07,309 INFO ___FILE_ONLY___ ═
329
+ 2023-07-20 13:27:07,312 INFO ___FILE_ONLY___ ═
330
+ 2023-07-20 13:27:07,314 INFO ___FILE_ONLY___ ═
331
+ 2023-07-20 13:27:09,215 INFO ___FILE_ONLY___ ═
332
+ 2023-07-20 13:27:09,227 INFO ___FILE_ONLY___ ═
333
+ 2023-07-20 13:27:09,242 INFO ___FILE_ONLY___ ═
334
+ 2023-07-20 13:27:09,255 INFO ___FILE_ONLY___ ═
335
+ 2023-07-20 13:27:09,267 INFO ___FILE_ONLY___ ═
336
+ 2023-07-20 13:27:09,284 INFO ___FILE_ONLY___ ═
337
+ 2023-07-20 13:27:09,302 INFO ___FILE_ONLY___ ═
338
+ 2023-07-20 13:27:09,323 INFO ___FILE_ONLY___ ═
339
+ 2023-07-20 13:27:09,340 INFO ___FILE_ONLY___ ═
340
+ 2023-07-20 13:27:09,352 INFO ___FILE_ONLY___ ═
341
+ 2023-07-20 13:27:09,371 INFO ___FILE_ONLY___ ═
342
+ 2023-07-20 13:27:09,391 INFO ___FILE_ONLY___ ═
343
+ 2023-07-20 13:27:09,406 INFO ___FILE_ONLY___ ═
344
+ 2023-07-20 13:27:09,528 INFO ___FILE_ONLY___ ═
345
+ 2023-07-20 13:27:09,542 INFO ___FILE_ONLY___ ═
346
+ 2023-07-20 13:27:09,637 INFO ___FILE_ONLY___ ═
347
+ 2023-07-20 13:27:09,659 INFO ___FILE_ONLY___ ═
348
+ 2023-07-20 13:27:09,680 INFO ___FILE_ONLY___ ═
349
+ 2023-07-20 13:27:09,698 INFO ___FILE_ONLY___ ═
350
+ 2023-07-20 13:27:09,734 INFO ___FILE_ONLY___ ═
351
+ 2023-07-20 13:27:09,748 INFO ___FILE_ONLY___ ═
352
+ 2023-07-20 13:27:09,761 INFO ___FILE_ONLY___ ═
353
+ 2023-07-20 13:27:09,777 INFO ___FILE_ONLY___ ═
354
+ 2023-07-20 13:27:09,794 INFO ___FILE_ONLY___ ═
355
+ 2023-07-20 13:27:09,813 INFO ___FILE_ONLY___ ═
356
+ 2023-07-20 13:27:09,831 INFO ___FILE_ONLY___ ═
357
+ 2023-07-20 13:27:10,450 INFO ___FILE_ONLY___ ═
358
+ 2023-07-20 13:27:10,740 INFO ___FILE_ONLY___ ═
359
+ 2023-07-20 13:27:10,751 INFO ___FILE_ONLY___ ═
360
+ 2023-07-20 13:27:10,763 INFO ___FILE_ONLY___ ═
361
+ 2023-07-20 13:27:10,763 INFO ___FILE_ONLY___ ╝
362
+
363
+ 2023-07-20 13:27:10,809 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
364
+
365
+ 2023-07-20 13:27:10,809 INFO ___FILE_ONLY___ ╠═ Installing: Cloud Storage Command Line Tool ═╣
366
+
367
+ 2023-07-20 13:27:10,809 INFO ___FILE_ONLY___ ╚
368
+ 2023-07-20 13:27:10,814 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
369
+ 2023-07-20 13:27:10,919 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-gsutil-20230707144938.tar.gz HTTP/1.1" 200 11823782
370
+ 2023-07-20 13:27:11,063 INFO ___FILE_ONLY___ ═
371
+ 2023-07-20 13:27:11,063 INFO ___FILE_ONLY___ ═
372
+ 2023-07-20 13:27:11,064 INFO ___FILE_ONLY___ ═
373
+ 2023-07-20 13:27:11,064 INFO ___FILE_ONLY___ ═
374
+ 2023-07-20 13:27:11,065 INFO ___FILE_ONLY___ ═
375
+ 2023-07-20 13:27:11,066 INFO ___FILE_ONLY___ ═
376
+ 2023-07-20 13:27:11,066 INFO ___FILE_ONLY___ ═
377
+ 2023-07-20 13:27:11,067 INFO ___FILE_ONLY___ ═
378
+ 2023-07-20 13:27:11,067 INFO ___FILE_ONLY___ ═
379
+ 2023-07-20 13:27:11,068 INFO ___FILE_ONLY___ ═
380
+ 2023-07-20 13:27:11,068 INFO ___FILE_ONLY___ ═
381
+ 2023-07-20 13:27:11,069 INFO ___FILE_ONLY___ ═
382
+ 2023-07-20 13:27:11,069 INFO ___FILE_ONLY___ ═
383
+ 2023-07-20 13:27:11,070 INFO ___FILE_ONLY___ ═
384
+ 2023-07-20 13:27:11,070 INFO ___FILE_ONLY___ ═
385
+ 2023-07-20 13:27:11,071 INFO ___FILE_ONLY___ ═
386
+ 2023-07-20 13:27:11,071 INFO ___FILE_ONLY___ ═
387
+ 2023-07-20 13:27:11,072 INFO ___FILE_ONLY___ ═
388
+ 2023-07-20 13:27:11,072 INFO ___FILE_ONLY___ ═
389
+ 2023-07-20 13:27:11,073 INFO ___FILE_ONLY___ ═
390
+ 2023-07-20 13:27:11,073 INFO ___FILE_ONLY___ ═
391
+ 2023-07-20 13:27:11,074 INFO ___FILE_ONLY___ ═
392
+ 2023-07-20 13:27:11,074 INFO ___FILE_ONLY___ ═
393
+ 2023-07-20 13:27:11,075 INFO ___FILE_ONLY___ ═
394
+ 2023-07-20 13:27:11,075 INFO ___FILE_ONLY___ ═
395
+ 2023-07-20 13:27:11,076 INFO ___FILE_ONLY___ ═
396
+ 2023-07-20 13:27:11,076 INFO ___FILE_ONLY___ ═
397
+ 2023-07-20 13:27:11,077 INFO ___FILE_ONLY___ ═
398
+ 2023-07-20 13:27:11,078 INFO ___FILE_ONLY___ ═
399
+ 2023-07-20 13:27:11,078 INFO ___FILE_ONLY___ ═
400
+ 2023-07-20 13:27:11,765 INFO ___FILE_ONLY___ ═
401
+ 2023-07-20 13:27:11,794 INFO ___FILE_ONLY___ ═
402
+ 2023-07-20 13:27:11,819 INFO ___FILE_ONLY___ ═
403
+ 2023-07-20 13:27:11,848 INFO ___FILE_ONLY___ ═
404
+ 2023-07-20 13:27:11,865 INFO ___FILE_ONLY___ ═
405
+ 2023-07-20 13:27:11,885 INFO ___FILE_ONLY___ ═
406
+ 2023-07-20 13:27:11,906 INFO ___FILE_ONLY___ ═
407
+ 2023-07-20 13:27:11,929 INFO ___FILE_ONLY___ ═
408
+ 2023-07-20 13:27:11,953 INFO ___FILE_ONLY___ ═
409
+ 2023-07-20 13:27:11,989 INFO ___FILE_ONLY___ ═
410
+ 2023-07-20 13:27:12,021 INFO ___FILE_ONLY___ ═
411
+ 2023-07-20 13:27:12,046 INFO ___FILE_ONLY___ ═
412
+ 2023-07-20 13:27:12,083 INFO ___FILE_ONLY___ ═
413
+ 2023-07-20 13:27:12,115 INFO ___FILE_ONLY___ ═
414
+ 2023-07-20 13:27:12,146 INFO ___FILE_ONLY___ ═
415
+ 2023-07-20 13:27:12,169 INFO ___FILE_ONLY___ ═
416
+ 2023-07-20 13:27:12,187 INFO ___FILE_ONLY___ ═
417
+ 2023-07-20 13:27:12,209 INFO ___FILE_ONLY___ ═
418
+ 2023-07-20 13:27:12,232 INFO ___FILE_ONLY___ ═
419
+ 2023-07-20 13:27:12,258 INFO ___FILE_ONLY___ ═
420
+ 2023-07-20 13:27:12,286 INFO ___FILE_ONLY___ ═
421
+ 2023-07-20 13:27:12,312 INFO ___FILE_ONLY___ ═
422
+ 2023-07-20 13:27:12,335 INFO ___FILE_ONLY___ ═
423
+ 2023-07-20 13:27:12,383 INFO ___FILE_ONLY___ ═
424
+ 2023-07-20 13:27:12,410 INFO ___FILE_ONLY___ ═
425
+ 2023-07-20 13:27:12,436 INFO ___FILE_ONLY___ ═
426
+ 2023-07-20 13:27:12,455 INFO ___FILE_ONLY___ ═
427
+ 2023-07-20 13:27:12,474 INFO ___FILE_ONLY___ ═
428
+ 2023-07-20 13:27:12,497 INFO ___FILE_ONLY___ ═
429
+ 2023-07-20 13:27:12,518 INFO ___FILE_ONLY___ ═
430
+ 2023-07-20 13:27:12,518 INFO ___FILE_ONLY___ ╝
431
+
432
+ 2023-07-20 13:27:12,572 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
433
+
434
+ 2023-07-20 13:27:12,572 INFO ___FILE_ONLY___ ╠═ Installing: Cloud Storage Command Line Tool (Platform... ═╣
435
+
436
+ 2023-07-20 13:27:12,572 INFO ___FILE_ONLY___ ╚
437
+ 2023-07-20 13:27:12,577 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
438
+ 2023-07-20 13:27:12,674 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-gsutil-nix-20220920185015.tar.gz HTTP/1.1" 200 1851
439
+ 2023-07-20 13:27:12,675 INFO ___FILE_ONLY___ ══════════════════════════════
440
+ 2023-07-20 13:27:12,676 INFO ___FILE_ONLY___ ══════════════════════════════
441
+ 2023-07-20 13:27:12,676 INFO ___FILE_ONLY___ ╝
442
+
443
+ 2023-07-20 13:27:12,685 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
444
+
445
+ 2023-07-20 13:27:12,685 INFO ___FILE_ONLY___ ╠═ Installing: Default set of gcloud commands ═╣
446
+
447
+ 2023-07-20 13:27:12,685 INFO ___FILE_ONLY___ ╚
448
+ 2023-07-20 13:27:12,689 INFO ___FILE_ONLY___ ════════════════════════════════════════════════════════════
449
+ 2023-07-20 13:27:12,689 INFO ___FILE_ONLY___ ╝
450
+
451
+ 2023-07-20 13:27:12,691 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
452
+
453
+ 2023-07-20 13:27:12,692 INFO ___FILE_ONLY___ ╠═ Installing: Google Cloud CLI Core Libraries (Platform... ═╣
454
+
455
+ 2023-07-20 13:27:12,692 INFO ___FILE_ONLY___ ╚
456
+ 2023-07-20 13:27:12,696 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
457
+ 2023-07-20 13:27:12,795 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-core-nix-20220920185015.tar.gz HTTP/1.1" 200 2221
458
+ 2023-07-20 13:27:12,796 INFO ___FILE_ONLY___ ══════════════════════════════
459
+ 2023-07-20 13:27:12,797 INFO ___FILE_ONLY___ ═══════════════
460
+ 2023-07-20 13:27:12,797 INFO ___FILE_ONLY___ ═══════════════
461
+ 2023-07-20 13:27:12,798 INFO ___FILE_ONLY___ ╝
462
+
463
+ 2023-07-20 13:27:12,853 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
464
+
465
+ 2023-07-20 13:27:12,854 INFO ___FILE_ONLY___ ╠═ Installing: Google Cloud CRC32C Hash Tool ═╣
466
+
467
+ 2023-07-20 13:27:12,854 INFO ___FILE_ONLY___ ╚
468
+ 2023-07-20 13:27:12,858 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
469
+ 2023-07-20 13:27:13,022 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-gcloud-crc32c-linux-x86_64-20230707144938.tar.gz HTTP/1.1" 200 1272169
470
+ 2023-07-20 13:27:13,082 INFO ___FILE_ONLY___ ═
471
+ 2023-07-20 13:27:13,082 INFO ___FILE_ONLY___ ═
472
+ 2023-07-20 13:27:13,082 INFO ___FILE_ONLY___ ═
473
+ 2023-07-20 13:27:13,082 INFO ___FILE_ONLY___ ═
474
+ 2023-07-20 13:27:13,082 INFO ___FILE_ONLY___ ═
475
+ 2023-07-20 13:27:13,083 INFO ___FILE_ONLY___ ═
476
+ 2023-07-20 13:27:13,083 INFO ___FILE_ONLY___ ═
477
+ 2023-07-20 13:27:13,083 INFO ___FILE_ONLY___ ═
478
+ 2023-07-20 13:27:13,083 INFO ___FILE_ONLY___ ═
479
+ 2023-07-20 13:27:13,083 INFO ___FILE_ONLY___ ═
480
+ 2023-07-20 13:27:13,083 INFO ___FILE_ONLY___ ═
481
+ 2023-07-20 13:27:13,083 INFO ___FILE_ONLY___ ═
482
+ 2023-07-20 13:27:13,083 INFO ___FILE_ONLY___ ═
483
+ 2023-07-20 13:27:13,084 INFO ___FILE_ONLY___ ═
484
+ 2023-07-20 13:27:13,084 INFO ___FILE_ONLY___ ═
485
+ 2023-07-20 13:27:13,084 INFO ___FILE_ONLY___ ═
486
+ 2023-07-20 13:27:13,084 INFO ___FILE_ONLY___ ═
487
+ 2023-07-20 13:27:13,084 INFO ___FILE_ONLY___ ═
488
+ 2023-07-20 13:27:13,084 INFO ___FILE_ONLY___ ═
489
+ 2023-07-20 13:27:13,084 INFO ___FILE_ONLY___ ═
490
+ 2023-07-20 13:27:13,084 INFO ___FILE_ONLY___ ═
491
+ 2023-07-20 13:27:13,085 INFO ___FILE_ONLY___ ═
492
+ 2023-07-20 13:27:13,085 INFO ___FILE_ONLY___ ═
493
+ 2023-07-20 13:27:13,085 INFO ___FILE_ONLY___ ═
494
+ 2023-07-20 13:27:13,085 INFO ___FILE_ONLY___ ═
495
+ 2023-07-20 13:27:13,085 INFO ___FILE_ONLY___ ═
496
+ 2023-07-20 13:27:13,085 INFO ___FILE_ONLY___ ═
497
+ 2023-07-20 13:27:13,085 INFO ___FILE_ONLY___ ═
498
+ 2023-07-20 13:27:13,085 INFO ___FILE_ONLY___ ═
499
+ 2023-07-20 13:27:13,085 INFO ___FILE_ONLY___ ═
500
+ 2023-07-20 13:27:13,119 INFO ___FILE_ONLY___ ═══════════════
501
+ 2023-07-20 13:27:13,120 INFO ___FILE_ONLY___ ═══════════════
502
+ 2023-07-20 13:27:13,120 INFO ___FILE_ONLY___ ╝
503
+
504
+ 2023-07-20 13:27:13,129 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
505
+
506
+ 2023-07-20 13:27:13,129 INFO ___FILE_ONLY___ ╠═ Installing: Google Cloud CRC32C Hash Tool ═╣
507
+
508
+ 2023-07-20 13:27:13,129 INFO ___FILE_ONLY___ ╚
509
+ 2023-07-20 13:27:13,134 INFO ___FILE_ONLY___ ════════════════════════════════════════════════════════════
510
+ 2023-07-20 13:27:13,134 INFO ___FILE_ONLY___ ╝
511
+
512
+ 2023-07-20 13:27:13,136 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
513
+
514
+ 2023-07-20 13:27:13,136 INFO ___FILE_ONLY___ ╠═ Installing: anthoscli ═╣
515
+
516
+ 2023-07-20 13:27:13,136 INFO ___FILE_ONLY___ ╚
517
+ 2023-07-20 13:27:13,141 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
518
+ 2023-07-20 13:27:13,254 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-anthoscli-linux-x86_64-20230707144938.tar.gz HTTP/1.1" 200 71750428
519
+ 2023-07-20 13:27:14,274 INFO ___FILE_ONLY___ ═
520
+ 2023-07-20 13:27:14,277 INFO ___FILE_ONLY___ ═
521
+ 2023-07-20 13:27:14,280 INFO ___FILE_ONLY___ ═
522
+ 2023-07-20 13:27:14,283 INFO ___FILE_ONLY___ ═
523
+ 2023-07-20 13:27:14,286 INFO ___FILE_ONLY___ ═
524
+ 2023-07-20 13:27:14,289 INFO ___FILE_ONLY___ ═
525
+ 2023-07-20 13:27:14,292 INFO ___FILE_ONLY___ ═
526
+ 2023-07-20 13:27:14,295 INFO ___FILE_ONLY___ ═
527
+ 2023-07-20 13:27:14,298 INFO ___FILE_ONLY___ ═
528
+ 2023-07-20 13:27:14,301 INFO ___FILE_ONLY___ ═
529
+ 2023-07-20 13:27:14,304 INFO ___FILE_ONLY___ ═
530
+ 2023-07-20 13:27:14,307 INFO ___FILE_ONLY___ ═
531
+ 2023-07-20 13:27:14,309 INFO ___FILE_ONLY___ ═
532
+ 2023-07-20 13:27:14,312 INFO ___FILE_ONLY___ ═
533
+ 2023-07-20 13:27:14,315 INFO ___FILE_ONLY___ ═
534
+ 2023-07-20 13:27:14,318 INFO ___FILE_ONLY___ ═
535
+ 2023-07-20 13:27:14,321 INFO ___FILE_ONLY___ ═
536
+ 2023-07-20 13:27:14,323 INFO ___FILE_ONLY___ ═
537
+ 2023-07-20 13:27:14,326 INFO ___FILE_ONLY___ ═
538
+ 2023-07-20 13:27:14,329 INFO ___FILE_ONLY___ ═
539
+ 2023-07-20 13:27:14,331 INFO ___FILE_ONLY___ ═
540
+ 2023-07-20 13:27:14,334 INFO ___FILE_ONLY___ ═
541
+ 2023-07-20 13:27:14,337 INFO ___FILE_ONLY___ ═
542
+ 2023-07-20 13:27:14,340 INFO ___FILE_ONLY___ ═
543
+ 2023-07-20 13:27:14,342 INFO ___FILE_ONLY___ ═
544
+ 2023-07-20 13:27:14,345 INFO ___FILE_ONLY___ ═
545
+ 2023-07-20 13:27:14,348 INFO ___FILE_ONLY___ ═
546
+ 2023-07-20 13:27:14,351 INFO ___FILE_ONLY___ ═
547
+ 2023-07-20 13:27:14,354 INFO ___FILE_ONLY___ ═
548
+ 2023-07-20 13:27:14,357 INFO ___FILE_ONLY___ ═
549
+ 2023-07-20 13:27:16,587 INFO ___FILE_ONLY___ ══════════
550
+ 2023-07-20 13:27:16,592 INFO ___FILE_ONLY___ ═════════
551
+ 2023-07-20 13:27:16,619 INFO ___FILE_ONLY___ ═══════════
552
+ 2023-07-20 13:27:16,620 INFO ___FILE_ONLY___ ╝
553
+
554
+ 2023-07-20 13:27:16,641 INFO ___FILE_ONLY___ ╔═════════════════════════════════��══════════════════════════╗
555
+
556
+ 2023-07-20 13:27:16,642 INFO ___FILE_ONLY___ ╠═ Installing: anthoscli ═╣
557
+
558
+ 2023-07-20 13:27:16,642 INFO ___FILE_ONLY___ ╚
559
+ 2023-07-20 13:27:16,646 INFO ___FILE_ONLY___ ════════════════════════════════════════════════════════════
560
+ 2023-07-20 13:27:16,647 INFO ___FILE_ONLY___ ╝
561
+
562
+ 2023-07-20 13:27:16,649 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
563
+
564
+ 2023-07-20 13:27:16,649 INFO ___FILE_ONLY___ ╠═ Installing: gcloud cli dependencies ═╣
565
+
566
+ 2023-07-20 13:27:16,649 INFO ___FILE_ONLY___ ╚
567
+ 2023-07-20 13:27:16,654 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
568
+ 2023-07-20 13:27:16,756 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-gcloud-deps-linux-x86_64-20210416153011.tar.gz HTTP/1.1" 200 104
569
+ 2023-07-20 13:27:16,757 INFO ___FILE_ONLY___ ══════════════════════════════
570
+ 2023-07-20 13:27:16,757 INFO ___FILE_ONLY___ ══════════════════════════════
571
+ 2023-07-20 13:27:16,757 INFO ___FILE_ONLY___ ╝
572
+
573
+ 2023-07-20 13:27:16,766 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
574
+
575
+ 2023-07-20 13:27:16,767 INFO ___FILE_ONLY___ ╠═ Creating backup and activating new installation ═╣
576
+
577
+ 2023-07-20 13:27:16,767 INFO ___FILE_ONLY___ ╚
578
+ 2023-07-20 13:27:16,767 DEBUG root Attempting to move directory [/tools/google-cloud-sdk] to [/tools/google-cloud-sdk.staging/.install/.backup]
579
+ 2023-07-20 13:27:16,767 INFO ___FILE_ONLY___ ══════════════════════════════
580
+ 2023-07-20 13:27:16,767 DEBUG root Attempting to move directory [/tools/google-cloud-sdk.staging] to [/tools/google-cloud-sdk]
581
+ 2023-07-20 13:27:16,767 INFO ___FILE_ONLY___ ══════════════════════════════
582
+ 2023-07-20 13:27:16,767 INFO ___FILE_ONLY___ ╝
583
+
584
+ 2023-07-20 13:27:16,771 DEBUG root Updating notification cache...
585
+ 2023-07-20 13:27:16,772 INFO ___FILE_ONLY___
586
+
587
+ 2023-07-20 13:27:16,774 INFO ___FILE_ONLY___ Performing post processing steps...
588
+ 2023-07-20 13:27:16,775 DEBUG root Executing command: ['python3', '-S', '/tools/google-cloud-sdk/lib/gcloud.py', 'components', 'post-process']
589
+ 2023-07-20 13:27:42,251 DEBUG ___FILE_ONLY___
590
+ 2023-07-20 13:27:42,252 DEBUG ___FILE_ONLY___
591
+ 2023-07-20 13:27:42,313 INFO ___FILE_ONLY___
592
+ Update done!
593
+
594
+
595
+ 2023-07-20 13:27:42,317 DEBUG root Chosen display Format:none
596
+ 2023-07-20 13:27:42,317 INFO root Display format: "none"
.config/logs/2023.07.20/13.27.17.369260.log ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ 2023-07-20 13:27:17,370 DEBUG root Loaded Command Group: ['gcloud', 'components']
2
+ 2023-07-20 13:27:17,372 DEBUG root Loaded Command Group: ['gcloud', 'components', 'post_process']
3
+ 2023-07-20 13:27:17,375 DEBUG root Running [gcloud.components.post-process] with arguments: []
4
+ 2023-07-20 13:27:42,071 DEBUG root Chosen display Format:none
5
+ 2023-07-20 13:27:42,072 INFO root Display format: "none"
.config/logs/2023.07.20/13.27.43.121533.log ADDED
@@ -0,0 +1,169 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2023-07-20 13:27:43,122 DEBUG root Loaded Command Group: ['gcloud', 'components']
2
+ 2023-07-20 13:27:43,125 DEBUG root Loaded Command Group: ['gcloud', 'components', 'update']
3
+ 2023-07-20 13:27:43,128 DEBUG root Running [gcloud.components.update] with arguments: [--quiet: "True", COMPONENT-IDS:8: "['gcloud', 'core', 'bq', 'gsutil', 'compute', 'preview', 'alpha', 'beta']"]
4
+ 2023-07-20 13:27:43,129 INFO ___FILE_ONLY___ Beginning update. This process may take several minutes.
5
+
6
+ 2023-07-20 13:27:43,135 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
7
+ 2023-07-20 13:27:43,233 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components-2.json HTTP/1.1" 200 208282
8
+ 2023-07-20 13:27:43,252 WARNING root Component [preview] no longer exists.
9
+ 2023-07-20 13:27:43,252 WARNING root Component [compute] no longer exists.
10
+ 2023-07-20 13:27:43,253 INFO ___FILE_ONLY___
11
+
12
+ 2023-07-20 13:27:43,254 INFO ___FILE_ONLY___
13
+ Your current Google Cloud CLI version is: 439.0.0
14
+
15
+ 2023-07-20 13:27:43,254 INFO ___FILE_ONLY___ Installing components from version: 439.0.0
16
+
17
+ 2023-07-20 13:27:43,254 INFO ___FILE_ONLY___
18
+
19
+ 2023-07-20 13:27:43,254 DEBUG root Chosen display Format:table[box,title="These components will be removed."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
20
+ 2023-07-20 13:27:43,255 DEBUG root Chosen display Format:table[box,title="These components will be updated."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
21
+ 2023-07-20 13:27:43,255 DEBUG root Chosen display Format:table[box,title="These components will be installed."](details.display_name:label=Name:align=left,version.version_string:label=Version:align=right,data.size.size(zero="",min=1048576):label=Size:align=right)
22
+ 2023-07-20 13:27:43,257 INFO ___FILE_ONLY___ ┌──────────────────────────────────────────────┐
23
+ 2023-07-20 13:27:43,257 INFO ___FILE_ONLY___
24
+
25
+ 2023-07-20 13:27:43,257 INFO ___FILE_ONLY___ │ These components will be installed. │
26
+ 2023-07-20 13:27:43,257 INFO ___FILE_ONLY___
27
+
28
+ 2023-07-20 13:27:43,257 INFO ___FILE_ONLY___ ├───────────────────────┬────────────┬─────────┤
29
+ 2023-07-20 13:27:43,257 INFO ___FILE_ONLY___
30
+
31
+ 2023-07-20 13:27:43,257 INFO ___FILE_ONLY___ │ Name │ Version │ Size │
32
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___
33
+
34
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___ ├───────────────────────┼────────────┼─────────┤
35
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___
36
+
37
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___ │
38
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___ gcloud Alpha Commands
39
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___
40
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___ │
41
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___ 2023.07.14
42
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___
43
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___ │
44
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___ < 1 MiB
45
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___
46
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___ │
47
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___
48
+
49
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___ │
50
+ 2023-07-20 13:27:43,258 INFO ___FILE_ONLY___ gcloud Beta Commands
51
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___
52
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___ │
53
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___ 2023.07.14
54
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___
55
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___ │
56
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___ < 1 MiB
57
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___
58
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___ │
59
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___
60
+
61
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___ └───────────────────────┴────────────┴─────────┘
62
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___
63
+
64
+ 2023-07-20 13:27:43,259 INFO ___FILE_ONLY___
65
+
66
+ 2023-07-20 13:27:43,264 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
67
+ 2023-07-20 13:27:43,365 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/RELEASE_NOTES HTTP/1.1" 200 1040023
68
+ 2023-07-20 13:27:43,443 INFO ___FILE_ONLY___ For the latest full release notes, please visit:
69
+ https://cloud.google.com/sdk/release_notes
70
+
71
+
72
+ 2023-07-20 13:27:43,446 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
73
+
74
+ 2023-07-20 13:27:43,446 INFO ___FILE_ONLY___ ╠═ Creating update staging area ═╣
75
+
76
+ 2023-07-20 13:27:43,446 INFO ___FILE_ONLY___ ╚
77
+ 2023-07-20 13:27:43,446 INFO ___FILE_ONLY___ ══════
78
+ 2023-07-20 13:27:44,031 INFO ___FILE_ONLY___ ══════
79
+ 2023-07-20 13:27:44,031 INFO ___FILE_ONLY___ ══════
80
+ 2023-07-20 13:27:44,433 INFO ___FILE_ONLY___ ═
81
+ 2023-07-20 13:27:44,633 INFO ___FILE_ONLY___ ═
82
+ 2023-07-20 13:27:44,749 INFO ___FILE_ONLY___ ═
83
+ 2023-07-20 13:27:44,845 INFO ___FILE_ONLY___ ═
84
+ 2023-07-20 13:27:45,073 INFO ___FILE_ONLY___ ═
85
+ 2023-07-20 13:27:45,164 INFO ___FILE_ONLY___ ═
86
+ 2023-07-20 13:27:45,230 INFO ___FILE_ONLY___ ═
87
+ 2023-07-20 13:27:45,332 INFO ___FILE_ONLY___ ═
88
+ 2023-07-20 13:27:45,399 INFO ___FILE_ONLY___ ═
89
+ 2023-07-20 13:27:45,473 INFO ___FILE_ONLY___ ═
90
+ 2023-07-20 13:27:45,576 INFO ___FILE_ONLY___ ═
91
+ 2023-07-20 13:27:45,636 INFO ___FILE_ONLY___ ═
92
+ 2023-07-20 13:27:45,727 INFO ___FILE_ONLY___ ═
93
+ 2023-07-20 13:27:45,900 INFO ___FILE_ONLY___ ═
94
+ 2023-07-20 13:27:45,999 INFO ___FILE_ONLY___ ═
95
+ 2023-07-20 13:27:46,032 INFO ___FILE_ONLY___ ═
96
+ 2023-07-20 13:27:46,071 INFO ___FILE_ONLY___ ═
97
+ 2023-07-20 13:27:46,105 INFO ___FILE_ONLY___ ═
98
+ 2023-07-20 13:27:46,146 INFO ___FILE_ONLY___ ═
99
+ 2023-07-20 13:27:46,227 INFO ___FILE_ONLY___ ═
100
+ 2023-07-20 13:27:46,276 INFO ___FILE_ONLY___ ═
101
+ 2023-07-20 13:27:46,366 INFO ___FILE_ONLY___ ═
102
+ 2023-07-20 13:27:46,606 INFO ___FILE_ONLY___ ═
103
+ 2023-07-20 13:27:46,736 INFO ___FILE_ONLY___ ═
104
+ 2023-07-20 13:27:46,934 INFO ___FILE_ONLY___ ═
105
+ 2023-07-20 13:27:47,011 INFO ___FILE_ONLY___ ═
106
+ 2023-07-20 13:27:47,069 INFO ___FILE_ONLY___ ═
107
+ 2023-07-20 13:27:47,132 INFO ___FILE_ONLY___ ═
108
+ 2023-07-20 13:27:47,189 INFO ___FILE_ONLY___ ═
109
+ 2023-07-20 13:27:47,234 INFO ___FILE_ONLY___ ═
110
+ 2023-07-20 13:27:47,283 INFO ___FILE_ONLY___ ═
111
+ 2023-07-20 13:27:47,332 INFO ___FILE_ONLY___ ═
112
+ 2023-07-20 13:27:47,388 INFO ___FILE_ONLY___ ═
113
+ 2023-07-20 13:27:47,461 INFO ___FILE_ONLY___ ═
114
+ 2023-07-20 13:27:47,523 INFO ___FILE_ONLY___ ═
115
+ 2023-07-20 13:27:47,574 INFO ___FILE_ONLY___ ═
116
+ 2023-07-20 13:27:47,629 INFO ___FILE_ONLY___ ═
117
+ 2023-07-20 13:27:47,794 INFO ___FILE_ONLY___ ═
118
+ 2023-07-20 13:27:47,847 INFO ___FILE_ONLY___ ═
119
+ 2023-07-20 13:27:47,901 INFO ___FILE_ONLY___ ═
120
+ 2023-07-20 13:27:47,980 INFO ___FILE_ONLY___ ═
121
+ 2023-07-20 13:27:48,034 INFO ___FILE_ONLY___ ═
122
+ 2023-07-20 13:27:48,034 INFO ___FILE_ONLY___ ╝
123
+
124
+ 2023-07-20 13:27:49,851 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
125
+
126
+ 2023-07-20 13:27:49,852 INFO ___FILE_ONLY___ ╠═ Installing: gcloud Alpha Commands ═╣
127
+
128
+ 2023-07-20 13:27:49,852 INFO ___FILE_ONLY___ ╚
129
+ 2023-07-20 13:27:49,856 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
130
+ 2023-07-20 13:27:50,010 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-alpha-20230714124024.tar.gz HTTP/1.1" 200 800
131
+ 2023-07-20 13:27:50,011 INFO ___FILE_ONLY___ ══════════════════════════════
132
+ 2023-07-20 13:27:50,013 INFO ___FILE_ONLY___ ══════════════════════════════
133
+ 2023-07-20 13:27:50,013 INFO ___FILE_ONLY___ ╝
134
+
135
+ 2023-07-20 13:27:50,021 INFO ___FILE_ONLY___ ╔════════════════════════════════════════════════════════════╗
136
+
137
+ 2023-07-20 13:27:50,021 INFO ___FILE_ONLY___ ╠═ Installing: gcloud Beta Commands ═╣
138
+
139
+ 2023-07-20 13:27:50,021 INFO ___FILE_ONLY___ ╚
140
+ 2023-07-20 13:27:50,025 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): dl.google.com:443
141
+ 2023-07-20 13:27:50,121 DEBUG urllib3.connectionpool https://dl.google.com:443 "GET /dl/cloudsdk/channels/rapid/components/google-cloud-sdk-beta-20230714124024.tar.gz HTTP/1.1" 200 797
142
+ 2023-07-20 13:27:50,122 INFO ___FILE_ONLY___ ══════════════════════════════
143
+ 2023-07-20 13:27:50,123 INFO ___FILE_ONLY___ ══════════════════════════════
144
+ 2023-07-20 13:27:50,123 INFO ___FILE_ONLY___ ╝
145
+
146
+ 2023-07-20 13:27:50,133 INFO ___FILE_ONLY___ ��════════════════════════════════════════════════════════════╗
147
+
148
+ 2023-07-20 13:27:50,133 INFO ___FILE_ONLY___ ╠═ Creating backup and activating new installation ═╣
149
+
150
+ 2023-07-20 13:27:50,133 INFO ___FILE_ONLY___ ╚
151
+ 2023-07-20 13:27:50,133 DEBUG root Attempting to move directory [/tools/google-cloud-sdk] to [/tools/google-cloud-sdk.staging/.install/.backup]
152
+ 2023-07-20 13:27:50,133 INFO ___FILE_ONLY___ ══════════════════════════════
153
+ 2023-07-20 13:27:50,134 DEBUG root Attempting to move directory [/tools/google-cloud-sdk.staging] to [/tools/google-cloud-sdk]
154
+ 2023-07-20 13:27:50,134 INFO ___FILE_ONLY___ ══════════════════════════════
155
+ 2023-07-20 13:27:50,134 INFO ___FILE_ONLY___ ╝
156
+
157
+ 2023-07-20 13:27:50,138 DEBUG root Updating notification cache...
158
+ 2023-07-20 13:27:50,138 INFO ___FILE_ONLY___
159
+
160
+ 2023-07-20 13:27:50,140 INFO ___FILE_ONLY___ Performing post processing steps...
161
+ 2023-07-20 13:27:50,141 DEBUG root Executing command: ['python3', '-S', '/tools/google-cloud-sdk/lib/gcloud.py', 'components', 'post-process']
162
+ 2023-07-20 13:28:15,915 DEBUG ___FILE_ONLY___
163
+ 2023-07-20 13:28:15,916 DEBUG ___FILE_ONLY___
164
+ 2023-07-20 13:28:15,929 INFO ___FILE_ONLY___
165
+ Update done!
166
+
167
+
168
+ 2023-07-20 13:28:15,932 DEBUG root Chosen display Format:none
169
+ 2023-07-20 13:28:15,933 INFO root Display format: "none"
.config/logs/2023.07.20/13.27.50.747950.log ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ 2023-07-20 13:27:50,749 DEBUG root Loaded Command Group: ['gcloud', 'components']
2
+ 2023-07-20 13:27:50,751 DEBUG root Loaded Command Group: ['gcloud', 'components', 'post_process']
3
+ 2023-07-20 13:27:50,753 DEBUG root Running [gcloud.components.post-process] with arguments: []
4
+ 2023-07-20 13:28:15,731 DEBUG root Chosen display Format:none
5
+ 2023-07-20 13:28:15,732 INFO root Display format: "none"
.config/logs/2023.07.20/13.28.16.714039.log ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ 2023-07-20 13:28:16,716 DEBUG root Loaded Command Group: ['gcloud', 'config']
2
+ 2023-07-20 13:28:16,746 DEBUG root Loaded Command Group: ['gcloud', 'config', 'set']
3
+ 2023-07-20 13:28:16,749 DEBUG root Running [gcloud.config.set] with arguments: [SECTION/PROPERTY: "component_manager/disable_update_check", VALUE: "true"]
4
+ 2023-07-20 13:28:16,750 INFO ___FILE_ONLY___ Updated property [component_manager/disable_update_check].
5
+
6
+ 2023-07-20 13:28:16,751 DEBUG root Chosen display Format:default
7
+ 2023-07-20 13:28:16,752 INFO root Display format: "default"
8
+ 2023-07-20 13:28:16,752 DEBUG root SDK update checks are disabled.
.config/logs/2023.07.20/13.28.17.509819.log ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ 2023-07-20 13:28:17,511 DEBUG root Loaded Command Group: ['gcloud', 'config']
2
+ 2023-07-20 13:28:17,542 DEBUG root Loaded Command Group: ['gcloud', 'config', 'set']
3
+ 2023-07-20 13:28:17,545 DEBUG root Running [gcloud.config.set] with arguments: [SECTION/PROPERTY: "compute/gce_metadata_read_timeout_sec", VALUE: "0"]
4
+ 2023-07-20 13:28:17,546 INFO ___FILE_ONLY___ Updated property [compute/gce_metadata_read_timeout_sec].
5
+
6
+ 2023-07-20 13:28:17,547 DEBUG root Chosen display Format:default
7
+ 2023-07-20 13:28:17,548 INFO root Display format: "default"
8
+ 2023-07-20 13:28:17,548 DEBUG root SDK update checks are disabled.
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ sample_data/mnist_test.csv filter=lfs diff=lfs merge=lfs -text
37
+ sample_data/mnist_train_small.csv filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2023 Andrej
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md CHANGED
@@ -1,12 +1,158 @@
1
  ---
2
- title: Play With Baby Llama2
3
- emoji: 😻
4
- colorFrom: yellow
5
- colorTo: indigo
6
  sdk: gradio
7
  sdk_version: 3.38.0
8
- app_file: app.py
9
- pinned: false
10
  ---
11
 
12
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: play_with_baby_llama2
3
+ app_file: baby_llama2.py
 
 
4
  sdk: gradio
5
  sdk_version: 3.38.0
 
 
6
  ---
7
 
8
+ ## llama2.c
9
+
10
+ Have you ever wanted to inference a baby [Llama 2](https://ai.meta.com/llama/) model in pure C? No? Well, now you can!
11
+
12
+ <img src="assets/llama_cute.jpg" width="300" height="300">
13
+
14
+ With this code you can train the Llama 2 LLM architecture from scratch in PyTorch, then save the weights to a raw binary file, then load that into one ~simple 500-line C file ([run.c](run.c)) that inferences the model, simply in fp32 for now. On my cloud Linux devbox a dim 288 6-layer 6-head model (~15M params) inferences at ~100 tok/s in fp32, and about the same on my M1 MacBook Air. I was somewhat pleasantly surprised that one can run reasonably sized models (few ten million params) at highly interactive rates with an approach this simple.
15
+
16
+ Please note that this is just a weekend project: I took nanoGPT, tuned it to implement the Llama-2 architecture instead of GPT-2, and the meat of it was writing the C inference engine in [run.c](run.c). As such, this is not really meant to be a production-grade library right now.
17
+
18
+ Hat tip to [llama.cpp](https://github.com/ggerganov/llama.cpp) for inspiring this project. I wanted something super minimal so I chose to hard-code the llama-2 architecture, stick to fp32, and just roll one inference file of pure C with no dependencies.
19
+
20
+ ## feel the magic
21
+
22
+ Let's just run a baby Llama 2 model in C. You need a model checkpoint. Download this 15M parameter model I trained on the [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset (~58MB download) and place it into the default checkpoint directory `out`:
23
+
24
+ ```bash
25
+ wget https://karpathy.ai/llama2c/model.bin -P out
26
+ ```
27
+
28
+ (if that doesn't work try [google drive](https://drive.google.com/file/d/1aTimLdx3JktDXxcHySNrZJOOk8Vb1qBR/view?usp=share_link)). Compile and run the C code:
29
+
30
+ ```bash
31
+ gcc -O3 -o run run.c -lm
32
+ ./run out/model.bin
33
+ ```
34
+
35
+ You'll see the text stream a sample. On my M1 MacBook Air this runs at ~100 tokens/s, not bad for super naive fp32 single-threaded C code. See [performance](#performance) for compile flags that can significantly speed this up. Sample output:
36
+
37
+ *Once upon a time, there was a boy named Timmy. Timmy loved to play sports with his friends. He was very good at throwing and catching balls. One day, Timmy's mom gave him a new shirt to wear to a party. Timmy thought it was impressive and asked his mom to explain what a shirt could be for. "A shirt is like a special suit for a basketball game," his mom said. Timmy was happy to hear that and put on his new shirt. He felt like a soldier going to the army and shouting. From that day on, Timmy wore his new shirt every time he played sports with his friends at the party. Once upon a time, there was a little girl named Lily. She loved to play outside with her friends. One day, Lily and her friend Emma were playing with a ball. Emma threw the ball too hard and it hit Lily's face. Lily felt embarrassed and didn't want to play anymore.
38
+ Emma asked Lily what was wrong, and Lily told her about her memory. Emma told Lily that she was embarrassed because she had thrown the ball too hard. Lily felt bad
39
+ achieved tok/s: 98.746993347843922*
40
+
41
+ **Update**: I've now also uploaded a bigger checkpoint. This one is dim 512, 8 layers, 8 heads and context length 1024, a ~44M param Transformer. It trained for 200K iterations batch size 32 on 4XA100 40GB GPUs in ~8 hours. You can use this bigger and more powerful checkpoint like so:
42
+
43
+ ```bash
44
+ wget https://karpathy.ai/llama2c/model44m.bin -P out44m
45
+ ./run out44m/model44m.bin
46
+ ```
47
+
48
+ On my MacBook Air compiled with $ gcc -Ofast -o run run.c -lm this ran at ~150 tok/s. Still way too fast! I have to train an even bigger checkpoint... This model samples more coherent and diverse stories:
49
+
50
+ *Once upon a time, there was a little girl named Lily. She loved playing with her toys on top of her bed. One day, she decided to have a tea party with her stuffed animals. She poured some tea into a tiny teapot and put it on top of the teapot. Suddenly, her little brother Max came into the room and wanted to join the tea party too. Lily didn't want to share her tea and she told Max to go away. Max started to cry and Lily felt bad. She decided to yield her tea party to Max and they both shared the teapot. But then, something unexpected happened. The teapot started to shake and wiggle. Lily and Max were scared and didn't know what to do. Suddenly, the teapot started to fly towards the ceiling and landed on the top of the bed. Lily and Max were amazed and they hugged each other. They realized that sharing was much more fun than being selfish. From that day on, they always shared their tea parties and toys.*
51
+
52
+ ## howto
53
+
54
+ It should be possible to load the weights released by Meta but I haven't tried because the inference speed, even of the 7B model, would probably be not great with this baby single-threaded C program. So in this repo we focus on more narrow applications, and train the same architecture but from scratch, in this case on the TinyStories dataset for fun.
55
+
56
+ First let's download and pretokenize some source dataset, e.g. I like [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) so this is the only example currently available in this repo. But it should be very easy to add datasets, see the code.
57
+
58
+ ```bash
59
+ python tinystories.py download
60
+ python tinystories.py pretokenize
61
+ ```
62
+
63
+ Then train our model:
64
+
65
+ ```bash
66
+ python train.py
67
+ ```
68
+
69
+ See the train.py script for more exotic launches and hyperparameter overrides. I didn't tune the hyperparameters, I expect simple hyperparameter exploration should give better models. Totally understand if you want to skip model training, for simple demo just download my pretrained model and save it into the directory `out`:
70
+
71
+ ```bash
72
+ wget https://karpathy.ai/llama2c/model.bin -P out
73
+ ```
74
+
75
+ Once we have the model.bin file, we can inference in C. Compile the C code first:
76
+
77
+ ```bash
78
+ gcc -O3 -o run run.c -lm
79
+ ```
80
+
81
+ You can now run it simply as
82
+
83
+ ```bash
84
+ ./run out/model.bin
85
+ ```
86
+
87
+ Watch the tokens stream by, fun! We can also run the PyTorch inference script for comparison (to run, add [model.ckpt](https://drive.google.com/file/d/1SM0rMxzy7babB-v4MfTg1GFqOCgWar5w/view?usp=share_link) to /out if you haven't already):
88
+
89
+ ```bash
90
+ python sample.py
91
+ ```
92
+
93
+ Which gives the same results. More detailed testing will be done in `test_all.py`, run as:
94
+
95
+ ```bash
96
+ $ pytest
97
+ ```
98
+
99
+ Currently you will need two files to test or sample: the [model.bin](https://drive.google.com/file/d/1aTimLdx3JktDXxcHySNrZJOOk8Vb1qBR/view?usp=share_link) file and the [model.ckpt](https://drive.google.com/file/d/1SM0rMxzy7babB-v4MfTg1GFqOCgWar5w/view?usp=share_link) file from PyTorch training I ran earlier. I have to think through running the tests without having to download 200MB of data.
100
+
101
+ ## performance
102
+
103
+ *(NOTE: this guide is not great because I personally spend a lot of my time in Python land and don't have an amazing understanding of a lot of these features and flags. If someone does and is willing to help document and briefly describe some of these and their tradeoffs, I'd welcome a PR)*
104
+
105
+ There are many ways to potentially speed up this code depending on your system. Here we document a few together with a high-level guide on what they do. Here's again the default way to compile, but using -O3:
106
+
107
+ ```bash
108
+ gcc -O3 -o run run.c -lm
109
+ ```
110
+
111
+ -O3 includes optimizations that are expensive in terms of compile time and memory usage. Including vectorization, loop unrolling, and predicting branches. Here's a few more to try.
112
+
113
+ `-Ofast` Run additional optimizations which may break compliance with the C/IEEE specifications, in addition to `-O3`. See [the GCC docs](https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html) for more information.
114
+
115
+ `-ffast-math` breaks IEEE compliance, e.g. allowing reordering of operations, disables a bunch of checks for e.g. NaNs (assuming they don't happen), enables reciprocal approximations, disables signed zero, etc. However, there is a good reason to be suspicious of this setting, one good writeup is here: ["Beware of fast-math"](https://simonbyrne.github.io/notes/fastmath/).
116
+
117
+ `-funsafe-math-optimizations` a more limited form of -ffast-math, that still breaks IEEE compliance but doesn't have all of the numeric/error handling changes from `-ffasth-math`. See [the GCC docs](https://gcc.gnu.org/wiki/FloatingPointMath) for more information.
118
+
119
+ `-march=native` Compile the program to use the architecture of the machine you're compiling on rather than a more generic CPU. This may enable additional optimizations and hardware-specific tuning such as improved vector instructions/width.
120
+
121
+ Putting a few of these together, the fastest throughput I saw so far on my MacBook Air (M1) is with:
122
+
123
+ ```bash
124
+ gcc -Ofast -o run run.c -lm
125
+ ```
126
+
127
+ Also, I saw someone report higher throughput replacing `gcc` with `clang`.
128
+
129
+ **OpenMP** Big improvements can also be achieved by compiling with OpenMP, which "activates" the `#pragma omp parallel for` inside the matmul. You can compile e.g. like so:
130
+
131
+ ```bash
132
+ clang -Ofast -fopenmp -march=native run.c -lm -o run
133
+ ```
134
+
135
+ (I believe you can swap clang/gcc, and may try to leave out -march=native). Then when you run inference, make sure to use OpenMP flags to set the number of threads, e.g.:
136
+
137
+ ```bash
138
+ OMP_NUM_THREADS=4 ./run out/model.bin
139
+ ```
140
+
141
+ Depending on your system resources you may want to tweak these hyperparameters. (TODO: I am not intimitely familiar with OpenMP and its configuration, if someone would like to flesh out this section I would welcome a PR).
142
+
143
+ ## unsorted todos
144
+
145
+ - why is there a leading space in C sampling code when we `./run`?
146
+ - todo multiquery support? doesn't seem as useful for smaller models that run on CPU (?)
147
+ - todo support inferencing beyond max_seq_len steps, have to think through the kv cache
148
+ - why is MFU so low (~10%) on my A100 40GB for training?
149
+ - weird errors with torch.compile and wandb when using DDP
150
+ - make more better tests to decrease yolo
151
+
152
+ ## ack
153
+
154
+ I trained the llama2.c storyteller models on a 4X A100 40GB box graciously provided by the excellent [Lambda labs](https://lambdalabs.com/service/gpu-cloud), thank you.
155
+
156
+ ## License
157
+
158
+ MIT
baby_llama2.py ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import subprocess
2
+ import gradio as gr
3
+ from translate import Translator
4
+
5
+ def generate_text():
6
+ cmd = ['./run', 'model.bin']
7
+ result = subprocess.run(cmd, stdout=subprocess.PIPE, text=True)
8
+ translator= Translator(from_lang='en', to_lang='zh-cn')
9
+
10
+ # Split the output into sentences
11
+ sentences = result.stdout.split('. ')
12
+
13
+ # Translate each sentence and join them back together
14
+ translation = '. '.join(translator.translate(sentence) for sentence in sentences)
15
+
16
+ return translation
17
+
18
+ iface = gr.Interface(
19
+ fn=generate_text,
20
+ inputs=[],
21
+ outputs="text",
22
+ submit_label="开始生成",
23
+ title="和小羊驼一起玩"
24
+ )
25
+ iface.launch()
configurator.py ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Poor Man's Configurator. Probably a terrible idea. Example usage:
3
+ $ python train.py config/override_file.py --batch_size=32
4
+ this will first run config/override_file.py, then override batch_size to 32
5
+
6
+ The code in this file will be run as follows from e.g. train.py:
7
+ >>> exec(open('configurator.py').read())
8
+
9
+ So it's not a Python module, it's just shuttling this code away from train.py
10
+ The code in this script then overrides the globals()
11
+
12
+ I know people are not going to love this, I just really dislike configuration
13
+ complexity and having to prepend config. to every single variable. If someone
14
+ comes up with a better simple Python solution I am all ears.
15
+ """
16
+
17
+ import sys
18
+ from ast import literal_eval
19
+
20
+ for arg in sys.argv[1:]:
21
+ if '=' not in arg:
22
+ # assume it's the name of a config file
23
+ assert not arg.startswith('--')
24
+ config_file = arg
25
+ print(f"Overriding config with {config_file}:")
26
+ with open(config_file) as f:
27
+ print(f.read())
28
+ exec(open(config_file).read())
29
+ else:
30
+ # assume it's a --key=value argument
31
+ assert arg.startswith('--')
32
+ key, val = arg.split('=')
33
+ key = key[2:]
34
+ if key in globals():
35
+ try:
36
+ # attempt to eval it it (e.g. if bool, number, or etc)
37
+ attempt = literal_eval(val)
38
+ except (SyntaxError, ValueError):
39
+ # if that goes wrong, just use the string
40
+ attempt = val
41
+ # ensure the types match ok
42
+ assert type(attempt) == type(globals()[key])
43
+ # cross fingers
44
+ print(f"Overriding: {key} = {attempt}")
45
+ globals()[key] = attempt
46
+ else:
47
+ raise ValueError(f"Unknown config key: {key}")
import subprocess.py ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import subprocess
2
+
3
+ # Define the command as a list
4
+ import gradio as gr
5
+
6
+ def generate_text():
7
+
8
+
9
+ cmd = ['./run', 'model.bin']
10
+
11
+ # Use subprocess.run to execute the command
12
+ result = subprocess.run(cmd, stdout=subprocess.PIPE, text=True)
13
+ # 在这里运行你的模型，生成文本
14
+ # 以下是一个示例
15
+ text = "这是生成的文本。"
16
+ title="运行来让小羊驼跑起来"
17
+ return result
18
+
19
+ iface = gr.Interface(fn=generate_text, inputs=[], outputs="text",submit_label="让小羊驼跑起来",title="和小羊驼一起玩")
20
+ iface.launch()
21
+
22
+
23
+ cmd = ['./run', 'model.bin']
24
+
25
+ # Use subprocess.run to execute the command
26
+ result = subprocess.run(cmd, stdout=subprocess.PIPE, text=True)
27
+
28
+ # Print the output
29
+ print(result.stdout)
model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cd590644d963867a2b6e5a1107f51fad663c41d79c149fbecbbb1f95fa81f49a
3
+ size 60816028
model.py ADDED
@@ -0,0 +1,370 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright (c) Meta Platforms, Inc. and affiliates.
2
+ # This software may be used and distributed according to the terms of the Llama 2 Community License Agreement.
3
+
4
+ import math
5
+ import struct
6
+ import inspect
7
+ from dataclasses import dataclass
8
+ from typing import Any, Optional, Tuple
9
+
10
+ import numpy as np
11
+ import torch
12
+ import torch.nn.functional as F
13
+ from torch import nn
14
+
15
+ @dataclass
16
+ class ModelArgs:
17
+ dim: int = 4096
18
+ n_layers: int = 32
19
+ n_heads: int = 32
20
+ n_kv_heads: Optional[int] = None
21
+ vocab_size: int = -1 # defined later by tokenizer
22
+ multiple_of: int = 256 # make SwiGLU hidden layer size multiple of large power of 2
23
+ norm_eps: float = 1e-5
24
+ max_seq_len: int = 2048
25
+ dropout: float = 0.0
26
+
27
+
28
+ class RMSNorm(torch.nn.Module):
29
+ def __init__(self, dim: int, eps: float):
30
+ super().__init__()
31
+ self.eps = eps
32
+ self.weight = nn.Parameter(torch.ones(dim))
33
+
34
+ def _norm(self, x):
35
+ return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
36
+
37
+ def forward(self, x):
38
+ output = self._norm(x.float()).type_as(x)
39
+ return output * self.weight
40
+
41
+
42
+ def precompute_freqs_cis(dim: int, end: int, theta: float = 10000.0):
43
+ freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))
44
+ t = torch.arange(end, device=freqs.device) # type: ignore
45
+ freqs = torch.outer(t, freqs).float() # type: ignore
46
+ freqs_cis = torch.polar(torch.ones_like(freqs), freqs) # complex64
47
+ return freqs_cis
48
+
49
+
50
+ def reshape_for_broadcast(freqs_cis: torch.Tensor, x: torch.Tensor):
51
+ ndim = x.ndim
52
+ assert 0 <= 1 < ndim
53
+ assert freqs_cis.shape == (x.shape[1], x.shape[-1])
54
+ shape = [d if i == 1 or i == ndim - 1 else 1 for i, d in enumerate(x.shape)]
55
+ return freqs_cis.view(*shape)
56
+
57
+
58
+ def apply_rotary_emb(
59
+ xq: torch.Tensor,
60
+ xk: torch.Tensor,
61
+ freqs_cis: torch.Tensor,
62
+ ) -> Tuple[torch.Tensor, torch.Tensor]:
63
+ xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2))
64
+ xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2))
65
+ freqs_cis = reshape_for_broadcast(freqs_cis, xq_)
66
+ xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(3)
67
+ xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(3)
68
+ return xq_out.type_as(xq), xk_out.type_as(xk)
69
+
70
+ def repeat_kv(x: torch.Tensor, n_rep: int) -> torch.Tensor:
71
+ """torch.repeat_interleave(x, dim=2, repeats=n_rep)"""
72
+ bs, slen, n_kv_heads, head_dim = x.shape
73
+ if n_rep == 1:
74
+ return x
75
+ return (
76
+ x[:, :, :, None, :]
77
+ .expand(bs, slen, n_kv_heads, n_rep, head_dim)
78
+ .reshape(bs, slen, n_kv_heads * n_rep, head_dim)
79
+ )
80
+
81
+ class Attention(nn.Module):
82
+ def __init__(self, args: ModelArgs):
83
+ super().__init__()
84
+ self.n_kv_heads = args.n_heads if args.n_kv_heads is None else args.n_kv_heads
85
+ model_parallel_size = 1
86
+ self.n_local_heads = args.n_heads // model_parallel_size
87
+ self.n_local_kv_heads = self.n_kv_heads // model_parallel_size
88
+ self.n_rep = self.n_local_heads // self.n_local_kv_heads
89
+ self.head_dim = args.dim // args.n_heads
90
+ self.wq = nn.Linear(args.dim, args.n_heads * self.head_dim, bias=False)
91
+ self.wk = nn.Linear(args.dim, self.n_kv_heads * self.head_dim, bias=False)
92
+ self.wv = nn.Linear(args.dim, self.n_kv_heads * self.head_dim, bias=False)
93
+ self.wo = nn.Linear(args.n_heads * self.head_dim, args.dim, bias=False)
94
+ self.attn_dropout = nn.Dropout(args.dropout)
95
+ self.resid_dropout = nn.Dropout(args.dropout)
96
+ self.dropout = args.dropout
97
+
98
+ # use flash attention or a manual implementation?
99
+ self.flash = hasattr(torch.nn.functional, 'scaled_dot_product_attention')
100
+ if not self.flash:
101
+ print("WARNING: using slow attention. Flash Attention requires PyTorch >= 2.0")
102
+ mask = torch.full((1, 1, args.max_seq_len, args.max_seq_len), float("-inf"))
103
+ mask = torch.triu(mask, diagonal=1)
104
+ self.register_buffer("mask", mask)
105
+
106
+ def forward(
107
+ self,
108
+ x: torch.Tensor,
109
+ freqs_cis: torch.Tensor,
110
+ ):
111
+ bsz, seqlen, _ = x.shape
112
+
113
+ # QKV
114
+ xq, xk, xv = self.wq(x), self.wk(x), self.wv(x)
115
+ xq = xq.view(bsz, seqlen, self.n_local_heads, self.head_dim)
116
+ xk = xk.view(bsz, seqlen, self.n_local_kv_heads, self.head_dim)
117
+ xv = xv.view(bsz, seqlen, self.n_local_kv_heads, self.head_dim)
118
+
119
+ # RoPE relative positional embeddings
120
+ xq, xk = apply_rotary_emb(xq, xk, freqs_cis)
121
+
122
+ # grouped multiquery attention: expand out keys and values
123
+ xk = repeat_kv(xk, self.n_rep) # (bs, seqlen, n_local_heads, head_dim)
124
+ xv = repeat_kv(xv, self.n_rep) # (bs, seqlen, n_local_heads, head_dim)
125
+
126
+ # make heads into a batch dimension
127
+ xq = xq.transpose(1, 2) # (bs, n_local_heads, seqlen, head_dim)
128
+ xk = xk.transpose(1, 2)
129
+ xv = xv.transpose(1, 2)
130
+
131
+ # flash implementation
132
+ if self.flash:
133
+ output = torch.nn.functional.scaled_dot_product_attention(xq, xk, xv, attn_mask=None, dropout_p=self.dropout if self.training else 0, is_causal=True)
134
+ else:
135
+ # manual implementation
136
+ scores = torch.matmul(xq, xk.transpose(2, 3)) / math.sqrt(self.head_dim)
137
+ scores = scores + self.mask[:, :, :seqlen, :seqlen] # (bs, n_local_heads, seqlen, cache_len + seqlen)
138
+ scores = F.softmax(scores.float(), dim=-1).type_as(xq)
139
+ scores = self.attn_dropout(scores)
140
+ output = torch.matmul(scores, xv) # (bs, n_local_heads, seqlen, head_dim)
141
+
142
+ # restore time as batch dimension and concat heads
143
+ output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)
144
+
145
+ # final projection into the residual stream
146
+ output = self.wo(output)
147
+ output = self.resid_dropout(output)
148
+ return output
149
+
150
+
151
+ class FeedForward(nn.Module):
152
+ def __init__(self, dim: int, hidden_dim: int, multiple_of: int, dropout: float):
153
+ super().__init__()
154
+ hidden_dim = int(2 * hidden_dim / 3)
155
+ hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of)
156
+ self.w1 = nn.Linear(dim, hidden_dim, bias=False)
157
+ self.w2 = nn.Linear(hidden_dim, dim, bias=False)
158
+ self.w3 = nn.Linear(dim, hidden_dim, bias=False)
159
+ self.dropout = nn.Dropout(dropout)
160
+
161
+ def forward(self, x):
162
+ return self.dropout(self.w2(F.silu(self.w1(x)) * self.w3(x)))
163
+
164
+
165
+ class TransformerBlock(nn.Module):
166
+ def __init__(self, layer_id: int, args: ModelArgs):
167
+ super().__init__()
168
+ self.n_heads = args.n_heads
169
+ self.dim = args.dim
170
+ self.head_dim = args.dim // args.n_heads
171
+ self.attention = Attention(args)
172
+ self.feed_forward = FeedForward(
173
+ dim=args.dim,
174
+ hidden_dim=4 * args.dim,
175
+ multiple_of=args.multiple_of,
176
+ dropout=args.dropout,
177
+ )
178
+ self.layer_id = layer_id
179
+ self.attention_norm = RMSNorm(args.dim, eps=args.norm_eps)
180
+ self.ffn_norm = RMSNorm(args.dim, eps=args.norm_eps)
181
+
182
+ def forward(self, x, freqs_cis):
183
+ h = x + self.attention.forward(self.attention_norm(x), freqs_cis)
184
+ out = h + self.feed_forward.forward(self.ffn_norm(h))
185
+ return out
186
+
187
+
188
+ class Transformer(nn.Module):
189
+ def __init__(self, params: ModelArgs):
190
+ super().__init__()
191
+ self.params = params
192
+ self.vocab_size = params.vocab_size
193
+ self.n_layers = params.n_layers
194
+
195
+ self.tok_embeddings = nn.Embedding(params.vocab_size, params.dim)
196
+ self.dropout = nn.Dropout(params.dropout)
197
+ self.layers = torch.nn.ModuleList()
198
+ for layer_id in range(params.n_layers):
199
+ self.layers.append(TransformerBlock(layer_id, params))
200
+ self.norm = RMSNorm(params.dim, eps=params.norm_eps)
201
+ self.output = nn.Linear(params.dim, params.vocab_size, bias=False)
202
+
203
+ # share the unembedding parameters with the embedding parameters
204
+ self.tok_embeddings.weight = self.output.weight # https://paperswithcode.com/method/weight-tying
205
+
206
+ # some useful precompute for the RoPE relative positional embeddings. TODO why * 2 here? confuse
207
+ freqs_cis = precompute_freqs_cis(self.params.dim // self.params.n_heads, self.params.max_seq_len * 2)
208
+ self.register_buffer("freqs_cis", freqs_cis, persistent=False)
209
+
210
+ # init all weights
211
+ self.apply(self._init_weights)
212
+ # apply special scaled init to the residual projections, per GPT-2 paper
213
+ for pn, p in self.named_parameters():
214
+ if pn.endswith('w3.weight') or pn.endswith('wo.weight'):
215
+ torch.nn.init.normal_(p, mean=0.0, std=0.02/math.sqrt(2 * params.n_layers))
216
+
217
+ def _init_weights(self, module):
218
+ if isinstance(module, nn.Linear):
219
+ torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
220
+ if module.bias is not None:
221
+ torch.nn.init.zeros_(module.bias)
222
+ elif isinstance(module, nn.Embedding):
223
+ torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
224
+
225
+ def forward(self, tokens, targets=None):
226
+ _bsz, seqlen = tokens.shape
227
+ h = self.tok_embeddings(tokens)
228
+ h = self.dropout(h)
229
+ freqs_cis = self.freqs_cis[:seqlen]
230
+
231
+ for layer in self.layers:
232
+ h = layer(h, freqs_cis)
233
+ h = self.norm(h)
234
+
235
+ if targets is not None:
236
+ # if we are given some desired targets also calculate the loss
237
+ logits = self.output(h)
238
+ loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1), ignore_index=-1)
239
+ else:
240
+ # inference-time mini-optimization: only forward the output on the very last position
241
+ logits = self.output(h[:, [-1], :]) # note: using list [-1] to preserve the time dim
242
+ loss = None
243
+
244
+ return logits, loss
245
+
246
+ def configure_optimizers(self, weight_decay, learning_rate, betas, device_type):
247
+ # start with all of the candidate parameters
248
+ param_dict = {pn: p for pn, p in self.named_parameters()}
249
+ # filter out those that do not require grad
250
+ param_dict = {pn: p for pn, p in param_dict.items() if p.requires_grad}
251
+ # create optim groups. Any parameters that is 2D will be weight decayed, otherwise no.
252
+ # i.e. all weight tensors in matmuls + embeddings decay, all biases and layernorms don't.
253
+ decay_params = [p for n, p in param_dict.items() if p.dim() >= 2]
254
+ nodecay_params = [p for n, p in param_dict.items() if p.dim() < 2]
255
+ optim_groups = [
256
+ {'params': decay_params, 'weight_decay': weight_decay},
257
+ {'params': nodecay_params, 'weight_decay': 0.0}
258
+ ]
259
+ num_decay_params = sum(p.numel() for p in decay_params)
260
+ num_nodecay_params = sum(p.numel() for p in nodecay_params)
261
+ print(f"num decayed parameter tensors: {len(decay_params)}, with {num_decay_params:,} parameters")
262
+ print(f"num non-decayed parameter tensors: {len(nodecay_params)}, with {num_nodecay_params:,} parameters")
263
+ # Create AdamW optimizer and use the fused version if it is available
264
+ fused_available = 'fused' in inspect.signature(torch.optim.AdamW).parameters
265
+ use_fused = fused_available and device_type == 'cuda'
266
+ extra_args = dict(fused=True) if use_fused else dict()
267
+ optimizer = torch.optim.AdamW(optim_groups, lr=learning_rate, betas=betas, **extra_args)
268
+ print(f"using fused AdamW: {use_fused}")
269
+
270
+ return optimizer
271
+
272
+ def estimate_mfu(self, fwdbwd_per_iter, dt):
273
+ """ estimate model flops utilization (MFU) in units of A100 bfloat16 peak FLOPS """
274
+ # first estimate the number of flops we do per iteration.
275
+ # see PaLM paper Appendix B as ref: https://arxiv.org/abs/2204.02311
276
+ N = sum(p.numel() for p in self.parameters())
277
+ cfg = self.params
278
+ L, H, Q, T = cfg.n_layers, cfg.n_heads, cfg.dim//cfg.n_heads, cfg.max_seq_len
279
+ flops_per_token = 6*N + 12*L*H*Q*T
280
+ flops_per_fwdbwd = flops_per_token * T
281
+ flops_per_iter = flops_per_fwdbwd * fwdbwd_per_iter
282
+ # express our flops throughput as ratio of A100 bfloat16 peak flops
283
+ flops_achieved = flops_per_iter * (1.0/dt) # per second
284
+ flops_promised = 312e12 # A100 GPU bfloat16 peak flops is 312 TFLOPS
285
+ mfu = flops_achieved / flops_promised
286
+ return mfu
287
+
288
+ @torch.inference_mode()
289
+ def generate(self, idx, max_new_tokens, temperature=1.0, top_k=None):
290
+ """
291
+ Take a conditioning sequence of indices idx (LongTensor of shape (b,t)) and complete
292
+ the sequence max_new_tokens times, feeding the predictions back into the model each time.
293
+ Most likely you'll want to make sure to be in model.eval() mode of operation for this.
294
+ Also note this is a super inefficient version of sampling with no key/value cache.
295
+ """
296
+ for _ in range(max_new_tokens):
297
+ # if the sequence context is growing too long we must crop it at block_size
298
+ idx_cond = idx if idx.size(1) <= self.params.max_seq_len else idx[:, -self.params.max_seq_len:]
299
+ # forward the model to get the logits for the index in the sequence
300
+ logits, _ = self(idx_cond)
301
+ logits = logits[:, -1, :] # crop to just the final time step
302
+ if temperature == 0.0:
303
+ # "sample" the single most likely index
304
+ _, idx_next = torch.topk(logits, k=1, dim=-1)
305
+ else:
306
+ # pluck the logits at the final step and scale by desired temperature
307
+ logits = logits / temperature
308
+ # optionally crop the logits to only the top k options
309
+ if top_k is not None:
310
+ v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
311
+ logits[logits < v[:, [-1]]] = -float('Inf')
312
+ # apply softmax to convert logits to (normalized) probabilities
313
+ probs = F.softmax(logits, dim=-1)
314
+ idx_next = torch.multinomial(probs, num_samples=1)
315
+ # append sampled index to the running sequence and continue
316
+ idx = torch.cat((idx, idx_next), dim=1)
317
+
318
+ return idx
319
+
320
+ def export(self, filepath='model.bin'):
321
+ """export the model weights in fp32 into .bin file to be read from C"""
322
+ f = open(filepath, 'wb')
323
+
324
+ def serialize(t):
325
+ d = t.detach().cpu().view(-1).numpy().astype(np.float32)
326
+ b = struct.pack(f'{len(d)}f', *d)
327
+ f.write(b)
328
+
329
+ # first write out the header
330
+ hidden_dim = self.layers[0].feed_forward.w1.weight.shape[0]
331
+ p = self.params
332
+ n_kv_heads = p.n_heads if p.n_kv_heads is None else p.n_kv_heads
333
+ header = struct.pack('iiiiiii', p.dim, hidden_dim, p.n_layers, p.n_heads,
334
+ n_kv_heads, p.vocab_size, p.max_seq_len)
335
+ f.write(header)
336
+
337
+ # next write out the embedding weights
338
+ serialize(self.tok_embeddings.weight)
339
+
340
+ # now all the layers
341
+ # attention weights
342
+ for layer in self.layers:
343
+ serialize(layer.attention_norm.weight)
344
+ for layer in self.layers:
345
+ serialize(layer.attention.wq.weight)
346
+ for layer in self.layers:
347
+ serialize(layer.attention.wk.weight)
348
+ for layer in self.layers:
349
+ serialize(layer.attention.wv.weight)
350
+ for layer in self.layers:
351
+ serialize(layer.attention.wo.weight)
352
+ # ffn weights
353
+ for layer in self.layers:
354
+ serialize(layer.ffn_norm.weight)
355
+ for layer in self.layers:
356
+ serialize(layer.feed_forward.w1.weight)
357
+ for layer in self.layers:
358
+ serialize(layer.feed_forward.w2.weight)
359
+ for layer in self.layers:
360
+ serialize(layer.feed_forward.w3.weight)
361
+ # final rmsnorm
362
+ serialize(self.norm.weight)
363
+ # note: no need to write final classifier weights due to weight sharing
364
+ # freqs_cis
365
+ serialize(self.freqs_cis.real[:p.max_seq_len])
366
+ serialize(self.freqs_cis.imag[:p.max_seq_len])
367
+
368
+ # write to binary file
369
+ f.close()
370
+ print(f"wrote {filepath}")
requirements.txt ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ numpy==1.23.5
2
+ pytest==7.4.0
3
+ Requests==2.31.0
4
+ sentencepiece==0.1.99
5
+ tiktoken==0.3.3
6
+ torch==2.0.1
7
+ tqdm==4.64.1
8
+ wandb==0.15.5
run ADDED
Binary file (29.7 kB). View file
 
run.c ADDED
@@ -0,0 +1,490 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /*
2
+ Inference for Llama-2 Transformer model in pure C.
3
+
4
+ Example compile: (see README for more details)
5
+ $ gcc -O3 -o run run.c -lm
6
+
7
+ Then run with:
8
+ $ ./run
9
+ */
10
+
11
+ #include <stdio.h>
12
+ #include <stdlib.h>
13
+ #include <time.h>
14
+ #include <math.h>
15
+ #include <string.h>
16
+ #include <sys/time.h>
17
+
18
+ // ----------------------------------------------------------------------------
19
+ // Transformer and RunState structs, and related memory management
20
+
21
+ typedef struct {
22
+ int dim; // transformer dimension
23
+ int hidden_dim; // for ffn layers
24
+ int n_layers; // number of layers
25
+ int n_heads; // number of query heads
26
+ int n_kv_heads; // number of key/value heads (can be < query heads because of multiquery)
27
+ int vocab_size; // vocabulary size, usually 256 (byte-level)
28
+ int seq_len; // max sequence length
29
+ } Config;
30
+
31
+ typedef struct {
32
+ // token embedding table
33
+ float* token_embedding_table; // (vocab_size, dim)
34
+ // weights for rmsnorms
35
+ float* rms_att_weight; // (layer, dim) rmsnorm weights
36
+ float* rms_ffn_weight; // (layer, dim)
37
+ // weights for matmuls
38
+ float* wq; // (layer, dim, dim)
39
+ float* wk; // (layer, dim, dim)
40
+ float* wv; // (layer, dim, dim)
41
+ float* wo; // (layer, dim, dim)
42
+ // weights for ffn
43
+ float* w1; // (layer, hidden_dim, dim)
44
+ float* w2; // (layer, dim, hidden_dim)
45
+ float* w3; // (layer, hidden_dim, dim)
46
+ // final rmsnorm
47
+ float* rms_final_weight; // (dim,)
48
+ // freq_cis for RoPE relatively positional embeddings
49
+ float* freq_cis_real; // (seq_len, dim/2)
50
+ float* freq_cis_imag; // (seq_len, dim/2)
51
+ } TransformerWeights;
52
+
53
+ typedef struct {
54
+ // current wave of activations
55
+ float *x; // activation at current time stamp (dim,)
56
+ float *xb; // same, but inside a residual branch (dim,)
57
+ float *xb2; // an additional buffer just for convenience (dim,)
58
+ float *hb; // buffer for hidden dimension in the ffn (hidden_dim,)
59
+ float *hb2; // buffer for hidden dimension in the ffn (hidden_dim,)
60
+ float *q; // query (dim,)
61
+ float *k; // key (dim,)
62
+ float *v; // value (dim,)
63
+ float *att; // buffer for scores/attention values (seq_len,)
64
+ float *logits; // output logits
65
+ // kv cache
66
+ float* key_cache; // (layer, seq_len, dim)
67
+ float* value_cache; // (layer, seq_len, dim)
68
+ } RunState;
69
+
70
+ void malloc_run_state(RunState* s, Config* p) {
71
+ // we calloc instead of malloc to keep valgrind happy
72
+ s->x = calloc(p->dim, sizeof(float));
73
+ s->xb = calloc(p->dim, sizeof(float));
74
+ s->xb2 = calloc(p->dim, sizeof(float));
75
+ s->hb = calloc(p->hidden_dim, sizeof(float));
76
+ s->hb2 = calloc(p->hidden_dim, sizeof(float));
77
+ s->q = calloc(p->dim, sizeof(float));
78
+ s->k = calloc(p->dim, sizeof(float));
79
+ s->v = calloc(p->dim, sizeof(float));
80
+ s->att = calloc(p->seq_len, sizeof(float));
81
+ s->logits = calloc(p->vocab_size, sizeof(float));
82
+ s->key_cache = calloc(p->n_layers * p->seq_len * p->dim, sizeof(float));
83
+ s->value_cache = calloc(p->n_layers * p->seq_len * p->dim, sizeof(float));
84
+ // ensure all mallocs went fine
85
+ if (!s->x || !s->xb || !s->xb2 || !s->hb || !s->hb2 || !s->q
86
+ || !s->k || !s->v || !s->att || !s->logits || !s->key_cache
87
+ || !s->value_cache) {
88
+ printf("malloc failed!\n");
89
+ exit(1);
90
+ }
91
+ }
92
+
93
+ void free_run_state(RunState* s) {
94
+ free(s->x);
95
+ free(s->xb);
96
+ free(s->xb2);
97
+ free(s->hb);
98
+ free(s->hb2);
99
+ free(s->q);
100
+ free(s->k);
101
+ free(s->v);
102
+ free(s->att);
103
+ free(s->logits);
104
+ free(s->key_cache);
105
+ free(s->value_cache);
106
+ }
107
+
108
+ void malloc_weights(TransformerWeights* w, Config* p) {
109
+ // we calloc instead of malloc to keep valgrind happy
110
+ w->token_embedding_table = calloc(p->vocab_size * p->dim, sizeof(float));
111
+ w->rms_att_weight = calloc(p->n_layers * p->dim, sizeof(float));
112
+ w->rms_ffn_weight = calloc(p->n_layers * p->dim, sizeof(float));
113
+ w->wq = calloc(p->n_layers * p->dim * p->dim, sizeof(float));
114
+ w->wk = calloc(p->n_layers * p->dim * p->dim, sizeof(float));
115
+ w->wv = calloc(p->n_layers * p->dim * p->dim, sizeof(float));
116
+ w->wo = calloc(p->n_layers * p->dim * p->dim, sizeof(float));
117
+ w->w1 = calloc(p->n_layers * p->hidden_dim * p->dim, sizeof(float));
118
+ w->w2 = calloc(p->n_layers * p->dim * p->hidden_dim, sizeof(float));
119
+ w->w3 = calloc(p->n_layers * p->hidden_dim * p->dim, sizeof(float));
120
+ w->rms_final_weight = calloc(p->dim, sizeof(float));
121
+ w->freq_cis_real = calloc(p->seq_len * p->dim / 2, sizeof(float));
122
+ w->freq_cis_imag = calloc(p->seq_len * p->dim / 2, sizeof(float));
123
+ // ensure all mallocs went fine
124
+ if (!w->token_embedding_table || !w->rms_att_weight || !w->rms_ffn_weight
125
+ || !w->wq || !w->wk || !w->wv || !w->wo || !w->w1 || !w->w2 || !w->w3 ||
126
+ !w->rms_final_weight || !w->freq_cis_real || !w->freq_cis_imag) {
127
+ printf("malloc failed!\n");
128
+ exit(1);
129
+ }
130
+ }
131
+
132
+ void free_weights(TransformerWeights* w) {
133
+ free(w->token_embedding_table);
134
+ free(w->rms_att_weight);
135
+ free(w->rms_ffn_weight);
136
+ free(w->wq);
137
+ free(w->wk);
138
+ free(w->wv);
139
+ free(w->wo);
140
+ free(w->w1);
141
+ free(w->w2);
142
+ free(w->w3);
143
+ free(w->rms_final_weight);
144
+ free(w->freq_cis_real);
145
+ free(w->freq_cis_imag);
146
+ }
147
+
148
+ // ----------------------------------------------------------------------------
149
+ // initialization: read from checkpoint
150
+
151
+ int checkpoint_init_weights(TransformerWeights *w, Config* p, FILE* f) {
152
+ if (fread(w->token_embedding_table, sizeof(float), p->vocab_size * p->dim, f) != p->vocab_size * p->dim) return 1;
153
+ if (fread(w->rms_att_weight, sizeof(float), p->n_layers * p->dim, f) != p->n_layers * p->dim) return 1;
154
+ if (fread(w->wq, sizeof(float), p->n_layers * p->dim * p->dim, f) != p->n_layers * p->dim * p->dim) return 1;
155
+ if (fread(w->wk, sizeof(float), p->n_layers * p->dim * p->dim, f) != p->n_layers * p->dim * p->dim) return 1;
156
+ if (fread(w->wv, sizeof(float), p->n_layers * p->dim * p->dim, f) != p->n_layers * p->dim * p->dim) return 1;
157
+ if (fread(w->wo, sizeof(float), p->n_layers * p->dim * p->dim, f) != p->n_layers * p->dim * p->dim) return 1;
158
+ if (fread(w->rms_ffn_weight, sizeof(float), p->n_layers * p->dim, f) != p->n_layers * p->dim) return 1;
159
+ if (fread(w->w1, sizeof(float), p->n_layers * p->dim * p->hidden_dim, f) != p->n_layers * p->dim * p->hidden_dim) return 1;
160
+ if (fread(w->w2, sizeof(float), p->n_layers * p->hidden_dim * p->dim, f) != p->n_layers * p->hidden_dim * p->dim) return 1;
161
+ if (fread(w->w3, sizeof(float), p->n_layers * p->dim * p->hidden_dim, f) != p->n_layers * p->dim * p->hidden_dim) return 1;
162
+ if (fread(w->rms_final_weight, sizeof(float), p->dim, f) != p->dim) return 1;
163
+ int head_size = p->dim / p->n_heads;
164
+ if (fread(w->freq_cis_real, sizeof(float), p->seq_len * head_size / 2, f) != p->seq_len * head_size / 2) return 1;
165
+ if (fread(w->freq_cis_imag, sizeof(float), p->seq_len * head_size / 2, f) != p->seq_len * head_size / 2) return 1;
166
+ return 0;
167
+ }
168
+
169
+
170
+ // ----------------------------------------------------------------------------
171
+ // neural net blocks
172
+
173
+ void accum(float *a, float *b, int size) {
174
+ for (int i = 0; i < size; i++) {
175
+ a[i] += b[i];
176
+ }
177
+ }
178
+
179
+ void rmsnorm(float* o, float* x, float* weight, int size) {
180
+ // calculate sum of squares
181
+ float ss = 0.0f;
182
+ for (int j = 0; j < size; j++) {
183
+ ss += x[j] * x[j];
184
+ }
185
+ ss /= size;
186
+ ss += 1e-5f;
187
+ ss = 1.0f / sqrt(ss);
188
+ // normalize and scale
189
+ for (int j = 0; j < size; j++) {
190
+ o[j] = weight[j] * (ss * x[j]);
191
+ }
192
+ }
193
+
194
+ void softmax(float* x, int size) {
195
+ // find max value (for numerical stability)
196
+ float max_val = x[0];
197
+ for (int i = 1; i < size; i++) {
198
+ if (x[i] > max_val) {
199
+ max_val = x[i];
200
+ }
201
+ }
202
+ // exp and sum
203
+ float sum = 0.0f;
204
+ for (int i = 0; i < size; i++) {
205
+ x[i] = exp(x[i] - max_val);
206
+ sum += x[i];
207
+ }
208
+ // normalize
209
+ for (int i = 0; i < size; i++) {
210
+ x[i] /= sum;
211
+ }
212
+ }
213
+
214
+ void matmul(float* xout, float* x, float* w, int n, int d) {
215
+ // W (d,n) @ x (n,) -> xout (d,)
216
+ #pragma omp parallel for
217
+ for (int i = 0; i < d; i++) {
218
+ float val = 0.0f;
219
+ for (int j = 0; j < n; j++) {
220
+ val += w[i * n + j] * x[j];
221
+ }
222
+ xout[i] = val;
223
+ }
224
+ }
225
+
226
+ void transformer(int token, int pos, Config* p, RunState* s, TransformerWeights* w) {
227
+
228
+ // a few convenience variables
229
+ float *x = s->x;
230
+ int dim = p->dim;
231
+ int hidden_dim = p->hidden_dim;
232
+ int head_size = dim / p->n_heads;
233
+
234
+ // copy the token embedding into x
235
+ float* content_row = &(w->token_embedding_table[token * dim]);
236
+ memcpy(x, content_row, dim*sizeof(*x));
237
+
238
+ // pluck out the "pos" row of freq_cis_real and freq_cis_imag
239
+ float* freq_cis_real_row = w->freq_cis_real + pos * head_size / 2;
240
+ float* freq_cis_imag_row = w->freq_cis_imag + pos * head_size / 2;
241
+
242
+ // forward all the layers
243
+ for(int l = 0; l < p->n_layers; l++) {
244
+
245
+ // attention rmsnorm
246
+ rmsnorm(s->xb, x, w->rms_att_weight + l*dim, dim);
247
+
248
+ // qkv matmuls for this position
249
+ matmul(s->q, s->xb, w->wq + l*dim*dim, dim, dim);
250
+ matmul(s->k, s->xb, w->wk + l*dim*dim, dim, dim);
251
+ matmul(s->v, s->xb, w->wv + l*dim*dim, dim, dim);
252
+
253
+ // apply RoPE rotation to the q and k vectors for each head
254
+ for (int h = 0; h < p->n_heads; h++) {
255
+ // get the q and k vectors for this head
256
+ float* q = s->q + h * head_size;
257
+ float* k = s->k + h * head_size;
258
+ // rotate q and k by the freq_cis_real and freq_cis_imag
259
+ for (int i = 0; i < head_size; i+=2) {
260
+ float q0 = q[i];
261
+ float q1 = q[i+1];
262
+ float k0 = k[i];
263
+ float k1 = k[i+1];
264
+ float fcr = freq_cis_real_row[i/2];
265
+ float fci = freq_cis_imag_row[i/2];
266
+ q[i] = q0 * fcr - q1 * fci;
267
+ q[i+1] = q0 * fci + q1 * fcr;
268
+ k[i] = k0 * fcr - k1 * fci;
269
+ k[i+1] = k0 * fci + k1 * fcr;
270
+ }
271
+ }
272
+
273
+ // save key,value at this time step (pos) to our kv cache
274
+ int loff = l * p->seq_len * dim; // kv cache layer offset for convenience
275
+ float* key_cache_row = s->key_cache + loff + pos * dim;
276
+ float* value_cache_row = s->value_cache + loff + pos * dim;
277
+ memcpy(key_cache_row, s->k, dim*sizeof(*key_cache_row));
278
+ memcpy(value_cache_row, s->v, dim*sizeof(*value_cache_row));
279
+
280
+ // multihead attention. iterate over all heads
281
+ for (int h = 0; h < p->n_heads; h++) {
282
+ // get the query vector for this head
283
+ float* q = s->q + h * head_size;
284
+ // iterate over all timesteps, including the current one
285
+ for (int t = 0; t <= pos; t++) {
286
+ // get the key vector for this head and at this timestep
287
+ float* k = s->key_cache + loff + t * dim + h * head_size;
288
+ // calculate the attention score as the dot product of q and k
289
+ float score = 0.0f;
290
+ for (int i = 0; i < head_size; i++) {
291
+ score += q[i] * k[i];
292
+ }
293
+ score /= sqrtf(head_size);
294
+ // save the score to the attention buffer
295
+ s->att[t] = score;
296
+ }
297
+
298
+ // softmax the scores to get attention weights, from 0..pos inclusively
299
+ softmax(s->att, pos + 1);
300
+
301
+ // weighted sum of the values, store back into xb
302
+ for (int i = 0; i < head_size; i++) {
303
+ float val = 0.0f;
304
+ for (int t = 0; t <= pos; t++) {
305
+ val += s->att[t] * s->value_cache[loff + t * dim + h * head_size + i]; // note bad locality
306
+ }
307
+ s->xb[h * head_size + i] = val;
308
+ }
309
+ }
310
+
311
+ // final matmul to get the output of the attention
312
+ matmul(s->xb2, s->xb, w->wo + l*dim*dim, dim, dim);
313
+
314
+ // residual connection back into x
315
+ accum(x, s->xb2, dim);
316
+
317
+ // ffn rmsnorm
318
+ rmsnorm(s->xb, x, w->rms_ffn_weight + l*dim, dim);
319
+
320
+ // Now for FFN in PyTorch we have: self.w2(F.silu(self.w1(x)) * self.w3(x))
321
+ // first calculate self.w1(x) and self.w3(x)
322
+ matmul(s->hb, s->xb, w->w1 + l*dim*hidden_dim, dim, hidden_dim);
323
+ matmul(s->hb2, s->xb, w->w3 + l*dim*hidden_dim, dim, hidden_dim);
324
+
325
+ // F.silu; silu(x)=x*σ(x),where σ(x) is the logistic sigmoid
326
+ for (int i = 0; i < hidden_dim; i++) {
327
+ s->hb[i] = s->hb[i] * (1.0f / (1.0f + expf(-s->hb[i])));
328
+ }
329
+
330
+ // elementwise multiply with w3(x)
331
+ for (int i = 0; i < hidden_dim; i++) {
332
+ s->hb[i] = s->hb[i] * s->hb2[i];
333
+ }
334
+
335
+ // final matmul to get the output of the ffn
336
+ matmul(s->xb, s->hb, w->w2 + l*dim*hidden_dim, hidden_dim, dim);
337
+
338
+ // residual connection
339
+ accum(x, s->xb, dim);
340
+ }
341
+
342
+ // final rmsnorm
343
+ rmsnorm(x, x, w->rms_final_weight, dim);
344
+
345
+ // classifier into logits
346
+ matmul(s->logits, x, w->token_embedding_table, p->dim, p->vocab_size);
347
+ }
348
+
349
+ int sample(float* probabilities, int n) {
350
+ // sample index from probabilities, they must sum to 1
351
+ float r = (float)rand() / (float)RAND_MAX;
352
+ float cdf = 0.0f;
353
+ for (int i = 0; i < n; i++) {
354
+ cdf += probabilities[i];
355
+ if (r < cdf) {
356
+ return i;
357
+ }
358
+ }
359
+ return n - 1; // in case of rounding errors
360
+ }
361
+
362
+ int argmax(float* v, int n) {
363
+ // return argmax of v in elements 0..n
364
+ int max_i = 0;
365
+ float max_p = v[0];
366
+ for (int i = 1; i < n; i++) {
367
+ if (v[i] > max_p) {
368
+ max_i = i;
369
+ max_p = v[i];
370
+ }
371
+ }
372
+ return max_i;
373
+ }
374
+
375
+ // ----------------------------------------------------------------------------
376
+
377
+ long time_in_ms() {
378
+ struct timeval time;
379
+ gettimeofday(&time, NULL);
380
+ return time.tv_sec * 1000 + time.tv_usec / 1000;
381
+ }
382
+
383
+ int main(int argc, char *argv[]) {
384
+
385
+ // poor man's C argparse
386
+ char *checkpoint = NULL;
387
+ float temperature = 0.9f;
388
+ // 'checkpoint' is necessary arg
389
+ if (argc < 2) {
390
+ printf("Usage: %s <checkpoint_file> [temperature] [seed]\n", argv[0]);
391
+ return 1;
392
+ }
393
+ checkpoint = argv[1];
394
+ // temperature is optional
395
+ if (argc >= 3) {
396
+ temperature = atof(argv[2]);
397
+ }
398
+ // seed is optional
399
+ if (argc >= 4) {
400
+ unsigned int seed = atoi(argv[3]);
401
+ srand(seed);
402
+ } else {
403
+ time_t current_time;
404
+ time(&current_time);
405
+ srand((unsigned int)current_time);
406
+ }
407
+
408
+ // read in the model.bin file
409
+ Config config;
410
+ TransformerWeights weights;
411
+ {
412
+ FILE *file = fopen(checkpoint, "rb");
413
+ if (!file) {
414
+ printf("Unable to open the checkpoint file %s!\n", checkpoint);
415
+ return 1;
416
+ }
417
+ // read in the config header
418
+ if(fread(&config, sizeof(Config), 1, file) != 1) { return 1; }
419
+ // read in the Transformer weights
420
+ malloc_weights(&weights, &config);
421
+ if(checkpoint_init_weights(&weights, &config, file)) { return 1; }
422
+ fclose(file);
423
+ }
424
+
425
+ // read in the tokenizer.bin file
426
+ char** vocab = (char**)malloc(config.vocab_size * sizeof(char*));
427
+ {
428
+ FILE *file = fopen("tokenizer.bin", "rb");
429
+ if (!file) {
430
+ printf("Unable to open the tokenizer file tokenizer.bin! Run "
431
+ "python tokenizer.py to convert tokenizer.model -> tokenizer.bin\n");
432
+ return 1;
433
+ }
434
+ int len;
435
+ for (int i = 0; i < config.vocab_size; i++) {
436
+ if(fread(&len, sizeof(int), 1, file) != 1) { return 1; }
437
+ vocab[i] = (char *)malloc(len + 1);
438
+ if(fread(vocab[i], len, 1, file) != 1) { return 1; }
439
+ vocab[i][len] = '\0'; // add the string terminating token
440
+ }
441
+ fclose(file);
442
+ }
443
+
444
+ // create and init the application RunState
445
+ RunState state;
446
+ malloc_run_state(&state, &config);
447
+
448
+ // the current position we are in
449
+ long start = time_in_ms();
450
+
451
+ int next;
452
+ int token = 1; // 1 = BOS token in Llama-2 sentencepiece
453
+ int pos = 0;
454
+ while (pos < config.seq_len) {
455
+
456
+ // forward the transformer to get logits for the next token
457
+ transformer(token, pos, &config, &state, &weights);
458
+
459
+ // sample the next token
460
+ if(temperature == 0.0f) {
461
+ // greedy argmax sampling
462
+ next = argmax(state.logits, config.vocab_size);
463
+ } else {
464
+ // apply the temperature to the logits
465
+ for (int q=0; q<config.vocab_size; q++) { state.logits[q] /= temperature; }
466
+ // apply softmax to the logits to get the probabilities for next token
467
+ softmax(state.logits, config.vocab_size);
468
+ // we now want to sample from this distribution to get the next token
469
+ next = sample(state.logits, config.vocab_size);
470
+ }
471
+ printf("%s", vocab[next]);
472
+ fflush(stdout);
473
+
474
+ // advance forward
475
+ token = next;
476
+ pos++;
477
+ }
478
+ printf("\n");
479
+
480
+ // report our achieved tok/s
481
+ long end = time_in_ms();
482
+ printf("achieved tok/s: %f\n", config.seq_len / (double)(end-start)*1000);
483
+
484
+ // memory cleanup
485
+ free_run_state(&state);
486
+ free_weights(&weights);
487
+ for (int i = 0; i < config.vocab_size; i++) { free(vocab[i]); }
488
+ free(vocab);
489
+ return 0;
490
+ }
run.exe ADDED
Binary file (81.5 kB). View file
 
sample.py ADDED
@@ -0,0 +1,69 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Sample from the trained model with PyTorch
3
+ """
4
+ import os
5
+ import pickle
6
+ from contextlib import nullcontext
7
+ import torch
8
+ import tiktoken
9
+ from model import ModelArgs, Transformer
10
+ from tokenizer import Tokenizer
11
+
12
+ # -----------------------------------------------------------------------------
13
+ out_dir = 'out' # ignored if init_from is not 'resume'
14
+ start = "" # or "<|endoftext|>" or etc. Can also specify a file, use as: "FILE:prompt.txt"
15
+ num_samples = 1 # number of samples to draw
16
+ max_new_tokens = 100 # number of tokens generated in each sample
17
+ temperature = 1.0 # 1.0 = no change, < 1.0 = less random, > 1.0 = more random, in predictions
18
+ top_k = 300 # retain only the top_k most likely tokens, clamp others to have 0 probability
19
+ seed = 1337
20
+ device = 'cuda' if torch.cuda.is_available() else 'cpu' # examples: 'cpu', 'cuda', 'cuda:0', 'cuda:1', etc.
21
+ #dtype = 'bfloat16' if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else 'float16' # 'float32' or 'bfloat16' or 'float16'
22
+ dtype = "float32"
23
+ compile = False # use PyTorch 2.0 to compile the model to be faster
24
+ exec(open('configurator.py').read()) # overrides from command line or config file
25
+ # -----------------------------------------------------------------------------
26
+
27
+ torch.manual_seed(seed)
28
+ torch.cuda.manual_seed(seed)
29
+ torch.backends.cuda.matmul.allow_tf32 = True # allow tf32 on matmul
30
+ torch.backends.cudnn.allow_tf32 = True # allow tf32 on cudnn
31
+ device_type = 'cuda' if 'cuda' in device else 'cpu' # for later use in torch.autocast
32
+ ptdtype = {'float32': torch.float32, 'bfloat16': torch.bfloat16, 'float16': torch.float16}[dtype]
33
+ ctx = nullcontext() if device_type == 'cpu' else torch.amp.autocast(device_type=device_type, dtype=ptdtype)
34
+
35
+ # init from a model saved in a specific directory
36
+ ckpt_path = os.path.join(out_dir, 'ckpt.pt')
37
+ checkpoint = torch.load(ckpt_path, map_location=device)
38
+ gptconf = ModelArgs(**checkpoint['model_args'])
39
+ model = Transformer(gptconf)
40
+ state_dict = checkpoint['model']
41
+ unwanted_prefix = '_orig_mod.'
42
+ for k,v in list(state_dict.items()):
43
+ if k.startswith(unwanted_prefix):
44
+ state_dict[k[len(unwanted_prefix):]] = state_dict.pop(k)
45
+ model.load_state_dict(state_dict, strict=False)
46
+
47
+ model.eval()
48
+ model.to(device)
49
+ if compile:
50
+ print("Compiling the model...")
51
+ model = torch.compile(model) # requires PyTorch 2.0 (optional)
52
+
53
+ # load the tokenizer
54
+ enc = Tokenizer()
55
+
56
+ # encode the beginning of the prompt
57
+ if start.startswith('FILE:'):
58
+ with open(start[5:], 'r', encoding='utf-8') as f:
59
+ start = f.read()
60
+ start_ids = enc.encode(start, bos=True, eos=False)
61
+ x = (torch.tensor(start_ids, dtype=torch.long, device=device)[None, ...])
62
+
63
+ # run generation
64
+ with torch.no_grad():
65
+ with ctx:
66
+ for k in range(num_samples):
67
+ y = model.generate(x, max_new_tokens, temperature=temperature, top_k=top_k)
68
+ print(enc.decode(y[0].tolist()))
69
+ print('---------------')
sample_data/README.md ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ This directory includes a few sample datasets to get you started.
2
+
3
+ * `california_housing_data*.csv` is California housing data from the 1990 US
4
+ Census; more information is available at:
5
+ https://developers.google.com/machine-learning/crash-course/california-housing-data-description
6
+
7
+ * `mnist_*.csv` is a small sample of the
8
+ [MNIST database](https://en.wikipedia.org/wiki/MNIST_database), which is
9
+ described at: http://yann.lecun.com/exdb/mnist/
10
+
11
+ * `anscombe.json` contains a copy of
12
+ [Anscombe's quartet](https://en.wikipedia.org/wiki/Anscombe%27s_quartet); it
13
+ was originally described in
14
+
15
+ Anscombe, F. J. (1973). 'Graphs in Statistical Analysis'. American
16
+ Statistician. 27 (1): 17-21. JSTOR 2682899.
17
+
18
+ and our copy was prepared by the
19
+ [vega_datasets library](https://github.com/altair-viz/vega_datasets/blob/4f67bdaad10f45e3549984e17e1b3088c731503d/vega_datasets/_data/anscombe.json).
sample_data/anscombe.json ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {"Series":"I", "X":10.0, "Y":8.04},
3
+ {"Series":"I", "X":8.0, "Y":6.95},
4
+ {"Series":"I", "X":13.0, "Y":7.58},
5
+ {"Series":"I", "X":9.0, "Y":8.81},
6
+ {"Series":"I", "X":11.0, "Y":8.33},
7
+ {"Series":"I", "X":14.0, "Y":9.96},
8
+ {"Series":"I", "X":6.0, "Y":7.24},
9
+ {"Series":"I", "X":4.0, "Y":4.26},
10
+ {"Series":"I", "X":12.0, "Y":10.84},
11
+ {"Series":"I", "X":7.0, "Y":4.81},
12
+ {"Series":"I", "X":5.0, "Y":5.68},
13
+
14
+ {"Series":"II", "X":10.0, "Y":9.14},
15
+ {"Series":"II", "X":8.0, "Y":8.14},
16
+ {"Series":"II", "X":13.0, "Y":8.74},
17
+ {"Series":"II", "X":9.0, "Y":8.77},
18
+ {"Series":"II", "X":11.0, "Y":9.26},
19
+ {"Series":"II", "X":14.0, "Y":8.10},
20
+ {"Series":"II", "X":6.0, "Y":6.13},
21
+ {"Series":"II", "X":4.0, "Y":3.10},
22
+ {"Series":"II", "X":12.0, "Y":9.13},
23
+ {"Series":"II", "X":7.0, "Y":7.26},
24
+ {"Series":"II", "X":5.0, "Y":4.74},
25
+
26
+ {"Series":"III", "X":10.0, "Y":7.46},
27
+ {"Series":"III", "X":8.0, "Y":6.77},
28
+ {"Series":"III", "X":13.0, "Y":12.74},
29
+ {"Series":"III", "X":9.0, "Y":7.11},
30
+ {"Series":"III", "X":11.0, "Y":7.81},
31
+ {"Series":"III", "X":14.0, "Y":8.84},
32
+ {"Series":"III", "X":6.0, "Y":6.08},
33
+ {"Series":"III", "X":4.0, "Y":5.39},
34
+ {"Series":"III", "X":12.0, "Y":8.15},
35
+ {"Series":"III", "X":7.0, "Y":6.42},
36
+ {"Series":"III", "X":5.0, "Y":5.73},
37
+
38
+ {"Series":"IV", "X":8.0, "Y":6.58},
39
+ {"Series":"IV", "X":8.0, "Y":5.76},
40
+ {"Series":"IV", "X":8.0, "Y":7.71},
41
+ {"Series":"IV", "X":8.0, "Y":8.84},
42
+ {"Series":"IV", "X":8.0, "Y":8.47},
43
+ {"Series":"IV", "X":8.0, "Y":7.04},
44
+ {"Series":"IV", "X":8.0, "Y":5.25},
45
+ {"Series":"IV", "X":19.0, "Y":12.50},
46
+ {"Series":"IV", "X":8.0, "Y":5.56},
47
+ {"Series":"IV", "X":8.0, "Y":7.91},
48
+ {"Series":"IV", "X":8.0, "Y":6.89}
49
+ ]
sample_data/california_housing_test.csv ADDED
The diff for this file is too large to render. See raw diff
 
sample_data/california_housing_train.csv ADDED
The diff for this file is too large to render. See raw diff
 
sample_data/mnist_test.csv ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:51c292478d94ec3a01461bdfa82eb0885d262eb09e615679b2d69dedb6ad09e7
3
+ size 18289443
sample_data/mnist_train_small.csv ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ef64781aa03180f4f5ce504314f058f5d0227277df86060473d973cf43b033e
3
+ size 36523880
test_all.py ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Run simply with
3
+ $ pytest
4
+ """
5
+ import os
6
+ import pytest # pip install pytest
7
+ import subprocess
8
+
9
+ import torch
10
+ from model import ModelArgs, Transformer
11
+
12
+ def test_argmax_inference():
13
+ """
14
+ Only the simplest test for now: run inference with temperature 0
15
+ (for determinism) in both C and PyTorch, and see that the sampled tokens
16
+ are the same.
17
+ """
18
+ test_ckpt_dir = "out" # TODO create a dummy test checkpoint for this?
19
+
20
+ # run C version
21
+ model_path = os.path.join(test_ckpt_dir, "model.bin")
22
+ command = ["./run", model_path, "0.0"]
23
+ proc = subprocess.Popen(command, stdout=subprocess.PIPE)
24
+ c_tokens = []
25
+ for line in proc.stdout:
26
+ token = int(line.decode('utf-8').strip())
27
+ c_tokens.append(token)
28
+ proc.wait()
29
+ #print(c_tokens)
30
+
31
+ # run PyTorch version
32
+ device = "cuda" if torch.cuda.is_available() else "cpu"
33
+ ckpt_path = os.path.join(test_ckpt_dir, "ckpt.pt")
34
+ checkpoint = torch.load(ckpt_path, map_location=device)
35
+ gptconf = ModelArgs(**checkpoint['model_args'])
36
+ model = Transformer(gptconf)
37
+ state_dict = checkpoint['model']
38
+ unwanted_prefix = '_orig_mod.'
39
+ for k,v in list(state_dict.items()):
40
+ if k.startswith(unwanted_prefix):
41
+ state_dict[k[len(unwanted_prefix):]] = state_dict.pop(k)
42
+ model.load_state_dict(state_dict, strict=False)
43
+ model.eval()
44
+ model.to(device)
45
+ x = torch.tensor([[1]], dtype=torch.long, device=device) # 1 is BOS
46
+ with torch.inference_mode():
47
+ y = model.generate(x, max_new_tokens=gptconf.max_seq_len, temperature=0.0)
48
+ pt_tokens = y[0].tolist()
49
+ pt_tokens = pt_tokens[1:] # remove BOS
50
+ #print(pt_tokens)
51
+
52
+ # compare
53
+ assert c_tokens == pt_tokens
tinystories.py ADDED
@@ -0,0 +1,166 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Download, preprocess and serve the TinyStories dataset as a DataLoader.
3
+ """
4
+
5
+ import argparse
6
+ import glob
7
+ import json
8
+ import os
9
+ import random
10
+ from typing import List
11
+ from concurrent.futures import ThreadPoolExecutor, as_completed
12
+
13
+ import numpy as np
14
+ import requests
15
+ import torch
16
+ import torch.distributed as dist
17
+ from tqdm import tqdm
18
+
19
+ from tokenizer import Tokenizer
20
+
21
+ DATA_CACHE_DIR = "data"
22
+
23
+ def download_file(url: str, fname: str, chunk_size=1024):
24
+ """Helper function to download a file from a given url"""
25
+ resp = requests.get(url, stream=True)
26
+ total = int(resp.headers.get("content-length", 0))
27
+ with open(fname, "wb") as file, tqdm(
28
+ desc=fname,
29
+ total=total,
30
+ unit="iB",
31
+ unit_scale=True,
32
+ unit_divisor=1024,
33
+ ) as bar:
34
+ for data in resp.iter_content(chunk_size=chunk_size):
35
+ size = file.write(data)
36
+ bar.update(size)
37
+
38
+
39
+ def download():
40
+ """Downloads the dataset to disk."""
41
+ os.makedirs(DATA_CACHE_DIR, exist_ok=True)
42
+
43
+ # download the TinyStories dataset, unless it's already downloaded
44
+ data_url = "https://huggingface.co/datasets/roneneldan/TinyStories/resolve/main/TinyStories_all_data.tar.gz"
45
+ data_filename = os.path.join(DATA_CACHE_DIR, "TinyStories_all_data.tar.gz")
46
+ if not os.path.exists(data_filename):
47
+ print(f"Downloading {data_url} to {data_filename}...")
48
+ download_file(data_url, data_filename)
49
+ else:
50
+ print(f"{data_filename} already exists, skipping download...")
51
+
52
+ # unpack the tar.gz file into all the data shards (json files)
53
+ data_dir = os.path.join(DATA_CACHE_DIR, "TinyStories_all_data")
54
+ if not os.path.exists(data_dir):
55
+ os.makedirs(data_dir, exist_ok=True)
56
+ print(f"Unpacking {data_filename}...")
57
+ os.system(f"tar -xzf {data_filename} -C {data_dir}")
58
+ else:
59
+ print(f"{data_dir} already exists, skipping unpacking...")
60
+
61
+ # print a single example just for debugging and such
62
+ shard_filenames = sorted(glob.glob(os.path.join(data_dir, "*.json")))
63
+ with open(shard_filenames[0], "r") as f:
64
+ data = json.load(f)
65
+ print("Download done.")
66
+ print(f"Number of shards: {len(shard_filenames)}")
67
+ print(f"Example story:\n{data[0]}")
68
+
69
+ def pretokenize():
70
+ enc = Tokenizer()
71
+
72
+ def process_shard(shard):
73
+ with open(shard, "r") as f:
74
+ data = json.load(f)
75
+ all_tokens = []
76
+ for example in tqdm(data):
77
+ text = example["story"]
78
+ text = text.strip() # get rid of leading/trailing whitespace
79
+ tokens = enc.encode(text, bos=True, eos=False) # encode the text, use BOS
80
+ all_tokens.extend(tokens)
81
+ # convert to uint16 nparray
82
+ all_tokens = np.array(all_tokens, dtype=np.uint16)
83
+ # write to disk
84
+ tokenized_filename = shard.replace(".json", ".bin")
85
+ with open(tokenized_filename, "wb") as f:
86
+ f.write(all_tokens.tobytes())
87
+ print(f"Saved {tokenized_filename}")
88
+
89
+ # iterate the shards and tokenize all of them one by one
90
+ data_dir = os.path.join(DATA_CACHE_DIR, "TinyStories_all_data")
91
+ shard_filenames = sorted(glob.glob(os.path.join(data_dir, "*.json")))
92
+
93
+ # process all the shards in a threadpool
94
+ with ThreadPoolExecutor(max_workers=8) as executor:
95
+ executor.map(process_shard, shard_filenames)
96
+
97
+ print("Done.")
98
+
99
+
100
+ class PretokDataset(torch.utils.data.IterableDataset):
101
+ """Loads pretokenized examples from disk and yields them as PyTorch tensors."""
102
+
103
+ def __init__(self, split, max_seq_len):
104
+ super().__init__()
105
+ self.split = split
106
+ self.max_seq_len = max_seq_len
107
+
108
+ def __iter__(self):
109
+ # get worker info within a DataLoader
110
+ worker_info = torch.utils.data.get_worker_info()
111
+ worker_id = worker_info.id if worker_info else 0
112
+ # get DDP rank info
113
+ rank = dist.get_rank() if dist.is_initialized() else 0
114
+ # combine the worker_id and worker_rank to create a unique seed for rng
115
+ seed = 42 + worker_id + 1337 * rank
116
+ rng = random.Random(seed)
117
+ print(f"Created a PretokDataset with rng seed {seed}")
118
+ data_dir = os.path.join(DATA_CACHE_DIR, "TinyStories_all_data")
119
+ shard_filenames = sorted(glob.glob(os.path.join(data_dir, "*.bin")))
120
+ # train/test split. let's use only shard 0 for test split, rest train
121
+ shard_filenames = shard_filenames[1:] if self.split == "train" else shard_filenames[:1]
122
+ while True:
123
+ rng.shuffle(shard_filenames)
124
+ for shard in shard_filenames:
125
+ # open the dataset for reading but keep it on disk with memmap
126
+ m = np.memmap(shard, dtype=np.uint16, mode="r")
127
+ num_batches = len(m) // self.max_seq_len
128
+ num_batches -= 1 # drop the last partial batch
129
+ assert num_batches > 0, "this shard is way too small? investigate."
130
+ ixs = list(range(num_batches))
131
+ rng.shuffle(ixs)
132
+ for ix in ixs:
133
+ start = ix * self.max_seq_len
134
+ end = start + self.max_seq_len + 1
135
+ # calling .astype will copy the data into a new numpy array, now in RAM
136
+ chunk = torch.from_numpy((m[start:end]).astype(np.int64))
137
+ x = chunk[:-1]
138
+ y = chunk[1:]
139
+ yield x, y
140
+
141
+
142
+ class Task:
143
+
144
+ @staticmethod
145
+ def iter_batches(split, batch_size, max_seq_len, device, num_workers=0):
146
+ ds = PretokDataset(split, max_seq_len)
147
+ dl = torch.utils.data.DataLoader(
148
+ ds, batch_size=batch_size, pin_memory=True, num_workers=num_workers
149
+ )
150
+ for x, y in dl:
151
+ x = x.to(device, non_blocking=True)
152
+ y = y.to(device, non_blocking=True)
153
+ yield x, y
154
+
155
+
156
+ if __name__ == "__main__":
157
+ parser = argparse.ArgumentParser()
158
+ parser.add_argument("stage", type=str, choices=["download", "train_tokenizer", "pretokenize"])
159
+ args = parser.parse_args()
160
+
161
+ # depending on the stage call the appropriate function
162
+ fun = {
163
+ "download": download,
164
+ "pretokenize": pretokenize,
165
+ }
166
+ fun[args.stage]()
tokenizer.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7d606805ba3c66fb99952e063d7227e38e696f34267bfa03d44b590f0490a14b
3
+ size 304713
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
3
+ size 499723
tokenizer.py ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Taken from llama code and lightly modified
2
+ # Copyright (c) Meta Platforms, Inc. and affiliates.
3
+ # This software may be used and distributed according to the terms of the Llama 2 Community License Agreement.
4
+
5
+ import os
6
+ from logging import getLogger
7
+ from typing import List
8
+
9
+ from sentencepiece import SentencePieceProcessor
10
+
11
+ TOKENIZER_MODEL = "tokenizer.model" # the llama sentencepiece tokenizer model
12
+ TOKENIZER_BIN = "tokenizer.bin" # binary version of the tokenizer for inference in C
13
+
14
+ class Tokenizer:
15
+ def __init__(self):
16
+ model_path = TOKENIZER_MODEL
17
+ assert os.path.isfile(model_path), model_path
18
+ self.sp_model = SentencePieceProcessor(model_file=model_path)
19
+ #print(f"Loaded SentencePiece model from {model_path}")
20
+
21
+ # BOS / EOS token IDs
22
+ self.n_words: int = self.sp_model.vocab_size()
23
+ self.bos_id: int = self.sp_model.bos_id()
24
+ self.eos_id: int = self.sp_model.eos_id()
25
+ self.pad_id: int = self.sp_model.pad_id()
26
+ #print(f"#words: {self.n_words} - BOS ID: {self.bos_id} - EOS ID: {self.eos_id}")
27
+ assert self.sp_model.vocab_size() == self.sp_model.get_piece_size()
28
+
29
+ def encode(self, s: str, bos: bool, eos: bool) -> List[int]:
30
+ assert type(s) is str
31
+ t = self.sp_model.encode(s)
32
+ if bos:
33
+ t = [self.bos_id] + t
34
+ if eos:
35
+ t = t + [self.eos_id]
36
+ return t
37
+
38
+ def decode(self, t: List[int]) -> str:
39
+ return self.sp_model.decode(t)
40
+
41
+ def export(self):
42
+ tokens = []
43
+ for i in range(self.n_words):
44
+
45
+ # decode the token and light postprocessing
46
+ t = self.sp_model.id_to_piece(i)
47
+ if i == self.bos_id:
48
+ t = '\n<s>\n'
49
+ elif i == self.eos_id:
50
+ t = '\n</s>\n'
51
+ elif len(t) == 6 and t.startswith('<0x') and t.endswith('>'):
52
+ t = chr(int(t[3:5], 16)) # e.g. make '<0x01>' into '\x01'
53
+ t = t.replace('▁', ' ') # sentencepiece uses this as the whitespace
54
+
55
+ tokens.append(t)
56
+
57
+ with open(TOKENIZER_BIN, 'wb') as f:
58
+ for token in tokens:
59
+ bytes = token.encode('utf-8')
60
+ f.write((len(bytes)).to_bytes(4, 'little')) # write length of bytes
61
+ f.write(bytes) # write token bytes
62
+
63
+ if __name__ == "__main__":
64
+ t = Tokenizer()
65
+ t.export()
train.py ADDED
@@ -0,0 +1,331 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ This training script can be run both on a single gpu in debug mode,
3
+ and also in a larger training run with distributed data parallel (ddp).
4
+
5
+ To run on a single GPU small debug run, example:
6
+ $ python -m train.py --compile=False --eval_iters=10 --batch_size=8
7
+
8
+ To run with DDP on 4 gpus on 1 node, example:
9
+ $ torchrun --standalone --nproc_per_node=4 train.py
10
+
11
+ To run with DDP on 4 gpus across 2 nodes, example:
12
+ - Run on the first (master) node with example IP 123.456.123.456:
13
+ $ torchrun --nproc_per_node=8 --nnodes=2 --node_rank=0 --master_addr=123.456.123.456 --master_port=1234 train.py
14
+ - Run on the worker node:
15
+ $ torchrun --nproc_per_node=8 --nnodes=2 --node_rank=1 --master_addr=123.456.123.456 --master_port=1234 train.py
16
+ (If your cluster does not have Infiniband interconnect prepend NCCL_IB_DISABLE=1)
17
+ """
18
+
19
+ import math
20
+ import os
21
+ import time
22
+ from contextlib import nullcontext
23
+ from datetime import datetime
24
+ from functools import partial
25
+
26
+ import torch
27
+ from model import Transformer, ModelArgs
28
+ from torch.distributed import destroy_process_group, init_process_group
29
+ from torch.nn.parallel import DistributedDataParallel as DDP
30
+
31
+ from tinystories import Task
32
+
33
+ # -----------------------------------------------------------------------------
34
+ # I/O
35
+ out_dir = "out"
36
+ eval_interval = 2000
37
+ log_interval = 1
38
+ eval_iters = 100
39
+ eval_only = False # if True, script exits right after the first eval
40
+ always_save_checkpoint = False # if True, always save a checkpoint after each eval
41
+ init_from = "scratch" # 'scratch' or 'resume'
42
+ # wandb logging
43
+ wandb_log = False # disabled by default
44
+ wandb_project = "llamac"
45
+ wandb_run_name = "run" + datetime.now().strftime("%Y_%m_%d_%H_%M_%S")
46
+ # data
47
+ batch_size = 128 # if gradient_accumulation_steps > 1, this is the micro-batch size
48
+ max_seq_len = 256
49
+ # model
50
+ dim = 288
51
+ n_layers = 6
52
+ n_heads = 6
53
+ multiple_of = 32
54
+ dropout = 0.0
55
+ # adamw optimizer
56
+ gradient_accumulation_steps = 4 # used to simulate larger batch sizes
57
+ learning_rate = 5e-4 # max learning rate
58
+ max_iters = 100000 # total number of training iterations
59
+ weight_decay = 1e-1
60
+ beta1 = 0.9
61
+ beta2 = 0.95
62
+ grad_clip = 1.0 # clip gradients at this value, or disable if == 0.0
63
+ # learning rate decay settings
64
+ decay_lr = True # whether to decay the learning rate
65
+ warmup_iters = 1000 # how many steps to warm up for
66
+ # system
67
+ device = "cuda" # examples: 'cpu', 'cuda', 'cuda:0', 'cuda:1' etc., or try 'mps' on macbooks
68
+ dtype = "bfloat16" # float32|bfloat16|float16
69
+ compile = True # use PyTorch 2.0 to compile the model to be faster
70
+ # -----------------------------------------------------------------------------
71
+ config_keys = [
72
+ k
73
+ for k, v in globals().items()
74
+ if not k.startswith("_") and isinstance(v, (int, float, bool, str))
75
+ ]
76
+ exec(open("configurator.py").read()) # overrides from command line or config file
77
+ config = {k: globals()[k] for k in config_keys} # will be useful for logging
78
+ # -----------------------------------------------------------------------------
79
+
80
+ # fixing some hyperparams to sensible defaults
81
+ lr_decay_iters = max_iters # should be ~= max_iters per Chinchilla
82
+ min_lr = 0.0 # minimum learning rate, should be ~= learning_rate/10 per Chinchilla
83
+
84
+ # various inits, derived attributes, I/O setup
85
+ ddp = int(os.environ.get("RANK", -1)) != -1 # is this a ddp run?
86
+ if ddp:
87
+ init_process_group(backend="nccl")
88
+ ddp_rank = int(os.environ["RANK"])
89
+ ddp_local_rank = int(os.environ["LOCAL_RANK"])
90
+ ddp_world_size = int(os.environ["WORLD_SIZE"])
91
+ device = f"cuda:{ddp_local_rank}"
92
+ torch.cuda.set_device(device)
93
+ master_process = ddp_rank == 0 # this process will do logging, checkpointing etc.
94
+ seed_offset = ddp_rank # each process gets a different seed
95
+ # world_size number of processes will be training simultaneously, so we can scale
96
+ # down the desired gradient accumulation iterations per process proportionally
97
+ assert gradient_accumulation_steps % ddp_world_size == 0
98
+ gradient_accumulation_steps //= ddp_world_size
99
+ else:
100
+ # if not ddp, we are running on a single gpu, and one process
101
+ master_process = True
102
+ seed_offset = 0
103
+ ddp_world_size = 1
104
+ tokens_per_iter = gradient_accumulation_steps * ddp_world_size * batch_size * max_seq_len
105
+ if master_process:
106
+ print(f"tokens per iteration will be: {tokens_per_iter:,}")
107
+ print(f"breaks down as: {gradient_accumulation_steps} grad accum steps * {ddp_world_size} processes * {batch_size} batch size * {max_seq_len} max seq len")
108
+
109
+ if master_process:
110
+ os.makedirs(out_dir, exist_ok=True)
111
+ torch.manual_seed(1337 + seed_offset)
112
+ torch.backends.cuda.matmul.allow_tf32 = True # allow tf32 on matmul
113
+ torch.backends.cudnn.allow_tf32 = True # allow tf32 on cudnn
114
+ device_type = "cuda" if "cuda" in device else "cpu" # for later use in torch.autocast
115
+ # note: float16 data type will automatically use a GradScaler
116
+ ptdtype = {"float32": torch.float32, "bfloat16": torch.bfloat16, "float16": torch.float16}[dtype]
117
+ ctx = (
118
+ nullcontext()
119
+ if device_type == "cpu"
120
+ else torch.amp.autocast(device_type=device_type, dtype=ptdtype)
121
+ )
122
+
123
+ # task-specific setup
124
+ iter_batches = partial(
125
+ Task.iter_batches,
126
+ batch_size=batch_size,
127
+ max_seq_len=max_seq_len,
128
+ device=device,
129
+ num_workers=0,
130
+ )
131
+
132
+ # init these up here, can override if init_from='resume' (i.e. from a checkpoint)
133
+ iter_num = 0
134
+ best_val_loss = 1e9
135
+
136
+ # model init
137
+ model_args = dict(
138
+ dim=dim,
139
+ n_layers=n_layers,
140
+ n_heads=n_heads,
141
+ n_kv_heads=n_heads,
142
+ vocab_size=32000,
143
+ multiple_of=multiple_of,
144
+ max_seq_len=max_seq_len,
145
+ #dropout=dropout,
146
+ ) # start with model_args from command line
147
+ if init_from == "scratch":
148
+ # init a new model from scratch
149
+ print("Initializing a new model from scratch")
150
+ gptconf = ModelArgs(**model_args)
151
+ model = Transformer(gptconf)
152
+ elif init_from == "resume":
153
+ print(f"Resuming training from {out_dir}")
154
+ # resume training from a checkpoint.
155
+ ckpt_path = os.path.join(out_dir, "ckpt.pt")
156
+ checkpoint = torch.load(ckpt_path, map_location=device)
157
+ checkpoint_model_args = checkpoint["model_args"]
158
+ # force these config attributes to be equal otherwise we can't even resume training
159
+ # the rest of the attributes (e.g. dropout) can stay as desired from command line
160
+ for k in ["dim", "n_layers", "n_heads", "n_kv_heads", "vocab_size", "multiple_of", "max_seq_len"]:
161
+ model_args[k] = checkpoint_model_args[k]
162
+ # create the model
163
+ gptconf = ModelArgs(**model_args)
164
+ model = Transformer(gptconf)
165
+ state_dict = checkpoint["model"]
166
+ # fix the keys of the state dictionary :(
167
+ # honestly no idea how checkpoints sometimes get this prefix, have to debug more
168
+ unwanted_prefix = "_orig_mod."
169
+ for k, v in list(state_dict.items()):
170
+ if k.startswith(unwanted_prefix):
171
+ state_dict[k[len(unwanted_prefix) :]] = state_dict.pop(k)
172
+ model.load_state_dict(state_dict)
173
+ iter_num = checkpoint["iter_num"]
174
+ best_val_loss = checkpoint["best_val_loss"]
175
+ model.to(device)
176
+
177
+ # initialize a GradScaler. If enabled=False scaler is a no-op
178
+ scaler = torch.cuda.amp.GradScaler(enabled=(dtype == "float16"))
179
+
180
+ # optimizer
181
+ optimizer = model.configure_optimizers(weight_decay, learning_rate, (beta1, beta2), device_type)
182
+ if init_from == "resume":
183
+ optimizer.load_state_dict(checkpoint["optimizer"])
184
+ checkpoint = None # free up memory
185
+
186
+ # compile the model
187
+ if compile:
188
+ print("compiling the model... (takes a ~minute)")
189
+ unoptimized_model = model
190
+ model = torch.compile(model) # requires PyTorch 2.0
191
+
192
+ # wrap model into DDP container
193
+ if ddp:
194
+ # Ignore the `freqs_cis` buffer so that DDP does not broadcast it at
195
+ # construction time since NCCL does not support `ComplexFloat`
196
+ prefix = "_orig_mod." if compile else ""
197
+ model._ddp_params_and_buffers_to_ignore = {prefix + "freqs_cis"}
198
+ model = DDP(model, device_ids=[ddp_local_rank])
199
+
200
+ # helps estimate an arbitrarily accurate loss over either split using many batches
201
+ @torch.no_grad()
202
+ def estimate_loss():
203
+ out = {}
204
+ model.eval()
205
+ for split in ["train", "val"]:
206
+ batch_iter = iter_batches(split)
207
+ losses = torch.zeros(eval_iters) # keep on CPU
208
+ for k in range(eval_iters):
209
+ X, Y = next(batch_iter)
210
+ with ctx:
211
+ logits, loss = model(X, Y)
212
+ losses[k] = loss.item()
213
+ out[split] = losses.mean()
214
+ model.train()
215
+ return out
216
+
217
+ # learning rate decay scheduler (cosine with warmup)
218
+ def get_lr(it):
219
+ # 1) linear warmup for warmup_iters steps
220
+ if it < warmup_iters:
221
+ return learning_rate * it / warmup_iters
222
+ # 2) if it > lr_decay_iters, return min learning rate
223
+ if it > lr_decay_iters:
224
+ return min_lr
225
+ # 3) in between, use cosine decay down to min learning rate
226
+ decay_ratio = (it - warmup_iters) / (lr_decay_iters - warmup_iters)
227
+ assert 0 <= decay_ratio <= 1
228
+ coeff = 0.5 * (1.0 + math.cos(math.pi * decay_ratio)) # coeff ranges 0..1
229
+ return min_lr + coeff * (learning_rate - min_lr)
230
+
231
+ # logging
232
+ if wandb_log and master_process:
233
+ import wandb
234
+ wandb.init(project=wandb_project, name=wandb_run_name, config=config)
235
+
236
+ # training loop
237
+ train_batch_iter = iter_batches("train")
238
+ X, Y = next(train_batch_iter) # fetch the very first batch
239
+ t0 = time.time()
240
+ local_iter_num = 0 # number of iterations in the lifetime of this process
241
+ raw_model = model.module if ddp else model # unwrap DDP container if needed
242
+ running_mfu = -1.0
243
+ while True:
244
+ # determine and set the learning rate for this iteration
245
+ lr = get_lr(iter_num) if decay_lr else learning_rate
246
+ for param_group in optimizer.param_groups:
247
+ param_group["lr"] = lr
248
+
249
+ # evaluate the loss on train/val sets and write checkpoints
250
+ if iter_num % eval_interval == 0 and master_process:
251
+ losses = estimate_loss()
252
+ print(f"step {iter_num}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}")
253
+ if wandb_log:
254
+ try:
255
+ wandb.log(
256
+ {
257
+ "iter": iter_num,
258
+ "tokens": iter_num * tokens_per_iter,
259
+ "loss/train": losses["train"],
260
+ "loss/val": losses["val"],
261
+ "lr": lr,
262
+ "mfu": running_mfu * 100, # convert to percentage
263
+ }
264
+ )
265
+ except Exception as e:
266
+ print(f"logging to wandb failed: {e}")
267
+ if losses["val"] < best_val_loss or always_save_checkpoint:
268
+ best_val_loss = losses["val"]
269
+ if iter_num > 0:
270
+ checkpoint = {
271
+ "model": raw_model.state_dict(),
272
+ "optimizer": optimizer.state_dict(),
273
+ "model_args": model_args,
274
+ "iter_num": iter_num,
275
+ "best_val_loss": best_val_loss,
276
+ "config": config,
277
+ }
278
+ print(f"saving checkpoint to {out_dir}")
279
+ torch.save(checkpoint, os.path.join(out_dir, "ckpt.pt"))
280
+ raw_model.export(os.path.join(out_dir, "model.bin"))
281
+ if iter_num == 0 and eval_only:
282
+ break
283
+
284
+ # forward backward update, with optional gradient accumulation to simulate larger batch size
285
+ # and using the GradScaler if data type is float16
286
+ for micro_step in range(gradient_accumulation_steps):
287
+ if ddp:
288
+ # in DDP training we only need to sync gradients at the last micro step.
289
+ # the official way to do this is with model.no_sync() context manager, but
290
+ # I really dislike that this bloats the code and forces us to repeat code
291
+ # looking at the source of that context manager, it just toggles this variable
292
+ model.require_backward_grad_sync = micro_step == gradient_accumulation_steps - 1
293
+ with ctx:
294
+ logits, loss = model(X, Y)
295
+ loss = loss / gradient_accumulation_steps
296
+ # immediately async prefetch next batch while model is doing the forward pass on the GPU
297
+ X, Y = next(train_batch_iter)
298
+ # backward pass, with gradient scaling if training in fp16
299
+ scaler.scale(loss).backward()
300
+ # clip the gradient
301
+ if grad_clip != 0.0:
302
+ scaler.unscale_(optimizer)
303
+ torch.nn.utils.clip_grad_norm_(model.parameters(), grad_clip)
304
+ # step the optimizer and scaler if training in fp16
305
+ scaler.step(optimizer)
306
+ scaler.update()
307
+ # flush the gradients as soon as we can, no need for this memory anymore
308
+ optimizer.zero_grad(set_to_none=True)
309
+
310
+ # timing and logging
311
+ t1 = time.time()
312
+ dt = t1 - t0
313
+ t0 = t1
314
+ if iter_num % log_interval == 0 and master_process:
315
+ # get loss as float, scale up due to the divide above. note: this is a CPU-GPU sync point
316
+ lossf = loss.item() * gradient_accumulation_steps
317
+ if local_iter_num >= 5: # let the training loop settle a bit
318
+ mfu = raw_model.estimate_mfu(batch_size * gradient_accumulation_steps, dt)
319
+ running_mfu = mfu if running_mfu == -1.0 else 0.9 * running_mfu + 0.1 * mfu
320
+ print(
321
+ f"{iter_num} | loss {lossf:.4f} | lr {lr:e} | {dt*1000:.2f}ms | mfu {running_mfu*100:.2f}%"
322
+ )
323
+ iter_num += 1
324
+ local_iter_num += 1
325
+
326
+ # termination conditions
327
+ if iter_num > max_iters:
328
+ break
329
+
330
+ if ddp:
331
+ destroy_process_group()