Duplicate from gbueno86/Cathallama-70B

Co-authored-by: Guilherme Bueno <gbueno86@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,145 @@
+---
+license: llama3.1
+language:
+- en
+library_name: transformers
+tags:
+- mergekit
+- merge
+base_model:
+- meta-llama/Meta-Llama-3.1-70B-Instruct
+- turboderp/Cat-Llama-3-70B-instruct
+- Nexusflow/Athene-70B
+---
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/649dc85249ae3a68334adcc6/KxaiZ7rDKkYlix99O9j5H.png)
+
+**Cathallama**
+=====================================
+
+Awesome model, my new daily driver.
+
+Edit: I am seeing a lot of token generations pointing to unknown unicode addresses that didn't show up during testing for this model, so I have stopped using it and I am working on a new version.
+
+**Notable Performance**
+
+* 9% overall success rate increase on MMLU-PRO over LLaMA 3.1 70b at Q4_0
+* Strong performance in MMLU-PRO categories overall
+* Great performance during manual testing
+
+**Creation workflow**
+=====================
+**Models merged**
+* meta-llama/Meta-Llama-3.1-70B-Instruct
+* turboderp/Cat-Llama-3-70B-instruct
+* Nexusflow/Athene-70B
+
+```
+flowchart TD
+    A[Nexusflow_Athene] -->|Merge with| B[Meta-Llama-3.1]
+    C[turboderp_Cat] -->|Merge with| D[Meta-Llama-3.1]
+    B -->| | E[Merge]
+    D -->| | E[Merge]
+    E[Merge] -->|Result| F[Cathallama]
+```
+
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/649dc85249ae3a68334adcc6/bBcB194tAtsZjPUnI1pDQ.png)
+
+**Testing**
+=====================
+
+**Hyperparameters**
+---------------
+
+* **Temperature**: 0.0 for automated, 0.9 for manual
+* **Penalize repeat sequence**: 1.05
+* **Consider N tokens for penalize**: 256
+* **Penalize repetition of newlines**
+* **Top-K sampling**: 40
+* **Top-P sampling**: 0.95
+* **Min-P sampling**: 0.05
+
+**LLaMAcpp Version**
+------------------
+
+* b3527-2-g2d5dd7bb
+* -fa -ngl -1 -ctk f16 --no-mmap
+
+**Tested Files**
+------------------
+
+* Cathallama-70B.Q4_0.gguf
+* Nexusflow_Athene-70B.Q4_0.gguf
+* turboderp_Cat-Llama-3-70B-instruct.Q4_0.gguf
+* Meta-Llama-3.1-70B-Instruct.Q4_0.gguf
+
+**Tests**
+--------------
+
+
+**Manual testing**
+
+| Category | Test Case | Cathallama-70B.Q4_0.gguf | Nexusflow_Athene-70B.Q4_0.gguf | turboderp_Cat-Llama-3-70B-instruct.Q4_0.gguf | Meta-Llama-3.1-70B-Instruct.Q4_0.gguf |
+| --- | --- | --- | --- | --- | --- |
+| **Common Sense** | Ball on cup | OK | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> | OK |
+|  | Big duck small horse | <span style="color: red;">KO</span> | OK | <span style="color: red;">KO</span> | OK |
+|  | Killers | OK | OK | <span style="color: red;">KO</span> | OK |
+|  | Strawberry r's | OK | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> |
+|  | 9.11 or 9.9 bigger | <span style="color: red;">KO</span> | OK | OK | <span style="color: red;">KO</span> |
+|  | Dragon or lens | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> |
+|  | Shirts | OK | OK | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> |
+|  | Sisters | OK | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> |
+|  | Jane faster | OK | OK | OK | OK |
+| **Programming** | JSON | OK | OK | OK | OK |
+|  | Python snake game | OK | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> |
+| **Math** | Door window combination | OK | OK | <span style="color: red;">KO</span> | <span style="color: red;">KO</span> |
+| **Smoke** | Poem | OK | OK | OK | OK |
+|  | Story | OK | OK | KO | OK |
+
+*Note: See [sample_generations.txt](https://huggingface.co/gbueno86/Cathallama-70B/blob/main/sample_generations.txt) on the main folder of the repo for the raw generations.*
+
+**MMLU-PRO**
+
+| Model | Success % |
+| --- | --- |
+| Cathallama-70B.Q4_0.gguf | **51.0%** |
+| turboderp_Cat-Llama-3-70B-instruct.Q4_0.gguf | 37.0% |
+| Nexusflow_Athene-70B.Q4_0.gguf | 41.0% |
+| Meta-Llama-3.1-70B-Instruct.Q4_0.gguf | 42.0% |
+
+| MMLU-PRO category| Cathallama-70B.Q4_0.gguf | Nexusflow_Athene-70B.Q4_0.gguf | turboderp_Cat-Llama-3-70B-instruct.Q4_0.gguf | Meta-Llama-3.1-70B-Instruct.Q4_0.gguf |
+| --- | --- | --- | --- | --- |
+| Business | **50.0%** | 45.0% | 20.0% | 40.0% |
+| Law | **40.0%** | 30.0% | 30.0% | 35.0% |
+| Psychology | **85.0%** | 80.0% | 70.0% | 75.0% |
+| Biology | 80.0% | 70.0% | **85.0%** | 80.0% |
+| Chemistry | **55.0%** | 40.0% | 35.0% | 35.0% |
+| History | **65.0%** | 60.0% | 55.0% | **65.0%** |
+| Other | **55.0%** | 50.0% | 45.0% | 50.0% |
+| Health | **75.0%** | 40.0% | 60.0% | 65.0% |
+| Economics | **80.0%** | 75.0% | 65.0% | 70.0% |
+| Math | **45.0%** | 35.0% | 15.0% | 40.0% |
+| Physics | **50.0%** | 45.0% | 45.0% | 45.0% |
+| Computer Science | **60.0%** | 55.0% | 55.0% | **60.0%** |
+| Philosophy | 55.0% | **60.0%** | 45.0% | 50.0% |
+| Engineering | 35.0% | **40.0%** | 25.0% | 35.0% |
+
+*Note: MMLU-PRO Overall tested with 100 questions. Categories testes with 20 questions from each category.*
+
+**PubmedQA**
+
+ Model Name | Success% |
+| --- | --- |
+| Cathallama-70B.Q4_0.gguf| 73.00% |
+| turboderp_Cat-Llama-3-70B-instruct.Q4_0.gguf | **76.00%** |
+| Nexusflow_Athene-70B.Q4_0.gguf | 67.00% |
+| Meta-Llama-3.1-70B-Instruct.Q4_0.gguf | 72.00% |
+
+
+**Request**
+--------------
+If you are hiring in the EU or can sponsor a visa, PM me :D
+
+
+PS. Thank you mradermacher for the GGUFs!

config.json

@@ -0,0 +1,29 @@
+{
+  "_name_or_path": "/Volumes/external2/models/Cat-a-Llama-v2",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 128000,
+  "eos_token_id": 128001,
+  "hidden_act": "silu",
+  "hidden_size": 8192,
+  "initializer_range": 0.02,
+  "intermediate_size": 28672,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 64,
+  "num_hidden_layers": 80,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.43.3",
+  "use_cache": true,
+  "vocab_size": 128256
+}

llama.log

@@ -0,0 +1 @@
+[1723093853] warming up the model with an empty run

mergekit_config.yml

@@ -0,0 +1,16 @@
+base_model: /Volumes/external2/models/Cat-a-Llama-v2
+dtype: bfloat16
+merge_method: slerp
+parameters:
+  t:
+  - filter: self_attn
+    value: [0.0, 0.5, 0.3, 0.7, 1.0]
+  - filter: mlp
+    value: [1.0, 0.5, 0.7, 0.3, 0.0]
+  - value: 0.5
+slices:
+- sources:
+  - layer_range: [0, 80]
+    model: /Volumes/external2/models/Cat-a-Llama-v2
+  - layer_range: [0, 80]
+    model: /Volumes/external2/models/Athallama-3.1-70B

model-00001-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:772833f59582abd0c83dbe984e123fe28324bb383e8b4565067eae5dbab04273
+size 2101346432

model-00002-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:485662acd4cb0fada549c0981aa85878edb4da34ff2e92fd5b65bb0936a4c1c1
+size 2101346448

model-00003-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bc2c18b7381238928686993bdb4506585796ef685316f9fc75d0442e65a6718
+size 939540840

model-00004-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0feb82b1811ccddb3a027aa021191ca9e9fc9d81c38ef7370b7a97242b03922
+size 771785536

model-00005-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0020ab57a1b97102081afbe580c807cc2ea7f636f9790ab5de625a4579dd2262
+size 939524360

model-00006-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7ca1b69e20040dcf802afd7da12a1833439e053285c4152ab4f5f6e13dded7c
+size 771785536

model-00007-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd81c985e6377a806ad58b2a2788f64169c3d5782ddb50672cffba411074ddda
+size 939524360

model-00008-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9894f85af1146934b0b2611daea494639a762b30ece46a9e09b8a8eafb895e34
+size 771785544

model-00009-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:668d7e6ee945ab849df4a91166e86c46c6f2e2b54c9f785e19d70283f0c47096
+size 939524360

model-00010-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:200ea9d30b4749c2d097ced4c74e4e04058f85a58e25904b9dbb6b08102cd85d
+size 771785544

model-00011-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66746fd7962854951590d0b3bd7ac37ab7328dbc93fdef9aa84c437b1bbc9593
+size 939524360

model-00012-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3532fea014e968c303c066593ffe2b0633c2504524bf9be4b1c1b2b7ff17c2fb
+size 771785544

model-00013-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3709ce0c95a466389f400c4729ecd04572d119924ce481ad6f5f88651cc5842c
+size 939524360

model-00014-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9003182803853d889fde3ab1193986dd811e5b33cb43f6cddc2b430f2f3148e
+size 771785544

model-00015-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8dbdbde3f7acde05cffb5b0f2a4752095ae7a977fab69ae0c4ea15080be2c4f4
+size 939524360

model-00016-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:449c0d73c38cc96af8200570ff4f4828dc1e98a81c34ba5f09fcadaccdbd08f3
+size 771785544

model-00017-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91280d5b556cbe7de7a487d98732f9b98c6d0bea4c28eb70c0cb182969babcd1
+size 939524360

model-00018-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46534e6d48a33ff635d9959e08f30e6af978207f179f16f5fb17d5108a7cff05
+size 771785544

model-00019-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd63072c726b4134aed48bd9204b8067a2773abb091d70389de421f94cf0e9e0
+size 939524360

model-00020-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19ea8969532cc93025c319946699d6ae084e1d9ebcdd23fa132f5e4c467bcf3f
+size 771785544

model-00021-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33075ce0f389d87ecc61e7278f6841141e3e697ba2c0b5d63a4e34122f71b909
+size 939524360

model-00022-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aba989c87830290c515599993ae04232780a09eaae07ec4c8d677688a203917a
+size 771785544

model-00023-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee0ee0bb4aa30fb43bdc8f40872449a9478e9a33f4c55f993883e2da2288db1d
+size 939524360

model-00024-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d265467d5401d17ee05d8535ca433052ca6ad08f6b0103231dcfaa94e1597d1d
+size 771785544

model-00025-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ea1e90e45e681af8bfd964d7895cc4603f5decbad83b565d3664b9546881e31
+size 939524360

model-00026-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f2dc82110a3369f6f0416001b8af13855491efb53bea2ed0e3f08b11766faf33
+size 771785544

model-00027-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e83c574ee6441b7c0c2db27102bd416a748f13ce3e1d0c58513ab7e60f35ee6a
+size 939524360

model-00028-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0dc265b68be106652bb9b860f71b44779ba6bf9bda085fd61381efeae293363f
+size 771785536

model-00029-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0598a7b0a207b0d7e04961342d8124d7e543505f773f85d4a0c98588f96d90df
+size 939524360

model-00030-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e7f31ea3778a658626bf336ef838a0ea89b094044c9077af6e866a6d856a617
+size 771785544

model-00031-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ca94309b7de61726d210bcbe21d1113347d7479f4920910d6e06d50001ab271
+size 939524360

model-00032-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b32b78307130d5b038c43710a5a2ef8222adb00030d03469604902a1e6c6967a
+size 771785544

model-00033-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98be568e311fa34263cedf2716cf4b1f5e0b9a8d1dc74e8bba7635f10d66c170
+size 939524360

model-00034-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17eebf3515cb0ba8dd2ff128c97f8f574aa0d2c6dd7403e52e6bcb0e11f97780
+size 771785544

model-00035-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bbec538d68c8683299f8d294a74f667d940c947be45784e3d91ac389c3dde32
+size 939524360

model-00036-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d39cc570727c0b882c1456e8951a9075dd02fe92172b47d259063c4c4b93654
+size 771785544

model-00037-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91819e85d5848d87b1e9232d1c93b057d2ac7e975f0dadf3a99a7d42d889b43a
+size 939524360

model-00038-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70bbe31f5cf0e6b53e6cbc1eefd40615162c4a406e7843a5cd2a46169e84f0e0
+size 771785544

model-00039-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e49ccf29c8469a0afd5bd264387399f16d998ab1fbdf67d0c0ba324912e2f118
+size 939524360

model-00040-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cdecf3778464c2895c6ad5a90768e38110f37cf45685f91a83d44a180857bf51
+size 771785544

model-00041-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:435180d5c4cac7d6c02c366d80cfcf7257a3d87a7b80e4a1b3dcf1aa123cd995
+size 939524360

model-00042-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fdd4e95a12e079ce4016cbb2ac9590d4be5f7139c19c0cdd2630f2846f1d47bc
+size 771785544

model-00043-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45ca2da2cc80b560578da1604444543b4da13b54c5ba0a2fc627c78306c16235
+size 939524360

model-00044-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67aae86c7b0b2421a0f1d3039302bd283beb21df878ca26b276d0279b46532d6
+size 771785544

model-00045-of-00162.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a611b643f37f0dbd8ad41679e59bfead3971696e57785f387e217ca729b71242
+size 939524360