Hugging Face's logo

softwareweaver
/
Mistral-Large-Extra

Text Generation
Transformers
Safetensors
mistral
mergekit
Merge
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
softwareweaver commited on
Commit
5494636
•
1 Parent(s): 96ae9a2

Upload folder using huggingface_hub

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +41 -0
  2. config.json +27 -0
  3. mergekit_config.yml +9 -0
  4. model-00001-of-00081.safetensors +3 -0
  5. model-00002-of-00081.safetensors +3 -0
  6. model-00003-of-00081.safetensors +3 -0
  7. model-00004-of-00081.safetensors +3 -0
  8. model-00005-of-00081.safetensors +3 -0
  9. model-00006-of-00081.safetensors +3 -0
  10. model-00007-of-00081.safetensors +3 -0
  11. model-00008-of-00081.safetensors +3 -0
  12. model-00009-of-00081.safetensors +3 -0
  13. model-00010-of-00081.safetensors +3 -0
  14. model-00011-of-00081.safetensors +3 -0
  15. model-00012-of-00081.safetensors +3 -0
  16. model-00013-of-00081.safetensors +3 -0
  17. model-00014-of-00081.safetensors +3 -0
  18. model-00015-of-00081.safetensors +3 -0
  19. model-00016-of-00081.safetensors +3 -0
  20. model-00017-of-00081.safetensors +3 -0
  21. model-00018-of-00081.safetensors +3 -0
  22. model-00019-of-00081.safetensors +3 -0
  23. model-00020-of-00081.safetensors +3 -0
  24. model-00021-of-00081.safetensors +3 -0
  25. model-00022-of-00081.safetensors +3 -0
  26. model-00023-of-00081.safetensors +3 -0
  27. model-00024-of-00081.safetensors +3 -0
  28. model-00025-of-00081.safetensors +3 -0
  29. model-00026-of-00081.safetensors +3 -0
  30. model-00027-of-00081.safetensors +3 -0
  31. model-00028-of-00081.safetensors +3 -0
  32. model-00029-of-00081.safetensors +3 -0
  33. model-00030-of-00081.safetensors +3 -0
  34. model-00031-of-00081.safetensors +3 -0
  35. model-00032-of-00081.safetensors +3 -0
  36. model-00033-of-00081.safetensors +3 -0
  37. model-00034-of-00081.safetensors +3 -0
  38. model-00035-of-00081.safetensors +3 -0
  39. model-00036-of-00081.safetensors +3 -0
  40. model-00037-of-00081.safetensors +3 -0
  41. model-00038-of-00081.safetensors +3 -0
  42. model-00039-of-00081.safetensors +3 -0
  43. model-00040-of-00081.safetensors +3 -0
  44. model-00041-of-00081.safetensors +3 -0
  45. model-00042-of-00081.safetensors +3 -0
  46. model-00043-of-00081.safetensors +3 -0
  47. model-00044-of-00081.safetensors +3 -0
  48. model-00045-of-00081.safetensors +3 -0
  49. model-00046-of-00081.safetensors +3 -0
  50. model-00047-of-00081.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model:
3
+ - mistralai/Mistral-Large-Instruct-2407
4
+ library_name: transformers
5
+ tags:
6
+ - mergekit
7
+ - merge
8
+
9
+ ---
10
+ # Mistral-Large-Extra
11
+
12
+ Experimental Merge to verify if bigger models are better.
13
+ Great for Creative Writing and other AI questions.
14
+
15
+ This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
16
+
17
+ ## Merge Details
18
+ ### Merge Method
19
+
20
+ This model was merged using the passthrough merge method.
21
+
22
+ ### Models Merged
23
+
24
+ The following models were included in the merge:
25
+ * [mistralai/Mistral-Large-Instruct-2407](https://huggingface.co/mistralai/Mistral-Large-Instruct-2407)
26
+
27
+ ### Configuration
28
+
29
+ The following YAML configuration was used to produce this model:
30
+
31
+ ```yaml
32
+ dtype: bfloat16
33
+ merge_method: passthrough
34
+ slices:
35
+ - sources:
36
+ - layer_range: [0, 70]
37
+ model: mistralai/Mistral-Large-Instruct-2407
38
+ - sources:
39
+ - layer_range: [17, 87]
40
+ model: mistralai/Mistral-Large-Instruct-2407
41
+ ```
config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "mistralai/Mistral-Large-Instruct-2407",
3
+ "architectures": [
4
+ "MistralForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "head_dim": 128,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 12288,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 28672,
14
+ "max_position_embeddings": 131072,
15
+ "model_type": "mistral",
16
+ "num_attention_heads": 96,
17
+ "num_hidden_layers": 140,
18
+ "num_key_value_heads": 8,
19
+ "rms_norm_eps": 1e-05,
20
+ "rope_theta": 1000000.0,
21
+ "sliding_window": null,
22
+ "tie_word_embeddings": false,
23
+ "torch_dtype": "bfloat16",
24
+ "transformers_version": "4.44.2",
25
+ "use_cache": true,
26
+ "vocab_size": 32768
27
+ }
mergekit_config.yml ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ dtype: bfloat16
2
+ merge_method: passthrough
3
+ slices:
4
+ - sources:
5
+ - layer_range: [0, 70]
6
+ model: mistralai/Mistral-Large-Instruct-2407
7
+ - sources:
8
+ - layer_range: [17, 87]
9
+ model: mistralai/Mistral-Large-Instruct-2407
model-00001-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1fd03ff284e4324bd3a31511d1bbaab644cc30dff53db64c37acbcac7f91769c
3
+ size 4378928504
model-00002-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5b95ba9950ba6d154a2d4d8379a7b0a487770b4d1b0809f1fd76f554a58fec90
3
+ size 4907411088
model-00003-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f723328d474667d0128c4dae515e96bc354978ec5e5e0a131e0e2144a7205cdd
3
+ size 4806747904
model-00004-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5df0b636a686027a11922d0a5265982869dd1a6d99dd216e926e719e778046d0
3
+ size 4831938544
model-00005-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:26079a878ffb3be350f2f3be6dcc2b3c8b94a809fbc1055878d6bb16ea5e38cd
3
+ size 4831938552
model-00006-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ef67c44f4b550bb9b01184f0a13b13d3e276021bd49a33b0328e0ff218392fa5
3
+ size 4882245144
model-00007-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4091f23c312b0c811c18d6951662a80bafa709c2106c3a6b23661e016d28d547
3
+ size 4831938552
model-00008-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9f21043cceda33555cd70ab071447b1a2f95c4cc963576e8459c83957c00fd89
3
+ size 4831938552
model-00009-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2c6fd12dae8dcef95c0c493b6bcfe39387f5a3a4becf6741c27b74c7742c2b5c
3
+ size 4831913864
model-00010-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3e71ed274847e2314befd072d8cdf015d66ce9fac35052c00aa0619248c3691a
3
+ size 4882245136
model-00011-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:932ee5be804d2fe6f8ffc1ddcc438a6bb78e2f8bf20fe2ba5063825140824579
3
+ size 4831938552
model-00012-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:629f6d8a57282334645816cd1c850cb41d9d43bec621011dcea12035540b21dc
3
+ size 4831938552
model-00013-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4ef2a18a5aaeea448c27a6a458862c17b5901c58f10d6f8356d03be66c9bb452
3
+ size 4831938552
model-00014-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:40b7c54afa344a3bdaf7ce66ca3bf7e0552bbad5bd9215f341ed87bd07331235
3
+ size 4882220464
model-00015-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:36602c402f3d744c68f9bb7f0cb77ac06695c32ba81b7a33e674703514628561
3
+ size 4932601720
model-00016-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0937923ce304c1159b5b37acc4f7e8bc5b17ef3834c5bca06e19ee53348ffe50
3
+ size 4781557264
model-00017-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:96c8460ce039feb67aa1adde3122b4e843907cdd125e654075dc76e0fd0d0d5d
3
+ size 4831938552
model-00018-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4bfdead35ecf6626e79e2210943947b5f7309e7922f989057f79ce3babc69731
3
+ size 4831938544
model-00019-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca489d1b0475ca389a8464990b2d6b2022de3be1c9a4dc59636c7d0ce2bfc1a8
3
+ size 4831938552
model-00020-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ddeb5704efb932a02ee363f3693beb6c27b1aa58ff1bb9492b0a8885c5bf9d94
3
+ size 4831938552
model-00021-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a03dfd3456271e9c701f2891017ad1e4fa8605c8d67d42a0ed56469707191e5b
3
+ size 4831913864
model-00022-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:57b31dc56a169dce2304c5037f3190fbb14b36e70220c7d43fbd7e4f387f7ea8
3
+ size 4882245136
model-00023-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8bb33b41140ad0b5cb80c4cb52050bc84f147fe5b05ae920f47a1985e7c1f8ed
3
+ size 4831938552
model-00024-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:065debd9edaeba44788139829ae367c81ac3ec997c522a42f09955d1260c1934
3
+ size 4831938552
model-00025-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d7d954e049e6137722cd97ac4d9a9e4699dbab644f3aaae0c6d2ac142d055aa3
3
+ size 4831938552
model-00026-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f735183710b8998f78111020609043551e8ace66564ba2c1f5017673c6d9998c
3
+ size 4882220464
model-00027-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3964a0b1dd93a8e6d339d7995a1263597d7ddbc5275c6932cad64ad99ea2c6a4
3
+ size 4932601720
model-00028-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e923a70f786102ee7f09f4e40b51f37e5bab1bc4af8ff6a923ece16a19544839
3
+ size 4781557264
model-00029-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2d9fbceda7aacfe24e7e8c79cd30e1f7c7796cc5170e076662f76243ade4509d
3
+ size 4831938552
model-00030-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c77888c64d1a4083ed6f7889240c989e467fd1f011ad0b738bd7dfddb289211c
3
+ size 4831938544
model-00031-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c72a4f46cb5f04d0b1762278a74c7d915e2c1ebf04c44ef0091f19372ae60d18
3
+ size 4831938552
model-00032-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5da099df90a848cf2745719c9b80dde7c6af4afb296c0013da41911af47b8bee
3
+ size 4831938552
model-00033-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c214b8ab3f4a63acf2599c09704fe5634607f6c05e3817e9eebc8490dde9ddf2
3
+ size 4831913864
model-00034-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:632b1134f2ba28096d04d13cba73f57554a7d5d72d01277bd606c9de07d514ce
3
+ size 4882245136
model-00035-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8470d281fcb0e5e2414b0e392cb037be84e974816bc271a2e9dd0df02070213b
3
+ size 4831938552
model-00036-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3348acfcc5544d4d2dad8470a56c35242564988d9851add05bdfec41d77c25fb
3
+ size 4831938552
model-00037-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3060def9d86b603b739e299e2c57ca7cd3333ff321cb44b27d4aed5dc8cf7772
3
+ size 4831938552
model-00038-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:433f9364f1ed807ccf2945c4cd40d2c9e761a69aa3250f4d1de94e6046501211
3
+ size 4882220464
model-00039-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84edd9d506b39457d76bbddc367d287fd3479e91ac396a5865ea0c9775eaed30
3
+ size 4932601720
model-00040-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d49f3d01fc88f858714b6cb092a0c341da4a1c4db916d4f0815019271f284f62
3
+ size 4781557264
model-00041-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b8ef9617a23df660088013f8a516e2a7f12c8653ec897fba5bb53291f84cfcd7
3
+ size 4831938552
model-00042-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:64e20578d893635735c00341342c5b3c5b75cc5f5b7ba3af9a8c35a1aca13bd7
3
+ size 4831938544
model-00043-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5d16c2c8d6c11103cacd87c8d274b727e0f1c533155182feaa4a928397af1148
3
+ size 4831938552
model-00044-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:497fcf057c88bb0d5e11e2cf2db75a36fd4f67f630ec044ededb7f427f7982af
3
+ size 4831938560
model-00045-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9ef63e6dd229302d2b74aa58d4a180924c4abe61dbca20c124184e3572d4a915
3
+ size 4831913872
model-00046-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3d8bb1f15c2093a9f4b29849adae4649f1e397a906ad7cd00c418af4e7a1d427
3
+ size 4882245128
model-00047-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:07f28de42ab5fa6bde65d4e74ff02de1e07aefeffe1a8c70b1f610c500e40b14
3
+ size 4831938552