Upload folder using huggingface_hub

README.md

@@ -0,0 +1,72 @@
+---
+license: apache-2.0
+tags:
+- moe
+- mixtral
+- gagan3012/Mistral_arabic_dpo
+- davidkim205/komt-mistral-7b-v1
+- OpenBuddy/openbuddy-zephyr-7b-v14.1
+- manishiitg/open-aditi-hi-v1
+---
+
+# Multilingual-mistral-asian
+
+This model is a Mixure of Experts (MoE) made with [mergekit](https://github.com/cg123/mergekit) (mixtral branch). It uses the following base models:
+* [gagan3012/Mistral_arabic_dpo](https://huggingface.co/gagan3012/Mistral_arabic_dpo)
+* [davidkim205/komt-mistral-7b-v1](https://huggingface.co/davidkim205/komt-mistral-7b-v1)
+* [OpenBuddy/openbuddy-zephyr-7b-v14.1](https://huggingface.co/OpenBuddy/openbuddy-zephyr-7b-v14.1)
+* [manishiitg/open-aditi-hi-v1](https://huggingface.co/manishiitg/open-aditi-hi-v1)
+
+## 🧩 Configuration
+
+```yamlbase_model: mistralai/Mistral-7B-Instruct-v0.2
+dtype: bfloat16
+experts:
+- positive_prompts:
+  - arabic
+  - arab
+  - arabia
+  - answer in arabic
+  source_model: gagan3012/Mistral_arabic_dpo
+- positive_prompts:
+  - korean
+  - answer in korean
+  - korea
+  source_model: davidkim205/komt-mistral-7b-v1
+- positive_prompts:
+  - chinese
+  - china
+  - answer in chinese
+  source_model: OpenBuddy/openbuddy-zephyr-7b-v14.1
+- positive_prompts:
+  - hindi
+  - india
+  - hindu
+  - answer in hindi
+  source_model: manishiitg/open-aditi-hi-v1
+gate_mode: hidden
+```
+
+## 💻 Usage
+
+```python
+!pip install -qU transformers bitsandbytes accelerate
+
+from transformers import AutoTokenizer
+import transformers
+import torch
+
+model = "gagan3012/Multilingual-mistral-asian"
+
+tokenizer = AutoTokenizer.from_pretrained(model)
+pipeline = transformers.pipeline(
+    "text-generation",
+    model=model,
+    model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
+)
+
+messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
+prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
+print(outputs[0]["generated_text"])
+```

config.json

@@ -0,0 +1,30 @@
+{
+  "_name_or_path": "mistralai/Mistral-7B-Instruct-v0.2",
+  "architectures": [
+    "MixtralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "model_type": "mixtral",
+  "num_attention_heads": 32,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "num_local_experts": 4,
+  "output_router_logits": false,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 1000000.0,
+  "router_aux_loss_coef": 0.001,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.36.2",
+  "use_cache": true,
+  "vocab_size": 32000
+}

model-1.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1ab1e84474c0d342c2df483137067e03138fc207460463551daf2d3329e4018
+size 9919813704

model-2.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:acb360354ac66a9dad0e3736f0a59df4565b93debf6ae799c657f160c0c8448a
+size 9982454720

model-3.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eab6fac246c7d7efa46599dd5cc0352efa86af38906c2b12a44600c53909f2f8
+size 9982454752

model-4.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac63e34c632f9d04d0de500c71735db22bebb560cc7cff58f32420d5ceb851af
+size 9982454720