|
--- |
|
license: apache-2.0 |
|
language: |
|
- fr |
|
- it |
|
- de |
|
- es |
|
- en |
|
inference: false |
|
--- |
|
# Model Card for Mixtral-Fusion-4x7B-Instruct-v0.1 |
|
This model is an experimental model created by merging [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) experts. |
|
|
|
# How we merged experts |
|
We simply take the average of every two experts.weight. |
|
The same goes for gate.weight. |
|
**Unfortunately, this model has a large hallucination. Look extraction version. -> [mmnga/Mixtral-Extraction-4x7B-Instruct-v0.1](https://huggingface.co/mmnga/Mixtral-Extraction-4x7B-Instruct-v0.1)** |
|
|
|
# How To Convert |
|
use colab cpu-high-memory. |
|
[convert_mixtral_8x7b_to_4x7b.ipynb](https://huggingface.co/mmnga/Mixtral-Fusion-4x7B-Instruct-v0.1/blob/main/notebook/convert_mixtral_8x7b_to_4x7b.ipynb) |
|
|
|
# Usage |
|
~~~python |
|
pip install git+https://github.com/huggingface/transformers --upgrade |
|
pip install torch accelerate bitsandbytes flash_attn |
|
~~~ |
|
|
|
~~~python |
|
from transformers import AutoTokenizer, AutoModelForCausalLM, MixtralForCausalLM |
|
import torch |
|
|
|
model_name_or_path = "mmnga/Mixtral-Fusion-4x7B-Instruct-v0.1" |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path) |
|
model = MixtralForCausalLM.from_pretrained(model_name_or_path, load_in_8bit=True) |
|
|
|
text = "Tell me what's for dinner tonight. " |
|
inputs = tokenizer(text, return_tensors="pt") |
|
|
|
outputs = model.generate(**inputs, max_new_tokens=128) |
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
|
~~~ |
|
|
|
|
