Mixllama3-8x8b-Instruct-v0.1 based on LLaMA 3

An experimental MoE (Mixture of Experts) model based on the LLaMA-3-8B. MixLLaMA3-8x8b combines 8 fine-tuned LLaMA 8B models, each specialized in a specific set of tasks. By leveraging the strengths of each expert model, Mixllama3-8x8b aims to deliver enhanced performance and adaptability across a wide range of applications.

Disclaimer

This model is a research experiment and may generate incorrect or harmful content. The model's outputs should not be taken as factual or representative of the views of the model's creator or any other individual.

The model's creator is not responsible for any harm or damage caused by the model's outputs.

Merge Details

base_model: meta-llama/Meta-Llama-3-8B-Instruct
experts:
  - source_model: meta-llama/Meta-Llama-3-8B-Instruct
    positive_prompts:
    - "assistant"
  - source_model: Muhammad2003/Llama3-8B-OpenHermes-DPO
    positive_prompts:
    - "python"
  - source_model: cognitivecomputations/dolphin-2.9-llama3-8b
    positive_prompts:
    - "chat"
  - source_model: orpo-explorers/hf-llama3-8b-orpo-v0.1.4
    positive_prompts:
    - "code"
  - source_model: Locutusque/llama-3-neural-chat-v1-8b
    positive_prompts:
    - "math"
  - source_model: mlabonne/Llama-3-SLERP-8B
    positive_prompts:
    - "AI"
  - source_model: meta-llama/Meta-Llama-3-8B
    positive_prompts:
    - "explain"
  - source_model: dreamgen/opus-v1.2-llama-3-8b
    positive_prompts:
    - "Role playing"
gate_mode: cheap_embed
dtype: float16