metadata

base_model:
  - yuvraj17/Llama-3-8B-spectrum-25
  - ruggsea/Llama3-stanford-encyclopedia-philosophy-QA
  - arcee-ai/Llama-3.1-SuperNova-Lite
tags:
  - merge
  - mergekit
  - lazymergekit
  - yuvraj17/Llama-3-8B-spectrum-25
  - ruggsea/Llama3-stanford-encyclopedia-philosophy-QA
  - arcee-ai/Llama-3.1-SuperNova-Lite
license: apache-2.0
language:
  - en
pipeline_tag: text-classification

Llama3-8B-SuperNova-Spectrum-dare_ties

Llama3-8B-SuperNova-Spectrum-dare_ties is a DARE_TIES merge of the following models using LazyMergekit:

DARE_TIES Merging

TIES Merging

TIES Merging, introduced by Yadav et al. (2023), is a method for merging multiple specialized models into one general-purpose model. It solves two key challenges:

Redundancy Removal: Identifies and eliminates overlapping or unnecessary information between models, making the final model more efficient.
Conflict Resolution: Reconciles differences between models by creating a unified sign vector that represents the most dominant direction of change across all models.

DARE Merging

Introduced by Yu et al. (2023), DARE uses an approach similar to TIES with two main differences:

Weight Pruning: Randomly resets some fine-tuned weights to their original values, reducing model complexity.
Weight Scaling: Adjusts the remaining weights by scaling and combining them with the base model's weights to maintain consistent performance.

Mergekit’s implementation of this method has two flavours: with the sign election step of TIES (dare_ties) or without (dare_linear).

For more information refer this Merge Large Language Models with MergeKit by Maxime Labonne

🧩 Configuration

models:
  - model: NousResearch/Meta-Llama-3-8B
    # No parameters necessary for base model
  - model: yuvraj17/Llama-3-8B-spectrum-25
    parameters:
      density: 0.56
      weight: 0.12
  - model: ruggsea/Llama3-stanford-encyclopedia-philosophy-QA
    parameters:
      density: 0.56
      weight: 0.12
  - model: arcee-ai/Llama-3.1-SuperNova-Lite
    parameters:
      density: 0.58
      weight: 0.55
merge_method: dare_ties
base_model: NousResearch/Meta-Llama-3-8B
dtype: bfloat16

💻 Usage

!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "yuvraj17/Llama3-8B-SuperNova-Spectrum-dare_ties"
messages = [{"role": "user", "content": "What is a large language model?"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

🏆 Evaluation Scores

Coming soon

Special thanks & Reference

Maxime Labonne for their easy-to-use colab-notebook Merging LLMs with MergeKit and Blog
Authors of Mergekit