File size: 8,541 Bytes
5e98f64 bc6cbd8 0e081d5 5e98f64 ccdfe0e 5e98f64 bc6cbd8 ccdfe0e bc6cbd8 ccdfe0e bc6cbd8 ccdfe0e bc6cbd8 5e98f64 bc6cbd8 998d15b bc6cbd8 998d15b 4534a6e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 |
---
base_model:
- yuvraj17/Llama-3-8B-spectrum-25
- ruggsea/Llama3-stanford-encyclopedia-philosophy-QA
- arcee-ai/Llama-3.1-SuperNova-Lite
tags:
- merge
- mergekit
- lazymergekit
- yuvraj17/Llama-3-8B-spectrum-25
- ruggsea/Llama3-stanford-encyclopedia-philosophy-QA
- arcee-ai/Llama-3.1-SuperNova-Lite
license: apache-2.0
language:
- en
pipeline_tag: text-generation
---
# Llama3-8B-SuperNova-Spectrum-dare_ties
Llama3-8B-SuperNova-Spectrum-dare_ties is a `dare_ties` merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
* [yuvraj17/Llama-3-8B-spectrum-25](https://huggingface.co/yuvraj17/Llama-3-8B-spectrum-25)
* [ruggsea/Llama3-stanford-encyclopedia-philosophy-QA](https://huggingface.co/ruggsea/Llama3-stanford-encyclopedia-philosophy-QA)
* [arcee-ai/Llama-3.1-SuperNova-Lite](https://huggingface.co/arcee-ai/Llama-3.1-SuperNova-Lite)
## DARE_TIES Merging
### TIES Merging
[TIES](https://arxiv.org/abs/2306.01708) Merging, introduced by Yadav et al. (2023), is a method for merging multiple specialized models into one general-purpose model. It solves two key challenges:
* **Redundancy Removal**: Identifies and eliminates overlapping or unnecessary information between models, making the final model more efficient.
* **Conflict Resolution**: Reconciles differences between models by creating a unified sign vector that represents the most dominant direction of change across all models.
**TRIES** stands for **T**R**I**M, **E**LECT **S**IGN & MERGE (TIES-MERGING).
<figure>
<img src="https://cdn-uploads.huggingface.co/production/uploads/66137d95e8d2cda230ddcea6/2vBgcGko-tcsaAkLUzHnU.png" width="1000" height="768">
<figcaption> How TIES-Merging Works <a href="//arxiv.org/pdf/2306.01708">Reference</a> </figcaption>
</figure>
### DARE Merging
Introduced by Yu et al. (2023), [DARE](https://arxiv.org/abs/2311.03099) uses an approach similar to TIES with two main differences:
* **Weight Pruning**: Randomly resets some fine-tuned weights to their original values, reducing model complexity.
* **Weight Scaling**: Adjusts the remaining weights by scaling and combining them with the base model's weights to maintain consistent performance.
**DARE** stands for **D**ROP **A**ND **RE**SCALE
Mergekit’s implementation of DARE-Merging has two flavours: with the sign election step of TIES (`dare_ties`) or without (`dare_linear`). I have chosen `dare_ties` for this merge.
For more information refer this [Merge Large Language Models with MergeKit by Maxime Labonne](https://towardsdatascience.com/merge-large-language-models-with-mergekit-2118fb392b54)
Also, if you want to get in-depth knowledge about Model-Merging and its different types, I highly recommend this [YouTube Video by Julien Simon](https://youtu.be/cvOpX75Kz4M?si=d5crVWSxcjvNUm6a)
## 🧩 Configuration
```yaml
models:
- model: NousResearch/Meta-Llama-3-8B
# No parameters necessary for base model
- model: yuvraj17/Llama-3-8B-spectrum-25
parameters:
density: 0.56
weight: 0.12
- model: ruggsea/Llama3-stanford-encyclopedia-philosophy-QA
parameters:
density: 0.56
weight: 0.12
- model: arcee-ai/Llama-3.1-SuperNova-Lite
parameters:
density: 0.58
weight: 0.55
merge_method: dare_ties
base_model: NousResearch/Meta-Llama-3-8B
dtype: bfloat16
```
## 💻 Usage
```python
!pip install -qU transformers accelerate
from transformers import AutoTokenizer
import transformers
import torch
model = "yuvraj17/Llama3-8B-SuperNova-Spectrum-dare_ties"
messages = [{"role": "user", "content": "What is a large language model?"}]
tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
"text-generation",
model=model,
torch_dtype=torch.float16,
device_map="auto",
)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```
## 🏆 Evaluation Scores
### Nous
| Model |AGIEval|TruthfulQA|Bigbench|
|----------------------------------------------------------------------------------------------------------------|------:|---------:|-------:|
|[Llama3-8B-SuperNova-Spectrum-dare_ties](https://huggingface.co/yuvraj17/Llama3-8B-SuperNova-Spectrum-dare_ties)| 38.32| 57.15| 43.91|
### AGIEval
| Task |Version| Metric |Value| |Stderr|
|------------------------------|------:|--------|----:|---|-----:|
|agieval_aqua_rat | 0|acc |20.47|± | 2.54|
| | |acc_norm|18.50|± | 2.44|
|agieval_logiqa_en | 0|acc |35.94|± | 1.88|
| | |acc_norm|35.64|± | 1.88|
|agieval_lsat_ar | 0|acc |21.74|± | 2.73|
| | |acc_norm|20.00|± | 2.64|
|agieval_lsat_lr | 0|acc |41.37|± | 2.18|
| | |acc_norm|40.98|± | 2.18|
|agieval_lsat_rc | 0|acc |59.11|± | 3.00|
| | |acc_norm|56.13|± | 3.03|
|agieval_sat_en | 0|acc |63.59|± | 3.36|
| | |acc_norm|60.19|± | 3.42|
|agieval_sat_en_without_passage| 0|acc |40.29|± | 3.43|
| | |acc_norm|37.38|± | 3.38|
|agieval_sat_math | 0|acc |38.64|± | 3.29|
| | |acc_norm|37.73|± | 3.28|
Average: 38.32%
### TruthfulQA
| Task |Version|Metric|Value| |Stderr|
|-------------|------:|------|----:|---|-----:|
|truthfulqa_mc| 1|mc1 |38.43|± | 1.7|
| | |mc2 |57.15|± | 1.5|
Average: 57.15%
### Bigbench
| Task |Version| Metric |Value| |Stderr|
|------------------------------------------------|------:|---------------------|----:|---|-----:|
|bigbench_causal_judgement | 0|multiple_choice_grade|58.42|± | 3.59|
|bigbench_date_understanding | 0|multiple_choice_grade|70.73|± | 2.37|
|bigbench_disambiguation_qa | 0|multiple_choice_grade|30.23|± | 2.86|
|bigbench_geometric_shapes | 0|multiple_choice_grade|47.35|± | 2.64|
| | |exact_str_match | 0.00|± | 0.00|
|bigbench_logical_deduction_five_objects | 0|multiple_choice_grade|29.00|± | 2.03|
|bigbench_logical_deduction_seven_objects | 0|multiple_choice_grade|21.00|± | 1.54|
|bigbench_logical_deduction_three_objects | 0|multiple_choice_grade|51.33|± | 2.89|
|bigbench_movie_recommendation | 0|multiple_choice_grade|33.20|± | 2.11|
|bigbench_navigate | 0|multiple_choice_grade|55.40|± | 1.57|
|bigbench_reasoning_about_colored_objects | 0|multiple_choice_grade|66.35|± | 1.06|
|bigbench_ruin_names | 0|multiple_choice_grade|45.76|± | 2.36|
|bigbench_salient_translation_error_detection | 0|multiple_choice_grade|28.26|± | 1.43|
|bigbench_snarks | 0|multiple_choice_grade|62.43|± | 3.61|
|bigbench_sports_understanding | 0|multiple_choice_grade|50.30|± | 1.59|
|bigbench_temporal_sequences | 0|multiple_choice_grade|48.00|± | 1.58|
|bigbench_tracking_shuffled_objects_five_objects | 0|multiple_choice_grade|23.60|± | 1.20|
|bigbench_tracking_shuffled_objects_seven_objects| 0|multiple_choice_grade|17.66|± | 0.91|
|bigbench_tracking_shuffled_objects_three_objects| 0|multiple_choice_grade|51.33|± | 2.89|
Average: 43.91%
## Special thanks & Reference
- Maxime Labonne for their easy-to-use colab-notebook [Merging LLMs with MergeKit](https://github.com/mlabonne/llm-course/blob/main/Mergekit.ipynb), [Blog](https://towardsdatascience.com/merge-large-language-models-with-mergekit-2118fb392b54) and [LLM-AutoEva Notebookl](https://github.com/mlabonne/llm-autoeval)
- Authors of [Mergekit](https://github.com/arcee-ai/mergekit) |