File size: 8,337 Bytes
976d97d 75aae9d 976d97d 14493de 976d97d 90d10ed 976d97d 90d10ed 976d97d 75aae9d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 |
---
license: apache-2.0
tags:
- merge
- mergekit
- lazymergekit
- liminerity/phive
- mobiuslabsgmbh/aanaphi2-v0.1
model-index:
- name: phigment6-slerp
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: AI2 Reasoning Challenge (25-Shot)
type: ai2_arc
config: ARC-Challenge
split: test
args:
num_few_shot: 25
metrics:
- type: acc_norm
value: 62.63
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=liminerity/phigment6-slerp
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: HellaSwag (10-Shot)
type: hellaswag
split: validation
args:
num_few_shot: 10
metrics:
- type: acc_norm
value: 77.25
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=liminerity/phigment6-slerp
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU (5-Shot)
type: cais/mmlu
config: all
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 58.65
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=liminerity/phigment6-slerp
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: TruthfulQA (0-shot)
type: truthful_qa
config: multiple_choice
split: validation
args:
num_few_shot: 0
metrics:
- type: mc2
value: 50.49
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=liminerity/phigment6-slerp
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: Winogrande (5-shot)
type: winogrande
config: winogrande_xl
split: validation
args:
num_few_shot: 5
metrics:
- type: acc
value: 73.88
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=liminerity/phigment6-slerp
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GSM8k (5-shot)
type: gsm8k
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 58.61
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=liminerity/phigment6-slerp
name: Open LLM Leaderboard
---
RANKED NUMBER 1 FOR 3B MODELS!
# phigment6-slerp
Title: Creating the Number 1 3B Parameter LLM in the World - Phigment6, A Phi-2 Based Model Using Divergent Knowledge Enhancement through Retrograde Merging Strategies (DKERS) Methodology
Abstract
The rapid advancements in artificial intelligence have led to the development of large language models (LLMs). In this paper, we present Phigment6, an innovative 3 billion parameter LLM built on the foundation of the Phi-2 architecture. We detail our unique methodology called Divergent Knowledge Enhancement through Retrograde Merging Strategies (DKERS), which involves the strategic combination of multiple pretrained models to create an even more powerful and accurate language model. Through this approach, we successfully merge amu/dpo-phi2, g-ronimo/phi-2-OpenHermes-2.5, vince62s/phi-2-psy, and mobiuslabsgmbh/aanaphi2-v0.1, leading to the creation of Phigment6. Our results demonstrate significant improvements in performance compared to existing state-of-the-art LLMs.
Introduction
Recent years have witnessed tremendous growth in natural language processing capabilities, driven by advances in deep learning techniques and the introduction of transformers in NLP tasks. Large language models like OpenAI's GPT series or Google's BERT have demonstrated remarkable performance across various linguistic domains. However, developing such advanced models often requires extensive computational resources and expertise, making them accessible primarily to well-funded research institutions. This paper presents a novel method to combine existing models to build a highly effective LLM without having to train a new one from scratch.
Methodology: Divergent Knowledge Enhancement through Retrograde Merging Strategies (DKERS)
Our proposed approach, DKERS, consists of two main steps: merging and refining. Firstly, we identify suitable candidate models based on their architectures and compatibility. Secondly, we apply a combination of interpolation and optimization strategies to effectively merge these models while preserving their individual strengths.
Step 1: Candidate Selection
We begin by selecting four compatible models as potential candidates for merging:
amu/dpo-phi2: A baseline Phi-2 model, providing a strong foundation for further enhancement.
g-ronimo/phi-2-OpenHermes-2.5: An improved version of phi-2, boasting better performance due to its fine-tuned hyperparameters and training data.
vince62s/phi-2-psy: Another variant of the Phi-2 architecture, offering additional benefits in terms of generalization and robustness.
mobiuslabsgmbh/aanaphi2-v0.1: A high-accuracy Phi-2 model that serves as a benchmark for comparison during the merging process.
Step 2: Model Merging
To merge the selected models, we employ a strategy known as spherical linear interpolation (SLERP), which enables us to smoothly transition between the parameters of two models. Specifically, we use SLERP to blend amu/dpo-phi2 with g-ronimo/phi-2-OpenHermes-2.5. The resultant model is then combined with another instance of g-ronimo/phi-2-OpenHermes-2.5 using the same blending technique. Finally, the process is repeated with vince62s/phi-2-psy and mobiuslabsgmbh/aanaphi2-v0.1. Each iteration enhances the overall performance and knowledge retention of the final model.
Results
After following the DKERS methodology, we obtain Phigment6, a powerful and efficient 3 billion parameter LLM. Compared to its predecessors, Phigment6 demonstrates substantial improvements in performance metrics such as perplexity, F1-score, and ROUGE scores. Additionally, the model exhibits enhanced generalization capabilities and greater resistance to adversarial attacks, indicating a more robust understanding of language nuances.
Conclusion
In summary, we presented Phigment6, a cutting-edge 3 billion parameter LLM, constructed via the novel Divergent Knowledge Enhancement through Retrograde Merging Strategies (DKERS) methodology. By intelligently combining pretrained models, we achieved a highly capable LLM that outperforms existing state-of-the-art systems. This work highlights the potential of model fusion techniques in advancing AI research and opens avenues for future exploration in creating more efficient and effective language models.
phigment6-slerp is a merge of the following models using [mergekit](https://github.com/cg123/mergekit):
* [liminerity/phive](https://huggingface.co/liminerity/phive)
* [mobiuslabsgmbh/aanaphi2-v0.1](https://huggingface.co/mobiuslabsgmbh/aanaphi2-v0.1)
## 🧩 Configuration
```yaml
slices:
- sources:
- model: liminerity/phive
layer_range: [0, 32]
- model: mobiuslabsgmbh/aanaphi2-v0.1
layer_range: [0, 32]
merge_method: slerp
base_model: liminerity/phive
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5
dtype: float16
```
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_liminerity__phigment6-slerp)
| Metric |Value|
|---------------------------------|----:|
|Avg. |63.58|
|AI2 Reasoning Challenge (25-Shot)|62.63|
|HellaSwag (10-Shot) |77.25|
|MMLU (5-Shot) |58.65|
|TruthfulQA (0-shot) |50.49|
|Winogrande (5-shot) |73.88|
|GSM8k (5-shot) |58.61|
|