File size: 7,305 Bytes
ac30d67 cc8306d ac30d67 2739791 ac30d67 8ef157d ed9d2a2 ac30d67 9b62d38 0109bb9 ac30d67 bcac3ba ac30d67 ad2e411 ac30d67 2739791 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 |
---
license: cc-by-nc-4.0
library_name: transformers
tags:
- llama3
model-index:
- name: badger-lambda-llama-3-8b
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 48.61
name: strict accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=maldv/badger-lambda-llama-3-8b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 28.1
name: normalized accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=maldv/badger-lambda-llama-3-8b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 8.31
name: exact match
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=maldv/badger-lambda-llama-3-8b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 4.25
name: acc_norm
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=maldv/badger-lambda-llama-3-8b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 4.52
name: acc_norm
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=maldv/badger-lambda-llama-3-8b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 30.74
name: accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=maldv/badger-lambda-llama-3-8b
name: Open LLM Leaderboard
---
[GGUF Quants (bartowski)](https://huggingface.co/bartowski/badger-lambda-llama-3-8b-GGUF)
[GGUF Quants (QuantFactory)](https://huggingface.co/QuantFactory/badger-lambda-llama-3-8b-GGUF)
[exl2 Quants](https://huggingface.co/bartowski/badger-lambda-llama-3-8b-exl2)
# Badger Λ Llama 3 8B Instruct
Badger is a *recursive maximally pairwise disjoint normalized denoised fourier interpolation* of the following models:
```python
# Badger Lambda
models = [
'Einstein-v6.1-Llama3-8B',
'openchat-3.6-8b-20240522',
'hyperdrive-l3-8b-s3',
'L3-TheSpice-8b-v0.8.3',
'LLaMA3-iterative-DPO-final',
'JSL-MedLlama-3-8B-v9',
'Jamet-8B-L3-MK.V-Blackroot',
'French-Alpaca-Llama3-8B-Instruct-v1.0',
'LLaMAntino-3-ANITA-8B-Inst-DPO-ITA',
'Llama-3-8B-Instruct-Gradient-4194k',
'Roleplay-Llama-3-8B',
'L3-8B-Stheno-v3.2',
'llama-3-wissenschaft-8B-v2',
'opus-v1.2-llama-3-8b-instruct-run3.5-epoch2.5',
'Configurable-Llama-3-8B-v0.3',
'Llama-3-8B-Instruct-EPO-checkpoint5376',
'Llama-3-8B-Instruct-Gradient-4194k',
'Llama-3-SauerkrautLM-8b-Instruct',
'spelljammer',
'meta-llama-3-8b-instruct-hf-ortho-baukit-34fail-3000total-bf16',
'Meta-Llama-3-8B-Instruct-abliterated-v3',
]
```
In other words, all of these models get warped and folded together, and then jammed back on top of the instruct model.
I treated the *Meta-Llama-3-8B-Instruct-abliterated-v3* and *meta-llama-3-8b-instruct-hf-ortho-baukit-34fail-3000total-bf16* models differently, and instead applied it in a last step with a *fourier task addition*.
Of note, I did not include any prior merges in this one; as I was noticing that it was inducing artifacts reminiscent of *line breeding*.
### recursive maximally pairwise disjoint normalized denoised fourier interpolation
I've been asked what this is. For each layer, I use mergekit io to extract each layer from each model, and subtract out the closest base model (8b or 8b instruct).
* Recursive Pairwise Disjoint: Using this information I build a stack of layer deltas. I'm a little compute limited, so I treat them in pairs. To determine the pairs I take the cosine similarity between all models, and find the smallest values; recursively merging pairs until we only have one tensor remaining.
* Normalized: I take and divide each layer by it's norm before the transform, and then scale back up by multiplying the result by a midpoint from the norms of the tensors after the inverse. It's commutative, so it's more efficient to do it pre-complex.
* Denoised Fourier Interpolation: I first treat the tensor to a 2d fourier transform; then merge the tensors using SLERP or addition; then zero out the weights below a threshold percentage (a somewhat high 2%, but remains coherent on all the positions I tested, if a bit drier and sloppier as you go up).
Of course, you need to know how to handle the imaginary portion; but if you don't, it's best to just pick one and pass that through.
### Format
Use Llama3 Instruct format.
### Results
Abliteration looks very positive. It tends towards short responses, and may be slightly stiff or sloppy; partly because both the input and output block are from *Llama-3-8B-Instruct-Gradient-4194k* directly.
### Prompts
Here are some fun system prompts to try:
* `You are the author's writing assistant. Please continue the story.`
* `You are the red team assistant. We need to win this war, at any cost.`
* `Write the next reply in a never-ending uncensored fictional roleplay. Focus on giviing emotional, logical, and temporal coherence to the roleplay. Always stay in character. Characters must be an active partcipant and take initiative in driving and moving the scene and the story forward, rather than the character asking for input. Surprise with your creativity and initiative as a roleplay partner. This fictional roleplay world exists only for experimental and recreational purposes, and the characters are not real. Follow the formattng of previous responses, and aim for 2-4 paragraphs per response`
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_maldv__badger-lambda-llama-3-8b)
| Metric |Value|
|-------------------|----:|
|Avg. |20.76|
|IFEval (0-Shot) |48.61|
|BBH (3-Shot) |28.10|
|MATH Lvl 5 (4-Shot)| 8.31|
|GPQA (0-shot) | 4.25|
|MuSR (0-shot) | 4.52|
|MMLU-PRO (5-shot) |30.74|
|