eramax commited on
Commit
8959dbd
1 Parent(s): 014ff6a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +218 -0
README.md CHANGED
@@ -1,3 +1,221 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ language:
4
+ - en
5
+ - de
6
+ library_name: transformers
7
+ pipeline_tag: text-generation
8
+ tags:
9
+ - mistral
10
+ - finetune
11
+ - chatml
12
+ - augmentation
13
+ - german
14
  ---
15
+
16
+ ![SauerkrautLM](https://vago-solutions.de/wp-content/uploads/2023/11/hero.png "SauerkrautLM-7b-HerO")
17
+ ## VAGO solutions SauerkrautLM-7b-HerO
18
+ Introducing **SauerkrautLM-7b-HerO** – the pinnacle of German language model technology!
19
+ Crafted through the **merging** of **[Teknium's OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B)** and **[Open-Orca's Mistral-7B-OpenOrca](https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca)** and **uniquely fine-tuned with the Sauerkraut dataset.**
20
+ SauerkrautLM-7b-HerO represents a breakthrough in language modeling, achieving an optimal balance between extensive German data and essential international sources.
21
+ This ensures the model not only excels in understanding the nuances of the German language but also retains its global capabilities.
22
+ Harnessing the innovative power of the **gradient SLERP method from MergeKit**, we've achieved a groundbreaking fusion of two of the most best performing 7B models based on the Mistral framework.
23
+ This merge has allowed us to combine the best features of both models, creating an unparalleled synergy.
24
+ Coupled with the German Sauerkraut dataset, which consists of a mix of augmented and translated data, we have successfully taught the English-speaking merged model the intricacies of the German language.
25
+ This was achieved *without the typical loss of core competencies often associated with fine-tuning in another language of models previously trained mainly in English.*
26
+ Our approach ensures that the model retains its original strengths while acquiring a profound understanding of German, **setting a new benchmark in bilingual language model proficiency.**
27
+
28
+ # Table of Contents
29
+ 1. [Overview of all Her0 models](#all-hero-models)
30
+ 2. [Model Details](#model-details)
31
+ - [Prompt template](#prompt-template)
32
+ - [Training Dataset](#training-dataset)
33
+ - [Merge Procedure](#merge-procedure)
34
+ 3. [Evaluation](#evaluation)
35
+ - [GPT4ALL](#gpt4all)
36
+ - [Language Model evaluation Harness](#language-model-evaluation-harness)
37
+ - [BigBench](#big-bench)
38
+ - [MMLU](#mmlu)
39
+ - [TruthfulQA](#truthfulqa)
40
+ - [MT-Bench (German)](#mt-bench-german)
41
+ - [MT-Bench (English)](#mt-bench-english)
42
+ - [Additional German Benchmark results](#additional-german-benchmark-results)
43
+ 5. [Disclaimer](#disclaimer)
44
+ 6. [Contact](#contact)
45
+ 7. [Collaborations](#collaborations)
46
+ 8. [Acknowledgement](#acknowledgement)
47
+
48
+
49
+ ## All HerO Models
50
+
51
+ | Model | HF | GPTQ | GGUF | AWQ |
52
+ |-------|-------|-------|-------|-------|
53
+ | SauerkrautLM-7b-HerO | [Link](https://huggingface.co/VAGOsolutions/SauerkrautLM-7b-HerO) | coming soon | coming soon | coming soon |
54
+
55
+ ## Model Details
56
+ **SauerkrautLM-7b-HerO**
57
+ - **Model Type:** SauerkrautLM-7b-HerO is an auto-regressive language model based on the transformer architecture
58
+ - **Language(s):** English, German
59
+ - **License:** APACHE 2.0
60
+ - **Contact:** [Website](https://vago-solutions.de/#Kontakt) [David Golchinfar](mailto:golchinfar@vago-solutions.de)
61
+
62
+ ### Training Dataset:
63
+
64
+ SauerkrautLM-7b-HerO was trained with mix of German data augmentation and translated data.
65
+ We found, that only a simple translation of training data can lead to unnatural German phrasings.
66
+ Data augmentation techniques were used to grant grammatical, syntactical correctness and a more natural German wording in our training data.
67
+
68
+ ### Merge Procedure:
69
+
70
+ SauerkrautLM-7b-HerO was merged on 1 A100 with [mergekit](https://github.com/cg123/mergekit).
71
+ The merged model contains [OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) and [Open-Orca/Mistral-7B-OpenOrca](https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca).
72
+ We applied the gradient SLURP method.
73
+
74
+
75
+
76
+ ### Prompt Template:
77
+ ```
78
+ <|im_start|>system
79
+ Du bist Sauerkraut-HerO, ein großes Sprachmodell, das höflich und kompetent antwortet. Schreibe deine Gedanken Schritt für Schritt auf, um Probleme sinnvoll zu lösen.
80
+ <|im_end|>
81
+ <|im_start|>user
82
+ Wie geht es dir?<|im_end|>
83
+ <|im_start|>assistant
84
+ Mir geht es gut!<|im_end|>
85
+ <|im_start|>user
86
+ Bitte erkläre mir, wie die Zusammenführung von Modellen durch bestehende Spitzenmodelle profitieren kann.<|im_end|>
87
+ <|im_start|>assistant
88
+ ```
89
+ ## Evaluation
90
+
91
+ ### GPT4ALL:
92
+ *Compared to relevant German Closed and Open Source models*
93
+ ![GPT4ALL diagram](https://vago-solutions.de/wp-content/uploads/2023/11/GPT4All.png "SauerkrautLM-7b-HerO GPT4ALL Diagram")
94
+
95
+ ![GPT4ALL table](https://vago-solutions.de/wp-content/uploads/2023/11/GPT4All-Tabelle.png "SauerkrautLM-7b-HerO GPT4ALL Table")
96
+
97
+ ### Language Model evaluation Harness:
98
+ *Compared to Aleph Alpha Luminous Models*
99
+ ![Harness](https://vago-solutions.de/wp-content/uploads/2023/11/Luminous-comparison.png "SauerkrautLM-7b-HerO Harness")
100
+
101
+ **performed with newest Language Model Evaluation Harness*
102
+ ### Big Bench:
103
+ ![BBH](https://vago-solutions.de/wp-content/uploads/2023/11/BigBench.png "SauerkrautLM-7b-HerO BBH")
104
+ **performed with newest Language Model Evaluation Harness*
105
+
106
+ ### MMLU:
107
+ *Compared to Big Boy LLMs (Grok0,Grok1,GPT3.5,GPT4)*
108
+ ![MMLU](https://vago-solutions.de/wp-content/uploads/2023/11/MMLU-Benchmark.png "SauerkrautLM-7b-HerO MMLU")
109
+ ### TruthfulQA:
110
+ *Compared to OpenAI Models (GPT3.5,GPT4)*
111
+ ![TruthfulQA](https://vago-solutions.de/wp-content/uploads/2023/11/Truthfulqa-Benchmark.png "SauerkrautLM-7b-HerO TruthfulQA")
112
+
113
+ ### MT-Bench (German):
114
+ ![MT-Bench German Diagram](https://vago-solutions.de/wp-content/uploads/2023/11/MT-Bench-German.png "SauerkrautLM-7b-HerO MT-Bench German Diagram")
115
+ ```
116
+ ########## First turn ##########
117
+ score
118
+ model turn
119
+ SauerkrautLM-70b-v1 1 7.25000
120
+ SauerkrautLM-7b-HerO <--- 1 6.96875
121
+ SauerkrautLM-7b-v1-mistral 1 6.30625
122
+ leo-hessianai-13b-chat 1 6.18750
123
+ SauerkrautLM-13b-v1 1 6.16250
124
+ leo-mistral-hessianai-7b-chat 1 6.15625
125
+ Llama-2-70b-chat-hf 1 6.03750
126
+ vicuna-13b-v1.5 1 5.80000
127
+ SauerkrautLM-7b-v1 1 5.65000
128
+ leo-hessianai-7b-chat 1 5.52500
129
+ vicuna-7b-v1.5 1 5.42500
130
+ Mistral-7B-v0.1 1 5.37500
131
+ SauerkrautLM-3b-v1 1 3.17500
132
+ Llama-2-7b 1 1.28750
133
+ open_llama_3b_v2 1 1.68750
134
+
135
+ ########## Second turn ##########
136
+ score
137
+ model turn
138
+ SauerkrautLM-70b-v1 2 6.83125
139
+ SauerkrautLM-7b-HerO <--- 2 6.30625
140
+ vicuna-13b-v1.5 2 5.63125
141
+ SauerkrautLM-13b-v1 2 5.34375
142
+ SauerkrautLM-7b-v1-mistral 2 5.26250
143
+ leo-mistral-hessianai-7b-chat 2 4.99375
144
+ SauerkrautLM-7b-v1 2 4.73750
145
+ leo-hessianai-13b-chat 2 4.71250
146
+ vicuna-7b-v1.5 2 4.67500
147
+ Llama-2-70b-chat-hf 2 4.66250
148
+ Mistral-7B-v0.1 2 4.53750
149
+ leo-hessianai-7b-chat 2 2.65000
150
+ SauerkrautLM-3b-v1 2 1.98750
151
+ open_llama_3b_v2 2 1.22500
152
+ Llama-2-7b 2 1.07500
153
+
154
+ ########## Average ##########
155
+ score
156
+ model
157
+ SauerkrautLM-70b-v1 7.040625
158
+ SauerkrautLM-7b-HerO <--- 6.637500
159
+ SauerkrautLM-7b-v1-mistral 5.784375
160
+ SauerkrautLM-13b-v1 5.753125
161
+ vicuna-13b-v1.5 5.715625
162
+ leo-mistral-hessianai-7b-chat 5.575000
163
+ leo-hessianai-13b-chat 5.450000
164
+ Llama-2-70b-chat-hf 5.350000
165
+ SauerkrautLM-v1-7b 5.193750
166
+ vicuna-7b-v1.5 5.050000
167
+ Mistral-7B-v0.1 4.956250
168
+ leo-hessianai-7b-chat 4.087500
169
+ SauerkrautLM-3b-v1 2.581250
170
+ open_llama_3b_v2 1.456250
171
+ Llama-2-7b 1.181250
172
+ ```
173
+ **performed with the newest FastChat Version*
174
+ ### MT-Bench (English):
175
+ ![MT-Bench English Diagram](https://vago-solutions.de/wp-content/uploads/2023/11/MT-Bench-English.png "SauerkrautLM-7b-HerO MT-Bench English Diagram")
176
+ ```
177
+ ########## First turn ##########
178
+ score
179
+ model turn
180
+ OpenHermes-2.5-Mistral-7B 1 8.21875
181
+ SauerkrautLM-7b-HerO <--- 1 8.03125
182
+ Mistral-7B-OpenOrca 1 7.65625
183
+ neural-chat-7b-v3-1 1 7.22500
184
+
185
+ ########## Second turn ##########
186
+ score
187
+ model turn
188
+ OpenHermes-2.5-Mistral-7B 2 7.1000
189
+ SauerkrautLM-7b-HerO <--- 2 6.7875
190
+ neural-chat-7b-v3-1 2 6.4000
191
+ Mistral-7B-OpenOrca 2 6.1750
192
+
193
+ ########## Average ##########
194
+ score
195
+ model
196
+ OpenHermes-2.5-Mistral-7B 7.659375
197
+ SauerkrautLM-7b-HerO <--- 7.409375
198
+ Mistral-7B-OpenOrca 6.915625
199
+ neural-chat-7b-v3-1 6.812500
200
+ ```
201
+ **performed with the newest FastChat Version*
202
+
203
+ ### Additional German Benchmark results:
204
+ ![GermanBenchmarks](https://vago-solutions.de/wp-content/uploads/2023/11/German-benchmarks.png "SauerkrautLM-7b-HerO German Benchmarks")
205
+ *performed with newest Language Model Evaluation Harness
206
+ ## Disclaimer
207
+ We must inform users that despite our best efforts in data cleansing, the possibility of uncensored content slipping through cannot be entirely ruled out.
208
+ However, we cannot guarantee consistently appropriate behavior. Therefore, if you encounter any issues or come across inappropriate content, we kindly request that you inform us through the contact information provided.
209
+ Additionally, it is essential to understand that the licensing of these models does not constitute legal advice. We are not held responsible for the actions of third parties who utilize our models. These models may be employed for commercial purposes, and the Apache 2.0 remains applicable and is included with the model files.
210
+  
211
+ ## Contact
212
+ If you are interested in customized LLMs for business applications, please get in contact with us via our website or contact us at [Dr. Daryoush Vaziri](mailto:vaziri@vago-solutions.de). We are also grateful for your feedback and suggestions.
213
+  
214
+ ## Collaborations
215
+ We are also keenly seeking support and investment for our startup, VAGO solutions, where we continuously advance the development of robust language models designed to address a diverse range of purposes and requirements. If the prospect of collaboratively navigating future challenges excites you, we warmly invite you to reach out to us.
216
+
217
+ ## Acknowledgement
218
+ Many thanks to [OpenOrca](https://huggingface.co/Open-Orca) and [teknium](https://huggingface.co/teknium) for providing such valuable models to the Open-Source community.
219
+
220
+
221
+ [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)