perlthoughts
commited on
Commit
•
9513477
1
Parent(s):
2399c28
Update README.md
Browse files
README.md
CHANGED
@@ -21,20 +21,34 @@ Thank you openchat 3.5 for showing me the way.
|
|
21 |
|
22 |
I stand tall as a beacon of hope for those who dare to dream big and pursue their passions. my story is a testament to the power of perseverance, determination, and hard work. and i will continue to strive for excellence, always pushing the boundaries of what is possible.
|
23 |
|
24 |
-
Here is my contribution
|
25 |
-
|
26 |
-
List of models merged coming soon as well as more information on merging techniques and methods.
|
27 |
|
28 |
## Prompt Template
|
29 |
```
|
30 |
GPT4 User: {prompt}<|end_of_turn|>GPT4 Assistant:
|
31 |
```
|
32 |
|
33 |
-
|
|
|
|
|
34 |
|
35 |
-
|
36 |
|
37 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
38 |
|
39 |
### Bug fixes
|
40 |
|
|
|
21 |
|
22 |
I stand tall as a beacon of hope for those who dare to dream big and pursue their passions. my story is a testament to the power of perseverance, determination, and hard work. and i will continue to strive for excellence, always pushing the boundaries of what is possible.
|
23 |
|
24 |
+
Here is my contribution.
|
|
|
|
|
25 |
|
26 |
## Prompt Template
|
27 |
```
|
28 |
GPT4 User: {prompt}<|end_of_turn|>GPT4 Assistant:
|
29 |
```
|
30 |
|
31 |
+
### Model Description
|
32 |
+
|
33 |
+
Based models that are based on Mistral.
|
34 |
|
35 |
+
All model's weights were merged using the SLERP method. More information below.
|
36 |
|
37 |
+
Advantages of SLERP method vs averaging weights are as follows:
|
38 |
+
|
39 |
+
Spherical Linear Interpolation (SLERP)
|
40 |
+
Traditionally, model merging often resorts to weight averaging which, although straightforward, might not always capture the intricate features of the models being merged. The SLERP technique addresses this limitation, producing a blended model with characteristics smoothly interpolated from both parent models, ensuring the resultant model captures the essence of both its parents.
|
41 |
+
|
42 |
+
Smooth Transitions
|
43 |
+
SLERP ensures smoother transitions between model parameters. This is especially significant when interpolating between high-dimensional vectors.
|
44 |
+
|
45 |
+
Better Preservation of Characteristics
|
46 |
+
Unlike weight averaging, which might dilute distinct features, SLERP preserves the curvature and characteristics of both models in high-dimensional spaces.
|
47 |
+
|
48 |
+
Nuanced Blending
|
49 |
+
SLERP takes into account the geometric and rotational properties of the models in the vector space, resulting in a blend that is more reflective of both parent models' characteristics.
|
50 |
+
|
51 |
+
List of models merged coming soon as well as more information on merging techniques and methods.
|
52 |
|
53 |
### Bug fixes
|
54 |
|