xxx777xxxASD commited on
Commit
9ea7923
1 Parent(s): 7719f5b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +51 -0
README.md CHANGED
@@ -1,3 +1,54 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ language:
4
+ - en
5
+ tags:
6
+ - merge
7
  ---
8
+ Test merge
9
+
10
+ ### Loyal Mistral Maid v0.2
11
+ ```
12
+ slices:
13
+ - sources:
14
+ - model: Mistral_Instruct_SelfMerge
15
+ layer_range: [0, 48]
16
+ - model: Loyal_Toppy_Maid
17
+ layer_range: [0, 48]
18
+ merge_method: slerp
19
+ base_model: Mistral_Instruct_SelfMerge
20
+ parameters:
21
+ t:
22
+ - filter: self_attn
23
+ value: [0, 0.5, 0.3, 0.7, 1]
24
+ - filter: mlp
25
+ value: [1, 0.5, 0.7, 0.3, 0]
26
+ - value: 0.5 # fallback for rest of tensors
27
+ dtype: bfloat16
28
+ ```
29
+
30
+ ### Loyal Toppy Maid
31
+ ```
32
+ slices:
33
+ - sources:
34
+ - model: Undi95/Toppy-M-7B
35
+ layer_range: [0, 24]
36
+ - sources:
37
+ - model: SanjiWatsuki/Loyal-Macaroni-Maid-7B
38
+ layer_range: [8, 32]
39
+ merge_method: passthrough
40
+ dtype: bfloat16
41
+ ```
42
+
43
+ ### Mistral_Instruct_SelfMerge
44
+ ```
45
+ slices:
46
+ - sources:
47
+ - model: mistralai/Mistral-7B-Instruct-v0.2
48
+ layer_range: [0, 24]
49
+ - sources:
50
+ - model: mistralai/Mistral-7B-Instruct-v0.2
51
+ layer_range: [8, 32]
52
+ merge_method: passthrough
53
+ dtype: bfloat16
54
+ ```