Update README.md

README.md

@@ -1,3 +1,58 @@
 ---
 license: mit
+language:
+- ja
+pipeline_tag: sentence-similarity
 ---
+
+This model was created by merging [intfloat/e5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct) and [stabilityai/japanese-stablelm-base-gamma-7b](https://huggingface.co/stabilityai/japanese-stablelm-base-gamma-7b).  
+See [intfloat/e5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct) page for model usage.
+
+The steps to merge are as follows.
+
+1. Load intfloat/e5-mistral-7b-instruct as a "MistralForCausalLM" class and save_pretrained as is.
+
+Because e5-mistral-7b-instruct is made with the "MistralModel" class, it could not be merged with "MistraForCausalLM" as is.  
+In my environment, I had to load into the CPU, not the GPU, or I would get an error.
+
+```
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "intfloat/e5-mistral-7b-instruct"
+model = AutoModelForCausalLM.from_pretrained(model_id)#, device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+model.save_pretrained("./e5-mistral-7b-instruct_with_lm_head")
+```
+
+2. Merge using [mergekit](https://github.com/cg123/mergekit) with the following yaml configuration
+
+merge_config.yaml
+``` 
+models:
+  - model: stabilityai/japanese-stablelm-base-gamma-7b
+  - model: ./e5-mistral-7b-instruct_with_lm_head
+base_model: stabilityai/japanese-stablelm-base-gamma-7b
+parameters:
+  t:
+    - filter: self_attn
+      value: [0.75, 0.25]
+    - filter: mlp
+      value: [0.75, 0.25]
+    - value: 0.5 # fallback for rest of tensors
+
+merge_method: slerp
+dtype: float16
+```
+
+I tried the "linear", "slerp", and "task_arithmetic" merging methods, and this setting seemed to be the best.  
+The choice of "t" parameters was set to use more japanese-stablelm-base-gamma-7b for the layer closer to the input to take advantage of Japanese word understanding, 
+and more e5-mistral-7b-instruct for the layer closer to the output to generate good embeddings.  
+As for the "ties" method, I could not find any parameters for density and weight that worked properly.
+
+3. Copy settings related to pad_token from the e5-mistral-7b-instruct repository.
+
+* config.json
+* tokenizer.json
+* tokenizer.model
+* tokenizer_config.json