Update README.md

README.md

@@ -28,7 +28,64 @@ This model is a Korean vocabulary-extended version of [upstage/SOLAR-10.7B-v1.0]
 
 ### Technical Deep Dive
 
-TBU
+Here’s a glimpse into our technical approach:
+
+```python
+# number_of_old_tokens is the size of tokenizer before vocab extension. For example, in case of EEVE-Korean-10.8B-v1.0, number_of_old_tokens is 32000.
+def freeze_partial_embedding_hook(grad):
+    grad[:number_of_old_tokens] = 0
+    return grad
+
+for name, param in model.named_parameters():
+    if ("lm_head" in name or "embed_tokens" in name) and "original" not in name:
+        param.requires_grad = True
+        if "embed_tokens" in name:
+            param.register_hook(freeze_partial_embedding_hook)
+    else:
+        param.requires_grad = False
+```
+
+Our strategy involved a selective freeze of model parameters. Specifically, we kept most parameters of the base model unchanged while focusing on enhancing the Korean language capabilities. Through our experiments, we discovered:
+
+1. Freezing the `embed_tokens` layer for existing tokens is crucial to maintain overall performance.
+2. Unfreezing the `lm_head` layer for existing tokens actually boosts performance.
+
+As a result, we froze the internal layers and the first 32,000 `embed_tokens`, directing our training efforts on a rich mix of Korean and multi-lingual corpora. This balanced approach has notably improved the model’s proficiency in Korean, without compromising its original language capabilities.
+
+### Usage and Limitations
+
+Keep in mind that this model hasn't been fine-tuned with instruction-based training. While it excels in Korean language tasks, we advise careful consideration and further training for specific applications.
+
+### Training Details
+
+Our model’s training was comprehensive and diverse:
+
+- **Data Sources:**
+  - English to Korean paragraph pairs: 5.86%
+  - Multi-lingual corpus (primarily English): 10.69%
+  - Korean web content: 83.46%
+
+- **Vocabulary Expansion:**
+  We meticulously selected 8,960 Korean tokens based on their frequency in our Korean web corpus. This process involved multiple rounds of tokenizer training, manual curation, and token frequency analysis, ensuring a rich and relevant vocabulary for our model.
+
+    1. **Initial Tokenizer Training:** We trained an intermediate tokenizer on a Korean web corpus, with a vocabulary of 40,000 tokens.
+    
+    2. **Extraction of New Korean Tokens:** From the intermediate tokenizer, we identified all Korean tokens not present in the original SOLAR's tokenizer.
+
+    3. **Manual Tokenizer Construction:** We then built the target tokenizer, focusing on these new Korean tokens.
+
+    4. **Frequency Analysis:** Using the target tokenizer, we processed a 100GB Korean corpus to count each token's frequency.
+
+    5. **Refinement of Token List:** We removed tokens appearing less than 6,000 times, ensuring to secure enough tokens to train models later.
+
+    6. **Inclusion of Single-Letter Characters:** Counted missing Korean single-letter characters and added them to the target tokenizer that appeared more than 6,000 times.
+
+    7. **Iterative Refinement:** We repeated steps 2 to 6 until there were no tokens to drop or add.
+
+    8. **Training Bias Towards New Tokens:** Our training data was biased to include more texts with new tokens, for effective learning.
+
+This rigorous approach ensured a comprehensive and contextually rich Korean vocabulary for the model.
+
 
 ### Usage and Limitations