Update README.md

README.md

@@ -1,3 +1,145 @@
----
-license: llama3
----
+---
+license: llama3
+datasets:
+- legacy-datasets/wikipedia
+language:
+- en
+- ko
+library_name: transformers
+pipeline_tag: text-generation
+---
+## Model Details
+
+This model was continually pretrained from the [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B), using English and Korean datasets.
+The goal is to enhance its proficiency in Korean while maintaining its English language capabilities from the original model.
+
+### Datasets
+
+We sampled 16B tokens from the following datasets for training:
+
+<table>
+  <tr>
+   <td><strong>Sources</strong>
+   </td>
+   <td><strong>Tokens (Llama-3-8B)</strong>
+   </td>
+  </tr>
+  <tr>
+   <td>AI-Hub
+   </td>
+   <td>9.2B
+   </td>
+  </tr>
+  <tr>
+   <td>Modu Corpus
+   </td>
+   <td>5.8B
+   </td>
+  </tr>
+  <tr>
+   <td>Wikipedia
+   </td>
+   <td>5.4B
+   </td>
+  </tr>
+</table>
+
+### Hyperparameters
+
+<table>
+  <tr>
+   <td><strong>Learning rate</strong></td>
+   <td><strong>Optimizer</strong></td>
+   <td><strong>Betas</strong></td>
+   <td><strong>Weight decay</strong></td>
+   <td><strong>Warm-up ratio</strong></td>
+  </tr>
+  <tr>
+   <td>3e-5</td>
+   <td>AdamW</td>
+   <td>(0.9, 0.95)</td>
+   <td>0.1</td>
+   <td>0.05</td>
+  </tr>
+</table>
+
+## Intended Use
+
+This model has not been fine-tuned, so you will need to train it on your own dataset before using it.
+
+## Evaluations
+
+We evaluated this model using both English and Korean benchmarks, and compared it with similar models that were continually pretrained from the [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B).
+
+<table>
+  <tr>
+   <td></td>
+   <td colspan="4"><strong>English</strong></td>
+   <td colspan="3"><strong>Korean</strong></td>
+  </tr>
+  <tr>
+   <td><strong>Model</strong></td>
+   <td><strong>MMLU (5 shots)</strong></td>
+   <td><strong>HellaSwag (10 shots)</strong></td>
+   <td><strong>GSM8K (8 shots, CoT)</strong></td>
+   <td><strong>BBH (3 shots, CoT)</strong></td>
+   <td><strong>KMMLU (5 shots)</strong></td>
+   <td><strong>HAE-RAE (5 shots)</strong></td>
+   <td><strong>KoBEST (5 shots)</strong></td>
+  </tr>
+  <tr>
+   <td>meta-llama/Meta-Llama-3-8B</td>
+   <td><strong>65.1</strong></td>
+   <td><strong>82.1</strong></td>
+   <td><strong>52.0</strong></td>
+   <td><strong>61.9</strong></td>
+   <td>40.2</td>
+   <td>61.1</td> 
+   <td>69.2</td>  
+  </tr>
+  <tr>
+   <td>saltlux/Ko-Llama3-Luxia-8B</td>
+   <td>57.1</td>
+   <td>77.1</td>
+   <td>32.3</td>
+   <td>51.8</td>
+   <td>39.4</td>
+   <td>69.2</td> 
+   <td>71.9</td>  
+  </tr>
+  <tr>
+   <td>beomi/Llama-3-Open-Ko-8B</td>
+   <td>56.2</td>
+   <td>77.4</td>
+   <td>31.5</td>
+   <td>46.8</td>
+   <td>40.3</td>
+   <td>68.1</td> 
+   <td><u>72.1</u></td>  
+  </tr>
+  <tr>
+   <td>beomi/Llama-3-KoEn-8B</td>
+   <td>52.5</td>
+   <td>77.7</td>
+   <td>21.2</td>
+   <td>43.2</td>
+   <td><u>40.8</u></td>
+   <td><u>71.3</u></td> 
+   <td><strong>73.8</strong></td>  
+  </tr>
+  <tr>
+   <td><strong>tesser/Tesser-Llama-3-Ko-8B</strong></td>
+   <td><u>60.5</u></td>
+   <td><u>79.8</u></td>
+   <td><u>40.3</u></td>
+   <td><u>56.3</u></td>
+   <td><strong>42.5</strong></td>
+   <td><strong>72.1</strong></td> 
+   <td><strong>73.8</strong></td>  
+  </tr>
+</table>
+
+
+## License
+
+This model follows the original [Llama-3 license](https://llama.meta.com/llama3/license/).