metterian commited on
Commit
c95001f
1 Parent(s): 3d31b0f

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +34 -0
README.md ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - ko
5
+ ---
6
+ # KULLM-Polyglot-12.8B-v2
7
+
8
+ This model is a fine-tuned version of [EleutherAI/polyglot-ko-12.8b](https://huggingface.co/EleutherAI/polyglot-ko-12.8b) on a KULLM v2
9
+
10
+ Detail Codes are available at [KULLM Github Repository](https://github.com/nlpai-lab/KULLM)
11
+
12
+
13
+ ## Training procedure
14
+
15
+ ### Training hyperparameters
16
+
17
+ The following hyperparameters were used during training:
18
+
19
+ - learning_rate: 3e-4
20
+ - train_batch_size: 6
21
+ - seed: 42
22
+ - distributed_type: multi-GPU (A100 80G)
23
+ - num_devices: 4
24
+ - gradient_accumulation_steps: 21.3
25
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
26
+ - lr_scheduler_type: linear
27
+ - num_epochs: 8.0
28
+
29
+ ### Framework versions
30
+
31
+ - Transformers 4.28.1
32
+ - Pytorch 2.0.0+cu117
33
+ - Datasets 2.11.0
34
+ - Tokenizers 0.13.3