Fitsum Gaim commited on
Commit
8eb3c17
1 Parent(s): 1f0a04c

Add model card

Browse files
Files changed (1) hide show
  1. README.md +22 -0
README.md ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: ti
3
+ widget:
4
+ - text: "ዓቕሚ መንእሰይ ኤርትራ [MASK] ተራእዩ"
5
+ ---
6
+
7
+ # Pre-trained ELECTRA small for Tigrinya Language
8
+
9
+ We pre-train ELECTRA small on the [TLMD](https://zenodo.org/record/5139094) dataset, with over 40 million tokens.
10
+
11
+ Contained are trained Flax and PyTorch models.
12
+
13
+
14
+ ## Hyperparameters
15
+
16
+ The hyperparameters corresponding to model sizes mentioned above are as follows:
17
+
18
+ | Model Size | L | AH | HS | FFN | P | Seq |
19
+ |------------|----|----|-----|------|------|------|
20
+ | BASE | 12 | 4 | 256 | 1024 | 14M | 512 |
21
+
22
+ (L = number of layers; AH = number of attention heads; HS = hidden size; FFN = feedforward network dimension; P = number of parameters; Seq = maximum sequence length.)