Update README.md
Browse files
README.md
CHANGED
|
@@ -6,7 +6,7 @@ datasets:
|
|
| 6 |
metrics:
|
| 7 |
- accuracy
|
| 8 |
model-index:
|
| 9 |
-
- name:
|
| 10 |
results:
|
| 11 |
- task:
|
| 12 |
name: Text Classification
|
|
@@ -19,31 +19,70 @@ model-index:
|
|
| 19 |
- name: Accuracy
|
| 20 |
type: accuracy
|
| 21 |
value: 0.7926136255264282
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
---
|
| 23 |
|
| 24 |
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
| 25 |
should probably proofread and complete it, then remove this comment. -->
|
| 26 |
|
| 27 |
-
#
|
| 28 |
|
| 29 |
-
This model is a fine-tuned version of [tartuNLP/EstBERT](https://huggingface.co/tartuNLP/EstBERT) on the
|
| 30 |
-
|
|
|
|
|
|
|
| 31 |
- Loss: 2.2440
|
| 32 |
- Accuracy: 0.7926
|
| 33 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
## Model description
|
| 35 |
|
| 36 |
-
|
| 37 |
|
| 38 |
## Intended uses & limitations
|
| 39 |
|
| 40 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
## Training and evaluation data
|
| 43 |
|
| 44 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
|
| 46 |
## Training procedure
|
|
|
|
|
|
|
| 47 |
|
| 48 |
### Training hyperparameters
|
| 49 |
|
|
@@ -60,73 +99,74 @@ The following hyperparameters were used during training:
|
|
| 60 |
- mixed_precision_training: Native AMP
|
| 61 |
|
| 62 |
### Training results
|
|
|
|
| 63 |
|
| 64 |
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|
| 65 |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
|
| 66 |
-
| 0.836 |
|
| 67 |
-
| 0.5336 |
|
| 68 |
-
| 0.2913 |
|
| 69 |
-
| 0.1048 |
|
| 70 |
-
| 0.0424 |
|
| 71 |
-
| 0.0262 |
|
| 72 |
-
| 0.0169 |
|
| 73 |
-
| 0.0048 |
|
| 74 |
-
| 0.0084 |
|
| 75 |
-
| 0.0087 |
|
| 76 |
-
| 0.0082 |
|
| 77 |
-
| 0.0104 |
|
| 78 |
-
| 0.0115 |
|
| 79 |
-
| 0.0111 |
|
| 80 |
-
| 0.0028 |
|
| 81 |
-
| 0.0032 |
|
| 82 |
-
| 0.0107 |
|
| 83 |
-
| 0.0056 |
|
| 84 |
-
| 0.0078 |
|
| 85 |
-
| 0.0041 |
|
| 86 |
-
| 0.0087 |
|
| 87 |
-
| 0.0113 |
|
| 88 |
-
| 0.0025 |
|
| 89 |
-
| 0.0083 |
|
| 90 |
-
| 0.0104 |
|
| 91 |
-
| 0.0041 |
|
| 92 |
-
| 0.002 |
|
| 93 |
-
| 0.001 |
|
| 94 |
-
| 0.003 |
|
| 95 |
-
| 0.006 |
|
| 96 |
-
| 0.0005 |
|
| 97 |
-
| 0.0007 |
|
| 98 |
-
| 0.0088 |
|
| 99 |
-
| 0.0 |
|
| 100 |
-
| 0.0 |
|
| 101 |
-
| 0.0 |
|
| 102 |
-
| 0.0 |
|
| 103 |
-
| 0.0 |
|
| 104 |
-
| 0.0 |
|
| 105 |
-
| 0.0087 |
|
| 106 |
-
| 0.0021 |
|
| 107 |
-
| 0.0054 |
|
| 108 |
-
| 0.0051 |
|
| 109 |
-
| 0.0 |
|
| 110 |
-
| 0.0019 |
|
| 111 |
-
| 0.0044 |
|
| 112 |
-
| 0.0006 |
|
| 113 |
-
| 0.0007 |
|
| 114 |
-
| 0.0001 |
|
| 115 |
-
| 0.0 |
|
| 116 |
-
| 0.0 |
|
| 117 |
-
| 0.0031 |
|
| 118 |
-
| 0.0 |
|
| 119 |
-
| 0.0 |
|
| 120 |
-
| 0.0 |
|
| 121 |
-
| 0.0001 |
|
| 122 |
-
| 0.0007 |
|
| 123 |
-
| 0.0 |
|
| 124 |
-
| 0.0 |
|
| 125 |
-
| 0.0 |
|
| 126 |
-
| 0.0013 |
|
| 127 |
-
| 0.0001 |
|
| 128 |
-
| 0.0011 |
|
| 129 |
-
| 0.0002 |
|
| 130 |
|
| 131 |
|
| 132 |
### Framework versions
|
|
@@ -135,3 +175,6 @@ The following hyperparameters were used during training:
|
|
| 135 |
- Pytorch 1.10.1+cu113
|
| 136 |
- Datasets 1.16.1
|
| 137 |
- Tokenizers 0.10.3
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
metrics:
|
| 7 |
- accuracy
|
| 8 |
model-index:
|
| 9 |
+
- name: EstBERT128_Sentiment
|
| 10 |
results:
|
| 11 |
- task:
|
| 12 |
name: Text Classification
|
|
|
|
| 19 |
- name: Accuracy
|
| 20 |
type: accuracy
|
| 21 |
value: 0.7926136255264282
|
| 22 |
+
widget:
|
| 23 |
+
- text: "Enam kui kümme aastat tagasi tegutses huumorisaates «Wremja» inspektor Kukeke, kes kogu aeg vingus väikese palga pärast ja vaatas, mida saaks töö juurest koju tassida. Stsenaristid Andrus Kivirähk ja Mart Juur olid Kukekese isikusse kokku valanud kõik, mis 1990. aastate Eesti politseinikke halvast küljest iseloomustas."
|
| 24 |
+
example_title: "negative"
|
| 25 |
+
- text: "Isiklikult kohtasin natukegi Kukekese moodi politseinikku viimati kaheksa aasta eest Lätis. Eranditult kõik viimase kümnendi kokkupuuted politseiametnikega on kinnitanud: vaatamata raskustele on Eesti riik suutnud korrakaitsjateks värvata inimesi, kes on arukad, kohusetundlikud, lugupidamist sisendavas füüsilises vormis ja hea väljendusoskusega."
|
| 26 |
+
example_title: "positive"
|
| 27 |
+
- text: "Pisut retooriline küsimus, kelle või mille jaoks on Estonian Ai, nõuab taas vastust. Oleme jõudnud olukorda, kus vastus peaks olema juba konkreetne. Siinkohal tuleks hoiduda rahvusliku lennukompanii mõistest, mis pärineb ajast, kui lennundusäri oli peaaegu sajaprotsendiliselt riigi kontrolli all ning riigid ja nende grupeeringud reguleerisid äärmise põhjalikkusega lennundusturgu."
|
| 28 |
+
example_title: "neutral"
|
| 29 |
---
|
| 30 |
|
| 31 |
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
| 32 |
should probably proofread and complete it, then remove this comment. -->
|
| 33 |
|
| 34 |
+
# EstBERT128_sentiment
|
| 35 |
|
| 36 |
+
This model is a fine-tuned version of [tartuNLP/EstBERT](https://huggingface.co/tartuNLP/EstBERT) on the reduced version of the [Estonian Valence corpus](https://figshare.com/articles/dataset/Estonian_Valence_Corpus_Eesti_valentsikorpus/24517054), where the items with Mixed labels were removed.
|
| 37 |
+
The data (containing Positive, Negative and Neutral labels) was split into 70/10/20 train/dev/test splits.
|
| 38 |
+
|
| 39 |
+
It achieves the following results on the developments split:
|
| 40 |
- Loss: 2.2440
|
| 41 |
- Accuracy: 0.7926
|
| 42 |
|
| 43 |
+
It achieves the following results on the test split:
|
| 44 |
+
- Loss: 2.7633
|
| 45 |
+
- Accuracy: 0.7479
|
| 46 |
+
|
| 47 |
## Model description
|
| 48 |
|
| 49 |
+
A single linear layer classifier is fit on top of the last layer [CLS] token representation of the EstBERT model. The model is fully fine-tuned during training.
|
| 50 |
|
| 51 |
## Intended uses & limitations
|
| 52 |
|
| 53 |
+
This model is intended to be used as it is. We hope that it can prove to be useful to somebody but we do not guarantee that the model is useful for anything or that the predictions are accurate on new data.
|
| 54 |
+
|
| 55 |
+
## Citation information
|
| 56 |
+
If you use this model, please cite:
|
| 57 |
+
|
| 58 |
+
```
|
| 59 |
+
@inproceedings{tanvir2021estbert,
|
| 60 |
+
title={EstBERT: A Pretrained Language-Specific BERT for Estonian},
|
| 61 |
+
author={Tanvir, Hasan and Kittask, Claudia and Eiche, Sandra and Sirts, Kairit},
|
| 62 |
+
booktitle={Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)},
|
| 63 |
+
pages={11--19},
|
| 64 |
+
year={2021}
|
| 65 |
+
}
|
| 66 |
+
```
|
| 67 |
|
| 68 |
## Training and evaluation data
|
| 69 |
|
| 70 |
+
The model was trained and evaluated on the sentiment categories of the [Estonian Valence corpus](https://figshare.com/articles/dataset/Estonian_Valence_Corpus_Eesti_valentsikorpus/24517054).
|
| 71 |
+
The data was split into train/dev/test parts with 70/10/20 proportions.
|
| 72 |
+
|
| 73 |
+
The Estonian Valence corpus has four sentiment labels:
|
| 74 |
+
- positive
|
| 75 |
+
- negative
|
| 76 |
+
- neutral
|
| 77 |
+
- mixed
|
| 78 |
+
|
| 79 |
+
Following [Pajupuu et al., 2016](https://www.researchgate.net/profile/Hille-Pajupuu/publication/303837298_Identifying_Polarity_in_Different_Text_Types/links/575711e308ae05c1ec16ce05/Identifying-Polarity-in-Different-Text-Types.pdf), the items with mixed labels were removed.
|
| 80 |
+
Thus, the model was trained and evaluated on the reduced version of the dataset containing only three labels (positive, negative and neutral).
|
| 81 |
+
|
| 82 |
|
| 83 |
## Training procedure
|
| 84 |
+
The model was trained for maximu 100 epochs using early stopping procedure. After every epoch, the accuracy was calculated on the development set.
|
| 85 |
+
If the development set accuracy did not improve for 20 epochs, the training was stopped.
|
| 86 |
|
| 87 |
### Training hyperparameters
|
| 88 |
|
|
|
|
| 99 |
- mixed_precision_training: Native AMP
|
| 100 |
|
| 101 |
### Training results
|
| 102 |
+
The final model was taken after 44th epoch.
|
| 103 |
|
| 104 |
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|
| 105 |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
|
| 106 |
+
| 0.836 | 1 | 38 | 0.6966 | 0.7216 |
|
| 107 |
+
| 0.5336 | 2 | 76 | 0.5948 | 0.7699 |
|
| 108 |
+
| 0.2913 | 3 | 114 | 0.7197 | 0.7358 |
|
| 109 |
+
| 0.1048 | 4 | 152 | 0.9570 | 0.7557 |
|
| 110 |
+
| 0.0424 | 5 | 190 | 1.2144 | 0.7528 |
|
| 111 |
+
| 0.0262 | 6 | 228 | 1.2675 | 0.7727 |
|
| 112 |
+
| 0.0169 | 7 | 266 | 1.4788 | 0.75 |
|
| 113 |
+
| 0.0048 | 8 | 304 | 1.5053 | 0.7699 |
|
| 114 |
+
| 0.0084 | 9 | 342 | 1.5368 | 0.7614 |
|
| 115 |
+
| 0.0087 | 10 | 380 | 1.6678 | 0.7699 |
|
| 116 |
+
| 0.0082 | 11 | 418 | 1.7598 | 0.7642 |
|
| 117 |
+
| 0.0104 | 12 | 456 | 1.6951 | 0.7528 |
|
| 118 |
+
| 0.0115 | 13 | 494 | 1.7123 | 0.7727 |
|
| 119 |
+
| 0.0111 | 14 | 532 | 1.7577 | 0.7528 |
|
| 120 |
+
| 0.0028 | 15 | 570 | 1.7383 | 0.7727 |
|
| 121 |
+
| 0.0032 | 16 | 608 | 2.0254 | 0.7727 |
|
| 122 |
+
| 0.0107 | 17 | 646 | 2.2123 | 0.7415 |
|
| 123 |
+
| 0.0056 | 18 | 684 | 1.9406 | 0.7614 |
|
| 124 |
+
| 0.0078 | 19 | 722 | 2.2002 | 0.7642 |
|
| 125 |
+
| 0.0041 | 20 | 760 | 2.0157 | 0.7670 |
|
| 126 |
+
| 0.0087 | 21 | 798 | 2.1228 | 0.7642 |
|
| 127 |
+
| 0.0113 | 22 | 836 | 2.3692 | 0.7727 |
|
| 128 |
+
| 0.0025 | 23 | 874 | 2.2211 | 0.75 |
|
| 129 |
+
| 0.0083 | 24 | 912 | 2.2120 | 0.7841 |
|
| 130 |
+
| 0.0104 | 25 | 950 | 2.1478 | 0.7614 |
|
| 131 |
+
| 0.0041 | 26 | 988 | 2.1118 | 0.7756 |
|
| 132 |
+
| 0.002 | 27 | 1026 | 1.9929 | 0.7699 |
|
| 133 |
+
| 0.001 | 28 | 1064 | 2.0295 | 0.7841 |
|
| 134 |
+
| 0.003 | 29 | 1102 | 2.3142 | 0.7699 |
|
| 135 |
+
| 0.006 | 30 | 1140 | 2.2957 | 0.7642 |
|
| 136 |
+
| 0.0005 | 31 | 1178 | 2.0661 | 0.7642 |
|
| 137 |
+
| 0.0007 | 32 | 1216 | 2.4220 | 0.7614 |
|
| 138 |
+
| 0.0088 | 33 | 1254 | 2.2842 | 0.7614 |
|
| 139 |
+
| 0.0 | 34 | 1292 | 2.4060 | 0.7585 |
|
| 140 |
+
| 0.0 | 35 | 1330 | 2.2088 | 0.7585 |
|
| 141 |
+
| 0.0 | 36 | 1368 | 2.2181 | 0.7614 |
|
| 142 |
+
| 0.0 | 37 | 1406 | 2.2560 | 0.7784 |
|
| 143 |
+
| 0.0 | 38 | 1444 | 2.4803 | 0.7585 |
|
| 144 |
+
| 0.0 | 39 | 1482 | 2.1163 | 0.7812 |
|
| 145 |
+
| 0.0087 | 40 | 1520 | 2.3410 | 0.75 |
|
| 146 |
+
| 0.0021 | 41 | 1558 | 2.3583 | 0.75 |
|
| 147 |
+
| 0.0054 | 42 | 1596 | 2.3546 | 0.7642 |
|
| 148 |
+
| 0.0051 | 43 | 1634 | 2.2295 | 0.7812 |
|
| 149 |
+
| 0.0 | 44 | 1672 | 2.2440 | 0.7926 |
|
| 150 |
+
| 0.0019 | 45 | 1710 | 2.3248 | 0.7784 |
|
| 151 |
+
| 0.0044 | 46 | 1748 | 2.3058 | 0.7841 |
|
| 152 |
+
| 0.0006 | 47 | 1786 | 2.3588 | 0.7784 |
|
| 153 |
+
| 0.0007 | 48 | 1824 | 2.6541 | 0.7670 |
|
| 154 |
+
| 0.0001 | 49 | 1862 | 2.4621 | 0.7614 |
|
| 155 |
+
| 0.0 | 50 | 1900 | 2.4696 | 0.7727 |
|
| 156 |
+
| 0.0 | 51 | 1938 | 2.4981 | 0.7670 |
|
| 157 |
+
| 0.0031 | 52 | 1976 | 2.6702 | 0.7670 |
|
| 158 |
+
| 0.0 | 53 | 2014 | 2.4448 | 0.7756 |
|
| 159 |
+
| 0.0 | 54 | 2052 | 2.4214 | 0.7756 |
|
| 160 |
+
| 0.0 | 55 | 2090 | 2.4308 | 0.7841 |
|
| 161 |
+
| 0.0001 | 56 | 2128 | 2.5869 | 0.7642 |
|
| 162 |
+
| 0.0007 | 57 | 2166 | 2.4803 | 0.7727 |
|
| 163 |
+
| 0.0 | 58 | 2204 | 2.4557 | 0.7784 |
|
| 164 |
+
| 0.0 | 59 | 2242 | 2.4702 | 0.7784 |
|
| 165 |
+
| 0.0 | 60 | 2280 | 2.5165 | 0.7784 |
|
| 166 |
+
| 0.0013 | 61 | 2318 | 2.6322 | 0.7727 |
|
| 167 |
+
| 0.0001 | 62 | 2356 | 2.6253 | 0.7756 |
|
| 168 |
+
| 0.0011 | 63 | 2394 | 2.6303 | 0.7841 |
|
| 169 |
+
| 0.0002 | 64 | 2432 | 2.5646 | 0.7614 |
|
| 170 |
|
| 171 |
|
| 172 |
### Framework versions
|
|
|
|
| 175 |
- Pytorch 1.10.1+cu113
|
| 176 |
- Datasets 1.16.1
|
| 177 |
- Tokenizers 0.10.3
|
| 178 |
+
|
| 179 |
+
### Contact
|
| 180 |
+
Kairit Sirts: kairit.sirts@ut.ee
|