File size: 64,357 Bytes
c9ff422 fc9726c c9ff422 1252ca6 44fc6d6 1252ca6 44fc6d6 1252ca6 44fc6d6 1252ca6 44fc6d6 1252ca6 44fc6d6 1252ca6 44fc6d6 1252ca6 44fc6d6 1252ca6 44fc6d6 1252ca6 44fc6d6 fe2a101 44fc6d6 1252ca6 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 |
---
language: et
license: cc-by-4.0
widget:
- text: "Eesti President on Alar Karis."
---
# Estonian NER model based on EstBERT
This model is a fine-tuned version of [tartuNLP/EstBERT](https://huggingface.co/tartuNLP/EstBERT) on the Estonian NER dataset. The model was trained by tartuNLP, the NLP research group at the institute of Computer Science at the University of Tartu.
It achieves the following results on the test set:
- Loss: 0.3565
- Precision: 0.7612
- Recall: 0.7744
- F1: 0.7678
- Accuracy: 0.9672
The entity-level results are as follows:
| | Precision | Recall | F1 | Number |
|---------| --------- | ------- | ------- | ------- |
| DATE | 0.7278 | 0.7258 | 0.7268 | 372 |
| EVENT | 0.3721 | 0.5714 | 0.4507 | 28 |
| GPE | 0.8679 | 0.8369 | 0.8521 | 840 |
| LOC | 0.6545 | 0.4832 | 0.5560 | 149 |
| MONEY | 0.6625 | 0.6023 | 0.6310 | 88 |
| ORG | 0.6761 | 0.7267 | 0.7005 | 589 |
| PER | 0.8255 | 0.9068 | 0.8642 | 751 |
| PERCENT | 1.0 | 0.9589 | 0.9790 | 73 |
| PROD | 0.6030 | 0.5430 | 0.5714 | 221 |
| TIME | 0.5682 | 0.5556 | 0.5618 | 45 |
| TITLE | 0.7 | 0.8063 | 0.7494 | 191 |
## How to use
You can use this model with Transformers pipeline for NER. Post-processing of results may be necessary as the model occasionally tags subword tokens as entities.
```
from transformers import BertTokenizer, BertForTokenClassification
from transformers import pipeline
tokenizer = BertTokenizer.from_pretrained('tartuNLP/EstBERT_NER')
bertner = BertForTokenClassification.from_pretrained('tartuNLP/EstBERT_NER')
nlp = pipeline("ner", model=bertner, tokenizer=tokenizer)
text = "Kaia Kanepi (WTA 57.) langes USA-s Charlestonis toimuval WTA 500 kategooria tenniseturniiril konkurentsist kaheksandikfinaalis, kaotades poolatarile Magda Linette'ile (WTA 64.) 3 : 6, 6 : 4, 2 : 6."
ner_results = nlp(text)
tokens=tokenizer(text)
tokens=tokenizer.convert_ids_to_tokens(tokens['input_ids'])
print(f'tokens: {tokens}')
print(f'NER model:{ner_results}')
```
```
tokens: ['[CLS]', 'kai', '##a', 'kanepi', '(', 'w', '##ta', '57', '.', ')', 'langes', 'usa', '-', 's', 'cha', '##rl', '##est', '##onis', 'toimuval', 'w', '##ta', '500', 'kategooria', 'tennise', '##turniiril', 'konkurentsist', 'kaheksandik', '##finaalis', ',', 'kaotades', 'poola', '##tari', '##le', 'ma', '##gda', 'line', '##tte', "'", 'ile', '(', 'w', '##ta', '64', '.', ')', '3', ':', '6', ',', '6', ':', '4', ',', '2', ':', '6', '.', '[SEP]']
```
```
NER model: [{'entity': 'B-PER', 'score': 0.99999887, 'index': 1, 'word': 'kai', 'start': None, 'end': None}, {'entity': 'B-PER', 'score': 0.97371966, 'index': 2, 'word': '##a', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.99999815, 'index': 3, 'word': 'kanepi', 'start': None, 'end': None}, {'entity': 'B-ORG', 'score': 0.63085276, 'index': 5, 'word': 'w', 'start': None, 'end': None}, {'entity': 'B-GPE', 'score': 0.99999934, 'index': 11, 'word': 'usa', 'start': None, 'end': None}, {'entity': 'B-GPE', 'score': 0.9999685, 'index': 14, 'word': 'cha', 'start': None, 'end': None}, {'entity': 'I-GPE', 'score': 0.8875574, 'index': 15, 'word': '##rl', 'start': None, 'end': None}, {'entity': 'I-GPE', 'score': 0.9996168, 'index': 16, 'word': '##est', 'start': None, 'end': None}, {'entity': 'I-GPE', 'score': 0.9992657, 'index': 17, 'word': '##onis', 'start': None, 'end': None}, {'entity': 'B-EVENT', 'score': 0.99999064, 'index': 19, 'word': 'w', 'start': None, 'end': None}, {'entity': 'I-EVENT', 'score': 0.9772493, 'index': 20, 'word': '##ta', 'start': None, 'end': None}, {'entity': 'I-EVENT', 'score': 0.99999076, 'index': 21, 'word': '500', 'start': None, 'end': None}, {'entity': 'I-EVENT', 'score': 0.99955636, 'index': 22, 'word': 'kategooria', 'start': None, 'end': None}, {'entity': 'B-TITLE', 'score': 0.8771319, 'index': 30, 'word': 'poola', 'start': None, 'end': None}, {'entity': 'B-PER', 'score': 0.99999785, 'index': 33, 'word': 'ma', 'start': None, 'end': None}, {'entity': 'B-PER', 'score': 0.9998398, 'index': 34, 'word': '##gda', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.9999987, 'index': 35, 'word': 'line', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.9999976, 'index': 36, 'word': '##tte', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.99999285, 'index': 37, 'word': "'", 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.9999794, 'index': 38, 'word': 'ile', 'start': None, 'end': None}, {'entity': 'B-ORG', 'score': 0.7664479, 'index': 40, 'word': 'w', 'start': None, 'end': None}]
```
## Intended uses & limitations
This model can be used to find named entities from Estonian texts. The model is free to use for anyone. TartuNLP does not guarantee that the model is useful for anyone or anything. TartuNLP is not responsible for any results it generates.
## Training and evaluation data
The model was trained on two Estonian NER datasets:
- [The Reannotated Estonian NER corpus](https://metashare.ut.ee/repository/browse/reannotated-estonian-ner-corpus/bd43f1f614a511eca6e4fa163e9d45477d086613d2894fd5af79bf13e3f13594/)
- [The New Estonian NER corpus](https://metashare.ut.ee/repository/browse/new-estonian-ner-corpus/98b6706c963c11eba6e4fa163e9d45470bcd0533b6994c93ab8b8c628516ffed/)
Both datasets have been annotated with the same annotation scheme. For training this model, the datasets were joined.
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 1024
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
- lr_scheduler_type: polynomial
- max num_epochs: 150
- early stopping limit: 20
- early stopping tol: 0.0001
- mixed_precision_training: Native AMP
### Training results
The final model was saved after epoch 53 (shown in bold) where the overall F1 was the highest on the development set.
| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1 | Accuracy | Date Precision | Date Recall | Date F1 | Date Number | Event Precision | Event Recall | Event F1 | Event Number | Gpe Precision | Gpe Recall | Gpe F1 | Gpe Number | Loc Precision | Loc Recall | Loc F1 | Loc Number | Money Precision | Money Recall | Money F1 | Money Number | Org Precision | Org Recall | Org F1 | Org Number | Per Precision | Per Recall | Per F1 | Per Number | Percent Precision | Percent Recall | Percent F1 | Percent Number | Prod Precision | Prod Recall | Prod F1 | Prod Number | Time Precision | Time Recall | Time F1 | Time Number | Title Precision | Title Recall | Title F1 | Title Number | Overall Precision | Overall Recall | Overall F1 | Overall Accuracy |
|:-------------:|:-----:|:-----:|:---------------:|:---------:|:------:|:------:|:--------:|:--------------:|:-----------:|:-------:|:-----------:|:---------------:|:------------:|:--------:|:------------:|:-------------:|:----------:|:------:|:----------:|:-------------:|:----------:|:------:|:----------:|:---------------:|:------------:|:--------:|:------------:|:-------------:|:----------:|:------:|:----------:|:-------------:|:----------:|:------:|:----------:|:-----------------:|:--------------:|:----------:|:--------------:|:--------------:|:-----------:|:-------:|:-----------:|:--------------:|:-----------:|:-------:|:-----------:|:---------------:|:------------:|:--------:|:------------:|:-----------------:|:--------------:|:----------:|:----------------:|
| 0.3252 | 1 | 1061 | 0.1628 | 0.6835 | 0.6083 | 0.6437 | 0.9526 | 0.5910 | 0.6022 | 0.5965 | 372 | 0.0 | 0.0 | 0.0 | 28 | 0.8073 | 0.7631 | 0.7846 | 840 | 0.1389 | 0.0336 | 0.0541 | 149 | 0.4217 | 0.3977 | 0.4094 | 88 | 0.5381 | 0.5280 | 0.5330 | 589 | 0.7917 | 0.8655 | 0.8270 | 751 | 0.6471 | 0.3014 | 0.4112 | 73 | 0.2581 | 0.0724 | 0.1131 | 221 | 0.1429 | 0.0889 | 0.1096 | 45 | 0.7805 | 0.6702 | 0.7211 | 191 | 0.6835 | 0.6083 | 0.6437 | 0.9526 |
| 0.1513 | 2 | 2122 | 0.1332 | 0.6906 | 0.7329 | 0.7111 | 0.9615 | 0.6185 | 0.7366 | 0.6724 | 372 | 0.0857 | 0.1071 | 0.0952 | 28 | 0.7874 | 0.8595 | 0.8219 | 840 | 0.4767 | 0.2752 | 0.3489 | 149 | 0.6848 | 0.7159 | 0.7000 | 88 | 0.6158 | 0.6231 | 0.6194 | 589 | 0.7770 | 0.9001 | 0.8341 | 751 | 0.9565 | 0.9041 | 0.9296 | 73 | 0.5 | 0.3620 | 0.4199 | 221 | 0.3571 | 0.3333 | 0.3448 | 45 | 0.6033 | 0.7644 | 0.6744 | 191 | 0.6906 | 0.7329 | 0.7111 | 0.9615 |
| 0.1131 | 3 | 3183 | 0.1281 | 0.7224 | 0.7338 | 0.7280 | 0.9638 | 0.7054 | 0.7339 | 0.7194 | 372 | 0.1053 | 0.1429 | 0.1212 | 28 | 0.8013 | 0.85 | 0.8250 | 840 | 0.5476 | 0.3087 | 0.3948 | 149 | 0.6386 | 0.6023 | 0.6199 | 88 | 0.6371 | 0.6469 | 0.6420 | 589 | 0.8235 | 0.8762 | 0.8490 | 751 | 0.9859 | 0.9589 | 0.9722 | 73 | 0.5148 | 0.3937 | 0.4462 | 221 | 0.5116 | 0.4889 | 0.5 | 45 | 0.6245 | 0.7749 | 0.6916 | 191 | 0.7224 | 0.7338 | 0.7280 | 0.9638 |
| 0.0884 | 4 | 4244 | 0.1354 | 0.7283 | 0.7386 | 0.7334 | 0.9639 | 0.6785 | 0.6694 | 0.6739 | 372 | 0.1795 | 0.25 | 0.2090 | 28 | 0.8231 | 0.8310 | 0.8270 | 840 | 0.6020 | 0.3960 | 0.4777 | 149 | 0.6092 | 0.6023 | 0.6057 | 88 | 0.6473 | 0.7012 | 0.6732 | 589 | 0.8351 | 0.8628 | 0.8487 | 751 | 1.0 | 0.9726 | 0.9861 | 73 | 0.5899 | 0.4751 | 0.5263 | 221 | 0.4524 | 0.4222 | 0.4368 | 45 | 0.6 | 0.7853 | 0.6803 | 191 | 0.7283 | 0.7386 | 0.7334 | 0.9639 |
| 0.0685 | 5 | 5305 | 0.1383 | 0.7224 | 0.7696 | 0.7453 | 0.9644 | 0.6635 | 0.7473 | 0.7029 | 372 | 0.26 | 0.4643 | 0.3333 | 28 | 0.8259 | 0.8357 | 0.8308 | 840 | 0.5913 | 0.4564 | 0.5152 | 149 | 0.6437 | 0.6364 | 0.64 | 88 | 0.6540 | 0.7284 | 0.6892 | 589 | 0.8070 | 0.8961 | 0.8492 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.5693 | 0.5204 | 0.5437 | 221 | 0.5192 | 0.6 | 0.5567 | 45 | 0.6320 | 0.7644 | 0.6919 | 191 | 0.7224 | 0.7696 | 0.7453 | 0.9644 |
| 0.0532 | 6 | 6366 | 0.1493 | 0.7099 | 0.7613 | 0.7347 | 0.9631 | 0.6727 | 0.6962 | 0.6843 | 372 | 0.2308 | 0.5357 | 0.3226 | 28 | 0.8242 | 0.8262 | 0.8252 | 840 | 0.5877 | 0.4497 | 0.5095 | 149 | 0.6410 | 0.5682 | 0.6024 | 88 | 0.6232 | 0.7470 | 0.6795 | 589 | 0.8087 | 0.8895 | 0.8472 | 751 | 0.9672 | 0.8082 | 0.8806 | 73 | 0.5107 | 0.5385 | 0.5242 | 221 | 0.6190 | 0.5778 | 0.5977 | 45 | 0.6371 | 0.7906 | 0.7056 | 191 | 0.7099 | 0.7613 | 0.7347 | 0.9631 |
| 0.0403 | 7 | 7427 | 0.1592 | 0.7239 | 0.7592 | 0.7411 | 0.9642 | 0.6923 | 0.7016 | 0.6969 | 372 | 0.2857 | 0.5714 | 0.3810 | 28 | 0.8272 | 0.8262 | 0.8267 | 840 | 0.5752 | 0.4362 | 0.4962 | 149 | 0.6265 | 0.5909 | 0.6082 | 88 | 0.6402 | 0.6978 | 0.6677 | 589 | 0.8404 | 0.8762 | 0.8579 | 751 | 0.9859 | 0.9589 | 0.9722 | 73 | 0.5257 | 0.6018 | 0.5612 | 221 | 0.5870 | 0.6 | 0.5934 | 45 | 0.6235 | 0.8063 | 0.7032 | 191 | 0.7239 | 0.7592 | 0.7411 | 0.9642 |
| 0.0304 | 8 | 8488 | 0.1738 | 0.7301 | 0.7484 | 0.7392 | 0.9644 | 0.6866 | 0.6774 | 0.6820 | 372 | 0.3409 | 0.5357 | 0.4167 | 28 | 0.8393 | 0.8083 | 0.8235 | 840 | 0.5882 | 0.4698 | 0.5224 | 149 | 0.6429 | 0.6136 | 0.6279 | 88 | 0.6608 | 0.6978 | 0.6788 | 589 | 0.8268 | 0.8708 | 0.8482 | 751 | 0.9595 | 0.9726 | 0.9660 | 73 | 0.5351 | 0.5520 | 0.5434 | 221 | 0.5208 | 0.5556 | 0.5376 | 45 | 0.6204 | 0.7958 | 0.6972 | 191 | 0.7301 | 0.7484 | 0.7392 | 0.9644 |
| 0.0234 | 9 | 9549 | 0.1860 | 0.7248 | 0.7625 | 0.7432 | 0.9641 | 0.6947 | 0.7097 | 0.7021 | 372 | 0.2963 | 0.5714 | 0.3902 | 28 | 0.8317 | 0.8298 | 0.8308 | 840 | 0.5913 | 0.4564 | 0.5152 | 149 | 0.6118 | 0.5909 | 0.6012 | 88 | 0.6361 | 0.7063 | 0.6693 | 589 | 0.8410 | 0.8735 | 0.8570 | 751 | 0.9859 | 0.9589 | 0.9722 | 73 | 0.5212 | 0.6109 | 0.5625 | 221 | 0.5417 | 0.5778 | 0.5591 | 45 | 0.6414 | 0.7958 | 0.7103 | 191 | 0.7248 | 0.7625 | 0.7432 | 0.9641 |
| 0.0178 | 10 | 10610 | 0.2037 | 0.7434 | 0.7383 | 0.7408 | 0.9640 | 0.7159 | 0.6774 | 0.6961 | 372 | 0.2857 | 0.4286 | 0.3429 | 28 | 0.8333 | 0.8333 | 0.8333 | 840 | 0.6262 | 0.4497 | 0.5234 | 149 | 0.6324 | 0.4886 | 0.5513 | 88 | 0.6568 | 0.6757 | 0.6661 | 589 | 0.8291 | 0.8722 | 0.8501 | 751 | 1.0 | 0.8219 | 0.9023 | 73 | 0.5672 | 0.5158 | 0.5403 | 221 | 0.5 | 0.5333 | 0.5161 | 45 | 0.6952 | 0.7644 | 0.7282 | 191 | 0.7434 | 0.7383 | 0.7408 | 0.9640 |
| 0.0147 | 11 | 11671 | 0.2114 | 0.7440 | 0.7233 | 0.7335 | 0.9643 | 0.7009 | 0.6613 | 0.6805 | 372 | 0.3030 | 0.3571 | 0.3279 | 28 | 0.8352 | 0.8024 | 0.8185 | 840 | 0.6238 | 0.4228 | 0.504 | 149 | 0.65 | 0.5909 | 0.6190 | 88 | 0.6436 | 0.6469 | 0.6452 | 589 | 0.8407 | 0.8575 | 0.8490 | 751 | 0.9315 | 0.9315 | 0.9315 | 73 | 0.5812 | 0.5023 | 0.5388 | 221 | 0.5476 | 0.5111 | 0.5287 | 45 | 0.6835 | 0.7801 | 0.7286 | 191 | 0.7440 | 0.7233 | 0.7335 | 0.9643 |
| 0.0118 | 12 | 12732 | 0.2218 | 0.7331 | 0.7532 | 0.7430 | 0.9649 | 0.7119 | 0.6909 | 0.7012 | 372 | 0.3488 | 0.5357 | 0.4225 | 28 | 0.8325 | 0.8405 | 0.8365 | 840 | 0.5303 | 0.4698 | 0.4982 | 149 | 0.65 | 0.5909 | 0.6190 | 88 | 0.6690 | 0.6587 | 0.6638 | 589 | 0.8178 | 0.8908 | 0.8528 | 751 | 0.9677 | 0.8219 | 0.8889 | 73 | 0.5408 | 0.5701 | 0.5551 | 221 | 0.5102 | 0.5556 | 0.5319 | 45 | 0.6567 | 0.8010 | 0.7217 | 191 | 0.7331 | 0.7532 | 0.7430 | 0.9649 |
| 0.0093 | 13 | 13793 | 0.2283 | 0.7495 | 0.7359 | 0.7427 | 0.9644 | 0.7163 | 0.6989 | 0.7075 | 372 | 0.3810 | 0.5714 | 0.4571 | 28 | 0.8612 | 0.7905 | 0.8243 | 840 | 0.6111 | 0.4430 | 0.5136 | 149 | 0.6145 | 0.5795 | 0.5965 | 88 | 0.6775 | 0.6740 | 0.6757 | 589 | 0.8346 | 0.8802 | 0.8568 | 751 | 0.9710 | 0.9178 | 0.9437 | 73 | 0.5619 | 0.5339 | 0.5476 | 221 | 0.4 | 0.4889 | 0.4400 | 45 | 0.6812 | 0.7382 | 0.7085 | 191 | 0.7495 | 0.7359 | 0.7427 | 0.9644 |
| 0.0079 | 14 | 14854 | 0.2383 | 0.7371 | 0.7490 | 0.7430 | 0.9647 | 0.6727 | 0.7016 | 0.6868 | 372 | 0.3261 | 0.5357 | 0.4054 | 28 | 0.8453 | 0.8 | 0.8220 | 840 | 0.5963 | 0.4362 | 0.5039 | 149 | 0.625 | 0.5682 | 0.5952 | 88 | 0.6634 | 0.6927 | 0.6777 | 589 | 0.8433 | 0.8815 | 0.8620 | 751 | 0.9853 | 0.9178 | 0.9504 | 73 | 0.5427 | 0.5747 | 0.5582 | 221 | 0.5814 | 0.5556 | 0.5682 | 45 | 0.6513 | 0.8115 | 0.7226 | 191 | 0.7371 | 0.7490 | 0.7430 | 0.9647 |
| 0.0068 | 15 | 15915 | 0.2511 | 0.7255 | 0.7359 | 0.7306 | 0.9639 | 0.6826 | 0.6532 | 0.6676 | 372 | 0.3590 | 0.5 | 0.4179 | 28 | 0.8295 | 0.8167 | 0.8230 | 840 | 0.5263 | 0.4698 | 0.4965 | 149 | 0.6575 | 0.5455 | 0.5963 | 88 | 0.6549 | 0.6604 | 0.6577 | 589 | 0.8242 | 0.8802 | 0.8513 | 751 | 0.9833 | 0.8082 | 0.8872 | 73 | 0.5398 | 0.5520 | 0.5459 | 221 | 0.36 | 0.4 | 0.3789 | 45 | 0.6511 | 0.8010 | 0.7183 | 191 | 0.7255 | 0.7359 | 0.7306 | 0.9639 |
| 0.0061 | 16 | 16976 | 0.2497 | 0.7253 | 0.7690 | 0.7465 | 0.9648 | 0.6824 | 0.6989 | 0.6906 | 372 | 0.3333 | 0.5357 | 0.4110 | 28 | 0.8473 | 0.8321 | 0.8396 | 840 | 0.4583 | 0.5168 | 0.4858 | 149 | 0.6494 | 0.5682 | 0.6061 | 88 | 0.6556 | 0.7368 | 0.6938 | 589 | 0.8382 | 0.8828 | 0.8599 | 751 | 0.9841 | 0.8493 | 0.9118 | 73 | 0.5341 | 0.6380 | 0.5814 | 221 | 0.5 | 0.5333 | 0.5161 | 45 | 0.6622 | 0.7801 | 0.7163 | 191 | 0.7253 | 0.7690 | 0.7465 | 0.9648 |
| 0.0054 | 17 | 18037 | 0.2554 | 0.7323 | 0.7625 | 0.7471 | 0.9650 | 0.6870 | 0.6962 | 0.6916 | 372 | 0.3421 | 0.4643 | 0.3939 | 28 | 0.8463 | 0.8262 | 0.8361 | 840 | 0.5902 | 0.4832 | 0.5314 | 149 | 0.6753 | 0.5909 | 0.6303 | 88 | 0.6640 | 0.7148 | 0.6885 | 589 | 0.8317 | 0.8948 | 0.8621 | 751 | 0.9437 | 0.9178 | 0.9306 | 73 | 0.5210 | 0.5611 | 0.5403 | 221 | 0.5 | 0.5111 | 0.5055 | 45 | 0.6102 | 0.8115 | 0.6966 | 191 | 0.7323 | 0.7625 | 0.7471 | 0.9650 |
| 0.005 | 18 | 19098 | 0.2601 | 0.7273 | 0.7747 | 0.7503 | 0.9654 | 0.6970 | 0.7608 | 0.7275 | 372 | 0.2830 | 0.5357 | 0.3704 | 28 | 0.8320 | 0.8488 | 0.8403 | 840 | 0.5841 | 0.4430 | 0.5038 | 149 | 0.6477 | 0.6477 | 0.6477 | 88 | 0.6378 | 0.6995 | 0.6672 | 589 | 0.8501 | 0.8908 | 0.8700 | 751 | 0.9722 | 0.9589 | 0.9655 | 73 | 0.5323 | 0.5973 | 0.5629 | 221 | 0.4444 | 0.4444 | 0.4444 | 45 | 0.624 | 0.8168 | 0.7075 | 191 | 0.7273 | 0.7747 | 0.7503 | 0.9654 |
| 0.0044 | 19 | 20159 | 0.2602 | 0.7369 | 0.7616 | 0.7490 | 0.9656 | 0.7124 | 0.7124 | 0.7124 | 372 | 0.3415 | 0.5 | 0.4058 | 28 | 0.8239 | 0.8631 | 0.8430 | 840 | 0.6355 | 0.4564 | 0.5313 | 149 | 0.6667 | 0.6136 | 0.6391 | 88 | 0.6517 | 0.6638 | 0.6577 | 589 | 0.8405 | 0.8842 | 0.8618 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.5144 | 0.5656 | 0.5388 | 221 | 0.5217 | 0.5333 | 0.5275 | 45 | 0.6550 | 0.7853 | 0.7143 | 191 | 0.7369 | 0.7616 | 0.7490 | 0.9656 |
| 0.004 | 20 | 21220 | 0.2677 | 0.7347 | 0.7702 | 0.7520 | 0.9658 | 0.7374 | 0.7097 | 0.7233 | 372 | 0.2857 | 0.4286 | 0.3429 | 28 | 0.8466 | 0.8345 | 0.8405 | 840 | 0.6050 | 0.4832 | 0.5373 | 149 | 0.6667 | 0.6136 | 0.6391 | 88 | 0.6593 | 0.7131 | 0.6852 | 589 | 0.8240 | 0.8975 | 0.8591 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.4981 | 0.5837 | 0.5375 | 221 | 0.5102 | 0.5556 | 0.5319 | 45 | 0.6371 | 0.8272 | 0.7198 | 191 | 0.7347 | 0.7702 | 0.7520 | 0.9658 |
| 0.0034 | 21 | 22281 | 0.2743 | 0.7386 | 0.7717 | 0.7548 | 0.9657 | 0.6984 | 0.7097 | 0.704 | 372 | 0.3784 | 0.5 | 0.4308 | 28 | 0.8475 | 0.8333 | 0.8403 | 840 | 0.6333 | 0.5101 | 0.5651 | 149 | 0.6190 | 0.5909 | 0.6047 | 88 | 0.6512 | 0.7385 | 0.6921 | 589 | 0.8428 | 0.8921 | 0.8668 | 751 | 0.9846 | 0.8767 | 0.9275 | 73 | 0.5513 | 0.5837 | 0.5670 | 221 | 0.5106 | 0.5333 | 0.5217 | 45 | 0.6379 | 0.8115 | 0.7143 | 191 | 0.7386 | 0.7717 | 0.7548 | 0.9657 |
| 0.0033 | 22 | 23342 | 0.2788 | 0.7418 | 0.7520 | 0.7469 | 0.9652 | 0.7143 | 0.6989 | 0.7065 | 372 | 0.3182 | 0.5 | 0.3889 | 28 | 0.8367 | 0.8298 | 0.8332 | 840 | 0.6168 | 0.4430 | 0.5156 | 149 | 0.6235 | 0.6023 | 0.6127 | 88 | 0.6758 | 0.6689 | 0.6724 | 589 | 0.8327 | 0.8815 | 0.8564 | 751 | 0.9714 | 0.9315 | 0.9510 | 73 | 0.5458 | 0.5928 | 0.5683 | 221 | 0.4783 | 0.4889 | 0.4835 | 45 | 0.6637 | 0.7853 | 0.7194 | 191 | 0.7418 | 0.7520 | 0.7469 | 0.9652 |
| 0.0033 | 23 | 24403 | 0.2831 | 0.7342 | 0.7535 | 0.7437 | 0.9650 | 0.6981 | 0.6962 | 0.6972 | 372 | 0.3784 | 0.5 | 0.4308 | 28 | 0.8499 | 0.8024 | 0.8255 | 840 | 0.5034 | 0.4966 | 0.5 | 149 | 0.6067 | 0.6136 | 0.6102 | 88 | 0.6581 | 0.6961 | 0.6766 | 589 | 0.8350 | 0.8961 | 0.8645 | 751 | 0.9714 | 0.9315 | 0.9510 | 73 | 0.5424 | 0.5792 | 0.5602 | 221 | 0.3774 | 0.4444 | 0.4082 | 45 | 0.7048 | 0.7749 | 0.7382 | 191 | 0.7342 | 0.7535 | 0.7437 | 0.9650 |
| 0.0029 | 24 | 25464 | 0.2931 | 0.7544 | 0.7380 | 0.7461 | 0.9648 | 0.7365 | 0.6989 | 0.7172 | 372 | 0.3590 | 0.5 | 0.4179 | 28 | 0.8535 | 0.7976 | 0.8246 | 840 | 0.5849 | 0.4161 | 0.4863 | 149 | 0.6622 | 0.5568 | 0.6049 | 88 | 0.6672 | 0.6706 | 0.6689 | 589 | 0.8474 | 0.8802 | 0.8635 | 751 | 0.9701 | 0.8904 | 0.9286 | 73 | 0.5550 | 0.5475 | 0.5513 | 221 | 0.4889 | 0.4889 | 0.4889 | 45 | 0.7023 | 0.7906 | 0.7438 | 191 | 0.7544 | 0.7380 | 0.7461 | 0.9648 |
| 0.0028 | 25 | 26525 | 0.2899 | 0.7489 | 0.7574 | 0.7531 | 0.9654 | 0.7021 | 0.7097 | 0.7059 | 372 | 0.3902 | 0.5714 | 0.4638 | 28 | 0.8635 | 0.8131 | 0.8375 | 840 | 0.6182 | 0.4564 | 0.5251 | 149 | 0.6471 | 0.625 | 0.6358 | 88 | 0.6613 | 0.6995 | 0.6799 | 589 | 0.8454 | 0.9028 | 0.8731 | 751 | 0.9583 | 0.9452 | 0.9517 | 73 | 0.5681 | 0.5475 | 0.5576 | 221 | 0.4222 | 0.4222 | 0.4222 | 45 | 0.6608 | 0.7853 | 0.7177 | 191 | 0.7489 | 0.7574 | 0.7531 | 0.9654 |
| 0.0023 | 26 | 27586 | 0.2922 | 0.7413 | 0.7532 | 0.7472 | 0.9649 | 0.6897 | 0.6989 | 0.6943 | 372 | 0.35 | 0.5 | 0.4118 | 28 | 0.85 | 0.8298 | 0.8398 | 840 | 0.6161 | 0.4631 | 0.5287 | 149 | 0.6486 | 0.5455 | 0.5926 | 88 | 0.6486 | 0.6927 | 0.6700 | 589 | 0.8457 | 0.8828 | 0.8638 | 751 | 0.9853 | 0.9178 | 0.9504 | 73 | 0.5636 | 0.5611 | 0.5624 | 221 | 0.3958 | 0.4222 | 0.4086 | 45 | 0.6638 | 0.7958 | 0.7238 | 191 | 0.7413 | 0.7532 | 0.7472 | 0.9649 |
| 0.0021 | 27 | 28647 | 0.2967 | 0.7514 | 0.7568 | 0.7541 | 0.9656 | 0.7081 | 0.7043 | 0.7062 | 372 | 0.3659 | 0.5357 | 0.4348 | 28 | 0.8547 | 0.8190 | 0.8365 | 840 | 0.5641 | 0.4430 | 0.4962 | 149 | 0.6582 | 0.5909 | 0.6228 | 88 | 0.6677 | 0.7097 | 0.6881 | 589 | 0.8459 | 0.8842 | 0.8646 | 751 | 0.9710 | 0.9178 | 0.9437 | 73 | 0.5806 | 0.5701 | 0.5753 | 221 | 0.4898 | 0.5333 | 0.5106 | 45 | 0.7089 | 0.7906 | 0.7475 | 191 | 0.7514 | 0.7568 | 0.7541 | 0.9656 |
| 0.0025 | 28 | 29708 | 0.2957 | 0.7335 | 0.7622 | 0.7475 | 0.9651 | 0.7060 | 0.7231 | 0.7145 | 372 | 0.3077 | 0.4286 | 0.3582 | 28 | 0.8459 | 0.8429 | 0.8444 | 840 | 0.5069 | 0.4899 | 0.4983 | 149 | 0.6438 | 0.5341 | 0.5839 | 88 | 0.6838 | 0.7012 | 0.6924 | 589 | 0.8413 | 0.8895 | 0.8647 | 751 | 0.9552 | 0.8767 | 0.9143 | 73 | 0.4901 | 0.5611 | 0.5232 | 221 | 0.3818 | 0.4667 | 0.42 | 45 | 0.6580 | 0.7958 | 0.7204 | 191 | 0.7335 | 0.7622 | 0.7475 | 0.9651 |
| 0.0023 | 29 | 30769 | 0.3049 | 0.7455 | 0.7544 | 0.7499 | 0.9654 | 0.6997 | 0.7392 | 0.7190 | 372 | 0.3182 | 0.5 | 0.3889 | 28 | 0.8483 | 0.8119 | 0.8297 | 840 | 0.5630 | 0.5101 | 0.5352 | 149 | 0.6579 | 0.5682 | 0.6098 | 88 | 0.6791 | 0.7114 | 0.6949 | 589 | 0.8583 | 0.8628 | 0.8606 | 751 | 0.9853 | 0.9178 | 0.9504 | 73 | 0.5234 | 0.5566 | 0.5395 | 221 | 0.4565 | 0.4667 | 0.4615 | 45 | 0.7009 | 0.7853 | 0.7407 | 191 | 0.7455 | 0.7544 | 0.7499 | 0.9654 |
| 0.0018 | 30 | 31830 | 0.3042 | 0.7415 | 0.7679 | 0.7544 | 0.9654 | 0.6935 | 0.7419 | 0.7169 | 372 | 0.3333 | 0.5 | 0.4 | 28 | 0.8563 | 0.8226 | 0.8391 | 840 | 0.5878 | 0.5168 | 0.55 | 149 | 0.6582 | 0.5909 | 0.6228 | 88 | 0.6677 | 0.7470 | 0.7051 | 589 | 0.8544 | 0.8828 | 0.8684 | 751 | 0.9710 | 0.9178 | 0.9437 | 73 | 0.5300 | 0.5204 | 0.5251 | 221 | 0.4375 | 0.4667 | 0.4516 | 45 | 0.6417 | 0.8063 | 0.7146 | 191 | 0.7415 | 0.7679 | 0.7544 | 0.9654 |
| 0.0017 | 31 | 32891 | 0.3071 | 0.7540 | 0.7481 | 0.7510 | 0.9660 | 0.7083 | 0.7312 | 0.7196 | 372 | 0.4054 | 0.5357 | 0.4615 | 28 | 0.8552 | 0.8226 | 0.8386 | 840 | 0.6311 | 0.4362 | 0.5159 | 149 | 0.6220 | 0.5795 | 0.6 | 88 | 0.6734 | 0.6757 | 0.6746 | 589 | 0.8626 | 0.8775 | 0.8700 | 751 | 0.9855 | 0.9315 | 0.9577 | 73 | 0.5307 | 0.5475 | 0.5390 | 221 | 0.3830 | 0.4 | 0.3913 | 45 | 0.7019 | 0.7644 | 0.7318 | 191 | 0.7540 | 0.7481 | 0.7510 | 0.9660 |
| 0.0018 | 32 | 33952 | 0.3190 | 0.7499 | 0.7553 | 0.7526 | 0.9656 | 0.7182 | 0.7124 | 0.7152 | 372 | 0.3333 | 0.5357 | 0.4110 | 28 | 0.8586 | 0.7952 | 0.8257 | 840 | 0.6116 | 0.4966 | 0.5481 | 149 | 0.6463 | 0.6023 | 0.6235 | 88 | 0.6805 | 0.6978 | 0.6890 | 589 | 0.8360 | 0.8895 | 0.8619 | 751 | 0.9855 | 0.9315 | 0.9577 | 73 | 0.5633 | 0.5837 | 0.5733 | 221 | 0.5106 | 0.5333 | 0.5217 | 45 | 0.6711 | 0.8010 | 0.7303 | 191 | 0.7499 | 0.7553 | 0.7526 | 0.9656 |
| 0.0018 | 33 | 35013 | 0.3094 | 0.7460 | 0.7774 | 0.7614 | 0.9665 | 0.7147 | 0.7473 | 0.7306 | 372 | 0.3659 | 0.5357 | 0.4348 | 28 | 0.8556 | 0.8393 | 0.8474 | 840 | 0.6273 | 0.4631 | 0.5328 | 149 | 0.6506 | 0.6136 | 0.6316 | 88 | 0.6787 | 0.7351 | 0.7058 | 589 | 0.8344 | 0.8988 | 0.8654 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.5702 | 0.6063 | 0.5877 | 221 | 0.3036 | 0.3778 | 0.3366 | 45 | 0.6567 | 0.8010 | 0.7217 | 191 | 0.7460 | 0.7774 | 0.7614 | 0.9665 |
| 0.0015 | 34 | 36074 | 0.3091 | 0.7441 | 0.7759 | 0.7597 | 0.9665 | 0.7113 | 0.7285 | 0.7198 | 372 | 0.3404 | 0.5714 | 0.4267 | 28 | 0.8266 | 0.8512 | 0.8387 | 840 | 0.5405 | 0.5369 | 0.5387 | 149 | 0.6707 | 0.625 | 0.6471 | 88 | 0.6856 | 0.7182 | 0.7015 | 589 | 0.8517 | 0.8868 | 0.8689 | 751 | 1.0 | 0.9452 | 0.9718 | 73 | 0.5752 | 0.5882 | 0.5817 | 221 | 0.3878 | 0.4222 | 0.4043 | 45 | 0.6830 | 0.8010 | 0.7373 | 191 | 0.7441 | 0.7759 | 0.7597 | 0.9665 |
| 0.0015 | 35 | 37135 | 0.3185 | 0.7487 | 0.7619 | 0.7552 | 0.9660 | 0.6982 | 0.7339 | 0.7156 | 372 | 0.3415 | 0.5 | 0.4058 | 28 | 0.8685 | 0.8179 | 0.8424 | 840 | 0.5504 | 0.4765 | 0.5108 | 149 | 0.6353 | 0.6136 | 0.6243 | 88 | 0.6636 | 0.7267 | 0.6937 | 589 | 0.8654 | 0.8815 | 0.8734 | 751 | 1.0 | 0.9315 | 0.9645 | 73 | 0.55 | 0.5475 | 0.5488 | 221 | 0.3673 | 0.4 | 0.3830 | 45 | 0.6937 | 0.8063 | 0.7458 | 191 | 0.7487 | 0.7619 | 0.7552 | 0.9660 |
| 0.0015 | 36 | 38196 | 0.3203 | 0.7438 | 0.7649 | 0.7542 | 0.9660 | 0.6961 | 0.7204 | 0.7081 | 372 | 0.3659 | 0.5357 | 0.4348 | 28 | 0.8617 | 0.8381 | 0.8497 | 840 | 0.5203 | 0.5168 | 0.5185 | 149 | 0.6667 | 0.5909 | 0.6265 | 88 | 0.6710 | 0.7063 | 0.6882 | 589 | 0.8495 | 0.8868 | 0.8678 | 751 | 0.9710 | 0.9178 | 0.9437 | 73 | 0.5561 | 0.5385 | 0.5471 | 221 | 0.42 | 0.4667 | 0.4421 | 45 | 0.6568 | 0.8115 | 0.7260 | 191 | 0.7438 | 0.7649 | 0.7542 | 0.9660 |
| 0.0013 | 37 | 39257 | 0.3298 | 0.7315 | 0.7732 | 0.7518 | 0.9656 | 0.6915 | 0.7231 | 0.7070 | 372 | 0.3333 | 0.5714 | 0.4211 | 28 | 0.8654 | 0.8190 | 0.8416 | 840 | 0.4793 | 0.5436 | 0.5094 | 149 | 0.6582 | 0.5909 | 0.6228 | 88 | 0.6656 | 0.7267 | 0.6948 | 589 | 0.8289 | 0.9028 | 0.8642 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.5574 | 0.5928 | 0.5746 | 221 | 0.4043 | 0.4222 | 0.4130 | 45 | 0.6408 | 0.8220 | 0.7202 | 191 | 0.7315 | 0.7732 | 0.7518 | 0.9656 |
| 0.0012 | 38 | 40318 | 0.3311 | 0.7533 | 0.7610 | 0.7571 | 0.9664 | 0.7060 | 0.7231 | 0.7145 | 372 | 0.3571 | 0.5357 | 0.4286 | 28 | 0.8613 | 0.8357 | 0.8483 | 840 | 0.6339 | 0.4765 | 0.5441 | 149 | 0.6543 | 0.6023 | 0.6272 | 88 | 0.6528 | 0.7182 | 0.6839 | 589 | 0.8424 | 0.8828 | 0.8622 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.6031 | 0.5294 | 0.5639 | 221 | 0.4130 | 0.4222 | 0.4176 | 45 | 0.7122 | 0.7644 | 0.7374 | 191 | 0.7533 | 0.7610 | 0.7571 | 0.9664 |
| 0.0012 | 39 | 41379 | 0.3328 | 0.7444 | 0.7553 | 0.7498 | 0.9657 | 0.6818 | 0.7258 | 0.7031 | 372 | 0.3478 | 0.5714 | 0.4324 | 28 | 0.8561 | 0.8143 | 0.8347 | 840 | 0.6055 | 0.4430 | 0.5116 | 149 | 0.6582 | 0.5909 | 0.6228 | 88 | 0.6715 | 0.7046 | 0.6877 | 589 | 0.8461 | 0.8708 | 0.8583 | 751 | 0.9706 | 0.9041 | 0.9362 | 73 | 0.5665 | 0.5973 | 0.5815 | 221 | 0.4082 | 0.4444 | 0.4255 | 45 | 0.6770 | 0.8010 | 0.7338 | 191 | 0.7444 | 0.7553 | 0.7498 | 0.9657 |
| 0.0014 | 40 | 42440 | 0.3415 | 0.7421 | 0.7437 | 0.7429 | 0.9641 | 0.6931 | 0.7043 | 0.6987 | 372 | 0.3488 | 0.5357 | 0.4225 | 28 | 0.8422 | 0.8262 | 0.8341 | 840 | 0.6190 | 0.4362 | 0.5118 | 149 | 0.6622 | 0.5568 | 0.6049 | 88 | 0.6888 | 0.6350 | 0.6608 | 589 | 0.8175 | 0.8828 | 0.8489 | 751 | 1.0 | 0.9178 | 0.9571 | 73 | 0.5584 | 0.5837 | 0.5708 | 221 | 0.4043 | 0.4222 | 0.4130 | 45 | 0.6580 | 0.7958 | 0.7204 | 191 | 0.7421 | 0.7437 | 0.7429 | 0.9641 |
| 0.0013 | 41 | 43501 | 0.3401 | 0.7501 | 0.7487 | 0.7494 | 0.9651 | 0.6915 | 0.7231 | 0.7070 | 372 | 0.3421 | 0.4643 | 0.3939 | 28 | 0.8545 | 0.8179 | 0.8358 | 840 | 0.6346 | 0.4430 | 0.5217 | 149 | 0.6812 | 0.5341 | 0.5987 | 88 | 0.6728 | 0.6808 | 0.6768 | 589 | 0.8380 | 0.8748 | 0.8560 | 751 | 0.9710 | 0.9178 | 0.9437 | 73 | 0.5860 | 0.5701 | 0.5780 | 221 | 0.4423 | 0.5111 | 0.4742 | 45 | 0.6787 | 0.7853 | 0.7282 | 191 | 0.7501 | 0.7487 | 0.7494 | 0.9651 |
| 0.0011 | 42 | 44562 | 0.3468 | 0.7426 | 0.7687 | 0.7554 | 0.9650 | 0.6965 | 0.7527 | 0.7235 | 372 | 0.3488 | 0.5357 | 0.4225 | 28 | 0.8667 | 0.8202 | 0.8428 | 840 | 0.6408 | 0.4430 | 0.5238 | 149 | 0.6709 | 0.6023 | 0.6347 | 88 | 0.6902 | 0.7148 | 0.7023 | 589 | 0.8404 | 0.8975 | 0.8680 | 751 | 0.9444 | 0.9315 | 0.9379 | 73 | 0.5191 | 0.6154 | 0.5631 | 221 | 0.3469 | 0.3778 | 0.3617 | 45 | 0.6210 | 0.8063 | 0.7016 | 191 | 0.7426 | 0.7687 | 0.7554 | 0.9650 |
| 0.0015 | 43 | 45623 | 0.3440 | 0.7566 | 0.7422 | 0.7493 | 0.9648 | 0.6937 | 0.7366 | 0.7145 | 372 | 0.3846 | 0.5357 | 0.4478 | 28 | 0.8608 | 0.8095 | 0.8344 | 840 | 0.6082 | 0.3960 | 0.4797 | 149 | 0.7 | 0.5568 | 0.6203 | 88 | 0.6766 | 0.6570 | 0.6667 | 589 | 0.8317 | 0.8881 | 0.8590 | 751 | 0.9701 | 0.8904 | 0.9286 | 73 | 0.6224 | 0.5520 | 0.5851 | 221 | 0.3913 | 0.4 | 0.3956 | 45 | 0.7081 | 0.7749 | 0.74 | 191 | 0.7566 | 0.7422 | 0.7493 | 0.9648 |
| 0.0011 | 44 | 46684 | 0.3354 | 0.7565 | 0.7640 | 0.7602 | 0.9664 | 0.7062 | 0.7366 | 0.7211 | 372 | 0.3659 | 0.5357 | 0.4348 | 28 | 0.8483 | 0.8452 | 0.8468 | 840 | 0.6095 | 0.4295 | 0.5039 | 149 | 0.6883 | 0.6023 | 0.6424 | 88 | 0.6880 | 0.6740 | 0.6810 | 589 | 0.8517 | 0.8948 | 0.8727 | 751 | 0.9710 | 0.9178 | 0.9437 | 73 | 0.6238 | 0.5928 | 0.6079 | 221 | 0.3830 | 0.4 | 0.3913 | 45 | 0.65 | 0.8168 | 0.7239 | 191 | 0.7565 | 0.7640 | 0.7602 | 0.9664 |
| 0.0011 | 45 | 47745 | 0.3347 | 0.7485 | 0.7622 | 0.7553 | 0.9655 | 0.7088 | 0.7392 | 0.7237 | 372 | 0.3636 | 0.5714 | 0.4444 | 28 | 0.8603 | 0.8286 | 0.8441 | 840 | 0.5882 | 0.4698 | 0.5224 | 149 | 0.6023 | 0.6023 | 0.6023 | 88 | 0.6770 | 0.6689 | 0.6729 | 589 | 0.8417 | 0.8921 | 0.8662 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.6037 | 0.5928 | 0.5982 | 221 | 0.4583 | 0.4889 | 0.4731 | 45 | 0.6275 | 0.8115 | 0.7078 | 191 | 0.7485 | 0.7622 | 0.7553 | 0.9655 |
| 0.0011 | 46 | 48806 | 0.3421 | 0.7481 | 0.7640 | 0.7559 | 0.9657 | 0.7261 | 0.7339 | 0.7299 | 372 | 0.3171 | 0.4643 | 0.3768 | 28 | 0.8570 | 0.8202 | 0.8382 | 840 | 0.5691 | 0.4698 | 0.5147 | 149 | 0.6429 | 0.6136 | 0.6279 | 88 | 0.6769 | 0.7114 | 0.6937 | 589 | 0.8311 | 0.8908 | 0.8599 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.5714 | 0.5611 | 0.5662 | 221 | 0.5 | 0.5556 | 0.5263 | 45 | 0.6638 | 0.7958 | 0.7238 | 191 | 0.7481 | 0.7640 | 0.7559 | 0.9657 |
| 0.0009 | 47 | 49867 | 0.3487 | 0.7496 | 0.7604 | 0.7550 | 0.9656 | 0.7158 | 0.7043 | 0.7100 | 372 | 0.3409 | 0.5357 | 0.4167 | 28 | 0.86 | 0.8190 | 0.8390 | 840 | 0.5496 | 0.4832 | 0.5143 | 149 | 0.7162 | 0.6023 | 0.6543 | 88 | 0.6745 | 0.7284 | 0.7004 | 589 | 0.8346 | 0.8802 | 0.8568 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.5566 | 0.5339 | 0.5450 | 221 | 0.5349 | 0.5111 | 0.5227 | 45 | 0.6828 | 0.8115 | 0.7416 | 191 | 0.7496 | 0.7604 | 0.7550 | 0.9656 |
| 0.0009 | 48 | 50928 | 0.3470 | 0.7414 | 0.7649 | 0.7529 | 0.9651 | 0.7092 | 0.7473 | 0.7277 | 372 | 0.3333 | 0.5357 | 0.4110 | 28 | 0.8541 | 0.8226 | 0.8381 | 840 | 0.5847 | 0.4631 | 0.5169 | 149 | 0.6835 | 0.6136 | 0.6467 | 88 | 0.6801 | 0.7148 | 0.6970 | 589 | 0.8319 | 0.8895 | 0.8597 | 751 | 0.9571 | 0.9178 | 0.9371 | 73 | 0.5307 | 0.5475 | 0.5390 | 221 | 0.4583 | 0.4889 | 0.4731 | 45 | 0.6364 | 0.8063 | 0.7113 | 191 | 0.7414 | 0.7649 | 0.7529 | 0.9651 |
| 0.0011 | 49 | 51989 | 0.3389 | 0.7435 | 0.7664 | 0.7547 | 0.9659 | 0.6957 | 0.7312 | 0.7130 | 372 | 0.3590 | 0.5 | 0.4179 | 28 | 0.8561 | 0.8286 | 0.8421 | 840 | 0.6636 | 0.4899 | 0.5637 | 149 | 0.6136 | 0.6136 | 0.6136 | 88 | 0.6732 | 0.6995 | 0.6861 | 589 | 0.8251 | 0.8921 | 0.8573 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.5746 | 0.5928 | 0.5835 | 221 | 0.4348 | 0.4444 | 0.4396 | 45 | 0.6390 | 0.8063 | 0.7130 | 191 | 0.7435 | 0.7664 | 0.7547 | 0.9659 |
| 0.0009 | 50 | 53050 | 0.3557 | 0.7490 | 0.7640 | 0.7564 | 0.9659 | 0.6948 | 0.6855 | 0.6901 | 372 | 0.3947 | 0.5357 | 0.4545 | 28 | 0.8584 | 0.8298 | 0.8438 | 840 | 0.6455 | 0.4765 | 0.5483 | 149 | 0.6933 | 0.5909 | 0.6380 | 88 | 0.6745 | 0.7317 | 0.7020 | 589 | 0.8296 | 0.8948 | 0.8610 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.6082 | 0.5339 | 0.5687 | 221 | 0.4043 | 0.4222 | 0.4130 | 45 | 0.6270 | 0.8272 | 0.7133 | 191 | 0.7490 | 0.7640 | 0.7564 | 0.9659 |
| 0.0008 | 51 | 54111 | 0.3492 | 0.7516 | 0.7601 | 0.7558 | 0.9662 | 0.7104 | 0.6989 | 0.7046 | 372 | 0.3714 | 0.4643 | 0.4127 | 28 | 0.8545 | 0.8321 | 0.8432 | 840 | 0.6496 | 0.5101 | 0.5714 | 149 | 0.625 | 0.5682 | 0.5952 | 88 | 0.6722 | 0.6893 | 0.6806 | 589 | 0.8413 | 0.8895 | 0.8647 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.5611 | 0.5611 | 0.5611 | 221 | 0.4792 | 0.5111 | 0.4946 | 45 | 0.6724 | 0.8168 | 0.7376 | 191 | 0.7516 | 0.7601 | 0.7558 | 0.9662 |
| 0.0008 | 52 | 55172 | 0.3432 | 0.7526 | 0.7625 | 0.7575 | 0.9661 | 0.7044 | 0.7366 | 0.7201 | 372 | 0.3571 | 0.5357 | 0.4286 | 28 | 0.8610 | 0.8262 | 0.8433 | 840 | 0.6140 | 0.4698 | 0.5323 | 149 | 0.6667 | 0.5909 | 0.6265 | 88 | 0.6766 | 0.6927 | 0.6846 | 589 | 0.8403 | 0.8895 | 0.8642 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.5849 | 0.5611 | 0.5727 | 221 | 0.46 | 0.5111 | 0.4842 | 45 | 0.6681 | 0.8115 | 0.7329 | 191 | 0.7526 | 0.7625 | 0.7575 | 0.9661 |
| **0.0006** | **53** | **56233** | **0.3565** | **0.7615** | **0.7747** | **0.7681** | **0.9672** | **0.7305** | **0.7285** | **0.7295** | **372** | **0.3721** | **0.5714** | **0.4507** | **28** | **0.8679** | **0.8369** | **0.8521** | **840** | **0.6545** | **0.4832** | **0.5560** | **149** | **0.6625** | **0.6023** | **0.6310** | **88** | **0.6761** | **0.7267** | **0.7005** | **589** | **0.8255** | **0.9068** | **0.8642** | **751** | **1.0** | **0.9589** | **0.9790** | **73** | **0.6030** | **0.5430** | **0.5714** | **221** | **0.5682** | **0.5556** | **0.5618** | **45** | **0.7** | **0.8063** | **0.7494** | **191** | **0.7615** | **0.7747** | **0.7681** | **0.9672** |
| 0.0008 | 54 | 57294 | 0.3480 | 0.7590 | 0.7631 | 0.7610 | 0.9668 | 0.7452 | 0.7312 | 0.7381 | 372 | 0.3409 | 0.5357 | 0.4167 | 28 | 0.8589 | 0.8190 | 0.8385 | 840 | 0.5935 | 0.4899 | 0.5368 | 149 | 0.7027 | 0.5909 | 0.6420 | 88 | 0.6924 | 0.6842 | 0.6883 | 589 | 0.8432 | 0.8948 | 0.8682 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.5856 | 0.5882 | 0.5869 | 221 | 0.5102 | 0.5556 | 0.5319 | 45 | 0.6513 | 0.8115 | 0.7226 | 191 | 0.7590 | 0.7631 | 0.7610 | 0.9668 |
| 0.0008 | 55 | 58355 | 0.3568 | 0.7601 | 0.7622 | 0.7612 | 0.9663 | 0.7228 | 0.7151 | 0.7189 | 372 | 0.3571 | 0.5357 | 0.4286 | 28 | 0.8429 | 0.8429 | 0.8429 | 840 | 0.6634 | 0.4497 | 0.536 | 149 | 0.7 | 0.5568 | 0.6203 | 88 | 0.6828 | 0.7165 | 0.6993 | 589 | 0.8655 | 0.8828 | 0.8741 | 751 | 0.9853 | 0.9178 | 0.9504 | 73 | 0.5909 | 0.5294 | 0.5585 | 221 | 0.5106 | 0.5333 | 0.5217 | 45 | 0.6429 | 0.8010 | 0.7133 | 191 | 0.7601 | 0.7622 | 0.7612 | 0.9663 |
| 0.0009 | 56 | 59416 | 0.3498 | 0.7542 | 0.7580 | 0.7561 | 0.9661 | 0.7178 | 0.7043 | 0.7110 | 372 | 0.3409 | 0.5357 | 0.4167 | 28 | 0.8379 | 0.8429 | 0.8404 | 840 | 0.6634 | 0.4497 | 0.536 | 149 | 0.6322 | 0.625 | 0.6286 | 88 | 0.6895 | 0.6825 | 0.6860 | 589 | 0.8513 | 0.8842 | 0.8674 | 751 | 0.9577 | 0.9315 | 0.9444 | 73 | 0.5613 | 0.5385 | 0.5497 | 221 | 0.5111 | 0.5111 | 0.5111 | 45 | 0.6667 | 0.8063 | 0.7299 | 191 | 0.7542 | 0.7580 | 0.7561 | 0.9661 |
| 0.0007 | 57 | 60477 | 0.3486 | 0.7479 | 0.7711 | 0.7593 | 0.9663 | 0.7143 | 0.7392 | 0.7266 | 372 | 0.3571 | 0.5357 | 0.4286 | 28 | 0.8417 | 0.8417 | 0.8417 | 840 | 0.5923 | 0.5168 | 0.5520 | 149 | 0.6667 | 0.6136 | 0.6391 | 88 | 0.6720 | 0.7165 | 0.6935 | 589 | 0.8562 | 0.8802 | 0.8680 | 751 | 0.9714 | 0.9315 | 0.9510 | 73 | 0.5670 | 0.5747 | 0.5708 | 221 | 0.4583 | 0.4889 | 0.4731 | 45 | 0.6623 | 0.8010 | 0.7251 | 191 | 0.7479 | 0.7711 | 0.7593 | 0.9663 |
| 0.0007 | 58 | 61538 | 0.3497 | 0.7539 | 0.7744 | 0.7640 | 0.9667 | 0.7143 | 0.7392 | 0.7266 | 372 | 0.3659 | 0.5357 | 0.4348 | 28 | 0.8449 | 0.8429 | 0.8439 | 840 | 0.6429 | 0.4832 | 0.5517 | 149 | 0.6667 | 0.5909 | 0.6265 | 88 | 0.6708 | 0.7267 | 0.6976 | 589 | 0.8499 | 0.8975 | 0.8731 | 751 | 0.9714 | 0.9315 | 0.9510 | 73 | 0.6108 | 0.5611 | 0.5849 | 221 | 0.5 | 0.4889 | 0.4944 | 45 | 0.6525 | 0.8063 | 0.7213 | 191 | 0.7539 | 0.7744 | 0.7640 | 0.9667 |
| 0.0008 | 59 | 62599 | 0.3581 | 0.7474 | 0.7762 | 0.7615 | 0.9662 | 0.7183 | 0.7473 | 0.7325 | 372 | 0.3409 | 0.5357 | 0.4167 | 28 | 0.8439 | 0.8429 | 0.8434 | 840 | 0.5467 | 0.5503 | 0.5485 | 149 | 0.6709 | 0.6023 | 0.6347 | 88 | 0.6693 | 0.7250 | 0.6960 | 589 | 0.8454 | 0.8881 | 0.8662 | 751 | 0.9714 | 0.9315 | 0.9510 | 73 | 0.5961 | 0.5475 | 0.5708 | 221 | 0.5 | 0.5333 | 0.5161 | 45 | 0.6769 | 0.8115 | 0.7381 | 191 | 0.7474 | 0.7762 | 0.7615 | 0.9662 |
| 0.0007 | 60 | 63660 | 0.3636 | 0.7494 | 0.7676 | 0.7584 | 0.9662 | 0.7016 | 0.7204 | 0.7109 | 372 | 0.3488 | 0.5357 | 0.4225 | 28 | 0.8489 | 0.8357 | 0.8422 | 840 | 0.6 | 0.4832 | 0.5353 | 149 | 0.6538 | 0.5795 | 0.6145 | 88 | 0.6828 | 0.7199 | 0.7008 | 589 | 0.8476 | 0.8815 | 0.8642 | 751 | 0.9714 | 0.9315 | 0.9510 | 73 | 0.5579 | 0.5882 | 0.5727 | 221 | 0.4762 | 0.4444 | 0.4598 | 45 | 0.6797 | 0.8220 | 0.7441 | 191 | 0.7494 | 0.7676 | 0.7584 | 0.9662 |
| 0.0008 | 61 | 64721 | 0.3646 | 0.7538 | 0.7574 | 0.7556 | 0.9660 | 0.6854 | 0.7204 | 0.7025 | 372 | 0.3659 | 0.5357 | 0.4348 | 28 | 0.8573 | 0.8369 | 0.8470 | 840 | 0.6306 | 0.4698 | 0.5385 | 149 | 0.6667 | 0.5909 | 0.6265 | 88 | 0.6896 | 0.6978 | 0.6937 | 589 | 0.8495 | 0.8722 | 0.8607 | 751 | 0.9714 | 0.9315 | 0.9510 | 73 | 0.5728 | 0.5520 | 0.5622 | 221 | 0.375 | 0.4 | 0.3871 | 45 | 0.6830 | 0.8010 | 0.7373 | 191 | 0.7538 | 0.7574 | 0.7556 | 0.9660 |
| 0.0006 | 62 | 65782 | 0.3697 | 0.7510 | 0.7460 | 0.7485 | 0.9651 | 0.6885 | 0.7070 | 0.6976 | 372 | 0.4286 | 0.5357 | 0.4762 | 28 | 0.8663 | 0.7869 | 0.8247 | 840 | 0.5902 | 0.4832 | 0.5314 | 149 | 0.6757 | 0.5682 | 0.6173 | 88 | 0.6667 | 0.6927 | 0.6794 | 589 | 0.8432 | 0.8948 | 0.8682 | 751 | 0.9851 | 0.9041 | 0.9429 | 73 | 0.5829 | 0.5566 | 0.5694 | 221 | 0.3673 | 0.4 | 0.3830 | 45 | 0.6995 | 0.7801 | 0.7376 | 191 | 0.7510 | 0.7460 | 0.7485 | 0.9651 |
| 0.0006 | 63 | 66843 | 0.3661 | 0.7504 | 0.7502 | 0.7503 | 0.9655 | 0.6909 | 0.6909 | 0.6909 | 372 | 0.4286 | 0.5357 | 0.4762 | 28 | 0.8571 | 0.8143 | 0.8352 | 840 | 0.5814 | 0.5034 | 0.5396 | 149 | 0.6582 | 0.5909 | 0.6228 | 88 | 0.7013 | 0.6655 | 0.6829 | 589 | 0.8348 | 0.8948 | 0.8638 | 751 | 0.9571 | 0.9178 | 0.9371 | 73 | 0.5570 | 0.5747 | 0.5657 | 221 | 0.3830 | 0.4 | 0.3913 | 45 | 0.6786 | 0.7958 | 0.7325 | 191 | 0.7504 | 0.7502 | 0.7503 | 0.9655 |
| 0.0006 | 64 | 67904 | 0.3711 | 0.7404 | 0.7628 | 0.7514 | 0.9656 | 0.6911 | 0.7097 | 0.7003 | 372 | 0.3784 | 0.5 | 0.4308 | 28 | 0.8455 | 0.8405 | 0.8430 | 840 | 0.6 | 0.5034 | 0.5474 | 149 | 0.65 | 0.5909 | 0.6190 | 88 | 0.6667 | 0.7029 | 0.6843 | 589 | 0.8350 | 0.8961 | 0.8645 | 751 | 0.9714 | 0.9315 | 0.9510 | 73 | 0.5673 | 0.5339 | 0.5501 | 221 | 0.2917 | 0.3111 | 0.3011 | 45 | 0.6568 | 0.8115 | 0.7260 | 191 | 0.7404 | 0.7628 | 0.7514 | 0.9656 |
| 0.0007 | 65 | 68965 | 0.3672 | 0.7377 | 0.7696 | 0.7533 | 0.9661 | 0.7005 | 0.7419 | 0.7206 | 372 | 0.3333 | 0.5357 | 0.4110 | 28 | 0.8433 | 0.8393 | 0.8413 | 840 | 0.5839 | 0.5369 | 0.5594 | 149 | 0.6506 | 0.6136 | 0.6316 | 88 | 0.6840 | 0.7131 | 0.6983 | 589 | 0.8412 | 0.8815 | 0.8609 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.5427 | 0.5747 | 0.5582 | 221 | 0.3019 | 0.3556 | 0.3265 | 45 | 0.6360 | 0.7958 | 0.7070 | 191 | 0.7377 | 0.7696 | 0.7533 | 0.9661 |
| 0.0005 | 66 | 70026 | 0.3768 | 0.7496 | 0.7520 | 0.7508 | 0.9657 | 0.6903 | 0.7070 | 0.6985 | 372 | 0.3415 | 0.5 | 0.4058 | 28 | 0.8454 | 0.8333 | 0.8393 | 840 | 0.6372 | 0.4832 | 0.5496 | 149 | 0.6795 | 0.6023 | 0.6386 | 88 | 0.6914 | 0.6655 | 0.6782 | 589 | 0.8483 | 0.8788 | 0.8633 | 751 | 0.9577 | 0.9315 | 0.9444 | 73 | 0.5714 | 0.5792 | 0.5753 | 221 | 0.3 | 0.3333 | 0.3158 | 45 | 0.6696 | 0.7958 | 0.7273 | 191 | 0.7496 | 0.7520 | 0.7508 | 0.9657 |
| 0.0007 | 67 | 71087 | 0.3682 | 0.7461 | 0.7664 | 0.7561 | 0.9656 | 0.7094 | 0.7285 | 0.7188 | 372 | 0.3409 | 0.5357 | 0.4167 | 28 | 0.8563 | 0.8369 | 0.8465 | 840 | 0.6290 | 0.5235 | 0.5714 | 149 | 0.6974 | 0.6023 | 0.6463 | 88 | 0.6935 | 0.6876 | 0.6905 | 589 | 0.8363 | 0.8842 | 0.8595 | 751 | 0.9437 | 0.9178 | 0.9306 | 73 | 0.5175 | 0.6018 | 0.5565 | 221 | 0.4694 | 0.5111 | 0.4894 | 45 | 0.6483 | 0.8010 | 0.7166 | 191 | 0.7461 | 0.7664 | 0.7561 | 0.9656 |
| 0.0005 | 68 | 72148 | 0.3815 | 0.7590 | 0.7416 | 0.7502 | 0.9654 | 0.7092 | 0.7016 | 0.7054 | 372 | 0.4054 | 0.5357 | 0.4615 | 28 | 0.8489 | 0.8095 | 0.8288 | 840 | 0.6796 | 0.4698 | 0.5556 | 149 | 0.6456 | 0.5795 | 0.6108 | 88 | 0.6801 | 0.6570 | 0.6684 | 589 | 0.8476 | 0.8815 | 0.8642 | 751 | 0.9571 | 0.9178 | 0.9371 | 73 | 0.615 | 0.5566 | 0.5843 | 221 | 0.4348 | 0.4444 | 0.4396 | 45 | 0.6759 | 0.7644 | 0.7174 | 191 | 0.7590 | 0.7416 | 0.7502 | 0.9654 |
| 0.0006 | 69 | 73209 | 0.3919 | 0.7494 | 0.7487 | 0.7491 | 0.9650 | 0.6888 | 0.6962 | 0.6925 | 372 | 0.3590 | 0.5 | 0.4179 | 28 | 0.8416 | 0.8095 | 0.8252 | 840 | 0.5865 | 0.5235 | 0.5532 | 149 | 0.6901 | 0.5568 | 0.6164 | 88 | 0.6950 | 0.6808 | 0.6878 | 589 | 0.8490 | 0.8908 | 0.8694 | 751 | 1.0 | 0.9041 | 0.9496 | 73 | 0.5662 | 0.5611 | 0.5636 | 221 | 0.3265 | 0.3556 | 0.3404 | 45 | 0.6881 | 0.7853 | 0.7335 | 191 | 0.7494 | 0.7487 | 0.7491 | 0.9650 |
| 0.0006 | 70 | 74270 | 0.3704 | 0.7587 | 0.7619 | 0.7603 | 0.9666 | 0.6891 | 0.7151 | 0.7018 | 372 | 0.3947 | 0.5357 | 0.4545 | 28 | 0.8376 | 0.8536 | 0.8455 | 840 | 0.6697 | 0.4899 | 0.5659 | 149 | 0.6420 | 0.5909 | 0.6154 | 88 | 0.7018 | 0.6791 | 0.6903 | 589 | 0.8491 | 0.8842 | 0.8663 | 751 | 0.9857 | 0.9452 | 0.9650 | 73 | 0.6219 | 0.5656 | 0.5924 | 221 | 0.3913 | 0.4 | 0.3956 | 45 | 0.6802 | 0.7906 | 0.7312 | 191 | 0.7587 | 0.7619 | 0.7603 | 0.9666 |
| 0.0005 | 71 | 75331 | 0.3841 | 0.7501 | 0.7634 | 0.7567 | 0.9659 | 0.7005 | 0.6855 | 0.6929 | 372 | 0.4054 | 0.5357 | 0.4615 | 28 | 0.8531 | 0.8298 | 0.8413 | 840 | 0.6293 | 0.4899 | 0.5509 | 149 | 0.6410 | 0.5682 | 0.6024 | 88 | 0.6774 | 0.7165 | 0.6964 | 589 | 0.8264 | 0.9001 | 0.8617 | 751 | 0.9706 | 0.9041 | 0.9362 | 73 | 0.5882 | 0.5882 | 0.5882 | 221 | 0.4545 | 0.4444 | 0.4494 | 45 | 0.6864 | 0.7906 | 0.7348 | 191 | 0.7501 | 0.7634 | 0.7567 | 0.9659 |
| 0.0005 | 72 | 76392 | 0.3830 | 0.7605 | 0.7496 | 0.7550 | 0.9655 | 0.7036 | 0.6828 | 0.6930 | 372 | 0.3824 | 0.4643 | 0.4194 | 28 | 0.8618 | 0.8238 | 0.8424 | 840 | 0.6542 | 0.4698 | 0.5469 | 149 | 0.6582 | 0.5909 | 0.6228 | 88 | 0.6935 | 0.6723 | 0.6828 | 589 | 0.8476 | 0.8815 | 0.8642 | 751 | 0.9577 | 0.9315 | 0.9444 | 73 | 0.5830 | 0.5882 | 0.5856 | 221 | 0.4043 | 0.4222 | 0.4130 | 45 | 0.6892 | 0.8010 | 0.7409 | 191 | 0.7605 | 0.7496 | 0.7550 | 0.9655 |
| 0.0006 | 73 | 77453 | 0.3839 | 0.7611 | 0.7547 | 0.7579 | 0.9661 | 0.712 | 0.7177 | 0.7149 | 372 | 0.3429 | 0.4286 | 0.3810 | 28 | 0.8494 | 0.8393 | 0.8443 | 840 | 0.6542 | 0.4698 | 0.5469 | 149 | 0.6538 | 0.5795 | 0.6145 | 88 | 0.6877 | 0.6655 | 0.6764 | 589 | 0.8428 | 0.8921 | 0.8668 | 751 | 0.9710 | 0.9178 | 0.9437 | 73 | 0.6257 | 0.5294 | 0.5735 | 221 | 0.4468 | 0.4667 | 0.4565 | 45 | 0.6814 | 0.8063 | 0.7386 | 191 | 0.7611 | 0.7547 | 0.7579 | 0.9661 |
### Framework versions
- Transformers 4.16.2
- Pytorch 1.10.2+cu113
- Datasets 1.18.3
- Tokenizers 0.11.0
|