diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -1,3 +1,7331 @@ ---- -license: cc-by-4.0 ---- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ner/README.md at master · D3auuvc/ner + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ Skip to content + + + + + + + + + + + + + + +
+ +
+ + + + + + + +
+ + + +
+ + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ + + + D3auuvc  /   + ner  /   + +
+
+ + + +
+ + +
+ + + + + + +
+
+ Tip: + Type # to search pull requests +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Type # to search issues +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Type # to search discussions +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Type ! to search projects +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Type @ to search teams +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Type @ to search people and organizations +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Type > to activate command mode +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Go to your accessibility settings to change your keyboard shortcuts +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Type author:@me to search your content +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Type is:pr to filter to pull requests +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Type is:issue to filter to issues +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Type is:project to filter to projects +
+
+ Type ? for help and tips +
+
+ + +
+
+ Tip: + Type is:open to filter to open content +
+
+ Type ? for help and tips +
+
+ + +
+ +
+
+ We’ve encountered an error and some results aren't available at this time. Type a new search or try again later. +
+ + + No results matched your search + + + + + + + + + + + + + + + + + + Search for issues and pull requests + + # + + + + Search for issues, pull requests, discussions, and projects + + # + + + + Search for organizations, repositories, and users + + @ + + + + Search for projects + + ! + + + + Search for files + + / + + + + Activate command mode + + > + + + + Search your issues, pull requests, and discussions + + # author:@me + + + + Search your issues, pull requests, and discussions + + # author:@me + + + + Filter to pull requests + + # is:pr + + + + Filter to issues + + # is:issue + + + + Filter to discussions + + # is:discussion + + + + Filter to projects + + # is:project + + + + Filter to open issues, pull requests, and discussions + + # is:open + + + + + + + + + + + + + + + + + + +
+ +
+ + + + + + + + + + +
+ + + + +
+
+
+ + + + + + + + + +
+ +
+ +
+

+ + + / + + ner + + + Public +

+ + forked from ksirts/ner + + +
+ +
    + + + +
  • + +
    + + + + + + + Watch + + + 0 + + + + +
    +
    +

    Notifications

    + +
    + +
    +
    + + + + + + + + +
    + + +
    + + + + + Get push notifications on iOS or Android. + +
    +
    +
    +     + + + + +
    +
    +     + + + +
    • + +
  • + Fork + 1 + + +
    • + +
  • + + +
    +
    + + +
    +
    + +
    +
    + + + + +
    + +
    +
    + + + + + + + +
    + +
    +
    +     +
    +
    +
    • + + + +
+ +
+ +
+
+ + + + +
+ + + +
+ Open in github.dev + Open in a new github.dev tab + + + + + + +
+ + +
+ + + + + + +Permalink + +
+ +
+
+ + + master + + + + +
+
+
+ Switch branches/tags + +
+ + + +
+ +
+ +
+ + +
+ +
+ + + + + + + + + + + + + + + +
+ + + + +
+
+ +
+ +
+ +

+ ner/README.md +

+ Go to file + + +
+ + + + + +
    +
  • + + Go to file + +
    • +
  • +
    • +
  • +
  • + + + Copy path + +
    • +
  • + + + + Copy permalink + + +
    • +
+
+
+ + + + + + + + + +
+ +
+
+
 
+
+ +
+
 
+ Cannot retrieve contributors at this time +
+
+ + + + + + + + + + + + + +
+ +
+ +
+ + + + + + +
+ +
+ +
+ +
+ Estonian NER model based on EstBERT + How to use + Intended uses & limitations + Training and evaluation data + Training procedure + Training hyperparameters + Training results + Framework versions +
+
+ +
+ + +
+ + 177 lines (145 sloc) + + 62.8 KB +
+ +
+
+ + + + +
+ + +
+ Raw + + Blame + +
+ +
+ + + + + + + + + + + + + + + +
+ +
+
+ +
+
+ +
+
+ + + + + +
+
+
+ + +
+

Estonian NER model based on EstBERT

+

This model is a fine-tuned version of tartuNLP/EstBERT on the Estonian NER dataset. The model was trained by tartuNLP, the NLP research group at the institute of Computer Science at the University of Tartu.

+

It achieves the following results on the test set:

+
    +
  • Loss: 0.3565
    • +
  • Precision: 0.7612
    • +
  • Recall: 0.7744
    • +
  • F1: 0.7678
    • +
  • Accuracy: 0.9672
    • +
+

The entity-level results are as follows:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PrecisionRecallF1Number
DATE0.72780.72580.7268372
EVENT0.37210.57140.450728
GPE0.86790.83690.8521840
LOC0.65450.48320.5560149
MONEY0.66250.60230.631088
ORG0.67610.72670.7005589
PER0.82550.90680.8642751
PERCENT1.00.95890.979073
PROD0.60300.54300.5714221
TIME0.56820.55560.561845
TITLE0.70.80630.7494191
+

How to use

+

You can use this model with Transformers pipeline for NER. Post-processing of results may be necessary as the model occasionally tags subword tokens as entities.

+
from transformers import BertTokenizer, BertForTokenClassification
+from transformers import pipeline
+
+tokenizer = BertTokenizer.from_pretrained('tartuNLP/EstBERT_NER')
+bertner = BertForTokenClassification.from_pretrained('tartuNLP/EstBERT_NER')
+
+nlp = pipeline("ner", model=bertner, tokenizer=tokenizer)
+
+text = "Kaia Kanepi (WTA 57.) langes USA-s Charlestonis toimuval WTA 500 kategooria tenniseturniiril konkurentsist kaheksandikfinaalis, kaotades poolatarile Magda Linette'ile (WTA 64.) 3 : 6, 6 : 4, 2 : 6."
+
+ner_results = new_nlp(text)
+
+tokens=tokenizer(text)
+tokens=tokenizer.convert_ids_to_tokens(tokens['input_ids'])
+
+
+print(f'tokens: {tokens}')
+print(f'NER model:{ner_results}')
+
+
+
tokens: ['[CLS]', 'kai', '##a', 'kanepi', '(', 'w', '##ta', '57', '.', ')', 'langes', 'usa', '-', 's', 'cha', '##rl', '##est', '##onis', 'toimuval', 'w', '##ta', '500', 'kategooria', 'tennise', '##turniiril', 'konkurentsist', 'kaheksandik', '##finaalis', ',', 'kaotades', 'poola', '##tari', '##le', 'ma', '##gda', 'line', '##tte', "'", 'ile', '(', 'w', '##ta', '64', '.', ')', '3', ':', '6', ',', '6', ':', '4', ',', '2', ':', '6', '.', '[SEP]']
+
+
+
NER model: [{'entity': 'B-PER', 'score': 0.99999887, 'index': 1, 'word': 'kai', 'start': None, 'end': None}, {'entity': 'B-PER', 'score': 0.97371966, 'index': 2, 'word': '##a', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.99999815, 'index': 3, 'word': 'kanepi', 'start': None, 'end': None}, {'entity': 'B-ORG', 'score': 0.63085276, 'index': 5, 'word': 'w', 'start': None, 'end': None}, {'entity': 'B-GPE', 'score': 0.99999934, 'index': 11, 'word': 'usa', 'start': None, 'end': None}, {'entity': 'B-GPE', 'score': 0.9999685, 'index': 14, 'word': 'cha', 'start': None, 'end': None}, {'entity': 'I-GPE', 'score': 0.8875574, 'index': 15, 'word': '##rl', 'start': None, 'end': None}, {'entity': 'I-GPE', 'score': 0.9996168, 'index': 16, 'word': '##est', 'start': None, 'end': None}, {'entity': 'I-GPE', 'score': 0.9992657, 'index': 17, 'word': '##onis', 'start': None, 'end': None}, {'entity': 'B-EVENT', 'score': 0.99999064, 'index': 19, 'word': 'w', 'start': None, 'end': None}, {'entity': 'I-EVENT', 'score': 0.9772493, 'index': 20, 'word': '##ta', 'start': None, 'end': None}, {'entity': 'I-EVENT', 'score': 0.99999076, 'index': 21, 'word': '500', 'start': None, 'end': None}, {'entity': 'I-EVENT', 'score': 0.99955636, 'index': 22, 'word': 'kategooria', 'start': None, 'end': None}, {'entity': 'B-TITLE', 'score': 0.8771319, 'index': 30, 'word': 'poola', 'start': None, 'end': None}, {'entity': 'B-PER', 'score': 0.99999785, 'index': 33, 'word': 'ma', 'start': None, 'end': None}, {'entity': 'B-PER', 'score': 0.9998398, 'index': 34, 'word': '##gda', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.9999987, 'index': 35, 'word': 'line', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.9999976, 'index': 36, 'word': '##tte', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.99999285, 'index': 37, 'word': "'", 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.9999794, 'index': 38, 'word': 'ile', 'start': None, 'end': None}, {'entity': 'B-ORG', 'score': 0.7664479, 'index': 40, 'word': 'w', 'start': None, 'end': None}]
+
+

Intended uses & limitations

+

This model can be used to find named entities from Estonian texts. The model is free to use for anyone. TartuNLP does not guarantee that the model is useful for anyone or anything. TartuNLP is not responsible for any results it generates.

+

Training and evaluation data

+

The model was trained on two Estonian NER datasets:

+ +

Both datasets have been annotated with the same annotation scheme. For training this model, the datasets were joined.

+

Training procedure

+

Training hyperparameters

+

The following hyperparameters were used during training:

+
    +
  • learning_rate: 1e-05
    • +
  • train_batch_size: 16
    • +
  • eval_batch_size: 16
    • +
  • seed: 1024
    • +
  • optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
    • +
  • lr_scheduler_type: polynomial
    • +
  • max num_epochs: 150
    • +
  • early stopping limit: 20
    • +
  • early stopping tol: 0.0001
    • +
  • mixed_precision_training: Native AMP
    • +
+

Training results

+

The final model was saved after epoch 53 (shown in bold) where the overall F1 was the highest on the development set.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Training LossEpochStepValidation LossPrecisionRecallF1AccuracyDate PrecisionDate RecallDate F1Date NumberEvent PrecisionEvent RecallEvent F1Event NumberGpe PrecisionGpe RecallGpe F1Gpe NumberLoc PrecisionLoc RecallLoc F1Loc NumberMoney PrecisionMoney RecallMoney F1Money NumberOrg PrecisionOrg RecallOrg F1Org NumberPer PrecisionPer RecallPer F1Per NumberPercent PrecisionPercent RecallPercent F1Percent NumberProd PrecisionProd RecallProd F1Prod NumberTime PrecisionTime RecallTime F1Time NumberTitle PrecisionTitle RecallTitle F1Title NumberOverall PrecisionOverall RecallOverall F1Overall Accuracy
0.3252110610.16280.68350.60830.64370.95260.59100.60220.59653720.00.00.0280.80730.76310.78468400.13890.03360.05411490.42170.39770.4094880.53810.52800.53305890.79170.86550.82707510.64710.30140.4112730.25810.07240.11312210.14290.08890.1096450.78050.67020.72111910.68350.60830.64370.9526
0.1513221220.13320.69060.73290.71110.96150.61850.73660.67243720.08570.10710.0952280.78740.85950.82198400.47670.27520.34891490.68480.71590.7000880.61580.62310.61945890.77700.90010.83417510.95650.90410.9296730.50.36200.41992210.35710.33330.3448450.60330.76440.67441910.69060.73290.71110.9615
0.1131331830.12810.72240.73380.72800.96380.70540.73390.71943720.10530.14290.1212280.80130.850.82508400.54760.30870.39481490.63860.60230.6199880.63710.64690.64205890.82350.87620.84907510.98590.95890.9722730.51480.39370.44622210.51160.48890.5450.62450.77490.69161910.72240.73380.72800.9638
0.0884442440.13540.72830.73860.73340.96390.67850.66940.67393720.17950.250.2090280.82310.83100.82708400.60200.39600.47771490.60920.60230.6057880.64730.70120.67325890.83510.86280.84877511.00.97260.9861730.58990.47510.52632210.45240.42220.4368450.60.78530.68031910.72830.73860.73340.9639
0.0685553050.13830.72240.76960.74530.96440.66350.74730.70293720.260.46430.3333280.82590.83570.83088400.59130.45640.51521490.64370.63640.64880.65400.72840.68925890.80700.89610.84927510.98570.94520.9650730.56930.52040.54372210.51920.60.5567450.63200.76440.69191910.72240.76960.74530.9644
0.0532663660.14930.70990.76130.73470.96310.67270.69620.68433720.23080.53570.3226280.82420.82620.82528400.58770.44970.50951490.64100.56820.6024880.62320.74700.67955890.80870.88950.84727510.96720.80820.8806730.51070.53850.52422210.61900.57780.5977450.63710.79060.70561910.70990.76130.73470.9631
0.0403774270.15920.72390.75920.74110.96420.69230.70160.69693720.28570.57140.3810280.82720.82620.82678400.57520.43620.49621490.62650.59090.6082880.64020.69780.66775890.84040.87620.85797510.98590.95890.9722730.52570.60180.56122210.58700.60.5934450.62350.80630.70321910.72390.75920.74110.9642
0.0304884880.17380.73010.74840.73920.96440.68660.67740.68203720.34090.53570.4167280.83930.80830.82358400.58820.46980.52241490.64290.61360.6279880.66080.69780.67885890.82680.87080.84827510.95950.97260.9660730.53510.55200.54342210.52080.55560.5376450.62040.79580.69721910.73010.74840.73920.9644
0.0234995490.18600.72480.76250.74320.96410.69470.70970.70213720.29630.57140.3902280.83170.82980.83088400.59130.45640.51521490.61180.59090.6012880.63610.70630.66935890.84100.87350.85707510.98590.95890.9722730.52120.61090.56252210.54170.57780.5591450.64140.79580.71031910.72480.76250.74320.9641
0.017810106100.20370.74340.73830.74080.96400.71590.67740.69613720.28570.42860.3429280.83330.83330.83338400.62620.44970.52341490.63240.48860.5513880.65680.67570.66615890.82910.87220.85017511.00.82190.9023730.56720.51580.54032210.50.53330.5161450.69520.76440.72821910.74340.73830.74080.9640
0.014711116710.21140.74400.72330.73350.96430.70090.66130.68053720.30300.35710.3279280.83520.80240.81858400.62380.42280.5041490.650.59090.6190880.64360.64690.64525890.84070.85750.84907510.93150.93150.9315730.58120.50230.53882210.54760.51110.5287450.68350.78010.72861910.74400.72330.73350.9643
0.011812127320.22180.73310.75320.74300.96490.71190.69090.70123720.34880.53570.4225280.83250.84050.83658400.53030.46980.49821490.650.59090.6190880.66900.65870.66385890.81780.89080.85287510.96770.82190.8889730.54080.57010.55512210.51020.55560.5319450.65670.80100.72171910.73310.75320.74300.9649
0.009313137930.22830.74950.73590.74270.96440.71630.69890.70753720.38100.57140.4571280.86120.79050.82438400.61110.44300.51361490.61450.57950.5965880.67750.67400.67575890.83460.88020.85687510.97100.91780.9437730.56190.53390.54762210.40.48890.4400450.68120.73820.70851910.74950.73590.74270.9644
0.007914148540.23830.73710.74900.74300.96470.67270.70160.68683720.32610.53570.4054280.84530.80.82208400.59630.43620.50391490.6250.56820.5952880.66340.69270.67775890.84330.88150.86207510.98530.91780.9504730.54270.57470.55822210.58140.55560.5682450.65130.81150.72261910.73710.74900.74300.9647
0.006815159150.25110.72550.73590.73060.96390.68260.65320.66763720.35900.50.4179280.82950.81670.82308400.52630.46980.49651490.65750.54550.5963880.65490.66040.65775890.82420.88020.85137510.98330.80820.8872730.53980.55200.54592210.360.40.3789450.65110.80100.71831910.72550.73590.73060.9639
0.006116169760.24970.72530.76900.74650.96480.68240.69890.69063720.33330.53570.4110280.84730.83210.83968400.45830.51680.48581490.64940.56820.6061880.65560.73680.69385890.83820.88280.85997510.98410.84930.9118730.53410.63800.58142210.50.53330.5161450.66220.78010.71631910.72530.76900.74650.9648
0.005417180370.25540.73230.76250.74710.96500.68700.69620.69163720.34210.46430.3939280.84630.82620.83618400.59020.48320.53141490.67530.59090.6303880.66400.71480.68855890.83170.89480.86217510.94370.91780.9306730.52100.56110.54032210.50.51110.5055450.61020.81150.69661910.73230.76250.74710.9650
0.00518190980.26010.72730.77470.75030.96540.69700.76080.72753720.28300.53570.3704280.83200.84880.84038400.58410.44300.50381490.64770.64770.6477880.63780.69950.66725890.85010.89080.87007510.97220.95890.9655730.53230.59730.56292210.44440.44440.4444450.6240.81680.70751910.72730.77470.75030.9654
0.004419201590.26020.73690.76160.74900.96560.71240.71240.71243720.34150.50.4058280.82390.86310.84308400.63550.45640.53131490.66670.61360.6391880.65170.66380.65775890.84050.88420.86187510.98570.94520.9650730.51440.56560.53882210.52170.53330.5275450.65500.78530.71431910.73690.76160.74900.9656
0.00420212200.26770.73470.77020.75200.96580.73740.70970.72333720.28570.42860.3429280.84660.83450.84058400.60500.48320.53731490.66670.61360.6391880.65930.71310.68525890.82400.89750.85917510.98570.94520.9650730.49810.58370.53752210.51020.55560.5319450.63710.82720.71981910.73470.77020.75200.9658
0.003421222810.27430.73860.77170.75480.96570.69840.70970.7043720.37840.50.4308280.84750.83330.84038400.63330.51010.56511490.61900.59090.6047880.65120.73850.69215890.84280.89210.86687510.98460.87670.9275730.55130.58370.56702210.51060.53330.5217450.63790.81150.71431910.73860.77170.75480.9657
0.003322233420.27880.74180.75200.74690.96520.71430.69890.70653720.31820.50.3889280.83670.82980.83328400.61680.44300.51561490.62350.60230.6127880.67580.66890.67245890.83270.88150.85647510.97140.93150.9510730.54580.59280.56832210.47830.48890.4835450.66370.78530.71941910.74180.75200.74690.9652
0.003323244030.28310.73420.75350.74370.96500.69810.69620.69723720.37840.50.4308280.84990.80240.82558400.50340.49660.51490.60670.61360.6102880.65810.69610.67665890.83500.89610.86457510.97140.93150.9510730.54240.57920.56022210.37740.44440.4082450.70480.77490.73821910.73420.75350.74370.9650
0.002924254640.29310.75440.73800.74610.96480.73650.69890.71723720.35900.50.4179280.85350.79760.82468400.58490.41610.48631490.66220.55680.6049880.66720.67060.66895890.84740.88020.86357510.97010.89040.9286730.55500.54750.55132210.48890.48890.4889450.70230.79060.74381910.75440.73800.74610.9648
0.002825265250.28990.74890.75740.75310.96540.70210.70970.70593720.39020.57140.4638280.86350.81310.83758400.61820.45640.52511490.64710.6250.6358880.66130.69950.67995890.84540.90280.87317510.95830.94520.9517730.56810.54750.55762210.42220.42220.4222450.66080.78530.71771910.74890.75740.75310.9654
0.002326275860.29220.74130.75320.74720.96490.68970.69890.69433720.350.50.4118280.850.82980.83988400.61610.46310.52871490.64860.54550.5926880.64860.69270.67005890.84570.88280.86387510.98530.91780.9504730.56360.56110.56242210.39580.42220.4086450.66380.79580.72381910.74130.75320.74720.9649
0.002127286470.29670.75140.75680.75410.96560.70810.70430.70623720.36590.53570.4348280.85470.81900.83658400.56410.44300.49621490.65820.59090.6228880.66770.70970.68815890.84590.88420.86467510.97100.91780.9437730.58060.57010.57532210.48980.53330.5106450.70890.79060.74751910.75140.75680.75410.9656
0.002528297080.29570.73350.76220.74750.96510.70600.72310.71453720.30770.42860.3582280.84590.84290.84448400.50690.48990.49831490.64380.53410.5839880.68380.70120.69245890.84130.88950.86477510.95520.87670.9143730.49010.56110.52322210.38180.46670.42450.65800.79580.72041910.73350.76220.74750.9651
0.002329307690.30490.74550.75440.74990.96540.69970.73920.71903720.31820.50.3889280.84830.81190.82978400.56300.51010.53521490.65790.56820.6098880.67910.71140.69495890.85830.86280.86067510.98530.91780.9504730.52340.55660.53952210.45650.46670.4615450.70090.78530.74071910.74550.75440.74990.9654
0.001830318300.30420.74150.76790.75440.96540.69350.74190.71693720.33330.50.4280.85630.82260.83918400.58780.51680.551490.65820.59090.6228880.66770.74700.70515890.85440.88280.86847510.97100.91780.9437730.53000.52040.52512210.43750.46670.4516450.64170.80630.71461910.74150.76790.75440.9654
0.001731328910.30710.75400.74810.75100.96600.70830.73120.71963720.40540.53570.4615280.85520.82260.83868400.63110.43620.51591490.62200.57950.6880.67340.67570.67465890.86260.87750.87007510.98550.93150.9577730.53070.54750.53902210.38300.40.3913450.70190.76440.73181910.75400.74810.75100.9660
0.001832339520.31900.74990.75530.75260.96560.71820.71240.71523720.33330.53570.4110280.85860.79520.82578400.61160.49660.54811490.64630.60230.6235880.68050.69780.68905890.83600.88950.86197510.98550.93150.9577730.56330.58370.57332210.51060.53330.5217450.67110.80100.73031910.74990.75530.75260.9656
0.001833350130.30940.74600.77740.76140.96650.71470.74730.73063720.36590.53570.4348280.85560.83930.84748400.62730.46310.53281490.65060.61360.6316880.67870.73510.70585890.83440.89880.86547510.98570.94520.9650730.57020.60630.58772210.30360.37780.3366450.65670.80100.72171910.74600.77740.76140.9665
0.001534360740.30910.74410.77590.75970.96650.71130.72850.71983720.34040.57140.4267280.82660.85120.83878400.54050.53690.53871490.67070.6250.6471880.68560.71820.70155890.85170.88680.86897511.00.94520.9718730.57520.58820.58172210.38780.42220.4043450.68300.80100.73731910.74410.77590.75970.9665
0.001535371350.31850.74870.76190.75520.96600.69820.73390.71563720.34150.50.4058280.86850.81790.84248400.55040.47650.51081490.63530.61360.6243880.66360.72670.69375890.86540.88150.87347511.00.93150.9645730.550.54750.54882210.36730.40.3830450.69370.80630.74581910.74870.76190.75520.9660
0.001536381960.32030.74380.76490.75420.96600.69610.72040.70813720.36590.53570.4348280.86170.83810.84978400.52030.51680.51851490.66670.59090.6265880.67100.70630.68825890.84950.88680.86787510.97100.91780.9437730.55610.53850.54712210.420.46670.4421450.65680.81150.72601910.74380.76490.75420.9660
0.001337392570.32980.73150.77320.75180.96560.69150.72310.70703720.33330.57140.4211280.86540.81900.84168400.47930.54360.50941490.65820.59090.6228880.66560.72670.69485890.82890.90280.86427510.98570.94520.9650730.55740.59280.57462210.40430.42220.4130450.64080.82200.72021910.73150.77320.75180.9656
0.001238403180.33110.75330.76100.75710.96640.70600.72310.71453720.35710.53570.4286280.86130.83570.84838400.63390.47650.54411490.65430.60230.6272880.65280.71820.68395890.84240.88280.86227510.98570.94520.9650730.60310.52940.56392210.41300.42220.4176450.71220.76440.73741910.75330.76100.75710.9664
0.001239413790.33280.74440.75530.74980.96570.68180.72580.70313720.34780.57140.4324280.85610.81430.83478400.60550.44300.51161490.65820.59090.6228880.67150.70460.68775890.84610.87080.85837510.97060.90410.9362730.56650.59730.58152210.40820.44440.4255450.67700.80100.73381910.74440.75530.74980.9657
0.001440424400.34150.74210.74370.74290.96410.69310.70430.69873720.34880.53570.4225280.84220.82620.83418400.61900.43620.51181490.66220.55680.6049880.68880.63500.66085890.81750.88280.84897511.00.91780.9571730.55840.58370.57082210.40430.42220.4130450.65800.79580.72041910.74210.74370.74290.9641
0.001341435010.34010.75010.74870.74940.96510.69150.72310.70703720.34210.46430.3939280.85450.81790.83588400.63460.44300.52171490.68120.53410.5987880.67280.68080.67685890.83800.87480.85607510.97100.91780.9437730.58600.57010.57802210.44230.51110.4742450.67870.78530.72821910.75010.74870.74940.9651
0.001142445620.34680.74260.76870.75540.96500.69650.75270.72353720.34880.53570.4225280.86670.82020.84288400.64080.44300.52381490.67090.60230.6347880.69020.71480.70235890.84040.89750.86807510.94440.93150.9379730.51910.61540.56312210.34690.37780.3617450.62100.80630.70161910.74260.76870.75540.9650
0.001543456230.34400.75660.74220.74930.96480.69370.73660.71453720.38460.53570.4478280.86080.80950.83448400.60820.39600.47971490.70.55680.6203880.67660.65700.66675890.83170.88810.85907510.97010.89040.9286730.62240.55200.58512210.39130.40.3956450.70810.77490.741910.75660.74220.74930.9648
0.001144466840.33540.75650.76400.76020.96640.70620.73660.72113720.36590.53570.4348280.84830.84520.84688400.60950.42950.50391490.68830.60230.6424880.68800.67400.68105890.85170.89480.87277510.97100.91780.9437730.62380.59280.60792210.38300.40.3913450.650.81680.72391910.75650.76400.76020.9664
0.001145477450.33470.74850.76220.75530.96550.70880.73920.72373720.36360.57140.4444280.86030.82860.84418400.58820.46980.52241490.60230.60230.6023880.67700.66890.67295890.84170.89210.86627510.98570.94520.9650730.60370.59280.59822210.45830.48890.4731450.62750.81150.70781910.74850.76220.75530.9655
0.001146488060.34210.74810.76400.75590.96570.72610.73390.72993720.31710.46430.3768280.85700.82020.83828400.56910.46980.51471490.64290.61360.6279880.67690.71140.69375890.83110.89080.85997510.98570.94520.9650730.57140.56110.56622210.50.55560.5263450.66380.79580.72381910.74810.76400.75590.9657
0.000947498670.34870.74960.76040.75500.96560.71580.70430.71003720.34090.53570.4167280.860.81900.83908400.54960.48320.51431490.71620.60230.6543880.67450.72840.70045890.83460.88020.85687510.98570.94520.9650730.55660.53390.54502210.53490.51110.5227450.68280.81150.74161910.74960.76040.75500.9656
0.000948509280.34700.74140.76490.75290.96510.70920.74730.72773720.33330.53570.4110280.85410.82260.83818400.58470.46310.51691490.68350.61360.6467880.68010.71480.69705890.83190.88950.85977510.95710.91780.9371730.53070.54750.53902210.45830.48890.4731450.63640.80630.71131910.74140.76490.75290.9651
0.001149519890.33890.74350.76640.75470.96590.69570.73120.71303720.35900.50.4179280.85610.82860.84218400.66360.48990.56371490.61360.61360.6136880.67320.69950.68615890.82510.89210.85737510.98570.94520.9650730.57460.59280.58352210.43480.44440.4396450.63900.80630.71301910.74350.76640.75470.9659
0.000950530500.35570.74900.76400.75640.96590.69480.68550.69013720.39470.53570.4545280.85840.82980.84388400.64550.47650.54831490.69330.59090.6380880.67450.73170.70205890.82960.89480.86107510.98570.94520.9650730.60820.53390.56872210.40430.42220.4130450.62700.82720.71331910.74900.76400.75640.9659
0.000851541110.34920.75160.76010.75580.96620.71040.69890.70463720.37140.46430.4127280.85450.83210.84328400.64960.51010.57141490.6250.56820.5952880.67220.68930.68065890.84130.88950.86477510.98570.94520.9650730.56110.56110.56112210.47920.51110.4946450.67240.81680.73761910.75160.76010.75580.9662
0.000852551720.34320.75260.76250.75750.96610.70440.73660.72013720.35710.53570.4286280.86100.82620.84338400.61400.46980.53231490.66670.59090.6265880.67660.69270.68465890.84030.88950.86427510.98570.94520.9650730.58490.56110.57272210.460.51110.4842450.66810.81150.73291910.75260.76250.75750.9661
0.000653562330.35650.76150.77470.76810.96720.73050.72850.72953720.37210.57140.4507280.86790.83690.85218400.65450.48320.55601490.66250.60230.6310880.67610.72670.70055890.82550.90680.86427511.00.95890.9790730.60300.54300.57142210.56820.55560.5618450.70.80630.74941910.76150.77470.76810.9672
0.000854572940.34800.75900.76310.76100.96680.74520.73120.73813720.34090.53570.4167280.85890.81900.83858400.59350.48990.53681490.70270.59090.6420880.69240.68420.68835890.84320.89480.86827510.98570.94520.9650730.58560.58820.58692210.51020.55560.5319450.65130.81150.72261910.75900.76310.76100.9668
0.000855583550.35680.76010.76220.76120.96630.72280.71510.71893720.35710.53570.4286280.84290.84290.84298400.66340.44970.5361490.70.55680.6203880.68280.71650.69935890.86550.88280.87417510.98530.91780.9504730.59090.52940.55852210.51060.53330.5217450.64290.80100.71331910.76010.76220.76120.9663
0.000956594160.34980.75420.75800.75610.96610.71780.70430.71103720.34090.53570.4167280.83790.84290.84048400.66340.44970.5361490.63220.6250.6286880.68950.68250.68605890.85130.88420.86747510.95770.93150.9444730.56130.53850.54972210.51110.51110.5111450.66670.80630.72991910.75420.75800.75610.9661
0.000757604770.34860.74790.77110.75930.96630.71430.73920.72663720.35710.53570.4286280.84170.84170.84178400.59230.51680.55201490.66670.61360.6391880.67200.71650.69355890.85620.88020.86807510.97140.93150.9510730.56700.57470.57082210.45830.48890.4731450.66230.80100.72511910.74790.77110.75930.9663
0.000758615380.34970.75390.77440.76400.96670.71430.73920.72663720.36590.53570.4348280.84490.84290.84398400.64290.48320.55171490.66670.59090.6265880.67080.72670.69765890.84990.89750.87317510.97140.93150.9510730.61080.56110.58492210.50.48890.4944450.65250.80630.72131910.75390.77440.76400.9667
0.000859625990.35810.74740.77620.76150.96620.71830.74730.73253720.34090.53570.4167280.84390.84290.84348400.54670.55030.54851490.67090.60230.6347880.66930.72500.69605890.84540.88810.86627510.97140.93150.9510730.59610.54750.57082210.50.53330.5161450.67690.81150.73811910.74740.77620.76150.9662
0.000760636600.36360.74940.76760.75840.96620.70160.72040.71093720.34880.53570.4225280.84890.83570.84228400.60.48320.53531490.65380.57950.6145880.68280.71990.70085890.84760.88150.86427510.97140.93150.9510730.55790.58820.57272210.47620.44440.4598450.67970.82200.74411910.74940.76760.75840.9662
0.000861647210.36460.75380.75740.75560.96600.68540.72040.70253720.36590.53570.4348280.85730.83690.84708400.63060.46980.53851490.66670.59090.6265880.68960.69780.69375890.84950.87220.86077510.97140.93150.9510730.57280.55200.56222210.3750.40.3871450.68300.80100.73731910.75380.75740.75560.9660
0.000662657820.36970.75100.74600.74850.96510.68850.70700.69763720.42860.53570.4762280.86630.78690.82478400.59020.48320.53141490.67570.56820.6173880.66670.69270.67945890.84320.89480.86827510.98510.90410.9429730.58290.55660.56942210.36730.40.3830450.69950.78010.73761910.75100.74600.74850.9651
0.000663668430.36610.75040.75020.75030.96550.69090.69090.69093720.42860.53570.4762280.85710.81430.83528400.58140.50340.53961490.65820.59090.6228880.70130.66550.68295890.83480.89480.86387510.95710.91780.9371730.55700.57470.56572210.38300.40.3913450.67860.79580.73251910.75040.75020.75030.9655
0.000664679040.37110.74040.76280.75140.96560.69110.70970.70033720.37840.50.4308280.84550.84050.84308400.60.50340.54741490.650.59090.6190880.66670.70290.68435890.83500.89610.86457510.97140.93150.9510730.56730.53390.55012210.29170.31110.3011450.65680.81150.72601910.74040.76280.75140.9656
0.000765689650.36720.73770.76960.75330.96610.70050.74190.72063720.33330.53570.4110280.84330.83930.84138400.58390.53690.55941490.65060.61360.6316880.68400.71310.69835890.84120.88150.86097510.98570.94520.9650730.54270.57470.55822210.30190.35560.3265450.63600.79580.70701910.73770.76960.75330.9661
0.000566700260.37680.74960.75200.75080.96570.69030.70700.69853720.34150.50.4058280.84540.83330.83938400.63720.48320.54961490.67950.60230.6386880.69140.66550.67825890.84830.87880.86337510.95770.93150.9444730.57140.57920.57532210.30.33330.3158450.66960.79580.72731910.74960.75200.75080.9657
0.000767710870.36820.74610.76640.75610.96560.70940.72850.71883720.34090.53570.4167280.85630.83690.84658400.62900.52350.57141490.69740.60230.6463880.69350.68760.69055890.83630.88420.85957510.94370.91780.9306730.51750.60180.55652210.46940.51110.4894450.64830.80100.71661910.74610.76640.75610.9656
0.000568721480.38150.75900.74160.75020.96540.70920.70160.70543720.40540.53570.4615280.84890.80950.82888400.67960.46980.55561490.64560.57950.6108880.68010.65700.66845890.84760.88150.86427510.95710.91780.9371730.6150.55660.58432210.43480.44440.4396450.67590.76440.71741910.75900.74160.75020.9654
0.000669732090.39190.74940.74870.74910.96500.68880.69620.69253720.35900.50.4179280.84160.80950.82528400.58650.52350.55321490.69010.55680.6164880.69500.68080.68785890.84900.89080.86947511.00.90410.9496730.56620.56110.56362210.32650.35560.3404450.68810.78530.73351910.74940.74870.74910.9650
0.000670742700.37040.75870.76190.76030.96660.68910.71510.70183720.39470.53570.4545280.83760.85360.84558400.66970.48990.56591490.64200.59090.6154880.70180.67910.69035890.84910.88420.86637510.98570.94520.9650730.62190.56560.59242210.39130.40.3956450.68020.79060.73121910.75870.76190.76030.9666
0.000571753310.38410.75010.76340.75670.96590.70050.68550.69293720.40540.53570.4615280.85310.82980.84138400.62930.48990.55091490.64100.56820.6024880.67740.71650.69645890.82640.90010.86177510.97060.90410.9362730.58820.58820.58822210.45450.44440.4494450.68640.79060.73481910.75010.76340.75670.9659
0.000572763920.38300.76050.74960.75500.96550.70360.68280.69303720.38240.46430.4194280.86180.82380.84248400.65420.46980.54691490.65820.59090.6228880.69350.67230.68285890.84760.88150.86427510.95770.93150.9444730.58300.58820.58562210.40430.42220.4130450.68920.80100.74091910.76050.74960.75500.9655
0.000673774530.38390.76110.75470.75790.96610.7120.71770.71493720.34290.42860.3810280.84940.83930.84438400.65420.46980.54691490.65380.57950.6145880.68770.66550.67645890.84280.89210.86687510.97100.91780.9437730.62570.52940.57352210.44680.46670.4565450.68140.80630.73861910.76110.75470.75790.9661
+

Framework versions

+
    +
  • Transformers 4.16.2
    • +
  • Pytorch 1.10.2+cu113
    • +
  • Datasets 1.18.3
    • +
  • Tokenizers 0.11.0
    • +
+
+
+ +
+ + + + + +
+ + +
+ + +
 +
+ + + + +
+ +
+ + +
+ +
+
+ +
+ + + + + + + + + + + + + + + + + + + + + +