* Documents that contained words from a selection of the Dutch and English [List of Dirty Naught Obscene and Otherwise Bad Words](https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words) are removed
* Sentences with less than 3 words are removed
* Sentences with a word of more than 1000 characters are removed
* Documents with less than 5 sentences are removed
* Documents with "javascript", "lorum ipsum", "terms of use", "privacy policy", "cookie policy", "uses cookies",
"use of cookies", "use cookies", "elementen ontbreken", "deze printversie" are removed.


## Models


* The first model, `t5-base-dutch` is a re-training of the Dutch T5 base v1.0 model trained during the Flax/Jax community
week. With training complete, accuracy was improved from 0,64 to 0,70.
* The second two models are a uncased and cased version of `t5-v1.1-base`, again pre-trained from scratch on Dutch,
with a tokenizer also trained from scratch. The t5 v1.1 models are slightly different from the t5 models, and the
base models are trained with a dropout of 0.0. For fine-tuning it is intended to set this back to 0.1.
* The large cased model is a pre-trained Dutch version of `t5-v1.1-large`. Training of t5-v1.1-large proved difficult.
Without dropout regularization, the training would diverge at a certain point. With dropout training went better,
be it much slower than training the t5-model. At some point convergance was too slow to warrant further training.
The latest checkpoint, training scripts and metrics are available for reference. For actual fine-tuning the cased
base model is probably the better choice.


|                            | model   | train seq len | acc      | loss     | batch size | epochs | steps   | dropout | optim     | lr   | duration |
|----------------------------|---------|---------------|----------|----------|------------|--------|---------|---------|-----------|------|----------|
| t5-base-dutch              | T5      | 512           | 0,70     | 1,38     | 128        | 1      | 528481  | 0.1     | adafactor | 5e-3 | 2d 9h    |
| t5-v1.1-base-dutch-uncased | t5-v1.1 | 1024          | 0,73     | 1,20     | 64         | 2      | 1014525 | 0.0     | adafactor | 5e-3 | 5d 5h    |
| t5-v1.1-base-dutch-cased   | t5-v1.1 | 1024          | **0,78** | **0,96** | 64         | 2      | 1210000 | 0.0     | adafactor | 5e-3 | 6d 6h    |
| t5-v1.1-large-dutch-cased  | t5-v1.1 | 512           | 0,76     | 1,07     | 64         | 1      | 1120000 | 0.1     | adafactor | 5e-3 | 86 13h   |


The cased t5-v1.1 Dutch models were fine-tuned on summarizing the CNN Daily Mail dataset.


|                              | model   | input len | target len | Rouge1 | Rouge2 | RougeL | RougeLsum | Test Gen Len | epochs | batch size | steps | duration |
|------------------------------|---------|-----------|------------|--------|--------|--------|-----------|--------------|--------|------------|-------|----------|
| t5-v1.1-base-dutch-cnn-test  | t5-v1.1 | 1024      | 96         | 34,8   | 13,6   | 25,2   | 32,1      | 79           | 6      | 64         | 26916 | 2h 40m   |
| t5-v1.1-large-dutch-cnn-test | t5-v1.1 | 1024      | 96         | 34,4   | 13,6   | 25,3   | 31,7      | 81           | 5      | 16         | 89720 | 11h      |


## Acknowledgements


This project would not have been possible without compute generously provided by Google through the
[TPU Research Cloud](https://sites.research.google/trc/). The HuggingFace 🤗 ecosystem was also
instrumental in many, if not all parts of the training. The following repositories where helpful in setting up the TPU-VM,
and getting an idea what sensible hyper-parameters are for training gpt2 from scratch.


* [Gsarti's Pretrain and Fine-tune a T5 model with Flax on GCP](https://github.com/gsarti/t5-flax-gcp)
* [Flax/Jax Community week t5-base-dutch](https://huggingface.co/flax-community/t5-base-dutch)


Created by [Yeb Havinga](https://www.linkedin.com/in/yeb-havinga-86530825/)


model name: t5-base-dutch


model author: yhavinga


task: text2text-generation


tags: pytorch, tf, jax, tensorboard, t5, text2text-generation, nl, dataset:yhavinga/mc4_nl_cleaned, arxiv:1910.10683, transformers, seq2seq, lm-head, license:apache-2.0, autonlp_compatible


language:


- nl
datasets:
- yhavinga/mc4_nl_cleaned
tags:
- seq2seq
- lm-head
license: apache-2.0
inference: false


# T5-base pre-trained on cleaned Dutch mC4 🇳🇱


A [T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) v1.1 base model pre-trained from scratch on [Dutch mC4](https://huggingface.co/datasets/yhavinga/mc4_nl_cleaned).


* Pre-trained T5 models need to be finetuned before they can be used for downstream tasks, therefore the inference widget on the right has been turned off.
* T5 paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)


## Tokenizer


* SentencePiece tokenizer trained from scratch for Dutch on mC4 nl cleaned with scripts from the Huggingface
Transformers [Flax examples](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling).


## Dataset


All models listed below are trained on of the `full` configuration (39B tokens) of
[cleaned Dutch mC4](https://huggingface.co/datasets/yhavinga/mc4_nl_cleaned),
which is the original mC4, except


## Models


TL;DR: [yhavinga/t5-v1.1-base-dutch-cased](https://huggingface.co/yhavinga/t5-v1.1-base-dutch-cased) is the best model.


* `yhavinga/t5-base-dutch` is a re-training of the Dutch T5 base v1.0 model trained during the summer 2021
Flax/Jax community week. Accuracy was improved from 0.64 to 0.70.
* The two T5 v1.1 base models are an uncased and cased version of `t5-v1.1-base`, again pre-trained from scratch on Dutch,
with a tokenizer also trained from scratch. The t5 v1.1 models are slightly different from the t5 models, and the
base models are trained with a dropout of 0.0. For fine-tuning it is intended to set this back to 0.1.
* The large cased model is a pre-trained Dutch version of `t5-v1.1-large`. Training of t5-v1.1-large proved difficult.
Without dropout regularization, the training would diverge at a certain point. With dropout training went better,
be it much slower than training the t5-model. At some point convergance was too slow to warrant further training.
The latest checkpoint, training scripts and metrics are available for reference. For actual fine-tuning the cased
base model is probably the better choice.


|                                                                                | model   | train seq len | acc      | loss     | batch size | epochs | steps   | dropout | optim     | lr   | duration |
|---------------------------------------------------------------------------------------------------|---------|---------------|----------|----------|------------|--------|---------|---------|-----------|------|----------|
| [yhavinga/t5-base-dutch](https://huggingface.co/yhavinga/t5-base-dutch)                           | T5      | 512           | 0,70     | 1,38     | 128        | 1      | 528481  | 0.1     | adafactor | 5e-3 | 2d 9h    |
| [yhavinga/t5-v1.1-base-dutch-uncased](https://huggingface.co/yhavinga/t5-v1.1-base-dutch-uncased) | t5-v1.1 | 1024          | 0,73     | 1,20     | 64         | 2      | 1014525 | 0.0     | adafactor | 5e-3 | 5d 5h    |
| [yhavinga/t5-v1.1-base-dutch-cased](https://huggingface.co/yhavinga/t5-v1.1-base-dutch-cased)     | t5-v1.1 | 1024          | **0,78** | **0,96** | 64         | 2      | 1210000 | 0.0     | adafactor | 5e-3 | 6d 6h    |
| [yhavinga/t5-v1.1-large-dutch-cased](https://huggingface.co/yhavinga/t5-v1.1-large-dutch-cased)   | t5-v1.1 | 512           | 0,76     | 1,07     | 64         | 1      | 1120000 | 0.1     | adafactor | 5e-3 | 86 13h   |


|                                                                                | model   | input len | target len | Rouge1 | Rouge2 | RougeL | RougeLsum | Test Gen Len | epochs | batch size | steps | duration |
|-------------------------------------------------------------------------------------------------------|---------|-----------|------------|--------|--------|--------|-----------|--------------|--------|------------|-------|----------|
| [yhavinga/t5-v1.1-base-dutch-cnn-test](https://huggingface.co/yhavinga/t5-v1.1-base-dutch-cnn-test)   | t5-v1.1 | 1024      | 96         | 34,8   | 13,6   | 25,2   | 32,1      | 79           | 6      | 64         | 26916 | 2h 40m   |
| [yhavinga/t5-v1.1-large-dutch-cnn-test](https://huggingface.co/yhavinga/t5-v1.1-large-dutch-cnn-test) | t5-v1.1 | 1024      | 96         | 34,4   | 13,6   | 25,3   | 31,7      | 81           | 5      | 16         | 89720 | 11h      |


## Acknowledgements


This project would not have been possible without compute generously provided by Google through the
[TPU Research Cloud](https://sites.research.google/trc/). The HuggingFace 🤗 ecosystem was also
instrumental in many, if not all parts of the training. The following repositories where helpful in setting up the TPU-VM,
and training the models:


* [Gsarti's Pretrain and Fine-tune a T5 model with Flax on GCP](https://github.com/gsarti/t5-flax-gcp)
* [HUggingFace Flax MLM examples](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
* [Flax/Jax Community week t5-base-dutch](https://huggingface.co/flax-community/t5-base-dutch)


model name: t5-v1.1-base-dutch-uncased


model author: yhavinga


task: text2text-generation


tags: jax, tensorboard, t5, text2text-generation, nl, dataset:yhavinga/mc4_nl_cleaned, arxiv:1910.10683, transformers, seq2seq, lm-head, license:apache-2.0, autonlp_compatible


language:


- nl
datasets:
- yhavinga/mc4_nl_cleaned
- ml6team/cnn_dailymail_nl
tags:
- summarization
- t5
- seq2seq
license: apache-2.0
pipeline_tag: summarization
widget:
- text: "Het Van Goghmuseum in Amsterdam heeft vier kostbare prenten verworven van Mary Cassatt, de Amerikaanse impressionistische kunstenaar en tijdgenoot van Vincent van Gogh. Dat heeft het museum woensdagmiddag op een persconferentie bekendgemaakt. Het gaat om drie grote kleurenetsen en een zwart-wit litho met voorstellingen van vrouwen. Voor deze prenten, die afkomstig zijn van een Amerikaanse verzamelaar, betaalde het museum ruim 1,4 miljoen euro. Drie grote fondsen en een aantal particulieren hebben samen de aankoopsom beschikbaar gesteld. Mary Stevenson Cassatt (1844-1926) woonde en werkte lange tijd in Frankrijk. Ze staat met haar impressionistische schilderijen en tekeningen te boek als een van de vernieuwers van de Parijse kunstwereld in de late negentiende eeuw. Het Van Goghmuseum rekent haar prenten „tot het mooiste wat op grafisch gebied in het fin de siècle is geproduceerd”. De drie aangekochte kleurenetsen – Het doorpassen, De brief en Badende vrouw – komen uit een serie van tien waarmee Cassatt haar naam als (prent)kunstenaar definitief vestigde. Ze maakte de etsen na een bezoek in 1890 aan een tentoonstelling van Japanse prenten in Parijs. Over die expositie schreef de Amerikaanse aan haar vriendin Berthe Morisot, een andere vrouwelijke impressionist: „We kunnen de Japanse prenten in de Beaux-Arts gaan bekijken. Echt, die mag je niet missen. Als je kleurenprenten wilt maken, is er niets mooiers voorstelbaar. Ik droom ervan en denk nergens anders meer aan dan aan kleur op koper."
- text: "Afgelopen zaterdagochtend werden Hunga Tonga en Hunga Hapai opnieuw twee aparte eilanden toen de vulkaan met een hevige explosie uitbarstte. De aanloop tot de uitbarsting begon al eind vorig jaar met kleinere explosies. Begin januari nam de activiteit af en dachten geologen dat de vulkaan tot rust was gekomen. Toch barstte hij afgelopen zaterdag opnieuw uit, veel heviger dan de uitbarstingen ervoor. Vlák voor deze explosie stortte het kilometerslange verbindingsstuk in en verdween onder het water. De eruptie duurde acht minuten. De wolk van as en giftige gasdeeltjes, zoals zwaveloxide, die daarbij vrijkwam, reikte tot dertig kilometer hoogte en was zo’n vijfhonderd kilometer breed. Ter vergelijking: de pluimen uit de recente vulkaanuitbarsting op La Palma reikten maximaal zo’n vijf kilometer hoog. De hoofdstad van Tonga, vijfenzestig kilometer verderop is bedekt met een dikke laag as. Dat heeft bijvoorbeeld gevolgen voor de veiligheid van het drinkwater op Tonga. De uitbarsting van de onderzeese vulkaan in de eilandstaat Tonga afgelopen zaterdag was bijzonder heftig. De eruptie veroorzaakte een tsunami die reikte van Nieuw-Zeeland tot de Verenigde Staten en in Nederland ging de luchtdruk omhoog. Geologen verwachten niet dat de vulkaan op Tonga voor een lange wereldwijde afkoeling zorgt, zoals bij andere hevige vulkaanuitbarstingen het geval is geweest. De vulkaan ligt onder water tussen de onbewoonde eilandjes Hunga Tonga (0,39 vierkante kilometer) en Hunga Ha’apai (0,65 vierkante kilometer). Magma dat bij kleinere uitbarsting in 2009 en 2014 omhoog kwam, koelde af en vormde een verbindingsstuk tussen de twee eilanden in. Een explosie van een onderwatervulkaan als die bij Tonga is heftiger dan bijvoorbeeld die uitbarsting op La Palma. „Dat komt doordat het vulkanisme hier veroorzaakt wordt door subductie: de Pacifische plaat zinkt onder Tonga de aardmantel in en neemt water mee omlaag”, zegt hoogleraar paleogeografie Douwe van Hinsbergen van de Universiteit Utrecht. „Dit water komt met magma als gas, als waterdamp, mee omhoog. Dat voert de druk onder de aardkost enorm op. Arwen Deuss, geowetenschapper aan de Universiteit Utrecht, vergelijkt het met een fles cola. „Wanneer je een fles cola schudt, zal het gas er met veel geweld uitkomen. Dat is waarschijnlijk wat er gebeurd is op Tonga, maar we weten het niet precies.”"


# T5 v1.1 Base finetuned for CNN news summarization in Dutch 🇳🇱


This model is [t5-v1.1-base-dutch-cased](https://huggingface.co/yhavinga/t5-v1.1-base-dutch-cased) finetuned on [CNN Dailymail NL](https://huggingface.co/datasets/ml6team/cnn_dailymail_nl)


For a demo of the Dutch CNN summarization models, head over to the Hugging Face Spaces for
the **[Netherformer 📰](https://huggingface.co/spaces/flax-community/netherformer)** example application!


Rouge scores for this model are listed below.


## Tokenizer


## Dataset


* Documents that contained words from a selection of the Dutch and English [List of Dirty Naught Obscene and Otherwise Bad Words](https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words) are removed
* Sentences with less than 3 words are removed
* Sentences with a word of more than 1000 characters are removed
* Documents with less than 5 sentences are removed
* Documents with "javascript", "lorum ipsum", "terms of use", "privacy policy", "cookie policy", "uses cookies",
"use of cookies", "use cookies", "elementen ontbreken", "deze printversie" are removed.


## Models


The cased t5-v1.1 Dutch models were fine-tuned on summarizing the CNN Daily Mail dataset.


## Acknowledgements


Created by [Yeb Havinga](https://www.linkedin.com/in/yeb-havinga-86530825/)


model name: t5-v1.1-base-dutch-cnn-test


model author: yhavinga


task: summarization


tags: pytorch, jax, tensorboard, t5, text2text-generation, nl, dataset:yhavinga/mc4_nl_cleaned, dataset:ml6team/cnn_dailymail_nl, transformers, summarization, seq2seq, license:apache-2.0, autonlp_compatible


language: nl
widget:


- text: "In het jaar 2030 zullen we"
- text: "Toen ik gisteren volledig in de ban was van"
- text: "Studenten en leraren van de Bogazici Universiteit in de Turkse stad Istanbul"
- text: "In Israël was een strenge lockdown"
tags:
- gpt2-medium
- gpt2
pipeline_tag: text-generation
datasets:
- yhavinga/mc4_nl_cleaned


# GPT2-Medium pre-trained on cleaned Dutch mC4 🇳🇱


A GPT2 medium-sized model (345M parameters) trained from scratch on Dutch, with perplexity 15.1 on cleaned Dutch mC4.


## How To Use


You can use this GPT2-model directly with a pipeline for text generation.


```python
MODEL_DIR='yhavinga/gpt2-medium-dutch'
from transformers import pipeline, GPT2Tokenizer, GPT2LMHeadModel
tokenizer = GPT2Tokenizer.from_pretrained(MODEL_DIR)
model = GPT2LMHeadModel.from_pretrained(MODEL_DIR)
generator = pipeline('text-generation', model, tokenizer=tokenizer, config={'max_length':100})

generated_text = generator('In Antwerpen heeft zich gisteren', max_length=100, do_sample=True, top_k=40, top_p=0.95, repetition_penalty=2.0))
```


*"In Antwerpen heeft zich gisteren" - " een dramatische ontknoping voorgedaan in de Vlaamse deelregering. De VLD, die sinds afgelopen woensdag aan het bewind is in Vlaams-Waals gebied (de zogenaamde gewestelijke en niet rechtstreeks met Vlaanderen samenwerkende gewesten), krijgt toch geen meerderheidszetels bij verkiezingen voor gemeenteraadsverkiezingen in oktober of november volgend jaar in Westmalle, Berchem, Tervuren enz., aldus premier Jean-Pierre Van Cauwenberghe van Wallonië vandaag"*


## Tokenizer


* BPE tokenizer trained from scratch for Dutch on mC4 nl cleaned with scripts from the Huggingface
Transformers [Flax examples](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling).


## Dataset


This model was trained on of the `full` configuration (33B tokens) of
[cleaned Dutch mC4](https://huggingface.co/datasets/yhavinga/mc4_nl_cleaned),
which is the original mC4, except


## Models


TL;DR: [yhavinga/gpt2-medium-dutch](https://huggingface.co/yhavinga/gpt2-medium-dutch) is the best model.


* The models with `a`/`b` in the step-column have been trained to step `a` of a total of `b` steps.


|                                                                                | model   | params | train seq len | ppl  | loss | batch size | epochs | steps           | optim     | lr     | duration | config    |
|-----------------------------------------------------------------------------------|---------|--------|---------------|------|------|------------|--------|-----------------|-----------|--------|----------|-----------|
| [yhavinga/gpt-neo-125M-dutch](https://huggingface.co/yhavinga/gpt-neo-125M-dutch) | gpt neo | 125M   | 512           | 20.9 | 3.04 | 128        | 1      | 190000/558608          | adafactor | 2.4e-3 | 1d 12h   | full |
| [yhavinga/gpt2-medium-dutch](https://huggingface.co/yhavinga/gpt2-medium-dutch)   | gpt2    | 345M   | 512           | 15.1 | 2.71 | 128        | 1      | 320000/520502   | adafactor | 8e-4   | 7d 2h    | full      |
| [yhavinga/gpt2-large-dutch](https://huggingface.co/yhavinga/gpt2-large-dutch)     | gpt2    | 762M   | 512           | 15.1 | 2.72 | 32         | 1      | 1100000/2082009 | adafactor | 3.3e-5 | 8d 15h   | large     |
| [yhavinga/gpt-neo-1.3B-dutch](https://huggingface.co/yhavinga/gpt-neo-1.3B-dutch) | gpt neo | 1.3B   | 512           | 16.0 | 2.77 | 16         | 1      | 960000/3049896  | adafactor | 5e-4   | 7d 11h   | full      |


## Acknowledgements


This project would not have been possible without compute generously provided by Google through the
[TPU Research Cloud](https://sites.research.google/trc/). The HuggingFace 🤗 ecosystem was also
instrumental in most, if not all, parts of the training. The following repositories where helpful in setting up the TPU-VM,
and training the models:


* [Gsarti's Pretrain and Fine-tune a T5 model with Flax on GCP](https://github.com/gsarti/t5-flax-gcp)
* [HUggingFace Flax MLM examples](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
* [gpt2-medium-persian](https://huggingface.co/flax-community/gpt2-medium-persian)
* [gpt2-medium-indonesian](https://huggingface.co/flax-community/gpt2-medium-persian)


model name: gpt2-medium-dutch


model author: yhavinga


task: text-generation


tags: pytorch, jax, tensorboard, gpt2, text-generation, nl, dataset:yhavinga/mc4_nl_cleaned, transformers, gpt2-medium


language:


- en
- nl
tags:
- translation
datasets:
- opus_books
metrics:
- bleu
widget:
- text: "translate English to Dutch: The International Space Station (ISS) will continue working until 2030, before plunging into the Pacific Ocean in early 2031, according to Nasa. In a report this week, the US space agency said the ISS would crash into a part of the ocean known. This is the point furthest from land on planet Earth, also known as the spacecraft cemetery. Many old satellites and other space debris have crashed there, including the Russian space station Mir in 2001."


Test model


model name: mt5-base-test


task: translation


tags: pytorch, mt5, text2text-generation, en, nl, dataset:opus_books, transformers, translation, autonlp_compatible


tags:


- conversational


# Jake Peralta DialoGPT model


model name: DialoGPT-small-jake


model author: gizmo-dev


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


license: mit
tags:


- generated_from_trainer
datasets:
- conll2003
metrics:
- precision
- recall
- f1
- accuracy
model_index:
- name: xlm-roberta-base-finetuned-ner
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: conll2003
type: conll2003
args: conll2003
metric:
name: Accuracy
type: accuracy
value: 0.9463931352557172


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# xlm-roberta-base-finetuned-ner


This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the conll2003 dataset.
It achieves the following results on the evaluation set:


- Loss: 0.2327
- Precision: 0.7363
- Recall: 0.7265
- F1: 0.7314
- Accuracy: 0.9464


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 5
- eval_batch_size: 5
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 300


### Training results


| Training Loss | Epoch | Step   | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:------:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.9632        | 1.0   | 2809   | 0.9072          | 0.0       | 0.0    | 0.0    | 0.7879   |
| 0.7652        | 2.0   | 5618   | 0.7899          | 0.0       | 0.0    | 0.0    | 0.7880   |
| 0.7118        | 3.0   | 8427   | 0.7207          | 0.1429    | 0.0004 | 0.0007 | 0.7883   |
| 0.6548        | 4.0   | 11236  | 0.6674          | 0.2934    | 0.0107 | 0.0206 | 0.7929   |
| 0.622         | 5.0   | 14045  | 0.6234          | 0.3741    | 0.0460 | 0.0819 | 0.8064   |
| 0.5802        | 6.0   | 16854  | 0.5871          | 0.4617    | 0.1024 | 0.1677 | 0.8230   |
| 0.5605        | 7.0   | 19663  | 0.5557          | 0.5237    | 0.1611 | 0.2464 | 0.8380   |
| 0.5445        | 8.0   | 22472  | 0.5297          | 0.5631    | 0.2264 | 0.3230 | 0.8514   |
| 0.5138        | 9.0   | 25281  | 0.5075          | 0.5925    | 0.2896 | 0.3890 | 0.8634   |
| 0.5029        | 10.0  | 28090  | 0.4879          | 0.6077    | 0.3405 | 0.4364 | 0.8730   |
| 0.4813        | 11.0  | 30899  | 0.4716          | 0.6194    | 0.3822 | 0.4727 | 0.8807   |
| 0.4606        | 12.0  | 33708  | 0.4564          | 0.6306    | 0.4184 | 0.5030 | 0.8873   |
| 0.4616        | 13.0  | 36517  | 0.4431          | 0.6396    | 0.4482 | 0.5271 | 0.8929   |
| 0.4366        | 14.0  | 39326  | 0.4315          | 0.6441    | 0.4681 | 0.5422 | 0.8968   |
| 0.4334        | 15.0  | 42135  | 0.4218          | 0.6489    | 0.4830 | 0.5538 | 0.8995   |
| 0.4259        | 16.0  | 44944  | 0.4112          | 0.6485    | 0.5070 | 0.5691 | 0.9040   |
| 0.3912        | 17.0  | 47753  | 0.4031          | 0.6526    | 0.5159 | 0.5763 | 0.9058   |
| 0.4274        | 18.0  | 50562  | 0.3955          | 0.6557    | 0.5294 | 0.5858 | 0.9083   |
| 0.4034        | 19.0  | 53371  | 0.3885          | 0.6608    | 0.5407 | 0.5948 | 0.9106   |
| 0.3952        | 20.0  | 56180  | 0.3819          | 0.6620    | 0.5523 | 0.6022 | 0.9126   |
| 0.3862        | 21.0  | 58989  | 0.3755          | 0.6622    | 0.5652 | 0.6099 | 0.9148   |
| 0.3887        | 22.0  | 61798  | 0.3698          | 0.6662    | 0.5725 | 0.6158 | 0.9163   |
| 0.3764        | 23.0  | 64607  | 0.3648          | 0.6671    | 0.5788 | 0.6198 | 0.9176   |
| 0.3791        | 24.0  | 67416  | 0.3599          | 0.6686    | 0.5838 | 0.6234 | 0.9185   |
| 0.3684        | 25.0  | 70225  | 0.3551          | 0.6684    | 0.5926 | 0.6282 | 0.9201   |
| 0.3573        | 26.0  | 73034  | 0.3515          | 0.6717    | 0.5954 | 0.6312 | 0.9210   |
| 0.367         | 27.0  | 75843  | 0.3470          | 0.6711    | 0.6022 | 0.6348 | 0.9221   |
| 0.3714        | 28.0  | 78652  | 0.3433          | 0.6735    | 0.6085 | 0.6393 | 0.9233   |
| 0.3594        | 29.0  | 81461  | 0.3400          | 0.6738    | 0.6109 | 0.6408 | 0.9239   |
| 0.3626        | 30.0  | 84270  | 0.3366          | 0.6765    | 0.6159 | 0.6448 | 0.9249   |
| 0.3519        | 31.0  | 87079  | 0.3334          | 0.6765    | 0.6183 | 0.6461 | 0.9254   |
| 0.3591        | 32.0  | 89888  | 0.3305          | 0.6767    | 0.6220 | 0.6482 | 0.9263   |
| 0.3424        | 33.0  | 92697  | 0.3279          | 0.6785    | 0.6243 | 0.6502 | 0.9268   |
| 0.3514        | 34.0  | 95506  | 0.3249          | 0.6794    | 0.6290 | 0.6532 | 0.9277   |
| 0.3463        | 35.0  | 98315  | 0.3226          | 0.6806    | 0.6302 | 0.6544 | 0.9279   |
| 0.3516        | 36.0  | 101124 | 0.3200          | 0.6812    | 0.6327 | 0.6561 | 0.9283   |
| 0.3307        | 37.0  | 103933 | 0.3178          | 0.6809    | 0.6355 | 0.6574 | 0.9290   |
| 0.343         | 38.0  | 106742 | 0.3155          | 0.6830    | 0.6384 | 0.6599 | 0.9294   |
| 0.3415        | 39.0  | 109551 | 0.3134          | 0.6842    | 0.6416 | 0.6622 | 0.9299   |
| 0.3307        | 40.0  | 112360 | 0.3112          | 0.6843    | 0.6437 | 0.6634 | 0.9305   |
| 0.3428        | 41.0  | 115169 | 0.3093          | 0.6839    | 0.6455 | 0.6641 | 0.9310   |
| 0.3348        | 42.0  | 117978 | 0.3074          | 0.6849    | 0.6474 | 0.6656 | 0.9312   |
| 0.3282        | 43.0  | 120787 | 0.3057          | 0.6848    | 0.6486 | 0.6662 | 0.9316   |
| 0.3346        | 44.0  | 123596 | 0.3040          | 0.6860    | 0.6497 | 0.6674 | 0.9318   |
| 0.3349        | 45.0  | 126405 | 0.3023          | 0.6867    | 0.6524 | 0.6691 | 0.9324   |
| 0.3323        | 46.0  | 129214 | 0.3010          | 0.6895    | 0.6522 | 0.6703 | 0.9326   |
| 0.3258        | 47.0  | 132023 | 0.2992          | 0.6901    | 0.6549 | 0.6720 | 0.9331   |
| 0.3276        | 48.0  | 134832 | 0.2978          | 0.6915    | 0.6563 | 0.6734 | 0.9334   |
| 0.3345        | 49.0  | 137641 | 0.2962          | 0.6916    | 0.6585 | 0.6746 | 0.9337   |
| 0.3138        | 50.0  | 140450 | 0.2949          | 0.6926    | 0.6594 | 0.6756 | 0.9339   |
| 0.3285        | 51.0  | 143259 | 0.2935          | 0.6928    | 0.6601 | 0.6760 | 0.9340   |
| 0.3135        | 52.0  | 146068 | 0.2925          | 0.6931    | 0.6610 | 0.6767 | 0.9343   |
| 0.3206        | 53.0  | 148877 | 0.2914          | 0.6955    | 0.6624 | 0.6785 | 0.9346   |
| 0.3105        | 54.0  | 151686 | 0.2899          | 0.6953    | 0.6638 | 0.6792 | 0.9348   |
| 0.3045        | 55.0  | 154495 | 0.2887          | 0.6968    | 0.6651 | 0.6806 | 0.9352   |
| 0.3082        | 56.0  | 157304 | 0.2875          | 0.6985    | 0.6680 | 0.6829 | 0.9356   |
| 0.3229        | 57.0  | 160113 | 0.2865          | 0.6998    | 0.6688 | 0.6840 | 0.9357   |
| 0.3113        | 58.0  | 162922 | 0.2855          | 0.7012    | 0.6704 | 0.6855 | 0.9361   |
| 0.3047        | 59.0  | 165731 | 0.2843          | 0.7010    | 0.6713 | 0.6858 | 0.9362   |
| 0.3028        | 60.0  | 168540 | 0.2833          | 0.7024    | 0.6728 | 0.6873 | 0.9365   |
| 0.3082        | 61.0  | 171349 | 0.2826          | 0.7050    | 0.6740 | 0.6892 | 0.9367   |
| 0.3054        | 62.0  | 174158 | 0.2814          | 0.7038    | 0.6752 | 0.6892 | 0.9369   |
| 0.3124        | 63.0  | 176967 | 0.2804          | 0.7043    | 0.6770 | 0.6904 | 0.9372   |
| 0.3138        | 64.0  | 179776 | 0.2796          | 0.7034    | 0.6783 | 0.6906 | 0.9374   |
| 0.3022        | 65.0  | 182585 | 0.2787          | 0.7045    | 0.6785 | 0.6912 | 0.9374   |
| 0.3142        | 66.0  | 185394 | 0.2778          | 0.7059    | 0.6792 | 0.6923 | 0.9377   |
| 0.3043        | 67.0  | 188203 | 0.2770          | 0.7074    | 0.6808 | 0.6938 | 0.9381   |
| 0.3053        | 68.0  | 191012 | 0.2762          | 0.7069    | 0.6813 | 0.6938 | 0.9381   |
| 0.3147        | 69.0  | 193821 | 0.2752          | 0.7065    | 0.6832 | 0.6947 | 0.9383   |
| 0.2998        | 70.0  | 196630 | 0.2746          | 0.7086    | 0.6831 | 0.6956 | 0.9384   |
| 0.2951        | 71.0  | 199439 | 0.2739          | 0.7087    | 0.6831 | 0.6957 | 0.9385   |
| 0.3087        | 72.0  | 202248 | 0.2733          | 0.7089    | 0.6838 | 0.6961 | 0.9385   |
| 0.3059        | 73.0  | 205057 | 0.2723          | 0.7087    | 0.6867 | 0.6976 | 0.9389   |
| 0.2983        | 74.0  | 207866 | 0.2717          | 0.7080    | 0.6860 | 0.6968 | 0.9389   |
| 0.2994        | 75.0  | 210675 | 0.2710          | 0.7094    | 0.6867 | 0.6978 | 0.9390   |
| 0.3056        | 76.0  | 213484 | 0.2706          | 0.7090    | 0.6854 | 0.6970 | 0.9389   |
| 0.3118        | 77.0  | 216293 | 0.2698          | 0.7099    | 0.6869 | 0.6982 | 0.9391   |
| 0.296         | 78.0  | 219102 | 0.2691          | 0.7093    | 0.6886 | 0.6988 | 0.9393   |
| 0.3111        | 79.0  | 221911 | 0.2687          | 0.7111    | 0.6885 | 0.6996 | 0.9395   |
| 0.2961        | 80.0  | 224720 | 0.2678          | 0.7103    | 0.6895 | 0.6997 | 0.9397   |
| 0.3043        | 81.0  | 227529 | 0.2674          | 0.7111    | 0.6899 | 0.7003 | 0.9399   |
| 0.2924        | 82.0  | 230338 | 0.2667          | 0.7125    | 0.6920 | 0.7021 | 0.9401   |
| 0.2947        | 83.0  | 233147 | 0.2660          | 0.7107    | 0.6920 | 0.7012 | 0.9402   |
| 0.3035        | 84.0  | 235956 | 0.2656          | 0.7126    | 0.6922 | 0.7023 | 0.9402   |
| 0.3034        | 85.0  | 238765 | 0.2648          | 0.7133    | 0.6937 | 0.7034 | 0.9404   |
| 0.297         | 86.0  | 241574 | 0.2645          | 0.7143    | 0.6946 | 0.7043 | 0.9406   |
| 0.2943        | 87.0  | 244383 | 0.2639          | 0.7145    | 0.6955 | 0.7049 | 0.9407   |
| 0.2929        | 88.0  | 247192 | 0.2636          | 0.7125    | 0.6940 | 0.7031 | 0.9406   |
| 0.2974        | 89.0  | 250001 | 0.2628          | 0.7149    | 0.6975 | 0.7061 | 0.9410   |
| 0.2917        | 90.0  | 252810 | 0.2626          | 0.7143    | 0.6949 | 0.7045 | 0.9408   |
| 0.3031        | 91.0  | 255619 | 0.2620          | 0.7147    | 0.6958 | 0.7051 | 0.9409   |
| 0.3053        | 92.0  | 258428 | 0.2612          | 0.7149    | 0.6977 | 0.7062 | 0.9411   |
| 0.2921        | 93.0  | 261237 | 0.2610          | 0.7164    | 0.6969 | 0.7065 | 0.9411   |
| 0.2934        | 94.0  | 264046 | 0.2606          | 0.7160    | 0.6969 | 0.7063 | 0.9412   |
| 0.2863        | 95.0  | 266855 | 0.2601          | 0.7160    | 0.6973 | 0.7066 | 0.9412   |
| 0.2918        | 96.0  | 269664 | 0.2595          | 0.7167    | 0.6986 | 0.7076 | 0.9413   |
| 0.2926        | 97.0  | 272473 | 0.2591          | 0.7171    | 0.7004 | 0.7086 | 0.9415   |
| 0.2844        | 98.0  | 275282 | 0.2588          | 0.7171    | 0.6997 | 0.7083 | 0.9414   |
| 0.2924        | 99.0  | 278091 | 0.2585          | 0.7175    | 0.6986 | 0.7080 | 0.9414   |
| 0.2931        | 100.0 | 280900 | 0.2580          | 0.7178    | 0.6997 | 0.7086 | 0.9415   |
| 0.289         | 101.0 | 283709 | 0.2575          | 0.7184    | 0.7013 | 0.7098 | 0.9417   |
| 0.2892        | 102.0 | 286518 | 0.2570          | 0.7178    | 0.7024 | 0.7100 | 0.9418   |
| 0.285         | 103.0 | 289327 | 0.2567          | 0.7184    | 0.7013 | 0.7098 | 0.9417   |
| 0.2809        | 104.0 | 292136 | 0.2565          | 0.7192    | 0.7013 | 0.7102 | 0.9418   |
| 0.2802        | 105.0 | 294945 | 0.2561          | 0.7198    | 0.7014 | 0.7105 | 0.9420   |
| 0.2878        | 106.0 | 297754 | 0.2556          | 0.7192    | 0.7022 | 0.7106 | 0.9419   |
| 0.2853        | 107.0 | 300563 | 0.2554          | 0.7201    | 0.7017 | 0.7108 | 0.9420   |
| 0.2871        | 108.0 | 303372 | 0.2549          | 0.7203    | 0.7038 | 0.7119 | 0.9422   |
| 0.2904        | 109.0 | 306181 | 0.2545          | 0.7205    | 0.7043 | 0.7123 | 0.9422   |
| 0.2848        | 110.0 | 308990 | 0.2543          | 0.7203    | 0.7031 | 0.7116 | 0.9423   |
| 0.2933        | 111.0 | 311799 | 0.2538          | 0.7198    | 0.7046 | 0.7121 | 0.9423   |
| 0.2885        | 112.0 | 314608 | 0.2534          | 0.7198    | 0.7056 | 0.7126 | 0.9425   |
| 0.2813        | 113.0 | 317417 | 0.2532          | 0.7205    | 0.7058 | 0.7131 | 0.9425   |
| 0.2858        | 114.0 | 320226 | 0.2528          | 0.7202    | 0.7067 | 0.7134 | 0.9426   |
| 0.2871        | 115.0 | 323035 | 0.2525          | 0.7216    | 0.7075 | 0.7145 | 0.9427   |
| 0.2725        | 116.0 | 325844 | 0.2522          | 0.7220    | 0.7065 | 0.7142 | 0.9428   |
| 0.2887        | 117.0 | 328653 | 0.2519          | 0.7222    | 0.7068 | 0.7144 | 0.9428   |
| 0.2773        | 118.0 | 331462 | 0.2514          | 0.7211    | 0.7079 | 0.7145 | 0.9428   |
| 0.2831        | 119.0 | 334271 | 0.2513          | 0.7227    | 0.7078 | 0.7152 | 0.9429   |
| 0.2924        | 120.0 | 337080 | 0.2508          | 0.7239    | 0.7091 | 0.7164 | 0.9431   |
| 0.2944        | 121.0 | 339889 | 0.2507          | 0.7244    | 0.7090 | 0.7166 | 0.9431   |
| 0.2887        | 122.0 | 342698 | 0.2506          | 0.7248    | 0.7088 | 0.7167 | 0.9431   |
| 0.2826        | 123.0 | 345507 | 0.2501          | 0.7247    | 0.7100 | 0.7173 | 0.9432   |
| 0.2795        | 124.0 | 348316 | 0.2500          | 0.7247    | 0.7090 | 0.7167 | 0.9431   |
| 0.2855        | 125.0 | 351125 | 0.2496          | 0.7259    | 0.7104 | 0.7180 | 0.9433   |
| 0.2797        | 126.0 | 353934 | 0.2494          | 0.7244    | 0.7101 | 0.7171 | 0.9433   |
| 0.2804        | 127.0 | 356743 | 0.2491          | 0.7247    | 0.7097 | 0.7171 | 0.9433   |
| 0.286         | 128.0 | 359552 | 0.2488          | 0.7238    | 0.7096 | 0.7166 | 0.9433   |
| 0.2785        | 129.0 | 362361 | 0.2487          | 0.7237    | 0.7091 | 0.7163 | 0.9432   |
| 0.284         | 130.0 | 365170 | 0.2484          | 0.7238    | 0.7104 | 0.7170 | 0.9434   |
| 0.2757        | 131.0 | 367979 | 0.2480          | 0.725     | 0.7117 | 0.7183 | 0.9436   |
| 0.286         | 132.0 | 370788 | 0.2477          | 0.7248    | 0.7117 | 0.7182 | 0.9436   |
| 0.2874        | 133.0 | 373597 | 0.2476          | 0.7249    | 0.7115 | 0.7181 | 0.9436   |
| 0.2796        | 134.0 | 376406 | 0.2474          | 0.7249    | 0.7119 | 0.7183 | 0.9437   |
| 0.2851        | 135.0 | 379215 | 0.2471          | 0.7247    | 0.7115 | 0.7180 | 0.9437   |
| 0.2833        | 136.0 | 382024 | 0.2469          | 0.7255    | 0.7124 | 0.7189 | 0.9438   |
| 0.2859        | 137.0 | 384833 | 0.2466          | 0.7261    | 0.7126 | 0.7193 | 0.9438   |
| 0.2903        | 138.0 | 387642 | 0.2464          | 0.7261    | 0.7131 | 0.7195 | 0.9439   |
| 0.2836        | 139.0 | 390451 | 0.2462          | 0.7262    | 0.7129 | 0.7195 | 0.9439   |
| 0.282         | 140.0 | 393260 | 0.2461          | 0.7258    | 0.7119 | 0.7188 | 0.9438   |
| 0.2886        | 141.0 | 396069 | 0.2459          | 0.7254    | 0.7127 | 0.7190 | 0.9439   |
| 0.2759        | 142.0 | 398878 | 0.2457          | 0.7259    | 0.7130 | 0.7194 | 0.9439   |
| 0.2701        | 143.0 | 401687 | 0.2455          | 0.7267    | 0.7132 | 0.7199 | 0.9439   |
| 0.2872        | 144.0 | 404496 | 0.2452          | 0.7262    | 0.7135 | 0.7198 | 0.9440   |
| 0.2797        | 145.0 | 407305 | 0.2451          | 0.7264    | 0.7135 | 0.7199 | 0.9440   |
| 0.2798        | 146.0 | 410114 | 0.2449          | 0.7256    | 0.7130 | 0.7192 | 0.9440   |
| 0.2677        | 147.0 | 412923 | 0.2446          | 0.7264    | 0.7139 | 0.7201 | 0.9441   |
| 0.2713        | 148.0 | 415732 | 0.2445          | 0.7264    | 0.7129 | 0.7196 | 0.9440   |
| 0.2736        | 149.0 | 418541 | 0.2442          | 0.7268    | 0.7141 | 0.7204 | 0.9442   |
| 0.2807        | 150.0 | 421350 | 0.2440          | 0.7270    | 0.7143 | 0.7206 | 0.9442   |
| 0.2777        | 151.0 | 424159 | 0.2437          | 0.7269    | 0.7151 | 0.7210 | 0.9443   |
| 0.2703        | 152.0 | 426968 | 0.2437          | 0.7279    | 0.7153 | 0.7215 | 0.9444   |
| 0.2701        | 153.0 | 429777 | 0.2434          | 0.7277    | 0.7153 | 0.7214 | 0.9444   |
| 0.2693        | 154.0 | 432586 | 0.2433          | 0.7271    | 0.7148 | 0.7209 | 0.9443   |
| 0.2894        | 155.0 | 435395 | 0.2430          | 0.7275    | 0.7158 | 0.7216 | 0.9445   |
| 0.2855        | 156.0 | 438204 | 0.2430          | 0.7290    | 0.7165 | 0.7227 | 0.9446   |
| 0.2874        | 157.0 | 441013 | 0.2428          | 0.7292    | 0.7178 | 0.7235 | 0.9448   |
| 0.2745        | 158.0 | 443822 | 0.2427          | 0.7296    | 0.7171 | 0.7233 | 0.9448   |
| 0.2842        | 159.0 | 446631 | 0.2424          | 0.7294    | 0.7180 | 0.7236 | 0.9448   |
| 0.281         | 160.0 | 449440 | 0.2423          | 0.7293    | 0.7177 | 0.7234 | 0.9448   |
| 0.2655        | 161.0 | 452249 | 0.2421          | 0.7293    | 0.7183 | 0.7237 | 0.9448   |
| 0.2701        | 162.0 | 455058 | 0.2419          | 0.7287    | 0.7170 | 0.7228 | 0.9447   |
| 0.2787        | 163.0 | 457867 | 0.2418          | 0.7286    | 0.7170 | 0.7227 | 0.9446   |
| 0.2779        | 164.0 | 460676 | 0.2416          | 0.7287    | 0.7174 | 0.7230 | 0.9447   |
| 0.2926        | 165.0 | 463485 | 0.2416          | 0.7299    | 0.7172 | 0.7235 | 0.9447   |
| 0.2751        | 166.0 | 466294 | 0.2413          | 0.7298    | 0.7185 | 0.7241 | 0.9449   |
| 0.2756        | 167.0 | 469103 | 0.2412          | 0.7299    | 0.7185 | 0.7242 | 0.9449   |
| 0.2792        | 168.0 | 471912 | 0.2411          | 0.7301    | 0.7184 | 0.7242 | 0.9449   |
| 0.2722        | 169.0 | 474721 | 0.2409          | 0.7305    | 0.7190 | 0.7247 | 0.9450   |
| 0.2719        | 170.0 | 477530 | 0.2408          | 0.7306    | 0.7184 | 0.7245 | 0.9449   |
| 0.2736        | 171.0 | 480339 | 0.2407          | 0.7307    | 0.7188 | 0.7247 | 0.9450   |
| 0.2805        | 172.0 | 483148 | 0.2404          | 0.7311    | 0.7199 | 0.7255 | 0.9451   |
| 0.2762        | 173.0 | 485957 | 0.2402          | 0.7313    | 0.7205 | 0.7259 | 0.9452   |
| 0.2717        | 174.0 | 488766 | 0.2402          | 0.7316    | 0.7195 | 0.7255 | 0.9451   |
| 0.2657        | 175.0 | 491575 | 0.2400          | 0.7314    | 0.7195 | 0.7254 | 0.9451   |
| 0.276         | 176.0 | 494384 | 0.2398          | 0.7309    | 0.7197 | 0.7253 | 0.9452   |
| 0.2767        | 177.0 | 497193 | 0.2397          | 0.7314    | 0.7202 | 0.7258 | 0.9452   |
| 0.2672        | 178.0 | 500002 | 0.2396          | 0.7309    | 0.7197 | 0.7252 | 0.9451   |
| 0.2727        | 179.0 | 502811 | 0.2395          | 0.7316    | 0.7202 | 0.7258 | 0.9453   |
| 0.2746        | 180.0 | 505620 | 0.2394          | 0.7314    | 0.7202 | 0.7258 | 0.9453   |
| 0.2704        | 181.0 | 508429 | 0.2392          | 0.7312    | 0.7203 | 0.7257 | 0.9453   |
| 0.2927        | 182.0 | 511238 | 0.2392          | 0.7314    | 0.7199 | 0.7256 | 0.9453   |
| 0.2705        | 183.0 | 514047 | 0.2391          | 0.7316    | 0.7199 | 0.7257 | 0.9453   |
| 0.2668        | 184.0 | 516856 | 0.2390          | 0.7318    | 0.7198 | 0.7258 | 0.9453   |
| 0.2562        | 185.0 | 519665 | 0.2388          | 0.7307    | 0.7187 | 0.7246 | 0.9451   |
| 0.2642        | 186.0 | 522474 | 0.2387          | 0.7314    | 0.7197 | 0.7255 | 0.9452   |
| 0.2688        | 187.0 | 525283 | 0.2385          | 0.7316    | 0.7205 | 0.7260 | 0.9453   |
| 0.284         | 188.0 | 528092 | 0.2384          | 0.7313    | 0.7202 | 0.7257 | 0.9453   |
| 0.2656        | 189.0 | 530901 | 0.2383          | 0.7321    | 0.7210 | 0.7265 | 0.9454   |
| 0.2724        | 190.0 | 533710 | 0.2383          | 0.7324    | 0.7215 | 0.7269 | 0.9455   |
| 0.2815        | 191.0 | 536519 | 0.2382          | 0.7322    | 0.7209 | 0.7265 | 0.9454   |
| 0.2847        | 192.0 | 539328 | 0.2380          | 0.7320    | 0.7219 | 0.7269 | 0.9455   |
| 0.2686        | 193.0 | 542137 | 0.2379          | 0.7323    | 0.7223 | 0.7273 | 0.9456   |
| 0.2641        | 194.0 | 544946 | 0.2378          | 0.7326    | 0.7220 | 0.7272 | 0.9455   |
| 0.2871        | 195.0 | 547755 | 0.2377          | 0.7319    | 0.7220 | 0.7269 | 0.9455   |
| 0.2682        | 196.0 | 550564 | 0.2376          | 0.7331    | 0.7222 | 0.7276 | 0.9456   |
| 0.2772        | 197.0 | 553373 | 0.2376          | 0.7328    | 0.7216 | 0.7272 | 0.9456   |
| 0.2781        | 198.0 | 556182 | 0.2375          | 0.7330    | 0.7218 | 0.7273 | 0.9456   |
| 0.2612        | 199.0 | 558991 | 0.2373          | 0.7328    | 0.7223 | 0.7275 | 0.9456   |
| 0.2788        | 200.0 | 561800 | 0.2372          | 0.7332    | 0.7225 | 0.7278 | 0.9457   |
| 0.2797        | 201.0 | 564609 | 0.2371          | 0.7326    | 0.7223 | 0.7274 | 0.9456   |
| 0.2641        | 202.0 | 567418 | 0.2370          | 0.7331    | 0.7226 | 0.7278 | 0.9457   |
| 0.2742        | 203.0 | 570227 | 0.2369          | 0.7334    | 0.7225 | 0.7279 | 0.9457   |
| 0.2622        | 204.0 | 573036 | 0.2369          | 0.7339    | 0.7234 | 0.7286 | 0.9458   |
| 0.2732        | 205.0 | 575845 | 0.2367          | 0.7333    | 0.7232 | 0.7282 | 0.9457   |
| 0.264         | 206.0 | 578654 | 0.2366          | 0.7334    | 0.7230 | 0.7282 | 0.9457   |
| 0.27          | 207.0 | 581463 | 0.2366          | 0.7339    | 0.7231 | 0.7284 | 0.9457   |
| 0.2808        | 208.0 | 584272 | 0.2364          | 0.7331    | 0.7227 | 0.7279 | 0.9457   |
| 0.2881        | 209.0 | 587081 | 0.2364          | 0.7333    | 0.7228 | 0.7280 | 0.9457   |
| 0.2723        | 210.0 | 589890 | 0.2364          | 0.7335    | 0.7232 | 0.7283 | 0.9457   |
| 0.2696        | 211.0 | 592699 | 0.2362          | 0.7332    | 0.7236 | 0.7284 | 0.9458   |
| 0.2729        | 212.0 | 595508 | 0.2362          | 0.7334    | 0.7236 | 0.7284 | 0.9458   |
| 0.265         | 213.0 | 598317 | 0.2361          | 0.7332    | 0.7235 | 0.7283 | 0.9458   |
| 0.2816        | 214.0 | 601126 | 0.2360          | 0.7329    | 0.7236 | 0.7283 | 0.9458   |
| 0.273         | 215.0 | 603935 | 0.2359          | 0.7339    | 0.7241 | 0.7290 | 0.9458   |
| 0.2681        | 216.0 | 606744 | 0.2359          | 0.7338    | 0.7239 | 0.7288 | 0.9458   |
| 0.2648        | 217.0 | 609553 | 0.2358          | 0.7342    | 0.7242 | 0.7292 | 0.9459   |
| 0.269         | 218.0 | 612362 | 0.2357          | 0.7341    | 0.7237 | 0.7289 | 0.9458   |
| 0.277         | 219.0 | 615171 | 0.2357          | 0.7346    | 0.7239 | 0.7292 | 0.9458   |
| 0.266         | 220.0 | 617980 | 0.2356          | 0.7344    | 0.7246 | 0.7295 | 0.9460   |
| 0.2737        | 221.0 | 620789 | 0.2355          | 0.7345    | 0.7249 | 0.7297 | 0.9460   |
| 0.2779        | 222.0 | 623598 | 0.2356          | 0.7345    | 0.7239 | 0.7292 | 0.9459   |
| 0.2834        | 223.0 | 626407 | 0.2354          | 0.7349    | 0.7249 | 0.7298 | 0.9460   |
| 0.273         | 224.0 | 629216 | 0.2354          | 0.7349    | 0.7249 | 0.7299 | 0.9460   |
| 0.2691        | 225.0 | 632025 | 0.2353          | 0.7345    | 0.7251 | 0.7298 | 0.9460   |
| 0.2696        | 226.0 | 634834 | 0.2352          | 0.7345    | 0.7254 | 0.7299 | 0.9461   |
| 0.2643        | 227.0 | 637643 | 0.2352          | 0.7350    | 0.7244 | 0.7296 | 0.9460   |
| 0.2685        | 228.0 | 640452 | 0.2351          | 0.7349    | 0.7250 | 0.7300 | 0.9461   |
| 0.2818        | 229.0 | 643261 | 0.2350          | 0.7346    | 0.7250 | 0.7298 | 0.9461   |
| 0.2848        | 230.0 | 646070 | 0.2349          | 0.7349    | 0.7255 | 0.7302 | 0.9462   |
| 0.2781        | 231.0 | 648879 | 0.2349          | 0.7349    | 0.7258 | 0.7303 | 0.9462   |
| 0.2633        | 232.0 | 651688 | 0.2348          | 0.7350    | 0.7251 | 0.7301 | 0.9461   |
| 0.2694        | 233.0 | 654497 | 0.2348          | 0.7349    | 0.7252 | 0.7300 | 0.9461   |
| 0.2595        | 234.0 | 657306 | 0.2347          | 0.7348    | 0.7251 | 0.7299 | 0.9461   |
| 0.2732        | 235.0 | 660115 | 0.2346          | 0.7347    | 0.7249 | 0.7297 | 0.9461   |
| 0.2728        | 236.0 | 662924 | 0.2346          | 0.7346    | 0.7248 | 0.7296 | 0.9461   |
| 0.2673        | 237.0 | 665733 | 0.2345          | 0.7346    | 0.7249 | 0.7297 | 0.9461   |
| 0.2694        | 238.0 | 668542 | 0.2345          | 0.7351    | 0.7251 | 0.7301 | 0.9461   |
| 0.2721        | 239.0 | 671351 | 0.2345          | 0.7353    | 0.7256 | 0.7304 | 0.9462   |
| 0.264         | 240.0 | 674160 | 0.2344          | 0.7351    | 0.7255 | 0.7303 | 0.9462   |
| 0.267         | 241.0 | 676969 | 0.2343          | 0.7352    | 0.7256 | 0.7304 | 0.9462   |
| 0.2728        | 242.0 | 679778 | 0.2343          | 0.7355    | 0.7256 | 0.7305 | 0.9462   |
| 0.2697        | 243.0 | 682587 | 0.2343          | 0.7354    | 0.7255 | 0.7304 | 0.9462   |
| 0.2688        | 244.0 | 685396 | 0.2342          | 0.7352    | 0.7253 | 0.7302 | 0.9462   |
| 0.2741        | 245.0 | 688205 | 0.2341          | 0.7354    | 0.7259 | 0.7306 | 0.9462   |
| 0.2834        | 246.0 | 691014 | 0.2341          | 0.7353    | 0.7255 | 0.7304 | 0.9462   |
| 0.2706        | 247.0 | 693823 | 0.2341          | 0.7357    | 0.7262 | 0.7309 | 0.9462   |
| 0.2686        | 248.0 | 696632 | 0.2340          | 0.7355    | 0.7258 | 0.7306 | 0.9462   |
| 0.2655        | 249.0 | 699441 | 0.2340          | 0.7350    | 0.7253 | 0.7301 | 0.9462   |
| 0.274         | 250.0 | 702250 | 0.2340          | 0.7350    | 0.7251 | 0.7300 | 0.9462   |
| 0.2728        | 251.0 | 705059 | 0.2339          | 0.7349    | 0.7252 | 0.7300 | 0.9462   |
| 0.2696        | 252.0 | 707868 | 0.2338          | 0.7354    | 0.7259 | 0.7306 | 0.9463   |
| 0.2678        | 253.0 | 710677 | 0.2338          | 0.7350    | 0.7254 | 0.7302 | 0.9462   |
| 0.279         | 254.0 | 713486 | 0.2337          | 0.7350    | 0.7256 | 0.7303 | 0.9462   |
| 0.2523        | 255.0 | 716295 | 0.2337          | 0.7350    | 0.7255 | 0.7302 | 0.9462   |
| 0.2722        | 256.0 | 719104 | 0.2336          | 0.7351    | 0.7257 | 0.7304 | 0.9462   |
| 0.2794        | 257.0 | 721913 | 0.2335          | 0.7348    | 0.7254 | 0.7301 | 0.9462   |
| 0.279         | 258.0 | 724722 | 0.2335          | 0.7345    | 0.7253 | 0.7299 | 0.9462   |
| 0.2676        | 259.0 | 727531 | 0.2335          | 0.7351    | 0.7256 | 0.7303 | 0.9463   |
| 0.261         | 260.0 | 730340 | 0.2335          | 0.7354    | 0.7259 | 0.7306 | 0.9463   |
| 0.2674        | 261.0 | 733149 | 0.2334          | 0.7350    | 0.7258 | 0.7304 | 0.9463   |
| 0.2742        | 262.0 | 735958 | 0.2334          | 0.7351    | 0.7258 | 0.7304 | 0.9463   |
| 0.2592        | 263.0 | 738767 | 0.2333          | 0.7351    | 0.7259 | 0.7305 | 0.9463   |
| 0.2729        | 264.0 | 741576 | 0.2333          | 0.7351    | 0.7259 | 0.7305 | 0.9463   |
| 0.2775        | 265.0 | 744385 | 0.2333          | 0.7355    | 0.7262 | 0.7308 | 0.9463   |
| 0.2695        | 266.0 | 747194 | 0.2333          | 0.7356    | 0.7263 | 0.7309 | 0.9463   |
| 0.2674        | 267.0 | 750003 | 0.2332          | 0.7356    | 0.7263 | 0.7310 | 0.9463   |
| 0.2522        | 268.0 | 752812 | 0.2332          | 0.7354    | 0.7260 | 0.7307 | 0.9463   |
| 0.2621        | 269.0 | 755621 | 0.2332          | 0.7361    | 0.7265 | 0.7313 | 0.9464   |
| 0.2813        | 270.0 | 758430 | 0.2331          | 0.7360    | 0.7265 | 0.7313 | 0.9464   |
| 0.2629        | 271.0 | 761239 | 0.2331          | 0.7360    | 0.7265 | 0.7313 | 0.9464   |
| 0.2762        | 272.0 | 764048 | 0.2331          | 0.7359    | 0.7263 | 0.7311 | 0.9463   |
| 0.2599        | 273.0 | 766857 | 0.2331          | 0.7362    | 0.7264 | 0.7313 | 0.9464   |
| 0.2795        | 274.0 | 769666 | 0.2331          | 0.7362    | 0.7264 | 0.7313 | 0.9464   |
| 0.2628        | 275.0 | 772475 | 0.2330          | 0.7360    | 0.7260 | 0.7309 | 0.9463   |
| 0.2762        | 276.0 | 775284 | 0.2330          | 0.7360    | 0.7261 | 0.7310 | 0.9463   |
| 0.2657        | 277.0 | 778093 | 0.2330          | 0.7361    | 0.7261 | 0.7310 | 0.9463   |
| 0.2673        | 278.0 | 780902 | 0.2330          | 0.7360    | 0.7259 | 0.7309 | 0.9463   |
| 0.2718        | 279.0 | 783711 | 0.2330          | 0.7361    | 0.7261 | 0.7311 | 0.9464   |
| 0.2631        | 280.0 | 786520 | 0.2329          | 0.7356    | 0.7257 | 0.7306 | 0.9463   |
| 0.2744        | 281.0 | 789329 | 0.2329          | 0.7359    | 0.7260 | 0.7309 | 0.9463   |
| 0.2848        | 282.0 | 792138 | 0.2329          | 0.7360    | 0.7261 | 0.7310 | 0.9464   |
| 0.271         | 283.0 | 794947 | 0.2329          | 0.7359    | 0.7262 | 0.7310 | 0.9464   |
| 0.262         | 284.0 | 797756 | 0.2328          | 0.7359    | 0.7262 | 0.7310 | 0.9464   |
| 0.2622        | 285.0 | 800565 | 0.2328          | 0.7359    | 0.7263 | 0.7310 | 0.9464   |
| 0.2679        | 286.0 | 803374 | 0.2328          | 0.7359    | 0.7263 | 0.7310 | 0.9464   |
| 0.2616        | 287.0 | 806183 | 0.2328          | 0.7359    | 0.7263 | 0.7311 | 0.9464   |
| 0.2721        | 288.0 | 808992 | 0.2328          | 0.7360    | 0.7264 | 0.7312 | 0.9464   |
| 0.2693        | 289.0 | 811801 | 0.2328          | 0.7361    | 0.7264 | 0.7312 | 0.9464   |
| 0.2645        | 290.0 | 814610 | 0.2328          | 0.7361    | 0.7264 | 0.7312 | 0.9464   |
| 0.2728        | 291.0 | 817419 | 0.2328          | 0.7361    | 0.7264 | 0.7312 | 0.9464   |
| 0.2637        | 292.0 | 820228 | 0.2327          | 0.7362    | 0.7264 | 0.7313 | 0.9464   |
| 0.2713        | 293.0 | 823037 | 0.2327          | 0.7363    | 0.7265 | 0.7314 | 0.9464   |
| 0.2623        | 294.0 | 825846 | 0.2327          | 0.7363    | 0.7265 | 0.7314 | 0.9464   |
| 0.2667        | 295.0 | 828655 | 0.2327          | 0.7363    | 0.7265 | 0.7314 | 0.9464   |
| 0.2679        | 296.0 | 831464 | 0.2327          | 0.7363    | 0.7265 | 0.7314 | 0.9464   |
| 0.2595        | 297.0 | 834273 | 0.2327          | 0.7363    | 0.7265 | 0.7314 | 0.9464   |
| 0.2609        | 298.0 | 837082 | 0.2327          | 0.7363    | 0.7265 | 0.7314 | 0.9464   |
| 0.2616        | 299.0 | 839891 | 0.2327          | 0.7363    | 0.7265 | 0.7314 | 0.9464   |
| 0.2713        | 300.0 | 842700 | 0.2327          | 0.7363    | 0.7265 | 0.7314 | 0.9464   |


### Framework versions


- Transformers 4.9.2
- Pytorch 1.9.0+cu102
- Datasets 1.11.0
- Tokenizers 0.10.3


model name: XLMRFineTuneonEnglishNERFrozenBase30epochs


model author: vitvit


task: token-classification


tags: pytorch, xlm-roberta, token-classification, dataset:conll2003, transformers, license:mit, generated_from_trainer, autonlp_compatible


tags:


- generated_from_trainer
datasets:
- wikiann
metrics:
- precision
- recall
- f1
- accuracy
model_index:
- name: xlm-roberta-base-finetuned-ner
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: wikiann
type: wikiann
args: he
metric:
name: Accuracy
type: accuracy
value: 0.9449884563330945


# xlm-roberta-base-finetuned-ner


This model was trained from scratch on the wikiann dataset.
It achieves the following results on the evaluation set:


- Loss: 0.5647
- Precision: 0.8684
- Recall: 0.8656
- F1: 0.8670
- Accuracy: 0.9450


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 3
- eval_batch_size: 3
- seed: 42
- distributed_type: multi-GPU
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 100


### Training results


| Training Loss | Epoch | Step   | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:------:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.3537        | 1.0   | 6667   | 0.3621          | 0.7951    | 0.8054 | 0.8002 | 0.9187   |
| 0.2468        | 2.0   | 13334  | 0.3024          | 0.8341    | 0.8451 | 0.8396 | 0.9359   |
| 0.1705        | 3.0   | 20001  | 0.3255          | 0.8401    | 0.8328 | 0.8364 | 0.9365   |
| 0.1388        | 4.0   | 26668  | 0.3530          | 0.8438    | 0.8527 | 0.8482 | 0.9389   |
| 0.0979        | 5.0   | 33335  | 0.3980          | 0.8445    | 0.8542 | 0.8494 | 0.9390   |
| 0.0946        | 6.0   | 40002  | 0.3863          | 0.8500    | 0.8622 | 0.8560 | 0.9426   |
| 0.0908        | 7.0   | 46669  | 0.3991          | 0.8519    | 0.8633 | 0.8576 | 0.9420   |
| 0.0712        | 8.0   | 53336  | 0.4065          | 0.8617    | 0.8551 | 0.8584 | 0.9424   |
| 0.0568        | 9.0   | 60003  | 0.4348          | 0.8441    | 0.8663 | 0.8551 | 0.9413   |
| 0.0448        | 10.0  | 66670  | 0.4661          | 0.8429    | 0.8603 | 0.8515 | 0.9404   |
| 0.0687        | 11.0  | 73337  | 0.4482          | 0.8561    | 0.8621 | 0.8591 | 0.9431   |
| 0.0552        | 12.0  | 80004  | 0.4527          | 0.8499    | 0.8619 | 0.8558 | 0.9405   |
| 0.059         | 13.0  | 86671  | 0.4688          | 0.8564    | 0.8592 | 0.8578 | 0.9428   |
| 0.0362        | 14.0  | 93338  | 0.4593          | 0.8705    | 0.8615 | 0.8660 | 0.9451   |
| 0.0407        | 15.0  | 100005 | 0.4661          | 0.8647    | 0.8674 | 0.8660 | 0.9449   |
| 0.0278        | 16.0  | 106672 | 0.4794          | 0.8670    | 0.8707 | 0.8688 | 0.9457   |
| 0.0425        | 17.0  | 113339 | 0.5056          | 0.8548    | 0.8698 | 0.8622 | 0.9440   |
| 0.0251        | 18.0  | 120006 | 0.4630          | 0.8658    | 0.8603 | 0.8630 | 0.9442   |
| 0.0207        | 19.0  | 126673 | 0.5077          | 0.8515    | 0.8574 | 0.8544 | 0.9420   |
| 0.0245        | 20.0  | 133340 | 0.5130          | 0.8630    | 0.8646 | 0.8638 | 0.9437   |
| 0.051         | 21.0  | 140007 | 0.5233          | 0.8578    | 0.8644 | 0.8611 | 0.9423   |
| 0.0381        | 22.0  | 146674 | 0.5269          | 0.8688    | 0.8635 | 0.8661 | 0.9433   |
| 0.0144        | 23.0  | 153341 | 0.5137          | 0.8572    | 0.8668 | 0.8620 | 0.9443   |
| 0.0237        | 24.0  | 160008 | 0.5121          | 0.8741    | 0.8552 | 0.8645 | 0.9443   |
| 0.0175        | 25.0  | 166675 | 0.5019          | 0.8665    | 0.8725 | 0.8695 | 0.9467   |
| 0.0268        | 26.0  | 173342 | 0.5247          | 0.8597    | 0.8696 | 0.8646 | 0.9433   |
| 0.0128        | 27.0  | 180009 | 0.5075          | 0.8696    | 0.8704 | 0.8700 | 0.9461   |
| 0.0299        | 28.0  | 186676 | 0.5066          | 0.8647    | 0.8636 | 0.8641 | 0.9444   |
| 0.018         | 29.0  | 193343 | 0.5421          | 0.8677    | 0.8609 | 0.8643 | 0.9432   |
| 0.0264        | 30.0  | 200010 | 0.5023          | 0.8479    | 0.8731 | 0.8603 | 0.9424   |
| 0.0169        | 31.0  | 206677 | 0.5215          | 0.8672    | 0.8653 | 0.8662 | 0.9435   |
| 0.0185        | 32.0  | 213344 | 0.5184          | 0.8698    | 0.8630 | 0.8664 | 0.9457   |
| 0.0159        | 33.0  | 220011 | 0.4930          | 0.8653    | 0.8662 | 0.8657 | 0.9448   |
| 0.026         | 34.0  | 226678 | 0.4976          | 0.8579    | 0.8794 | 0.8685 | 0.9456   |
| 0.016         | 35.0  | 233345 | 0.5671          | 0.8517    | 0.8689 | 0.8602 | 0.9421   |
| 0.0186        | 36.0  | 240012 | 0.4881          | 0.8706    | 0.8752 | 0.8729 | 0.9467   |
| 0.0253        | 37.0  | 246679 | 0.5351          | 0.8621    | 0.8725 | 0.8673 | 0.9447   |
| 0.0086        | 38.0  | 253346 | 0.5759          | 0.8742    | 0.8612 | 0.8677 | 0.9440   |
| 0.0157        | 39.0  | 260013 | 0.5362          | 0.8549    | 0.8696 | 0.8622 | 0.9436   |
| 0.0107        | 40.0  | 266680 | 0.5734          | 0.8730    | 0.8582 | 0.8655 | 0.9438   |
| 0.0139        | 41.0  | 273347 | 0.4995          | 0.8622    | 0.8729 | 0.8675 | 0.9457   |
| 0.0141        | 42.0  | 280014 | 0.5567          | 0.8651    | 0.8671 | 0.8661 | 0.9448   |
| 0.0146        | 43.0  | 286681 | 0.5124          | 0.8673    | 0.8691 | 0.8682 | 0.9460   |
| 0.0125        | 44.0  | 293348 | 0.5511          | 0.8568    | 0.8758 | 0.8662 | 0.9440   |
| 0.0153        | 45.0  | 300015 | 0.5385          | 0.8597    | 0.8720 | 0.8658 | 0.9445   |
| 0.017         | 46.0  | 306682 | 0.5302          | 0.8633    | 0.8714 | 0.8673 | 0.9448   |
| 0.0121        | 47.0  | 313349 | 0.5302          | 0.8604    | 0.8666 | 0.8635 | 0.9441   |
| 0.0136        | 48.0  | 320016 | 0.5639          | 0.8481    | 0.8677 | 0.8578 | 0.9404   |
| 0.0107        | 49.0  | 326683 | 0.5403          | 0.8731    | 0.8648 | 0.8689 | 0.9457   |
| 0.0083        | 50.0  | 333350 | 0.5615          | 0.8770    | 0.8581 | 0.8675 | 0.9431   |
| 0.0121        | 51.0  | 340017 | 0.5489          | 0.8512    | 0.8730 | 0.8620 | 0.9439   |
| 0.0079        | 52.0  | 346684 | 0.5328          | 0.8599    | 0.8736 | 0.8667 | 0.9458   |
| 0.0139        | 53.0  | 353351 | 0.5572          | 0.8665    | 0.8631 | 0.8648 | 0.9441   |
| 0.0138        | 54.0  | 360018 | 0.5128          | 0.8662    | 0.8740 | 0.8701 | 0.9468   |
| 0.014         | 55.0  | 366685 | 0.5603          | 0.8798    | 0.8662 | 0.8730 | 0.9460   |
| 0.0319        | 56.0  | 373352 | 0.5508          | 0.8631    | 0.8688 | 0.8659 | 0.9427   |
| 0.0152        | 57.0  | 380019 | 0.5716          | 0.8596    | 0.8644 | 0.8620 | 0.9429   |
| 0.0249        | 58.0  | 386686 | 0.5692          | 0.8595    | 0.8749 | 0.8671 | 0.9453   |
| 0.0161        | 59.0  | 393353 | 0.5483          | 0.8665    | 0.8715 | 0.8690 | 0.9463   |
| 0.0157        | 60.0  | 400020 | 0.5588          | 0.8603    | 0.8800 | 0.8701 | 0.9463   |
| 0.0247        | 61.0  | 406687 | 0.5265          | 0.8510    | 0.8662 | 0.8585 | 0.9417   |
| 0.0069        | 62.0  | 413354 | 0.5578          | 0.8681    | 0.8679 | 0.8680 | 0.9459   |
| 0.0254        | 63.0  | 420021 | 0.5756          | 0.8620    | 0.8646 | 0.8633 | 0.9435   |
| 0.0182        | 64.0  | 426688 | 0.5323          | 0.8651    | 0.8762 | 0.8707 | 0.9458   |
| 0.0237        | 65.0  | 433355 | 0.5342          | 0.8592    | 0.8724 | 0.8657 | 0.9443   |
| 0.0234        | 66.0  | 440022 | 0.5458          | 0.8653    | 0.8679 | 0.8666 | 0.9437   |
| 0.0159        | 67.0  | 446689 | 0.5166          | 0.8781    | 0.8624 | 0.8702 | 0.9448   |
| 0.0204        | 68.0  | 453356 | 0.5499          | 0.8658    | 0.8723 | 0.8690 | 0.9452   |
| 0.0117        | 69.0  | 460023 | 0.5573          | 0.8572    | 0.8714 | 0.8642 | 0.9432   |
| 0.0062        | 70.0  | 466690 | 0.5887          | 0.8592    | 0.8675 | 0.8633 | 0.9422   |
| 0.0123        | 71.0  | 473357 | 0.5138          | 0.8600    | 0.8699 | 0.8649 | 0.9448   |
| 0.0079        | 72.0  | 480024 | 0.5548          | 0.8610    | 0.8724 | 0.8666 | 0.9447   |
| 0.0061        | 73.0  | 486691 | 0.5872          | 0.8476    | 0.8675 | 0.8574 | 0.9415   |
| 0.0129        | 74.0  | 493358 | 0.5520          | 0.8727    | 0.8595 | 0.8661 | 0.9449   |
| 0.0159        | 75.0  | 500025 | 0.5427          | 0.8611    | 0.8674 | 0.8642 | 0.9435   |
| 0.0258        | 76.0  | 506692 | 0.5402          | 0.8672    | 0.8702 | 0.8687 | 0.9448   |
| 0.0151        | 77.0  | 513359 | 0.5589          | 0.8681    | 0.8704 | 0.8693 | 0.9457   |
| 0.0075        | 78.0  | 520026 | 0.5754          | 0.8613    | 0.8682 | 0.8647 | 0.9438   |
| 0.0076        | 79.0  | 526693 | 0.5709          | 0.8608    | 0.8646 | 0.8627 | 0.9445   |
| 0.0196        | 80.0  | 533360 | 0.5252          | 0.8714    | 0.8706 | 0.8710 | 0.9461   |
| 0.0123        | 81.0  | 540027 | 0.5857          | 0.8637    | 0.8631 | 0.8634 | 0.9437   |
| 0.0205        | 82.0  | 546694 | 0.5805          | 0.8642    | 0.8655 | 0.8648 | 0.9431   |
| 0.0065        | 83.0  | 553361 | 0.5815          | 0.8619    | 0.8626 | 0.8622 | 0.9431   |
| 0.0128        | 84.0  | 560028 | 0.6305          | 0.8498    | 0.8646 | 0.8571 | 0.9402   |
| 0.0118        | 85.0  | 566695 | 0.5620          | 0.8648    | 0.8682 | 0.8665 | 0.9445   |
| 0.0173        | 86.0  | 573362 | 0.5714          | 0.8655    | 0.8657 | 0.8656 | 0.9442   |
| 0.0107        | 87.0  | 580029 | 0.5845          | 0.8603    | 0.8649 | 0.8626 | 0.9418   |
| 0.0218        | 88.0  | 586696 | 0.5259          | 0.8708    | 0.8697 | 0.8703 | 0.9449   |
| 0.0039        | 89.0  | 593363 | 0.5809          | 0.8800    | 0.8648 | 0.8723 | 0.9465   |
| 0.0076        | 90.0  | 600030 | 0.5852          | 0.8744    | 0.8615 | 0.8679 | 0.9443   |
| 0.008         | 91.0  | 606697 | 0.5540          | 0.8689    | 0.8683 | 0.8686 | 0.9454   |
| 0.0114        | 92.0  | 613364 | 0.5836          | 0.8578    | 0.8639 | 0.8609 | 0.9422   |
| 0.0245        | 93.0  | 620031 | 0.5808          | 0.8735    | 0.8672 | 0.8703 | 0.9450   |
| 0.0142        | 94.0  | 626698 | 0.5846          | 0.8630    | 0.8692 | 0.8661 | 0.9429   |
| 0.0013        | 95.0  | 633365 | 0.5495          | 0.8656    | 0.8605 | 0.8630 | 0.9432   |
| 0.0093        | 96.0  | 640032 | 0.6049          | 0.8660    | 0.8656 | 0.8658 | 0.9436   |
| 0.012         | 97.0  | 646699 | 0.5802          | 0.8633    | 0.8618 | 0.8626 | 0.9427   |
| 0.0042        | 98.0  | 653366 | 0.5851          | 0.8571    | 0.8658 | 0.8615 | 0.9422   |
| 0.0143        | 99.0  | 660033 | 0.5619          | 0.8671    | 0.8626 | 0.8649 | 0.9437   |
| 0.0173        | 100.0 | 666700 | 0.5647          | 0.8684    | 0.8656 | 0.8670 | 0.9450   |


### Framework versions


- Transformers 4.9.2
- Pytorch 1.9.0+cu111
- Datasets 1.11.0
- Tokenizers 0.10.3


model name: xlm-roberta-base-finetuned-heb_HebrewSentiment


model author: vitvit


task: token-classification


tags: pytorch, xlm-roberta, token-classification, dataset:wikiann, transformers, generated_from_trainer, autonlp_compatible


license: mit
tags:


- generated_from_trainer
datasets:
- conll2003
metrics:
- precision
- recall
- f1
- accuracy
model_index:
- name: xlm-roberta-base-finetuned-ner
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: conll2003
type: conll2003
args: conll2003
metric:
name: Accuracy
type: accuracy
value: 0.9882987313361343


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# xlm-roberta-base-finetuned-ner


This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the conll2003 dataset.
It achieves the following results on the evaluation set:


- Loss: 0.1202
- Precision: 0.9447
- Recall: 0.9536
- F1: 0.9492
- Accuracy: 0.9883


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 5
- eval_batch_size: 5
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 30


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:-----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.1023        | 1.0   | 2809  | 0.0724          | 0.9338    | 0.9363 | 0.9351 | 0.9850   |
| 0.0596        | 2.0   | 5618  | 0.0760          | 0.9295    | 0.9359 | 0.9327 | 0.9848   |
| 0.0406        | 3.0   | 8427  | 0.0740          | 0.9346    | 0.9410 | 0.9378 | 0.9863   |
| 0.0365        | 4.0   | 11236 | 0.0676          | 0.9368    | 0.9490 | 0.9428 | 0.9870   |
| 0.0279        | 5.0   | 14045 | 0.0737          | 0.9453    | 0.9476 | 0.9464 | 0.9877   |
| 0.0147        | 6.0   | 16854 | 0.0812          | 0.9413    | 0.9515 | 0.9464 | 0.9878   |
| 0.0138        | 7.0   | 19663 | 0.0893          | 0.9425    | 0.9525 | 0.9475 | 0.9876   |
| 0.0158        | 8.0   | 22472 | 0.1066          | 0.9362    | 0.9464 | 0.9412 | 0.9862   |
| 0.0092        | 9.0   | 25281 | 0.1026          | 0.9391    | 0.9511 | 0.9451 | 0.9869   |
| 0.0073        | 10.0  | 28090 | 0.1001          | 0.9442    | 0.9503 | 0.9472 | 0.9879   |
| 0.0069        | 11.0  | 30899 | 0.1103          | 0.9399    | 0.9511 | 0.9455 | 0.9871   |
| 0.0073        | 12.0  | 33708 | 0.1170          | 0.9383    | 0.9481 | 0.9432 | 0.9876   |
| 0.0054        | 13.0  | 36517 | 0.1068          | 0.9407    | 0.9491 | 0.9448 | 0.9875   |
| 0.0048        | 14.0  | 39326 | 0.1096          | 0.9438    | 0.9518 | 0.9477 | 0.9879   |
| 0.0042        | 15.0  | 42135 | 0.1187          | 0.9442    | 0.9523 | 0.9483 | 0.9884   |
| 0.0037        | 16.0  | 44944 | 0.1162          | 0.9384    | 0.9521 | 0.9452 | 0.9875   |
| 0.0039        | 17.0  | 47753 | 0.1046          | 0.9435    | 0.9477 | 0.9456 | 0.9878   |
| 0.0025        | 18.0  | 50562 | 0.1063          | 0.9501    | 0.9549 | 0.9525 | 0.9889   |
| 0.0021        | 19.0  | 53371 | 0.0992          | 0.9533    | 0.9572 | 0.9553 | 0.9895   |
| 0.0019        | 20.0  | 56180 | 0.1216          | 0.9404    | 0.9524 | 0.9464 | 0.9876   |
| 0.0021        | 21.0  | 58989 | 0.1080          | 0.9430    | 0.9478 | 0.9454 | 0.9880   |
| 0.0032        | 22.0  | 61798 | 0.1109          | 0.9436    | 0.9512 | 0.9474 | 0.9881   |
| 0.0115        | 23.0  | 64607 | 0.1161          | 0.9412    | 0.9475 | 0.9443 | 0.9874   |
| 0.001         | 24.0  | 67416 | 0.1216          | 0.9446    | 0.9518 | 0.9481 | 0.9882   |
| 0.0004        | 25.0  | 70225 | 0.1145          | 0.9478    | 0.9527 | 0.9503 | 0.9888   |
| 0.0005        | 26.0  | 73034 | 0.1217          | 0.9479    | 0.9531 | 0.9505 | 0.9887   |
| 0.0007        | 27.0  | 75843 | 0.1199          | 0.9452    | 0.9561 | 0.9506 | 0.9887   |
| 0.0053        | 28.0  | 78652 | 0.1187          | 0.9440    | 0.9510 | 0.9475 | 0.9881   |
| 0.0014        | 29.0  | 81461 | 0.1207          | 0.9461    | 0.9540 | 0.9500 | 0.9884   |
| 0.0023        | 30.0  | 84270 | 0.1202          | 0.9447    | 0.9536 | 0.9492 | 0.9883   |


### Framework versions


- Transformers 4.9.2
- Pytorch 1.9.0+cu102
- Datasets 1.11.0
- Tokenizers 0.10.3


model name: xlm-roberta-base-finetuned-ner


model author: vitvit


task: token-classification


tags: pytorch, xlm-roberta, token-classification, dataset:conll2003, transformers, license:mit, generated_from_trainer, autonlp_compatible


license: mit
tags:


- generated_from_trainer
datasets:
- conll2003
metrics:
- precision
- recall
- f1
- accuracy
model_index:
- name: xlm-roberta-base-finetuned-ner
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: conll2003
type: conll2003
args: conll2003
metric:
name: Accuracy
type: accuracy
value: 0.90090188725725


# xlm-roberta-base-finetuned-ner


- Loss: 0.4181
- Precision: 0.6464
- Recall: 0.4904
- F1: 0.5577
- Accuracy: 0.9009
- It just needs more training time


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:-----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.9474        | 1.0   | 2809  | 0.9105          | 0.0       | 0.0    | 0.0    | 0.7879   |
| 0.7728        | 2.0   | 5618  | 0.8002          | 0.0       | 0.0    | 0.0    | 0.7879   |
| 0.7209        | 3.0   | 8427  | 0.7329          | 0.1818    | 0.0002 | 0.0004 | 0.7881   |
| 0.6666        | 4.0   | 11236 | 0.6824          | 0.27      | 0.0050 | 0.0099 | 0.7903   |
| 0.6372        | 5.0   | 14045 | 0.6416          | 0.3302    | 0.0261 | 0.0484 | 0.7988   |
| 0.5982        | 6.0   | 16854 | 0.6084          | 0.4188    | 0.0686 | 0.1179 | 0.8128   |
| 0.5812        | 7.0   | 19663 | 0.5800          | 0.4799    | 0.1152 | 0.1858 | 0.8266   |
| 0.5684        | 8.0   | 22472 | 0.5569          | 0.5255    | 0.1647 | 0.2508 | 0.8380   |
| 0.5389        | 9.0   | 25281 | 0.5375          | 0.5564    | 0.2128 | 0.3078 | 0.8482   |
| 0.5307        | 10.0  | 28090 | 0.5205          | 0.5749    | 0.2550 | 0.3533 | 0.8567   |
| 0.5106        | 11.0  | 30899 | 0.5064          | 0.5916    | 0.2916 | 0.3906 | 0.8636   |
| 0.4921        | 12.0  | 33708 | 0.4938          | 0.6033    | 0.3236 | 0.4212 | 0.8698   |
| 0.4967        | 13.0  | 36517 | 0.4825          | 0.6106    | 0.3544 | 0.4485 | 0.8758   |
| 0.4707        | 14.0  | 39326 | 0.4733          | 0.6199    | 0.3753 | 0.4676 | 0.8798   |
| 0.4704        | 15.0  | 42135 | 0.4654          | 0.6246    | 0.3927 | 0.4823 | 0.8830   |
| 0.4654        | 16.0  | 44944 | 0.4574          | 0.6285    | 0.4159 | 0.5006 | 0.8871   |
| 0.4314        | 17.0  | 47753 | 0.4514          | 0.6321    | 0.4240 | 0.5075 | 0.8887   |
| 0.47          | 18.0  | 50562 | 0.4459          | 0.6358    | 0.4380 | 0.5187 | 0.8911   |
| 0.4486        | 19.0  | 53371 | 0.4410          | 0.6399    | 0.4480 | 0.5271 | 0.8929   |
| 0.4411        | 20.0  | 56180 | 0.4367          | 0.6413    | 0.4561 | 0.5331 | 0.8944   |
| 0.4333        | 21.0  | 58989 | 0.4328          | 0.6411    | 0.4644 | 0.5386 | 0.8959   |
| 0.4402        | 22.0  | 61798 | 0.4295          | 0.6425    | 0.4687 | 0.5420 | 0.8968   |
| 0.4287        | 23.0  | 64607 | 0.4268          | 0.6442    | 0.4735 | 0.5458 | 0.8978   |
| 0.4336        | 24.0  | 67416 | 0.4245          | 0.6441    | 0.4771 | 0.5482 | 0.8985   |
| 0.4243        | 25.0  | 70225 | 0.4224          | 0.6454    | 0.4817 | 0.5517 | 0.8993   |
| 0.4153        | 26.0  | 73034 | 0.4209          | 0.6469    | 0.4846 | 0.5541 | 0.8998   |
| 0.4286        | 27.0  | 75843 | 0.4197          | 0.6467    | 0.4865 | 0.5553 | 0.9002   |
| 0.436         | 28.0  | 78652 | 0.4188          | 0.6466    | 0.4887 | 0.5566 | 0.9006   |
| 0.427         | 29.0  | 81461 | 0.4183          | 0.6465    | 0.4900 | 0.5575 | 0.9008   |
| 0.4317        | 30.0  | 84270 | 0.4181          | 0.6464    | 0.4904 | 0.5577 | 0.9009   |


### Framework versions


model name: XLMRFineTuneonEnglishNERFrozenBase


model author: vitvit


task: token-classification


This model is a T5-large reranker fine-tuned on the MS MARCO passage dataset for 100k steps (or 10 epochs).


For more details on how to use it, check the following links:


- [A simple reranking example](https://github.com/castorini/pygaggle#a-simple-reranking-example)
- [Rerank MS MARCO passages](https://github.com/castorini/pygaggle/blob/master/docs/experiments-msmarco-passage-subset.md)
- [Rerank Robust04 documents](https://github.com/castorini/pygaggle/blob/master/docs/experiments-robust04-monot5-gpu.md)


Paper describing the model: [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://www.aclweb.org/anthology/2020.findings-emnlp.63/)


model name: monot5-large-msmarco


model author: castorini


task: feature-extraction


tags: pytorch, jax, t5, feature-extraction, transformers


This model is converted from the original ANCE [repo](https://github.com/microsoft/ANCE) and fitted into Pyserini:


> Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, Arnold Overwijk. [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://arxiv.org/pdf/2007.00808.pdf)


For more details on how to use it, check our experiments in [Pyserini](https://github.com/castorini/pyserini/blob/master/docs/experiments-ance.md)


model name: ance-dpr-context-multi


tags: pytorch, dpr, arxiv:2007.00808, transformers


This model is a T5-base reranker fine-tuned on the MS MARCO passage dataset for 100k steps (or 10 epochs).


For better zero-shot performance (i.e., inference on other datasets), we recommend using `castorini/monot5-base-msmarco-10k`.


model name: monot5-base-msmarco


task: text2text-generation


tags: pytorch, jax, t5, text2text-generation, transformers, autonlp_compatible


## Hugging Face's logo


language:


- om
- am
- rw
- rn
- ha
- ig
- pcm
- so
- sw
- ti
- yo
- multilingual


# afriberta_large


## Model description


AfriBERTa large is a pretrained multilingual language model with around 126 million parameters.
The model has 10 layers, 6 attention heads, 768 hidden units and 3072 feed forward size.
The model was pretrained on 11 African languages namely - Afaan Oromoo (also called Oromo), Amharic, Gahuza (a mixed language containing Kinyarwanda and Kirundi), Hausa, Igbo, Nigerian Pidgin, Somali, Swahili, Tigrinya and Yorùbá.
The model has been shown to obtain competitive downstream performances on text classification and Named Entity Recognition on several African languages, including those it was not pretrained on.


## Intended uses & limitations


#### How to use


You can use this model with Transformers for any downstream task.
For example, assuming we want to finetune this model on a token classification task, we do the following:


```python
>>> from transformers import AutoTokenizer, AutoModelForTokenClassification
>>> model = AutoModelForTokenClassification.from_pretrained("castorini/afriberta_large")
>>> tokenizer = AutoTokenizer.from_pretrained("castorini/afriberta_large")
# we have to manually set the model max length because it is an imported sentencepiece model, which huggingface does not properly support right now
>>> tokenizer.model_max_length = 512
```


#### Limitations and bias


- This model is possibly limited by its training dataset which are majorly obtained from news articles from a specific span of time. Thus, it may not generalize well.
- This model is trained on very little data (less than 1 GB), hence it may not have seen enough data to learn very complex linguistic relations.


## Training data


The model was trained on an aggregation of datasets from the BBC news website and Common Crawl.


## Training procedure


For information on training procedures, please refer to the AfriBERTa [paper]() or [repository](https://github.com/keleog/afriberta)


### BibTeX entry and citation info


```
@inproceedings{ogueji-etal-2021-small,
    title = "Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages",
    author = "Ogueji, Kelechi  and
      Zhu, Yuxin  and
      Lin, Jimmy",
    booktitle = "Proceedings of the 1st Workshop on Multilingual Representation Learning",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.mrl-1.11",
    pages = "116--126",
}
```


model name: afriberta_large


model author: castorini


task: fill-mask


tags: pytorch, xlm-roberta, fill-mask, om, am, rw, rn, ha, ig, pcm, so, sw, ti, yo, multilingual, transformers, autonlp_compatible


This model is to reproduce a variant of TCT-ColBERT-V2 dense retrieval models described in the following paper:


> Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. [In-Batch Negatives for Knowledge Distillation with Tightly-CoupledTeachers for Dense Retrieval.](https://cs.uwaterloo.ca/~jimmylin/publications/Lin_etal_2021_RepL4NLP.pdf) *RepL4NLP 2021*.


You can find our reproduction report in Pyserini [here](https://github.com/castorini/pyserini/blob/master/docs/experiments-tct_colbert-v2.md).


model name: tct_colbert-v2-hn-msmarco


task: feature-extraction


tags: pytorch, bert, feature-extraction, transformers, infinity_compatible


For more information, check [doc2query.ai](http://doc2query.ai)


model name: doc2query-t5-base-msmarco


task: text2text-generation


This model is converted from the original DKRR [repo](https://github.com/facebookresearch/FiD) and ported into Pyserini:


> @misc{izacard2020distilling,
title={Distilling Knowledge from Reader to Retriever for Question Answering},
author={Gautier Izacard and Edouard Grave},
year={2020},
eprint={2012.04584},
archivePrefix={arXiv},
primaryClass={cs.CL}
}


model name: dkrr-dpr-nq-retriever


tags: pytorch, bert, feature-extraction, arxiv:2012.04584, transformers, infinity_compatible


language: en
tags:


- azbert
- pretraining
- fill-mask
widget:
- text: "$f$ $($ $x$ [MASK] $y$ $)$"
example_title: "mathy"
- text: "$x$ [MASK] $x$ $equal$ $2$ $x$"
example_title: "mathy"
- text: "Proof by [MASK] that $n$ $fact$ $gt$ $3$ $n$ for $n$ $gt$ $6$"
example_title: "mathy"
- text: "Proof by induction that $n$ [MASK] $gt$ $3$ $n$ for $n$ $gt$ $6$"
example_title: "mathy"
- text: "The goal of life is [MASK]."
example_title: "philosophical"
license: mit


## About


Here we share a pretrained BERT model that is aware of math tokens. The math tokens are treated specially and tokenized using [pya0](https://github.com/approach0/pya0), which adds very limited new tokens for latex markup (total vocabulary is just 31,061).


This model is trained on 4 x 2 Tesla V100 with a total batch size of 64, using Math StackExchange data with 2.7 million sentence pairs trained for 7 epochs.


### Usage


Download and try it out


```sh
pip install pya0==0.3.2
wget https://vault.cs.uwaterloo.ca/s/gqstFZmWHCLGXe3/download -O ckpt.tar.gz
mkdir -p ckpt
tar xzf ckpt.tar.gz -C ckpt --strip-components=1
python test.py --test_file test.txt
```


### Test file format


Modify the test examples in `test.txt` to play with it.


The test file is tab-separated, the first column is additional positions you want to mask for the right-side sentence (useful for masking tokens in math markups). A zero means no additional mask positions.


### Example output


### Upload to huggingface


This repo is hosted on [Github](https://github.com/approach0/azbert), and only mirrored at [huggingface](https://huggingface.co/castorini/azbert-base).


To upload to huggingface, use the `upload2hgf.sh` script.
Before runnig this script, be sure to check:


* check points for model and tokenizer are created under `./ckpt` folder
* model contains all the files needed: `config.json` and `pytorch_model.bin`
* tokenizer contains all the files needed: `added_tokens.json`, `special_tokens_map.json`, `tokenizer_config.json`, `vocab.txt` and `tokenizer.json`
* no `tokenizer_file` field in `tokenizer_config.json` (sometimes it is located locally at `~/.cache`)
* `git-lfs` is installed
* having git-remote named `hgf` reference to `https://huggingface.co/castorini/azbert-base`


model name: azbert-base


model author: castorini


task: fill-mask


tags: pytorch, tensorboard, bert, pretraining, en, transformers, azbert, fill-mask, license:mit, autonlp_compatible, infinity_compatible


This model is converted from the original ANCE [repo](https://github.com/microsoft/ANCE) and fitted into Pyserini:


> Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, Arnold Overwijk. [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://arxiv.org/pdf/2007.00808.pdf)


For more details on how to use it, check our experiments in [Pyserini](https://github.com/castorini/pyserini/blob/master/docs/experiments-ance.md)


model name: ance-dpr-question-multi


task: feature-extraction


tags: pytorch, dpr, feature-extraction, arxiv:2007.00808, transformers


## Hugging Face's logo


language:


- om
- am
- rw
- rn
- ha
- ig
- pcm
- so
- sw
- ti
- yo
- multilingual


# afriberta_base


## Model description


AfriBERTa base is a pretrained multilingual language model with around 111 million parameters.
The model has 8 layers, 6 attention heads, 768 hidden units and 3072 feed forward size.
The model was pretrained on 11 African languages namely - Afaan Oromoo (also called Oromo), Amharic, Gahuza (a mixed language containing Kinyarwanda and Kirundi), Hausa, Igbo, Nigerian Pidgin, Somali, Swahili, Tigrinya and Yorùbá.
The model has been shown to obtain competitive downstream performances on text classification and Named Entity Recognition on several African languages, including those it was not pretrained on.


## Intended uses & limitations


#### How to use


You can use this model with Transformers for any downstream task.
For example, assuming we want to finetune this model on a token classification task, we do the following:


```python
>>> from transformers import AutoTokenizer, AutoModelForTokenClassification
>>> model = AutoModelForTokenClassification.from_pretrained("castorini/afriberta_base")
>>> tokenizer = AutoTokenizer.from_pretrained("castorini/afriberta_base")
# we have to manually set the model max length because it is an imported sentencepiece model, which huggingface does not properly support right now
>>> tokenizer.model_max_length = 512
```


#### Limitations and bias


- This model is possibly limited by its training dataset which are majorly obtained from news articles from a specific span of time. Thus, it may not generalize well.
- This model is trained on very little data (less than 1 GB), hence it may not have seen enough data to learn very complex linguistic relations.


## Training data


The model was trained on an aggregation of datasets from the BBC news website and Common Crawl.


## Training procedure


For information on training procedures, please refer to the AfriBERTa [paper]() or [repository](https://github.com/keleog/afriberta)


### BibTeX entry and citation info


```
@inproceedings{ogueji-etal-2021-small,
    title = "Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages",
    author = "Ogueji, Kelechi  and
      Zhu, Yuxin  and
      Lin, Jimmy",
    booktitle = "Proceedings of the 1st Workshop on Multilingual Representation Learning",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.mrl-1.11",
    pages = "116--126",
}
```


model name: afriberta_base


model author: castorini


task: fill-mask


tags: pytorch, xlm-roberta, fill-mask, om, am, rw, rn, ha, ig, pcm, so, sw, ti, yo, multilingual, transformers, autonlp_compatible


This model is a T5-3B reranker fine-tuned on the MS MARCO passage dataset for 100k steps (or 10 epochs).


For more details on how to use it, check [pygaggle.ai](pygaggle.ai)


Paper describing the model: [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://www.aclweb.org/anthology/2020.findings-emnlp.63/)


model name: monot5-3b-msmarco


task: feature-extraction


tags: pytorch, t5, feature-extraction, transformers


This model is converted from the original ANCE [repo](https://github.com/microsoft/ANCE) and fitted into Pyserini:


> Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, Arnold Overwijk. [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://arxiv.org/pdf/2007.00808.pdf)


For more details on how to use it, check our experiments in [Pyserini](https://github.com/castorini/pyserini/blob/master/docs/experiments-ance.md)


model name: ance-msmarco-doc-maxp


tags: pytorch, roberta, arxiv:2007.00808, transformers, infinity_compatible


This model is to reproduce the TCT-ColBERT dense retrieval described in the following paper:


> Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. [Distilling Dense Representations for Ranking using Tightly-Coupled Teachers.](https://arxiv.org/abs/2010.11386) arXiv:2010.11386, October 2020.


For more details on how to use it, check our experiments in [Pyserini](https://github.com/castorini/pyserini/blob/master/docs/experiments-tct_colbert.md)


model name: tct_colbert-msmarco


model author: castorini


tags: pytorch, arxiv:2010.11386, transformers


This model is a T5-base reranker fine-tuned on the MS MARCO passage dataset for 10k steps (or 1 epoch).


This model usually has a better zero-shot performance than `monot5-base-msmarco`, i.e., it performs better on datasets different from MS MARCO.


For more details on how to use it, check the following links:


- [A simple reranking example](https://github.com/castorini/pygaggle#a-simple-reranking-example)
- [Rerank MS MARCO passages](https://github.com/castorini/pygaggle/blob/master/docs/experiments-msmarco-passage-subset.md)
- [Rerank Robust04 documents](https://github.com/castorini/pygaggle/blob/master/docs/experiments-robust04-monot5-gpu.md)


model name: monot5-base-msmarco-10k


task: text2text-generation


tags: pytorch, jax, t5, text2text-generation, transformers, autonlp_compatible


This model is to reproduce a variant of TCT-ColBERT-V2 dense retrieval models described in the following paper:


> Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. [In-Batch Negatives for Knowledge Distillation with Tightly-CoupledTeachers for Dense Retrieval.](https://cs.uwaterloo.ca/~jimmylin/publications/Lin_etal_2021_RepL4NLP.pdf) *RepL4NLP 2021*.


You can find our reproduction report in Pyserini [here](https://github.com/castorini/pyserini/blob/master/docs/experiments-tct_colbert-v2.md).


model name: tct_colbert-v2-hnp-msmarco


task: feature-extraction


tags: pytorch, bert, feature-extraction, transformers, infinity_compatible


# Model Description


This checkpoint is a direct conversion of [BERT_Large_trained_on_MSMARCO.zip](https://drive.google.com/open?id=1crlASTMlsihALlkabAQP6JTYIZwC1Wm8) from the original [repo](https://github.com/nyu-dl/dl4marco-bert/).
The corresponding model class is BertForSequenceClassification, and its purpose is for MS MARCO passage ranking.
Please find the original repo for more detail of its training settings regarding hyperparameter/device/data.


model name: monobert-large-msmarco-finetune-only


model author: castorini


task: text-classification


tags: pytorch, jax, bert, text-classification, transformers, infinity_compatible


This model is to reproduce Contextualized Query Embeddings for Conversational Search described in the following paper:


> Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. [Contextualized Query Embeddings for Conversational Search.](https://cs.uwaterloo.ca/~jimmylin/publications/Lin_etal_EMNLP2021.pdf) EMNLP, Nov 2021.


This model is finetuend only on query ecoder with frezzed passage encoder. The starting point is the [tct_colbert-msmarco](https://huggingface.co/castorini/tct_colbert-msmarco/tree/main). The detailed usage of the model will be out soon on [Chatty Goose](https://github.com/castorini/chatty-goose). You can also check the fine-tuning and inference using tensorflow in our [CQE repo](https://github.com/castorini/CQE)


model name: tct_colbert-v2-msmarco-cqe


model name: tct_colbert-v2-msmarco


This model is a T5-3B reranker pre-finetuned on the MS MARCO passage dataset for 10K steps (or 1 epoch) on the pairwise task and then finetuned on MedMARCO (from [Sledge-Z paper](https://www.aclweb.org/anthology/2020.emnlp-main.341.pdf)) for 1K steps on the pairwise task.


For more details on how to use it, check [pygaggle.ai](pygaggle.ai)!


Paper describing the model: [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://arxiv.org/abs/2101.05667)


model name: duot5-3b-med-msmarco


task: feature-extraction


tags: pytorch, t5, feature-extraction, arxiv:2101.05667, transformers


This model is converted from the original BPR [repo](https://github.com/studio-ousia/bpr) and fitted into Pyserini:


> Ikuya Yamada, Akari Asai, and Hannaneh Hajishirzi. 2021. Efficient passage retrieval with hashing for open-domain question answering. arXiv:2106.00882.


model name: bpr-nq-ctx-encoder


model author: castorini


tags: pytorch, dpr, transformers


This model is a T5-large reranker fine-tuned on the MS MARCO passage dataset for 10k steps (or 1 epoch).


This model usually has a better zero-shot performance than `monot5-large-msmarco`, i.e., it performs better on datasets different from MS MARCO.


For more details on how to use it, check the following links:


- [A simple reranking example](https://github.com/castorini/pygaggle#a-simple-reranking-example)
- [Rerank MS MARCO passages](https://github.com/castorini/pygaggle/blob/master/docs/experiments-msmarco-passage-subset.md)
- [Rerank Robust04 documents](https://github.com/castorini/pygaggle/blob/master/docs/experiments-robust04-monot5-gpu.md)


Paper describing the model: [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://www.aclweb.org/anthology/2020.findings-emnlp.63/)


model name: monot5-large-msmarco-10k


task: text2text-generation


tags: pytorch, jax, t5, text2text-generation, transformers, autonlp_compatible


This model is a T5-base reranker fine-tuned on the MS MARCO passage dataset for 10k steps (or 1 epoch) and then fine-tuned again on MedMARCO (from [Sledge-Z paper](https://www.aclweb.org/anthology/2020.emnlp-main.341.pdf) for 1k steps.


For more details on how to use it, check [pygaggle.ai](pygaggle.ai)


model name: monot5-base-med-msmarco


task: feature-extraction


tags: pytorch, jax, t5, feature-extraction, transformers


This model is converted from the original ANCE [repo](https://github.com/microsoft/ANCE) and fitted into Pyserini:


> Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, Arnold Overwijk. [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://arxiv.org/pdf/2007.00808.pdf)


For more details on how to use it, check our experiments in [Pyserini](https://github.com/castorini/pyserini/blob/master/docs/experiments-ance.md)


model name: ance-msmarco-doc-firstp


model author: castorini


tags: pytorch, roberta, arxiv:2007.00808, transformers, infinity_compatible


This model is a T5-base pairwise reranker fine-tuned on MS MARCO passage dataset for 50k steps (or 5 epochs).


Paper describing the model: [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://arxiv.org/pdf/2101.05667.pdf)


model name: duot5-base-msmarco


tags: pytorch, jax, t5, text2text-generation, arxiv:2101.05667, transformers, autonlp_compatible


This model is to reproduce a variant of TCT-ColBERT-V2 dense retrieval models described in the following paper:


> Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. [In-Batch Negatives for Knowledge Distillation with Tightly-CoupledTeachers for Dense Retrieval.](https://cs.uwaterloo.ca/~jimmylin/publications/Lin_etal_2021_RepL4NLP.pdf) *RepL4NLP 2021*.


Specifically, this checkpoint is finetuned for MS MARCO-V2 passage ranking, and we use this checkpoint as our ``trained'' model for TREC DL 2021 submissions.
The initial checkpoint is from a previous one [tct_colbert-v2-hnp-msmarco](https://huggingface.co/castorini/tct_colbert-v2-hnp-msmarco) trained on [MS MARCO](https://github.com/microsoft/MSMARCO-Passage-Ranking).
For fine-tuning, we construct our training data for MS MARCO-V2 passage ranking using this [script](https://github.com/castorini/pyserini/blob/master/scripts/msmarco_v2/generate_train_triplet.py).


model name: tct_colbert-v2-hnp-msmarco-r2


tags: pytorch, bert, feature-extraction, transformers, infinity_compatible


This model is trained for conversational question rewriting.


Usage:


Source text format: ${HISTORY} ||| ${CURRENT_QUESTION}


example from [CANARD](https://sites.google.com/view/qanta/projects/canard):
Frank Zappa ||| Disbandment ||| What group disbanded ||| Zappa and the Mothers of Invention ||| When did they disband?


Target text:
When did Zappa and the Mothers of Invention disband?


You can find our guide to reproduce the training in this [repo](https://github.com/castorini/chatty-goose/blob/c7d0cd8c45354b09b5fb930ab0b5af8be2e5772b/docs/t5_finetuning.md).


model name: t5-base-canard


model author: castorini


task: text2text-generation


tags: pytorch, jax, t5, text2text-generation, transformers, autonlp_compatible


This model is a T5-3B reranker, initialized with our pointwise ranker, [castorini/monot5-3b-msmarco](https://huggingface.co/castorini/monot5-3b-msmarco), and finetuned on the MS MARCO passage dataset for 50K steps (or 5 epochs) on the pairwise reranking task.


For more details on how to use it, check [pygaggle.ai](pygaggle.ai)!


Paper describing the model: [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://arxiv.org/abs/2101.05667)


model name: duot5-3b-msmarco


task: feature-extraction


tags: pytorch, t5, feature-extraction, arxiv:2101.05667, transformers


This model is converted from the original BPR [repo](https://github.com/studio-ousia/bpr) and fitted into Pyserini:


> Ikuya Yamada, Akari Asai, and Hannaneh Hajishirzi. 2021. Efficient passage retrieval with hashing for open-domain question answering. arXiv:2106.00882.


model name: bpr-nq-question-encoder


tags: pytorch, dpr, feature-extraction, transformers


## Hugging Face's logo


language:


- om
- am
- rw
- rn
- ha
- ig
- pcm
- so
- sw
- ti
- yo
- multilingual


# afriberta_small


## Model description


AfriBERTa small is a pretrained multilingual language model with around 97 million parameters.
The model has 4 layers, 6 attention heads, 768 hidden units and 3072 feed forward size.
The model was pretrained on 11 African languages namely - Afaan Oromoo (also called Oromo), Amharic, Gahuza (a mixed language containing Kinyarwanda and Kirundi), Hausa, Igbo, Nigerian Pidgin, Somali, Swahili, Tigrinya and Yorùbá.
The model has been shown to obtain competitive downstream performances on text classification and Named Entity Recognition on several African languages, including those it was not pretrained on.


## Intended uses & limitations


#### How to use


You can use this model with Transformers for any downstream task.
For example, assuming we want to finetune this model on a token classification task, we do the following:


```python
>>> from transformers import AutoTokenizer, AutoModelForTokenClassification
>>> model = AutoModelForTokenClassification.from_pretrained("castorini/afriberta_small")
>>> tokenizer = AutoTokenizer.from_pretrained("castorini/afriberta_small")
# we have to manually set the model max length because it is an imported trained sentencepiece model, which huggingface does not properly support right now
>>> tokenizer.model_max_length = 512
```


#### Limitations and bias


- This model is possibly limited by its training dataset which are majorly obtained from news articles from a specific span of time. Thus, it may not generalize well.
- This model is trained on very little data (less than 1 GB), hence it may not have seen enough data to learn very complex linguistic relations.


## Training data


The model was trained on an aggregation of datasets from the BBC news website and Common Crawl.


## Training procedure


For information on training procedures, please refer to the AfriBERTa [paper]() or [repository](https://github.com/keleog/afriberta)


### BibTeX entry and citation info


```
@inproceedings{ogueji-etal-2021-small,
    title = "Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages",
    author = "Ogueji, Kelechi  and
      Zhu, Yuxin  and
      Lin, Jimmy",
    booktitle = "Proceedings of the 1st Workshop on Multilingual Representation Learning",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.mrl-1.11",
    pages = "116--126",
}
```


model name: afriberta_small


model author: castorini


task: fill-mask


tags: pytorch, xlm-roberta, fill-mask, om, am, rw, rn, ha, ig, pcm, so, sw, ti, yo, multilingual, transformers, autonlp_compatible


For more information, check [doc2query.ai](http://doc2query.ai)


model name: doc2query-t5-large-msmarco


task: text2text-generation


tags: pytorch, jax, t5, text2text-generation, transformers, autonlp_compatible


This model is a T5-3B reranker fine-tuned on the MS MARCO passage dataset for 10K steps (or 1 epoch) and then fine-tuned again on MedMARCO (from [Sledge-Z paper](https://www.aclweb.org/anthology/2020.emnlp-main.341.pdf)) for 1K steps.


For more details on how to use it, check [pygaggle.ai](pygaggle.ai)!


Paper describing the model: [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://www.aclweb.org/anthology/2020.findings-emnlp.63/)


model name: monot5-3b-med-msmarco


task: feature-extraction


tags: pytorch, t5, feature-extraction, transformers


This model is converted from the original ANCE [repo](https://github.com/microsoft/ANCE) and fitted into Pyserini:


> Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, Arnold Overwijk. [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://arxiv.org/pdf/2007.00808.pdf)


For more details on how to use it, check our experiments in [Pyserini](https://github.com/castorini/pyserini/blob/master/docs/experiments-ance.md)


model name: ance-msmarco-passage


model author: castorini


tags: pytorch, roberta, arxiv:2007.00808, transformers, infinity_compatible


language:


- en
tags:
- mental-health
license: apache-2.0
datasets:
- PubMed


# Psych-Search


Psych-Search is a work in progress to bring cutting edge NLP to mental health practitioners. The model detailed here serves as a foundation for traditional classification models as well as NLU models for a Psych-Search application. The goal of the Psych-Search Application is to use a combination of traditional text classification models to expand the scope of the MESH taxonomy with the inclusion of relevant categories for mental health pracitioners designing suicide prevention programs for adolescent communities within the United States, as well as the automatic extraction and standardization of entities such as risk factors and protective factors.


Our first expansion efforts to the MESH taxonomy include categories:


- Prevention Strategies
- Protective Factors


We are actively looking for partners on this work and would love to hear from you! Please ping us at nlp4good@gmail.com.


## Model description


This model is an extension of [allenai/scibert_scivocab_uncased](https://huggingface.co/allenai/scibert_scivocab_uncased). Continued pretraining was done using SciBERT as the base model using abstract text only from Pyschology and Psychiatry PubMed research. Training was done on approximately 3.5 million papers for 10 epochs and evaluated on a task similar to BioASQ Task A.


## Intended uses & limitations


#### How to use


```python
from transformers import AutoTokenizer, AutoModel

mname = "nlp4good/psych-search"
tokenizer = AutoTokenizer.from_pretrained(mname)
model = AutoModel.from_pretrained(mname)
```


### Limitations and bias


This model was trained on all PubMed abstracts categorized under [Psychology and Psychiatry](https://meshb.nlm.nih.gov/treeView). As of March 1, this corresponds to approximately 3.2 million papers that contains abstract text. Of these 3.2 million papers, relevant sparse mental health categories were back translated to increase the representation of certain mental health categories.


There are several limitation with this dataset including large discrepancies in the number of papers associated with [Sexual and Gender Minorities](https://meshb.nlm.nih.gov/record/ui?ui=D000072339). The training data consisted of the following breakdown across gender groups:


Female | Male | Sexual and Gender Minorities
-------|---------|----------
1,896,301 | 1,945,279 | 4,529


Similar discrepancies are present within [Ethnic Groups](https://meshb.nlm.nih.gov/record/ui?ui=D005006) as defined within the MESH taxonomy: