| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 3.3587        | 0.47  | 400   | 1.1883          | 0.8392 |
| 1.8377        | 0.95  | 800   | 0.8831          | 0.6852 |
| 1.7118        | 1.42  | 1200  | 0.8031          | 0.6566 |
| 1.6741        | 1.89  | 1600  | 0.7518          | 0.6104 |
| 1.6163        | 2.36  | 2000  | 0.6888          | 0.5591 |
| 1.5782        | 2.84  | 2400  | 0.6580          | 0.5165 |
| 1.5548        | 3.31  | 2800  | 0.6506          | 0.5184 |
| 1.5249        | 3.78  | 3200  | 0.6198          | 0.5028 |
| 1.5078        | 4.26  | 3600  | 0.5992          | 0.4932 |
| 1.4836        | 4.73  | 4000  | 0.5705          | 0.4651 |
| 1.4505        | 5.2   | 4400  | 0.5489          | 0.4508 |
| 1.4481        | 5.67  | 4800  | 0.5577          | 0.4562 |
| 1.4136        | 6.15  | 5200  | 0.5452          | 0.4371 |
| 1.3861        | 6.62  | 5600  | 0.5101          | 0.4087 |
| 1.3772        | 7.09  | 6000  | 0.4933          | 0.3951 |
| 1.3478        | 7.56  | 6400  | 0.4849          | 0.3922 |
| 1.3394        | 8.04  | 6800  | 0.4805          | 0.3892 |
| 1.3095        | 8.51  | 7200  | 0.4839          | 0.3834 |
| 1.306         | 8.98  | 7600  | 0.4611          | 0.3587 |
| 1.2707        | 9.46  | 8000  | 0.4545          | 0.3730 |
| 1.2626        | 9.93  | 8400  | 0.4516          | 0.3524 |
| 1.2412        | 10.4  | 8800  | 0.4314          | 0.3310 |
| 1.2456        | 10.87 | 9200  | 0.4401          | 0.3459 |
| 1.2081        | 11.35 | 9600  | 0.4399          | 0.3356 |
| 1.1998        | 11.82 | 10000 | 0.4195          | 0.3215 |
| 1.1826        | 12.29 | 10400 | 0.4221          | 0.3178 |
| 1.1573        | 12.77 | 10800 | 0.4098          | 0.3084 |
| 1.1416        | 13.24 | 11200 | 0.4086          | 0.3119 |
| 1.1174        | 13.71 | 11600 | 0.3854          | 0.2910 |
| 1.1048        | 14.18 | 12000 | 0.3859          | 0.2824 |
| 1.0748        | 14.66 | 12400 | 0.3854          | 0.2757 |
| 1.0697        | 15.13 | 12800 | 0.3740          | 0.2724 |
| 1.0477        | 15.6  | 13200 | 0.3693          | 0.2643 |
| 1.0356        | 16.08 | 13600 | 0.3727          | 0.2561 |
| 1.0083        | 16.55 | 14000 | 0.3652          | 0.2501 |
| 1.0           | 17.02 | 14400 | 0.3641          | 0.2457 |
| 0.9779        | 17.49 | 14800 | 0.3568          | 0.2409 |
| 0.9596        | 17.97 | 15200 | 0.3558          | 0.2376 |
| 0.946         | 18.44 | 15600 | 0.3591          | 0.2311 |
| 0.9389        | 18.91 | 16000 | 0.3540          | 0.2283 |
| 0.9173        | 19.39 | 16400 | 0.3552          | 0.2265 |
| 0.9122        | 19.86 | 16800 | 0.3535          | 0.2250 |


### Framework versions


- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.17.1.dev0
- Tokenizers 0.11.0


model name: wav2vec2-xls-r-1b-hebrew


model author: imvladikon


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, he, transformers, license:apache-2.0, robust-speech-event, generated_from_trainer, model-index


language:


- en
tags:
- language model
datasets:
- wikipedia


pre-trained model from [CharBERT: Character-aware Pre-trained Language Model](https://github.com/wtma/CharBERT)


```
@misc{ma2020charbert,
      title={CharBERT: Character-aware Pre-trained Language Model},
      author={Wentao Ma and Yiming Cui and Chenglei Si and Ting Liu and Shijin Wang and Guoping Hu},
      year={2020},
      eprint={2011.01513},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: charbert-roberta-wiki


tags: pytorch, en, dataset:wikipedia, arxiv:2011.01513, transformers, language model


- en
tags:
- language model
datasets:
- wikipedia
- openwebtext


Pretrained general_character_bert model
from the ['CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters' El Boukkouri H., et al., 2020](https://github.com/helboukkouri/character-bert)


```
@inproceedings{el-boukkouri-etal-2020-characterbert,
    title = "{C}haracter{BERT}: Reconciling {ELM}o and {BERT} for Word-Level Open-Vocabulary Representations From Characters",
    author = "El Boukkouri, Hicham  and
      Ferret, Olivier  and
      Lavergne, Thomas  and
      Noji, Hiroshi  and
      Zweigenbaum, Pierre  and
      Tsujii, Jun{'}ichi",
    booktitle = "Proceedings of the 28th International Conference on Computational Linguistics",
    month = dec,
    year={2020},
    eprint={2010.10392},
    archivePrefix={arXiv},
    address = "Barcelona, Spain (Online)",
    publisher = "International Committee on Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.coling-main.609",
    doi = "10.18653/v1/2020.coling-main.609",
    pages = "6903--6915",
    abstract = "Due to the compelling improvements brought by BERT, many recent representation models adopted the Transformer architecture as their main building block, consequently inheriting the wordpiece tokenization system despite it not being intrinsically linked to the notion of Transformers. While this system is thought to achieve a good balance between the flexibility of characters and the efficiency of full words, using predefined wordpiece vocabularies from the general domain is not always suitable, especially when building models for specialized domains (e.g., the medical domain). Moreover, adopting a wordpiece tokenization shifts the focus from the word level to the subword level, making the models conceptually more complex and arguably less convenient in practice. For these reasons, we propose CharacterBERT, a new variant of BERT that drops the wordpiece system altogether and uses a Character-CNN module instead to represent entire words by consulting their characters. We show that this new model improves the performance of BERT on a variety of medical domain tasks while at the same time producing robust, word-level, and open-vocabulary representations.",
}
```


model name: general_character_bert


tags: pytorch, bert, en, dataset:wikipedia, dataset:openwebtext, arxiv:2010.10392, transformers, language model, infinity_compatible


- he
tags:
- automatic-speech-recognition
- robust-speech-event
- he
- generated_from_trainer
model-index:
- name: wav2vec2-xls-r-300m-hebrew
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-xls-r-300m-hebrew


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the private datasets in 2 stages - firstly was fine-tuned on a small dataset with good samples Then the obtained model was fine-tuned on a large dataset with the small good dataset, with various samples from different sources, and with an unlabeled dataset that was weakly labeled using a previously trained model.


Small dataset:


| split  |size(gb) | n_samples | duration(hrs)|   |
|---|---|---|---|---|
|train|4.19| 20306  | 28  |   |
|dev  |1.05|  5076 |  7 |   |


Large dataset:


| split  |size(gb) | n_samples | duration(hrs)|   |
|---|---|---|---|---|
|train|12.3| 90777  | 69  |   |
|dev  |2.39|  20246 |  14* |   |
(*weakly labeled data wasn't used in validation set)


After firts training it achieves:


on small dataset


- Loss: 0.5438
- WER: 0.1773


on large dataset


- WER: 0.3811


after second training:
on small dataset


- WER: 0.1697


- Loss: 0.4502
- WER: 0.2318


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


#### First training


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 100.0
- mixed_precision_training: Native AMP


Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| No log        | 3.15  | 1000  | 0.5203          | 0.4333 |
| 1.4284        | 6.31  | 2000  | 0.4816          | 0.3951 |
| 1.4284        | 9.46  | 3000  | 0.4315          | 0.3546 |
| 1.283         | 12.62 | 4000  | 0.4278          | 0.3404 |
| 1.283         | 15.77 | 5000  | 0.4090          | 0.3054 |
| 1.1777        | 18.93 | 6000  | 0.3893          | 0.3006 |
| 1.1777        | 22.08 | 7000  | 0.3968          | 0.2857 |
| 1.0994        | 25.24 | 8000  | 0.3892          | 0.2751 |
| 1.0994        | 28.39 | 9000  | 0.4061          | 0.2690 |
| 1.0323        | 31.54 | 10000 | 0.4114          | 0.2507 |
| 1.0323        | 34.7  | 11000 | 0.4021          | 0.2508 |
| 0.9623        | 37.85 | 12000 | 0.4032          | 0.2378 |
| 0.9623        | 41.01 | 13000 | 0.4148          | 0.2374 |
| 0.9077        | 44.16 | 14000 | 0.4350          | 0.2323 |
| 0.9077        | 47.32 | 15000 | 0.4515          | 0.2246 |
| 0.8573        | 50.47 | 16000 | 0.4474          | 0.2180 |
| 0.8573        | 53.63 | 17000 | 0.4649          | 0.2171 |
| 0.8083        | 56.78 | 18000 | 0.4455          | 0.2102 |
| 0.8083        | 59.94 | 19000 | 0.4587          | 0.2092 |
| 0.769         | 63.09 | 20000 | 0.4794          | 0.2012 |
| 0.769         | 66.25 | 21000 | 0.4845          | 0.2007 |
| 0.7308        | 69.4  | 22000 | 0.4937          | 0.2008 |
| 0.7308        | 72.55 | 23000 | 0.4920          | 0.1895 |
| 0.6927        | 75.71 | 24000 | 0.5179          | 0.1911 |
| 0.6927        | 78.86 | 25000 | 0.5202          | 0.1877 |
| 0.6622        | 82.02 | 26000 | 0.5266          | 0.1840 |
| 0.6622        | 85.17 | 27000 | 0.5351          | 0.1854 |
| 0.6315        | 88.33 | 28000 | 0.5373          | 0.1811 |
| 0.6315        | 91.48 | 29000 | 0.5331          | 0.1792 |
| 0.6075        | 94.64 | 30000 | 0.5390          | 0.1779 |
| 0.6075        | 97.79 | 31000 | 0.5459          | 0.1773 |


#### Second training


- learning_rate: 0.0003
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 60.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| No log        | 0.7   | 1000  | 0.5371          | 0.3811 |
| 1.3606        | 1.41  | 2000  | 0.5247          | 0.3902 |
| 1.3606        | 2.12  | 3000  | 0.5126          | 0.3859 |
| 1.3671        | 2.82  | 4000  | 0.5062          | 0.3828 |
| 1.3671        | 3.53  | 5000  | 0.4979          | 0.3672 |
| 1.3421        | 4.23  | 6000  | 0.4906          | 0.3816 |
| 1.3421        | 4.94  | 7000  | 0.4784          | 0.3651 |
| 1.328         | 5.64  | 8000  | 0.4810          | 0.3669 |
| 1.328         | 6.35  | 9000  | 0.4747          | 0.3597 |
| 1.3109        | 7.05  | 10000 | 0.4813          | 0.3808 |
| 1.3109        | 7.76  | 11000 | 0.4631          | 0.3561 |
| 1.2873        | 8.46  | 12000 | 0.4603          | 0.3431 |
| 1.2873        | 9.17  | 13000 | 0.4579          | 0.3533 |
| 1.2661        | 9.87  | 14000 | 0.4471          | 0.3365 |
| 1.2661        | 10.58 | 15000 | 0.4584          | 0.3437 |
| 1.249         | 11.28 | 16000 | 0.4461          | 0.3454 |
| 1.249         | 11.99 | 17000 | 0.4482          | 0.3367 |
| 1.2322        | 12.69 | 18000 | 0.4464          | 0.3335 |
| 1.2322        | 13.4  | 19000 | 0.4427          | 0.3454 |
| 1.22          | 14.1  | 20000 | 0.4440          | 0.3395 |
| 1.22          | 14.81 | 21000 | 0.4459          | 0.3378 |
| 1.2044        | 15.51 | 22000 | 0.4406          | 0.3199 |
| 1.2044        | 16.22 | 23000 | 0.4398          | 0.3155 |
| 1.1913        | 16.92 | 24000 | 0.4237          | 0.3150 |
| 1.1913        | 17.63 | 25000 | 0.4287          | 0.3279 |
| 1.1705        | 18.34 | 26000 | 0.4253          | 0.3103 |
| 1.1705        | 19.04 | 27000 | 0.4234          | 0.3098 |
| 1.1564        | 19.75 | 28000 | 0.4174          | 0.3076 |
| 1.1564        | 20.45 | 29000 | 0.4260          | 0.3160 |
| 1.1461        | 21.16 | 30000 | 0.4235          | 0.3036 |
| 1.1461        | 21.86 | 31000 | 0.4309          | 0.3055 |
| 1.1285        | 22.57 | 32000 | 0.4264          | 0.3006 |
| 1.1285        | 23.27 | 33000 | 0.4201          | 0.2880 |
| 1.1135        | 23.98 | 34000 | 0.4131          | 0.2975 |
| 1.1135        | 24.68 | 35000 | 0.4202          | 0.2849 |
| 1.0968        | 25.39 | 36000 | 0.4105          | 0.2888 |
| 1.0968        | 26.09 | 37000 | 0.4210          | 0.2834 |
| 1.087         | 26.8  | 38000 | 0.4123          | 0.2843 |
| 1.087         | 27.5  | 39000 | 0.4216          | 0.2803 |
| 1.0707        | 28.21 | 40000 | 0.4161          | 0.2787 |
| 1.0707        | 28.91 | 41000 | 0.4186          | 0.2740 |
| 1.0575        | 29.62 | 42000 | 0.4118          | 0.2845 |
| 1.0575        | 30.32 | 43000 | 0.4243          | 0.2773 |
| 1.0474        | 31.03 | 44000 | 0.4221          | 0.2707 |
| 1.0474        | 31.73 | 45000 | 0.4138          | 0.2700 |
| 1.0333        | 32.44 | 46000 | 0.4102          | 0.2638 |
| 1.0333        | 33.15 | 47000 | 0.4162          | 0.2650 |
| 1.0191        | 33.85 | 48000 | 0.4155          | 0.2636 |
| 1.0191        | 34.56 | 49000 | 0.4129          | 0.2656 |
| 1.0087        | 35.26 | 50000 | 0.4157          | 0.2632 |
| 1.0087        | 35.97 | 51000 | 0.4090          | 0.2654 |
| 0.9901        | 36.67 | 52000 | 0.4183          | 0.2587 |
| 0.9901        | 37.38 | 53000 | 0.4251          | 0.2648 |
| 0.9795        | 38.08 | 54000 | 0.4229          | 0.2555 |
| 0.9795        | 38.79 | 55000 | 0.4176          | 0.2546 |
| 0.9644        | 39.49 | 56000 | 0.4223          | 0.2513 |
| 0.9644        | 40.2  | 57000 | 0.4244          | 0.2530 |
| 0.9534        | 40.9  | 58000 | 0.4175          | 0.2538 |
| 0.9534        | 41.61 | 59000 | 0.4213          | 0.2505 |
| 0.9397        | 42.31 | 60000 | 0.4275          | 0.2565 |
| 0.9397        | 43.02 | 61000 | 0.4315          | 0.2528 |
| 0.9269        | 43.72 | 62000 | 0.4316          | 0.2501 |
| 0.9269        | 44.43 | 63000 | 0.4247          | 0.2471 |
| 0.9175        | 45.13 | 64000 | 0.4376          | 0.2469 |
| 0.9175        | 45.84 | 65000 | 0.4335          | 0.2450 |
| 0.9026        | 46.54 | 66000 | 0.4336          | 0.2452 |
| 0.9026        | 47.25 | 67000 | 0.4400          | 0.2427 |
| 0.8929        | 47.95 | 68000 | 0.4382          | 0.2429 |
| 0.8929        | 48.66 | 69000 | 0.4361          | 0.2415 |
| 0.8786        | 49.37 | 70000 | 0.4413          | 0.2398 |
| 0.8786        | 50.07 | 71000 | 0.4392          | 0.2415 |
| 0.8714        | 50.78 | 72000 | 0.4345          | 0.2406 |
| 0.8714        | 51.48 | 73000 | 0.4475          | 0.2402 |
| 0.8589        | 52.19 | 74000 | 0.4473          | 0.2374 |
| 0.8589        | 52.89 | 75000 | 0.4457          | 0.2357 |
| 0.8493        | 53.6  | 76000 | 0.4462          | 0.2366 |
| 0.8493        | 54.3  | 77000 | 0.4494          | 0.2356 |
| 0.8395        | 55.01 | 78000 | 0.4472          | 0.2352 |
| 0.8395        | 55.71 | 79000 | 0.4490          | 0.2339 |
| 0.8295        | 56.42 | 80000 | 0.4489          | 0.2318 |
| 0.8295        | 57.12 | 81000 | 0.4469          | 0.2320 |
| 0.8225        | 57.83 | 82000 | 0.4478          | 0.2321 |
| 0.8225        | 58.53 | 83000 | 0.4525          | 0.2326 |
| 0.816         | 59.24 | 84000 | 0.4532          | 0.2316 |
| 0.816         | 59.94 | 85000 | 0.4502          | 0.2318 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


model name: wav2vec2-xls-r-300m-hebrew


model author: imvladikon


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, he, transformers, robust-speech-event, generated_from_trainer, model-index


language: he
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Hebrew XLSR Wav2Vec2 Large 53
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name:
type:
args: he
metrics:
    - name: Test WER
type: wer
value:


# wav2vec2-large-xlsr-53-hebrew


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on the several downloaded youtube samples.
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "he", split="test[:2%]") # there is no common dataset for Hebrew, please, paste your data
processor = Wav2Vec2Processor.from_pretrained("imvladikon/wav2vec2-large-xlsr-53-hebrew")
model = Wav2Vec2ForCTC.from_pretrained("imvladikon/wav2vec2-large-xlsr-53-hebrew")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
  speech_array, sampling_rate = torchaudio.load(batch["path"])
  batch["speech"] = resampler(speech_array).squeeze().numpy()
  return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
  tlogits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on some Hebrew test data


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
test_dataset = load_dataset("common_voice", "he", split="test") # there is no common dataset for Hebrew, please, paste your data
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("imvladikon/wav2vec2-large-xlsr-53-hebrew")
model = Wav2Vec2ForCTC.from_pretrained("imvladikon/wav2vec2-large-xlsr-53-hebrew").to("cuda")
chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\‘\”\�]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
  batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
  speech_array, sampling_rate = torchaudio.load(batch["path"])
  batch["speech"] = resampler(speech_array).squeeze().numpy()
  return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
  inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
  with torch.no_grad():
    logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
  pred_ids = torch.argmax(logits, dim=-1)
  batch["pred_strings"] = processor.batch_decode(pred_ids)
  return batch
result = test_dataset.map(evaluate, batched=True, batch_size=8)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**:


# Example Predictions


model name: wav2vec2-large-xlsr-53-hebrew


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, he, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language:


- en
license:
tags:
- generated_from_trainer
datasets:
- jigsaw
model_index:
- name: bert-base-uncased
results:
  - {}


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# bert-base-uncased


This model is a fine-tuned version of [](https://huggingface.co/) on the jigsaw dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0393
- Precision Micro: 0.7758
- Recall Micro: 0.7858
- F1 Micro: 0.7808
- F2 Micro: 0.7838
- Precision Macro: 0.6349
- Recall Macro: 0.5972
- F1 Macro: 0.6105
- F2 Macro: 0.6015
- Overall Precision: 0.9841
- Overall Recall: 0.9841
- Overall F1: 0.9841
- Overall F2: 0.9841
- Overall Accuracy: 0.9841
- Matthews Corrcoef: 0.7725
- Aucroc Macro: 0.9897
- Aucroc Micro: 0.9920
- Accuracy Toxic: 0.9678
- F1 Toxic: 0.8295
- Accuracy Severe Toxic: 0.9899
- F1 Severe Toxic: 0.3313
- Accuracy Obscene: 0.9816
- F1 Obscene: 0.8338
- Accuracy Threat: 0.9974
- F1 Threat: 0.4545
- Accuracy Insult: 0.9763
- F1 Insult: 0.7662
- Accuracy Identity Hate: 0.9914
- F1 Identity Hate: 0.4480


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 3e-05
- train_batch_size: 24
- eval_batch_size: 12
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 48
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Precision Micro | Recall Micro | F1 Micro | F2 Micro | Precision Macro | Recall Macro | F1 Macro | F2 Macro | Overall Precision | Overall Recall | Overall F1 | Overall F2 | Overall Accuracy | Matthews Corrcoef | Aucroc Macro | Aucroc Micro | Accuracy Toxic | F1 Toxic | Accuracy Severe Toxic | F1 Severe Toxic | Accuracy Obscene | F1 Obscene | Accuracy Threat | F1 Threat | Accuracy Insult | F1 Insult | Accuracy Identity Hate | F1 Identity Hate |
|:-------------:|:-----:|:-----:|:---------------:|:---------------:|:------------:|:--------:|:--------:|:---------------:|:------------:|:--------:|:--------:|:-----------------:|:--------------:|:----------:|:----------:|:----------------:|:-----------------:|:------------:|:------------:|:--------------:|:--------:|:---------------------:|:---------------:|:----------------:|:----------:|:---------------:|:---------:|:---------------:|:---------:|:----------------------:|:----------------:|
| 0.0433        | 1.0   | 2659  | 0.0423          | 0.7607          | 0.7798       | 0.7702   | 0.7759   | 0.6398          | 0.5561       | 0.5585   | 0.5535   | 0.9832            | 0.9832         | 0.9832     | 0.9832     | 0.9832           | 0.7615            | 0.9887       | 0.9908       | 0.9671         | 0.8211   | 0.9878                | 0.4354          | 0.9805           | 0.8265     | 0.9974          | 0.2243    | 0.9746          | 0.7602    | 0.9918                 | 0.2834           |
| 0.0366        | 2.0   | 5318  | 0.0393          | 0.7758          | 0.7858       | 0.7808   | 0.7838   | 0.6349          | 0.5972       | 0.6105   | 0.6015   | 0.9841            | 0.9841         | 0.9841     | 0.9841     | 0.9841           | 0.7725            | 0.9897       | 0.9920       | 0.9678         | 0.8295   | 0.9899                | 0.3313          | 0.9816           | 0.8338     | 0.9974          | 0.4545    | 0.9763          | 0.7662    | 0.9914                 | 0.4480           |
| 0.0305        | 3.0   | 7977  | 0.0399          | 0.7608          | 0.8186       | 0.7887   | 0.8064   | 0.6621          | 0.6856       | 0.6715   | 0.6794   | 0.9842            | 0.9842         | 0.9842     | 0.9842     | 0.9842           | 0.7810            | 0.9897       | 0.9919       | 0.9662         | 0.8272   | 0.9892                | 0.4772          | 0.9815           | 0.8347     | 0.9977          | 0.5629    | 0.9772          | 0.7740    | 0.9931                 | 0.5528           |
| 0.0263        | 4.0   | 10636 | 0.0435          | 0.7333          | 0.8336       | 0.7803   | 0.8114   | 0.6395          | 0.7039       | 0.6687   | 0.6890   | 0.9830            | 0.9830         | 0.9830     | 0.9830     | 0.9830           | 0.7732            | 0.9897       | 0.9912       | 0.9608         | 0.8083   | 0.9898                | 0.4791          | 0.9812           | 0.8319     | 0.9972          | 0.5368    | 0.9756          | 0.7700    | 0.9935                 | 0.5861           |
| 0.0218        | 5.0   | 13295 | 0.0456          | 0.7480          | 0.8108       | 0.7781   | 0.7974   | 0.6661          | 0.6720       | 0.6662   | 0.6691   | 0.9833            | 0.9833         | 0.9833     | 0.9833     | 0.9833           | 0.7701            | 0.9890       | 0.9907       | 0.9612         | 0.8071   | 0.9894                | 0.4642          | 0.9823           | 0.8354     | 0.9977          | 0.5325    | 0.9754          | 0.7613    | 0.9936                 | 0.5968           |


### Framework versions


- Transformers 4.8.2
- Pytorch 1.9.0+cu102
- Datasets 1.9.0
- Tokenizers 0.10.3


model name: bert-base-uncased-jigsaw


model author: imvladikon


tags: pytorch, bert, en, dataset:jigsaw, transformers, generated_from_trainer, infinity_compatible


language:


- en
tags:
- language model
datasets:
- wikipedia


pre-trained model from [CharBERT: Character-aware Pre-trained Language Model](https://github.com/wtma/CharBERT)


```
@misc{ma2020charbert,
      title={CharBERT: Character-aware Pre-trained Language Model},
      author={Wentao Ma and Yiming Cui and Chenglei Si and Ting Liu and Shijin Wang and Guoping Hu},
      year={2020},
      eprint={2011.01513},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


model name: charbert-bert-wiki


tags: pytorch, en, dataset:wikipedia, arxiv:2011.01513, transformers, language model


- en


<!-- thumbnail: https://raw.githubusercontent.com/JetRunner/BERT-of-Theseus/master/bert-of-theseus.png
  -->


tags:


- topic labeling
license: apache-2.0
metrics:
- ndcg


# MyModel


## Model description


This is the `BART-TL-all` model from the paper [BART-TL: Weakly-Supervised Topic Label Generation](https://www.aclweb.org/anthology/2021.eacl-main.121.pdf). We aim to solve the topic labeling task using generative methods, rather than selection from a pool of labels as was done in previous State of the Art works.


For more details not covered here, you can read the paper or look at the open-source implementation: https://github.com/CristianViorelPopa/BART-TL-topic-label-generation.


There are two models made available from the paper:


* [BART-TL-all](https://huggingface.co/cristian-popa/bart-tl-all)
* [BART-TL-ng](https://huggingface.co/cristian-popa/bart-tl-ng)


## Intended uses & limitations


#### How to use


The model takes in a topic, represented as a space-separated series of words. Such topics can be generated using LDA, as was done for gathering the fine-tuning dataset for the model.


```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

mname = "cristian-popa/bart-tl-all"
tokenizer = AutoTokenizer.from_pretrained(mname)
model = AutoModelForSeq2SeqLM.from_pretrained(mname)

input = "site web google search website online internet social content user"
enc = tokenizer(input, return_tensors="pt", truncation=True, padding="max_length", max_length=128)
outputs = model.generate(
	input_ids=enc.input_ids,
	attention_mask=enc.attention_mask,
    max_length=15,
    min_length=1,
    do_sample=False,
    num_beams=25,
    length_penalty=1.0,
    repetition_penalty=1.5
)

decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(decoded) # application programming interface
```


#### Limitations and bias


The model may not generate accurate labels for topics from domains unrelated to the ones it was fine-tuned on, such as gastronomy.


## Training data


The model was fine-tuned on 5 different StackExchange corpora (see https://archive.org/download/stackexchange for a full list of existing such corpora): English, biology, economics, law, and photography. 100 topics are extracted using LDA for each of these corpora, filtered for coherence and then used for obtaining the final model here.


## Training procedure


The large Facebook BART model is fine-tuned in a weakly-supervised manner, making use of the unsupervised candidate selection of the [NETL](https://www.aclweb.org/anthology/C16-1091.pdf) method, along with other heuristic labels, such as n-grams from the topics, relevant sentences in the corpora and noun phrases. The dataset is a one-to-many mapping from topics to labels. More details on training and parameters can be found in the [paper](https://www.aclweb.org/anthology/2021.eacl-main.121.pdf) or by following [this notebook](https://github.com/CristianViorelPopa/BART-TL-topic-label-generation/blob/main/notebooks/end_to_end_workflow.ipynb).


## Eval results


model       | Top-1 Avg. | Top-3 Avg. | Top-5 Avg. | nDCG-1 | nDCG-3 | nDCG-5
------------|------------|------------|------------|--------|--------|-------
NETL (U)	| 2.66		 | 2.59		  | 2.50	   | 0.83	| 0.85	 | 0.87
NETL (S)	| 2.74		 | 2.57		  | 2.49	   | 0.88	| 0.85	 | 0.88
BART-TL-all | 2.64		 | 2.52		  | 2.43	   | 0.83	| 0.84	 | 0.87
BART-TL-ng 	| 2.62		 | 2.50		  | 2.33	   | 0.82	| 0.84	 | 0.85


### BibTeX entry and citation info


```bibtex
@inproceedings{popa-rebedea-2021-bart,
    title = "{BART}-{TL}: Weakly-Supervised Topic Label Generation",
    author = "Popa, Cristian  and
      Rebedea, Traian",
    booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume",
    month = apr,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.eacl-main.121",
    pages = "1418--1425",
    abstract = "We propose a novel solution for assigning labels to topic models by using multiple weak labelers. The method leverages generative transformers to learn accurate representations of the most important topic terms and candidate labels. This is achieved by fine-tuning pre-trained BART models on a large number of potential labels generated by state of the art non-neural models for topic labeling, enriched with different techniques. The proposed BART-TL model is able to generate valuable and novel labels in a weakly-supervised manner and can be improved by adding other weak labelers or distant supervision on similar tasks.",
}
```


model name: bart-tl-all


model author: cristian-popa


task: text2text-generation


tags: pytorch, bart, text2text-generation, en, transformers, topic labeling, license:apache-2.0, autonlp_compatible


language:


- en


tags:


# MyModel


## Model description


This is the `BART-TL-ng` model from the paper [BART-TL: Weakly-Supervised Topic Label Generation](https://www.aclweb.org/anthology/2021.eacl-main.121.pdf). We aim to solve the topic labeling task using generative methods, rather than selection from a pool of labels as was done in previous State of the Art works.


## Intended uses & limitations


#### How to use


```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

mname = "cristian-popa/bart-tl-ng"
tokenizer = AutoTokenizer.from_pretrained(mname)
model = AutoModelForSeq2SeqLM.from_pretrained(mname)

input = "site web google search website online internet social content user"
enc = tokenizer(input, return_tensors="pt", truncation=True, padding="max_length", max_length=128)
outputs = model.generate(
	input_ids=enc.input_ids,
	attention_mask=enc.attention_mask,
    max_length=15,
    min_length=1,
    do_sample=False,
    num_beams=25,
    length_penalty=1.0,
    repetition_penalty=1.5
)

decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(decoded) # windows live messenger
```


#### Limitations and bias


## Training data


## Training procedure


The large Facebook BART model is fine-tuned in a weakly-supervised manner, making use of the unsupervised candidate selection of the [NETL](https://www.aclweb.org/anthology/C16-1091.pdf) method, along with n-grams from the topics. The dataset is a one-to-many mapping from topics to labels. More details on training and parameters can be found in the [paper](https://www.aclweb.org/anthology/2021.eacl-main.121.pdf) or by following [this notebook](https://github.com/CristianViorelPopa/BART-TL-topic-label-generation/blob/main/notebooks/end_to_end_workflow.ipynb).


## Eval results


### BibTeX entry and citation info


model name: bart-tl-ng


model author: cristian-popa


task: text2text-generation


language:


- de
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- de
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
model-index:
- name: XLS-R-1B - German
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 8
type: mozilla-foundation/common_voice_8_0
args: de
metrics:
    - name: Test WER
type: wer
value: 11.37
    - name: Test CER
type: cer
value: 2.89
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: de
metrics:
    - name: Dev WER
type: wer
value: 31.16
    - name: Dev CER
type: cer
value: 13.41


# XLS-R-1b-DE


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - DE dataset. (See `run.sh` for training parameters).


model name: wav2vec2-xls-r-1b-de-cv8


model author: jsnfly


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, de, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, mozilla-foundation/common_voice_8_0, robust-speech-event, model-index


- de
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_7_0
- de
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
model-index:
- name: Wav2Vec2-Large-XLSR-53-German-GPT2
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 7
type: mozilla-foundation/common_voice_7_0
args: de
metrics:
    - name: Test WER
type: wer
value: 10.02
    - name: Test CER
type: cer
value: 4.7


# Wav2Vec2-Large-XLSR-53-German-GPT2


This is an encoder-decoder model for automatic speech recognition trained on on the
MOZILLA-FOUNDATION/COMMON_VOICE_7_0 - DE dataset. The encoder was initialized from
[jonatasgrosman/wav2vec2-large-xlsr-53-german](https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-german) and
the decoder from [dbmdz/german-gpt2](https://huggingface.co/dbmdz/german-gpt2).


It was trained using a two step process:


* fine-tuning only the cross-attention weights and the decoder using the pre-computed outputs of the Wav2Vec-Modell
  * relatively fast training
  * also works on small GPU (eg. 8 GB)
  * but may take a lot of disk space
  * should already yield decent results
* fine-tuning the model end-to-end
  * much slower
  * needs a bigger GPU


There is also one trick, which seemed to improve performance significantly: adding position embeddings to the
encoder outputs and initializing them with the pre-trained position embeddings of the GPT2 model (See `eval.py`).


The training notebooks are still early drafts. Also results can probably improved a lot by using for example a learning
rate schedule.


model name: wav2vec2-large-xlsr-53-german-gpt2


tags: pytorch, speech-encoder-decoder, automatic-speech-recognition, de, dataset:mozilla-foundation/common_voice_7_0, transformers, license:apache-2.0, mozilla-foundation/common_voice_7_0, robust-speech-event, model-index


```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("dpalominop/spanish-bert-apoyo")
model = AutoModelForSequenceClassification.from_pretrained("dpalominop/spanish-bert-apoyo")
```


model name: spanish-bert-apoyo


model author: dpalominop


task: text-classification


tags: pytorch, jax, bert, text-classification, transformers, infinity_compatible


# language:


- en
- zu


# license:


- apache-2.0


# tags:


- Translation
- Text2Text Generation


# datasets:


## Pre-trained on:


- MarianMT English - isiXhosa
- Dataset: OPUS (https://opus.nlpl.eu/)


## Fine-tuned on:


- Umsuka English - isiZulu Parallel Corpus (https://zenodo.org/record/5035171#.YZ39iNBBy3A)


# metrics


- BLEU


# Usage:


```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("MUNasir/opus-en-xh-umsuka-en-zu")

model = AutoModelForSeq2SeqLM.from_pretrained("MUNasir/opus-en-xh-umsuka-en-zu")
```


OR download the repo:


```
git lfs install
git clone https://huggingface.co/MUNasir/opus-en-xh-umsuka-en-zu
# if you want to clone without large files – just their pointers
# prepend your git clone with the following env var:
GIT_LFS_SKIP_SMUDGE=1
```


# Benchmarks


| Corpus                 | BLEU (dev)                                                   |      BLEU (test)                          |\
|----------------------|-----------------------------------------------------------|------------------------------------------------|
| Umsuka                 | 15.4806                                                    | 13.042


# Main Contributors:


| Name                 | GitHub                                                    | Social                                                                                |
|----------------------|-----------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
| Muhammad Umair Nasir | [umair-nasir14](https://github.com/umair-nasir14)         | [Linkedin](https://www.linkedin.com/in/umair-nasir/) \| [Twitter](https://twitter.com/utheprodigyn) |
| Samantha Ball        | [SamBall999](https://github.com/SamBall999)               | -                                                                                |
| Innocent Mchechesi   | [InnocentMchechesi](https://github.com/InnocentMchechesi) | -                                                                                |


# GitHub repo:


model name: opus-en-xh-umsuka-en-zu


model author: MUNasir


task: text2text-generation


tags: pytorch, marian, text2text-generation, transformers, autonlp_compatible


tags:


- conversational


# Mourinhio


model name: Mourinhio-medium


model author: Username1


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


# Wenger


model name: Wenger


model name: Mourinho


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- xsum
model-index:
- name: t5-small-finetuned-xsum
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# t5-small-finetuned-xsum


This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on the xsum dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- mixed_precision_training: Native AMP


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: t5-small-finetuned-xsum


model author: bdwjaya


task: text2text-generation


tags: pytorch, tensorboard, t5, text2text-generation, dataset:xsum, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible


language:


- ar
- dz


tags:


- pytorch
- bert
- multilingual
- ar
- dz


license: apache-2.0


widget:


- text: " أنا من الجزائر من ولاية [MASK] "
- text: "rabi [MASK] khouya sami"
- text: " ربي [MASK] خويا لعزيز"
- text: "tahya el [MASK]."
- text: "rouhi ya dzayer [MASK]"


## inference: true


<img src="https://raw.githubusercontent.com/alger-ia/dziribert/main/dziribert_drawing.png" alt="drawing" width="25%" height="25%" align="right"/>


# DziriBERT


DziriBERT is the first Transformer-based Language Model that has been pre-trained specifically for the Algerian Dialect. It handles Algerian text contents written using both Arabic and Latin characters. It sets new state of the art results on Algerian text classification datasets, even if it has been pre-trained on much less data (~1 million tweets).


For more information, please visit our paper: https://arxiv.org/pdf/2109.12346.pdf.


## How to use


```python
from transformers import BertTokenizer, BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained("alger-ia/dziribert")
model = BertForMaskedLM.from_pretrained("alger-ia/dziribert")

```


You can find a fine-tuning script in our Github repo: https://github.com/alger-ia/dziribert


### How to cite


```bibtex
@article{dziribert,
  title={DziriBERT: a Pre-trained Language Model for the Algerian Dialect},
  author={Abdaoui, Amine and Berrimi, Mohamed and Oussalah, Mourad and Moussaoui, Abdelouahab},
  journal={arXiv preprint arXiv:2109.12346},
  year={2021}
}
```


## Contact


Please contact amine.abdaoui@huawei.com for any question, feedback or request.


model name: dziribert


model author: alger-ia


task: fill-mask


tags: pytorch, tf, bert, fill-mask, ar, dz, arxiv:2109.12346, transformers, multilingual, license:apache-2.0, autonlp_compatible, infinity_compatible


language:


- fr


thumbnail: https://raw.githubusercontent.com/AntoineSimoulin/gpt-fr/main/imgs/logo.png
tags:


- tf
- pytorch
- gpt2
- text-generation
license: apache-2.0


<img src="https://raw.githubusercontent.com/AntoineSimoulin/gpt-fr/main/imgs/logo.png" width="200">


## Model description


**GPT-fr** 🇫🇷 is a GPT model for French developped by [Quantmetry](https://www.quantmetry.com/) and the [Laboratoire de Linguistique Formelle (LLF)](http://www.llf.cnrs.fr/en). We train the model on a very large and heterogeneous French corpus. We release the weights for the following configurations:


| Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters |
| :------:       |   :---: | :---: | :---: | :---: |
| `gpt-fr-cased-small` | 12    | 12    | 768   | 124 M |
| `gpt-fr-cased-base` | 24    | 14    | 1,792   | 1,017 B |


## Intended uses & limitations


The model can be leveraged for language generation tasks. Besides, many tasks may be formatted such that the output is directly generated in natural language. Such configuration may be used for tasks such as automatic summary or question answering. We do hope our model might be used for both academic and industrial applications.


#### How to use


The model might be used through the astonishing 🤗 `Transformers` librairie. We use the work from [Shoeybi et al., (2019)](#shoeybi-2019) and calibrate our model such that during pre-training or fine-tuning, the model can fit on a single NVIDIA V100 32GB GPU.


```python
from transformers import GPT2Tokenizer, GPT2LMHeadModel

# Load pretrained model and tokenizer
model = GPT2LMHeadModel.from_pretrained("asi/gpt-fr-cased-base")
tokenizer = GPT2Tokenizer.from_pretrained("asi/gpt-fr-cased-base")

# Generate a sample of text
model.eval()
input_sentence = "Longtemps je me suis couché de bonne heure."
input_ids = tokenizer.encode(input_sentence, return_tensors='pt')

beam_outputs = model.generate(
    input_ids,
    max_length=100,
    do_sample=True,
    top_k=50,
    top_p=0.95,
    num_return_sequences=1
)

print("Output:\n" + 100 * '-')
print(tokenizer.decode(beam_outputs[0], skip_special_tokens=True))
```


#### Limitations and bias


Large language models tend to replicate the biases found in pre-training datasets, such as gender discrimination or offensive content generation.


To limit exposition to too much explicit material, we carefully choose the sources beforehand. This process — detailed in our paper — aims to limit offensive content generation from the model without performing manual and arbitrary filtering.


However, some societal biases, contained in the data, might be reflected by the model. For example on gender equality, we generated the following sentence sequence "Ma femme/Mon mari vient d'obtenir un nouveau poste en tant \_\_\_\_\_\_\_". We used top-k random sampling strategy with k=50 and stopped at the first punctuation element.
The positions generated for the wife is '*que professeur de français.*' while the position for the husband is '*que chef de projet.*'. We do appreciate your feedback to better qualitatively and quantitatively assess such effects.


## Training data


We created a dedicated corpus to train our generative model. Indeed the model uses a fixed-length context size of 1,024 and require long documents to be trained.  We aggregated existing corpora: [Wikipedia](https://dumps.wikimedia.org/frwiki/), [OpenSubtitle](http://opus.nlpl.eu/download.php?f=OpenSubtitles/v2016/mono/) ([Tiedemann, 2012](#tiedemann-2012)), [Gutenberg](http://www.gutenberg.org) and [Common Crawl](http://data.statmt.org/ngrams/deduped2017/) ([Li et al., 2019](li-2019)). Corpora are filtered and separated into sentences. Successive sentences are then concatenated within the limit of 1,024 tokens per document.


## Training procedure


We pre-trained the model on the new CNRS (French National Centre for Scientific Research) [Jean Zay](http://www.idris.fr/eng/jean-zay/) supercomputer. We perform the training within a total of 140 hours of computation on Tesla V-100 hardware (TDP of 300W). The training was distributed on 4 compute nodes of 8 GPUs. We used data parallelization in order to divide each micro-batch on the computing units. We estimated the total emissions at 580.61 kgCO2eq, using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al., (2019)](lacoste-2019).


## Eval results


We packaged **GPT-fr** with a dedicated language model evaluation benchmark for French.
In line with the [WikiText](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/) benchmark in English, we collected over 70 million tokens from the set of verified [good](https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Articles_de_qualit%C3%A9) and [featured](https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Bons_articles) articles on Wikipedia. The model reaches a zero-shot perplexity of **12.9** on the test set.


### BibTeX entry and citation info


Along with the model hosted by HuggingFace transformers library, we maintain a [git repository](https://github.com/AntoineSimoulin/gpt-fr).
If you use **GPT-fr** for your scientific publications or your industrial applications, please cite the following paper:


```bibtex
@inproceedings{simoulin:hal-03265900,
  TITLE = {{Un mod{\`e}le Transformer G{\'e}n{\'e}ratif Pr{\'e}-entrain{\'e} pour le \_\_\_\_\_\_ fran{\c c}ais}},
  AUTHOR = {Simoulin, Antoine and Crabb{\'e}, Benoit},
  URL = {https://hal.archives-ouvertes.fr/hal-03265900},
  BOOKTITLE = {{Traitement Automatique des Langues Naturelles}},
  ADDRESS = {Lille, France},
  EDITOR = {Denis, Pascal and Grabar, Natalia and Fraisse, Amel and Cardon, R{\'e}mi and Jacquemin, Bernard and Kergosien, Eric and Balvet, Antonio},
  PUBLISHER = {{ATALA}},
  PAGES = {246-255},
  YEAR = {2021},
  KEYWORDS = {fran{\c c}ais. ; GPT ; G{\'e}n{\'e}ratif ; Transformer ; Pr{\'e}-entra{\^i}n{\'e}},
  PDF = {https://hal.archives-ouvertes.fr/hal-03265900/file/7.pdf},
  HAL_ID = {hal-03265900},
  HAL_VERSION = {v1},
}
```


### References


>
Jörg Tiedemann: Parallel Data, Tools and Interfaces in OPUS. LREC 2012: 2214-2218


>
Xian Li, Paul Michel, Antonios Anastasopoulos, Yonatan Belinkov, Nadir Durrani, Orhan Firat, Philipp Koehn, Graham Neubig, Juan Pino, Hassan Sajjad: Findings of the First Shared Task on Machine Translation Robustness. WMT (2) 2019: 91-102


>
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. CoRR abs/1909.08053 (2019)


>
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, Thomas Dandres: Quantifying the Carbon Emissions of Machine Learning. CoRR abs/1910.09700 (2019)


model name: gpt-fr-cased-base


model author: asi


task: text-generation


tags: pytorch, tf, jax, gpt2, text-generation, fr, transformers, license:apache-2.0


language:


- fr


tags:


- tf
- pytorch
- gpt2
- text-generation
license: apache-2.0
thumbnail: https://raw.githubusercontent.com/AntoineSimoulin/gpt-fr/main/imgs/logo.png


<img src="https://raw.githubusercontent.com/AntoineSimoulin/gpt-fr/main/imgs/logo.png" width="200">


## Model description


**GPT-fr** 🇫🇷 is a GPT model for French developped by [Quantmetry](https://www.quantmetry.com/) and the [Laboratoire de Linguistique Formelle (LLF)](http://www.llf.cnrs.fr/en). We train the model on a very large and heterogeneous French corpus. We release the weights for the following configurations:


| Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters |
| :------:       |   :---: | :---: | :---: | :---: |
| `gpt-fr-cased-small` | 12    | 12    | 768   | 124 M |
| `gpt-fr-cased-base` | 24    | 14    | 1,792   | 1,017 B |


## Intended uses & limitations


The model can be leveraged for language generation tasks. Besides, many tasks may be formatted such that the output is directly generated in natural language. Such configuration may be used for tasks such as automatic summary or question answering. We do hope our model might be used for both academic and industrial applications.


#### How to use


The model might be used through the astonishing 🤗 `Transformers` librairie:


```python
from transformers import GPT2Tokenizer, GPT2LMHeadModel

# Load pretrained model and tokenizer
model = GPT2LMHeadModel.from_pretrained("asi/gpt-fr-cased-small")
tokenizer = GPT2Tokenizer.from_pretrained("asi/gpt-fr-cased-small")

# Generate a sample of text
model.eval()
input_sentence = "Longtemps je me suis couché de bonne heure."
input_ids = tokenizer.encode(input_sentence, return_tensors='pt')

beam_outputs = model.generate(
    input_ids,
    max_length=100,
    do_sample=True,
    top_k=50,
    top_p=0.95,
    num_return_sequences=1
)

print("Output:\n" + 100 * '-')
print(tokenizer.decode(beam_outputs[0], skip_special_tokens=True))
```


#### Limitations and bias


Large language models tend to replicate the biases found in pre-training datasets, such as gender discrimination or offensive content generation.


To limit exposition to too much explicit material, we carefully choose the sources beforehand. This process — detailed in our paper — aims to limit offensive content generation from the model without performing manual and arbitrary filtering.


However, some societal biases, contained in the data, might be reflected by the model. For example on gender equality, we generated the following sentence sequence "Ma femme/Mon mari vient d'obtenir un nouveau poste. A partir de demain elle/il sera \_\_\_\_\_\_\_" and observed the model generated distinct positions given the subject gender. We used top-k random sampling strategy with k=50 and stopped at the first punctuation element.
The positions generated for the wife is '*femme de ménage de la maison*' while the position for the husband is '*à la tête de la police*'. We do appreciate your feedback to better qualitatively and quantitatively assess such effects.


## Training data


We created a dedicated corpus to train our generative model. Indeed the model uses a fixed-length context size of 1,024 and require long documents to be trained.  We aggregated existing corpora: [Wikipedia](https://dumps.wikimedia.org/frwiki/), [OpenSubtitle](http://opus.nlpl.eu/download.php?f=OpenSubtitles/v2016/mono/) ([Tiedemann, 2012](#tiedemann-2012)), [Gutenberg](http://www.gutenberg.org). Corpora are filtered and separated into sentences. Successive sentences are then concatenated within the limit of 1,024 tokens per document.


## Training procedure


We pre-trained the model on a TPU v2-8 using the amazing [Google Colab](https://colab.research.google.com) inter-server.


## Eval results


We packaged **GPT-fr** with a dedicated language model evaluation benchmark.
In line with the [WikiText](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/) benchmark in English, we collected over 70 million tokens from the set of verified [good](https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Articles_de_qualit%C3%A9) and [featured](https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Bons_articles) articles on French Wikipedia. The model reaches a zero-shot perplexity of **109.2** on the test set.


### BibTeX entry and citation info


Along with the model hosted by HuggingFace transformers library, we maintain a [git repository](https://github.com/AntoineSimoulin/gpt-fr).
If you use **GPT-fr** for your scientific publications or your industrial applications, please cite the following paper:


```bibtex
@inproceedings{simoulin:hal-03265900,
  TITLE = {{Un mod{\`e}le Transformer G{\'e}n{\'e}ratif Pr{\'e}-entrain{\'e} pour le \_\_\_\_\_\_ fran{\c c}ais}},
  AUTHOR = {Simoulin, Antoine and Crabb{\'e}, Benoit},
  URL = {https://hal.archives-ouvertes.fr/hal-03265900},
  BOOKTITLE = {{Traitement Automatique des Langues Naturelles}},
  ADDRESS = {Lille, France},
  EDITOR = {Denis, Pascal and Grabar, Natalia and Fraisse, Amel and Cardon, R{\'e}mi and Jacquemin, Bernard and Kergosien, Eric and Balvet, Antonio},
  PUBLISHER = {{ATALA}},
  PAGES = {246-255},
  YEAR = {2021},
  KEYWORDS = {fran{\c c}ais. ; GPT ; G{\'e}n{\'e}ratif ; Transformer ; Pr{\'e}-entra{\^i}n{\'e}},
  PDF = {https://hal.archives-ouvertes.fr/hal-03265900/file/7.pdf},
  HAL_ID = {hal-03265900},
  HAL_VERSION = {v1},
}
```


### References


>
Jörg Tiedemann: Parallel Data, Tools and Interfaces in OPUS. LREC 2012: 2214-2218


model name: gpt-fr-cased-small


model author: asi


task: text-generation


tags: pytorch, tf, jax, gpt2, fr, transformers, text-generation, license:apache-2.0


tags:


- conversational


# DialoGPT-small model trained on dialogue from Rick and Morty


### [Chat to me on Chai!](https://chai.ml/chat/share/_bot_de374c84-9598-4848-996b-736d0cc02f6b)


Make your own Rick bot [here](https://colab.research.google.com/drive/1o5LxBspm-C28HQvXN-PRQavapDbm5WjG?usp=sharing)


model name: RickBot


model author: rsedlr


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


# RickBot built for [Chai](https://chai.ml/)


Make your own [here](https://colab.research.google.com/drive/1o5LxBspm-C28HQvXN-PRQavapDbm5WjG?usp=sharing)


model name: RickBotExample


language:


- ar
thumbnail: "wav2vec2-large-xls-r-300m-arabic fine-tuned for Modern Standard Arabic"
tags:
- automatic-speech-recognition
- robust-speech-event
license: "apache-2.0"
datasets:
- Mozilla/common_voice_7_0
metrics:
- WER


model name: wav2vec2-large-xls-r-300m-arabic


model author: phantomcoder1996


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, ar, dataset:Mozilla/common_voice_7_0, transformers, robust-speech-event, license:apache-2.0


- ar
thumbnail: "wav2vec2-large-xls-r fine tuned on common voice data for Modern Standard Arabic"
tags:
- robust-speech-event
- automatic-speech-recognition
license: "apache-2.0"
datasets:
- mozilla-foundation/common_voice_7_0
metrics:
- WER


model name: wav2vec2-large-xls-r-300m-arabic-colab


tags: pytorch, wav2vec2, automatic-speech-recognition, ar, dataset:mozilla-foundation/common_voice_7_0, transformers, robust-speech-event, license:apache-2.0


license: apache-2.0
tags:


- generated_from_keras_callback
model-index:
- name: market_positivity_model
results: []


<!-- This model card has been generated automatically according to the information Keras had access to. You should
probably proofread and complete it, then remove this comment. -->


# market_positivity_model


This model is a fine-tuned version of [hfl/chinese-roberta-wwm-ext](https://huggingface.co/hfl/chinese-roberta-wwm-ext) on an unknown dataset.
It achieves the following results on the evaluation set:


- Train Loss: 0.3171
- Train Sparse Categorical Accuracy: 0.8870
- Validation Loss: 0.3344
- Validation Sparse Categorical Accuracy: 0.8768
- Epoch: 2


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- optimizer: {'name': 'Adam', 'learning_rate': 5e-05, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-07, 'amsgrad': False}
- training_precision: float32


### Training results


| Train Loss | Train Sparse Categorical Accuracy | Validation Loss | Validation Sparse Categorical Accuracy | Epoch |
|:----------:|:---------------------------------:|:---------------:|:--------------------------------------:|:-----:|
| 0.4282     | 0.8574                            | 0.3700          | 0.8720                                 | 0     |
| 0.3583     | 0.8694                            | 0.3345          | 0.8672                                 | 1     |
| 0.3171     | 0.8870                            | 0.3344          | 0.8768                                 | 2     |


### Framework versions


- Transformers 4.16.0
- TensorFlow 2.7.0
- Datasets 1.18.1
- Tokenizers 0.11.0


model name: market_positivity_model


model author: leetdavid


task: text-classification


tags: tf, bert, text-classification, transformers, license:apache-2.0, generated_from_keras_callback, model-index, infinity_compatible


license: apache-2.0
tags:


- generated_from_keras_callback
model-index:
- name: importance_model
results: []


# importance_model


- Train Loss: 0.5902
- Train Sparse Categorical Accuracy: 0.7910
- Validation Loss: 0.8534
- Validation Sparse Categorical Accuracy: 0.7372
- Epoch: 2


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


| Train Loss | Train Sparse Categorical Accuracy | Validation Loss | Validation Sparse Categorical Accuracy | Epoch |
|:----------:|:---------------------------------:|:---------------:|:--------------------------------------:|:-----:|
| 0.9209     | 0.6502                            | 0.8464          | 0.7055                                 | 0     |
| 0.7112     | 0.7633                            | 0.8572          | 0.7332                                 | 1     |
| 0.5902     | 0.7910                            | 0.8534          | 0.7372                                 | 2     |


### Framework versions


model name: importance_model


model author: leetdavid


task: text-classification


tags:


- flair
- token-classification
- sequence-tagger-model
language: en
datasets:
- conll2000
widget:
- text: "The happy man has been eating at the diner"


## English Chunking in Flair (default model)


This is the standard phrase chunking model for English that ships with [Flair](https://github.com/flairNLP/flair/).


F1-Score: **96,48** (CoNLL-2000)


Predicts 4 tags:


| **tag**                        | **meaning** |
|---------------------------------|-----------|
| ADJP     | adjectival |
| ADVP     | adverbial |
| CONJP     | conjunction |
| INTJ     | interjection |
| LST     | list marker |
| NP     | noun phrase |
| PP     | prepositional |
| PRT     | particle |
| SBAR      | subordinate clause |
| VP      | verb phrase |


Based on [Flair embeddings](https://www.aclweb.org/anthology/C18-1139/) and LSTM-CRF.


### Demo: How to use in Flair


Requires: **[Flair](https://github.com/flairNLP/flair/)** (`pip install flair`)


```python
from flair.data import Sentence
from flair.models import SequenceTagger

# load tagger
tagger = SequenceTagger.load("flair/chunk-english")

# make example sentence
sentence = Sentence("The happy man has been eating at the diner")

# predict NER tags
tagger.predict(sentence)

# print sentence
print(sentence)

# print predicted NER spans
print('The following NER tags are found:')
# iterate over entities and print
for entity in sentence.get_spans('np'):
    print(entity)

```


This yields the following output:


```
Span [1,2,3]: "The happy man"   [− Labels: NP (0.9958)]
Span [4,5,6]: "has been eating"   [− Labels: VP (0.8759)]
Span [7]: "at"   [− Labels: PP (1.0)]
Span [8,9]: "the diner"   [− Labels: NP (0.9991)]

```


So, the spans "*The happy man*" and "*the diner*" are labeled as **noun phrases** (NP) and "*has been eating*" is labeled as a **verb phrase** (VP) in the sentence "*The happy man has been eating at the diner*".


### Training: Script to train this model


The following Flair script was used to train this model:


```python
from flair.data import Corpus
from flair.datasets import CONLL_2000
from flair.embeddings import WordEmbeddings, StackedEmbeddings, FlairEmbeddings

# 1. get the corpus
corpus: Corpus = CONLL_2000()

# 2. what tag do we want to predict?
tag_type = 'np'

# 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)

# 4. initialize each embedding we use
embedding_types = [

    # contextual string embeddings, forward
    FlairEmbeddings('news-forward'),

    # contextual string embeddings, backward
    FlairEmbeddings('news-backward'),
]

# embedding stack consists of Flair and GloVe embeddings
embeddings = StackedEmbeddings(embeddings=embedding_types)

# 5. initialize sequence tagger
from flair.models import SequenceTagger

tagger = SequenceTagger(hidden_size=256,
                        embeddings=embeddings,
                        tag_dictionary=tag_dictionary,
                        tag_type=tag_type)

# 6. initialize trainer
from flair.trainers import ModelTrainer

trainer = ModelTrainer(tagger, corpus)

# 7. run training
trainer.train('resources/taggers/chunk-english',
              train_with_dev=True,
              max_epochs=150)
```


### Cite


Please cite the following paper when using this model.


```
@inproceedings{akbik2018coling,
  title={Contextual String Embeddings for Sequence Labeling},
  author={Akbik, Alan and Blythe, Duncan and Vollgraf, Roland},
  booktitle = {{COLING} 2018, 27th International Conference on Computational Linguistics},
  pages     = {1638--1649},
  year      = {2018}
}
```


### Issues?


The Flair issue tracker is available [here](https://github.com/flairNLP/flair/issues/).


model name: chunk-english


model author: flair


task: token-classification


tags: pytorch, en, dataset:conll2000, flair, token-classification, sequence-tagger-model


tags:


- flair
- token-classification
- sequence-tagger-model
language:
- en
- de
- nl
- es
- multilingual
datasets:
- conll2003
widget:
- text: "George Washington ging nach Washington"


## 4-Language NER in Flair (English, German, Dutch and Spanish)


This is the standard 4-class NER model for 4 CoNLL-03 languages that ships with [Flair](https://github.com/flairNLP/flair/). Also kind of works for related languages like French.


F1-Score: **92,16** (CoNLL-03 English), **87,33** (CoNLL-03 German revised), **88,96** (CoNLL-03 Dutch), **86,65** (CoNLL-03 Spanish)


Predicts 4 tags:


| **tag**                        | **meaning** |
|---------------------------------|-----------|
| PER         | person name |
| LOC         | location name |
| ORG         | organization name |
| MISC         | other name |


Based on [Flair embeddings](https://www.aclweb.org/anthology/C18-1139/) and LSTM-CRF.


### Demo: How to use in Flair


Requires: **[Flair](https://github.com/flairNLP/flair/)** (`pip install flair`)


```python
from flair.data import Sentence
from flair.models import SequenceTagger

# load tagger
tagger = SequenceTagger.load("flair/ner-multi")

# make example sentence in any of the four languages
sentence = Sentence("George Washington ging nach Washington")

# predict NER tags
tagger.predict(sentence)

# print sentence
print(sentence)

# print predicted NER spans
print('The following NER tags are found:')
# iterate over entities and print
for entity in sentence.get_spans('ner'):
    print(entity)

```


This yields the following output:


```
Span [1,2]: "George Washington"   [− Labels: PER (0.9977)]
Span [5]: "Washington"   [− Labels: LOC (0.9895)]
```


So, the entities "*George Washington*" (labeled as a **person**) and "*Washington*" (labeled as a **location**) are found in the sentence "*George Washington ging nach Washington*".


### Training: Script to train this model


The following Flair script was used to train this model:


```python
from flair.data import Corpus
from flair.datasets import CONLL_03, CONLL_03_GERMAN, CONLL_03_DUTCH, CONLL_03_SPANISH
from flair.embeddings import WordEmbeddings, StackedEmbeddings, FlairEmbeddings

# 1. get the multi-language corpus
corpus: Corpus = MultiCorpus([
    CONLL_03(),         # English corpus
    CONLL_03_GERMAN(),  # German corpus
    CONLL_03_DUTCH(),   # Dutch corpus
    CONLL_03_SPANISH(), # Spanish corpus
    ])

# 2. what tag do we want to predict?
tag_type = 'ner'

# 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)

# 4. initialize each embedding we use
embedding_types = [

    # GloVe embeddings
    WordEmbeddings('glove'),

    # FastText embeddings
    WordEmbeddings('de'),

    # contextual string embeddings, forward
    FlairEmbeddings('multi-forward'),

    # contextual string embeddings, backward
    FlairEmbeddings('multi-backward'),
]

# embedding stack consists of Flair and GloVe embeddings
embeddings = StackedEmbeddings(embeddings=embedding_types)

# 5. initialize sequence tagger
from flair.models import SequenceTagger

tagger = SequenceTagger(hidden_size=256,
                        embeddings=embeddings,
                        tag_dictionary=tag_dictionary,
                        tag_type=tag_type)

# 6. initialize trainer
from flair.trainers import ModelTrainer

trainer = ModelTrainer(tagger, corpus)

# 7. run training
trainer.train('resources/taggers/ner-multi',
              train_with_dev=True,
              max_epochs=150)
```


### Cite


Please cite the following paper when using this model.


```
@misc{akbik2019multilingual,
  title={Multilingual sequence labeling with one model},
  author={Akbik, Alan and Bergmann, Tanja and Vollgraf, Roland}
  booktitle = {{NLDL} 2019, Northern Lights Deep Learning Workshop},
  year      = {2019}
}
```


```
@inproceedings{akbik2018coling,
  title={Contextual String Embeddings for Sequence Labeling},
  author={Akbik, Alan and Blythe, Duncan and Vollgraf, Roland},
  booktitle = {{COLING} 2018, 27th International Conference on Computational Linguistics},
  pages     = {1638--1649},
  year      = {2018}
}
```


model name: ner-multi


model author: flair


task: token-classification


tags: pytorch, en, de, nl, es, multilingual, dataset:conll2003, flair, token-classification, sequence-tagger-model


tags:


- flair
- token-classification
- sequence-tagger-model
language: en
datasets:
- ontonotes
widget:
- text: "On September 1st George Washington won 1 dollar."


## English NER in Flair (Ontonotes default model)


This is the 18-class NER model for English that ships with [Flair](https://github.com/flairNLP/flair/).


F1-Score: **89.27** (Ontonotes)


Predicts 18 tags:


| **tag**                        | **meaning** |
|---------------------------------|-----------|
| CARDINAL    | cardinal value |
| DATE         | date value |
| EVENT         | event name |
| FAC         | building name |
| GPE         | geo-political entity |
| LANGUAGE         | language name |
| LAW         | law name |
| LOC         | location name |
| MONEY         | money name |
| NORP         | affiliation |
| ORDINAL         | ordinal value |
| ORG         | organization name |
| PERCENT         | percent value |
| PERSON         | person name |
| PRODUCT         | product name |
| QUANTITY         | quantity value |
| TIME         | time value |
| WORK_OF_ART         | name of work of art |


### Demo: How to use in Flair


```python
from flair.data import Sentence
from flair.models import SequenceTagger

# load tagger
tagger = SequenceTagger.load("flair/ner-english-ontonotes")

# make example sentence
sentence = Sentence("On September 1st George Washington won 1 dollar.")

# predict NER tags
tagger.predict(sentence)

# print sentence
print(sentence)

# print predicted NER spans
print('The following NER tags are found:')
# iterate over entities and print
for entity in sentence.get_spans('ner'):
    print(entity)

```


This yields the following output:


```
Span [2,3]: "September 1st"   [− Labels: DATE (0.8824)]
Span [4,5]: "George Washington"   [− Labels: PERSON (0.9604)]
Span [7,8]: "1 dollar"   [− Labels: MONEY (0.9837)]
```


So, the entities "*September 1st*" (labeled as a **date**), "*George Washington*" (labeled as a **person**) and "*1 dollar*" (labeled as a **money**) are found in the sentence "*On September 1st George Washington won 1 dollar*".


```python
from flair.data import Corpus
from flair.datasets import ColumnCorpus
from flair.embeddings import WordEmbeddings, StackedEmbeddings, FlairEmbeddings

# 1. load the corpus (Ontonotes does not ship with Flair, you need to download and reformat into a column format yourself)
corpus: Corpus = ColumnCorpus(
                "resources/tasks/onto-ner",
                column_format={0: "text", 1: "pos", 2: "upos", 3: "ner"},
                tag_to_bioes="ner",
            )

# 2. what tag do we want to predict?
tag_type = 'ner'

# 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)

# 4. initialize each embedding we use
embedding_types = [

    # GloVe embeddings
    WordEmbeddings('en-crawl'),

    # contextual string embeddings, forward
    FlairEmbeddings('news-forward'),

    # contextual string embeddings, backward
    FlairEmbeddings('news-backward'),
]

# embedding stack consists of Flair and GloVe embeddings
embeddings = StackedEmbeddings(embeddings=embedding_types)

# 5. initialize sequence tagger
from flair.models import SequenceTagger

tagger = SequenceTagger(hidden_size=256,
                        embeddings=embeddings,
                        tag_dictionary=tag_dictionary,
                        tag_type=tag_type)

# 6. initialize trainer
from flair.trainers import ModelTrainer

trainer = ModelTrainer(tagger, corpus)

# 7. run training
trainer.train('resources/taggers/ner-english-ontonotes',
              train_with_dev=True,
              max_epochs=150)
```


### Cite


### Issues?


The Flair issue tracker is available [here](https://github.com/flairNLP/flair/issues/).


model name: ner-english-ontonotes


model author: flair


task: token-classification


tags: pytorch, en, dataset:ontonotes, flair, token-classification, sequence-tagger-model


tags:


- flair
- token-classification
- sequence-tagger-model
language: en de nl es
datasets:
- conll2003
widget:
- text: "George Washington ging nach Washington"


## 4-Language NER in Flair (English, German, Dutch and Spanish)


This is the fast 4-class NER model for 4 CoNLL-03 languages that ships with [Flair](https://github.com/flairNLP/flair/). Also kind of works for related languages like French.


F1-Score: **91,51** (CoNLL-03 English), **85,72** (CoNLL-03 German revised), **86,22** (CoNLL-03 Dutch), **85,78** (CoNLL-03 Spanish)


Predicts 4 tags:


| **tag**                        | **meaning** |
|---------------------------------|-----------|
| PER         | person name |
| LOC         | location name |
| ORG         | organization name |
| MISC         | other name |


Based on [Flair embeddings](https://www.aclweb.org/anthology/C18-1139/) and LSTM-CRF.


### Demo: How to use in Flair


Requires: **[Flair](https://github.com/flairNLP/flair/)** (`pip install flair`)


```python
from flair.data import Sentence
from flair.models import SequenceTagger

# load tagger
tagger = SequenceTagger.load("flair/ner-multi-fast")

# make example sentence in any of the four languages
sentence = Sentence("George Washington ging nach Washington")

# predict NER tags
tagger.predict(sentence)

# print sentence
print(sentence)

# print predicted NER spans
print('The following NER tags are found:')
# iterate over entities and print
for entity in sentence.get_spans('ner'):
    print(entity)

```


This yields the following output:


```
Span [1,2]: "George Washington"   [− Labels: PER (0.9977)]
Span [5]: "Washington"   [− Labels: LOC (0.9895)]
```


So, the entities "*George Washington*" (labeled as a **person**) and "*Washington*" (labeled as a **location**) are found in the sentence "*George Washington ging nach Washington*".


### Training: Script to train this model


The following Flair script was used to train this model:


```python
from flair.data import Corpus
from flair.datasets import CONLL_03, CONLL_03_GERMAN, CONLL_03_DUTCH, CONLL_03_SPANISH
from flair.embeddings import WordEmbeddings, StackedEmbeddings, FlairEmbeddings

# 1. get the multi-language corpus
corpus: Corpus = MultiCorpus([
    CONLL_03(),         # English corpus
    CONLL_03_GERMAN(),  # German corpus
    CONLL_03_DUTCH(),   # Dutch corpus
    CONLL_03_SPANISH(), # Spanish corpus
    ])

# 2. what tag do we want to predict?
tag_type = 'ner'

# 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)

# 4. initialize each embedding we use
embedding_types = [

    # GloVe embeddings
    WordEmbeddings('glove'),

    # FastText embeddings
    WordEmbeddings('de'),

    # contextual string embeddings, forward
    FlairEmbeddings('multi-forward-fast'),

    # contextual string embeddings, backward
    FlairEmbeddings('multi-backward-fast'),
]

# embedding stack consists of Flair and GloVe embeddings
embeddings = StackedEmbeddings(embeddings=embedding_types)

# 5. initialize sequence tagger
from flair.models import SequenceTagger

tagger = SequenceTagger(hidden_size=256,
                        embeddings=embeddings,
                        tag_dictionary=tag_dictionary,
                        tag_type=tag_type)

# 6. initialize trainer
from flair.trainers import ModelTrainer

trainer = ModelTrainer(tagger, corpus)

# 7. run training
trainer.train('resources/taggers/ner-multi-fast',
              train_with_dev=True,
              max_epochs=150)
```


### Cite


Please cite the following papers when using this model.


```
@misc{akbik2019multilingual,
  title={Multilingual sequence labeling with one model},
  author={Akbik, Alan and Bergmann, Tanja and Vollgraf, Roland}
  booktitle = {{NLDL} 2019, Northern Lights Deep Learning Workshop},
  year      = {2019}
}
```


```
@inproceedings{akbik2018coling,
  title={Contextual String Embeddings for Sequence Labeling},
  author={Akbik, Alan and Blythe, Duncan and Vollgraf, Roland},
  booktitle = {{COLING} 2018, 27th International Conference on Computational Linguistics},
  pages     = {1638--1649},
  year      = {2018}
}
```


model name: ner-multi-fast


model author: flair


task: token-classification


tags: pytorch, en de nl es, dataset:conll2003, flair, token-classification, sequence-tagger-model


tags:


- flair
- token-classification
- sequence-tagger-model
language: en
datasets:
- ontonotes
widget:
- text: "George returned to Berlin to return his hat."


## English Verb Disambiguation in Flair (default model)


This is the standard verb disambiguation model for English that ships with [Flair](https://github.com/flairNLP/flair/).


F1-Score: **89,34** (Ontonotes) - predicts [Proposition Bank verb frames](http://verbs.colorado.edu/propbank/framesets-english-aliases/).


### Demo: How to use in Flair


```python
from flair.data import Sentence
from flair.models import SequenceTagger

# load tagger
tagger = SequenceTagger.load("flair/frame-english")

# make example sentence
sentence = Sentence("George returned to Berlin to return his hat.")

# predict NER tags
tagger.predict(sentence)

# print sentence
print(sentence)

# print predicted NER spans
print('The following frame tags are found:')
# iterate over entities and print
for entity in sentence.get_spans('frame'):
    print(entity)

```


This yields the following output:


```
Span [2]: "returned"   [− Labels: return.01 (0.9951)]
Span [6]: "return"   [− Labels: return.02 (0.6361)]
```


So, the word "*returned*" is labeled as **return.01** (as in *go back somewhere*) while "*return*" is labeled as **return.02** (as in *give back something*) in the sentence "*George returned to Berlin to return his hat*".


```python
from flair.data import Corpus
from flair.datasets import ColumnCorpus
from flair.embeddings import WordEmbeddings, StackedEmbeddings, FlairEmbeddings

# 1. load the corpus (Ontonotes does not ship with Flair, you need to download and reformat into a column format yourself)
corpus = ColumnCorpus(
    "resources/tasks/srl", column_format={1: "text", 11: "frame"}
)

# 2. what tag do we want to predict?
tag_type = 'frame'

# 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)

# 4. initialize each embedding we use
embedding_types = [

    BytePairEmbeddings("en"),

    FlairEmbeddings("news-forward"),

    FlairEmbeddings("news-backward"),
]

# embedding stack consists of Flair and GloVe embeddings
embeddings = StackedEmbeddings(embeddings=embedding_types)

# 5. initialize sequence tagger
from flair.models import SequenceTagger

tagger = SequenceTagger(hidden_size=256,
                        embeddings=embeddings,
                        tag_dictionary=tag_dictionary,
                        tag_type=tag_type)

# 6. initialize trainer
from flair.trainers import ModelTrainer

trainer = ModelTrainer(tagger, corpus)

# 7. run training
trainer.train('resources/taggers/frame-english',
              train_with_dev=True,
              max_epochs=150)
```


### Cite


Please cite the following paper when using this model.


```
@inproceedings{akbik2019flair,
  title={FLAIR: An easy-to-use framework for state-of-the-art NLP},
  author={Akbik, Alan and Bergmann, Tanja and Blythe, Duncan and Rasul, Kashif and Schweter, Stefan and Vollgraf, Roland},
  booktitle={{NAACL} 2019, 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)},
  pages={54--59},
  year={2019}
}

```


### Issues?


The Flair issue tracker is available [here](https://github.com/flairNLP/flair/issues/).


model name: frame-english


model author: flair


task: token-classification


tags: pytorch, en, dataset:ontonotes, flair, token-classification, sequence-tagger-model


tags:


- flair
- token-classification
- sequence-tagger-model
language: da
datasets:
- DaNE
widget:
- text: "Jens Peter Hansen kommer fra Danmark"


# Danish NER in Flair (default model)


This is the standard 4-class NER model for Danish that ships with [Flair](https://github.com/flairNLP/flair/).


F1-Score: **81.78** (DaNER)


Predicts 4 tags:


| **tag**                        | **meaning** |
|---------------------------------|-----------|
| PER         | person name |
| LOC         | location name |
| ORG         | organization name |
| MISC         | other name |


Based on Transformer embeddings and LSTM-CRF.


# Demo: How to use in Flair


Requires: **[Flair](https://github.com/flairNLP/flair/)** (`pip install flair`)


```python
from flair.data import Sentence
from flair.models import SequenceTagger

# load tagger
tagger = SequenceTagger.load("flair/ner-danish")

# make example sentence
sentence = Sentence("Jens Peter Hansen kommer fra Danmark")

# predict NER tags
tagger.predict(sentence)

# print sentence
print(sentence)

# print predicted NER spans
print('The following NER tags are found:')
# iterate over entities and print
for entity in sentence.get_spans('ner'):
    print(entity)

```


This yields the following output:


```
Span [1,2,3]: "Jens Peter Hansen"   [− Labels: PER (0.9961)]
Span [6]: "Danmark"   [− Labels: LOC (0.9816)]
```


So, the entities "*Jens Peter Hansen*" (labeled as a **person**) and "*Danmark*" (labeled as a **location**) are found in the sentence "*Jens Peter Hansen kommer fra Danmark*".


### Training: Script to train this model


The model was trained by the [DaNLP project](https://github.com/alexandrainst/danlp) using the [DaNE corpus](https://github.com/alexandrainst/danlp/blob/master/docs/docs/datasets.md#danish-dependency-treebank-dane-dane). Check their repo for more information.


The following Flair script may be used to train such a model:


```python
from flair.data import Corpus
from flair.datasets import DANE
from flair.embeddings import WordEmbeddings, StackedEmbeddings, FlairEmbeddings

# 1. get the corpus
corpus: Corpus = DANE()

# 2. what tag do we want to predict?
tag_type = 'ner'

# 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)

# 4. initialize each embedding we use
embedding_types = [

    # GloVe embeddings
    WordEmbeddings('da'),

    # contextual string embeddings, forward
    FlairEmbeddings('da-forward'),

    # contextual string embeddings, backward
    FlairEmbeddings('da-backward'),
]

# embedding stack consists of Flair and GloVe embeddings
embeddings = StackedEmbeddings(embeddings=embedding_types)

# 5. initialize sequence tagger
from flair.models import SequenceTagger

tagger = SequenceTagger(hidden_size=256,
                        embeddings=embeddings,
                        tag_dictionary=tag_dictionary,
                        tag_type=tag_type)

# 6. initialize trainer
from flair.trainers import ModelTrainer

trainer = ModelTrainer(tagger, corpus)

# 7. run training
trainer.train('resources/taggers/ner-danish',
              train_with_dev=True,
              max_epochs=150)
```


### Cite


Please cite the following papers when using this model.


```
@inproceedings{akbik-etal-2019-flair,
    title = "{FLAIR}: An Easy-to-Use Framework for State-of-the-Art {NLP}",
    author = "Akbik, Alan  and
      Bergmann, Tanja  and
      Blythe, Duncan  and
      Rasul, Kashif  and
      Schweter, Stefan  and
      Vollgraf, Roland",
    booktitle = "Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics (Demonstrations)",
    year = "2019",
    url = "https://www.aclweb.org/anthology/N19-4010",
    pages = "54--59",
}
```


And check the [DaNLP project](https://github.com/alexandrainst/danlp) for more information.


### Issues?


The Flair issue tracker is available [here](https://github.com/flairNLP/flair/issues/).


model name: ner-danish


model author: flair


task: token-classification


tags: pytorch, da, dataset:DaNE, flair, token-classification, sequence-tagger-model


tags:


- flair
- token-classification
- sequence-tagger-model
language: en
datasets:
- ontonotes
widget:
- text: "I love Berlin."


## English Universal Part-of-Speech Tagging in Flair (default model)


This is the standard universal part-of-speech tagging model for English that ships with [Flair](https://github.com/flairNLP/flair/).


F1-Score: **98,6** (Ontonotes)


Predicts universal POS tags:


| **tag**                        | **meaning** |
|---------------------------------|-----------|
|ADJ |  adjective |
|   ADP |  adposition |
|   ADV |  adverb |
|   AUX |  auxiliary |
|   CCONJ |  coordinating conjunction |
|   DET |  determiner |
|   INTJ |  interjection |
|   NOUN |  noun |
|   NUM |  numeral |
|   PART |  particle |
|   PRON |  pronoun |
|   PROPN |  proper noun |
|   PUNCT |  punctuation |
|   SCONJ |  subordinating conjunction |
|   SYM |  symbol |
|   VERB |  verb |
|   X |  other |


Based on [Flair embeddings](https://www.aclweb.org/anthology/C18-1139/) and LSTM-CRF.


### Demo: How to use in Flair


Requires: **[Flair](https://github.com/flairNLP/flair/)** (`pip install flair`)


```python
from flair.data import Sentence
from flair.models import SequenceTagger

# load tagger
tagger = SequenceTagger.load("flair/upos-english")

# make example sentence
sentence = Sentence("I love Berlin.")

# predict NER tags
tagger.predict(sentence)

# print sentence
print(sentence)

# print predicted NER spans
print('The following NER tags are found:')
# iterate over entities and print
for entity in sentence.get_spans('pos'):
    print(entity)

```


This yields the following output:


```
Span [1]: "I"   [− Labels: PRON (0.9996)]
Span [2]: "love"   [− Labels: VERB (1.0)]
Span [3]: "Berlin"   [− Labels: PROPN (0.9986)]
Span [4]: "."   [− Labels: PUNCT (1.0)]
```


So, the word "*I*" is labeled as a **pronoun** (PRON),  "*love*" is labeled as a **verb** (VERB) and "*Berlin*" is labeled as a **proper noun** (PROPN) in the sentence "*I love Berlin*".


### Training: Script to train this model


The following Flair script was used to train this model:


```python
from flair.data import Corpus
from flair.datasets import ColumnCorpus
from flair.embeddings import WordEmbeddings, StackedEmbeddings, FlairEmbeddings

# 1. load the corpus (Ontonotes does not ship with Flair, you need to download and reformat into a column format yourself)
corpus: Corpus = ColumnCorpus(
                "resources/tasks/onto-ner",
                column_format={0: "text", 1: "pos", 2: "upos", 3: "ner"},
                tag_to_bioes="ner",
            )

# 2. what tag do we want to predict?
tag_type = 'upos'

# 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)

# 4. initialize each embedding we use
embedding_types = [

    # contextual string embeddings, forward
    FlairEmbeddings('news-forward'),

    # contextual string embeddings, backward
    FlairEmbeddings('news-backward'),
]

# embedding stack consists of Flair and GloVe embeddings
embeddings = StackedEmbeddings(embeddings=embedding_types)

# 5. initialize sequence tagger
from flair.models import SequenceTagger

tagger = SequenceTagger(hidden_size=256,
                        embeddings=embeddings,
                        tag_dictionary=tag_dictionary,
                        tag_type=tag_type)

# 6. initialize trainer
from flair.trainers import ModelTrainer

trainer = ModelTrainer(tagger, corpus)

# 7. run training
trainer.train('resources/taggers/upos-english',
              train_with_dev=True,
              max_epochs=150)
```


### Cite


Please cite the following paper when using this model.


```
@inproceedings{akbik2018coling,
  title={Contextual String Embeddings for Sequence Labeling},
  author={Akbik, Alan and Blythe, Duncan and Vollgraf, Roland},
  booktitle = {{COLING} 2018, 27th International Conference on Computational Linguistics},
  pages     = {1638--1649},
  year      = {2018}
}
```


### Issues?


model name: upos-english


model author: flair


task: token-classification


tags: pytorch, en, dataset:ontonotes, flair, token-classification, sequence-tagger-model


tags:


- flair
- token-classification
- sequence-tagger-model
language: es
datasets:
- conll2003
widget:
- text: "George Washington fue a Washington"


## Spanish NER in Flair (large model)


This is the large 4-class NER model for Spanish that ships with [Flair](https://github.com/flairNLP/flair/).


F1-Score: **90,54** (CoNLL-03 Spanish)


Predicts 4 tags:


| **tag**                        | **meaning** |
|---------------------------------|-----------|
| PER         | person name |
| LOC         | location name |
| ORG         | organization name |
| MISC         | other name |


Based on document-level XLM-R embeddings and [FLERT](https://arxiv.org/pdf/2011.06993v1.pdf/).


### Demo: How to use in Flair


```python
from flair.data import Sentence
from flair.models import SequenceTagger

# load tagger
tagger = SequenceTagger.load("flair/ner-spanish-large")

# make example sentence
sentence = Sentence("George Washington fue a Washington")

# predict NER tags
tagger.predict(sentence)

# print sentence
print(sentence)

# print predicted NER spans
print('The following NER tags are found:')
# iterate over entities and print
for entity in sentence.get_spans('ner'):
    print(entity)

```


This yields the following output:


```
Span [1,2]: "George Washington"   [− Labels: PER (1.0)]
Span [5]: "Washington"   [− Labels: LOC (1.0)]
```


So, the entities "*George Washington*" (labeled as a **person**) and "*Washington*" (labeled as a **location**) are found in the sentence "*George Washington fue a Washington*".


```python
import torch

# 1. get the corpus
from flair.datasets import CONLL_03_SPANISH

corpus = CONLL_03_SPANISH()

# 2. what tag do we want to predict?
tag_type = 'ner'

# 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)

# 4. initialize fine-tuneable transformer embeddings WITH document context
from flair.embeddings import TransformerWordEmbeddings

embeddings = TransformerWordEmbeddings(
    model='xlm-roberta-large',
    layers="-1",
    subtoken_pooling="first",
    fine_tune=True,
    use_context=True,
)

# 5. initialize bare-bones sequence tagger (no CRF, no RNN, no reprojection)
from flair.models import SequenceTagger

tagger = SequenceTagger(
    hidden_size=256,
    embeddings=embeddings,
    tag_dictionary=tag_dictionary,
    tag_type='ner',
    use_crf=False,
    use_rnn=False,
    reproject_embeddings=False,
)

# 6. initialize trainer with AdamW optimizer
from flair.trainers import ModelTrainer

trainer = ModelTrainer(tagger, corpus, optimizer=torch.optim.AdamW)

# 7. run training with XLM parameters (20 epochs, small LR)
from torch.optim.lr_scheduler import OneCycleLR

trainer.train('resources/taggers/ner-spanish-large',
              learning_rate=5.0e-6,
              mini_batch_size=4,
              mini_batch_chunk_size=1,
              max_epochs=20,
              scheduler=OneCycleLR,
              embeddings_storage_mode='none',
              weight_decay=0.,
              )

)
```


### Cite


```
@misc{schweter2020flert,
    title={FLERT: Document-Level Features for Named Entity Recognition},
    author={Stefan Schweter and Alan Akbik},
    year={2020},
    eprint={2011.06993},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```


### Issues?


The Flair issue tracker is available [here](https://github.com/flairNLP/flair/issues/).


model name: ner-spanish-large


model author: flair


task: token-classification


tags: pytorch, es, dataset:conll2003, arxiv:2011.06993, flair, token-classification, sequence-tagger-model


tags:


- flair
- token-classification
- sequence-tagger-model
language: en
datasets:
- ontonotes
widget:
- text: "I love Berlin."


## English Part-of-Speech Tagging in Flair (default model)


This is the standard part-of-speech tagging model for English that ships with [Flair](https://github.com/flairNLP/flair/).


F1-Score: **98,19** (Ontonotes)


Predicts fine-grained POS tags:


| **tag**                        | **meaning** |
|---------------------------------|-----------|
|ADD        | Email |
|AFX        | Affix |
|CC         | Coordinating conjunction  |
|CD         | Cardinal number |
|DT         | Determiner |
|EX         | Existential there |
|FW         | Foreign word |
|HYPH       | Hyphen |
|IN        | Preposition or subordinating conjunction |
|JJ         | Adjective |
|JJR        |Adjective, comparative |
|JJS        | Adjective, superlative |
|LS         | List item marker  |
|MD         | Modal |
|NFP        | Superfluous punctuation |
|NN        | Noun, singular or mass |
|NNP        |Proper noun, singular |
|NNPS       | Proper noun, plural |
|NNS        |Noun, plural |
|PDT        | Predeterminer |
|POS        | Possessive ending |
|PRP        | Personal pronoun |
|PRP$       | Possessive pronoun |
|RB         | Adverb |
|RBR        | Adverb, comparative |
|RBS        | Adverb, superlative |
|RP         | Particle |
|SYM        | Symbol |
|TO         | to |
|UH         | Interjection |
|VB         | Verb, base form |
|VBD       | Verb, past tense |
|VBG        | Verb, gerund or present participle |
|VBN        | Verb, past participle |
|VBP        | Verb, non-3rd person singular present |
|VBZ        | Verb, 3rd person singular present |
|WDT        | Wh-determiner |
|WP        | Wh-pronoun |
|WP$        | Possessive wh-pronoun |
|WRB        | Wh-adverb |
|XX         | Unknown |


Based on [Flair embeddings](https://www.aclweb.org/anthology/C18-1139/) and LSTM-CRF.


### Demo: How to use in Flair


Requires: **[Flair](https://github.com/flairNLP/flair/)** (`pip install flair`)


```python
from flair.data import Sentence
from flair.models import SequenceTagger

# load tagger
tagger = SequenceTagger.load("flair/pos-english")

# make example sentence
sentence = Sentence("I love Berlin.")

# predict NER tags
tagger.predict(sentence)

# print sentence
print(sentence)

# print predicted NER spans
print('The following NER tags are found:')
# iterate over entities and print
for entity in sentence.get_spans('pos'):
    print(entity)

```


This yields the following output:


```
Span [1]: "I"   [− Labels: PRP (1.0)]
Span [2]: "love"   [− Labels: VBP (1.0)]
Span [3]: "Berlin"   [− Labels: NNP (0.9999)]
Span [4]: "."   [− Labels: . (1.0)]

```


So, the word "*I*" is labeled as a **pronoun** (PRP),  "*love*" is labeled as a **verb** (VBP) and "*Berlin*" is labeled as a **proper noun** (NNP) in the sentence "*I love Berlin*".


### Training: Script to train this model


The following Flair script was used to train this model:


```python
from flair.data import Corpus
from flair.datasets import ColumnCorpus
from flair.embeddings import WordEmbeddings, StackedEmbeddings, FlairEmbeddings

# 1. load the corpus (Ontonotes does not ship with Flair, you need to download and reformat into a column format yourself)
corpus: Corpus = ColumnCorpus(
                "resources/tasks/onto-ner",
                column_format={0: "text", 1: "pos", 2: "upos", 3: "ner"},
                tag_to_bioes="ner",
            )

# 2. what tag do we want to predict?
tag_type = 'pos'

# 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)

# 4. initialize each embedding we use
embedding_types = [

    # contextual string embeddings, forward
    FlairEmbeddings('news-forward'),

    # contextual string embeddings, backward
    FlairEmbeddings('news-backward'),
]

# embedding stack consists of Flair and GloVe embeddings
embeddings = StackedEmbeddings(embeddings=embedding_types)

# 5. initialize sequence tagger
from flair.models import SequenceTagger

tagger = SequenceTagger(hidden_size=256,
                        embeddings=embeddings,
                        tag_dictionary=tag_dictionary,
                        tag_type=tag_type)

# 6. initialize trainer
from flair.trainers import ModelTrainer

trainer = ModelTrainer(tagger, corpus)

# 7. run training
trainer.train('resources/taggers/pos-english',
              train_with_dev=True,
              max_epochs=150)
```


### Cite


Please cite the following paper when using this model.


```
@inproceedings{akbik2018coling,
  title={Contextual String Embeddings for Sequence Labeling},
  author={Akbik, Alan and Blythe, Duncan and Vollgraf, Roland},
  booktitle = {{COLING} 2018, 27th International Conference on Computational Linguistics},
  pages     = {1638--1649},
  year      = {2018}
}
```


### Issues?


model name: pos-english


model author: flair


task: token-classification


tags: pytorch, en, dataset:ontonotes, flair, token-classification, sequence-tagger-model


tags:


- flair
- token-classification
- sequence-tagger-model
language: en
datasets:
- conll2003
widget:
- text: "George Washington went to Washington"


## English NER in Flair (default model)


This is the standard 4-class NER model for English that ships with [Flair](https://github.com/flairNLP/flair/).


F1-Score: **93,06** (corrected CoNLL-03)


Predicts 4 tags:


| **tag**                        | **meaning** |
|---------------------------------|-----------|
| PER         | person name |
| LOC         | location name |
| ORG         | organization name |
| MISC         | other name |


### Demo: How to use in Flair


```python
from flair.data import Sentence
from flair.models import SequenceTagger

# load tagger
tagger = SequenceTagger.load("flair/ner-english")

# make example sentence
sentence = Sentence("George Washington went to Washington")

# predict NER tags
tagger.predict(sentence)

# print sentence
print(sentence)

# print predicted NER spans
print('The following NER tags are found:')
# iterate over entities and print
for entity in sentence.get_spans('ner'):
    print(entity)

```


This yields the following output:


```
Span [1,2]: "George Washington"   [− Labels: PER (0.9968)]
Span [5]: "Washington"   [− Labels: LOC (0.9994)]
```


So, the entities "*George Washington*" (labeled as a **person**) and "*Washington*" (labeled as a **location**) are found in the sentence "*George Washington went to Washington*".


```python
from flair.data import Corpus
from flair.datasets import CONLL_03
from flair.embeddings import WordEmbeddings, StackedEmbeddings, FlairEmbeddings

# 1. get the corpus
corpus: Corpus = CONLL_03()

# 2. what tag do we want to predict?
tag_type = 'ner'

# 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)

# 4. initialize each embedding we use
embedding_types = [

    # GloVe embeddings
    WordEmbeddings('glove'),

    # contextual string embeddings, forward
    FlairEmbeddings('news-forward'),

    # contextual string embeddings, backward
    FlairEmbeddings('news-backward'),
]

# embedding stack consists of Flair and GloVe embeddings
embeddings = StackedEmbeddings(embeddings=embedding_types)

# 5. initialize sequence tagger
from flair.models import SequenceTagger

tagger = SequenceTagger(hidden_size=256,
                        embeddings=embeddings,
                        tag_dictionary=tag_dictionary,
                        tag_type=tag_type)

# 6. initialize trainer
from flair.trainers import ModelTrainer

trainer = ModelTrainer(tagger, corpus)

# 7. run training
trainer.train('resources/taggers/ner-english',
              train_with_dev=True,
              max_epochs=150)
```


### Cite


### Issues?


The Flair issue tracker is available [here](https://github.com/flairNLP/flair/issues/).


model name: ner-english


model author: flair


task: token-classification


tags: pytorch, en, dataset:conll2003, flair, token-classification, sequence-tagger-model


tags:


## English NER in Flair (large model)


This is the large 4-class NER model for English that ships with [Flair](https://github.com/flairNLP/flair/).


F1-Score: **94,36** (corrected CoNLL-03)


Predicts 4 tags: