---
language: sv
license: mit
---

## Swedish BERT models for sentiment analysis
 [Recorded Future](https://www.recordedfuture.com/) together with [AI Sweden](https://www.ai.se/en) releases two language models for sentiment analysis in Swedish. The two models are based on the [KB\/bert-base-swedish-cased](https://huggingface.co/KB/bert-base-swedish-cased) model and has been fine-tuned to solve a multi-label sentiment analysis task.

The models have been fine-tuned for the sentiments fear and violence. The models output three floats corresponding to the labels "Negative", "Weak sentiment", and "Strong Sentiment" at the respective indexes.
The models have been trained on Swedish data with a conversational focus, collected from various internet sources and forums.

The models are only trained on Swedish data and only supports inference of Swedish input texts. The models inference metrics for all non-Swedish inputs are not defined, these inputs are considered as out of domain data.

The current models are supported at Transformers version >= 4.3.3 and Torch version 1.8.0, compatibility with older versions are not verified.

### Swedish-Sentiment-Fear

The model can be imported from the transformers library by running

    from transformers import BertForSequenceClassification, BertTokenizerFast

    tokenizer = BertTokenizerFast.from_pretrained("RecordedFuture/Swedish-Sentiment-Fear")
    classifier_fear= BertForSequenceClassification.from_pretrained("RecordedFuture/Swedish-Sentiment-Fear")

When the model and tokenizer are initialized the model can be used for inference.

#### Sentiment definitions
#### The strong sentiment includes but are not limited to
Texts that:

 - Hold an expressive emphasis on fear and/ or anxiety

#### The weak sentiment includes but are not limited to
Texts that:

- Express fear and/ or anxiety in a neutral way

#### Verification metrics

During training, the model had maximized validation metrics at the following classification breakpoint.

| Classification Breakpoint | F-score | Precision | Recall |
|:-------------------------:|:-------:|:---------:|:------:|
|               0.45 |  0.8754 |   0.8618  | 0.8895 |

#### Swedish-Sentiment-Violence
The model be can imported from the transformers library by running

    from transformers import BertForSequenceClassification, BertTokenizerFast

    tokenizer = BertTokenizerFast.from_pretrained("RecordedFuture/Swedish-Sentiment-Violence")
    classifier_violence = BertForSequenceClassification.from_pretrained("RecordedFuture/Swedish-Sentiment-Violence")

When the model and tokenizer are initialized the model can be used for inference.

### Sentiment definitions
#### The strong sentiment includes but are not limited to
Texts that:
 -  Referencing highly violent acts
-   Hold an aggressive tone
#### The weak sentiment includes but are not limited to
Texts that:
-   Include general violent statements that do not fall under the strong sentiment
#### Verification metrics
During training, the model had maximized validation metrics at the following classification breakpoint.

| Classification Breakpoint | F-score | Precision | Recall |
|:-------------------------:|:-------:|:---------:|:------:|
|            0.35           |  0.7677 |   0.7456  |  0.791 |


from transformers import MT5ForConditionalGeneration, AutoTokenizer
model = MT5ForConditionalGeneration.from_pretrained("Parth/mT5-question-generator")
tokenizer = AutoTokenizer.from_pretrained("google/mt5-base")


---
language: fon
datasets:
- fon_dataset
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Fon XLSR Wav2Vec2 Large 53
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: fon
      type: fon_dataset
      args: fon
    metrics:
       - name: Test WER
         type: wer
         value: 14.97%
---
# Wav2Vec2-Large-XLSR-53-Fon

Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on [Fon (or Fongbe)](https://en.wikipedia.org/wiki/Fon_language) using the [Fon Dataset](https://github.com/laleye/pyFongbe/tree/master/data).

When using this model, make sure that your speech input is sampled at 16kHz.

## Usage

The model can be used directly (without a language model) as follows:

```python
import json
import random
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

#Load test_dataset from saved files in folder
from datasets import load_dataset, load_metric

#for test
for root, dirs, files in os.walk(test/):
  test_dataset= load_dataset("json", data_files=[os.path.join(root,i) for i in files],split="train")

#Remove unnecessary chars
chars_to_ignore_regex = '[\\,\\?\\.\\!\\-\\;\\:\\"\\“\\%\\‘\\”]'
def remove_special_characters(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower() + " "
    return batch

test_dataset = test_dataset.map(remove_special_characters)

processor = Wav2Vec2Processor.from_pretrained("chrisjay/wav2vec2-large-xlsr-53-fon")
model = Wav2Vec2ForCTC.from_pretrained("chrisjay/wav2vec2-large-xlsr-53-fon")

#No need for resampling because audio dataset already at 16kHz
#resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the audio files as arrays
def speech_file_to_array_fn(batch):
  speech_array, sampling_rate = torchaudio.load(batch["path"])
  batch["speech"]=speech_array.squeeze().numpy()
  return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
  tlogits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits

predicted_ids = torch.argmax(logits, dim=-1)

print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```

## Evaluation

The model can be evaluated as follows on our unique Fon test data.

```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

for root, dirs, files in os.walk(test/):
  test_dataset = load_dataset("json", data_files=[os.path.join(root,i) for i in files],split="train")

chars_to_ignore_regex = '[\\,\\?\\.\\!\\-\\;\\:\\"\\“\\%\\‘\\”]'
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower() + " "
    return batch

test_dataset = test_dataset.map(remove_special_characters)
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("chrisjay/wav2vec2-large-xlsr-53-fon")
model = Wav2Vec2ForCTC.from_pretrained("chrisjay/wav2vec2-large-xlsr-53-fon")
model.to("cuda")

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = speech_array[0].numpy()
    batch["sampling_rate"] = sampling_rate
    batch["target_text"] = batch["sentence"]
    return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)

#Evaluation on test dataset
def evaluate(batch):
  inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

  with torch.no_grad():
    logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits

  pred_ids = torch.argmax(logits, dim=-1)
  batch["pred_strings"] = processor.batch_decode(pred_ids)
  return batch

result = test_dataset.map(evaluate, batched=True, batch_size=8)

print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))

```

**Test Result**: 14.97 %

## Training

The [Fon dataset](https://github.com/laleye/pyFongbe/tree/master/data) was split into `train`(8235 samples), `validation`(1107 samples), and `test`(1061 samples).

The script used for training can be found [here](https://colab.research.google.com/drive/11l6qhJCYnPTG1TQZ8f3EvKB9z12TQi4g?usp=sharing)

# Collaborators on this project

  - Chris C. Emezue  ([Twitter](https://twitter.com/ChrisEmezue))|(chris.emezue@gmail.com)
  - Bonaventure F.P. Dossou (HuggingFace Username: [bonadossou](https://huggingface.co/bonadossou))|([Twitter](https://twitter.com/bonadossou))|(femipancrace.dossou@gmail.com)

## This is a joint project continuing our research on [OkwuGbé: End-to-End Speech Recognition for Fon and Igbo](https://arxiv.org/abs/2103.07762)


---
language: african-languages
tags:
- african-languages
- machine-translation
- text
license: apache-2.0
model-index:
- name: Masakhane Benchmark Models
  results:
  - task:
      name: Machine Translation
      type: machine-translation
    dataset:
      name: masakhane benchmarks
      args: african-languages

---
# Interacting with the Masakhane Benchmark Models

I created this demo for very easy interaction with the [benchmark models on Masakhane](https://github.com/masakhane-io/masakhane-mt/tree/master/benchmarks) which were trained with [JoeyNMT](https://github.com/chrisemezue/joeynmt)(my forked version).

To access the space click [here](https://huggingface.co/spaces/chrisjay/masakhane-benchmarks).

To include your language, all you need to do is:
1. Create a folder in the format *src-tgt/main* for your language pair, if it does not exist.
2. Inside the *main* folder put the following files:
    1. model checkpoint. Rename it to `best.ckpt`.
    2. `config.yaml` file. This is the JoeyNMT config file which loads the model an pre-processing parameters.
    3. `src_vocab.txt` file.
    4. `trg_vocab.txt` file.

The space currently supports these languages:

| source language | target language |
|:---------------:|:---------------:|
| English         | Swahili         |
| English         | Afrikaans       |
| English         | Arabic          |
| English         | Urhobo          |
| English         | Ẹ̀dó             |
| Efik            | English         |
| English         | Hausa           |
| English         | Igbo            |
| English         | Fon             |
| English         | Twi             |
| English         | Dendi           |
| English         | Ẹ̀sán             |
| English         | Isoko           |
| English         | Kamba           |
| English         | Luo           |
| English         | Southern Ndebele  |
| English         | Tshivenda           |
| Shona           | English         |
| Swahili         | English         |
| Yoruba          | English         |

TO DO:
1. Include more languages from the benchmark.


Indonesian BERT Base Sentiment Classifier is a sentiment-text-classification model. The model was originally the pre-trained [IndoBERT Base Model (phase1 - uncased)](https://huggingface.co/indobenchmark/indobert-base-p1) model using [Prosa sentiment dataset](https://github.com/indobenchmark/indonlu/tree/master/dataset/smsa_doc-sentiment-prosa)

## How to Use
### As Text Classifier
```python
from transformers import pipeline
from transformers import AutoTokenizer, AutoModelForSequenceClassification

pretrained= "mdhugol/indonesia-bert-sentiment-classification"

model = AutoModelForSequenceClassification.from_pretrained(pretrained)
tokenizer = AutoTokenizer.from_pretrained(pretrained)

sentiment_analysis = pipeline("sentiment-analysis", model=model, tokenizer=tokenizer)

label_index = {'LABEL_0': 'positive', 'LABEL_1': 'neutral', 'LABEL_2': 'negative'}

pos_text = "Sangat bahagia hari ini"
neg_text = "Dasar anak sialan!! Kurang ajar!!"

result = sentiment_analysis(pos_text)
status = label_index[result[0]['label']]
score = result[0]['score']
print(f'Text: {pos_text} | Label : {status} ({score * 100:.3f}%)')

result = sentiment_analysis(neg_text)
status = label_index[result[0]['label']]
score = result[0]['score']
print(f'Text: {neg_text} | Label : {status} ({score * 100:.3f}%)')
```


---
license: gpl-3.0
tags:
- generated_from_trainer
datasets:
- mim_gold_ner
metrics:
- precision
- recall
- f1
- accuracy
widget:
- text: Systurnar Guðrún og Monique átu einar á McDonalds og horfðu á Stöð 2, þar glitti í Bruce Willis leika í Die Hard 2.
model-index:
- name: IceBERT-finetuned-ner
  results:
  - task:
      name: Token Classification
      type: token-classification
    dataset:
      name: mim_gold_ner
      type: mim_gold_ner
      args: mim-gold-ner
    metrics:
    - name: Precision
      type: precision
      value: 0.89397115028973
    - name: Recall
      type: recall
      value: 0.8664117576771418
    - name: F1
      type: f1
      value: 0.8799757281553399
    - name: Accuracy
      type: accuracy
      value: 0.9854156499755994
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# IceBERT-finetuned-ner

This model is a fine-tuned version of [vesteinn/IceBERT](https://huggingface.co/vesteinn/IceBERT) on the mim_gold_ner dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0802
- Precision: 0.8940
- Recall: 0.8664
- F1: 0.8800
- Accuracy: 0.9854

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.0528        | 1.0   | 2904 | 0.0779          | 0.8829    | 0.8504 | 0.8663 | 0.9831   |
| 0.0274        | 2.0   | 5808 | 0.0784          | 0.8802    | 0.8585 | 0.8692 | 0.9839   |
| 0.0162        | 3.0   | 8712 | 0.0802          | 0.8940    | 0.8664 | 0.8800 | 0.9854   |

### Framework versions

- Transformers 4.11.1
- Pytorch 1.9.0+cu102
- Datasets 1.12.1
- Tokenizers 0.10.3


---
license: agpl-3.0
tags:
- generated_from_trainer
datasets:
- mim_gold_ner
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: XLMR-ENIS-finetuned-ner
  results:
  - task:
      name: Token Classification
      type: token-classification
    dataset:
      name: mim_gold_ner
      type: mim_gold_ner
      args: mim-gold-ner
    metrics:
    - name: Precision
      type: precision
      value: 0.8714268909540054
    - name: Recall
      type: recall
      value: 0.842296759522456
    - name: F1
      type: f1
      value: 0.8566142460684552
    - name: Accuracy
      type: accuracy
      value: 0.9827189115812273
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# XLMR-ENIS-finetuned-ner

This model is a fine-tuned version of [vesteinn/XLMR-ENIS](https://huggingface.co/vesteinn/XLMR-ENIS) on the mim_gold_ner dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0955
- Precision: 0.8714
- Recall: 0.8423
- F1: 0.8566
- Accuracy: 0.9827

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.0561        | 1.0   | 2904 | 0.0939          | 0.8481    | 0.8205 | 0.8341 | 0.9804   |
| 0.031         | 2.0   | 5808 | 0.0917          | 0.8652    | 0.8299 | 0.8472 | 0.9819   |
| 0.0186        | 3.0   | 8712 | 0.0955          | 0.8714    | 0.8423 | 0.8566 | 0.9827   |

### Framework versions

- Transformers 4.11.1
- Pytorch 1.9.0+cu102
- Datasets 1.12.1
- Tokenizers 0.10.3


---
language: ru
tags:
- text-generation

---

# GPT2 - RUS


---
language: ru
tags:
- text-generation
license: apache-2.0
datasets:
- bookcorpus
- wikipedia
---

# LSTM


# random-roberta-base

We introduce random-roberta-base, which is a unpretrained version of RoBERTa model. The weight of random-roberta-base is randomly initiated and this can be particularly useful when we aim to train a language model from scratch or benchmark the effect of pretraining.

It's important to note that tokenizer of random-roberta-base is the same as roberta-base because it's not a trivial task to get a random tokenizer and it's less meaningful compared to the random weight.

A debatable advantage of pulling random-roberta-base from Huggingface is to avoid using random seed in order to obtain the same randomness at each time.

The code to obtain such a random model:

```python

from transformers import AutoTokenizer, AutoModelForSequenceClassification

def get_blank_model_from_hf(model_name="bert-base-cased"):
    model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=5)
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model.base_model.init_weights()
    model_name = "random-" + model_name
    base_model= model.base_model
    return base_model, tokenizer, model_name
```


# random-albert-base-v2

We introduce random-albert-base-v2, which is a unpretrained version of Albert model. The weight of random-albert-base-v2 is randomly initiated and this can be particularly useful when we aim to train a language model from scratch or benchmark the effect of pretraining.

It's important to note that tokenizer of random-albert-base-v2 is the same as albert-base-v2 because it's not a trivial task to get a random tokenizer and it's less meaningful compared to the random weight.

A debatable advantage of pulling random-albert-base-v2 from Huggingface is to avoid using random seed in order to obtain the same randomness at each time.

The code to obtain a such random model:

```python

from transformers import AutoTokenizer, AutoModelForSequenceClassification

def get_blank_model_from_hf(model_name="bert-base-cased"):
    model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=5)
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model.base_model.init_weights()
    model_name = "random-" + model_name
    base_model= model.base_model
    return base_model, tokenizer, model_name
```


# random-roberta-tiny

We introduce random-roberta-tiny, which is a unpretrained version of a mini RoBERTa model(2 layer and 128 heads). The weight of random-roberta-tiny is randomly initiated and this can be particularly useful when we aim to train a language model from scratch or benchmark the effect of pretraining.

It's important to note that tokenizer of random-roberta-tiny is the same as roberta-base because it's not a trivial task to get a random tokenizer and it's less meaningful compared to the random weight.

A debatable advantage of pulling random-roberta-tiny from Huggingface is to avoid using random seed in order to obtain the same randomness at each time.

The code to obtain such random model:

```python

from transformers import RobertaConfig, RobertaModel

def get_custom_blank_roberta(h=768, l=12):

    # Initializing a RoBERTa configuration
    configuration = RobertaConfig(num_attention_heads=h, num_hidden_layers=l)

    # Initializing a model from the configuration
    model = RobertaModel(configuration)

    return model

rank="tiny"
h=128
l=2
model_type = "roberta"
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model_name ="random-"+model_type+"-"+rank
model = get_custom_blank_roberta(h, l)
```


The weight of this model is randomly initiated and this can be particularly useful when we aim to train a language model from scratch or benchmark the effect of pretraining.

It's important to note that tokenizer of this random model is the same as the original pretrained model because it's not a trivial task to get a random tokenizer and it's less meaningful compared to the random weight.

A debatable advantage of pulling this model from Huggingface is to avoid using random seed in order to obtain the same randomness at each time.

The code to obtain such a random model:

```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# @Filename : random_model
# @Date : 2021-12-13-10-08

import os
from transformers import AutoModel, AutoTokenizer

def auto_name_blank_model(model_url:str):
    original_model_name:str = os.path.basename(model_url)
    return "blank_"+original_model_name

pretrained_model_url="google/bert_uncased_L-2_H-128_A-2"

tokenizer = AutoTokenizer.from_pretrained(pretrained_model_url)
model = AutoModel.from_pretrained(pretrained_model_url)
model.init_weights()
new_repo_name:str = auto_name_blank_model(pretrained_model_url)
model.push_to_hub(new_repo_name)
tokenizer.push_to_hub(new_repo_name)

# uploading files to an existing repo will overwrite the files without prompt.
```


# random-roberta-mini

We introduce random-roberta-mini, which is a unpretrained version of a mini RoBERTa model(4 layer and 256 heads). The weight of random-roberta-mini is randomly initiated and this can be particularly useful when we aim to train a language model from scratch or benchmark the effect of pretraining.

It's important to note that tokenizer of random-roberta-mini is the same as roberta-base because it's not a trivial task to get a random tokenizer and it's less meaningful compared to the random weight.

A debatable advantage of pulling random-roberta-mini from Huggingface is to avoid using random seed in order to obtain the same randomness at each time.

The code to obtain such random model:

```python

from transformers import RobertaConfig, RobertaModel

def get_custom_blank_roberta(h=768, l=12):

    # Initializing a RoBERTa configuration
    configuration = RobertaConfig(num_attention_heads=h, num_hidden_layers=l)

    # Initializing a model from the configuration
    model = RobertaModel(configuration)

    return model

rank="mini"
h=256
l=4
model_type = "roberta"
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model_name ="random-"+model_type+"-"+rank
model = get_custom_blank_roberta(h, l)
```


---
language:
- en
tags:
- legal
license: apache-2.0
metrics:
- precision
- recall
---

# LEGAL-ROBERTA
We introduce LEGAL-ROBERTA, which is a domain-specific language representation model fine-tuned on large-scale legal corpora(4.6 GB).

## Demo

'This \<mask\> Agreement is between General Motors and John Murray .'

| Model        | top1 | top2    |  top3   | top4    | top5 |
| ------------ | ---- | --- | --- | --- | -------- |
| Bert         |  new    |   current  | proposed    |  marketing   |    joint      |
| legalBert    |  settlement    | letter    |  dealer   |  master   |   supplemental       |
| legalRoberta | License     |  Settlement   |  Contract   |  license   |   Trust       |

> LegalRoberta captures the case

'The applicant submitted that her husband was subjected to treatment amounting to \<mask\> whilst in the custody of Adana Security Directorate'

| Model        | top1 | top2    |  top3   | top4    |  top5 |
| ------------ | ---- | --- | --- | --- | -------- |
| Bert    |  torture    |  rape   |  abuse   |  death   |     violence     |
| legalBert    |  torture    | detention    | arrest    |  rape   |   death       |
| legalRoberta | torture     |  abuse   |  insanity   |   cruelty  |    confinement      |

'Establishing a system for the identification and registration of \<mask\> animals and regarding the labeling of beef and beef products .':

| Model        | top1 | top2    |  top3   | top4    | top5 |
| ------------ | ---- | --- | --- | --- | -------- |
| Bert         |  farm    |  livestock   | draft    | domestic    |   wild       |
| legalBert    |   live   |  beef   |  farm   | pet    |      dairy    |
| legalRoberta | domestic     |  all   |  beef   |   wild  |    registered      |

## Load Pretrained Model

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("saibo/legal-roberta-base")
model = AutoModel.from_pretrained("saibo/legal-roberta-base")
```

## Training data

The training data consists of 3 origins:

1. Patent Litigations (https://www.kaggle.com/uspto/patent-litigations): This dataset covers over 74k cases across 52 years and over 5 million relevant documents. 5 different files detail the litigating parties, their attorneys, results, locations, and dates.
    1. *1.57GB*
    2. abbrev:PL
    3. *clean 1.1GB*

2. Caselaw Access Project (CAP) (https://case.law/): Following 360 years of United States case law, Caselaw Access Project (CAP) API and bulk data services includes 40 million pages of U.S. court decisions and almost 6.5 million individual cases.
    1. *raw 5.6*
    2. abbrev:CAP
    3. *clean 2.8GB*
3. Google Patents Public Data (https://www.kaggle.com/bigquery/patents): The Google Patents Public Data contains a collection of publicly accessible, connected database tables for empirical analysis of the international patent system.
    1. *BigQuery (https://www.kaggle.com/sohier/beyond-queries-exploring-the-bigquery-api)*
    2. abbrev:GPPD(1.1GB,patents-public-data.uspto_oce_litigation.documents)
    3. *clean 1GB*

## Training procedure
We start from a pretrained ROBERTA-BASE model and fine-tune it on the legal corpus.

Fine-tuning configuration:
- lr = 5e-5(with lr decay, ends at 4.95e-8)
- num_epoch = 3
- Total steps = 446500
- Total_flos = 2.7365e18

Loss starts at 1.850 and ends at 0.880
The perplexity after fine-tuning on legal corpus = 2.2735

Device:
2*GeForce GTX TITAN X computeCapability: 5.2

## Eval results
We benchmarked the model on two downstream tasks: Multi-Label Classification for Legal Text and Catchphrase Retrieval with Legal Case Description.

1.LMTC, Legal Multi-Label Text Classification

Dataset:

Labels shape:    4271
Frequent labels: 739
Few labels:      3369
Zero labels:     163

Hyperparameters:
- lr: 1e-05
- batch_size: 4
- max_sequence_size: 512
- max_label_size: 15
- few_threshold: 50
- epochs: 10
- dropout:0.1
- early stop:yes
- patience: 3

## Limitations:
In the Masked Language Model showroom, the tokens have the prefix **Ġ**. This seems to be wired but I haven't yet been able to fix it.
I know in the case of BPE tokenizer(ROBERTA's tokenizer), the symbol Ġ means the end of a new token, and the majority of tokens in the vocabs of pre-trained tokenizers start with Ġ.

For example
```python
import transformers
tokenizer = transformers.RobertaTokenizer.from_pretrained('roberta-base')
print(tokenizer.tokenize('I love salad'))
```
Outputs:

```
['I', 'Ġlove', 'Ġsalad']
```

The pretraining of LegalRoBERTa was restricted by the size of legal corpora available and the number of pretraining steps is small compared to the popular domain adapted models. This makes legalRoBERTa significantly **under-trained**.

## BibTeX entry and citation info


---
language: "English"
license: apache-2.0
tags:
- token-classification
- NER
- Biomedical
- Diseases
datasets:
- BC5CDR-diseases
- ncbi_disease
---
BioBERT model fine-tuned in NER task with BC5CDR-diseases and NCBI-diseases corpus

This was fine-tuned in order to use it in a BioNER/BioNEN system which is available at: https://github.com/librairy/bio-ner


---
language: "English"
license: apache-2.0
tags:
- token-classification
- NER
- Biomedical
- Genetics
datasets:
- JNLPBA
- BC2GM
---
BioBERT model fine-tuned in NER task with JNLPBA and BC2GM corpus for genetic class entities.

This was fine-tuned in order to use it in a BioNER/BioNEN system which is available at: https://github.com/librairy/bio-ner


---
language: "English"
tags:
- token-classification
- NER
- Biomedical
- Chemicals
datasets:
- BC5CDR-chemicals
- BC4CHEMD
license: apache-2.0
---

BioBERT model fine-tuned in NER task with BC5CDR-chemicals and BC4CHEMD corpus.

This was fine-tuned in order to use it in a BioNER/BioNEN system which is available at: https://github.com/librairy/bio-ner


---
tags:
- conversational
---

# My Awesome Model


### Model Description
GPT-J 6B is a transformer model designed using EleutherAI's replication of the GPT-3 architecture. GPT-J refers to the class of models, while 6B represents the number of parameters of this particular pre-trained model.

The original GPT-J-6B model is trained with TPUs, which is not easy to use for normal users. Thus, through a converting script, we convert the TPU version GPT-J-6B into GPU version, which could be load and fine-tuned with GPUs.

As we have tried, the model can be loaded with 1 GPU with 16G memory to do inference. For fine-tune, we used 8 * 32G GPUs with DeepSpeed library to distribute the model, data and gradients, in order to allocate the huge amount of model parameters.


---
tags:
- conversational
---

# Rick and Morty DialoGPT Model


---
tags:
- conversational
---

# Personal DialoGPT Model


---
license: apache-2.0
tags:
- generated_from_trainer
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: distilbert-base-uncased-finetuned-ner
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-uncased-finetuned-ner

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.9988
- Precision: 0.3
- Recall: 0.6
- F1: 0.4
- Accuracy: 0.7870

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| No log        | 1.0   | 84   | 0.8399          | 0.2105    | 0.4    | 0.2759 | 0.75     |
| No log        | 2.0   | 168  | 0.9664          | 0.3       | 0.6    | 0.4    | 0.7870   |
| No log        | 3.0   | 252  | 0.9988          | 0.3       | 0.6    | 0.4    | 0.7870   |

### Framework versions

- Transformers 4.16.2
- Pytorch 1.10.0+cu111
- Datasets 1.18.2
- Tokenizers 0.11.0


## Usage:
```
from sentence_transformers import models
from sentence_transformers import SentenceTransformer

word_embedding_model = models.Transformer('Cro-CoV-cseBERT')
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(),
                               pooling_mode_mean_tokens=True,
                               pooling_mode_cls_token=False,
                               pooling_mode_max_tokens=False)
model = SentenceTransformer(modules=[word_embedding_model, pooling_model], device='')  ## device = 'gpu' or 'cpu'
texts_emb = model.encode(texts)
```

## Datasets:
https://github.com/InfoCoV/InfoCoV

## Paper:
Please cite https://www.mdpi.com/2076-3417/11/21/10442


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: distilgpt2-finetuned-wikitext2
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilgpt2-finetuned-wikitext2

This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on the None dataset.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0

### Framework versions

- Transformers 4.12.3
- Pytorch 1.10.0+cu111
- Datasets 1.15.1
- Tokenizers 0.10.3


---
tags:
- conversational
---

# test DialoGPT Model


language: en
license: bsd
datasets:
- bookcorpus
- wikipedia
---

# SqueezeBERT pretrained model

This model, `squeezebert-uncased`, is a pretrained model for the English language using a masked language modeling (MLM) and Sentence Order Prediction (SOP) objective.
SqueezeBERT was introduced in [this paper](https://arxiv.org/abs/2006.11316). This model is case-insensitive. The model architecture is similar to BERT-base, but with the pointwise fully-connected layers replaced with [grouped convolutions](https://blog.yani.io/filter-group-tutorial/).
The authors found that SqueezeBERT is 4.3x faster than `bert-base-uncased` on a Google Pixel 3 smartphone.

## Pretraining

### Pretraining data
- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of thousands of unpublished books
- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia)

### Pretraining procedure
The model is pretrained using the Masked Language Model (MLM) and Sentence Order Prediction (SOP) tasks.
(Author's note: If you decide to pretrain your own model, and you prefer to train with MLM only, that should work too.)

From the SqueezeBERT paper:
> We pretrain SqueezeBERT from scratch (without distillation) using the [LAMB](https://arxiv.org/abs/1904.00962) optimizer, and we employ the hyperparameters recommended by the LAMB authors: a global batch size of 8192, a learning rate of 2.5e-3, and a warmup proportion of 0.28. Following the LAMB paper's recommendations, we pretrain for 56k steps with a maximum sequence length of 128 and then for 6k steps with a maximum sequence length of 512.

## Finetuning

The SqueezeBERT paper results from 2 approaches to finetuning the model:
- "finetuning without bells and whistles" -- after pretraining the SqueezeBERT model, finetune it on each GLUE task
- "finetuning with bells and whistles" -- after pretraining the SqueezeBERT model, finetune it on a MNLI with distillation from a teacher model. Then, use the MNLI-finetuned SqueezeBERT model as a student model to finetune on each of the other GLUE tasks (e.g. RTE, MRPC, …) with distillation from a task-specific teacher model.

A detailed discussion of the hyperparameters used for finetuning is provided in the appendix of the [SqueezeBERT paper](https://arxiv.org/abs/2006.11316).
Note that finetuning SqueezeBERT with distillation is not yet implemented in this repo. If the author (Forrest Iandola - forrest.dnn@gmail.com) gets enough encouragement from the user community, he will add example code to Transformers for finetuning SqueezeBERT with distillation.

This model, `squeezebert/squeezebert-uncased`, has been pretrained but not finetuned. For most text classification tasks, we recommend using squeezebert-mnli-headless as a starting point.

### How to finetune
To try finetuning SqueezeBERT on the [MRPC](https://www.microsoft.com/en-us/download/details.aspx?id=52398) text classification task, you can run the following command:
```
./utils/download_glue_data.py

python examples/text-classification/run_glue.py \
    --model_name_or_path squeezebert-base-headless \
    --task_name mrpc \
    --data_dir ./glue_data/MRPC \
    --output_dir ./models/squeezebert_mrpc \
    --overwrite_output_dir \
    --do_train \
    --do_eval \
    --num_train_epochs 10 \
    --learning_rate 3e-05 \
    --per_device_train_batch_size 16 \
    --save_steps 20000

```

## BibTeX entry and citation info
```
@article{2020_SqueezeBERT,
     author = {Forrest N. Iandola and Albert E. Shaw and Ravi Krishna and Kurt W. Keutzer},
     title = {{SqueezeBERT}: What can computer vision teach NLP about efficient neural networks?},
     journal = {arXiv:2006.11316},
     year = {2020}
}
```


language: en
license: bsd
datasets:
- bookcorpus
- wikipedia
---

# SqueezeBERT pretrained model

This model, `squeezebert-mnli-headless`, has been pretrained for the English language using a masked language modeling (MLM) and Sentence Order Prediction (SOP) objective and finetuned on the [Multi-Genre Natural Language Inference (MNLI)](https://cims.nyu.edu/~sbowman/multinli/) dataset. This is a "headless" model with the final classification layer removed, and this will allow Transformers to automatically reinitialize the final classification layer before you begin finetuning on your data.
SqueezeBERT was introduced in [this paper](https://arxiv.org/abs/2006.11316). This model is case-insensitive. The model architecture is similar to BERT-base, but with the pointwise fully-connected layers replaced with [grouped convolutions](https://blog.yani.io/filter-group-tutorial/).
The authors found that SqueezeBERT is 4.3x faster than `bert-base-uncased` on a Google Pixel 3 smartphone.

## Pretraining

### Pretraining data
- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of thousands of unpublished books
- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia)

### Pretraining procedure
The model is pretrained using the Masked Language Model (MLM) and Sentence Order Prediction (SOP) tasks.
(Author's note: If you decide to pretrain your own model, and you prefer to train with MLM only, that should work too.)

From the SqueezeBERT paper:
> We pretrain SqueezeBERT from scratch (without distillation) using the [LAMB](https://arxiv.org/abs/1904.00962) optimizer, and we employ the hyperparameters recommended by the LAMB authors: a global batch size of 8192, a learning rate of 2.5e-3, and a warmup proportion of 0.28. Following the LAMB paper's recommendations, we pretrain for 56k steps with a maximum sequence length of 128 and then for 6k steps with a maximum sequence length of 512.

## Finetuning

The SqueezeBERT paper presents 2 approaches to finetuning the model:
- "finetuning without bells and whistles" -- after pretraining the SqueezeBERT model, finetune it on each GLUE task
- "finetuning with bells and whistles" -- after pretraining the SqueezeBERT model, finetune it on a MNLI with distillation from a teacher model. Then, use the MNLI-finetuned SqueezeBERT model as a student model to finetune on each of the other GLUE tasks (e.g. RTE, MRPC, …) with distillation from a task-specific teacher model.

A detailed discussion of the hyperparameters used for finetuning is provided in the appendix of the [SqueezeBERT paper](https://arxiv.org/abs/2006.11316).
Note that finetuning SqueezeBERT with distillation is not yet implemented in this repo. If the author (Forrest Iandola - forrest.dnn@gmail.com) gets enough encouragement from the user community, he will add example code to Transformers for finetuning SqueezeBERT with distillation.

This model, `squeezebert/squeezebert-mnli-headless`, is the "finetuned with bells and whistles" MNLI-finetuned SqueezeBERT model. In this particular model, we have removed the final classification layer -- in other words, it is "headless." We recommend using this model if you intend to finetune the model on your own data. Using this model means that your final layer will automatically be reinitialized when you start finetuning on your data.

### How to finetune
To try finetuning SqueezeBERT on the [MRPC](https://www.microsoft.com/en-us/download/details.aspx?id=52398) text classification task, you can run the following command:
```
./utils/download_glue_data.py

python examples/text-classification/run_glue.py \
    --model_name_or_path squeezebert-base-headless \
    --task_name mrpc \
    --data_dir ./glue_data/MRPC \
    --output_dir ./models/squeezebert_mrpc \
    --overwrite_output_dir \
    --do_train \
    --do_eval \
    --num_train_epochs 10 \
    --learning_rate 3e-05 \
    --per_device_train_batch_size 16 \
    --save_steps 20000

```

## BibTeX entry and citation info
```
@article{2020_SqueezeBERT,
     author = {Forrest N. Iandola and Albert E. Shaw and Ravi Krishna and Kurt W. Keutzer},
     title = {{SqueezeBERT}: What can computer vision teach NLP about efficient neural networks?},
     journal = {arXiv:2006.11316},
     year = {2020}
}
```


language: en
license: bsd
datasets:
- bookcorpus
- wikipedia
---

# SqueezeBERT pretrained model

This model, `squeezebert-mnli`, has been pretrained for the English language using a masked language modeling (MLM) and Sentence Order Prediction (SOP) objective and finetuned on the [Multi-Genre Natural Language Inference (MNLI)](https://cims.nyu.edu/~sbowman/multinli/) dataset.
SqueezeBERT was introduced in [this paper](https://arxiv.org/abs/2006.11316). This model is case-insensitive. The model architecture is similar to BERT-base, but with the pointwise fully-connected layers replaced with [grouped convolutions](https://blog.yani.io/filter-group-tutorial/).
The authors found that SqueezeBERT is 4.3x faster than `bert-base-uncased` on a Google Pixel 3 smartphone.

## Pretraining

### Pretraining data
- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of thousands of unpublished books
- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia)

### Pretraining procedure
The model is pretrained using the Masked Language Model (MLM) and Sentence Order Prediction (SOP) tasks.
(Author's note: If you decide to pretrain your own model, and you prefer to train with MLM only, that should work too.)

From the SqueezeBERT paper:
> We pretrain SqueezeBERT from scratch (without distillation) using the [LAMB](https://arxiv.org/abs/1904.00962) optimizer, and we employ the hyperparameters recommended by the LAMB authors: a global batch size of 8192, a learning rate of 2.5e-3, and a warmup proportion of 0.28. Following the LAMB paper's recommendations, we pretrain for 56k steps with a maximum sequence length of 128 and then for 6k steps with a maximum sequence length of 512.

## Finetuning

The SqueezeBERT paper presents 2 approaches to finetuning the model:
- "finetuning without bells and whistles" -- after pretraining the SqueezeBERT model, finetune it on each GLUE task
- "finetuning with bells and whistles" -- after pretraining the SqueezeBERT model, finetune it on a MNLI with distillation from a teacher model. Then, use the MNLI-finetuned SqueezeBERT model as a student model to finetune on each of the other GLUE tasks (e.g. RTE, MRPC, …) with distillation from a task-specific teacher model.

A detailed discussion of the hyperparameters used for finetuning is provided in the appendix of the [SqueezeBERT paper](https://arxiv.org/abs/2006.11316).
Note that finetuning SqueezeBERT with distillation is not yet implemented in this repo. If the author (Forrest Iandola - forrest.dnn@gmail.com) gets enough encouragement from the user community, he will add example code to Transformers for finetuning SqueezeBERT with distillation.

This model, `squeezebert/squeezebert-mnli`, is the "trained with bells and whistles" MNLI-finetuned SqueezeBERT model.

### How to finetune
To try finetuning SqueezeBERT on the [MRPC](https://www.microsoft.com/en-us/download/details.aspx?id=52398) text classification task, you can run the following command:
```
./utils/download_glue_data.py

python examples/text-classification/run_glue.py \
    --model_name_or_path squeezebert-base-headless \
    --task_name mrpc \
    --data_dir ./glue_data/MRPC \
    --output_dir ./models/squeezebert_mrpc \
    --overwrite_output_dir \
    --do_train \
    --do_eval \
    --num_train_epochs 10 \
    --learning_rate 3e-05 \
    --per_device_train_batch_size 16 \
    --save_steps 20000

```

## BibTeX entry and citation info
```
@article{2020_SqueezeBERT,
     author = {Forrest N. Iandola and Albert E. Shaw and Ravi Krishna and Kurt W. Keutzer},
     title = {{SqueezeBERT}: What can computer vision teach NLP about efficient neural networks?},
     journal = {arXiv:2006.11316},
     year = {2020}
}
```


---
tags:
- spacy
- token-classification
language:
- de
license: cc-by-4.0
model-index:
- name: de_MRP_NER
  results:
  - task:
      name: NER
      type: token-classification
    metrics:
    - name: NER Precision
      type: precision
      value: 0.905255631
    - name: NER Recall
      type: recall
      value: 0.8568527919
    - name: NER F Score
      type: f_score
      value: 0.8803894298
---
NER Model for 'Ministerratsprotokolle'

| Feature | Description |
| --- | --- |
| **Name** | `de_MRP_NER` |
| **Version** | `0.0.0` |
| **spaCy** | `>=3.1.0,<3.2.0` |
| **Default Pipeline** | `tok2vec`, `ner` |
| **Components** | `tok2vec`, `ner` |
| **Vectors** | 0 keys, 0 unique vectors (0 dimensions) |
| **Sources** | n/a |
| **License** | `cc-by` |
| **Author** | [Peter Andorfer]() |

### Label Scheme

<details>

<summary>View label scheme (4 labels for 1 components)</summary>

| Component | Labels |
| --- | --- |
| **`ner`** | `GPE`, `LOC`, `ORG`, `PER` |

</details>

### Accuracy

| Type | Score |
| --- | --- |
| `ENTS_F` | 88.04 |
| `ENTS_P` | 90.53 |
| `ENTS_R` | 85.69 |
| `TOK2VEC_LOSS` | 40077.56 |
| `NER_LOSS` | 77727.57 |


---
tags:
- spacy
- token-classification
language:
- de
license: cc-by-nc-4.0
model-index:
- name: de_RTA_NER
  results:
  - task:
      name: NER
      type: token-classification
    metrics:
    - name: NER Precision
      type: precision
      value: 0.8630136986
    - name: NER Recall
      type: recall
      value: 0.8743253662
    - name: NER F Score
      type: f_score
      value: 0.8686327078
---
Regensburger Reichstag von 1576

| Feature | Description |
| --- | --- |
| **Name** | `de_RTA_NER` |
| **Version** | `0.0.0` |
| **spaCy** | `>=3.1.0,<3.2.0` |
| **Default Pipeline** | `tok2vec`, `ner` |
| **Components** | `tok2vec`, `ner` |
| **Vectors** | 0 keys, 0 unique vectors (0 dimensions) |
| **Sources** | n/a |
| **License** | `https://creativecommons.org/licenses/by-nc/4.0/` |
| **Author** | [n/a](https://reichstagsakten-1576.uni-graz.at) |

### Label Scheme

<details>

<summary>View label scheme (4 labels for 1 components)</summary>

| Component | Labels |
| --- | --- |
| **`ner`** | `DATE`, `LOC`, `PER`, `TIME` |

</details>

### Accuracy

| Type | Score |
| --- | --- |
| `ENTS_F` | 86.86 |
| `ENTS_P` | 86.30 |
| `ENTS_R` | 87.43 |
| `TOK2VEC_LOSS` | 43588.74 |
| `NER_LOSS` | 95573.96 |


---
language:
- km
license: apache-2.0
tags:
- automatic-speech-recognition
- openslr
- robust-speech-event
- km
- generated_from_trainer
model-index:
- name: xls-r-300m-km
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: OpenSLR km
      type: openslr
      args: km
    metrics:
       - name: Test WER
         type: wer
         value: 25.70
       - name: Test CER
         type: cer
         value: 7.03
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Robust Speech Event - Dev Data
      type: speech-recognition-community-v2/dev_data
      args: km
    metrics:
       - name: Test WER
         type: wer
         value: 25.70
       - name: Test CER
         type: cer
         value: 7.03
---

#

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the openslr dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3281
- Wer: 0.3462

# Evaluation results on OpenSLR "test" (self-split 10%) (Running ./eval.py):
- WER: 0.3216977389924633
- CER: 0.08653361193169537

# Evaluation results with language model on OpenSLR "test" (self-split 10%) (Running ./eval.py):
- WER: 0.257040856802856
- CER: 0.07025001801282513

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 100
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 5.0795        | 5.47  | 400  | 4.4121          | 1.0    |
| 3.5658        | 10.95 | 800  | 3.5203          | 1.0    |
| 3.3689        | 16.43 | 1200 | 2.8984          | 0.9996 |
| 2.01          | 21.91 | 1600 | 1.0041          | 0.7288 |
| 1.6783        | 27.39 | 2000 | 0.6941          | 0.5989 |
| 1.527         | 32.87 | 2400 | 0.5599          | 0.5282 |
| 1.4278        | 38.35 | 2800 | 0.4827          | 0.4806 |
| 1.3458        | 43.83 | 3200 | 0.4429          | 0.4532 |
| 1.2893        | 49.31 | 3600 | 0.4156          | 0.4330 |
| 1.2441        | 54.79 | 4000 | 0.4020          | 0.4040 |
| 1.188         | 60.27 | 4400 | 0.3777          | 0.3866 |
| 1.1628        | 65.75 | 4800 | 0.3607          | 0.3858 |
| 1.1324        | 71.23 | 5200 | 0.3534          | 0.3604 |
| 1.0969        | 76.71 | 5600 | 0.3428          | 0.3624 |
| 1.0897        | 82.19 | 6000 | 0.3387          | 0.3567 |
| 1.0625        | 87.66 | 6400 | 0.3339          | 0.3499 |
| 1.0601        | 93.15 | 6800 | 0.3288          | 0.3446 |
| 1.0474        | 98.62 | 7200 | 0.3281          | 0.3462 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


---
language:
- en
license: apache-2.0
tags:
- automatic-speech-recognition
- librispeech_asr
- robust-speech-event
- en
- generated_from_trainer
model-index:
- name: XLS-R-300M - English
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: LibriSpeech ASR
      type: librispeech_asr
      args: clean
    metrics:
       - name: Test WER
         type: wer
         value: 12.29
       - name: Test CER
         type: cer
         value: 3.34
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Robust Speech Event - Dev Data
      type: speech-recognition-community-v2/dev_data
      args: en
    metrics:
       - name: Validation WER
         type: wer
         value: 36.75
       - name: Validation CER
         type: cer
         value: 14.83
---

#

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the librispeech_asr dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1444
- Wer: 0.1167

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 50
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 2.9365        | 4.17  | 500  | 2.9398          | 0.9999 |
| 1.5444        | 8.33  | 1000 | 0.5947          | 0.4289 |
| 1.1367        | 12.5  | 1500 | 0.2751          | 0.2366 |
| 0.9972        | 16.66 | 2000 | 0.2032          | 0.1797 |
| 0.9118        | 20.83 | 2500 | 0.1786          | 0.1479 |
| 0.8664        | 24.99 | 3000 | 0.1641          | 0.1408 |
| 0.8251        | 29.17 | 3500 | 0.1537          | 0.1267 |
| 0.793         | 33.33 | 4000 | 0.1525          | 0.1244 |
| 0.785         | 37.5  | 4500 | 0.1470          | 0.1184 |
| 0.7612        | 41.66 | 5000 | 0.1446          | 0.1177 |
| 0.7478        | 45.83 | 5500 | 0.1449          | 0.1176 |
| 0.7443        | 49.99 | 6000 | 0.1444          | 0.1167 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


---
language:
- km
license: apache-2.0
tags:
- automatic-speech-recognition
- openslr
- robust-speech-event
- km
- generated_from_trainer
model-index:
- name: wav2vec2-xls-r-1b-km
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: OpenSLR km
      type: openslr
      args: km
    metrics:
       - name: Test WER
         type: wer
         value: 32.13
       - name: Test CER
         type: cer
         value: 9.35
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Robust Speech Event - Dev Data
      type: speech-recognition-community-v2/dev_data
      args: km
    metrics:
       - name: Test WER
         type: wer
         value: 32.13
       - name: Test CER
         type: cer
         value: 9.35
---
#

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) on the openslr dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4239
- Wer: 0.4221

# Evaluation results on OpenSLR "test" (self-split 10%) (Running ./eval.py):
- WER: 0.4490281634272114
- CER: 0.12198285179047481

# Evaluation results on OpenSLR "test" with LM ngram (self-split 10%) (Running ./eval.py):
- WER: 0.32130107100357
- CER: 0.09345053678218891

# Note
- Since this dataset is small (4 hours of voice recording), we decided not to train that for too long to avoid overfitting and under-generalization.
- This model performs worse than its 300M-variant. Probably, we don't explore the hyper-parameter enough?

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 75
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.5671        | 5.47  | 400  | 12.0218         | 1.0    |
| 3.5159        | 10.95 | 800  | 10.6337         | 1.0    |
| 2.4543        | 16.43 | 1200 | 1.8256          | 0.9839 |
| 1.9437        | 21.91 | 1600 | 1.1237          | 0.9173 |
| 1.696         | 27.39 | 2000 | 0.8246          | 0.7700 |
| 1.5342        | 32.87 | 2400 | 0.6433          | 0.6594 |
| 1.4509        | 38.35 | 2800 | 0.5500          | 0.5787 |
| 1.3478        | 43.83 | 3200 | 0.5070          | 0.4907 |
| 1.3096        | 49.31 | 3600 | 0.4692          | 0.4726 |
| 1.2532        | 54.79 | 4000 | 0.4448          | 0.4479 |
| 1.2291        | 60.27 | 4400 | 0.4374          | 0.4366 |
| 1.196         | 65.75 | 4800 | 0.4314          | 0.4310 |
| 1.1862        | 71.23 | 5200 | 0.4239          | 0.4221 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


---
language:
- ja
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- generated_from_trainer
- ja
- robust-speech-event
datasets:
- common_voice
model-index:
- name: XLS-R-300M - Japanese
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice 8
      type: mozilla-foundation/common_voice_8_0
      args: ja
    metrics:
       - name: Test WER
         type: wer
         value: 54.05
       - name: Test CER
         type: cer
         value: 27.54
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Robust Speech Event - Dev Data
      type: speech-recognition-community-v2/dev_data
      args: ja
    metrics:
       - name: Validation WER
         type: wer
         value: 48.77
       - name: Validation CER
         type: cer
         value: 24.87
---

#

This model is for transcribing audio into Hiragana, one format of Japanese language.

This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the `mozilla-foundation/common_voice_8_0 dataset`. Note that the following results are achieved by:
- Modify `eval.py` to suit the use case.
- Since kanji and katakana shares the same sound as hiragana, we convert all texts to hiragana using [pykakasi](https://pykakasi.readthedocs.io) and tokenize them using [fugashi](https://github.com/polm/fugashi).

It achieves the following results on the evaluation set:
- Loss: 0.7751
- Cer: 0.2227

# Evaluation results (Running ./eval.py):

| Model    | Metric | Common-Voice-8/test | speech-recognition-community-v2/dev-data   |
|:--------:|:------:|:-------------------:|:------------------------------------------:|
| w/o LM   | WER    | 0.5964              | 0.5532                                     |
|          | CER    | 0.2944              | 0.2629                                     |
| w/  LM   | WER    | 0.5405              | 0.4877                                     |
|          | CER    | **0.2754**              | **0.2487**                                     |

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- training_steps: 4000
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Cer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 4.4081        | 1.6   | 500   | 4.0983          | 1.0    |
| 3.303         | 3.19  | 1000  | 3.3563          | 1.0    |
| 3.1538        | 4.79  | 1500  | 3.2066          | 0.9239 |
| 2.1526        | 6.39  | 2000  | 1.1597          | 0.3355 |
| 1.8726        | 7.98  | 2500  | 0.9023          | 0.2505 |
| 1.7817        | 9.58  | 3000  | 0.8219          | 0.2334 |
| 1.7488        | 11.18 | 3500  | 0.7915          | 0.2222 |
| 1.7039        | 12.78 | 4000  | 0.7751          | 0.2227 |
| Stop & Train  |       |       |                 |        |
| 1.6571        | 15.97 | 5000  | 0.6788          | 0.1685 |
| 1.520400      | 19.16 | 6000  | 0.6095          | 0.1409 |
| 1.448200      | 22.35 | 7000  | 0.5843          | 0.1430 |
| 1.385400      | 25.54 | 8000  | 0.5699          | 0.1263 |
| 1.354200      | 28.73 | 9000  | 0.5686          | 0.1219 |
| 1.331500      | 31.92 | 10000 | 0.5502          | 0.1144 |
| 1.290800      | 35.11 | 11000 | 0.5371          | 0.1140 |
| Stop & Train  |       |       |                 |        |
| 1.235200      | 38.30 | 12000 | 0.5394          | 0.1106 |

### Framework versions

- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


---
license: apache-2.0
tags:
- generated_from_trainer
model-index:
- name: daniel_asr
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# daniel_asr

This model is a fine-tuned version of [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4565
- Wer: 0.3423

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 30
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.4909        | 4.0   | 500  | 1.3485          | 0.8887 |
| 0.5887        | 8.0   | 1000 | 0.4957          | 0.4641 |
| 0.2207        | 12.0  | 1500 | 0.4621          | 0.3971 |
| 0.125         | 16.0  | 2000 | 0.4339          | 0.3756 |
| 0.0829        | 20.0  | 2500 | 0.4618          | 0.3613 |
| 0.0601        | 24.0  | 3000 | 0.4564          | 0.3535 |
| 0.0456        | 28.0  | 3500 | 0.4565          | 0.3423 |

### Framework versions

- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


---
tags:
- generated_from_trainer
model-index:
- name: bangla_asr
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bangla_asr

This model is a fine-tuned version of [Harveenchadha/vakyansh-wav2vec2-bengali-bnm-200](https://huggingface.co/Harveenchadha/vakyansh-wav2vec2-bengali-bnm-200) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 157.8652
- Wer: 0.4507

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 60
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 2601.5363     | 7.46  | 500  | 259.6630        | 0.6863 |
| 417.7386      | 14.93 | 1000 | 156.6117        | 0.5275 |
| 262.9455      | 22.39 | 1500 | 155.0886        | 0.5006 |
| 178.7715      | 29.85 | 2000 | 155.1077        | 0.4840 |
| 132.448       | 37.31 | 2500 | 163.8623        | 0.4770 |
| 116.3943      | 44.78 | 3000 | 161.5531        | 0.4609 |
| 87.1653       | 52.24 | 3500 | 165.6857        | 0.4597 |
| 80.5606       | 59.7  | 4000 | 157.8652        | 0.4507 |

### Framework versions

- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


---
language:
- ar
- en
license: apache-2.0
datasets:
- 4Dialects
- MADAR
- CSCS
thumbnail: https://www.informatik.hu-berlin.de/en/forschung-en/gebiete/ml-en/resolveuid/a6f82e0d7fa446a59c902cac4cafa9cb/@@images/image/preview
tags:
- flair
- token-classification
- sequence-tagger-model
- Dialectal Arabic
- Code-Switching
- Code-Mixing
metrics:
- f1
widget:
- text: "طلعوا جماعة الممانعة بالسياسة ما بيعرفوا ولا بالصحة بيعرفوا ولا حتى بالدين"
- text: "أعلم أن هذا يبدو غير عادل ، لكن لا يمكن أن يكون هناك ظلم"
- text: "أنا عارف أن الموضوع ده شكله مش عادل ، بس لا يمكن أن يكون فيه ظلم"
---

# Arabic Flair + fastText Part-of-Speech tagging Model (Egyptian and Levant)
Pretrained Part-of-Speech tagging model built on a joint corpus written in Egyptian and Levantine (Jordanian, Lebanese, Palestinian, Syrian) dialects with code-switching of Egyptian Arabic and English. The model is trained using [Flair](https://aclanthology.org/C18-1139/) (forward+backward)and [fastText](https://fasttext.cc) embeddings.

# Pretraining Corpora:
This sequence labeling model was pretrained on three corpora jointly:
1. [4 Dialects](https://huggingface.co/datasets/viewer/?dataset=arabic_pos_dialect)
A Dialectal Arabic Datasets containing four dialects of Arabic, Egyptian (EGY), Levantine (LEV), Gulf (GLF), and Maghrebi (MGR). Each dataset consists of a set of 350 manually segmented and PoS tagged tweets.
2. [UD South Levantine Arabic MADAR](https://universaldependencies.org/treebanks/ajp_madar/index.html)
A Dataset with 100 manually-annotated sentences taken from the [MADAR](https://camel.abudhabi.nyu.edu/madar/) (Multi-Arabic Dialect Applications and Resources) project by [Shorouq Zahra](mailto:shorouqjzahra@gmail.com).
3. Parts of the Cairo Students Code-Switch (CSCS) corpus developed for ["Collection and Analysis of Code-switch Egyptian Arabic-English Speech Corpus"](https://aclanthology.org/L18-1601.pdf) by Hamed et al.

# Usage
```python
from flair.data import Sentence
from flair.models import SequenceTagger

tagger = SequenceTagger.load("megantosh/flair-arabic-dialects-codeswitch-egy-lev")
sentence = Sentence('عمرو عادلي أستاذ للاقتصاد السياسي المساعد في الجامعة الأمريكية  بالقاهرة .')
tagger.predict(sentence)
for entity in sentence.get_spans('pos'):
    print(entity)
```

Due to the right-to-left in left-to-right context, some formatting errors might occur. and your code might appear like [this](https://ibb.co/ky20Lnq), (link accessed on 2020-10-27)

<!--# Example

# Tagset-->

# Scores & Tagset
<details>

| |precision |   recall | f1-score |  support|
|--|-----------|------|-------------|--------------|
|INTJ |    0.8182   | 0.9000    |0.8571    |    10|
|OUN   |  0.9009   | 0.9402    |0.9201      | 435|
|NUM    | 0.9524   | 0.8333   | 0.8889       | 24|
|ADJ     |0.8762   | 0.7603  |  0.8142      | 121|
|ADP     |0.9903    |0.9623 |   0.9761       |106|
| CCONJ |    0.9600   | 0.9730 |   0.9664 |       74|
|PROPN |    0.9333   | 0.9333  |  0.9333  |      15|
| ADV  |   0.9135   | 0.8051  |  0.8559   |    118|
|VERB   |  0.8852    | 0.9231 |   0.9038   |    117|
|PRON    | 0.9620    | 0.9465 |   0.9542    |   187|
|SCONJ |    0.8571   | 0.9474  |  0.9000      |  19|
|PART  |   0.9350   | 0.9791   | 0.9565       | 191|
| DET   |  0.9348    | 0.9149  |  0.9247 |       47|
|PUNCT    | 1.0000    | 1.0000  |  1.0000  |      35|
| AUX  |   0.9286    | 0.9811  |  0.9541   |     53|
|MENTION   |  0.9231   |  1.0000  |  0.9600    |    12|
|     V    | 0.8571   | 0.8780    | 0.8675     |   82|
| FUT-PART+V+PREP+PRON     |1.0000   | 0.0000   | 0.0000       |  1|
|  PROG-PART+V+PRON+PREP+PRON |     0.0000  |  1.0000  |  0.0000       |  0|
|ADJ+NSUFF |    0.6111   | 0.8462   | 0.7097 |       26|
|NOUN+NSUFF  |   0.8182   | 0.8438   | 0.8308  |      64|
|PREP+PRON   |  0.9565   | 0.9565   | 0.9565   |     23|
|                   PUNC    | 0.9941   | 1.0000   | 0.9971    |   169|
|                    EOS     |1.0000   | 1.0000   | 1.0000    |   70|
|             NOUN+PRON   |  0.6986   | 0.8500   | 0.7669      |  60|
|                V+PRON    | 0.7258   | 0.8036   | 0.7627       | 56|
|            PART+PRON    | 1.0000   | 0.9474   | 0.9730    |    19|
|          PROG-PART+V    | 0.8333   | 0.9302   | 0.8791 |       43|
|            DET+NOUN    | 0.9625   | 1.0000   | 0.9809  |      77|
|     NOUN+NSUFF+PRON    | 0.9091   | 0.7143   | 0.8000   |     14|
|     PROG-PART+V+PRON    | 0.7083   | 0.9444   | 0.8095    |    18|
|      PREP+NOUN+NSUFF    | 0.6667   | 0.4000   | 0.5000         5|
|     NOUN+NSUFF+NSUFF    | 1.0000   | 0.0000   | 0.0000 |        3|
|                CONJ    | 0.9722   | 1.0000   | 0.9859  |      35|
|        V+PRON+PRON    | 0.6364   | 0.5833   | 0.6087   |     12|
|           FOREIGN    | 0.6667   | 0.6667   | 0.6667    |     3|
|        PREP+NOUN    | 0.6316   | 0.7500  |  0.6857 |       16|
|  DET+NOUN+NSUFF    | 0.9000   | 0.9310  |  0.9153  |      29|
|  DET+ADJ+NSUFF    | 1.0000   | 0.5714  |  0.7273   |      7|
|     CONJ+PRON    | 1.0000   | 0.8750  |  0.9333     |    8|
|    NOUN+CASE    | 0.0000   | 0.0000  |  0.0000    |     2|
|     DET+ADJ    | 1.0000   | 0.6667  |  0.8000      |   6|
|       PREP    | 1.0000   | 0.9718  |  0.9857  |      71|
|  CONJ+FUT-PART+V    | 0.0000   | 0.0000  |  0.0000   |      1|
|            CONJ+V    | 0.6667   | 0.7500  |  0.7059    |     8|
|         FUT-PART    | 1.0000   | 1.0000  |  1.0000     |    2|
|             ADJ+PRON    | 1.0000   | 0.0000  |  0.0000      |   8|
|   CONJ+PREP+NOUN+PRON    | 1.0000   | 0.0000  |  0.0000       |  1|
|        CONJ+NOUN+PRON    | 0.3750   | 1.0000  |  0.5455      |   3|
|              PART+ADJ    | 1.0000   | 0.0000  |  0.0000       |  1|
|             PART+NOUN    | 0.5000   | 1.0000  |  0.6667        | 1|
|       CONJ+PREP+NOUN    | 1.0000   | 0.0000  |  0.0000       |  1|
|           CONJ+NOUN    | 0.7000   | 0.7778  |  0.7368  |       9|
|                URL    | 1.0000   | 1.0000   | 1.0000 |        3|
|     CONJ+FUT-PART    | 1.0000   | 0.0000   | 0.0000  |       1|
|       FUT-PART+V    | 0.8571   | 0.6000   | 0.7059   |     10|
|      PREP+NOUN+NSUFF+NSUFF    | 1.0000   | 0.0000    | 0.0000   |      1|
|                      HASH    | 1.0000   | 0.9412   | 0.9697     |   17|
|            ADJ+PREP+PRON    | 1.0000   | 0.0000   | 0.0000  |       3|
|          PREP+NOUN+PRON    | 0.0000   | 0.0000   | 0.0000   |      1|
|                   EMOT    | 1.0000   | 0.8889   | 0.9412    |    18|
|             CONJ+PREP    | 1.0000   | 0.7500   | 0.8571     |    4|
|  PREP+DET+NOUN+NSUFF    | 1.0000   | 0.7500   | 0.8571      |   4|
| PRON+DET+NOUN+NSUFF    | 0.0000   | 1.0000   | 0.0000       |  0|
|        V+PREP+PRON    | 1.0000   | 0.0000   | 0.0000        | 5|
|  V+PRON+PREP+PRON    | 0.0000   | 1.0000   | 0.0000         | 0|
|  CONJ+NOUN+NSUFF    | 0.5000   | 0.5000   | 0.5000 |        2|
|      V+NEG-PART    | 1.0000   | 0.0000   | 0.0000  |       2|
|  PREP+DET+NOUN    | 0.9091   | 1.0000   | 0.9524   |     10|
|        PREP+V    | 1.0000   | 0.0000   | 0.0000    |     2|
|    CONJ+PART    | 1.0000   | 0.7778   | 0.8750     |    9|
| CONJ+V+PRON    | 1.0000   | 1.0000   | 1.0000 |        5|
|    PROG-PART+V+PREP+PRON    | 1.0000   | 0.5000   | 0.6667  |       2|
|    PREP+NOUN+NSUFF+PRON    | 1.0000   | 1.0000   | 1.0000   |      1|
|               ADJ+CASE    | 1.0000   | 0.0000    | 0.0000   |      1|
|        PART+NOUN+PRON    | 1.0000   | 1.0000   | 1.0000     |    1|
|               PART+V    | 1.0000   | 0.0000  |  0.0000      |   3|
|         PART+V+PRON    | 0.0000   | 1.0000  |  0.0000       |  0|
|    FUT-PART+V+PRON    | 0.0000   | 1.0000  |  0.0000        | 0|
|FUT-PART+V+PRON+PRON    | 1.0000   | 0.0000  |  0.0000  |       1|
|     CONJ+PREP+PRON    | 1.0000   | 0.0000  |  0.0000   |      1|
|CONJ+V+PRON+PREP+PRON    | 1.0000   | 0.0000  |  0.0000    |     1|
|    CONJ+V+PREP+PRON    | 0.0000   | 1.0000  |  0.0000     |    0|
|CONJ+DET+NOUN+NSUFF    | 1.0000   | 0.0000  |  0.0000      |   1|
|     CONJ+DET+NOUN    | 0.6667   | 1.0000  |  0.8000    |     2|
| CONJ+PREP+DET+NOUN   |  1.0000  |  1.0000 |   1.0000  |       1|
|       PREP+PART    | 1.0000   | 0.0000  |  0.0000  |       2|
|      PART+V+PRON+NEG-PART    | 0.3333   | 0.3333  |  0.3333         | 3|
|          PART+V+NEG-PART    | 0.3333   | 0.5000  |  0.4000        | 2|
|      PART+PREP+NEG-PART    | 1.0000   | 1.0000  |  1.0000       |  3|
| PART+PROG-PART+V+NEG-PART    | 1.0000   | 0.3333   | 0.5000      |   3|
| PREP+DET+NOUN+NSUFF+PREP+PRON   |  1.0000  |  0.0000  |  0.0000    |     1|
|         PREP+PRON+DET+NOUN    | 0.0000   | 1.0000    | 0.0000   |      0|
|                PART+NSUFF    | 1.0000   | 0.0000    | 0.0000  |       1|
|    CONJ+PROG-PART+V+PRON    | 1.0000   | 1.0000   | 1.0000    |     1|
|          PART+PREP+PRON    | 1.0000   | 0.0000   | 0.0000   |      1|
|         CONJ+PART+PREP    | 1.0000   | 0.0000    | 0.0000        | 1|
|             NUM+NSUFF    | 0.6667   | 0.6667   | 0.6667        | 3|
| CONJ+PART+V+PRON+NEG-PART   |  1.0000  |  1.0000  |  1.0000      |   1|
|     PART+NOUN+NEG-PART    | 1.0000   | 1.0000   | 1.0000      |   1|
|        CONJ+ADJ+NSUFF     | 1.0000  |  0.0000  |  0.0000    |     1|
|             PREP+ADJ     | 1.0000  |  0.0000  |  0.0000   |      1|
|      ADJ+NSUFF+PRON     | 1.0000  |  0.0000  |  0.0000  |       2|
|   CONJ+PROG-PART+V    | 1.0000   | 0.0000   | 0.0000   |      1|
| CONJ+PART+PROG-PART+V+PREP+PRON+NEG-PART   |  1.0000  |  0.0000  |  0.0000 |        1|
|          CONJ+PART+PREP+PRON+NEG-PART    | 0.0000   | 1.0000   | 0.0000 |        0|
|                       PREP+PART+PRON    | 1.0000   | 0.0000   | 0.0000    |     1|
|                      CONJ+ADV+NSUFF    | 1.0000   | 0.0000    |0.0000   |      1|
|                           CONJ+ADV    | 0.0000   | 1.0000   | 0.0000  |       0|
|           PART+NOUN+PRON+NEG-PART    | 0.0000   | 1.0000  |  0.0000 |        0|
|                         CONJ+ADJ    | 1.0000   | 1.0000 |   1.0000 |         1|

</details>

- F-score (micro): 0.8974
- F-score (macro): 0.5188
- Accuracy (incl. no class): 0.901

Expand details below to show class scores for each tag. Note that tag compounds (a tag made for multiple agglutinated parts of speech) are considered as separate ones.

# Citation
*if you use this model in your work, please consider citing this work:*
```latex
@unpublished{MMHU21
author = "M. Megahed",
title = "Sequence Labeling Architectures in Diglossia",
note = "In review",
}
```


---
language: ar
license: apache-2.0
datasets:
- AQMAR
- ANERcorp
thumbnail: https://www.informatik.hu-berlin.de/en/forschung-en/gebiete/ml-en/resolveuid/a6f82e0d7fa446a59c902cac4cafa9cb/@@images/image/preview
tags:
- flair
- Text Classification
- token-classification
- sequence-tagger-model
metrics:
- f1
widget:
- text: "اختارها خيري بشارة كممثلة، دون سابقة معرفة أو تجربة تمثيلية، لتقف بجانب فاتن حمامة في فيلم «يوم مر ويوم حلو» (1988) وهي ما زالت شابة لم تتخطَ عامها الثاني"
---
# Arabic NER Model for AQMAR dataset
Training was conducted over 86 epochs, using a linear decaying learning rate of 2e-05, starting from 0.3 and a batch size of 48 with fastText and Flair forward and backward embeddings.

## Original Dataset:
- [AQMAR](http://www.cs.cmu.edu/~ark/ArabicNER/)

## Results:
- F1-score (micro) 0.9323
- F1-score (macro) 0.9272

|      | True Posititves  | False Positives | False Negatives | Precision | Recall | class-F1 |
|------|-----|----|----|---------|--------|----------|
| LOC  | 164 | 7  | 13 | 0.9591  | 0.9266 | 0.9425   |
| MISC | 398 | 22 | 37 |  0.9476 | 0.9149 | 0.9310   |
| ORG  | 65  | 6  | 9  | 0.9155  | 0.8784 | 0.8966   |
| PER  | 199 | 13 | 13 | 0.9387  | 0.9387 | 0.9387   |

---

# Usage
```python
from flair.data import Sentence
from flair.models import SequenceTagger
import pyarabic.araby as araby
from icecream import ic

arTagger = SequenceTagger.load('megantosh/flair-arabic-MSA-aqmar')

sentence = Sentence('George Washington went to Washington .')
arSentence = Sentence('عمرو عادلي أستاذ للاقتصاد السياسي المساعد في الجامعة الأمريكية  بالقاهرة .')

# predict NER tags
tagger.predict(sentence)
arTagger.predict(arSentence)

# print sentence with predicted tags
ic(sentence.to_tagged_string)
ic(arSentence.to_tagged_string)

```

# Example
see an example from a [similar NER model in Flair](https://huggingface.co/megantosh/flair-arabic-multi-ner)

# Model Configuration
```python
  (embeddings): StackedEmbeddings(
    (list_embedding_0): WordEmbeddings('ar')
    (list_embedding_1): FlairEmbeddings(
      (lm): LanguageModel(
        (drop): Dropout(p=0.1, inplace=False)
        (encoder): Embedding(7125, 100)
        (rnn): LSTM(100, 2048)
        (decoder): Linear(in_features=2048, out_features=7125, bias=True)
      )
    )
    (list_embedding_2): FlairEmbeddings(
      (lm): LanguageModel(
        (drop): Dropout(p=0.1, inplace=False)
        (encoder): Embedding(7125, 100)
        (rnn): LSTM(100, 2048)
        (decoder): Linear(in_features=2048, out_features=7125, bias=True)
      )
    )
  )
  (word_dropout): WordDropout(p=0.05)
  (locked_dropout): LockedDropout(p=0.5)
  (embedding2nn): Linear(in_features=4396, out_features=4396, bias=True)
  (rnn): LSTM(4396, 256, batch_first=True, bidirectional=True)
  (linear): Linear(in_features=512, out_features=14, bias=True)
  (beta): 1.0
  (weights): None
  (weight_tensor) None
)"
2021-03-31 22:19:50,654 ----------------------------------------------------------------------------------------------------
2021-03-31 22:19:50,654 Corpus: "Corpus: 3025 train + 336 dev + 373 test sentences"
2021-03-31 22:19:50,654 ----------------------------------------------------------------------------------------------------
2021-03-31 22:19:50,654 Parameters:
2021-03-31 22:19:50,654  - learning_rate: "0.3"
2021-03-31 22:19:50,654  - mini_batch_size: "48"
2021-03-31 22:19:50,654  - patience: "3"
2021-03-31 22:19:50,654  - anneal_factor: "0.5"
2021-03-31 22:19:50,654  - max_epochs: "150"
2021-03-31 22:19:50,654  - shuffle: "True"
2021-03-31 22:19:50,654  - train_with_dev: "False"
2021-03-31 22:19:50,654  - batch_growth_annealing: "False"
2021-03-31 22:19:50,655 ------------------------------------
 ```
Due to the right-to-left in left-to-right context, some formatting errors might occur. and your code might appear like [this](https://ibb.co/ky20Lnq), (link accessed on 2020-10-27)

 # Citation
*if you use this model in your work, please consider citing this work:*
```latex
@unpublished{MMHU21
author = "M. Megahed",
title = "Sequence Labeling Architectures in Diglossia",
note = "In Review",
}
```


---
language:
- ar
- en
license: apache-2.0
datasets:
- AQMAR
- ANERcorp
thumbnail: https://www.informatik.hu-berlin.de/en/forschung-en/gebiete/ml-en/resolveuid/a6f82e0d7fa446a59c902cac4cafa9cb/@@images/image/preview
tags:
- flair
- Text Classification
- token-classification
- sequence-tagger-model
metrics:
- f1
widget:
- text: أعرف كل شيء عن جيجي
- text: ترتقي شريحة M1 Pro وشريحة M1 Max ببنية شريحة M1 المذهلة إلى مستويات جديدة، إذ تأتيان للمرة الأولى ببنية نظام متكامل في شريحة (SoC) إلى جهاز نوت بوك للمحترفين.
- text: "اختارها خيري بشارة كممثلة، دون سابقة معرفة أو تجربة تمثيلية، لتقف بجانب فاتن حمامة في فيلم «يوم مر ويوم حلو» (1988) وهي ما زالت شابة لم تتخطَ عامها الثاني"
---
# Arabic NER Model using Flair Embeddings
Training was conducted over 94 epochs, using a linear decaying learning rate of 2e-05, starting from 0.225 and a batch size of 32 with GloVe and Flair forward and backward embeddings.

## Original Datasets:
- [AQMAR](http://www.cs.cmu.edu/~ark/ArabicNER/)
- [ANERcorp](http://curtis.ml.cmu.edu/w/courses/index.php/ANERcorp)

## Results:
- F1-score (micro) 0.8666
- F1-score (macro) 0.8488

|      | Named Entity Type    | True Posititves  | False Positives | False Negatives | Precision | Recall | class-F1 |
|------|-|----|----|----|-----------|--------|----------|
| LOC  | Location| 539 | 51 | 68 | 0.9136    | 0.8880 | 0.9006   |
| MISC | Miscellaneous|408 | 57 | 89 | 0.8774    | 0.8209 | 0.8482   |
| ORG  | Organisation|167 | 43 | 64 | 0.7952    | 0.7229 | 0.7574   |
| PER  | Person (no title)|501 | 65 | 60 | 0.8852    | 0.8930 | 0.8891   |

---

# Usage
```python
from flair.data import Sentence
from flair.models import SequenceTagger
import pyarabic.araby as araby
from icecream import ic

tagger = SequenceTagger.load("julien-c/flair-ner")
arTagger = SequenceTagger.load('megantosh/flair-arabic-multi-ner')

sentence = Sentence('George Washington went to Washington .')
arSentence = Sentence('عمرو عادلي أستاذ للاقتصاد السياسي المساعد في الجامعة الأمريكية  بالقاهرة .')

# predict NER tags
tagger.predict(sentence)
arTagger.predict(arSentence)

# print sentence with predicted tags
ic(sentence.to_tagged_string)
ic(arSentence.to_tagged_string)

```

# Example
```bash
2021-07-07 14:30:59,649 loading file /Users/mega/.flair/models/flair-ner/f22eb997f66ae2eacad974121069abaefca5fe85fce71b49e527420ff45b9283.941c7c30b38aef8d8a4eb5c1b6dd7fe8583ff723fef457382589ad6a4e859cfc
2021-07-07 14:31:04,654 loading file /Users/mega/.flair/models/flair-arabic-multi-ner/c7af7ddef4fdcc681fcbe1f37719348afd2862b12aa1cfd4f3b93bd2d77282c7.242d030cb106124f7f9f6a88fb9af8e390f581d42eeca013367a86d585ee6dd6
ic| sentence.to_tagged_string: <bound method Sentence.to_tagged_string of Sentence: "George Washington went to Washington ."   [− Tokens: 6  − Token-Labels: "George <B-PER> Washington <E-PER> went to Washington <S-LOC> ."]>
ic| arSentence.to_tagged_string: <bound method Sentence.to_tagged_string of Sentence: "عمرو عادلي أستاذ للاقتصاد السياسي المساعد في الجامعة الأمريكية بالقاهرة ."   [− Tokens: 11  − Token-Labels: "عمرو <B-PER> عادلي <I-PER> أستاذ للاقتصاد السياسي المساعد في الجامعة <B-ORG> الأمريكية <I-ORG> بالقاهرة <B-LOC> ."]>
ic| entity: <PER-span (1,2): "George Washington">
ic| entity: <LOC-span (5): "Washington">
ic| entity: <PER-span (1,2): "عمرو عادلي">
ic| entity: <ORG-span (8,9): "الجامعة الأمريكية">
ic| entity: <LOC-span (10): "بالقاهرة">
ic| sentence.to_dict(tag_type='ner'):
{"text":"عمرو عادلي أستاذ للاقتصاد السياسي المساعد في الجامعة الأمريكية  بالقاهرة .",
"labels":[],
{"entities":[{{{
               "text":"عمرو عادلي",
               "start_pos":0,
               "end_pos":10,
               "labels":[PER (0.9826)]},
            {"text":"الجامعة الأمريكية",
               "start_pos":45,
               "end_pos":62,
               "labels":[ORG (0.7679)]},
            {"text":"بالقاهرة",
               "start_pos":64,
               "end_pos":72,
               "labels":[LOC (0.8079)]}]}
"text":"George Washington went to Washington .",
"labels":[],
"entities":[{
           {"text":"George Washington",
            "start_pos":0,
            "end_pos":17,
            "labels":[PER (0.9968)]},
           {"text":"Washington""start_pos":26,
            "end_pos":36,
            "labels":[LOC (0.9994)]}}]}
```

# Model Configuration
```python
SequenceTagger(
  (embeddings): StackedEmbeddings(
    (list_embedding_0): WordEmbeddings('glove')
    (list_embedding_1): FlairEmbeddings(
      (lm): LanguageModel(
        (drop): Dropout(p=0.1, inplace=False)
        (encoder): Embedding(7125, 100)
        (rnn): LSTM(100, 2048)
        (decoder): Linear(in_features=2048, out_features=7125, bias=True)
      )
    )
    (list_embedding_2): FlairEmbeddings(
      (lm): LanguageModel(
        (drop): Dropout(p=0.1, inplace=False)
        (encoder): Embedding(7125, 100)
        (rnn): LSTM(100, 2048)
        (decoder): Linear(in_features=2048, out_features=7125, bias=True)
      )
    )
  )
  (word_dropout): WordDropout(p=0.05)
  (locked_dropout): LockedDropout(p=0.5)
  (embedding2nn): Linear(in_features=4196, out_features=4196, bias=True)
  (rnn): LSTM(4196, 256, batch_first=True, bidirectional=True)
  (linear): Linear(in_features=512, out_features=15, bias=True)
  (beta): 1.0
  (weights): None
  (weight_tensor) None
 ```
 Due to the right-to-left in left-to-right context, some formatting errors might occur. and your code might appear like [this](https://ibb.co/ky20Lnq), (link accessed on 2020-10-27)

 # Citation
*if you use this model in your work, please consider citing this work:*
```latex
@unpublished{MMHU21
author = "M. Megahed",
title = "Sequence Labeling Architectures in Diglossia",
note = "In preparation",
}
```


---
tags:
- conversational
---

# harry potter DialogGPT Model


---
tags:
- image-classification
- pytorch
- huggingpics
metrics:
- accuracy

model-index:
- name: ice_cream
  results:
  - task:
      name: Image Classification
      type: image-classification
    metrics:
      - name: Accuracy
        type: accuracy
        value: 0.5166666507720947
---

# ice_cream

Autogenerated by HuggingPics🤗🖼

Create your own image classifier for **anything** by running [the demo on Google Colab](https://colab.research.google.com/github/nateraw/huggingpics/blob/main/HuggingPics.ipynb).

Report any issues with the demo at the [github repo](https://github.com/nateraw/huggingpics).

## Example Images

#### chocolate ice cream

![chocolate ice cream](images/chocolate_ice_cream.jpg)

#### vanilla ice cream

![vanilla ice cream](images/vanilla_ice_cream.jpg)


基于有赞商品评论语料训练的bert模型。


基于有赞商品标题语料训练的bert模型。


clip-product-title-chinese

## 基于有赞商品图片和标题语料训练的clip模型。

## Usage
使用模型前，请 git clone https://github.com/youzanai/trexpark.git

```python
import torch
from src.clip.clip import ClipProcesserChinese, ClipChineseModel
import requests
from PIL import Image

clip_processor = ClipProcesserChinese.from_pretrained('youzanai/clip-product-title-chinese')
model = ClipChineseModel.from_pretrained('youzanai/clip-product-title-chinese')

url = 'http://img.yzcdn.cn/upload_files/2015/04/21/0140dac4657f874f2acff9294b28088c.jpg'
img = Image.open(requests.get(url, stream=True).raw).convert('RGB')
imgs = [img]
texts = ['运动鞋', '红色连衣裙', '黑色连衣裙', '大衣', '文具']

f = clip_processor(texts, imgs, return_tensors='pt', truncation=True, padding=True)
del f['token_type_ids']
with torch.no_grad():
    out = model(**f)
logits_per_image, logits_per_text = out['logits_per_image'], out['logits_per_text']

print(logits_per_image.softmax(dim=-1).cpu().detach().numpy())

# 结果： [[1.1700666e-07 9.9948394e-01 5.1582896e-04 4.7687358e-11 6.9604440e-08]]
```


---
language: fr
license: mit
datasets:
- flaubert
metrics:
- flue
tags:
- bert
- language-model
- flaubert
- flue
- french
- flaubert-small
- cased
---

# FlauBERT: Unsupervised Language Model Pre-training for French

**FlauBERT** is a French BERT trained on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) [Jean Zay](http://www.idris.fr/eng/jean-zay/ ) supercomputer.

Along with FlauBERT comes [**FLUE**](https://github.com/getalp/Flaubert/tree/master/flue): an evaluation setup for French NLP systems similar to the popular GLUE benchmark. The goal is to enable further reproducible experiments in the future and to share models and progress on the French language.For more details please refer to the [official website](https://github.com/getalp/Flaubert).

## FlauBERT models

| Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters |
| :------:       |   :---: | :---: | :---: | :---: |
| `flaubert-small-cased` | 6    | 8    | 512   | 54 M |
| `flaubert-base-uncased`  | 12  | 12  | 768  | 137 M |
| `flaubert-base-cased`   | 12   | 12      | 768   | 138 M |
| `flaubert-large-cased`  | 24   | 16     | 1024 | 373 M |

**Note:** `flaubert-small-cased` is partially trained so performance is not guaranteed. Consider using it for debugging purpose only.

## Using FlauBERT with Hugging Face's Transformers

```python
import torch
from transformers import FlaubertModel, FlaubertTokenizer

# Choose among ['flaubert/flaubert_small_cased', 'flaubert/flaubert_base_uncased',
#               'flaubert/flaubert_base_cased', 'flaubert/flaubert_large_cased']
modelname = 'flaubert/flaubert_base_cased'

# Load pretrained model and tokenizer
flaubert, log = FlaubertModel.from_pretrained(modelname, output_loading_info=True)
flaubert_tokenizer = FlaubertTokenizer.from_pretrained(modelname, do_lowercase=False)
# do_lowercase=False if using cased models, True if using uncased ones

sentence = "Le chat mange une pomme."
token_ids = torch.tensor([flaubert_tokenizer.encode(sentence)])

last_layer = flaubert(token_ids)[0]
print(last_layer.shape)
# torch.Size([1, 8, 768])  -> (batch size x number of tokens x embedding dimension)

# The BERT [CLS] token correspond to the first hidden state of the last layer
cls_embedding = last_layer[:, 0, :]
```

**Notes:** if your `transformers` version is <=2.10.0, `modelname` should take one
of the following values:

```
['flaubert-small-cased', 'flaubert-base-uncased', 'flaubert-base-cased', 'flaubert-large-cased']
```

## References

If you use FlauBERT or the FLUE Benchmark for your scientific publication, or if you find the resources in this repository useful, please cite one of the following papers:

[LREC paper](http://www.lrec-conf.org/proceedings/lrec2020/pdf/2020.lrec-1.302.pdf)
```
@InProceedings{le2020flaubert,
  author    = {Le, Hang  and  Vial, Lo\"{i}c  and  Frej, Jibril  and  Segonne, Vincent  and  Coavoux, Maximin  and  Lecouteux, Benjamin  and  Allauzen, Alexandre  and  Crabb\'{e}, Beno\^{i}t  and  Besacier, Laurent  and  Schwab, Didier},
  title     = {FlauBERT: Unsupervised Language Model Pre-training for French},
  booktitle = {Proceedings of The 12th Language Resources and Evaluation Conference},
  month     = {May},
  year      = {2020},
  address   = {Marseille, France},
  publisher = {European Language Resources Association},
  pages     = {2479--2490},
  url       = {https://www.aclweb.org/anthology/2020.lrec-1.302}
}
```

[TALN paper](https://hal.archives-ouvertes.fr/hal-02784776/)
```
@inproceedings{le2020flaubert,
  title         = {FlauBERT: des mod{\`e}les de langue contextualis{\'e}s pr{\'e}-entra{\^\i}n{\'e}s pour le fran{\c{c}}ais},
  author        = {Le, Hang and Vial, Lo{\"\i}c and Frej, Jibril and Segonne, Vincent and Coavoux, Maximin and Lecouteux, Benjamin and Allauzen, Alexandre and Crabb{\'e}, Beno{\^\i}t and Besacier, Laurent and Schwab, Didier},
  booktitle     = {Actes de la 6e conf{\'e}rence conjointe Journ{\'e}es d'{\'E}tudes sur la Parole (JEP, 31e {\'e}dition), Traitement Automatique des Langues Naturelles (TALN, 27e {\'e}dition), Rencontre des {\'E}tudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (R{\'E}CITAL, 22e {\'e}dition). Volume 2: Traitement Automatique des Langues Naturelles},
  pages         = {268--278},
  year          = {2020},
  organization  = {ATALA}
}
```


---
language: fr
license: mit
datasets:
- flaubert
metrics:
- flue
tags:
- bert
- language-model
- flaubert
- flue
- french
- bert-base
- flaubert-base
- cased
---

# FlauBERT: Unsupervised Language Model Pre-training for French

**FlauBERT** is a French BERT trained on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) [Jean Zay](http://www.idris.fr/eng/jean-zay/ ) supercomputer.

Along with FlauBERT comes [**FLUE**](https://github.com/getalp/Flaubert/tree/master/flue): an evaluation setup for French NLP systems similar to the popular GLUE benchmark. The goal is to enable further reproducible experiments in the future and to share models and progress on the French language.For more details please refer to the [official website](https://github.com/getalp/Flaubert).

## FlauBERT models

| Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters |
| :------:       |   :---: | :---: | :---: | :---: |
| `flaubert-small-cased` | 6    | 8    | 512   | 54 M |
| `flaubert-base-uncased`  | 12  | 12  | 768  | 137 M |
| `flaubert-base-cased`   | 12   | 12      | 768   | 138 M |
| `flaubert-large-cased`  | 24   | 16     | 1024 | 373 M |

**Note:** `flaubert-small-cased` is partially trained so performance is not guaranteed. Consider using it for debugging purpose only.

## Using FlauBERT with Hugging Face's Transformers

```python
import torch
from transformers import FlaubertModel, FlaubertTokenizer

# Choose among ['flaubert/flaubert_small_cased', 'flaubert/flaubert_base_uncased',
#               'flaubert/flaubert_base_cased', 'flaubert/flaubert_large_cased']
modelname = 'flaubert/flaubert_base_cased'

# Load pretrained model and tokenizer
flaubert, log = FlaubertModel.from_pretrained(modelname, output_loading_info=True)
flaubert_tokenizer = FlaubertTokenizer.from_pretrained(modelname, do_lowercase=False)
# do_lowercase=False if using cased models, True if using uncased ones

sentence = "Le chat mange une pomme."
token_ids = torch.tensor([flaubert_tokenizer.encode(sentence)])

last_layer = flaubert(token_ids)[0]
print(last_layer.shape)
# torch.Size([1, 8, 768])  -> (batch size x number of tokens x embedding dimension)

# The BERT [CLS] token correspond to the first hidden state of the last layer
cls_embedding = last_layer[:, 0, :]
```

**Notes:** if your `transformers` version is <=2.10.0, `modelname` should take one
of the following values:

```
['flaubert-small-cased', 'flaubert-base-uncased', 'flaubert-base-cased', 'flaubert-large-cased']
```

## References

If you use FlauBERT or the FLUE Benchmark for your scientific publication, or if you find the resources in this repository useful, please cite one of the following papers:

[LREC paper](http://www.lrec-conf.org/proceedings/lrec2020/pdf/2020.lrec-1.302.pdf)
```
@InProceedings{le2020flaubert,
  author    = {Le, Hang  and  Vial, Lo\"{i}c  and  Frej, Jibril  and  Segonne, Vincent  and  Coavoux, Maximin  and  Lecouteux, Benjamin  and  Allauzen, Alexandre  and  Crabb\'{e}, Beno\^{i}t  and  Besacier, Laurent  and  Schwab, Didier},
  title     = {FlauBERT: Unsupervised Language Model Pre-training for French},
  booktitle = {Proceedings of The 12th Language Resources and Evaluation Conference},
  month     = {May},
  year      = {2020},
  address   = {Marseille, France},
  publisher = {European Language Resources Association},
  pages     = {2479--2490},
  url       = {https://www.aclweb.org/anthology/2020.lrec-1.302}
}
```

[TALN paper](https://hal.archives-ouvertes.fr/hal-02784776/)
```
@inproceedings{le2020flaubert,
  title         = {FlauBERT: des mod{\`e}les de langue contextualis{\'e}s pr{\'e}-entra{\^\i}n{\'e}s pour le fran{\c{c}}ais},
  author        = {Le, Hang and Vial, Lo{\"\i}c and Frej, Jibril and Segonne, Vincent and Coavoux, Maximin and Lecouteux, Benjamin and Allauzen, Alexandre and Crabb{\'e}, Beno{\^\i}t and Besacier, Laurent and Schwab, Didier},
  booktitle     = {Actes de la 6e conf{\'e}rence conjointe Journ{\'e}es d'{\'E}tudes sur la Parole (JEP, 31e {\'e}dition), Traitement Automatique des Langues Naturelles (TALN, 27e {\'e}dition), Rencontre des {\'E}tudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (R{\'E}CITAL, 22e {\'e}dition). Volume 2: Traitement Automatique des Langues Naturelles},
  pages         = {268--278},
  year          = {2020},
  organization  = {ATALA}
}
```


---
language: fr
license: mit
datasets:
- flaubert
metrics:
- flue
tags:
- bert
- language-model
- flaubert
- flue
- french
- bert-large
- flaubert-large
- cased
---

# FlauBERT: Unsupervised Language Model Pre-training for French

**FlauBERT** is a French BERT trained on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) [Jean Zay](http://www.idris.fr/eng/jean-zay/ ) supercomputer.

Along with FlauBERT comes [**FLUE**](https://github.com/getalp/Flaubert/tree/master/flue): an evaluation setup for French NLP systems similar to the popular GLUE benchmark. The goal is to enable further reproducible experiments in the future and to share models and progress on the French language.For more details please refer to the [official website](https://github.com/getalp/Flaubert).

## FlauBERT models

| Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters |
| :------:       |   :---: | :---: | :---: | :---: |
| `flaubert-small-cased` | 6    | 8    | 512   | 54 M |
| `flaubert-base-uncased`  | 12  | 12  | 768  | 137 M |
| `flaubert-base-cased`   | 12   | 12      | 768   | 138 M |
| `flaubert-large-cased`  | 24   | 16     | 1024 | 373 M |

**Note:** `flaubert-small-cased` is partially trained so performance is not guaranteed. Consider using it for debugging purpose only.

## Using FlauBERT with Hugging Face's Transformers

```python
import torch
from transformers import FlaubertModel, FlaubertTokenizer

# Choose among ['flaubert/flaubert_small_cased', 'flaubert/flaubert_base_uncased',
#               'flaubert/flaubert_base_cased', 'flaubert/flaubert_large_cased']
modelname = 'flaubert/flaubert_base_cased'

# Load pretrained model and tokenizer
flaubert, log = FlaubertModel.from_pretrained(modelname, output_loading_info=True)
flaubert_tokenizer = FlaubertTokenizer.from_pretrained(modelname, do_lowercase=False)
# do_lowercase=False if using cased models, True if using uncased ones

sentence = "Le chat mange une pomme."
token_ids = torch.tensor([flaubert_tokenizer.encode(sentence)])

last_layer = flaubert(token_ids)[0]
print(last_layer.shape)
# torch.Size([1, 8, 768])  -> (batch size x number of tokens x embedding dimension)

# The BERT [CLS] token correspond to the first hidden state of the last layer
cls_embedding = last_layer[:, 0, :]
```

**Notes:** if your `transformers` version is <=2.10.0, `modelname` should take one
of the following values:

```
['flaubert-small-cased', 'flaubert-base-uncased', 'flaubert-base-cased', 'flaubert-large-cased']
```

## References

If you use FlauBERT or the FLUE Benchmark for your scientific publication, or if you find the resources in this repository useful, please cite one of the following papers:

[LREC paper](http://www.lrec-conf.org/proceedings/lrec2020/pdf/2020.lrec-1.302.pdf)
```
@InProceedings{le2020flaubert,
  author    = {Le, Hang  and  Vial, Lo\"{i}c  and  Frej, Jibril  and  Segonne, Vincent  and  Coavoux, Maximin  and  Lecouteux, Benjamin  and  Allauzen, Alexandre  and  Crabb\'{e}, Beno\^{i}t  and  Besacier, Laurent  and  Schwab, Didier},
  title     = {FlauBERT: Unsupervised Language Model Pre-training for French},
  booktitle = {Proceedings of The 12th Language Resources and Evaluation Conference},
  month     = {May},
  year      = {2020},
  address   = {Marseille, France},
  publisher = {European Language Resources Association},
  pages     = {2479--2490},
  url       = {https://www.aclweb.org/anthology/2020.lrec-1.302}
}
```

[TALN paper](https://hal.archives-ouvertes.fr/hal-02784776/)
```
@inproceedings{le2020flaubert,
  title         = {FlauBERT: des mod{\`e}les de langue contextualis{\'e}s pr{\'e}-entra{\^\i}n{\'e}s pour le fran{\c{c}}ais},
  author        = {Le, Hang and Vial, Lo{\"\i}c and Frej, Jibril and Segonne, Vincent and Coavoux, Maximin and Lecouteux, Benjamin and Allauzen, Alexandre and Crabb{\'e}, Beno{\^\i}t and Besacier, Laurent and Schwab, Didier},
  booktitle     = {Actes de la 6e conf{\'e}rence conjointe Journ{\'e}es d'{\'E}tudes sur la Parole (JEP, 31e {\'e}dition), Traitement Automatique des Langues Naturelles (TALN, 27e {\'e}dition), Rencontre des {\'E}tudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (R{\'E}CITAL, 22e {\'e}dition). Volume 2: Traitement Automatique des Langues Naturelles},
  pages         = {268--278},
  year          = {2020},
  organization  = {ATALA}
}
```