- en
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- boolq
metrics:
- accuracy
model_index:
- name: distilbert-base-uncased-boolq
results:
  - task:
name: Question Answering
type: question-answering
dataset:
name: boolq
type: boolq
args: default
metric:
name: Accuracy
type: accuracy
value: 0.7314984709480122


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-boolq


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the boolq dataset.
It achieves the following results on the evaluation set:


- Loss: 1.2071
- Accuracy: 0.7315


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 0.6506        | 1.0   | 531  | 0.6075          | 0.6681   |
| 0.575         | 2.0   | 1062 | 0.5816          | 0.6978   |
| 0.4397        | 3.0   | 1593 | 0.6137          | 0.7253   |
| 0.2524        | 4.0   | 2124 | 0.8124          | 0.7466   |
| 0.126         | 5.0   | 2655 | 1.1437          | 0.7370   |


### Framework versions


- Transformers 4.8.2
- Pytorch 1.8.1+cu111
- Datasets 1.8.0
- Tokenizers 0.10.3


model name: distilbert-base-uncased-qa-boolq


model author: andi611


task: text-classification


tags: pytorch, distilbert, text-classification, en, dataset:boolq, transformers, license:apache-2.0, generated_from_trainer, infinity_compatible


tags:


- generated_from_trainer
datasets:
- conll2003
model_index:
- name: distilbert-base-uncased-squad2-with-ner-with-neg-with-repeat
results:
  - task:
name: Question Answering
type: question-answering
dataset:
name: conll2003
type: conll2003
args: conll2003


# distilbert-base-uncased-squad2-with-ner-with-neg-with-repeat


This model is a fine-tuned version of [twmkn9/distilbert-base-uncased-squad2](https://huggingface.co/twmkn9/distilbert-base-uncased-squad2) on the conll2003 dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


### Framework versions


model name: distilbert-base-uncased-squad2-with-ner-with-neg-with-repeat


model author: andi611


task: question-answering


tags: pytorch, distilbert, question-answering, dataset:conll2003, transformers, generated_from_trainer, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- conll2003
metrics:
- precision
- recall
- f1
- accuracy
model_index:
- name: distilbert-base-uncased-ner
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: conll2003
type: conll2003
args: conll2003
metric:
name: Accuracy
type: accuracy
value: 0.985193893275295


# distilbert-base-uncased-ner


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the conll2003 dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0664
- Precision: 0.9332
- Recall: 0.9423
- F1: 0.9377
- Accuracy: 0.9852


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 3e-05
- train_batch_size: 16
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 0.2042        | 1.0   | 878  | 0.0636          | 0.9230    | 0.9253 | 0.9241 | 0.9822   |
| 0.0428        | 2.0   | 1756 | 0.0577          | 0.9286    | 0.9370 | 0.9328 | 0.9841   |
| 0.0199        | 3.0   | 2634 | 0.0606          | 0.9364    | 0.9401 | 0.9383 | 0.9851   |
| 0.0121        | 4.0   | 3512 | 0.0641          | 0.9339    | 0.9380 | 0.9360 | 0.9847   |
| 0.0079        | 5.0   | 4390 | 0.0664          | 0.9332    | 0.9423 | 0.9377 | 0.9852   |


### Framework versions


model name: distilbert-base-uncased-ner-conll2003


model author: andi611


task: token-classification


tags: pytorch, tensorboard, distilbert, token-classification, dataset:conll2003, transformers, license:apache-2.0, generated_from_trainer, autonlp_compatible, infinity_compatible


# ArcheoBERTje


A Dutch BERT model for the Archaeology domain


This model is based on the Dutch BERTje model by wietsedv (https://github.com/wietsedv/bertje).


We further finetuned BERTje with a corpus of roughly 60k Dutch excavation reports (~650 million tokens) from the DANS data archive (https://easy.dans.knaw.nl/ui/home).


model name: ArcheoBERTje


model author: alexbrandsen


task: fill-mask


tags: pytorch, jax, bert, fill-mask, transformers, autonlp_compatible, infinity_compatible


# ArcheoBERTje-NER


A Dutch BERT model for Named Entity Recognition in the Archaeology domain


This is the [ArcheoBERTje](https://huggingface.co/alexbrandsen/ArcheoBERTje) model finetuned for NER, targeting the following entities:


- Time periods
- Places
- Artefacts
- Contexts
- Materials
- Species


model name: ArcheoBERTje-NER


tags: pytorch, jax, bert, token-classification, transformers, autonlp_compatible, infinity_compatible


license: mit
tags:


- generated_from_trainer
datasets:
- glue
metrics:
- matthews_correlation
model_index:
- name: roberta-base-finetuned-cola
results:
  - task:
name: Text Classification
type: text-classification
dataset:
name: glue
type: glue
args: cola
metric:
name: Matthews Correlation
type: matthews_correlation
value: 0.557882735147727


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# roberta-base-finetuned-cola


This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the glue dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4716
- Matthews Correlation: 0.5579


## Model description


More information needed


## Intended uses & limitations


```python
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("jxuhf/roberta-base-finetuned-cola")
```


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2


### Training results


| Training Loss | Epoch | Step | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------:|
| 0.4981        | 1.0   | 535  | 0.5162          | 0.5081               |
| 0.314         | 2.0   | 1070 | 0.4716          | 0.5579               |


### Framework versions


- Transformers 4.9.0
- Pytorch 1.9.0+cu102
- Datasets 1.10.2
- Tokenizers 0.10.3


model name: roberta-base-finetuned-cola


model author: jxuhf


task: text-classification


tags: pytorch, tensorboard, roberta, text-classification, dataset:glue, transformers, license:mit, generated_from_trainer, infinity_compatible


language: de
datasets:


- common_voice
metrics:
- wer
- cer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 German with LM by Florian Zimmermeister
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice de
type: common_voice
args: de
metrics:
    - name: Test WER
type: wer
value: 5.7467896819046755
    - name: Test CER
type: cer
value: 1.8980142607670552


**Test Result**


| Model | WER | CER |
| ------------- | ------------- | ------------- |
| flozi00/wav2vec2-large-xlsr-53-german-with-lm | **5.7467896819046755%** | **1.8980142607670552%** |


## Evaluation


The model can be evaluated as follows on the German test data of Common Voice.


```python
import torchaudio.functional as F
import torch
from transformers import AutoModelForCTC, AutoProcessor
import re
from datasets import load_dataset, load_metric

CHARS_TO_IGNORE = [",", "?", "¿", ".", "!", "¡", ";", "；", ":", '""', "%", '"', "�", "ʿ", "·", "჻", "~", "՞",
                   "؟", "،", "।", "॥", "«", "»", "„", "“", "”", "「", "」", "‘", "’", "《", "》", "(", ")", "[", "]",
                   "{", "}", "=", "`", "_", "+", "<", ">", "…", "–", "°", "´", "ʾ", "‹", "›", "©", "®", "—", "→", "。",
                   "、", "﹂", "﹁", "‧", "～", "﹏", "，", "｛", "｝", "（", "）", "［", "］", "【", "】", "‥", "〽",
                   "『", "』", "〝", "〟", "⟨", "⟩", "〜", "：", "！", "？", "♪", "؛", "/", "\\", "º", "−", "^", "ʻ", "ˆ"]

chars_to_ignore_regex = f"[{re.escape(''.join(CHARS_TO_IGNORE))}]"

counter = 0
wer_counter = 0
cer_counter = 0

def main():
    model = AutoModelForCTC.from_pretrained("flozi00/wav2vec2-large-xlsr-53-german-with-lm")
    processor = AutoProcessor.from_pretrained("flozi00/wav2vec2-large-xlsr-53-german-with-lm")

    wer = load_metric("wer")
    cer = load_metric("cer")

    ds = load_dataset("common_voice", "de", split="test")
    #ds = ds.select(range(100))

    def calculate_metrics(batch):
        global counter, wer_counter, cer_counter
        resampled_audio = F.resample(torch.tensor(batch["audio"]["array"]), 48_000, 16_000).numpy()

        input_values = processor(resampled_audio, return_tensors="pt", sampling_rate=16_000).input_values

        with torch.no_grad():
            logits = model(input_values).logits.numpy()[0]

        decoded = processor.decode(logits)
        pred = decoded.text

        ref = re.sub(chars_to_ignore_regex, "", batch["sentence"]).upper()

        wer_result = wer.compute(predictions=[pred], references=[ref])
        cer_result = cer.compute(predictions=[pred], references=[ref])

        counter += 1
        wer_counter += wer_result
        cer_counter += cer_result

        print(f"WER: {(wer_counter/counter)*100} | CER: {(cer_counter/counter)*100}")

        return batch

    ds.map(calculate_metrics, remove_columns=ds.column_names)

main()
```


Credits:


The Acoustic model is an copy of [jonatasgrosman's model](https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-german) I used to train an matching kenlm language model for


model name: wav2vec2-large-xlsr-53-german-with-lm


model author: flozi00


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, de, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


- common_voice
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 German by Florian Zimmermeister
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice de
type: common_voice
args: de\
metrics:
    - name: Test WER
type: wer
value: 15.36


# Wav2Vec2-Large-XLSR-53-German


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) in German using the [Common Voice](https://huggingface.co/datasets/common_voice)
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "de", split="test[:2%]").
processor = Wav2Vec2Processor.from_pretrained("flozi00/wav2vec-xlsr-german")
model = Wav2Vec2ForCTC.from_pretrained("flozi00/wav2vec-xlsr-german")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


The model can be evaluated as follows on the {language} test data of Common Voice.


```python
import datasets
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import soundfile as sf
import torch
from jiwer import wer
import librosa
import re
import json
import time

chars_to_ignore_regex = '[\\\\,\\\\?\\\\.\\\\!\\\\-\\\\;\\\\:\\\\"\\\\“\\\\%\\\\‘\\\\”\\\\�]'
pattern = re.compile("[A-Za-z0-9 ]+")

#try:
#    os.remove("dataset.json")
#except:
#    pass

modelName = "flozi00/wav2vec-xlsr-german"
model = Wav2Vec2ForCTC.from_pretrained(modelName).to("cuda")
tokenizer = Wav2Vec2Processor.from_pretrained(modelName)

print("loading and cleaning")
val_dataset = datasets.load_dataset("common_voice","de", split="validation")
val_dataset = val_dataset.remove_columns(["client_id","up_votes","down_votes","age","gender","accent","locale","segment"])
val_dataset = val_dataset.rename_column("path","file")
val_dataset = val_dataset.rename_column("sentence","text")
#val_dataset = val_dataset.filter(lambda example, indice: indice % 30 == 0, with_indices=True)

print(val_dataset)

def map_to_array(batch):
    batch["text"] = re.sub(chars_to_ignore_regex, '', batch["text"]).upper() + " "
    try:
        speech_array, sampling_rate = sf.read(batch["file"]+ ".wav")
    except:
        speech_array, sampling_rate = librosa.load(batch["file"], sr=16000, res_type='kaiser_fast')
        sf.write(batch["file"] + ".wav", speech_array, sampling_rate, subtype='PCM_24')
    batch["speech"] = speech_array
    return batch

print("reading audio")
start = time.time()
val_dataset = val_dataset.map(map_to_array)
print(time.time()-start)
val_dataset = val_dataset.filter(lambda example: True if pattern.fullmatch(example["text"]) is not None else False)

def map_to_pred(batch):
    inputs = tokenizer(batch["speech"], return_tensors="pt", padding="longest")
    input_values = inputs.input_values.to("cuda")
    attention_mask = inputs.attention_mask.to("cuda")

    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits

    predicted_ids = torch.argmax(logits, dim=-1)
    transcription = tokenizer.batch_decode(predicted_ids)
    batch["transcription"] = transcription
    return batch

print("predicting")
result = val_dataset.map(map_to_pred, batched=True, batch_size=32, remove_columns=["speech"])

print("WER:", wer(result["text"], result["transcription"]))

```


**Test Result**: 15.36 %


model name: wav2vec-xlsr-german


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, de, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


license: cc-by-nc-4.0
tags:


- generated_from_trainer
metrics:
- accuracy
model_index:
name: wav2vec2-speechcommands-german


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-speechcommands-german


This model is a fine-tuned version of [facebook/wav2vec2-base-10k-voxpopuli-ft-de](https://huggingface.co/facebook/wav2vec2-base-10k-voxpopuli-ft-de) on an unkown dataset.
It achieves the following results on the evaluation set:


- Loss: 0.2481
- Accuracy: 0.9180


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0
- mixed_precision_training: Native AMP


### Training results


### Framework versions


- Transformers 4.9.1
- Pytorch 1.8.1
- Datasets 1.7.0
- Tokenizers 0.10.2


model name: wav2vec2-speechcommands-german


model author: flozi00


tags: pytorch, tensorboard, wav2vec2, transformers, license:cc-by-nc-4.0, generated_from_trainer


language: de
tags:


- grammar
widget:
- text: "correct german grammar: es ist schön so viele tolle menschen um sich zu haben denn ohne sie wäre es nicht so schön"


example outputs:


input: ich liebe das leben --> output: Ich liebe das Leben.


input: es ist schön so viele tolle menschen um sich zu haben denn ohne sie wäre es nicht so schön --> output: Es ist schön, so viele tolle Menschen, um sich zu haben, denn ohne sie wäre es nicht so schön.


input: der kunde hat ausdrücklich nach dirk verlangt weil er den rabatt haben möchte --> output: Der Kunde hat ausdrücklich nach Dirk verlangt, weil er den Rabatt haben möchte.


the data can be prepared like this:
the broken_text is used as input, while the text is the output


```python

import re
import phonetics
import random

chars_to_ignore_regex = "[^A-Za-z0-9\ö\ä\ü\Ö\Ä\Ü\ß\-,;.:?! ]+"
broken_chars_to_ignore_regex = "[^A-Za-z0-9\ö\ä\ü\Ö\Ä\Ü\ß\- ]+"

def do_manipulation(string):
    text = re.sub(chars_to_ignore_regex, '', string)
    broken_text = re.sub(broken_chars_to_ignore_regex, "", text.lower())

    if(random.randint(0,100) >= 50):
        for xyz in range(int(len(broken_text.split(" "))/4)):
            if(random.randint(0,100) > 30):
                randc = random.choice(broken_text.split(" "))
                if(random.randint(0,10) > 4):
                    broken_text = broken_text.replace(randc, ''.join(random.choice('abcdefghijklmnopqrstuvxyz') for _ in range(len(randc))).lower())
                else:
                    broken_text = broken_text.replace(randc, phonetics.metaphone(randc).lower())

    return text, broken_text

```


model name: byt5-german-grammar


task: text2text-generation


tags: pytorch, t5, text2text-generation, de, transformers, grammar, autonlp_compatible


language:


- input
- output
tags:
- generated_from_trainer
model-index:
- name: t5-skills
results: []


# t5-skills


This model is a fine-tuned version of [flozi00/t5-skills](https://huggingface.co/flozi00/t5-skills) on the None dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 5e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1.0


### Training results


### Framework versions


- Transformers 4.12.5
- Pytorch 1.8.1
- Datasets 1.14.0
- Tokenizers 0.10.2


model name: t5-skills


model author: flozi00


task: text2text-generation


tags: pytorch, tensorboard, t5, text2text-generation, input, output, transformers, generated_from_trainer, model-index, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- glue
metrics:
- matthews_correlation
model-index:
- name: distilbert-base-uncased-finetuned-cola
results:
  - task:
name: Text Classification
type: text-classification
dataset:
name: glue
type: glue
args: cola
metrics:
    - name: Matthews Correlation
type: matthews_correlation
value: 0.5373281885173845


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# distilbert-base-uncased-finetuned-cola


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the glue dataset.
It achieves the following results on the evaluation set:


- Loss: 0.7637
- Matthews Correlation: 0.5373


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------:|
| 0.5306        | 1.0   | 535  | 0.5156          | 0.4063               |
| 0.3524        | 2.0   | 1070 | 0.5249          | 0.5207               |
| 0.2417        | 3.0   | 1605 | 0.6514          | 0.5029               |
| 0.1762        | 4.0   | 2140 | 0.7637          | 0.5373               |
| 0.1252        | 5.0   | 2675 | 0.8746          | 0.5291               |


### Framework versions


- Transformers 4.13.0
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-cola


model author: jery33


task: text-classification


tags: pytorch, tensorboard, distilbert, text-classification, dataset:glue, transformers, license:apache-2.0, generated_from_trainer, model-index, infinity_compatible


tags:


- conversational


# Dialogue From Persona 3


model name: DialoGPTChatbot


model author: LaiJY


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


# Jesse (Breaking Bad) DialoGPT Model


model name: DialoGPT-small-jesse


model author: Canadiancaleb


# Walter (Breaking Bad) DialoGPT Model


model name: DialoGPT-small-walter


- question-answering
datasets:
- squad
- anukaver/EstQA


# Question answering model for Estonian


This is a question answering model based on XLM-Roberta base model. It is fine-tuned subsequentially on:


1. English SQuAD v1.1
1. SQuAD v1.1 translated into Estonian
1. Small native Estonian dataset (800 samples)


The model has retained good multilingual properties and can be used for extractive QA tasks in all languages included in XLM-Roberta. The performance is best in the fine-tuning languages of Estonian and English.


| Tested on | F1 | EM |
| ----------- | --- | --- |
| EstQA test set | 82.4 | 75.3 |
| SQuAD v1.1 dev set | 86.9 | 77.9 |


The Estonian dataset used for fine-tuning and validating results is available in https://huggingface.co/datasets/anukaver/EstQA/ (version 1.0)


model name: xlm-roberta-est-qa


model author: anukaver


task: question-answering


tags: pytorch, xlm-roberta, question-answering, dataset:squad, dataset:anukaver/EstQA, transformers, autonlp_compatible


language:


- hu
tags:
- token-classification
license: gpl
metrics:
- F1
widget:
- text: "A jótékonysági szervezet által idézett Forbes-adatok szerint a világ tíz leggazdagabb embere: Elon Musk (Tesla, SpaceX), Jeff Bezos (Amazon, Blue Origin), Bernard Arnault és családja (LVMH, azaz Louis Vuitton és Moët Hennessy), Bill Gates (Microsoft), Larry Ellison (Oracle), Larry Page (Google), Sergey Brin (Google), Mark Zuckerberg (Facebook), Steve Ballmer (Microsoft) és Warren Buffett (befektető).
Miközben vagyonuk együttesen 700 milliárdról másfél ezer milliárd dollárra nőtt 2020 márciusa és 2021 novembere között, jelentős eltérések vannak közöttük: Musk vagyona több mint 1000 százalékos, míg Gatesé szerényebb, 30 százalékos növekedést mutatott."
inference:
parameters:
aggregation_strategy: "first"


# Hungarian named entity recognition model with four entity types: PER ORG LOC MISC


- Pretrained model used: SZTAKI-HLT/hubert-base-cc
- Finetuned on NYTK-NerKor Corpus


## Limitations


- max_seq_length = 448


model name: nerkor-hubert


model author: novakat


task: token-classification


tags: pytorch, bert, hu, transformers, token-classification, license:gpl, autonlp_compatible, infinity_compatible


# Hungarian named entity recognition model with OntoNotes5 + more entity types


- Pretrained model used: SZTAKI-HLT/hubert-base-cc
- Finetuned on NerKor+CARS-ONPP Corpus


## Training data


The underlying corpus, [NerKor+CARS-ONPP](https://github.com/novakat/NYTK-NerKor-Cars-OntoNotesPP), was derived from [NYTK-NerKor](https://github.com/nytud/NYTK-NerKor), a Hungarian gold standard named entity annotated corpus containing about 1 million tokens.
It includes a small addition of 12k tokens of text (individual sentences) concerning motor vehicles (cars, buses, motorcycles) from the news archive of [hvg.hu](hvg.hu).
While the annotation in NYTK-NerKor followed the CoNLL2002 labelling standard with just four NE categories (`PER`, `LOC`, `MISC`, `ORG`), this version of the corpus features over 30 entity types, including all entity types used in the [OntoNotes 5.0] English NER annotation.
The new annotation elaborates on subtypes of the `LOC` and `MISC` entity types, and includes annotation for non-names like times and dates, quantities, languages and nationalities or religious or political groups. The annotation was elaborated with further entity subtypes not present in the Ontonotes 5 annotation (see below).


## Tags derived from the OntoNotes 5.0 annotation


Names are annotated according to the following set of types:


| | |
|---|---------|
|`PER` | = PERSON People, including fictional |
|`FAC` | = FACILITY Buildings, airports, highways, bridges, etc. |
|`ORG` | = ORGANIZATION Companies, agencies, institutions, etc. |
|`GPE` | Geopolitical entites: countries, cities, states |
|`LOC` | = LOCATION Non-GPE locations, mountain ranges, bodies of water |
|`PROD` | = PRODUCT Vehicles, weapons, foods, etc. (Not services) |
|`EVENT` | Named hurricanes, battles, wars, sports events, etc. |
|`WORK_OF_ART` | Titles of books, songs, etc. |
|`LAW` | Named documents made into laws  |


The following are also annotated in a style similar to names:


| | |
|---|---------|
| `NORP` | Nationalities or religious or political groups |
| `LANGUAGE` | Any named language |
| `DATE` | Absolute or relative dates or periods |
| `TIME` | Times smaller than a day |
| `PERCENT` | Percentage (including "%") |
| `MONEY` | Monetary values, including unit |
| `QUANTITY` | Measurements, as of weight or distance |
| `ORDINAL` | "first", "second" |
| `CARDINAL` | Numerals that do not fall under another type |


## Additional tags (not in OntoNotes 5)


Further subtypes of names of type `MISC`:


| | |
|-|-|
|`AWARD`| Awards and prizes |
| `CAR` | Cars and trucks |
|`MEDIA`| Media outlets, TV channels, news portals|
|`SMEDIA`| Social media platforms|
|`PROJ`| Projects and initiatives |
|`MISC`| Unresolved subtypes of MISC entities |
|`MISC-ORG`| Organization-like unresolved subtypes of MISC entities |


Further non-name entities:


| | |
|-|-|
|`DUR` |Time duration
|`ID`| identifier


model name: nerkor-cars-onpp-hubert


model author: novakat


task: token-classification


## language: fr


# CamemBERT: a Tasty French Language Model


## Introduction


[CamemBERT](https://arxiv.org/abs/1911.03894) is a state-of-the-art language model for French based on the RoBERTa model.


It is now available on Hugging Face in 6 different versions with varying number of parameters, amount of pretraining data and pretraining data source domains.


For further information or requests, please go to [Camembert Website](https://camembert-model.fr/)


## Pre-trained models


| Model                          | #params                        | Arch. | Training data                     |
|--------------------------------|--------------------------------|-------|-----------------------------------|
| `camembert-base` | 110M   | Base  | OSCAR (138 GB of text)            |
| `camembert/camembert-large`              | 335M    | Large | CCNet (135 GB of text)            |
| `camembert/camembert-base-ccnet`         | 110M    | Base  | CCNet (135 GB of text)            |
| `camembert/camembert-base-wikipedia-4gb` | 110M    | Base  | Wikipedia (4 GB of text)          |
| `camembert/camembert-base-oscar-4gb`     | 110M    | Base  | Subsample of OSCAR (4 GB of text) |
| `camembert/camembert-base-ccnet-4gb`     | 110M    | Base  | Subsample of CCNet (4 GB of text) |


## How to use CamemBERT with HuggingFace


##### Load CamemBERT and its sub-word tokenizer :


```python
from transformers import CamembertModel, CamembertTokenizer

# You can replace "camembert-base" with any other model from the table, e.g. "camembert/camembert-large".
tokenizer = CamembertTokenizer.from_pretrained("camembert/camembert-base-oscar-4gb")
camembert = CamembertModel.from_pretrained("camembert/camembert-base-oscar-4gb")

camembert.eval()  # disable dropout (or leave in train mode to finetune)

```


##### Filling masks using pipeline


```python
from transformers import pipeline

camembert_fill_mask  = pipeline("fill-mask", model="camembert/camembert-base-oscar-4gb", tokenizer="camembert/camembert-base-oscar-4gb")
>>> results = camembert_fill_mask("Le camembert est <mask> !")
# results
#[{'sequence': '<s> Le camembert est parfait!</s>', 'score': 0.04089554399251938, 'token': 1654},
#{'sequence': '<s> Le camembert est délicieux!</s>', 'score': 0.037193264812231064, 'token': 7200},
#{'sequence': '<s> Le camembert est prêt!</s>', 'score': 0.025467922911047935, 'token': 1415},
#{'sequence': '<s> Le camembert est meilleur!</s>', 'score': 0.022812040522694588, 'token': 528},
#{'sequence': '<s> Le camembert est différent!</s>', 'score': 0.017135459929704666, 'token': 2935}]

```


##### Extract contextual embedding features from Camembert output


```python
import torch
# Tokenize in sub-words with SentencePiece
tokenized_sentence = tokenizer.tokenize("J'aime le camembert !")
# ['▁J', "'", 'aime', '▁le', '▁ca', 'member', 't', '▁!']

# 1-hot encode and add special starting and end tokens
encoded_sentence = tokenizer.encode(tokenized_sentence)
# [5, 121, 11, 660, 16, 730, 25543, 110, 83, 6]
# NB: Can be done in one step : tokenize.encode("J'aime le camembert !")

# Feed tokens to Camembert as a torch tensor (batch dim 1)
encoded_sentence = torch.tensor(encoded_sentence).unsqueeze(0)
embeddings, _ = camembert(encoded_sentence)
# embeddings.detach()
# embeddings.size torch.Size([1, 10, 768])
#tensor([[[-0.1120, -0.1464,  0.0181,  ..., -0.1723, -0.0278,  0.1606],
#         [ 0.1234,  0.1202, -0.0773,  ..., -0.0405, -0.0668, -0.0788],
#         [-0.0440,  0.0480, -0.1926,  ...,  0.1066, -0.0961,  0.0637],
#         ...,
```


##### Extract contextual embedding features from all Camembert layers


```python
from transformers import CamembertConfig
# (Need to reload the model with new config)
config = CamembertConfig.from_pretrained("camembert/camembert-base-oscar-4gb", output_hidden_states=True)
camembert = CamembertModel.from_pretrained("camembert/camembert-base-oscar-4gb", config=config)

embeddings, _, all_layer_embeddings = camembert(encoded_sentence)
#  all_layer_embeddings list of len(all_layer_embeddings) == 13 (input embedding layer + 12 self attention layers)
all_layer_embeddings[5]
# layer 5 contextual embedding : size torch.Size([1, 10, 768])
#tensor([[[-0.1584, -0.1207, -0.0179,  ...,  0.5457,  0.1491, -0.1191],
#         [-0.1122,  0.3634,  0.0676,  ...,  0.4395, -0.0470, -0.3781],
#         [-0.2232,  0.0019,  0.0140,  ...,  0.4461, -0.0233,  0.0735],
#         ...,
```


## Authors


CamemBERT was trained and evaluated by Louis Martin\*, Benjamin Muller\*, Pedro Javier Ortiz Suárez\*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.


## Citation


If you use our work, please cite:


```bibtex
@inproceedings{martin2020camembert,
  title={CamemBERT: a Tasty French Language Model},
  author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year={2020}
}
```


model name: camembert-base-oscar-4gb


model author: camembert


tags: pytorch, camembert, fr, arxiv:1911.03894, transformers


## language: fr


## Introduction


## Pre-trained models


```python
from transformers import CamembertModel, CamembertTokenizer

# You can replace "camembert-base" with any other model from the table, e.g. "camembert/camembert-large".
tokenizer = CamembertTokenizer.from_pretrained("camembert/camembert-base-wikipedia-4gb")
camembert = CamembertModel.from_pretrained("camembert/camembert-base-wikipedia-4gb")

camembert.eval()  # disable dropout (or leave in train mode to finetune)

```


##### Filling masks using pipeline


```python
from transformers import pipeline

camembert_fill_mask  = pipeline("fill-mask", model="camembert/camembert-base-wikipedia-4gb", tokenizer="camembert/camembert-base-wikipedia-4gb")
results = camembert_fill_mask("Le camembert est un fromage de <mask>!")
# results
#[{'sequence': '<s> Le camembert est un fromage de chèvre!</s>', 'score': 0.4937814474105835, 'token': 19370},
#{'sequence': '<s> Le camembert est un fromage de brebis!</s>', 'score': 0.06255942583084106, 'token': 30616},
#{'sequence': '<s> Le camembert est un fromage de montagne!</s>', 'score': 0.04340197145938873, 'token': 2364},
# {'sequence': '<s> Le camembert est un fromage de Noël!</s>', 'score': 0.02823255956172943, 'token': 3236},
#{'sequence': '<s> Le camembert est un fromage de vache!</s>', 'score': 0.021357402205467224, 'token': 12329}]
```


```python
import torch
# Tokenize in sub-words with SentencePiece
tokenized_sentence = tokenizer.tokenize("J'aime le camembert !")
# ['▁J', "'", 'aime', '▁le', '▁ca', 'member', 't', '▁!']

# 1-hot encode and add special starting and end tokens
encoded_sentence = tokenizer.encode(tokenized_sentence)
# [5, 221, 10, 10600, 14, 8952, 10540, 75, 1114, 6]
# NB: Can be done in one step : tokenize.encode("J'aime le camembert !")

# Feed tokens to Camembert as a torch tensor (batch dim 1)
encoded_sentence = torch.tensor(encoded_sentence).unsqueeze(0)
embeddings, _ = camembert(encoded_sentence)
# embeddings.detach()
# embeddings.size torch.Size([1, 10, 768])
#tensor([[[-0.0928,  0.0506, -0.0094,  ..., -0.2388,  0.1177, -0.1302],
#         [ 0.0662,  0.1030, -0.2355,  ..., -0.4224, -0.0574, -0.2802],
#         [-0.0729,  0.0547,  0.0192,  ..., -0.1743,  0.0998, -0.2677],
#         ...,
```


```python
from transformers import CamembertConfig
# (Need to reload the model with new config)
config = CamembertConfig.from_pretrained("camembert/camembert-base-wikipedia-4gb", output_hidden_states=True)
camembert = CamembertModel.from_pretrained("camembert/camembert-base-wikipedia-4gb", config=config)

embeddings, _, all_layer_embeddings = camembert(encoded_sentence)
#  all_layer_embeddings list of len(all_layer_embeddings) == 13 (input embedding layer + 12 self attention layers)
all_layer_embeddings[5]
# layer 5 contextual embedding : size torch.Size([1, 10, 768])
#tensor([[[-0.0059, -0.0227,  0.0065,  ..., -0.0770,  0.0369,  0.0095],
#         [ 0.2838, -0.1531, -0.3642,  ..., -0.0027, -0.8502, -0.7914],
#         [-0.0073, -0.0338, -0.0011,  ...,  0.0533, -0.0250, -0.0061],
#         ...,
```


model name: camembert-base-wikipedia-4gb


# CamemBERT: a Tasty French Language Model


[CamemBERT](https://arxiv.org/abs/1911.03894) is a state-of-the-art language model for French based on the RoBERTa model.


It is now available on Hugging Face in 6 different versions with varying number of parameters, amount of pretraining data and pretraining data source domains.


For further information or requests, please go to [Camembert Website](https://camembert-model.fr/)


| Model                          | #params                        | Arch. | Training data                     |
|--------------------------------|--------------------------------|-------|-----------------------------------|
| `camembert-base` | 110M   | Base  | OSCAR (138 GB of text)            |
| `camembert/camembert-large`              | 335M    | Large | CCNet (135 GB of text)            |
| `camembert/camembert-base-ccnet`         | 110M    | Base  | CCNet (135 GB of text)            |
| `camembert/camembert-base-wikipedia-4gb` | 110M    | Base  | Wikipedia (4 GB of text)          |
| `camembert/camembert-base-oscar-4gb`     | 110M    | Base  | Subsample of OSCAR (4 GB of text) |
| `camembert/camembert-base-ccnet-4gb`     | 110M    | Base  | Subsample of CCNet (4 GB of text) |


## How to use CamemBERT with HuggingFace


##### Load CamemBERT and its sub-word tokenizer :


```python
from transformers import CamembertModel, CamembertTokenizer

# You can replace "camembert-base" with any other model from the table, e.g. "camembert/camembert-large".
tokenizer = CamembertTokenizer.from_pretrained("camembert/camembert-base-ccnet")
camembert = CamembertModel.from_pretrained("camembert/camembert-base-ccnet")

camembert.eval()  # disable dropout (or leave in train mode to finetune)

```


```python
from transformers import pipeline

camembert_fill_mask  = pipeline("fill-mask", model="camembert/camembert-base-ccnet", tokenizer="camembert/camembert-base-ccnet")
results = camembert_fill_mask("Le camembert est <mask> :)")
# results
#[{'sequence': '<s> Le camembert est bon :)</s>', 'score': 0.14011502265930176, 'token': 305},
# {'sequence': '<s> Le camembert est délicieux :)</s>', 'score': 0.13929404318332672, 'token': 11661},
# {'sequence': '<s> Le camembert est excellent :)</s>', 'score': 0.07010319083929062, 'token': 3497},
# {'sequence': '<s> Le camembert est parfait :)</s>', 'score': 0.025885622948408127, 'token': 2528},
# {'sequence': '<s> Le camembert est top :)</s>', 'score': 0.025684962049126625, 'token': 2328}]
```


##### Extract contextual embedding features from Camembert output


```python
import torch
# Tokenize in sub-words with SentencePiece
tokenized_sentence = tokenizer.tokenize("J'aime le camembert !")
# ['▁J', "'", 'aime', '▁le', '▁cam', 'ember', 't', '▁!']

# 1-hot encode and add special starting and end tokens
encoded_sentence = tokenizer.encode(tokenized_sentence)
# [5, 133, 22, 1250, 16, 12034, 14324, 81, 76, 6]
# NB: Can be done in one step : tokenize.encode("J'aime le camembert !")

# Feed tokens to Camembert as a torch tensor (batch dim 1)
encoded_sentence = torch.tensor(encoded_sentence).unsqueeze(0)
embeddings, _ = camembert(encoded_sentence)
# embeddings.detach()
# embeddings.size torch.Size([1, 10, 768])
#tensor([[[ 0.0667, -0.2467,  0.0954,  ...,  0.2144,  0.0279,  0.3621],
#         [-0.0472,  0.4092, -0.6602,  ...,  0.2095,  0.1391, -0.0401],
#         [ 0.1911, -0.2347, -0.0811,  ...,  0.4306, -0.0639,  0.1821],
#         ...,
```


##### Extract contextual embedding features from all Camembert layers


```python
from transformers import CamembertConfig
# (Need to reload the model with new config)
config = CamembertConfig.from_pretrained("camembert/camembert-base-ccnet", output_hidden_states=True)
camembert = CamembertModel.from_pretrained("camembert/camembert-base-ccnet", config=config)

embeddings, _, all_layer_embeddings = camembert(encoded_sentence)
#  all_layer_embeddings list of len(all_layer_embeddings) == 13 (input embedding layer + 12 self attention layers)
all_layer_embeddings[5]
# layer 5 contextual embedding : size torch.Size([1, 10, 768])
#tensor([[[ 0.0057, -0.1022,  0.0163,  ..., -0.0675, -0.0360,  0.1078],
#         [-0.1096, -0.3344, -0.0593,  ...,  0.1625, -0.0432, -0.1646],
#         [ 0.3751, -0.3829,  0.0844,  ...,  0.1067, -0.0330,  0.3334],
#         ...,
```


## Authors


CamemBERT was trained and evaluated by Louis Martin\*, Benjamin Muller\*, Pedro Javier Ortiz Suárez\*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.


## Citation


If you use our work, please cite:


```bibtex
@inproceedings{martin2020camembert,
  title={CamemBERT: a Tasty French Language Model},
  author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year={2020}
}
```


model name: camembert-base-ccnet


model author: camembert


tags: pytorch, camembert, fr, arxiv:1911.03894, transformers


## language: fr


## Introduction


## Pre-trained models


```python
from transformers import CamembertModel, CamembertTokenizer

# You can replace "camembert-base" with any other model from the table, e.g. "camembert/camembert-large".
tokenizer = CamembertTokenizer.from_pretrained("camembert/camembert-base-ccnet-4gb")
camembert = CamembertModel.from_pretrained("camembert/camembert-base-ccnet-4gb")

camembert.eval()  # disable dropout (or leave in train mode to finetune)

```


##### Filling masks using pipeline


```python
from transformers import pipeline

camembert_fill_mask  = pipeline("fill-mask", model="camembert/camembert-base-ccnet-4gb", tokenizer="camembert/camembert-base-ccnet-4gb")
results = camembert_fill_mask("Le camembert est-il  <mask> ?")
# results
#[{'sequence': '<s> Le camembert est-il sain?</s>', 'score': 0.07001790404319763, 'token': 10286},
#{'sequence': '<s> Le camembert est-il français?</s>', 'score': 0.057594332844018936, 'token': 384},
#{'sequence': '<s> Le camembert est-il bon?</s>', 'score': 0.04098724573850632, 'token': 305},
#{'sequence': '<s> Le camembert est-il périmé?</s>', 'score': 0.03486393392086029, 'token': 30862},
#{'sequence': '<s> Le camembert est-il cher?</s>', 'score': 0.021535946056246758, 'token': 1604}]

```


```python
import torch
# Tokenize in sub-words with SentencePiece
tokenized_sentence = tokenizer.tokenize("J'aime le camembert !")
# ['▁J', "'", 'aime', '▁le', '▁ca', 'member', 't', '▁!']

# 1-hot encode and add special starting and end tokens
encoded_sentence = tokenizer.encode(tokenized_sentence)
# [5, 133, 22, 1250, 16, 12034, 14324, 81, 76, 6]
# NB: Can be done in one step : tokenize.encode("J'aime le camembert !")

# Feed tokens to Camembert as a torch tensor (batch dim 1)
encoded_sentence = torch.tensor(encoded_sentence).unsqueeze(0)
embeddings, _ = camembert(encoded_sentence)
# embeddings.detach()
# embeddings.size torch.Size([1, 10, 768])
#tensor([[[ 0.0331,  0.0095, -0.2776,  ...,  0.2875, -0.0827, -0.2467],
#         [-0.1348,  0.0478, -0.5409,  ...,  0.8330,  0.0467,  0.0662],
#         [ 0.0920, -0.0264,  0.0177,  ...,  0.1112,  0.0108, -0.1123],
#         ...,
```


```python
from transformers import CamembertConfig
# (Need to reload the model with new config)
config = CamembertConfig.from_pretrained("camembert/camembert-base-ccnet-4gb", output_hidden_states=True)
camembert = CamembertModel.from_pretrained("camembert/camembert-base-ccnet-4gb", config=config)

embeddings, _, all_layer_embeddings = camembert(encoded_sentence)
#  all_layer_embeddings list of len(all_layer_embeddings) == 13 (input embedding layer + 12 self attention layers)
all_layer_embeddings[5]
# layer 5 contextual embedding : size torch.Size([1, 10, 768])
#tensor([[[-0.0144,  0.1855,  0.4895,  ..., -0.1537,  0.0107, -0.2293],
#         [-0.6664, -0.0880, -0.1539,  ...,  0.3635,  0.4047,  0.1258],
#         [ 0.0511,  0.0540,  0.2545,  ...,  0.0709, -0.0288, -0.0779],
#         ...,
```


model name: camembert-base-ccnet-4gb


# CamemBERT: a Tasty French Language Model


[CamemBERT](https://arxiv.org/abs/1911.03894) is a state-of-the-art language model for French based on the RoBERTa model.


It is now available on Hugging Face in 6 different versions with varying number of parameters, amount of pretraining data and pretraining data source domains.


For further information or requests, please go to [Camembert Website](https://camembert-model.fr/)


| Model                          | #params                        | Arch. | Training data                     |
|--------------------------------|--------------------------------|-------|-----------------------------------|
| `camembert-base` | 110M   | Base  | OSCAR (138 GB of text)            |
| `camembert/camembert-large`              | 335M    | Large | CCNet (135 GB of text)            |
| `camembert/camembert-base-ccnet`         | 110M    | Base  | CCNet (135 GB of text)            |
| `camembert/camembert-base-wikipedia-4gb` | 110M    | Base  | Wikipedia (4 GB of text)          |
| `camembert/camembert-base-oscar-4gb`     | 110M    | Base  | Subsample of OSCAR (4 GB of text) |
| `camembert/camembert-base-ccnet-4gb`     | 110M    | Base  | Subsample of CCNet (4 GB of text) |


## How to use CamemBERT with HuggingFace


##### Load CamemBERT and its sub-word tokenizer :


```python
from transformers import CamembertModel, CamembertTokenizer

# You can replace "camembert-base" with any other model from the table, e.g. "camembert/camembert-large".
tokenizer = CamembertTokenizer.from_pretrained("camembert/camembert-large")
camembert = CamembertModel.from_pretrained("camembert/camembert-large")

camembert.eval()  # disable dropout (or leave in train mode to finetune)

```


```python
from transformers import pipeline

camembert_fill_mask  = pipeline("fill-mask", model="camembert/camembert-large", tokenizer="camembert/camembert-large")
results = camembert_fill_mask("Le camembert est <mask> :)")
# results
#[{'sequence': '<s> Le camembert est bon :)</s>', 'score': 0.15560828149318695, 'token': 305},
#{'sequence': '<s> Le camembert est excellent :)</s>', 'score': 0.06821336597204208, 'token': 3497},
#{'sequence': '<s> Le camembert est délicieux :)</s>', 'score': 0.060438305139541626, 'token': 11661},
#{'sequence': '<s> Le camembert est ici :)</s>', 'score': 0.02023460529744625, 'token': 373},
#{'sequence': '<s> Le camembert est meilleur :)</s>', 'score': 0.01778135634958744, 'token': 876}]
```


##### Extract contextual embedding features from Camembert output


```python
import torch
# Tokenize in sub-words with SentencePiece
tokenized_sentence = tokenizer.tokenize("J'aime le camembert !")
# ['▁J', "'", 'aime', '▁le', '▁cam', 'ember', 't', '▁!']

# 1-hot encode and add special starting and end tokens
encoded_sentence = tokenizer.encode(tokenized_sentence)
# [5, 133, 22, 1250, 16, 12034, 14324, 81, 76, 6]
# NB: Can be done in one step : tokenize.encode("J'aime le camembert !")

# Feed tokens to Camembert as a torch tensor (batch dim 1)
encoded_sentence = torch.tensor(encoded_sentence).unsqueeze(0)
embeddings, _ = camembert(encoded_sentence)
# embeddings.detach()
# torch.Size([1, 10, 1024])
#tensor([[[-0.1284,  0.2643,  0.4374,  ...,  0.1627,  0.1308, -0.2305],
#         [ 0.4576, -0.6345, -0.2029,  ..., -0.1359, -0.2290, -0.6318],
#         [ 0.0381,  0.0429,  0.5111,  ..., -0.1177, -0.1913, -0.1121],
#         ...,
```


##### Extract contextual embedding features from all Camembert layers


```python
from transformers import CamembertConfig
# (Need to reload the model with new config)
config = CamembertConfig.from_pretrained("camembert/camembert-large", output_hidden_states=True)
camembert = CamembertModel.from_pretrained("camembert/camembert-large", config=config)

embeddings, _, all_layer_embeddings = camembert(encoded_sentence)
#  all_layer_embeddings list of len(all_layer_embeddings) == 25 (input embedding layer + 24 self attention layers)
all_layer_embeddings[5]
# layer 5 contextual embedding : size torch.Size([1, 10, 1024])
#tensor([[[-0.0600,  0.0742,  0.0332,  ..., -0.0525, -0.0637, -0.0287],
#         [ 0.0950,  0.2840,  0.1985,  ...,  0.2073, -0.2172, -0.6321],
#         [ 0.1381,  0.1872,  0.1614,  ..., -0.0339, -0.2530, -0.1182],
#         ...,
```


## Authors


CamemBERT was trained and evaluated by Louis Martin\*, Benjamin Muller\*, Pedro Javier Ortiz Suárez\*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.


## Citation


If you use our work, please cite:


```bibtex
@inproceedings{martin2020camembert,
  title={CamemBERT: a Tasty French Language Model},
  author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year={2020}
}
```


model name: camembert-large


model author: camembert


tags: pytorch, camembert, fr, arxiv:1911.03894, transformers


tags:


- stanza
- token-classification
library_name: stanza
language:
- fi
license: apache-2.0


# Stanza model for Finnish (fi)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:19:25.792


model name: stanza-fi


model author: stanfordnlp


task: token-classification


tags: fi, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- da
license: apache-2.0


# Stanza model for Danish (da)


Last updated 2022-01-12 03:57:53.395


model name: stanza-da


tags: da, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- ja
license: apache-2.0


# Stanza model for Japanese (ja)


Last updated 2021-12-16 08:35:14.470


model name: stanza-ja


tags: ja, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- zh-hant
license: apache-2.0


# Stanza model for Traditional_Chinese (zh-hant)


Last updated 2021-12-16 09:00:54.591


model name: stanza-zh-hant


model author: stanfordnlp


task: token-classification


tags: zh-hant, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- sl
license: apache-2.0


# Stanza model for Slovenian (sl)


Last updated 2021-12-16 08:50:18.645


model name: stanza-sl


tags: sl, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- is
license: apache-2.0


# Stanza model for Icelandic (is)


Last updated 2021-12-16 08:31:55.416


model name: stanza-is


tags: is, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- fro
license: apache-2.0


# Stanza model for Old_French (fro)


Last updated 2021-12-16 08:23:15.810


model name: stanza-fro


model author: stanfordnlp


task: token-classification


tags: fro, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- gd
license: apache-2.0


# Stanza model for Scottish_Gaelic (gd)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:24:29.962


model name: stanza-gd


tags: gd, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- my
license: apache-2.0


# Stanza model for Myanmar (my)


Last updated 2021-12-16 08:40:58.686


model name: stanza-my


tags: my, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- vi
license: apache-2.0


# Stanza model for Vietnamese (vi)


Last updated 2021-12-16 08:57:58.879


model name: stanza-vi


model author: stanfordnlp


task: token-classification


tags: vi, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- cop
license: apache-2.0


# Stanza model for Coptic (cop)


Last updated 2021-12-16 08:02:06.198


model name: stanza-cop


tags: cop, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- sv
license: apache-2.0


# Stanza model for Swedish (sv)


Last updated 2022-01-05 09:22:03.398


model name: stanza-sv


tags: sv, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- ta
license: apache-2.0


# Stanza model for Tamil (ta)


Last updated 2021-12-16 08:53:25.868


model name: stanza-ta


model author: stanfordnlp


task: token-classification


tags: ta, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- en
license: apache-2.0


# Stanza model for English (en)


Last updated 2022-01-05 02:34:10.864


model name: stanza-en


tags: en, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- hyw
license: apache-2.0


# Stanza model for Western_Armenian (hyw)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:30:25.226


model name: stanza-hyw


tags: hyw, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- got
license: apache-2.0


# Stanza model for Gothic (got)


Last updated 2021-12-16 08:25:58.985


model name: stanza-got


model author: stanfordnlp


task: token-classification


tags: got, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- th
license: apache-2.0


# Stanza model for Thai (th)


Last updated 2021-12-16 08:54:33.903


model name: stanza-th


tags: th, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- ar
license: apache-2.0


# Stanza model for Arabic (ar)


Last updated 2021-12-16 07:59:16.931


model name: stanza-ar


tags: ar, stanza, token-classification, license:apache-2.0


tags:


- corenlp
library_tag: corenlp
language:
- french
license: gpl-2.0


# Core NLP model for french


CoreNLP is your one stop shop for natural language processing in Java! CoreNLP enables users to derive linguistic annotations for text, including token and sentence boundaries, parts of speech, named entities, numeric and time values, dependency and constituency parses, coreference, sentiment, quote attributions, and relations.
Find more about it in [our website](https://stanfordnlp.github.io/CoreNLP) and our [GitHub repository](https://github.com/stanfordnlp/CoreNLP).


This card and repo were automatically prepared with `hugging_corenlp.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2022-01-21 06:20:59.095


model name: corenlp-french


model author: stanfordnlp


tags: french, corenlp, license:gpl-2.0


- stanza
- token-classification
library_name: stanza
language:
- la
license: apache-2.0


# Stanza model for Latin (la)


Last updated 2021-12-16 08:36:45.438


model name: stanza-la


task: token-classification


tags: la, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- multilingual
license: apache-2.0


# Stanza model for multilingual


Last updated 2021-12-16 08:40:28.073


model name: stanza-multilingual


tags: multilingual, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- id
license: apache-2.0


# Stanza model for Indonesian (id)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:31:11.366


model name: stanza-id


model author: stanfordnlp


tags: id, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- swl
license: apache-2.0


# Stanza model for Swedish_Sign_Language (swl)


Last updated 2021-12-16 08:52:52.226


model name: stanza-swl


task: token-classification


tags: swl, stanza, token-classification, license:apache-2.0


- corenlp
library_tag: corenlp
language:
- english-extra
license: gpl-2.0


# Core NLP model for english-extra


Last updated 2022-01-21 06:20:36.619


model name: corenlp-english-extra


tags: english-extra, corenlp, license:gpl-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- af
license: apache-2.0


# Stanza model for Afrikaans (af)


Last updated 2021-12-16 07:58:30.340


model name: stanza-af


model author: stanfordnlp


tags: af, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- sme
license: apache-2.0


# Stanza model for North_Sami (sme)


Last updated 2021-12-16 08:50:53.295


model name: stanza-sme


task: token-classification


tags: sme, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- ga
license: apache-2.0


# Stanza model for Irish (ga)


Last updated 2021-12-16 08:23:54.826


model name: stanza-ga


tags: ga, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- mr
license: apache-2.0


# Stanza model for Marathi (mr)


Last updated 2021-12-16 08:39:38.605


model name: stanza-mr


model author: stanfordnlp


tags: mr, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- uk
license: apache-2.0


# Stanza model for Ukrainian (uk)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2022-01-09 17:13:41.706


model name: stanza-uk


task: token-classification


tags: uk, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- lv
license: apache-2.0


# Stanza model for Latvian (lv)


Last updated 2021-12-16 08:38:39.256


model name: stanza-lv


tags: lv, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- es
license: apache-2.0


# Stanza model for Spanish (es)


Last updated 2021-12-16 08:15:03.040


model name: stanza-es


model author: stanfordnlp


tags: es, stanza, token-classification, license:apache-2.0


- corenlp
library_tag: corenlp
language:
- hungarian
license: gpl-2.0


# Core NLP model for hungarian


CoreNLP is your one stop shop for natural language processing in Java! CoreNLP enables users to derive linguistic annotations for text, including token and sentence boundaries, parts of speech, named entities, numeric and time values, dependency and constituency parses, coreference, sentiment, quote attributions, and relations.
Find more about it in [our website](https://stanfordnlp.github.io/CoreNLP) and our [GitHub repository](https://github.com/stanfordnlp/CoreNLP).


This card and repo were automatically prepared with `hugging_corenlp.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2022-01-21 06:21:20.694


model name: corenlp-hungarian


tags: hungarian, corenlp, license:gpl-2.0


- corenlp
library_tag: corenlp
language:
- german
license: gpl-2.0


# Core NLP model for german


Last updated 2022-01-21 06:21:10.250


model name: corenlp-german


tags: german, corenlp, license:gpl-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- pl
license: apache-2.0


# Stanza model for Polish (pl)


Last updated 2021-12-16 08:45:06.927


model name: stanza-pl


model author: stanfordnlp


task: token-classification


tags: pl, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- hy
license: apache-2.0


# Stanza model for Armenian (hy)


Last updated 2021-12-16 08:29:34.308


model name: stanza-hy


tags: hy, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- et
license: apache-2.0


# Stanza model for Estonian (et)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:16:19.711


model name: stanza-et


tags: et, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- nn
license: apache-2.0


# Stanza model for Norwegian_Nynorsk (nn)


Last updated 2022-01-01 08:27:00.049


model name: stanza-nn


model author: stanfordnlp


task: token-classification


tags: nn, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- mt
license: apache-2.0


# Stanza model for Maltese (mt)


Last updated 2021-12-16 08:40:06.408


model name: stanza-mt


tags: mt, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- ca
license: apache-2.0


# Stanza model for Catalan (ca)


Last updated 2021-12-16 08:01:35.836


model name: stanza-ca


tags: ca, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- fa
license: apache-2.0


# Stanza model for Persian (fa)


Last updated 2021-12-16 08:18:10.380


model name: stanza-fa


model author: stanfordnlp


task: token-classification


tags: fa, stanza, token-classification, license:apache-2.0


- corenlp
library_tag: corenlp
language:
- italian
license: gpl-2.0


# Core NLP model for italian


CoreNLP is your one stop shop for natural language processing in Java! CoreNLP enables users to derive linguistic annotations for text, including token and sentence boundaries, parts of speech, named entities, numeric and time values, dependency and constituency parses, coreference, sentiment, quote attributions, and relations.
Find more about it in [our website](https://stanfordnlp.github.io/CoreNLP) and our [GitHub repository](https://github.com/stanfordnlp/CoreNLP).


This card and repo were automatically prepared with `hugging_corenlp.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2022-01-21 06:21:31.273


model name: corenlp-italian


tags: italian, corenlp, license:gpl-2.0


- corenlp
library_tag: corenlp
language:
- arabic
license: gpl-2.0


# Core NLP model for arabic


Last updated 2022-01-21 06:19:50.729


model name: corenlp-arabic


tags: arabic, corenlp, license:gpl-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- wo
license: apache-2.0


# Stanza model for Wolof (wo)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:58:34.733


model name: stanza-wo


model author: stanfordnlp


task: token-classification


tags: wo, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- ug
license: apache-2.0


# Stanza model for Uyghur (ug)


Last updated 2021-12-16 08:56:11.632


model name: stanza-ug


tags: ug, stanza, token-classification, license:apache-2.0


Hosts models for Stanford's GloVe distribution


model name: glove


tags:


- stanza
- token-classification
library_name: stanza
language:
- nl
license: apache-2.0


# Stanza model for Dutch (nl)


Last updated 2021-12-16 08:42:29.646


model name: stanza-nl


tags: nl, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- cs
license: apache-2.0


# Stanza model for Czech (cs)


Last updated 2021-12-16 08:03:06.703


model name: stanza-cs


model author: stanfordnlp


task: token-classification


tags: cs, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- hu
license: apache-2.0


# Stanza model for Hungarian (hu)


Last updated 2021-12-16 08:28:49.195


model name: stanza-hu


tags: hu, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- he
license: apache-2.0


# Stanza model for Hebrew (he)


Last updated 2021-12-16 08:27:10.478


model name: stanza-he


tags: he, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- lt
license: apache-2.0


# Stanza model for Lithuanian (lt)


Last updated 2021-12-16 08:37:52.573


model name: stanza-lt


model author: stanfordnlp


task: token-classification


tags: lt, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- el
license: apache-2.0


# Stanza model for Greek (el)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:08:29.621


model name: stanza-el


tags: el, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- pcm
license: apache-2.0


# Stanza model for Naija (pcm)


Last updated 2021-12-16 08:44:31.734


model name: stanza-pcm


tags: pcm, stanza, token-classification, license:apache-2.0


- corenlp
library_tag: corenlp
language:
- CoreNLP
license: gpl-2.0


# Core NLP model for CoreNLP


CoreNLP is your one stop shop for natural language processing in Java! CoreNLP enables users to derive linguistic annotations for text, including token and sentence boundaries, parts of speech, named entities, numeric and time values, dependency and constituency parses, coreference, sentiment, quote attributions, and relations.
Find more about it in [our website](https://stanfordnlp.github.io/CoreNLP) and our [GitHub repository](https://github.com/stanfordnlp/CoreNLP).


This card and repo were automatically prepared with `hugging_corenlp.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2022-01-21 06:19:39.291


model name: CoreNLP


model author: stanfordnlp


tags: CoreNLP, corenlp, license:gpl-2.0


- stanza
- token-classification
library_name: stanza
language:
- te
license: apache-2.0


# Stanza model for Telugu (te)


Last updated 2021-12-16 08:54:04.266


model name: stanza-te


task: token-classification


tags: te, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- cy
license: apache-2.0


# Stanza model for Welsh (cy)


Last updated 2021-12-16 08:04:50.143


model name: stanza-cy


tags: cy, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- hr
license: apache-2.0


# Stanza model for Croatian (hr)


Last updated 2021-12-16 08:28:13.395


model name: stanza-hr


model author: stanfordnlp


tags: hr, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- orv
license: apache-2.0


# Stanza model for Old_East_Slavic (orv)


Last updated 2021-12-16 08:44:04.352


model name: stanza-orv


task: token-classification


tags: orv, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- sr
license: apache-2.0


# Stanza model for Serbian (sr)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:51:22.678


model name: stanza-sr


tags: sr, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- tr
license: apache-2.0


# Stanza model for Turkish (tr)


Last updated 2021-12-16 08:55:17.280


model name: stanza-tr


model author: stanfordnlp


tags: tr, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- gl
license: apache-2.0


# Stanza model for Galician (gl)


Last updated 2021-12-16 08:25:16.544


model name: stanza-gl


task: token-classification


tags: gl, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- fo
license: apache-2.0


# Stanza model for Faroese (fo)


Last updated 2021-12-16 08:20:22.823


model name: stanza-fo


tags: fo, stanza, token-classification, license:apache-2.0


- corenlp
library_tag: corenlp
language:
- spanish
license: gpl-2.0


# Core NLP model for spanish


CoreNLP is your one stop shop for natural language processing in Java! CoreNLP enables users to derive linguistic annotations for text, including token and sentence boundaries, parts of speech, named entities, numeric and time values, dependency and constituency parses, coreference, sentiment, quote attributions, and relations.
Find more about it in [our website](https://stanfordnlp.github.io/CoreNLP) and our [GitHub repository](https://github.com/stanfordnlp/CoreNLP).


This card and repo were automatically prepared with `hugging_corenlp.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2022-01-21 06:23:44.181


model name: corenlp-spanish


model author: stanfordnlp


tags: spanish, corenlp, license:gpl-2.0


- stanza
- token-classification
library_name: stanza
language:
- be
license: apache-2.0


# Stanza model for Belarusian (be)


Last updated 2021-12-16 08:00:05.014


model name: stanza-be


task: token-classification


tags: be, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- lzh
license: apache-2.0


# Stanza model for Classical_Chinese (lzh)


Last updated 2021-12-16 08:39:08.764


model name: stanza-lzh


tags: lzh, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- ur
license: apache-2.0


# Stanza model for Urdu (ur)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:57:22.924


model name: stanza-ur


model author: stanfordnlp


tags: ur, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- hi
license: apache-2.0


# Stanza model for Hindi (hi)


Last updated 2021-12-16 08:27:42.048


model name: stanza-hi


task: token-classification


tags: hi, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- sk
license: apache-2.0


# Stanza model for Slovak (sk)


Last updated 2021-12-16 08:49:42.413


model name: stanza-sk


tags: sk, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- zh-hans
license: apache-2.0


# Stanza model for Simplified_Chinese (zh-hans)


Last updated 2022-01-05 02:55:46.354


model name: stanza-zh-hans


model author: stanfordnlp


tags: zh-hans, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- pt
license: apache-2.0


# Stanza model for Portuguese (pt)


Last updated 2021-12-16 08:45:49.769


model name: stanza-pt


task: token-classification


tags: pt, stanza, token-classification, license:apache-2.0


tags:


- corenlp
library_tag: corenlp
language:
- english-default
license: gpl-2.0


# Core NLP model for english-default


CoreNLP is your one stop shop for natural language processing in Java! CoreNLP enables users to derive linguistic annotations for text, including token and sentence boundaries, parts of speech, named entities, numeric and time values, dependency and constituency parses, coreference, sentiment, quote attributions, and relations.
Find more about it in [our website](https://stanfordnlp.github.io/CoreNLP) and our [GitHub repository](https://github.com/stanfordnlp/CoreNLP).


This card and repo were automatically prepared with `hugging_corenlp.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2022-01-21 06:20:22.667


model name: corenlp-english-default


tags: english-default, corenlp, license:gpl-2.0


- stanza
- token-classification
library_name: stanza
language:
- bg
license: apache-2.0


# Stanza model for Bulgarian (bg)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:00:54.101


model name: stanza-bg


model author: stanfordnlp


task: token-classification


tags: bg, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- grc
license: apache-2.0


# Stanza model for Ancient_Greek (grc)


Last updated 2021-12-16 08:26:38.150


model name: stanza-grc


tags: grc, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- fr
license: apache-2.0


# Stanza model for French (fr)


Last updated 2021-12-16 08:21:47.037


model name: stanza-fr


tags: fr, stanza, token-classification, license:apache-2.0


- corenlp
library_tag: corenlp
language:
- english-kbp
license: gpl-2.0


# Core NLP model for english-kbp


Last updated 2022-01-21 06:20:48.310


model name: corenlp-english-kbp


model author: stanfordnlp


tags: english-kbp, corenlp, license:gpl-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- cu
license: apache-2.0


# Stanza model for Old_Church_Slavonic (cu)


Last updated 2021-12-16 08:04:11.187


model name: stanza-cu


task: token-classification


tags: cu, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- ro
license: apache-2.0


# Stanza model for Romanian (ro)


Last updated 2021-12-16 08:47:02.893


model name: stanza-ro


tags: ro, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- qtd
license: apache-2.0


# Stanza model for Turkish_German (qtd)


Last updated 2021-12-16 08:46:23.831


model name: stanza-qtd


model author: stanfordnlp


tags: qtd, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- ko
license: apache-2.0


# Stanza model for Korean (ko)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:35:51.359


model name: stanza-ko


task: token-classification


tags: ko, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- it
license: apache-2.0


# Stanza model for Italian (it)


Last updated 2022-01-05 03:20:13.649


model name: stanza-it


tags: it, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- eu
license: apache-2.0


# Stanza model for Basque (eu)


Last updated 2021-12-16 08:17:07.631


model name: stanza-eu


model author: stanfordnlp


tags: eu, stanza, token-classification, license:apache-2.0


tags:


- corenlp
library_tag: corenlp
language:
- chinese
license: gpl-2.0


# Core NLP model for chinese


CoreNLP is your one stop shop for natural language processing in Java! CoreNLP enables users to derive linguistic annotations for text, including token and sentence boundaries, parts of speech, named entities, numeric and time values, dependency and constituency parses, coreference, sentiment, quote attributions, and relations.
Find more about it in [our website](https://stanfordnlp.github.io/CoreNLP) and our [GitHub repository](https://github.com/stanfordnlp/CoreNLP).


This card and repo were automatically prepared with `hugging_corenlp.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2022-01-21 06:20:04.060


model name: corenlp-chinese


tags: chinese, corenlp, license:gpl-2.0


- stanza
- token-classification
library_name: stanza
language:
- nb
license: apache-2.0


# Stanza model for Norwegian (nb)


Last updated 2022-01-01 08:19:02.594


model name: stanza-nb


task: token-classification


tags: nb, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- ru
license: apache-2.0


# Stanza model for Russian (ru)


Last updated 2021-12-16 08:48:08.915


model name: stanza-ru


model author: stanfordnlp


tags: ru, stanza, token-classification, license:apache-2.0


tags:


- stanza
- token-classification
library_name: stanza
language:
- sa
license: apache-2.0


# Stanza model for Sanskrit (sa)


Last updated 2021-12-16 08:49:08.097


model name: stanza-sa


tags: sa, stanza, token-classification, license:apache-2.0


- stanza
- token-classification
library_name: stanza
language:
- de
license: apache-2.0


# Stanza model for German (de)


Stanza is a collection of accurate and efficient tools for the linguistic analysis of many human languages. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages of your choosing.
Find more about it in [our website](https://stanfordnlp.github.io/stanza) and our [GitHub repository](https://github.com/stanfordnlp/stanza).


This card and repo were automatically prepared with `hugging_stanza.py` in the `stanfordnlp/huggingface-models` repo


Last updated 2021-12-16 08:06:59.557


model name: stanza-de


task: token-classification


tags: de, stanza, token-classification, license:apache-2.0


## language: mn


# ALBERT-Mongolian


[pretraining repo link](https://github.com/bayartsogt-ya/albert-mongolian)


## Model description


Here we provide pretrained ALBERT model and trained SentencePiece model for Mongolia text. Training data is the Mongolian wikipedia corpus from Wikipedia Downloads and Mongolian News corpus.


## Evaluation Result:


```
loss = 1.7478163
masked_lm_accuracy = 0.6838185
masked_lm_loss = 1.6687671
sentence_order_accuracy = 0.998125
sentence_order_loss = 0.007942731
```


## Fine-tuning Result on Eduge Dataset:


```
                          precision    recall  f1-score   support

            байгал орчин       0.85      0.83      0.84       999
               боловсрол       0.80      0.80      0.80       873
                   спорт       0.98      0.98      0.98      2736
               технологи       0.88      0.93      0.91      1102
                 улс төр       0.92      0.85      0.89      2647
              урлаг соёл       0.93      0.94      0.94      1457
                   хууль       0.89      0.87      0.88      1651
             эдийн засаг       0.83      0.88      0.86      2509
              эрүүл мэнд       0.89      0.92      0.90      1159

                accuracy                           0.90     15133
               macro avg       0.89      0.89      0.89     15133
            weighted avg       0.90      0.90      0.90     15133
```


## Reference


1. [ALBERT - official repo](https://github.com/google-research/albert)
1. [WikiExtrator](https://github.com/attardi/wikiextractor)
1. [Mongolian BERT](https://github.com/tugstugi/mongolian-bert)
1. [ALBERT - Japanese](https://github.com/alinear-corp/albert-japanese)
1. [Mongolian Text Classification](https://github.com/sharavsambuu/mongolian-text-classification)
1. [You's paper](https://arxiv.org/abs/1904.00962)


## Citation


```
@misc{albert-mongolian,
  author = {Bayartsogt Yadamsuren},
  title = {ALBERT Pretrained Model on Mongolian Datasets},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/bayartsogt-ya/albert-mongolian/}}
}
```


## For More Information


Please contact by bayartsogtyadamsuren@icloud.com


model name: albert-mongolian


model author: bayartsogt


task: fill-mask


tags: pytorch, tf, albert, fill-mask, mn, arxiv:1904.00962, transformers, autonlp_compatible, infinity_compatible


library_name: fairseq
task: text-to-speech
tags:


- fairseq
- audio
- text-to-speech
language: mn
datasets:
- mbspeech
widget:
- text: "миний нэрийг баярцогт гэдэг"
example_title: "Say my name!"
- text: "би монгол улсын нийслэл, улаанбаатар хотод амьдардаг"
example_title: "Where I am from?"
- text: "энэхүү өгөгдлийг нээлттэй болгосон, болор соофтынхонд баярлалаа"
example_title: "Thank you!"
- text: "энэхүү ажлын ихэнх хэсгийг, төгөлдөр ах хийсэн болно"
example_title: "Shout out to original creater"


# tts_transformer-mn-mbspeech


[Transformer](https://arxiv.org/abs/1809.08895) text-to-speech model from fairseq S^2 ([paper](https://arxiv.org/abs/2109.06912)/[code](https://github.com/pytorch/fairseq/tree/main/examples/speech_synthesis)):


- Mongolian
- Single-speaker male voice
- Trained on [MBSpeech](https://github.com/tugstugi/mongolian-nlp/blob/master/datasets/MBSpeech-1.0-csv.zip)


model name: tts_transformer-mn-mbspeech


task: text-to-speech


tags: mn, dataset:mbspeech, arxiv:1809.08895, arxiv:2109.06912, fairseq, audio, text-to-speech


language: mn
datasets:


- common_voice mn
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 Mongolian V1 by Bayartsogt
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice mn
type: common_voice
args: mn
metrics:
    - name: Test WER
type: wer
value: 34.64


# Wav2Vec2-Large-XLSR-53-Mongolian-v1


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Mongolian using the [Common Voice](https://huggingface.co/datasets/common_voice).


When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "mn", split="test[:2%]")
processor = Wav2Vec2Processor.from_pretrained("bayartsogt/wav2vec2-large-xlsr-mongolian-v1")
model = Wav2Vec2ForCTC.from_pretrained("bayartsogt/wav2vec2-large-xlsr-mongolian-v1")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
     speech_array, sampling_rate = torchaudio.load(batch["path"])
     batch["speech"] = resampler(speech_array).squeeze().numpy()
     return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
     logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the Mongolian test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

test_dataset = load_dataset("common_voice", "mn", split="test")
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("bayartsogt/wav2vec2-large-xlsr-mongolian-v1")
model = Wav2Vec2ForCTC.from_pretrained("bayartsogt/wav2vec2-large-xlsr-mongolian-v1")
model.to("cuda")

chars_to_ignore_regex = '[\!\"\'\,\.\«\»\?\-]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
     batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
     speech_array, sampling_rate = torchaudio.load(batch["path"])
     batch["speech"] = resampler(speech_array).squeeze().numpy()
     return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
     inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

     with torch.no_grad():
          logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits

     pred_ids = torch.argmax(logits, dim=-1)
     batch["pred_strings"] = processor.batch_decode(pred_ids)
     return batch

result = test_dataset.map(evaluate, batched=True, batch_size=8)

print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 34.64 %


## Training


The Common Voice `train` dataset was used for training as well as ... and ...


model name: wav2vec2-large-xlsr-mongolian-v1


model author: bayartsogt


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, mn, dataset:common_voice mn, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


|fold|accuracy|
|-|-|
| fold 0 |  0.974197247706422 |
| fold 1 |  0.9678899082568807 |
| fold 2 |  0.9724770642201835 |
| fold 3 |  0.9701834862385321 |
| fold 4 |  0.9736238532110092 |
| OOF Acc |  0.9716743119266055 |


```
synset_word
ав        1.000000
ам        0.931507
баг       0.980000
байр      0.943548
бараа     0.964789
гар       0.950210
гол       0.938731
гүн       0.912088
зах       0.946667
зуу       0.995798
зүрх      0.918367
мөнгө     0.973333
нуруу     0.968750
нүд       1.000000
нүүр      0.987805
салбар    0.963636
сар       0.996627
сум       0.816667
тэрэг     0.822581
түүх      0.980237
төр       0.998428
хий       0.993077
хураа     0.858268
хэлбэр    0.727273
хөндий    1.000000
шат       1.000000
эм        1.000000
эрүүл     1.000000
dtype: float64
```


model name: mlub-bert-large-uncased-tr5do30ep25


tags:


|fold|accuracy|
|-|-|
| fold 0 |  0.9730504587155964 |
| fold 1 |  0.9690366972477065 |
| fold 2 |  0.970756880733945 |
| fold 3 |  0.9684633027522935 |
| fold 4 |  0.9719036697247706 |
| OOF Acc |  0.9706422018348624 |


model name: mlub-bert-large-cased-tr5do30ep25s42


language: mn
datasets:


- common_voice
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 Mongolian by Bayartsogt
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice mn
type: common_voice
args: mn\
metrics:
    - name: Test WER
type: wer
value: 45.82


# Wav2Vec2-Large-XLSR-53-Mongolian


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Mongolian using the [Common Voice](https://huggingface.co/datasets/common_voice)
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

test_dataset = load_dataset("common_voice", "mn", split="test[:2%]")

processor = Wav2Vec2Processor.from_pretrained("bayartsogt/wav2vec2-large-xlsr-mongolian")
model = Wav2Vec2ForCTC.from_pretrained("bayartsogt/wav2vec2-large-xlsr-mongolian")

resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
\\tspeech_array, sampling_rate = torchaudio.load(batch["path"])
\\tbatch["speech"] = resampler(speech_array).squeeze().numpy()
\\treturn batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
\\tlogits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits

predicted_ids = torch.argmax(logits, dim=-1)

print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

test_dataset = load_dataset("common_voice", "mn", split="test")
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("bayartsogt/wav2vec2-large-xlsr-mongolian")
model = Wav2Vec2ForCTC.from_pretrained("bayartsogt/wav2vec2-large-xlsr-mongolian")
model.to("cuda")

chars_to_ignore_regex = '[\\\\,\\\\?\\\\.\\\\!\\\\-\\\\;\\\\:\\\\"\\\\“\\\\%\\\\‘\\\\”\\\\�\\\\'h\\\\«\\\\»]'

resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
\\tbatch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
\\tspeech_array, sampling_rate = torchaudio.load(batch["path"])
\\tbatch["speech"] = resampler(speech_array).squeeze().numpy()
\\treturn batch

test_dataset = test_dataset.map(speech_file_to_array_fn)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
\\tinputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

\\twith torch.no_grad():
\\t\\tlogits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits

    pred_ids = torch.argmax(logits, dim=-1)
\\tbatch["pred_strings"] = processor.batch_decode(pred_ids)
\\treturn batch

result = test_dataset.map(evaluate, batched=True, batch_size=8)

print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 45.82%


## Training


❌ The Common Voice `train`, `validation`, and ... datasets were used for training as well as ... and ...  # TODO: adapt to state all the datasets that were used for training.


❌ The script used for training can be found [here](...) # TODO: fill in a link to your training script here. If you trained your model in a colab, simply fill in the link here. If you trained the model locally, it would be great if you could upload the training script on github and paste the link here.


model name: wav2vec2-large-xlsr-mongolian


model author: bayartsogt


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, mn, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


# StructBERT: Un-Official Copy


Official Repository Link: https://github.com/alibaba/AliceMind/tree/main/StructBERT


**Claimer**


* This model card is not produced by [AliceMind Team](https://github.com/alibaba/AliceMind/)


## Reproduce HFHub models:


Download model/tokenizer vocab


```bash
wget https://raw.githubusercontent.com/alibaba/AliceMind/main/StructBERT/config/large_bert_config.json && mv large_bert_config.json config.json
wget https://raw.githubusercontent.com/alibaba/AliceMind/main/StructBERT/config/vocab.txt
wget https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/StructBERT/en_model && mv en_model pytorch_model.bin
```


```python
from transformers import AutoConfig, AutoModelForMaskedLM, AutoTokenizer

config = AutoConfig.from_pretrained("./config.json")
model = AutoModelForMaskedLM.from_pretrained(".", config=config)
tokenizer = AutoTokenizer.from_pretrained(".", config=config)

model.push_to_hub("structbert-large")
tokenizer.push_to_hub("structbert-large")
```


# StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding


## Introduction


We extend BERT to a new model, StructBERT, by incorporating language structures into pre-training.
Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential
order of words and sentences, which leverage language structures at the word and sentence levels,
respectively.


## Pre-trained models


|Model | Description | #params | Download |
|------------------------|-------------------------------------------|------|------|
|structbert.en.large | StructBERT using the BERT-large architecture | 340M | [structbert.en.large](https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/StructBERT/en_model) |
|structroberta.en.large | StructRoBERTa continue training from RoBERTa | 355M | Coming soon |
|structbert.ch.large | Chinese StructBERT; BERT-large architecture | 330M | [structbert.ch.large](https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/StructBERT/ch_model) |


## Results


The results of GLUE & CLUE tasks can be reproduced using the hyperparameters listed in the following "Example usage" section.


#### structbert.en.large


[GLUE benchmark](https://gluebenchmark.com/leaderboard)


|Model| MNLI | QNLIv2 | QQP | SST-2 | MRPC |
|--------------------|-------|-------|-------|-------|-------|
|structbert.en.large |86.86% |93.04% |91.67% |93.23% |86.51% |


#### structbert.ch.large


[CLUE benchmark](https://www.cluebenchmarks.com/)


|Model | CMNLI | OCNLI | TNEWS | AFQMC |
|--------------------|-------|-------|-------|-------|
|structbert.ch.large |84.47% |81.28% |68.67% |76.11% |


## Example usage


#### Requirements and Installation


* [PyTorch](https://pytorch.org/) version >= 1.0.1
* Install other libraries via


```
pip install -r requirements.txt
```


* For faster training install NVIDIA's [apex](https://github.com/NVIDIA/apex) library


#### Finetune MNLI


```
python run_classifier_multi_task.py \
  --task_name MNLI \
  --do_train \
  --do_eval \
  --do_test \
  --amp_type O1 \
  --lr_decay_factor 1 \
  --dropout 0.1 \
  --do_lower_case \
  --detach_index -1 \
  --core_encoder bert \
  --data_dir path_to_glue_data \
  --vocab_file config/vocab.txt \
  --bert_config_file config/large_bert_config.json \
  --init_checkpoint path_to_pretrained_model \
  --max_seq_length 128 \
  --train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --fast_train \
  --gradient_accumulation_steps 1 \
  --output_dir path_to_output_dir
```


## Citation


If you use our work, please cite:


```
@article{wang2019structbert,
  title={Structbert: Incorporating language structures into pre-training for deep language understanding},
  author={Wang, Wei and Bi, Bin and Yan, Ming and Wu, Chen and Bao, Zuyi and Xia, Jiangnan and Peng, Liwei and Si, Luo},
  journal={arXiv preprint arXiv:1908.04577},
  year={2019}
}
```


model name: structbert-large


model author: bayartsogt


task: fill-mask