```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='albert-xxlarge-v2')
>>> unmasker("The man worked as a [MASK].")

[
   {
      "sequence":"[CLS] the man worked as a chauffeur.[SEP]",
      "score":0.029577180743217468,
      "token":28744,
      "token_str":"â–chauffeur"
   },
   {
      "sequence":"[CLS] the man worked as a janitor.[SEP]",
      "score":0.028865724802017212,
      "token":29477,
      "token_str":"â–janitor"
   },
   {
      "sequence":"[CLS] the man worked as a shoemaker.[SEP]",
      "score":0.02581118606030941,
      "token":29024,
      "token_str":"â–shoemaker"
   },
   {
      "sequence":"[CLS] the man worked as a blacksmith.[SEP]",
      "score":0.01849772222340107,
      "token":21238,
      "token_str":"â–blacksmith"
   },
   {
      "sequence":"[CLS] the man worked as a lawyer.[SEP]",
      "score":0.01820771023631096,
      "token":3672,
      "token_str":"â–lawyer"
   }
]

>>> unmasker("The woman worked as a [MASK].")

[
   {
      "sequence":"[CLS] the woman worked as a receptionist.[SEP]",
      "score":0.04604868218302727,
      "token":25331,
      "token_str":"â–receptionist"
   },
   {
      "sequence":"[CLS] the woman worked as a janitor.[SEP]",
      "score":0.028220869600772858,
      "token":29477,
      "token_str":"â–janitor"
   },
   {
      "sequence":"[CLS] the woman worked as a paramedic.[SEP]",
      "score":0.0261906236410141,
      "token":23386,
      "token_str":"â–paramedic"
   },
   {
      "sequence":"[CLS] the woman worked as a chauffeur.[SEP]",
      "score":0.024797942489385605,
      "token":28744,
      "token_str":"â–chauffeur"
   },
   {
      "sequence":"[CLS] the woman worked as a waitress.[SEP]",
      "score":0.024124596267938614,
      "token":13678,
      "token_str":"â–waitress"
   }
]
```


This bias will also affect all fine-tuned versions of this model.


## Training data


The ALBERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
headers).


## Training procedure


### Preprocessing


The texts are lowercased and tokenized using SentencePiece and a vocabulary size of 30,000. The inputs of the model are
then of the form:


```
[CLS] Sentence A [SEP] Sentence B [SEP]
```


### Training


The ALBERT procedure follows the BERT setup.


The details of the masking procedure for each sentence are the following:


- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.


## Evaluation results


When fine-tuned on downstream tasks, the ALBERT models achieve the following results:


|                | Average  | SQuAD1.1 | SQuAD2.0 | MNLI     | SST-2    | RACE     |
|----------------|----------|----------|----------|----------|----------|----------|
|V2              |
|ALBERT-base     |82.3      |90.2/83.2 |82.1/79.3 |84.6      |92.9      |66.8      |
|ALBERT-large    |85.7      |91.8/85.2 |84.9/81.8 |86.5      |94.9      |75.2      |
|ALBERT-xlarge   |87.9      |92.9/86.4 |87.9/84.1 |87.9      |95.4      |80.7      |
|ALBERT-xxlarge  |90.9      |94.6/89.1 |89.8/86.9 |90.6      |96.8      |86.8      |
|V1              |
|ALBERT-base     |80.1      |89.3/82.3 | 80.0/77.1|81.6      |90.3      | 64.0     |
|ALBERT-large    |82.4      |90.6/83.9 | 82.3/79.4|83.5      |91.7      | 68.5     |
|ALBERT-xlarge   |85.5      |92.5/86.1 | 86.1/83.1|86.4      |92.4      | 74.8     |
|ALBERT-xxlarge  |91.0      |94.8/89.3 | 90.2/87.4|90.8      |96.9      | 86.5     |


### BibTeX entry and citation info


```bibtex
@article{DBLP:journals/corr/abs-1909-11942,
  author    = {Zhenzhong Lan and
               Mingda Chen and
               Sebastian Goodman and
               Kevin Gimpel and
               Piyush Sharma and
               Radu Soricut},
  title     = {{ALBERT:} {A} Lite {BERT} for Self-supervised Learning of Language
               Representations},
  journal   = {CoRR},
  volume    = {abs/1909.11942},
  year      = {2019},
  url       = {http://arxiv.org/abs/1909.11942},
  archivePrefix = {arXiv},
  eprint    = {1909.11942},
  timestamp = {Fri, 27 Sep 2019 13:04:21 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1909-11942.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```


model name: albert-xxlarge-v2


task: fill-mask


tags: pytorch, tf, albert, fill-mask, en, dataset:bookcorpus, dataset:wikipedia, arxiv:1909.11942, transformers, exbert, license:apache-2.0, autonlp_compatible, infinity_compatible


language: en
tags:


- exbert
license: mit
datasets:
- bookcorpus
- wikipedia


# RoBERTa base model


Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
[this paper](https://arxiv.org/abs/1907.11692) and first released in
[this repository](https://github.com/pytorch/fairseq/tree/master/examples/roberta). This model is case-sensitive: it
makes a difference between english and English.


Disclaimer: The team releasing RoBERTa did not write a model card for this model so this model card has been written by
the Hugging Face team.


model name: distilbert-base-uncased-finetuned-emotion


model author: kjackson


tags: en, dataset:bookcorpus, dataset:wikipedia, arxiv:1907.11692, exbert, license:mit


pipeline_tag: sentence-similarity
tags:


- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers


# whaleloops/phrase-bert


This is the official repository for the EMNLP 2021 long paper [Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration](https://arxiv.org/abs/2109.06304). We provide [code](https://github.com/sf-wa-326/phrase-bert-topic-model) for training and evaluating Phrase-BERT in addition to the datasets used in the paper.


## Usage (Sentence-Transformers)


Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:


```
pip install -U sentence-transformers
```


Our model is tested on pytorch=1.9.0, tranformers=4.8.1, sentence-tranformers = 2.1.0 TODO


Then you can use the model like this:


```python
from sentence_transformers import SentenceTransformer
phrase_list = [ 'play an active role', 'participate actively', 'active lifestyle']

model = SentenceTransformer('whaleloops/phrase-bert')
phrase_embs = model.encode( phrase_list )
[p1, p2, p3] = phrase_embs
```


As in sentence-BERT, the default output is a list of numpy arrays:


```
for phrase, embedding in zip(phrase_list, phrase_embs):
    print("Phrase:", phrase)
    print("Embedding:", embedding)
    print("")
```


An example of computing the dot product of phrase embeddings:


```
import numpy as np
print(f'The dot product between phrase 1 and 2 is: {np.dot(p1, p2)}')
print(f'The dot product between phrase 1 and 3 is: {np.dot(p1, p3)}')
print(f'The dot product between phrase 2 and 3 is: {np.dot(p2, p3)}')
```


An example of computing cosine similarity of phrase embeddings:


```
import torch
from torch import nn
cos_sim = nn.CosineSimilarity(dim=0)
print(f'The cosine similarity between phrase 1 and 2 is: {cos_sim( torch.tensor(p1), torch.tensor(p2))}')
print(f'The cosine similarity between phrase 1 and 3 is: {cos_sim( torch.tensor(p1), torch.tensor(p3))}')
print(f'The cosine similarity between phrase 2 and 3 is: {cos_sim( torch.tensor(p2), torch.tensor(p3))}')
```


The output should look like:


```
The dot product between phrase 1 and 2 is: 218.43600463867188
The dot product between phrase 1 and 3 is: 165.48483276367188
The dot product between phrase 2 and 3 is: 160.51708984375
The cosine similarity between phrase 1 and 2 is: 0.8142536282539368
The cosine similarity between phrase 1 and 3 is: 0.6130303144454956
The cosine similarity between phrase 2 and 3 is: 0.584893524646759
```


## Evaluation


Given the lack of a unified phrase embedding evaluation benchmark, we collect the following five phrase semantics evaluation tasks, which are described further in our paper:


* Turney [[Download](https://storage.googleapis.com/phrase-bert/turney/data.txt) ]
* BiRD [[Download](https://storage.googleapis.com/phrase-bert/bird/data.txt)]
* PPDB [[Download](https://storage.googleapis.com/phrase-bert/ppdb/examples.json)]
* PPDB-filtered [[Download](https://storage.googleapis.com/phrase-bert/ppdb_exact/examples.json)]
* PAWS-short [[Download Train-split](https://storage.googleapis.com/phrase-bert/paws_short/train_examples.json) ] [[Download Dev-split](https://storage.googleapis.com/phrase-bert/paws_short/dev_examples.json) ] [[Download Test-split](https://storage.googleapis.com/phrase-bert/paws_short/test_examples.json) ]


Change `config/model_path.py` with the model path according to your directories and


* For evaluation on Turney, run `python eval_turney.py`
* For evaluation on BiRD, run `python eval_bird.py`
* for evaluation on PPDB / PPDB-filtered / PAWS-short, run `eval_ppdb_paws.py` with:

  ```
  nohup python  -u eval_ppdb_paws.py \
      --full_run_mode \
      --task <task-name> \
      --data_dir <input-data-dir> \
      --result_dir <result-storage-dr> \
      >./output.txt 2>&1 &
  ```


## Train your own Phrase-BERT


If you would like to go beyond using the pre-trained Phrase-BERT model, you may train your own Phrase-BERT using data from the domain you are interested in. Please refer to
`phrase-bert/phrase_bert_finetune.py`


The datasets we used to fine-tune Phrase-BERT are here: [training data csv file](https://storage.googleapis.com/phrase-bert/phrase-bert-ft-data/pooled_context_para_triples_p%3D0.8_train.csv) and [validation data csv file](https://storage.googleapis.com/phrase-bert/phrase-bert-ft-data/pooled_context_para_triples_p%3D0.8_valid.csv).


To re-produce the trained Phrase-BERT, please run:


    export INPUT_DATA_PATH=<directory-of-phrasebert-finetuning-data>
    export TRAIN_DATA_FILE=<training-data-filename.csv>
    export VALID_DATA_FILE=<validation-data-filename.csv>
    export INPUT_MODEL_PATH=bert-base-nli-stsb-mean-tokens
    export OUTPUT_MODEL_PATH=<directory-of-saved-model>

    python -u phrase_bert_finetune.py \
        --input_data_path $INPUT_DATA_PATH \
        --train_data_file $TRAIN_DATA_FILE \
        --valid_data_file $VALID_DATA_FILE \
        --input_model_path $INPUT_MODEL_PATH \
        --output_model_path $OUTPUT_MODEL_PATH


## Citation:


Please cite us if you find this useful:


```
@inproceedings{phrasebertwang2021,
    author={Shufan Wang and Laure Thompson and Mohit Iyyer},
    Booktitle = {Empirical Methods in Natural Language Processing},
    Year = "2021",
    Title={Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration}
}
```


model name: phrase-bert


model author: whaleloops


task: sentence-similarity


tags: pytorch, bert, feature-extraction, arxiv:2109.06304, sentence-transformers, sentence-similarity, transformers, infinity_compatible


Info here: https://github.com/josephrocca/openai-clip-js


model name: openai-clip-js


model author: rocca


tags: onnx


Bad Modell for Research Purposes!


model name: wav2vec2-large-xlsr-german


model author: unknownTransformer


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, transformers


UzRoBerta model.


Pre-prepared model in Uzbek (Cyrillic script) to model the masked language and predict the next sentences.


Training data.


UzBERT model was pretrained on ≈167K news articles (≈568Mb).


model name: uztext_568Mb_Roberta_BPE


model author: rifkat


task: fill-mask


tags: pytorch, roberta, fill-mask, transformers, autonlp_compatible, infinity_compatible


Ushbu model, HuggingFace-da RoBERTa transformatorini amalga oshirishga asoslangan. Bizning RoBERTa dasturimiz 12 ta diqqat boshi va 6 ta qatlamdan foydalanadi, natijada 72 ta aniq e'tibor mexanizmlari paydo bo'ladi. Biz har bir kirish satridagi tokenlarning 15 foizini niqoblaydigan RoBERTa-dan dastlabki tekshirish protsedurasini qabul qildik. Biz maksimal 52K tokenli lug'atdan va maksimal 512 ta ketma-ketlik uzunligidan foydalanganmiz. Biz 1M PubChem to'plamlarida 10 ta davr uchun o'qitdik. Loss funksiya 2.9 dan 0.33 gacha tushdi. Ushbu modelni sizga taqdim qilamiz.


model name: pubchem_1M


Pre-prepared model in Uzbek (Cyrillic and latin script) to model the masked language and predict the next sentences.


UzBERT model was pretrained on ≈2M news articles (≈3Gb).


model name: uztext-3Gb-BPE-Roberta


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-dansk-CV-80
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-300m-dansk-CV-80


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) for Danish, using the [mozilla-foundation/common_voice_8_0](https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0) dataset.


It achieves the following results on the evaluation set:


- eval_loss: 0.6394
- eval_wer: 0.3682
- eval_runtime: 104.0466
- eval_samples_per_second: 13.359
- eval_steps_per_second: 1.672
- epoch: 21.28
- step: 2000


## Model description


ASR Danish model


## Intended uses & limitations


More information needed


## Training and evaluation data


Danish subset of [mozilla-foundation/common_voice_8_0](https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0)


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 30
- mixed_precision_training: Native AMP


### Framework versions


- Transformers 4.16.1
- Pytorch 1.10.0+cu111
- Datasets 1.18.2
- Tokenizers 0.11.0


model name: wav2vec2-large-xls-r-300m-dansk-CV-80


model author: vachonni


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:common_voice, transformers, license:apache-2.0, generated_from_trainer, model-index


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-large-xls-r-300m-da-colab
results: []


# wav2vec2-large-xls-r-300m-da-colab


This model is a fine-tuned version of [Alvenir/wav2vec2-base-da](https://huggingface.co/Alvenir/wav2vec2-base-da) on the None dataset.


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


### Training results


### Framework versions


- Transformers 4.11.3
- Pytorch 1.10.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: wav2vec2-large-xls-r-300m-da-colab


model author: vachonni


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, transformers, license:apache-2.0, generated_from_trainer, model-index


language: pl
datasets:


- enelpol/czywiesz
task_categories:
- question_answering
task_ids:
- open-domain-qa
multilinguality:
- monolingual
size_categories:
- 1k<n<10K


This is the question encoder for the Polish DPR question answering model. The full model consists of two encoders.
Please read [context encoder documentation](https://huggingface.co/enelpol/czywiesz-context) to get the details of the model.


model name: czywiesz-question


model author: enelpol


task: feature-extraction


tags: pytorch, bert, feature-extraction, pl, dataset:enelpol/czywiesz, transformers, infinity_compatible


- enelpol/czywiesz


# Model description


The model was created for selective question answering in Polish. I.e. it is used to find passages containing the answers to the given question.


It is used to encode the contexts (aka passages) in the DPR bi-encoder architecture. The architecture requires two separate models.
The question part has to be encoded with the corresponding [question encoder](https://huggingface.co/enelpol/czywiesz-question).


The model was created by fine-tuning [Herbert base cased](https://huggingface.co/allegro/herbert-base-cased) on "Czywiesz" dataset.
[Czywiesz](https://clarin-pl.eu/dspace/handle/11321/39) dataset contains questions and Wikipedia articles extracted from the Polish Wikipedia.


# Usage


It is the easiest to use the model with the [Haystack framework](https://haystack.deepset.ai/overview/intro).


```python
from haystack.document_stores import FAISSDocumentStore
from haystack.retriever import DensePassageRetriever

document_store = FAISSDocumentStore(faiss_index_factory_str="Flat")

retriever = DensePassageRetriever(
    document_store=document_store,
    query_embedding_model="enelpol/czywiesz-question",
    passage_embedding_model="enelpol/czywiesz-context"
)

for document in documents:
    document_store.write_documents([document])

document_store.udpate_embeddings(retriever)
document_store.save("contexts.faiss")

```


model name: czywiesz-context


tags:


- conversational


# Bapibot


model name: bapismall


model author: NanniKirby


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


model name: DialoGPT-medium-bapi


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- wmt16
metrics:
- bleu
model-index:
- name: t5-small-finetuned-en-to-ro-fp16_off
results:
  - task:
name: Sequence-to-sequence Language Modeling
type: text2text-generation
dataset:
name: wmt16
type: wmt16
args: ro-en
metrics:
    - name: Bleu
type: bleu
value: 5.9132


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# t5-small-finetuned-en-to-ro-fp16_off


This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on the wmt16 dataset.
It achieves the following results on the evaluation set:


- Loss: 1.8351
- Bleu: 5.9132
- Gen Len: 18.2656


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1


### Training results


| Training Loss | Epoch | Step | Validation Loss | Bleu   | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:-------:|
| 0.8501        | 1.0   | 7629 | 1.8351          | 5.9132 | 18.2656 |


### Framework versions


- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


model name: t5-small-finetuned-en-to-ro-fp16_off


model author: eliotm


task: text2text-generation


tags: pytorch, tensorboard, t5, text2text-generation, dataset:wmt16, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- wmt16
metrics:
- bleu
model-index:
- name: t5-small-finetuned-en-to-ro-lr0.001
results:
  - task:
name: Sequence-to-sequence Language Modeling
type: text2text-generation
dataset:
name: wmt16
type: wmt16
args: ro-en
metrics:
    - name: Bleu
type: bleu
value: 5.8837


# t5-small-finetuned-en-to-ro-lr0.001


- Loss: 1.8309
- Bleu: 5.8837
- Gen Len: 18.2656


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.01
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Bleu   | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:-------:|
| 0.9442        | 1.0   | 7629 | 1.8309          | 5.8837 | 18.2656 |


### Framework versions


model name: t5-small-finetuned-en-to-ro-lr0.001


model author: eliotm


task: text2text-generation


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- wmt16
metrics:
- bleu
model-index:
- name: t5-small-finetuned-en-to-ro-lr_2e-6
results:
  - task:
name: Sequence-to-sequence Language Modeling
type: text2text-generation
dataset:
name: wmt16
type: wmt16
args: ro-en
metrics:
    - name: Bleu
type: bleu
value: 7.2935


# t5-small-finetuned-en-to-ro-lr_2e-6


- Loss: 1.4232
- Bleu: 7.2935
- Gen Len: 18.2521


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-06
- train_batch_size: 10
- eval_batch_size: 10
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 0.04375
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Bleu   | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:-------:|
| 0.6703        | 0.04  | 2671 | 1.4232          | 7.2935 | 18.2521 |


### Framework versions


model name: t5-small-finetuned-en-to-ro-lr_2e-6


model author: eliotm


task: text2text-generation


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- wmt16
metrics:
- bleu
model-index:
- name: t5-small-finetuned-en-to-ro-LR_1e-3
results:
  - task:
name: Sequence-to-sequence Language Modeling
type: text2text-generation
dataset:
name: wmt16
type: wmt16
args: ro-en
metrics:
    - name: Bleu
type: bleu
value: 7.1606


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# t5-small-finetuned-en-to-ro-LR_1e-3


This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) on the wmt16 dataset.
It achieves the following results on the evaluation set:


- Loss: 1.5215
- Bleu: 7.1606
- Gen Len: 18.2451


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.001
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Bleu   | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:-------:|
| 0.6758        | 1.0   | 7629 | 1.5215          | 7.1606 | 18.2451 |


### Framework versions


- Transformers 4.12.5
- Pytorch 1.10.0+cu111
- Datasets 1.16.1
- Tokenizers 0.10.3


model name: t5-small-finetuned-en-to-ro-LR_1e-3


model author: eliotm


task: text2text-generation


tags: pytorch, tensorboard, t5, text2text-generation, dataset:wmt16, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible


# Sentiment Analysis in Spanish


## beto-sentiment-analysis


Repository: [https://github.com/finiteautomata/pysentimiento/](https://github.com/finiteautomata/pysentimiento/)
Model trained with TASS 2020 corpus (around ~5k tweets) of several dialects of Spanish. Base model is [BETO](https://github.com/dccuchile/beto), a BERT model trained in Spanish.
Uses `POS`, `NEG`, `NEU` labels.
**Coming soon**: a brief paper describing the model and training.
Enjoy! 🤗


model name: jd-resume-model


model author: nikunjbjj


task: text-classification


tags: pytorch, jax, bert, text-classification, transformers, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- squad
model-index:
- name: distilbert-base-uncased-finetuned-squad
results:
  - task:
name: Question Answering
type: question-answering
dataset:
name: squad
type: squad
args: plain_text


# distilbert-base-uncased-finetuned-squad


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the squad dataset.
It achieves the following results on the evaluation set:


- Loss: 1.1582


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3


### Training results


| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.2176        | 1.0   | 5533  | 1.1429          |
| 0.9425        | 2.0   | 11066 | 1.1196          |
| 0.7586        | 3.0   | 16599 | 1.1582          |


### Framework versions


- Transformers 4.10.0
- Pytorch 1.9.0+cu102
- Datasets 1.11.0
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-squad


model author: Hoang


task: question-answering


tags: pytorch, tensorboard, distilbert, question-answering, dataset:squad, transformers, license:apache-2.0, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


library_name: superb
benchmark: superb
task: asr
datasets:


- superb
tags:
- automatic-speech-recognition
- osanseviero/hubert_base
widget:
- example_title: Librispeech sample 1
src: https://cdn-media.huggingface.co/speech_samples/sample1.flac


# Fine-tuned s3prl model for ASR


model name: superb-s3prl-osanseviero__hubert_base-asr-c61a5cff


model author: leo19941227


task: automatic-speech-recognition


tags: tensorboard, dataset:superb, superb, automatic-speech-recognition, osanseviero/hubert_base


tags:


- conversational


# Santa Chatbot


model name: Dialog-santa


model author: KringleClaus


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


language:


- pt
tags:
- generated_from_trainer
datasets:
- lener_br
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: checkpoints
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: lener_br
type: lener_br
metrics:
    - name: F1
type: f1
value: 0.9082022949426265
    - name: Precision
type: precision
value: 0.8975220495590088
    - name: Recall
type: recall
value: 0.9191397849462366
    - name: Accuracy
type: accuracy
value: 0.9808310603867311
    - name: Loss
type: loss
value: 0.1228889599442482
widget:
- text: "Ao Instituto Médico Legal da jurisdição do acidente ou da residência cumpre fornecer, no prazo de 90 dias, laudo à vítima (art. 5, § 5, Lei n. 6.194/74  de 19 de dezembro de 1974), função técnica que pode ser suprida por prova pericial realizada por ordem do juízo da causa, ou por prova técnica realizada no âmbito administrativo que se mostre coerente com os demais elementos de prova constante dos autos."
- text: "Acrescento que não há de se falar em violação do artigo 114, § 3º, da Constituição Federal, posto que referido dispositivo revela-se impertinente, tratando da possibilidade de ajuizamento de dissídio coletivo pelo Ministério Público do Trabalho nos casos de greve em atividade essencial."
- text: "Todavia, entendo que extrair da aludida norma o sentido expresso na redação acima implica desconstruir o significado do texto constitucional, o que é absolutamente vedado ao intérprete. Nesse sentido, cito Dimitri Dimoulis: ‘(...) ao intérprete não é dado escolher significados que não estejam abarcados pela moldura da norma. Interpretar não pode significar violentar a norma.’ (Positivismo Jurídico. São Paulo: Método, 2006, p. 220).59. Dessa forma, deve-se tomar o sentido etimológico como limite da atividade interpretativa, a qual não pode superado, a ponto de destruir a própria norma a ser interpretada. Ou, como diz Konrad Hesse, ‘o texto da norma é o limite insuperável da atividade interpretativa.’ (Elementos de Direito Constitucional da República Federal da Alemanha, Porto Alegre: Sergio Antonio Fabris, 2003, p. 71)."


## (BERT large) NER model in the legal domain in Portuguese (LeNER-Br)


**ner-bert-large-portuguese-cased-lenerbr** is a NER model (token classification) in the legal domain in Portuguese that was finetuned on 20/12/2021 in Google Colab from the model [pierreguillou/bert-large-cased-pt-lenerbr](https://huggingface.co/pierreguillou/bert-large-cased-pt-lenerbr) on the dataset [LeNER_br](https://huggingface.co/datasets/lener_br) by using a NER objective.


Due to the small size of the finetuning dataset, the model overfitted before to reach the end of training. Here are the overall final metrics on the validation dataset (*note: see the paragraph "Validation metrics by Named Entity" to get detailed metrics*):


- **f1**: 0.9082022949426265
- **precision**: 0.8975220495590088
- **recall**: 0.9191397849462366
- **accuracy**: 0.9808310603867311
- **loss**: 0.1228889599442482


Check as well the [base version of this model](https://huggingface.co/pierreguillou/ner-bert-base-cased-pt-lenerbr) with a f1 of 0.893.


**Note**: the model [pierreguillou/bert-large-cased-pt-lenerbr](https://huggingface.co/pierreguillou/bert-large-cased-pt-lenerbr) is a language model that was created through the finetuning of the model [BERTimbau large](https://huggingface.co/neuralmind/bert-large-portuguese-cased) on the dataset [LeNER-Br language modeling](https://huggingface.co/datasets/pierreguillou/lener_br_finetuning_language_model) by using a MASK objective. This first specialization of the language model before finetuning on the NER task allows to get a better NER model.


## Blog post


[NLP | Modelos e Web App para Reconhecimento de Entidade Nomeada (NER) no domínio jurídico brasileiro](https://medium.com/@pierre_guillou/nlp-modelos-e-web-app-para-reconhecimento-de-entidade-nomeada-ner-no-dom%C3%ADnio-jur%C3%ADdico-b658db55edfb) (29/12/2021)


## Widget & App


You can test this model into the widget of this page.


Use as well the [NER App](https://huggingface.co/spaces/pierreguillou/ner-bert-pt-lenerbr) that allows comparing the 2 BERT models (base and large) fitted in the NER task with the legal LeNER-Br dataset.


## Using the model for inference in production


```
# install pytorch: check https://pytorch.org/
# !pip install transformers
from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch

# parameters
model_name = "pierreguillou/ner-bert-large-cased-pt-lenerbr"
model = AutoModelForTokenClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

input_text = "Acrescento que não há de se falar em violação do artigo 114, § 3º, da Constituição Federal, posto que referido dispositivo revela-se impertinente, tratando da possibilidade de ajuizamento de dissídio coletivo pelo Ministério Público do Trabalho nos casos de greve em atividade essencial."

# tokenization
inputs = tokenizer(input_text, max_length=512, truncation=True, return_tensors="pt")
tokens = inputs.tokens()

# get predictions
outputs = model(**inputs).logits
predictions = torch.argmax(outputs, dim=2)

# print predictions
for token, prediction in zip(tokens, predictions[0].numpy()):
    print((token, model.config.id2label[prediction]))
```


You can use pipeline, too. However, it seems to have an issue regarding to the max_length of the input sequence.


```
!pip install transformers
import transformers
from transformers import pipeline

model_name = "pierreguillou/ner-bert-large-cased-pt-lenerbr"

ner = pipeline(
    "ner",
    model=model_name
)

ner(input_text)
```


## Training procedure


### Notebook


The notebook of finetuning ([HuggingFace_Notebook_token_classification_NER_LeNER_Br.ipynb](https://github.com/piegu/language-models/blob/master/HuggingFace_Notebook_token_classification_NER_LeNER_Br.ipynb)) is in github.


### Hyperparameters


# batch, learning rate...


- per_device_batch_size = 2
- gradient_accumulation_steps = 2
- learning_rate = 2e-5
- num_train_epochs = 10
- weight_decay = 0.01
- optimizer = AdamW
- betas = (0.9,0.999)
- epsilon = 1e-08
- lr_scheduler_type = linear
- seed = 42


# save model & load best model


- save_total_limit = 7
- logging_steps = 500
- eval_steps = logging_steps
- evaluation_strategy = 'steps'
- logging_strategy = 'steps'
- save_strategy = 'steps'
- save_steps = logging_steps
- load_best_model_at_end = True
- fp16 = True


# get best model through a metric


- metric_for_best_model = 'eval_f1'
- greater_is_better = True


### Training results


```
Num examples = 7828
Num Epochs = 20
Instantaneous batch size per device = 2
Total train batch size (w. parallel, distributed & accumulation) = 4
Gradient Accumulation steps = 2
Total optimization steps = 39140

Step   Training Loss  Validation Loss  Precision  Recall    F1        Accuracy
500    0.250000       0.140582         0.760833   0.770323  0.765548  0.963125
1000   0.076200       0.117882         0.829082   0.817849  0.823428  0.966569
1500   0.082400       0.150047         0.679610   0.914624  0.779795  0.957213
2000   0.047500       0.133443         0.817678   0.857419  0.837077  0.969190
2500   0.034200       0.230139         0.895672   0.845591  0.869912  0.964070
3000   0.033800       0.108022         0.859225   0.887312  0.873043  0.973700
3500   0.030100       0.113467         0.855747   0.885376  0.870310  0.975879
4000   0.029900       0.118619         0.850207   0.884946  0.867229  0.974477
4500   0.022500       0.124327         0.841048   0.890968  0.865288  0.975041
5000   0.020200       0.129294         0.801538   0.918925  0.856227  0.968077
5500   0.019700       0.128344         0.814222   0.908602  0.858827  0.969250
6000   0.024600       0.182563         0.908087   0.866882  0.887006  0.968565
6500   0.012600       0.159217         0.829883   0.913763  0.869806  0.969357
7000   0.020600       0.183726         0.854557   0.893333  0.873515  0.966447
7500   0.014400       0.141395         0.777716   0.905161  0.836613  0.966828
8000   0.013400       0.139378         0.873042   0.899140  0.885899  0.975772
8500   0.014700       0.142521         0.864152   0.901505  0.882433  0.976366

9000   0.010900       0.122889         0.897522   0.919140  0.908202  0.980831

9500   0.013500       0.143407         0.816580   0.906667  0.859268  0.973395
10000  0.010400       0.144946         0.835608   0.908387  0.870479  0.974629
10500  0.007800       0.143086         0.847587   0.910108  0.877735  0.975985
11000  0.008200       0.156379         0.873778   0.884301  0.879008  0.976321
11500  0.008200       0.133356         0.901193   0.910108  0.905628  0.980328
12000  0.006900       0.133476         0.892202   0.920215  0.905992  0.980572
12500  0.006900       0.129991         0.890159   0.904516  0.897280  0.978683
```


### Validation metrics by Named Entity


```
{'JURISPRUDENCIA': {'f1': 0.8135593220338984,
  'number': 657,
  'precision': 0.865979381443299,
  'recall': 0.7671232876712328},
 'LEGISLACAO': {'f1': 0.8888888888888888,
  'number': 571,
  'precision': 0.8952042628774423,
  'recall': 0.882661996497373},
 'LOCAL': {'f1': 0.850467289719626,
  'number': 194,
  'precision': 0.7777777777777778,
  'recall': 0.9381443298969072},
 'ORGANIZACAO': {'f1': 0.8740635033892258,
  'number': 1340,
  'precision': 0.8373205741626795,
  'recall': 0.914179104477612},
 'PESSOA': {'f1': 0.9836677554829678,
  'number': 1072,
  'precision': 0.9841269841269841,
  'recall': 0.9832089552238806},
 'TEMPO': {'f1': 0.9669669669669669,
  'number': 816,
  'precision': 0.9481743227326266,
  'recall': 0.9865196078431373},
 'overall_accuracy': 0.9808310603867311,
 'overall_f1': 0.9082022949426265,
 'overall_precision': 0.8975220495590088,
 'overall_recall': 0.9191397849462366}
```


model name: ner-bert-large-cased-pt-lenerbr


model author: pierreguillou


task: token-classification


tags: pytorch, bert, token-classification, pt, dataset:lener_br, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


language: pt


widget:


- text: "Quem era Jim Henson? Jim Henson era um"
- text: "Em um achado chocante, o cientista descobriu um"
- text: "Barack Hussein Obama II, nascido em 4 de agosto de 1961, é"
- text: "Corrida por vacina contra Covid-19 já tem"
license: mit
datasets:
- wikipedia


# GPorTuguese-2: a Language Model for Portuguese text generation (and more NLP tasks...)


## Introduction


GPorTuguese-2 (Portuguese GPT-2 small) is a state-of-the-art language model for Portuguese based on the GPT-2 small model.


It was trained on Portuguese Wikipedia using **Transfer Learning and Fine-tuning techniques** in just over a day, on one GPU NVIDIA V100 32GB and with a little more than 1GB of training data.


It is a proof-of-concept that it is possible to get a state-of-the-art language model in any language with low ressources.


It was fine-tuned from the [English pre-trained GPT-2 small](https://huggingface.co/gpt2) using the Hugging Face libraries (Transformers and Tokenizers) wrapped into the [fastai v2](https://dev.fast.ai/) Deep Learning framework. All the fine-tuning fastai v2 techniques were used.


It is now available on Hugging Face. For further information or requests, please go to "[Faster than training from scratch — Fine-tuning the English GPT-2 in any language with Hugging Face and fastai v2 (practical case with Portuguese)](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787)".


## Model


| Model                   | #params | Model file (pt/tf) | Arch.       | Training /Validation data (text)         |
|-------------------------|---------|--------------------|-------------|------------------------------------------|
| `gpt2-small-portuguese` | 124M    | 487M / 475M        | GPT-2 small | Portuguese Wikipedia (1.28 GB / 0.32 GB) |


## Evaluation results


In a little more than a day (we only used one GPU NVIDIA V100 32GB; through a Distributed Data Parallel (DDP) training mode, we could have divided by three this time to 10 hours, just with 2 GPUs), we got a loss of 3.17, an **accuracy of 37.99%** and a **perplexity of 23.76** (see the validation results table below).


| after ... epochs | loss | accuracy (%) | perplexity | time by epoch | cumulative time |
|------------------|------|--------------|------------|---------------|-----------------|
|         0        | 9.95 |      9.90    |  20950.94  |    00:00:00   |     00:00:00    |
|         1        | 3.64 |     32.52    |     38.12  |     5:48:31   |      5:48:31    |
|         2        | 3.30 |     36.29    |     27.16  |     5:38:18   |     11:26:49    |
|         3        | 3.21 |     37.46    |     24.71  |     6:20:51   |     17:47:40    |
|         4        | 3.19 |     37.74    |     24.21  |     6:06:29   |     23:54:09    |
|         5        | 3.17 |     37.99    |     23.76  |     6:16:22   |     30:10:31    |


## GPT-2


*Note: information copied/pasted from [Model: gpt2 >> GPT-2](https://huggingface.co/gpt2#gpt-2)*


Pretrained model on English language using a causal language modeling (CLM) objective. It was introduced in this [paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) and first released at this [page](https://openai.com/blog/better-language-models/) (February 14, 2019).


Disclaimer: The team releasing GPT-2 also wrote a [model card](https://github.com/openai/gpt-2/blob/master/model_card.md) for their model. Content from this model card has been written by the Hugging Face team to complete the information they provided and give specific examples of bias.


## Model description


*Note: information copied/pasted from [Model: gpt2 >> Model description](https://huggingface.co/gpt2#model-description)*


GPT-2 is a transformers model pretrained on a very large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it was trained to guess the next word in sentences.


More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence, shifted one token (word or piece of word) to the right. The model uses internally a mask-mechanism to make sure the predictions for the token `i` only uses the inputs from `1` to `i` but not the future tokens.


This way, the model learns an inner representation of the English language that can then be used to extract features useful for downstream tasks. The model is best at what it was pretrained for however, which is generating texts from a prompt.


## How to use GPorTuguese-2 with HuggingFace (PyTorch)


The following code use PyTorch. To use TensorFlow, check the below corresponding paragraph.


### Load GPorTuguese-2 and its sub-word tokenizer (Byte-level BPE)


```python
from transformers import AutoTokenizer, AutoModelWithLMHead
import torch

tokenizer = AutoTokenizer.from_pretrained("pierreguillou/gpt2-small-portuguese")
model = AutoModelWithLMHead.from_pretrained("pierreguillou/gpt2-small-portuguese")

# Get sequence length max of 1024
tokenizer.model_max_length=1024

model.eval()  # disable dropout (or leave in train mode to finetune)
```


### Generate one word


```python
# input sequence
text = "Quem era Jim Henson? Jim Henson era um"
inputs = tokenizer(text, return_tensors="pt")

# model output
outputs = model(**inputs, labels=inputs["input_ids"])
loss, logits = outputs[:2]
predicted_index = torch.argmax(logits[0, -1, :]).item()
predicted_text = tokenizer.decode([predicted_index])

# results
print('input text:', text)
print('predicted text:', predicted_text)

# input text: Quem era Jim Henson? Jim Henson era um
# predicted text:  homem
```


### Generate one full sequence


```python
# input sequence
text = "Quem era Jim Henson? Jim Henson era um"
inputs = tokenizer(text, return_tensors="pt")

# model output using Top-k sampling text generation method
sample_outputs = model.generate(inputs.input_ids,
                                pad_token_id=50256,
                                do_sample=True,
                                max_length=50, # put the token number you want
                                top_k=40,
                                num_return_sequences=1)

# generated sequence
for i, sample_output in enumerate(sample_outputs):
    print(">> Generated text {}\n\n{}".format(i+1, tokenizer.decode(sample_output.tolist())))

# >> Generated text
# Quem era Jim Henson? Jim Henson era um executivo de televisão e diretor de um grande estúdio de cinema mudo chamado Selig,
# depois que o diretor de cinema mudo Georges Seuray dirigiu vários filmes para a Columbia e o estúdio.
```


## How to use GPorTuguese-2 with HuggingFace (TensorFlow)


The following code use TensorFlow. To use PyTorch, check the above corresponding paragraph.


```python
from transformers import AutoTokenizer, TFAutoModelWithLMHead
import tensorflow as tf

tokenizer = AutoTokenizer.from_pretrained("pierreguillou/gpt2-small-portuguese")
model = TFAutoModelWithLMHead.from_pretrained("pierreguillou/gpt2-small-portuguese")

# Get sequence length max of 1024
tokenizer.model_max_length=1024

model.eval()  # disable dropout (or leave in train mode to finetune)
```


```python
# input sequence
text = "Quem era Jim Henson? Jim Henson era um"
inputs = tokenizer.encode(text, return_tensors="tf")

# model output using Top-k sampling text generation method
outputs = model.generate(inputs, eos_token_id=50256, pad_token_id=50256,
                         do_sample=True,
                         max_length=40,
                         top_k=40)
print(tokenizer.decode(outputs[0]))

# >> Generated text
# Quem era Jim Henson? Jim Henson era um amigo familiar da família. Ele foi contratado pelo seu pai
# para trabalhar como aprendiz no escritório de um escritório de impressão, e então começou a ganhar dinheiro

```


## Limitations and bias


The training data used for this model come from Portuguese Wikipedia. We know it contains a lot of unfiltered content from the internet, which is far from neutral. As the openAI team themselves point out in their model card:


> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases that require the generated text to be true. Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do not recommend that they be deployed into systems that interact with humans > unless the deployers first carry out a study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race, and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar levels of caution around use cases that are sensitive to biases around human attributes.


## Author


Portuguese GPT-2 small was trained and evaluated by [Pierre GUILLOU](https://www.linkedin.com/in/pierreguillou/) thanks to the computing power of the GPU (GPU NVIDIA V100 32 Go) of the [AI Lab](https://www.linkedin.com/company/ailab-unb/) (University of Brasilia) to which I am attached as an Associate Researcher in NLP and the participation of its directors in the definition of NLP strategy, Professors Fabricio Ataides Braz and Nilton Correia da Silva.


## Citation


If you use our work, please cite:


```bibtex
@inproceedings{pierre2020gpt2smallportuguese,
  title={GPorTuguese-2 (Portuguese GPT-2 small): a Language Model for Portuguese text generation (and more NLP tasks...)},
  author={Pierre Guillou},
  year={2020}
}
```


model name: gpt2-small-portuguese


model author: pierreguillou


task: text-generation


tags: pytorch, tf, jax, gpt2, text-generation, pt, dataset:wikipedia, transformers, license:mit


language: pt
license: mit
tags:


- question-answering
- bert
- bert-large
- pytorch
datasets:
- brWaC
- squad
- squad_v1_pt
metrics:
- squad
widget:
- text: "Quando começou a pandemia de Covid-19 no mundo?"
context: "A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). O vírus tem origem zoonótica e o primeiro caso conhecido da doença remonta a dezembro de 2019 em Wuhan, na China."
- text: "Onde foi descoberta a Covid-19?"
context: "A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). O vírus tem origem zoonótica e o primeiro caso conhecido da doença remonta a dezembro de 2019 em Wuhan, na China."


# Portuguese BERT large cased QA (Question Answering), finetuned on SQUAD v1.1


![Exemple of what can do the Portuguese BERT large cased QA (Question Answering), finetuned on SQUAD v1.1](https://miro.medium.com/max/5256/1*QxyeAjT2V1OfE2B6nEcs3w.png)


## Introduction


The model was trained on the dataset SQUAD v1.1 in portuguese from the [Deep Learning Brasil group](http://www.deeplearningbrasil.com.br/).


The language model used is the [BERTimbau Large](https://huggingface.co/neuralmind/bert-large-portuguese-cased) (aka "bert-large-portuguese-cased") from [Neuralmind.ai](https://neuralmind.ai/): BERTimbau is a pretrained BERT model for Brazilian Portuguese that achieves state-of-the-art performances on three downstream NLP tasks: Named Entity Recognition, Sentence Textual Similarity and Recognizing Textual Entailment. It is available in two sizes: Base and Large.


## Informations on the method used


All the informations are in the blog post : [NLP | Como treinar um modelo de Question Answering em qualquer linguagem baseado no BERT large, melhorando o desempenho do modelo utilizando o BERT base? (estudo de caso em português)](https://medium.com/@pierre_guillou/nlp-como-treinar-um-modelo-de-question-answering-em-qualquer-linguagem-baseado-no-bert-large-1c899262dd96)


## Notebook in GitHub


## Performance


The results obtained are the following:


```
f1 = 84.43 (against 82.50 for the base model)
exact match = 72.68 (against 70.49 for the base model)
```


## How to use the model... with Pipeline


```python
import transformers
from transformers import pipeline

# source: https://pt.wikipedia.org/wiki/Pandemia_de_COVID-19
context = r"""
A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19,
uma doença respiratória causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2).
O vírus tem origem zoonótica e o primeiro caso conhecido da doença remonta a dezembro de 2019 em Wuhan, na China.
Em 20 de janeiro de 2020, a Organização Mundial da Saúde (OMS) classificou o surto
como Emergência de Saúde Pública de Âmbito Internacional e, em 11 de março de 2020, como pandemia.
Em 18 de junho de 2021, 177 349 274 casos foram confirmados em 192 países e territórios,
com 3 840 181 mortes atribuídas à doença, tornando-se uma das pandemias mais mortais da história.
Os sintomas de COVID-19 são altamente variáveis, variando de nenhum a doenças com risco de morte.
O vírus se espalha principalmente pelo ar quando as pessoas estão perto umas das outras.
Ele deixa uma pessoa infectada quando ela respira, tosse, espirra ou fala e entra em outra pessoa pela boca, nariz ou olhos.
Ele também pode se espalhar através de superfícies contaminadas.
As pessoas permanecem contagiosas por até duas semanas e podem espalhar o vírus mesmo se forem assintomáticas.
"""

model_name = 'pierreguillou/bert-large-cased-squad-v1.1-portuguese'
nlp = pipeline("question-answering", model=model_name)

question = "Quando começou a pandemia de Covid-19 no mundo?"

result = nlp(question=question, context=context)

print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")

# Answer: 'dezembro de 2019', score: 0.5087, start: 290, end: 306
```


## How to use the model... with the Auto classes


```python
from transformers import AutoTokenizer, AutoModelForQuestionAnswering

tokenizer = AutoTokenizer.from_pretrained("pierreguillou/bert-large-cased-squad-v1.1-portuguese")
model = AutoModelForQuestionAnswering.from_pretrained("pierreguillou/bert-large-cased-squad-v1.1-portuguese")
```


Or just clone the model repo:


```python
git lfs install
git clone https://huggingface.co/pierreguillou/bert-large-cased-squad-v1.1-portuguese

# if you want to clone without large files – just their pointers
# prepend your git clone with the following env var:

GIT_LFS_SKIP_SMUDGE=1
```


## Limitations and bias


The training data used for this model come from Portuguese SQUAD. It could contain a lot of unfiltered content, which is far from neutral, and biases.


## Author


Portuguese BERT large cased QA (Question Answering), finetuned on SQUAD v1.1 was trained and evaluated by [Pierre GUILLOU](https://www.linkedin.com/in/pierreguillou/) thanks to the Open Source code, platforms and advices of many organizations ([link to the list](https://medium.com/@pierre_guillou/nlp-como-treinar-um-modelo-de-question-answering-em-qualquer-linguagem-baseado-no-bert-large-1c899262dd96#c2f5)). In particular: [Hugging Face](https://huggingface.co/), [Neuralmind.ai](https://neuralmind.ai/), [Deep Learning Brasil group](http://www.deeplearningbrasil.com.br/) and [AI Lab](https://ailab.unb.br/).


## Citation


If you use our work, please cite:


```bibtex
@inproceedings{pierreguillou2021bertlargecasedsquadv11portuguese,
  title={Portuguese BERT large cased QA (Question Answering), finetuned on SQUAD v1.1},
  author={Pierre Guillou},
  year={2021}
}
```


model name: bert-large-cased-squad-v1.1-portuguese


model author: pierreguillou


task: question-answering


tags: pytorch, tf, bert, question-answering, pt, dataset:brWaC, dataset:squad, dataset:squad_v1_pt, transformers, license:mit, bert-large, autonlp_compatible, infinity_compatible


language:


- pt
tags:
- generated_from_trainer
datasets:
- pierreguillou/lener_br_finetuning_language_model
model-index:
- name: checkpoints
results:
  - task:
name: Fill Mask
type: fill-mask
dataset:
name: pierreguillou/lener_br_finetuning_language_model
type: pierreguillou/lener_br_finetuning_language_model
metrics:
    - name: Loss
type: loss
value: 1.352389
widget:
- text: "Com efeito, se tal fosse possível, o Poder [MASK] – que não dispõe de função legislativa – passaria a desempenhar atribuição que lhe é institucionalmente estranha (a de legislador positivo), usurpando, desse modo, no contexto de um sistema de poderes essencialmente limitados, competência que não lhe pertence, com evidente transgressão ao princípio constitucional da separação de poderes."


## (BERT base) Language modeling in the legal domain in Portuguese (LeNER-Br)


**bert-base-cased-pt-lenerbr** is a Language Model in the legal domain in Portuguese that was finetuned on 20/12/2021 in Google Colab from the model [BERTimbau base](https://huggingface.co/neuralmind/bert-base-portuguese-cased) on the dataset [LeNER-Br language modeling](https://huggingface.co/datasets/pierreguillou/lener_br_finetuning_language_model) by using a MASK objective.


You can check as well the [version large of this model](https://huggingface.co/pierreguillou/bert-large-cased-pt-lenerbr).


## Blog post


This language model is used to get a NER model on the Portuguese judicial domain. You can check the fine-tuned NER model at [pierreguillou/ner-bert-base-cased-pt-lenerbr](https://huggingface.co/pierreguillou/ner-bert-base-cased-pt-lenerbr).


All informations and links are in this blog post: [NLP | Modelos e Web App para Reconhecimento de Entidade Nomeada (NER) no domínio jurídico brasileiro](https://medium.com/@pierre_guillou/nlp-modelos-e-web-app-para-reconhecimento-de-entidade-nomeada-ner-no-dom%C3%ADnio-jur%C3%ADdico-b658db55edfb) (29/12/2021)


## Widget & APP


You can test this model into the widget of this page.


## Using the model for inference in production


```
# install pytorch: check https://pytorch.org/
# !pip install transformers
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("pierreguillou/bert-base-cased-pt-lenerbr")
model = AutoModelForMaskedLM.from_pretrained("pierreguillou/bert-base-cased-pt-lenerbr")
```


## Training procedure


## Notebook


The notebook of finetuning ([Finetuning_language_model_BERtimbau_LeNER_Br.ipynb](https://github.com/piegu/language-models/blob/master/Finetuning_language_model_BERtimbau_LeNER_Br.ipynb)) is in github.


### Training results


```
Num examples = 3227
Num Epochs = 5
Instantaneous batch size per device = 8
Total train batch size (w. parallel, distributed & accumulation) = 8
Gradient Accumulation steps = 1
Total optimization steps = 2020

Step	Training Loss	Validation Loss
100	 1.988700	     1.616412
200	 1.724900	     1.561100
300	 1.713400	     1.499991
400	 1.687400	     1.451414
500	 1.579700	     1.433665
600	 1.556900	     1.407338
700	 1.591400	     1.421942
800	 1.546000	     1.406395
900	 1.510100	     1.352389
1000	1.507100     	1.394799
1100	1.462200     	1.36809373471
```


model name: bert-base-cased-pt-lenerbr


model author: pierreguillou


task: fill-mask


tags: pytorch, bert, fill-mask, pt, dataset:pierreguillou/lener_br_finetuning_language_model, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


language:


- pt
tags:
- text2text-generation
- t5
- pytorch
- qa
datasets:
- squad
- squad_v1_pt
metrics:
- precision
- recall
- f1
- accuracy
- squad
model-index:
- name: checkpoints
results:
  - task:
name: text2text-generation
type: text2text-generation
dataset:
name: squad
type: squad
metrics:
    - name: f1
type: f1
value: 79.3
    - name: exact-match
type: exact-match
value: 67.3983
widget:
- text: "question: Quando começou a pandemia de Covid-19 no mundo? context: A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano."
- text: "question: Onde foi descoberta a Covid-19? context: A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano."


# T5 base finetuned for Question Answering (QA) on SQUaD v1.1 Portuguese


![Exemple of what can do with a T5 model (for example: Question Answering finetuned on SQUAD v1.1 in Portuguese)](https://miro.medium.com/max/2000/1*zp9niaQzWNo8Pipd8zvL1w.png)


## Introduction


**t5-base-qa-squad-v1.1-portuguese** is a QA model (Question Answering) in Portuguese that was finetuned on 27/01/2022 in Google Colab from the model [unicamp-dl/ptt5-base-portuguese-vocab](https://huggingface.co/unicamp-dl/ptt5-base-portuguese-vocab) of Neuralmind on the dataset SQUAD v1.1 in portuguese from the [Deep Learning Brasil group](http://www.deeplearningbrasil.com.br/) by using a Test2Text-Generation objective.


Due to the small size of T5 base and finetuning dataset, the model overfitted before to reach the end of training. Here are the overall final metrics on the validation dataset:


- **f1**: 79.3
- **exact_match**: 67.3983


Check our other QA models in Portuguese finetuned on SQUAD v1.1:


- [Portuguese BERT base cased QA](https://huggingface.co/pierreguillou/bert-base-cased-squad-v1.1-portuguese)
- [Portuguese BERT large cased QA](https://huggingface.co/pierreguillou/bert-large-cased-squad-v1.1-portuguese)
- [Portuguese ByT5 small QA](https://huggingface.co/pierreguillou/byt5-small-qa-squad-v1.1-portuguese)


## Blog post


[NLP nas empresas | Como eu treinei um modelo T5 em português na tarefa QA no Google Colab](https://medium.com/@pierre_guillou/nlp-nas-empresas-como-eu-treinei-um-modelo-t5-em-portugu%C3%AAs-na-tarefa-qa-no-google-colab-e8eb0dc38894) (27/01/2022)


## Widget & App


Use as well the [QA App | T5 base pt](https://huggingface.co/spaces/pierreguillou/question-answering-portuguese-t5-base) that allows using the model T5 base finetuned on the QA task with the SQuAD v1.1 pt dataset.


```
# install pytorch: check https://pytorch.org/
# !pip install transformers
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# model & tokenizer
model_name = "pierreguillou/t5-base-qa-squad-v1.1-portuguese"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# parameters
max_target_length=32
num_beams=1
early_stopping=True

input_text  = 'question: Quando foi descoberta a Covid-19? context: A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano.'
label = '1 de dezembro de 2019'

inputs = tokenizer(input_text, return_tensors="pt")

outputs = model.generate(inputs["input_ids"],
                             max_length=max_target_length,
                             num_beams=num_beams,
                             early_stopping=early_stopping
                            )
pred = tokenizer.decode(outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)

print('true answer |', label)
print('pred        |', pred)
```


You can use pipeline, too. However, it seems to have an issue regarding to the max_length of the input sequence.


```
!pip install transformers
import transformers
from transformers import pipeline

# model
model_name = "pierreguillou/t5-base-qa-squad-v1.1-portuguese"

# parameters
max_target_length=32
num_beams=1
early_stopping=True
clean_up_tokenization_spaces=True

input_text  = 'question: Quando foi descoberta a Covid-19? context: A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano.'
label = '1 de dezembro de 2019'

text2text = pipeline(
                    "text2text-generation",
                     model=model_name,
                     max_length=max_target_length,
                     num_beams=num_beams,
                     early_stopping=early_stopping,
                     clean_up_tokenization_spaces=clean_up_tokenization_spaces
                    )

pred = text2text(input_text)

print('true answer |', label)
print('pred        |', pred)
```


## Training procedure


### Notebook


The notebook of finetuning ([HuggingFace_Notebook_t5-base-portuguese-vocab_question_answering_QA_squad_v11_pt.ipynb](https://github.com/piegu/language-models/blob/master/HuggingFace_Notebook_t5_base_portuguese_vocab_question_answering_QA_squad_v11_pt.ipynb)) is in github.


### Hyperparameters


```
# do training and evaluation
do_train = True
do_eval= True

# batch
batch_size = 4
gradient_accumulation_steps = 3
per_device_train_batch_size = batch_size
per_device_eval_batch_size = per_device_train_batch_size*16

# LR, wd, epochs
learning_rate = 1e-4
weight_decay = 0.01
num_train_epochs = 10
fp16 = True

# logs
logging_strategy = "steps"
logging_first_step = True
logging_steps = 3000     # if logging_strategy = "steps"
eval_steps = logging_steps

# checkpoints
evaluation_strategy = logging_strategy
save_strategy = logging_strategy
save_steps = logging_steps
save_total_limit = 3

# best model
load_best_model_at_end = True
metric_for_best_model = "f1" #"loss"
if metric_for_best_model == "loss":
  greater_is_better = False
else:
  greater_is_better = True

# evaluation
num_beams = 1
```


### Training results


```
Num examples = 87510
Num Epochs = 10
Instantaneous batch size per device = 4
Total train batch size (w. parallel, distributed & accumulation) = 12
Gradient Accumulation steps = 3
Total optimization steps = 72920

Step 	Training Loss  Exact Match	F1
3000 	0.776100	     61.807001  	75.114517
6000 	0.545900	     65.260170  	77.468930
9000 	0.460500	     66.556291  	78.491938
12000	0.393400	     66.821192  	78.745397
15000	0.379800  	   66.603595  	78.815515
18000	0.298100  	   67.578051  	79.287899
21000	0.303100  	   66.991485  	78.979669
24000	0.251600  	   67.275307  	78.929923

27000	0.237500	     66.972564  	79.333612

30000	0.220500 	    66.915799  	79.236574
33000	0.182600	     67.029328  	78.964212
36000	0.190600 	    66.982025  	79.086125

```


model name: t5-base-qa-squad-v1.1-portuguese


model author: pierreguillou


task: text2text-generation


tags: pytorch, t5, text2text-generation, pt, dataset:squad, dataset:squad_v1_pt, transformers, qa, model-index, autonlp_compatible


language: pt
license: apache-2.0
tags:


- text2text-generation
- byt5
- pytorch
- qa
datasets: squad
metrics: squad
widget:
- text: 'question: "Quando começou a pandemia de Covid-19 no mundo?" context: "A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano."'
- text: 'question: "Onde foi descoberta a Covid-19?" context: "A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano."'


# ByT5 small finetuned for Question Answering (QA) on SQUaD v1.1 Portuguese


![Exemple of what can do the Portuguese ByT5 small QA (Question Answering), finetuned on SQUAD v1.1](https://miro.medium.com/max/2000/1*te5MmdesAHCmg4KmK8zD3g.png)


- [Portuguese BERT base cased QA](https://huggingface.co/pierreguillou/bert-base-cased-squad-v1.1-portuguese)
- [Portuguese BERT large cased QA](https://huggingface.co/pierreguillou/bert-large-cased-squad-v1.1-portuguese)
- [Portuguese T5 base QA](https://huggingface.co/pierreguillou/t5-base-qa-squad-v1.1-portuguese)


## Introduction


The model was trained on the dataset SQUAD v1.1 in portuguese from the [Deep Learning Brasil group](http://www.deeplearningbrasil.com.br/) on Google Colab from the language model [ByT5 small](https://huggingface.co/google/byt5-small) of Google.


## About ByT5


ByT5 is a tokenizer-free version of [Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) and generally follows the architecture of [MT5](https://huggingface.co/google/mt5-small). ByT5 was only pre-trained on [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual) excluding any supervised training with an average span-mask of 20 UTF-8 characters. Therefore, this model has to be fine-tuned before it is useable on a downstream task.


ByT5 works especially well on noisy text data,*e.g.*, `google/byt5-small` significantly outperforms [mt5-small](https://huggingface.co/google/mt5-small) on [TweetQA](https://arxiv.org/abs/1907.06292).


Paper: [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626)


## Informations on the method used


All the informations are in the blog post : ...


## Notebooks in Google Colab & GitHub


- Google Colab: ...
- GitHub: ...


## Performance


The results obtained are the following:


```
f1 = ...
exact match = ...
```


## How to use the model... with Pipeline


```python
import transformers
from transformers import pipeline

model_name = 'pierreguillou/byt5-small-qa-squad-v1.1-portuguese'
nlp = pipeline("text2text-generation", model=model_name)

# source: https://pt.wikipedia.org/wiki/Pandemia_de_COVID-19
input_text = r"""
question: "Quando começou a pandemia de Covid-19 no mundo?"
context: "A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano."
"""
input_text = input_text.replace('\n','')
input_text

# question: "Quando começou a pandemia de Covid-19 no mundo?" context: "A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano."

result = nlp(input_text)
result

# [{'generated_text': '1 de dezembro de 2019'}]
```


## How to use the model... with the Auto classes


```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_name = 'pierreguillou/byt5-small-qa-squad-v1.1-portuguese'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# source: https://pt.wikipedia.org/wiki/Pandemia_de_COVID-19
input_text = r"""
question: "Quando começou a pandemia de Covid-19 no mundo?"
context: "A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano."
"""
input_text = input_text.replace('\n','')
input_text

# question: "Quando começou a pandemia de Covid-19 no mundo?" context: "A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano."

input_ids = tokenizer(input_text, return_tensors='pt').input_ids
outputs = model.generate(
    input_ids,
    max_length=64,
    num_beams=1
    )

result = tokenizer.decode(outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)
result

# 1 de dezembro de 2019
```


## Limitations and bias


The training data used for this model come from Portuguese SQUAD. It could contain a lot of unfiltered content, which is far from neutral, and biases.


## Author


Portuguese ByT5 small QA (Question Answering), finetuned on SQUAD v1.1 was trained and evaluated by [Pierre GUILLOU](https://www.linkedin.com/in/pierreguillou/) thanks to the Open Source code, platforms and advices of many organizations. In particular: [Google AI](https://huggingface.co/google), [Hugging Face](https://huggingface.co/), [Deep Learning Brasil group](http://www.deeplearningbrasil.com.br/) and [Google Colab](https://colab.research.google.com/).


## Citation


If you use our work, please cite:


```bibtex
@inproceedings{pierreguillou2021byt5smallsquadv11portuguese,
  title={Portuguese ByT5 small QA (Question Answering), finetuned on SQUAD v1.1},
  author={Pierre Guillou},
  year={2021}
}
```


model name: byt5-small-qa-squad-v1.1-portuguese


model author: pierreguillou


task: text2text-generation


tags: pytorch, t5, text2text-generation, pt, dataset:squad, arxiv:1907.06292, arxiv:2105.13626, transformers, license:apache-2.0, byt5, qa, autonlp_compatible


language:


- pt
tags:
- generated_from_trainer
datasets:
- lener_br
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: checkpoints
results:
  - task:
name: Token Classification
type: token-classification
dataset:
name: lener_br
type: lener_br
metrics:
    - name: F1
type: f1
value: 0.8926146010186757
    - name: Precision
type: precision
value: 0.8810222036028488
    - name: Recall
type: recall
value: 0.9045161290322581
    - name: Accuracy
type: accuracy
value: 0.9759397808828684
    - name: Loss
type: loss
value: 0.18803243339061737
widget:
- text: "Ao Instituto Médico Legal da jurisdição do acidente ou da residência cumpre fornecer, no prazo de 90 dias, laudo à vítima (art. 5, § 5, Lei n. 6.194/74  de 19 de dezembro de 1974), função técnica que pode ser suprida por prova pericial realizada por ordem do juízo da causa, ou por prova técnica realizada no âmbito administrativo que se mostre coerente com os demais elementos de prova constante dos autos."
- text: "Acrescento que não há de se falar em violação do artigo 114, § 3º, da Constituição Federal, posto que referido dispositivo revela-se impertinente, tratando da possibilidade de ajuizamento de dissídio coletivo pelo Ministério Público do Trabalho nos casos de greve em atividade essencial."
- text: "Dispõe sobre o estágio de estudantes; altera a redação do art. 428 da Consolidação das Leis do Trabalho – CLT, aprovada pelo Decreto-Lei no 5.452, de 1o de maio de 1943, e a Lei no 9.394, de 20 de dezembro de 1996; revoga as Leis nos 6.494, de 7 de dezembro de 1977, e 8.859, de 23 de março de 1994, o parágrafo único do art. 82 da Lei no 9.394, de 20 de dezembro de 1996, e o art. 6o da Medida Provisória  no 2.164-41, de 24 de agosto de 2001; e dá outras providências."


## (BERT base) NER model in the legal domain in Portuguese (LeNER-Br)


**ner-bert-base-portuguese-cased-lenerbr** is a NER model (token classification) in the legal domain in Portuguese that was finetuned on 20/12/2021 in Google Colab from the model [pierreguillou/bert-base-cased-pt-lenerbr](https://huggingface.co/pierreguillou/bert-base-cased-pt-lenerbr) on the dataset [LeNER_br](https://huggingface.co/datasets/lener_br) by using a NER objective.


Due to the small size of BERTimbau base and finetuning dataset, the model overfitted before to reach the end of training. Here are the overall final metrics on the validation dataset (*note: see the paragraph "Validation metrics by Named Entity" to get detailed metrics*):


- **f1**: 0.8926146010186757
- **precision**: 0.8810222036028488
- **recall**: 0.9045161290322581
- **accuracy**: 0.9759397808828684
- **loss**: 0.18803243339061737


Check as well the [large version of this model](https://huggingface.co/pierreguillou/ner-bert-large-cased-pt-lenerbr) with a f1 of 0.908.


**Note**: the model [pierreguillou/bert-base-cased-pt-lenerbr](https://huggingface.co/pierreguillou/bert-base-cased-pt-lenerbr) is a language model that was created through the finetuning of the model [BERTimbau base](https://huggingface.co/neuralmind/bert-base-portuguese-cased) on the dataset [LeNER-Br language modeling](https://huggingface.co/datasets/pierreguillou/lener_br_finetuning_language_model) by using a MASK objective. This first specialization of the language model before finetuning on the NER task improved a bit the model quality. To prove it, here are the results of the NER model finetuned from the model [BERTimbau base](https://huggingface.co/neuralmind/bert-base-portuguese-cased) (a non-specialized language model):


- **f1**: 0.8716487228203504
- **precision**: 0.8559286898839138
- **recall**: 0.8879569892473118
- **accuracy**: 0.9755893153732458
- **loss**: 0.1133928969502449


## Blog post


[NLP | Modelos e Web App para Reconhecimento de Entidade Nomeada (NER) no domínio jurídico brasileiro](https://medium.com/@pierre_guillou/nlp-modelos-e-web-app-para-reconhecimento-de-entidade-nomeada-ner-no-dom%C3%ADnio-jur%C3%ADdico-b658db55edfb) (29/12/2021)


## Widget & App


You can test this model into the widget of this page.


Use as well the [NER App](https://huggingface.co/spaces/pierreguillou/ner-bert-pt-lenerbr) that allows comparing the 2 BERT models (base and large) fitted in the NER task with the legal LeNER-Br dataset.


## Using the model for inference in production


```
# install pytorch: check https://pytorch.org/
# !pip install transformers
from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch

# parameters
model_name = "pierreguillou/ner-bert-base-cased-pt-lenerbr"
model = AutoModelForTokenClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

input_text = "Acrescento que não há de se falar em violação do artigo 114, § 3º, da Constituição Federal, posto que referido dispositivo revela-se impertinente, tratando da possibilidade de ajuizamento de dissídio coletivo pelo Ministério Público do Trabalho nos casos de greve em atividade essencial."

# tokenization
inputs = tokenizer(input_text, max_length=512, truncation=True, return_tensors="pt")
tokens = inputs.tokens()

# get predictions
outputs = model(**inputs).logits
predictions = torch.argmax(outputs, dim=2)

# print predictions
for token, prediction in zip(tokens, predictions[0].numpy()):
    print((token, model.config.id2label[prediction]))
```


You can use pipeline, too. However, it seems to have an issue regarding to the max_length of the input sequence.


```
!pip install transformers
import transformers
from transformers import pipeline

model_name = "pierreguillou/ner-bert-base-cased-pt-lenerbr"

ner = pipeline(
    "ner",
    model=model_name
)

ner(input_text)
```


## Training procedure


### Notebook


The notebook of finetuning ([HuggingFace_Notebook_token_classification_NER_LeNER_Br.ipynb](https://github.com/piegu/language-models/blob/master/HuggingFace_Notebook_token_classification_NER_LeNER_Br.ipynb)) is in github.


### Hyperparameters


#### batch, learning rate...


- per_device_batch_size = 2
- gradient_accumulation_steps = 2
- learning_rate = 2e-5
- num_train_epochs = 10
- weight_decay = 0.01
- optimizer = AdamW
- betas = (0.9,0.999)
- epsilon = 1e-08
- lr_scheduler_type = linear
- seed = 7


#### save model & load best model


- save_total_limit = 2
- logging_steps = 300
- eval_steps = logging_steps
- evaluation_strategy = 'steps'
- logging_strategy = 'steps'
- save_strategy = 'steps'
- save_steps = logging_steps
- load_best_model_at_end = True
- fp16 = True


#### get best model through a metric


- metric_for_best_model = 'eval_f1'
- greater_is_better = True


### Training results


```
Num examples = 7828
Num Epochs = 10
Instantaneous batch size per device = 2
Total train batch size (w. parallel, distributed & accumulation) = 4
Gradient Accumulation steps = 2
Total optimization steps = 19570

Step	Training Loss Validation Loss      Precision     Recall  	 F1      	Accuracy
300	  0.127600	  0.178613	        0.722909	  0.741720	 0.732194	0.948802
600	  0.088200	  0.136965	        0.733636	  0.867742	 0.795074	0.963079
900	  0.078000	  0.128858	        0.791912	  0.838065	 0.814335	0.965243
1200 	0.077800      0.126345	        0.815400	  0.865376	 0.839645	0.967849
1500 	0.074100      0.148207	        0.779274	  0.895914	 0.833533	0.960184
1800 	0.059500      0.116634	        0.830829	  0.868172	 0.849090	0.969342
2100 	0.044500      0.208459	        0.887150	  0.816559	 0.850392	0.960535
2400 	0.029400      0.136352	        0.867821	  0.851398	 0.859531	0.970271
2700 	0.025000      0.165837	        0.814881	  0.878495	 0.845493	0.961235
3000 	0.038400      0.120629	        0.811719	  0.893763	 0.850768	0.971506
3300 	0.026200      0.175094	        0.823435	  0.882581	 0.851983	0.962957
3600 	0.025600      0.178438	        0.881095	  0.886022	 0.883551	0.963689
3900 	0.041000      0.134648	        0.789035	  0.916129	 0.847846	0.967681
4200 	0.026700      0.130178	        0.821275	  0.903226	 0.860303	0.972313
4500 	0.018500      0.139294	        0.844016	  0.875054	 0.859255	0.971140
4800 	0.020800      0.197811	        0.892504	  0.873118	 0.882705	0.965883
5100 	0.019300      0.161239	        0.848746	  0.888172	 0.868012	0.967849
5400 	0.024000      0.139131	        0.837507	  0.913333	 0.873778	0.970591
5700 	0.018400      0.157223	        0.899754	  0.864731	 0.881895	0.970210
6000 	0.023500      0.137022	        0.883018	  0.873333	 0.878149	0.973243
6300 	0.009300      0.181448	        0.840490	  0.900860	 0.869628	0.968290
6600 	0.019200      0.173125	        0.821316	  0.896559	 0.857290	0.966736
6900 	0.016100      0.143160	        0.789938	  0.904946	 0.843540	0.968245
7200 	0.017000      0.145755	        0.823274	  0.897634	 0.858848	0.969037
7500 	0.012100      0.159342	        0.825694	  0.883226	 0.853491	0.967468
7800 	0.013800      0.194886            0.861237	  0.859570	 0.860403	0.964771
8100 	0.008000      0.140271	        0.829914	  0.896129	 0.861752	0.971567
8400 	0.010300      0.143318	        0.826844	  0.908817	 0.865895	0.973578
8700 	0.015000      0.143392	        0.847336	  0.889247	 0.867786	0.973365
9000 	0.006000      0.143512	        0.847795	  0.905591	 0.875741	0.972892
9300 	0.011800      0.138747	        0.827133	  0.894194	 0.859357	0.971673
9600 	0.008500      0.159490	        0.837030	  0.909032	 0.871546	0.970028
9900 	0.010700      0.159249	        0.846692	  0.910968	 0.877655	0.970546
10200	0.008100	  0.170069  	      0.848288	  0.900645	 0.873683	0.969113
10500	0.004800	  0.183795	        0.860317	  0.899355	 0.879403	0.969570
10800	0.010700	  0.157024	        0.837838	  0.906667	 0.870894	0.971094
11100	0.003800	  0.164286	        0.845312	  0.880215	 0.862410	0.970744
11400	0.009700	  0.204025	        0.884294	  0.887527	 0.885907	0.968854
11700	0.008900	  0.162819	        0.829415	  0.887742	 0.857588	0.970530
12000	0.006400	  0.164296	        0.852666	  0.901075	 0.876202	0.971414
12300	0.007100	  0.143367	        0.852959	  0.895699	 0.873807	0.973669
12600	0.015800	  0.153383	        0.859224	  0.900430	 0.879345	0.972679
12900	0.006600	  0.173447	        0.869954	  0.899140	 0.884306	0.970927
13200	0.006800	  0.163234  	      0.856849	  0.897204	 0.876563	0.971795
13500	0.003200	  0.167164	        0.850867	  0.907957	 0.878485	0.971231
13800	0.003600	  0.148950  	      0.867801	  0.910538	 0.888656	0.976961
14100	0.003500	  0.155691  	      0.847621	  0.907957	 0.876752	0.974127
14400	0.003300	  0.157672	        0.846553	  0.911183	 0.877680	0.974584
14700	0.002500	  0.169965	        0.847804	  0.917634	 0.881338	0.973045
15000	0.003400	  0.177099  	      0.842199	  0.912473	 0.875929	0.971155
15300	0.006000	  0.164151  	      0.848928	  0.911183	 0.878954	0.973258
15600	0.002400	  0.174305	        0.847437	  0.906667	 0.876052	0.971765
15900	0.004100	  0.174561  	      0.852929	  0.907957	 0.879583	0.972907
16200	0.002600	  0.172626	        0.843263	  0.907097	 0.874016	0.972100
16500	0.002100	  0.185302	        0.841108	  0.907312	 0.872957	0.970485
16800	0.002900	  0.175638	        0.840557	  0.909247	 0.873554	0.971704
17100	0.001600	  0.178750	        0.857056	  0.906452	 0.881062	0.971765
17400	0.003900	  0.188910	        0.853619	  0.907957	 0.879950	0.970835
17700	0.002700	  0.180822	        0.864699	  0.907097	 0.885390	0.972283
18000	0.001300	  0.179974	        0.868150	  0.906237	 0.886785	0.973060

18300	0.000800	  0.188032	        0.881022	  0.904516	 0.892615	0.972572

18600	0.002700	  0.183266	        0.868601	  0.901290	 0.884644	0.972298
18900	0.001600	  0.180301	        0.862041	  0.903011	 0.882050	0.972344
19200	0.002300	  0.183432	        0.855370	  0.904301	 0.879155	0.971109
19500	0.001800	  0.183381	        0.854501	  0.904301	 0.878696	0.971186
```


### Validation metrics by Named Entity


```
Num examples = 1177

{'JURISPRUDENCIA': {'f1': 0.7016574585635359,
  'number': 657,
  'precision': 0.6422250316055625,
  'recall': 0.7732115677321156},
 'LEGISLACAO': {'f1': 0.8839681133746677,
  'number': 571,
  'precision': 0.8942652329749103,
  'recall': 0.8739054290718039},
 'LOCAL': {'f1': 0.8253968253968254,
  'number': 194,
  'precision': 0.7368421052631579,
  'recall': 0.9381443298969072},
 'ORGANIZACAO': {'f1': 0.8934049079754601,
  'number': 1340,
  'precision': 0.918769716088328,
  'recall': 0.8694029850746269},
 'PESSOA': {'f1': 0.982653539615565,
  'number': 1072,
  'precision': 0.9877474081055608,
  'recall': 0.9776119402985075},
 'TEMPO': {'f1': 0.9657657657657657,
  'number': 816,
  'precision': 0.9469964664310954,
  'recall': 0.9852941176470589},
 'overall_accuracy': 0.9725722644643211,
 'overall_f1': 0.8926146010186757,
 'overall_precision': 0.8810222036028488,
 'overall_recall': 0.9045161290322581}
```


model name: ner-bert-base-cased-pt-lenerbr


model author: pierreguillou


task: token-classification


tags: pytorch, bert, token-classification, pt, dataset:lener_br, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


language:


- pt
tags:
- generated_from_trainer
datasets:
- pierreguillou/lener_br_finetuning_language_model
model-index:
- name: checkpoints
results:
  - task:
name: Fill Mask
type: fill-mask
dataset:
name: pierreguillou/lener_br_finetuning_language_model
type: pierreguillou/lener_br_finetuning_language_model
metrics:
    - name: Loss
type: loss
value: 1.127950
widget:
- text: "Com efeito, se tal fosse possível, o Poder [MASK] – que não dispõe de função legislativa – passaria a desempenhar atribuição que lhe é institucionalmente estranha (a de legislador positivo), usurpando, desse modo, no contexto de um sistema de poderes essencialmente limitados, competência que não lhe pertence, com evidente transgressão ao princípio constitucional da separação de poderes."


## (BERT large) Language modeling in the legal domain in Portuguese (LeNER-Br)


**bert-large-cased-pt-lenerbr** is a Language Model in the legal domain in Portuguese that was finetuned on 20/12/2021 in Google Colab from the model [BERTimbau large](https://huggingface.co/neuralmind/bert-large-portuguese-cased) on the dataset [LeNER-Br language modeling](https://huggingface.co/datasets/pierreguillou/lener_br_finetuning_language_model) by using a MASK objective.


You can check as well the [version base of this model](https://huggingface.co/pierreguillou/bert-base-cased-pt-lenerbr).


## Widget & APP


You can test this model into the widget of this page.


## Blog post


This language model is used to get a NER model on the Portuguese judicial domain. You can check the fine-tuned NER model at [pierreguillou/ner-bert-large-cased-pt-lenerbr](https://huggingface.co/pierreguillou/ner-bert-large-cased-pt-lenerbr).


All informations and links are in this blog post: [NLP | Modelos e Web App para Reconhecimento de Entidade Nomeada (NER) no domínio jurídico brasileiro](https://medium.com/@pierre_guillou/nlp-modelos-e-web-app-para-reconhecimento-de-entidade-nomeada-ner-no-dom%C3%ADnio-jur%C3%ADdico-b658db55edfb) (29/12/2021)


## Using the model for inference in production


```
# install pytorch: check https://pytorch.org/
# !pip install transformers
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("pierreguillou/bert-large-cased-pt-lenerbr")
model = AutoModelForMaskedLM.from_pretrained("pierreguillou/bert-large-cased-pt-lenerbr")
```


## Training procedure


## Notebook


The notebook of finetuning ([Finetuning_language_model_BERtimbau_LeNER_Br.ipynb](https://github.com/piegu/language-models/blob/master/Finetuning_language_model_BERtimbau_LeNER_Br.ipynb)) is in github.


### Training results


```
Num examples = 3227
Num Epochs = 5
Instantaneous batch size per device = 2
Total train batch size (w. parallel, distributed & accumulation) = 8
Gradient Accumulation steps = 4
Total optimization steps = 2015

Step	Training Loss	Validation Loss
100   1.616700      1.366015
200   1.452000      1.312473
300   1.431100      1.253055
400   1.407500      1.264705
500   1.301900      1.243277
600   1.317800      1.233684
700   1.319100      1.211826
800   1.303800      1.190818
900   1.262800      1.171898
1000  1.235900      1.146275
1100  1.221900      1.149027
1200  1.226200      1.127950
1300  1.201700      1.172729
1400  1.198200      1.145363
```


model name: bert-large-cased-pt-lenerbr


model author: pierreguillou


task: fill-mask


tags: pytorch, bert, fill-mask, pt, dataset:pierreguillou/lener_br_finetuning_language_model, transformers, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


language: pt
license: mit
tags:


- question-answering
- bert
- bert-base
- pytorch
datasets:
- brWaC
- squad
- squad_v1_pt
metrics:
- squad
widget:
- text: "Quando começou a pandemia de Covid-19 no mundo?"
context: "A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano."
- text: "Onde foi descoberta a Covid-19?"
context: "A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano."


# Portuguese BERT base cased QA (Question Answering), finetuned on SQUAD v1.1


![Exemple of what can do the Portuguese BERT base cased QA (Question Answering), finetuned on SQUAD v1.1](https://miro.medium.com/max/2000/1*te5MmdesAHCmg4KmK8zD3g.png)


## Introduction


The model was trained on the dataset SQUAD v1.1 in portuguese from the [Deep Learning Brasil group](http://www.deeplearningbrasil.com.br/) on Google Colab.


The language model used is the [BERTimbau Base](https://huggingface.co/neuralmind/bert-base-portuguese-cased) (aka "bert-base-portuguese-cased") from [Neuralmind.ai](https://neuralmind.ai/): BERTimbau Base is a pretrained BERT model for Brazilian Portuguese that achieves state-of-the-art performances on three downstream NLP tasks: Named Entity Recognition, Sentence Textual Similarity and Recognizing Textual Entailment. It is available in two sizes: Base and Large.


## Informations on the method used


All the informations are in the blog post : [NLP | Modelo de Question Answering em qualquer idioma baseado no BERT base (estudo de caso em português)](https://medium.com/@pierre_guillou/nlp-modelo-de-question-answering-em-qualquer-idioma-baseado-no-bert-base-estudo-de-caso-em-12093d385e78)


## Notebooks in Google Colab & GitHub


- Google Colab: [colab_question_answering_BERT_base_cased_squad_v11_pt.ipynb](https://colab.research.google.com/drive/18ueLdi_V321Gz37x4gHq8mb4XZSGWfZx?usp=sharing)
- GitHub: [colab_question_answering_BERT_base_cased_squad_v11_pt.ipynb](https://github.com/piegu/language-models/blob/master/colab_question_answering_BERT_base_cased_squad_v11_pt.ipynb)


## Performance


The results obtained are the following:


```
f1 = 82.50
exact match = 70.49
```


## How to use the model... with Pipeline


```python
import transformers
from transformers import pipeline

# source: https://pt.wikipedia.org/wiki/Pandemia_de_COVID-19
context = r"""
A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19,
uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2).
A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China,
em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano.
Acredita-se que o vírus tenha uma origem zoonótica, porque os primeiros casos confirmados
tinham principalmente ligações ao Mercado Atacadista de Frutos do Mar de Huanan, que também vendia animais vivos.
Em 11 de março de 2020, a Organização Mundial da Saúde declarou o surto uma pandemia. Até 8 de fevereiro de 2021,
pelo menos 105 743 102 casos da doença foram confirmados em pelo menos 191 países e territórios,
com cerca de 2 308 943 mortes e 58 851 440 pessoas curadas.
"""

model_name = 'pierreguillou/bert-base-cased-squad-v1.1-portuguese'
nlp = pipeline("question-answering", model=model_name)

question = "Quando começou a pandemia de Covid-19 no mundo?"

result = nlp(question=question, context=context)

print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")

# Answer: '1 de dezembro de 2019', score: 0.713, start: 328, end: 349
```


## How to use the model... with the Auto classes


```python
from transformers import AutoTokenizer, AutoModelForQuestionAnswering

tokenizer = AutoTokenizer.from_pretrained("pierreguillou/bert-base-cased-squad-v1.1-portuguese")
model = AutoModelForQuestionAnswering.from_pretrained("pierreguillou/bert-base-cased-squad-v1.1-portuguese")
```


Or just clone the model repo:


```python
git lfs install
git clone https://huggingface.co/pierreguillou/bert-base-cased-squad-v1.1-portuguese

# if you want to clone without large files – just their pointers
# prepend your git clone with the following env var:

GIT_LFS_SKIP_SMUDGE=1
```


## Limitations and bias


The training data used for this model come from Portuguese SQUAD. It could contain a lot of unfiltered content, which is far from neutral, and biases.


## Author


Portuguese BERT base cased QA (Question Answering), finetuned on SQUAD v1.1 was trained and evaluated by [Pierre GUILLOU](https://www.linkedin.com/in/pierreguillou/) thanks to the Open Source code, platforms and advices of many organizations ([link to the list](https://medium.com/@pierre_guillou/nlp-modelo-de-question-answering-em-qualquer-idioma-baseado-no-bert-base-estudo-de-caso-em-12093d385e78#c572)). In particular: [Hugging Face](https://huggingface.co/), [Neuralmind.ai](https://neuralmind.ai/), [Deep Learning Brasil group](http://www.deeplearningbrasil.com.br/), [Google Colab](https://colab.research.google.com/) and [AI Lab](https://ailab.unb.br/).


## Citation


If you use our work, please cite:


```bibtex
@inproceedings{pierreguillou2021bertbasecasedsquadv11portuguese,
  title={Portuguese BERT base cased QA (Question Answering), finetuned on SQUAD v1.1},
  author={Pierre Guillou},
  year={2021}
}
```


model name: bert-base-cased-squad-v1.1-portuguese


model author: pierreguillou


task: question-answering


tags: pytorch, tf, jax, bert, question-answering, pt, dataset:brWaC, dataset:squad, dataset:squad_v1_pt, transformers, license:mit, bert-base, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-base-timit-demo-colab
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-base-timit-demo-colab


This model is a fine-tuned version of [facebook/hubert-large-ls960-ft](https://huggingface.co/facebook/hubert-large-ls960-ft) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0280
- Wer: 0.0082


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.1152        | 1.42  | 500   | 0.0416          | 0.0159 |
| 0.0803        | 2.83  | 1000  | 0.0372          | 0.0144 |
| 0.0672        | 4.25  | 1500  | 0.0345          | 0.0119 |
| 0.0564        | 5.67  | 2000  | 0.0338          | 0.0106 |
| 0.0513        | 7.08  | 2500  | 0.0307          | 0.0100 |
| 0.0448        | 8.5   | 3000  | 0.0343          | 0.0098 |
| 0.0374        | 9.92  | 3500  | 0.0300          | 0.0084 |
| 0.0368        | 11.33 | 4000  | 0.0314          | 0.0086 |
| 0.0388        | 12.75 | 4500  | 0.0283          | 0.0089 |
| 0.0277        | 14.16 | 5000  | 0.0302          | 0.0089 |
| 0.0298        | 15.58 | 5500  | 0.0298          | 0.0089 |
| 0.0271        | 17.0  | 6000  | 0.0320          | 0.0098 |
| 0.024         | 18.41 | 6500  | 0.0286          | 0.0088 |
| 0.0236        | 19.83 | 7000  | 0.0284          | 0.0084 |
| 0.0238        | 21.25 | 7500  | 0.0290          | 0.0086 |
| 0.0227        | 22.66 | 8000  | 0.0284          | 0.0093 |
| 0.0198        | 24.08 | 8500  | 0.0280          | 0.0088 |
| 0.0225        | 25.5  | 9000  | 0.0281          | 0.0086 |
| 0.018         | 26.91 | 9500  | 0.0280          | 0.0082 |
| 0.0178        | 28.33 | 10000 | 0.0280          | 0.0082 |
| 0.0209        | 29.75 | 10500 | 0.0280          | 0.0082 |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.9.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: wav2vec2-base-timit-demo-colab


model author: rkmt


task: automatic-speech-recognition


tags: pytorch, tensorboard, hubert, automatic-speech-recognition, transformers, license:apache-2.0, generated_from_trainer, model-index


language:


- id
datasets:
- allenai/c4


# Indonesian T5 Base


T5 (Text-to-Text Transfer Transformer) model pretrained on Indonesian mC4 with [extra filtering](https://github.com/Wikidepia/indonesian_datasets/tree/master/dump/mc4). This model is pre-trained only and needs to be fine-tuned to be used for specific tasks.


## Pretraining Details


Trained for 1M steps following [`google/t5-v1_1-base`](https://huggingface.co/google/t5-v1_1-base).


## Model Performance


TBD


## Limitations and bias


This model also has the problem of biased (unethical, harmful, biased) output results due to the bias of the content of the training data, which is associated with the language model using a large-scale corpus. There is potential. Assuming that this problem may occur, please be careful to use it only for applications that do not cause damage.


## Acknowledgement


Thanks to Tensorflow Research Cloud for providing TPU v3-8s.


model name: IndoT5-base


model author: Wikidepia


task: text2text-generation


tags: pytorch, t5, text2text-generation, id, dataset:allenai/c4, transformers, autonlp_compatible


## language: id
inference: false


# SQuAD IndoBERT-Lite Base Model


Fine-tuned IndoBERT-Lite from IndoBenchmark using Translated SQuAD datasets.


## How to use


### Using pipeline