tags:


- object-detection
library_name: keras


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- optimizer: {'name': 'Adam', 'learning_rate': 0.001, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-07, 'amsgrad': False}
- training_precision: float32


## Training Metrics


| Epochs | Train Loss | Validation Loss |
|--- |--- |--- |
| 1| 0.09961754083633423|  0.09323594719171524|
| 2| 0.09265559166669846|  0.09110038727521896|
| 3| 0.09106099605560303|  0.08995018154382706|
| 4| 0.09000983089208603|  0.08897411823272705|
| 5| 0.0892450213432312|  0.08839458972215652|


## Model Plot


<details>
<summary>View Model Plot</summary>


![Model Image](./model.png)


</details>


model name: model-card-toggle-test4


model author: merve


task: object-detection


tags: tensorboard, keras, object-detection


tags:


- Keras


# TODO: Fill this model card


model name: conv-autoencoder


tags: keras, Keras


Pre-trained to have better reasoning ability, try this if you are working with task like QA. For more details please see https://openreview.net/forum?id=cGB7CMFtrSx


This is based on bert-base-uncased model and pre-trained for text input


model name: ReasonBERT-BERT


model author: Anonymous


task: feature-extraction


tags: pytorch, bert, feature-extraction, transformers, infinity_compatible


This is based on roberta-base model and pre-trained for text input


model name: ReasonBERT-RoBERTa


tags: pytorch, roberta, feature-extraction, transformers, infinity_compatible


This is based on tapas-base(no_reset) model and pre-trained for table input


model name: ReasonBERT-TAPAS


tags: pytorch, tapas, feature-extraction, transformers


tags:


- conversational


This is an improved version of the Joshua bot


model name: DialoGPT-medium-joshua


model author: ThatSkyFox


task: conversational


tags: gpt2, text-generation, transformers, conversational


#This is a chatbot trained on the transcript of the game "The World Ends with You"


model name: DialoGPT-small-joshua


tags: pytorch, gpt2, text-generation, transformers, conversational


license: mit
tags:


- generated_from_trainer
model-index:
- name: output_mlm
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# output_mlm


This model is a fine-tuned version of [roberta-large](https://huggingface.co/roberta-large) on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 1.2024


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3.0


### Training results


| Training Loss | Epoch | Step   | Validation Loss |
|:-------------:|:-----:|:------:|:---------------:|
| 1.5832        | 0.19  | 15000  | 1.4992          |
| 1.5325        | 0.39  | 30000  | 1.4653          |
| 1.4979        | 0.58  | 45000  | 1.4359          |
| 1.4715        | 0.77  | 60000  | 1.4039          |
| 1.4448        | 0.97  | 75000  | 1.3877          |
| 1.4191        | 1.16  | 90000  | 1.3603          |
| 1.3988        | 1.35  | 105000 | 1.3425          |
| 1.3699        | 1.54  | 120000 | 1.3230          |
| 1.3493        | 1.74  | 135000 | 1.3012          |
| 1.3201        | 1.93  | 150000 | 1.2773          |
| 1.2993        | 2.12  | 165000 | 1.2617          |
| 1.2745        | 2.32  | 180000 | 1.2490          |
| 1.2614        | 2.51  | 195000 | 1.2283          |
| 1.2424        | 2.7   | 210000 | 1.2152          |
| 1.2296        | 2.9   | 225000 | 1.2052          |


### Framework versions


- Transformers 4.11.2
- Pytorch 1.9.0
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: roberta-large-dialog-narrative


model author: lacai


task: fill-mask


tags: pytorch, tensorboard, roberta, fill-mask, transformers, license:mit, generated_from_trainer, model-index, autonlp_compatible, infinity_compatible


Base model: [microsoft/DialoGPT-small](https://huggingface.co/microsoft/DialoGPT-small)


Fine tuned for dialogue response generation on the [Schema Guided Dialogue Dataset](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue) (Rastogi et al., 2019)


Three additional special tokens were added during the fine-tuning process:


- <|pad|> padding token
- <|user|> speaker control token to prompt user responses
- <|system|> speaker control token to prompt system responses


model name: DialoGPT-small-SGD


task: text-generation


tags: pytorch, tensorboard, gpt2, text-generation, transformers


Fine tuned for dialogue response generation on the [Persuasion For Good Dataset](https://gitlab.com/ucdavisnlp/persuasionforgood) (Wang et al., 2019)


The following Dialogues were excluded:


- Those with donation amounts outside of the task range of [$0, $2].
- Those where a donation of 0 was made at the end of the task but a non-zero amount was pledged in the dialogue.
- Those with more than 800 words.


Stats:


- Training set: 519 dialogues
- Validation set: 58 dialogues
- ~20 utterances per dialogue


model name: DialoGPT-small-PFG


Base model: [gpt2-xl](https://huggingface.co/gpt2-xl)


Domain-adapted for dialogue response and narrative generation on a [narrative-aligned variant](https://github.com/AbrahamSanders/gutenberg-dialog#download-narrative-aligned-datasets) of the [Gutenberg Dialogue Dataset (Csaky & Recski, 2021)](https://aclanthology.org/2021.eacl-main.11.pdf)


Fine-tuned for dialogue response generation on [Persuasion For Good (Wang et al., 2019)](https://aclanthology.org/P19-1566.pdf) ([dataset](https://gitlab.com/ucdavisnlp/persuasionforgood))


model name: gpt2-xl-dialog-narrative-persuasion


model author: lacai


tags: pytorch, gpt2, text-generation, transformers


Base model: [microsoft/DialoGPT-large](https://huggingface.co/microsoft/DialoGPT-large)


model name: DialoGPT-large-PFG


tags:


- generated_from_trainer
metrics:
- spearmanr
model-index:
- name: koelectra-sts-v0.4
results:
  - task:
name: Text Classification
type: text-classification
metrics:
    - name: Spearmanr
type: spearmanr
value: 0.9286505242442783


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# koelectra-sts-v0.4


This model was trained from scratch on an unknown dataset.
It achieves the following results on the evaluation set:


- Loss: 0.3368
- Pearson: 0.9303
- Spearmanr: 0.9287


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss | Pearson | Spearmanr |
|:-------------:|:-----:|:----:|:---------------:|:-------:|:---------:|
| 0.0345        | 1.0   | 730  | 0.3368          | 0.9303  | 0.9287    |
| 0.0343        | 2.0   | 1460 | 0.3368          | 0.9303  | 0.9287    |
| 0.0337        | 3.0   | 2190 | 0.3368          | 0.9303  | 0.9287    |
| 0.0345        | 4.0   | 2920 | 0.3368          | 0.9303  | 0.9287    |
| 0.0347        | 5.0   | 3650 | 0.3368          | 0.9303  | 0.9287    |


### Framework versions


- Transformers 4.10.0
- Pytorch 1.10.1+cu113
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: koelectra-sts-v0.4


model author: Ketzu


task: text-classification


tags: pytorch, tensorboard, electra, text-classification, transformers, generated_from_trainer, model-index, infinity_compatible


## language: hu
metrics: rouge


We publish an abstractive summarizer for Hungarian, an
encoder-decoder model initialized with [huBERT](huggingface.co/SZTAKI-HLT/hubert-base-cc), and fine-tuned on the
[ELTE.DH](https://elte-dh.hu/) corpus of former Hungarian news portals. The model produces fluent output in the correct topic, but it hallucinates frequently.
Our quantitative evaluation on automatic and human transcripts of news
(with automatic and human-made punctuation, [Tündik et al. (2019)](https://www.isca-speech.org/archive/interspeech_2019/tundik19_interspeech.html), [Tündik and Szaszák (2019)](https://www.isca-speech.org/archive/interspeech_2019/szaszak19_interspeech.html)) shows that the model is
robust with respect to errors in either automatic speech recognition or
automatic punctuation restoration. In fine-tuning and inference, we followed [a jupyter notebook by Patrick von
Platen](https://github.com/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb). Most hyper-parameters are the same as those by von Platen, but we
found it advantageous to change the minimum length of the summary to 8 word-
pieces (instead of 56), and the number of beams in beam search to 5 (instead
of 4). Our model was fine-tuned on a server of the [SZTAKI-HLT](hlt.bme.hu/) group, which kindly
provided access to it.


model name: foszt2oszt


model author: BME-TMIT


task: text2text-generation


tags: pytorch, encoder-decoder, text2text-generation, hu, transformers, autonlp_compatible


# Building a HuggingFace Transformer NLP Model


## Running this Repo


model name: slai_transformer


model author: aogara


tags:


language:


- ja


名言推論モデル


model name: meigen_generate_Japanese


model author: Momerio


task: text-generation


tags: pytorch, gpt2, text-generation, ja, transformers


language: es
tags:


- token-classification
- nli
- pytorch
datasets:
- our own
pipeline_tag: token-classification
widget:
- text: "La recaudación del ministerio del interior fue de 20,000,000 euros así constatado por el artículo 24 de la Constitución Española."


model name: NER_LAW_MONEY4


model author: pitoneros


task: token-classification


tags: pytorch, roberta, token-classification, es, dataset:our own, transformers, nli, autonlp_compatible, infinity_compatible


- conversational


#Lelouch DialoGPT model


model name: DialoGPT-medium-Lelouch


model author: Nova


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


tags:


# Tony Stark dialoGPT model


model name: DialoGPT-small-tonystarkscript


model author: Navya2608


# Rachel Green DialoGPT Model


model name: DialoGPT-medium-rachel


# Chandler Bing DialoGPT Model


model name: DialoGPT-medium-chandler


#hi


model name: DDDC2


model author: Biasface


- conversational


model name: DDDC


license: cc-by-4.0
tags:


- generated_from_trainer
datasets:
- amazon_reviews_multi
metrics:
- accuracy
model_index:
- name: roberta-base-bne-finetuned-amazon_reviews_multi
results:
  - task:
name: Text Classification
type: text-classification
dataset:
name: amazon_reviews_multi
type: amazon_reviews_multi
args: es
metric:
name: Accuracy
type: accuracy
value: 0.9335


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# roberta-base-bne-finetuned-amazon_reviews_multi


This model is a fine-tuned version of [BSC-TeMU/roberta-base-bne](https://huggingface.co/BSC-TeMU/roberta-base-bne) on the amazon_reviews_multi dataset.
It achieves the following results on the evaluation set:


- Loss: 0.2275
- Accuracy: 0.9335


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2


### Training results


| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 0.1909        | 1.0   | 1250 | 0.1717          | 0.9333   |
| 0.0932        | 2.0   | 2500 | 0.2275          | 0.9335   |


### Framework versions


- Transformers 4.9.2
- Pytorch 1.9.0+cu102
- Datasets 1.11.0
- Tokenizers 0.10.3


model name: roberta-base-bne-finetuned-amazon_reviews_multi


model author: Hormigo


task: text-classification


tags: pytorch, tensorboard, roberta, text-classification, dataset:amazon_reviews_multi, transformers, license:cc-by-4.0, generated_from_trainer, infinity_compatible


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- glue
metrics:
- matthews_correlation
model-index:
- name: distilbert-base-uncased-finetuned-cola
results:
  - task:
name: Text Classification
type: text-classification
dataset:
name: glue
type: glue
args: cola
metrics:
    - name: Matthews Correlation
type: matthews_correlation
value: 0.5451837431775948


# distilbert-base-uncased-finetuned-cola


This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the glue dataset.
It achieves the following results on the evaluation set:


- Loss: 0.8508
- Matthews Correlation: 0.5452


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5


### Training results


| Training Loss | Epoch | Step | Validation Loss | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------:|
| 0.5221        | 1.0   | 535  | 0.5370          | 0.4246               |
| 0.3462        | 2.0   | 1070 | 0.5157          | 0.5183               |
| 0.2332        | 3.0   | 1605 | 0.6324          | 0.5166               |
| 0.1661        | 4.0   | 2140 | 0.7616          | 0.5370               |
| 0.1263        | 5.0   | 2675 | 0.8508          | 0.5452               |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.0+cu111
- Datasets 1.14.0
- Tokenizers 0.10.3


model name: distilbert-base-uncased-finetuned-cola


model author: hchc


task: text-classification


tags: pytorch, tensorboard, distilbert, text-classification, dataset:glue, transformers, license:apache-2.0, generated_from_trainer, model-index, infinity_compatible


language:


- es
license: apache-2.0
tags:
- "national library of spain"
- "spanish"
- "bne"
- "capitel"
- "pos"
datasets:
- "bne"
- "capitel"\
metrics:
- "f1"
widget:
- text: "Festival de San Sebastián: Johnny Depp recibirá el premio Donostia en pleno rifirrafe judicial con Amber Heard"
- text: "El alcalde de Vigo, Abel Caballero, ha comenzado a colocar las luces de Navidad en agosto."
- text: "Gracias a los datos de la BNE, se ha podido lograr este modelo del lenguaje."
- text: "El Tribunal Superior de Justicia se pronunció ayer: \"Hay base legal dentro del marco jurídico actual\"."
inference:
parameters:
aggregation_strategy: "first"


# Spanish RoBERTa-base trained on BNE finetuned for CAPITEL Part of Speech (POS) dataset


RoBERTa-base-bne is a transformer-based masked language model for the Spanish language. It is based on the [RoBERTa](https://arxiv.org/abs/1907.11692) base model and has been pre-trained using the largest Spanish corpus known to date, with a total of 570GB of clean and deduplicated text processed for this work, compiled from the web crawlings performed by the  [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) from 2009 to 2019.


Original pre-trained model can be found here: https://huggingface.co/PlanTL-GOB-ES/roberta-base-bne


## Dataset


The dataset used is the one from the [CAPITEL competition at IberLEF 2020](https://sites.google.com/view/capitel2020) (sub-task 2).


## Evaluation and results


F1 Score: 0.9846 (average of 5 runs).


For evaluation details visit our [GitHub repository](https://github.com/PlanTL-GOB-ES/lm-spanish).


## Citing


Check out our paper for all the details: https://arxiv.org/abs/2107.07253


```
@misc{gutierrezfandino2021spanish,
      title={Spanish Language Models},
      author={Asier Gutiérrez-Fandiño and Jordi Armengol-Estapé and Marc Pàmies and Joan Llop-Palao and Joaquín Silveira-Ocampo and Casimiro Pio Carrino and Aitor Gonzalez-Agirre and Carme Armentano-Oller and Carlos Rodriguez-Penagos and Marta Villegas},
      year={2021},
      eprint={2107.07253},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


## Funding


This work was partially funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL, and the Future of Computing Center, a Barcelona Supercomputing Center and IBM initiative (2020).


## Disclaimer


The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.


When third  parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence.


In no event shall the owner of the models (SEDIA – State Secretariat for digitalization and artificial intelligence) nor the creator (BSC – Barcelona Supercomputing Center) be liable for any results arising from the use made by third parties of these models.


Los modelos publicados en este repositorio tienen una finalidad generalista y están a disposición de terceros. Estos modelos pueden tener sesgos y/u otro tipo de distorsiones indeseables.


Cuando terceros desplieguen o proporcionen sistemas y/o servicios a otras partes usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) o se conviertan en usuarios de los modelos, deben tener en cuenta que es su responsabilidad mitigar los riesgos derivados de su uso y, en todo caso, cumplir con la normativa aplicable, incluyendo la normativa en materia de uso de inteligencia artificial.


En ningún caso el propietario de los modelos (SEDIA – Secretaría de Estado de Digitalización e Inteligencia Artificial) ni el creador (BSC – Barcelona Supercomputing Center) serán responsables de los resultados derivados del uso que hagan terceros de estos modelos.


model name: roberta-base-bne-capitel-pos


model author: PlanTL-GOB-ES


task: token-classification


tags: pytorch, roberta, token-classification, es, dataset:bne, dataset:capitel, arxiv:1907.11692, arxiv:2107.07253, transformers, license:apache-2.0, national library of spain, spanish, bne, capitel, pos, autonlp_compatible, infinity_compatible


language:


- es
license: apache-2.0
tags:
- "national library of spain"
- "spanish"
- "bne"
datasets:
- "bne"\
metrics:
- "ppl"


# GPT2-large trained with data from National Library of Spain (BNE)


## Model Description


GPT2-large-bne is a transformer-based model for the Spanish language. It is based on the [GPT-2](http://www.persagen.com/files/misc/radford2019language.pdf) model and has been pre-trained using the largest Spanish corpus known to date, with a total of 570GB of clean and deduplicated text processed for this work, compiled from the web crawlings performed by the  [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) from 2009 to 2019.


## Training corpora and preprocessing


The [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) crawls all .es domains once a year. The training corpus consists of 59TB of WARC files from these crawls, carried out from 2009 to 2019.


To obtain a high-quality training corpus, the corpus has been preprocessed with a pipeline of operations, including among the others, sentence splitting, language detection, filtering of bad-formed sentences and deduplication of repetitive contents. During the process document boundaries are kept. This resulted into 2TB of Spanish clean corpus. Further global deduplication among the corpus is applied, resulting into 570GB of text.


Some of the statistics of the corpus:


| Corpora | Number of documents | Number of tokens | Size (GB) |
|---------|---------------------|------------------|-----------|
| BNE     |         201,080,084 |  135,733,450,668 |     570GB |


## Tokenization and pre-training


The training corpus has been tokenized using a byte version of Byte-Pair Encoding (BPE) used in the original [GPT-2](http://www.persagen.com/files/misc/radford2019language.pdf) model with a vocabulary size of 50,262 tokens. The GPT2-large-bne pre-training consists of an autoregressive language model training that follows the approach of the GPT-2. The training lasted a total of 10 days with 32 computing nodes each one with 4 NVIDIA V100 GPUs of 16GB VRAM.


## Evaluation and results


For evaluation details visit our [GitHub repository](https://github.com/PlanTL-GOB-ES/lm-spanish).


## Citing


Check out our paper for all the details: https://arxiv.org/abs/2107.07253


```
@misc{gutierrezfandino2021spanish,
      title={Spanish Language Models},
      author={Asier Gutiérrez-Fandiño and Jordi Armengol-Estapé and Marc Pàmies and Joan Llop-Palao and Joaquín Silveira-Ocampo and Casimiro Pio Carrino and Aitor Gonzalez-Agirre and Carme Armentano-Oller and Carlos Rodriguez-Penagos and Marta Villegas},
      year={2021},
      eprint={2107.07253},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


## Funding


This work was funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL.


## Disclaimer


The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.


When third  parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence.


In no event shall the owner of the models (SEDIA – State Secretariat for digitalization and artificial intelligence) nor the creator (BSC – Barcelona Supercomputing Center) be liable for any results arising from the use made by third parties of these models.


Los modelos publicados en este repositorio tienen una finalidad generalista y están a disposición de terceros. Estos modelos pueden tener sesgos y/u otro tipo de distorsiones indeseables.


Cuando terceros desplieguen o proporcionen sistemas y/o servicios a otras partes usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) o se conviertan en usuarios de los modelos, deben tener en cuenta que es su responsabilidad mitigar los riesgos derivados de su uso y, en todo caso, cumplir con la normativa aplicable, incluyendo la normativa en materia de uso de inteligencia artificial.


En ningún caso el propietario de los modelos (SEDIA – Secretaría de Estado de Digitalización e Inteligencia Artificial) ni el creador (BSC – Barcelona Supercomputing Center) serán responsables de los resultados derivados del uso que hagan terceros de estos modelos.


model name: gpt2-large-bne


model author: PlanTL-GOB-ES


task: text-generation


tags: pytorch, gpt2, text-generation, es, dataset:bne, arxiv:2107.07253, transformers, license:apache-2.0, national library of spain, spanish, bne


language:


- es
license: apache-2.0
tags:
- "national library of spain"
- "spanish"
- "bne"
datasets:
- "bne"\
metrics:
- "ppl"


# GPT2-base trained with data from National Library of Spain (BNE)


## Model Description


GPT2-base-bne is a transformer-based model for the Spanish language. It is based on the [GPT-2](http://www.persagen.com/files/misc/radford2019language.pdf) model and has been pre-trained using the largest Spanish corpus known to date, with a total of 570GB of clean and deduplicated text processed for this work, compiled from the web crawlings performed by the  [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) from 2009 to 2019.


## Training corpora and preprocessing


The [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) crawls all .es domains once a year. The training corpus consists of 59TB of WARC files from these crawls, carried out from 2009 to 2019.


To obtain a high-quality training corpus, the corpus has been preprocessed with a pipeline of operations, including among the others, sentence splitting, language detection, filtering of bad-formed sentences and deduplication of repetitive contents. During the process document boundaries are kept. This resulted into 2TB of Spanish clean corpus. Further global deduplication among the corpus is applied, resulting into 570GB of text.


Some of the statistics of the corpus:


| Corpora | Number of documents | Number of tokens | Size (GB) |
|---------|---------------------|------------------|-----------|
| BNE     |         201,080,084 |  135,733,450,668 |     570GB |


## Tokenization and pre-training


The training corpus has been tokenized using a byte version of Byte-Pair Encoding (BPE) used in the original [GPT-2](http://www.persagen.com/files/misc/radford2019language.pdf) model with a vocabulary size of 50,262 tokens. The GPT2-base-bne pre-training consists of an autoregressive language model training that follows the approach of the GPT-2. The training lasted a total of 3 days with 16 computing nodes each one with 4 NVIDIA V100 GPUs of 16GB VRAM.


## Evaluation and results


For evaluation details visit our [GitHub repository](https://github.com/PlanTL-GOB-ES/lm-spanish).


## Citing


Check out our paper for all the details: https://arxiv.org/abs/2107.07253


```
@misc{gutierrezfandino2021spanish,
      title={Spanish Language Models},
      author={Asier Gutiérrez-Fandiño and Jordi Armengol-Estapé and Marc Pàmies and Joan Llop-Palao and Joaquín Silveira-Ocampo and Casimiro Pio Carrino and Aitor Gonzalez-Agirre and Carme Armentano-Oller and Carlos Rodriguez-Penagos and Marta Villegas},
      year={2021},
      eprint={2107.07253},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


## Funding


This work was funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL.


## Disclaimer


The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.


When third  parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence.


In no event shall the owner of the models (SEDIA – State Secretariat for digitalization and artificial intelligence) nor the creator (BSC – Barcelona Supercomputing Center) be liable for any results arising from the use made by third parties of these models.


Los modelos publicados en este repositorio tienen una finalidad generalista y están a disposición de terceros. Estos modelos pueden tener sesgos y/u otro tipo de distorsiones indeseables.


Cuando terceros desplieguen o proporcionen sistemas y/o servicios a otras partes usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) o se conviertan en usuarios de los modelos, deben tener en cuenta que es su responsabilidad mitigar los riesgos derivados de su uso y, en todo caso, cumplir con la normativa aplicable, incluyendo la normativa en materia de uso de inteligencia artificial.


En ningún caso el propietario de los modelos (SEDIA – Secretaría de Estado de Digitalización e Inteligencia Artificial) ni el creador (BSC – Barcelona Supercomputing Center) serán responsables de los resultados derivados del uso que hagan terceros de estos modelos.


model name: gpt2-base-bne


model author: PlanTL-GOB-ES


task: text-generation


tags: pytorch, gpt2, text-generation, es, dataset:bne, arxiv:2107.07253, transformers, license:apache-2.0, national library of spain, spanish, bne


language:


- es
license: apache-2.0
tags:
- "national library of spain"
- "spanish"
- "bne"
- "qa"
- "question answering"
datasets:
- "PlanTL-GOB-ES/SQAC"\
metrics:
- "f1"


# Spanish RoBERTa-base trained on BNE finetuned for Spanish Question Answering Corpus (SQAC) dataset.


RoBERTa-base-bne is a transformer-based masked language model for the Spanish language. It is based on the [RoBERTa](https://arxiv.org/abs/1907.11692) base model and has been pre-trained using the largest Spanish corpus known to date, with a total of 570GB of clean and deduplicated text processed for this work, compiled from the web crawlings performed by the  [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) from 2009 to 2019.


Original pre-trained model can be found here: https://huggingface.co/PlanTL-GOB-ES/roberta-base-bne


## Dataset


The dataset used is the [SQAC corpus](https://huggingface.co/datasets/PlanTL-GOB-ES/SQAC).


## Evaluation and results


F1 Score: 0.7923 (average of 5 runs).


## Citing


## Funding


This work was partially funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL, and the Future of Computing Center, a Barcelona Supercomputing Center and IBM initiative (2020).


## Disclaimer


model name: roberta-base-bne-sqac


model author: PlanTL-GOB-ES


task: question-answering


tags: pytorch, roberta, question-answering, es, dataset:PlanTL-GOB-ES/SQAC, arxiv:1907.11692, arxiv:2107.07253, transformers, license:apache-2.0, national library of spain, spanish, bne, qa, question answering, autonlp_compatible, infinity_compatible


language:


- es
license: apache-2.0
tags:
- "national library of spain"
- "spanish"
- "bne"
- "capitel"
- "pos"
datasets:
- "bne"
- "capitel"\
metrics:
- "f1"
widget:
- text: "Festival de San Sebastián: Johnny Depp recibirá el premio Donostia en pleno rifirrafe judicial con Amber Heard"
- text: "El alcalde de Vigo, Abel Caballero, ha comenzado a colocar las luces de Navidad en agosto."
- text: "Gracias a los datos de la BNE, se ha podido lograr este modelo del lenguaje."
- text: "El Tribunal Superior de Justicia se pronunció ayer: \"Hay base legal dentro del marco jurídico actual\"."
inference:
parameters:
aggregation_strategy: "first"


# Spanish RoBERTa-large trained on BNE finetuned for CAPITEL Part of Speech (POS) dataset


RoBERTa-large-bne is a transformer-based masked language model for the Spanish language. It is based on the [RoBERTa](https://arxiv.org/abs/1907.11692) large model and has been pre-trained using the largest Spanish corpus known to date, with a total of 570GB of clean and deduplicated text processed for this work, compiled from the web crawlings performed by the  [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) from 2009 to 2019.


Original pre-trained model can be found here: https://huggingface.co/BSC-TeMU/roberta-large-bne


## Dataset


The dataset used is the one from the [CAPITEL competition at IberLEF 2020](https://sites.google.com/view/capitel2020) (sub-task 2).


## Evaluation and results


F1 Score: 0.9851 (average of 5 runs).


For evaluation details visit our [GitHub repository](https://github.com/PlanTL-GOB-ES/lm-spanish).


## Citing


Check out our paper for all the details: https://arxiv.org/abs/2107.07253


```
@misc{gutierrezfandino2021spanish,
      title={Spanish Language Models},
      author={Asier Gutiérrez-Fandiño and Jordi Armengol-Estapé and Marc Pàmies and Joan Llop-Palao and Joaquín Silveira-Ocampo and Casimiro Pio Carrino and Aitor Gonzalez-Agirre and Carme Armentano-Oller and Carlos Rodriguez-Penagos and Marta Villegas},
      year={2021},
      eprint={2107.07253},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


## Funding


## Disclaimer


The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.


When third  parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence.


In no event shall the owner of the models (SEDIA – State Secretariat for digitalization and artificial intelligence) nor the creator (BSC – Barcelona Supercomputing Center) be liable for any results arising from the use made by third parties of these models.


Los modelos publicados en este repositorio tienen una finalidad generalista y están a disposición de terceros. Estos modelos pueden tener sesgos y/u otro tipo de distorsiones indeseables.


Cuando terceros desplieguen o proporcionen sistemas y/o servicios a otras partes usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) o se conviertan en usuarios de los modelos, deben tener en cuenta que es su responsabilidad mitigar los riesgos derivados de su uso y, en todo caso, cumplir con la normativa aplicable, incluyendo la normativa en materia de uso de inteligencia artificial.


En ningún caso el propietario de los modelos (SEDIA – Secretaría de Estado de Digitalización e Inteligencia Artificial) ni el creador (BSC – Barcelona Supercomputing Center) serán responsables de los resultados derivados del uso que hagan terceros de estos modelos.


model name: roberta-large-bne-capitel-pos


model author: PlanTL-GOB-ES


task: token-classification


tags: pytorch, roberta, token-classification, es, dataset:bne, dataset:capitel, arxiv:1907.11692, arxiv:2107.07253, transformers, license:apache-2.0, national library of spain, spanish, bne, capitel, pos, autonlp_compatible, infinity_compatible


language:


- es
license: apache-2.0
tags:
- "national library of spain"
- "spanish"
- "bne"
datasets:
- "bne"\
metrics:
- "ppl"
widget:
- text: "Este año las campanadas de La Sexta las <mask> Pedroche y Chicote."
- text: "El artista Antonio Orozco es un colaborador de La <mask>."
- text: "Gracias a los datos de la BNE se ha podido <mask> este modelo del lenguaje."
- text: "Hay base legal dentro del marco <mask> actual."


# RoBERTa large trained with data from National Library of Spain (BNE)


## Model Description


## Training corpora and preprocessing


The [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) crawls all .es domains once a year. The training corpus consists of 59TB of WARC files from these crawls, carried out from 2009 to 2019.


To obtain a high-quality training corpus, the corpus has been preprocessed with a pipeline of operations, including among the others, sentence splitting, language detection, filtering of bad-formed sentences and deduplication of repetitive contents. During the process document boundaries are kept. This resulted into 2TB of Spanish clean corpus. Further global deduplication among the corpus is applied, resulting into 570GB of text.


Some of the statistics of the corpus:


| Corpora | Number of documents | Number of tokens | Size (GB) |
|---------|---------------------|------------------|-----------|
| BNE     |         201,080,084 |  135,733,450,668 |     570GB |


## Tokenization and pre-training


The training corpus has been tokenized using a byte version of Byte-Pair Encoding (BPE) used in the original [RoBERTA](https://arxiv.org/abs/1907.11692) model with a vocabulary size of 50,262 tokens. The RoBERTa-large-bne pre-training consists of a masked language model training that follows the approach employed for the RoBERTa large. The training lasted a total of 96 hours with 32 computing nodes each one with 4 NVIDIA V100 GPUs of 16GB VRAM.


## Evaluation and results


For evaluation details visit our [GitHub repository](https://github.com/PlanTL-GOB-ES/lm-spanish).


## Citing


Check out our paper for all the details: https://arxiv.org/abs/2107.07253


```
@misc{gutierrezfandino2021spanish,
      title={Spanish Language Models},
      author={Asier Gutiérrez-Fandiño and Jordi Armengol-Estapé and Marc Pàmies and Joan Llop-Palao and Joaquín Silveira-Ocampo and Casimiro Pio Carrino and Aitor Gonzalez-Agirre and Carme Armentano-Oller and Carlos Rodriguez-Penagos and Marta Villegas},
      year={2021},
      eprint={2107.07253},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


## Funding


This work was funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL.


## Disclaimer


The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.


When third  parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence.


In no event shall the owner of the models (SEDIA – State Secretariat for digitalization and artificial intelligence) nor the creator (BSC – Barcelona Supercomputing Center) be liable for any results arising from the use made by third parties of these models.


Los modelos publicados en este repositorio tienen una finalidad generalista y están a disposición de terceros. Estos modelos pueden tener sesgos y/u otro tipo de distorsiones indeseables.


Cuando terceros desplieguen o proporcionen sistemas y/o servicios a otras partes usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) o se conviertan en usuarios de los modelos, deben tener en cuenta que es su responsabilidad mitigar los riesgos derivados de su uso y, en todo caso, cumplir con la normativa aplicable, incluyendo la normativa en materia de uso de inteligencia artificial.


En ningún caso el propietario de los modelos (SEDIA – Secretaría de Estado de Digitalización e Inteligencia Artificial) ni el creador (BSC – Barcelona Supercomputing Center) serán responsables de los resultados derivados del uso que hagan terceros de estos modelos.


model name: roberta-large-bne


model author: PlanTL-GOB-ES


task: fill-mask


tags: pytorch, roberta, fill-mask, es, dataset:bne, arxiv:1907.11692, arxiv:2107.07253, transformers, license:apache-2.0, national library of spain, spanish, bne, autonlp_compatible, infinity_compatible


language:


- es
license: apache-2.0
tags:
- "national library of spain"
- "spanish"
- "bne"
- "capitel"
- "ner"
datasets:
- "bne"
- "capitel"\
metrics:
- "f1"
inference:
parameters:
aggregation_strategy: "first"


# Spanish RoBERTa-base trained on BNE finetuned for CAPITEL Named Entity Recognition (NER) dataset.


RoBERTa-base-bne is a transformer-based masked language model for the Spanish language. It is based on the [RoBERTa](https://arxiv.org/abs/1907.11692) base model and has been pre-trained using the largest Spanish corpus known to date, with a total of 570GB of clean and deduplicated text processed for this work, compiled from the web crawlings performed by the  [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) from 2009 to 2019.


Original pre-trained model can be found here: https://huggingface.co/BSC-TeMU/roberta-base-bne


## Dataset


The dataset used is the one from the [CAPITEL competition at IberLEF 2020](https://sites.google.com/view/capitel2020) (sub-task 1).


## Evaluation and results


F1 Score: 0.8960


## Citing


## Funding


This work was partially funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL, and the Future of Computing Center, a Barcelona Supercomputing Center and IBM initiative (2020).


## Disclaimer


model name: roberta-base-bne-capitel-ner


model author: PlanTL-GOB-ES


task: token-classification


tags: pytorch, roberta, token-classification, es, dataset:bne, dataset:capitel, arxiv:1907.11692, arxiv:2107.07253, transformers, license:apache-2.0, national library of spain, spanish, bne, capitel, ner, autonlp_compatible, infinity_compatible


language:


Original pre-trained model can be found here: https://huggingface.co/PlanTL-GOB-ES/roberta-base-bne


**IMPORTANT ABOUT THIS MODEL:** We modified the dataset to make this model more robust to general Spanish input. In the Spanish language all the name entities are capitalized, as this dataset has been elaborated by experts, it is totally correct in terms of Spanish language. We randomly took some entities and we lower-cased some of them for the model to learn not only that the named entities are capitalized, but also the structure of a sentence that should contain a named entity. For instance: "My name is [placeholder]", this [placeholder] should be a named entity even though it is not written capitalized. The model trained on the original capitel dataset can be found here: https://huggingface.co/PlanTL-GOB-ES/roberta-base-bne-capitel-ner


Examples:


This model:


- "Me llamo asier y vivo en barcelona todo el año." → "Me llamo {asier:S-PER} y vivo en {barcelona:S-LOC} todo el año."
- "Hoy voy a visitar el parc güell tras salir del barcelona supercomputing center." → "Hoy voy a visitar el {park:B-LOC} {güell:E-LOC} tras salir del {barcelona:B-ORG} {supercomputing:I-ORG} {center:E-ORG}."


Model trained on original data:


- "Me llamo asier y vivo en barcelona todo el año." → "Me llamo asier y vivo en barcelona todo el año." (nothing)
- "Hoy voy a visitar el parc güell tras salir del barcelona supercomputing center." → "Hoy voy a visitar el parc güell tras salir del barcelona supercomputing center." (nothing)


## Evaluation and results


F1 Score: 0.8867


For evaluation details visit our [GitHub repository](https://github.com/PlanTL-GOB-ES/lm-spanish).


## Citing


Check out our paper for all the details: https://arxiv.org/abs/2107.07253


```
@misc{gutierrezfandino2021spanish,
      title={Spanish Language Models},
      author={Asier Gutiérrez-Fandiño and Jordi Armengol-Estapé and Marc Pàmies and Joan Llop-Palao and Joaquín Silveira-Ocampo and Casimiro Pio Carrino and Aitor Gonzalez-Agirre and Carme Armentano-Oller and Carlos Rodriguez-Penagos and Marta Villegas},
      year={2021},
      eprint={2107.07253},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


## Funding


## Disclaimer


The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.


When third  parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence.


In no event shall the owner of the models (SEDIA – State Secretariat for digitalization and artificial intelligence) nor the creator (BSC – Barcelona Supercomputing Center) be liable for any results arising from the use made by third parties of these models.


Los modelos publicados en este repositorio tienen una finalidad generalista y están a disposición de terceros. Estos modelos pueden tener sesgos y/u otro tipo de distorsiones indeseables.


Cuando terceros desplieguen o proporcionen sistemas y/o servicios a otras partes usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) o se conviertan en usuarios de los modelos, deben tener en cuenta que es su responsabilidad mitigar los riesgos derivados de su uso y, en todo caso, cumplir con la normativa aplicable, incluyendo la normativa en materia de uso de inteligencia artificial.


En ningún caso el propietario de los modelos (SEDIA – Secretaría de Estado de Digitalización e Inteligencia Artificial) ni el creador (BSC – Barcelona Supercomputing Center) serán responsables de los resultados derivados del uso que hagan terceros de estos modelos.


model name: roberta-base-bne-capitel-ner-plus


model author: PlanTL-GOB-ES


task: token-classification


language:


- es
tags:
- biomedical
- spanish
license: apache-2.0
metrics:
- ppl
widget:
- text: "El único antecedente personal a reseñar era la <mask> arterial."
- text: "Las radiologías óseas de cuerpo entero no detectan alteraciones <mask>, ni alteraciones vertebrales."
- text: "En el <mask> toraco-abdómino-pélvico no se encontraron hallazgos patológicos de interés."


# Biomedical language model for Spanish


Biomedical pretrained language model for Spanish. For more details about the corpus, the pretraining and the evaluation, check the official [repository](https://github.com/PlanTL-GOB-ES/lm-biomedical-clinical-es) and read our [preprint](https://arxiv.org/abs/2109.03570) "*Carrino, C. P., Armengol-Estapé, J., Gutiérrez-Fandiño, A., Llop-Palao, J., Pàmies, M., Gonzalez-Agirre, A., & Villegas, M. (2021). Biomedical and Clinical Language Models for Spanish: On the Benefits of Domain-Specific Pretraining in a Mid-Resource Scenario.*".


## Tokenization and model pretraining


This model is a [RoBERTa-based](https://github.com/pytorch/fairseq/tree/master/examples/roberta) model trained on a
**biomedical** corpus in Spanish collected from several sources (see next section).
The training corpus has been tokenized using a byte version of [Byte-Pair Encoding (BPE)](https://github.com/openai/gpt-2)
used in the original [RoBERTA](https://github.com/pytorch/fairseq/tree/master/examples/roberta) model with a vocabulary size of 52,000 tokens. The pretraining consists of a masked language model training at the subword level following the approach employed for the RoBERTa base model with the same hyperparameters as in the original work. The training lasted a total of 48 hours with 16 NVIDIA V100 GPUs of 16GB DDRAM, using Adam optimizer with a peak learning rate of 0.0005 and an effective batch size of 2,048 sentences.


## Training corpora and preprocessing


The training corpus is composed of several biomedical corpora in Spanish, collected from publicly available corpora and crawlers.
To obtain a high-quality training corpus, a cleaning pipeline with the following operations has been applied:


- data parsing in different formats
  - sentence splitting
  - language detection
  - filtering of ill-formed sentences
  - deduplication of repetitive contents
  - keep the original document boundaries


Finally, the corpora are concatenated and further global deduplication among the corpora have been applied.
The result is a medium-size biomedical corpus for Spanish composed of about 963M tokens. The table below shows some basic statistics of the individual cleaned corpora:


| Name                                                                                | No. tokens  | Description                                                                                |
|-----------------------------------------------------------------------------------------|-------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Medical crawler](https://zenodo.org/record/4561970)                                    | 745,705,946 | Crawler of more than 3,000 URLs belonging to Spanish biomedical and health domains.                                                                                |
| Clinical cases misc.                                                                    | 102,855,267 | A miscellany of medical content, essentially clinical cases. Note that a clinical case report is a scientific publication where medical practitioners share patient cases and it is different from a clinical note or document.                                                                                |
| [Scielo](https://github.com/PlanTL-GOB-ES/SciELO-Spain-Crawler)                        | 60,007,289  | Publications written in Spanish crawled from the Spanish SciELO server in 2017.                                                                                |
| [BARR2_background](https://temu.bsc.es/BARR2/downloads/background_set.raw_text.tar.bz2) | 24,516,442  | Biomedical Abbreviation Recognition and Resolution (BARR2) containing Spanish clinical case study sections from a variety of clinical disciplines.                                                                                |
| Wikipedia_life_sciences                                                                 | 13,890,501  | Wikipedia articles crawled 04/01/2021 with the [Wikipedia API python library](https://pypi.org/project/Wikipedia-API/) starting from the "Ciencias\_de\_la\_vida" category up to a maximum of 5 subcategories. Multiple links to the same articles are then discarded to avoid repeating content.                                                                                |
| Patents                                                                                | 13,463,387  | Google Patent in Medical Domain for Spain (Spanish). The accepted codes (Medical Domain) for Json files of patents are: "A61B", "A61C","A61F", "A61H", "A61K", "A61L","A61M", "A61B", "A61P".                                                        |
| [EMEA](http://opus.nlpl.eu/download.php?f=EMEA/v3/moses/en-es.txt.zip)                  | 5,377,448   | Spanish-side documents extracted from parallel corpora made out of PDF documents from the European Medicines Agency.                                                                                |
| [mespen_Medline](https://zenodo.org/record/3562536#.YTt1fH2xXbR)                        | 4,166,077   | Spanish-side articles extracted from a collection of Spanish-English parallel corpus consisting of biomedical scientific literature.  The collection of parallel resources are aggregated from the MedlinePlus source. |
| PubMed                                                                                | 1,858,966   | Open-access articles from the PubMed repository crawled in 2017.                                                                                |


## Evaluation and results


The model has been evaluated on the Named Entity Recognition (NER) using the following datasets:


- [PharmaCoNER](https://zenodo.org/record/4270158): is a track on chemical and drug mention recognition from Spanish medical texts (for more info see: https://temu.bsc.es/pharmaconer/).
- [CANTEMIST](https://zenodo.org/record/3978041#.YTt5qH2xXbQ): is a shared task specifically focusing on named entity recognition of tumor morphology, in Spanish (for more info see: https://zenodo.org/record/3978041#.YTt5qH2xXbQ).
- ICTUSnet: consists of 1,006 hospital discharge reports of patients admitted for stroke from 18 different Spanish hospitals. It contains more than 79,000 annotations for 51 different kinds of variables.


The evaluation results are compared against the [mBERT](https://huggingface.co/bert-base-multilingual-cased) and [BETO](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased) models:


| F1 - Precision - Recall | roberta-base-biomedical-es | mBERT                   | BETO                    |
|---------------------------|----------------------------|-------------------------------|-------------------------|
| PharmaCoNER               | **89.48** - **87.85** - **91.18**    | 87.46 - 86.50 - 88.46 | 88.18 - 87.12 - 89.28 |
| CANTEMIST                 | **83.87** - **81.70** - **86.17**    | 82.61 - 81.12 - 84.15 | 82.42 - 80.91 - 84.00 |
| ICTUSnet                  | **88.12** - **85.56** - **90.83**    | 86.75 - 83.53 - 90.23 | 85.95 - 83.10 - 89.02 |


## Intended uses & limitations


The model is ready-to-use only for masked language modelling to perform the Fill Mask task (try the inference API or read the next section)


However, the is intended to be fine-tuned on downstream tasks such as Named Entity Recognition or Text Classification.


## Cite


If you use our models, please cite our latest preprint:


```bibtex

@misc{carrino2021biomedical,
      title={Biomedical and Clinical Language Models for Spanish: On the Benefits of Domain-Specific Pretraining in a Mid-Resource Scenario},
      author={Casimiro Pio Carrino and Jordi Armengol-Estapé and Asier Gutiérrez-Fandiño and Joan Llop-Palao and Marc Pàmies and Aitor Gonzalez-Agirre and Marta Villegas},
      year={2021},
      eprint={2109.03570},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

```


If you use our Medical Crawler corpus, please cite the preprint:


```bibtex

@misc{carrino2021spanish,
      title={Spanish Biomedical Crawled Corpus: A Large, Diverse Dataset for Spanish Biomedical Language Models},
      author={Casimiro Pio Carrino and Jordi Armengol-Estapé and Ona de Gibert Bonet and Asier Gutiérrez-Fandiño and Aitor Gonzalez-Agirre and Martin Krallinger and Marta Villegas},
      year={2021},
      eprint={2109.07765},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

```


## How to use


```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("PlanTL-GOB-ES/roberta-base-biomedical-es")

model = AutoModelForMaskedLM.from_pretrained("PlanTL-GOB-ES/roberta-base-biomedical-es")

from transformers import pipeline

unmasker = pipeline('fill-mask', model="PlanTL-GOB-ES/roberta-base-biomedical-es")

unmasker("El único antecedente personal a reseñar era la <mask> arterial.")
```


```
# Output
[
  {
    "sequence": " El único antecedente personal a reseñar era la hipertensión arterial.",
    "score": 0.9855039715766907,
    "token": 3529,
    "token_str": " hipertensión"
  },
  {
    "sequence": " El único antecedente personal a reseñar era la diabetes arterial.",
    "score": 0.0039140828885138035,
    "token": 1945,
    "token_str": " diabetes"
  },
  {
    "sequence": " El único antecedente personal a reseñar era la hipotensión arterial.",
    "score": 0.002484665485098958,
    "token": 11483,
    "token_str": " hipotensión"
  },
  {
    "sequence": " El único antecedente personal a reseñar era la Hipertensión arterial.",
    "score": 0.0023484621196985245,
    "token": 12238,
    "token_str": " Hipertensión"
  },
  {
    "sequence": " El único antecedente personal a reseñar era la presión arterial.",
    "score": 0.0008009297889657319,
    "token": 2267,
    "token_str": " presión"
  }
]
```


## Funding


This work was funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL.


## Disclaimer


The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.


When third  parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence.


In no event shall the owner of the models (SEDIA – State Secretariat for digitalization and artificial intelligence) nor the creator (BSC – Barcelona Supercomputing Center) be liable for any results arising from the use made by third parties of these models.


Los modelos publicados en este repositorio tienen una finalidad generalista y están a disposición de terceros. Estos modelos pueden tener sesgos y/u otro tipo de distorsiones indeseables.


Cuando terceros desplieguen o proporcionen sistemas y/o servicios a otras partes usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) o se conviertan en usuarios de los modelos, deben tener en cuenta que es su responsabilidad mitigar los riesgos derivados de su uso y, en todo caso, cumplir con la normativa aplicable, incluyendo la normativa en materia de uso de inteligencia artificial.


En ningún caso el propietario de los modelos (SEDIA – Secretaría de Estado de Digitalización e Inteligencia Artificial) ni el creador (BSC – Barcelona Supercomputing Center) serán responsables de los resultados derivados del uso que hagan terceros de estos modelos.


model name: roberta-base-biomedical-es


model author: PlanTL-GOB-ES


task: fill-mask


tags: pytorch, roberta, fill-mask, es, arxiv:2109.03570, arxiv:2109.07765, transformers, biomedical, spanish, license:apache-2.0, autonlp_compatible, infinity_compatible


language:


- es
license: apache-2.0
tags:
- legal
- spanish
datasets:
- legal_ES
- temu_legal\
metrics:
- ppl
widget:
- text: "La ley fue <mask> finalmente."
- text: "El Tribunal <mask> desestimó el recurso de amparo."
- text: "Hay base legal dentro del marco <mask> actual."


# Spanish Legal-domain RoBERTa


There are few models trained for the Spanish language. Some of the models have been trained with a low resource, unclean corpora. The ones derived from the Spanish National Plan for Language Technologies are proficient solving several tasks and have been trained using large scale clean corpora. However, the Spanish Legal domain language could be think of an independent language on its own. We therefore created a Spanish Legal model from scratch trained exclusively on legal corpora.


## Citing


```
@misc{gutierrezfandino2021legal,
      title={Spanish Legalese Language Model and Corpora},
      author={Asier Gutiérrez-Fandiño and Jordi Armengol-Estapé and Aitor Gonzalez-Agirre and Marta Villegas},
      year={2021},
      eprint={2110.12201},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


For more information visit our [GitHub repository](https://github.com/PlanTL-GOB-ES/lm-legal-es)


## Funding


## Disclaimer


model name: RoBERTalex


tags: pytorch, roberta, fill-mask, es, dataset:legal_ES, dataset:temu_legal, arxiv:2110.12201, transformers, license:apache-2.0, legal, spanish, autonlp_compatible, infinity_compatible


- es
tags:
- biomedical
- clinical
- spanish
license: apache-2.0
metrics:
- ppl
widget:
- text: "El único antecedente personal a reseñar era la <mask> arterial."
- text: "Las radiologías óseas de cuerpo entero no detectan alteraciones <mask>, ni alteraciones vertebrales."
- text: "En el <mask> toraco-abdómino-pélvico no se encontraron hallazgos patológicos de interés."


# Biomedical-clinical language model for Spanish


Biomedical pretrained language model for Spanish. For more details about the corpus, the pretraining and the evaluation, check the official [repository](https://github.com/PlanTL-GOB-ES/lm-biomedical-clinical-es) and read our [preprint](https://arxiv.org/abs/2109.03570) "*Carrino, C. P., Armengol-Estapé, J., Gutiérrez-Fandiño, A., Llop-Palao, J., Pàmies, M., Gonzalez-Agirre, A., & Villegas, M. (2021). Biomedical and Clinical Language Models for Spanish: On the Benefits of Domain-Specific Pretraining in a Mid-Resource Scenario.*".


## Tokenization and model pretraining


This model is a [RoBERTa-based](https://github.com/pytorch/fairseq/tree/master/examples/roberta) model trained on a
**biomedical-clinical** corpus in Spanish collected from several sources (see next section).
The training corpus has been tokenized using a byte version of [Byte-Pair Encoding (BPE)](https://github.com/openai/gpt-2)
used in the original [RoBERTA](https://github.com/pytorch/fairseq/tree/master/examples/roberta) model with a vocabulary size of 52,000 tokens. The pretraining consists of a masked language model training at the subword level following the approach employed for the RoBERTa base model with the same hyperparameters as in the original work. The training lasted a total of 48 hours with 16 NVIDIA V100 GPUs of 16GB DDRAM, using Adam optimizer with a peak learning rate of 0.0005 and an effective batch size of 2,048 sentences.


## Training corpora and preprocessing


The training corpus is composed of several biomedical corpora in Spanish, collected from publicly available corpora and crawlers, and a real-world clinical corpus collected from more than 278K clinical documents and notes. To obtain a high-quality training corpus while retaining the idiosyncrasies of the clinical language, a cleaning pipeline has been applied only to the biomedical corpora, keeping the clinical corpus uncleaned. Essentially, the cleaning operations used are:


- data parsing in different formats
  - sentence splitting
  - language detection
  - filtering of ill-formed sentences
  - deduplication of repetitive contents
  - keep the original document boundaries


Then, the biomedical corpora are concatenated and further global deduplication among the biomedical corpora have been applied.
Eventually, the clinical corpus is concatenated to the cleaned biomedical corpus resulting in a medium-size biomedical-clinical corpus for Spanish composed of more than 1B tokens. The table below shows some basic statistics of the individual cleaned corpora:


| Name                                                                                | No. tokens  | Description                                                                                |
|-----------------------------------------------------------------------------------------|-------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Medical crawler](https://zenodo.org/record/4561970)                                    | 745,705,946 | Crawler of more than 3,000 URLs belonging to Spanish biomedical and health domains.                                                                                |
| Clinical cases misc.                                                                    | 102,855,267 | A miscellany of medical content, essentially clinical cases. Note that a clinical case report is a scientific publication where medical practitioners share patient cases and it is different from a clinical note or document.                                                                                |
| Clinical notes/documents                                                                | 91,250,080 | Collection of more than 278K clinical documents, including discharge reports, clinical course notes and X-ray reports, for a total of 91M tokens.                                                                                |
| [Scielo](https://github.com/PlanTL-GOB-ES/SciELO-Spain-Crawler)                        | 60,007,289  | Publications written in Spanish crawled from the Spanish SciELO server in 2017.                                                                                |
| [BARR2_background](https://temu.bsc.es/BARR2/downloads/background_set.raw_text.tar.bz2) | 24,516,442  | Biomedical Abbreviation Recognition and Resolution (BARR2) containing Spanish clinical case study sections from a variety of clinical disciplines.                                                                                |
| Wikipedia_life_sciences                                                                 | 13,890,501  | Wikipedia articles crawled 04/01/2021 with the [Wikipedia API python library](https://pypi.org/project/Wikipedia-API/) starting from the "Ciencias\_de\_la\_vida" category up to a maximum of 5 subcategories. Multiple links to the same articles are then discarded to avoid repeating content.                                                                                |
| Patents                                                                                | 13,463,387  | Google Patent in Medical Domain for Spain (Spanish). The accepted codes (Medical Domain) for Json files of patents are: "A61B", "A61C","A61F", "A61H", "A61K", "A61L","A61M", "A61B", "A61P".                                                        |
| [EMEA](http://opus.nlpl.eu/download.php?f=EMEA/v3/moses/en-es.txt.zip)                  | 5,377,448   | Spanish-side documents extracted from parallel corpora made out of PDF documents from the European Medicines Agency.                                                                                |
| [mespen_Medline](https://zenodo.org/record/3562536#.YTt1fH2xXbR)                        | 4,166,077   | Spanish-side articles extracted from a collection of Spanish-English parallel corpus consisting of biomedical scientific literature.  The collection of parallel resources are aggregated from the MedlinePlus source. |
| PubMed                                                                                | 1,858,966   | Open-access articles from the PubMed repository crawled in 2017.                                                                                |


## Evaluation and results


The model has been evaluated on the Named Entity Recognition (NER) using the following datasets:


- [PharmaCoNER](https://zenodo.org/record/4270158): is a track on chemical and drug mention recognition from Spanish medical texts (for more info see: https://temu.bsc.es/pharmaconer/).
- [CANTEMIST](https://zenodo.org/record/3978041#.YTt5qH2xXbQ): is a shared task specifically focusing on named entity recognition of tumor morphology, in Spanish (for more info see: https://zenodo.org/record/3978041#.YTt5qH2xXbQ).
- ICTUSnet: consists of 1,006 hospital discharge reports of patients admitted for stroke from 18 different Spanish hospitals. It contains more than 79,000 annotations for 51 different kinds of variables.


The evaluation results are compared against the [mBERT](https://huggingface.co/bert-base-multilingual-cased) and [BETO](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased) models:


| F1 - Precision - Recall | roberta-base-biomedical-clinical-es | mBERT                   | BETO                    |
|---------------------------|----------------------------|-------------------------------|-------------------------|
| PharmaCoNER               | **90.04** - **88.92** - **91.18**    | 87.46 - 86.50 - 88.46 | 88.18 - 87.12 - 89.28 |
| CANTEMIST                 | **83.34** - **81.48** - **85.30**    | 82.61 - 81.12 - 84.15 | 82.42 - 80.91 - 84.00 |
| ICTUSnet                  | **88.08** - **84.92** - **91.50**    | 86.75 - 83.53 - 90.23 | 85.95 - 83.10 - 89.02 |


## Intended uses & limitations


The model is ready-to-use only for masked language modelling to perform the Fill Mask task (try the inference API or read the next section)


However, the is intended to be fine-tuned on downstream tasks such as Named Entity Recognition or Text Classification.


## Cite


If you use our models, please cite our latest preprint:


```bibtex

@misc{carrino2021biomedical,
      title={Biomedical and Clinical Language Models for Spanish: On the Benefits of Domain-Specific Pretraining in a Mid-Resource Scenario},
      author={Casimiro Pio Carrino and Jordi Armengol-Estapé and Asier Gutiérrez-Fandiño and Joan Llop-Palao and Marc Pàmies and Aitor Gonzalez-Agirre and Marta Villegas},
      year={2021},
      eprint={2109.03570},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

```


If you use our Medical Crawler corpus, please cite the preprint:


```bibtex

@misc{carrino2021spanish,
      title={Spanish Biomedical Crawled Corpus: A Large, Diverse Dataset for Spanish Biomedical Language Models},
      author={Casimiro Pio Carrino and Jordi Armengol-Estapé and Ona de Gibert Bonet and Asier Gutiérrez-Fandiño and Aitor Gonzalez-Agirre and Martin Krallinger and Marta Villegas},
      year={2021},
      eprint={2109.07765},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

```


## How to use


```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("PlanTL-GOB-ES/roberta-base-biomedical-es")

model = AutoModelForMaskedLM.from_pretrained("PlanTL-GOB-ES/roberta-base-biomedical-es")

from transformers import pipeline

unmasker = pipeline('fill-mask', model="PlanTL-GOB-ES/roberta-base-biomedical-es")

unmasker("El único antecedente personal a reseñar era la <mask> arterial.")
```


```
# Output
[
  {
    "sequence": " El único antecedente personal a reseñar era la hipertensión arterial.",
    "score": 0.9855039715766907,
    "token": 3529,
    "token_str": " hipertensión"
  },
  {
    "sequence": " El único antecedente personal a reseñar era la diabetes arterial.",
    "score": 0.0039140828885138035,
    "token": 1945,
    "token_str": " diabetes"
  },
  {
    "sequence": " El único antecedente personal a reseñar era la hipotensión arterial.",
    "score": 0.002484665485098958,
    "token": 11483,
    "token_str": " hipotensión"
  },
  {
    "sequence": " El único antecedente personal a reseñar era la Hipertensión arterial.",
    "score": 0.0023484621196985245,
    "token": 12238,
    "token_str": " Hipertensión"
  },
  {
    "sequence": " El único antecedente personal a reseñar era la presión arterial.",
    "score": 0.0008009297889657319,
    "token": 2267,
    "token_str": " presión"
  }
]
```


## Funding


This work was funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL.


## Disclaimer


The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.


When third  parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence.


In no event shall the owner of the models (SEDIA – State Secretariat for digitalization and artificial intelligence) nor the creator (BSC – Barcelona Supercomputing Center) be liable for any results arising from the use made by third parties of these models.


Los modelos publicados en este repositorio tienen una finalidad generalista y están a disposición de terceros. Estos modelos pueden tener sesgos y/u otro tipo de distorsiones indeseables.


Cuando terceros desplieguen o proporcionen sistemas y/o servicios a otras partes usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) o se conviertan en usuarios de los modelos, deben tener en cuenta que es su responsabilidad mitigar los riesgos derivados de su uso y, en todo caso, cumplir con la normativa aplicable, incluyendo la normativa en materia de uso de inteligencia artificial.


En ningún caso el propietario de los modelos (SEDIA – Secretaría de Estado de Digitalización e Inteligencia Artificial) ni el creador (BSC – Barcelona Supercomputing Center) serán responsables de los resultados derivados del uso que hagan terceros de estos modelos.


model name: roberta-base-biomedical-clinical-es


model author: PlanTL-GOB-ES


task: fill-mask


tags: pytorch, roberta, fill-mask, es, arxiv:2109.03570, arxiv:2109.07765, transformers, biomedical, clinical, spanish, license:apache-2.0, autonlp_compatible, infinity_compatible


language:


- es
license: apache-2.0
tags:
- "national library of spain"
- "spanish"
- "bne"
- "capitel"
- "ner"
datasets:
- "bne"
- "capitel"\
metrics:
- "f1"
inference:
parameters:
aggregation_strategy: "first"


# Spanish RoBERTa-large trained on BNE finetuned for CAPITEL Named Entity Recognition (NER) dataset.


RoBERTa-large-bne is a transformer-based masked language model for the Spanish language. It is based on the [RoBERTa](https://arxiv.org/abs/1907.11692) large model and has been pre-trained using the largest Spanish corpus known to date, with a total of 570GB of clean and deduplicated text processed for this work, compiled from the web crawlings performed by the  [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) from 2009 to 2019.


Original pre-trained model can be found here: https://huggingface.co/BSC-TeMU/roberta-large-bne


## Dataset


The dataset used is the one from the [CAPITEL competition at IberLEF 2020](https://sites.google.com/view/capitel2020) (sub-task 1).


## Evaluation and results


F1 Score: 0.8998


For evaluation details visit our [GitHub repository](https://github.com/PlanTL-GOB-ES/lm-spanish).


## Citing


Check out our paper for all the details: https://arxiv.org/abs/2107.07253


```
@misc{gutierrezfandino2021spanish,
      title={Spanish Language Models},
      author={Asier Gutiérrez-Fandiño and Jordi Armengol-Estapé and Marc Pàmies and Joan Llop-Palao and Joaquín Silveira-Ocampo and Casimiro Pio Carrino and Aitor Gonzalez-Agirre and Carme Armentano-Oller and Carlos Rodriguez-Penagos and Marta Villegas},
      year={2021},
      eprint={2107.07253},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


## Funding


This work was partially funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL, and the Future of Computing Center, a Barcelona Supercomputing Center and IBM initiative (2020).


## Disclaimer


The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.


When third  parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence.


In no event shall the owner of the models (SEDIA – State Secretariat for digitalization and artificial intelligence) nor the creator (BSC – Barcelona Supercomputing Center) be liable for any results arising from the use made by third parties of these models.


Los modelos publicados en este repositorio tienen una finalidad generalista y están a disposición de terceros. Estos modelos pueden tener sesgos y/u otro tipo de distorsiones indeseables.


Cuando terceros desplieguen o proporcionen sistemas y/o servicios a otras partes usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) o se conviertan en usuarios de los modelos, deben tener en cuenta que es su responsabilidad mitigar los riesgos derivados de su uso y, en todo caso, cumplir con la normativa aplicable, incluyendo la normativa en materia de uso de inteligencia artificial.


En ningún caso el propietario de los modelos (SEDIA – Secretaría de Estado de Digitalización e Inteligencia Artificial) ni el creador (BSC – Barcelona Supercomputing Center) serán responsables de los resultados derivados del uso que hagan terceros de estos modelos.


model name: roberta-large-bne-capitel-ner


model author: PlanTL-GOB-ES


task: token-classification


tags: pytorch, roberta, token-classification, es, dataset:bne, dataset:capitel, arxiv:1907.11692, arxiv:2107.07253, transformers, license:apache-2.0, national library of spain, spanish, bne, capitel, ner, autonlp_compatible, infinity_compatible


language:


- es
license: apache-2.0
tags:
- "national library of spain"
- "spanish"
- "bne"
datasets:
- "bne"\
metrics:
- "ppl"
widget:
- text: "Este año las campanadas de La Sexta las presentará <mask>."
- text: "David Broncano es un presentador de La <mask>."
- text: "Gracias a los datos de la BNE se ha podido <mask> este modelo del lenguaje."
- text: "Hay base legal dentro del marco <mask> actual."


# RoBERTa base trained with data from National Library of Spain (BNE)


## Model Description


RoBERTa-base-bne is a transformer-based masked language model for the Spanish language. It is based on the [RoBERTa](https://arxiv.org/abs/1907.11692) base model and has been pre-trained using the largest Spanish corpus known to date, with a total of 570GB of clean and deduplicated text processed for this work, compiled from the web crawlings performed by the  [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) from 2009 to 2019.


## Training corpora and preprocessing


The [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) crawls all .es domains once a year. The training corpus consists of 59TB of WARC files from these crawls, carried out from 2009 to 2019.


To obtain a high-quality training corpus, the corpus has been preprocessed with a pipeline of operations, including among the others, sentence splitting, language detection, filtering of bad-formed sentences and deduplication of repetitive contents. During the process document boundaries are kept. This resulted into 2TB of Spanish clean corpus. Further global deduplication among the corpus is applied, resulting into 570GB of text.


Some of the statistics of the corpus:


| Corpora | Number of documents | Number of tokens | Size (GB) |
|---------|---------------------|------------------|-----------|
| BNE     |         201,080,084 |  135,733,450,668 |     570GB |


## Tokenization and pre-training


The training corpus has been tokenized using a byte version of Byte-Pair Encoding (BPE) used in the original [RoBERTA](https://arxiv.org/abs/1907.11692) model with a vocabulary size of 50,262 tokens. The RoBERTa-base-bne pre-training consists of a masked language model training that follows the approach employed for the RoBERTa base. The training lasted a total of 48 hours with 16 computing nodes each one with 4 NVIDIA V100 GPUs of 16GB VRAM.


## Evaluation and results


For evaluation details visit our [GitHub repository](https://github.com/PlanTL-GOB-ES/lm-spanish).


## Citing


Check out our paper for all the details: https://arxiv.org/abs/2107.07253


```
@misc{gutierrezfandino2021spanish,
      title={Spanish Language Models},
      author={Asier Gutiérrez-Fandiño and Jordi Armengol-Estapé and Marc Pàmies and Joan Llop-Palao and Joaquín Silveira-Ocampo and Casimiro Pio Carrino and Aitor Gonzalez-Agirre and Carme Armentano-Oller and Carlos Rodriguez-Penagos and Marta Villegas},
      year={2021},
      eprint={2107.07253},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


## Funding


This work was funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL.


## Disclaimer


The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.


When third  parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence.


In no event shall the owner of the models (SEDIA – State Secretariat for digitalization and artificial intelligence) nor the creator (BSC – Barcelona Supercomputing Center) be liable for any results arising from the use made by third parties of these models.


Los modelos publicados en este repositorio tienen una finalidad generalista y están a disposición de terceros. Estos modelos pueden tener sesgos y/u otro tipo de distorsiones indeseables.


Cuando terceros desplieguen o proporcionen sistemas y/o servicios a otras partes usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) o se conviertan en usuarios de los modelos, deben tener en cuenta que es su responsabilidad mitigar los riesgos derivados de su uso y, en todo caso, cumplir con la normativa aplicable, incluyendo la normativa en materia de uso de inteligencia artificial.


En ningún caso el propietario de los modelos (SEDIA – Secretaría de Estado de Digitalización e Inteligencia Artificial) ni el creador (BSC – Barcelona Supercomputing Center) serán responsables de los resultados derivados del uso que hagan terceros de estos modelos.


model name: roberta-base-bne


model author: PlanTL-GOB-ES


task: fill-mask


tags: pytorch, roberta, fill-mask, es, dataset:bne, arxiv:1907.11692, arxiv:2107.07253, transformers, license:apache-2.0, national library of spain, spanish, bne, autonlp_compatible, infinity_compatible


language: "ca"
tags:


- masked-lm
- BERTa
- catalan
widget:
- text: "El Català és una llengua molt <mask>."
- text: "Salvador Dalí va viure a <mask>."
- text: "La Costa Brava té les millors <mask> d'Espanya."
- text: "El cacaolat és un batut de <mask>."
- text: "<mask> és la capital de la Garrotxa."
- text: "Vaig al <mask> a buscar bolets."
- text: "Antoni Gaudí vas ser un <mask> molt important per la ciutat."
- text: "Catalunya és una referència en <mask> a nivell europeu."
license: apache-2.0


# BERTa: RoBERTa-based Catalan language model


## BibTeX  citation


If you use any of these resources (datasets or models) in your work, please cite our latest paper:


```bibtex
@inproceedings{armengol-estape-etal-2021-multilingual,
    title = "Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? {A} Comprehensive Assessment for {C}atalan",
    author = "Armengol-Estap{\'e}, Jordi  and
      Carrino, Casimiro Pio  and
      Rodriguez-Penagos, Carlos  and
      de Gibert Bonet, Ona  and
      Armentano-Oller, Carme  and
      Gonzalez-Agirre, Aitor  and
      Melero, Maite  and
      Villegas, Marta",
    booktitle = "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021",
    month = aug,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.findings-acl.437",
    doi = "10.18653/v1/2021.findings-acl.437",
    pages = "4933--4946",
}
```


## Model description


BERTa is a transformer-based masked language model for the Catalan language.
It is based on the [RoBERTA](https://github.com/pytorch/fairseq/tree/master/examples/roberta) base model
and has been trained on a medium-size corpus collected from publicly available corpora and crawlers.


## Training corpora and preprocessing


The training corpus consists of several corpora gathered from web crawling and public corpora.


The publicly available corpora are:


1. the Catalan part of the [DOGC](http://opus.nlpl.eu/DOGC-v2.php) corpus, a set of documents from the Official Gazette of the Catalan Government
1. the [Catalan Open Subtitles](http://opus.nlpl.eu/download.php?f=OpenSubtitles/v2018/mono/OpenSubtitles.raw.ca.gz), a collection of translated movie subtitles
1. the non-shuffled version of the Catalan part of the [OSCAR](https://traces1.inria.fr/oscar/) corpus \\\\cite{suarez2019asynchronous},
a collection of monolingual corpora, filtered from [Common Crawl](https://commoncrawl.org/about/)
1. The [CaWac](http://nlp.ffzg.hr/resources/corpora/cawac/) corpus, a web corpus of Catalan built from the .cat top-level-domain in late 2013
the non-deduplicated version
1. the [Catalan Wikipedia articles](https://ftp.acc.umu.se/mirror/wikimedia.org/dumps/cawiki/20200801/) downloaded on 18-08-2020.


The crawled corpora are:


6. The Catalan General Crawling, obtained by crawling the 500 most popular .cat and .ad domains
6. the Catalan Government Crawling, obtained by crawling the .gencat domain and subdomains, belonging to the Catalan Government
6. the ACN corpus with 220k news items from March 2015 until October 2020, crawled from the [Catalan News Agency](https://www.acn.cat/)


To obtain a high-quality training corpus, each corpus have preprocessed with a pipeline of operations, including among the others,
sentence splitting, language detection, filtering of bad-formed sentences and deduplication of repetitive contents.
During the process, we keep document boundaries are kept.
Finally, the corpora are concatenated and further global deduplication among the corpora is applied.
The final training corpus consists of about 1,8B tokens.


## Tokenization and pretraining


The training corpus has been tokenized using a byte version of [Byte-Pair Encoding (BPE)](https://github.com/openai/gpt-2)
used in the original [RoBERTA](https://github.com/pytorch/fairseq/tree/master/examples/roberta) model with a vocabulary size of 52,000 tokens.
The BERTa pretraining consists of a masked language model training that follows the approach employed for the RoBERTa base model
with the same hyperparameters as in the original work.
The training lasted a total of 48 hours with 16 NVIDIA V100 GPUs of 16GB DDRAM.


## Evaluation


## CLUB benchmark


The BERTa model has been fine-tuned on the downstream tasks of the Catalan Language Understanding Evaluation benchmark (CLUB),
that has been created along with the model.


It contains the following tasks and their related datasets:


1. Part-of-Speech Tagging (POS)

   Catalan-Ancora: from the [Universal Dependencies treebank](https://github.com/UniversalDependencies/UD_Catalan-AnCora) of the well-known Ancora corpus
1. Named Entity Recognition (NER)

   **[AnCora Catalan 2.0.0](https://zenodo.org/record/4762031#.YKaFjqGxWUk)**: extracted named entities from the original [Ancora](https://doi.org/10.5281/zenodo.4762030) version,
filtering out some unconventional ones, like book titles, and transcribed them into a standard CONLL-IOB format
1. Text Classification (TC)

   **[TeCla](https://doi.org/10.5281/zenodo.4627197)**: consisting of 137k news pieces from the Catalan News Agency ([ACN](https://www.acn.cat/)) corpus
1. Semantic Textual Similarity (STS)

   **[Catalan semantic textual similarity](https://doi.org/10.5281/zenodo.4529183)**: consisting of more than 3000 sentence pairs, annotated with the semantic similarity between them,
scraped from the [Catalan Textual Corpus](https://doi.org/10.5281/zenodo.4519349)
1. Question Answering (QA):

   **[ViquiQuAD](https://doi.org/10.5281/zenodo.4562344)**: consisting of more than 15,000 questions outsourced from Catalan Wikipedia randomly chosen from a set of 596 articles that were originally written in Catalan.

   **[XQuAD](https://doi.org/10.5281/zenodo.4526223)**: the Catalan translation of XQuAD, a multilingual collection of manual translations of 1,190 question-answer pairs from English Wikipedia used only as a *test set*


Here are the train/dev/test splits of the datasets:


| Task (Dataset) | Total | Train | Dev  | Test |
|:--|:--|:--|:--|:--|
| NER (Ancora)  |13,581 | 10,628 | 1,427 | 1,526 |
| POS (Ancora)| 16,678 | 13,123 | 1,709 | 1,846 |
| STS         | 3,073 | 2,073 | 500 | 500 |
| TC (TeCla) |  137,775 | 110,203 | 13,786 |  13,786|
| QA (ViquiQuAD) | 14,239  | 11,255  | 1,492  | 1,429 |


*The fine-tuning on downstream tasks have been performed with the HuggingFace [**Transformers**](https://github.com/huggingface/transformers) library*


## Results


Below the evaluation results on the CLUB tasks compared with the multilingual mBERT, XLM-RoBERTa models and
the Catalan WikiBERT-ca model


| Task        | NER (F1)      | POS (F1)   | STS (Pearson)   | TC (accuracy) | QA (ViquiQuAD) (F1/EM)  | QA (XQuAD) (F1/EM) |
| ------------|:-------------:| -----:|:------|:-------|:------|:----|
| BERTa       | **88.13** | **98.97** | **79.73** | **74.16** | **86.97/72.29** | **68.89/48.87** |
| mBERT       | 86.38 | 98.82 | 76.34 | 70.56 | 86.97/72.22 | 67.15/46.51 |
| XLM-RoBERTa | 87.66 | 98.89 | 75.40 | 71.68 | 85.50/70.47 | 67.10/46.42 |
| WikiBERT-ca | 77.66 | 97.60 | 77.18 | 73.22 | 85.45/70.75 | 65.21/36.60 |


## Intended uses & limitations


The model is ready-to-use only for masked language modelling to perform the Fill Mask task (try the inference API or read the next section)
However, the is intended to be fine-tuned on non-generative downstream tasks such as Question Answering, Text Classification or Named Entity Recognition.


## Using BERTa


## Load model and tokenizer


```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("PlanTL-GOB-ES/roberta-base-ca-cased")

model = AutoModelForMaskedLM.from_pretrained("PlanTL-GOB-ES/roberta-base-ca-cased")
```


## Fill Mask task


Below, an example of how to use the masked language modelling task with a pipeline.


```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='PlanTL-GOB-ES/roberta-base-ca-cased')
>>> unmasker("Situada a la costa de la mar Mediterrània, <mask> s'assenta en una plana formada "
             "entre els deltes de les desembocadures dels rius Llobregat, al sud-oest, "
             "i Besòs, al nord-est, i limitada pel sud-est per la línia de costa,"
             "i pel nord-oest per la serralada de Collserola "
             "(amb el cim del Tibidabo, 516,2 m, com a punt més alt) que segueix paral·lela "
             "la línia de costa encaixant la ciutat en un perímetre molt definit.")

[
  {
    "sequence": " Situada a la costa de la mar Mediterrània, <mask> s'assenta en una plana formada "
                "entre els deltes de les desembocadures dels rius Llobregat, al sud-oest, "
                "i Besòs, al nord-est, i limitada pel sud-est per la línia de costa,"
                "i pel nord-oest per la serralada de Collserola "
                "(amb el cim del Tibidabo, 516,2 m, com a punt més alt) que segueix paral·lela "
                "la línia de costa encaixant la ciutat en un perímetre molt definit.",
    "score": 0.4177263379096985,
    "token": 734,
    "token_str": " Barcelona"
  },
  {
    "sequence": " Situada a la costa de la mar Mediterrània, <mask> s'assenta en una plana formada "
                "entre els deltes de les desembocadures dels rius Llobregat, al sud-oest, "
                "i Besòs, al nord-est, i limitada pel sud-est per la línia de costa,"
                "i pel nord-oest per la serralada de Collserola "
                "(amb el cim del Tibidabo, 516,2 m, com a punt més alt) que segueix paral·lela "
                "la línia de costa encaixant la ciutat en un perímetre molt definit.",
    "score": 0.10696165263652802,
    "token": 3849,
    "token_str": " Badalona"
  },
  {
    "sequence": " Situada a la costa de la mar Mediterrània, <mask> s'assenta en una plana formada "
                "entre els deltes de les desembocadures dels rius Llobregat, al sud-oest, "
                "i Besòs, al nord-est, i limitada pel sud-est per la línia de costa,"
                "i pel nord-oest per la serralada de Collserola "
                "(amb el cim del Tibidabo, 516,2 m, com a punt més alt) que segueix paral·lela "
                "la línia de costa encaixant la ciutat en un perímetre molt definit.",
    "score": 0.08135009557008743,
    "token": 19349,
    "token_str": " Collserola"
  },
  {
   "sequence": " Situada a la costa de la mar Mediterrània, <mask> s'assenta en una plana formada "
                "entre els deltes de les desembocadures dels rius Llobregat, al sud-oest, "
                "i Besòs, al nord-est, i limitada pel sud-est per la línia de costa,"
                "i pel nord-oest per la serralada de Collserola "
                "(amb el cim del Tibidabo, 516,2 m, com a punt més alt) que segueix paral·lela "
                "la línia de costa encaixant la ciutat en un perímetre molt definit.",
    "score": 0.07330769300460815,
    "token": 4974,
    "token_str": " Terrassa"
  },
  {
    "sequence": " Situada a la costa de la mar Mediterrània, <mask> s'assenta en una plana formada "
                "entre els deltes de les desembocadures dels rius Llobregat, al sud-oest, "
                "i Besòs, al nord-est, i limitada pel sud-est per la línia de costa,"
                "i pel nord-oest per la serralada de Collserola "
                "(amb el cim del Tibidabo, 516,2 m, com a punt més alt) que segueix paral·lela "
                "la línia de costa encaixant la ciutat en un perímetre molt definit.",
    "score": 0.03317456692457199,
    "token": 14333,
    "token_str": " Gavà"
  }
]
```


This model was originally published as [bsc/roberta-base-ca-cased](https://huggingface.co/bsc/roberta-base-ca-cased).


## Funding


This work was funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL.


## Disclaimer


The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.


When third  parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence.


In no event shall the owner of the models (SEDIA – State Secretariat for digitalization and artificial intelligence) nor the creator (BSC – Barcelona Supercomputing Center) be liable for any results arising from the use made by third parties of these models.


Los modelos publicados en este repositorio tienen una finalidad generalista y están a disposición de terceros. Estos modelos pueden tener sesgos y/u otro tipo de distorsiones indeseables.


Cuando terceros desplieguen o proporcionen sistemas y/o servicios a otras partes usando alguno de estos modelos (o utilizando sistemas basados en estos modelos) o se conviertan en usuarios de los modelos, deben tener en cuenta que es su responsabilidad mitigar los riesgos derivados de su uso y, en todo caso, cumplir con la normativa aplicable, incluyendo la normativa en materia de uso de inteligencia artificial.


En ningún caso el propietario de los modelos (SEDIA – Secretaría de Estado de Digitalización e Inteligencia Artificial) ni el creador (BSC – Barcelona Supercomputing Center) serán responsables de los resultados derivados del uso que hagan terceros de estos modelos.


model name: roberta-base-ca


model author: PlanTL-GOB-ES


task: fill-mask


tags: pytorch, roberta, fill-mask, ca, transformers, masked-lm, BERTa, catalan, license:apache-2.0, autonlp_compatible, infinity_compatible


language:


- es
license: apache-2.0
tags:
- "national library of spain"
- "spanish"
- "bne"
- "qa"
- "question answering"
datasets:
- "PlanTL-GOB-ES/SQAC"\
metrics:
- "f1"


# Spanish RoBERTa-large trained on BNE finetuned for Spanish Question Answering Corpus (SQAC) dataset.


RoBERTa-large-bne is a transformer-based masked language model for the Spanish language. It is based on the [RoBERTa](https://arxiv.org/abs/1907.11692) large model and has been pre-trained using the largest Spanish corpus known to date, with a total of 570GB of clean and deduplicated text processed for this work, compiled from the web crawlings performed by the  [National Library of Spain (Biblioteca Nacional de España)](http://www.bne.es/en/Inicio/index.html) from 2009 to 2019.


Original pre-trained model can be found here: https://huggingface.co/BSC-TeMU/roberta-large-bne


## Dataset


The dataset used is the [SQAC corpus](https://huggingface.co/datasets/PlanTL-GOB-ES/SQAC).


## Evaluation and results


F1 Score: 0.7993 (average of 5 runs).


For evaluation details visit our [GitHub repository](https://github.com/PlanTL-GOB-ES/lm-spanish).


## Citing


Check out our paper for all the details: https://arxiv.org/abs/2107.07253


```
@misc{gutierrezfandino2021spanish,
      title={Spanish Language Models},
      author={Asier Gutiérrez-Fandiño and Jordi Armengol-Estapé and Marc Pàmies and Joan Llop-Palao and Joaquín Silveira-Ocampo and Casimiro Pio Carrino and Aitor Gonzalez-Agirre and Carme Armentano-Oller and Carlos Rodriguez-Penagos and Marta Villegas},
      year={2021},
      eprint={2107.07253},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


## Funding


This work was partially funded by the Spanish State Secretariat for Digitalization and Artificial Intelligence (SEDIA) within the framework of the Plan-TL, and the Future of Computing Center, a Barcelona Supercomputing Center and IBM initiative (2020).


## Disclaimer


model name: roberta-large-bne-sqac


model author: PlanTL-GOB-ES


task: question-answering


tags: pytorch, roberta, question-answering, es, dataset:PlanTL-GOB-ES/SQAC, arxiv:1907.11692, arxiv:2107.07253, transformers, license:apache-2.0, national library of spain, spanish, bne, qa, question answering, autonlp_compatible, infinity_compatible


tags:


- conversational


#StupidEdwin


model name: StupidEdwin


model author: Snaky


task: conversational


tags: pytorch, gpt2, text-generation, transformers, conversational


language: lt
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: XLSR Wav2Vec2 Lithuanian by Enes Burak Dundar
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice lt
type: common_voice
args: lt
metrics:
    - name: Test WER
type: wer
value: 35.87


# Wav2Vec2-Large-XLSR-53-Lithuanian


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Lithuanian using the [Common Voice](https://huggingface.co/datasets/common_voice)
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows: