|
--- |
|
language: es |
|
thumbnail: https://i.imgur.com/DUlT077.jpg |
|
widget: |
|
- text: "España es un país muy <mask> en la UE" |
|
--- |
|
|
|
# RuPERTa: the Spanish RoBERTa 🎃<img src="https://abs-0.twimg.com/emoji/v2/svg/1f1ea-1f1f8.svg" alt="spain flag" width="25"/> |
|
|
|
RuPERTa-base (uncased) is a [RoBERTa model](https://github.com/pytorch/fairseq/tree/master/examples/roberta) trained on a *uncased* verison of [big Spanish corpus](https://github.com/josecannete/spanish-corpora). |
|
RoBERTa iterates on BERT's pretraining procedure, including training the model longer, with bigger batches over more data; removing the next sentence prediction objective; training on longer sequences; and dynamically changing the masking pattern applied to the training data. |
|
The architecture is the same as `roberta-base`: |
|
|
|
`roberta.base:` **RoBERTa** using the **BERT-base architecture 125M** params |
|
|
|
## Benchmarks 🧾 |
|
WIP (I continue working on it) 🚧 |
|
|
|
| Task/Dataset | F1 | Precision | Recall | Fine-tuned model | Reproduce it | |
|
| -------- | ----: | --------: | -----: | --------------------------------------------------------------------------------------: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | |
|
| POS | 97.39 | 97.47 | 97.32 | [RuPERTa-base-finetuned-pos](https://huggingface.co/mrm8488/RuPERTa-base-finetuned-pos) | [](https://colab.research.google.com/github/mrm8488/shared_colab_notebooks/blob/master/RuPERTa_base_finetuned_POS.ipynb) |
|
| NER | 77.55 | 75.53 | 79.68 | [RuPERTa-base-finetuned-ner](https://huggingface.co/mrm8488/RuPERTa-base-finetuned-ner) | |
|
| SQUAD-es v1 | to-do | | |[RuPERTa-base-finetuned-squadv1](https://huggingface.co/mrm8488/RuPERTa-base-finetuned-squadv1) |
|
| SQUAD-es v2 | to-do | | |[RuPERTa-base-finetuned-squadv2](https://huggingface.co/mrm8488/RuPERTa-base-finetuned-squadv2) |
|
|
|
## Model in action 🔨 |
|
|
|
### Usage for POS and NER 🏷 |
|
|
|
```python |
|
import torch |
|
from transformers import AutoModelForTokenClassification, AutoTokenizer |
|
|
|
id2label = { |
|
"0": "B-LOC", |
|
"1": "B-MISC", |
|
"2": "B-ORG", |
|
"3": "B-PER", |
|
"4": "I-LOC", |
|
"5": "I-MISC", |
|
"6": "I-ORG", |
|
"7": "I-PER", |
|
"8": "O" |
|
} |
|
|
|
tokenizer = AutoTokenizer.from_pretrained('mrm8488/RuPERTa-base-finetuned-ner') |
|
model = AutoModelForTokenClassification.from_pretrained('mrm8488/RuPERTa-base-finetuned-ner') |
|
|
|
text ="Julien, CEO de HF, nació en Francia." |
|
|
|
input_ids = torch.tensor(tokenizer.encode(text)).unsqueeze(0) |
|
|
|
outputs = model(input_ids) |
|
last_hidden_states = outputs[0] |
|
|
|
for m in last_hidden_states: |
|
for index, n in enumerate(m): |
|
if(index > 0 and index <= len(text.split(" "))): |
|
print(text.split(" ")[index-1] + ": " + id2label[str(torch.argmax(n).item())]) |
|
|
|
# Output: |
|
''' |
|
Julien,: I-PER |
|
CEO: O |
|
de: O |
|
HF,: B-ORG |
|
nació: I-PER |
|
en: I-PER |
|
Francia.: I-LOC |
|
''' |
|
``` |
|
|
|
For **POS** just change the `id2label` dictionary and the model path to [mrm8488/RuPERTa-base-finetuned-pos](https://huggingface.co/mrm8488/RuPERTa-base-finetuned-pos) |
|
|
|
### Fast usage for LM with `pipelines` 🧪 |
|
|
|
```python |
|
from transformers import AutoModelWithLMHead, AutoTokenizer |
|
model = AutoModelWithLMHead.from_pretrained('mrm8488/RuPERTa-base') |
|
tokenizer = AutoTokenizer.from_pretrained("mrm8488/RuPERTa-base", do_lower_case=True) |
|
|
|
from transformers import pipeline |
|
|
|
pipeline_fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer) |
|
|
|
pipeline_fill_mask("España es un país muy <mask> en la UE") |
|
``` |
|
|
|
```json |
|
[ |
|
{ |
|
"score": 0.1814306527376175, |
|
"sequence": "<s> españa es un país muy importante en la ue</s>", |
|
"token": 1560 |
|
}, |
|
{ |
|
"score": 0.024842597544193268, |
|
"sequence": "<s> españa es un país muy fuerte en la ue</s>", |
|
"token": 2854 |
|
}, |
|
{ |
|
"score": 0.02473250962793827, |
|
"sequence": "<s> españa es un país muy pequeño en la ue</s>", |
|
"token": 2948 |
|
}, |
|
{ |
|
"score": 0.023991240188479424, |
|
"sequence": "<s> españa es un país muy antiguo en la ue</s>", |
|
"token": 5240 |
|
}, |
|
{ |
|
"score": 0.0215945765376091, |
|
"sequence": "<s> españa es un país muy popular en la ue</s>", |
|
"token": 5782 |
|
} |
|
] |
|
``` |
|
|
|
## Acknowledgments |
|
|
|
I thank [🤗/transformers team](https://github.com/huggingface/transformers) for answering my doubts and Google for helping me with the [TensorFlow Research Cloud](https://www.tensorflow.org/tfrc) program. |
|
|
|
> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488) |
|
|
|
> Made with <span style="color: #e25555;">♥</span> in Spain |
|
|
