roberta-eus-cc100-base-cased upload

README.md

@@ -1,3 +1,41 @@
----
-license: cc-by-nc-4.0
----
+---
+language: eu
+license: cc-by-nc-4.0
+tags:
+- basque
+- roberta
+---
+
+# Roberta-eus cc100 base cased
+
+This is a RoBERTa model for Basque model presented in [Does corpus quality really matter for low-resource languages?](https://arxiv.org/abs/2203.08111). There are several models for Basque using the RoBERTa architecture, using different corpora:
+
+- roberta-eus-euscrawl-base-cased: Basque RoBERTa model trained on Euscrawl, a corpus created using tailored crawling from Basque sites, and which is distributed under a CC-BY license. EusCrawl It contains 12,528k documents and 423M tokens.
+- roberta-eus-euscrawl-large-cased: RoBERTa large trained on EusCrawl.
+- roberta-eus-mC4-base-cased: Basque RoBERTa model trained on the Basque portion of mc4 dataset.
+- roberta-eus-CC100-base-cased: Basque RoBERTa model trained on  Basque portion of cc100 dataset.
+
+The models have been tested on five different downstream tasks for Basque: Topic classification, Sentiment analysis, Stance detection, Named Entity Recognition (NER), and Question Answering (refer to the [paper](https://arxiv.org/abs/2203.08111) for more details). See summary of results below:
+
+
+| Model                            | Topic class. | Sentiment | Stance det. |     NER  |     QA   | Average  |
+|----------------------------------|--------------|-----------|-------------|----------|----------|----------|
+| roberta-eus-euscrawl-base-cased  |         76.2 |      77.7 |        57.4 |    86.8  |    34.6  |    66.5  |
+| roberta-eus-euscrawl-large-cased |     **77.6** |      78.8 |        62.9 | **87.2** | **38.3** | **69.0** |
+| roberta-eus-mC4-base-cased       |         75.3 |  **80.4** |        59.1 |    86.0  |    35.2  |    67.2  |
+| roberta-eus-CC100-base-cased     |         76.2 |      78.8 |    **63.4** |    85.2  |    35.8  |    67.9  |
+
+
+If you use any of these models, please cite the following paper:
+
+```
+@misc{artetxe2022euscrawl,
+ title={Does corpus quality really matter for low-resource languages?},
+ author={Mikel Artetxe, Itziar Aldabe, Rodrigo Agerri,
+         Olatz Perez-de-Viñaspre, Aitor Soroa},
+ year={2022},
+ eprint={2203.08111},
+ archivePrefix={arXiv},
+ primaryClass={cs.CL}
+}
+```

config.json

@@ -0,0 +1,23 @@
+{
+  "architectures": [
+    "RobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "type_vocab_size": 1,
+  "vocab_size": 50005
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ebbd5df103ea564b6fc2ab110c17288ba259df9b593ea154013f05be06fc2db
+size 651891543

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a9e5b2e29cc7476d1df3e44efb38c20d3974d70a0c2dbfbf1c9edc184a5d86f
+size 1156403

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "pad_token": "<pad>", "cls_token": "<s>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true}}

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case":false, "bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "cls_token": "<s>", "pad_token": "<pad>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "sp_model_kwargs": {}, "tokenizer_class": "XLMRobertaTokenizer"}