ixa-ehu commited on
Commit
4dab3e4
1 Parent(s): 3970e17

roberta-eus-cc100-base-cased upload

Browse files

README.md CHANGED
@@ -1,3 +1,41 @@
1
- ---
2
- license: cc-by-nc-4.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: eu
3
+ license: cc-by-nc-4.0
4
+ tags:
5
+ - basque
6
+ - roberta
7
+ ---
8
+
9
+ # Roberta-eus cc100 base cased
10
+
11
+ This is a RoBERTa model for Basque model presented in [Does corpus quality really matter for low-resource languages?](https://arxiv.org/abs/2203.08111). There are several models for Basque using the RoBERTa architecture, using different corpora:
12
+
13
+ - roberta-eus-euscrawl-base-cased: Basque RoBERTa model trained on Euscrawl, a corpus created using tailored crawling from Basque sites, and which is distributed under a CC-BY license. EusCrawl It contains 12,528k documents and 423M tokens.
14
+ - roberta-eus-euscrawl-large-cased: RoBERTa large trained on EusCrawl.
15
+ - roberta-eus-mC4-base-cased: Basque RoBERTa model trained on the Basque portion of mc4 dataset.
16
+ - roberta-eus-CC100-base-cased: Basque RoBERTa model trained on Basque portion of cc100 dataset.
17
+
18
+ The models have been tested on five different downstream tasks for Basque: Topic classification, Sentiment analysis, Stance detection, Named Entity Recognition (NER), and Question Answering (refer to the [paper](https://arxiv.org/abs/2203.08111) for more details). See summary of results below:
19
+
20
+
21
+ | Model | Topic class. | Sentiment | Stance det. | NER | QA | Average |
22
+ |----------------------------------|--------------|-----------|-------------|----------|----------|----------|
23
+ | roberta-eus-euscrawl-base-cased | 76.2 | 77.7 | 57.4 | 86.8 | 34.6 | 66.5 |
24
+ | roberta-eus-euscrawl-large-cased | **77.6** | 78.8 | 62.9 | **87.2** | **38.3** | **69.0** |
25
+ | roberta-eus-mC4-base-cased | 75.3 | **80.4** | 59.1 | 86.0 | 35.2 | 67.2 |
26
+ | roberta-eus-CC100-base-cased | 76.2 | 78.8 | **63.4** | 85.2 | 35.8 | 67.9 |
27
+
28
+
29
+ If you use any of these models, please cite the following paper:
30
+
31
+ ```
32
+ @misc{artetxe2022euscrawl,
33
+ title={Does corpus quality really matter for low-resource languages?},
34
+ author={Mikel Artetxe, Itziar Aldabe, Rodrigo Agerri,
35
+ Olatz Perez-de-Viñaspre, Aitor Soroa},
36
+ year={2022},
37
+ eprint={2203.08111},
38
+ archivePrefix={arXiv},
39
+ primaryClass={cs.CL}
40
+ }
41
+ ```
config.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "RobertaForMaskedLM"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.1,
6
+ "bos_token_id": 0,
7
+ "eos_token_id": 2,
8
+ "gradient_checkpointing": false,
9
+ "hidden_act": "gelu",
10
+ "hidden_dropout_prob": 0.1,
11
+ "hidden_size": 768,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 3072,
14
+ "layer_norm_eps": 1e-05,
15
+ "max_position_embeddings": 514,
16
+ "model_type": "roberta",
17
+ "num_attention_heads": 12,
18
+ "num_hidden_layers": 12,
19
+ "pad_token_id": 1,
20
+ "position_embedding_type": "absolute",
21
+ "type_vocab_size": 1,
22
+ "vocab_size": 50005
23
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ebbd5df103ea564b6fc2ab110c17288ba259df9b593ea154013f05be06fc2db
3
+ size 651891543
sentencepiece.bpe.model ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6a9e5b2e29cc7476d1df3e44efb38c20d3974d70a0c2dbfbf1c9edc184a5d86f
3
+ size 1156403
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
1
+ {"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "pad_token": "<pad>", "cls_token": "<s>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true}}
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
1
+ {"do_lower_case":false, "bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "cls_token": "<s>", "pad_token": "<pad>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "sp_model_kwargs": {}, "tokenizer_class": "XLMRobertaTokenizer"}