File size: 2,316 Bytes
8226539 8b507ed 8226539 283c62c 8226539 95cd723 8226539 f7396e3 95cd723 8226539 95cd723 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 |
---
language:
- it
pipeline_tag: fill-mask
library_name: transformers
widget:
- example_title: Esempio 1
text: >-
Più tardi, durante la sua carriera, la gravidanza la costrinse a cancellare
le sue apparizioni programmate nell'autunno del 1935. Il 29 marzo 1936 entrò
in una clinica di Londra in [MASK] della nascita del suo bimbo
- example_title: Esempio 2
text: >-
Nella stagione 2017-2018 il Karlsruhe, allenato da Alois Schwartz, [MASK] il
campionato di 3. Liga al 3º posto.
- example_title: Esempio 3
text: >-
Conchita Supervía, nome alla nascita María de la Concepción Supervía Pascual
(Barcellona, 8 dicembre 1895[1] – Londra, 30 marzo 1936), è [MASK] un
mezzosoprano spagnolo molto famosa nel panorama operistico europeo e
statunitense, attiva anche in campo concertistico
license: apache-2.0
datasets:
- wikipedia
metrics:
- perplexity
---
**bertina-3M** is an italian foundational model based on bert, pretrained from scratch on 30GB of italian Wikipedia articles (10M sentences, 329M tokens).
It has 3M parameters and uses a 512 context window size.
The project is still a work in progress, new versions will come with time.
Use it as a foundational model to be finetuned on specific italian tasks.
# Training
- epochs: 4
- lr: 4e-4
- optim: AdamW (beta_1=0.8)
- weight_decay: 1e-2
- Dev set perplexity: 19 (it's a 12MB model!)
# Evaluation (UINAUIL)
Following the [UINAUIL setup](https://github.com/valeriobasile/uinauil/tree/main) we can summarise the following results on BERTINA-3M:
**CLASSIFICATION TASKS**
```
task,type,p,r,f1,acc
haspeede,classification,0.699,0.687,0.680,0.685
ironita,classification,0.701,0.701,0.701,0.701
sentipolc,classification,0.649,0.588,0.587,0.560
```
**ENTAILMENT TASKS**
```
task,type,p,r,f1,acc
textualentailment,entailment,0.423,0.530,0.401,0.530
```
**SEQUENCE TASKS**
```
task,type,acc
eventi,NER,0.835
facta,NER,0.967
```
# License
BERTINA-3M can be freely used for research and commercial purposes.
# Citation
If you're using BERTINA-3M in your scientific work, please cite with:
```
@misc{
Sciancalepore,
title={mascit/bertina-3M},
url={https://huggingface.co/mascIT/bertina-3M},
journal={mascIT/bertina-3M · Hugging Face},
publisher={mascIT},
author={Sciancalepore, Mauro}
}
``` |