metadata

language:
  - it
pipeline_tag: fill-mask

This model (based on bert-tiny) has been trained from scratch on 5k italian Wikipedia articles and on a wide collection of italian words and dictionary definitions.

The project is still a work in progress, new versions will come soon.

Training

epochs: 200
lr: 1e-5
optim: AdamW
weight_decay: 1e-3

Eval

perplexity: 50