How to use this model directly from the
from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("iarfmoose/roberta-small-bulgarian") model = AutoModelForMaskedLM.from_pretrained("iarfmoose/roberta-small-bulgarian")
This model can be used for cloze tasks (masked language modeling) or finetuned on other tasks in Bulgarian.
The training data is unfiltered text from the internet and may contain all sorts of biases.
This model was trained on the following data:
The model was pretrained using a masked language-modeling objective with dynamic masking as described here
It was trained for 160k steps. The batch size was limited to 8 due to GPU memory limitations.